PHOTO CREDIT: MWANGI KIRUBI EVALUATION External Performance Evaluation of the East Africa Energy Program (EAEP) June 2023 This publication was produced with the support of the United States Agency for International Development (USAID) Power Africa initiative under the terms of the Power Africa Contract Evaluation Services Task Order No. 720-674-23-F-00001. EXTERNAL PERFORMANCE EVALUATION OF EAEP | JUNE 2023 | i Report Title: External Performance Evaluation of the East Africa Energy Program (EAEP) Lead Author: Matthew Savage Supporting Authors: Karin Sosis, Benjamin Kachero, Sujata Ram, Nicole Economu, and Isabel Morford-Cheibub Submitted by: ICF Activity Title: Power Africa Contract Evaluation Services USAID Task Order Number: 720-674-23-F-00001 Completion Date: June 9, 2023 Photo Caption: A Kenya Power employee working on a grid connection at Kasarani area, Kenya. The authors’ views expressed in this publication do not necessarily reflect the views of the United States Agency for International Development or the United States Government. EXTERNAL PERFORMANCE EVALUATION OF EAEP | JUNE 2023 | ii ACKNOWLEDGEMENTS On behalf of the ICF Evaluation Team, I, Mark Wagner (Evaluation Oversight and Quality Assurance Officer), would like to extend my sincere appreciation to the many USAID staff, Implementing Partners, host country government representatives, development partners, utility and energy companies, and other key informants who contributed their valuable time and insights to support the evaluation process. I am particularly grateful to the USAID staff and Implementing Partners who connected our team to program stakeholders, provided key documentation, and improved our understanding of Power Africa’s experiences, successes, and challenges, as well as to the Contracting Officer’s Representative, Yugi Nair, and the Alternate Contracting Officer’s Representative, Ngonidzashe Makaza, for their continued support and the trust they placed in us throughout the process. EXTERNAL PERFORMANCE EVALUATION OF EAEP | JUNE 2023 | iii TABLE OF CONTENTS ACRONYMS AND ABBREVIATIONS IV EXECUTIVE SUMMARY V 1. INTRODUCTION 1 1.1 BACKGROUND ON EVALUATION 1 1.2 ORGANIZATION OF THE REPORT 2 2. EVALUATION METHODS 3 2.1 EVALUATION QUESTIONS AND SCOPE 3 2.2 EVALUATION METHODOLOGY 4 2.3 EVALUATION LIMITATIONS 4 3. FINDINGS 5 3.1 RELEVANCE AND COHERENCE 5 3.2 PROGRAM DELIVERY: EFFICIENCY AND EFFECTIVENESS 6 3.3 DELIVERY OF PROGRAM RESULTS: OUTCOMES, IMPACT, SUSTAINABILITY 10 3.4 KEY BARRIERS TO RESULTS DELIVERY 13 4. CONCLUSIONS AND RECOMMENDATIONS 15 4.1 CONCLUSIONS/LESSONS LEARNED 15 4.2 RECOMMENDATIONS 18 ANNEXES 22 ANNEX A: EVALUATION STATEMENT OF WORK 22 ANNEX B: EVALUATION DESIGN AND MATRIX 26 ANNEX C: DATA COLLECTION INSTRUMENTS 38 ANNEX D: SOURCES OF INFORMATION 42 ANNEX E: DISCLOSURE OF ANY CONFLICTS OF INTEREST 44 ANNEX F: STATEMENTS OF DIFFERENCES (IF APPLICABLE) 44 EXTERNAL PERFORMANCE EVALUATION OF EAEP | JUNE 2023 | iv ACRONYMS AND ABBREVIATIONS AfDB African Development Bank COR Contracting Officer’s Representative COVID-19 Coronavirus SARS-CoV-2 2019 DFI Development Finance Institution DOAG Development Objective Agreement EAEP East Africa Energy Program EAPP Eastern Africa Power Pool EEU Ethiopian Electricity Utility HICD Human and Institutional Capacity Development IP Implementing Partner KPLC Kenya Power and Lighting Company MDB Multilateral Development Bank MEL Monitoring, Evaluation, and Learning MOU Memorandum of Understanding NELSAP Nile Equatorial Lakes Subsidiary Action Program NPSP Nigeria Power Sector Program PA Power Africa PAIS Power Africa Information System PAOP Power Africa Off-Grid Program PATRP Power Africa Transactions and Reforms Program PATT Power Africa Tracking Tool PPA Power Purchase Agreement PPP Public-Private Partnership RTI RTI International SSA Sub-Saharan Africa TOC Theory of Change USAID United States Agency for International Development USTDA United States Trade and Development Agency WAEP West Africa Energy Program WiAP Women in African Power WB World Bank EXTERNAL PERFORMANCE EVALUATION OF EAEP | JUNE 2023 | v EXECUTIVE SUMMARY EVALUATION OVERVIEW The United States Agency for International Development (USAID) Power Africa (PA) Program Office contracted ICF to conduct external performance evaluations of four PA programs, including the East Africa Energy Program (EAEP), to improve program performance management and efficiency. The lessons drawn from these evaluations are directed toward technical teams, activity managers, and senior management. ICF developed an overarching evaluation design framework for all four PA programs, which included five focus areas: (1) relevance and coherence of the theory of change and program design; (2) efficiency and effectiveness of program delivery; (3) outcomes, impact, and sustainability of program delivery; (4) key barriers to the delivery of results; and (5) lessons learned and recommendations. The evaluation team employed a combination of data collection and analysis methods. The team reviewed relevant program and external documents and triangulated findings from the desk review with data collected through 59 semi-structured key informant interviews. SUMMARY OF EAEP EAEP is an implementing mechanism that was awarded to RTI International in December 2018. It is designed to optimize the power supply, increase grid-based power connections, strengthen utilities and other power sector entities, and increase regional power trade. EAEP’s theory of change (TOC) assumes that by expanding affordable and reliable electricity services, EAEP will support development priorities, including inclusive growth, increased security, and improved health and education outcomes, through community engagement. The project is implemented in ten countries in East Africa. These include Burundi, the Democratic Republic of the Congo (DRC), Djibouti, Eritrea, Ethiopia, Kenya, Rwanda, Somalia (including Puntland and Somaliland), Tanzania, and Uganda. The TOC for this project assumed that by expanding affordable and reliable electricity services in East Africa, EAEP interventions would aid in the achievement of the goal of supporting development priorities, including inclusive economic growth, increased security, and improved health and education outcomes in the region. The implementing mechanism was expected to address critical opportunities across the following four objectives: (1) optimized power supply, (2) increased grid-based power connections, (3) strengthened utilities and other power sector entities, and (4) increased regional power trade. KEY TAKEAWAYS The Power Africa model, as implemented through EAEP, offers a sensible and flexible model for power sector engagement in East Africa. The program structure has allowed for sufficient breadth and geographical flexibility to identify demand-led opportunities across the region and work in an adaptive manner with a range of beneficiaries. Despite the pipeline legacy of the Power Africa Transactions and Reforms Program, the level of political, economic, and social disruption within the region has underlined the need for a degree of flexibility in how and where targets are delivered. EXTERNAL PERFORMANCE EVALUATION OF EAEP | JUNE 2023 | vi The contract is a useful guiding framework for prioritization; however, this may be at the expense of more strategic and long-term engagement approaches. Program targets create a healthy set of incentives to identify and develop new opportunities where roadblocks are encountered. However, the need to deliver the contract targets may have taken the focus of the program away from a more systemic approach to addressing complex country challenges with partners in an integrated manner. Perceptions of long-term, in-country continuity and presence are important for building credibility and trust, even if progress may be slow. The implementation team has conducted the program in an efficient manner, with strong management systems and processes to deliver a broad range of interventions. EAEP is universally regarded as having been efficiently and effectively managed under challenging circumstances. Robust management and reporting systems, combined with the use of well-regarded “Pause and Reflect” sessions and the use of advisers, have meant that the program has maintained a strong overview of progress and challenges across its workstreams and has been able to adapt and pivot accordingly. The program has been more successful in reaching some objectives than others. EAEP has been particularly successful in supporting the delivery of new connections, providing the most of any of the Power Africa programs. Its work on gender and utility turnaround also has been successful, with significant performance improvements for those utilities and in those regions where it engaged. However, progress on new generation and transmission capacity (both national and regional) has been slower than expected for a number of reasons at both the project and political levels, including potentially unrealistic target setting at the program outset. The attribution of results to EAEP activities varies in strength across the objectives; however, EAEP has sought to provide robust evidence as to its relative contribution. There are varying levels of contribution and attribution for the reported results of EAEP, with connections having perhaps the weakest overall contribution. Given the robust reporting requirements for EAEP, the program has made a significant effort to detail the linkages between activities and outcomes, with a view toward ensuring that these can be captured by Power Africa in the future as and when transactions reach financial close. LESSONS Timescales for large transactions are unpredictable in the East Africa region due, in part, to complex political, economic, and security environments. Large capital-intensive transactions in generation and transmission infrastructure are, by their nature, complex and can require significant time to reach financial close and also are exposed to changing political, sectoral, and economic circumstances. As a result, large transactions originated by EAEP are highly likely to be grandfathered in any subsequent program and/or supported by other parties. This requires a level of programmatic integration over time; effective handover between program contractors, ensuring continuity and political engagement; and an acknowledgment of the value of both preparing and accelerating transaction progress, as well as achieving financial close. Power Africa is just one of the many stakeholders in the East Africa region that is engaged in policy, transaction support, and utility reform, which requires strong partnerships and coordination. To achieve large-scale change, a strong focus on stakeholder relationships and coordinated sustained coalitions is necessary. In this context, the technical assistance provided by EAEP is valuable. EXTERNAL PERFORMANCE EVALUATION OF EAEP | JUNE 2023 | vii However, Power Africa needs to establish close relationships with large-scale financiers to facilitate transactions and with political partners to help overcome policy and regulatory obstacles. Because many development finance institutions (DFIs) provide technical assistance, EAEP must identify its relative niche while seeking to add value to processes rather than duplicating/complicating them. Upstream planning support is critical in unlocking effective downstream transactions. EAEP has indirectly generated a significant proportion of transaction impact through its upstream planning support to policymakers and other national or regional bodies. This support has enabled the creation of frameworks that allow for more confident investment decision-making by policymakers and utilities. Currently, the program is following potential opportunities for masterplans and updates in Kenya (distribution), Tanzania, and Uganda. The contribution of this work to downstream investments needs to be strengthened and recognized. EAEP has added value in identifying common challenges and creating standardized approaches to allow for scale and consistency. EAEP’s ability to add value has been significantly enhanced by its use of diagnostics and the creation of standardized technical assistance approaches. The program identified common challenges among different types of beneficiaries, particularly at the utility level. This allowed EAEP to tailor engagement processes that leveraged core approaches and knowledge effectively. Discussions around additional generation require confidence among decision-makers regarding the robustness of future demand. There is a strong need for partners to have confidence in future growth in power demand (e.g., expanding access and distribution, supporting larger energy￾intensive baseload customers) to unlock generation decision-making. A greater focus on strengthening connections and demand visibility is key to unlocking generation investment in some countries. Institutional buy-in and long-term relationships are key to gaining traction on policy and utility reform. Some of EAEP’s most important successes have been built on upfront investment in beneficiary-led models, building trust and ensuring that partners understood the Power Africa approach. Of significant importance is the perception that Power Africa is there to provide long-term, facilitative support rather than offering a one-time solution or a donor-driven agenda. The EAEP team invested heavily in tailoring solutions to need (rather than just delivering off-the-shelf support). Where this has been less successful (e.g., the Tanzania Electric Supply Company Limited), the outcomes and impacts also have suffered. RECOMMENDATIONS EAEP was due to close out at the end of fiscal year 2022, and even with the current extension, there is now a limited opportunity to engage in course correction. Nevertheless, there are several recommendations that flow from the findings and conclusions that might be considered in relation to both the procurement of a new contractor and more detailed implementation design and delivery planning. RECOMMENDATION 1: Create a smarter approach to setting targets and program priorities. The discussion of program priorities and boundaries needs to be undertaken in a smarter manner between Power Africa and the contractor, based on a clear understanding of evolving opportunities and challenges. Early agreement and buy-in with beneficiaries are vital, with USAID Missions EXTERNAL PERFORMANCE EVALUATION OF EAEP | JUNE 2023 | viii and Power Africa needing to play a more proactive role in engaging with key stakeholders and enhancing the political economy of program delivery. • ACTION 1: Ensure that programmatic targets are realistic given the pipeline, scale of resourcing and prevailing market, and policy challenges. To determine whether targets are realistic, USAID should model the new program on the conversation rate and project timeframes within EAEP. • ACTION 2: Increase Power Africa and USAID strategic engagement on key outcomes in the region (beyond numerical targets) and help build a more integrated narrative of change. • ACTION 3: Achieve early buy-in and secure memoranda of understanding between Power Africa, USAID, the implementation team, counterparts and beneficiaries for engagement with clear sense of scope and results. RECOMMENDATION 2: Strengthen the focus on the value of upstream enabling environment work to facilitate downstream transactions with the ability to recognize contributions where evidenced. Upstream enabling environment support has been key in creating a conducive environment for downstream transaction development. Weak decision-making capacity in key partner institutions, compounded by political economy challenges, has created blockages to moving forward with strategic transactions. • ACTION 1: Encourage a renewed focus around upstream enabling environment work (e.g., master planning) in the new phase of programming. • ACTION 2: Facilitate systems that recognize the value of enabling environment work toward downstream targets where supported/validated by beneficiaries. RECOMMENDATION 3: Maintain a focus on expanding connections and increasing engagement regarding demand and productive use. There is currently limited confidence among policymakers with regard to regional power trading as a reliable opportunity to underpin the financial case for investment in generation. Power Africa should continue to target expanding connections because there are significant opportunities to expand networks in Burundi, DRC, Ethiopia, Somalia, Tanzania, and Uganda, as well as explore the value of working with end users on promoting demand stimulation, particularly in countries experiencing a surplus. • ACTION 1: Include a renewed focus on connections as part of the new Power Africa program. • ACTION 2: Explore options for engaging in expanding downstream productive use because this provides an anchor for new generation and utility sustainability. RECOMMENDATION 4: Continue with the embedded advisor model: The embedded advisor model has proven to be critical to EAEP’s success, allowing for more integrated, consistent, and ongoing support to beneficiaries. While some concerns have been expressed around both the cost and the sustainability of the model once the advisor leaves, combined with capacity building and training, it has nevertheless proven to be very effective. • ACTION 1: Explore the continued use of embedded advisors in the new program. RECOMMENDATION 5: Work with DFIs to ensure that there is momentum for delivering large infrastructure investments, in particular, transmission infrastructure projects, in a timely manner. For large critical investments (both generation and regional transmission), there is a EXTERNAL PERFORMANCE EVALUATION OF EAEP | JUNE 2023 | ix significant role played by multilateral development banks and other DFIs in the planning and execution. The positioning of EAEP in the context of these transactions needs to be done in a coordinated manner that ensures added value. • ACTION 1: Ensure Power Africa and implementation team prioritization of DFI-supported transactions. RECOMMENDATION 6: Improve synergies between the program and the Power Africa coalition/strategy. Work at the program level in the region, while engaging selectively with Power Africa partners for task delivery, does not appear to be closely integrated with a higher level coalition approach or a unified strategy among the partners. More could be done to bring together the coalition and the program, particularly as related to strategic transactions and political economy challenges. • ACTION 1: Strengthen proactive work with the Implementing Partner to ensure that synergies between EAEP (and its successor program) and the wider Power Africa coalition are maximized for the delivery of results. RECOMMENDATION 7: Build sustainability plans from an early stage in implementation. EAEP devised its sustainability plans in the third year, whereas it would have been better to co-create these plans from the onset of the program. This collaborative strategy would have facilitated the delineation of EAEP’s objectives and scope, as well as the identification of synergies for achieving the desired outcomes. In future programming, it is important that greater effort is made upfront to build in sustainability considerations. • ACTION 1: Work with the Implementing Partner to ensure that long-term sustainability plans and objectives are adequately reflected from the inception phase, with a common understanding of successful exit benchmarks. RECOMMENDATION 8: Continue and strengthen work on gender; however, broaden the focus on female entrepreneurship and leadership. The program has made significant progress in enhancing gender inclusivity within the power sector by raising awareness and fostering momentum for women’s participation. There is the potential to expand this focus to encompass entrepreneurship and wider economic development, creating opportunities on both the supply and demand sides. EAEP could incorporate this type of training through the contractor. Alternatively, the respective Missions could collaborate with networks or other organizations to provide support for employment and economic growth initiatives. • ACTION 1: Maintain Power Africa approach to enhancing access, focusing on the sustainability of the existing networks, and ongoing support for apprenticeships. • ACTION 2: Include support for female entrepreneurship (both the supply and demand sides), both through the program and by leveraging infrastructure already created by EAEP. RECOMMENDATION 9: Greater operational clarity and timeliness around budgets and contractor status by USAID/Power Africa. EAEP faced significant challenges in managing visibility around annual budgeting processes, as well as a high degree of uncertainty around contractor tax status. Greater focus needs to be placed on minimizing the potential operationally disruptive aspects of the program through early clarification of financing envelopes and flows (not least in order to support EXTERNAL PERFORMANCE EVALUATION OF EAEP | JUNE 2023 | x beneficiary expectations around the scale and timelines of support), as well as the status of the contractor relative to the countries in which it operates. • ACTION 1: Develop a more transparent and predictable funding framework to allow for better strategic planning, resource allocation, and beneficiary support. • ACTION 2: Support Implementing Partners in resolving issues related to taxation by vouching for and explaining the role of the contractor within the United States Government program to the relevant host country authorities. EXTERNAL PERFORMANCE EVALUATION OF EAEP | JUNE 2023 | 1 1. INTRODUCTION 1.1BACKGROUND ON EVALUATION The United States Agency for International Development (USAID) Power Africa (PA) Program Office has contracted ICF to conduct external performance evaluations of four PA programs, including the East Africa Energy Program (EAEP), to improve program performance management and efficiency. PA seeks to increase access to electricity throughout Sub-Saharan Africa (SSA) with the addition of millions of new connections and tens of thousands of megawatts in new and cleaner power generation. To date, PA has delivered significant transformative impacts in the SSA region, not only in expanding energy services but also in benefiting multiple additional development priorities. The lessons drawn from this evaluation are directed toward technical teams, activity managers, and senior management. DESCRIPTION OF PROGRAM EAEP is designed to optimize the power supply, increase grid-based power connections, strengthen utilities and other power sector entities, and increase the regional power trade in East Africa. EAEP’s theory of change (TOC) assumes that by expanding affordable and reliable electricity services, EAEP will support development priorities, including inclusive growth, increased security, and improved health and education outcomes. The program is implemented in ten countries in East Africa. These include Burundi, the Democratic Republic of the Congo (DRC), Djibouti, Eritrea, Ethiopia, Kenya, Rwanda, Somalia (including Puntland and Somaliland), Tanzania, and Uganda. EAEP is an implementing mechanism that was awarded to RTI International (RTI) in December 2018. In each of the ten countries, the implementation team is expected to provide technical assistance and capacity building in the focus countries while collaborating closely with its consortium members: Khulisa Management Services, Tetra Tech, Fieldstone Africa, Center for Climate Strategies, and Norton Rose Fulbright. THEORY OF CHANGE The TOC for this program assumed that by expanding affordable and reliable electricity services in East Africa, EAEP interventions would aid in the achievement of the overall goal of supporting development priorities, including inclusive economic growth, increased security, and improved health and education outcomes in the region. EAEP was expected to address critical opportunities across the following four objectives: 1. Optimized power supply; 2. Increased grid-based power connections; 3. Strengthened utilities and other power sector entities; and, 4. Increased regional power trade. EAEP also was expected to promote gender equity, environmental mitigation and standards, and community engagement across all objectives. The proposed implementation approach aimed to strengthen EXTERNAL PERFORMANCE EVALUATION OF EAEP | JUNE 2023 | 2 and leverage the EAEP regional platform to maximize contact points among regional stakeholders, while simultaneously addressing four common regional dimensions: 1. Workforce development and institutional strengthening via a regional training platform that delivers common proficiency to stakeholders in key technical areas; 2. Improvement in national and regional planning and policy through diagnostic tools, private sector￾driven improvements and data-driven analysis, and the introduction of best practices; 3. Coordination and information sharing through the Power Africa Information System, Women in African Power (WiAP), Eastern Africa Power Pool (EAPP), and other regional institutions; and 4. Facilitation of public-private dialogue through the program’s regional footprint, more embedded utility and transaction advisors to maintain real momentum on deal flow, and relations built with the private sector. The implementation team was expected to provide overall project direction, cross-cutting activities, and coordination with Power Africa and the Kenya donor community across four project offices, ten countries, and five subcontractors by establishing a harmonized “One Team” management approach. Each project office managed one of the four program objectives, with two offices in Nairobi, Kenya, for objectives 1 and 3 and two offices in Addis Ababa, Ethiopia, for objectives 2 and 4. East Africa has the second strongest power sector in SSA, with important gains made over the past decade in utility performance, access to energy (both on- and off-grid), and available generation capacity. Significant progress was made under the Power Africa Transactions and Reforms Program (PATRP), which is Power Africa’s first implementing mechanism that ran from 2013 to 2018. However, regional trends mask the significant remaining challenges across the ten countries targeted by EAEP. National priorities, human and institutional capacity, and the sophistication of systems differ greatly as does the resulting approach required to accompany each nation on its journey to energy equity and more self-reliance. A far greater lagging challenge is regional integration of the technical and human systems that permit the exchange of power. Despite nominal institutions, plans, and dialogue about regional integration, East Africa is still in the initial stages of building a real sustainable energy market, with significant gaps in the required transmission infrastructure, harmonized grid codes and operating protocols, and market platforms for the regional exchange of power. Only when national systems are meaningfully integrated will competition drive down consumer tariffs and improve regional reliability and broader access to energy. EAEP is designed to be implemented by prioritizing cost-effective and direct contact with all stakeholders to facilitate needs assessment and develop dynamic, responsive interventions that fit into a cohesive, integrated, regional approach to systems interconnection and harmonized planning. It seeks to engage in jointly managed projects to facilitate political buy-in and country-led development, and to assist EAEP stakeholders on their journey to self-reliance. 1.2ORGANIZATION OF THE REPORT The remainder of the report is organized as follows: • Section 2 summarizes the evaluation methods, including the questions used to guide this evaluation; EXTERNAL PERFORMANCE EVALUATION OF EAEP | JUNE 2023 | 3 • Section 3 presents the key findings, organized into subtopics addressing relevance and coherence, efficiency and effectiveness of program delivery, outcomes and impacts to date, and key barriers affecting the delivery of results; • Section 4 includes conclusions, lessons learned; and recommendations for the future; and • Annexes contain the statement of work, evaluation design, data collection instruments, and sources of information. 2. EVALUATION METHODS 2.1EVALUATION QUESTIONS AND SCOPE ICF was tasked with conducting a performance evaluation of the USAID-funded program that provides technical services to implement the EAEP task order. The evaluation sought to (1) determine, to date, the extent to which the contract has achieved its intended objectives and outputs; (2) assess the technical and program management of the implementation mechanism; (3) highlight lessons for USAID in facilitating coordination between Implementing Partners (IPs); (4) highlight major gaps and challenges that may require adjustments in program implementation; and (5) inform the design of potential future activities. USAID provided the following illustrative evaluation questions in the statement of work (see Annex A): • In its fourth and final year of project implementation, is EAEP on target to achieve the goals set in the task order? Do the assumptions in the TOC hold true or does the TOC need to be adjusted? If adjustments are needed, please provide concrete recommendations for the types and scope of project implementation adjustments. • Identify achievements, challenges, best practices, lessons learned, and recommendations from program implementation to date for the four project objectives: o Objective 1: Optimized power supply; o Objective 2: Increased grid-based power connections; o Objective 3: Strengthened utilities and other power sector entities; and o Objective 4: Increased regional power trade. • What are the lessons learned from the “one team” management approach across the various countries, teams, and sub-partners in achieving the expected results? What are the recommendations for improving the results for the rest of the program implementation years. • How effective were the IPs’ performance management tools in terms of timeliness, accuracy of reporting, availability of information, tracking target achievement, and so forth? • Evaluate and describe the learning and adapting approach identified by RTI in implementing and documenting learning, knowledge management, and adaptability through its quarterly pause-and￾reflect sessions. How have these learnings supported program implementation? • How has the COVID-19 pandemic affected the project in achieving its implementation milestones and the overall project performance? How did the partner monitor and document performance change? EXTERNAL PERFORMANCE EVALUATION OF EAEP | JUNE 2023 | 4 2.2EVALUATION METHODOLOGY EVALUATION DESIGN ICF developed an overarching evaluation design framework for all four PA programs, which included five focus areas: (1) relevance and coherence of the TOC and program design; (2) efficiency and effectiveness of program delivery; (3) outcomes, impact, and sustainability of program delivery; (4) key barriers to the delivery of results; and (5) lessons learned and recommendations (see the evaluation framework in Annex B). The evaluation sought to understand how the program is addressing ongoing challenges, given the different national priorities; gaps in human and institutional capacity; and the fragmented systems required to reach energy equity. The evaluation examined how the program’s platform addresses the challenge of regional integration of technical and human systems by considering the four regional program dimensions mentioned previously. The evaluation team explored whether and to what extent EAEP has demonstrated results of greater reliability, lower prices, and increased access and whether these outcomes are well correlated to “meaningfully integrated national systems” to achieve results. The team also assessed cross￾cutting issues to determine effectiveness and identify models and areas of improvement across countries. To identify program efficiencies and lessons, the evaluation team reviewed the program’s “One-Team” management approach for lessons that can be shared with USAID and IPs. DATA COLLECTION AND ANALYSIS The evaluation team employed a combination of data collection and analysis methods to assess EAEP in the context of the evaluation matrix. The team started with a desk review of relevant program documents, including program implementation plans, annual work plans, annual and quarterly performance reports, performance monitoring plans and systems, and technical reports. The team also reviewed external publications on the energy sector and other inventions in the similar sub-sector and geographic area to consider other best practices and models. The team triangulated findings from the desk review with data collected through 59 semi-structured interviews with key informants. A full list of these interviews, which were conducted via video platforms (e.g., Google Meet, Zoom), is available in Annex D. See Annex C for the data collection instruments. 2.3EVALUATION LIMITATIONS The evaluation team is confident that it collected sufficient evidence to make strong and robust conclusions. However, certain limitations on the data exist, including the following: • Social response bias. The evaluation team is confident that overall, most interviewees were frank and forthcoming. Nevertheless, it also is important to acknowledge that some were, at least partly, presenting material in a manner that was consistent with their own institutional interests. Such bias is normal in qualitative research and is one reason why it is critical to interview a range of individuals and triangulate the results. The number of key informant interviews (KIIs) conducted across a range of different stakeholder groups allows for improved accuracy. • Insufficient data to evaluate outcomes and impacts. It was not possible to assess the ultimate contributions to long-term resilience and sustainability because these effects will be evidenced over decades, beyond the implementation timeframes of the programs. In order to evaluate the sustainability of interventions past the life of the programs, the evaluation therefore relies on data and qualitative information about outputs and immediate and intermediate outcomes as reported, consistent with the TOC for each program. EXTERNAL PERFORMANCE EVALUATION OF EAEP | JUNE 2023 | 5 3. FINDINGS 3.1RELEVANCE AND COHERENCE EAEP was sensibly framed around four key objectives, and this provided a useful structure around which to organize and deliver Power Africa outcomes. EAEP’s TOC encompassed the entire energy sector chain of generation, transmission, and distribution, and envisaged cooperation with a broad range of stakeholders to deliver its outcomes. EAEP’s structure around four objectives is considered relevant in terms of addressing Power Africa’s priority areas related to on-grid electrification. The model appropriately addressed partner (e.g., utilities, ministries, regulators, private companies) needs and complemented ongoing donor and government efforts to increase electricity access and capacity. The model also allowed for flexibility in adapting to the changing objectives and priorities of partners as markets and the wider political economy evolved. However, the structure around the four objectives also led to an element of siloing in operational delivery, undermining a more integrated and impactful approach, particularly in relation to utility engagement. The TOC, although very high level, generally remained relevant over time; however, there were some changes in delivery priorities and approach during implementation. The overall structure of the program continued to provide a solid foundation for delivery, although there were some changes in priorities and emphasis over time. Priority areas of support changed as a result of COVID-19, which created economic instability, a greater emphasis on affordable versus cost-reflective tariff prices, and less interest in securing generation due to oversupply challenges. The role of different program components also became more integrated over time. For example, the connections and utility performance activities became more aligned, and both contributed significantly to overall connections outcomes, overcoming the initial siloing effects. Likewise, the linkages between upstream enabling environment work and downstream transactions also evolved to become more integrated. EAEP’s approach aimed for a high degree of coherence and complementarity with the broader energy transition efforts. This included collaboration with the USAID Power Africa interagency coalition, donor organizations, and private sector partners. EAEP sought to collaborate with these entities to maximize resources and generate a more significant impact. Key partners included the multilateral development banks (MDBs), such as the African Development Bank (AfDB) and World Bank (WB), where these institutions acted as the lead on large-scale capital infrastructure (e.g., transmission infrastructure). Other development finance institutes (DFIs), such as Agence Française de Développement (AFD), also were important for transaction delivery. EAEP’s role in relation to larger funding institutions is complex; however, the program was able to establish itself primarily as a facilitator and transaction accelerator, adding value to existing DFI capital and technical assistance support. This often involved supporting the government, regulator, or utility to build confidence in the transaction, or supporting utilities and developers to access and accelerate DFI funding (e.g., Rwanda transmission studies for the Rwanda Energy Group and AfDB). The demand-driven model allowed for a degree of regional thematic flexibility to adapt to changing stakeholder needs and political realities. Both the implementation team and the beneficiaries recognized the value of running a demand-driven program broadly informed by stakeholder needs and shaped collaboratively with partner institutions. This flexibility was particularly important given the challenges linked to a range of factors, including pandemics (e.g., COVID-19, Ebola), conflict (Ethiopia), EXTERNAL PERFORMANCE EVALUATION OF EAEP | JUNE 2023 | 6 political instability (e.g., elections), policy and regulatory instability (e.g., tariff, power purchase agreement [PPA] renegotiation), and regional political economy challenges (e.g., Burundi, DRC, Rwanda, Uganda). The demand-led model also was central in creating trust and building longer term relationships with key institutions. The TOC underestimated the importance of political economy, capacity, and oversupply challenges. Although the TOC was directionally robust, it failed to envisage some of the potential challenges associated with the scale and timing of the targets set in the contract, particularly around the delivery of new generation and transmission capacity. Aside from assuming that the existing pipeline from PATRP would serve as the basis for achieving the established targets, the TOC was quite basic and included little articulation of other assumptions. It failed to explicitly identify and articulate potential political economy challenges and did not include strategies for the mitigation of potential risks. Moreover, it possibly underestimated the policy, regulatory, and capacity barriers that significantly contributed to the delay of transactions. Key decision-makers held perceptions of oversupply and unaffordability concerning new generation investments, which were not adequately considered during the planning and target-setting stages. Over time, EAEP pivoted from transaction support to upstream work, with a greater focus on enabling environments to facilitate and unlock downstream investments. However, the timeline associated with upstream work is greater than that of the downstream investments, meaning that the results are likely to appear during the next program’s implementation rather than by the end of the EAEP implementation period. When the program began, there was an assumption that transactions would be primarily driven through technical assistance to developers and other parties, such as through the provision of financial, technical, and legal support. However, it became clear early in the program that while this support can be useful, the key barriers lie in the policy, regulatory, and political economy spheres. As a result, the program pivoted upstream to work with policymakers and regulators to unlock downstream investment (while maintaining a transaction focus). This involved both capacity development and interventions during the planning phase that would allow decision-makers to become comfortable not only with individual investments but also with related endeavors (e.g., around adding renewable capacity, supporting interregional trade and national transmission infrastructure). The program envisaged a core focus on a small number of key markets but evolved over time into a more geographically disbursed program to reflect political and sector realities. Initially, it was assumed that the bulk of the results for the program would be delivered from the pipeline in three key countries (i.e., Ethiopia, Kenya, and Tanzania), primarily building on work undertaken early by PATRP. However, due to political, policy, and conflict challenges in core markets, the program evolved into a much more distributed country model with a higher number of smaller engagements to meet the targets. This strategy was heavily driven by the need to meet contractual targets and came at the expense of a concentration on key market reforms and development in the region but had the co-benefit of widening the program activities into new markets and working with a broader range of utilities. 3.2PROGRAM DELIVERY: EFFICIENCY AND EFFECTIVENESS Management of the program is regarded as having been efficient and effective by both external stakeholders and program staff, with a strong focus on systems and process control. The management approach employed by EAEP has been deemed highly efficient by both the internal team and by external stakeholders. Management places its trust in the technical and cross-cutting teams to fulfill EXTERNAL PERFORMANCE EVALUATION OF EAEP | JUNE 2023 | 7 their respective mandates and objectives. To ensure consistent progress, management offers support as needed and conducts regular check-in meetings with various teams to assess advancement and identify areas requiring additional assistance. The Senior Management Team (SMT) implemented an Activity Approval Form (AAF) process, which provided team members with a framework to develop tasks that align with broader work plans and contractual deliverables. This operational methodology was informed by monitoring and evaluation indicators, contractual minimum outputs, and budget cost controls, all geared toward achieving the desired results. All staff members received training on the AAF process and contributed to the project’s procurement and contract management operations. The systems, management style, and processes are assessed as excellent by the implementation team and partners. EAEP has used a range of systems, information management tools, and reporting methods that are regarded as high quality by both the internal team and external stakeholders, including USAID country Missions. Standard tools and systems encompass a cloud-based shared drive for document storage, the Wrike platform for task management, and consolidated contacts for clients and staff. These standardized systems have proven to be valuable for ensuring that reporting obligations are fulfilled in a timely manner. Team members submit weekly updates, using bullet points, on their respective portfolios to the objective and cross-cutting team leads. These leads then finalize their weekly reports with assistance from the SMT and communications advisor before submitting them to USAID. The structure of these weekly updates is mirrored in EAEP’s monthly reports, quarterly reports, and annual documents. This streamlined process simplifies the identification of validation criteria for monitoring and evaluation, enabling the early detection of challenges and successes. Moreover, the standardized systems and tools facilitate multi-step quality assurance and quality control processes, ensuring that EAEP’s communiqués and reports are of high quality. EAEP has taken a strong approach toward adaptive management, which has helped improve processes and the technical assistance offered over time. EAEP management has effectively employed adaptive management to deliver results throughout the program. Adaptive management has been crucial for navigating the complex, evolving energy sector and for addressing challenges during implementation. EAEP has taken a flexible approach, adjusting objectives and priorities based on partner needs, market conditions, and external factors, such as the COVID-19 pandemic. For example, EAEP realigned resources to adapt to USAID Mission buy-ins while meeting contractual minimum outputs. Management proactively learned from the successes and challenges, incorporating lessons into future activities and adjusting strategies as needed. EAEP hosted quarterly pause-and-reflect sessions to review the results and gaps and plan next steps, which have proven to be valuable both for implementation staff (to avoid siloing) and Power Africa staff, who are invited to participate. Feedback was systematically collected and addressed to support continuous improvement (e.g., on training events). Finally, EAEP established systems for regular monitoring and evaluation (M&E) of activities and outcomes, enabling data￾driven decisions and adaptability. For example, the M&E team conducted an analysis on “delays in financial close for generation projects” to support the Objective 1 team, share findings with stakeholders, and pivot strategies to secure higher levels of megawatt transactions. EAEP has deployed a strong approach toward developing and replicating advisory models, which has helped with program efficiency, scale, and sustainability. Despite adopting a flexible, demand-driven approach, EAEP has aimed to standardize and enhance its offerings, such as Human and Institutional Capacity Development (HICD) and advisory support models. EAEP has proven to be highly effective in capacity building, with some partners adopting its training models for future use. For example, EXTERNAL PERFORMANCE EVALUATION OF EAEP | JUNE 2023 | 8 Kenya Power, the country’s distribution utility, requested that EAEP train its learning and development team on managing and delivering effective training programs. Following EAEP’s intervention, Kenya Power incorporated the USAID HICD framework into its training activities. This framework emphasizes optimal performance and results from training resources, while considering both the individual and institutional aspects of the operating environment. Furthermore, EAEP conducted assessments of partners’ capacity and performance gaps, followed by post-assessment reviews. Through these processes, EAEP fostered a proactive capacity-building culture within partner organizations, focusing on individual and institutional change. The advisor model has been considered to be effective, particularly when advisors were embedded. Advisors have been a key component in the program’s ability to deliver results. For example, transaction advisors across the region facilitated the acceleration of more than 1,600 megawatts (MW) of generation and 500 kilometers (km) of transmission transactions toward financial close. These advisors assisted partners, such as private sector entities, governments, and utilities, in addressing roadblocks and challenges. Transaction advisors played a crucial role in identifying country-specific challenges and proposing strategic technical assistance to advance partner priorities. Additionally, EAEP’s utility turnaround team incorporated embedded advisors in selected utilities to improve commercial performance. These utility advisors employed diagnostic tools to plan, communicate, and monitor turnaround work plan opportunities, which required senior leadership buy-in from the utilities. After obtaining buy-in, the embedded advisors supported pilot zones in reducing losses and increasing revenue. Their contributions were invaluable in fostering performance monitoring and active intervention cultures in utilities that adopted their methodologies, extending their application across other service territories and expanding pilots. The Power Africa partnership model has been useful in building linkages between the program and United States Government (USG) agencies and companies seeking political and economic development opportunities. The Power Africa partnership model within EAEP plays a significant role in connecting a broad range of institutions, including USG agencies and private sector companies to unlock access to African markets. EAEP conducts market research, matchmaking, and due diligence for U.S. companies while providing insights on energy deals with USG agencies, such as the U.S. International Development Finance Corporation, U.S. Trade and Development Agency (USTDA), and the U.S. State Department. U.S. companies’ increased awareness of Power Africa’s toolbox is a result of outreach efforts by the U.S. Department of Commerce, USTDA, and the Power Africa Coordinator’s Office partnership team. This model acknowledges the numerous stakeholders involved in the energy sector. By participating in donor coordination meetings, private sector outreach, and regular meetings with country- or region-specific energy working groups, EAEP effectively communicates stakeholder needs and shifting priorities to USAID Missions, Power Africa, and other partners. The relationship is, to some extent, opportunistic and reciprocal in that USG entities also will reach out to EAEP (usually through USAID) when they become aware of an opportunity to request support from EAEP (e.g., the September 2022 visit by the Burundi president to the United States). EAEP also has been able to engage with a broad range of donors to improve collaboration. The donor model has been particularly successful at the country level, where routine donor meetings in the energy sector foster collaboration with support from EAEP. This approach promotes resource efficiency, reduces duplicated efforts, and considers the diverse interests and priorities of all stakeholders during activity implementation. For example, EAEP hosted donor coordination meetings with multilateral EXTERNAL PERFORMANCE EVALUATION OF EAEP | JUNE 2023 | 9 and bilateral development partners, provided consulting services, prepared capacity-building matrices, and represented USAID interests in various energy working groups. EAEP facilitated quarterly donor coordination meetings in Somalia for the energy sector working group, offered consulting services for rebundling support decisions in Uganda to the donor working group, developed a capacity-building matrix and provided inputs for the Rwanda working group, and consistently represented USAID interests in the Burundi donor energy group. EAEP’s participation in donor groups was subject to USAID activity managers and country leads, with the program providing deliverables to the Missions as needed. Although EAEP did not extensively engage in the DRC, Kenya, or Tanzania working groups, it still provided deliverables to the Missions as required. The team was able to respond well to manage and mitigate the impacts of COVID-19. Regular internal meetings facilitated ongoing situation assessment, performance tracking, and strategy development to adapt to the pandemic’s challenges. External meetings enabled EAEP to collaborate with partners in addressing COVID-related obstacles. For example, EAEP assisted the Kenya Electricity Generating Company in implementing mitigation measures and protocols, thus ensuring work continuity. Post￾COVID-19, EAEP consistently explored opportunities to support utilities through back-to-work initiatives, health care and supply chain connectivity, and safety training. Internally, EAEP carried out an extensive employee survey to measure the pandemic’s impact and gather feedback on optimal support measures for effective work continuation. This process enabled EAEP to comprehend the challenges that its staff encountered and adjust their strategies accordingly. There has been a general pattern of lack of visibility on the scale of funding and timing from Power Africa, which has hampered EAEP operational implementation. EAEP faced budget uncertainty starting in its second year because the Power Africa Coordinator’s Office did not clearly communicate budget availability. This led to significant delays in commitments for scopes of work with utilities, companies, and governments, resulting in ongoing tension due to insufficient funds for the originally envisioned assignments. Mission buy-ins further complicated matters because they were not budget “add-ons” but instead secured work for specific activities, many of which did not contribute to output result areas. This issue was exacerbated by Power Africa hosting all energy activities in operating countries but requesting buy-ins from USAID Missions due to inadequate funding for a regional mechanism. Missions were unprepared for this process, and their activity managers were typically not in economic growth teams, making it difficult to advocate for Power Africa projects. Following early budget reductions, the program had to be reframed within a significantly lower funding level. Annual allocations made planning difficult, impacting program close-out. Although EAEP was set to conclude in 2022, the agreement to extend the program’s timeline came late, leading to early departures and necessitating staff realignment and responsibility adjustments. In Quarter 1 (Q1) of 2023, some staff members took on additional tasks and duties in order to address these challenges. The EAEP team also faced challenges around a lack of intergovernmental agreement on contractor status and taxes, which were not supported by the Power Africa structure. The lack of clear tax policies or agreements, such as the value-added tax (VAT) in Kenya and tax exemptions in other focus countries, created challenges for EAEP’s implementation and effectiveness. Power Africa, an initiative based in Washington, D.C., and USAID/Southern Africa, does not have Mission-based relationships or memoranda of understanding (MOUs) for funding (e.g., Development Objective Agreements [DOAGs]). This situation makes it difficult for IPs to receive Mission support for exemptions, and Power Africa is unwilling to cover the cost differences. The regional DOAG with the AfDB does not EXTERNAL PERFORMANCE EVALUATION OF EAEP | JUNE 2023 | 10 extend in practice to implementers of Power Africa task orders, causing uncertainty regarding the tax and VAT status for purchasing. Such factors can significantly impact how an IP prioritizes the countries in which it operates. 3.3DELIVERY OF PROGRAM RESULTS: OUTCOMES, IMPACT, SUSTAINABILITY EAEP is on track to meet minimum outputs from the contract, although original targets have proven to be a challenge for some outcome areas. EAEP has been effective in meeting its intended purpose, objectives, and outcomes as outlined in the “minimum outputs” from the contract. The minimum outputs were updated halfway through the implementation period to reflect the realities on the ground for attribution/control over higher level outcomes in megawatts to financial close and terawatt￾hours for trade indicators. As a result, EAEP is now responsible for advancing transactions rather than simply ensuring their financial close. This has provided the flexibility to support earlier stage projects, which was a request from most partners and USAID Missions. However, together with a range of challenges, this shift also means that the results of the program have been mixed, with some results and target areas being delivered more effectively than others. EAEP SUCCESSES The following key program successes have been identified by a range of stakeholders: • EAEP has been successful in reaching its targets for new connections, although evidence for the strength of the contribution is variable. At the end of Q1 fiscal year 2023, the program reported 4,204,029 on-grid connections, allowing EAEP to surpass its target of 4 million new connections, the most of any of the Power Africa mechanisms. EAEP has taken an aggressive approach to identify utilities to deliver this target and has used a regional engagement framework to share challenges. EAEP co-developed scopes of work to improve the performance of utilities and created technical assistance activities to leverage existing priorities and donor initiatives throughout the region. This work was enabled by two different workstreams (objectives 2 and 3), which collectively supported utilities on both connectivity and performance, with the bulk of the reported connections delivered through the utility performance workstream rather than through support for direct connections. However, of all the targets, the strength of the evidence for EAEP contribution/attribution to connections is perhaps the weakest as the program has generally counted all new connections for the given utility regardless of the level or type of support provided by EAEP. • The program also was very successful in relation to its utility turnaround and performance objectives. In terms of the operational performance of the utilities, EAEP is currently reporting improvements of 200 percent in the reduction of aggregate, technical, commercial, and collection (ATC&C) losses, relative to the contract life-of-program targets. EAEP undertook diagnostic approaches to identify key issues at the utility level to understand both operational performance and challenges in extending access. As a result, the program was able to identify a core set of approaches common to different utilities that the team could operationalize through technical assistance. These included procurement and contract management, software and planning, performance improvement planning, and financial management. Once identified, EAEP was able to establish capacity-building models and coaching that could be tailored to utilities throughout the region for a low-cost solution to maximize the counting of connections. As a EXTERNAL PERFORMANCE EVALUATION OF EAEP | JUNE 2023 | 11 result, EAEP has achieved the most on-grid connections counted under any Power Africa mechanism. • Gender has been a highly successful component of the program. EAEP was able to build on Power Africa’s previous gender equity efforts with the WiAP network and USAID’s Engendering Utilities by expanding their models broadly across Ethiopia, Kenya, Rwanda, and Tanzania, as well as partially in DRC, Somalia, and Uganda. EAEP had the opportunity to manage $1.6 million in funds from the White House-led Women’s Global Development Prosperity Fund to support the Women in Rwandan Energy (WIRE) initiative. This initiative demonstrated the impact of gender equality and inclusivity goals in energy funding. For example, more than 200 apprentices were sponsored to take on work assignments (e.g., planning engineering, legal support, finance) in private and public companies, and more than 100 have since obtained employment. The successful female apprenticeship program across the region has brought women into energy companies, improved companies’ understanding of gender considerations, and supported workforce readiness for the apprentices. Simultaneously, EAEP supported women in energy networks to establish foundations, promote membership, and build partnerships with local and international companies to improve the presence of women in energy. These efforts were all aligned with overall gender equity policy development and training across dozens of ministries, companies, and organizations throughout the region, bringing large-scale gender equity programming in energy to the region. • All stakeholders recognized the value of EAEP’s efforts to support the capacity building of both staff and institutions across its suite of interventions. EAEP contributed to wider institutional capacity, systems, and skills, despite high turnover within key partner institutions. EAEP supported HICD and gender equity, among other cross-cutting areas, across nearly every intervention. The HICD model, although USAID-developed, had been used infrequently in previous energy sector capacity-strengthening efforts. EAEP was able to apply the HICD model to foster communication across country sectors, regional bodies, and even small divisions, such as public-private partnership (PPP) units, to self-diagnose gaps, identify solutions, and build implementation plans. EAEP has become a model for rolling out the HICD work in energy. It offers a model for sustainable capacity building to stop replicating previous energy interventions from donors and encourage ownership among partners receiving technical assistance. EAEP CHALLENGES In other areas, the program has faced challenges (detailed below) that have affected the delivery of results. • The program has struggled to achieve its target with regard to new generation capacity. The life-of-program target of 1,500 MW has proven to be challenging to meet. Despite early expectations of sufficient pipeline to meet the target, only 272 MW had reached financial close by the end of 2022, with no further capacity added in Q1 2023. EAEP provided PPP training, PPA negotiation training and legal support, financial modeling support, and even grid-integration assistance for stakeholders to support more megawatts. It also engaged upstream on enabling environment efforts where it was clear that transaction support would be insufficient. Key challenges arose from delays associated with the Kenya Presidential PPA Task Force deciding to renegotiate and delay PPAs that had been identified (600+ MW), resulting in at least 24 months of delays. There also were significant delays and cancellations arising from the Ethiopia portfolio EXTERNAL PERFORMANCE EVALUATION OF EAEP | JUNE 2023 | 12 (450+ MW) as a result of civil conflict and changing government priorities. Despite these challenges, EAEP has continued to advance transactions toward financial close in Kenya in advance of potential reconsideration and final approval by the government and has worked with private developers in Ethiopia. The program also was able to initiate new transactions in Burundi, DRC, Djibouti, Rwanda, and Tanzania, in addition to those grandfathered from PARTP. It is nevertheless unlikely that EAEP will meet the target within the timeframe of the program. During Q1 2023, EAEP actively supported 642 MW to move toward financial close. Overall, EAEP has identified 1,600 MW of transactions that have advanced significantly toward financial close in the Power Africa Tracking Tool (PATT) (with levels of justification) and the program continues to work in key countries on potential opportunities to advance transactions. • EAEP has struggled to deliver transmission line capacity and is projected to fall short of its target of 500 km of lines at financial close and 1,300 km commissioned. EAEP has supported 109 km of transmission lines to reach financial close and 308 km toward commissioning at the national levels through direct technical assistance. The 109 km in Rwanda (achieved only in Q1 2023) were the result of supporting studies and procurement preparation for the utility and the AfDB financing to be aligned. This $37 million investment was supported by transmission feasibility studies, environmental and social impact assessments (ESIAs), resettlement action plans (RAPs), and tender documents. • The regional trade target of 2 terawatt-hours has proven to be difficult to achieve and is unlikely to be met. EAEP entered a crowded donor space with support to EAPP, the Nile Equatorial Lakes Subsidiary Program, and other regional bodies involved in energy trade and integration. Through constant consultation and mapping with donors, EAEP was able to help advance regional initiatives in the market, legal, and operations committees of the power pool member utilities to support regional trade. In Q1 2023, the Ethiopia–Kenya segment of the Ethiopia–Kenya–Tanzania line began trading power for the first time, with full commissioning anticipated to take place by the end of 2023. Additionally, in Q1 2023, EAEP completed a feasibility study draft for the Tanzania–Malawi 400-kilovolt (kV) interconnection and a prefeasibility study draft for the Tanzania–DRC 400-kV interconnection Tanzania Electric Supply Company Limited (TANESCO) championed both lines’ studies, with the goal of obtaining financing and ultimately connecting Tanzania to the Southern African Power Pool and EAPP. EAEP continued to push the studies toward approval, with all parties to proceed to the next steps of securing financing and development. Additionally, EAEP continued the commissioning process for stages 1 and 2 of the Rwanda–Uganda 220-kV interconnector. The sustainability of EAEP interventions presents a mixed picture, with some outcomes likely requiring future programmatic support. EAEP has achieved various results, some of which are likely to be sustainable, while others may need ongoing support or further interventions. EAEP’s success in capacity building among partner organizations, utilities, and government ministries has fostered a culture of proactive capacity building, focusing on individual and institutional change rather than a one￾time opportunity for technical assistance. Moreover, EAEP has facilitated the development and implementation of progressive policies, regulatory reforms, and master plans to guide future decision￾making for utilities and governments. The program also has improved utility performance by emphasizing performance metrics, such as ATC&C losses. This focus on performance monitoring and intervention equips utilities to maintain and enhance their operations, contributing to sustainability. However, additional support is needed for transaction support to advance the pipeline because financing and risk gaps persist in stalling generation and transmission transactions. The sustainability of some results also EXTERNAL PERFORMANCE EVALUATION OF EAEP | JUNE 2023 | 13 may be affected by external factors such as political changes, economic conditions, or unforeseen challenges. In these situations, continuous monitoring and support are crucial for ensuring the sustainability of the program’s outcomes. There are differing levels of contribution to the above results given that EAEP’s engagement has varied. Many institutions are engaging directly with beneficiaries in each of the countries and Power Africa is one of several sources of technical assistance and advisory support. Beneficiary stakeholders shared that EAEP has taken a robust approach to trying to capture the contribution narrative for all results claimed within the results framework. However, the strength of the contribution inevitably varies by the type of engagement. Usually, EAEP and Power Africa may be only engaging in one element of a more complex transaction or reform process. The role is clearer in relation to single, large-scale transactions than it is in wider distribution or connection activities. 3.4KEY BARRIERS TO RESULTS DELIVERY LONG PROJECT TIMELINES AND PROCESSES There are challenges in advancing large-scale, capital-intensive transactions within EAEP timeframes: • Long timelines associated with large-scale generation and transmission transactions. Large-scale generation and transmission transactions take many years to plan even before they reach financial close. One of the greatest challenges that EAEP has faced was identifying those transactions that had realistic timelines to hit financial close during the lifetime of the program (with the original EAEP end date of December 2022. The majority of EAEP’s transaction advisors therefore focused on generation transactions grandfathered from PATRP, which were more advanced and more likely to meet financial close targets during program implementation. The approach to pursuing the PATRP legacy portfolio was reasonable given the circumstances. The need to pivot mid-program left limited time to see new transactions through to financial close. With perfect hindsight, it is possible that the results would have been further along if the strategy had been different from the start; however, this would have been difficult to justify in the context of where the program was in 2019. As a result, EAEP lacked an incentive initially to commit resources to early-stage transactions and pipeline development because these results would fall outside of its results framework. In the end, many of these grandfathered transactions were impacted by chronic delays resulting from a range of factors (set out in more detail below), forcing the program to seek new transactions that were then challenging to complete within the program timeframes. • Long and bureaucratic processes associated with MDB/DFI-led transmission projects. For transmission projects, the timelines are even longer than those for generation due to the significant capital investment requirements and interregional political relationships for cross￾border trade. Utilities needed to adhere to the MDB standards for transmission feasibility studies, ESIAs, RAPs, and detailed design before they could be tendered. For example, this process took 2 years in Rwanda for two lines, which were a total of 109 km, due to constant updates, questioning, and concerns about AfDB approval. While developing the requisite studies, EAEP attempted to identify ways to accelerate the process and develop more streamlined methodologies; however, the program is ultimately dependent upon external factors regarding whether these transactions proceed. EXTERNAL PERFORMANCE EVALUATION OF EAEP | JUNE 2023 | 14 POLITICAL AND REGULATORY CHALLENGES The program has faced external challenges in the political, regulatory, and security sphere, where it has little control or influence: • Regulatory uncertainty, tariff reviews, and stalling PPAs. There has been significant instability in the policy and regulatory environment in EAEP countries in relation to power purchase regimes and tariff setting. This has been particularly true in those countries that were expected to deliver the bulk of the transaction volumes. PPAs were delayed, with governments delaying regulatory approvals that would enable financial close on projects to review tariffs or sector structures. For example, Rwanda and Tanzania put in place 10 cents per kilowatt-hour tariff caps when EAEP started, even asking for renegotiations with existing projects. In Kenya, a Presidential Task Force instituted a moratorium on all PPAs for more than 1 year as it reviewed tariffs, power needs, and market design in 2021. • Political and security factors. EAEP has operated in a challenging political and security context, including in the largest countries in the region expected to deliver the greatest impact. Civil conflict in Ethiopia led to staff evacuations and eventually resulted in program closure, forcing the need to identify new utilities for partnership. Election periods in Kenya and Uganda, as well as transitions between outgoing and incoming governments, resulted in delays for EAEP with projects stalled for significant periods of time. Operations in Somaliland were challenging as USAID is unable to recognize Somaliland as an independent government nor make it a direct recipient of technical assistance funds; however, there was nevertheless the need for a political MOU to be signed. This created significant delays and required a different delivery model. Interregional political and security challenges also affected transaction development, particularly for regional interconnectors (e.g., the Shango–Mbarara transmission line between Rwanda and Uganda). • Weak delivery of national commitments. EAEP, to some extent, was guided by government commitments in developing its work plans, which turned out not to have been delivered. This was particularly true of national commitments to universal electricity access and associated connections programs. However, national plans often were not well implemented and fell short of their targets. For example, in Uganda, the electricity connections program (ECP), a subsidized free connections program, was halted suddenly after reaching 300,000 connections in 2019–2020 due to non-payment by the government to utilities and service providers. ECP was only revived in late 2022. Likewise, Ethiopia committed to one million connections per year; however, in practice, it never exceeded more than 450,000, despite universal access targets and a large population. • Perceptions of oversupply in the region. Although the installed electrical capacity in the region is comparatively low, especially when considering the unmet potential demand, utility planners and regional organizations often perceived that there was an oversupply. This is primarily a result of lack of demand due to low levels of access and a lack of power-intensive customers or of challenges within the transmission and distribution infrastructure. In addition, the large differential between peak and off-peak demand also creates concerns around the economic and financial sustainability for new baseload generation infrastructure. This has created a lack of willingness to invest in new capacity (e.g., in Uganda), even where there are opportunities for export to countries with power deficits. EXTERNAL PERFORMANCE EVALUATION OF EAEP | JUNE 2023 | 15 PANDEMIC AND NATURAL DISASTERS The COVID-19 pandemic significantly disrupted program implementation, as detailed below: • COVID-19 was a key challenge to implementation. COVID-19 has been a significant barrier to program implementation and overall performance. The pandemic affected the normal functioning of partner organizations and their employees, with some becoming infected or affected, which, in turn, affected the overall performance of these organizations. Furthermore, COVID-19 led to disruptions in global and local supply chains and logistics, causing delays in obtaining the necessary supplies for program activities. For example, some distribution companies were unable to procure meters and other supplies, resulting in reduced electricity connectivity rates. This contributed to Uganda’s ability to supply only 30,000 connections annually, despite its projection of supplying 300,000 connections annually. The pandemic created an unprecedented challenge for EAEP and its partners, hindering their ability to achieve objectives and targets in a timely manner. In addition to COVID-19, there also was closure of parts of Uganda in 2022 due to the Ebola outbreak, which further hindered program activities and delayed the achievement of objectives. 4. CONCLUSIONS AND RECOMMENDATIONS 4.1CONCLUSIONS/LESSONS LEARNED CONCLUSIONS The Power Africa model, as implemented through EAEP, offers a sensible and flexible model for power sector engagement in East Africa. The structure of the program, built around a range of objectives (e.g., generation, transmission, distribution, regional connectivity) and working with a broad range of stakeholders (e.g., government, utilities, developers), has allowed for sufficient breadth and geographical flexibility to identify demand-led opportunities across the region and work in an adaptive manner with a range of beneficiaries. Despite the pipeline legacy of PATRP, the level of political, economic, and social disruption within the region has underlined the need for a degree of flexibility in how and where targets are delivered. The contract is a useful guiding framework for prioritization; however, this may be at the expense of more strategic and long-term engagement approaches. Program targets create a healthy set of incentives to identify and develop new opportunities where roadblocks are encountered. However, the need to deliver the contract targets may have taken the focus of the program away from a more systemic approach to addressing complex country challenges with partners in an integrated manner. The contract creates incentives toward “paths of least resistance” or “the lowest hanging fruit” most likely to support outcomes. This has resulted in a large number of interventions across a broad number of countries, making it more difficult to create an integrated narrative at a country or regional level. Perceptions of long-term, in-country continuity and presence are important to build credibility and trust, even if progress may be slow. The implementation team has conducted the program in an efficient manner, with strong management systems and processes to deliver a broad range of interventions. EAEP is EXTERNAL PERFORMANCE EVALUATION OF EAEP | JUNE 2023 | 16 universally regarded as having been efficiently and effectively managed under challenging circumstances (i.e., COVID-19 and political and economic instability). The program was able to adapt to new delivery models (e.g., country delivery approach, remote working). Robust management and reporting systems, combined with the use of well-regarded pause-and-reflect sessions, have meant that the program has maintained a strong overview of progress and challenges across its workstreams, and has been able to adapt and pivot accordingly. The use of advisors also has been a key success factor, as has EAEP’s ability to reach out to relevant USG and Power Africa institutional partners where necessary to progress work. The program has been more successful in reaching some objectives than others. EAEP has been particularly successful in supporting the delivery of new connections, providing the most of any of the Power Africa programs. Its work on utility turnaround also has been successful, with significant performance improvements for those utilities and in those regions where it engaged. The program’s work on gender also has been groundbreaking, with practical delivery of both training and institutional development, resulting in real-world outcomes for young women in the power sector in multiple countries. However, progress on new generation and transmission capacity (both national and regional) has been slower than expected for a number of reasons at both the project and political levels, including potentially unrealistic target setting at the program outset. The attribution of results to EAEP activities varies in strength across the objectives; however, EAEP has sought to provide robust evidence regarding its relative contribution. There are varying levels of contribution and attribution for the reported results of EAEP, with connections having perhaps the overall weakest evidence of contribution. Many of the transactions were already inherited from PATRP. Given the more robust reporting requirements for EAEP (compared with PATRP), the program has made significant efforts to detail the linkages between activities and outcomes with a view to ensuring that these can be captured by Power Africa in the future as and when transactions reach financial close. LESSONS LEARNED Timescales for large transactions are unpredictable in the East Africa region, in part, due to a complex political, economic, and security environment. Large capital-intensive transactions in generation and transmission infrastructure are, by their nature, complex and can require significant time to reach financial close. As such, they also are exposed to changing political, sectoral, and economic circumstances, which can delay implementation or change the value of investments over time. As a result, large transactions originated by EAEP are highly likely to be grandfathered in any subsequent program or supported by other parties. This would require a level of programmatic integration over time, an effective handover between program contractors, a strong role for Power Africa and USAID in ensuring continuity and political engagement, and acknowledgement of the value of both preparing and accelerating transaction progress, as well as achieving financial close. Power Africa is just one of many stakeholders in the East Africa region engaged in policy, transaction support, and utility reform, requiring strong partnerships and coordination. To achieve large-scale change, a strong focus on stakeholder relationships and coordinated, sustained coalitions is necessary. In this context, the technical assistance provided by EAEP is valuable and the program was able to engage with other technical assistance partners and development partner coordination groups in a number of countries. However, Power Africa still needs to establish closer relationships with large-scale financiers to facilitate transactions and with political partners to help EXTERNAL PERFORMANCE EVALUATION OF EAEP | JUNE 2023 | 17 overcome policy and regulatory obstacles. For larger scale transactions, Power Africa often acts as a capacity builder to DFIs. WB and AfDB dominate transmission and interconnector development efforts in East Africa and the Horn of Africa region. EAEP found AfDB, AFD, and WB receptive to collaboration in Rwanda, DRC, and Burundi, respectively. Because many DFIs provide technical assistance, EAEP must identify its relative niche while seeking to add value to processes rather than duplicating or complicating them. Often it is a challenge to persuade DFIs to rely on a third-party provider of technical assistance, such as EAEP, to help address critical barriers for large-scale infrastructure due to the complexity of coordination and control, as well as political relationships. Upstream planning support is critical in unlocking effective downstream transactions. EAEP has indirectly generated a significant proportion of transaction impact through its upstream planning support to policymakers and other national or regional bodies. This support has enabled the creation of frameworks that allow for more confident investment decision-making by policymakers and utilities. For example, EAEP provided support to Ethiopia’s master plan, which facilitated a more integrated and data￾driven approach to planning. This involved undertaking demand forecasts and integrating environmental impact and social aspects, which was a major improvement from earlier ad hoc decision-making on generation, transmission, and distribution. Similarly, EAEP supported Rwanda in creating resource assessment methodologies for future generation development, which provides utilities with a stronger sense of the relative cost of national generation development and energy imports. Currently, the program is following potential opportunities for master plans and updates in Kenya, Tanzania, and Uganda. The contribution of this work to downstream investments needs to be strengthened and recognized. EAEP has added value in identifying common challenges and creating standardized approaches to allow for scale and consistency. EAEP’s ability to add value has been significantly enhanced by its use of diagnostics and the creation of standardized technical assistance approaches. The program identified common challenges among different types of beneficiaries, particularly at the utility level. This allowed EAEP to tailor engagement processes that leveraged core approaches and knowledge effectively. For example, most of the countries had ambitious targets for universal access plans at the national economic planning level. However, the utilities were left to determine how to work with numerous donors providing funds and loans to achieve those targets. After the second year of implementation, EAEP identified common issues that prevented utilities from achieving their connection targets. These issues included lengthy procurement processes and mismatched skill sets, insufficient contract-management capacity among utility staff for service or materials providers, technology-adoption delays due to a lack of basic computers, internet, and planning software, the neglect of customer service in existing and former state-owned utilities, and the chaotic and disparate nature of data collection across an entire utility. Once these gaps were identified, EAEP developed rapid solutions that aligned with the needs of utility staff, unlocking connections data. Discussions around additional generation require confidence among decision-makers regarding the robustness of future demand. Many policymakers are caught up in short-term concerns around oversupply and the economic performance of utilities, which get in the way of long-term sector investment. Therefore, there is a strong need for partners to have confidence in future growth in power demand (e.g., expanding access and distribution, supporting larger energy-intensive baseload customers) to unlock generation decision-making. For example, Uganda has a 600-MW surplus, creating concerns around future capacity investment; however, less than 30 percent of its population has access to electricity. Exporting to neighboring countries to alleviate generation deficits is a viable option; EXTERNAL PERFORMANCE EVALUATION OF EAEP | JUNE 2023 | 18 however, there is insufficient confidence in regional trading to use this as a basis for investment decision￾making related to new power generation. A greater focus on strengthening connections, upgrading distribution infrastructure, and stimulating baseload demand is therefore key to unlocking generation investment in some countries. Institutional buy-in and long-term relationships are key to gaining traction on policy and utility reform. Some of EAEP’s most important successes have been built on upfront investment in beneficiary-led models, building trust and ensuring that partners understood the Power Africa approach. Of significant importance is the perception that Power Africa is there to provide long-term support rather than offering a one-time solution or donor-driven agenda. The EAEP team invested heavily in tailoring solutions to need rather than just delivering off-the-shelf support. Examples include the extensive use of diagnostics for utility turnaround (e.g., in Kenya, Ethiopia, Zanzibar). The provision of support helped unlock senior management buy-in, which resulted in better access to data and reporting. Where these relationship elements have been less successful (e.g., TANESCO), outcomes and impacts also have suffered. 4.2RECOMMENDATIONS EAEP was due to close out at the end of fiscal year 2022, and even with the current extension, there is now limited opportunity to engage in a course correction. Several of the workstreams and country engagements have now been effectively stopped. In addition, the next phase of EAEP has already been tendered, meaning that there also are limited opportunities for influencing the overall structure and terms of reference for the new program. Nevertheless, there are several recommendations that flow from the findings and conclusions that might be considered in relation to both the procurement of a new contractor and more detailed implementation design and delivery planning. RECOMMENDATION 1: Create a smarter approach to setting targets and program priorities. Initial target setting (e.g., around generation and transmission) for EAEP was optimistic and at the high end of what was realistic given the political economy challenges. In addition, the program structure around targets, rather than strategic country outcomes, may have encouraged more piecemeal results. The discussion of both program priorities and boundaries needs to be undertaken in a smarter manner between Power Africa and the implementation team, based on a clear understanding of evolving opportunities and challenges. In a resource-constrained environment, there may be value in doing a smaller number of things more effectively on a greater scale. Some geographical and sub-sectoral concentration may enhance momentum and critical mass. Targets should not create perverse incentives to look for low hanging fruit, or to try and be present everywhere at the expense of diluting resources. Early agreement and buy-in with beneficiaries are vital, with USAID Missions and Power Africa needing to play a more proactive role in engaging with key stakeholders and enhancing the political economy of program delivery. • ACTION 1: Ensure that programmatic targets are realistic given the pipeline, scale of resourcing and the prevailing market, and policy challenges. To determine whether targets are realistic, USAID should model the new program on the conversation rate and project timeframes within EAEP. Generation and transmission targets should be at least 50 percent lower than the current ones, barring any significant shift in the political economy. • ACTION 2: Increase Power Africa and USAID strategic engagement on key outcomes in the region (beyond numerical targets) and help build a more integrated narrative of change. This can be done by developing country strategies and theories of change that value and contribute not EXTERNAL PERFORMANCE EVALUATION OF EAEP | JUNE 2023 | 19 only to the ability to reach financial close but also opportunities to contribute to the development of other important goals, such as innovative products and services. While addressing these opportunities may not be easily quantifiable, and therefore currently not included in Power Africa indicators and targets, USAID can still work toward aligning its efforts more closely with the broader context of sector development. This will allow USAID to contribute to more upstream engagement (i.e., through integrated sector planning support). • ACTION 3: Achieve early buy-in and secure memoranda of understanding between Power Africa, USAID, the implementation team, counterparts and beneficiaries for engagement with a clear sense of scope and results. RECOMMENDATION 2: Strengthen the focus on the value of upstream enabling environment work to facilitate downstream transactions with the ability to recognize contributions where evidenced. Upstream enabling environment support has been key in creating a conducive environment for downstream transaction development. Weak decision-making capacity in key partner institutions, compounded by political economy challenges, has created blockages to moving forward with strategic transactions, forcing the reallocation of EAEP resources toward new opportunities across a broader range of countries. However, the relatively long causal pathways between upstream policy and planning and individual downstream transactions also are not well captured or rewarded in the existing results frameworks. The value of progress in the enabling environment should be recognized. • ACTION 1: Encourage a renewed focus around upstream enabling environment work (e.g., master planning) in the new phase of programming. • ACTION 2: Facilitate systems that recognize the value of enabling environment work toward downstream targets where supported or validated by beneficiaries. RECOMMENDATION 3: Maintain the focus on expanding connections and increasing engagement regarding demand and productive use. Against a backdrop of a lack of universal access and limited demand from energy-intensive customers, including the emerging market for e-mobility, policymakers and utilities are concerned about the impact of additional generation, whether on the financial sustainability of utilities or on public subsidies. There is currently limited confidence among policymakers with regard to regional power trading as a reliable opportunity to underpin the financial case for investment in generation. Power Africa should maintain a strong focus on expanding connections because there are significant opportunities to expand networks in Burundi, DRC, Ethiopia, Somalia, Tanzania, and Uganda, as well as explore the value of working with end users on promoting demand stimulation, particularly in countries experiencing a surplus. • ACTION 1: Include a renewed focus on connections as part of the new Power Africa program. • ACTION 2: Explore options for engaging in expanding downstream productive use because this provides an anchor for new generation and utility sustainability. RECOMMENDATION 4: Continue with the embedded advisor model. The embedded advisor model has proven to be critical to EAEP’s success, allowing for more integrated, consistent, and ongoing support to beneficiaries. This builds trust and allows for more critical and strategic engagement with institutional management, particularly around intangibles such as culture change and performance. The model has been particularly successful at the utility level but also in the ministry context. While some concerns have been expressed around both the cost and the sustainability of the model once the advisor leaves, combined with capacity building and training, it has nevertheless proven to be very effective. EXTERNAL PERFORMANCE EVALUATION OF EAEP | JUNE 2023 | 20 • ACTION 1: Power Africa and the implementation team to explore the continued use of embedded advisors in the new program phase. RECOMMENDATION 5: Work with DFIs to ensure that there is momentum for delivering large infrastructure investments, in particular transmission infrastructure projects, in a timely manner. For large critical investments (both generation and regional transmission), there is a significant role played by multilateral development banks and other DFIs in the planning and execution. DFIs know which projects will have the political buy-in and bankability to reach financial close and commissioning in the next 5 years. If a project does not already have a DFI interest, then it is unlikely that it will be commissioned within the next phase of the program. The positioning of EAEP in the context of these transactions needs to be done in a coordinated manner that ensures added value. • ACTION 1: Ensure Power Africa and implementation team prioritization of DFI-supported transactions. RECOMMENDATION 6: Improve synergies between the program and the Power Africa coalition. While EAEP operates as a stand-alone program, it does so in the context of the broader set of Power Africa partners (e.g., USG agencies, donors, private sector) who are collectively responsible for supporting sector-level transformation. Work at the program level in the region, while engaging selectively with Power Africa partners for task delivery, does not appear to be closely integrated with a higher level coalition approach or a unified strategy among the partners. While Power Africa may lack the scale and influence to act as the main point of alignment among a broad range of partners, more could be done to bring together the coalition and the program, particularly as related to strategic transactions and political economy challenges. Furthermore, care will have to be taken to ensure that efforts do not duplicate wider coordination and emerging partnership efforts (e.g., Global Energy Alliance) where there is overlap. It is unlikely that this can be done by the IP, so this effort will need to be supported and led primarily by the USAID/Power Africa team, both at the regional and headquarters levels. • ACTION 1: Strengthen proactive work with the IP to ensure that synergies between EAEP (and its successor program) and the wider Power Africa coalition are maximized for the delivery of results. RECOMMENDATION 7: Build sustainability plans from an early stage in implementation. The emphasis on the sustainability of EAEP outcomes was introduced late in the program’s development. EAEP devised its sustainability plans in the third year, whereas it would have been better to co-create these plans from the onset of the program. This collaborative strategy would have facilitated the delineation of EAEP’s objectives and scope, as well as the identification of synergies for achieving desired outcomes. It is important that, in future programming, greater effort is made upfront to build in sustainability considerations. • ACTION 1: Work with the IP to ensure that long-term sustainability plans and objectives are adequately reflected from the inception phase, with common understanding of successful exit benchmarks. These sustainability plans might include a greater focus on strengthening country￾level collaboration (e.g., through enhanced donor coordination), ensuring institutional sustainability (e.g., through train-the-trainer methodologies and skills transfer), or creating shared long-term visions of institutional progress (e.g., organizational health assessments and HICD validation workshops). EXTERNAL PERFORMANCE EVALUATION OF EAEP | JUNE 2023 | 21 RECOMMENDATION 8: Continue and strengthen work on gender; however, broaden the focus on female entrepreneurship and leadership. The program has made significant progress in enhancing gender inclusivity within the power sector by raising awareness and fostering momentum for women’s participation. There is potential to expand this focus to encompass entrepreneurship and wider economic development, creating opportunities on both the supply and demand sides. On the supply side, the program could support female-led energy supply and service businesses, while on the demand and productive use sides, it could promote productive uses of energy for both on-grid and off-grid applications. There is a high degree of enthusiasm for entrepreneurship, with many network members, apprentices, and interns expressing interest. Support could be given to green job opportunities and access to entrepreneurship-related resources. EAEP could incorporate this type of training through the implementation team; however, it also might require longer term partnerships between Power Africa or the Missions to ensure continuity between program phases. Alternatively, the respective USAID Missions could collaborate with networks such as PowerHer, ActivateHer, the Tanzania Women in Energy Network, or other organizations to provide support for employment and economic growth initiatives. • ACTION 1: Maintain the Power Africa approach to enhancing access, focusing on the sustainability of the existing networks and ongoing support for apprenticeships. • ACTION 2: Include support for female entrepreneurship (both the supply and demand sides), both through the program and by leveraging infrastructure already created by EAEP. RECOMMENDATION 9: Greater operational clarity and timeliness around budgets and contractor status by Power Africa. EAEP faced significant challenges in managing visibility around annual budgeting processes, as well as a high degree of uncertainty around contractor tax status. Greater focus needs to be placed on minimizing the potential operationally disruptive aspects of the program through early clarification of financing envelopes and flows (not least in order to support beneficiary expectations around the scale and timelines of support), as well as the status of the contractor relative to the countries in which it operates. • ACTION 1: Develop a more transparent and predictable funding framework to allow for better strategic planning, resource allocation, and beneficiary support. • ACTION 2: Support IPs in resolving issues related to taxation by vouching for and explaining the role of the contractor within the USG program to the relevant host country authorities. EXTERNAL PERFORMANCE EVALUATION OF EAEP | JUNE 2023 | 22 ANNEXES ANNEX A: EVALUATION STATEMENT OF WORK EAST AFRICA ENERGY PROGRAM (EAEP) The Contractor must conduct a Performance Evaluation of the USAID-funded program that provides technical services to implement the four-year EAEP, under a Power Africa Indefinite Delivery, Indefinite Quantity (IDIQ) Contract. I. BACKGROUND OF THE PROJECT Implementing Organization RTI International Total Estimated Cost $73,850,693.00 (Cost-Plus-Fixed-Fee Completion) Period of Performance December 17, 2018 to December 16, 2022 Place of Performance for PA Task Order East Africa - Scope includes Burundi, the Democratic Republic of Congo, Djibouti, Eritrea, Ethiopia, Kenya, Rwanda, Somalia (including Puntland and Somaliland), Tanzania, and Uganda II. DESCRIPTION OF THE EAEP PROJECT AND THEORY OF CHANGE EAEP is an implementing mechanism that was awarded to RTI in December 2018. The project is implemented in 10 countries in East Africa. These include: Burundi, the Democratic Republic of Congo, Djibouti, Eritrea, Ethiopia, Kenya, Rwanda, Somalia (including Puntland and Somaliland), Tanzania, and Uganda. In each of the 10 countries, the contractor is expected to provide technical assistance and capacity building in the focus countries while collaborating closely with its consortium members: Khulisa Management Services, Tetra Tech, Fieldstone Africa, The Center for Climate Strategies, and Norton Rose Fulbright. EAEP is designed to be implemented by prioritizing cost￾effective, direct contact with all stakeholders to facilitate needs assessment and develop dynamic, responsive interventions that fit into a cohesive, integrated, regional approach to systems interconnection and harmonized planning. It seeks to engage in jointly managed projects to facilitate political buy-in, country-led development, and assist EAEP stakeholders on their journey to self-reliance. Since RTI is working in ten countries, the partner is expected to use a one-team management approach. EXTERNAL PERFORMANCE EVALUATION OF EAEP | JUNE 2023 | 23 The project focuses on the activities listed in the table below. In addition, the project provides support to the Power Africa Coordinator’s Office and USAID Missions. Objective 1: Objective 1: Optimized Power Supply Activity 1.1: Bringing new power generation capacity to financial close Activity 1.2: Accelerating national transmission projects Activity 1.3: Supporting the governments and utilities to improve their power generation, transmission, and distribution planning Activity 1.4: Improving legal and regulatory frameworks to stimulate private-sector investment Activity 1.5: Improve capacity of governments and utilities to negotiate, produce, and manage power generation and transmission projects Objective 2: Increased Grid-Based Power Connections Activity 2.1: Ethiopia–Build capacity of the Ethiopian Electric Utility (EEU) to install over 1 million new meters annually Activity 2.2: Support the implementation of Kenya’s new Energy Bill Activity 2.3: Kenya—Support the Kenya Power and Lighting Company (KPLC) in the implementation of its 2017–2023 corporate strategy Activity 2.4: Help stimulate demand for on-grid power Activity 2.5: Support utilities and rural electrification agencies in reducing the cost of connections and time required to connect to the grid Objective 3: Strengthened Utilities and Other Power Sector Entities Activity 3.1: Ethiopia—Assistance to the EEU Activity 3.2: Kenya—Assistance to KPLC Activity 3.3: Rwanda: Assistance to the The Rwanda Energy Group/Energy Utility Corporation Limited (REG/EUCL) Activity 3.4: Implement Uniform System of Accounts (“USofA”) Activity 3.5: Developing and advocating for cost-reflective tariffs Activity 3.6: Assist utilities in attracting private-sector financing Objective 4: Increased Regional Power Trade Activity 4.1: Build capacity of the EAPP member utilities and generation companies to improve overall operational and technical readiness for power trading Activity 4.2: Harmonize and strengthen member state legal and regulatory frameworks to support cross-border electricity trading Activity 4.3: Improve regional coordination and planning at EAPP, the Nile Equatorial Lakes Subsidiary Action Program (NELSAP), energy ministries, and member state electric utilities Activity 4.4: Assist in the development and implementation of high-priority regional power infrastructure projects Activity 4.5: Assist member countries in identifying and developing new off-takers for the purchase of surplus power Activity 4.6: Help establish and operationalize two control area agreements EXTERNAL PERFORMANCE EVALUATION OF EAEP | JUNE 2023 | 24 East Africa has the second-strongest power sector in SSA, with important gains made over the past decade in utility performance, access to energy (both on- and off-grid), and available generation capacity, including significant progress made under the PATRP. Yet, regional trends mask the significant remaining challenges across the ten countries targeted by EAEP. National priorities, human and institutional capacity, and sophistication of systems differ greatly, as does the resulting approach required to accompany each nation on its journey to energy equity and more self-reliance. A far greater lagging challenge is regional integration of the technical and human systems that permit the exchange of power: despite nominal institutions, plans, and dialogue about regional integration, East Africa is still in the initial stages of building a real sustainable energy market, with significant gaps in the required transmission infrastructure, grid codes and operating protocols, and market platforms for regional exchange of power. Only when national systems are meaningfully integrated, will competition drive down consumer tariffs and improve regional reliability and broader access to energy. The TOC for this project assumed that by expanding affordable and reliable electricity services in East Africa, EAEP interventions would aid in the achievement of the goal of supporting development priorities, including inclusive economic growth, increased security, and improved health and education outcomes in the region. The implementing mechanism was expected to implement this activity to address critical opportunities across the following four objectives: 1. Optimized power supply; 2. Increased grid-based power connections; 3. Strengthened utilities and other power sector entities; and, 4. Increased regional power trade. EAEP was expected to promote gender equity, environmental mitigation and standards, and community engagement across all objectives. The contractor’s proposed implementation approach aims to strengthen and leverage the EAEP regional platform to maximize contact points among regional stakeholders, while simultaneously addressing four common regional dimensions: 1. Workforce development and institutional strengthening via a regional training platform that delivers common proficiency to stakeholders in key technical areas; 2. Improvement in national and regional planning and policy through diagnostic tools, private sector-driven improvements and data-driven analysis, and introduction of best practices; 3. Coordination and information sharing through the Power Africa Information System (“PAIS”), Women in African Power (“WIAP”), Eastern Africa Power Pool (“EAPP”), and other regional institutions; and, 4. Facilitation of public-private dialogue through the program’s regional footprint, more embedded utility and transaction advisors to maintain real momentum on deal flow, and relations built with the private sector. The contractor was expected to address the challenge of four project offices, ten countries, and five sub-contractors by establishing a harmonized “One Team” management approach. The contractor was required to establish four project offices; managing objectives 1 and 3 from Nairobi, Kenya, and managing objectives 2 and 4 from Addis Ababa, Ethiopia. The contractor was also tasked to provide overall project direction, cross-cutting activities, and coordination with Power Africa and the donor community from Kenya. EXTERNAL PERFORMANCE EVALUATION OF EAEP | JUNE 2023 | 25 III. EVALUATION QUESTIONS FOR EAEP Evaluation questions will be finalized in collaboration with the USAID post-award process. The following evaluation questions are illustrative: Q1. In its fourth and final year of project implementation, is EAEP on target to achieve the goals set in the task order? Do the assumptions in the theory of change hold true or does the theory of change need to be adjusted? If adjustments are needed, please provide concrete recommendations for the types and scope of project implementation adjustments. Q2. Identify achievements, challenges, best practices, lessons learned and recommendations from program implementation to date, for the four project objectives: Objective 1. Optimized power supply; Objective 2. Increased grid-based power connections; Objective 3. Strengthened utilities and other power sector entities; and Objective 4. Increased regional power trade. Q3. What are the lessons learned from the “one team” management approach across the various countries, teams and sub-partners in achieving the expected results? What are the recommendations to improve results for the rest of the program implementation years. Q4. How effective was the IPs performance management tools in terms of timeliness, accuracy of reporting, availability of information, tracking target achievement, etc? Q5. Evaluate and describe the learning and adapting approach identified by RTI in implementing and documenting learning, knowledge management and adaptability through its quarterly pause-and- reflect sessions. How have these learnings supported program implementation? Q6. How has the COVID-19 pandemic affected the project in achieving its implementation milestones and the overall project performance? How did the partner monitor and document performance changes? EXTERNAL PERFORMANCE EVALUATION OF EAEP | JUNE 2023 | 26 ANNEX B: EVALUATION DESIGN AND MATRIX Updated to reflect final team organization and other minor corrections. 1. INTRODUCTION The U.S. Agency for International Development (USAID) Power Africa Program Office, located within the U.S. Mission to South Africa, has contracted ICF to conduct external performance evaluations of four USAID-funded programs to improve program performance management and efficiency. These evaluations will help Power Africa meet Agency requirements, make improved and more informed strategic and management decisions around program implementation, and advance Agency learning. The evaluations will: 1) determine, to date, the extent to which the contract has achieved its intended objectives and outputs; 2) assess the technical and program management of the implementation mechanism; 3) highlight lessons for USAID in facilitating coordination with IPs; 4) highlight major gaps and challenges that may require adjustments in program implementation; and 5) inform the design of potential future activities. The lessons drawn from these evaluations will be important to the technical teams, activity managers and senior management. 1.1 ABOUT USAID POWER AFRICA USAID Power Africa seeks to increase access to electricity throughout the region with the addition of millions of new connections and tens of thousands of megawatts in new and cleaner power generation. To date, USAID Power Africa has delivered significant transformative impacts in the Sub-Saharan Africa region, not only in expanding energy services but also in benefitting multiple additional development priorities related to promoting economic growth, improving access to health and education, increasing productivity and sustainability in agriculture and livelihoods, and recognizing the needs of local communities. To evaluate effectiveness, efficiency, and relevance, and to identify best practices and lessons learned, USAID Power Africa has commissioned independent evaluations of the following four programs: ● The West Africa Energy Program (WAEP); ● The Nigeria Power Sector Program (NPSP); ● The East Africa Energy Program (EAEP); and ● The Power Africa Off-Grid Program (PAOP). Each evaluation will be specifically tailored to address the program’s theory of change and expected outcomes, while recognizing USAID Power Africa’s overall mandate to increase access to reliable, affordable, and sustainable power through its three strategic pillars: (1) New Power Generation Capacity; (2) Increased Electrical Connections; and (3) Improved Enabling Environment. 1.2 USAID POWER AFRICA PROGRAMS TO BE EVALUATED Each USAID Power Africa program brings together a unique set of interventions, technical partners, and country stakeholders that address the region’s most pressing energy needs and priorities for energy equity and self-reliance. EXTERNAL PERFORMANCE EVALUATION OF EAEP | JUNE 2023 | 27 The following is a brief description of each program and its respective theory of change (TOC) and outcomes as well as areas of inquiry which provide a framework for the evaluation design. Additional program details and geographic scope are provided in Exhibit 1. Exhibit 1. ICF's expertise extends to all program geographies. West Africa Energy Program ● IP: Deloitte Consulting LLP ● POP: July 2019-July 2023 ● TEC: $73,850,693 CLIN 1 $54,581,750 CLIN 2 (Ghana) $19,268,943 ● Countries: Benin, Burkina Faso, Cameroon, Cape Verde, Central African Republic, Chad, Congo, Cote d’Ivoire, DRC, Equatorial Guinea, Gabon, The Gambia, Ghana, Guinea-Bissau, Liberia, Mali, Mauritania, Niger, São Tomé and Príncipe, Senegal, Sierra Leone, and Togo Nigeria Power Sector Program ● IP: Deloitte Consulting LLP ● POP: April 2018-March 2023; Extended to September 30, 2024 ● TEC: $109,258,794 ● Country: Nigeria EAEP ● IP: RTI International ● POP: December 2018-December 2022; extended to March 2023 ● TEC: $73,850,693 ● Countries: Burundi, DRC, Djibouti, Eretria, Ethiopia, Kenya, Rwanda, Somalia (Punland and Somaliland), Tanzania, and Uganda Power Africa Off-Grid Program ● IP: RTI International POP: November 2018-August 2023 ● TEC: $49,992,361 ● Countries: Benin, Burkina Faso, Cameroon, Cape Verde, Central African Republic, Chad, Cote d’Ivoire, DRC, Ethiopia, The Gambia, Ghana, Guinea-Bissau, Kenya, Lesotho, Liberia, Mali, Madagascar, Mauritania, Mozambique, Niger, Rwanda, Senegal, Sierra Leone, Somalia, Tanzania, Togo, Uganda, Zambia IP = Implementing Partner; POP = Period of Performance; TEC = Total Estimated Cost EXTERNAL PERFORMANCE EVALUATION OF EAEP | JUNE 2023 | 28 The East Africa Energy Program (EAEP) is designed to optimize the power supply, increase grid￾based power connections, strengthen utilities and other power sector entities, and increase regional power trade. EAEP’s TOC assumes that by expanding affordable and reliable electricity services, EAEP will support development priorities including inclusive growth, increase security, and improve health and education outcomes through community engagement. The Nigeria Power Sector Program (NPSP) implemented in coordination with USAID/Nigeria is designed to address specific constraints in the energy sector across four outcomes: private sector investment; development of the off-grid market; improved enabling environment for private sector participation; and improved liquidity of the distribution sector. The program aims to accelerate energy transactions that lead to increased power generation and increased connections to end-users by working with government partners to create the policy, legal, and regulatory frameworks necessary for private sector investment and financial sustainability within the power sector. Power Africa Off-Grid Program (PAOP), or Beyond-the-Grid, supports USAID Power Africa’s goal of doubling electricity access across Sub-Saharan Africa by 2030 by increasing the number of households and businesses connected to electricity via on-grid or off-grid (household solar and micro￾grids) solutions. PAOP’s TOC assumes that improving markets and regulatory mechanisms and access to private financing will yield a subsequent increase in the total number of off-grid connections and better development outcomes. Program activities focus on increasing the number of off-grid connections with targets of 25 million to 30 million new connections by 2030, increasing private sector financing, and improving the enabling regulatory environment for investment. The West Africa Energy Program (WAEP) is designed to expand the support of and access to affordable and reliable grid-connected electricity services across West Africa and advance development outcomes of inclusive growth, security, and improved health and education. WAEP’s outcomes also include strengthening the performance of national utilities and power sector entities and launching a regional power market. Similar outcomes, except for the regional market, are also elaborated separately for Ghana. WAEP’s TOC proposes to achieve these outcomes by aligning energy reform and electrification goals with investment opportunities, local and regional resources, and increased human capital. 1.3 EVALUATION OBJECTIVES AND SCOPE This section of the work plan lays out the scope and objectives of the four separate evaluations. Specific evaluation objectives and scope are described below for each of the four program evaluations. For each evaluation, it is anticipated that a 25-page final report will be generated and will focus on summary findings and recommendations with annexes as appropriate. East Africa Energy Program (EAEP): A key focus area for the evaluation is determining how the program is addressing ongoing challenges, given national priorities, gaps in human and institutional capacity, and fragmented systems required to reach energy equity. The evaluation will examine how the program’s platform addresses the challenge of regional integration of technical and human systems by examining four program dimensions: workforce development and institutional strengthening; improvement in national and regional planning and policy; coordination and information sharing through information systems; and the facilitation of public-private dialogue. ICF will explore whether and to what extent EAEP has demonstrated results of greater reliability, lower prices, and increased access and if EXTERNAL PERFORMANCE EVALUATION OF EAEP | JUNE 2023 | 29 these outcomes are well correlated to “meaningfully integrated national systems” to achieve results. The evaluation will explore cross-cutting issues to determine effectiveness and identify models and areas of improvement across countries. To identify program efficiencies and lessons, ICF will review the program’s ‘One-Team’ management approach for lessons that can be shared with RTI International and partners. Nigeria Power Sector Program (NPSP): Evaluation focus areas will address how NPSP’s activities have responded to the lack of electrical access as a critical constraint on Nigeria’s economic development and an underlying cause of poverty. This focus includes determining the extent to which program activities have evolved to address longer-term sustainability as well as how effective the IP’s management approach was in achieving Nigeria’s rural electrification goals in a manner that addresses energy equity. We will determine the extent to which power and investment targets aided the program to address unintended systemic issues necessary for self-reliance and financial stability. The evaluation will also examine NPSP’s performance in response to disruptions to markets and enabling environments. Power Africa Off-Grid Program (PAOP), or Beyond-the-Grid: The evaluation design will assess the PAOP TOC for relevance and gather contextual information from private companies, governments, and investors from each region to determine whether PAOP is on track to achieve its targets as well as the feasibility of achieving those targets. We will analyze the program’s effectiveness in increasing the number of off-grid companies active in Sub-Saharan Africa and determine whether PAOP has been successful in driving down sector costs, making energy more accessible. We propose to give particular emphasis to assessing program delivery and identifying best practices and models that can be shared across the region. West Africa Energy Program (WAEP): The evaluation will examine the extent to which the program expanded the support of and access to affordable and reliable grid-connected electricity services across West Africa and advanced development outcomes of inclusive growth, security, and improved health and education. We will determine if WAEP’s outcomes also include strengthening the performance of national utilities and power sector entities and launching a regional power market were on track. Similar outcomes, except for the regional market, are also elaborated separately for Ghana, will be reviewed. Finally, WAEP’s TOC which proposes to achieve these outcomes by aligning energy reform and electrification goals with investment opportunities, local and regional resources, and increased human capital will be reviewed as well as assessing the effectiveness of the program delivery and contractor’s monitoring and documentation process. 1.4 EVALUATION TEAM MEMBERS AND RESPONSIBILITIES Each individual evaluation team has a combination of members that have technical and evaluation experience. In addition to the individual evaluation teams, there is the ICF Backstopping Team that is composed of individuals external to the evaluation teams. Evaluation Team Structure All four evaluation teams will be structured in the same way, with the caveat that the PAOP Team will have an additional evaluation specialist to account for the change in the PAOP Team Lead position. One independent consultant will serve as the Team Lead for each evaluation. These Leads are remote staff EXTERNAL PERFORMANCE EVALUATION OF EAEP | JUNE 2023 | 30 who are not based in any of the program countries. They will lead data collection, analysis, and triangulation, as well as drafting the final evaluation report for their respective team. Each Team Lead will be supported by at least one evaluation specialist and one energy expert. These support individuals will compile resources for the desk study, identify key stakeholders, and contribute to data collection, analysis, and triangulation. They will also assist in the preparation of the final evaluation report for their respective team. Backstopping Team Structure As shown in Exhibit 2 below, the evaluation teams will receive evaluation, technical, and administrative support, and oversight from the Backstopping Team. The Backstopping Team will facilitate meeting planning, scheduling, and other logistical support related to the Power Africa MEL Team. This may include support related to data collection, such as scheduling key informant interviews, as needed to support in-country specialists. The Backstopping Team will also provide evaluation support, such as reviewing data collection instruments, and technical support, such as verifying sectoral knowledge, to the evaluation teams, as needed. Exhibit 2: Evaluation Team Organization EXTERNAL PERFORMANCE EVALUATION OF EAEP | JUNE 2023 | 31 1.5 COORDINATION WITH USAID POWER AFRICA Close coordination with the Power Africa MEL Team will be essential to the timely and effective execution of this assignment. This coordination will be led by the Backstopping Team, though the evaluation teams may engage occasionally with the Power Africa MEL Team. Meanwhile, the evaluation teams will primarily engage with the Task Order CORs and Implementing Partners for each of the four Power Africa programs to be evaluated. After finalization of the work plan, Team Leads will be responsible for sharing weekly updates with the Backstopping Team and the Power Africa MEL Team. The Backstopping Team will also package and deliver monthly progress reports to be shared with the Power Africa MEL Team after finalization of the work plan. 1.6 OVERVIEW OF WORK SCHEDULE AND DELIVERABLES The evaluation will be conducted over a 23-week period that began on January 3, 2023 and concludes on June 9, 2023. The key evaluation deliverables and milestones are listed below, and Table 1 below further presents the evaluation work schedule. Milestones and Deliverables. All dates shown are in 2023. ▪ Kick-off meeting with USAID: January 10 ▪ Draft evaluation work plan submitted to the Power Africa MEL Team: January 13 ▪ Branding and marking plan: January 13 ▪ Draft evaluation design: January 24 ▪ Work plan returned by USAID: January 27 ▪ Evaluation design approved by USAID: February 7 ▪ Final draft of work plan and evaluation design: February 10 ▪ Work plan approved by USAID: February 17 ▪ Finalization of document library: March 3 ▪ Finalize data collection: March 30 ▪ Finish data analysis and interpretation: April 14 ▪ ICF-USAID collaborative workshop: April 21 ▪ Evaluation Report Outlines submitted to Power Africa MEL Team: April 21 ▪ Draft reports #1 submitted to Power Africa MEL Team: May 5 ▪ Presentation to Power Africa MEL Team: May 5 ▪ Receive comments from Power Africa MEL Team: May 12 ▪ Draft reports #2 submitted to Power Africa MEL Team: May 19 ▪ Receive comments from Power Africa MEL Team: June 2 ▪ Final evaluation reports submitted to Power Africa MEL Team: June 9 EXTERNAL PERFORMANCE EVALUATION OF EAEP | JUNE 2023 | 32 Exhibit 3: Evaluation Work Schedule W0 W1 W2 W3 W4 W5 W6 W7 W8 W9 W10 W11 W12 W13 W14 W15 W16 W17 W18 W19 W20 W21 W22 W23 Contract signed Contract start date Kick off meeting with PA MEL Team Branding and Marking Plan submitted to PA MEL Team Work Plan submitted to PA MEL Team (first draft) Work Plan reviewed by PA MEL Team and returned with comments Evaluation Design submitted to PA MEL Team (first draft) Evaluation Design reviewed and approved by PA MEL Team Final draft of Work Plan submitted to PA MEL Team Final draft of Work Plan reviewed and approved by PA MEL Team Determine list of key informants Develop evaluation instruments Establish document library EXTERNAL PERFORMANCE EVALUATION OF EAEP | JUNE 2023 | 33 W0 W1 W2 W3 W4 W5 W6 W7 W8 W9 W10 W11 W12 W13 W14 W15 W16 W17 W18 W19 W20 W21 W22 W23 Data collection activities Data analyses ICF-USAID collaborative workshop Develop Evaluation Report outlines Draft report #1 submission Presentation to PA MEL Team USAID review of reports Draft report #2 submission USAID review of reports Final reports submission USAID review and approve the final report EXTERNAL PERFORMANCE EVALUATION OF EAEP | JUNE 2023 | 34 2. EVALUATION DESIGN AND METHODOLOGY ICF’s detailed evaluation design elaborates a clear plan for conducting each program evaluation. The design has been informed by each individual program’s theory of change and expected outcomes and addresses evaluation questions that have been refined during consultations with the USAID Power Africa MEL Team. For this evaluation design, a detailed program-specific evaluation matrix has been developed that sets out the key evaluation questions and sub-questions, indicators, data collection and analysis methods, and sources and types of data (see Annex 1). Each individual evaluation will tailor the overarching evaluation design to the program context and will be informed by the desk review and internal consultations. All aspects are subject to revision, including the sub-questions, data sources, and design strategies. Data availability and other factors may inform a prioritization or elaboration of the evaluation questions. The evaluation design elaborates the data collection methods, including documents to be reviewed and proposed selection criteria for identifying key informants, focus group members, and survey respondents, identifying respondents that represent stakeholders from partner organizations, key government stakeholders, civil society and private sector entities, and other organizations involved in the sector (e.g., other development partners or sector experts). The selection criteria for key informants and respondents will be informed by the program’s interventions and key assumptions identified in the program’s theory of change. Program-specific data collection plans will present a framework and methodology for data collection and analysis and present a strong rationale for identified number of key informant interviews, group discussions, and surveys. These plans will be shared in the final evaluation report. 2.1 DATA COLLECTION METHODS AND TOOLS Evaluation teams will employ a combination of the following primary types of data collection. The final set of methods will be determined by the desk review and approved set of key questions and areas of inquiry for each evaluation. Document Review. Before collecting data and finalizing evaluation designs and data collection plans, ICF will review relevant program documents, including program implementation plans, annual Work Plans, annual and quarterly performance reports, performance monitoring plans and systems, technical reports, and other relevant program publications. We will also review external publications on the energy sector and other interventions in the similar sub-sector and geographic area to consider other best practices and models. Document review will inform evaluation team planning and provide background and context for probing questions associated with the key informant interview and focus group discussions related to individual evaluation questions. Evaluation team members will take notes to highlight key learnings and knowledge gaps; these will be shared, triangulated, and discussed as a team, and we will incorporate the findings into the evaluation reports. Key Informant Interviews (KIIs). ICF will develop semi-structured interview guides, tailored by program-specific stakeholder type. These guides will be based on the document review, which will help provide additional nuance for each program, and the evaluation questions outlined in the evaluation matrix below. We will iteratively test and improve these interview guides. The basic structure of the EXTERNAL PERFORMANCE EVALUATION OF EAEP | JUNE 2023 | 35 interview guides, with examples, was provided in draft form on January 31. The final guides will be shared as an annex in the final evaluation report, to allow for mid-course corrections, as needed. Semi-structured interviewing enables the interviewer to follow an agreed-upon set of questions, while retaining flexibility to adapt the process depending on the interviewee and to probe where a fruitful line of questioning emerges. ICF will design the interview guides to align to the specific questions and sub￾questions that the evaluation aims to answer. The evaluation will benefit from our highly experienced interviewers who bring methodological rigor and technical expertise of the energy sector, enabling a richer, more holistic, and nuanced account of factors influencing each program’s performance. The ICF Team will schedule interviews via preferred video platforms (e.g., Google Meet, Microsoft Teams, Zoom, Skype, or WhatsApp), taking internet access limitations into consideration. Interviewers will take detailed, typed interview notes, and, when feasible and with the consent of interviewees, we will record KIIs to facilitate validation. Evaluation teams will anonymize interview notes in line with standard evaluation ethics and coded in software to facilitate qualitative analysis. Focus Group Discussions (FGDs). ICF will design FGD guides according to the program-specific evaluation questions. Guides and lists of questions will be tested iteratively to improve their efficacy. These FGD guides will follow the illustrative interview guide provided to USAID on January 31. The final guides will be shared as an annex in the final evaluation report. Focus groups will take advantage of group dynamics and will be gender sensitive, allowing for greater security and open dialogue. Discussion sessions with key stakeholders will be conducted in small groups of six to eight participants to explore topics related to their knowledge, attitudes, practices, and other relevant insights specific to program￾level activities. Given that FGDs will need to take place virtually, the ICF Team will consider bandwidth constraints in determining the final number and scope of groups for each evaluation. Survey. As deemed useful, ICF will design a brief online survey to collect additional information on relevance, effectiveness, impact, and sustainability criteria that is reflective of the evaluation questions. As part of the evaluation design process for each evaluation, we will work closely with the USAID Power MEL Team to explore the usefulness and feasibility of a survey. The final decision for including a survey will be contingent upon whether contact information of respondents is readily available and the relevancy of the final evaluation design. The use of surveys will also be dependent upon initial desk reviews and key informant interviews and will vary by program. We would plan to use both discrete answer choices (Likert-scale) for quantitative analysis and open-ended questions to gather richer qualitative information, with follow-up for any special or outlier comments requiring clarification or explanation. All individual responses provided through the survey would be treated as strictly confidential and will not be shared outside the ICF Team; aggregated results would be shared with USAID Power Africa MEL Team and included in the evaluation report. ICF would provide a comprehensive methodology of each survey in each evaluation design document. In developing and analyzing the results, ICF will draw on the expertise of our In-country Specialists. 2.2 DATA ANALYSIS The evaluation team will use several analytical methods over the course of individual evaluations, to identify key evidence-based findings. EXTERNAL PERFORMANCE EVALUATION OF EAEP | JUNE 2023 | 36 Qualitative Data Analysis ICF will conduct an in-depth, structured data analysis process to ensure that findings are credible, valid, and evidence based. The qualitative data analysis will also identify data that help to interpret and triangulate the findings from the analysis of quantitative data and case studies. The data review process will focus on the use of content analysis and pattern observation to draw understanding and context from the information. Evaluation teams will use descriptive analysis to identify specific content that speaks to the overall program interventions in the program areas as well as program-specific themes. In addition, any recordings of KIIs and FGDs will provide teams with direct quotes from respondents that can be used as supporting material for findings and conclusions in the draft and final evaluation reports. Data coding is an exercise to thematically group data (the content of the transcripts) to further support qualitative data analysis; how the data are coded becomes the basis for analysis. Each ICF evaluation team will develop a set of thematic codes specific to each evaluation. Each team will code (contingent upon final evaluation designs) the qualitative data and identify emergent themes using the Dedoose analysis software, a cross-platform computer application that will be used to analyze qualitative and quantitative data as a part of a mixed-methods evaluation. Individual teams will develop deductive codes that correspond to the research questions outlined in the interview transcripts. The teams will also incorporate inductive codes, capturing additional emergent or unexpected themes. With the codebook in place, the teams will read the translated transcript with each statement or exchange in the transcript thematically coded. The Team Leads will oversee this process and periodically check the coding of the other team members supporting this task. The evaluation teams will analyze data using content and contribution analysis. Content analysis will be used to identify themes and trends within and across respondent groups. We will also bring further understanding to contextual and environmental factors and their influence on behaviors, as possible. Contribution analysis will explore causal mechanisms and verify theories of change based upon identifying critical underlying assumptions. Quantitative Methods ICF will use a mix of data analysis methods to review program-level quantitative data. The ICF evaluation teams will analyze each program’s performance monitoring data, obtained through data management systems (USAID Power Africa Power Africa Information System [PAIS] and Power Africa Tracking Tool [PATT] systems) to determine whether annual program targets have been met. Baseline figures for all performance monitoring indicators will be compared to endline results for activities that are at the end of their program cycles. Each ICF evaluation team will also analyze disaggregated data to provide a deeper assessment of results focusing on issues of energy equity, gender (where relevant), self-reliance and other cross-cutting issues. These data analyses will help in identifying critical assumptions, challenges, and gaps in addressing each program’s theory of change. The ICF teams will use other methods such as trend analysis to compare year-to-year figures across standard indicators of particular interest. The evaluation teams will triangulate the results of the qualitative and quantitative analyses during a working group session to identify key findings and conclusions. This process will ensure that all findings are supported by multiple evidence sources and analyses. EXTERNAL PERFORMANCE EVALUATION OF EAEP | JUNE 2023 | 37 2.3 DATA QUALITY ASSURANCE The evaluation teams will adhere to high professional standards in collecting and analyzing data at every stage. Interviews will be attended by at least two evaluation team members (barring exceptional circumstances), who will each take detailed notes. The quality and validity of these notes will be assured in the process of consolidating them into a single file documenting each individual or group interview, which will be shared with the entire evaluation team. Throughout, the teams will discuss and develop emergent findings in an iterative process. 3. DATA COLLECTION SCHEDULE Each Team Lead will conduct an initial interview with the Contracting Officer’s Representative (COR) for their respective program. Based on this initial interview, the Team Leads will review the detailed list of stakeholders provided by the USAID Power Africa MEL Team and will determine an appropriate interview schedule. A meeting will also be scheduled with USAID Power African Monitoring Specialist to better understand and obtain access to the PATT and PIAS data for all programs. In addition to the priority informants identified by the Power Africa MEL Team, the evaluation teams intend to contact other individuals for interviews. Both the interview schedule and the final list of stakeholders will be included as an annex in the Final Evaluation Report, as both will evolve throughout the data collection process. Each Team Lead expects to interview up to 50 stakeholders that fall into the following groups: NGOs/CSOs, development partners, government ministries, public and private utility companies, and energy service providers/other private sector actors. Additional stakeholder groups may be identified during the data collection process. EXTERNAL PERFORMANCE EVALUATION OF EAEP | JUNE 2023 | 38 ANNEX C: DATA COLLECTION INSTRUMENTS INTERVIEW GUIDES GENERAL INTRODUCTION FOR ALL INTERVIEWS Provide a brief introduction that covers: • Short introductions of role and positions within organizations and relation to EAEP; • The aims of the interview and the evaluation; [See optional talking points below] o USAID has contracted ICF to conduct an evaluation of Power Africa’s EAEP along with evaluations of three other programs. o The current schedule has us wrapping up data collection by early April, with a draft report in May and final in June. o The evaluations focus on appropriateness of the design, overall performance, effectiveness of different components and adaptive management. • The expected length of the interview (e.g., 30, 45, or 60 minutes); and • Confidentiality —i.e., that comments made by the interviewee will not be attributable; results of the evaluation will be published; and results may inform future programming. • Obtain consent if the interview will be recorded. BACKGROUND ON EAEP FOR EVALUATIONS [Use as needed to remind yourself or the interviewee of key facts] EAEP is a $65.5 million program running from 2018—2022 and recently extended to September 2023. It is designed to optimize the power supply, increase grid-based power connections, strengthen utilities and other power sector entities, and increase regional power trade. EAEP’s TOC assumes that by expanding affordable and reliable electricity services, EAEP will support development priorities including inclusive growth, increase security, and improve health and education outcomes through community engagement. The project is implemented in 10 countries in East Africa. These include Burundi, the Democratic Republic of Congo, Djibouti, Eritrea, Ethiopia, Kenya, Rwanda, Somalia (including Puntland and Somaliland), Tanzania, and Uganda. EAEP is an implementing mechanism that was awarded to RTI in December 2018. In each of the 10 countries, the contractor is expected to provide technical assistance and capacity building in the focus countries while collaborating closely with its consortium members: Khulisa Management Services, Tetra Tech, Fieldstone Africa, The Center for Climate Strategies, and Norton Rose Fulbright. The implementing mechanism was expected to implement this activity to address critical opportunities across the following four objectives: 1. Optimized power supply; 2. Increased grid-based power connections; 3. Strengthened utilities and other power sector entities; and, 4. Increased regional power trade. EXTERNAL PERFORMANCE EVALUATION OF EAEP | JUNE 2023 | 39 PROTOCOL FOR KEY INFORMANT INTERVIEWS: INSIDER: SENIOR IMPLEMENTATION TEAM, POWER AFRICA, USAID STAFF Date: Name: Country: Title/Role: Sex: ☐ Male ☐Female Organization: Interviewer(s): Relevance, Coherence: • Were the theory of change and underlying assumptions robust? • How relevant was the program design and structure (e.g., right partners/partnership model, addressing relevant needs and opportunities)? • How coherent and complimentary was the program with wider energy transition efforts (e.g., USAID Power Africa interagency coalition, donors, and private sector partners)? • Did the program theory of change over time? • Do the assumptions in the theory of change hold true or does the theory of change need to be adjusted? If adjustments are needed, what concrete recommendations can be made for the types and scope of program implementation adjustments? Program Delivery: Efficiency and Effectiveness • How efficient were the operational systems? o How efficient has the management approach and team been to delivering results? o What has been the quality and usefulness of systems, information management and tools and reporting o What has been the program’s added value and effectiveness of specific design elements? • What is the program’s overall effectiveness of its technical assistance and capacity building initiatives? o What has been the added value of advisors in program delivery? o What is the effectiveness of the program’s stakeholder model? o What is the effectiveness of the US partnership model? o What is the effectiveness of the donor model? • Adaptive management o To what extent has management been able to learn and bring adaptive management to deliver results? Delivery of Program Results: Outcomes, Impact, Sustainability • How effective has the program been in achieving the intended purpose, objectives, and outcomes? Has the program met, or is it likely to meet outcome targets as set out in the results framework as envisaged? • Has the program been effective in promoting inclusivity and gender equality? • What have been the key transformational outcomes of the program (e.g., systems change, scaling, acceleration)? • Are there any additional co-benefits? Are outcomes aligned with wider social, economic, and environmental considerations? • Have there been any unintended consequences? • Are results likely to be sustainable? Key Barriers to Results Delivery EXTERNAL PERFORMANCE EVALUATION OF EAEP | JUNE 2023 | 40 • What have been the key operational barriers to delivering results? • What are the program’s specific challenges? [prompt – identify 2-3 major challenges/barriers] • What effect has COVID-19 had on program implementation and overall performance? [prompt – has COVID-19 been a significant barrier?] • How were/are these COVID-19 related impacts being monitored and documented (addressing ongoing performance changes)? Recommendations and Lessons Learned • What are the program’s achievements, challenges, best practices, lessons learned and recommendations from program implementation to date? • What worked and what didn’t not work well. • What should be done differently in the future? Why? Closing • Any other comments? Is there something we missed or that you expected we’d also talk about today? PROTOCOL FOR KEY INFORMANT INTERVIEWS: EXTERNALS (BENEFICIARIES, PEERS, INTERNATIONAL PARTNERS) Date: Name: Country: Title/Role: Sex: ☐ Male ☐Female Organization: Interviewer(s): Background • Name and role • Organization and its main area of focus? Type of engagement • Describe your engagement with Power Africa • What was the nature of the relationship with Power Africa? • What issues was PA trying to address or solve? • Which support or services did the program provide? • Were any other partners involved and if so, which and in what way? • Did the relationship evolve over time? or • Role as peer organization or expert in energy field if not a beneficiary. Experience of engagement with Power Africa • What was your practical experience of engaging with the Power Africa team? • Were they strategic in their outlook and seeking to add value? • Were they responsive, professional? • Were any processes or reporting requirements straightforward? • Were they flexible and able to adapt to your needs or changing circumstances? • To what extent did COVID-19 impact upon the engagement? EXTERNAL PERFORMANCE EVALUATION OF EAEP | JUNE 2023 | 41 Results and impact • What were the benefits of engaging with Power Africa for your organization? • What were the results (energy sector, cross cutting (e.g. gender) • In what way has Power Africa had a broader impact on the power sector in your country? • Do you think that Power Africa results are sustainable over time? • What are the key barriers that remain in your country/sector? Recommendations • What are the key lessons, best practices should be considered for the next phase (2022-26)? • What topics or issues should Power Africa focus on going forward in your country? • How might Power Africa improve its operating model and structure? Closing • Any other comments? Is there something we missed or that you expected we’d also talk about today? EXTERNAL PERFORMANCE EVALUATION OF EAEP | JUNE 2023 | 42 ANNEX D: SOURCES OF INFORMATION LIST OF ORGANIZATIONS CONSULTED The following table lists the organizations represented in the 59 KIIs conducted. In several cases, such as for Implementing Partners and USAID, there are multiple informants per organization. Organization U.S. Government – 7 Kilembe Investment Ltd USAID/Power Africa KIS Host country government – 6 Kyegegwa Rural Electricity Cooperative Society Ltd EPRA (Kenya) OffGridBox ERA (Uganda) Tanzania Electric Supply Company Limited (TANESCO) MININFRA (Rwanda) Uganda Electricity Transmission Company (UETCL) Puntland Electricity Development Agency (Somaliland) Waamo Energy Service Company (WESCO) Public or private energy company – 20 Other – 6 Africa Solar Industry Association (AFSIA) Energy for Growth Amahoro Energy Ltd Ethiopian Women in Energy (EWiEN) ARC PowerHer Berbera Energy Company (BEC) TaWOED East African Power Development partner (public and private) – 3 Energy Development Corporation Limited (EDCL) GIZ Engie SIDA Finer Green WB HydroNeo IPs – 17 Kalangala Infrastructure Services Limited (KIS) RTI Kenergy Tetratech EXTERNAL PERFORMANCE EVALUATION OF EAEP | JUNE 2023 | 43 Organization Kenya Electricity Generating Company Ltd BIBLIOGRAPHY OF DOCUMENTS REVIEWED The evaluation team primarily reviewed project documents (i.e., annual work plans, quarterly progress reports, portfolio reviews) that were provided to ICF by the EAEP COR and the implementing team. In addition to these program documents, the evaluation team consulted the following public reports: Aynte, A., E. Chen, and D. Mozersky. 2022. “Powering Ahead: The United Nations and Somalia’s Renewable Energy Opportunity.” Washington, D.C.: Stimpson. East Africa Power Tool. 2022a. “A Road Map for Increasing Regional Power Trade in Eastern Africa.” U.S. Agency for International Development. —. 2022b. “Final EAPP Market Design Final Report.” U.S. Agency for International Development. International Energy Agency. 2022. “Africa Energy Outlook 2022.” World Energy Outlook Special Report. International Renewable Energy Agency. 2021. “Planning and Prospects for Renewable Power: Eastern and Southern Africa.” International Renewable Energy Agency and African Development Bank Group. 2022. “Renewable Energy Market Analysis: Africa and its Regions.” Manirambona, E., S. M. Talai, and S. K. Kimutai. 2022. “A review of sustainable planning of Burundian energy sector in East Africa.” Energy Strategy Reviews 43. Netherlands Enterprise Agency. 2019. “Final Energy report Tanzania.” U.S. Agency for International Development. 2016. “Development of Kenya’s Power Sector, 2015–2020.” —. 2022. “East Africa Energy Program Fact Sheet.” U.S. Energy Information Administration. 2020. “Off-Grid Electricity Development in Africa: Uncertainties and Potential Implications for Electric Power Markets.” Washington, D.C.: U.S. Department of Energy. World Bank Group. 2019. “Learning from Power Sector Reform: The Case of Kenya.” Policy Research Working Paper 8819. DATABASES REVIEWED The evaluation team reviewed EAEP’s performance monitoring data, obtained through two data management systems: • USAID Power Africa Power Africa Information System • Power Africa Tracking Tool EXTERNAL PERFORMANCE EVALUATION OF EAEP | JUNE 2023 | 44 ANNEX E: DISCLOSURE OF ANY CONFLICTS OF INTEREST None. ANNEX F: STATEMENTS OF DIFFERENCES (IF APPLICABLE) None.