NOT FOR DISTRIBUTION April 2023 PHOTO CREDIT: TOLU JETHRO BADE EVALUATION External Performance Evaluation of the West Africa Energy Program (WAEP) June 2023 This publication was produced with the support of the United States Agency for International Development (USAID) Power Africa initiative under the terms of the Power Africa Contract Evaluation Services Task Order No. 720-674-23-F-00001. EXTERNAL PERFORMANCE EVALUATION OF WAEP | JUNE 2023 | i Report Title: External Performance Evaluation of the West Africa Energy Program (WAEP) Lead Author: Anne-Marie Verbeken Supporting Authors: Benoit Payard, Moustafa Sowou, Sujata Ram, Nicole Economu, and Isabel Morford￾Cheibub. Submitted by: ICF Activity Title: Power Africa Contract Evaluation Services USAID Task Order Number: 720-674-23-F-00001 Completion Date: June 9, 2023 Photo Caption: Drone view of the installed solar panels at the Dambala community health center in Sierra Leone. The authors’ views expressed in this publication do not necessarily reflect the views of the United States Agency for International Development or the United States Government. EXTERNAL PERFORMANCE EVALUATION OF WAEP | JUNE 2023 | ii ACKNOWLEDGEMENTS On behalf of the ICF Evaluation Team, I, Mark Wagner (Evaluation Oversight and Quality Assurance Officer), would like to extend my sincere appreciation to the many USAID staff, Implementing Partners, host country government representatives, development partners, utility and energy companies, and other key informants who contributed their valuable time and insights to support the evaluation process. I am particularly grateful to the USAID staff and Implementing Partners who connected our team to program stakeholders, provided key documentation, and improved our understanding of Power Africa’s experiences, successes, and challenges, as well as to the Contracting Officer’s Representative, Yugi Nair, and Alternate Contracting Officer’s Representative, Ngonidzashe Makaza, for their continued support and the trust they placed in us throughout the process. EXTERNAL PERFORMANCE EVALUATION OF WAEP | JUNE 2023 | iii TABLE OF CONTENTS ACRONYMS AND ABBREVIATIONS IV EXECUTIVE SUMMARY V 1. INTRODUCTION 1 1.1 BACKGROUND ON EVALUATION 1 1.2 ORGANIZATION OF THE REPORT 2 2. EVALUATION METHODS 2 2.1 EVALUATION QUESTIONS AND SCOPE 2 2.2 EVALUATION METHODOLOGY 3 2.3 EVALUATION LIMITATIONS 3 3. FINDINGS 4 3.1 RELEVANCE AND COHERENCE 4 3.2 PROGRAM DELIVERY: EFFICIENCY AND EFFECTIVENESS 7 3.3 DELIVERY OF PROGRAM RESULTS: OUTCOMES, IMPACT, SUSTAINABILITY 9 3.4 KEY BARRIERS TO THE DELIVERY OF RESULTS 15 4. CONCLUSIONS AND RECOMMENDATIONS 16 4.1 CONCLUSIONS/LESSONS LEARNED 16 4.2 RECOMMENDATIONS 18 ANNEXES 21 ANNEX A: EVALUATION STATEMENT OF WORK 21 ANNEX B: EVALUATION DESIGN AND MATRIX 25 ANNEX C: DATA COLLECTION INSTRUMENTS 37 ANNEX D: SOURCES OF INFORMATION 43 ANNEX E: DISCLOSURE OF ANY CONFLICTS OF INTEREST 46 ANNEX F: STATEMENTS OF DIFFERENCES 46 EXTERNAL PERFORMANCE EVALUATION OF WAEP | JUNE 2023 | iv ACRONYMS AND ABBREVIATIONS AfDB African Development Bank BOAD Banque Ouest Africaine de Développement (West African Development Bank) CLIN Contract Line Item Number COR Contracting Officer’s Representative COVID-19 Coronavirus SARS-CoV-2 2019 DFC United States International Development Finance Corporation DFI Development Finance Institution ESRP Energy Sector Recovery Program FY Fiscal Year GIS Geographic Information System IP Implementing Partner IPP Independent Power Project KPI Key Performance Indicator MCC Millenium Challenge Corporation MW Megawatt NREL National Renewable Energy Laboratory PA Power Africa PATRP Power Africa Transactions and Reforms Program PPA Power Purchase Agreement PV Photovoltaic SONABEL Société Nationale d’Électricité du Burkina TBI Tony Blair Institute T&D Transmission and Distribution TOC Theory of Change TWh Terawatt-Hour USAID United States Agency for International Development USG United States Government USTDA United States Trade and Development Agency WAEP West Africa Energy Program WB World Bank EXTERNAL PERFORMANCE EVALUATION OF WAEP | JUNE 2023 | v EXECUTIVE SUMMARY EVALUATION OVERVIEW The United States Agency for International Development (USAID) Power Africa (PA) Program Office contracted ICF to conduct external performance evaluations of four PA programs, including the West Africa Energy Program (WAEP), to improve program performance management and efficiency. The lessons drawn from these evaluations are directed toward technical teams, activity managers, and senior management. ICF developed an overarching evaluation design framework for all four PA programs, which included five focus areas: (1) relevance and coherence of the theory of change and program design; (2) efficiency and effectiveness of program delivery; (3) outcomes, impact, and sustainability of program delivery; (4) key barriers to the delivery of results; and (5) lessons learned and recommendations. The evaluation team employed a combination of data collection and analysis methods. The team reviewed relevant program and external documents and triangulated findings from the desk review with data collected through more than 80 semi-structured key informant interviews. SUMMARY OF WAEP Deloitte Consulting LLP was awarded WAEP under Task Order No. 720-674-18-F-00003 of Power Africa Indefinite Delivery, Indefinite Quantity Contract No. 720-674-18-D-00003. WAEP is a four-year program with a total estimated cost of $73,850,693 and a period of performance from July 15, 2019 to July 14, 2023. Under Contract Line Item Number (CLIN) 1, WAEP focuses on 23 countries in the West Africa Regional Mission, and under CLIN 2, WAEP focuses separately on Ghana. WAEP is designed to expand the support of and access to affordable and reliable grid-connected electricity services across West Africa and advance the development outcomes of inclusive growth, security, and improved health and education. WAEP’s outcomes include strengthening the performance of national utilities and power sector entities and launching a regional power market under CLIN 1. Similar outcomes, except for the regional market, also are elaborated separately for Ghana in CLIN 2. CLIN 1 is meant to increase electricity availability and access in West Africa while objectively quantifying and measuring progress toward four key outcomes: • Outcome 1: Supply of power increased • Outcome 2: Access to reliable and affordable grid-based power increased • Outcome 3: Performance of national utilities and power sector entities improved • Outcome 4: Launch of a regional power market accelerated CLIN 2 is meant to increase electricity availability and access in Ghana while objectively quantifying and measuring progress toward three key outcomes: • Outcome 1: Optimized supply of power • Outcome 2: Access to reliable and affordable grid-based power increased • Outcome 3: Performance of national utilities and power sector entities improved KEY TAKEAWAYS WAEP would benefit from a theory of change (TOC) that enhances the coherence of the program by articulating its own position and comparative advantages in the ecosystem of United States Government (USG) agencies and other relevant development partners in the sector. An EXTERNAL PERFORMANCE EVALUATION OF WAEP | JUNE 2023 | vi improved TOC also would pay greater attention to the feasibility of proposed actions in terms of the resources and conditions for their effectiveness. The program also would benefit from a greater examination of the assumptions made in order to avoid unrealistic targets and timelines. WAEP’s first three years of implementation were overshadowed by the COVID-19 pandemic, directly impacting investment, slowing electrification efforts, and constraining program implementation as a result of the lockdowns. The effects of the pandemic continue to be felt and were made worse by the war in Ukraine, with growing debt and inflation pressures, and a higher cost of capital due to higher interest rates. It is not possible to disentangle the effects of the pandemic from ongoing trends in the sector, such as stagnant growth in transmission and distribution and a decline in the growth of new generation capacity. These factors need to be considered in the evaluation of the performance of WAEP. The need for technical and financial resources in the West African power sector are immense and unmet. The evaluation team found that there is demand for technical partners such as WAEP who can provide targeted and advanced technical expertise, a range of advisory services, and capacity building, including a wide variety of training. A significant number of beneficiary stakeholders valued WAEP’s support and wanted more of it, with the feedback on relevance broadly correlating with responsiveness and timeliness, level of engagement and skills transfer, and added value and impact. Although the operating model was highly valued by independent power project (IPP) developers, the need to deliver results led to an approach with a lighter touch that focused on advising government counterparts regarding resolving bottlenecks in the evaluation and negotiation of more advanced private power generation projects by ministries of energy or utilities. Stakeholders, in particular the private sector but also government counterparts, placed a high value on WAEP’s unique flexibility, responsiveness, deep engagement, and expertise in moving power generation projects forward and finding solutions. However, the resource-intensive, early-stage project development support model came under pressure from Power Africa management because the results were too slow to lead to a financial close within one to two years. While deeply attractive to government counterparts and private developers alike, the operating model of having a deep bench of expert consultants who can be deployed for flexible assignments to address client technical, financial, and legal advisory needs can be resource intensive while not delivering immediate results. However, a range of stakeholders were of the view that there is high value in supporting early-stage projects to create a pipeline of high-quality power projects that can reach financial close in the future. While support for government counterparts to advance power generation projects is impactful, stakeholders believe that there is a critical need to support early-stage projects, in particular local IPP developers. A significant number of stakeholders expressed a desire for a more consultative approach with greater stakeholder engagement throughout the program to ensure that support is tailored to the country and best suited to meet technical and financial needs, rather than to simply meet WAEP targets. Stakeholders also pointed to limited absorption capacity and identified further efforts and conditions that were needed for training and capacity building to be effective. The evaluation also revealed a funding gap in the form of grants that would have enabled the piloting or implementation of WAEP’s recommendations and advice, such as the piloting of productive use of energy business models in Ghana, the testing and piloting of the ready board solution, or the acquisition of necessary software and hardware. Development partners and other stakeholders often do not understand what support WAEP provides, even when they have worked alongside the program. The lack of understanding appears EXTERNAL PERFORMANCE EVALUATION OF WAEP | JUNE 2023 | vii to partially derive from a lack of clarity on the nature of WAEP’s support and its exact role in the sector and in specific projects, as well as from a lack of stakeholder consultation. These stakeholders are familiar with the U.S. International Development Finance Corporation (DFC) and the Millennium Challenge Corporation; however, they generally do not know what WAEP does beyond providing transaction advisory services in power generation projects. WAEP also seems to have struggled to articulate and communicate a coherent role for itself in fostering regional electricity trade. Overall, WAEP’s visibility among development partners is low and there is a need to clarify its role in the broader energy sector. WAEP underachieved in power generation, improving utility performance, and increasing regional electricity trade but performed moderately on transmission lines and connections. WAEP performed better in Ghana under CLIN 2, where the program benefited from existing USAID networks and support, as well as embedded advisors, some of whom were former Power Africa Transactions and Reforms Program transaction advisors. WAEP was able to provide high-impact financial advisory services at the highest level and directly engage in helping to create new institutions to reform how financial flows in the energy sector are managed among other achievements. Power Africa may have underestimated the weak enabling environment and lack of institutional capacities that are hindering the progress of IPP or hindering the improved performance of utilities, while overestimating its ability to overcome these barriers. Nevertheless, WAEP was able to identify and design solutions to address at least some of the performance problems of select utilities. Attribution claims made related to kilometers of transmission and the number of connections were questioned by some stakeholders and development partners because grid connections and transmission and distribution (T&D) are entirely publicly funded by governments, development finance institutions (DFIs), and bilateral aid agencies that also provide technical assistance. WAEP’s ability to claim connections is entirely reliant on these existing T&D and electrification programs. While stakeholders generally recognized WAEP’s contribution in terms of supporting government counterparts and energy companies in expediting and managing T&D and electrification programs, attribution was considered tenuous or unsubstantiated. Stakeholders perceived that the program would benefit from documenting intermediary contributions rather than solely focusing on high-level outcomes to enhance credibility and demonstrate results. This could be done by creating intermediate metrics that indicate progress toward congressional targets to reflect the actual timeframes to bring projects to a financial close and into operation in West Africa. USAID Missions, with their existing government relations and ability to raise the program’s profile, were seen as instrumental in enabling WAEP to engage government counterparts and position the implementation team to provide high-level support and claim results. Similarly, having in-country embedded advisors was seen as critical to the delivery of results. Another avenue for engaging government counterparts was through the African Development Bank partnership, where WAEP had embedded advisors who provided critical technical and financial advisory support. Collaboration with the Tony Blair Institute for Global Change (TBI) also allowed WAEP to provide valued support to regional entities and government counterparts. TBI has seemingly played a key role in enabling and coordinating WAEP’s support for some regional entities and government counterparts, including through the provision of experts. The expectation that WAEP would be able to leverage the USG ecosystem to access funding or investment was not entirely fulfilled, except for instances of collaboration with the U.S. Trade and Development Agency (USTDA), which funded feasibility studies for early-stage power generation projects supported by WAEP. WAEP and USTDA had a natural complementarity, with WAEP bringing a pipeline of eligible early-stage power generation projects in need of feasibility studies for USTDA to fund. EXTERNAL PERFORMANCE EVALUATION OF WAEP | JUNE 2023 | viii There also was an expectation that WAEP would enable DFC investment in IPP projects because the two entities share pipelines; however, DFC did not invest in any WAEP projects during the implementation period. DFC was not interviewed for this evaluation, so it is unclear what prevented DFC from investing. RECOMMENDATIONS RECOMMENDATION 1: Clarify and focus WAEP’s mission. WAEP would benefit from a clearer, more focused mission that can be communicated to stakeholders and be understood by the development partners. A more focused mission also could inform its operating model by determining the range and quantity of technical expertise required and how that expertise could be efficiently deployed. • ACTION 1: Consider and discuss how WAEP could develop a clearer and more focused mission incorporating depth, range, and quantity of expertise. RECOMMENDATION 2: Consider greater adaptative management. WAEP program implementation would benefit from greater adaptive management of the program to accommodate shifting political priorities and evolving market and institutional dynamics. • ACTION 1: Adopt a less detailed statement of work and put in place a less onerous process for work plan revisions. • ACTION 2: Ensure that there are adequate staffing resources for adaptive management included in the work plan, budget, and Monitoring, Evaluation, and Learning plan for the next program. RECOMMENDATION 3: Adopt more realistic targets and outcomes. WAEP should base its targets on analyses of recent trends and data on generation capacity, transmission, and connections, while considering its limited resources. Similarly, outcomes such as the reduction of aggregate losses are not realistic because technical losses are particularly hard to reduce without major investments and efforts beyond what is available to Power Africa. Furthermore, other key performance indicators (KPIs), such as the reduction of cost per connection to the utility, also are not realistic because such data are not collected by the utilities. • ACTION 1: Revise the methodology to identify more appropriate indicator targets. • ACTION 2: Revise KPIs under Outcome 3 to reflect what is achievable based on the available data and resources. RECOMMENDATION 4: Adopt intermediate performance metrics. While reaching financial close for power generation projects is a reasonable milestone and metric for added power generation capacity, WAEP should add a secondary metric that relates to building a pipeline of power generation projects that are expected to reach financial close in the future. This metric would reflect both the foundational work that is required for IPPs, in particular, support for local developers, and the intergenerational aspect of supporting a pipeline of IPP projects. This would enhance WAEP’s impact and sustainability and fill a critical gap in support. Such intermediate metrics would require information on progress regarding the development of projects, detailed explanations of the remaining obstacles to financial close, and recommended approaches for how those obstacles would be resolved. Similarly, intermediate metrics should be considered for transmission and connections outcomes, which are all reliant on existing publicly financed grid expansion and electrification projects and programs co￾financed by DFIs and bilateral agencies. A performance-related metric could be adopted that reflects more effective project management and the resulting impacts and benefits. A higher level KPI could be applied to cases in which WAEP provides transaction advisory services to an IPP equivalent for transmission (i.e., an independent transmission project or a concession). An intermediate metric for fostering regional EXTERNAL PERFORMANCE EVALUATION OF WAEP | JUNE 2023 | ix electricity trade, which reflects the many conditions that must be met before regional electricity trade can take place, also would also be appropriate because all transmission lines and substations are being publicly funded. • ACTION 1: Explore the adoption of intermediary metrics informed by the experiences and lessons learned from the implementation of WAEP. RECOMMENDATION 5: Facilitate access to grant funding. The lack of access to grants for piloting solutions and implementing technical advice, including the acquisition of software and equipment, has been a handicap for the program. Grants also would benefit early-stage IPP projects. Access to grants would make WAEP more effective and enhance the program’s impact and sustainability. • ACTION 1: Identify what conditions are necessary for grants to be effective, then provide grants under those conditions. RECOMMENDATION 6: Given the success of the embedded advisor model, expand it where appropriate. Having an embedded advisor was considered critical for making an impact across WAEP countries. • ACTION 1: Identify where and how the embedded advisor model could be expanded. RECOMMENDATION 7: Enhance coherence and coordination. More emphasis should be placed on stakeholder consultation and engagement throughout the different stages of a workstream—from the conceptual design stage of technical assistance and capacity building to the adoption of technical advice. This would include holding stakeholder consultations that involve development partners to ensure better coherence and coordination. • ACTION 1: Invest in stakeholder consultations at various appropriate levels. RECOMMENDATION 8: Explore replicable technical assistance and conditions for replicability. Successful technical assistance and capacity-building efforts should be assessed and socialized to better understand how they could be applied to the WAEP context. • ACTION 1: Analyze lessons learned to identify replicable or scalable practices, the resources they would require, and the conditions for replicating and scaling them. RECOMMENDATION 9: Move from capacity building to skills transfer. Skills transfer can be achieved through co-construction and more tailored training, which is more easily adopted and absorbed than capacity building and would enhance sustainability. This approach would be more resource intensive, which would have to be considered when developing budgets and metrics. • ACTION 1: Replace capacity building with skills transfer where appropriate and effective. Skills transfer is more oriented toward co-development and co-creation, as well as support for the application and integration of new tools or methods in the organization. RECOMMENDATION 10: Formulate a coherent strategy to support the regional electricity market. Feedback from stakeholders points to the lack of visibility, consultation, and coherence in WAEP’s support for the regional electricity market. • ACTION 1: Develop a strategy that articulates WAEP’s position and potential contribution in the crowded ecosystem of DFIs and other agencies that support the regional electricity market and puts more emphasis on stakeholder consultation and engagement. EXTERNAL PERFORMANCE EVALUATION OF WAEP | JUNE 2023 | 1 1. INTRODUCTION 1.1BACKGROUND ON EVALUATION The United States Agency for International Development (USAID) Power Africa (PA) Program Office has contracted ICF to conduct external performance evaluations of four PA programs, including the West Africa Energy Program (WAEP), to improve program performance management and efficiency. PA seeks to increase access to electricity throughout Sub-Saharan Africa (SSA) with the addition of millions of new connections and tens of thousands of megawatts in new and cleaner power generation. To date, PA has delivered significant transformative impacts in the SSA region, not only in expanding energy services but also in benefiting multiple additional development priorities. The lessons drawn from this evaluation are directed toward technical teams, activity managers, and senior management. DESCRIPTION OF PROGRAM WAEP is designed to expand the support of and access to affordable and reliable grid-connected electricity services across West Africa and advance the development outcomes of inclusive growth, security, and improved health and education. WAEP’s outcomes include strengthening the performance of national utilities and power sector entities and launching a regional power market (under Contract Line Item [CLIN] 1 1). Similar outcomes, except for the regional market, also are elaborated separately for Ghana in CLIN 2. CLIN 1 is meant to increase electricity availability and access in West Africa while objectively quantifying and measuring progress toward four key outcomes: • Outcome 1: Supply of power increased • Outcome 2: Access to reliable and affordable grid-based power increased • Outcome 3: Performance of national utilities and power sector entities improved • Outcome 4: Launch of a regional power market accelerated CLIN 2 is meant to increase electricity availability and access in Ghana, specifically, while objectively quantifying and measuring progress toward three key outcomes: • Outcome 1: Optimized supply of power • Outcome 2: Access to reliable and affordable grid-based power increased • Outcome 3: Performance of national utilities and power sector entities improved THEORY OF CHANGE WAEP’s theory of change (TOC) proposed to achieve these outcomes by aligning energy reform and electrification goals with investment opportunities, local and regional resources, and increased human capital. It was designed to expand the supply of and access to affordable and reliable grid-connected electricity services in West Africa, with the goal of advancing development priorities, including inclusive economic growth, security, and improved health and education outcomes. 1 The West Africa Regional Mission includes Benin, Burkina Faso, Cameroon, Cape Verde, Central African Republic, Chad, Congo, Côte d’Ivoire, the Democratic Republic of the Congo, Equatorial Guinea, Gabon, The Gambia, Ghana, Guinea, Guinea￾Bissau, Liberia, Mali, Mauritania, Niger, São Tomé & Príncipe, Senegal, Sierra Leone, and Togo. EXTERNAL PERFORMANCE EVALUATION OF WAEP | JUNE 2023 | 2 1.2ORGANIZATION OF REPORT The report is structured in the following manner: • Section 2 summarizes the evaluation methods, including the questions used to guide this evaluation; • Section 3 presents the key findings, organized into subtopics addressing relevance and coherence, efficiency and effectiveness of program delivery, outcomes and impacts to date, and, lastly, key barriers affecting the delivery of results; • Section 4 includes conclusions, lessons learned, and recommendations for the future; and • Annexes contain the statement of work, evaluation design, data collection instruments, and sources of information. 2. EVALUATION METHODS 2.1EVALUATION QUESTIONS AND SCOPE ICF was tasked with conducting a performance evaluation of the USAID-funded program that provides technical services to implement the four-year WAEP task order under a Power Africa indefinite delivery, indefinite quantity contract. The evaluation sought to (1) determine, to date, the extent to which the contract has achieved its intended objectives and outputs; (2) assess the technical and program management of the implementation mechanism; (3) highlight lessons for USAID in facilitating coordination among Implementing Partners (IPs); (4) highlight major gaps and challenges that may require adjustments in program implementation; and (5) inform the design of potential future activities. USAID provided the following illustrative evaluation questions in the statement of work (see Annex A): • To what extent have WAEP’s technical approach and interventions been successful in achieving the program’s intended outcomes for both CLINs? What additional interventions or strategies could have fully enabled the achievement of the intended outcomes? • What compelling narrative and evidence is there for WAEP (CLIN – Regional and CLIN 2 – Ghana) to indicate the following: o What worked well? Please provide examples of how it was accomplished. o What did not work well? Please concretely describe what did not work well and why. o What was learned? o What should be done differently in the future? Why? o Please describe actions taken to ensure opportunities for regular strategic activity integration of national (Ghana) and regional programming and resource leveraging of WAEP assets. • To what extent has the level of Power Africa’s available funds affected WAEP’s implementation and/or its ability to achieve results? • How did the contractor monitor and document performance changes? What adjustments (if any) were made to adapt to or mitigate changing conditions? • How has WAEP worked with interagency partners to leverage resources with other development partners to extend its influence? Similarly, has private sector engagement resulted in a significant leveraging of resources or funding, and what development results (including unintended results) has this coordination and collaboration contributed to? • How has the COVID-19 pandemic affected program implementation and the overall project performance? EXTERNAL PERFORMANCE EVALUATION OF WAEP | JUNE 2023 | 3 2.2EVALUATION METHODOLOGY EVALUATION DESIGN ICF developed an overarching evaluation design framework for all four PA programs, which included five focus areas: (1) relevance and coherence of the TOC and program design; (2) efficiency and effectiveness of program delivery; (3) outcomes, impact, and sustainability of program delivery; (4) key barriers to the delivery of results; and (5) lessons learned and recommendations. (See the evaluation framework in Annex B.) The evaluation examined the extent to which the program expanded the support of and access to affordable and reliable grid-connected electricity services across West Africa and advanced the development outcomes of inclusive growth, security, and improved health and education. It sought to determine whether WAEP’s outcomes, including strengthening the performance of national utilities and power sector entities and launching a regional power market, were on track. Similar outcomes, except for the regional market, were reviewed separately for Ghana. Finally, WAEP’s TOC, which proposes to achieve these outcomes by aligning energy reform and electrification goals with investment opportunities, local and regional resources, and increased human capital, was reviewed to assess the effectiveness of the program delivery and contractor’s monitoring and documentation process. DATA COLLECTION AND ANALYSIS The evaluation team employed a combination of data collection and analysis methods to assess WAEP in the context of the evaluation matrix. The team started with a desk review of relevant program documents, including program implementation plans, annual work plans, annual and quarterly performance reports, performance monitoring plans and systems, and technical reports. The team also reviewed external publications on the energy sector and other inventions in the similar subsector and geographic area to consider other best practices and models. The team triangulated the findings from the desk review with data collected through 84 semi-structured interviews with key informants. A full list of these interviews, which were conducted via video platforms (e.g., Google Meet, Zoom), is available in Annex D. See Annex C for the data collection instruments. 2.3EVALUATION LIMITATIONS The evaluation team is confident that it collected sufficient evidence to make strong and robust conclusions. However, certain limitations on the data exist, including the following: • Social response bias. The evaluation team is confident that, overall, most interviewees were frank and forthcoming. Nevertheless, it also is important to acknowledge that some were, at least partly, presenting material in a manner that was consistent with their own institutional interests. Such bias is normal in qualitative research and is one reason why it is critical to interview a range of individuals and triangulate the results. • Insufficient data to evaluate outcomes and impacts. It was not possible to assess the ultimate contributions to long-term resilience and sustainability because these effects will be evidenced over decades, beyond the implementation timeframes of the programs. In order to evaluate the sustainability of interventions past the life of the programs, the evaluation therefore relies on data and qualitative information about outputs and immediate and intermediate outcomes as reported, consistent with the TOC for each program. EXTERNAL PERFORMANCE EVALUATION OF WAEP | JUNE 2023 | 4 • Access to important stakeholders. Due to the timing and compressed schedule, the evaluation was limited by the inability to schedule interviews with a number of host government stakeholders due to unresponsiveness, including in Burkina Faso, the Democratic Republic of the Congo, Guinea, Liberia, and Sierra Leone, despite multiple attempts to contact them directly. 3. FINDINGS 3.1RELEVANCE AND COHERENCE This section discusses the findings on the relevance of WAEP’s efforts to expand the supply of and access to affordable and reliable grid-connected electricity services, as well as WAEP’s coherence with the programs and initiatives of other relevant United States Government (USG) agencies and development partners. WAEP has developed a good reputation among recipient government partners due to its efforts to address the need for technical and financial resources in the West African energy sector. This need is immense and unmet, as evidenced by the limited progress made by West and Central African countries in increasing access to electricity in recent years. According to World Bank (WB), only three countries are on track to achieve universal energy access by 2030. Various bilateral and multilateral agencies, including development finance institutions (DFIs), provide financial and technical resources according to their priorities and mandate to support country sector planning and program implementation and investment. Although it is a crowded space, with other active USG agencies (i.e., Millennium Challenge Corporation [MCC], U.S. International Development Finance Corporation [DFC], and U.S. Trade and Development Agency [USTDA]), WAEP has developed a profile among recipient government counterparts as a technical assistance program that provides much needed training, capacity building, and advisory services to national host government partners. It also has been identified by independent power project (IPP) developers as a valuable partner in the development of power generation projects. There is potential for WAEP to address additional needs, such as IPP development support from the early stage to financial close, by collaborating with development partners. While WAEP has thus far limited its close collaboration with DFIs and other non-USG development partners to the African Development Bank (AfDB) and the Tony Blair Institute (TBI), other agencies (i.e., WB and InfraCo Africa) have expressed interest in working collaboratively on IPPs through various transaction advisory services, including highly valuable legal advisory services. PERCEPTION OF RELEVANCE CORRELATED WITH LEVEL OF ENGAGEMENT While a majority of the beneficiary stakeholders acknowledge the value of WAEP support across a range of activities, including technical, legal, financial modeling, and business advisory services, their perceived relevance correlates with the intensity, level of engagement, and pertinence of technical advisory and capacity-building support. For example, generic and limited online training is perceived to be less relevant than training that is tailored to the context and needs of a country, is longer or more specialized, and/or is delivered in-person. Similarly, the perception of the relevance of advisory services (e.g., the development of studies and analyses) by host government partners correlates with the degree to which WAEP advisors have been embedded, the degree of consultation with relevant stakeholders, timeliness, and the level of WAEP engagement. Training and capacity-building efforts that stakeholders considered relevant include geographic information system (GIS) training and support for the design of dashboard and project management and coordination tools in EXTERNAL PERFORMANCE EVALUATION OF WAEP | JUNE 2023 | 5 ministries and utilities, which track electrification progress. In Ghana, specifically, financial modeling formed a critical input to the Energy Sector Recovery Program (ESRP) and the Gas Clearing House, leading to a reduction in energy sector financial shortfalls. Financial modeling and evaluation support were equally valued in the context of energy generation projects. Other much valued public sector types of support include support for negotiations with private energy companies and IPP developers. Insufficient consultation contributed to a negative assessment of WAEP’s coherence by host country government, other direct beneficiaries, and development partner stakeholders. While some partners view WAEP as providing valuable advisory services, especially on legal matters, there also is a perception that WAEP support tends to be supply driven and too focused on meeting targets. 2 Furthermore, it is seen as lacking in coherence, partly due to insufficient stakeholder consultation. Both development partners and beneficiary entities commented that WAEP failed to gain buy-in from the various regional actors in the West African energy sector because the program did not adequately consult with its peers or beneficiaries to better understand the environment and priority needs. This was particularly concerning to Power Africa’s development partners who have already made major investments over multiple years in infrastructure, technical assistance, and capacity-building efforts, and who did not want to see their work duplicated or attributed to WAEP. Other development partners indicated to the evaluation team that they did not know much or anything about WAEP’s involvement in the regional market. While the COVID-19 pandemic did affect the ability of the WAEP team to conduct in-person consultations, the evaluation team did not find evidence of WAEP effectively switching to online consultations. At the other end of the spectrum, the perception exists of high-value and timely support delivered over an appropriate period to yield high-impact results. In Ghana, for example, WAEP provided exceedingly valued high-level support to the Ministry of Energy and other host government entities to help design and implement the ESRP. Another example is WAEP’s support for AfDB’s Desert to Power) initiative, where its embedded advisors advanced the initiative by providing highly valued technical expertise. Other examples include various power generation projects where WAEP provided critical legal and financial advisory support to host government parties or wide-ranging technical advisory and project development support to IPP developers. The perception of WAEP support among private partners does not differ meaningfully from that of public partners, except in their interest in and emphasis on the importance of early-stage project development support of IPP developers and business advisory support of regional energy companies. PERCEPTION OF RELEVANCE AND COHERENCE AMONG DEVELOPMENT PARTNERS AND USG ENTITIES 3 WAEP built on the Power Africa Transactions and Reforms Program’s (PATRP) achievements and experience in developing a pipeline of and advancing power generation projects. Although WAEP expanded on PATRP’s scope by including transmission, grid connections, utility performance, and regional power trading goals, nearly 4 years into implementation, WAEP (and Power Africa more broadly) remains known primarily for its transaction advisory services among development 2 The evaluation team recognizes that WAEP’s targets are congressional mandates; however, this distinction did not appear to be clear to various stakeholders. 3 The evaluation team did not receive adequate feedback from private sector entities to report on their perception of WAEP’s relevance and coherence. EXTERNAL PERFORMANCE EVALUATION OF WAEP | JUNE 2023 | 6 partners. The visibility of its support for grid connections, transmission lines, utility performance, and the regional power market remains low. WAEP’s communication and coordination with development partners and its participation in donor coordination meetings appear to have been limited, despite the potential for synergy and reducing the risk of duplication of efforts. Development partners and other stakeholders in the energy sector are unsure of WAEP’s role and contributions, highlighting the need for a clearer mission and focus. They were unsure of the nature of WAEP’s expertise and technical assistance, what the benefit was from engaging with the program, and where WAEP fits in the broader development landscape. However, these same stakeholders have generally indicated that the need for support to expand power generation is vast and there is scope for more collaboration with WAEP, in particular, but not limited to, power generation projects. Coordination and collaboration with other USG entities in the delivery of its outcomes occurred mainly with USTDA, which was WAEP’s main partner in its power supply transaction advisory activities. WAEP has been able to provide a pipeline of power projects to USTDA, which, in turn, has been able to advance projects through feasibility studies. In Ghana, WAEP also has collaborated closely with the National Renewable Energy Laboratory (NREL) in support of the development of public renewable energy projects, as well as for non-wire alternatives such as distributed photovoltaics and net metering policies. In countries where WAEP had embedded advisors, USG entities also benefited from the in-depth sector knowledge provided during the consultation phase of new programs. For example, MCC spoke highly of WAEP’s embedded advisor in Guinea who helped to design an MCC program prior to the coup. LEVERAGING USG AGENCIES WAEP struggled to tap into additional USG resources after funding allocations for WAEP implementation were slow to materialize. A significant number of informants perceived that the budget cuts made after activities had already been negotiated with counterparts undermined WAEP as a credible and reliable partner. Furthermore, in terms of leveraging financial resources from USG entities, no investments were made by DFC during the implementation period. The evaluation team was unable to identify a reason for this lack of investment, which certainly affected WAEP and is a challenge that Power Africa is working to address across SSA. The implementation team also noted that USAID Mission support was important and should be stronger and more clearly defined. POLICY REVERSAL WAEP’s relevance was affected mid-implementation by a USG-wide policy to no longer support gas-to-power projects and instead focus on renewable energy. WAEP chose to no longer support gas-to-power projects, although exceptions on a case-by-case basis are possible. This policy and its implementation are unclear to host country stakeholders and the implementation teams alike. Stakeholders in a number of countries with domestic gas production and recent discoveries expressed concern about the new policy and its effect on expanding the power supply. The relevance of WAEP’s role in power generation projects risks being reduced in those countries that seek to exploit their domestic gas resources for power generation to meet growing electricity demand, including through fuel switching. EXTERNAL PERFORMANCE EVALUATION OF WAEP | JUNE 2023 | 7 3.2PROGRAM DELIVERY: EFFICIENCY AND EFFECTIVENESS WAEP had a slow start as it took time to reach a common understanding and agreement among Power Africa, the contracting officer’s representatives (CORs), and the contractor for CLIN 1 and CLIN 2, which was not helped by early changes in contractor personnel. This was particularly the case for CLIN 1 because the new chief of party sought to do a stocktaking of the pipeline and reassess targets and projects. Furthermore, while CLIN 2 had the benefit of retaining former PATRP personnel who were already familiar with the program, country, in-country networks, and government counterparts, CLIN 1 staff had to dedicate resources to overcoming the initial learning curve. The COVID￾19 pandemic also affected CLIN 1 more than CLIN 2 because travel between countries in the region was significantly affected. PROGRAM FLEXIBILITY The implementation team noted a need for greater flexibility from USAID to allow for future realignment in response to shifting government priorities and evolving market realities. This was especially the case for reallocating resources to mirror workflow shifts (e.g., after the moratorium on power purchase agreements (PPAs) in Ghana, the PPA indicator was still being included and evaluated in quarterly progress reports). Although it would have been difficult to revise indicators given their congressionally mandated status, flexibility would have allowed a shift in, or inclusion of, other indicators. The team also noted that a less detailed statement of work and work plans would have been more beneficial given the shifting political priorities and evolving institutional and market dynamics at the country and regional levels. Although the implementation team did realign activities based on quarterly feedback from USAID and changing realities, the process used by the COR to adapt the work plan or targets to address new realities and developments was considered overly lengthy and rigid. For example, it took 2 years to reduce the 500-megawatt (MW) target for CLIN 2 to 300 MW. Nevertheless, CLIN 2 executed a successful turn in program focus from transaction advisory services under Outcome 1 to energy sector reform-related financial advisory services. EFFECTIVENESS OF WAEP’S OPERATING MODEL Many stakeholders who received in-depth support found the program to be efficient and responsive in its delivery approach. This is despite the initial challenge of and resulting delay from reaching an agreement between Power Africa and the contractor on the statement of work for CLIN 1 and CLIN 2, as well as the impacts of COVID-19 (i.e., office closures and travel bans forcing interactions and consultations online during the first part and a large part of the second year of implementation). The overall positive sentiment from stakeholders reflects the comparative advantage of the program’s operating model, which can rapidly deploy experts from a deep bench of senior consultants to deliver advisory services where needed. The result is the perception of a uniquely flexible, value-driven, and responsive delivery model compared with other technical assistance programs that often undertake lengthy procurement processes. Once an agreement is reached with an IPP developer or government counterpart entity, WAEP provides analytical inputs and finds solutions to problems to help projects reach financial close. Despite the efficiency and responsiveness, some development partners questioned the effectiveness of partnering with a consulting firm for implementation. These doubts relate to both the perceived quality of consulting firm expertise and their lack of ownership, compared with in-house implementation in EXTERNAL PERFORMANCE EVALUATION OF WAEP | JUNE 2023 | 8 combination with independent experts. However, the benefit of using highly specialized consulting services, such as those of EnerNex, was recognized. The original Outcome 1 delivery model was not strong enough to meet the demand for support to IPP developers. The delivery model came under pressure as resource-intensive support for IPP developers did not result in financial close and thus WAEP’s ability to claim additional megawatts. The model was more tailored to local and less sophisticated private developers, which required extensive resources to advance power plant development because local IPP developers generally need more technical or financial advisory support than compared with international IPP developers. To ensure more rapid achievement of results, the implementation team was forced to adopt a more resource-efficient approach with a greater likelihood of success by pivoting toward supporting ministries of energy or utilities in their evaluation of proposed power projects. As these ministries are often important bottlenecks for concluding IPP projects, this pivot was considered valid. This support also involved developing tools and providing input and support for ministries’ negotiations with private developers. The program went from a ratio of 70 percent IPPs to 30 percent government counterpart support in the first 2 years, to a ratio of 25 percent IPPs to 75 percent government counterpart support thereafter. At the same time, WAEP also had to take an approach with a lighter touch to transaction advisory support as resources became more thinly spread following a 33 percent reduction in the overall total estimated cost of the program. Although the implementation team did make significant progress toward several KPIs, this budget cut created a risk of eroding claims of attribution. A significant number of stakeholders observed that the effectiveness of WAEP had been damaged by the unexpected budget cuts, resulting in cancellation of agreed-upon activities. In some cases, damage control was undertaken to the extent possible; however, in other cases, the lack of communication resulted in a loss of credibility that also risks undermining WAEP’s future effectiveness. An example of a lighter touch approach was the provision of a financial model by WAEP to Société Nationale d’Électricité du Burkina (SONABEL) in Burkina Faso. Rather than being directly involved through transaction advisory services, WAEP provided SONABEL with a tool that allowed them to evaluate solar IPP projects. WAEP’s partnership with AfDB provided another pathway to support power generation, transmission, and grid connections projects that proved to be effective toward meeting its targets and engaging with government counterparts. EFFECTIVENESS OF TRAINING AND CAPACITY-BUILDING ACTIVITIES Nearly all stakeholders commented on the limited effectiveness of the online trainings that were offered during and after the COVID-19 pandemic and which were seen as being too generic. The practice of online webinars also continued. Stakeholders expressed a need for training tailored to the context of the country, including relevant case studies and training follow-up to support the adoption and integration of new tools and practices. While it was acknowledged that project management training could be provided online in larger groups, other training, such as project finance, requires a more tailored approach in order to be effective. Both government counterpart and implementation team stakeholders perceived that skills transfer, whereby the government and utility personnel are engaged in co-creations of solutions or tools, whether conducted in-person or online, is more effective than traditional capacity building. EXTERNAL PERFORMANCE EVALUATION OF WAEP | JUNE 2023 | 9 ROLE OF USAID MISSIONS AND OTHER ENTITIES IN ENABLING WAEP ENGAGEMENT WITH GOVERNMENT STAKEHOLDERS AND EMBEDDED ADVISOR MODEL Many stakeholders (e.g., implementation team, other USAID staff, development partners) emphasized the importance of USAID Missions’ role in effectively facilitating WAEP engagement with government counterparts. To be effective and able to contribute to government plans and policies (e.g., the sector’s Master Plan or Integrated Resource Plan), or to access and collect data, the program had to develop a relationship with the ministries of energy, utilities, and regulators in the focus countries. Because the sector is quite politicized and competitive, with a range of development partners engaged in technical assistance, it was considered important to have the USAID Missions facilitate connections for WAEP and elevate its profile. Ghana is a good example of a country where pre-existing relationships from the Mission paved the way for high-level and effective engagement in the design and implementation of the ESRP. Similarly, WAEP benefited from other groups’ relationships with government counterparts and regional entities that positioned the program to effectively execute its work plan. Unlike other aspects of the program, the embedded advisor model was universally considered to be effective and necessary to have an impact in a given country. Having an embedded advisor also was more effective for coordination with other USG entities and development partners. ACCESS TO GRANT FUNDING AND LEVERAGING USG PARTNERSHIP The implementation team and beneficiary stakeholders perceived that WAEP’s ability to achieve Outcomes 2, 3, and 4 could have benefited from small grant funding opportunities. Although WAEP included a Grants Under Contract mechanism for the West Africa Power Pool, the administrative costs associated with using that mechanism were deemed to be too high and therefore it was not utilized. Furthermore, the implementation team and beneficiary stakeholders noted that, in some cases, technical assistance alone was not sufficient. Instead, having grant funding to complement the technical assistance would have been more beneficial. This was particularly true for activities that would benefit from piloting and testing recommendations on a limited scale or that require software or hardware acquisition to ensure implementation. Beneficiaries also expected that WAEP would be able to leverage small grant funding from USAID and other USG partners; however, it was unsuccessful in doing so. 3.3DELIVERY OF PROGRAM RESULTS: OUTCOMES, IMPACT, SUSTAINABILITY The evaluation team was not able to verify each result due to the non-responsiveness of a number of government counterparts and public energy companies. In contrast, the responsiveness of private developers was high. Countries where some or all results could be verified include Benin, Burkina Faso, Cameroon, Cape Verde, Chad, Côte d’Ivoire, Ghana, Mauritania, Niger, Senegal, and Togo. The evaluation team was also able to verify the results through interviews with a number of regional entities and AfDB. OVERALL PERFORMANCE Although WAEP’s results under CLIN 1 are moderately on track to meet some targets, it is underperforming across all outcomes. For example, megawatt generation under Outcome 1 has underachieved by a large margin, while connections under Outcome 2 reached just under 50 percent of the target at the end of fiscal year (FY) 2022. Other targets for which WAEP is underperforming include average cost to utility per connection reduced (0 out of 2 countries) under Outcome 2; aggregate losses EXTERNAL PERFORMANCE EVALUATION OF WAEP | JUNE 2023 | 10 (total technical and non-technical electricity losses) (1 out of 3 utilities), cost recovery improved in at least two utilities (0 out of 2), and utility performance (frequency of outages, duration of outages) (0 out of 2 utilities) under Outcome 3; and terawatt-hours (TWh) of additional cross-border power trade (0.50216 TWh out of 3.5 TWh) under Outcome 4. While WAEP supports five CLIN 1 countries and Ghana (under CLIN 2) to improve the cost recovery of utilities and reduce commercial losses, its results reflect the overall challenges that utilities in Africa face related to improving their performance, cost recovery, and reducing aggregate losses. One specific challenge is the limited number of utilities that are transparently and consistently reporting on financial and operational performance. The result obtained under CLIN 1 of one utility having reduced aggregate losses occurred because of WAEP support for a private sector partnership model for the Liberia Electricity Corporation. With regard to the reduction of cost per connection, the main issue is that utilities do not collect such cost data. The limited results for cross-border electricity trading reflects the delay in several regional interconnection projects that were affected by the COVID-19 pandemic and weather￾related events, as well as the long timeframe and efforts required to put in place the necessary mechanisms and implement reforms to enable a regional power market. CLIN 2 has performed better in terms of results, likely because it focuses on only one country; however, it also underachieved in power generation as a result of the moratorium in place since 2019 on PPAs. It overachieved in the reduction of the cost of wholesale power, kilometers of power lines constructed or rehabilitated, the reduction in the cost of connection for end-users, and cost-reflective adjudication capacity. IMPACT AND SUSTAINABILITY There is more evidence of impact for CLIN 2 than for CLIN 1. The evaluation did not collect evidence of impact for either CLIN other than through interviews with government counterparts, energy companies, and development partners, and through limited documentary evidence. With a few exceptions, power plants supported by WAEP have not yet been commissioned, while there was very limited information beyond the numbers of connections claimed. It is too early for many CLIN 1 activities to have an impact, in particular those that require operationalization to take effect. More evidence of impacts was found for CLIN 2, where WAEP technical advisory services have led to energy sector debt reduction, the adoption of a net metering policy, new institutions and practices such as the Cash Waterfall Mechanism and the Gas Clearing House, and the commissioning of a hybrid solar photovoltaic (PV) plant, which in turn led to the development of a next phase and potential replication by another utility. The continuation of benefits as a result of WAEP support is contingent upon the absorption, integration, and continued use of new practices and tools, and the adoption and implementation of new plans, laws, and policies proposed by WAEP. It is too early to assess the sustainability of WAEP’s outcomes except those that are linked to hardware such as connections added, transmission lines constructed/rehabilitated, and power plants that are constructed and supplying power. In the case of transaction advisory support to power generation projects that have reached or are progressing toward financial close, there is a reasonable likelihood of sustainability even though the time lag between this milestone and construction and commissioning is considerable. Most of the results under the outcomes related to reforms and performance are at too early of a stage to assess sustainability. In most cases, the associated activities are in the piloting or testing phase or have yet to be implemented. One notable exception can be seen in Ghana, where WAEP played a crucial role in developing and EXTERNAL PERFORMANCE EVALUATION OF WAEP | JUNE 2023 | 11 implementing both the National Gas Clearing House and the ESRP. Both initiatives benefited from the constant availability of WAEP’s embedded advisor and both were considered to be sustainable by stakeholders, at least until the power sector is reformed. The net metering policy that was adopted in Ghana is another example of WAEP support that led to an observable impact. ATTRIBUTION VERSUS CONTRIBUTION The target-driven nature of the program has resulted in a continuous search for opportunities to claim connections, kilometers of power lines, and megawatts of generation. Although this is the result of congressional mandates, the implementation team wondered whether the emphasis on targets may undermine the credibility of the program as the explanatory link between claims and technical assistance or training becomes more tenuous. While most stakeholders underscored the value of the technical assistance and training provided, attribution was questioned. WAEP measured the results in connections based on project management and the GIS training that it offered to utilities and ministries, support for the design and deployment of a dashboard tool to monitor electrification programs, support for an inventory management system, and the mobilization of funding. CLIN 1 also claimed connections based on support to regional transmission companies. Development partners who have financed major T&D projects and electricity access programs questioned the practice of claiming connections and kilometers of power lines based on technical assistance and capacity building. It would be beneficial to develop additional indicators that support the tracking of direct support provided through technical assistance. While WAEP’s technical and capacity-building support for electrification was universally valued, some beneficiaries questioned the linkage made between connections achieved and WAEP’s technical and capacity-building support for GIS-enabled project tracking and management. They were of the view that these claims devalued WAEP’s investment and technical assistance projects and could create the sense that power lines and connections could be added by supporting monitoring tools and project management capacity building. This contrasts with transaction advisory support, which played a critical role in realizing private power generation projects. All results were affected by the pandemic, which limited interaction with ministries and utilities in 2020 and 2021, slowed down all electricity infrastructure projects, reduced investment, and further worsened the financial performance of many utilities. OUTCOME 1 RESULTS The targets for generation capacity added by both CLIN 1 and CLIN 2 were considered unrealistic by a majority of the stakeholders. Only 1,914 MW of the CLIN 1 target of 8,000 MW have been achieved, and only 50 MW of the CLIN 2 target of 350 MW have been achieved. Originally set at 500 MW, the target for CLIN 2 was reduced to 350 MW following the government’s moratorium on PPAs. A total of 50 MW were realized through a hybrid solar PV public project of Bui Power with the support of NREL. Another 100-MW phase is under development and could start construction this year. The total generation capacity added in West Africa, including Nigeria, during an 11-year period between 2010 and 2021 was 15,000 MW. This corresponds to an average of 1,363 MW per year, or 5,455 MW over 4 years. The WAEP target was based on a pipeline of generation projects identified in a U.S. Trade Outlook report and assumed that WAEP would pursue projects in each West African country. The evaluation team could not assess whether WAEP had sufficient human resources to provide transaction advisory services to each project due to a lack of budget and personnel data, as well as figures showing the average time spent per transaction. A total of 14 projects that received varying levels of transaction EXTERNAL PERFORMANCE EVALUATION OF WAEP | JUNE 2023 | 12 support by CLIN 1 had reached financial close at the time of the evaluation. WAEP’s claims to these additional megawatts were made based on providing transaction advisory support, which included legal services to unlock a Banque Ouest Africaine de Développement (West African Development Bank [BOAD]) loan for an evacuation line; the provision of a financial tool to six solar PV projects in Burkina Faso; and technical advisory support for the Gambia River Basin Development Organization, which evacuated 450 MW of power from the Souapiti Hydropower Station in Guinea. Much more technical advisory support at varying levels of resource intensity was provided to power generation projects that have not yet reached financial close but could reach this stage in Year 4 or afterwards. Results also were affected by a number of external factors (i.e., COVID-19, global supply chains, deteriorating macroeconomic conditions and rising debt, higher interest rates affecting the capital cost of projects, and several coups and political developments) that prevented WAEP from working in Burkina Faso, Guinea, and Mali and impacted the achievement of megawatt results by reducing the number of countries in which WAEP could operate. Finally, a USG policy shift mid-program effectively excluded gas￾related generation, meaning that larger gas projects could not be counted. Even without such barriers, power generation projects are generally slow to develop in West Africa, with some projects taking 10 or more years to reach financial closure or the construction stage. Projects often remain in the permitting stage for years as they struggle to identify sufficient funds and meet financiers’ conditions. The number of bankable projects is limited, while building a pipeline is resource and time consuming. Overall investment in power generation in Africa fell after 2018, and growth in generation capacity decreased. Capacity growth in West Africa has slowed after peaking at 21 percent from 2011 to 2012 with the addition of large oil and diesel power plants in Mali and Burkina Faso that year, while for 2020–2021, capacity grew just 4.2 percent (Bloomberg New Economy Forum 2022). The wholesale cost of generation was reduced by 20 percent in at least three WAEP countries. Results were achieved in two countries under CLIN 1, with reductions of 44 percent in Chad and 63 percent in Liberia. In Ghana, the wholesale cost of generation was reduced by 34 percent. Wholesale generation cost reductions were achieved through a fuel switch (Ghana), importation of cheaper electricity (Liberia), or through the addition of renewable energy to reduce the operational costs of generation (Chad). CLIN 1 underperformed on the kilometers of transmission lines achieved (2,324 out of 3,800 kilometers), while CLIN 2 surpassed its target (245 out of 200 kilometers). CLIN 1 results for kilometers of transmission lines achieved come from WAEP’s technical assistance support to regional entities that are constructing regional transmission lines funded by development partners, including WB, Kreditanstalt für Wiederaufbau, AfDB, and BOAD. Most West African regional grid transmission projects have been under implementation for years and have incurred years of delay to complete. However, the transmission result was partially denied by one of the transmission companies and by development partners who financed the transmission lines. Meanwhile, the CLIN 2 results derive from WAEP’s assistance to GridCo in Ghana to upgrade and rehabilitate 245 kilometers of power lines through the provision of project management training, construction project management support, and GIS for power engineering. WAEP contributed to three Master Plans. Under CLIN 1, WAEP contributed to one Master Plan in Benin, through the preparation of a decarbonization strategy, and one in Côte d’Ivoire, through a study of loss reduction and economic evaluation of investments that could strengthen the existing master plan. Meanwhile, under CLIN 2, WAEP continued to support the Power Planning Technical Committee in the EXTERNAL PERFORMANCE EVALUATION OF WAEP | JUNE 2023 | 13 update and institutionalization of the Integrated Power System Master Plan as the official policy for procurement of new energy resources, following its successful inauguration in Year 1. OUTCOME 2 RESULTS WAEP’s ability to achieve new connections was bolstered by partnering with AfDB but was hindered by COVID-19. WAEP achieved just under 50 percent of the CLIN 1 target (1,625,167 out of 3,500,000 new connections) and 85 percent of the CLIN 2 target (255,900 out of 300,000 new connections) at the end of Year 3. The connections counted also include existing connections that were regularized. WAEP provided capacity-building support to AfDB projects in Niger and Burkina Faso, which are expected to lead to additional connections; however, COVID-19 has been a significant source of slowed progress. WAEP support for grid connections mostly focused on digitization using GIS to create informational dashboards to support ongoing electrification projects. These dashboards contributed to project efficiency, transparency, training, and management. In some cases, this support enabled the further mobilization of funding for electrification projects. By facilitating the monitoring and presentation of real-time electrification and regularization results, WAEP increased the potential to raise funds for electricity access. WAEP’s ability to achieve grid connections was dependent upon existing publicly financed projects, usually with support from other major donors and DFIs. Unlike private generation or mini-grid projects that do not require public funding, WAEP was entirely reliant on these existing programs and the willingness and interest of utilities to work with the program. Through this partnership with existing programs, WAEP succeeded in reducing the average time required to get a connection in one country out of three. Although the program was unsuccessful in meeting its target to reduce the average cost to utilities per connections, it saw greater success in reducing the cost of connections for end￾users. Most utilities in the region do not track the cost per connection, which is determined by a range of factors. Furthermore, the implementation team considered this reduction to require more resources than were available. However, reducing the cost to the end-user is a simpler endeavor and was achieved in Togo through the reduction of the upfront payment for the connection, and in Ghana through the design of a ready board and training of additional wiring professionals. OUTCOME 3 RESULTS WAEP was able to identify and design quick win solutions and provide capacity-building, training, and technical advisory services to improve performance for a number of utilities. Like other outcomes, CLIN 1 saw limited success compared with CLIN 2. In part because of budget cuts that led to activity cancellation, CLIN 1 achieved aggregate loss reduction in only one utility; the evaluation team was not able to identify which utility this was. Additionally, as with other WAEP focus areas, COVID￾19 affected the already precarious financial performance of most utilities in West Africa by raising operational costs. CLIN 2 performed better because of continued support for the ESRP, resulting in the reduction of the projected energy debt of $3.2 billion. WAEP supported advocacy for cost-reflective tariffs and claimed the impact of the subsequent rise in tariffs. WAEP also contributed to commercial loss reduction in Ghana’s electricity utility; however, the supporting data are not yet available. EXTERNAL PERFORMANCE EVALUATION OF WAEP | JUNE 2023 | 14 Outcome 3 includes some of WAEP’s most challenging targets, such as a reduction in the aggregate losses of utilities; improvement of cost recovery and utility performance (which requires investment in maintenance and capital expenditures); and good governance, including monitoring and reporting. The performance of utilities also is influenced by laws and regulations pertaining to the utilities. Furthermore, the implementation team mentioned inconsistencies in the definition and understanding of targets by Power Africa and some host country governments. OUTCOME 4 RESULTS The evaluation team found mixed evidence of WAEP’s contribution to the regional power trade. According to the FY 2022 Annual Report, WAEP achieved only 0.50216 out of 3.5 TWh of additional cross-border power trade. This underperformance stems from a variety of factors that are outside of WAEP’s control, including the impact of the COVID-19 pandemic and climate events on the construction of transmission lines and substations and the signing of PPAs. Stakeholders also perceived a lack of coherence and coordination by WAEP in relation to the regional power trade stemming from a lack of consultation with regional and national counterparts. WAEP provided varying levels of technical advisory services and capacity building to regional entities and companies engaged in fostering regional electricity trade and operationalizing a regional electricity market, including through collaboration with TBI and a highly specialized subcontractor. WAEP claims that it fostered three regional power utility trade relationships via technical assistance to regional companies, which is one more relationship than the target of two. However, while the technical assistance was considered valuable, the attribution of one of these relationships was questioned because the regional entity was not able to implement WAEP’s recommendations due to a lack of funds. CROSS-CUTTING ACTIVITIES AND RESULTS WAEP support for policy reform is considered relevant and effective in contributing to creating an enabling environment and removing barriers in the energy sector. WAEP defines policy reforms as those laws, policies, regulations, or standards which enhance energy sector governance and which are formally proposed, adopted, or implemented. Through the generation of energy transition roadmaps, implementation plans, and regulations, CLIN 1 has supported 24 out of 35 target policy reforms (68 percent) and CLIN 2 has supported 14 out of 22 policy reforms (64 percent). In some cases, such as in Ghana through the ESRP, WAEP made critical contributions at the highest level. Stakeholders universally value trainings and support related to gender inclusivity across a variety of aspects in the sustainable energy sector. Under CLIN 2, WAEP funded training for wiring artisans which led to an 80 percent increase in female wiring professionals (private and public) in northern Ghana. Additionally, through CLIN 1 training, WAEP conducted basic gender inclusivity trainings, as well as the Fostering Women’s Leadership in the Energy Sector trainings. The latter enabled eight women to transition into leadership roles in the sector, for which WAEP had a target of 20. Beneficiary feedback on WAEP’s training and capacity building, which is a significant component of WAEP’s activities, ranged from highly valuable and impactful to useful, but too general and not sufficiently contextualized. There was high demand for training especially among regulators and utilities. Under CLIN 1, WAEP trained 1,004 out of a target of 1,180 people by the end of FY2022. CLIN 2 far exceeded its target of 382 people trained, with 1,016 beneficiaries. EXTERNAL PERFORMANCE EVALUATION OF WAEP | JUNE 2023 | 15 WAEP mobilized $5.514 billion for energy projects under CLIN 1 and $73 million under CLIN 2. These amounts are not itemized, so the evaluation team could not determine how these results have been calculated. Based on specific examples in performance reports, mobilized finance includes the total cost of power generation projects to which WAEP provided direct advisory support and indirect support through technical assistance such as grid code alignment. 3.4 KEY BARRIERS TO DELIVERY OF RESULTS The COVID-19 pandemic slowed down the development of energy infrastructure and, in some cases, reversed progress made toward universal energy access. The COVID-19 pandemic had a significant impact on investment and project development and implementation in the electricity sector. The pandemic also delayed necessary tariff increases to allow cost recovery. The loss of revenues from commercial and industrial consumers during the pandemic, increases in non-payment, and emergency relief measures to protect end-users further exacerbated the debt held by utilities and their inability to cover operating costs, with cascading impacts on maintenance and investment in T&D and on the cost of PPAs. This impact on the development of energy infrastructure hindered WAEP’s ability to claim connections and therefore meet its targets. Deteriorating macroeconomic conditions affected WAEP’s ability to meet its targets. The pandemic has increased the debt burden of countries and worsened their fiscal revenue position, thereby reducing opportunities for public spending on energy access expansion, including through grid extension. Inflationary pressures exacerbated by the war in Ukraine and the higher cost of capital following rises in interest rates make infrastructure more costly, putting further pressure on indebted utilities and tight government budgets. WAEP operates in a weak enabling environment and under conditions of insufficient investment. According to the International Energy Agency’s Africa Energy Outlook 2022 report, investment in the SSA power sector and in end-use applications has been flat since 2014. Investment in T&D networks had already slowed by 2015 and has received little investment since the start of the COVID-19 pandemic. Broadly, demand for energy infrastructure finance is unmet, and has not increased in line with demand and needs since 2014. Regulatory, institutional, and legal frameworks, laws, and policies governing the power sector in West Africa are weak, which increases the risks associated with raising debt and equity for private power generation projects and slows down project development. Moreover, some governments are increasingly reluctant to issue sovereign guarantees, accept “take or pay” contracts, or allow PPAs in foreign currency, which adds new challenges for private investment in generation capacity. There is limited human and institutional capacity in the ministries of energy and utilities to evaluate and negotiate PPAs and to undertake sector planning, coordination, and monitoring of the sector. Similarly, regional entities are understaffed and lack advanced technical skills. Ministries, public utilities, and regional entities in the sector lack adequate resources and cannot act on recommendations that require software and equipment for implementation. While generation has been opened to privatization in most African countries, resulting in a growing number of IPP projects, mini-grid concessions, and the development of an off-grid market, privatization of T&D utilities has generally been resisted in West Africa and beyond. This political economic challenge prevents investment, contributes to delayed evacuation of power plants, and constrains power generation expansion. Additionally, coups led to the suspension of Power Africa EXTERNAL PERFORMANCE EVALUATION OF WAEP | JUNE 2023 | 16 activities in Guinea and Burkina Faso, and prevented WAEP from working in Mali, thereby affecting the results. 4. CONCLUSIONS AND RECOMMENDATIONS 4.1CONCLUSIONS/LESSONS LEARNED WAEP would benefit from a TOC that enhances the coherence of the program by articulating its own position and comparative advantages in the ecosystem of USG agencies and other relevant development partners in the sector. An improved TOC also would pay greater attention to the feasibility of proposed actions in terms of resources and conditions for their effectiveness. The program also would benefit from a greater examination of assumptions made to avoid unrealistic targets and timelines. WAEP’s first 3 years of implementation were overshadowed by the COVID-19 pandemic, directly impacting investment, slowing electrification efforts, and constraining program implementation as a result of lockdowns. The effects of the pandemic continue to be felt and were made worse by the war in Ukraine, with growing debt and inflation pressures, and a higher cost of capital due to higher interest rates. It is not possible to disentangle the effects of the pandemic from ongoing trends in the sector, such as stagnant growth in T&D and a decline in the growth of new generation capacity. These factors need to be considered in the evaluation of the performance of WAEP. The need for technical and financial resources in the West African power sector are immense and unmet. The evaluation team found that there is demand for technical partners such as WAEP who can provide targeted and advanced technical expertise, a range of advisory services, and capacity building, including a wide variety of training. A significant number of beneficiary stakeholders valued WAEP’s support and wanted more of it, with feedback on relevance broadly correlating with responsiveness and timeliness, level of engagement and skills transfer, and added value and impact. Although the operating model was highly valued by IPP developers, the need to deliver results led to an approach with a lighter touch focused on advising government counterparts in resolving bottlenecks in the evaluation and negotiation of more advanced private power generation projects by ministries of energy or utilities. Stakeholders, in particular the private sector but also government counterparts, placed a high value on WAEP’s unique flexibility, responsiveness, deep engagement, and expertise in moving power generation projects forward and finding solutions. However, the resource-intensive early-stage project development support model came under pressure from Power Africa management because the results were too slow to lead to a financial close within one to two years. While deeply attractive to government counterparts and private developers alike, the operating model of having a deep bench of expert consultants who can be deployed for flexible assignments to address client technical, financial, and legal advisory needs can be resource intensive while not delivering immediate results. However, a range of stakeholders were of the view that there is high value in supporting early-stage projects to create a pipeline of high-quality power projects that can reach financial close in the future. While support for government counterparts to advance power generation projects is impactful, stakeholders believe that there is a critical need to support early-stage projects, in particular local IPP developers. A significant number of stakeholders expressed the desire for a more consultative approach with greater stakeholder engagement throughout the program to ensure that support is tailored to EXTERNAL PERFORMANCE EVALUATION OF WAEP | JUNE 2023 | 17 the country and best suited to meet technical and financial needs, rather than to simply meet WAEP targets. Stakeholders also pointed to limited absorption capacity and identified further efforts and conditions that were needed for training and capacity building to be effective. The evaluation also revealed a funding gap in the form of grants that would have enabled the piloting or implementation of WAEP’s recommendations and advice, such as the piloting of productive use of energy business models in Ghana, the testing and piloting of the ready board solution, or the acquisition of necessary software and hardware. Development partners and other stakeholders often do not understand what support WAEP provides, even when they have worked alongside the program. The lack of understanding appears to partially derive from a lack of clarity on the nature of WAEP’s support and its exact role in the sector and in specific projects, as well as from a lack of stakeholder consultation. These stakeholders are familiar with DFC and MCC; however, they generally do not know what WAEP does beyond providing transaction advisory services for power generation projects. WAEP also seems to have struggled to articulate and communicate a coherent role for itself in fostering regional electricity trade. Overall, WAEP’s visibility and profile among development partners is low and there is a need to clarify its mission and role in the broader energy sector through additional meetings with partners. WAEP underachieved in power generation, improving utility performance, and increasing regional electricity trade; however, it performed moderately on transmission lines and connections. WAEP performed better in Ghana under CLIN 2, where the program benefited from existing USAID networks and support, as well as embedded advisors, some of whom were former PATRP transaction advisors. It was able to provide high-impact financial advisory services at the highest level and to directly engage in helping to create new institutions to reform how financial flows in the energy sector are managed, among other achievements. Power Africa may have underestimated the weak enabling environment and the lack of institutional capacities that are hindering IPP projects to advance or utilities to perform better, while overestimating its ability to overcome these barriers. Nevertheless, WAEP was able to identify and design solutions to address at least some of the performance problems of some utilities. Attribution claims made related to kilometers of transmission lines and the number of connections were questioned by some stakeholders and development partners because as grid connections and T&D are entirely publicly funded by governments, DFIs and bilateral aid agencies also provide technical assistance. WAEP’s ability to claim connections is entirely reliant on these existing T&D and electrification programs. While stakeholders generally recognized WAEP’s contribution in terms of supporting government counterparts and energy companies in expediting and project managing T&D and electrification programs, attribution was considered tenuous or unsubstantiated. Stakeholders perceived that the program would benefit from documenting intermediary contributions, rather than solely focusing on high-level outcomes, to enhance credibility and demonstrate results. This could be done by creating intermediate metrics that indicate progress toward congressional targets to reflect the actual timeframes to bring projects to a financial close and into operation in West Africa. USAID Missions, with their existing government relations and ability to raise the program profile, were seen as instrumental in enabling WAEP to engage government counterparts and positioning the implementation team to provide high-level support and claim results. Similarly, having in-country embedded advisors was seen as critical to the delivery of results. Another avenue for engaging government counterparts was through the AfDB partnership, where WAEP had embedded advisors and provides critical technical and financial advisory support. Collaboration with TBI EXTERNAL PERFORMANCE EVALUATION OF WAEP | JUNE 2023 | 18 also allowed WAEP to provide valued support to regional entities and government counterparts. TBI has seemingly played a key role in enabling and coordinating WAEP’s support for some regional entities and government counterparts, including through the provision of experts. The expectation that WAEP would be able to leverage the USG ecosystem to access funding or investment was not entirely fulfilled except for instances of collaboration with USTDA, which funded feasibility studies for early-stage power generation projects supported by WAEP. WAEP and USTDA had a natural complementarity, with WAEP bringing a pipeline of eligible early-stage power generation projects in need of feasibility studies for USTDA to fund. There also was an expectation that WAEP would enable DFC investment in IPP projects because the two entities share pipelines; however, DFC did not invest in any WAEP projects during the implementation period. DFC was not interviewed for this evaluation so it is unclear what prevented DFC from investing. 4.2RECOMMENDATIONS RECOMMENDATION 1: Clarify and focus WAEP’s mission. WAEP would benefit from a clearer, more focused mission that can be communicated to stakeholders and be understood by development partners. A more focused mission also could inform its operating model by determining the range and quantity of technical expertise required and how that expertise could be efficiently deployed. • ACTION 1: Consider and discuss how WAEP could develop a clearer and more focused mission incorporating depth, range, and quantity of expertise. RECOMMENDATION 2: Consider greater adaptative management. WAEP program implementation would benefit from greater adaptive management of the program to accommodate shifting political priorities and evolving market and institutional dynamics. • ACTION 1: Adopt a less detailed statement of work and put in place a less onerous process for work plan revisions. • ACTION 2: Ensure that there are adequate staffing resources for adaptive management included in the work plan, budget, and Monitoring, Evaluation, and Learning plan for the next program. RECOMMENDATION 3: Adopt more realistic targets and outcomes. WAEP should base its targets on analyses of recent trends and data on generation capacity, transmission, and connections, while considering its limited resources. Similarly, outcomes such as the reduction of aggregate losses are not realistic because technical losses are particularly hard to reduce without major investments and efforts beyond what is available to Power Africa. Furthermore, other key performance indicators (KPIs), such as the reduction of cost per connection to the utility, also are not realistic because such data are not collected by the utilities. • ACTION 1: Revise the methodology to identify more appropriate indicator targets. • ACTION 2: Revise KPIs under Outcome 3 to reflect what is achievable based on the available data and resources. RECOMMENDATION 4: Adopt intermediate performance metrics. While reaching financial close for power generation projects is a reasonable milestone and metric for added power generation capacity, WAEP should add a secondary metric that relates to building a pipeline of power generation projects that are expected to reach financial close in the future. This metric would reflect both the foundational work that is required for IPPs, in particular, support for local developers, and the EXTERNAL PERFORMANCE EVALUATION OF WAEP | JUNE 2023 | 19 intergenerational aspect of supporting a pipeline of IPP projects. This would enhance WAEP’s impact and sustainability and fill a critical gap in support. Such intermediate metrics would require information on the progress of project development, detailed explanations of the remaining obstacles to financial close, and recommended approaches for how those obstacles would be resolved. Similarly, intermediate metrics should be considered for transmission and connections outcomes, which are all reliant on existing publicly financed grid expansion and electrification projects and programs co￾financed by DFIs and bilateral agencies. A performance-related metric could be adopted that reflects more effective project management and the resulting impacts and benefits. A higher level KPI could be applied to cases in which WAEP provides transaction advisory services to an IPP equivalent for transmission (i.e., an independent transmission project or a concession). An intermediate metric for fostering regional electricity trade, which reflects the many conditions that must be met before regional electricity trade can take place, also would be appropriate because all transmission lines and substations are being publicly funded. • ACTION 1: Explore the adoption of intermediary metrics informed by the experiences and lessons learned from the implementation of WAEP. RECOMMENDATION 5: Facilitate access to grant funding. The lack of access to grants for piloting solutions and implementing technical advice, including the acquisition of software and equipment, has been a handicap for the program. Grants would also benefit early-stage IPP projects. Access to grants would make WAEP more effective and enhance the program’s impact and sustainability. • ACTION 1: Identify what conditions are necessary for grants to be effective, then provide grants under those conditions. RECOMMENDATION 6: Given the success of the embedded advisor model, expand it where appropriate. Having an embedded advisor was considered critical for making an impact across WAEP countries. • ACTION 1: Identify where and how the embedded advisor model could be expanded. RECOMMENDATION 7: Enhance coherence and coordination. More emphasis should be placed on stakeholder consultation and engagement throughout the different stages of a workstream—from the conceptual design stage of technical assistance and capacity building to the adoption of technical advice. This would include holding stakeholder consultations that involve development partners to ensure better coherence and coordination. • ACTION 1: Invest in stakeholder consultations at various appropriate levels. RECOMMENDATION 8: Explore replicable technical assistance and conditions for replicability. Successful technical assistance and capacity-building efforts should be assessed and socialized to better understand how they could be applied to the WAEP context. • ACTION 1: Analyze lessons learned to identify replicable or scalable practices, the resources they would require, and the conditions for replicating and scaling them. RECOMMENDATION 9: Move from capacity building to skills transfer. Skills transfer can be achieved through co-construction and more tailored training, which is more easily adopted and absorbed than capacity building and would enhance sustainability. This approach would be more resource intensive, which would have to be considered when developing budgets and metrics. EXTERNAL PERFORMANCE EVALUATION OF WAEP | JUNE 2023 | 20 • ACTION 1: Replace capacity building with skills transfer where appropriate and effective. Skills transfer is more oriented toward co-development and co-creation, as well as support for the application and integration of new tools or methods in the organization. RECOMMENDATION 10: Formulate a coherent strategy to support the regional electricity market. Feedback from stakeholders points to the lack of visibility, consultation, and coherence in WAEP’s support for the regional electricity market. • ACTION 1: Develop a strategy that articulates WAEP’s position and potential contribution in the crowded ecosystem of DFIs and other agencies that support the regional electricity market and puts more emphasis on stakeholder consultation and engagement. EXTERNAL PERFORMANCE EVALUATION OF WAEP | JUNE 2023 | 21 ANNEXES ANNEX A: EVALUATION STATEMENT OF WORK WEST AFRICA ENERGY PROGRAM (WAEP) Contractor must conduct a Performance Evaluation of the USAID-funded program that provides technical services to implement the four-year WAEP task order, under a Power Africa IDIQ Contract. I. BACKGROUND OF THE PROJECT CLIN = Contract Line Item Number II. DESCRIPTION OF THE WAEP PROGRAM USAID awarded the West Africa Energy Program to expand the supply of and access to affordable and reliable grid-connected electricity services in West Africa, with the ultimate goal of advancing development priorities, including inclusive economic growth, security, and improved health and education outcomes. Progress toward this ultimate goal is outlined in the Power Africa Roadmap, which functions as the master plan that describes how Power Africa, and its partners will fulfill its objectives. The purpose of this activity is to provide a range of technical assistance, capacity building, and transaction support to advance Power Africa’s objectives in the West Africa region. As a result of its unique country context, Ghana is addressed separately under this task order, with its own minimum expected results and illustrative activities. This task order does not include Nigeria. WAEP generally Implementing Organization Deloitte Consulting LLP Total Estimated Cost WAEP Total = $73,850,69 West Africa Regional CLIN1 = $54,581,750 Ghana CLIN2 = $19,268,943 Period of Performance July 15, 2019 to July 14, 2023 Place of Performance for PA Task Order CLIN 1 - West Africa Regional Mission including Benin, Burkina Faso, Cameroon, Cape Verde, Central African Republic, Chad, Congo, Côte d’Ivoire, the Democratic Republic of the Congo (“DRC”), Equatorial Guinea, Gabon, the Gambia, Ghana, Guinea, Guinea-Bissau, Liberia, Mali, Mauritania, Niger, Sao Tome & Principe, Senegal, Sierra Leone, and Togo. CLIN 2 - Ghana EXTERNAL PERFORMANCE EVALUATION OF WAEP | JUNE 2023 | 22 does not cover support for off-grid solutions or non-grid connected microgrid systems, except when otherwise directed by the TOC and in close collaboration with the implementer of Power Africa’s Beyond-the-Grid Task Order and the Power Africa Coordinator’s Office. WAEP (CLIN 1) is meant to increase electricity availability and access in West Africa while objectively quantifying and measuring progress towards four key outcomes: ▪ Outcome 1: Supply of power increased ▪ Outcome 2: Access to reliable and affordable grid-based power increased ▪ Outcome 3: Performance of national utilities and power sector entities improved ▪ Outcome 4: Launch of a regional power market accelerated As a result of activities and interventions targeting the above outcomes, the Program aims to meet the following goals: ▪ 8,000 MW of new generation brought to financial close ▪ 3,800 km of transmission line commissioned ▪ 3.5 million new connections to the grid ▪ Cost of generation reduced by 20% in three countries ▪ An improved enabling environment for energy sector activities WAEP (CLIN 2) is meant to increase electricity availability and access in Ghana while objectively quantifying and measuring progress towards three key outcomes: ▪ Outcome 1: Optimized supply of power ▪ Outcome 2: Access to reliable and affordable grid-based power increased ▪ Outcome 3: Performance of national utilities and power sector entities improved As a result of activities and interventions targeting the above outcomes, the Program aims to meet the following goals: ▪ Generation capacity (500 MW) ▪ Transmission commissioned (200 km) ▪ New connections (100,000) and Regularized connections (200,000) ▪ Wholesale cost of generation (reduced 20%) ▪ Improved enabling environment (Various) III. THEORY OF CHANGE WAEP will achieve the above outcomes by strategically aligning energy reform and electrification goals with investment opportunities, local and regional resources, and human capital. This includes employing a results-oriented framework for decision-making, which drives the identification, prioritization, and selection of intervention activities; and programming with the aim to unlock private sector investment and to move transactions forward to increase generation and access to electricity. WAEP cultivates enduring partnerships and alliances with local, regional, and national institutions, and will proactively engage with relevant communities, including through strategic outreach initiatives that focus on women’s empowerment and environmental sustainability. The result of these approaches will EXTERNAL PERFORMANCE EVALUATION OF WAEP | JUNE 2023 | 23 allow the Program the flexibility to implement planned or approved activities or explore opportunities and solutions as they arise as well as to establish enduring foundational capacity and buy-in with local communities and responsible agencies. WAEP implements targeted interventions to accomplish the below objectives: 1. Increase the supply of power in West Africa; 2. Enable expanded end-user connection to the grid; 3. Improve the technical and institutional capacity and performance of select national utilities and other relevant national power sector entities; and, 4. Accelerate the establishment of a high functioning regional power market. WAEP coordinates the delivery of services aimed at achieving these objectives with existing and anticipated work conducted by stakeholders, including the Power Africa Coordinator’s Office, other USAID programs, relevant USG entities and implementers, regional and national host country partners, the private sector, development partners, and civil society. WAEP is tasked to also mitigate critical barriers to attracting and sustaining necessary investment in electrical power generation, transmission, and distribution. Mitigating these barriers will facilitate expanded supply of and access to affordable and reliable grid-based electricity services in West Africa. As a result, it will improve broader development outcomes in other sectors, including economic growth, agriculture, health, education, and security. IV. EVALUATION QUESTIONS FOR WAEP Evaluation questions will be completed in collaboration with the Power Africa post-award process. It must be noted that the evaluation requires separate analyses and reported results for Contract Line Item Numbers (CLINs) 1 (Regional Programs) and 2 (Ghana). The following evaluation questions are illustrative: Q1. To what extent have WAEP’s technical approach and interventions been successful in achieving the program’s intended outcomes for both CLINs? What additional interventions or strategies could have fully enabled the achievement of the intended outcomes? Q2. What compelling narrative and evidence is there for WAEP (CLIN -Regional & CLIN 2 - Ghana) to indicate: • What worked well? Please provide examples of how it was accomplished? • What did not work well? Please concretely describe what did not work well and why? • What was learned? • What should be done differently in the future? Why? • Please describe actions taken to ensure opportunities for regular strategic activity integration of national (Ghana) and regional programming and resource leveraging of WAEP assets. Q3. To what extent has the level of Power Africa’ available funds affected WAEP’s implementation and/or its ability to achieve results? Q4. How did the contractor monitor and document performance changes? What adjustments (if any) were made to adapt to or mitigate changing conditions? EXTERNAL PERFORMANCE EVALUATION OF WAEP | JUNE 2023 | 24 Q5. How has WAEP worked with interagency partners to leverage resources with other development partners to extend its influence? Similarly, has private sector engagement resulted in a significant leverage of resources or funding, and what development results (including unintended results) has this coordination and collaboration contributed to? Q6. How has the COVID-19 pandemic affected program implementation and the overall project performance? EXTERNAL PERFORMANCE EVALUATION OF WAEP | JUNE 2023 | 25 ANNEX B: EVALUATION DESIGN AND MATRIX Updated to reflect final team organization and other minor corrections. I. INTRODUCTION The U.S Agency for International Develop (USAID) Power Africa Program Office, located within the US Mission to South Africa, has contracted ICF to conduct external performance evaluations of four USAID-funded programs to improve program performance management and efficiency. These evaluations will help Power Africa meet Agency requirements, make improved and more informed strategic and management decisions around program implementation, and advance Agency learning. The evaluations will: 1) determine, to date, the extent to which the contract has achieved its intended objectives and outputs; 2) assess the technical and program management of the implementation mechanism; 3) highlight lessons for USAID in facilitating coordination with implementing partners; 4) highlight major gaps and challenges that may require adjustments in program implementation; and 5) inform the design of potential future activities. The lessons drawn from these evaluations will be important to the technical teams, activity managers and senior management. 1.1ABOUT USAID POWER AFRICA USAID Power Africa seeks to increase access to electricity throughout the region with the addition of millions of new connections and tens of thousands of megawatts in new and cleaner power generation. To date, USAID Power Africa has delivered significant transformative impacts in the Sub-Saharan Africa region, not only in expanding energy services but also in benefitting multiple additional development priorities related to promoting economic growth, improving access to health and education, increasing productivity and sustainability in agriculture and livelihoods, and recognizing the needs of local communities. To evaluate effectiveness, efficiency, and relevance, and to identify best practices and lessons learned, USAID Power Africa has commissioned independent evaluations of the following four programs: ● The West Africa Energy Program (WEAP); ● The Nigeria Power Sector Program (NPSP); ● The East Africa Energy Program (EAEP); and ● The Power Africa Off-Grid Program (PAOP). Each evaluation will be specifically tailored to address the program’s theory of change and expected outcomes, while recognizing USAID Power Africa’s overall mandate to increase access to reliable, affordable, and sustainable power through its three strategic pillars: (1) New Power Generation Capacity; (2) Increased Electrical Connections; and (3) Improved Enabling Environment. 1.2USAID POWER AFRICA PROGRAMS TO BE EVALUATED Each USAID Power Africa program brings together a unique set of interventions, technical partners, and country stakeholders that address the region’s most pressing energy needs and priorities for energy equity and self-reliance. EXTERNAL PERFORMANCE EVALUATION OF WAEP | JUNE 2023 | 26 The following is a brief description of each program and its respective theory of change (TOC) and outcomes as well as areas of inquiry which provide a framework for the evaluation design. Additional program details and geographic scope are provided in Exhibit 1. Exhibit 1. ICF’s expertise extends to all program geographies. West Africa Energy Program ● IP: Deloitte Consulting LLP ● POP: July 2019-July 2023 ● TEC: $73,850,693 CLIN 1 $54,581,750 CLIN 2 (Ghana) $19,268,943 ● Countries: Benin, Burkina Faso, Cameroon, Cape Verde, Central African Republic, Chad, Congo, Cote d’Ivoire, DRC, Equatorial Guinea, Gabon, The Gambia, Ghana, Guinea-Bissau, Liberia, Mali, Mauritania, Niger, São Tomé and Príncipe, Senegal, Sierra Leone, and Togo Nigeria Power Sector Program ● IP: Deloitte Consulting LLP ● POP: April 2018-March 2023; Extended to September 30, 2024 ● TEC: $109,258,794 ● Country: Nigeria East Africa Energy Program ● IP: RTI International ● POP: December 2018-December 2022; extended to March 2023 ● TEC: $73,850,693 ● Countries: Burundi, DRC, Djibouti, Eretria, Ethiopia, Kenya, Rwanda, Somalia (Punland and Somalialand), Tanzania, and Uganda Power Africa Off-Grid Program ● IP: RTI International POP: November 2018-August 2023 ● TEC: $49,992,361 ● Countries: Benin, Burkina Faso, Cameroon, Cape Verde, Central African Republic, Chad, Cote d’Ivoire, DRC, Ethiopia, The Gambia, Ghana, Guinea-Bissau, Kenya, Lesotho, Liberia, Mali, Madagascar, Mauritania, Mozambique, Niger, Rwanda, Senegal, Sierra Leone, Somalia, Tanzania, Togo, Uganda, Zambia IP = Implementing Partner; POP = Period of Performance; TEC = Total Estimated Cost EXTERNAL PERFORMANCE EVALUATION OF WAEP | JUNE 2023 | 27 The East Africa Energy Program (EAEP) is designed to optimize the power supply, increase grid￾based power connections, strengthen utilities and other power sector entities, and increase regional power trade. EAEP’s TOC assumes that by expanding affordable and reliable electricity services, EAEP will support development priorities including inclusive growth, increase security, and improve health and education outcomes through community engagement. The Nigeria Power Sector Program (NPSP) implemented in coordination with USAID/Nigeria is designed to address specific constraints in the energy sector across four outcomes: private sector investment; development of the off-grid market; improved enabling environment for private sector participation; and improved liquidity of the distribution sector. The program aims to accelerate energy transactions that lead to increased power generation and increased connections to end-users by working with government partners to create the policy, legal, and regulatory frameworks necessary for private sector investment and financial sustainability within the power sector. Power Africa Off-Grid Program (PAOP), or Beyond-the-Grid, supports USAID Power Africa’s goal of doubling electricity access across Sub-Saharan Africa by 2030 by increasing the number of households and businesses connected to electricity via on-grid or off-grid (household solar and micro￾grids) solutions. PAOP’s TOC assumes that improving markets and regulatory mechanisms and access to private financing will yield a subsequent increase in the total number of off-grid connections and better development outcomes. Program activities focus on increasing the number of off-grid connections with targets of 25 million to 30 million new connections by 2030, increasing private sector financing, and improving the enabling regulatory environment for investment. The West Africa Energy Program (WAEP) is designed to expand the support of and access to affordable and reliable grid-connected electricity services across West Africa and advance development outcomes of inclusive growth, security, and improved health and education. WAEP’s outcomes also include strengthening the performance of national utilities and power sector entities and launching a regional power market. Similar outcomes, except for the regional market, are also elaborated separately for Ghana. WAEP’s TOC proposes to achieve these outcomes by aligning energy reform and electrification goals with investment opportunities, local and regional resources, and increased human capital. 1.3EVALUATION OBJECTIVES AND SCOPE This section of the work plan lays out the scope and objectives of the four separate evaluations. Specific evaluation objectives and scope are described below for each of the four program evaluations. For each evaluation, it is anticipated that a 25-page final report will be generated and will focus on summary findings and recommendations with annexes as appropriate. East Africa Energy Program (EAEP): A key focus area for the evaluation is determining how the program is addressing ongoing challenges, given national priorities, gaps in human and institutional capacity, and fragmented systems required to reach energy equity. The evaluation will examine how the program’s platform addresses the challenge of regional integration of technical and human systems by examining four program dimensions: workforce development and institutional strengthening; improvement in national and regional planning and policy; coordination and information sharing through information systems; and the facilitation of public-private dialogue. ICF will explore whether and to what extent EAEP has demonstrated results of greater reliability, lower prices, and increased access and if EXTERNAL PERFORMANCE EVALUATION OF WAEP | JUNE 2023 | 28 these outcomes are well correlated to “meaningfully integrated national systems” to achieve results. The evaluation will explore cross-cutting issues to determine effectiveness and identify models and areas of improvement across countries. To identify program efficiencies and lessons, ICF will review the program’s ‘One-Team’ management approach for lessons that can be shared with RTI International and partners. Nigeria Power Sector Program (NPSP): Evaluation focus areas will address how NPSP’s activities have responded to the lack of electrical access as a critical constraint on Nigeria’s economic development and an underlying cause of poverty. This focus includes determining the extent to which program activities have evolved to address longer-term sustainability as well as how effective the IP’s management approach was in achieving Nigeria’s rural electrification goals in a manner that addresses energy equity. We will determine the extent to which power and investment targets aided the program to address unintended systemic issues necessary for self-reliance and financial stability. The evaluation will also examine NPSP’s performance in response to disruptions to markets and enabling environments. Power Africa Off-Grid Program (PAOP), or Beyond-the-Grid: The evaluation design will assess the PAOP TOC for relevance and gather contextual information from private companies, governments, and investors from each region to determine whether PAOP is on track to achieve its targets as well as the feasibility of achieving those targets. We will analyze the program’s effectiveness in increasing the number of off-grid companies active in Sub-Saharan Africa and determine whether PAOP has been successful in driving down sector costs, making energy more accessible. We propose to give particular emphasis to assessing program delivery and identifying best practices and models that can be shared across the region. West Africa Energy Program (WAEP): The evaluation will examine the extent to which the program expanded the support of and access to affordable and reliable grid-connected electricity services across West Africa and advanced development outcomes of inclusive growth, security, and improved health and education. We will determine if WAEP’s outcomes also include strengthening the performance of national utilities and power sector entities and launching a regional power market were on track. Similar outcomes, except for the regional market, are also elaborated separately for Ghana, will be reviewed. Finally, WAEP’s TOC which proposes to achieve these outcomes by aligning energy reform and electrification goals with investment opportunities, local and regional resources, and increased human capital will be reviewed as well as assessing the effectiveness of the program delivery and contractor’s monitoring and documentation process. 1.4EVALUATION TEAM MEMBERS AND RESPONSIBILITIES Each individual evaluation team has a combination of members that have technical and evaluation experience. In addition to the individual evaluation teams, there is the ICF Backstopping Team that is composed of individuals external to the evaluation teams. Evaluation Team Structure All four Evaluation Teams will be structured in the same way, with the caveat that the PAOP Team will have an additional evaluation specialist to account for the change in the PAOP Team Lead position. One independent consultant will serve as the Team Lead for each evaluation. These Leads are remote staff EXTERNAL PERFORMANCE EVALUATION OF WAEP | JUNE 2023 | 29 who are not based in any of the program countries. They will lead data collection, analysis, and triangulation, as well as drafting the final evaluation report for their respective team. Each Team Lead will be supported by at least one evaluation specialist and one energy expert. These support individuals will compile resources for the desk study, identify key stakeholders, and contribute to data collection, analysis, and triangulation. They will also assist in the preparation of the final evaluation report for their respective team. Backstopping Team Structure As shown in Exhibit 2 below, the evaluation teams will receive evaluation, technical, and administrative support, and oversight from the Backstopping Team. The Backstopping Team will facilitate meeting planning, scheduling, and other logistical support related to the Power Africa Monitoring, Evaluation, and Learning (MEL) Team. This may include support related to data collection, such as scheduling key informant interviews, as needed to support in-country specialists. The Backstopping Team will also provide evaluation support, such as reviewing data collection instruments, and technical support, such as verifying sectoral knowledge, to the Evaluation Teams, as needed. Exhibit 2: Evaluation Team Organization EXTERNAL PERFORMANCE EVALUATION OF WAEP | JUNE 2023 | 30 1.5 COORDINATION WITH USAID POWER AFRICA Close coordination with the Power Africa MEL Team will be essential to the timely and effective execution of this assignment. This coordination will be led by the Backstopping Team, though the Evaluation Teams may engage occasionally with the Power Africa MEL Team. Meanwhile, the Evaluation Teams will primarily engage with the Task Order CORs and Implementing Partners for each of the four Power Africa programs to be evaluated. After finalization of the work plan, Team Leads will be responsible for sharing weekly updates with the Backstopping Team and the Power Africa MEL Team. The Backstopping Team will also package and deliver monthly progress reports to be shared with the Power Africa MEL Team after finalization of the work plan. 1.6OVERVIEW OF WORK SCHEDULE AND DELIVERABLES The evaluation will be conducted over a 23-week period that began on January 3, 2023 and concludes on June 9, 2023. The key evaluation deliverables and milestones are listed below, and Table 1 below further presents the evaluation work schedule. Milestones and Deliverables. All dates shown are in 2023. ▪ Kick-off meeting with USAID: January 10 ▪ Draft evaluation work plan submitted to the Power Africa MEL Team: January 13 ▪ Branding and marking plan: January 13 ▪ Draft evaluation design: January 24 ▪ Work plan returned by USAID: January 27 ▪ Evaluation design approved by USAID: February 7 ▪ Final draft of work plan and evaluation design: February 10 ▪ Work plan approved by USAID: February 17 ▪ Finalization of document library: March 3 ▪ Finalize data collection: March 30 ▪ Finish data analysis and interpretation: April 14 ▪ ICF-USAID collaborative workshop: April 21 ▪ Evaluation Report Outlines submitted to Power Africa MEL Team: April 21 ▪ Draft reports #1 submitted to Power Africa MEL Team: May 5 ▪ Presentation to Power Africa MEL Team: May 5 ▪ Receive comments from Power Africa MEL Team: May 12 ▪ Draft reports #2 submitted to Power Africa MEL Team: May 19 ▪ Receive comments from Power Africa MEL Team: June 2 ▪ Final evaluation reports submitted to Power Africa MEL Team: June 9 EXTERNAL PERFORMANCE EVALUATION OF WAEP | JUNE 2023 | 31 Exhibit 3: Evaluation Work Schedule W0 W1 W2 W3 W4 W5 W6 W7 W8 W9 W10 W11 W12 W13 W14 W15 W16 W17 W18 W19 W20 W21 W22 W23 Contract signed Contract start date Kick off meeting with PA MEL Team Branding and Marking Plan submitted to PA MEL Team Work Plan submitted to PA MEL Team (first draft) Work Plan reviewed by PA MEL Team and returned with comments Evaluation Design submitted to PA MEL Team (first draft) Evaluation Design reviewed and approved by PA MEL Team Final draft of Work Plan submitted to PA MEL Team Final draft of Work Plan reviewed and approved by PA MEL Team Determine list of key informants Develop evaluation instruments EXTERNAL PERFORMANCE EVALUATION OF WAEP | JUNE 2023 | 32 Establish document library Data collection activities Data analyses ICF-USAID collaborative workshop Develop Evaluation Report outlines Draft report #1 submission Presentation to PA MEL Team USAID review of reports Draft report #2 submission USAID review of reports Final reports submission USAID review and approve the final report EXTERNAL PERFORMANCE EVALUATION OF WAEP | JUNE 2023 | 33 2. EVALUATION DESIGN AND METHODOLOGY ICF’s detailed evaluation design elaborates a clear plan for conducting each program evaluation. The design has been informed by each individual program’s theory of change and expected outcomes and addresses evaluation questions that have been refined during consultations with the USAID Power Africa MEL Team. For this evaluation design, a detailed program-specific evaluation matrix has been developed that sets out the key evaluation questions and sub-questions, indicators, data collection and analysis methods, and sources and types of data (see Annex 1). Each individual evaluation will tailor the overarching evaluation design to the program context and will be informed by the desk review and internal consultations. All aspects are subject to revision, including the sub-questions, data sources, and design strategies. Data availability and other factors may inform a prioritization or elaboration of the evaluation questions. The evaluation design elaborates the data collection methods, including documents to be reviewed and proposed selection criteria for identifying key informants, focus group members, and survey respondents, identifying respondents that represent stakeholders from partner organizations, key government stakeholders, civil society and private sector entities, and other organizations involved in the sector (e.g., other development partners or sector experts). The selection criteria for key informants and respondents will be informed by the program’s interventions and key assumptions identified in the program’s theory of change. Program-specific data collection plans will present a framework and methodology for data collection and analysis and present a strong rationale for identified number of key informant interviews, group discussions, and surveys. These plans will be shared in the final evaluation report. 2.1DATA COLLECTION METHODS AND TOOLS Evaluation teams will employ a combination of the following primary types of data collection. The final set of methods will be determined by the desk review and approved set of key questions and areas of inquiry for each evaluation. Document Review. Before collecting data and finalizing evaluation designs and data collection plans, ICF will review relevant program documents, including program implementation plans, annual Work Plans, annual and quarterly performance reports, performance monitoring plans and systems, technical reports, and other relevant program publications. We will also review external publications on the energy sector and other interventions in the similar sub-sector and geographic area to consider other best practices and models. Document review will inform evaluation team planning and provide background and context for probing questions associated with the key informant interview and focus group discussions related to individual evaluation questions. Evaluation Team members will take notes to highlight key learnings and knowledge gaps; these will be shared, triangulated, and discussed as a team, and we will incorporate the findings into the evaluation reports. Key Informant Interviews (KIIs). ICF will develop semi-structured interview guides, tailored by program-specific stakeholder type. These guides will be based on the document review, which will help provide additional nuance for each program, and the evaluation questions outlined in the evaluation matrix below. We will iteratively test and improve these interview guides. The basic structure of the EXTERNAL PERFORMANCE EVALUATION OF WAEP | JUNE 2023 | 34 interview guides, with examples, was provided in draft form on January 31. The final guides will be shared as an annex in the final evaluation report, to allow for mid-course corrections, as needed. Semi-structured interviewing enables the interviewer to follow an agreed-upon set of questions, while retaining flexibility to adapt the process depending on the interviewee and to probe where a fruitful line of questioning emerges. ICF will design the interview guides to align to the specific questions and sub￾questions that the evaluation aims to answer. The evaluation will benefit from our highly experienced interviewers who bring methodological rigor and technical expertise of the energy sector, enabling a richer, more holistic, and nuanced account of factors influencing each program’s performance. The ICF Team will schedule interviews via preferred video platforms (e.g., Google Meet, Microsoft Teams, Zoom, Skype, or WhatsApp), taking internet access limitations into consideration. Interviewers will take detailed, typed interview notes, and, when feasible and with the consent of interviewees, we will record KIIs to facilitate validation. Evaluation teams will anonymize interview notes in line with standard evaluation ethics and coded in software to facilitate qualitative analysis. Focus Group Discussions (FGDs). ICF will design FGD guides according to the program-specific evaluation questions. Guides and lists of questions will be tested iteratively to improve their efficacy. These FGD guides will follow the illustrative interview guide provided to USAID on January 31. The final guides will be shared as an annex in the final evaluation report. Focus groups will take advantage of group dynamics and will be gender sensitive, allowing for greater security and open dialogue. Discussion sessions with key stakeholders will be conducted in small groups of six to eight participants to explore topics related to their knowledge, attitudes, practices, and other relevant insights specific to program￾level activities. Given that FGDs will need to take place virtually, the ICF Team will consider bandwidth constraints in determining the final number and scope of groups for each evaluation. Survey. As deemed useful, ICF will design a brief online survey to collect additional information on relevance, effectiveness, impact, and sustainability criteria that is reflective of the evaluation questions. As part of the evaluation design process for each evaluation, we will work closely with the USAID Power MEL Team to explore the usefulness and feasibility of a survey. The final decision for including a survey will be contingent upon whether contact information of respondents is readily available and the relevancy of the final evaluation design. The use of surveys will also be dependent upon initial desk reviews and key informant interviews and will vary by program. We would plan to use both discrete answer choices (Likert-scale) for quantitative analysis and open-ended questions to gather richer qualitative information, with follow-up for any special or outlier comments requiring clarification or explanation. All individual responses provided through the survey would be treated as strictly confidential and will not be shared outside the ICF Team; aggregated results would be shared with USAID Power Africa MEL Team and included in the evaluation report. ICF would provide a comprehensive methodology of each survey in each evaluation design document. In developing and analyzing the results, ICF will draw on the expertise of our In-country Specialists. 2.2DATA ANALYSIS The evaluation team will use several analytical methods over the course of individual evaluations, to identify key evidence-based findings. EXTERNAL PERFORMANCE EVALUATION OF WAEP | JUNE 2023 | 35 Qualitative Data Analysis ICF will conduct an in-depth, structured data analysis process to ensure that findings are credible, valid, and evidence based. The qualitative data analysis will also identify data that help to interpret and triangulate the findings from the analysis of quantitative data and case studies. The data review process will focus on the use of content analysis and pattern observation to draw understanding and context from the information. Evaluation teams will use descriptive analysis to identify specific content that speaks to the overall program interventions in the program areas as well as program-specific themes. In addition, any recordings of KIIs and FGDs will provide teams with direct quotes from respondents that can be used as supporting material for findings and conclusions in the draft and final evaluation reports. Data coding is an exercise to thematically group data (the content of the transcripts) to further support qualitative data analysis; how the data are coded becomes the basis for analysis. Each ICF Evaluation Team will develop a set of thematic codes specific to each evaluation. Each team will code (contingent upon final evaluation designs) the qualitative data and identify emergent themes using the Dedoose analysis software, a cross-platform computer application that will be used to analyze qualitative and quantitative data as a part of a mixed-methods evaluation. Individual teams will develop deductive codes that correspond to the research questions outlined in the interview transcripts. The teams will also incorporate inductive codes, capturing additional emergent or unexpected themes. With the codebook in place, the teams will read the translated transcript with each statement or exchange in the transcript thematically coded. The Team Leads will oversee this process and periodically check the coding of the other team members supporting this task. The Evaluation Teams will analyze data using content and contribution analysis. Content analysis will be used to identify themes and trends within and across respondent groups. We will also bring further understanding to contextual and environmental factors and their influence on behaviors, as possible. Contribution analysis will explore causal mechanisms and verify theories of change based upon identifying critical underlying assumptions. Quantitative Methods ICF will use a mix of data analysis methods to review program-level quantitative data. The ICF Evaluation Teams will analyze each program’s performance monitoring data, obtained through data management systems (USAID Power Africa Power Africa Information System [PAIS] and Power Africa Tracking Tool [PATT] systems) to determine whether annual program targets have been met. Baseline figures for all performance monitoring indicators will be compared to endline results for activities that are at the end of their program cycles. Each ICF Evaluation Team will also analyze disaggregated data to provide a deeper assessment of results focusing on issues of energy equity, gender (where relevant), self-reliance and other cross-cutting issues. These data analyses will help in identifying critical assumptions, challenges, and gaps in addressing each program’s theory of change. The ICF teams will use other methods such as trend analysis to compare year-to-year figures across standard indicators of particular interest. The Evaluation Teams will triangulate the results of the qualitative and quantitative analyses during a working group session to identify key findings and conclusions. This process will ensure that all findings are supported by multiple evidence sources and analyses. EXTERNAL PERFORMANCE EVALUATION OF WAEP | JUNE 2023 | 36 2.3DATA QUALITY ASSURANCE The Evaluation Teams will adhere to high professional standards in collecting and analyzing data at every stage. Interviews will be attended by at least two evaluation team members (barring exceptional circumstances), who will each take detailed notes. The quality and validity of these notes will be assured in the process of consolidating them into a single file documenting each individual or group interview, which will be shared with the entire evaluation team. Throughout, the teams will discuss and develop emergent findings in an iterative process. 3. DATA COLLECTION SCHEDULE Each Team Lead will conduct an initial interview with the Contracting Officer’s Representative (COR) for their respective program. Based on this initial interview, the Team Leads will review the detailed list of stakeholders provided by the USAID Power Africa MEL Team and will determine an appropriate interview schedule. A meeting will also be scheduled with USAID Power African Monitoring Specialist to better understand and obtain access to the PATT and PIAS data for all programs. In addition to the priority informants identified by the Power Africa MEL Team, the evaluation teams intend to contact other individuals for interviews. Both the interview schedule and the final list of stakeholders will be included as an annex in the Final Evaluation Report, as both will evolve throughout the data collection process. Each Team Lead expects to interview up to 50 stakeholders that fall into the following groups: NGOs/CSOs, development partners, government ministries, public and private utility companies, and energy service providers/other private sector actors. Additional stakeholder groups may be identified during the data collection process. EXTERNAL PERFORMANCE EVALUATION OF WAEP | JUNE 2023 | 37 ANNEX C: DATA COLLECTION INSTRUMENTS INTERVIEW GUIDE PROTOCOL FOR KEY INFORMANT INTERVIEWS: USAID STAFF Date: Name: Country: Title/Role: Sex: ☐ Male ☐Female Organization: USAID Interviewer(s): Role 1. What is your role in WAEP? Views on achievements/TOC 2. To what extent is the project on track to meet its outcomes? 3. In which area(s) has the project had the most success? 4. What is your view on the targets, quantitatively and qualitatively, that were set for the project? Were they appropriate or were they over ambitious? On what basis were they decided? 5. In your view, why were certain targets not achieved by a large margin? 6. What factors (both internal and external to the project) have hindered or helped in the achievement of the project’s expected results? 7. What were the challenges faced by the project and how did it evolve and adapt to changing conditions and realities? What changes in approach and activities were made? How did it affect outcomes? 8. How did COVID impact the implementation and outcomes of the project? 9. What role did the USAID missions play in achieving outcomes? 10. What role did partnerships, e.g., with AfDB, and collaboration with other development partners play in achieving results? 11. What were the challenges particular to the regional electricity market/trade outcomes? 12. What key assumptions did not hold true? a. E.g., related to conditions for private sector investment in generation and transmission and the ability of WAEP to address obstacles? b. E.g., related to an evolving market and expectations of expanding energy generation and access? c. E.g., related to policy reform? d. E.g., related to the regional electricity market? Views on implementation team/coordination 13. How has the contractor/implementation team performed? 14. In your view how did the contractor perform? EXTERNAL PERFORMANCE EVALUATION OF WAEP | JUNE 2023 | 38 15. How did the incoming contractor/implementation team engage with the targets, indicators and project pipeline? 16. Have there been issues with staffing/staff turnover? 17. Are they where they should be on budget, progress, etc.? 18. How effective are embedded and technical advisors at engaging different types of actors, e.g., private sector, host government entities, utilities, regional entities and other key partners? 19. What is the value of the embedded advisor model? E.g., in AfDB, and in countries? 20. How effectively has the project identified and reduced gaps regarding gender equity and women’s leadership role? 21. How are the overall project monitoring, evaluation and learning activities going? 22. How effective have the project’s monitoring and evaluation activities been in contributing to programming decisions? 23. How well suited are the project indicators for capturing intended outcomes? Are there indicators that should be modified or dropped? External coordination 24. How effective has the project been in coordinating with other US Government activities? How about those of other donors or major players? What are the synergies, potential for collaboration, and are there any overlaps or gaps? How has coordination and collaboration evolved and how would you like it to evolve? 25. Were there unexpected developments, e.g., related to the role of DFC or other entities, positive and negative? Lessons learnt/Improvements/next steps 26. What are key lessons learnt? 27. What adjustments, corrective actions, and/or areas for improvement are needed to enhance effectiveness, impact, and sustainability? 28. What is the current thinking on the next phase? Evaluation activities 29. What type of feedback would you like to come out of this evaluation? 30. Which stakeholders are most critical to interview? Closing 31. Any other comments? EXTERNAL PERFORMANCE EVALUATION OF WAEP | JUNE 2023 | 39 PROTOCOL FOR KEY INFORMANT INTERVIEWS: IMPLEMENTATION TEAM Date: Name: Country: Title/Role: Sex: ☐ Male ☐Female Organization: Interviewer(s): Roles 1. What is your role in the WAEP and for how long have you been in the position? Views on achievements/TOC [focus on their outcome area/activities] 2. To what extent is WAEP on track to meet its objectives [for this outcome area]? 3. What have been the most noteworthy achievements/successes? 4. What is your view on the targets that were set? 5. How have opportunities to achieve outcomes generally been identified? 6. How were outcomes attributed to activities, and were they any issues with attribution? 7. What were the challenges specific to identifying and realizing opportunities to achieve outcomes? 8. What types of support have been provided and which ones seem to be gaining the most traction and which ones have gained less traction and why? 9. What steps are being taken to ensure the sustainability of results (for your activities)? 10. Which types of support are replicable? 11. What key assumptions did not hold true? 12. What factors (both internal and external to the WAEP) have helped or hindered in the achievement of the WAEP’s expected results? 13. How has COVID-19 affected implementation and overall performance? 14. How have your activities evolved or adapted to changing conditions, realities, feedback from recipients and experience? Were there major changes in approach during implementation? Views on implementation team/coordination 15. How do you see the role of embedded advisors? What are the benefits of an embedded advisors and what are conditions for the model to be effective? Are there any downsides? 16. In what ways, if at all, do you work to identify and reduce gaps regarding gender equity and women’s leadership role as part of your WAEP activities? 17. How are the project monitoring, evaluation and learning activities going especially for your Outcome area(s)? 18. To what extent have the project’s monitoring and evaluation activities been contributing to programming decisions for your outcome area(s)? 19. How well suited are the project indicators and targets for capturing intended outcomes? Are there indicators that need to be modified or dropped? External coordination EXTERNAL PERFORMANCE EVALUATION OF WAEP | JUNE 2023 | 40 20. What was the extent and nature of coordination with other US Government activities or those of other major international players? 21. How is the coordination going? What were the synergies, gaps and overlap if any? What were the challenges related to coordination and collaboration? How has coordination and collaboration evolved? How would you like it to evolve? Lessons learnt/improvements/next steps 22. What would you like to see happen for your Outcome Area(s) in the next phase? Similar activities or shift in emphasis? On which activities would you have the project concentrate? 23. How are activities decided? Besides the work plan, how much flexibility is there within a year? Evaluation activities 24. Which stakeholders are most critical to add depth to what is in the reporting? Closing 25. Any other comments? PROTOCOL FOR KEY INFORMANT INTERVIEWS: INTERNATIONAL PARTNERS/OTHER Date: Name: Country: Title/Role: Sex: ☐ Male ☐Female Organization: Interviewer(s): Roles 1. Please briefly characterize your role and your relationship to the WAEP? Which WAEP activities are you most familiar with and for how long have you been in your position? 2. How do the activities of [your organization] link with WAEP activities? 3. Who are your main contacts on the WAEP team? 4. What was the nature of the coordination or collaboration, if any, with WAEP at the institutional/individual level? 5. [Where appropriate] Were you familiar with PATRP? In your view how does WAEP differ from PATRP? Views on achievements/TOC [focus on their outcome area/activities] 6. In your view, how relevant has WAEP been in your area of focus? EXTERNAL PERFORMANCE EVALUATION OF WAEP | JUNE 2023 | 41 7. What have been the most noteworthy successes and impacts? 8. What is your view on the targets set by WAEP? 9. What is your view on the outcomes claimed by WAEP? 10. Is WAEP’s role in your area of focus generally known and clear? 11. Is WAEP as a new program filling a gap, and if so, which gap? 12. In your view how effective has WAEP been? 13. How has WAEP’s approach and operating model affected its effectiveness and results? 14. What factors help or hinder progress in their activities to support WAEP’s objectives? 15. How well coordinated is WAEP with other donors, organizations, or other entities that are also providing support in these or similar areas? 16. Are there gaps or overlaps? Is there potential for more/improved coordination, and if so, how? Improvements 17. How could WAEP be strengthened? What should it focus on in a next phase? Evaluation activities 18. Is there anyone else in [your organization] you think it is important we speak with as well? 19. Any other comments? PROTOCOL FOR KEY INFORMANT INTERVIEWS: DIRECT BENEFICIARIES OF WAEP SUPPORT Date: Name: Country: Title/Role: Sex: ☐ Male ☐Female Organization: Interviewer(s): Roles 1. Please briefly characterize your role and your relationship to the WAEP? Which WAEP activities are you most familiar with and for how long have you been in your position? a. Who are your main contacts on the WAEP team? Views on achievements/relevance/ToC [focus on their outcome area/activities] 2. How has WAEP contributed to your department/program/company’s objectives, how relevant has WAEP been, what was achieved and how has WAEP’s support led to outcomes and targets? a. What have been the most noteworthy achievements? b. To what extent can the outcome(s) be attributed to WAEP activities? c. Overall, how effective is the support [your department] receiving from the WAEP? What results have you experienced because of that support? d. How has the capacity of [your department] improved through WAEP support? Please describe. EXTERNAL PERFORMANCE EVALUATION OF WAEP | JUNE 2023 | 42 e. What has been done by WAEP or [your department] to ensure the sustainability of the progress made? Which conditions, change or actions are needed to ensure sustainability of the TA/capacity building provided? f. How does WAEP and its approach differ from other donor TA? 3. What factors (both internal and external to the WAEP) have helped or hindered progress in their activities to support [your department]? a. How have the activities evolved or adapted to changing conditions and needs? b. Are there opportunities that may have been missed, in retrospect? What would you have them do differently going forward? 4. Which other donors, organizations, or other entities are also providing support in these or similar areas? a. How well coordinated are the efforts? Are there obvious gaps or overlaps? b. How could coordination be enhanced? Views on implementation team/coordination 5. What is your view on the skills/performance of the technical/embedded/transaction advisors? a. How was your interaction with the technical/embedded/transaction advisors? b. How were their activities decided? How collaborative was their approach? How effective has the approach been to date? 6. In what ways, if at all, does the WAEP work to identify and reduce gaps regarding gender equity and women’s leadership role as part of WAEP’s activities relating to [your department]? a. Please provide examples of successes. Improvements/next steps 7. How could WAEP be more effective and have more impact? What could be improved? What would you like to see happen in the next phase of the WAEP? Similar activities or shift in emphasis? On which activities would you have the WAEP concentrate? Evaluation activities 8. Is there anyone else in [your organization] you think it is important we speak with as well? a. [If needed:] How would you recommend they be approached? EXTERNAL PERFORMANCE EVALUATION OF WAEP | JUNE 2023 | 43 ANNEX D: SOURCES OF INFORMATION LIST OF ORGANIZATIONS CONSULTED The following table lists the organizations represented in the 84 KIIs conducted. In several cases, such as for implementation partners, USG agencies, development partners, public energy companies, host country governments and others, there are multiple interviews and informants per entity. Organization U.S. Government - 6 InfraCo Africa USAID GridWorks U.S. Trade and Development Agency Niger - NIGELEC Millennium Challenge Corporation Senegal - Enertec National Renewable Energy Laboratory (NREL) Senegal - Lekela Power Host country government - 16 Senegal - SENELEC Benin – Ministère de l’Energie - DGRE Sierra Leone – Milele Energy Benin - Autorité de Régulation de l’Electricité (ARE) Sierra Leone - Sewa Energy Burkina Faso - Authorité de Régulation du Secteur de l’Energie (ARSE) Sierra Leone - Serengeti Energy Cameroon - Agence de Régulation du Secteur de l’Electricité (ARSEL) Togo - CEET Chad - l’Autorité de régulation du secteur de l’énergie électrique (ARSE) du Tchad OMVG Côte d’Ivoire - Ministry of Energy – DGE (Department of Energy Generation) TransCo CSLG Ghana - Ministry of Energy Implementation Partners - 18 Ghana - Ministry of Finance Deloitte Ghana - Energy Sector Recovery Program (ESRP) Crossboundary Mauritania – Ministère du Pétrole, de l’Energie et des Mines EnerNex EXTERNAL PERFORMANCE EVALUATION OF WAEP | JUNE 2023 | 44 Organization Senegal – Commission de Régulation du Secteur de l’Electricité (CSRE) Development partner - 12 Togo - Ministry of Energy African Development Bank Togo – Regulatory Authority for the Electricity Sector (ARSE) World Bank Ghana - Wholesale Market Committee Agence Française de Développement (AFD) Ghana - Energy Commission Tony Blair Institute for Global Change Public or private energy company – 23 GIZ Cape Verde - ELECTRA SA Other - 9 Côte d’Ivoire - Ci-Energie WAPP ICC Côte d’Ivoire - SODEN ERERA Ghana - Ghana National Petroleum Company ECREEE Ghana - Grid Company (GRIDCo) Energy for Growth Hub Ghana – Volta River Authority (VRA) International Rescue Committee (IRC) Ghana - Bui Power Authority Ghana - Agribusiness AUB Dassan Ventures Ghana - National Gas Clearing House Ghana - Agribusiness Bulsa Agroforestry Liberia – Liberia Electric Company (LEC) Ghana - Agribusiness Y I Enterprise BIBLIOGRAPHY OF DOCUMENTS REVIEWED The evaluation team primarily reviewed project documents (i.e., annual work plans, quarterly progress reports, portfolio reviews) that were provided to ICF by the WAEP CLIN1 and CLIN2 CORs. In addition to these program documents, the evaluation team consulted the following public reports: Balabanyan, A., Semikolenova, Y., Singh, A., and Lee, M. A. 2021. ”Utility Performance and Behavior in Africa Today (UPBEAT): Summary Report.” Energy Sector Management Assistance Program Papers. Washington, D.C.: World Bank Group. Bloomberg New Economy Forum. 2022. “New Energy Outlook 2022.” Bloomberg Finance L.P. Bloomberg New Economy Forum. 2022. “Scaling-Up Renewable Energy in Africa: A NetZero Pathfinders Report.” Bloomberg Finance L.P. EXTERNAL PERFORMANCE EVALUATION OF WAEP | JUNE 2023 | 45 Energy Sector Reform Programme. 2022. “IMF Issues, Responses and Financial Projections.” Green Climate Fund and African Development Bank. 2021. “FP178: Desert to Power G5 Sahel Facility.” International Renewable Energy Agency and African Development Bank. 2022. “Renewable Energy Market Analysis: Africa and its Regions – A Summary for Policy Makers.” USAID Power Africa. n.d. “Results of Regional Technical Needs Assessment for Regulators.” West African Power Pool. 2023. “Solar Development in Sub-Saharan Africa – Phase 1 (Sahel).” World Bank Group. World Bank Group. 2015. “Project Appraisal Document for the OMVG Interconnection Project.” DATABASES REVIEWED None. EXTERNAL PERFORMANCE EVALUATION OF WAEP | JUNE 2023 | 46 ANNEX E: DISCLOSURE OF ANY CONFLICTS OF INTEREST None. ANNEX F: STATEMENTS OF DIFFERENCES None.