SARI/EI Phase IV Evaluation Final Report October 30, 2019 Contents I. Acronyms.............................................................................................................................................................. 1 II. Executive summary ........................................................................................................................................... 2 III. Background and context.................................................................................................................................. 6 The importance of CBET in South Asia................................................................................................................6 The evolution of CBET in South Asia....................................................................................................................7 The genesis and evolution of the USAID SARI/E program ........................................................................ 10 Purpose and rationale of the evaluation:......................................................................................................... 11 IV. Methodology: ....................................................................................................................................................12 Details of the DAC framework............................................................................................................................ 12 Inputs into the analysis .......................................................................................................................................... 13 Limitations of the evaluation approach............................................................................................................ 14 V. Findings and conclusions................................................................................................................................15 Summary of findings: .............................................................................................................................................. 15 Relevance: .................................................................................................................................................................. 16 Effectiveness & Efficiency:.................................................................................................................................... 22 Impact & Sustainability:......................................................................................................................................... 30 VI. Recommendations ...........................................................................................................................................36 Activities during Phase IV extension: ............................................................................................................... 37 Activities beyond 2022:......................................................................................................................................... 48 VII. Annexure ............................................................................................................................................................52 Stakeholder Survey................................................................................................................................................. 52 1 SARI/EI Phase IV Evaluation | Final Report I. ACRONYMS Acronym Definition AC Alternating current ADB Asian Development Bank ASEAN Association of Southeast Asian Nations BBIN Bangladesh, Bhutan, India, Nepal BD Bangladesh BIMSTEC Bay of Bengal Initiative for Multi-sectoral Technical and Economic Cooperation BT Bhutan CB Trans. Lines Cross border transmission lines CBET Cross border energy trade CERC Central Electricity regulatory commission (of India) CO2e Carbon dioxide equivalent DAC Development Assistance Committee of the OECD DC Direct current G2G Government to government GDP Gross National Product GHG Greenhouse gas GW Giga watt IN India IRADe Integrated Research and Action for Development [in energy] JV Joint venture kV Kilovolts M&E Monitoring and Evaluation MCC Millennium Challenge Corporation MOU Memorandum of understanding MTOE Million of tonnes of oil equivalent MW Mega watt NGO Non-governmental organization NP Nepal PDA Project Development Agreement PMO Prime Minister’s office PPA Power purchase agreement PSC Project steering committee PTA Power trading agreement SAARC South Asian Association for Regional Cooperation SAFER South Asia Forum of Electricity Regulators SAFIR South Asia Forum for infrastructure Regulation SAFSO South Asia Forum of System Operators SAFTU South Asian Forum of Transmission Utility SAPP South Africa Power Pool SARI South Asia Regional Integration for Energy integration SARPEX South Asian Regional Power Exchange SASEC South Asia Subregional Economic Cooperation TF Task Force TOR Terms of reference USAID United States Agency for International Development USD United States Dollar USG United States Government 2 SARI/EI Phase IV Evaluation | Final Report II. EXECUTIVE SUMMARY Energy needs in South Asia are rising due to a confluence of factors that include rising populations and fast-growing economies. The region’s energy sector is expected to be strained in meeting this rising need. Several solutions will have to work in concert to meeting the new demands. One innovation that is expected to contribute to meeting future regional energy needs is the creation of a inter-connected electricity grid in South Asia. This will seamlessly connect the energy grids of South Asian countries so that they can function as a larger whole. The interconnected grid would allow for greater diversification of both generation and load, which will improve utilization rates of generation assets, lower power costs for consumers, while increasing flexibility and reliability on the grid. Such an interconnected grid would be particularly beneficial in South Asia as these countries have complementary load profiles. Given the benefits of a regionally interconnected energy system, the region has slowly been increasing the volume of cross border energy trading (CBET). The normative CBET development journey USAID has been supporting South Asia’s CBET development journey over the past 20 years through its SARI/E program. SARI/E’s interventions have spanned four phases, with the first three phases focused on the Stage 1 of the CBET Journey, aimed primarily at changing mindsets. While phase 4 (SARI/EI), which spanned from 2012 to 2018, transitioned to preparing the sector to scale up CBET. This reflects the sector’s progression amid the transition from stage 1 to stage 2 of the CBET development journey. In line with the transition to Stage 2, stakeholders are beginning to seek interventions that supported CBET infrastructure development. SARI/EI, which was scheduled to end in 2018, has been extended for four years. In support of this extension, USAID has engaged Dalberg Advisors to conduct an evaluation of SARI/EI’s activities with three goals in mind: Evaluation of Phase IV (SARI/EI): This evaluation will provide a backward-looking view on the performance of the SARI/E project. Specifically, it offers a view of the implementing organization and its ability to produce the desired outcomes. Provide recommendations for the extension: Leveraging learning from the evaluation of Phase IV, this report will then take a view of Phase IV extension and offer recommendations on how to improve both the impact of the program and the effectiveness of IRADe as the implementing partner. 3 SARI/EI Phase IV Evaluation | Final Report Derive learnings to inform USAID’s future regional programming: SARI/EI has provided a wealth of learning that can be applied to other programming especially as USAID looks to incorporate more regional rather than national work in its interventions. EVALUATION OF PHASE IV Indicative theory of change from SARI Phase IV In phase IV, SARI/EI sought to (1) create consensus amongst stakeholders, (2) develop guidelines to guide regional CBET and (3) identify and help implement the CBET project. To do this they formed three task forces, each looking at different aspects of CBET development. 1/ Relevance: High. Overall, SARI’s activities were well aligned with its goal of helping South Asia along the CBET development journey. Specifically, SARI’s phase IV activities helped prepare the sector for scaled up CBET. For example: the creation of operating and technical standards should simplify how countries conduct CBET by providing stakeholders with a common language for mutual interactions. In performing these activities, SARI/EI has played a necessary role as a third-party convener given that South Asian countries often view regional programs championed by any one country with suspicion. SARI, being affiliated with USAID and thus seen as an honest broker, successfully broughttogether technical and political stakeholders from to participate in SARI/EI’s task forces and activities. This was necessary given that that SARI/EI wanted countries across South Asia to adopt its recommendations and thus crowding-in input from stakeholders of different nationalities was needed to ensure that a multiplicity of views was accounted for. In addition to getting the right people on board, the development of technical, operational and market-based standards was an appropriate next step for supporting the transition from Stage I to Stage II of the CBET development journey. Lastly, USAID decision to work with IRADe as the implementing partner was relevant to sectoral need. It reflected the need to marry USAID’s position as an outsider with a partner that had on-the-ground expertise with the South Asian power sector given that the sector was transitioning towards focusing on infrastructure development, which requires localized knowledge. 2/ Effectiveness and efficiency: High. IRADe, as the implementing partner, was governed through a series of terms of reference (TOR) documents which detailed USAID’s expectation for IRADe. This use of TORs was appropriate given that it was the first official partnership between USAID and IRADe. The clear instructions provided by the TOR helped IRADe to meet materially all its output targets with relatively high quality. This included the creation of the task forces and the publication of key research reports. These outputs were produced by IRADe under budget, but significantly over the allotted timeline. The slippage in the timeline reflected an unintended consequence of the fact that IRADe and Objective Scaled-up bilateral and multilateral cross-border electricity trading in South Asia. Mission Formalize and standardize enabling conditions for cross-border electricity trade and energy markets in the region. Software: Common legal and regulatory frameworks and guidelines established (TASK FORCE 1) Hardware: Interconnection opportunities identified and supported (TASK FORCE 2) Markets: Market rules and platforms established and supported (TASK FORCE 3) Outcomes Consensus among key decision makers to harmonize policies Common guidelines and regulations for CBET Outputs Activities Collect and analyze data: Collect data, research and analyze relevant issues CBET projects identified and increasingly implemented Prepare research outputs: Prepare relevant documents, reports, pilots, templates etc. as needed to build stakeholder awareness, engagement and commitment to CBET Engage stakeholders: Establish three TFs and the PSC, plan and hold meetings, consultations, workshops, conferences, study tours, and capacity building events 4 SARI/EI Phase IV Evaluation | Final Report USAID did not enter into a standard grant-maker/implementer contract. Instead, they operated under a cooperation agreement where IRADe was a co-branded partner. This meant that IRADe would be given greater latitude in its interventions, but also that IRADe would need to meet more stringent USAID guidelines. It took nearly a year for IRADe to bring its internal system to USAID standards, during which time the delivery of key of outputs was delayed. Survey Results related SARI’s outputs and influence 3/Impact:Medium. South Asia has seen progress along the CBET journey, in particular, there has been increasing discussion around the desire for bilateral and trilateral trading and to more trading through synchronized AC connections (which would create a true interconnection). SARI/E is a major contributor to this success as evidenced by the fact that SARI/E’s interventions are often brought up as being factors that encouraged the development of CBET. As a result, SARI/E’s brand is well known in the South Asia Energy ecosystem. While SARI/E’s has been an important contributor to CBET development, specific attribution of SARI/EI’s intervention to influencing policy has been less obvious despite stakeholders finding SARI’s work to insightful. This in part reflects that SARI/EI was focused on developing reports (which raised the quality of CBET related discussion/ thinking) rather than trying to directly influence stakeholders such as policymakers. Forward looking recommendations The report provides two sets of recommendations covering in detail recommendations that should be implemented quickly to improve impact and operational efficiency and a set of longer-term principles to guide programmatic development after 2022. 1/ The first set of recommendations covers SARI/EI extension which begins immediately and last until 2022. These recommendations lay out how SARI/EI, and IRADe as the implementing organization, can improve its impact and support South Asia during its transition from Stage I to Stage II of the CBET development journey. SARI should adopt the following actions to refine its programming to meet the changing needs: 1.Amplify SARI’s voice through structured communications: To date, insufficient dissemination of outputs and sporadic engagement with stakeholders have limited SARI/EI’s impact despite producing insightful analysis. SARI/EI needs to amplify it voice including with top politicians, key influencers, bureaucrats and funders to ensure that (1) its outputs are translated to impactful change and (2) governments provide the support needed to scale up CBET. 2. Invest deeply in training and capacity building: Given that CBET requires a new set of technical skills different from traditional utility operations, training is necessary lest the energy sector builds out infrastructure and standards which are not utilized because 5 SARI/EI Phase IV Evaluation | Final Report individuals don’t know how to leverage the new CBET systems. A mix of technical and broader training aimed at educating the sector on the evolving landscape is needed. 3. Shift focus to supporting infrastructure developed: SARI/EI should support efforts to translate success on establishing standards into on-the-ground implementation of CBET projects . This will help to promote the scaled up CBET infrastructure development needed to transition to Stage 2 of the CBET journey. Given that IRADe lacks specific expertise in developing infrastructure projects and engaging with the private sector, it should use the time (up to 2020) to build capacity while creating a database of development project to track progress in this sector over time. Adopting these changes will require a shift in how SARI operates. Firstly, SARI needs to move away from a demand-driven approach to strategically focus its interventions. During Phase IV, interventions adopted by SARI were supposed to reflect what task force members “demanded”. This created alignment between members’ desires and task force’s outputs. But as the sector moves away from creating regional standards towards implementation, a more intentional set of interventions to create the environment needed for infrastructure development is required. Secondly, while IRADe was successful in using terms of reference (TOR) documents to guide the generation of outputs, this process gave insufficient weight to creating end outcomes. Transitioning from a TOR approach to a MEL based governance approach which includes outputs, as well as outcomes, ensures that IRADe has the agency to drive outcomes. Finally, IRADe should localize its work to ensure that projects include the local nuances needed to support infrastructure development. These changes will also require SARI to develop new capabilities. The new activities and modifications to the operating structure will require a change to IRADe’s day-to-day work. To ensure success, new capabilities must be brought into IRADe to take advantage of SARI’s new direction. Much of the new capabilities can be developed through training of existing staff. But, some capabilities, such as the skill set needed to identify viable CBET project, should be brought in through external hires. Lastly, where appropriate, SARI should look to partner with other organizations to quickly develop capacities that may otherwise be too expensive to adopt. For example, SARI should deepen its existing relationship with USAID bilateral mission to develop a better understanding of local context. 2/ The second set of recommendation lays out guiding principles for how SARI/EI should operate beyond the extension: While South Asia will in the midst of transitioning from Stage I to Stage II of the CBET development journey during the SARI/EI extension, we believe that South Asia will be firmly within Stage II by 2022. The recommendations for SARI/EI’s actions after 2022 thus reflects guiding principles for how to support South Asia as it scales up CBET. Focus sharply on supporting execution, to get infrastructure projects off-the-ground. SARI should support efforts to translate success on establishing standards into on-the-ground implementation of CBET projects (including infrastructure development). This will help to promote the scaled up CBET development needed to transition to Stage 2 of the CBET journey. A SARI team can support developers at multiple points along the project development process to ensure that the projects progress smoothly. 6 SARI/EI Phase IV Evaluation | Final Report III. BACKGROUND AND CONTEXT The importance of CBET in South Asia Energy needs in South Asian countries are rising sharply, and domestic supply is unlikely to meet demand in the future. Over the past two decades, South Asia has been one of the fastest growing regions in the world, exhibiting material increases in both GDP-per-capita and population. These trends are projected to continue into the near future and drive considerable increases in energy demand.1 For example, total energy consumption in India—the region’s behemoth—is expected to rise from 746 mtoe in 2017 to 1,928 mtoe by 2040. 2 Similarly, in Bangladesh, demand is expected to rise from 9,479 MW in 2017 to 61,681 MW in 2040.3 To meet the rising demand, South Asian countries are racing to add new sources of energy to power their economies. Nepal and Bhutan are each adding approximately 5 GW of installed hydropower capacity over the next five years.4 India is projected to grow its installed capacity with a focus on solar and wind that will form approximately 50% of installed capacity and 35% of electricity generation by 2029-30.5 Power outages, however, continue to be a concern, resulting in higher-than-global-average loss of value for affected firms in Pakistan, Nepal, Afghanistan, and Bangladesh.6 Exhibit 1: Energy consumption by fuel type, % of Mtoe, India, 2017-40 (projected)7 The South Asian countries’ complementary load profiles and mix of generation assets create a major opportunity for cross-border electricity trade (CBET). The seasonal variations in the demand for electricity in South Asian countries see a high degree of complementarity. For instance, Nepal has excess supply of hydropower during the monsoon months, when there is high demand in India; meanwhile Nepal has high demand in winter months when India has relatively low demand. Furthermore, different countries in the region rely on different power sources for their domestic power supply – while India draws most of its power from coal, Bangladesh depends on gas, and Nepal and Bhutan on hydropower.8 1 ADB, “Energy Outlook for Asia and the Pacific,” 2013. 2 BP, “BP Energy Outlook,” 2019. 3 PWC, “Transforming the Power Sector in Bangladesh,” 2017. 4 Royal Government of Bhutan, “National Transmission Grid Master Plan (NTGMP) of Bhutan-2018,” 2018; International Hydropower Association website. 5 Central Electricity Authority, “Draft Report on Optimal Generation Capacity Mix for 2029-30,” 2019. 6 World Bank Data website 7 BP, “BP Energy Outlook,” 2019. 8 SARI/EI, “Cross-Border Electricity Trade in South Asia: Key Policy, Regulatory Issues/Challenges and the Way Forward,” 2016. 7 SARI/EI Phase IV Evaluation | Final Report Exhibit 2: Monthly load on select South Asian grids (2013-14)9 CBET offers several benefits to the region in the long-term. CBET provides a range of benefits that are aligned with the needs of the South Asian energy markets development. These benefits are economic, social, and environmental in nature. Economic benefits arise from more efficient use of generation resources which can lead to lower power costs that will benefit consumers and spur economic development. More broadly, CBET helps generate export revenues and spur economic growth. The diversification benefits of a larger grid build resilience in the system, reduce the need for spare capacity and lower capital expenditures while allowing for deeper integration of intermittent renewable energy. There is also greater energy reliability and the ensuing impacts on a wide range of services including health, education and water and sanitation. The evolution of CBET in South Asia Our research into the evolution of regional power pools globally and our conversations with regional power sector experts suggest a three-phase trajectory forCBET development. Our general framework for CBET development enables us to not only track and articulate South Asia’s progress toward a commercial, market-based CBET environment in terms of key outcomes and needs, but also map and understand SARI’s contributions and frame our recommendations within that essential broader regional context. 9 World Bank, “How Much Could South Asia Benefit from Regional Electricity Cooperation and Trade?” 2015. 8 SARI/EI Phase IV Evaluation | Final Report Figure 1: A normative view of how CBET develops10 In Stage I, regions move from having no or limitedCBET to setting up the first opportunistic bilateral links between nations. Initially, there is no knowledge of the potential or benefits for CBET. Stage I thus begins with generating awareness of the need for and potential benefits of cross-border trade. Stakeholders include government actors who consider cross-border links as a way to meet localized energy challenges. As mindset change there is opportunistic setting up of bilateral power trading, generally led by forward thinking politicians. As these connections focus on meeting specific needs, they are each idiosyncratic and unique. Stage II builds on the success of initial links, scaling up bilateral and multilateral trading. As initial trading links establish a proof-of-concept and other trading opportunities become apparent, a region moves into Stage II to support the scale up of CBET trading. This transitions the creation of new CBET links from being idiosyncratic projects towards one where the trading of power and the development of new projects can follow a formalized and standardized set of processes. This lowers the associated cost needed to develop new connections and allows for the development of greenfield projects without the direct intervention of a political champion (or similar stakeholder). Effectively, the development of CBET connections becomes a technical issue within utilities, based on power sector needs, rather than a project led by politicians. Progression along Stage II can be observed through advancement on four crucial paths: Aligning on common operating standards to create a common language for technocrats from different countries to interact with each other on CBET matters, developing physical infrastructure to allow for the transmission of power, developing the capacity of individuals to trade power, and most importantly, maintaining political commitment for CBET, as evidenced by supportive policies and a permissive operating environment. None of these factors are themselves sufficient for scaling up CBET, but collectively, they provide the building block for scaling up CBET in a formalized and standardized manner. In this stage, a region would expect to see the scale up of CBET, which would be evidenced by increasing bilateral and eventually multilateral trade. Key actors here include political and 10 This view of CBET development focuses on the cross-border connections and does not take into consideration the strength and readiness of the national grid. We note that an effective and stable national grid is a requirement for the scaled up CBET, but has not been included in this analysis as national level energy grid development is outside the scope of this document. 9 SARI/EI Phase IV Evaluation | Final Report bureaucratic representatives from country governments, think-tanks and thought leaders, civil society, utilities, and funders and investors. The ecosystem must support these actors by building and transferring knowledge, helping in project development and stakeholder coordination and creating and capacitating the right institutions. Seamless bilateral and multilateral trading paves the way for Stage 3 in which a region moves to market-based regional trading. CBET in Stage 3 represents the high-level of CBET where the different national grids are effectively integrated through these CBET connections. Trading of power is governed by agreed upon rules and guided by market prices rather than the actions of any individual national utility. The trading process is real-time, automatic and has extremely low transaction costs. Trading arrangements are governed by an international body that oversees the flow of cross border energy, rather than national level regulators. South Asia is only just approaching Stage II in its CBET journey. South Asia’s limited CBET is currently within Stage I and can be detailed by few physical transmission lines linking national grids, occasionally incompatible inter-country technical standards and operating guidelines and wavering political will for closer energy integration due to fears of losing national energy independence or of losing valuable energy resources. As a result, only about 3,000 MW (less than 1% of capacity) of power transmission infrastructure has been set up in South Asia which has over 360 GW of regional power capacity as of 2017. 11 India itself only integrated its regional grids into a national grid in 2015, and still runs a peak deficit of 0.8%12 . Figure 2: Development of CBET in South Asia Stakeholders have, however, moved from a place of no awareness of or interest in CBET to committing to CBET as a tool in their power strategy. Much of this progress has happened since the early 2000s. First, small-scale links helped kickstart a small amount of trading, primarily in response to immediate needs (e.g. the three 132kV lines between Nepal and Bihar, India). This is noteworthy because regional dynamics, with high levels of mistrust and changing geopolitical stances, make economic cooperation in South Asia difficult.13 However, as the benefits of cross-border trade became clearer, the openness to larger scale trade has seen rising interest as is evident from the 2016 guidelines for Cross-Border Trade of Electricity Regulations announced by the Central Electricity Regulatory Commission (CERC) of India, the game changing development of the 500MW Baharampur- Behramara line between India / Bangladesh and more recently the Dhalkebar￾11 SARI/EI, “Overview of SARI/EI Program, Progress & Key Findings and Way forward,” 2017. 12 The Hindu Business Line, “Peak power supply deficit halves over 5 years,” 2019. 13 Priyanka Kher, “Political Economy of Regional Integration in South Asia,” 2012; UNESCAP, “Unlocking the Potential of Regional Economic Cooperation and Integration in South Asia: Potential, Challenges and the Way forward,” 2016. 1970 2000 2010 2013 2016 2018 • Mid-1980s IN-NP: Low voltage interconnections between Nepal and states of UP/Bihar • 2015 IN: completed unification of regional grids • 2016 IN: CERC issued guidelines on CBET • 2008 IN: IEX began the first￾ever power exchange in India • 2009 IN-NP: MoU to develop the Dhalkebar￾Muzzafarpur line • 2010 IN-BD: MoU to enhance bilateral cooperation • 2013 IN-BD: Baharampur interconnection operational • 2014 IN-NP: Power Trade Agreement signed • 2014 IN-NP: Dhalkebar￾Muzzafarpur line operational • 2017 NP: publishing of the ERC ACT • 2018 IN: updated guidelines on CBET • 2003 IN: Electricity Act of India de-licensed generation of power • 2006 IN-BT: framework agreement signed 2005 • 1968 IN-BT: commissioning of first Hydroelectric project • 1971 IN-NP: 5MW power exchange to cater to isolated pockets Stage I: PROVING BENEFITS Transition to Stage II (FORMALIZING AND SCALING UP) No CBET Initial bilateral trading Scaled-up bilateral trading 10 SARI/EI Phase IV Evaluation | Final Report Muzzafarpur line between India / Nepal. Given this progression, we see South Asia as being in the midst of transitioning from Stage I to Stage II. Recent developments in regional geopolitics have also helped create a conducive environment for CBET. China’s growing interest in South Asia has made closer integration between South Asian countries more beneficial, pushing India specifically to more closely align with its neighbors and develop stronger regional ties; this will provide additional geopolitical support for developing CBET. India’s “Look East” policy, in which it focuses on its eastern neighbors and the BBIN and BIMSTEC sub￾regions can thus be understood as part of India’s broader effort to align closely with its neighbors, bolstering the platform for CBET in the region. The genesis and evolution of the USAID SARI/E program USAID has been supporting South Asia through the journey to develop CBET. Starting in 2000, USAID initiated the South Asian Regional Initiative for Energy (SARI/E) program spanning eight countries (Afghanistan, Bangladesh, Bhutan, India, the Maldives, Nepal, Pakistan, and Sri Lanka). SARI/E’s goal was to promote energy security in South Asia through greater CBET, energy market formation, and clean energy development. SARI/E has evolved over time and has spanned four phases over nearly two decades. Each of the phases have focused on a slightly different issues in the energy sector but has always centered around broader engagement in the South Asia region. Exhibit 3: Historical development of SARI/E From inception, SARI/E has systematically worked to broaden cooperation on energy issues throughout South Asia. As it related to CBET, the early phase worked to change stakeholders’ mindsets on relevance and effectiveness. Notably, momentum around developing an understanding on the need for CBET in the region began to pick up as a result of the mindset changing activities during first few phases of the program. SARI/E’s intervention in support of CBET to date has supported the steps laid out in red, in the graphic below. Phase 1 -3 interventions were within Stage I of the region’s CBET journey. During Phase 1, trading of power was limited to small amounts between India and Nepal/Bhutan and expanded to opportunistic trading between Indian - Bangladesh during phase 2 and 3. Phase IV was unique in that its interventions extended beyond stage 1 and were part of the process to formalize the CBET 11 SARI/EI Phase IV Evaluation | Final Report development process in Stage II. This marked that the CBET discussion in South Asia was transitioning from stage 1 to stage 2 during SARI/EI. Exhibit 4: South Asia’s current state representation in the normative stages of CBET development As of the date of this report, the original term of Phase IV has concluded, and SARI/E is currently in the first year of the phase 4 extension. As an extension was an uncertainty at the end of Phase IV, all activities were to conclude originally in 2017, later extended to 2018, leaving 2019 as a relatively clean slate for developing additional programming. Phase IV extension (2018-22) stemmed from a request by various task force members from Phase IV who suggested that a follow-on was needed to ensure momentum would be maintained and results sustained and scaled up across the region. The extension was approved in September 2018 and is expected to be the final phase of the SARI/E program. Purpose and rationale of the evaluation: This study was requested by USAID to conduct an evaluation of the activities and accomplishments of Phase IV of the SARI/E program between 2012- 2018 and IRADe’s performance as an implementing partner. This will help understand and learn from the results of the past six years of SARI/E’s operations. The evaluation has two broad goals: Evaluation of Phase IV (SARI/EI): This evaluation will provide a backward-looking view on the performance of the SARI/E project. Specifically, it offers a view of the implementing organization and its ability to produce the desired outcomes. Provide recommendations for the extension: Leveraging learning from the evaluation of Phase IV, this report will then take a view of Phase IV extension and offer recommendations on how to improve both the impact of the program and the effectiveness of IRADe as the implementing partner. Derive learnings to inform USAID’s future regional programming: SARI/EI has provided a wealth of learning that can be applied to other programming especially as USAID looks to incorporate more regional rather than national work in its interventions. 12 SARI/EI Phase IV Evaluation | Final Report IV. METHODOLOGY: The DAC framework provides the logical underpinning for the valuation of the Phase IV of SARI/E program.14 Details of the DAC framework The DAC framework is made up of five complementary areas of review: relevance, effectiveness, efficiency, impact, and sustainability. Some of these areas investigate overlapping issues where analyses support each other. Therefore, we have looked at pairs of complementary areas together, clubbing effectiveness and efficiency, as well as impact and sustainability. Following this framework provides a logical flow to our analysis and ensures all programmatic aspects are captured. Figure 3: Evaluation framework Relevance Relevance gauges the extent to which SARI/EI’s interventions are necessary and helpful for South Asia. Key research questions investigated under relevance include: 1. Were Phase IV’s goals and objectives relevant to South Asia’s CBET context and needs? 2. Was SARI/EI’s role relevant and additional to South Asia’s CBET context and needs? 3. Was Phase IV’s approach relevant to what was needed in South Asia’s CBET eco-system at the time? Effectiveness and Efficiency Effectiveness measures whether desired outputs have been produced in a timely and high-quality manner. Efficiency measures whether the SARI/EI was able to convert its resources (e.g. financial and human capital) into interventions in a way that maximized the use of those resources. Efficient programs can achieve the desired output using the least costly resources. Key research questions investigated under effectiveness and efficiency include: 1. Were planned outputs achieved according to plan? a. Were TFs established appropriately and with the right membership? 14 This is the same framework used in the 2013 evaluation (conducted at the end of phase III). To what extent was SARI/ EI relevant to South Asia’s CBET landscape and needs? To what extent was SARI/ EI and the TF approach effective and efficient? To what extent did SARI/ EI achieve its desired outcomes and impact? Were Phase IV’s goals and objectives relevant to South Asia’s CBET context and needs? Was USAID SARI’s role relevant/ additional to South Asia’s CBET context and needs? Was Phase IV’s approach relevant to what was needed in South Asia’s CBET eco-system at the time? To what extent has SARI/ EI (especially Phase IV) contributed to these outcomes and impact? How sustainable are these outcomes and impact, especially once SARI/ EI has ended? Were planned outputs achieved according to plan? Were the outputs/ deliverables of high quality? Were TFs established appropriately and with the right membership? Did the TF’s deliver on their plans related to “software”, “hardware”, and “markets”? Was the program implemented in a cost-effective and resource-efficient manner? Is there evidence of consensus, common guidelines and executable projects on the ground? Main questions Sub questions 13 SARI/EI Phase IV Evaluation | Final Report b. Did the TFs deliver on their plans related to “software”, “hardware”, and “markets”? c. Were the outputs/deliverables of high quality? 2. Was the program implemented in a cost-effective and resource-efficient manner? Impact and Sustainability Impact relates to whether SARI/EI was able to reach its longer-term outcomes and the resulting change in the sector that it envisioned. Sustainability is tied to the impact of the SARI/EI program and assesses whether the impact created will be sustained over time. Key research questions for impact and sustainability include: 1. Is there evidence of consensus, common guidelines and executable projects on the ground? 2. To what extent has SARI/ EI (especially Phase IV) contributed to these outcomes and impact? 3. How sustainable are these outcomes and impact, especially once SARI/ EI has ended? Inputs into the analysis Exhibit 5: Summary of Inputs into the analysis A major part of our analysis will be the survey which we conducted along with our interviews. Each of our interviewees were asked to complete the survey as part of the interview process (either before or after the interview). The survey consisted of 20 questions which were calibrated to test all aspects of the DAC framework. More details on each of the research methods can be found in the annex. The surveys were informed by 48 interviews across all types of stakeholders from government minister to multilateral organizations and private sector actors. Exhibit 6: Summary of 14 SARI/EI Phase IV Evaluation | Final Report Limitations of the evaluation approach 1. Impact is difficult to attribute: While we have provided a systemic approach to track the impact of SARI/EI, we note that direct attribution of SARI/EI’s intervention to improvements in CBET is challenging given that impacts are not felt immediately and there are a multitude of possible drivers which supported the creation of the desired impact. Nevertheless, we have done our best to show the contribution that SARI/EI is likely to have made towards improving regional CBET. 2. Interview list may reflect a selection bias: We have relied on USAID and IRADe to identify many of the individuals which we have interviewed. While we have also interviewed several individuals who are not in the SARI/EI ecosystem, we note that the majority of our interviews are from SARI/EI affiliated individuals and thus their views may be biased. 3. No Baseline for MEL: While MEL indicators were broadly reached, no baselines were provided for those indictors and thus there isn’t a way to compare against historical standards. 15 SARI/EI Phase IV Evaluation | Final Report V. FINDINGS AND CONCLUSIONS Summary of findings: Exhibit 7 – Summary of findings and conclusions Overall, we found that the SARI/EI program was relevant in the South Asian CBET context, was largely effective in achieving its goals, and partially impactful in the ecosystem. SARI/EI’s activities were relevant in terms of the objectives they set out to meet, the third-party role they embodied, and the demand-driven approach they utilized. SARI’s efforts were mostly effective, both in terms of onboarding and capacitating a local partner and setting up three task forces to establish consensus and guidelines. However, these successes did not fully translate into additional CBET infrastructure project development opportunities. Evidence of SARI’s intended impact has been observed in the increased momentum in the ecosystem around CBET, with clear progress towards scaled-up trading that is at least partly due to SARI programming. While progress for CBET in South Asia is clear, direct attribution of impact to SARI is difficult. Nevertheless, most stakeholders noted SARI/E’s historical positive contribution in advancing the regional CBET agenda. Key success factors that explain SARI’s relevance and effectiveness include greater India engagement, well-structured program design, and strong technical expertise and credibility. One of the goals for the SARI/EI program was to increase the Indian government’s engagement with CBET – this has been observed and has spurred engagement with other countries. The program’s design was structured to ensure cost- and time-efficiency and most outputs were produced as expected. These outputs were also largely deemed to be of high quality, underlining the esteem that IRADe’s work carries. IRADe itself has macroeconomic research capabilities and Indian energy experience that were leveraged with positive results. However, programmatic limitations such as insufficient operational capacity, lack of follow￾through and project development capabilities, and uncertainty regarding sustainability were observed. IRADe’s initial struggles to match the required operational capacity for an effective partnership with USAID sapped resources and pushed back timelines. They also did not have a well￾defined follow-through strategy for engagement or project development, limiting both the adoption of recommendations and the implementation of interconnection projects on the ground. The program also created impact that was not necessarily sustainable because capacity was not created at an 16 SARI/EI Phase IV Evaluation | Final Report institutional level, leading to an over-reliance on the commitment and knowledge of key actors in concerned institutions. Relevance: Figure 4: Illustrative understanding of SARI/EI Phase IV's Theory of Change Phase IV’s activities followed a broad Theory of Change outlined above.15 Given that the first three phases of SARI/EI had achieved mindset change around regional energy cooperation, Phase IV had the objective of scaled-up power trading in South Asia. Its mission, therefore, was to enable the transition from Stage I to Stage II of CBET development by formalizing and standardizing the conditions for trade. To this end, activities in data analysis, reporting, and stakeholder engagement on issues of software, hardware, and markets were intended to lead to stakeholder consensus, common guidelines, and implementable projects. The issues of software, hardware, and markets map to the task force structure designed for the program.We will use this understanding of the Theory of Change for Phase IV to anchor our analysis of the program’s relevance, as assessed in terms of its objectives, role, and approach. A summary of our findings on SARI/EI’s relevance can be found below: 15 This exhibit represents our understanding of phase IV’s Theory of Change and what the phase was able to accomplish. While the cooperative agreement did not articulate the program’s objectives and activities in these terms, the exhibit draws heavily from the program’s mandate as laid out in the agreement and the resulting terms of reference. The Theory of Change also draws from findings from stakeholder interviews, discussions, and analysis. Objective Scaled-up bilateral and multilateral cross-border electricity trading in South Asia. Mission Formalize and standardize enabling conditions for cross-border electricity trade and energy markets in the region. Software: Common legal and regulatory frameworks and guidelines established (TASK FORCE 1) Hardware: Interconnection opportunities identified and supported (TASK FORCE 2) Markets: Market rules and platforms established and supported (TASK FORCE 3) Outcomes Consensus among key decision makers to harmonize policies Common guidelines and regulations for CBET Outputs Activities Collect and analyze data: Collect data, research and analyze relevant issues CBET projects identified and increasingly implemented Prepare research outputs: Prepare relevant documents, reports, pilots, templates etc. as needed to build stakeholder awareness, engagement and commitment to CBET Engage stakeholders: Establish three TFs and the PSC, plan and hold meetings, consultations, workshops, conferences, study tours, and capacity building events 17 SARI/EI Phase IV Evaluation | Final Report Exhibit 8 – Summary of SARI/EI’s relevance Relevance of SARI/EI’s objectives: SARI/EI’s objective of developing standards for scaled-up CBET in Phase IV is in line with where the region is on its CBET journey. SARI/E’s success in the first three phases enabled the development of ad-hoc bilateral trading. Recognizing the need to transition to Stage II of CBET development which would see scaled-up bilateral and multilateral trading, SARI/E Phase IV activities focused on setting up common standards and procedures and providing a roadmap for CBET development. This allows new CBET projects to start from a point of mutually agreed-upon standards rather than idiosyncratic negotiations of terms, conditions, and procedures. Government would continue to be supportive of and enable CBET but would not be required to broker every interconnection. The sector’s readiness to transition from Stage I to Stage II was also evidenced by external factors reinforcing that SARI/EI’s interventions were meeting the sector’s needs. For example, the Nepali and Bangladeshi grids have increased in stability over the past few years, paving the way for more CBET connections. Similarly, the national integration of the Indian grid has been providing an example and a roadmap for how the grids of South Asian countries might be connected to the Indian grids given that the regional grids in India are often similar in size, if not larger than the national grids of some South Asian countries. Furthermore, countries are increasingly focused on adding renewable energy to their generation mix, such as India which plans to add 500 GW of renewable energy to its grid by 203016 and Bangladesh which has committed to sourcing 10% of their power from renewable sources by 2040 17 . Leveraging CBET maybe one way to meet these ambitious renewable energy goals as renewable power can be imported from countries like Nepal and Bhutan which have excess hydro capacity. This gradual increase in the use of cross border electricity trading is in line with how other power pools have developed. For example, the Southern Africa Power Pool (SAPP) first developed through a series of multilateral power connections before becoming a unified grid. SARI/EI thus focused on influencing key actors on three themes, each of which was relevant: regulations and policies (software), infrastructure (hardware), and markets. Regulations and infrastructure often exist within each country but are not set up to support CBET. In addition, markets are relevant as the agreed-upon end-goal for CBET – this would allow for flexible trading that responds to variable needs across the region. The development of market-based trading would take 16 Institute for Energy Economics and Financial analysis, “India ups renewable target to 500 GW by 2030”, 2019 17 Dalberg interviews 18 SARI/EI Phase IV Evaluation | Final Report place only after scaling up bilateral and multilateral trading but is relevant in South Asia for two reasons. Firstly, India has an emerging power exchange that provides an opportunity for learning. Secondly, mindset change takes time, as already noted, thus indicating a need for socializing the idea early on. Relevance of SARI/EI’s role: SARI/EI’s role as a third-party convener was necessary given that country level suggestions are often viewed with suspicion. Historically, cooperation between South Asian countries has been limited (e.g.: trade within South Asian countries only amounts to 8% of all South Asia’s trade).18 While individual countries and their policymakers may agree that CBET has a net positive impact, they might still view any CBET proposal involving a neighboring country with suspicion. Having a USAID program act as the convener, publishing economic support for CBET and discussing the creation of common standards is helpful. It allows countries to engage with a neutral but influential and esteemed third party that can act as a trusted broker. In phase IV, this was crucial to engage the right stakeholders in identifying common standards and building consensus. Furthermore, SARI/EI plays a different role from other third-party organizations in the region that have worked to further the CBET agenda. Other organizations working on CBET include bilateral donors like the Millennium Challenge Corporation (MCC), multilateral funders like the Asian Development Bank (ADB) and World Bank, and regional institutions such as SAARC and BIMSTEC. SARI/EI is unlike bilateral and multilateral donors in that the program does not directly invest in projects itself, thus being flexible enough to adapt to the ecosystem and conduct multiple kinds of activities (studies, convening, etc.). It is also unlike regional institutions in that these require buy-in and approval from each constituent member nation to proceed with any activity which can often be a time-consuming and difficult process. In this sense, SARI/EI occupies a unique position in the ecosystem and one that correctly engages with other third-party actors while remaining distinct in terms of its mandate. Relevance of SARI/EI’s approach: Design SARI/EI’s approach, anchored by a programmatic structure that engaged stakeholders at technical and decision-making levels in a consultative process, was appropriate for gathering inputs and buy-in. SARI/EI set up 2 kinds of committees in Phase IV (as a result of the political economy analysis done at the end of phase III): The Project Steering Committee (PSC) and the task forces (TFs). The latter were staffed by largely technical actors and personnel to ensure that the standards would be ready for adoption by the various countries. Additionally, the PSC was set up to achieve high-level political alignment across the region, accurately reflecting the ongoing need to engage with political stakeholders as the role of such political stakeholders transitioned from being the driving force behind a project to being supporters and implementers of the proposed CBET standards. One might note here that politicians themselves were not engaged on the PSC, but it was expected that PSC members would be suitable to engage their respective politicians. Engaging stakeholders in this programmatic governance enabled it to be demand-driven,which was crucial given its larger objectives. Given that Phase IV began in order to help the region transition from Stage I to Stage II of CBET development, it was crucial to bring stakeholders along and generate buy-in for each activity conducted. There was thus an implicit intention for the 18 UN, “International Trade Statistics Yearbook,” 2019. 19 SARI/EI Phase IV Evaluation | Final Report program to function in a demand-driven manner, whereby stakeholders expressed their requirements through the TF and PSC forums which the program took up and responded to as appropriate. The original goal was to have a purely demand-driven operation where suggested activities were entirely sourced from stakeholders. This did not function as intended – the project secretariat (IRADe) suggested activities and solicited inputs on their suggestions from the group of stakeholders. IRADe thus played a much larger role in driving TFs’ strategic direction than intended. Nevertheless, the process remained consultative and ensured that stakeholders were brought along every step of the journey. Governance The cooperative agreement between USAID and IRADe was the key document to assess the program’s results, as it was the basis on which the program was defined. The agreement outlines the required deliverables of the program along with other contractual terms. This was used as the guiding document of the program, to which we will measure the effectiveness and efficiency of the program’s achievements. It laid out the broad structure within which the program was to operate, particularly creating a mandate for the release of Terms of Reference (TOR) for each Task Force. The cooperative agreement broke down activities and outputs into the three pillars of enabling CBET, that were governed through TFs. Each TF had a TOR defining their overall objectives, activities and deliverables, all of which were developed by the secretariat (IRADe, as explained in the box at the end of this section). Similarly, TORs for each TF prioritizing their activities were developed by IRADe, as were the TORs for key activities (i.e., studies). Finally, a set of MEL indicators were established as part of the cooperative agreement. These indicators were proposed by IRADe and agreed upon by USAID. IRADe would report on them to USAID as part of their ongoing accountability to USAID. Task forces’ operations were well-structured as per terms of reference (TORs) which outlined deliverables required of IRADe. The use of TORs to guide the development of outputs was an appropriate strategy for implementing each task force as it set out the mandate for what a TF should accomplish over the duration of phase IV. As such, this was set up to serve as a ready reference for task forces to guide their outputs which took the form of reports and events. These outputs were useful in both publishing reports on common standards and sharing the same with stakeholders, thus working towards the outcomes of identifying guidelines and building consensus. Additionally, Phase IV brought in a new partner with whom USAID had not worked before (IRADe) – having the TORs as a guide helped establish a set of easily definable expectations for IRADe. The overall relevance of SARI/EI’s activities was muted due to the extensive use of narrowly defined TORs. The use of a TF level TORs were effective in creating outputs aligned with the SARI/EI’s objectives but did not offer sufficient incentives for IRADe to actively pursue the program’s true outcomes. For example, the TORs did not include a plan to translate outputs into outcomes such as converting momentum generated by Task force members into new development projects or to push regulators to adopt guideline. SARI/EI did have a MEL plan designed to cover a range of outputs and outcomes—however, IRADe used TOR rather than the MEL metrics (which were used primarily for reporting only) to gauge its success in meeting program requirement meaning that the few outcome indictors in the MEL framework were not important inputs to SARI/EI in its decision-making process. Additionally, the MEL outcomes Exhibit 9- Governance of SARI/EI 20 SARI/EI Phase IV Evaluation | Final Report metrics were too few and too broadly defined to change SARI/EI’s actions19 . For example, the MEL plan mentions targets for MOUs and agreements signed to promote CBET. Itis not clear that these MOUs are tied to the SARI/EI program (or even US government involvement more broadly). Furthermore, the wording does not consider the transition from Stage I to Stage II of CBET development, where government-to-government MOUs are replaced by standards-based agreements between utilities. Thus, while SARI/EI met its outcomes metrics as defined, these did not have the effect of influencing the design or direction of its interventions. Exhibit 10 - Rating of whether IRADe met MEL indicators Phase IV also brought in an Indian partner to co-own the SARI program which was appropriate given focus on building an enabling environment for CBET. An Indian partner was engaged, through a formal competitive process, to partner with USAID through a cooperative agreement (an agreement that treated IRADe as a co-branded partner rather than an implementer in a typical funder- implementer relationship) – this was needed as Indian buy-in was required to move the CBET agenda forward beyond ad-hoc small-scale trading. India is the largest country in the region and the only country that each of the other countries shares a border with (apart from Afghanistan which is not currently active in the CBET discussion). It was thus essential that India participated enthusiastically in the program. Bringing in IRADe, an organization with prominence in the Indian energy sector and the ability to influence Indian actors, helped SARI/EI Phase IV focuses on developing local capabilities on the ground with a stronger understanding of and engagement with local actors. Notably, the first three phases of the SARI/E program had relied on external consultants, primarily from the US, to help change mindsets, especially since the idea of CBET was one that had greater traction globally than in South Asia specifically. However, as Phase IV transitioned to creating an enabling environment for scaled-up bilateral and multilateral trading, it was helpful that the program transitioned from “foreign” consultants to a local counterpart in IRADe. Callout box: IRADe’s role in SARI/EI IRADe is a certified research agency based in India. Based out of New Delhi, Integrated Research and Action for Development [in energy] (“IRADe”) is an independent research 19 Please see the recommendations section for on how the MEL plan can be specifically improved. _________________ “Indian participation was at the lower level while other countries sent higher-level officials… The reason we hired a contractor from the region for Phase IV was that there was a higher chance of getting political support” - Former USAID personnel involved with SARI/E 21 SARI/EI Phase IV Evaluation | Final Report institute that conducts research and policy analysis to engage governments, NGOs and corporation across a broad range of social issues including energy, climate change, urban development, poverty, gender equality, agriculture, and food security. It is certified as a Research and Development organization as well as a Centre of Excellence by the ministries within the Government of India. IRADe was chosen to partner with USAID in a cooperative agreement for phase IV, acting as both project secretariat and implementing agency. IRADe is USAID’s implementing partner for SARI/EI and continues to be so for the extension. It works under a cooperative agreement with USAID, which makes IRADe a co-branded partner of SARI/EI alongside USAID. In its role as USAID’s partner it acts as (1) the implementing agency, providing programmatic oversight of SARI/EI such as in managing the financial accounts, but also as (2) the program secretariat, producing analytical reports and hosting stakeholder workshops. IRADe continues to partner with USAID in Phase IV extension, preserving the program structure but revising roles and responsibilities in tandem with evolving objectives. In Phase IV extension, IRADe will be reappraising its role as program partner, secretariat, and implementing agency. While its role will remain largely unchanged with regards to the program structure, its responsibilities will adapt to reflect the evolution of objectives between Phase IV and the extension. In the extension, the emphasis of IRADe’s role will shift from conducting studies and convening stakeholders to implementation and the development of institutional frameworks. Exhibit 11: Summary of differences between SARI/EI and the Phase IV extension 22 SARI/EI Phase IV Evaluation | Final Report Effectiveness & Efficiency: SARI/EI had both explicit and implicit objectives; the two goals were sometimes found to be at odds with each other. The explicit objective was a continuation of the work from prior SARI/E phases to advance South Asia along the CBET development pathway and create more CBET in the region. The other goal, which was not explicitly stated, was reflected in the prevailing sectoral conditions and USAID’s desire to develop a local Indian organization as the program’s implementing partner. This objective to develop a local partner was present at the time of launching Phase IV and reflected USAID’s internal desire and mandate to help countries along a “journey to self-reliance”. There was also a recognition that while SARI/E managed to garner interest from other countries on CBET, without India’s active interest and participation, there could be no traction on the CBET agenda given its central geographical location, large economy and advance energy grid. Working towards the stated objective while fulfilling the unstated objective required tradeoffs in the effectiveness and efficiency of the program. In this section we evaluate the program’s performance against both the stated and the unstated objectives. Explicit objective: Creating an “enabling” environment The cooperative agreement identified three desired outcomes from the program to achieve its objective of creating an enabling environment for regional energy trade. The program is therefore evaluated on its ability to 1) create consensus amongst key stakeholder, 2) establish guidelines for the long-term operations of CBET in South Asia and 3) identify infrastructure projects development. Exhibit 12 – Overview of the explicit objective Phase IV made strong progress in meeting its core objective but fell short on some key activities, especially on the third outcome of identifying projects and actions for implementation. SARI/EI established a strong case for CBET to key stakeholders in the region and was able to establish guidelines that facilitate market formation through its three task forces. Unfortunately, SARI/EI actions did not result in more actionable infrastructure development opportunities. To further assess the achievement of each of the desired outcomes, we examine the activities and outputs across each of three outcomes outlined. 23 SARI/EI Phase IV Evaluation | Final Report Outcome 1: Create consensus Exhibit 13 – Overview of the objective to create consensus SARI/EI identified key decision-makers to engage but did not create a comprehensive strategy for such engagement. At the outset, SARI/EI identified the ecosystem of actors that needed to be involved as part of its “Management plan”, aiming “to establish a consensus and capacity building exercise among the key decision makers in the electricity sector across the South Asia region.”20 Follow through on the plan is, however, unclear as there was some criticism from interviews in Nepal and Bangladesh that the program did not have the networks to engage the real influencers in the system. While the program did seek to create a “South Asia electricity market forum” as described in the cooperative agreement, there was no comprehensive engagement plan laid out to engage stakeholders over the course of the program. In terms of involving key decision￾makers, SARI/EI succeeded in engaging key stakeholder groups in Nepal, Bangladesh, and Bhutan in the PSC, but has had mixed feedback on engagement. The Project Steering Committee (PSC) was formed as part of Phase IV and sought representation from key Ministries at the secretary level as well as other high-level stakeholders in the ecosystem from all BBIN countries. SARI/EI’s events were attended by a variety of stakeholders in key organizations across the BBIN countries. Nonetheless, many stakeholders interviewed in Nepal and Bangladesh felt that despite hosting several events a year, further targeted momentum is needed. 20 Source: Cooperation agreement Table 1 24 SARI/EI Phase IV Evaluation | Final Report Outcome 2: Establish guidelines Exhibit 14 – Overview of the objective to establish guidelines SARI/EI was able to successfully establish the three task forces to drive the development of guidelines across each theme. The cooperative agreement laid out the plan to establish the task forces, identifying key organizations in the electricity sector to participate in the launch of the program and join the TFs. TF1 and TF2 were launched in July-August of 2013, and TF3 in April 2014. The Task Forces were able to bring in members from the key organizations across all BBIN countries. All three developed and approved a Terms of Reference (TOR) to guide the activities that each would undertake. Exhibit 15 – Summary of accomplishments and outputs delivered under each task force The Task Forces successfully engaged its members and conducted 82% of activities that were laid out in their TORs and produced 89% of the deliverables. The Terms of Reference for the Task Forces laid out a total of 28 activities, with six in Task Force 1, 14 in Task Force 2 and eight in Task Force 3. Task Force 1 completed five out of six activities completely, and partially completed the sixth, Task Force 2 completed 12 of the 14 activities, and Task Force 3 completed all eight. There were 19 deliverables laid out in the TOR, of which 16 were deemed complete. These 16 deliverables correspond to 12 reports published by the Task Forces. The remaining three items were partially completed. 25 SARI/EI Phase IV Evaluation | Final Report Exhibit 16 – Mapping of expected versus enacted roles The meeting of the TOR outputs goals was supposed to be guided by the task force members through a demand driven process, but IRADe ended up with a larger role than expected. The expectation was that the PSC was to be the “apex body that provides overall strategic direction”, and the Task Forces were meant to develop the agendas for the “demand driven studies”. The “demand driven” design of the program would ensure that SARI/EI’s priorities were aligned with those of the sector and the key actors in each of the countries. If the members had committed to the approach, SARI/EI’s outputs would have champions of those who “demanded” them in the first place. However, the PSC and TF roles were more passive than was needed for them to meet their stated goals. This in part reflected how the expectations set up when the committees set up. For example: The PSC met only about once a year as it was assumed that more frequent meetings were unnecessary given their role in providing broad guidance. Unfortunately, in reality, far greater PSC engagement was needed as the infrequent meetings meant that the PSC members were unfamiliar with SARI/EI’s programming and were not able to provide useful and actionable strategic direction. IRADe filled the void by providing much of the strategic direction for SARI/EI. This included setting the agenda for the task forces and the PSC ensuring that the TOR outputs were created on time and were of a high quality. Exhibit 17 – Results from Survey questions on SARI/EI’s outputs The quality of reports and events is perceived to be high by stakeholders. As per the survey conducted for the purpose of this evaluation, both SARI/EI’s reports and events are considered high￾quality by most stakeholders. In particular, over 60% of respondents strongly agreed that SARI/EI’s reports and events were both relevant and insightful. A smaller but still notable majority considered the reports’ methodology rigorous. Similarly, most respondents strongly agreed that events were engaging and well-curated. Only a small minority of respondents felt that reports and events were not of high quality (as measured along these parameters). Thus, SARI/EI went beyond achieving the promised outputs in terms of quantity, delivering on quality as well. 26 SARI/EI Phase IV Evaluation | Final Report Overall, IRADe was able to leverage its core capabilities well, and over the course of the program, demonstrated its ability to adapt and develop additional capabilities. IRADe’s team came in with no prior capabilities related to the topic of cross-border trade or the multidisciplinary skills required for the effective execution of the program. It, however, understood the goals of the program and acted on them, evidenced by its increase in events and activities over later years, its engagement with regional institutions, and its current prioritization of activities in Phase IV extension. It has now brought in stakeholders with experience in and understanding of ecosystem actors, as well as specialized communications capabilities. Outcome 3: Identify projects for development Exhibit 18 – Summary of projects for development While the SARI/EI program recognized the need for implementation activities, it did not develop a clear plan to conduct these activities. The cooperative agreement outlines several approaches and activities for consensus-building activities to bring actors in the ecosystem together “to take next steps” but provides no further guidance on an actionable strategy to ensure next steps are taken. This translated down into the Task Force TORs where there was little mention of project identification or defining action plans. Activities under TF2 include making suggestions on identifying projects without specifying a framework to do so. This lack of a plan was felt on the ground – many stakeholders found that while SARI/EI’s programs were useful, they did not have sufficient mechanisms for follow￾through. While SARI/EI worked to activate the CBET agenda in other regional forums such as SAARC and SAFIR and found positive engagement, it did not gain sufficient buy-in to carry on the work within the aegis of Phase IV (this is now a priority for the extension phase). SARI/EI showed some evidence of providing direct implementation support on certain bilateral issues but was not able to embed itself in execution support for projects. SARI/EI participated in several regional forums, including SAARC, SAFIR, BIMSTEC, and ASEAN, on a variety of topics. However, despite having access to stakeholders in key organizations, SARI/EI was not involved in supporting the implementation of any cross-border projects. Notably, they did not play a defined role in supporting organizations such as MCC or ADB who were engaged in developing cross-border projects between India and Nepal or Bangladesh. _________________ “SARI/EI has done its part by bringing all stakeholders together over the last many years. The respective Governments now need to pursue on the ground with scale-able actions based on the SARI/EI reports and recommendations for a closer and brighter South Asia.” - Task force 2 member 27 SARI/EI Phase IV Evaluation | Final Report Efficiency Exhibit 19 – Annual budget surplus for SARI/EI Exhibit 20 – Breakdown of SARI/EI’s surplus SARI/EI conducted all its activities within budget for each year, ending the program at 9% below budget for the whole period. After facing some teething issues in the first year, SARI/EI managed to consistently remain within budget for the remainder of the program. There were minor variations at a line item level, but all were under 10%, and the program was still below budget overall. The program focused on remaining within the budgetary constraints of USAID and did not push for flexibility to adapt its priorities to changing needs. Impressions from the program secretariat lead to the conclusion that USAID had strong involvement in aligning on the financial budgeting and monitoring processes. Many interviewees indicated that certain compliance policies inhibited the program’s flexibility to respond to demands, and that such policies were often seen as resource-intensive and cumbersome. Unfortunately, removal of such requirements are difficult given that USAID project as tax payer funded. That said, we believe that IRADe will be far more capable in meeting USAID’s requirements during the SARI/EI extension and thus freeing up time for interventions Exhibit 21 – Comparison of planned v. actual timelines The program met its timelines in each of the annual workplans though it fell behind on meeting the initial timelines set out in the cooperative agreement. Based on the high-level timeline laid out in the cooperative agreement, of establishing task forces in eight months, conducting visits and workshops for 40 months, and establishing guidelines and disseminating findings in the final 12 months, SARI/EI met most expectations, but with some delays. The program also completed 85% of its activities laid out in its annual workplans within each year. The Task Force TORs laid out more ambitious timelines that were not met, with TF 1 and TF 2 taking twice as long as anticipated in the initial timelines. It is notable, however, that the scope of work expanded to incorporate additional activities. 28 SARI/EI Phase IV Evaluation | Final Report Implicit objective: Developing a local partner Exhibit 22 – Summary of the Implicit objectives Alongside the goal of supporting CBET development in South Asia, USAID also had an unstated objective in response to its internal priorities. At the time that SARI/EI was set up, USAID was promoting the development of self-reliance 21 in aid recipient countries. In programmatic terms, this meant a pivot towards running SARI/EI with operational and governance structure that had increased local ownership to allow for on-the-ground capacity building. In this vein, SARI had a specific objective to work with, and develop, a local institution and engaged IRADe as part of this approach. Under SARI/EI, USAID and IRADe’s working relationship would be governed by a cooperation agreement22 which provided IRADe with far greater public profile, operational latitude and strategic leadership opportunities. This approach of providing local partners with a greater voice was not driven solely by USAID’s internal guidelines, as it also reflected the stage of CBET regional development at the beginning of Phase IV. Specifically, as the sector transitioned from stage 1 to stage 2, the was a need to evolve interventions from changing mindsets towards implementation. As such, a local partner with experience and connections in South Asia’s power sector was an asset. In particular, there was a recognition that South Asian CBET implementation would be impossible without greater engagement by India, the lynchpin country for any regional CBET plan given its geographical central position. IRADe met both requirement of having Asian energy sector expertise and Indian political clout. This is evidenced in that its leadership had strong and credible relationships with key actors in the Indian energy ecosystem such as in the erstwhile Planning Commission of the Government of India While IRADe was well suited to support the development of regional CBET, it found its strategic role in working with USAID challenging. Under the cooperative agreement, IRADe was required to take a more forward and strategic role, which was different from its traditional responsibilities of preparing research reports. Historically, as a grantee and implementer, IRADe role was output focused with success defined as the ability to efficiency develop high quality products. As discussed 21 USAID policy framework adopted a “Journey to self reliance” approach to its strategies, partnership models and programmatic practices to achieve a time when foreign assistance was no longer needed. 22 A cooperative agreement meant that IRADe and USAID would be co-branded partners of the SARI project rather than USAID being the project developer + funder and IRADe playing the role of an implementing organization. This gave IRADe the ability to speak to outside organizations as a branded partner and the agency to be USAID’s strategic partner. Practically, this meant that IRADe would be folded into the USAID financial infrastructure where their costs were paid on a reimbursement basis rather than as an upfront lump sum payment which would have been common practice for a grant given to an implementing organization. A program design to support transition of responsibilities to a local entity Selection of South Asian entity as a partner Selection of a partner with recognition and influence in the Indian ecosystem of actors Supporting to capability development as required in the partnering agency Program design Partner selection 1. 2. Partner capacity￾building 3. Phase IV was launched under a cooperative agreement, leading to: • Higher visibility and responsibility of the implementing partner • Limited choice of organizations interested in partnering • Limited strategic direction from USAID USAID selected an IRADe as its implementing partner, resulting in: • Recognition in the Indian ecosystem • Potential loss of perception as a neutral broker • Reduced engagement in other countries USAID invested resources to build IRADe’scapacity, resulting in: • Learning and capability development for IRADe • Increased time and resources for ramp-up Implications Unstated objective Phase IV was launched and designed to transition to a local partner who would come forth in a cooperative agreement, was implemented with a South Asian and specifically Indian partner as desired to encourage Indian interest, and was supported by USAID but with some time costs 29 SARI/EI Phase IV Evaluation | Final Report earlier, IRADe was able to achieve this aspect of its role, in part since SARI/EI’s outputs were aligned with its core strengths. But, as USAID’s partner, IRADe initially struggled in moving beyond an implementer/grantee mindset to developing strategies that considered producing higher level outcomes (e.g.: Influencing policy makers rather than creating reports). Furthermore, SARI faced operational and internal governance challenges in implementing the cooperation agreement. The cooperation agreement required stronger financial controls as funds were U.S. Tax Payers fund paid by USAID based on actual cost expensed. Thus, IRADe needed to develop new financial systems and maintain a level of internal bookkeeping that was more stringent than its previous experience as a grantee. As such, at the beginning of the phase IV, IRADe needed to spend significant time and effort to develop these capabilities with support from USAID. When combined with some leadership churn during the first few years of operations , SARI/EI’s operations were slowed in that first year. USAID a hands-on approach in helping IRADe improve its internal systems. For example, staff from USAID’s finance department worked directly with IRADe’s back office to build up their capabilities. Therefore, IRADe was able to reach proficiency in its financial reporting requirements within the first year, limiting the overall effect of its initial struggles. While IRADe initially struggled, it showed growth during phase IV and became an effective partner with support from USAID. Initially, the USAID has a senior team member that acted as an advisor and mentor who supported SARI’s development by providing strategic guidance. This guidance proved helpful in providing strategic thinking for the program, but this senior team member left early in Phase IV and his role not replaced. It was expected that senior members of the IRADe team would provide this strategic directly, but the fact that the program was focused on meeting the TOR requirements and that guidance of the program was supposed to be provide by the PSC mean that IRADE’s senior team members focused on implementation. Nevertheless, IRADe gradually assumed more strategic roles as phase IV progressed, as evidenced by the creation of the think tank forum, an intervention that was not part of the initial TOR and was designed to increase engagement rather than producing specific outputs. 30 SARI/EI Phase IV Evaluation | Final Report Impact & Sustainability: Exhibit 23 – Outline of impact and sustainability SARI/EI succeeded in convening stakeholders and developing appropriate standards for formalizing CBET development but was unable to convert these achievements into adoption of those standards or more rapid CBET infrastructure project development. Nonetheless, SARI/EI’s efforts helped move the region towards a more integrated regional grid, laying the groundwork for future bilateral trading. SARI/EI’s impact against its plan: SARI was broadly impactful in its efforts to formalizes and standardize cross border power trade through publishing relevant and insightful reports. IRADe worked with its tasks forces to publish reports that created a common language and way for utilities and stakeholders to work with each other 23. Starting with the Regional Regulatory Guidelines in Task Force 1, the Harmonization of Grid Codes in Task Force 2, and finally the Market Design Rules in Task Force 3, SARI/EI established guidelines for the prerequisite elements of scaling up cross border power trade in a formalize manner. It also conducted further studies regarding standardizing documentation (e.g.: PPAs, TSAs) and power pricing, amongst others. SARI/EI’s reports in Phase IV were well-regarded by stakeholders across the board and noted for their high quality and analytical rigor. SARI/EI’s key accomplishment in pushing infrastructure development forward was in its study on harmonizing grid codes, which has laid the groundwork for larger-scale connections between national grids. But, SARI/EI’s most notable contribution might have been in facilitating a mock-market exercise that clarified to non-Indian stakeholders how the Indian power exchange works and what the benefits of the same might be. SARI/EI’s ability to create impact in part reflects that its design was relevant to the needs of the sector and thus adherence to the implementation plan created relevant outputs. Furthermore, IRADe, through its strong connections and in leveraging the USAID/SARI brand, brought together a strong group of stakeholders representing the South Asian Power community (many of which were sent by their governments), which provided insights and perspectives needed to create reports that were insightful and relevant for the needs of the sector. 23 IRADe played multiple roles in this process. As the secretariat, IRADe worked to write and then publish the actual reports which included public dissemination and socializing the reports. IRADe also worked closely with the task forces who’s job was to decide the topics of the studies, oversaw the scope of the report, and review findings for accuracy and relevance. 31 SARI/EI Phase IV Evaluation | Final Report Exhibit 24 – Survey results related to SARI/EI’s outputs Exhibit 2524 – Survey Results related to SARI/EI’s influence While SARI/EI was able to generate outputs that were seen as insightful, limited progress was made in the adoption of guidelines or the influencing of policy limiting overall impact. Over 60% of survey respondent felt that outputs created by SARI/EI were insightful, but this did not directly translate to being actionable or influential towards key decision markers. As a result, despite helping to build awareness and knowledge on CBET, few respondents felt that SARI/EI had contributed meaningfully to the recent change in India’s CBET policy. This may reflect a feeling that SARI/E was successful in socializing the idea of CBET and its benefits through technical stakeholders who had sound industry and sector-specific knowledge, however, they were unable to reach political actors – ministers or key bureaucrats that work within influential ministries, for example. This points to a need for IRADe to increase political engagement, especially in light of the fact that IRADe ability to influence government was a primary reason for USAID to work with IRADe on SARI/EI. This higher level of political engagement is required because these are the actors that balance the needs of the entire trading relationship among countries and so are crucial decision-makers to convince. The importance of political will is underscored by the fact that stakeholders surveyed identified political will as the foremost barrier to CBET in South Asia today. Exhibit 26 – Most pressing challenges holding back the CBET agenda in South Asia This limited ability to reach high level political actors reflected a number of issues with SARI/EI’s interventions including: (1) emerging perception, among some stakeholders, of an India focus and/ or bias, and (2) limited engagement in other countries. With Phase IV being led by an Indian entity, SARI/EI in part lost some of its identity as a neutral broker limiting its ability to influence stakeholders in non-Indian countries. This was heightened by the fact that IRADe lacked experience and networks in non-Indian countries. As such, engagement with stakeholders outside of India was limited vis-à-vis SARI/EI’s interaction with Indian counterparts. 24 Note: Strongly agree is defined as the sum of the top 2 scores out of 6 on the survey (i.e. the sum of individuals who chose 5 and 6 for each question on the survey) 32 SARI/EI Phase IV Evaluation | Final Report The lack of on-the-ground presence also limited the ability for SARI/EI to meaningfully follow up with non-political stakeholders after they had received SARI/EI’s reports, restricting potential trickle-down effect on the broader ecosystem. As a result, engagement with local investors and funders, both from the multilateral and the private sector, was limited. For impact to meaningfully increase, SARI/EIwill need to transition from preparing macro level reports for the region into solving local issues. Reflecting this lower level of regional interaction, brand recognition of SARI/EI fell during the fourth phase – external stakeholders who were influential in national power ecosystems (e.g. Bangladesh) were unaware of SARI/EI’s activities or outputs during the phase. Communications more broadly was found to be lacking – there was limited interaction with civil society, sub-national governments, or the general public while interaction with national governments was not necessarily action-oriented. Nevertheless, SARI/EI was effective in laying the groundwork for the scaling up of CBET across the region. The presence of common standards provides the groundwork for creating further impact in the immediate future when these can be adopted and implemented. Furthermore, the convening of people in the development of these standards created opportunities for cross-border exchange of knowledge and building of trust and understanding among neighbors. This is again essential groundwork for further cooperation.While many stakeholders felt that SARI/EI helped to support the development of CBET, they felt that the next step in driving greater impact would be moving from reports and common standards to on-the-ground actionable projects. SARI/EI’s overall impact on South Asia’s journey to CBET: Stakeholders have observed an increase in the momentum towards an integrated regional grid in South Asia. In particular, there is more discussion today around trilateral or multilateral electricity trading within a legal framework that allows for this. There is also acknowledgement among stakeholders that the new connections that will be developed in the region will more likely be synchronized AC connections rather than high-voltage DC connections.25 This is especially the case for India-Bangladesh trade. The consensus around multilateral trade and the building of AC connections have not yet resulted in additional connections yet but signal the maturity of the individual grids and a greater comfort with CBET as a necessary option for electricity provision. Callout box: This momentum has built up at a varied pace in each country in the region The differences in countries’ progression towards market-based trading can be explained by the fact that each country began the phase in 2012 with its own unique electricity profile both domestically and in relation to CBET. Furthermore, the priorities for each country, given their 25 In South Asia, historical CBET connections were made through high voltage direct current lines. This created a physical separation between the national grids as the alternating current (AC) from the first grid had to be translated to direct current (DC) to be transmitted, and then converted back to AC to be used. This created additional infrastructure cost to build the lines and higher operating costs. A more efficient way to connect the grids would be with a wide area synchronous grid (also know as an Interconnection) where national grids are directly connected using AC lines. Interconnections allows power to flow freely between two grids, pooling power resources (and resulting in associated benefits, such as improved reliability). To date, South Asian countries have resisted creating CBET through an interconnection as it requires the two grids to be synchronized as the name suggests (i.e.: The grids must match voltage, phase sequence, phase angle, frequency and waveform) and national utilities did not have the technical capabilities to be confident that they could maintain such synchronization. Furthermore, the physical separation created by the DC line provide a backstop to stop the flow of power. Politically, such a separation was important as power within a synchronized grid would flow according to physical properties rather than political direction. _________________ “SARI is a well-known brand, but has been doing less these days” – USAID stakeholder in Nepal 33 SARI/EI Phase IV Evaluation | Final Report positions in the region, were different. In summary, the following progress in the power section and in CBET was observed in each country: 1. Bangladesh: Bangladesh’s electricity sector is extremely dynamic and has evolved in a few different ways. Generation capacity has increased to a point where the country now has surplus capacity. Transmission capacity has not kept pace and is now being opened up to private investment. Bangladesh has increased its targets for electricity it sources through CBET over the duration of Phase IV and is working on expanding the existing links with India. Bangladesh’s domestic grid has advanced to the point where it is seeking to connect with India through a synchronized connection rather than a HVDC line. Furthermore, it is eager to source renewable energy from Bhutan and Nepal to meet its renewable power targets. 2. Bhutan: Bhutan continues to see export of power to the India as a key way to earn revenues and drive economic development. Bhutan’s grid is tied to the Indian market given that it is was developed with Indian exports in mind. As such, it is the only market which has synchronized its grid with India. Power from Bhutan is sold through a standard PPA where excess power is sold to India. Bhutan is exploring the possibility of selling power to India through a market mechanism rather than a simple PPA but a lack of technical capacity continues to hinder this development. 3. India: India has moved from only considering bilateral trading to allowing multilateral trading where it plays the role of a “go-between” or in-between partner. In the meanwhile, it has integrated its 5 regional grids into a national grid through HVDC lines and seen the growth in power trading through market mechanisms. 4. Nepal: Nepal has seen a significant interest in the development of large-scale hydropower generation projects in the last few years, from local and international developers. It is also in the process of setting up an electricity regulatory commission, as well as splitting up its national utility (Nepal Electricity Authority) into three entities. Stakeholders appear to demonstrate positive sentiment over increasing trade of hydropower with India, as well as Bangladesh in the near future. 5. Sri Lanka: There has been limited CBET development in Sri Lanka as connections to India’s southern grid would require long (possible submerged) HVDC lines which are prohibitively expensive. Sri Lanka’s domestic power grid has improved, and they are seeking a direct synchronized connection with India to save money and expedite the time needed to make a connection. While discussions are ongoing, no progress was made during the fourth phase. 6. Pakistan and Afghanistan: While Pakistan and Afghanistan had representatives on the task forces and PSC, geopolitical concerns and India’s focus on the BIMSTEC region suggests that no progress has been made to further CBET in South Asia specifically in these countries. 34 SARI/EI Phase IV Evaluation | Final Report It is difficult to attribute systemic outcomes directly to SARI/EI as the initiative was one among a several notable efforts to boost CBET in the region. Impact is not directly attributable to SARI/EI because the ecosystem is complex with multiple actors. For instance, stakeholders surveyed provided mixed assessments of SARI/EI’s contribution to shifting India’s policy. Nepali stakeholders in particular felt that SARI/EI’s role in this was limited – this might be because there was a separate effort from high-level Nepali officials and eminent persons to change the Indian regulations. Alongside SARI/EI, other regional initiatives and institutions such as SASEC (of ADB), a group of eminent persons from India and Nepal, and the CERC in India provided recommendations on revising India’s draft regulations from 2016. There are also government-level groups such as the Joint Working Group of power ministry officials from South Asian countries that SARI/EI has not engaged with. It is noteworthy that representatives from the same institutions (government, bilateral, multilateral, academic) participate in many of these efforts, which further makes it hard to draw out SARI/EI’s specific contribution to the broader changes in the ecosystem. Sustainability of SARI/EI’s impact: Given the nature of the ecosystem in which SARI operates and the design of the program, the sustainability of phase IV’s impact is limited. Phase IV was designed to engage stakeholders in a consultative process to design and adopt guidelines for scaled-up trading. As such, interventions were primarily analytical in nature (and largely consisted of studies), targeting ecosystem-level concerns. While these issues required resolution, they are dynamic in nature – particularly because the ecosystem is dynamic. As actors in key positions across countries rotate and shift in and out of those positions and as countries’ domestic power sectors evolve and new challenges emerge. In this sense, a program that is designed to create knowledge with the assumption that this knowledge will be owned and leveraged by stakeholders because it is demand-driven, is set up to not be entirely sustainable. Ongoing updating of these standards will be needed to maintain their relevance. The sustainability of SARI’s impact was held back in part by the dynamic nature of the roles in the various public sector entities; this issue required more sustained engagement. Stakeholders report that people in critical positions in the electricity sector rotate frequently. While the mock market exercise done by Task￾Force 3 was successful. Key stakeholders suggested that it would have to be repeated for this reason. In such a situation, having artefacts such as reports in addition to events and workshops is critical as these can be recirculated among new stakeholders and revisited as such. However, doing so requires a strong dissemination strategy which SARI/EI lacked despite having the right setup for it. Therefore, sustained impact requires constant and repeated interactions. SARI/EIwas designed to end with Phase IV in 2018, but was extended for a few more years, creating an opportunity for a more concerted effort towards ensuring sustainability. SARI made efforts to extend their influence beyond the scheduled end of the program. These included preparing policy briefs that summarized takeaways from key studies. However, an extension was deemed necessary to enable the program to move further along and prepare better for a sustainable exit. In this sense, SARI/EI and the fourth phase was not very sustainable in and of itself, but the extension offers an opportunity for building sustainability of impact over the next few years. To that end, SARI/EI has laid the groundwork for sustain their interventions through future institutions such as the South Asian _________________ “There are limits to SARI’s ability to influence because the resources that SARI has are technical assistance resources which get you a seat at the table but do not get you to lead the charge.” - External observer formerly with SARI/E _________________ “Utilities were not set up for CBET – they don’t even have a separate cell to look into CBET – therefore, any wisdom and knowledge is specific to individuals and so the organizational or institutional knowledge is lost when these individuals are moved out.” - External observer in Bangladesh 35 SARI/EI Phase IV Evaluation | Final Report Forum of Transmission Utility (SAFTU) and South Asia Forum of System Operators (SAFSO) that will bring together utilities and system operators. 36 SARI/EI Phase IV Evaluation | Final Report VI. RECOMMENDATIONS We are at a critical juncture, both for CBET in South Asia and for SARI/E. We believe that SARI/E’s interventions must be maintained as the CBET development journey is far from complete. 26 As described above, the building blocks for a flourishing CBET market in South Asia are in place. But, as the region as-a-whole transitions from Stage I to Stage II in its CBET journey27, the onus is now to formalize and concretize the initial momentum (e.g. moving from white papers and workshops to actual regulations, protocols, and standards), and demonstrating real and tangible progress through increased bilateral (and eventually multilateral) power trade projects and agreements in the region. Exhibit 27: Recommendations for progressing South Asia along its CBET development journey The report makes two sets of recommendations. 1. The first covers SARI/EI extension which begins immediately and last until 2022. These recommendations lay out how SARI/EI, and IRADe as the implementing organization, can increase its impact and support South Asia during its transition from Stage I to Stage II of the CBET development journey. The recommendations for SARI/EI during the extension are shaped by two factors: (1) IRADe’s evolution to being a strong and competent program partner and (2) the transition of SARI/EI to be included within South Asia EDGE, a new USAID umbrella initiative that covers all US government energy programs in South Asia. 2. The second lays out guiding principles for how SARI/EI should operate beyond the extension: While South Asia will be in the midst of transitioning from Stage I to Stage II of the CBET development journey during the SARI/EI extension, we believe that South Asia will be firmly within Stage II by 2022. The recommendations for SARI/EI’s actions after 2022 thus reflects guiding principles for how to support South Asia as it scales up CBET. 26 In the event that this is not possible, USAID must prepare to transition the outputs of SARI/E to external institutions. Recommendations for this situation can be found in the annexure. 27 For clarity, we see the CBET sector to be transitioning from stage 1 to stage 2 of the CBET journey. Thus, during SARI/EI interventions related to both stage 1 and stage 2 were addressed. Thus, during this next few years the focus is on supporting this gradual transition and to set up the sector for success once it enters stage 2 fully. Stage I: PROVIDING BENEFITS Stage II: FORMALIZING AND SCALING UP Stage III: COMMERCIALIZING (MARKETS) No CBET: Stakeholders are not aware of CBET opportunities, focusing instead on national concerns Initial bilateral trading: Countries begin to trade through small￾scale connections in response to emergent electricity needs Scaled-up bilateral trading: Countries develop large-scale bilateral trading projects Multilateral trading: Multiple countries engage in trade through long￾term purchase agreements (OPTIONAL) Market-based trading: Public and private actors across the electricity value chain trade through power exchanges Operating procedures Standards based trading procedures Markets based trading procedures adopted Mindsets of technical engineers are changed Mindsets of key politicians are changed Benefits of CBET are main￾streamed National energy policies amended to included CBET Capacity for trading Political commitment Common operating standards establish￾ed CBET operating standards adopted by countries Common markets based guidelines developed and agreed upon International markets based operating standards adopted International regulatory bodies established Technical alignment Common technical standards establish￾ed CBET technical standards adopted by countries Technical capabilities and equipment Civil society agreement (land, enviro concerns) Financing CB Trans lines developed broadly Continued CB transmission lines developed Capacity in the system is developed to allow for standards based trading. Stakeholders which may need capacity development may include utilities, government officials, regulators and funders Systematic capacity developed to allow for markets based trading Political commitment to trade power including developing policies friendly to CBET and to enter into multilateral agreements Political commitment to cede a small portion of national sovereignty to allow for the development of markets based trading Trading utility to utility (public or private) through PPA, wheeling agreements and other agreed upon procedures Trading G2G via MOU or similar document Trading utility to utility automatically via market mechanism (PPA agreements may still exist) National regulatory bodies established to enforce standards SARI interventions Recommendations Physical infrastructure development 37 SARI/EI Phase IV Evaluation | Final Report As SARI/EI’s intervention transitions from Stage I to Stage II of South Asia’s CBET journey, overall success of the program will be increasingly tied to the level of USAID’s commitment. With sufficient commitment of USAID’s time and resources, SARI/E has the capability of being a high impact program as it will be building from a strong foundation developed over the past 20+ years. But, SARI/E’s success is likely to be quite binary especially when compared with other USAID programs, which may be able to attain moderate/ partial success. Unlike other programs where technical capabilities and funding may be the bottleneck to greater impact, in SARI/E it is its political influence which dictates whether it is ultimately successful. Thus, for SARI/E to be fruitful, USAID, as SARI/E’s most important political influencer, will need to be actively involved. Activities during Phase IV extension: Practically speaking, given that SARI/EI Phase IV extension has already been operational for a year at the time of the writing of this report, IRADe should be engaged to develop impactful interventions that would be immediately implementable within their current structure rather than trying to enact radical changes. SARI/EI is currently operating well, as has been reflected in the earlier review of SARI/EI’s activities. Furthermore, certain initiatives such as the issuances of RFPs to establish a number of new regional bodies have already been launched at the time of writing this report. Recognizing this, it is important to build on the strengths of these existing programs rather than to redesign the SARI/EI initiative. Furthermore, the extension will provide time to set up longer term SARI/E programming that will begin after 2022 during which time we would expect that the sector would have completed its transition to stage 2 of the CBET journey. Updates to programmatic design We recommend SARI/EI focusses on three priorities during the extension. (1) Amplify its voice through structured and strengthened communications and dissemination across key stakeholder groups. To date, insufficient dissemination of key outputs and sporadic engagement with important stakeholders have limited the overall impact of SARI/E’s work. Key actions to amplify SARI/E’s voice including leveraging new engagement channels and strategies to enhanced engagement with top politicians, key influencers such as think tanks, bureaucrats to increase the long-term institutional knowledge in the government and funders. (2) Invest deeply in training and capacity building. Capacity building was not a focus during phase four and as a result, the volume of training provided by SARI/EI was considerably lower when compared with previous phases. Given that CBET will be materially different from current energy transactions, further training to develop capacity within the sector is necessary for encouraging the use of CBET lest the sector builds out infrastructure and develops operating standards which are then not utilized. This would happen if individuals within utilities don’t know to leverage this new CBET systems. Key actions could include comprehensive training focusing on a mix of technical training for using the CBET systems as well broader training aimed to educating the sector on the evolving landscape. (3) Shift focus to supporting infrastructure development. SARI/EI should provide support to translate success on establishing standards into on-the-ground implementation of CBET projects (including infrastructure development). SARI/E can play a role in supporting this implementation by identifying, tracking, prioritizing and then sharing infrastructure development opportunities. In addition, SARI/EI’s existing interventions, namely the task forces, should be maintained to continue the current support for SARI/EI. This ongoing support is needed to ensure that the impact created by SARI/EI is not short-lived given the rapid rate at which the CBET sector is developing. For example, standards will need to be updated to reflect changes to the on-the-ground realities, new 38 SARI/EI Phase IV Evaluation | Final Report politicians will need to be informed when ministers’ portfolios change, and training will need to be refreshed when new technologies and processes are adopted. Given the dynamic nature of the sector, the task forces will be needed to continue their work into the near future. (1) Amplify its voice through structured and strengthened communications and dissemination across key stakeholder groups: SARI/EI’s impact on the ecosystem is limited by an unfocused approach to communications. Feedback from interviews suggests that SARI/EI developed insightful and useful analysis, but that this analysis was nevertheless not translated to new policies or regulations. This limited SARI/EI’s impact as the influence of its products ended at affecting mindsets and was not translated to tangible changes to the broader ecosystem. Furthermore, as shown in exhibit 28 below, the lack of a strategic communications plans meant that several influential stakeholders were insufficiently engaged, further limiting the ability for SARI/EI to affect ecosystem change. This lack of deeper engagement was particularly noteworthy as the stakeholders which lacked engagement were often the most influential to overall CBET development, such as high-level politicians. Finally, as SARI/E transitions towards scaling up regional infrastructure development, broader engagement with the general public, which has been extremely limited so far, is needed to develop a general understanding of the benefits of CBET. While the general public remains low on its influence on actual CBET, the public can have a disruptive effect on project development through NIMBY-ism or related concerns. A thoughtful approach on preempting the public’s concerns, such as by more strongly leveraging institutions through the think-tank forum, will ensure that new infrastructure projects are developed without issue. Exhibit 28: Mapping of select stakeholders28 (indicative only) As such, during Phase IV extension, we recommend that SARI/EI develop a strategic communications plan with the following goals: 1) to increase engagement with influential stakeholders to which SARI/EI has previously lacked engagement, 2) to deepen relationship with stakeholders which are both highly influential and already highly engaged with SARI/EI to encourage tangible changes in policies and 3) to develop a strategy to build broad-based support for CBET in the general public. This approach prioritizes engagement proportionately with the level of influence29 of 28 Indicative rating based on interview feedback and Dalberg analysis. Not inclusive of all stakeholders. 29 Influence is defined as the ability of the stakeholder helping to push the CBET journey forward 39 SARI/EI Phase IV Evaluation | Final Report specific stakeholders. We note that while SARI/EI should aim to engage with all stakeholders, SARI/EI should consider expanding partnerships (e.g. within the think-tank forum) with other sectoral organizations (e.g.: SAARC) to leverage the individual relationships and connections of different organizations. In this sense, this above analysis should only represent a starting point for the discussion on strategic communications as a strategic review of the operating landscape may allow for splitting of the responsibility for outreach amongst key institutions/ organizations. L O W E N G A G E M E N T , H I G H I N F L U E N C E S T A K E H O L D E R S : Low engagement, high influence stakeholders are lost opportunities for creating impact in the sector. SARI/EI should aim to increase engagement with these stakeholders as the return on greater engagement, as measured by impact on that stakeholder group, is likely to be high as they have limited exposure to SARI/EI’s messages possibly resulting in a knowledge gap that is may have resulted in ineffective or ill-informed decision making. SARI/EI should do an internal review to understand whether the lack of engagement was a result of misplaced focus or a lack of capabilities within the communications function. This will help to guide future engagement plans. In any case, as engagement has been low to date, strategies for engaging these stakeholders should focus on early, frequent and light touch interactions to allow SARI/EI to build rapport and develop a stronger relationship. Specific strategies for deepen engagement with key low engagement, high influence stakeholder groups are listed below: Top-level political actors: As South Asian countries often have hierarchical political decision￾making structures; it is important to engage with top level government decision-makers who can directly impact the development of CBET. Increasingly this means going beyond the usual channels of the Ministry of Energy/ Power and looking for champions with common interest whom have access to and can influence the highest level of the political establishment. This may include influential ministers (e.g.: foreign affairs, environment etc.), high level individuals within industrial companies or directly with the prime minister’s office (PMO). In particular, within the BBIN countries, we have observed that the PMO may be the most import executive authority on the development of CBET given the importance of power to economic development and the need to manage geopolitical factors when considered CBET decision. Historically, SARI/EI’s government interactions have not reached these upper level politicians limiting SARI/EI’s outreach effectiveness as messages meant for them can get muddled as they are passed up the chain of decision makers. To directly reach the top-level political actors, SARI/EI needs to leverage USAID’s strong network more deliberately, and make the outreach of key political actors purposefully so that SARI/EI can leverage the collective networks of the TFs and PSC. Furthermore, they should tweak messaging aimed at these individual to highlight the issues that resonates with them most, including 1) the geopolitical benefits of CBET and 2) immediate economic benefits. Influencers: Think-tanks, academics, and regional institutions influence the long-term development of CBET through their ability to convene stakeholders, provide sustained support across a number of years and conduct targeted research. While SARI/EI has worked with such influencers, for example through the think-tank forum, we recommend that these engagements be scaled up materially as the current level of engagement is low when compared to the benefits of closer coordination. In particular, there is insufficient communication between the various influencers on the strategic directions for the sector. This lack of coordination means influencers (think tanks, institutions etc.) are likely working on similar intervention resulting an inefficient use of sector-wide resources as the various interventions are unlikely to be synergistic and may in fact be overlapping/ redundant. SARI/EI should leverage a strong communications function to direct the sector beyond convening politicians and decision makers, but also in guiding the influencers. 40 SARI/EI Phase IV Evaluation | Final Report Leveraging USAID’s strong brand will help to provide legitimacy to SARI/EI’s action when speaking with these influencers. Bureaucrats in key roles: Engagement with bureaucrats at both the national and sub-national level for each country30 is sometimes overlooked as they are not as high-profile as politicians and often not seen as the key decision-makers. Nevertheless, they are highly influential to the long￾term development of CBET as they remain within the government even as politicians are voted in/out or change portfolios. In a similar to vein to the famous parable of the tortoise and the hare – whereby bureaucrats are akin to the tortoise and politicians are hares -influencing bureaucrats, while time-consuming and slow-moving, allows for the creation of a deep cache of knowledge within governments which eventually informs policy. Thus, the slow development of bureaucratic institutional knowledge may outweigh the role of politicians who may be able to create quick results, but momentum is lost when they invariably leave for different roles. Funders: Investment in CBET is difficult as investment risks are high, there are few benchmark transactions and ticket sizes are very large. As such, the number of funders with the necessary capacity to make investment is limited (e.g.: primarily limited to government bodies and multilateral organizations such as the ADB). SARI/EI has conducted research on investment opportunities in CBET and should continue to conduct such research and to actively communicate the results of such studies with investors (including lenders) to demystify the difficulty in making such investments. H I G H E N G A G E M E N T , H I G H I N F L U E N C E S T A K E H O L D E R S : Phase IV extension must leverage the engagement it has with the high-influence stakeholders to refine its focus on relevant activities and further its impact in the ecosystem. SARI/EI has actively engaged several stakeholders through its past phases, which has gained the program significant recognition. It must capitalize on its existing relationships to get a deeper understanding of the latest developments pertinent to CBET and seek to enhance their engagement to eventually become champions of SARI/EI’s efforts. Utility leadership: Key public utilities in the region have provided support through their membership in the Task Forces, and through its capacity building platforms. As such, SARI/EI is already actively engaged with utilities but should aim to deepen their relationship with a focus on encouraging utilities to adopt the standards that SARI/EI developed. This can be accomplished through a long-term engagement strategy where key members of the utility leadership are engaged by different SARI/EI stakeholders. Leveraging the relationships of PSC and TF members whom have existing relationship with utility leadership will be an important way to influence them. Deepening the influence on utilities can also be accomplished by being more proactively to bring in the utilities’ issues into SARI/EI’s TF conversations. This may result in SARI/EI’s outputs being more relevant to the utilities’ needs and increase the likelihood that its recommendations are adopted. Tailored efforts to address the needs of the utilities within each country will increase SARI/EI’s influence within them. Regulatory leadership: Similar to utility leadership discussed above, SARI/EI has engaged with the various regulatory bodies in the BBIN countries through the PSC and TFs and need to deepen these relationships to encourage the regulatory agencies to adopt the standards created by SARI/EI. Recommendations broadly mirror those for the utilities. 30 In this context we understand politicians to be the decision makers are they are elected officials. Top politicians often lead ministries such as the ministry of energy. On the other hand, bureaucrats are part of the government’s administrative body and generally don’t change based on elections given that they are the functional managers and are generally not elected. Bureaucrats serve politicians and may serve multiple generations of politicians which are more transitory. 41 SARI/EI Phase IV Evaluation | Final Report L O W I N F L U E N C E S T A K E H O L D E R S : Engagement with low influence stakeholders can be assigned fewer resources but must be maintained. The general public as well as the technical-level bureaucrats may not drive the agenda forward but can become roadblocks if they lack an understanding of CBET. For example, if the general public is uninformed, they may block development of CBET construction project or be unsupportive of governments with a CBET agenda. Therefore, SARI/EI should continue communications with these stakeholders through mass media formats such as television advertisements. Recognizing that mass marketing may be expensive and outside of the purview of SARI/EI’s mandate, a focused communications plan can instead provide content and support for politicians making public statements and/or mass marketing campaigns of their own. Callout box: SARI/E has built brand recognition amongst the SA Energy community; its brand should be maintained SARI/E has been working in South Asia on topics related to cross border energy trade for more than 20 years. Over this time, SARI/E has been successful in engaging with key stakeholders throughout the region and has helped to shape mindsets and build capacity of regional leaders. While the SARI/E brand is seen as an “American” brand, it is well known and respected throughout the South Asia energy sector. As SARI/E transitions to be a program under South Asia EDGE, USAID should make efforts to maintain the SARI/E brand as it will bring legitimacy to South Asia EDGE’s CBET interventions and show a sense of continuity with USAID’s work over the past 2 decades. (2) Invest deeply in training and capacity building: As the sector transitions from Stage I to Stage II, capacity-building will be increasingly critical to ensure that investments made by SARI/E in creating common operating standards and in infrastructure development are effectively used to trade power. Especially while standards are still nascent and cross border energy trade is a relatively new concept (reflecting that the sector is still transitioning from Stage I to Stage II), power sector professional will need to be constantly reminded on the benefit of CBET given the typical inertia individuals have to change. Additionally, technical people need to be trained on the functional activities needed to trade power within an organized framework (as per stage 2 of the CBET development journey) given that the common operating standards and technology might be new to them. Thus, during Phase 4 extension, SARI/EI should provide training that covers the technical requirements of CBET and also general knowledge on CBET to build general awareness and knowledge among civil society. The sector needs ongoing general skills development to create broader understanding of CBET. This is to build up general capacity throughout the sector to change mindsets and equip stakeholders along the CBET journey. To ensure that the training is relevant, the training should be tailored to the needs of each country. Furthermore, the training should be provided consistently and repetitively given that there is often turnover amongst stakeholders and thus periodic refreshers are necessary. To build out the training, SARI/EI should take stock of the existing capabilities across each country and identify priority areas to develop programs in line with their CBET-readiness. Currently, India is ahead of the curve in basic abilities and would benefit from learnings of best practices from other power pools. At the other end, Nepal may require significant support as it unbundles its national utility and builds up the ERC. 42 SARI/EI Phase IV Evaluation | Final Report The sector needs specialized training for core stakeholders such as utilities and regulators to build technical capacity. Without trained professionals, utilities, regulators and other power sector stakeholders would be unable to leverage the standards created by SARI/EI or the new physical infrastructure being constructed to trade power. While SARI/EI has already done some of this training during phase 4 (e.g.: through the mock exercises) further specialized training is needed to directly support the technical specialist in the power sector. Further, we believe providing technical training is an area where SARI/EI’s interventions can be additional as our review of the sector showed that there are few organizations that can offer this type of training. A further breakdown of the type of training needed by key stakeholder groups is show below: Politicians: As a key decision-maker, politicians must continually hone their understanding of the benefits of CBET, especially to develop long-term energy policies. They should be equipped to know what long term policies are needed to create a permissive environment for CBET. Regulatory agencies: As regulations are introduced and adopted, institutions such as regulatory and technical authorities need to build capacity to understand how to use and implement these standards effectively. They also need to be supported in developing an understanding of foreign regulations. Utilities: Management and technical officials at utilities need to be trained on applying standards in and the effect that such rule application will have on the technical quality of the power being traded. Funders: Private funders should be supported by so that they better understand how to manage investment risk of a CBET project. The goal of such trainings must be to ensure that trained stakeholders can become agents of change in their own organizations, further training their colleagues and other staff. (3) Shift focus to supporting infrastructure development: SARI/EI can identify a pipeline of potential projects. While SARI/EI lacks private sector or infrastructure project development expertise it can still support infrastructure development by leveraging its convening power, regional perspective, strong brand, and network of connections to collect and collate a pipeline of promising CBET project ideas. Most private sector players would be unwilling to develop and maintain such a database due to the uncertainty that useful projects will crystalize. Furthermore, if they did create such a database, they would be unwilling to share their intellectual property with the broader sector limiting its usefulness. SARI/EI being a neutral third party with long term interest in supporting the CBET development would be ideal to host this database. By developing and then sharing the pipeline with prospective developers, investors and construction firms, SARI/EI would highlight unknown, or under researched, CBET business and investment opportunities available in South Asia, catalyzing interest into the sector. As part of the process of creating this pipeline of opportunities, IRADe can leverage engagement with the infrastructure development community to build its internal capabilities with the goal of directly supporting the infrastructure development process once these capabilities are developed 31 . 31 We have provided recommendations ways to leverage these infrastructure development capabilities in a call out box in the section on activities beyond 2022. 43 SARI/EI Phase IV Evaluation | Final Report Update to operational structure: In order to achieve the goal of scaled-up bilateral trading and initial multilateral trading by 2022, SARI/EI will need to adopt four changes to its operational structure: Increased strategic guidance and inputs: During Phase IV, IRADe adopted a demand-driven approach to guide its strategic direction. This means that the members of the task forces collectively decided what topics to cover in their research and analysis. Acknowledging that the process was not perfect, and that IRADe did provide significant strategic guidance to the task forces, we note the IRADe nevertheless was not intentional in provide a strategic direction on what interventions were needed. This (semi) demand-driven approach was appropriate for Phase IV given that many of its interventions were focused on getting broad consensus on what was needed for the region. But as the sector moves towards implementation, IRADe will need to adopt a more intentional strategy as certain actions are needed to progress the sector forward (e.g.: to being more intentional in its communications or in choosing specific projects to help develop). To support this more intentional direction, IRADe will need to have a strong management office focused on understanding SARI/EI position/role within the sector and how SARI/EI should be positioned to meet sectoral needs. Recognizing that IRADe has not historically take the role of providing strategic direction, USAID should provide support to this management office. We recommend that the USAID’s SARI/EI team be provided with an additional manpower support so that the existing experienced staff can take a more hands-on approach in guiding SARI/EI. The expectation would be that USAID would be more active in helping IRADe prioritize its interventions while also leveraging its connections to catalyze and support specific interventions. Outcomes-focused governance structure: In Phase IV, IRADe was governed using a terms of reference (TOR)-driven approach, which sensible as it was the first time that IRADe, an organization with limited project delivery experience, was partnering with USAID. In fact, USAID was required to work closely with IRADe’s back office to bring their operations to an internationally acceptable standard and the TOR helped to keep IRADe on track – resulting in Phase IV being largely effective in delivering on promised outputs. As IRADe will continue as the USAID’s partner during the Phase IV extension, the governance structure should transition to one which is more focused on creating outcomes rather than ensuring that outputs (e.g.: reports/ analysis) are produced. This approach will allow IRADe flexibility in how it achieves it goals and ensures that IRADe’s and USAID’s goals are aligned. We recommend that USAID adopt an expanded monitoring and evaluation (MEL) approach to working with SARI/EI whereby IRADe’s has both outcome and output focused objectives. High level recommendations on using MEL as a management tool While IRADe already has a MEL plan, it is under-utilized as it is used as a reporting rather than as a management tool. Applying MEL as a management tool would require a more thoughtful setting of MEL indicators and then tracking IRADe’s performance against these indicators. Given the importance of such a management tool in aligning USAID and IRADe’s goals, IRADe should also be provided with training on the use and application of MEL. This will allow IRADe to better understand how the MEL governance structure will keep it accountable and empower IRADe to apply similar tactics to govern the task forces and sub￾contracts which it overseas. 44 SARI/EI Phase IV Evaluation | Final Report Exhibit 29: Overview of the types of indicators that should be tracked as part of an MEL plan 1. Be adaptable: IRADe has shown that it can be an effective problem solver (e.g.: creating the think tank forum to solve the lack of local engagement) when provided with discrete outcomes to achieve, but without specific instructions on how to reach that objective. To enable this problem-solving approach, USAID should focus on tracking whether IRADe help to achieve the desired outcomes rather than tracking whether specific outputs were attained. 2. Track “end point” outcomes: In order to properly guide IRADe, USAID will need to define success for the program so that IRADe has a clear understanding of its objectives. This requires setting indicator targets that, if reached, represents a sectoral change that USAID wanted to see. For SARI/EI, a suitable “end point” outcome may be the total volume of power traded in South Asia, combined with a second indicator that tracks that increase in the trade of power between each of the South Asian countries. This would provide a view of whether total power traded was increasing and whether that increase was focused on only a few country-to-country relationships. It is prudent to be careful when choosing the indicator that will define success for SARI/EI as poorly defined indicators may create skewed incentives. For example, the current set of SARI/EI indicators in the existing MEL plan covers the “quantity of cross border energy trade capacity”. Since this indicator focuses on capacity, it may encourage IRADe to support the development of new CBET lines even when there is a high likelihood that the lines are seldom used, wasting valuable resources. While the indicators “quantity of cross border energy trade capacity” is itself not poorly defined, it must be paired with other indicators such as “quantity of cross boarder energy traded” to ensure that unintended consequences are not inadvertently supported. 3. Track the progress along the impact journey: While tracking the “end point” outcome with a series of thoughtful indicators is useful for guiding IRADe, it is insufficient as a management tool as it does not allow for an understanding of how or why the outcomes were or were not achieved. For MEL indicators to be useful, the steps which lead to SARI/EI being successful should also be tracked. These outcomes often line up with the 45 SARI/EI Phase IV Evaluation | Final Report objectives of key intervention as they should be working towards the achievement of the “end point” outcome. In SARI/EI, indicative examples of such outcome indicators may include (1) the percentage of South Asian countries which have adopted common guidelines for the international trading of power and (2) the average time needed to complete a CBET project32.We would recommend that these outcome indicators reflect steps in the SARI/EI’s theory of change. Each vertical in the theory of change should be reflected in these outcome indicators as the theory of change. Empowering the PSC: In Phase IV, the project steering committee (PSC) achieved two complimentary goals. Firstly, they were an important source of information on the needs of the CBET sector from on-the-ground makers which informed SARI/EI’s work. Secondly, they served to develop and build capacity amongst its members. In this sense, the PSC was successful. But, a third use of the PSC was unsuccessful, and this was to use the PSC for disseminating SARI/EI’s findings and championing them in their respective national ecosystems. This lack of success in leveraging the PSC for communications reflected that they were used in a reactive manner – responding to questions presented to them by IRADe, rather than taking a more proactive approach. We recommend that IRADe push the PSC to adopt a more proactive stance to working with the SARI/E’s senior management to determine SARI/E’s strategic direction and increasing impact by leveraging the relationships its members possess. Given that PSC members are often influential government officials, it will be important for IRADe to adopt a flexible approach to increasing engagement with the PSC members, whom may have limited availability to meet in person. More frequent discussions outside of the regular PSC meetings may help in this regard as would application of technologies (e.g.: Video conferencing) to allow for inputs from PSC members without the need for travel/ in person meetings. Localizing the agenda: SARI/EI’s reports and workshop primarily focused on macro analysis reflecting the regional nature of CBET’s benefits. This focus was useful, when SARI/EI was tasked with created common standards that were expected to be adopted regionally. As SARI/E moves into Stage II where there is a greater focus on implementation and physical infrastructure development, the unit of analysis must localize as implementation occurs on-the-ground. For each country, we would recommend that SARI/E map the state of the power sector and the critical path to CBET-readiness, as well as map all the stakeholders and decision-makers involved. This scan should result in: • Workplan: A plan of action for addressing gaps in the ecosystem and to prioritize activities where SARI/E has a role to play. SARI/E should aim to develop a workplan for each country. • Engagement plan: The map of key decision-makers and influencers in the country the strategy to engage each entity. This should be closely tied to the overall strategic communications plan but providing a level of localized nuance such as by identifying the specific individuals in the national ecosystem which are important for engagement. • Resource plan: Based on the requirements identified for each country, SARI/E should determine the level of local resources required to meet country level work plans. Improving SARI/EI’s capacity: To facilitate the above recommendations, IRADe will needs to develop new capabilities. Developing these capabilities will involve training IRADe’s staff, some will require that SARI/EI bring on additional people with unique capabilities while others require SARI/E to partner with skilled organizations. In 32 Could be estimated by asking / surveying a cross cutting series of participants their expectations of the time needed to develop a specific existing cross border energy line if that process were to start today. 46 SARI/EI Phase IV Evaluation | Final Report any case, IRADe will need to be intentional in developing these capabilities as successful implementation of the above recommendations will involve skillsets that IRADe may not currently possess. Similar to its work in developing IRADe’s back office function, USAID should actively support the development of these skillsets as a long-term investment in the SARI/EI program and also for increasing capacity in the South Asian energy sector -- as part of USAID’s support for the region’s “journey to self-reliance”. How and which capabilities are developed should reflect IRADe’s current capacities. As such, the first step in building out SARI/EI’s capabilities is to complete an assessment of IRADe’s capabilities when compared with the level of proficiency needed to achieve the target objectives. Gaps identified in this process should be addressed as required. We have made the below recommendations for meeting our initials scan of perceived gaps but recognize that the below is neither a comprehensive list of the skills needed nor reflective of IRADe’s capabilities and should only represent a starting point for discussing capacity development. Training individuals: Individuals should be trained when the skills needed can be easily developed in a cost effective and timely manner. Such skills are less technical in nature and represent an approach to working that will improve IRADe’s activities. We believe that the below skills can be developed through a combination of short courses (either in person or online) and mentoring/ guidance from USAID. (1) Senior stakeholder engagement: For the Phase IV extension, SARI/EI has hired a new communications specialist. Given the importance of stakeholder engagement in creating a permissive environment for CBET development, we recommend that SARI/EI provide the specialist with training on influencing and engaging with high-level politicians. A public relations consultancy may be well suited for this training, which may also be attended by other IRADe or PSC member whom will be interact with senior stakeholders. (2) MEL basics: To allow IRADe’s staff to understanding how they will be held accountable to MEL metrics and for applying these concepts to improve their interactions with their own sub-contractors, IRADe’s staff should be provided training on the basics of MEL. Hire people: Where training individuals would be prohibitively expensive or difficult within the timeframe desired, it may be prudent to leverage expertise from external hires. Such individuals can be brought onboard as full-time staff or as external contractors, depending on SARI/EI’s immediate needs. The advantage of external hires is the ability to rapidly build up SARI/EI’s capacity, an important consideration given that the Phase IV extension is expected to be concluded by 2022. Partner with people: Where SARI/E doesn’t have a comparative advantage, it can be more effective to meeting its objectives by partnering with other organizations. (1) Training: SARI/E has historically partnered with USEA to provide training and should continue to do so rather than spending time to develop these capabilities in house. Where possible SARI/E should partner with local institutes, such as universities and think tanks to create training partnerships. This will help in logistics and coordination, but also help in socialize the CBET agenda. In terms of the training itself, developing training with local parties will help to bring a local flavor to the SARI/E’s making it more relevant and immediately applicable. (2) Working with multilateral regional organization: As a CBET leader and a recognized brand, SARI/EI should work to define the CBET agenda amongst the multilateral regional 47 SARI/EI Phase IV Evaluation | Final Report organizations. SARI/EI should aim to play an agenda setting role by publicly publishing it strategy and convening other organization to discuss how they can work cooperatively. This will help to prevent overlap in regional interventions. Furthermore, there may be some areas where regional organisation (e.g. BIMSTEC, SAARC) may have unique perspectives or connections, partnering with them may be helpful in catalyzing SARI/EI’s plan. (3) Working with the bilateral missions: As SARI/E looks to localize its interventions, SARI should leverage relationship with the in-country bilateral missions. While the current process for working with the bilateral missions is effective, a closer relationship with the bilateral missions can bring additional localized knowledge and expertise into the SARI/EI decision making process. Recognizing that the energy officers with the bilateral missions are stretch thin, SARI/EI should consider directly funding bilateral mission resources to create SARI/EI local coordinators33 to ensure that they have an on-the-ground resource to be their “eyes and ears” on local energy sector development. This is especially helpful as IRADe’s local presence in non-Indian markets is limited. Bilateral mission can be engaged to strengthen support to SARI/EI in the following (non￾exhaustive) ways34: • Local knowledge: Missions can inform SARI/E’s understanding on how to develop local projects. Specifically, it can help in understanding local laws (e.g. land rights, licensing, etc.) • Local connections: USAID’s local missions will have the ear of the governments, both at the national and sub-national levels (where relevant) and have local connections. These relationships will be helpful in supporting SARI/E’s communications effort. • Pipeline development: Missions, with an on-the-ground presence, will have a feel for emerging project development opportunities. • Capacity building: Many missions will have capacity building programs which SARI/E can leverage. For example, connections with local trainers and access to training spaces/ equipment. 33 Such funding may come from SARI/EI’s budget surplus. 34 This does not mean that SARI/EI did not engage with the bilateral missions in the ways presented. Rather, this list shows a series of different ways that bilateral missions could be engaged to improve the effectiveness of SARI/EI. For those bilateral missions that are already engaging in these actions, SARI/E should consider deepen integration and providing additional support to build on this strength. In areas where SARI/E has not yet engaged, this list provides way to being stronger cooperation. 48 SARI/EI Phase IV Evaluation | Final Report Activities beyond 2022: We believe that by 2022, the sector will have transitioned from Stage I to Stage II of the CBET journey and thus SARI/E’s focus should similarly recalibrate. In this environment, SARI/E should be supporting the scaling up of cross-border power transmission which requires a sharper focus on physical infrastructure development. Further, communications, training and capacity building activities should move beyond supporting the individual towards developing robust and sustainable institutional capabilities. SARI/E will need to consider what organizational structure will best support its activities under South Asia EDGE. In the extension phase up till 2022, SARI/EI will continue to function through the PSC and TFs. This is fitting given that the focus in this phase is to build incrementally on the strengths that have been established through SARI/EI. However, moving beyond 2022, these structures might need modification to better suit the needs of the ecosystem at the time. This includes re-considering the establishment of the cooperative agreement with IRADe in 2022 once the extension phase is concluded. We note that beyond 2022, SARI/E’s activities and organizational structure may need to differ materially from what is suggested in this section to reflect changes in internal priorities or sectoral outlook. Thus, this comments in this should be considered as a set of thought starters on how SARI/E may progress rather than a static plan. Therefore, USAID must ensure that their future plans for SARI/E are sufficiently flexible to adapt to changes, as required. Guiding thoughts for programmatic developments beyond 2022 1. SARI/E should consider tweaking its communications strategy to increasing engagement with the private sector and government bureaucrats and begin to deprioritize engagement with top politicians. If the sector develops as we envision and South Asia moves into Stage II of the CBET journey, CBET infrastructure development should increasingly be driven by private sector companies and guided by standardized rules, regulations and guidelines. This would mean that endorsements from top politicians would become less important in ensuring the success of CBET projects. We note that the above recommendations does not imply that top politicians are no longer important stakeholder and should be ignored, rather that under Stage II of the CBET journey: communications strategies should place greater relative emphasis on the private sector (the stakeholders doing the infrastructure development) and government bureaucrats (the stakeholders defining the standardize rules that guide the private sector). Furthermore, CBET development, which is tied to general energy sector development, will not be at the same level all countries. Thus while, a shift in the communications strategy for one country may raise the importance of the private sectors, top politicians may remain the most important stakeholders for countries which CBET sector development is slightly slower. The communications plan for any country must reflect its local needs, but a general view of increasing engagement with the private sector and government bureaucrats should be considered when the communications plan is refreshed in 2022. 2. Emerging from the interventions begun during the SARI/EI (and recommended above), SARI/E should transition from building the capacity in individuals to ensuring that there is strong institutional capacity to support CBET in the South Asian energy sector. 49 SARI/EI Phase IV Evaluation | Final Report a. Convert (and support) individuals trained by SARI/EI to be champions in important CBET related institutions and develop CBET related institutional capacity organically: A key part of our recommendations during the extension of SARI/EI is to increase training and capacity building for individuals throughoutthe South Asia energy sector. We believe that SARI/EI will be successful in training a significant number of technicians, bureaucrats, energy professional and government officials on the benefits of CBET and the technical skills needed to successfully trade power across international borders. After 2022, with a critical mass of trained individuals throughout the South Asian energy sector, SARI should set up programs that would enable these trained individuals to further disseminate their knowledge into the organization in which they are work. This would have the effect of converting individual knowledge into institutional knowledge. Examples of such programing may include capacity building for the trained individuals on how to conduct informal internal training for their peers. SARI/E may provide materials to equip this peer-to￾peer training. Alternately, SARI/E can provide training and materials to internal training teams that want to offer CBET related training to their staff. b. Develop technical courses to embed CBET in the broader learning agenda for technical energy experts. At the moment, mastery of CBET related technical skills in South Asia is rare. SARI/E can institutionalize the learning of CBET technical skills by supporting the formal inclusion of CBET related knowledge into energy courses at technical colleges/ universities. It can do this in a few ways: developing online courses that are easily accessible, designing or providing inputs into a university-level course at leading technical institutes in each of the focus countries, and/or providing certification to students who pass these courses. 3. SARI/E can support the development of the CBET infrastructure projects identified during the Phase IV extension.Whereas the previous focus was on sharing knowledge of prospective projects to catalyze the sector, SARI/EI’s longer term focus should be on directly supporting the development of CBET infrastructure projects by identifying bottlenecks in the development process and then providing direct support to project developers to address those bottlenecks. Further details in the callout box below. 50 SARI/EI Phase IV Evaluation | Final Report Callout box: Tactically, how can USAID SARI/E support the development of CBET infrastructure projects? As the support provided to each of the projects will be multifaceted, SARI/EI will need to develop a project support function that has a broad set of project development skills. This team will need to be staffed with individuals with experience in developing and financing power projects. Given that such skill set may be difficult to find, it may be prudent to engage with external consultants or hire external staff to meet any lack of technical expertise. An example of the type of support which SARI/EI can provide is shown below: 1) SARI/EI can support investors by conducting prefeasibility studies and providing appraisals of CBET projects. Investors require significant support to understand opportunities, given that investment ticket sizes are large and that there are few comparable projects to benchmark risk. SARI/E can help to mitigate some of this risk by conduct prefeasibility studies on project development opportunities. SARI/EI can also support an investor’s evaluation process by providing researched estimates on key investment assumptions such as the economics of transmission links, their power flows, and the working of markets. 2) SARI/EI can coordinate with governments to help international project developers navigate local requirements. Developing transmission interconnections will involve permissions from local governments and potentially require national-level or sub-national￾level approvals. Given SARI/EI’s familiarity with the ecosystem, it can support project developers in understanding the nuances in working with local governments and how to avoid legal roadblock. 3) SARI/EI can convene vendors such as EPC companies for project development. SARI/EI, and USAID, have a network of local entities with the capacity to offer products or services required in the development of the cross-border interconnections. SARI/EI can connect project developer with reputable local vendors helping to source the right partners. SARI/EI will need to develop new capabilities in order to properly support infrastructure projects. A summary of the key capacities to be developed are below: 1) Energy sector project development expertise: SARI/EI will need to create a dedicated team of energy sector project specialists to support regional CBET infrastructure projects. The team members will need energy sector project development experience to ensure that they can speak with regional CBET project developers as a specialist. This type of experience is difficult to train as the advice provided would be related to practical problem-solving experience. As such, SARI/EI will likely need to bring in this expertise from external sources. 2) Financial investment/ funder engagement expertise: SARI/EI’s support of project development will require discussions with funders such as commercial banks and investment funds. The ability to speak with these financial professionals thoughtfully will be important to helping these funders gain an understanding of the risk related to financing CBET projects and to mitigate those risks. It would be unlikely that SARI/EI’s staff will have the financial expertise to support external investors and thus SARI/EI will likely need to bring in this expertise from external sources. 3) Deep local knowledge: Project development teams are often locally based to ensure that they have a pulse on the local market. Furthermore, funders need to understand localized risk when making investment/loans. Therefore, for SARI/EI to be successful in supporting project development they would need to develop on-the-ground support teams in key markets, such as Nepal and Bangladesh. Notably, this goes beyond having a SARI/EI 51 SARI/EI Phase IV Evaluation | Final Report coordinator in bilateral missions as the project development team will need some of the technical and financial skills details in point 1 and 2 above. 52 SARI/EI Phase IV Evaluation | Final Report VII. ANNEXURE Stakeholder Survey Design and execution of the survey Dalberg designed and administered a short survey as part of the data-gathering exercise for this evaluation using Google forms. The survey had two aims: a) To gather evidence from a diverse range of stakeholders on a common set of quantifiable indicators and metrics across the five parameters of the DAC framework b) To generate a baseline using indicators that a future evaluation measures for the sake of comparison Keeping best practices in mind, the survey comprised questions that were either multiple-choice or scale-based in nature. It was created and first shared with stakeholders on the 13th of August, 2019. The form was closed on the 3rd of September, 2019. The survey was administered to present and past members of the TFs and the PSC, and circulated among external stakeholders who had interacted with SARI phase IV or were considered to be influential in the South Asian power ecosystem. A summary of key findings from the survey can be found below, followed by a list of the questions. 53 SARI/EI Phase IV Evaluation | Final Report Summary of findings Please find below some of the key findings that emerged from our analysis of the data from the survey. Background information Figure 5: Where are you based? (n = 31) Figure 6: Type of organization (n = 31) Figure 7: How have you engaged with SARI/EI between 2012 and 2018? (n = 31) CBET in South Asia Figure 8: (Q1) Which of the following is the most pressing challenge holding back the CBET agenda in South Asia? (n= 31) Figure 9: (Q2) Which of the following activities will provide the greatest boost to the CBET agenda? (select top 3) (n = 31) 1 7 13 4 2 4 India Bangladesh Nepal Bhutan Sri Lanka Other 10 2 4 1 1 13 Government Ministry or Agency Regulatory Authority Multilateral /bilateral Agency Utility Trading company Others 5 3 2 3 18 PSC TF1 TF2 TF3 External 12 10 4 4 1 Lack of harmonization in policy, legal and regulatory frameworks across countries Lack of transmission systems interconntection across countries Lack of an elecrticity trading market between countries Lack of political will / differences in geopolitical priorities Other 32% 32% 19% 19% 16% 13% 13% 10% 3% Creating forums for discussion and interactions between stakeholders across and within countries Developing guidelines for common market rules for power exchange Capacity building within public and private institutions Developing strong regional institutions for the purpose of energy trading Developing analyses to identify investment opportunities for CBET in South Asia Getting funding commitments from actors for infrastructure development Drafting guidelines for policy and regulation Anchoring the CBET agenda in existing regional institutions Building awareness of the benefits of CBET among key decision￾makers More evidence or analysis into CBET opportunities, benefits, and challenges in South Asia 0% Phase 4 interventions Other possible interventions 54 SARI/EI Phase IV Evaluation | Final Report Figure 10: (Q3) Which are the organisations that have been the most influential in supporting CBET in South Asia (select top 3) (n = 31) SARI/EI’s program effectiveness Figure 11: (Q4) To what extent do you agree or disagree with the following statements? (n = 31) Figure 12: (Q7) To what extent have SARI/EI's events contributed to your awareness and knowledge about CBET, and its benefits / opportunities in South Asia? (n = 31) SARI/EI National governments ADB World SAARC BIMSTEC Other Bank SASEC Think tanks 13% SAFIR None - market forces AIIB 3% 77% 65% 39% 29% 23% 16% 13% 10% 3% 0% 6% 3% 6% 6% 3% 6% 45% 32% 48% 29% 35% 39% 42% 61% 42% 58% 55% 48% 6% 3% 3% 6% 6% 6% 0% 4b. SARI/EI’s reports have relevant, additional, and insightful content 4a. SARI/EI’s reports are rigorous in their methodology 4c. The findings in SARI/EI’s reports are actionable 0% 0% 0% 4d. The content presented in SARI/EI’s events are relevant and insightful 0% 4e. SARI/EI’s events are engaging and well-curated 4f. SARI/EI’s events result in actionable insights Strongly Disagree Somewhat Disagree Somewhat Agree Strongly Agree Not applicable 1 1 2 8 11 8 1 2 3 4 5 6 1 = No contribution 6 = High contribution 55 SARI/EI Phase IV Evaluation | Final Report Figure 13: (Q8) To what extent do you agree or disagree with the following statements? (n = 31) Figure 14: (Q8a) To what extent do you agree or disagree with the following statements? (disaggregated by type of interaction with SARI/EI) – SARI/EI’s reports have had a significant influence on key stakeholders (policymakers, regulatory authorities, utilities, academic experts, funders, etc.) (n = 31) Figure 15: (Q8b) To what extent do you agree or disagree with the following statements? (disaggregated by type of interaction with SARI/EI) – SARI/EI’s events have had a significant influence on key stakeholders (policymakers, regulatory authorities, utilities, academic experts, funders, etc.) (n = 31) 3% 16% 10% 48% 58% 32% 29% 8a. SARI/EI’s reports have had a significant influence on key stakeholders (policymakers, regulatory authorities, utilities, academic experts, funders, etc.) 8b. SARI/EI’s events have had a significant influence on key stakeholders (policymakers, regulatory authorities, academic experts, utilities, funders, etc.) 0% 0% 3% Not applicable Strongly Disagree Somewhat Disagree Somewhat Agree Strongly Agree 1 1 3 2 1 2 1 2 1 3 10 4 Not applicable Strongly Disagree Somewhat Strongly Agree Disagree Somewhat Agree PSC TF1 TF2 TF3 External 1 1 3 2 1 1 1 1 2 1 1 12 4 Not applicable Strongly Disagree Somewhat Somewhat Agree Disagree Strongly Agree PSC TF1 TF2 TF3 External 56 SARI/EI Phase IV Evaluation | Final Report SARI/EI’s impact Figure 16: (Q10) How would you rate SARI/EI's overall influence on the ecosystem for developing CBET in South Asia? (n = 31) Figure 17: (Q11) How would you rate SARI/EI’s overall influence on policymakers for promoting CBET in South Asia? (n = 31) Figure 18: (Q12) To what extent did SARI/EI contribute to change in India's CBET policy? (n = 27) 1 2 18 8 2 1 2 3 4 5 6 0 1 = No contribution 6 = High contribution 1 1 1 18 9 1 1 2 3 4 5 6 1 = No contribution 6 = High contribution 1 5 6 12 3 1 2 3 4 5 6 0 1 = No contribution 6 = High contribution 57 SARI/EI Phase IV Evaluation | Final Report Figure 19: (Q12) To what extent did SARI/EI contribute to change in India's CBET policy? (disaggregated by country) (n = 27) Figure 20: (Q14) Do regulators, transmission and system operators, and private actors in each of the concerned countries have the required technical capabilities to implement CBET? (n = 31) General Feedback Figure 21: (Q15) How likely are you to recommend a SARI/EI report or event to colleagues and other stakeholders? (n = 31) 1 1 3 1 1 3 3 4 1 2 1 1 1 1 2 1 1 2 3 4 5 6 1 = No contribution 6 = High contribution India Other Bangladesh Sri Lanka Bhutan Nepal 35% 58% 6% Yes No Don’t know 26% 45% 29% 3 4 0% 2 0% 1 5 1 = Highly unlikely 5 = Highly likely 58 SARI/EI Phase IV Evaluation | Final Report Questionnaire 59 SARI/EI Phase IV Evaluation | Final Report 60 SARI/EI Phase IV Evaluation | Final Report 61 SARI/EI Phase IV Evaluation | Final Report 62 SARI/EI Phase IV Evaluation | Final Report 63 SARI/EI Phase IV Evaluation | Final Report