PERFORMANCE EVALUATION OF THE USAID–DEPARTMENT OF ENERGY INTERAGENCY AGREEMENT JANUARY 2020 This publication was produced at the request of the United States Agency for International Development for the E3 Analytics and Evaluation Project. It was prepared independently by Dr. Carolyn Fonseca and Isaac Morrison of Management Systems International, A Tetra Tech Company. PERFORMANCE EVALUATION OF THE USAID–DOE INTERAGENCY AGREEMENT | ii PERFORMANCE EVALUATION OF THE USAID–DEPARTMENT OF ENERGY INTERAGENCY AGREEMENT January 31, 2020 Contracted under AID-OAA-M-13-00017 E3 Analytics and Evaluation Project Cover Photo: National Renewable Energy Laboratory (NREL) solar panel. Credit: NREL. DISCLAIMER The author’s views expressed in this publication do not necessarily reflect the views of the United States Agency for International Development or the United States Government. PERFORMANCE EVALUATION OF THE USAID–DOE INTERAGENCY AGREEMENT | iii ACKNOWLEDGEMENTS The evaluation team thanks Dr. Andrea Watson and her team at NREL who generously provided their time and supported our efforts in this evaluation. We are grateful to NREL for hosting us in Colorado for data collection meetings and for making available so many of its staff to participate in this evaluation. We also thank Dr. Jen Leisch of USAID/E3/GCC for her guidance on the evaluation questions, helping to identify data sources, and assisting in coordinating interviews with Agency leadership. Additionally, we would like to give special recognition to Kate Faulhaber, the E3/GCC activity manager for this evaluation, for all her guidance and support throughout the evaluation and for ensuring our evaluation had access to the necessary documents, data, and individuals to carry out this effort. Finally, we thank the stakeholders involved in this partnership from around the globe who responded to our request to share their views and experiences. These stakeholders include representatives from USAID missions, DOE, host-country governments, industry, academia/research centers, implementing partners, and NGOs. Finally, the team is grateful to USAID leadership in the E3 Bureau, including Kathryn Stratos and Colin Green of E3/GCC as well as Dr. Bhavani Pathak, the USAID contracting officer’s representative for the E3 Analytics and Evaluation Project, for this opportunity to help the IAA access important data to guide future collaborative activities between USAID and DOE-NREL. PERFORMANCE EVALUATION OF THE USAID–DOE INTERAGENCY AGREEMENT | iv ABSTRACT USAID and the U.S. Department of Energy have held an interagency agreement (IAA) since 2009 that provides technical support to Agency missions and their in-country partners to achieve low-emissions goals, access clean power, and increase self-resilience in the energy sector. Under the IAA, the Department of Energy’s National Renewable Energy Laboratory (NREL) also creates and supports technical platforms and delivers technical assistance to help stakeholders achieve these goals. This performance evaluation identified areas of prioritization and opportunities to improve collaboration for the new five-year IAA between USAID and NREL. The evaluation focused on three questions to identify whether the IAA met missions’ needs, understand usage of IAA technical platforms, and capture strengths and weaknesses in the partnership. The evaluation used a mixed-methods approach including 274 surveys, 89 interviews, 2 focus groups, 1 process-mapping workshop, and a document review with a range of IAA stakeholders. Findings suggested the IAA provided cutting-edge technical support to missions and host-country governments, and that use of technical platforms is somewhat limited but users vary within sectors and organization types. It is likely there are training spillover effects, because survey respondents reported also using IAA technical platforms on which they had not received training. Barriers to use of technical platforms include bandwidth issues, users’ software and hardware limitations, competing platforms, and user capacity. Recommendations to USAID and NREL include revising IAA partner roles, creating a theory of change reflective of current work with a robust monitoring, evaluation, and learning strategy, and identifying new methods to capture diverse IAA benefits. PERFORMANCE EVALUATION OF THE USAID–DOE INTERAGENCY AGREEMENT | v CONTENTS ACKNOWLEDGEMENTS iii ABSTRACT iv CONTENTS v LIST OF TABLES vii LIST OF FIGURES vii ACRONYMS viii EXECUTIVE SUMMARY ix INTRODUCTION 1 IAA BACKGROUND 1 EVALUATION PURPOSE 3 EVALUATION QUESTIONS 3 METHODS AND LIMITATIONS 4 EVALUATION DESIGN 4 SAMPLE AND DATA COLLECTION 4 DATA ANALYSIS 6 TEAM COMPOSITION 6 LIMITATIONS 6 FINDINGS AND CONCLUSIONS 7 FINDINGS FOR EVALUATION QUESTION 1 7 CONCLUSIONS FOR EVALUATION QUESTION 1 10 FINDINGS FOR EVALUATION QUESTION 2 11 CONCLUSIONS FOR EVALUATION QUESTION 2 17 FINDINGS FOR EVALUATION QUESTION 3 18 CONCLUSIONS FOR EVALUATION QUESTION 3 20 RECOMMENDATIONS 22 EVALUATION QUESTION 1 – MEETING NEEDS AND IMPROVING BUY-IN 22 EVALUATION QUESTION 2 – TECHNICAL PLATFORM USE 22 EVALUATION QUESTION 3 – STRATEGIC PLANNING, COORDINATION, COMMUNICATION, AND M&E 23 PERFORMANCE EVALUATION OF THE USAID–DOE INTERAGENCY AGREEMENT | vi ANNEX A: EVALUATION STATEMENT OF WORK 25 ANNEX B: SUPPLEMENTARY INFORMATION ON EVALUATION METHODS, SAMPLES, AND LIMITATIONS 47 KEY INFORMANT INTERVIEWS 48 FOCUS GROUP DISCUSSIONS 51 SURVEY 52 PROCESS MAPPING WORKSHOP 60 DOCUMENT REVIEW 60 ANNEX C: SUMMARY FINDINGS BY EVALUATION QUESTION AND AREAS OF INTEREST 62 ANNEX D: EVALUATION INSTRUMENTS 66 KII INTERVIEW PROTOCOL: USAID/WASHINGTON 66 KII INTERVIEW PROTOCOL: USAID MISSIONS 71 KII INTERVIEW PROTOCOL: NREL CORE 76 KII INTERVIEW PROTOCOL: NREL TECH 81 KII INTERVIEW PROTOCOL: MISSION PARTNERS 86 KII INTERVIEW PROTOCOL: TOOL USERS 91 KII INTERVIEW PROTOCOL: IN-COUNTRY IMPLEMENTING PARTNERS 94 SURVEY NARRATIVE AND QUESTIONS 99 FGD PROTOCOL: NREL GROUP 1 106 FGD PROTOCOL: NREL GROUP 2 109 PROCESS MAPPING WORKSHOP PROTOCOL 113 ANNEX E: EVALUATION TEAM BIOS 117 PERFORMANCE EVALUATION OF THE USAID–DOE INTERAGENCY AGREEMENT | vii LIST OF TABLES Table 1: IAA Funding by Operating Unit, through Modification 10 1 Table 2: Sample by Data Source 5 Table 3: Survey Respondents Reporting Tool Use 11 Table 4: Number of Survey Respondent Tool Users by Stated Tool Used and Country 12 Table 5: Survey Responses about Tool Use and Training Attendance 14 Table 6: Data Sources by Eq and AOI 47 Table 7: Interviewees by Type and Data Collection Round 49 Table 8: Trainings Included in the Survey Sample 53 Table 9: Summary Findings by EQ and AOI 62 LIST OF FIGURES Figure 1: Distribution of the IAA Modification 10 Budget by Task 2 Figure 2: Number of Tool Users by Organization Type (n=153) 12 Figure 3: Number of Tool Users by Work Sector and Tool Type 13 Figure 4: Number of Tool Users by Work Sector and Single- or Multi-Tool Use 13 Figure 5: Survey Respondents by Purpose of Tool Use 14 Figure 6: Survey Responses to "What was your favorite feature(s) of the tool/platform?" 16 PERFORMANCE EVALUATION OF THE USAID–DOE INTERAGENCY AGREEMENT | viii ACRONYMS AAAS American Association for the Advancement of Science AOI Area of Interest AOR Agreement Officer’s Representative COR Contracting Officer’s Representative DEC Development Experience Clearinghouse (USAID) DIA Development Impact Assessment DOE Department of Energy DPV Distributed Photovoltaic E3 Bureau for Economic Growth, Education, and Environment (USAID) EC-LEDS Enhancing Capacity for Low-Emission Development Strategies EQ Evaluation Question FGD Focus Group Discussion GCC Office of Global Climate Change (USAID/E3) GtG Greening the Grid IAA Interagency Agreement IP Implementing Partner I-JEDI International Jobs and Economic Development Impacts KII Key Informant Interview MEL Monitoring Evaluation and Learning M&E Monitoring and Evaluation MSI Management Systems International NGO Nongovernmental Organization NREL National Renewable Energy Laboratory PV Photovoltaic RDMA Regional Development Mission for Asia RED-E Renewable Energy Data Explorer TA Technical Assistance USAID United States Agency for International Development PERFORMANCE EVALUATION OF THE USAID–DOE INTERAGENCY AGREEMENT | ix EXECUTIVE SUMMARY EVALUATION BACKGROUND AND PURPOSE In 2009, the United States Agency for International Development (USAID) signed an interagency agreement (IAA) with United States Department of Energy’s (DOE’s) Office of Policy and International Affairs and the National Renewable Energy Laboratory (NREL) to ensure that the best U.S. technical expertise is utilized to support global efforts to combat climate change. The IAA draws from a network of national energy lab staff to provide technical assistance to USAID missions and partner countries in the analysis, design, and implementation of clean energy greenhouse gas mitigation initiatives. The IAA designs technical platforms, tools, and trainings to assist key stakeholders in developing renewable energy resources by targeting specific technical or analytical barriers. USAID’s Office of Global Climate Change in the Bureau for Economic Growth, Education, and Environment commissioned this performance evaluation to inform future IAA work plans and strengthen its overall strategic management. EVALUATION METHODS AND LIMITATIONS The evaluation used a mixed-methods approach. From March 2018 to January 2019, the evaluation team collected five types of data (primary and secondary): semi-structured interviews, focus group discussions, a process mapping workshop, an online survey, and document review. The samples varied by data type including convenience, purposive, and census. Limitations in the findings included possible response bias from interviewees and survey respondents, as well sample selection bias. The findings are highly reflective of the samples and are not generalizable to the population of IAA stakeholders nor to a country. KEY FINDINGS AND CONCLUSIONS The evaluation answered three evaluation questions (EQs). EQ1: TO WHAT EXTENT HAS THE USAID-DOE IAA MET THE NEEDS OF MISSIONS, AND WHAT SPECIFIC OPPORTUNITIES EXIST TO IMPROVE THE BUY-IN PROGRAM? The interview data suggested that the IAA is meeting its primary goal of providing missions and in￾country partners with technical expertise. While the evaluation team is not able to confirm the extent to which the IAA met missions’ needs, interviewees from 12 of 14 missions stated positive views of NREL’s technical assistance. The IAA appeared to support and complement mission activities in country, as well as energy industry partners, in both new technology utilization and increased capacity in energy sectors. NREL respondents saw the IAA as a way to leverage additional funding, including direct activities between DOE and the mission, additional buy-in from other donors, and increased host-government funding. Mission staff, who often struggle with complex processes to procure technical expertise for in-country projects, value the mechanism for its ease of access to on-call data and expertise not otherwise found in country. In￾country partners indicated they view NREL teams as highly skilled and host-country government partners indicated a demand for additional NREL in-country presence. NREL activities appeared to be most effective in countries where (1) NREL has been present for many years and has established a reputation as a leader in the field, (2) USAID staff has strong capacity in the energy field, (3) the IP also has capacity in renewable energy or low-emission development strategies, and (4) NREL experts are fluent in the local language. PERFORMANCE EVALUATION OF THE USAID–DOE INTERAGENCY AGREEMENT | x Challenges were identified in ensuring that missions are aware of the IAA, the services it provides (including the breadth of technical expertise), and how to access the NREL network. Scheduling challenges were also noted, as NREL does not have enough staff to field multiple teams to support the same tool or platform and NREL technical experts are often booked well into the future. Nevertheless, mission respondents found that NREL was flexible with modifying SOWs and adapting to change. In addition, IAA processes for assessing missions’ needs are not clear, which likely result in missed opportunities to provide technical expertise. While the current activity intake process reflects USAID priorities and NREL’s availability, it does not necessarily reflect the expected impact of the requested TA. This may hinder the realization of the agencies’ goals for the IAA. EQ2: TO WHAT EXTENT ARE THE CLEAN ENERGY TECHNICAL PLATFORMS BEING USED EFFECTIVELY BY THEIR TARGET AUDIENCES FOR THEIR INTENDED PURPOSE? The survey data suggest tool use was limited in terms of the total number of users, if the intended outcome is increased tool users. However, this assumption is unknown; expectations of scale and size of use and users are not specified in IAA documentation. Additionally, the IAA’s monitoring of tool use has been poor and it has an incomplete list of users. Survey respondents who indicated they accessed or were trained on IAA tools were diverse within and across sectors and organization types. Tool users were not clustered around just one tool, which may indicate that users have multiple needs – even within the same type of sector or organization. Tool users also tend to be multiple-tool users, which may reflect the diverse needs of stakeholders for low-emission and renewable energy goals. This is important to know when designing targets and the implementation strategy for achieving IAA goals. Many survey respondents also reported using a tool for which they had not taken a training, suggesting there may be additional beneficiaries and target groups that are not reflected in the IAA objectives. The evaluation identified several types of tool use, such as how NREL supported the development of new quotas or standards in the renewable energy sector. Interviewees viewed this support as influential on national renewable energy policies. Data suggest the IAA has provided useful TA to diverse types of organizations, sectors, and users. However, the evaluation was not able to identify the underlying causes that led to tool use. In addition, having more users does not necessarily mean a higher return on the IAA investment. One tool user might have more impact on emission policies at the national level while large numbers of tool users might not be strategically placed to affect large-scale change. Respondents indicated a number of tool characteristics that facilitate use. These include free access, open sourced, ease of use, ability to access data not available in-country, and the unique features of the tool. There are opportunities for NREL and USAID to improve tool use by addressing barriers such as bandwidth requirements, country-specific model requirements, software or hardware limitations, limited user capacity, and lack of awareness of the tool. EQ3: WHAT ARE THE STRENGTHS AND WEAKNESSES OF THE STRATEGIC PLANNING, PROGRAM COORDINATION AND MANAGEMENT, COMMUNICATIONS, MONITORING AND REPORTING PROCESSES OF THE IAA? WHAT ARE SPECIFIC WAYS TO MAKE THE PROGRAM MORE EFFICIENT AND EFFECTIVE? The IAA’s clear strength stems from the fact that a USAID unit holds a formal agreement directly with another U.S. government entity. This arrangement can help strengthen new partnerships, encourages flexibility in achieving shared goals, and allows for adjustments when new priorities and budgets emerge. IAA partners are engaged and open to collaboration in achieving partnership goals. NREL staff find people are very positive about the work they do and the positive communications environment within the IAA strengthens planning and implementation processes. PERFORMANCE EVALUATION OF THE USAID–DOE INTERAGENCY AGREEMENT | xi The IAA’s main weaknesses also stem from the mechanism’s flexibility, which can result in unclear expectations, roles, and responsibilities for each agency. Current work under the IAA does not appear to align with the objectives in the original agreement, yet the language in the IAA has remained unchanged. Activity delays are also a persistent problem and dampen partner enthusiasm. This was particularly noted for delays related to travel approvals, staff availability, data access, and document/language clearance processes. The evaluation findings suggest that USAID and NREL understand “impact” and track programmatic effects differently, leading to divergent impact and M&E expectations. IAA products lack explicit causal pathways, which may underestimate the full portfolio of benefits produced through IAA activities or cause limited resources to be invested in ineffective activities. Without clear impact goals or an understanding of which variables and inputs are likely to be most effective in achieving desired impacts, USAID and NREL are hindered in making decisions about further investments in tools/platforms. The lack of a clear theory of change for the partnership further augments this issue. Without an explicit testable impact model, the IAA partners cannot effectively measure over time which investments lead to desired in-country effects. Current IAA monitoring practices are based on USAID indicators and are of limited utility for analyzing effectiveness, tracking change over time, or measuring tool use. For example, NREL trip reports include detailed information about tool use and implementation, and the effects of TA, but the report format requires significant transformation of the data for any mining or analysis. In addition, while some survey respondents reported using tools for which they had not taken an NREL training, current IAA M&E protocols do not capture spillover effects and thus hinder NREL’s ability to identify additional users, share new tools, and expand existing tool/platform use. RECOMMENDATIONS EVALUATION QUESTION 1 – MEETING NEEDS AND IMPROVING BUY-IN  E3/GCC should keep the IAA AOR located in Colorado, to facilitate coordination of activities with mission needs, permit the partnership to be agile, and provide NREL teams with immediate, direct access to USAID.  E3/GCC and NREL should co-locate NREL staff at USAID/Washington and missions, to strengthen access to NREL technical expertise.  E3/GCC and NREL should increase exchange opportunities to send NREL staff to the field for extended trips, and for in-country partners to visit NREL or other national laboratories.  USAID should promote awareness within the Agency of this IAA and similar partnerships.  USAID and NREL should co-fund an American Association for the Advancement of Science abroad.  NREL and E3/GCC should promote new kinds of IAA programming such as transportation expertise and online mentoring. EVALUATION QUESTION 2 – TECHNICAL PLATFORM USE  E3/GCC and NREL should rethink training on NREL tools to attract new types of users and new ways to use tools.  E3/GCC and NREL should pilot new ways to understand IAA benefits, such as a network analysis of trainees.  NREL should encourage opportunities for minorities and women to participate in training.  NREL should standardize its forms for IAA trainings and mission requests and needs. PERFORMANCE EVALUATION OF THE USAID–DOE INTERAGENCY AGREEMENT | xii  NREL and E3/GCC should consider creating a new position or enlisting support for a user experience specialist to prioritize the user experience for online technical platforms.  NREL should plan for the needs of future tool users and should consider issues such as country bandwidth requirements and emerging trends in virtual modeling. EVALUATION QUESTION 3 – STRATEGIC PLANNING, COORDINATION, COMMUNICATION, AND M&E  E3/GCC and NREL should jointly create a new MEL strategy for the IAA and host a workshop to create a new causal pathway reflective of partnership goals. The workshop should include a review of assumptions and expectations and identify which metrics would be most appropriate for long-term data capture.  E3/GCC and NREL should define metrics in greater detail. E3/GCC and NREL should also consider revising the IAA strategy for how the partners select and set country targets, which would address some of the timing issues around activity implementation.  E3/GCC and NREL should incorporate new methods of measuring and tracking IAA work and consider developing an online dashboard to help track website visits and users. The partners should also incorporate post-trip reporting documents into the formal IAA reporting process.  E3/GCC and NREL should train NREL teams on MEL strategies for development projects, which would help ensure NREL teams have more robust data collection practices to track and assess the impact of their work.  NREL should develop a strategic communications plan early in an activity.  In the next IAA modification, E3/GCC should consider explicitly defining key terms (e.g., impact) when these might differ by agency, assigning each partner tasks and roles that match their expertise, and clarifying any other areas where agency processes might differ and lead to misaligned expectations. These can include addressing ways to better facilitate travel approvals, data access, and other approval processes.  E3/GCC should align this IAA with the strengths and lessons learned from other IAAs. E3/GCC should host a workshop in which other IAA AORs and their partners discuss the strengths and weaknesses of IAAs and their perceptions about impact, especially when partnering with science agencies. Lessons can also be shared from evaluations of IAAs and similar partnerships. PERFORMANCE EVALUATION OF THE USAID–DOE INTERAGENCY AGREEMENT | 1 INTRODUCTION This report presents the results of a performance evaluation of the interagency agreement (IAA) that began in 2009 between the United States Agency for International Development (USAID) and the United States Department of Energy (DOE), including DOE’s National Renewable Energy Laboratory (NREL). USAID’s Office of Global Climate Change in the Bureau for Economic Growth, Education, and Environment (E3/GCC) commissioned this evaluation to inform future IAA work plans and strengthen overall IAA strategic management. The E3 Analytics and Evaluation Project1 designed and implemented the evaluation. Annex A provides USAID’s statement of work (SOW) for the evaluation. IAA BACKGROUND Recognizing the need for comprehensive technical assistance (TA) to support low-emissions growth activities across the globe, the U.S. government established a network of U.S. experts, agencies, and technical institutions to assist developing country partners in the clean energy and landscape sectors. In 2009, USAID and DOE’s Office of Policy and International Affairs signed an IAA2 enabling U.S. technical expertise from DOE to work with partner countries. The IAA draws from a network of national labs and their technical staff to provide TA to USAID missions and partner countries on the analysis, design, and implementation of clean energy, greenhouse gas–mitigation initiatives. In 2017, E3/GCC signed a follow-on five-year IAA directly with NREL. As of 2018, the IAA had undergone 10 modifications. USAID/E3 provides most of the funding for IAA activities, covering IAA management, the development of technical platforms, and assistance to missions. The IAA also allows for buy-ins from missions and other USAID operating units to access NREL TA. Table 1 shows total funding amounts from USAID operating units. Mission buy-ins have been increasing since the original IAA in 2009, while E3/GCC support has been generally consistent but with some spikes in funding. Missions such as India, Mexico, and the Regional Development Mission for Asia (RDMA) have multiple-year funding. TABLE 1: IAA FUNDING BY OPERATING UNIT, THROUGH MODIFICATION 10 USAID Operating Unit Total Funding Amount E3/GCC $168,452,020 Bangladesh Mission $19,600,000 India Mission $18,300,000 Haiti Mission $18,270,000 Mexico Mission $18,100,000 RDMA $17,800,000 Vietnam Mission $8,500,000 Philippines Mission $6,500,000 Kenya Mission $4,450,000 Brazil Mission $1,000,000 Jamaica Mission $550,000 East Africa Mission $500,000 Ecuador Mission $250,000 1 Management Systems International (MSI), a Tetra Tech Company, implements the E3 Analytics and Evaluation Project in partnership with Palladium and NORC at the University of Chicago. 2 Award number AEG-P-00-09-00003-00 PERFORMANCE EVALUATION OF THE USAID–DOE INTERAGENCY AGREEMENT | 2 The IAA has supported the creation of a variety of web-based technical platforms and tools in online or desktop format, to assist with the development of renewable energy resources by targeting specific technical or analytical barriers. These technical platforms and tools include: analytical tools that stakeholders can use or use in partnership with DOE laboratories; resources, information, and best practices; access to expert technical support; and trainings, workshops, and seminars. IAA-supported technical platforms and tools include Greening the Grid (GtG), geospatial toolkits, the Renewable Energy Data Explorer (RED-E), the Development Impact Assessment (DIA) toolkit, and the International Jobs and Economic Development Impacts (I-JEDI) tool. Annex A provides additional background information about each of these tools. In addition to developing technical platforms, NREL provides training on technical platform content to USAID staff and external stakeholders. This includes workshops, webinars, and custom trainings for policymakers and decision-makers as well as technical staff and practitioners. Other IAA training and outreach approaches include developing enhanced website and outreach material, improving coordination with other database organizers or data suppliers, and in-person or virtual participation in USAID staff trainings. The IAA’s original primary objectives as stated in the 2009 agreement were:  To provide TA for priority activities identified by USAID, including technical support to USAID missions and strategic countries and regions for global climate change clean energy programs and strategies including low emission development strategies (LEDS).  To assist priority USAID missions, countries, and regions in accessing the best U.S. technical expertise and expert resources to support global efforts to combat climate change.  To provide analytic and technical services to advance the whole of government effort supporting enhanced capacity for LEDS with up to 20 partner developing countries by 2013, including coordination and alignment with REDD+ related initiatives. Figure 1 breaks down the IAA modification 10 budget by tasks. FIGURE 1: DISTRIBUTION OF THE IAA MODIFICATION 10 BUDGET BY TASK $20,000 $50,000 $140,000 $150,000 $160,000 $280,000 $347,000 $350,000 $400,000 $435,202 $700,000 Database Integrated Resource Planning Technical Assistance Communications Mexico: Carbon Markets Geospatial Data Science and Visualization RDMA: Clean Power Asia India: SGWG Enterprise Geospatial Toolkit Renewable Energy Zones Greening the Grid Secretariat Energy Division: Distributed PV Pilots PERFORMANCE EVALUATION OF THE USAID–DOE INTERAGENCY AGREEMENT | 3 EVALUATION PURPOSE E3/GCC commissioned this performance evaluation to inform future IAA work plans and strengthen the overall strategic management of the agreement between E3/GCC and NREL. The evaluation provides IAA partners with recommendations on ways to continue providing missions and in-country partners cutting-edge TA, examples of tool use, and suggestions for improving key partnership dimensions critical in an effective collaboration. EVALUATION QUESTIONS This study answered three evaluation questions (EQs). USAID’s SOW also provided sub-questions for each EQ that conveyed areas of interest (AOIs) to inform the data collection instruments and key issues that the evaluation findings, conclusions, and recommendations should address. These are listed below and match those in USAID’s SOW. Annex C provides additional evaluation findings for each AOI. EQ1. To what extent has the USAID-DOE IAA met the needs of missions, and what specific opportunities exist to improve the buy-in program?  AOI 1.1 How, and to what extent, has this mechanism allowed USAID Mission and partner country access to NREL expertise and cutting-edge analysis, research, and deployment?  AOI 1.2 In what specific ways has this mechanism complemented capabilities of local contractors, partners, or other contract mechanisms used by the field Missions?  AOI 1.3 How can coordination between E3, NREL, Mission and local implementing partners be improved?  AOI 1.4 What types of technical expertise or assistance from NREL and other DOE Labs have been of highest value to partner governments or other Mission stakeholders? What type of technical assistance delivery models have been most effective?  AOI 1.5 In what ways have lessons been transferred to the broader USG low emissions development portfolio and what have the benefits been? EQ2. To what extent are the clean energy technical platforms being used effectively by their target audiences for their intended purpose?  AOI 2.1 How are these technical platforms and tools being used by target audiences and where/why might they be underutilized?  AOI 2.2 What were the primary drivers that informed the types of technical platforms and other resources that have been developed? How are these drivers and user needs evolving?  AOI 2.3 What types of policies, programs, and country capacity development are being informed or advanced through these technical platforms and related technical assistance?  AOI 2.4 Do host country counterparts credit the platforms, tools or data derived from them with helping to drive policy or regulatory change, improve planning processes, or drive investment decisions?  AOI 2.5 To what extent have host country counterparts, or “owners”, of the tools committed to the long-term operational costs required to keep model output current? PERFORMANCE EVALUATION OF THE USAID–DOE INTERAGENCY AGREEMENT | 4 EQ3. What are the strengths and weaknesses of the strategic planning, program coordination and management, communications, monitoring and reporting processes of the IAA? What are specific ways to make the program more efficient and effective?  AOI 3.1 Does NREL have the right number of staff to meet the needs of the IAA, and do staff have the right qualifications?  AOI 3.2 Does the broader NREL team of experts feel that they have clear direction when brought on to tasks and do they feel that their work has impact?  AOI 3.3 Does NREL have a robust strategy/theory of change for how TA provided in the work plan will contribute to on the ground mitigation results?  AOI 3.4 Do the results being monitoring under the IAA reflect the overall strategy and goals of the agreement?  AOI 3.5 What barriers does NREL face in implementing their work plan on time and as planned, that USAID can address?  AOI 3.6 How, and to what extent, has this mechanism helped to promote consistency and continuity of program technical support under the EC-LEDS program?  AOI 3.7 How well does NREL coordinate with other USG programs and agencies working in the same sector and country?  AOI 3.8 Are gender issues sufficiently addressed in work planning or implementation? METHODS AND LIMITATIONS EVALUATION DESIGN The evaluation team used a mixed-methods approach3 to answer the EQs, collecting both qualitative and quantitative data. There were five data sources: semi-structured key informant interviews (KIIs), focus group discussions (FGDs), a process-mapping workshop, an online survey, and secondary data from key documents. The team collected interview data in two waves (Annex B provides further details about the data collection process). The evaluation design process started in the fall of 2017, with data collection from March 2018 to January 2019. The evaluation focused on tool use across five technical platforms that USAID identified and included respondents who were active in IAA activities from 2015 to 2017. SAMPLE AND DATA COLLECTION USAID and NREL provided the sample list for KIIs and the trainee list for survey participants. With guidance from USAID, the evaluation team used these lists to select respondents and ensure the evaluation collected diverse data by country, respondent type, and technical platform or tool. The evaluation team conducted the first wave of data collection (KIIs, FGDs, and the process mapping workshop) between May and September 2018. The E3/GCC activity manager for this evaluation attended the process mapping workshop as a participant and several KIIs as an observer. During the preliminary analysis, the evaluation team identified two data gaps: (1) limited responses from host￾country government officials who had been partners in IAA activities; and (2) the small number of tool users shared in the sample that USAID and NREL provided. In agreement with E3/GCC, the team sought to fill these gaps through a second wave of data collection between October 2018 and February 2019, including additional KIIs and an online survey to capture more tool users. Annex A includes 3 The evaluation did not test causal linkages or infer statistical significance between the variables of interest, thus not measuring the IAA’s impact. The team did not measure the direct attribution of IAA activities on policy or programmatic changes related to the energy/power sector within a country. PERFORMANCE EVALUATION OF THE USAID–DOE INTERAGENCY AGREEMENT | 5 USAID’s supplemental SOW for this second wave. Table 2 summarizes the samples by data source. Annex B provides additional information about the samples for each data source and the data collection instruments. TABLE 2: SAMPLE BY DATA SOURCE Data Source Sample Framework Totals by Data Type Document and Data Review Key IAA documents that NREL and USAID shared Modifications, work plans, quarterly monitoring and performance reporting More than 50 documents Web analytics data that NREL and USAID provided (limited) Data on usage and user profiles for GtG, I-JEDI, and RED-E tools Insufficient/incomplete information for useful analysis of web use and user profiles KIIs (by phone and in person) USAID/Washington staff administering and participating in the IAA Census – IAA agreement officer’s representative; E3/GCC and E3 Office of Energy and Infrastructure staff 89 KIIs4 Staff from USAID bilateral or regional missions that bought into or received TA through the IAA from 2015–2017  Census – 11 buy-in missions (Bangladesh, Colombia, Gabon, Ghana, Haiti, India, Kenya, Mexico, Philippines, RDMA, Vietnam) plus additional missions  Convenience sample of buy-in and non-buy-in missions selected in consultation with USAID and NREL for staff availability and their knowledge of the IAA NREL core IAA management staff Census – NREL staff in Colorado who manage the IAA NREL technical staff Purposive sample of NREL staff in Colorado who provided TA or training under the IAA; supported the GtG, I￾JEDI, or RED-E tools; or worked on a pilot activity—with a target of 15 KII from a potential 38. In-country partners directly involved in IAA-supported activities Convenience sample identified in consultation with USAID and NREL Implementing partners (IPs) Purposive sample identified in consultation with USAID and NREL Tool and platform users Purposive sample of users involved directly and actively since 2015 with GtG, RED-E, I-JEDI, DPV, or DIA tools; could include people trained on using the tools. During wave 1 the team targeted 3 KIIs per tool. For wave 2 the team interviewed tool users identified through the online survey. Online Survey of Trainees People who attended NREL-supported trainings Census - all viable email addresses given by participants at 25 trainings from 2015-2018 were sent invitations 274 completed surveys (a 17.6 percent response rate) 4 Wave 1 included 69 KIIs (17.6 percent response rate); Wave 2: included 20 KIIs (a 42.6 percent response rate). PERFORMANCE EVALUATION OF THE USAID–DOE INTERAGENCY AGREEMENT | 6 Data Source Sample Framework Totals by Data Type to participate in the survey between October and November 2018 - 1,554 addresses FGDs and Workshop at NREL FGD 1: NREL technical staff NREL technical staff in Colorado who were knowledgeable about IAA activities. FGDs 1 and 2 covered different themes related to the evaluation questions. 1 FGD with 8 participants FGD 2: NREL core IAA management and technical staff NREL staff in Colorado who participated in the IAA. FGDs 1 and 2 covered different topics. 1 FGD with 8 participants Process-mapping workshop NREL and USAID staff in Colorado who worked on IAA activities 1 workshop with 25 participants DATA ANALYSIS The evaluation team conducted content analysis of the qualitative data it collected across response groups. First, the team coded transcripts via the MAXQDA qualitative analysis software using a coding scheme representative of the EQs and AOIs. Then, the team aggregated the data by response group and data source. The team assessed frequencies across the response groups. However, statistical testing of these was not possible due to small sample numbers. TEAM COMPOSITION The core evaluation team consisted of Dr. Carolyn Fonseca, the team leader and evaluation specialist; and Mr. Isaac Morrison, the research associate. Ms. Annie MacFadyen supported KII coding. Research support included Ms. Lyn Phang (survey data cleaning and descriptive statistics), Mr. Eric Shimono (document organization), and Mr. Jason Catanach (document cleaning and extraction). The E3 Analytics and Evaluation Project home office team supported the evaluation team with administrative assistance and quality assurance. The evaluation team members signed USAID’s conflict of interest disclosure statement, which are retained by MSI and available upon request. Annex D provides brief biographies of the core team members. LIMITATIONS This section summarizes the limitations of the evaluation and their implications on the results. Annex B provides additional information about the samples, their limitations, and the team’s mitigation strategies. RESPONSE BIAS During data collection, the evaluation team carefully assessed the data for response bias, particularly whether respondents may be providing socially desirable answers. This type of bias could be present in the NREL and IP samples as they receive direct funding from USAID. Although both samples were small, the lack of heterogeneity in the NREL sample increases the likelihood of a higher response bias compared to the IP sample (which included more diverse responses). The evaluation team also pre￾tested interview questions to help reduce this type of bias. RECALL BIAS Recall bias may be present when respondents have an inaccurate or incomplete recollection of past events. To reduce the likelihood of recall bias, the online survey covered only trainings conducted in the last two years. Some level of recall bias was evident from the large number of survey respondents who PERFORMANCE EVALUATION OF THE USAID–DOE INTERAGENCY AGREEMENT | 7 indicated they “do not remember” using a tool after training. This does not mean tools were not used after training, just that respondents could not accurately recall this when asked. It is important to note that the survey data had significant limitations and are not generalizable. SAMPLE SELECTION BIAS Selection bias in a sample can lead to distorted interpretation of findings that are not representative of the whole group. For KIIs, USAID and NREL provided contact information based on the level of engagement between NREL and the stakeholder. Missions with which NREL interacted more frequently typically had more stakeholders, and these stakeholders were more likely to respond to requests for interviews. For the survey, because GtG and RED-E trainings were represented in higher numbers between 2015 and 2018, the survey received more responses regarding those tools than other tools. Survey findings therefore are not generalizable across the IAA; they reflect only a specific group of users. DOCUMENTATION EVIDENCE Limited documentation on tool and platform web traffic made it difficult to determine user type, user location, and preferred features of tools. The team found detailed and useful data in the NREL trip debriefs, but the format of the debriefs made only a portion of the information usable, which limited a more robust analysis of IAA work. The team excluded from the online survey sample the lists of training attendees for which the sign-in forms were handwritten, or where participant names and contact information were not provided. The final survey sample was biased toward more standardized sign-in forms, thus potentially skewing the sample of respondents to those who took trainings with certain team leads. INSTRUMENTATION BIAS AND HUMAN ERROR The evaluation team carefully pretested all data collection instruments to reduce instrumentation bias. To reduce human error in coding, team members worked together closely and frequently reviewed the coding definitions to ensure consistent and reliable coding of KII text in MAXQDA. FINDINGS AND CONCLUSIONS FINDINGS FOR EVALUATION QUESTION 1 To what extent has the USAID DOE IAA met the needs of the missions, and what specific opportunities exist to improve the buy-in program? To answer EQ1, the evaluation team conducted KIIs with USAID staff and other IAA stakeholders. Their responses did not provide details on the extent to which needs were met but did produce general information on the kinds of IAA support provided and the level of satisfaction with this assistance. Interview data included responses from staff in 14 missions5 and 4 USAID/Washington units (E3/GCC, the E3/Office of Energy and Infrastructure, the Bureau for Latin America and the Caribbean, and the Bureau for Asia), and the findings reflect this sample. To ensure confidentiality as promised in the KIIs, the evaluation team presents missions’ views as a group, because some missions had only one respondent who could therefore easily be identified. 5 Bangladesh, Colombia, Ghana, Haiti, India, Jamaica, Kenya, Laos, Mexico, Philippines, RDMA, South Africa, Vietnam, and Zambia. PERFORMANCE EVALUATION OF THE USAID–DOE INTERAGENCY AGREEMENT | 8 MISSION VIEWS ON THE IAA Interviewees from 12 of the 14 missions stated positive views of NREL’s technical assistance. Only two missions gave responses suggesting the IAA did not fully address their technical need/request. Respondents from all 14 missions indicated the IAA provided a critical avenue through which to access otherwise unavailable technical expertise. Mission respondents felt they had a group of experts on call, and the time required to access them was significantly shorter than going through other contracting mechanisms. One mission found the IAA provided access to expertise in the resource management and environmental sector, which is not a sector for which the mission often received funding. Furthermore, NREL supported missions by helping increase the technical capacity of IPs through supplemental assistance, sophisticated data analysis, provision of complex data sets, and professional guidance, which led to stronger implementation of mission-funded work. Both IP and mission staff described how they specifically benefited from NREL technical expertise. Missions that had staff with limited technical expertise in the renewable energy, energy, and engineering sectors saw NREL staff as on-demand, in-house local experts who could field technical questions, guide missions in writing technical scopes of work, and help liaise with in-country technical partners. At least three mission interviewees stated that NREL’s work supported the mission by conducting analysis to support government processes to meet energy goals and build capacity in the industry to support the provision of renewable energy options. One mission gave an example in which NREL provided data to help make public transportation more efficient. For one mission, NREL’s value was in making the mission’s activity relevant to economic growth such as through use of the I-JEDI tool and calculation of jobs connected with electric vehicles. Others valued the NREL training, access to Plexos simulation software (used to integrate electric, water, and gas systems and for optimization modeling), and tools available through the IAA. “…[the IAA] allows us to address and support our host country objectives and requests in the area of renewable energy.” - USAID mission respondent “I think the idea that our different implementers bring different aspects that are useful, and NREL brings this really strong technical component that is hard to find. The idea is to try to link that in with folks that are more on the ground and can really ground the work that NREL is doing.” - USAID mission respondent “NREL came in handy, they sent us someone who can do stocktaking, talking to the various stakeholders, convening a number of meetings.” - USAID mission respondent PERFORMANCE EVALUATION OF THE USAID–DOE INTERAGENCY AGREEMENT | 9 Mission respondents found the IAA-supported trainings useful to the public sector in energy efficiency. For example, NREL’s capacity-building efforts helped the Kenyan government pursue energy efficiency in public buildings. Missions also found that the provision of technical expertise and uptake was more effective when NREL teams spoke the local language and gave training/TA in the local language. Mission staff found language skills to be particularly important as they noticed participants asking more complex questions during trainings in the local language. Mission interviewees had highly positive views of NREL teams working in-country. Missions consistently suggested NREL consider increasing face-to-face time, which often helped activities work more effectively, provided on-the ground expertise to mission staff, created opportunities for partners to leverage other work, and can lead to real-time answers during critical implementation periods. NREL participants in both FGDs also recognized this opportunity. IN-COUNTRY PARTNER VIEWS ON THE IAA The evaluation defined in-country partners as groups that participated directly in IAA activities. These groups include IPs, host-country government officials, NGOs, donors, the private sector, academics, and community groups that might have been stakeholders in IAA activities. IPs perceived the IAA as an effective mechanism for accessing technical expertise, which aligned with the views of mission staff. IPs also found the IAA effective because NREL SOWs were flexible in accommodating changes and NREL had formed strong relationships with country partners. IPs reported that the IAA helped missions by increasing their capacity, which in turn (1) reduced risk during the early stages of activity development, (2) permitted IPs to access technical experts – particularly in solar and wind, where capacity is generally lacking in most countries, and (3) allowed NREL to support IPs on activities that worked to build new renewable energy markets and help governments create underlying policy frameworks. NREL respondents discussed the advantages of the IAA as a way to leverage additional funding, including direct activities between DOE and the mission, additional buy-in from other donors, and increased host￾government funding to support the effort. For example, in India an NREL activity started with USAID/E3 funding but over time leveraged funds from the World Bank and the U.S. Department of State. NREL staff suggested the IAA was most effective in meeting missions’ needs when those missions had staff with some knowledge about the energy sector, when the host-country government had prioritized the area in which the NREL team was working, and when there was a clear path linking the IAA activity to a decision-making process. The two government respondents found direct conversations with NREL helpful and stated a desire to have more in-country NREL presence to help them attain their energy goals. These government respondents also highlighted NREL’s value as a technical partner with universities and utility companies, and in capacity-building efforts around government grid integration. CHALLENGES IN MEETING MISSION NEEDS Respondents were mostly positive about NREL’s support to missions and in-country partners, but some had negative experiences and faced challenges in accessing NREL. Challenges often centered around countries where NREL does not have a long history of working with in-country partners or local governments, or when SOWs with the mission were unclear – particularly about the products and timeline. For example, a mission respondent said that an activity SOW was not clear enough about NREL’s technical work to meet the mission’s needs. Another mission respondent reported that NREL did not know the local context and was not able to bring stakeholders together – a significant barrier to PERFORMANCE EVALUATION OF THE USAID–DOE INTERAGENCY AGREEMENT | 10 working in that country. This respondent felt that being connected to a local partner would reduce challenges that NREL might face in country. Respondents shared other challenges including limited familiarity with the details of the IAA mechanism – particularly with respect to the proper process for accessing NREL or knowing all the available services. Three missions reported barriers regarding awareness of NREL services available through the IAA. One mission respondent said that keeping the mission informed about services available from NREL and the network of DOE laboratories would be an ongoing challenge. An additional challenge that a government respondent raised was differing schedule priorities between NREL and the stakeholders it serves, which can produce incompatible timelines for final products. CONCLUSIONS FOR EVALUATION QUESTION 1 The interview data suggested that the IAA is meeting its primary goal of providing missions and in￾country partners with technical expertise. Mission staff, who often struggle with complex processes to procure technical expertise for in-country projects, value the mechanism for its ease of access to data and expertise not otherwise found in country. Challenges arise when NREL teams are requested in two places at the same time, because NREL does not have enough staff to field multiple teams to support the same tool or platform. For example, there is only one GtG team, and it is often booked far in advance. Tight scheduling limits NREL teams’ ability to extend a trip or leverage other efforts. Despite scheduling inflexibility, mission respondents found that NREL was flexible with modifying SOWs and adapting to change. There are probably missed opportunities to provide technical expertise, but IAA processes for assessing missions’ needs are not clear. The current process reflects USAID priorities and NREL’s availability, but not necessarily the expected impact of the requested TA. This may hinder the realization of the agencies’ goals for the IAA. NREL appears to be most effective under the IAA when it worked in countries where (1) it has been present for many years and has established a reputation as a leader in the field, (2) USAID staff has “[a challenge for the mission is] that we know that there is an agreement with NREL to make sure how can we make use of it. That's a really, really difficult job. I think it's done pretty well if you know who the correct people are. But I think that's just gonna be a continual challenge to make sure that the field or missions know what resources are available…that's just something to keep in mind.” - USAID mission respondent “…maybe in NREL, they have the same problem as we do, that they have few people doing many things. And in that case, I understand why they take so long into doing some things. But that's not our appreciation. Sometimes, I think that they go at their own pace and don't really get the idea of the sense of urgency or the specific dates that we would love to see” - Host-government respondent PERFORMANCE EVALUATION OF THE USAID–DOE INTERAGENCY AGREEMENT | 11 strong capacity in the energy field, (3) the IP also has capacity in renewable energy or low-emission development strategies, and (4) NREL experts are fluent in the local language. The challenges are in ensuring that missions are aware of the IAA, the services it provides (including the breadth of technical expertise), and how to access the NREL network. This is a common challenge facing many USAID-funded projects that require mission buy-in and direct engagement, because mission staff rotate every few years. FINDINGS FOR EVALUATION QUESTION 2 To what extent are the clean energy technical platforms being used effectively by their target audiences? To answer EQ2, the evaluation team drew from the online survey of training participants and KIIs. The Sample and Data Collection section and Annex B provide additional information on the survey sample. NUMBER AND TYPES OF TOOL USERS The survey and interview data suggested that IAA tools and products are being used in small numbers. The number of active users the evaluation identified reflects the sample, particularly for the survey data as only 274 (17.6 percent) of the 1,554 invited to participate (from the lists of specific trainings NREL carried out between 2015-2018) responded to the survey. Of the 274 individuals who responded to the survey, 153 (55.8 percent) stated using at least one tool while 121 (44.2 percent) had not used a tool (see Table 3). Additionally, of the 274 survey respondents 159 (58 percent) stated they had attended a training. Of the 274 respondents, 109 (39.8 percent) stated they had taken a training and also used at least one tool. TABLE 3: SURVEY RESPONDENTS REPORTING TOOL USE Response Number of Tools Used Total % “No, I did not use a tool” None 121 44.2 “Yes, I used at least one of the tools (or more)” 1 95 34.7 “Yes, I used at least one of the tools (or more)” 2 32 11.7 “Yes, I used at least one of the tools (or more)” 3 8 2.9 “Yes, I used at least one of the tools (or more)” 4 7 2.6 “Yes, I used at least one of the tools (or more)” 5 11 4.0 Total Number of Tool Users 153 Total Respondents 274 There were tool users across all tools and in multiple countries. Table 4 breaks down the number of users by tool type and country. Overall, GtG (83 users) and the RED-E (82 users) were the most used tools across all countries, followed by I-JEDI (38 users) and DPV (37 users). DIA (26 users) had the least users. The top 11 countries where tool users were clustered were: Philippines, India, Ghana, U.S., Bangladesh, Vietnam, Kenya, Sri Lanka, South Africa, Pakistan, and Colombia. Data on tool use from other countries showed only one to two users per country. PERFORMANCE EVALUATION OF THE USAID–DOE INTERAGENCY AGREEMENT | 12 TABLE 4: NUMBER OF SURVEY RESPONDENT TOOL USERS BY STATED TOOL USED AND COUNTRY Country GtG RED-E I-JEDI DPV DIA Total Users Philippines 17 14 2 5 3 41 India 13 3 1 2 2 21 Ghana 6 8 2 6 3 25 U.S. 7 5 5 2 2 21 Bangladesh 2 6 1 2 1 12 Vietnam 5 4 1 1 1 12 Kenya 0 10 1 0 0 11 Sri Lanka 5 3 1 1 0 10 South Africa 0 1 7 2 0 10 Pakistan 1 2 0 1 0 4 Colombia 0 3 0 1 0 4 Other countries 27 23 17 14 14 95 Total 83 82 38 37 26 The number of tools users varied by organization type. The survey data indicated that each of the five tools have users from all organizational types, with varying frequencies. Across the organization types, public-sector respondents were most commonly identified as tool users (see Figure 2). FIGURE 2: NUMBER OF TOOL USERS BY ORGANIZATION TYPE (N=153) Tool user numbers varied by sector. The survey data identified that most tool users were in the energy regulation/policy planning sector, which primarily includes government respondents. As shown in Figure 3, the type of tool used in the energy regulation/policy planning sector varied: 32 (30 percent) used GtG, 37 (34 percent) used RED-E, 17 (16 percent) used I-JEDI, 13 (12 percent) used DPV, and 9 (8 percent) used DIA. 54 33 23 21 12 3 7 Government/Public Sector Private/Industry Not-for-Profit Organization/Community Organization University/College/Academic or Research Institution International Donor/Assistance Organization Not Currently Employed (retired, between jobs, unemployed) Other* PERFORMANCE EVALUATION OF THE USAID–DOE INTERAGENCY AGREEMENT | 13 FIGURE 3: NUMBER OF TOOL USERS BY WORK SECTOR AND TOOL TYPE There were single-tool users in all sectors, but in several sectors, respondents had used multiple tools. As Figure 4 shows, the energy regulation and policy sector had the highest numbers of both single- and multi-tool users. FIGURE 4: NUMBER OF TOOL USERS BY WORK SECTOR AND SINGLE- OR MULTI￾TOOL USE Respondents had incomplete recollection of tool use, suggesting bias in the survey data on tool use. Across organization types, survey respondents stated they used the tools during and after trainings. However, many respondents could not remember if they had used the tool before or after the training, 32 19 13 8 3 8 17 1 5 9 0 6 37 4 16 11 3 11 9 1 4 6 0 6 13 2 8 8 2 4 Energy Regulation/ Policy/ Planning Energy Transmission/ Distribution Energy Generation Energy Efficiency Buildings/ Construction Other GtG I-JEDI RED-E DIA DPV 40 19 15 8 2 11 25 2 12 8 3 8 Energy Regulation/ Policy/Planning Energy Transmission/ Distribution Energy Generation Energy Efficiency Buildings/ Construction Other Single Tool Multiple Tools PERFORMANCE EVALUATION OF THE USAID–DOE INTERAGENCY AGREEMENT | 14 suggesting the potential for recall bias6 in the data. Table 5 also indicates an important and unexpected finding from the survey data, that there was a substantial number of tool users who had not taken a training but reported using a tool. This suggests there were spillover effects from IAA training (i.e., tool users learned of one tool during training for a different tool). However, the evaluation team was not able to measure the extent of these effects due to limitations in the evaluation scope and budget. TABLE 5: SURVEY RESPONSES ABOUT TOOL USE AND TRAINING ATTENDANCE Organization Type Used at Least One Tool Attended at Least One Training Used at Least One Tool and Attended the Training Government and Public Sector 54 56 37 Private Industry 33 41 24 International Donor or Assistance Organization 12 12 8 Not Currently Employed 3 1 1 NGOs and Community Organizations 23 21 18 University, College, and Academic or Research Institution 21 22 17 Other 7 6 4 Total 153 159 109 PURPOSES OF TOOL USE The survey and interview data suggested that IAA tools and products are being used for various purposes. Survey respondents’ most common answers on why they primarily used the tool(s) were “providing information to a decision maker” (30.6 percent) and for “professional growth” (27.6 percent) (see Figure 5). The survey data on primary tool use showed similar distribution patterns across the five tools/platforms. FIGURE 5: SURVEY RESPONDENTS BY PURPOSE OF TOOL USE 6 Recall bias is a type of systemic error in which participants have inaccurate or incomplete recollection. This bias in the data could suggest a trend that is not truly present. For example, a significant number of survey respondents could not remember using a tool, especially after the training. This does not mean tools were not used after training, just that the respondents could not accurately recall this when asked. Therefore, the evaluation team cannot state that a tool was not used after training, but that that data are limited in knowing the full extent of use. 71 64 36 27 25 9 Providing Information to a Decision Maker Professional Growth Setting a Standard/Quota/Code/Policy/Regulation Generating a Report Research and Journal Publications or Papers Other PERFORMANCE EVALUATION OF THE USAID–DOE INTERAGENCY AGREEMENT | 15 * n = 232 for Figure 5, as some survey responses had missing data and/or the respondent reported they did not use a tool. EXAMPLES OF TOOL USE Interviewees provided a number of examples in which IAA platforms/tools appear to have been used to advance policies, programs, or standards. According to respondents, NREL support helped create regulations, change standards and codes, and support decision-making processes through technical reports, analysis, and data. Interviewees (noted by type in parentheses below) shared the following examples of changes perceived to be influenced by NREL assistance via the IAA. Colombia (IP respondent) – The regulatory commission in Colombia received direct support from NREL to create the distributed general resolution, which outlines the country’s process to distributing energy. NREL conducted several workshops to support this effort. Part of this work helped the Colombian government set up long-term mechanisms in auctions to allow solar and wind projects to compete in Colombia. NREL TA also influenced a resolution on how distributive energy investors, communities, projects, and industries can design standards, connect, sell energy into the grid, get energy back from the grid, and how this energy demand and supply could be forecasted. The NREL expert who supported this activity was a specialist in electricity markets. According the IP, he was very precise and objective about his assessment. The policy adjustments that Colombia’s Ministry of Mines and Energy made were influenced by this TA. Guatemala (host-government respondent) – The respondent received training on I-JEDI then used that knowledge to help the Guatemala government write its new 20-year energy plan. Zambia (IP respondent) – Data collected and submitted to the rural electrification authority were used to develop a framework for the institution and helped it complete the intended nationally determined contribution document. Additionally, the private sector used I-JEDI to do analysis for the construction of a hydro plant. Another private company looked at calculations for the number of employees needed for its solar project. Philippines (IP respondent) – Through work with NREL, the wind prospector, and grid training, ceilings for wind and solar energy were changed. Prior estimates that informed the ceilings were based on limited models about the full capacity of the Philippines energy grid, and the prior ceilings underestimated the amount of wind and solar energy that the grid could accommodate. Kenya (mission respondent) – The Ministry of National Development Planning used DIA to set some national priorities, which spurred local institutions to use the tool as well. Additionally, IAA activities supported changes in Kenya’s minimum energy standards through awareness raising and assessments that NREL conducted. Vietnam (mission respondent) – NREL provided support to revise existing codes and advised on new codes for the Vietnam clean energy program. Jamaica and Mexico (USAID/Washington respondent) – NREL supported an assessment of the net billing policy and influenced some rooftop photovoltaic (PV) policies. NREL provided similar support in Mexico, where it influenced that country’s’ PV policy. FACTORS PROMOTING TOOL USE Interview data suggested tools were more likely to be used and adopted when IPs had strong existing capacity, when government priorities were aligned with the product/tool in a specific decision-making PERFORMANCE EVALUATION OF THE USAID–DOE INTERAGENCY AGREEMENT | 16 process, when modeling and higher-level analysis were needed, and when there was a highly specific target tool user. Interviewees indicated the following enabling factors for tool/platform use:  Free and open sourced  One of a kind  Easily shared  Easy to use  Has online resources  Provides new data The survey also asked respondents to share their favorite feature of the tool/platform. Figure 9 illustrates the responses in a word cloud, where a larger font size indicates a larger number of responses for that feature or issue. Data, renewable information, and analysis were among the most popular responses. FIGURE 6: SURVEY RESPONSES TO "WHAT WAS YOUR FAVORITE FEATURE(S) OF THE TOOL/PLATFORM?" FACTORS INHIBITING TOOL USE Interview data highlighted possible issues limiting tool use, which could contribute to the survey numbers on usage. The most common barriers interviewees shared related to online access to tools including hardware and software requirements, bandwidth, and language:  Competing tools  Bandwidth and other internet access issues  Lack of capacity  Language  Lack of awareness about the tool  Tool not customized for the country. Another challenge interviewees cited was feeling a lack of ownership of IAA-supported tools. DRIVERS OF TOOL DEVELOPMENT The IAA lacks an explicit theory of change related to tool use, and IAA modifications include little information about the target number of users for IAA tools/platforms, or about the process the IAA partners use to decide which mission requests are higher priority. NREL respondents indicated the PERFORMANCE EVALUATION OF THE USAID–DOE INTERAGENCY AGREEMENT | 17 creation of tools/platforms (or selection of a specific mission for TA) was driven by a balance between USAID priorities, available resources, mission requests, and country needs – which may not match the mission request. It may be necessary for the partners to develop a clear selection strategy or criteria to ensure IAA goals are achieved. EVOLVING USER NEEDS The IAA modifications indicated that the scale of IAA tasks increased over time. In the early modifications, missions bought in for small amounts (around $20,000 to $50,000). Mission buy-ins rose significantly over time, and some (e.g., RDMA and Mexico in modification 7) recently bought in for over $1 million. In missions where the IAA had significant involvement over multiple years (e.g., India, Colombia, Mexico, the Philippines), much of the current work is in advanced modeling and grid analysis of energy requirements. In newer missions where NREL has had a shorter presence or there is less country capacity around renewables, requests focus more on training and workshops – building blocks to reach low-emission and clean energy goals. The evaluation team asked respondents to share what tool needs they had now or anticipated for the future. Respondents discussed their needs but did not always differentiate between current and future needs. The most commonly mentioned needs were roof PV and microgrid solar support, integration of tool data and modeling into large systems (e.g., including energy with other socioeconomic data sets and conditions), electric cars, buses, and other transportation, and grid integration. Interviewees identified the following trends and needs:  Social education and awareness  Grid integration  Mobile tech  Politics and regulatory issues  Electric cars and buses  Battery and storage  System stability  Resiliency plans  Tool expansion to other sectors  Rooftop solar  Local versus national energy needs  Data needs CONCLUSIONS FOR EVALUATION QUESTION 2 The survey data suggest that tool use was small in scale but diverse. However, IAA expectations of the scale and size of tool use and users are not defined in any of its modifications or supporting documentation. Tool users varied within sectors and organization types. In addition, tool users were not clustered around just one tool, which may indicate that users have multiple needs – even within the same type of sector or organization. Tool users also tend to be multiple-tool users, which may reflect the diverse needs of stakeholders for low-emission and renewable energy goals. This is important to know when designing targets and the implementation strategy for achieving IAA goals. Many survey respondents also reported using a tool for which they had not taken a training, suggesting IAA tools are having impacts on groups that it did not target. PERFORMANCE EVALUATION OF THE USAID–DOE INTERAGENCY AGREEMENT | 18 The evaluation identified several purposes of tool use. Interviewees identified examples in which NREL supported the development of new quotas or standards in the renewable energy sector and influenced national policies in the renewable energy sector. This suggests the IAA is providing useful TA to diverse types of organizations, sectors, and users. However, the evaluation was not able to identify not the underlying causes that led to tool use. In addition, having more users does not necessarily mean a higher return on IAA investment. A policy change may have greater impact on emissions in a given country than tool use, and the number of tool users may be less instructive than the extent and types of tool use. Respondents indicated various characteristics of tools that facilitate use, including free access, open sourced, ease of use, ability to access data not available in-country, and the unique features of the tool. There are opportunities for NREL and USAID to improve tool use by addressing identified barriers such as bandwidth requirements, country-specific model requirements, software or hardware limitations, limited user capacity, and lack of awareness of the tool. FINDINGS FOR EVALUATION QUESTION 3 What are the strengths and weaknesses of the strategic planning, program coordination and management, communication, monitoring and reporting process of the IAA? What are the specific ways to make the program more efficient and effective? To answer EQ3, the evaluation team drew from the KIIs, FGDs, and process mapping workshop with NREL staff. Annex B provides additional information on these methods. STRATEGIC PLANNING Interviewees consistently highlighted several strengths of the IAA strategic planning process, including the ongoing stability of the relationship between the two partner agencies. This was in part due to a continued willingness to have difficult conversations related to their work. Additionally, interview data highlighted the partnership’s skill at targeting specific groups, its ability to keep missions engaged, and the overall strength of the scoping process, particularly with regards to capacity building needs. Weaknesses identified in the strategic planning process included the amount of time the planning process can take, which was a regular challenge for NREL in terms of losing activity momentum and excitement. Interviewees also reported that the strategic planning process can fall short of balancing the host-country government’s actual needs with USAID programming priorities for that country. Also, as noted in EQ1, NREL staff and host-country partners are sometimes unclear about how USAID funding processes work, leading assistance recipients to be impatient or have inaccurate expectations. PROGRAM COORDINATION AND MANAGEMENT Individual actors were a critical part of NREL’s success in program coordination and management. Interviewees consistently praised the USAID agreement officer’s representative (AOR, who is collocated at NREL offices in Colorado) as well as NREL management, and pointed to the indispensability of long-term NREL team members for portfolio continuity and good working relationships with host-government staff. This strength was even more pronounced when the client was struggling to meet its commitments or to understand the value of the TA. Getting partners to devote time and data is critical, and activity leads were seen as an essential part of that process. Additionally, interviewees regularly cited NREL’s reputation as an honest, apolitical broker as important for easing coordination and management challenges and for earning the trust of local partners. Despite these positive factors, coordination and management remain challenging because of the number and variety of partners involved and slow turnaround at multiple activity stages. Geographic and PERFORMANCE EVALUATION OF THE USAID–DOE INTERAGENCY AGREEMENT | 19 language barriers complicate the vital process of relationship building, and organizational requirements can involve extensive and unavoidable delays, even for fundamental documents like non-disclosure agreements and departmental circulars as well as final published website content. Coordinating the schedules of numerous individuals with diverse priorities and obligations adds a layer of need to the work in question. This creates a situation in which it is difficult to maintain momentum and urgency without also generating sufficient engagement from the local government. As with any federally funded activity, NREL staff require clearance to travel, work with other governments, and present findings online or in publications. Getting all these clearances can delay work and affect activity implementation timelines for missions. The timing required by in-country partners may not align with processes required by the U.S. government. COMMUNICATION Interviewees saw NREL’s positive internal communications environment as one of its principal assets. This includes both the cloud-based file-sharing systems that technical specialists use for exchanging and collaborating on documents and the comfortable working environment where team members commended “the ability to stick your head over someone else’s cube and ask a question.” NREL staff also praised the open conversation channels they share with USAID/Washington and mission counterparts. One team member credited the positive communications environment with contributing to successful program reporting. USAID/Washington and mission interviewees shared similar positive views about NREL‘s communications practice being proactive and effective. IPs and host-country partners also provided extensive positive feedback about NREL’s communications practice, although they were not without criticism. Even with positive views on internal communications, NREL staff recognized that communications and information channels with staff from USAID and other U.S. government agencies and outside actors were sometimes challenging. NREL staff also recognized that outwardly directed communications were more difficult than internal communications. While success stories are shared at workshops and through mission newsletters and social media, mission and IP respondents reported that they want to know more about IAA services and how to obtain NREL services directly. NREL staff also recognized that there was inadequate coordination over social media between USAID and DOE, although they saw potential for publicity and promotion of their work. NREL respondents also recognized the need for a faster clearance process for communications materials and other publications, which interviewees said lagged behind activity progress. “I honestly feel like the programs where we've had the most successes, we've been able to have that dialogue with them. So in most cases, I feel like a mission would come back and say, ‘We know who we want to support, we're hearing from the government, or based on the scoping report, we know grid integration of variable resources is a critical opportunity and we have our local on-the-ground subcontractors doing X, Y, and Z. We think there's a gap in A, B, and C. What do you guys think you can do in that space?’” - NREL respondent PERFORMANCE EVALUATION OF THE USAID–DOE INTERAGENCY AGREEMENT | 20 Government and IP interviewees saw the lack of face-to-face contact with NREL specialists over long stretches of time as a weakness. Nevertheless, IPs and host-country partners noted that host governments were often the weakest link in the communications chain and were less likely to initiate communication than IAA participants. MONITORING AND REPORTING PROCESSES USAID interviewees expressed positive views overall of NREL’s reporting practices. Some interviewees highlighted shortcomings in the monitoring of product users and use, but the IAA does not require such monitoring (except for specified indicators). NREL has met all IAA reporting requirements. The monitoring of product users and use varied among IAA activities. NREL teams discussed creating standard forms for tracking training and webinar attendees, but training and workshop data vary by activity. For example, there was no standard format for training participant lists and sign-in sheets. This lack of standardization can make it difficult for either agency to conduct self-assessments. IAA indicator data also do not provide comprehensive information about how stakeholders use NREL TA or how much their capacity might have increased. For example, for the indicator “Number of laws, policies, regulations, or standards addressing clean energy formally proposed, adopted, or implemented as supported by USG assistance,” the data do not indicate how this is achieved. There is no information on who the IAA activities affect or how policies or laws are affected, which limits NREL’s and USAID’s ability to prioritize IAA resources. Additionally, data on downloads, visits, and location of users do not exist for all tools. USAID interviewees mentioned wanting more standard M&E data and processes, such as those used in the development context, as well as data about tool use and users, more transparency about tool impacts, and more information about the kinds of stakeholder needs and requests. In addition, NREL staff prepare detailed reports on each travel assignment but NREL does not take full advantage of this rich data source to monitor IAA work because the information is not structured for analysis. NREL interviewees also indicated that teams hold robust discussions about activity progress and planning to improve implementation. The travel reports, task leads’ quarterly meetings, and standard activity indicators could provide useful material for the review of activities and self-assessment. CONCLUSIONS FOR EVALUATION QUESTION 3 The IAA’s clear strength stems from the fact that a USAID unit holds a formal agreement directly with another U.S. government entity. This arrangement can help strengthen new partnerships, encourages flexibility in achieving shared goals, and allows for adjustments when new priorities and budgets emerge. IAA partners are engaged and open to collaboration in achieving partnership goals. NREL staff find people are very positive about the work they do and the positive communications environment within the IAA strengthens planning and implementation processes. The IAA’s main weaknesses also stem from the mechanism’s flexibility, which can result in unclear expectations, roles, and responsibilities for each agency. Current work under the IAA does not appear to align with the objectives in the original agreement, yet the language in the IAA has remained unchanged. In addition, the evaluation findings suggest that USAID and NREL understand impact and track programmatic effects differently, leading to divergent impact and M&E expectations. IAA products lack explicit causal pathways, which may underestimate the full portfolio of benefits produced through IAA activities or cause limited resources to be invested in ineffective activities. Without clear impact goals or PERFORMANCE EVALUATION OF THE USAID–DOE INTERAGENCY AGREEMENT | 21 an understanding of which variables and inputs are likely to be most effective in achieving desired impacts, USAID and NREL are hindered in making decisions about further investments in tools/platforms. The lack of a clear theory of change for the partnership further augments this issue. Without an explicit testable impact model, the IAA partners cannot effectively measure over time which investments lead to desired in-country effects. Current IAA monitoring practices are based on USAID indicators and are of limited utility for analyzing effectiveness, tracking change over time, or measuring tool use. For example, NREL trip reports include detailed information about tool use and implementation, and the effects of TA, but the report format requires significant transformation of the data for any mining or analysis. In addition, while some survey respondents reported using tools for which they had not taken an NREL training, current IAA M&E protocols do not capture spillover effects and thus hinder NREL’s ability to identify additional users, share new tools, and expand existing tool/platform use. While USAID indicators used to calculate mitigation impacts may be based on reasonable assumptions, these assumptions need to be examined to ensure clear pathways to actual mitigation. For example, the installation of 16 kW of PV electrical generation is regarded as displacing a comparable amount of greenhouse gas emissions and is therefore calculated as reducing more than 410 tons of CO2 over a 10- year period. In most cases, however, indicators like, “Number of institutions with improved capacity to address climate change issues” or “Number of laws, policies, strategies, plans, agreements, or regulations addressing climate change and/or biodiversity conservation formally proposed, adopted, or implemented” describe positive changes in capability or effort but not in actual mitigation results. Expectations of use and user numbers also may be misplaced to cause changes in country-wide emission levels. PERFORMANCE EVALUATION OF THE USAID–DOE INTERAGENCY AGREEMENT | 22 RECOMMENDATIONS EVALUATION QUESTION 1 – MEETING NEEDS AND IMPROVING BUY-IN  E3/GCC should keep the IAA AOR in Colorado. Stationing the AOR at NREL in Colorado is successful and should be maintained. The IAA empowers the AOR to make changes, thereby reducing possible delays for clearance. Keeping the AOR at NREL will facilitate coordination of activities with mission needs, permit the partnership to be agile, and provide NREL teams with immediate, direct access to USAID.  E3/GCC and NREL should co-locate NREL staff at USAID/Washington and missions. NREL’s technical expertise can be accessed faster and easier if the experts are on site at missions that the IAA has supported. Experts’ presence in USAID will have similar effects to those gained from having the AOR at NREL in Colorado: improving coordination of technical experts in country, raising awareness in USAID of DOE expertise, and providing expertise for energy and power projects.  E3/GCC and NREL should increase exchange opportunities to send NREL staff to the field for extended trips, and for in-country partners to visit NREL in Colorado or other national laboratories. Time and distance make it difficult to leverage opportunities for IAA assistance. Respondents felt they gained significant knowledge about renewable technology through the in￾person TA NREL provided to missions and partners.  USAID should promote awareness within the Agency of this IAA and similar partnerships. USAID should revise its communications strategy for informing missions about IAAs. The USAID Mission Directors Conference might be a good opportunity to discuss this IAA and its services. A joint presentation with other IAAs could showcase where and how missions can access technical expertise from DOE, the Centers for Disease Control and Prevention, the U.S. Department of Agriculture, the National Aeronautics and Space Administration, the U.S. Forest Service, and other partners.  USAID and NREL should co-fund an American Association for the Advancement of Science (AAAS) fellowship abroad. Missions including those that bought into the IAA, are in critical regions or need relevant TA, or have with limited technical capacity in the energy sector could benefit from in-country NREL expertise. USAID and NREL could use the AAAS vetting process to identify scholars for the fellowships. Their training should be different from training in the general overseas program, including a requirement to spend time at NREL and USAID/Washington and take relevant USAID trainings before they travel abroad.  NREL and E3/GCC should promote new kinds of IAA programming. IAA activities have facilitated alternative studies and work in energy, such as smart infrastructure. This could be a continued area of opportunity for the IAA. Other activities with potential for IAA assistance include using NREL expertise in transportation to help countries improve efficiency through sensors on public buses. Countries sometimes take the advice of NREL teams on what activities to pursue; suggestions for activities beyond what a country has requested could be helpful when host-country governments are not aware of the types of support NREL can provide. One IP interviewee recommended that NREL provide online mentoring so partners can have more continuous access to NREL experts. EVALUATION QUESTION 2 – TECHNICAL PLATFORM USE  E3/GCC and NREL should rethink training on NREL tools. NREL should provide training on several tools during the same session, hosting diverse tool experts in country. The spillover effects revealed in the evaluation indicate that the target users may be more diverse than assumed. Training may also be a way to attract new types of users and new ways to use tools. PERFORMANCE EVALUATION OF THE USAID–DOE INTERAGENCY AGREEMENT | 23  E3/GCC and NREL should pilot new ways to understand IAA benefits. A network analysis of trainees could inform training needs. Access to and awareness of resources often correlates with an individual’s network; the IAA could explore how stakeholder participation in training leads to changes in stakeholder networks, and thus changes in a sector’s capacity. This is more significant for women, who tend to have more limited networks and fewer opportunities to expand their networks.7  NREL should encourage opportunities for minorities and women to participate in training. Training can have more female participants when childcare is offered at the training venue. People with families, disabilities, or special requirements may find online training easier to attend, and these trainings could be tailored to meet such users’ needs, thereby increasing usage and potentially the impacts of IAA tools. In some countries, women-only training might also be appropriate.  NREL should standardize its forms for IAA trainings and mission requests and needs. NREL should work with monitoring, evaluation, and learning (MEL) experts to standardize training forms, such as sign-in sheets. This would be a first step in creating a tracking system for mission requests and needs so demand trends could be analyzed and training resources could be allocated more effectively.  NREL and E3/GCC should consider creating a new position or enlisting support for a user experience specialist to prioritize the user experience. As more complex online platforms become available (including toolkits, resources, and reports), IAA partners will have additional web-based monitoring needs. A user experience specialist could track how online visitors experience the tools and could help design a more comprehensive monitoring strategy to feed into the MEL platform.  NREL should plan for the needs of future tool users. To explore the future needs of users for the online provision of IAA products and services, NREL should consider country bandwidth requirements, adjusting modeling applications to be used for training, and other emerging trends in virtual modeling. The gaming sector could help create options for stakeholders to test applications and consequences at various scales, like some of the models NREL is testing for its U.S.-based work. EVALUATION QUESTION 3 – STRATEGIC PLANNING, COORDINATION, COMMUNICATION, AND M&E  E3/GCC and NREL should jointly create a new MEL strategy for the IAA. E3/GCC and NREL should host a workshop to revise the IAA and create a new causal pathway reflective of partnership goals. The workshop should include a review of each partner’s assumptions and expectations, establish a new theory of change, and identify which metrics would be most appropriate for long-term data capture. The strategy should include language defining users and the targeted use; with improved MEL practices, the IAA will have better evidence to inform the prioritization of resources. Without an explicit, testable model, the IAA partners cannot measure which investments lead to the desired in-country effects over time and have differing assumptions about what it means to be a successful partnership.  E3/GCC and NREL should define metrics in greater detail. The IAA should include additional guidance on USAID requirements and expectations about metrics for impact, theory of change, and ways to monitor development work globally. The IAA currently does not include a theory of change guiding partnership goals or explicit causal pathways to achieve these 7 For example http://www.unesco.org/new/en/natural-sciences/priority-areas/gender-and-science/supporting-women￾scientists/strengthening-networks-of-women-scientists/; https://journals.sagepub.com/doi/10.1177/0162243904265895#articleShareContainer PERFORMANCE EVALUATION OF THE USAID–DOE INTERAGENCY AGREEMENT | 24 objectives. E3/GCC and NREL should also consider revising the IAA strategy for how the partners select and set country targets, which would address some of the timing issues around activity implementation.  E3/GCC and NREL should incorporate new methods of measuring and tracking IAA work. Methods of monitoring should reflect the robust metrics that NREL uses for its U.S.- based projects. E3/GCC and NREL may want to consider developing an online dashboard so participants and teams can easily submit information including tracking website visits and users. The partners should also incorporate post-trip reporting documents into the formal IAA reporting process.  E3/GCC and NREL should train NREL teams on MEL strategies for development projects. USAID and organizations such as The Evaluators’ Institute offer workshops and courses on MEL. With increased capacity, NREL teams can ensure they have more robust data collection practices to track and assess the impact of their work.  NREL should develop a strategic communications plan early in an activity, so the whole team considers communications from the outset.  In the next IAA modification, E3/GCC should consider explicitly defining key terms (e.g., impact) when these might differ by agency, assigning each partner tasks and roles that match their expertise, and clarifying any other areas where agency processes might differ and lead to misaligned expectations. These can include addressing ways to better facilitate travel approvals, data access, and other approval processes.  E3/GCC should align this IAA with the strengths and lessons learned from other IAAs. E3/GCC should host a workshop in which other IAA AORs and their partners discuss the strengths and weaknesses of IAAs and their perceptions about impact, especially when partnering with science agencies. This would help align expectations for IAAs. Lessons can also be shared from evaluations of IAAs and similar partnerships. In the case of SERVIR, a USAID￾NASA IAA, the two participating agencies measured success differently and the IAA did not explicitly clarify these differences. USAID changed its approach and moved the responsibility for tasks such as M&E, online platform maintenance and web trafficking, and communication and information dissemination to a third party contractor. This removed the burden on NASA to carry out tasks for which the agency had different definitions (for example, impact at NASA is focused on process and not just outcomes). Many IAAs have encountered obstacles and opportunities similar to those seen in this IAA. By examining these lessons learned, USAID could identify new ways to make science-development U.S. government partnerships more effective. PERFORMANCE EVALUATION OF THE USAID–DOE INTERAGENCY AGREEMENT | 25 ANNEX A: EVALUATION STATEMENT OF WORK PERFORMANCE EVALUATION OF THE USAID-DEPARTMENT OF ENERGY INTERAGENCY AGREEMENT October 7, 2017 1. INTRODUCTION A) Identifying Information 1. Program: USAID-Department of Energy Interagency Agreement 2. Award Number: AEG-P-00-09-00003-00 3. Award Dates: 2009-2017 4. Funding: $28,377,679 5: Implementer: Department of Energy The U.S. Agency for International Development’s Bureau of Economic Growth, Education and Environment (USAID/E3) seeks to undertake a performance evaluation of the USAID-Department of Energy Interagency Agreement (IAA). The IAA is implemented by the Department of Energy (DOE), including the National Renewable Energy Laboratory (NREL). The IAA was awarded in 2009 to the DOE, and 10 agreement modifications have been made to date. In 2017, a follow on IAA was awarded to the DOE lab in Golden directly (USAID-NREL IAA). B) Evaluation Information This evaluation will be conducted under the E3 Analytics and Evaluation (E3A) award. The USAID Activity Manager for this evaluation will be Kate Faulhaber. The Agreement Officer’s Representative (AOR) for the DOE award is Jen Leisch, who will be acting as the Analysis, Innovation and Demonstration (AID) technical point of contact for the evaluation. The Contract Officer’s Representative (COR) for the E3A award is Bhavani Pathak. 2. ACTIVITY BACKGROUND Guided by local needs and priorities, USAID partners with countries to improve air quality, increase energy access and security, lower energy costs, spur economic growth and job creation, and reduce greenhouse gas (GHG) emissions through the deployment of renewable energy and energy efficiency solutions. Recognizing the need for comprehensive technical assistance to support low emissions growth across the globe, the U.S. government has established a network of U.S. experts, agencies, and technical institutions to provide both targeted and cross-cutting technical assistance to developing country partners in the clean energy and landscapes sectors. The underlying goal of this effort is to ensure that the best U.S. technical expertise is being utilized to support global efforts to combat climate change. USAID works with the U.S. Department of Energy’s Office of Policy and International Affairs and the National Renewable Energy Laboratory to support the global effort to combat global climate change with a particular focus on the U.S. government’s program for Enhancing Capacity for Low Emission Development Strategies (EC-LEDS). With support by USAID, a network of National Lab technical experts was established to provide analytic and technical services to advance the use of clean energy PERFORMANCE EVALUATION OF THE USAID–DOE INTERAGENCY AGREEMENT | 26 technologies in developing countries. This network provides technical assistance to USAID Missions and their partner countries in the analysis, design and implementation of clean energy greenhouse gas mitigation initiatives. A) Technical Expertise The Department of Energy’s National Renewable Energy Laboratory was selected to partner with USAID due to its technical expertise in the fields of renewable energy, energy efficiency, geospatial analysis, and data analysis. Specifically, NREL's scientists and analysts support clean energy initiatives around the world to advance three key US Government strategic objectives: economic development, energy security, and environmental protection at home and abroad. NREL's international work includes three primary areas of expertise: research and development, energy analysis, and deployment and technology transfer. NREL collaborates with technical institutions and governments around the world to strengthen clean energy markets by conducting resource and technology assessments, sharing policy best practices, fostering business and investment partnerships, and building capacity through training and technical support. NREL also tests and analyzes energy efficiency and renewable energy systems, shares performance data and simulation models, and conducts fundamental research cooperation. B) Lab Consortia NREL also contributes technical expertise by providing access to a wide consortium of domestic and international experts. While the daily activities and overall strategic coordination of the IAA is carried out by a dedicated team of professionals at USAID and NREL, sector experts in data analysis, geospatial information, and other relevant fields are also able to contribute to specific tasks and deliverables. NREL collaborates with hundreds of partners across the U.S. and in over 30 countries to commercialize and deploy renewable energy and energy efficiency technologies from industry, academia, nonprofit organizations, federal agencies, and international institutions; as well as state, local, and tribal governments. 8 Overall, NREL has hundreds of active partnerships in both the private and public energy sectors that it can leverage for its agreement with USAID. As a government organization and part of the Department of Energy, NREL can access expertise in other DOE National Laboratories and counterparts in partner country governments. These relationships allow NREL to access on-demand technical assistance to respond to the needs of USAID offices and missions working in partner countries with rapidly evolving energy sectors. C)EC-LEDS Program and Evaluation Low emission development strategies (LEDS)9 are national development plans that promote sustainable social and economic development while reducing greenhouse gas emissions over the medium to long term. From 2009 to 2016, Enhancing Capacity for Low Emission Development Strategies (EC-LEDS) was the flagship U.S. government-led effort which assisted countries in developing and implementing LEDS. The program enhanced partner country efforts by: (1) providing targeted technical assistance; and, (2) 8 Transforming Energy Through Science. NREL. http://www.nrel.gov/docs/fy16osti/66385.pdf 9 An evaluation of the EC-LEDS program was conducted in 2016, and includes contributions of NREL in its analysis. Specific issues, such as the design of the EC-LEDS program, the scoping trips, and the whole of government approach have been addressed in the EC-LEDS evaluation, and will not be addressed directly in this evaluation. PERFORMANCE EVALUATION OF THE USAID–DOE INTERAGENCY AGREEMENT | 27 building a shared global knowledge base on LEDS. Through the EC-LEDS program, the U.S. supported more than 20 partner countries to develop or build on existing country programs, plans, and policies that support the unique national development goals of each country. It also complemented efforts of other international donors and organizations to support LEDS. In 2016, a performance evaluation was conducted of the entire EC-LEDS program. As part of this evaluation, performance metrics (greenhouse gas emission reductions, investment mobilized, etc.) and other outcomes were reviewed for each mission program through either a site visit or desk review. The effectiveness of the USG interagency approach of the EC-LEDS program, which includes DOE, USAID, the Department of State, USDA, and USFS was also reviewed as part of the EC-LEDS evaluation. Therefore, questions regarding interagency coordination or mission specific outcomes and metrics have been excluded from this evaluation to avoid a duplication of efforts. The EC-LEDS program was phased out at the beginning of FY17, but low emission development remains an important part of USAID’s development approach. Clean, low cost energy has significant impacts on economic growth, environmental quality, and air pollution, and remains a priority for our partner countries. A follow on agreement, the USAID-NREL IAA, began in 2017 and has continued to provide technical assistance in support of energy and development priorities. 3. ACTIVITY TASKS The USAID - DOE Interagency Agreement initially started with $500,000 of funding in 2009. For additional incremental funding, USAID developed an agreement Amendment or Modification that specified the additional level of funding and the work plan associated with those funds. The majority of funding for the IAA comes from the E3 Bureau for activities such as overall program management, the development of technical platforms and the support of specific mission programs. Toward the beginning of the IAA, a significant amount of effort was placed on planning and scoping specific country needs for the EC-LEDS program. This led to country specific technical assistance (TA) and then to tools to support decision-making and investment. Tools and lessons learned were gathered into cross-cutting technical platforms that provide the tools, resources, training, and access to experts on technical topics available to all partner countries. Another feature of the IAA is that it allows for buy￾ins from missions and other USAID offices or Bureaus. This allows missions or offices to provide additional funding for NREL to provide technical assistance services in country that meet specific needs and build off of existing support. USAID and DOE have agreed to a number of modifications to the original agreement. There have been 10 modifications to date. Each modification contains information regarding budget allocations for specific tasks, as well as a work plan, which includes detailed task descriptions and deliverables. These work plans are developed between both USAID and NREL, as well as any USAID missions that would like to obligate funds to the agreement. As information and priorities change, work plan implementation may be updated throughout the year, with USAID approval. Actual budget expenditure is tracked by NREL, and the actual resource allocation percentages through May 2017 are indicated for each task below. A) EC-LEDS Secretariat, Communications, Outreach (20%) Around 20% of the total IAA costs to date has been allocated to activities that support overall program management, monitoring and reporting, and external communications. This includes the initial EC-LEDS scoping trips and methodology design, as well as ongoing coordination, communication, and leadership PERFORMANCE EVALUATION OF THE USAID–DOE INTERAGENCY AGREEMENT | 28 activities. In addition, it includes all activities associated with monitoring and reporting and managing the Agency Priority Goal (APG) database for the entire program. EC-LEDS Secretariat – As part of the implementation of the IAA, NREL serves as the Technical Secretariat for the EC-LEDS program. The Technical Secretariat seeks to ensure senior level leadership and coordination of program activities. This task includes dedicated NREL staff supporting the implementation of the EC-LEDS program and management of NREL’s technical assistance to the program. It also includes program management tasks such as overseeing the overall budget, staffing, schedule, buy-ins, and subcontracts. In addition, the Secretariat has been central to coordinating the inter-agency collaboration within the EC-LEDS program. This includes the coordination of meetings of interagency members, NREL staff, and USAID missions. In addition, NREL coordinates monthly communications amongst the interagency partners via the EC-LEDS monthly highlights newsletter. NREL also contributes to USAID’s strategic vision for providing effective mitigation support and technical assistance to partner countries. Communications – This task is designed to facilitate learning across EC-LEDS countries including missions, Washington, and other stakeholders. It includes activities such as the development of success stories, infographics, fact sheets, the EC-LEDS website, and the monthly bulletin. Monitoring and Reporting – This includes the development of annual reports including reporting on custom and standard indicators and Agency Priority Goal (APG) milestones. It also includes the development of a monitoring framework and data collection database. NREL tracks IAA results using applicable Global Climate Change standard and custom indicators and provides this data to USAID via annual reports. Global Outreach – This includes a variety of targeted activities including supporting the Conference of Parties (COP) side events and working with the LEDS Global Partnership (LEDS GP). B) E3 Support for Missions (24%) and Mission Buy-Ins (41%) Under the IAA, experts in the NREL consortia provided technical assistance directly to target beneficiaries and mission based programs and activities. Work plans for these activities were agreed upon between the mission, E3 and NREL. Countries that received E3 pilot direct technical assistance at the start of the EC-LEDS program were Bangladesh, India, Vietnam, Gabon, Colombia, and Mexico. Additionally, missions had the opportunity to use their own funds to “buy in” to the IAA. These buy-ins are included in the Amendment and budget documentation provided, and the activities they include are consistent with the overall tasks associated with the original award. The table below lists the cumulative buy-ins by mission. Table 1 Mission Buy-Ins Key Buy-Ins by Mission Total Mexico $1,960,000 Haiti $1,827,000 Asia Regional $1,774,000 India $1,050,000 Bangladesh $960,000 Vietnam $850,000 Philippines $630,500 Kenya $445,000 PERFORMANCE EVALUATION OF THE USAID–DOE INTERAGENCY AGREEMENT | 29 Mission funded tasks may complement or expand on country specific activities funded by E3, complement their in-country programs, and leverage global activities, such as the development of technical platforms. Examples of mission funded tasks include:  Providing technical assistance to the Government of Mexico as it considers, analyzes, and plans a national carbon market.  Providing technical assistance to the Mexican Ministry of Energy (SENER) to meet installed capacity and minimum clean energy generation goals, implement electricity sector reform, and increase participation of private sector in the generation of electricity.  Supporting the US-India Sustainable Growth Working Group through development of and training for the Enterprise Geospatial Toolkit (eGsT) and eGsT Economic Potential Tool, as well as the Renewable Energy Zones and Geospatial Grid Integration technical platforms.  Facilitating a visiting scholar exchange program, where Indian climate and energy modelers visit participating national labs to learn about various technical issues and approaches in detail, in order to improve capabilities of respective Indian and U.S. models.  Supporting the USAID Clean Power Asia activity to expand and accelerate renewable energy deployment in Asia, particularly in the four Lower Mekong countries (Cambodia, Lao, Thailand, and Vietnam), by standardizing and harmonizing data protocols and access platforms. This section may also include subcontracts to in-country institutions or experts for EC-LEDS program coordination. Subcontracts are consistent with objectives of the relevant USAID Mission. Where needed, this fills staffing gaps for program coordination at USAID Missions, and allows for local USG engagement to remain focused on higher-level coordination and communication needs. This also includes subcontracts with leading international NGOs and technical institutions to bring unique skills, expertise, and networks to complement EC-LEDS program activities. C) Technical Platforms (13%) The IAA supports the development of a variety of tools and technical platforms online and/or in desktop format. The technical platforms include analytical tools that can be used by stakeholders or in partnership with U.S. DOE laboratories; resources, information and best practices; access to expert technical support, and capacity building through trainings, workshops, and webinars. These technical platforms support the development of renewable energy resources by targeting specific technical or analytical barriers. The primary global technical platforms supported by USAID and NREL are described below. In addition to developing the technical platforms, NREL provides training on the technical platform content to both internal and external USAID stakeholders. This training includes workshops, webinars, and custom training 1) for policy-makers and decision makers, and 2) for technical staff and practitioners. Other training and outreach approaches include developing enhanced website and outreach material, improved coordination with other database organizers or data suppliers, and in￾person or virtual participation in USAID staff trainings. Technical Platforms: Greening the Grid (GtG) offers information, guidance materials, and technical assistance to support developing countries in significantly scaling up the amount of variable renewable energy connected to the electricity grid. Resources provide 1) concise and comprehensive overviews of emerging practices for addressing grid-integration challenges through policy, market, and regulatory mechanisms and 2) guidance on applying these mechanisms to develop robust grid-integration road maps, grid integration PERFORMANCE EVALUATION OF THE USAID–DOE INTERAGENCY AGREEMENT | 30 modeling exercises, and grid operation and planning decision support. Greening the Grid also facilitates direct technical assistance tailored to the unique power system characteristics and priorities in each partner country. Integrating significant renewable energy to the grid requires new transmission to access the best wind and solar resources and accommodate generation from large projects. The renewable energy zone (REZ) approach provides a policy framework for planning new transmission to encourage utility-scale renewable energy development in areas that are most cost-effective and where risk is adequately low to attract private sector investment. By facilitating transmission planning in advance of generator approval and construction, the REZ approach helps to speed the deployment and utilization of renewable energy while minimizing impacts of variable renewable energy on the stability of the power system. The Renewable Energy Explorer is a technical platform that provides renewable energy data, analytical tools, and technical assistance to developers, policymakers, and decision makers in developing countries. It enables them to make meaningful decisions that support low-emission development and ultimately reduce greenhouse gas emissions. The platform is online and provides resources, information, tools and training through webinars. The Renewable Energy Data Explorer is the flagship tool of the Renewable Energy Explorer technical platforms, and facilitates renewable energy decision-making, investment, and deployment through a dynamic, online analytical tool. The RE Data Explorer is a web-based analytical tool that provides an intuitive, user-friendly interface to visualize renewable energy potential. It enables users to analyze solar, wind, and biomass potential under a variety of scenarios to inform high-level prospecting and integrated planning. It is currently available for 11 countries and more country tools are under development. The renewable energy data and access to the RE Data Explorer is available at the Renewable Energy Explorer website. Geospatial Toolkits are desk-based, open-source software applications that support integrated renewable energy planning with an intuitive map-based interface and analysis features. They are available for more than 20 countries. The geospatial toolkits are available for download from the National Renewable Energy Laboratory website. These tools were migrated to a web￾based platform and rebranded as the Renewable Energy Explorer in early 2017. The Development Impact Assessment (DIA) Toolkit helps practitioners and policymakers find tools and approaches to assess specific impacts of LEDS actions. DIA supports the evaluation and communication of links between low-emission development actions and a country's social, economic, and environmental development goals, such as improved public health, job creation and energy access. Robust development impact assessment processes inform decision-making and catalyze implementation of low emission development strategies (LEDS) and Nationally Determined Contributions. These tools and impact analyses can feed into broader DIA processes described on the website. The International Jobs and Economic Development Impacts (I-JEDI) Tool, developed by the National Renewable Energy Laboratory, supports practitioners and decision-makers in assessing job and other economic development impacts of LEDS actions in the energy sector. Users can input country￾specific data or use default international data included in the spreadsheet tool to support LEDS analysis. Grid-connected, Distributed Photovoltaic (DPV) technologies are increasingly permeating power markets around the world, due in part to the low cost of photovoltaic system hardware and scalability for deployment. This project supports the USAID Energy Division in identifying, developing and executing opportunities to assist in the formulation of DPV programs in developing countries as part of these countries strategies to improve energy reliability while also delivering educational content to PERFORMANCE EVALUATION OF THE USAID–DOE INTERAGENCY AGREEMENT | 31 USAID staff/contractors and in-country stakeholders. To date, the DPV technical platform has conducted several in-country pilots of the technology in order to demonstrate foundational codes, standards and interconnection processes; modeling and mitigating impacts on the electric distribution system; distributed generation tariff design; and more. The DPV technical platform is used to supplement in country technical assistance. 4. EVALUATION PURPOSE The USAID-DOE Interagency Agreement was initiated in 2009 as part of USAID’s increased focus on climate change adaptation and mitigation challenges around the world, and the launch of the U.S. government’s EC-LEDS program. In the eight years since the agreement went into effect, there have been significant changes to the global political and technical landscape. Many countries have signed the Paris Agreement10, submitted Intended Nationally Determined Contributions11, and experienced changes to political leadership and priorities. Technical and market advances include the dramatic decrease in the cost of renewables12, and the share of new renewable energy projects relative to fossil fuels13. Many other significant changes have occurred in financial markets, available technologies, and economies around the world. As USAID is ending the IAA with DOE and has transitioned to a new IAA directly with the NREL lab in Golden, it would like to conduct a stock-taking exercise to inform future work plans and overall IAA strategic management. As such, USAID and NREL managers are the primary audience for this evaluation. While the evaluation will be reviewing activities started under the USAID-DOE IAA, many of the core stakeholders and some of the current activities will continue under the new USAID-NREL IAA. The purpose of the evaluation is to help USAID and NREL management identify the tasks that should be prioritized going forward and identify what may no longer be relevant, given the current technical and political landscape discussed above and evolving mission priorities. It will also provide a review of IAA management, to improve performance and efficiency under the new IAA. To this end, while the evaluation should address past performance, the evaluation team must also provide forward-looking recommendations on possible strategies for improving the effectiveness and organization of the follow-on award. This evaluation will draw primarily on existing documentation and responses from surveys and interviews targeting specific stakeholder groups related to the IAA, such as USAID and NREL staff, and other beneficiaries in order to inform these questions. 5. EVALUATION QUESTIONS To guide this evaluation, USAID has identified the following evaluation questions. To support the evaluation team, further description of each evaluation question is provided below along with sub questions that convey areas of interest. This information is provided to help the evaluation team focus on the aspects of the evaluation questions that are most useful to USAID, to inform the development of interview and survey questions, and to inform the issues that the evaluation findings and recommendations should address. The sub questions do not need to be answered separately, but should be considered in answering the three evaluation questions. In its Evaluation Design Proposal, the evaluation team will take the area of interest into account and propose how it intends to define and 10 At the Paris climate conference (COP21) in December 2015, 195 countries adopted the first-ever universal, legally binding global climate deal. http://unfccc.int/paris_agreement/items/9485.php 11 INDCs are the primary means for governments to communicate internationally the steps they will take to address climate change in their own countries. http://www.wri.org/indc-definition 12 Bloomberg New Energy Finance. New Energy Outlook, 2017. https://about.bnef.com/new-energy-outlook/ 13 Ibid PERFORMANCE EVALUATION OF THE USAID–DOE INTERAGENCY AGREEMENT | 32 operationalize key terms and concepts for each evaluation question, for approval by the USAID Activity Manager prior to the initiation of data collection. EQ1. To what extent has the USAID-DOE IAA met the needs of missions, and what specific opportunities exist to improve the buy-in program? EQ2. To what extent are the clean energy technical platforms being used effectively by their target audiences for their intended purpose? EQ3. What are the strengths and weaknesses of the strategic planning, program coordination and management, communications, monitoring and reporting processes of the IAA? What are specific ways to make the program more efficient and effective? EQ 1) To what extent has the USAID-DOE IAA met the needs of missions, and what specific opportunities exist to improve the buy-in program? This question seeks to explore the benefits that missions and host country counterparts derived from the USAID-DOE IAA and hope to derive from the new USAID-NREL IAA. This question should also explore mission and host country stakeholder feedback on quality of assistance and ease of working with NREL. It should look for both strengths and weaknesses and suggest recommendations for improvement. This question will consider how well NREL has leveraged its existing network in support of the goals of the IAA. This network consists of both domestic and international relationships, and includes U.S. and partner government, private sector, academia, and other stakeholders. NREL and other DOE experts provide a variety of TA to USAID mission staff (e.g., program design), implementing partners, and local government and businesses. The delivery models for this assistance can be in person or remote (virtual), and require significant coordination and communication across all stakeholders. Areas of interest for EQ1 include: ● How, and to what extent, has this mechanism allowed USAID Mission and partner country access to NREL expertise and cutting edge analysis, research, and deployment? ● In what specific ways has this mechanism complemented capabilities of local contractors, partners, or other contract mechanisms used by the field Missions? ● How can coordination between E3, NREL, Mission and local implementing partners be improved? ● What types of technical expertise or assistance from NREL and other DOE Labs have been of highest value to partner governments or other Mission stakeholders? What types of technical assistant delivery models have been most effective? ● In what ways have lessons been transferred to the broader USG low emissions development portfolio and what have the benefits been? EQ 2) To what extent are the clean energy technical platforms being used effectively by their target audiences? This question seeks to explore the technical platform user and development portion of the IAA. USAID invests a considerable amount into various technical platforms. This evaluation seeks to assess the extent to which these tools are being used, by whom, and for what. See Section III C above for a definition of technical platforms and a description of each. PERFORMANCE EVALUATION OF THE USAID–DOE INTERAGENCY AGREEMENT | 33 This question will rely on existing web analytics data and feedback from NREL staff familiar with the tools, as well as tool users and other stakeholders. It should focus on trends and usage patterns, and suggest specific areas or actions for improvement. Areas of interest for EQ2 include:  How are these technical platforms and tools being used by target audiences and where/why might they be underutilized?  What were the primary drivers that informed the types of technical platforms and other resources that have been developed? How are these drivers and user needs evolving?  What types of policies, programs, and country capacity development are being informed or advanced through these technical platforms and related technical assistance?  Do host country counterparts credit the platforms, tools or data derived from them with helping to drive policy or regulatory change, improve planning processes, or drive investment decisions?  To what extent have host country counterparts, or “owners”, of the tools committed to the long term operational costs required to keep model output current? EQ 3) What are the strengths and weaknesses of the strategic planning, program coordination and management, communications, monitoring and reporting processes of the IAA? What are specific ways to make the program more efficient and effective? This question will look at the USAID/E3 and USAID/Mission relationship with NREL regarding overall strategic planning and management, staff and task organization, work planning and resource allocation, monitoring and reporting, communications, and communications products. It may also consider how missions view or use communications products such as newsletters, cases studies, monthly coordination calls, or web based reference materials. The current technical and strategic approach relies primarily on strategic guidance of E3 and NREL sector experts and interactions with USAID missions and partner governments, as well as consultations with USAID bureaus. This questions will rely on feedback from USAID and NREL stakeholders, but will also rely on a review of existing documentation, and a consideration of project design and implementation best practices (the USAID “Program Cycle”). The goal of this question is to improve the performance of the IAA by providing feedback and recommendations on strategic planning and implementation. It should also take into account program design and trends. It seeks to identify any changes that should be made to the operational structure of the agreement to maximize efficiency and effectiveness. The key stakeholders contributing to this question are USAID/W, NREL, and key USAID mission staff and partners. Areas of interest for EQ3 include:  Does NREL have the right number of staff to meet the needs of the IAA, and do staff have the right qualifications?  Does the broader NREL team of experts feel that they have clear direction when brought on to tasks and do they feel that their work has impact?  Does NREL have a robust strategy/theory of change for how TA provided in the work plan will contribute to on the ground mitigation results?  Do the results being monitoring under the IAA reflect the overall strategy and goals of the PERFORMANCE EVALUATION OF THE USAID–DOE INTERAGENCY AGREEMENT | 34 agreement?  What barriers does NREL face in implementing their work plan on time and as planned, that USAID can address?  How, and to what extent, has this mechanism helped to promote consistency and continuity of program technical support under the EC-LEDS program?  How well does NREL coordinate with other USG programs and agencies working in the same sector and country?  Are gender issues sufficiently addressed in work planning or implementation? 6. EVALUATION DESIGN AND METHODOLOGY The evaluation team is required to gather information on the program, analyze that information, and provide answers to the key evaluation questions. The main sources of information for the evaluation will be the document review, key informant interviews, surveys, a site visit, and consultations with relevant stakeholders. The evaluation team will work in conjunction with USAID and NREL to plan and implement the proposed evaluation. The evaluation team and USAID/E3 will be involved with design, planning, and logistics, but the evaluation team is expected to advise on approach and methods, and have responsibility for key evaluation duties and deliverables. The evaluation team should have no direct association to the activity or the implementing partner. While the evaluation should address past performance, the evaluation team must also provide forward￾looking recommendations on possible strategies for improving the effectiveness and organization of the follow-on award. Suggested areas to be more focused or expanded will be useful in order to achieve expected outcomes. The evaluation team must also address gender aspects of both the technical assistance provided by the DOE and its partners and of the activity beneficiaries, and consider the role of gender equality in clean energy. A) Data Collection and Analysis Methods Data requirements, collection methods, and required analyses will be determined by the evaluation team and USAID under the direction of the evaluation team lead. Consistent with ADS 203.3.1.6 guidance on evaluation methodologies, a combination of quantitative and qualitative methods in data collection and analysis must be employed by the evaluation team. Details on final datasets, collection methods (including interview questions, questionnaire form and key informants to be interviewed), and analytical framework(s) will be approved by the Activity Manager and Technical Lead as part of the evaluation design process. The evaluation team must begin its evaluation design process with a desk study of existing documents and information, followed by consultations with key stakeholders at USAID and NREL to further refine the implementation approach. This will be followed by interviews and surveys of partners, stakeholders, and beneficiaries in the program’s target countries. B) Stakeholder Groups The primary evaluation data sources will be existing activity documentation, including IAA work plans and online materials, and stakeholder surveys and interviews. For this evaluation, there are several stakeholder groups, each of which has different interactions and roles related to the implementation of the IAA. The evaluation team will develop data collection instruments for each question by reviewing the supporting documentation provided and targeting questions to specific stakeholder groups. PERFORMANCE EVALUATION OF THE USAID–DOE INTERAGENCY AGREEMENT | 35 The six stakeholder groups to be considered are the following: USAID/W Core Staff includes the AOR and members of the clean energy and energy teams in the Washington office E3 Bureau or regional offices that are familiar with implementation of the IAA. NREL Core Staff includes the NREL staff and technical experts who work exclusively or predominately on the IAA. This team is based in Golden, CO. USAID Mission Staff includes all USAID staff at field missions who have worked with NREL on any activities. Due to rotations of Foreign Service Officers to various missions, this list will prioritize current or recent assignments. Mission Partners includes locally based implementing partners, academics, government officials, private sector partners, or any other partner country stakeholders familiar enough with USAID or DOE activities or personnel to contribute to the evaluation. NREL Technical Experts includes NREL or DOE technical staff that have worked on deliverables under the IAA. The majority of these staff are based in Golden, CO. Tool Users includes anyone that has attended a workshop, webinar, or other training or outreach event related to one of the tools or toolkits developed under the IAA. The list of stakeholders and contact information has been generated from sign-in sheets and other documentation collected by NREL. The following table indicates the estimated number of stakeholders in each group, and the estimated target number of responses to the evaluation. The evaluation questions and data collection methods for each stakeholder group have also been identified, although not every method is applicable to each question. Depending on the final evaluation design, applicable groups for some questions and methods may be updated. Actual number of identified stakeholders and associated target responses will be included in the evaluation design. Table 2 Estimated Stakeholder Groups for Evaluation Stakeholder Group Total Target Evaluation Question(s) Data Collection Method USAID Washington Staff 10 8 EQ1-3 Interview NREL Core Staff 3 3 EQ1-3 Interview USAID Mission Staff 20 12 EQ1, EQ3 Interview Mission Partners 20 12 EQ1, EQ3 Interview NREL Technical Staff 30 15 EQ2-3 Focus Group, Interview Tool Users 500+ 100 EQ2 Survey C) Site Visit It is anticipated that there will be one site visit as part of the data collection process. In order to efficiently and effectively conduct key informant interviews and focus group interviews with NREL staff, the two person evaluation team may travel to Golden, Colorado for a week to meet with the AOR, PERFORMANCE EVALUATION OF THE USAID–DOE INTERAGENCY AGREEMENT | 36 core NREL staff, and other relevant technical experts. The visit will begin with introductions and an in￾brief with core NREL staff, and end with a short out-brief that discusses preliminary findings and impressions. The desired timeframe for the site visit is in the Schedule, below, but exact days may vary depending on availability. The evaluation team is responsible for making their own hotel, air travel, and local transportation arrangements in accordance with U.S. requirements for allowable carriers and per diems. The evaluation team is also responsible for coordinating the logistics for arranging and tracking the interview schedule for the week, however USAID and NREL can facilitate introductions and meeting spaces as necessary. D) Existing Information Sources Existing documentation, such as background information, work plans, existing performance information, and stakeholder contact lists will be provided by or around the kick-off of the evaluation by the evaluation team and USAID. A summary of these items is provided below. Additional information, particularly regarding specific tools/toolkits and the EC-LEDS program, is available on the NREL website. 1. Modifications 1-10 (includes work plans) 2. Budgetary Information 3. Recent Performance Reports 4. USAID and DOE Stakeholder Contact Information 5. Training and Webinar Contact Information 6. Web Analytics and Surveys related to the Technical Platforms (as available) 7. EC-LEDS Performance Evaluation Additional background information on the EC-LEDS program, USAID’s Climate Change program, and the various toolkits can also be found online at the following sites: Table 3 Relevant Websites Description Link Climatelinks https://www.climatelinks.org/ EC-LEDS https://www.ec-leds.org/ Greening the Grid (GtG) http://greeningthegrid.org/ Renewable Energy Data Explorer (RED-E) https://www.re-explorer.org/ Geospatial Toolkit (GsT) http://en.openei.org/wiki/Geospatial_Toolkit Development Impact Assessment (DIA) https://www.ec-leds.org/tools-page/development￾impact-assessment-tools International Jobs and Economic Development Impacts (I-JEDI) https://www.climatelinks.org/resources/international￾jobs-and-economic-development-impacts-i-jedi Team Composition The evaluation will be delivered by a small evaluation team comprised of two independent external consultants including an Evaluation Specialist and a Research Associate, and some oversight by the E3 Analytics Technical Director, Technical Manager, or Project Manager is expected. All members of the evaluation team will be subject to USAID approval. Preferred qualifications for the evaluation team are provided below. PERFORMANCE EVALUATION OF THE USAID–DOE INTERAGENCY AGREEMENT | 37 The Evaluation Specialist will be responsible for the overall implementation of the evaluation and ensuring that all expected tasks and deliverables are achieved on time and of high quality. S/he must have significant professional experience coordinating similarly complex evaluations, and leading evaluation teams. The candidate must have exceptional organizational, analytical, writing and presentation skills. S/he must be fluent in English and must have a master’s level degree with 10 years of technical experience in a relevant analytical field. It would be highly desirable for the Evaluation Specialist candidate to have direct knowledge and/or experience working with USAID rules, evaluation policy, regulations, and procedures. S/he will oversee the overall drafting of the evaluation framework, including methodology determinations, survey design, and key informant interviews; organization of calendar/travel/meetings; overseeing the desk study, interviews, and other data collection; and analyzing the data with input from team members and USAID to draft the evaluation report. Relevant experience and knowledge of climate change mitigation programs is preferred. The Research Associate will support the team leader in the implementation of the evaluation. S/he should have professional experience implementing similarly complex evaluations involving multiple stakeholders. The candidate must have exceptional organizational, analytical, writing and presentation skills. S/he must be fluent in English and should have a master’s level degree with 3 years of experience in a relevant analytical evaluation field. It would be highly desirable to have knowledge and/or experience working with USAID rules, regulations, and procedures, particularly requirements of clean energy programs. S/he will contribute to the overall drafting of the evaluation framework and participating in the desk study, interviews, and other data collection; and analyzing the data with input from team members and USAID to draft the evaluation report. E) Methodological Assumptions and Limitations There are several assumptions and limitations the evaluation team will encounter with regards to the scope of the evaluation. With regards to the period of performance, the evaluation may cover relevant information from the entire period of performance (2009-2017); however, given the evolving nature of the work plans over time, the evaluation should focus primarily on ongoing or recently completed activities. Stakeholder Availability - Not all stakeholders may be available to contribute to focus groups, interviews or surveys. Additionally, some of the stakeholder groups consist of only a few core people. Therefore it is important that every effort by made to contact and follow up with key stakeholders to maximize the chances that they will be able to participate in the evaluation. To mitigate this risk, USAID, NREL, and the evaluation team should work together to identify contact information for all key stakeholders in the evaluation design phase, to track progress in reaching key stakeholders, and to adhere to good practices regarding stakeholder coordination, outlined below. Selection Bias - The evaluation team will rely primarily on supporting documentation including project documents, websites and contact information of stakeholders provided by USAID and NREL. The selection of these data sources could be consciously or unconsciously biased. However, the evaluation team may look for additional relevant documentation and points of contact during its desk review and interview processes to mitigate any selection biases. 8. EVALUATION MANAGEMENT The evaluation team will be supervised by the Activity Manager in USAID/E3, while working closely with the USAID-DOE Interagency Agreement AOR/Technical Lead to gain in-depth information of the program activities. The Activity Manager will provide strategic direction and guidance throughout the evaluation process, including the development of the work plan, any data collection tools, and evaluation report outline, approach, and content. PERFORMANCE EVALUATION OF THE USAID–DOE INTERAGENCY AGREEMENT | 38 A) Communications Regular communication between the evaluation team and the USAID Activity Manager for this evaluation will be essential to the successful execution of the evaluation activities. To ensure proper communication and documentation of evaluation planning and completion, the process for meetings, approvals, and stakeholder communications is described below. Weekly Meetings - The evaluation team will be responsible for coordinating weekly meetings between the 2 person evaluation team described above, and the USAID Activity Manager. The USAID Technical Lead will also be invited to these meetings and additional USAID or NREL staff may be invited to participate, as needed. To keep track of weekly activities and roles and responsibilities, meeting notes or transcripts of team meetings shall be provided to the Activity Manager by the evaluation team within two business days. The meetings may be conducted in-person or by phone. Stakeholder Coordination - The Activity Manager needs to be cc’d on all correspondence with USAID or DOE staff, or other stakeholder groups. This will allow USAID to facilitate the coordination and participation of key stakeholders, particularly those who may not be familiar with the evaluation process or who may have conflicting schedules. While the evaluation team will be responsible for coordinating interviews, surveys, and presentations, the Activity Manager or Technical Lead can provide introductions or follow-up emails as necessary. Presentations and Dissemination - The Activity Manager and Technical Lead will help coordinate USAID and NREL’s participation in the evaluation kickoff, in-brief and out-brief presentations during the site visit, the initial presentation of findings and a final presentation (if requested). They will also coordinate the internal review of drafts and disseminate the final evaluation internally. Roles and Responsibilities - The following chart outlines the actions required of different stakeholders who need to be engaged at various stages of the evaluation planning, implementation, and approval processes. Table 4 Roles and Responsibilities Matrix Roles and Responsibilities E3A COR Activity Manager Technical Lead USAID/W core staff NREL core staff NREL technical staff USAID missions Other Beneficiaries Invited to evaluation kickoff and presentations X X X X X Attends weekly calls X X Invited to review evaluation deliverables X X X X X X Approve evaluation deliverables X X Respondent for interviews and/or surveys X X X X X X X Receive final evaluation report X X X X X X X Opportunity to provide statement of differences to final evaluation X X X X X PERFORMANCE EVALUATION OF THE USAID–DOE INTERAGENCY AGREEMENT | 39 B) Deliverables The main deliverables for this evaluation will be the evaluation design and the final evaluation report, which should be delivered electronically. These deliverables should be succinct and clearly written. The evaluation team should avoid duplication within and among the evaluation documents (scope, design and report). Supplementary information should be included as annexes for reference purposes. All draft reports must be submitted as complete drafts without sections or annexes omitted, e.g., the draft evaluation design must include all survey instruments and interview guides. Design and report outlines should be provided by the evaluation team and reviewed in conjunction with USAID as part of the planning and analysis processes, respectively. Each deliverable needs to be approved by the USAID Activity Manager and Technical Lead. The evaluation schedule and budget should be maintained throughout the evaluation, and updated as changes occur to reflect current actuals and expectations. Changes to the evaluation schedule or budget baseline described in the SOW need to be approved by the Activity Manager. The Draft and Final Evaluation Design Proposals are anticipated to be largely focused on the survey and interview instruments. The approach for each stakeholder group needs to be considered separately. It may be helpful to map survey and interview questions to specific EQs. The evaluation team should consider a balance of open-ended questions and ratings questions (e.g., using a Likert scale), or other techniques. The final evaluation design should be approved prior to the site visit. The Draft and Final Evaluation Reports should provide a clear, concise set of findings, conclusions and recommendations based on the data analysis. Preference is given to a shorter report, which avoids redundancy. The evaluation team’s Evaluation Design Proposal should include a proposed outline for the Final Evaluation Report, including proposed page lengths for each main section, for approval by the USAID Activity Manager. Additional figures (e.g., survey results) and supporting documents (SOW, design, stakeholder lists, bibliography, etc.) should be included as annexes. The evaluation team should clearly link findings to conclusions to recommendations in the final report. The evaluation team should incorporate charts and graphs to summarize data, where relevant, and use illustrative quotes in the final report. 9. SCHEDULE The following chart provides the anticipated timeframe for evaluation activities and deliverables. The period of performance of the evaluation is approximately five months. In the planning and implementation of the evaluation, the Activity Manager and evaluation team will take into account relevant local holidays as well as stakeholder availability. This schedule may be updated as necessary, subject to USAID approval. PERFORMANCE EVALUATION OF THE USAID–DOE INTERAGENCY AGREEMENT | 40 Table 5 Baseline Evaluation Schedule PERFORMANCE EVALUATION OF THE USAID–DOE INTERAGENCY AGREEMENT | 41 Performance Evaluation of the USAID-Department of Energy Interagency Agreement Amendment to the Scope of Work 8/29/2018 Summary The evaluation team has completed the initial data collection efforts for the USAID-Department of Energy Interagency Agreement performance evaluation as outlined in the Evaluation Design. During this process, the evaluation team identified gaps in data and information sources from two target stakeholder groups. Without additional information from these groups, the evaluation team may not have sufficient insight to fully respond to several of the evaluation questions, primarily: “EQ2. To what extent are the clean energy technical platforms being used effectively by their target audiences for their intended purpose;” and to some extent EQ1 and EQ3. [there is no guarantee that the second round of data collection will fully get all the data needed]. This amendment to the Evaluation Scope of Work allows the evaluation team additional time and funds to address these issues. Data Gaps The data gap includes two types of stakeholders: host government counterparts and tool users, both of which are in the In-Country Partners group. To date, the evaluation team has conducted key informant interviews (KIIs) based off a point of contact (POC) list compiled by National Renewable Energy Lab (NREL) representatives for each project work stream. Stakeholders on the POC lists were selected due to their familiarity with NREL’s activities. KII invitations were sent to a subset of the In-Country partners identified in the POC list, and the respondents were interviewed. Host Government Of the respondents, 6 were host country government officials from 4 countries: Mexico, Philippines, Bangladesh and India. For the purposes of this evaluation, host government counterparts are employees of government organizations in USAID partner countries, and quasi-governmental organizations, such as national utilities or regulators that have been privatized. Since the initial interviews, additional names have been added to the POC list by NREL in response to the evaluation team’s request. Tool Users Of the respondents to the initial KIIs, 6 were tool users (2 each for RED-E, I-JEDI, and GtG). For the purposes of the evaluation, tool users self-identify during the KIIs as a stakeholder that uses one of the tools or technical platforms in decision-making. Technical platforms “include analytical tools that can be used by stakeholders or in partnership with U.S. DOE laboratories; resources, information and best practices; access to expert technical support, and capacity building through trainings, workshops, and webinars.”14 Potential tool users can be identified through POC lists or attendance at a training, workshop, webinar, or other outreach event related to one of the tools or toolkits developed under the IAA.15 As a stakeholder type, the term “tool users” is not mutually exclusive, i.e., a host government or implementing partner stakeholders might also identify as a tool user. Given the sizable investment by 14 Evaluation Statement of Work: Performance Evaluation of the USAID-Department Of Energy Interagency Agreement, October 7, 2017, Page 6 15 Ibid, Page 12 PERFORMANCE EVALUATION OF THE USAID–DOE INTERAGENCY AGREEMENT | 42 USAID and NREL into designing and promoting these toolkits, the evaluation should include additional feedback from this important stakeholder group. Therefore additional outreach and data collection efforts are necessary to address this gap. Additional Data Collection The following additional data collection efforts are requested. The evaluation team will propose the methodology for each of these requests, in conjunction with feedback from USAID and NREL. A schedule and budget for the remainder of the evaluation activities will be proposed by the evaluation team (as attachments to this document) and agreed on by NREL and USAID. The expected budget for the additional data collection activities is approximately $40,000. Second Round of KIIs The evaluation team will conduct a second round of KIIs, focusing on POCs that are host government partners or have been identified as potential tool users. The target for second round KIIs is approximately 25 calls16 with in-country partners, depending on availability. It is possible additional tool users may be identified through the survey, if they self-identify as tool users and voluntarily provide contact information. Outreach to stakeholders may be facilitated by USAID or NREL as needed, and according to USAID mission staff preferences. The current POC list has been provided in the “Stakeholder POC list 8.21.2018.” Survey of Training Participants In order to identify additional tool users and assess the overall tool usage, the evaluation team will also design and conduct a short survey targeting training participants. The list of training participants includes participants, whose complete contact information has been provided17, of a recent,18 tool-related training, workshop, webinar or other outreach event.19 The current participant list is available as the “Master Trainee List 8.22.2018.” Additional discussions with USAID and NREL will help determine the final list of trainings to be included and confirm the stakeholder types that will receive the survey. The survey is anticipated to be a short, electronic survey which will identify basic stakeholder or organization information; levels of familiarity, satisfaction, and usage of the tools; specific comments regarding usage or feedback; or other relevant questions. The evaluation team will work with USAID and NREL to finalize the evaluation survey questions, outreach plan, and timeline. 16 Multiple individuals from the same organization who respond to KII invitations will be joined on a group call 17 This requires at least a name and valid email address 18 Most of the training information provided by NREL to date has been between 2015-2017 19 NREL to inform the list of training participant lists to be included in the survey PERFORMANCE EVALUATION OF THE USAID–DOE INTERAGENCY AGREEMENT | 43 Table 1. In-Country POCs by Country Country Already Interviewed Not yet Contacted Bangladesh 2 Chile 3 Colombia 2 1 Costa Rica 1 Gabon 2 Ghana 3 Guatemala 1 Haiti 1 India 1 Kazakhstan 1 Lao PDR 4 Mexico 5 2 Peru 1 Philippines 2 7 South Africa 2 Sri Lanka 1 Tajikistan 1 Thailand/RDMA 2 Uzbekistan 1 Vietnam 1 4 Zambia 2 1 TOTAL 20 34 Table 2. List of NREL Trainings Training Name Training Type Tool Abbr Training Country Training Date Type of file Gabon Energy Audit Training/workshop Energy Audits Gabon Apr-14 Word Conducting Energy Audits to Improve Energy Efficiency Training/workshop Energy Audits Kenya Apr-14 Word SGWG Workshop on Energy Data Management and Energy Modeling Training/workshop GtG India Apr-14 Excel SROI Tool Training Training/workshop SROI Tool Gabon Oct-14 Word IJEDI Workshop Training/workshop I-JEDI Colombia Mar-15 Handwritten Gabon Energy Audit Training Training/workshop Energy Audits Gabon Mar-15 Grid Integration Review Committee Training/workshop GtG India Oct-15 Excel Greening the Grid: Best Practices in Conducting Grid Integration Studies Webinar GtG Webinar Oct-15 Excel PERFORMANCE EVALUATION OF THE USAID–DOE INTERAGENCY AGREEMENT | 44 Training Name Training Type Tool Abbr Training Country Training Date Type of file Greening the Grid: Implementing Renewable Energy Zones for Integrated Transmission and Generation Planning Webinar GtG Webinar Dec-15 Excel Implementing RE Zones Integrated Transmissions and Generation Planning Webinar GtG Webinar Dec-15 Not provided Wind and Solar Forecasting Training - Mexico Training/workshop GtG Mexico Jan-16 Handwritten Wind and Solar Forecasting Training - Bangkok Training/workshop GtG Thailand Feb-16 Excel Sustainable Growth Working Group Energy Modeling Workshop Training/workshop GtG India Mar-16 Not provided I-JEDI Primer Mexico March 2016 Training/workshop I-JEDI Mexico Mar-16 Excel-survey Greening the Grid: Implementing Wind and Solar Power Forecasting Webinar GtG Webinar Apr-16 Excel Implementing Wind and Solar Power – GtG Webinar GtG Webinar May-16 Not provided LEDS LAC Regional Event - Benefits Working Group Training/workshop GsT/RED￾E and IJEDI LAC Jun-16 Not provided Asia Clean Energy Forum- Mapping the Future: Energizing Data￾Driven Policy Making and Investment in Renewable Power Training/workshop GsT Philippines Jun-16 Excel Asia Clean Energy Forum - Pre Forum, Scaling Up Clean Energy: Early Actions to Facilities Integration of Varibles Renewable Energy into Existing Power System Training/workshop GtG Philippines Jun-16 Excel India Geospatial Analysis Training/workshop RED-E India Aug-16 Excel Big Fast and Flexible: Grid Operations for Efficient Variable Renewable Integration Webinar GtG Webinar Oct-16 Excel IJEDI Mexico Training Training/workshop I-JEDI Mexico Nov-16 Not provided GsT RED-E Training Training/workshop RED-E Kenya Feb-17 Word PERFORMANCE EVALUATION OF THE USAID–DOE INTERAGENCY AGREEMENT | 45 Training Name Training Type Tool Abbr Training Country Training Date Type of file IJEDI Regional Webinar Feb 2017 Webinar I-JEDI Webinar Feb-17 Excel Acolgen Webinar - Advancing RE in Colombia Webinar ? Colombia Feb-17 Not provided IJEDI Regional Webinar April 2017 Webinar I-JEDI Webinar Apr-17 Excel RED-E India Econ Potential Webinar Webinar RED-E India Jun-17 Excel IJEDI Regional Webinar June 2017 Webinar I-JEDI Webinar Jun-17 Excel Asia Clean Energy Forum - Pre Forum, The Power System of Tomorrow: Pathways Towards Large Scale Integration of Variable Renewable Energy Training/workshop Philippines Jun-17 Excel Asia Clean Energy Forum 2017: Integrating Generation and Transmission Planning to Enable Renewable Energy Deployment Training/workshop Philippines Jun-17 Excel LEDS LAC Regional Event - Benefits Working Group Training/workshop GsT and I￾JEDI Jun-17 Not provided IJEDI Train-the-Trainer South Africa Training/workshop I-JEDI South Africa Jul-17 Excel NREL workshop Training/workshop Ghana Aug-17 Excel APEC Workshop on RE/CE Investment and Trade with public and private sector representatives in Hanoi, Vietnam Training/workshop Vietnam Aug-17 Introduction to Wind Resource Assessment & Vaisala Triton SoDAR Training Training/workshop GsT Bangladesh Aug-17 Excel LEDS GP Annual Event IJEDI Training - Economic impact training Training/workshop I-JEDI Oct-17 Not provided IJEDI Mexico Training Training/workshop I-JEDI Mexico Nov-17 SGWG Workshop on Energy Data Management and Energy Modeling Training/workshop GtG India Apr-18 Excel PERFORMANCE EVALUATION OF THE USAID–DOE INTERAGENCY AGREEMENT | 46 Training Name Training Type Tool Abbr Training Country Training Date Type of file Training for PLEXOS models -Southern and Western Regions Training/workshop GtG India India GtG: Regional Studies Modeling Teams Training/workshop GtG India Not Provided Excel SGWG Workshop on Energy Data Management and Energy Modeling: Geospatial Workshop Training/workshop GtG India Not Provided Excel iJEDI Mexico Impact Survey Results Training/workshop I-JEDI Mexico Not Provided Excel PERFORMANCE EVALUATION OF THE USAID–DOE INTERAGENCY AGREEMENT | 47 ANNEX B: SUPPLEMENTARY INFORMATION ON EVALUATION METHODS, SAMPLES, AND LIMITATIONS Details on the methods used for collecting data, as well as the protocols and processes used for each data collection activity, are described below, organized by data source:  DOC—Review of quarterly and annual reports, IAA modifications, trip reports, and web analytics20  FGD—Focus group discussion  KII—Key informant interview  PM—Process mapping workshop  OS—Online survey of trainees The evaluation design included a matrix with anticipated data sources. Table 7 shows the actual sources that supported the evaluation findings for each evaluation question. TABLE 6: DATA SOURCES BY EQ AND AOI Questions Data Source EQ1—To what extent has the USAID-DOE IAA met the needs of missions, and what specific opportunities exist to improve the buy-in program  DOC  KIIs AOI-1.1—How, and to what extent, has this mechanism allowed USAID Mission and partner country access to NREL expertise and cutting edge analysis, research, and deployment?  KIIs AOI-1.2—In what specific ways has this mechanism complemented capabilities of local contractors, partners, or other contract mechanisms used by the field Missions?  KIIs AOI-1.3—How can coordination between E3, NREL, Mission and local implementing partners be improved?  KIIs AOI-1.4—What types of technical expertise or assistance from NREL and other DOE Labs have been of highest value to partner governments or other Mission stakeholders? What type of technical assistance delivery models have been most effective?  KIIs AOI-1.5—In what ways have lessons been transferred to the broader USG low emissions development portfolio and what have the benefits been?  KIIs EQ2—To what extent are the clean energy technical platforms being used effectively by their target audiences?  DOC  KIIs  OS AOI-2.1—How are these technical platforms and tools being used by target audiences and where/why might they be underutilized?  KIIs  OS AOI-2.2—What were the primary drivers that informed the types of technical platforms and other resources that have been developed? How are these drivers and user needs evolving?  OS AOI-2.3—What types of policies, programs, and country capacity development are being informed or advanced through these technical platforms and related technical assistance?  OS AOI-2.4—Do host country counterparts credit the platforms, tools or data derived from them with helping to drive policy or regulatory change, improve planning processes, or drive investment decisions?  KIIs AOI-2.5—To what extent have host country counterparts, or “owners”, of the tools committed to the long term operational costs required to keep model output current?  KIIs 20 Data on web analytics was limited, with few data points for the tools that had recorded web traffic. The trip report data, although rich in information, was in a format that did not permit its full use in analysis. The evaluation budget did not permit the team to extract these data for analysis. PERFORMANCE EVALUATION OF THE USAID–DOE INTERAGENCY AGREEMENT | 48 Questions Data Source EQ3—What are the strengths and weaknesses of the strategic planning, program coordination and management, communications, monitoring and reporting processes of the IAA? What are specific ways to make the program more efficient and effective?  DOC  KIIs  PM  FGDs AOI-3.1—Does NREL have the right number of staff to meet the needs of the IAA, and do staff have the right qualifications?  KIIs  FGDs AOI-3.2—Does the broader NREL team of experts feel that they have clear direction when brought on to tasks and do they feel that their work has impact?  KIIs  FGDs  PM AOI-3.3—Does NREL have a robust strategy/theory of change for how TA provided in the work plan will contribute to on the ground mitigation results?  KIIs  PM  FGDs  DOC AOI-3.4—Do the results being monitoring under the IAA reflect the overall strategy and goals of the agreement?  KIIs  DOC AOI-3.5—What barriers does NREL face in implementing their work plan on time and as planned, that USAID can address?  PM  KIIs  FGDs AOI-3.6—How, and to what extent, has this mechanism helped to promote consistency and continuity of program technical support under the EC-LEDS program?  KIIs  DOC AOI-3.7—How well does NREL coordinate with other USG programs and agencies working in the same sector and country?  KIIs  FGDs AOI-3.8—Are gender issues sufficiently addressed in work planning or implementation?  KIIs  DOC KEY INFORMANT INTERVIEWS To capture information from knowledgeable stakeholders that was not available in documentation, the evaluation team contacted 145 potential interviewees and ultimately conducted 89 KIIs in person or by telephone or Skype (depending on the respondent’s availability and preference). The KIIs were intended primarily to answer EQ1 and EQ2, though respondents also shared data relevant to EQ3. Because different informant types were expected to have different areas of specialized knowledge, the team drafted seven versions of the interview protocols, each tailored to a different demographic group (see Annex D): Washington, DC-based USAID staff, overseas USAID Mission-based staff, NREL technical specialists, NREL core leadership, implementing partners, mission partners,21 and tool users.22 The first round of data collection took place between March and June 2018. The numbers of mission partners and tool users who participated, however, fell short of targets, and the evaluation team therefore conducted a second round of data collection in December 2018 and January 2019. Table 7 summarizes the numbers of interviewees by type. 21 Mission partners potentially include individuals from any of the following types of host-country organizations working directly or indirectly with NREL and USAID on this IAA: host country governments, parastatal organizations (such as power companies and regulatory institutions), academic institutions, NGOs, and host-country private sector actors. 22 The tool user protocol was an auxiliary supplement used in conjunction with the implementing partner and mission partner protocols when interviewees self-identified as tool users. PERFORMANCE EVALUATION OF THE USAID–DOE INTERAGENCY AGREEMENT | 49 TABLE 7: INTERVIEWEES BY TYPE AND DATA COLLECTION ROUND Interviewee Type Round 1 Round 2 NREL leadership 3 0 NREL technicala 21 0 USAID/Washington 7 0 USAID missions 17 1 Implementing partners 12b 2 Mission partners 9 c 17d Total 69 20 a. Two interviewees in this category were housed at PNNL, a different DOE lab with distinct but related expertise. Because the technical assistance being delivered was still coming through the USAID/NREL IAA, the KII protocol for NREL technical staff was still appropriate and viable. b. 1 tool user c. 3 tool users d. 10 tool users CODING PROCESS To code the text from the transcripts and analyze the KII data, the team developed a coding template using the software MAXQDA. The team developed the template during the initial stage of the document review, creating codes on the basis of observed thematic clustering. As KII and FGD transcripts were completed, they were incorporated into this process to ensure that the coding template was viable for all primary and secondary data. The simple coding scheme reflected the time and budgetary constraints. CODING SCHEME  EQ1 Descriptions of mission needs – EQ1 Mission needs met YES – EQ1 Mission needs met NO – EQ1 Mission needs met HOW – EQ1.1 IAA process  EQ1.1 IAA access to NREL – EQ1.1 Strength of buy-in process – EQ1.1 Weakness of buy-in process – EQ1.1 Financial access to NREL – EQ1.1 IAA limitations – EQ1.1 Collaboration examples  EQ1.2 Complementing contractor capacities  EQ1.2 Complementing other local capacities  EQ1.2 Complimenting Mission contract mechanisms  EQ1.2 Alternatives to NREL  EQ1.3 Coordination successes  EQ1.3 Coordination challenges  EQ1.4 Most valuable assistance  EQ1.4 Less valuable assistance  EQ1.4 Most effective delivery model  EQ1.4 Less effective delivery models  EQ1.4 Comments on the request process  EQ1.5 Using NREL tools/products in other countries  EQ1.5 Using NREL tools/products in other ways  EQ1.5 Cohesive EC-LEDS program example PERFORMANCE EVALUATION OF THE USAID–DOE INTERAGENCY AGREEMENT | 50  EQ2 Tool Use – EQ2 Tool Use example – EQ2 Measuring tool success – EQ2 Indicators/knowledge about users and use – EQ2 Target audience – EQ2 Expanding tool users – EQ2 Tool Characteristics positives/strengths – EQ2 Tool Characteristics challenges/weaknesses  EQ2.1 Reasons for Use  EQ2.1 Reasons for Underuse  EQ2.1 Reason for Nonuse  EQ2.1 Unverified/nonspecific use example  EQ2.2 Feedback platform/resource design  EQ2.2 Changing user needs – EQ2.2 Resources needed – EQ2.2 Challenges in sector  EQ2.3 Policy/Program/Regulatory change example – EQ2.3 Challenges accessing results/outcomes – EQ2.3 Reasons for policy/program change – EQ2.3 Reasons for no policy/program change  EQ2.3 CAPACITY change example  EQ2.4 Host country credits NREL tools/data  EQ2.5 Sustainability plan clear – EQ2.5 Contribution by partners – EQ2.5 Contribution by mission – EQ2.5 what should be sustained?  EQ3 Ways to make the program more efficient and effective – EQ3 Examples of program effectiveness – EQ3 Examples of ineffectiveness  EQ3 Strategic planning: Strengths/Positives  EQ3 Strategic planning: Weaknesses/Negatives  EQ3 Program Coord and Mgt: Strengths/Positives  EQ3 Program Coord and Mgt Weaknesses/Negatives  EQ3 Communications: Strengths/Positives  EQ3 Communications: Weaknesses/Negatives  EQ3 Communications: Logistics  EQ3 Communicating more broadly Sharing work  EQ3 Monitoring and reporting processes: Strengths/Positives  EQ3 Monitoring and reporting processes: Weaknesses/Negatives – EQ3.1 NREL: Staff availability – EQ3.1 NREL: Staff qualifications – EQ3.2 NREL: Clarity of tasking – EQ3.2 NREL: Feeling of Impact – EQ3.3 NREL: Clear theory of change/results framework – EQ3.4 Results being Monitored for Goals – EQ3.4 Knowledge of the overall strategy/goals – EQ3.4 Relation between activities and overarching goals? – EQ3.5 Barriers  EQ3.5 Logistics barriers PERFORMANCE EVALUATION OF THE USAID–DOE INTERAGENCY AGREEMENT | 51  EQ3.5 Bureaucratic barriers – EQ3.6 EC-LEDS priorities – EQ3.7 Engaged other USG actors – EQ3.8 Gender  EQ3.8 Gender discussed  EQ3.8 Gender incorporated into planning/implementation examples  EQ3.8 Gendered Interest Areas  EQ3.8 Gender not incorporated (why?)  Quotes  Comments on Output/Outcomes  Recommendations  Trends/Future Needs CODING BIAS Studies involving qualitative data can be plagued with severe bias during the coding of text for analysis. The team reviewed the coding scheme, tested it, and revised it through the coding process to ensure it was appropriate for the responses. Two individuals were the primary coders and remained in constant communication to ensure their coding aligned and data was coded in a standard format. In countries with different languages, cultures, and norms, respondents can have varying interpretations of questions and provide information that cannot be pooled. Some interviews were conducted in Spanish. Transcripts provided another layer of data quality by capturing the exact response and unbiased account of the response. To increase the likelihood of finding data the team oversampled and focused on respondents from four key countries: Colombia, Mexico, Philippines, and India. This has probably biased the data, i.e., the results reflect these stakeholders. FOCUS GROUP DISCUSSIONS The evaluation team conducted two FGDs with select NREL staff. These FGDs enabled an in-depth exploration of opinions and experiences regarding the IAA’s strategic planning, coordination and management, communication, and monitoring and reporting processes, as well as other technical topics. These discussions provided a comprehensive internal perspective on how the IAA pursued its objectives and what NREL staff perceive as working well or needing improvement. FGDs have the benefit of helping group members collaborate on views and perspectives reflective of their group and reducing data collection costs by acquiring input simultaneously. To ensure quality data from FGDs, the team followed social science standards of practice (e.g., pre-tested instrumentation, clear protocols for implementation, managing groupthink, ensuring participation by all members, focusing discussions around the questions of interest). Annex D provides the evaluation team’s FGD protocols for each group. SAMPLE Given the small number of available individuals, the team – with the assistance of an NREL facilitator – used a mix of purposive and convenience sampling to maximize the number of participants and perspectives for each FGD, without heavy additional demographic segmentation. The first of the FGDs, held on March 20, 2018, included seven NREL technical specialists, three male and four female. The second FDG was held on March 22, 2018 and featured six NREL technical specialists with an even number of male and female participants. PERFORMANCE EVALUATION OF THE USAID–DOE INTERAGENCY AGREEMENT | 52 FGD PROCESS Each FGD took approximately 90 minutes and took place at NREL’s offices in Golden, Colorado. The two evaluation team members conducted the FGDs with one responsible for keeping the interview on topic, engaging attendees, and prompting participants to expand on or clarify their responses. The second facilitator provided logistical support, setting up the venue, ensuring that signed informed consent forms were received, operating audio recording, and taking detailed notes of responses and group dynamics, including body language and expressions, during the FGD. Both FGDs followed the same general format, with topical variance reflected in the respective protocols:  Introduction: General questions about the roles of FGD participants in the IAA.  Overall experience: Experience with participating in the IAA and its most useful and successful components.  Outcomes and challenges: Main obstacles and challenges in executing respondents’ responsibilities under the IAA, and whether and how those have been dealt with.  Specific dimensions of the IAA: Group observations on four administrative and operational aspects of the IAA: (1) strategic planning, (2) coordination and management, (3) communication, and (4) monitoring and reporting processes. After the FGD, the facilitators met to debrief and expand their notes, including observations on the process and group dynamics. They then generated summary notes to be used for data analysis and report writing and had the FGD recordings transcribed verbatim for subsequent data analysis. LIMITATIONS Although focus groups are useful for collecting a broad range of perspectives on a topic in a limited time, it also presents limitations. Focus groups are less effective at capturing in-depth information than individual interviews, but these FGDs were conducted early in the data collection process and were followed by KIIs that allowed the team to probe more deeply into the issues raised in the focus groups. Additionally, there is always the possibility that members may not be comfortable to express their honest opinions on a subject, particularly when they are negative or if they diverge from another participant’s views—especially when there are power or authority imbalances in the group. To avoid this, the evaluation team limited attendance to technical staff and excluded NREL managers. SURVEY The survey was part of the second wave of data collection. The first KIIs left gaps in the data from tool users and government partners. A second round of data collection included interviews and an online survey of individuals who had taken NREL-supported training. The purpose of the survey was to identify more tool users to interview from people who had taken tool trainings. People taking the survey who stated they had used a tool could volunteer to participate in phone interviews. SURVEY SAMPLE The names and emails in the online survey sample were for people who had signed up for training between 2015 and 2018. USAID and NREL provided a list of trainings given on the five tools and platforms covered by this evaluation and their sign-in sheets. The evaluation team disregarded a training if the sign-in sheet was not the original but a PDF scan, or if names on the sheet were illegible. Table 8 lists the trainings for which the sign-in sheets were used to extract names of potential survey respondents. PERFORMANCE EVALUATION OF THE USAID–DOE INTERAGENCY AGREEMENT | 53 TABLE 8: TRAININGS INCLUDED IN THE SURVEY SAMPLE Training Name Training Type Training Date Training Location Energy Audit Training Training/workshop Apr 14 Gabon Training on Conducting Energy Audits to Improve Energy Efficiency Training/workshop May 14 Kenya SROI Training Training/workshop Oct 14 Gabon Gabon Energy Audit Training Training/workshop Mar 15 Gabon Grid Integration Review Committee Training/workshop Oct 15 India Greening the Grid: Best Practices in Conducting Grid Integration Studies Webinar Oct 15 Webinar Greening the Grid: Implementing Renewable Energy Zones for Integrated Transmission and Generation Planning Webinar Dec 15 Webinar Greening the Grid: Implementing Wind and Solar Power Forecasting Webinar Apr 16 Webinar Asia Clean Energy Forum- Mapping the Future: Energizing Data-Driven Policy Making and Investment in Renewable Power Training/workshop Jun 16 Philippines Asia Clean Energy Forum - Pre Forum, Scaling Up Clean Energy: Early Actions to Facilities Integration of Varibles Renewable Energy into Existing Power System Training/workshop Jun 16 Philippines India Geospatial Analyis Workshop Training/workshop Aug 16 India Big Fast and Flexible: Grid Operations for Efficient Variable Renewable Integration Webinar Oct 16 Webinar IJEDI Regional Webinar Webinar Feb 17 Webinar GsT RED-E Training Training/workshop Feb 17 Kenya IJEDI Regional Webinar Webinar Apr 17 Webinar IJEDI Regional Webinar Webinar Jun 17 Webinar RED-E India Econ Potential Webinar Webinar Jun 17 India Asia Clean Energy Forum, Pre Forum, The Power System of Tomorrow: Pathways Towards Large Scale Integration of Variable Renewable Energy Training/workshop Jun 17 Philippines Asia Clean Energy Forum 2017: Integrating Generation and Transmission Planning to Enable Renewable Energy Deployment Training/workshop Jun 17 Philippines IJEDI Training Training/workshop Jul 17 South Africa Introduction to Wind Resource Assessment & Vaisala Triton SoDAR Training Training/workshop Aug 17 Bangladesh NREL workshop Training/workshop Nov 17 Ghana SGWG Workshop on Energy Data Management and Energy Modeling Training/workshop Apr 18 India India GtG: Regional Studies Modeling Teams Training/workshop India iJEDI Mexico Impact Survey Results Training/workshop Mexico PERFORMANCE EVALUATION OF THE USAID–DOE INTERAGENCY AGREEMENT | 54 SUMMARY OF RESPONSES The survey started November 20, 2018, and closed January 1, 2019. There were 279 respondents. The responses to the online survey questions are summarized below. Q1. Do we have your consent for you to participate in this survey? Responses Percentage Yes 274 98.21% No 5 1.79% (Did not answer) 0 0.00% Total responses 279 100.00% Q2. Based on your current work, which of the following best represents your primary sector? Responses Percentage Energy Regulation/Policy/Planning 112 40.14% Energy Transmission/Distribution 38 13.62% Energy Generation 48 17.20% Energy Efficiency 26 9.32% Buildings/Construction 7 2.51% Transportation 4 1.43% Telecommunication/ Utilities 5 1.79% Other (Please specify) 37 13.26% (Did not answer) 2 0.72% Total responses 279 100.00% Q3. What kind of organization is your current employer? Responses Percentage Government/Public Sector 102 36.56% University/College/Academic or Research Institution 35 12.54% International Donor/Assistance Organization 24 8.60% Not-for-Profit Organization/Community Organization 36 12.90% Private/Industry 64 22.94% Not Currently Employed (retired, between jobs, unemployed) 5 1.79% Other (Please specify) 12 4.30% (Did not answer) 1 0.36% Total responses 279 100.00% Q4. Did you ATTEND training(s) on any of the following NREL supported tools/platforms between Jan. 2016 - Oct. 2018? Yes No Attende d a training before Jan. 2016 Do not rememb er Did not answer Total GtG 89 31.90% 115 41.22% 8 2.87% 23 8.24% 44 15.77% 279 I-JEDI 27 9.68% 154 55.20% 5 1.79% 19 6.81% 74 26.52% 279 RED-E 53 19.00% 134 48.03% 10 3.58% 20 7.17% 62 22.22% 279 DIA 12 4.3% 161 57.71% 3 1.08% 20 7.17% 83 29.75% 279 DPV 28 10.04% 149 53.41% 7 2.51% 21 7.53% 74 26.52% 279 PERFORMANCE EVALUATION OF THE USAID–DOE INTERAGENCY AGREEMENT | 55 Q5. What year did you attend/take the training(s)? Responses Percentage 2016 49 17.56% 2017 70 25.09% 2018 61 21.86% I cannot remember 47 16.85% None of the above 84 30.11% (Did not answer) 6 2.15% Total responses 317 100.00% Q6. How did you receive your training(s)? Responses Percentage In-person training/workshop 149 53.41% Webinar/Online training 63 22.58% Other (Please specify) 42 15.05% (Did not answer) 42 15.05% Total responses 296 100.00% Q7. To what extent do you agree or disagree with each statement: The training I received helped me in my work/job/projects. N=279 A B C D Strongly Agree 94 33.69% 107 38.35% 64 22.94% 94 33.69% Somewhat Agree 88 31.54% 69 24.73% 99 35.48% 84 30.11% Somewhat Disagree 9 3.23% 7 2.51% 22 7.89% 8 2.87% Strongly Disagree 3 1.08% 3 1.08% 6 2.15% 3 1.08% I did not attend tool(s) / platform(s) training 66 23.66% 75 26.88% 69 24.73% 67 24.01% Did not answer 19 6.81% 18 6.45% 19 6.81% 23 8.24% Total responses 279 100.00% 279 100.00% 279 100.00% 279 100.00% Q8. Have you USED any of the following NREL supported tools/platforms between Jan. 2016 - Oct. 2018? N=279 GtG % I-JEDI % RED￾E % DIA % DPV % Strongly agree 45 16.13 14 5.02 39 13.98 9 3.23 15 5.38 Somewhat agree 27 9.68 17 6.09 22 7.89 10 3.58 11 3.94 Somewhat disagree 12 4.30 7 2.51 22 7.89 7 2.51 11 3.94 Strongly disagree 137 49.10 160 57.35 128 45.88 162 58.06 151 54.12 I did not attend tool(s)/platform(s) training 16 5.73 18 6.45 15 5.38 19 6.81 21 7.53 Did not answer 42 15.05 63 22.58 53 19.00 72 25.81 70 25.09 Total responses 279 100.00 279 100.00 279 100.00 279 100.00 279 100.00 PERFORMANCE EVALUATION OF THE USAID–DOE INTERAGENCY AGREEMENT | 56 Q9. What was your primary use of the tool? N=279 GtG % I￾JEDI % RED￾E % DIA % DPV % For a report 15 5.38 9 3.23 10 3.58 5 1.79 4 1.43 To provide information to a decision maker 29 10.39 12 4.30 31 11.11 6 2.15 8 2.87 To help set a standard/quota/code/ policy/regulation 13 4.66 4 1.43 13 4.66 6 2.15 9 3.23 For professional growth 26 9.32 12 4.30 24 8.60 8 2.87 13 4.66 For research and journal publications/papers 9 3.23 7 2.51 11 3.94 2 0.72 8 2.87 I did not use this tool 89 31.90 121 43.37 88 31.54 125 44.80 115 41.22 Other 9 3.23 3 1.08 3 1.08 4 1.43 5 1.79 (Did not Answer) 89 31.20 111 39.78 99 35.48 123 44.09 117 41.94 Q10. How did you first hear about the tool? Responses Responses Percentage from a website 81 29.03% during the NREL training 99 35.48% through a colleague 47 16.85% Other (Please specify) 43 15.41% (Did not answer) 34 12.19% Total responses 304 100.00% Q11. What was your favorite feature(s) of the tool(s)/platform(s)? [A "feature" could be any specific aspect or characteristic of the tool/platform. ] Q12. Do you think you will use the tool(s)/platform(s) again? N=279 GtG I-JEDI RED-E DIA DPV Yes 85 30.47% 51 18.28% 105 37.63% 41 14.70% 57 20.43% No 15 5.35% 21 7.53% 14 5.02% 19 6.81% 16 5.73% Not applicable (did not use the tool) 61 21.86% 69 24.73% 57 20.43% 74 26.52% 73 26.16% Do not know 59 21.15% 55 19.71% 40 14.34% 51 18.28% 48 17.20% Did not answer 59 21.15% 83 29.75% 63 22.58% 94 33.69% 85 30.47% Q13. What additional feature or change would you recommend be added to the tool(s)/platform(s)? [short answer] Q14. Is there a tool you need in your field of work that currently does not exist? [Please share the type of features you would desire, purpose or use of the tool, and any other information that could be helpful.] [short answer] PERFORMANCE EVALUATION OF THE USAID–DOE INTERAGENCY AGREEMENT | 57 Q15. If you stated YES to using one of the tools/platforms in Q7, would you be willing to participate in a telephone/skype interview with our team so we can learn more about your tool experience? Responses Percentage Yes 53 19.00% No 141 50.54% (Did not answer) 85 30.47% Total responses 279 100.00% Q16. In what country do you currently reside? Country Responses Percentage Armenia 1 0.36% Bangladesh 11 3.94% Benin 2 0.72% Bermuda (UK) 1 0.36% Bosnia and Herzegovina 1 0.36% Brunei 1 0.36% Burkina Faso 1 0.36% Cambodia 1 0.36% Canada 4 1.43% Cape Verde 1 0.36% China 1 0.36% Colombia 6 2.15% Cook Islands (New Zealand) 1 0.36% Ecuador 1 0.36% El Salvador 4 1.43% Ethiopia 1 0.36% Federated States of Micronesia 1 0.36% Fiji 1 0.36% France 1 0.36% Gabon 1 0.36% Germany 5 1.79% Ghana 21 7.53% Guatemala 2 0.72% India 26 9.32% Indonesia 6 2.15% Italy 3 1.08% Ivory Coast 1 0.36% Kazakhstan 2 0.72% Kenya 17 6.09% Laos 1 0.36% Lebanon 1 0.36% Liberia 1 0.36% Malawi 1 0.36% Mauritius 1 0.36% Mexico 3 1.08% Nicaragua 1 0.36% Nigeria 5 1.79% Pakistan 6 2.15% Palestine 1 0.36% Papua New Guinea 3 1.08% Paraguay 1 0.36% PERFORMANCE EVALUATION OF THE USAID–DOE INTERAGENCY AGREEMENT | 58 Country Responses Percentage Philippines 49 17.56% Portugal 1 0.36% Sierra Leone 1 0.36% Singapore 1 0.36% South Africa 11 3.94% South Korea 4 1.43% Spain 1 0.36% Sri Lanka 11 3.94% Switzerland 1 0.36% Tanzania 1 0.36% Thailand 4 1.43% Trinidad and Tobago 1 0.36% Turks and Caicos Islands (UK) 1 0.36% United Kingdom 2 0.72% United States of America 21 7.53% Uruguay 3 1.08% Vietnam 9 3.23% (Did not answer) 7 2.51% Total responses 279 Q17. What is your gender? Responses Responses Percentage Male 217 77.78% Female 50 17.92% Do not wish to say 4 1.43% (Did not answer) 8 2.87% Total responses 279 Q18. Are you currently married? Responses Percentage No 71 25.45% Yes 181 64.87% Do not wish to say 14 5.02% (Did not answer) 13 4.66% Total responses 279 100.00% Q19. How many children do you have? Responses Percentage None (0) 93 33.33% 1 41 14.70% 2 80 28.67% 3 29 10.39% 4 or more 17 6.09% (Did not answer) 19 6.81% Total responses 279 100.00% PERFORMANCE EVALUATION OF THE USAID–DOE INTERAGENCY AGREEMENT | 59 Q20. What is your age? Responses Percentage 18-24 1 0.36% 25-34 84 30.11% 35-44 74 26.52% 45-54 53 19.00% 55-64 43 15.41% 65-74 13 4.66% 75 or older 2 0.72% (Did not answer) 9 3.23% Total responses Q21. What is the highest degree or level of school you have completed? Responses Percentage Less than high school degree 1 0.36% High School degree or equivalent 1 0.36% Some college but no degree 7 2.51% College degree (e.g. AA, BS, BA) 42 15.05% Vocational Degree 4 1.43% Masters degree (e.g. MA, MS, MPA, MBA) 142 50.90% Doctorate degree (PhD, EdD, other) 31 11.11% Professional Degree (e.g. Engineering, MD, Law, DDM, DMV, other) 43 15.41% Other (Please specify) 2 0.72% (Did not answer) 6 2.15% Total responses 279 100.00% Q22. How long have you been with your current employer? Responses Percentage Less than 6 months 10 3.58% more than 6 months but less than 1 year 9 3.23% more than 1 year but less than 5 years 84 30.11% more than 5 years 165 59.14% (Did not answer) 11 3.94% Total responses 279 100.00% Q23. Are you willing to share the name of the organization of your current employer? Responses Percentage Yes 177 63.44% No 89 31.90% (Did not answer) 13 4.66% Total responses 279 100.00% PERFORMANCE EVALUATION OF THE USAID–DOE INTERAGENCY AGREEMENT | 60 PROCESS MAPPING WORKSHOP In addition to the FGDs, the evaluation team conducted a process mapping workshop for IAA technical assistance delivery, noting both the intended processes and the actual processes and clarifying the differences. The workshop provided a unique opportunity to generate ideas with NREL staff to maximize the IAA’s impact, find solutions to problems, produce an action plan to reach targets, and inform the evaluation’s recommendations. Through a series of exercises, the team:  Mapped the causal pathway to impact as perceived by the group through a process mapping exercise from start to end, identifying the assumptions of the pathway.  Identified the mental models of the stakeholder groups regarding their prioritization of activities (i.e., the value of impact), insights into target impacts, and alignment between agencies.  Discussed and shared lessons learned about science development partnerships with USAID. SAMPLE This workshop included 14 participants from NREL technical and management staff who had worked on the IAA (8 female, 6 male). The workshop focus and mix of participants presented the opportunity to:  Clarify and discover the group’s mental models on how the IAA is achieving or could achieve the targeted impacts.  Have teams co-create and ideate by identifying innovative opportunities, technologies, and processes. LIMITATIONS The process mapping workshop featured many of the same limitations as a conventional focus group discussion, as described above. Additionally, the lack of familiarity with an exercise of this type can make it challenging for participants to engage fully until they reach a point of comfort with the process and objectives of the exercise. The participants appeared comfortable with each other, which facilitated successful execution of the workshop. DOCUMENT REVIEW Document review was an essential element of the evaluation process from the beginning, and throughout, as the technical assistance provided under this IAA is extensive. The evaluation team grouped the documents into four categories:  Category 1: Design documents, agreement modifications, project reports and evaluations were shared with the evaluation team by USAID and NREL and laid the groundwork for the evaluation team to understand the nature of the IAA and its components. These documents were helpful for answering evaluation questions and sub-questions.  Category 2: Project websites, white papers, worksheets, and other technical guidance documents drawn from NREL’s online tools and by NREL technical staff provided insight into the variety and depth of technical assistance being rendered, and helped the interviewers understand and engage with interviewees. Beyond this, however, they were not critical to the evaluation analysis.  Category 3: A limited number of interviewees for host country governments also provided documentary examples of how NREL guidance and assistance had been incorporated into planning, regulatory, or legislative documents. These examples were helpful in that they confirmed verified individual cases of technical assistance usage but were secondary to the KII information itself. PERFORMANCE EVALUATION OF THE USAID–DOE INTERAGENCY AGREEMENT | 61  Category 4: Workshop and training attendance sheets for a wide variety of online and in￾person NREL trainings were critical to drawing up the survey sample and interview participants for round two of data collection but were not needed for any additional analysis or consultation. REVIEW PROCESS The documents accumulated during the course of this evaluation contained a wide range of topics with substantially varied levels of relevance, so not all of them were handled in the same fashion. Only Category 1 documents were needed for the analysis process, while the others were consulted on an ad hoc basis. Documents needed for analysis were printed and reviewed for insight into specific evaluation questions, and essential information was marked by highlighter or margin notes. Because the evaluation relied largely on qualitative data, these documents were important for verifying and clarifying data collected through KIIs and FGDs. Although the documents did not lend themselves to coding, pieces of information were, nonetheless, incorporated into the analytic framework, and allowed the evaluation team to clarify and/or confirm some data collected during the KIIs. LIMITATIONS The large number of IAA-related documents presented a time burden for the team. In the interests of time, the evaluation team therefore prioritized internal USAID and NREL documents (Category 1 above), which may create an inherent bias favorable to those institutions and their IAA. To mitigate potential bias, these documents were used primarily to triangulate and verify information collected by other means, except when the documents are the sole source of the information. PERFORMANCE EVALUATION OF THE USAID–DOE INTERAGENCY AGREEMENT | 62 ANNEX C: SUMMARY FINDINGS BY EVALUATION QUESTION AND AREAS OF INTEREST Table 9 provides additional summary findings by each EQ and AOI in USAID’s SOW. TABLE 9: SUMMARY FINDINGS BY EQ AND AOI EQ/AOI Summary Findings EQ1. To what extent23 has the USAID DOE IAA met the needs of the missions, and what specific opportunities exist to improve the buy-in program?  IAA activities appear to be meeting mission needs by complementing existing mission activities in-country. Respondents highlighted this complementarity by pointing to increased implementing partner capacity, increased access to technical expertise, and increased technical assistance to in-country partners on new technical applications (e.g., modeling renewables for grid integration, assessing wind source potential, creating auction mechanisms). For example, mission respondents confirmed the NREL teams helped local energy sectors adopt new technology.  Mission staff found the mechanism practical and an easy way to access NREL technical expertise, which otherwise would be done through time-consuming procurement processes.  In-country partners24 suggested the IAA should increase the number of face-to￾face meetings; create a long-term in-country technical position to be held by NREL technical staff; encourage participation by the private sector to host and coordinate meetings; and expand the number of NREL teams that work on a single tool, because sometimes mission requests can overlap.  Suggestions by stakeholders to improve the buy-in process included (1) improve tracking of mission requests, (2) increase awareness of the IAA through more science exchanges of in-country partners at NREL in Colorado, and (3) improve use of mission media outlets to share IAA products and projects more broadly in the region. AOI 1.1 How, and to what extent, has this mechanism allowed USAID Mission and partner country access to NREL expertise and cutting-edge analysis, research, and deployment?  The IAA created an easily accessible mechanism, reducing the time required for missions to engage technical expertise, otherwise done through lengthy, complex procurements.  The mechanism complemented existing mission work.  The extent to which each mission request was met is difficult to analyze. The data with this information is not in a searchable format; the trip debriefs shared with the team contained detailed information about each trip and provision of TA to each country but were not formatted for analysis (because of the budget and timeframe). Also, the requests from missions are not tracked in a data￾mineable format for analysis, creating a gap in the data on the extent to which each mission request was met. AOI 1.2 In what specific ways has this mechanism complemented capabilities of local  Implementing partners suggested NREL training had increased their capacity and facilitated their work, specifically to support mission activities. 23 The data obtained in this evaluation is insufficient to measure the extent to which the IAA has met the mission’s needs. The data does provide information about how and if the IAA met mission needs. The IAA does not gather systematically data on mission requests and has not defined what constitutes meeting a mission need. 24 For this evaluation, in-country partners include implementing partners, host-government partners, NGOs, and other organizations collaborating on IAA-supported activities. These groups comprise the key stakeholders for the IAA. PERFORMANCE EVALUATION OF THE USAID–DOE INTERAGENCY AGREEMENT | 63 EQ/AOI Summary Findings contractors, partners, or other contract mechanisms used by the field Missions?  Respondents stated that access to NREL expertise complemented mission￾funded work through increased access to data, software resources, and expert analysis and advice. AOI 1.3 How can coordination between E3, NREL, Mission and local implementing partners be improved?  In-country partners25 interviewed suggested that the IAA increase the number of face-to-face meetings; create a long-term in-country technical position to be held by NREL technical staff; encourage participation by the private sector to host and coordinate meetings; and increase the number of NREL teams that work on a single tool, because sometimes mission requests can overlap. AOI 1.4 What types of technical expertise or assistance from NREL and other DOE Labs have been of highest value to partner governments or other Mission stakeholders? What type of technical assistance delivery models have been most effective?  The most common interview responses to this question referred broadly to benefitting from NREL’s technical expertise. Access to and instruction in Plexos software was also cited as highly valuable, particularly by host government partners. Less commonly mentioned but also present were the GtG and I-JEDI products, and NREL’s general trainings and workshops.  Regarding the second part of this question, interviewees did not discuss any other type of delivery models. Mission staff spoke positively about the ease of using the IAA and the speed of accessing NREL technical expertise when compared to traditional procurement processes. AOI 1.5 In what ways have lessons been transferred to the broader USG low emissions development portfolio and what have the benefits been?  Respondents did not specifically state knowledge transfer to the USG low￾emissions development portfolio directly, but they did discuss ways in which lessons have been shared. This included through mission newsletters, conferences and workshops, and social media at missions. EQ2. To what extent are the clean energy technical platforms being used effectively by their target audiences?  For the five tools assessed in this evaluation, survey data and KIIs suggested that target audiences (i.e., government/public sector, NGOs, academia, industry, and other in-country partners) used tools to meet diverse needs. Users stated tools were effective in helping them create renewable energy ceilings/quotas/standards, informing decision-makers, and helping to provide data for reports.  Survey data suggested the sectors were diverse in their use of tools, and use was not clustered around one type of tool.  The survey data from trainees suggested there are spillover effects; tool users who had not taken training were identified.  Current IAA practices do not fully capture the benefits of the activities.  Respondents stated that tools were most often used because they were free or open sourced, provided data not available in-country, and are often unique.  Barriers to tool use according to survey respondents included the presence of competing tools, bandwidth issues, software hardware limitations, user capacity, lack of country-specific customization, and lack of awareness of the tool. AOI 2.1. How are these technical platforms and tools being used by target audiences and where/why might they be underutilized?  Trainees said they use the tools to produce data, for reports, to inform decision-makers, papers, and publications, for personal growth, and to support standards and quotas.26  Only 34 percent of respondents (153 of the 274 surveyed) stated they had used at least one tool, and 121 (44 percent) stated they had not used a tool.  The top 11 countries using the tools, starting with the highest frequency of users were Philippines, India, Ghana, United States, Kenya, Sri Lanka, South Africa, Bangladesh, Vietnam, Pakistan, Colombia.  Survey respondents said they used the tools during and after training, but a large portion of respondents could not remember. 25 For this evaluation, in-country partners include implementing partners, host-government partners, NGOs, and other organizations collaborating on IAA supported activities. These groups comprise the key stakeholders for the IAA. 26 Tool use findings from survey data are not generalizable because this data reflects the biases of the training sample to more GtG and RED-E training and views of people who attended training. PERFORMANCE EVALUATION OF THE USAID–DOE INTERAGENCY AGREEMENT | 64 EQ/AOI Summary Findings AOI 2.2. What were the primary drivers that informed the types of technical platforms and other resources that have been developed? How are these drivers and user needs evolving?  Drivers of tool creation or application country support were highly affected by USAID priorities, and balanced by constraints created by NREL’s availability, budget limitations, and country needs.  Future requirements include continued grid integration, wind prospecting, and solar expertise. Other future trends identified were in electric buses and cars, battery storage, social education, rooftop photovoltaic, policy challenges, and renewables online. AOI 2.3. What types of policies, programs, and country capacity development are being informed or advanced through these technical platforms and related technical assistance?  NREL experts helped to influence policy change to increase ceilings for solar and wind energy, assisted net billing policy, established auction mechanisms, and provided data for new codes (among other examples). AOI 2.4. Do host country counterparts credit the platforms, tools or data derived from them with helping to drive policy or regulatory change, improve planning processes, or drive investment decisions?  Partners credit NREL technical expertise for helping to drive changes in the energy planning sector, producing analysis that meets standards, and increasing capacity for staff to implement better mission-funded projects. AOI 2.5. To what extent have host country counterparts, or “owners”, of the tools committed to the long-term operational costs required to keep model output current?  Across all respondent groups, none took direct ownership of the tools or platforms. Respondents did discuss when credit was given to them by NREL in reports, but they view the tools as DOE- and NREL-owned products. EQ3. What are the strengths and weaknesses of the strategic planning, program coordination and management, communication, monitoring and reporting process of the IAA? What are the specific ways to make the program more efficient and effective?  The current agreement does not appear to align with the objectives in the original agreement, yet the language in the IAA has remained unchanged.  Weaknesses identified reflect the weaknesses found in the mechanism itself, the interagency agreement. These can be general and lack of clarity about roles and responsibilities for each agency.  USAID and NREL understand impact differently, leading to divergent impact expectations.  The positive communication environment in the IAA is a strength in assisting implementation and planning processes. AOI 3.1 Does NREL have the right number of staff to meet the needs of the IAA, and do staff have the right qualifications?  NREL teams are viewed as experts in their field and there is continued demand for additional face-to-face presence in-country by stakeholders.  Respondents reported limitations on NREL’s ability to work beyond the original scope because of scheduling conflicts and budget limitations.  Limited experts working with the teams to track web traffic and online user experiences.  NREL as a laboratory does have the required expertise to continue helping missions and in-country partners. AOI 3.2 Does the broader NREL team of experts feel that they have clear direction when brought on to tasks and do they feel that their work has impact?  NREL staff have clear direction in the work they do and understand the impact of their work.  NREL staff expressed positive views about their work experience. AOI 3.3 Does NREL have a robust strategy/theory of change for how TA provided in the work plan will contribute to on the ground mitigation results?  The team was not able to find or identify a theory of change for this IAA, based on the data gathered for this evaluation. AOI 3.4 Do the results being monitoring under the IAA reflect the overall strategy and goals of the agreement?  The monitoring of the IAA is based primarily on reporting requirements for Congress. This is not enough to measure impact or track change over time. A more robust strategy is required to reflect the scope of work of the IAA and capture the diverse impacts and outcomes from this partnership. AOI 3.5 What barriers does NREL face in implementing their work plan on time and as planned, that USAID can address?  Both agencies can work to help teams standardize processes for travel clearance, publications, website content, and preassess data access barriers in country. PERFORMANCE EVALUATION OF THE USAID–DOE INTERAGENCY AGREEMENT | 65 EQ/AOI Summary Findings AOI 3.6 How, and to what extent, has this mechanism helped to promote consistency and continuity of program technical support under the EC-LEDS program?  NREL staff did not distinguish between the earlier EC-LEDS-focused IAA and the current agreement. The nature of the work was more important than the funding mechanism. AOI 3.7 How well does NREL coordinate with other USG programs and agencies working in the same sector and country  Trips for new countries should include early communications with other U.S. government agencies when appropriate, although not all agencies have been responsive or engaged when asked to participate. AOI 3.8 Are gender issues sufficiently addressed in work planning or implementation?  The IAA can improve how it integrates the needs of women and minorities to increase participation and thus potential IAA impact. PERFORMANCE EVALUATION OF THE USAID–DOE INTERAGENCY AGREEMENT | 66 ANNEX D: EVALUATION INSTRUMENTS KII INTERVIEW PROTOCOL: USAID/WASHINGTON  Paper Copy of the Interview Document (this document)  Recorder and batteries  Pen and paper  Confirmation of time and location of interview meeting DATE: Day _____ Month _____ 2018 RESEARCHER NAME: ☐ Carolyn Fonseca ☐ Isaac Morrison LOCATION/TIME: PARTICIPANTS: [see sign-in sheet] GENDER: ☐ Male _______ ☐ Female ______ Questions START HERE: 1. Introduction “Hello, my name is __________________ and I am an evaluation consultant for a Company called MSI (Management Systems International)” 2. Purpose “MSI has been contracted by USAID to conduct an independent and external evaluation of the Inter-Agency Agreement (IAA) between DOE NREL and USAID. This purpose of this evaluation is to assess the performance of the implementation of this collaboration and its activities” “We are conducting interviews to collect data from stakeholders, our experiences and perspectives about the IAA activities. We will be interviewing staff from both USAID and NREL, USAID missions, implementing partners, tool users and other in-country mission partners” “Findings from the evaluation will provide recommendations on how to improve future IAA efforts” 3. Interview Process “Today I will ask you some simple questions about your experiences and views” “The interview should take us about 45 minutes to 1 hour” 4. Consent “Before we start with introductions and interview questions, we are required to ask for your consent”  You have the right to not participate, this is completely voluntary  You have the right to stop at any point  If there is anything you do not understand, please ask me to clarify”  You have the right to decline to answer any question PERFORMANCE EVALUATION OF THE USAID–DOE INTERAGENCY AGREEMENT | 67  The information you provide is only for our evaluation and your responses will be protected. The information will be grouped so no one can be identified Consent “Do we have your consent to participate in this Focus Group?” ☐ YES ☐ NO Consent to Record “We would like to record this interview, ONLY for accuracy of our notes, and not to share with anyone else, including USAID or NREL” “Do we have your consent to RECORD?” ☐ YES ☐ NO START RECORDER: The Inter-Agency Agreement between USAID and DOE has supported a variety of activities initiated in 2009. In 2017, the two agencies singed a new agreement directly with the National Renewable Energy Lab (NREL), a Department of Energy laboratory in Golden CO. The evaluation will investigate which activities should be prioritized going forward, identify areas for change/improvement, and provide recommendations/guidance in future efforts under the new IAA.  In this interview we would like to focus on your experience, role, and perspective of IAA activities/products. SECTION KII QUESTION RESPONSES GENERAL INFORMATION 1. What has been your role/involvement under this IAA? (Connection to IAA and experience) (to be added by the researcher) 2. Can you confirm the name of your organization (to be added by the researcher) 3. How long have you been at _______ organization? (to be added by the researcher) 4. How long have you been on IAA activities? (to be added by the researcher) 5. Have you interacted directly with NREL staff? if yes, for what activity (to be added by the researcher) EQ1: To what extent has the USAID￾DOE IAA met the needs of missions, and what specific opportunities exist to improve the buy-in program? 6. Do you feel the needs of the Missions are being met by the IAA, if yes how, if no why not? (to be added by the researcher) 7. How are those needs identified and by who? AOI-1.1 How, and to what extent, has this mechanism allowed USAID Mission and partner country access to NREL expertise 8. What are the strengths and weaknesses of the “buy-in process”? (to be added by the researcher) 9. Have there been requests by Missions/partners which could not be met? Why? PERFORMANCE EVALUATION OF THE USAID–DOE INTERAGENCY AGREEMENT | 68 and cutting-edge analysis, research, and deployment? 10. Have any of the E3 units participated in the provision of technical assistance/expertise or been involved in any of the research activity with NREL or independently? AOI-1.2 In what specific ways has this mechanism complemented capabilities of local contractors, partners, or other contract mechanisms used by the field Missions? NONE (to be added by the researcher) AOI-1.3 How can coordination between E3, NREL, Mission and local implementing partners be improved? 11. How can coordination between E3, NREL, Mission and local implementing partners be improved? (to be added by the researcher) AOI-1.4 What types of technical expertise or assistance from NREL and other DOE Labs have been of highest value to partner governments or other Mission stakeholders? What type of technical assistance delivery models have been most effective? 12. What types of technical expertise or assistance from NREL and other DOE Labs have been of highest value to USAID E3 units? (to be added by the researcher) 13. How do USAID units engage with and make requests of NREL and DOE through this IAA? 14. What type of technical assistance delivery models have been most effective? AOI-1.5 In what ways have lessons been transferred to the broader USG low emissions development portfolio and what have the benefits been? 15. What are some examples where USAID engaged other agencies (beyond DOE) for expertise to meet the request from a mission or mission partner? (to be added by the researcher) 16. How does USAID share lessons learned from IAA funded activities within and outside USAID? 17. What are some challenges in sharing results and lessons learned of IAA activities? 18. Which other partner does USAID wish they had to meet the goals outlined in the IAA? EQ2: To what extent are the clean energy technical platforms being used effectively by their target audiences? 19. What technical platforms and products do you believe are being used, and by whom? (to be added by the researcher) 20. How do you know about these users and use? AOI-2.1 How are these technical platforms and tools being used by target audiences and where/why might they be underutilized? 21. What are some barriers to access and use? (to be added by the researcher) 22. Does USAID get any feedback, about these tools and platforms? And how? AOI-2.2 What were the primary drivers that informed the types of technical platforms and other resources that have been developed? How are these drivers and user needs evolving? 23. How does USAID track and respond to the evolving needs of the mission and users? (to be added by the researcher) PERFORMANCE EVALUATION OF THE USAID–DOE INTERAGENCY AGREEMENT | 69 AOI-2.3 What types of policies, programs, and country capacity development are being informed or advanced through these technical platforms and related technical assistance? 24. Are you aware of an instance in which a tool or product funded through this mechanism affected a policy or was used in a decision-making process? (to be added by the 25. What are some challenges in accessing researcher) product and outcome data? 26. What is your role in determining which capacity building efforts should be executed? AOI-2.4 Do host country counterparts credit the platforms, tools or data derived from them with helping to drive policy or regulatory change, improve planning processes, or drive investment decisions? NONE (to be added by the researcher) AOI-2.5 To what extent have host country counterparts, or “owners”, of the tools committed to the long-term operational costs required to keep model output current? 27. What type of in-kind resources have partners added to the IAA activities beyond the initial investment? (to be added by the 28. Who tracks these investments/leveraging researcher) efforts? 29. What is a sustainability model for an IAA product/platform tool? EQ3: What are the strengths and weaknesses of the strategic planning, program coordination and management, communications, monitoring and reporting processes of the IAA? What are specific ways to make the program more efficient and effective? 30. How does the management team at USAID select the projects for funding? (to be added by the researcher) 31. What is the role of the mission, NREL and other E3 units in the selection process? AOI-3.1 Does NREL have the right number of staff to meet the needs of the IAA, and do staff have the right qualifications? 32. Does NREL have the right number of staff to meet the needs of the IAA, and do staff have the right qualifications? (to be added by the researcher) 33. Do you believe USAID has the sufficient technical staff to manage and direct technical platforms? AOI-3.2 Does the broader NREL team of experts feel that they have clear direction when brought on to tasks and do they feel that their work has impact? 34. Do you feel NREL has clarity around the direction, implementation and impact of tasks? (to be added by the researcher) 35. Do you feel the agencies align regarding the target impact of IAA activities? AOI-3.3 Does NREL have a robust strategy/theory of change for how TA provided in the work plan will contribute to on the ground mitigation results? NONE (to be added by the researcher) AOI-3.4 Do the results being monitoring under the IAA reflect the overall strategy and goals of the agreement? 36. Do the results being monitoring under the IAA reflect the overall strategy and goals of the agreement? (to be added by the researcher) 37. What changes would you recommend USAID make to better understand the impact of IAA investments? PERFORMANCE EVALUATION OF THE USAID–DOE INTERAGENCY AGREEMENT | 70 AOI-3.5 What barriers does NREL face in implementing their work plan on time and as planned, that USAID can address? 38. What type of barriers does NREL face when implementing their work plan? (to be added by the researcher) 39. What changes could be made to improve the work plan timeline and help NREL meet the Mission requests within their time frame? 40. What additional actions could USAID take to help NREL implement their work plan? AOI-3.6 How, and to what extent, has this mechanism helped to promote consistency and continuity of program technical support under the EC-LEDS program? 41. Thinking of lessons learned, how, and to what extent, has this mechanism helped to promote consistency and continuity of program technical support under the EC-LEDS program? (to be added by the researcher) AOI-3.7 How well does NREL coordinate with other USG programs and agencies working in the same sector and country? NONE (to be added by the researcher) AOI-3.8 Are gender issues sufficiently addressed in work planning or implementation? 42. What is USAID’s understanding on how DOE processes incorporate gender issues into research activities? (to be added by the researcher) 43. What are some examples of where gender issues were integrated into a specific activity? 44. What are barriers for women to participate in any phase of the activities funded by the IAA? DEMOGRAPHICS 45. What is your highest level of education? 46. What is your field of expertise? (to be added by the researcher) Closing Remarks “We are now at the end of the interview”  “THANK YOU for your time and for sharing your experience”  “Do you have any questions for me, for our evaluation team?”  “Your participation in this interview will contribute greatly to the evaluation team’s ability better understand the effects of the IAA and its contributions to meet Mission needs”  “The information from this interview will be analyzed with other interviews aggregated. The results will not show your response with your name.”  “Recoding data will not be shared with USAID or NREL staff. Only the evaluation team will have the raw recorded data” STOP RECORDER: PERFORMANCE EVALUATION OF THE USAID–DOE INTERAGENCY AGREEMENT | 71 KII INTERVIEW PROTOCOL: USAID MISSIONS  Paper Copy of the Interview Document (this document)  Recorder and batteries  Pen and paper  Confirmation of time and location of interview meeting DATE: Day _____ Month _____ 2018 RESEARCHER NAME: ☐ Carolyn Fonseca ☐ Isaac Morrison LOCATION/TIME: PARTICIPANTS: [see sign-in sheet] GENDER: ☐ Male _______ ☐ Female ______ Questions START HERE: 1. Introduction “Hello, my name is __________________ and I am an evaluation consultant for a Company called MSI (Management Systems International)” 2. Purpose “MSI has been contracted by USAID to conduct an independent and external evaluation of the Inter-Agency Agreement (IAA) between DOE NREL and USAID. This purpose of this evaluation is to assess the performance of the implementation of this collaboration and its activities” “We are conducting interviews to collect data from stakeholders, our experiences and perspectives about the IAA activities. We will be interviewing staff from both USAID and NREL, USAID missions, implementing partners, tool users and other in-country mission partners” “Findings from the evaluation will provide recommendations on how to improve future IAA efforts” 3. Interview Process “Today I will ask you some simple questions about your experiences and views” “The interview should take us about 45 minutes to 1 hour” 4. Consent “Before we start with introductions and interview questions, we are required to ask for your consent”  You have the right to not participate, this is completely voluntary  You have the right to stop at any point  If there is anything you do not understand, please ask me to clarify” PERFORMANCE EVALUATION OF THE USAID–DOE INTERAGENCY AGREEMENT | 72  You have the right to decline to answer any question  The information you provide is only for our evaluation and your responses will be protected. The information will be grouped so no one can be identified Consent “Do we have your consent to participate in this Focus Group?” ☐ YES ☐ NO Consent to Record “We would like to record this interview, ONLY for accuracy of our notes, and not to share with anyone else, including USAID or NREL” “Do we have your consent to RECORD?” ☐ YES ☐ NO START RECORDER: The Inter-Agency Agreement between USAID and DOE has supported a variety of activities initiated in 2009. In 2017, the two agencies singed a new agreement directly with the National Renewable Energy Lab (NREL), a Department of Energy laboratory in Golden CO. The evaluation will investigate which activities should be prioritized going forward, identify areas for change/improvement, and provide recommendations/guidance in future efforts under the new IAA.  In this interview we would like to focus on your experience, role, and perspective of IAA activities/products. EQ AND AOI KII QUESTION RESPONSES GENERAL INFORMATION 1. What has been your role/involvement under this IAA? (Connection to IAA and experience) (to be added by the researcher) 2. Can you confirm the name of your organization (to be added by the researcher) 3. How long have you been at _______ organization? (to be added by the researcher) 4. How long have you been on IAA activities? (to be added by the researcher) 5. Have you interacted directly with NREL/DOE staff? if yes, for what activity? (to be added by the researcher) EQ1: To what extent has the USAID￾DOE IAA met the needs of missions, and what specific opportunities exist to improve the buy-in program? 6. Do you feel your needs are being met by the IAA, if yes why, if no why not? (to be added by the 7. How do you identify/prioritize your researcher) needs for IAA support requests? And by whom? AOI-1.1 How, and to what extent, has this mechanism allowed USAID Mission and partner country access to NREL expertise and cutting-edge analysis, research, and deployment? 8. Please describe how your Mission engages with the IAA process. (to be added by the researcher) 9. (For Buy-in Missions only) How does your Mission “buy-in” to the mechanism? 10. What are the strengths and weaknesses of this ‘buy-in process”? PERFORMANCE EVALUATION OF THE USAID–DOE INTERAGENCY AGREEMENT | 73 11. What type of expertise has your Mission requested? 12. What type of expertise has been requested the most, and why? 13. Have you made a request that was not met? What was it? 14. Have any of the technical Mission staff participated in the provision of technical assistance/expertise or been involved in any of the research activity with NREL or independently? AOI-1.2 In what specific ways has this mechanism complemented capabilities of local contractors, partners, or other contract mechanisms used by the field Missions? 15. How has the IAA mechanism complemented or increased the capacity of your local contractors (IPs and others), added technical capacity to your Mission staff, and/or complimented other contractual mechanism? (to be added by the researcher) AOI-1.3 How can coordination between E3, NREL, Mission and local implementing partners be improved? 16. How can coordination between E3, NREL, Mission and local implementing partners be improved? (to be added by the researcher) AOI-1.4 What types of technical expertise or assistance from NREL and other DOE Labs have been of highest value to partner governments or other Mission stakeholders? What type of technical assistance delivery models have been most effective? 17. What types of technical expertise or assistance from NREL and other DOE Labs have been of highest value to Missions? (to be added by the researcher) AOI-1.5 In what ways have lessons been transferred to the broader USG low emissions development portfolio and what have the benefits been? 18. What are some examples where USAID missions engaged other agencies (beyond DOE) for expertise or technical assistance? (to be added by the researcher) 19. What are some challenges in sharing results and lessons learned of IAA activities? EQ2: To what extent are the clean energy technical platforms being used effectively by their target audiences? 20. What technical platforms and products do you believe are being used, and by who? (to be added by the researcher) 21. How do you know about these users and use? 22. What would be an “effective” use of a platform or tool like? What would be the parameters in which you would assess effectiveness? 23. Can you give an example of “successful” IAA product in your country? What made it successful? 24. How does NREL engage you in the design process? 25. Who is currently not a target audience that could benefit from the IAA products and platforms? AOI-2.1 How are these technical platforms and tools being used by target 26. Please describe your interaction and experience with NREL and DOE staff. PERFORMANCE EVALUATION OF THE USAID–DOE INTERAGENCY AGREEMENT | 74 audiences and where/why might they be underutilized? 27. Does your mission collect information directly or indirectly about the tools/products created under the IAA (requested by your Mission)? (to be added by the researcher) 28. What type of feedback have your IPs and partners given your mission about IAA products/platforms? AOI-2.2 What were the primary drivers that informed the types of technical platforms and other resources that have been developed? How are these drivers and user needs evolving? 29. What development goal was the platform/product requested addressing? (to be added by the researcher) 30. What are some challenges if any in the product created? Did it meet your need? 31. What suggestions to you have for NREL to help improve the provision of technical expertise and products? AOI-2.3 What types of policies, programs, and country capacity development are being informed or advanced through these technical platforms and related technical assistance? 32. Are you aware of an instance in which a tool or product funded through this mechanism affected a policy or was used in a decision-making process? (to be added by the researcher) 33. What are some challenges in accessing products and output data? 34. What role does the mission play in determining which capacity building efforts are needed? 35. Is there a type of training you feel is missing from this program? AOI-2.4 Do host country counterparts credit the platforms, tools or data derived from them with helping to drive policy or regulatory change, improve planning processes, or drive investment decisions? NONE (to be added by the researcher) AOI-2.5 To what extent have host country counterparts, or “owners”, of the tools committed to the long-term operational costs required to keep model output current? 36. Beyond IAA funds, what other resources has your mission contributed to the platform/products? (to be added by the 37. What has been the partner response to researcher) IAA funded tools and products? 38. What is a sustainability model for IAA products? EQ3: What are the strengths and weaknesses of the strategic planning, program coordination and management, communications, monitoring and reporting processes of the IAA? What are specific ways to make the program more efficient and effective? NONE (to be added by the researcher) AOI-3.1 Does NREL have the right number of staff to meet the needs of the IAA, and do staff have the right qualifications? 39. Does NREL have the right number of staff to meet the needs of the IAA, and do staff have the right qualifications? (to be added by the researcher) AOI-3.2 Does the broader NREL team of experts feel that they have 40. Do you feel NREL is clear about your technical request? PERFORMANCE EVALUATION OF THE USAID–DOE INTERAGENCY AGREEMENT | 75 clear direction when brought on to tasks and do they feel that their work has impact? 41. Are there any barriers in communication with NREL technical staff? (to be added by the researcher) AOI-3.3 Does NREL have a robust strategy/theory of change for how TA provided in the work plan will contribute to on the ground mitigation results? NONE (to be added by the researcher) AOI-3.4 Do the results being monitoring under the IAA reflect the overall strategy and goals of the agreement? NONE (to be added by the researcher) AOI-3.5 What barriers does NREL face in implementing their work plan on time and as planned, that USAID can address? 42. What changes could be made to improve the work plan timeline and help NREL meet the Mission requests within their time frame? (to be added by the researcher) AOI-3.6 How, and to what extent, has this mechanism helped to promote consistency and continuity of program technical support under the EC-LEDS program? NONE (to be added by the researcher) AOI-3.7 How well does NREL coordinate with other USG programs and agencies working in the same sector and country? 43. Are you aware of any other USG partners in your country who are collaborating on the IAA activity? (to be added by the researcher) 44. What are some recommendations for changes that could improve coordination with other USG teams? AOI-3.8 Are gender issues sufficiently addressed in work planning or implementation? 45. Does the mission provide feedback to NREL about how groups may or may not be benefiting disproportionality? (to be added by the researcher) 46. Has NREL discussed gender effects for products/tools requested by your mission? DEMOGRAPHICS 47. What is your highest level of education? 48. What is your field of expertise? (to be added by the researcher) Closing Remarks  “We are now at the end of the interview”  “THANK YOU for your time and for sharing your experience”  “Do you have any questions for me, for our evaluation team?”  “Your participation in this interview will contribute greatly to the evaluation team’s ability better understand the effects of the IAA and its contributions to meet Mission needs”  “The information from this interview will be analyzed with other interviews aggregated. The results will not show your response with your name.”  “Recoding data will not be shared with USAID or NREL staff. Only the evaluation team will have the raw recorded data” STOP RECORDER: PERFORMANCE EVALUATION OF THE USAID–DOE INTERAGENCY AGREEMENT | 76 KII INTERVIEW PROTOCOL: NREL CORE  Paper Copy of the Interview Document (this document)  Recorder and batteries  Pen and paper  Confirmation of time and location of interview meeting DATE: Day _____ Month _____ 2018 RESEARCHER NAME: ☐ Carolyn Fonseca ☐ Isaac Morrison LOCATION/TIME: PARTICIPANTS: [see sign-in sheet] GENDER: ☐ Male _______ ☐ Female ______ Questions START HERE: 1. Introduction “Hello, my name is __________________ and I am an evaluation consultant for a Company called MSI (Management Systems International)” 2. Purpose “MSI has been contracted by USAID to conduct an independent and external evaluation of the Inter-Agency Agreement (IAA) between DOE NREL and USAID. This purpose of this evaluation is to assess the performance of the implementation of this collaboration and its activities” “We are conducting interviews to collect data from stakeholders, our experiences and perspectives about the IAA activities. We will be interviewing staff from both USAID and NREL, USAID missions, implementing partners, tool users and other in-country mission partners” “Findings from the evaluation will provide recommendations on how to improve future IAA efforts” 3. Interview Process “Today I will ask you some simple questions about your experiences and views” “The interview should take us about 45 minutes to 1 hour” 4. Consent “Before we start with introductions and interview questions, we are required to ask for your consent”  You have the right to not participate, this is completely voluntary  You have the right to stop at any point  If there is anything you do not understand, please ask me to clarify” PERFORMANCE EVALUATION OF THE USAID–DOE INTERAGENCY AGREEMENT | 77  You have the right to decline to answer any question  The information you provide is only for our evaluation and your responses will be protected. The information will be grouped so no one can be identified Consent “Do we have your consent to participate in this Focus Group?” ☐ YES ☐ NO Consent to Record “We would like to record this interview, ONLY for accuracy of our notes, and not to share with anyone else, including USAID or NREL” “Do we have your consent to RECORD?” ☐ YES ☐ NO START RECORDER: The Inter-Agency Agreement between USAID and DOE has supported a variety of activities initiated in 2009. In 2017, the two agencies singed a new agreement directly with the National Renewable Energy Lab (NREL), a Department of Energy laboratory in Golden CO. The evaluation will investigate which activities should be prioritized going forward, identify areas for change/improvement, and provide recommendations/guidance in future efforts under the new IAA.  In this interview we would like to focus on your experience, role, and perspective of IAA activities/products. EQ AND AOI KII QUESTION RESPONSES GENERAL INFORMATION 1. What has been your role/involvement under this IAA? (Connection to IAA and experience) (to be added by the researcher) 2. Can you confirm the name of your organization (to be added by the researcher) 3. How long have you been at _______ organization? (to be added by the researcher) 4. How long have you been on IAA activities? (to be added by the researcher) 5. Have you interacted directly with USAID staff (mission or DC?)? if yes, for what activity (to be added by the researcher) EQ1: To what extent has the USAID￾DOE IAA met the needs of missions, and what specific opportunities exist to improve the buy-in program? 6. Do you feel the needs of the Missions are being met by the IAA, if yes why, if no why not? (to be added by the researcher) AOI-1.1 How, and to what extent, has this mechanism allowed USAID Mission and partner country access to NREL expertise and cutting-edge analysis, research, and deployment? 7. Have there been requests by Missions/partners which could not be met? Why? (to be added by the researcher) PERFORMANCE EVALUATION OF THE USAID–DOE INTERAGENCY AGREEMENT | 78 EQ AND AOI KII QUESTION RESPONSES AOI-1.2 In what specific ways has this mechanism complemented capabilities of local contractors, partners, or other contract mechanisms used by the field Missions? NONE (to be added by the researcher) AOI-1.3 How can coordination between E3, NREL, Mission and local implementing partners be improved? 8. How can coordination between E3, NREL, Mission and local implementing partners be improved? (to be added by the researcher) AOI-1.4 What types of technical expertise or assistance from NREL and other DOE Labs have been of highest value to partner governments or other Mission stakeholders? What type of technical assistance delivery models have been most effective? 9. What types of technical expertise or assistance from NREL and other DOE labs have been of highest value to partner governments or other Mission stakeholders? (to be added by the researcher) 10. What type of technical assistance delivery models have been most effective? AOI-1.5 In what ways have lessons been transferred to the broader USG low emissions development portfolio and what have the benefits been? 11. How does the USG and/or other institutions agencies benefit from IAA activities? (to be added by the researcher) EQ2: To what extent are the clean energy technical platforms being used effectively by their target audiences? 12. Do you know if these platforms supported through the IAA mechanism are being used? (to be added by the researcher) 13. If yes, by who and how? 14. If yes, are they being used effectively (as defined by responsiveness, relevance and utility)? 15. If no, why? 16. Who do you think the target audiences are? AOI-2.1 How are these technical platforms and tools being used by target audiences and where/why might they be underutilized? NONE (to be added by the researcher) AOI-2.2 What were the primary drivers that informed the types of technical platforms and other resources that have been developed? How are these drivers and user needs evolving? NONE (to be added by the researcher) AOI-2.3 What types of policies, programs, and country capacity development are being informed or advanced through these technical platforms and related technical assistance? 17. Which platforms currently could use additional capacity building efforts? And why? (to be added by the researcher) 18. Do you have an example of someone applying their training? 19. Which tool do you perceived to be the most valuable tool, and why? 20. Do you know of specific instances where the tool or platform was used in a decision-making process? PERFORMANCE EVALUATION OF THE USAID–DOE INTERAGENCY AGREEMENT | 79 EQ AND AOI KII QUESTION RESPONSES AOI-2.4 Do host country counterparts credit the platforms, tools or data derived from them with helping to drive policy or regulatory change, improve planning processes, or drive investment decisions? NONE (to be added by the researcher) AOI-2.5 To what extent have host country counterparts, or “owners”, of the tools committed to the long-term operational costs required to keep model output current? 21. Are there any sustainability plans in place? (to be added by the researcher) 22. How have host country counterparts, or “owners”, of the tools assisted with the long term operational needs required to keep model output current? EQ3: What are the strengths and weaknesses of the strategic planning, program coordination and management, communications, monitoring and reporting processes of the IAA? What are specific ways to make the program more efficient and effective? NONE (to be added by the researcher) AOI-3.1 Does NREL have the right number of staff to meet the needs of the IAA, and do staff have the right qualifications? 23. Have you had requests for which you did not have the sufficient team size or composition (i.e. area of expertise, years of experience, other)? (to be added by the researcher) AOI-3.2 Does the broader NREL team of experts feel that they have clear direction when brought on to tasks and do they feel that their work has impact? NONE (to be added by the researcher) AOI-3.3 Does NREL have a robust strategy/theory of change for how TA provided in the work plan will contribute to on the ground mitigation results? NONE (to be added by the researcher) AOI-3.4 Do the results being monitoring under the IAA reflect the overall strategy and goals of the agreement? NONE (to be added by the researcher) AOI-3.5 What barriers does NREL face in implementing their work plan on time and as planned, that USAID can address? 24. What type of barriers does NREL face when implementing their work plan? (to be added by the researcher) 25. What changes could be made to improve the work plan timeline and help NREL meet the Mission requests within their time frame? AOI-3.6 How, and to what extent, has this mechanism helped to promote consistency and continuity of program technical support under the EC-LEDS program? NONE (to be added by the researcher) AOI-3.7 How well does NREL coordinate with other USG programs and agencies working in the same sector and country? NONE (to be added by the researcher) PERFORMANCE EVALUATION OF THE USAID–DOE INTERAGENCY AGREEMENT | 80 EQ AND AOI KII QUESTION RESPONSES AOI-3.8 Are gender issues sufficiently addressed in work planning or implementation? 26. What are DOE processes for incorporating gender issues into research activities? (to be added by the researcher) 27. What are some examples of where gender issues were integrated into a specific activity? 28. What are barriers for women to participate in any phase of the activities funded by the IAA? 29. Are there any other groups who may be affected disproportionally? DEMOGRAPHICS 30. What is your highest level of education? 31. What is your field of expertise? (to be added by the researcher) Closing Remarks  “We are now at the end of the interview”  “THANK YOU for your time and for sharing your experience”  “Do you have any questions for me, for our evaluation team?”  “Your participation in this interview will contribute greatly to the evaluation team’s ability better understand the effects of the IAA and its contributions to meet Mission needs”  “The information from this interview will be analyzed with other interviews aggregated. The results will not show your response with your name.”  “Recoding data will not be shared with USAID or NREL staff. Only the evaluation team will have the raw recorded data” STOP RECORDER: PERFORMANCE EVALUATION OF THE USAID–DOE INTERAGENCY AGREEMENT | 81 KII INTERVIEW PROTOCOL: NREL TECH  Paper Copy of the Interview Document (this document)  Recorder and batteries  Pen and paper  Confirmation of time and location of interview meeting DATE: Day _____ Month _____ 2018 RESEARCHER NAME: ☐ Carolyn Fonseca ☐ Isaac Morrison LOCATION/TIME: PARTICIPANTS: [see sign-in sheet] GENDER: ☐ Male _______ ☐ Female ______ Questions START HERE: 1. Introduction “Hello, my name is __________________ and I am an evaluation consultant for a Company called MSI (Management Systems International)” 2. Purpose “MSI has been contracted by USAID to conduct an independent and external evaluation of the Inter-Agency Agreement (IAA) between DOE NREL and USAID. This purpose of this evaluation is to assess the performance of the implementation of this collaboration and its activities” “We are conducting interviews to collect data from stakeholders, our experiences and perspectives about the IAA activities. We will be interviewing staff from both USAID and NREL, USAID missions, implementing partners, tool users and other in-country mission partners” “Findings from the evaluation will provide recommendations on how to improve future IAA efforts” 3. Interview Process “Today I will ask you some simple questions about your experiences and views” “The interview should take us about 45 minutes to 1 hour” 4. Consent “Before we start with introductions and interview questions, we are required to ask for your consent”  You have the right to not participate, this is completely voluntary  You have the right to stop at any point  If there is anything you do not understand, please ask me to clarify” PERFORMANCE EVALUATION OF THE USAID–DOE INTERAGENCY AGREEMENT | 82  You have the right to decline to answer any question  The information you provide is only for our evaluation and your responses will be protected. The information will be grouped so no one can be identified Consent “Do we have your consent to participate in this Focus Group?” ☐ YES ☐ NO Consent to Record “We would like to record this interview, ONLY for accuracy of our notes, and not to share with anyone else, including USAID or NREL” “Do we have your consent to RECORD?” ☐ YES ☐ NO START RECORDER: The Inter-Agency Agreement between USAID and DOE has supported a variety of activities initiated in 2009. In 2017, the two agencies singed a new agreement directly with the National Renewable Energy Lab (NREL), a Department of Energy laboratory in Golden CO. The evaluation will investigate which activities should be prioritized going forward, identify areas for change/improvement, and provide recommendations/guidance in future efforts under the new IAA.  In this interview we would like to focus on your experience, role, and perspective of IAA activities/products. EQ AND AOI KII QUESTION RESPONSES GENERAL INFORMATION 1. What has been your role/involvement under this IAA? (Connection to IAA and experience) (to be added by the researcher) 2. Can you confirm the name of your organization (to be added by the researcher) 3. How long have you been at _______ organization? (to be added by the researcher) 4. How long have you been on IAA activities? (to be added by the researcher) 5. Have you interacted directly with USAID staff (mission or DC?)? if yes, for what activity (to be added by the researcher) EQ1: To what extent has the USAID￾DOE IAA met the needs of missions, and what specific opportunities exist to improve the buy-in program? 6. Do you feel the needs of the Missions are being met by the IAA, if yes why, if no why not? (to be added by the researcher) PERFORMANCE EVALUATION OF THE USAID–DOE INTERAGENCY AGREEMENT | 83 AOI-1.1 How, and to what extent, has this mechanism allowed USAID Mission and partner country access to NREL expertise and cutting-edge analysis, research, and deployment? 7. What is your role in responding to technical requests (and how are you tasked on that)? (to be added by the researcher) AOI-1.2 In what specific ways has this mechanism complemented capabilities of local contractors, partners, or other contract mechanisms used by the field Missions? NONE (to be added by the researcher) AOI-1.3 How can coordination between E3, NREL, Mission and local implementing partners be improved? 8. How can coordination between E3, NREL, Mission and local implementing partners be improved? (to be added by the researcher) AOI-1.4 What types of technical expertise or assistance from NREL and other DOE Labs have been of highest value to partner governments or other Mission stakeholders? What type of technical assistance delivery models have been most effective? 9. What types of technical expertise or assistance from NREL and other DOE Labs have been of highest value to partner governments or other Mission stakeholders? (to be added by the researcher) 10. What is currently missing in terms of expertise/assistance? (to be added by the researcher) 11. What type of technical assistance delivery models have been most effective? (to be added by the researcher) AOI-1.5 In what ways have lessons been transferred to the broader USG low emissions development portfolio and what have the benefits been? NONE (to be added by the researcher) EQ2: To what extent are the clean energy technical platforms being used effectively by their target audiences? 12. Is someone collecting data about these tools and their use? (to be added by the researcher) 13. Do you know if these platforms supported through the IAA mechanism are being used? (to be added by the researcher) 14. If yes, by who and how? (to be added by the researcher) 15. If yes, are they being used effectively (as defined by responsiveness, relevance and utility)? (to be added by the researcher) 16. If no, why? (to be added by the researcher) AOI-2.1 How are these technical platforms and tools being used by target audiences and where/why might they be underutilized? 17. What is the expectation of use? (to be added by the researcher) AOI-2.2 What were the primary drivers that informed the types of technical platforms and other resources that have been developed? How are these drivers and user needs evolving? NONE (to be added by the researcher) AOI-2.3 What types of policies, programs, and country capacity development are being informed or 18. Which platforms currently could use additional capacity building efforts? And why? (to be added by the researcher) PERFORMANCE EVALUATION OF THE USAID–DOE INTERAGENCY AGREEMENT | 84 advanced through these technical platforms and related technical assistance? 19. Do you have an example of someone applying their training? (to be added by the researcher) 20. Which tool do you perceived to be the most valuable tool, and why? (to be added by the researcher) 21. Do you know of specific instances where the tool or platform was used in a decision-making process? (to be added by the researcher) 22. What aspects of a platform make it accessible? (to be added by the researcher) 23. Is there anything about the process of tool creation that would change? (to be added by the researcher) 24. How do you think your tool or product is being used? (to be added by the researcher) 25. Who do you think could use your tool that might not be using it now? (to be added by the researcher) AOI-2.4 Do host country counterparts credit the platforms, tools or data derived from them with helping to drive policy or regulatory change, improve planning processes, or drive investment decisions? NONE (to be added by the researcher) AOI-2.5 To what extent have host country counterparts, or “owners”, of the tools committed to the long-term operational costs required to keep model output current? 26. Have host country counterparts, or “owners”, of the tools assisted with the long-term operational needs required to keep model output current? If yes, how? (to be added by the researcher) EQ3: What are the strengths and weaknesses of the strategic planning, program coordination and management, communications, monitoring and reporting processes of the IAA? What are specific ways to make the program more efficient and effective? 27. What are some specific ways to make the program more effective? (to be added by the researcher) AOI-3.1 Does NREL have the right number of staff to meet the needs of the IAA, and do staff have the right qualifications? 28. Does NREL have the right number of staff to meet the needs of the IAA, and do staff have the right qualifications? (to be added by the researcher) AOI-3.2 Does the broader NREL team of experts feel that they have clear direction when brought on to tasks and do they feel that their work has impact? NONE (to be added by the researcher) AOI-3.3 Does NREL have a robust strategy/theory of change for how TA provided in the work plan will contribute to on the ground mitigation results? NONE (to be added by the researcher) PERFORMANCE EVALUATION OF THE USAID–DOE INTERAGENCY AGREEMENT | 85 AOI-3.4 Do the results being monitoring under the IAA reflect the overall strategy and goals of the agreement? NONE (to be added by the researcher) AOI-3.5 What barriers does NREL face in implementing their work plan on time and as planned, that USAID can address? 29. What type of barriers does NREL technical staff face when implementing their work plan? (to be added by the researcher) 30. What changes could be made to improve the work plan timeline and help NREL meet the Mission requests within their time frame? [by USAID, by NREL] (to be added by the researcher) AOI-3.6 How, and to what extent, has this mechanism helped to promote consistency and continuity of program technical support under the EC-LEDS program? 31. Have you applied any of the lessons you’ve learned from this program in other contexts? (to be added by the researcher) AOI-3.7 How well does NREL coordinate with other USG programs and agencies working in the same sector and country? NONE (to be added by the researcher) AOI-3.8 Are gender issues sufficiently addressed in work planning or implementation? 32. What are some examples of where gender issues were integrated into a specific activity/tool? (to be added by the researcher) 33. Have you been asked to build a tool for specific group in mind? What was that process like? (to be added by the researcher) 34. Are there any other groups who may be affected disproportionally? (to be added by the researcher) DEMOGRAPHICS 35. What is your highest level of education? 36. What is your field of expertise? (to be added by the researcher) Closing Remarks - “We are now at the end of the interview” - “THANK YOU for your time and for sharing your experience” - “Do you have any questions for me, for our evaluation team?” - “Your participation in this interview will contribute greatly to the evaluation team’s ability better understand the effects of the IAA and its contributions to meet Mission needs” - “The information from this interview will be analyzed with other interviews aggregated. The results will not show your response with your name.” - “Recoding data will not be shared with USAID or NREL staff. Only the evaluation team will have the raw recorded data” STOP RECORDER: PERFORMANCE EVALUATION OF THE USAID–DOE INTERAGENCY AGREEMENT | 86 KII INTERVIEW PROTOCOL: MISSION PARTNERS  Paper Copy of the Interview Document (this document)  Recorder and batteries  Pen and paper  Confirmation of time and location of interview meeting DATE: Day _____ Month _____ 2018 RESEARCHER NAME: ☐ Carolyn Fonseca ☐ Isaac Morrison LOCATION/TIME: PARTICIPANTS: [see sign-in sheet] GENDER: ☐ Male _______ ☐ Female ______ QuestionsRES START HERE: 1. Introduction “Hello, my name is __________________ and I am an evaluation consultant for a Company called MSI (Management Systems International)” 2. Purpose “MSI has been contracted by USAID to conduct an independent and external evaluation of the Inter-Agency Agreement (IAA) between DOE NREL and USAID. This purpose of this evaluation is to assess the performance of the implementation of this collaboration and its activities” “We are conducting interviews to collect data from stakeholders, our experiences and perspectives about the IAA activities. We will be interviewing staff from both USAID and NREL, USAID missions, implementing partners, tool users and other in-country mission partners” “Findings from the evaluation will provide recommendations on how to improve future IAA efforts” 3. Interview Process “Today I will ask you some simple questions about your experiences and views” “The interview should take us about 45 minutes to 1 hour” 4. Consent “Before we start with introductions and interview questions, we are required to ask for your consent”  You have the right to not participate, this is completely voluntary  You have the right to stop at any point  If there is anything you do not understand, please ask me to clarify” PERFORMANCE EVALUATION OF THE USAID–DOE INTERAGENCY AGREEMENT | 87  You have the right to decline to answer any question  The information you provide is only for our evaluation and your responses will be protected. The information will be grouped so no one can be identified Consent “Do we have your consent to participate in this Focus Group?” ☐ YES ☐ NO Consent to Record “We would like to record this interview, ONLY for accuracy of our notes, and not to share with anyone else, including USAID or NREL” “Do we have your consent to RECORD?” ☐ YES ☐ NO START RECORDER: The Inter-Agency Agreement between USAID and DOE has supported a variety of activities initiated in 2009. In 2017, the two agencies singed a new agreement directly with the National Renewable Energy Lab (NREL), a Department of Energy laboratory in Golden CO. The evaluation will investigate which activities should be prioritized going forward, identify areas for change/improvement, and provide recommendations/guidance in future efforts under the new IAA.  In this interview we would like to focus on your experience, role, and perspective of IAA activities/products. EQ AND AOI KII QUESTION RESPONSES GENERAL INFORMATION 1. What has been your role/involvement under this IAA? (Connection to IAA and experience) (to be added by the researcher) 2. Can you confirm the name of your organization (to be added by the researcher) 3. How long have you been at _______ organization? (to be added by the researcher) 4. How long have you been on IAA activities? (to be added by the researcher) 5. Have you interacted directly with USAID staff (mission or DC?)? if yes, for what activity (to be added by the researcher) EQ1: To what extent has the USAID￾DOE IAA met the needs of missions, and what specific opportunities exist to improve the buy-in program? NONE (to be added by the researcher) AOI-1.1 How, and to what extent, has this mechanism allowed USAID Mission and partner NONE (to be added by the researcher) PERFORMANCE EVALUATION OF THE USAID–DOE INTERAGENCY AGREEMENT | 88 country access to NREL expertise and cutting-edge analysis, research, and deployment? AOI-1.2 In what specific ways has this mechanism complemented capabilities of local contractors, partners, or other contract mechanisms used by the field Missions? 6. In what specific ways has this mechanism complemented capabilities of local contractors, partners, or other contract mechanisms used by the field Missions? (to be added by the researcher) How can coordination between E3, NREL, Mission and local implementing partners be improved? NONE (to be added by the researcher) AOI-1.3 What types of technical expertise or assistance from NREL and other DOE Labs have been of highest value to partner governments or other Mission stakeholders? What type of technical assistance delivery models have been most effective? 7. How can coordination between E3, NREL, Mission and local implementing partners be improved? (to be added by the researcher) 8. What types of technical expertise or assistance from NREL and other DOE Labs have you received/used and which one(s) have been of highest value? And why? 9. What type of technical assistance delivery models have been most effective? AOI-1.4 In what ways have lessons been transferred to the broader USG low emissions development portfolio and what have the benefits been? NONE (to be added by the researcher) EQ2: To what extent are the clean energy technical platforms being used effectively by their target audiences? 10. Are you aware of any of the following technical platforms/tools?  Climatelinks  EC-LEDS  Greening the Grid (GtG)  Renewable Energy Data Explorer (RED-E)  Geospatial Toolkit (GsT)  Development Impact Assessment (DIA)  International Jobs and Economic Development Impacts (I-JEDI) If yes, go to tool user protocol (to be added by the researcher) AOI-2.1 How are these technical platforms and tools being used by target audiences and where/why might they be underutilized? NONE – only if ”Tool User” (to be added by the researcher) AOI-2.2 What were the primary drivers that informed the types of technical platforms and other resources that have been NONE – only if ”Tool User” (to be added by the researcher) PERFORMANCE EVALUATION OF THE USAID–DOE INTERAGENCY AGREEMENT | 89 developed? How are these drivers and user needs evolving? AOI-2.3 What types of policies, programs, and country capacity development are being informed or advanced through these technical platforms and related technical assistance? NONE – only if ”Tool User” (to be added by the researcher) AOI-2.4 Do host country counterparts credit the platforms, tools or data derived from them with helping to drive policy or regulatory change, improve planning processes, or drive investment decisions? NONE – only if ”Tool User” (to be added by the researcher) AOI-2.5 To what extent have host country counterparts, or “owners”, of the tools committed to the long-term operational costs required to keep model output current? NONE – only if ”Tool User” (to be added by the researcher) EQ3: What are the strengths and weaknesses of the strategic planning, program coordination and management, communications, monitoring and reporting processes of the IAA? What are specific ways to make the program more efficient and effective? 11. Have you had any interaction with NREL staff (and DOE)? If yes, how was that experience? (to be added by the researcher) 12. Who do you communicate with regarding the results of your IAA activity? 13. Do you share your results/outputs/outcomes/impacts with the public? If yes, how? 14. Are there ways in which communication can be improved? AOI-3.1 Does NREL have the right number of staff to meet the needs of the IAA, and do staff have the right qualifications? 15. Have you participated in any of the strategic planning process? (to be added by the researcher) 16. What else could be improved regarding NREL’s TA and capacity building? AOI-3.2 Does the broader NREL team of experts feel that they have clear direction when brought on to tasks and do they feel that their work has impact? NONE (to be added by the researcher) AOI-3.3 Does NREL have a robust strategy/theory of change for how TA provided in the work plan will contribute to on the ground mitigation results? NONE (to be added by the researcher) AOI-3.4 Do the results being monitoring under the IAA reflect the overall strategy and goals of the agreement? NONE (to be added by the researcher) AOI-3.5 What barriers does NREL face in implementing their work plan on time and as planned, that USAID can address? 17. Are you required to provide any reporting documents to USAID? (to be added by 18. Have you conducted any assessments or the researcher) evaluations for this activity? (e.g. needs assessment, baseline, PE, IE, other)? PERFORMANCE EVALUATION OF THE USAID–DOE INTERAGENCY AGREEMENT | 90 19. What barriers does NREL face in implementing their work plan on time and as planned, that USAID can address? AOI-3.6 How, and to what extent, has this mechanism helped to promote consistency and continuity of program technical support under the EC-LEDS program? NONE (to be added by the researcher) AOI-3.7 How well does NREL coordinate with other USG programs and agencies working in the same sector and country? NONE (to be added by the researcher) AOI-3.8 Are gender issues sufficiently addressed in work planning or implementation? 20. What gender issues should IP, USAID, and/or the tool designers need to consider? (to be added by the researcher) 21. Are you aware of any barriers for women or other groups in participating or being beneficiaries from this tool/platform/product (IAA funded activity)? DEMOGRAPHICS 22. What is your highest level of education? 23. What is your field of expertise? (to be added by the researcher) Closing Remarks  “We are now at the end of the interview”  “THANK YOU for your time and for sharing your experience”  “Do you have any questions for me, for our evaluation team?”  “Your participation in this interview will contribute greatly to the evaluation team’s ability better understand the effects of the IAA and its contributions to meet Mission needs”  “The information from this interview will be analyzed with other interviews aggregated. The results will not show your response with your name.”  “Recording data will not be shared with USAID or NREL staff. Only the evaluation team will have the raw recorded data” STOP RECORDER: PERFORMANCE EVALUATION OF THE USAID–DOE INTERAGENCY AGREEMENT | 91 KII INTERVIEW PROTOCOL: TOOL USERS  Paper Copy of the Interview Document (this document)  Recorder and batteries  Pen and paper  Confirmation of time and location of interview meeting DATE: Day _____ Month _____ 2018 RESEARCHER NAME: ☐ Carolyn Fonseca ☐ Isaac Morrison LOCATION/TIME: PARTICIPANTS: [see sign-in sheet] GENDER: ☐ Male _______ ☐ Female ______ Questions START HERE: 1. Introduction “Hello, my name is __________________ and I am an evaluation consultant for a Company called MSI (Management Systems International)” 2. Purpose “MSI has been contracted by USAID to conduct an independent and external evaluation of the Inter-Agency Agreement (IAA) between DOE NREL and USAID. This purpose of this evaluation is to assess the performance of the implementation of this collaboration and its activities” “We are conducting interviews to collect data from stakeholders, our experiences and perspectives about the IAA activities. We will be interviewing staff from both USAID and NREL, USAID missions, implementing partners, tool users and other in-country mission partners” “Findings from the evaluation will provide recommendations on how to improve future IAA efforts” 3. Interview Process “Today I will ask you some simple questions about your experiences and views” “The interview should take us about 45 minutes to 1 hour” 4. Consent “Before we start with introductions and interview questions, we are required to ask for your consent”  You have the right to not participate, this is completely voluntary  You have the right to stop at any point  If there is anything you do not understand, please ask me to clarify PERFORMANCE EVALUATION OF THE USAID–DOE INTERAGENCY AGREEMENT | 92  You have the right to decline to answer any question  The information you provide is only for our evaluation and your responses will be protected. The information will be grouped so no one can be identified Consent “Do we have your consent to participate in this Focus Group?” ☐ YES ☐ NO Consent to Record “We would like to record this interview, ONLY for accuracy of our notes, and not to share with anyone else, including USAID or NREL” “Do we have your consent to RECORD?” ☐ YES ☐ NO START RECORDER: The Inter-Agency Agreement between USAID and DOE has supported a variety of activities initiated in 2009. In 2017, the two agencies singed a new agreement directly with the National Renewable Energy Lab (NREL), a Department of Energy laboratory in Golden CO. The evaluation will investigate which activities should be prioritized going forward, identify areas for change/improvement, and provide recommendations/guidance in future efforts under the new IAA.  In this interview we would like to focus on your experience, role, and perspective of IAA activities/products. EQ AND AOI KII QUESTION RESPONSES [any respondent who states they use/used a tool will get additional questions about tool use] EQ2: To what extent are the clean energy technical platforms being used effectively by their target audiences? 1. NOTE: some ”users” may also be partners, in which case the evaluation may include other questions as well related to being a partner. (to be added by the researcher) AOI-2.1 How are these technical platforms and tools being used by target audiences and where/why might they be underutilized? 2. Are you currently using any of the following: (list the tools and ask them about these)? (to be added by the researcher) 3. If no, have you used them in the past? If yes, how, if no, why? 4. If yes, which ones, and how are you using them? 5. How did you learn about this tool? 6. Why are you using this tool? 7. Did you receive training for the tool, if so by whom and when? 8. What characteristics about this tool(s) are most helpful to you? AOI-2.2 What were the primary drivers that informed the types of technical platforms and other resources that have 9. Which components of the tools would you like to change? (to be added by the researcher) 10. Are there other tools or resources that you use instead of or in addition to this tool? 11. What other resources would be helpful to ensure you use the tool properly (or use the tool more often)? PERFORMANCE EVALUATION OF THE USAID–DOE INTERAGENCY AGREEMENT | 93 been developed? How are these drivers and user needs evolving? 12. Have you had any issues using the tool and did you submit a request/notification about this issue to someone? If yes, to whom? 13. Have you had any interactions with the tool providers/support personnel/technical advisors for the tool (i.e. DOE/NREL, USAID, other)? If yes in what way? 14. Do you provide any input or feedback on the tool, and if so, how and to whom? 15. Is the need for this tool a consistent need, or has it changed over time? Do you expect it to change in the future? AOI-2.3 What types of policies, programs, and country capacity development are being informed or advanced through these technical platforms and related technical assistance? 16. Have you used this tool for decision-making, and if so how? (to be added by the researcher) 17. Have you shared outputs/products from the tool with decision-makers? 18. Do you know of any others who have used this this tool (or information from this tool) for decision-making or policy￾making? AOI-2.4 Do host country counterparts credit the platforms, tools or data derived from them with helping to drive policy or regulatory change, improve planning processes, or drive investment decisions? 19. Did data or other outputs from the tool end up in other documents? (to be added by the 20. PROBE: if you used in a decision-making context, please share researcher) with us in what content, under what conditions, during what stage in the decision-making process? AOI-2.5 To what extent have host country counterparts, or “owners”, of the tools committed to the long￾term operational costs required to keep model output current? 21. Have you or your organization contributed to the tool's continued functionality in any ways? If so, how? (to be added by the researcher) 22. Are you still receiving information regarding the tool and its changes? 23. What is your incentive to continue using this tool? 24. What would be required for this tool to be sustainable for the next 10 years? 25. What new tool do you need or would you recommend be created? Closing Remarks  “We are now at the end of the interview”  “THANK YOU for your time and for sharing your experience”  “Do you have any questions for me, for our evaluation team?”  “Your participation in this interview will contribute greatly to the evaluation team’s ability better understand the effects of the IAA and its contributions to meet Mission needs”  “The information from this interview will be analyzed with other interviews aggregated. The results will not show your response with your name.”  “Recoding data will not be shared with USAID or NREL staff. Only the evaluation team will have the raw recorded data” STOP RECORDER: PERFORMANCE EVALUATION OF THE USAID–DOE INTERAGENCY AGREEMENT | 94 KII INTERVIEW PROTOCOL: IN-COUNTRY IMPLEMENTING PARTNERS  Paper Copy of the Interview Document (this document)  Recorder and batteries  Pen and paper  Confirmation of time and location of interview meeting DATE: Day _____ Month _____ 2018 RESEARCHER NAME: ☐ Carolyn Fonseca ☐ Isaac Morrison LOCATION/TIME: PARTICIPANTS: [see sign-in sheet] GENDER: ☐ Male _______ ☐ Female ______ Questions START HERE: 1. Introduction “Hello, my name is __________________ and I am an evaluation consultant for a Company called MSI (Management Systems International)” 2. Purpose “MSI has been contracted by USAID to conduct an independent and external evaluation of the Inter-Agency Agreement (IAA) between DOE NREL and USAID. This purpose of this evaluation is to assess the performance of the implementation of this collaboration and its activities” “We are conducting interviews to collect data from stakeholders, our experiences and perspectives about the IAA activities. We will be interviewing staff from both USAID and NREL, USAID missions, implementing partners, tool users and other in-country mission partners” “Findings from the evaluation will provide recommendations on how to improve future IAA efforts” 3. Interview Process “Today I will ask you some simple questions about your experiences and views” “The interview should take us about 45 minutes to 1 hour” 4. Consent “Before we start with introductions and interview questions, we are required to ask for your consent”  You have the right to not participate, this is completely voluntary  You have the right to stop at any point  If there is anything you do not understand, please ask me to clarify” PERFORMANCE EVALUATION OF THE USAID–DOE INTERAGENCY AGREEMENT | 95  You have the right to decline to answer any question  The information you provide is only for our evaluation and your responses will be protected. The information will be grouped so no one can be identified Consent “Do we have your consent to participate in this Focus Group?” ☐ YES ☐ NO Consent to Record “We would like to record this interview, ONLY for accuracy of our notes, and not to share with anyone else, including USAID or NREL” “Do we have your consent to RECORD?” ☐ YES ☐ NO START RECORDER: The Inter-Agency Agreement between USAID and DOE has supported a variety of activities initiated in 2009. In 2017, the two agencies singed a new agreement directly with the National Renewable Energy Lab (NREL), a Department of Energy laboratory in Golden CO. The evaluation will investigate which activities should be prioritized going forward, identify areas for change/improvement, and provide recommendations/guidance in future efforts under the new IAA.  In this interview we would like to focus on your experience, role, and perspective of IAA activities/products. EQ AND AOI KII QUESTION RESPONSES GENERAL INFORMATION 1. What has been your role/involvement under this IAA? (Connection to IAA and experience) (to be added by the researcher) 2. Can you confirm the name of your organization (to be added by the researcher) 3. How long have you been at _______ organization? (to be added by the researcher) 4. How long have you been on IAA activities? (to be added by the researcher) 5. Have you interacted directly with USAID staff (mission or DC?)? if yes, for what activity (to be added by the researcher) EQ1: To what extent has the USAID￾DOE IAA met the needs of missions, and what specific opportunities exist to improve the buy-in program? NONE (to be added by the researcher) AOI-1.1 How, and to what extent, has this mechanism allowed USAID Mission and partner country access to 6. What type of expertise, cutting edge analysis, research, and deployment has NREL brought to your projects? (to be added by the researcher) PERFORMANCE EVALUATION OF THE USAID–DOE INTERAGENCY AGREEMENT | 96 NREL expertise and cutting-edge analysis, research, and deployment? 7. How has this complemented the capabilities of the local contractors, partners, or other contract mechanisms used by the field Missions? (to be added by the researcher) AOI-1.2 In what specific ways has this mechanism NONE (to be added by the researcher) AOI-1.3 How can coordination between E3, NREL, Mission and local implementing partners be improved? 8. ·How can coordination between E3, NREL, Mission and local implementing partners be improved? (to be added by the researcher) 9. Of the technical expertise or assistance you received from NREL and other DOE Labs, which is of the highest value, and why? 10. What type of technical assistance delivery modes have been most effective (responsiveness, relevance and utility)? What types of technical expertise or assistance from NREL and other DOE Labs have been of highest value to partner governments or other Mission stakeholders? What type of technical assistance delivery models have been most effective? 11. Do you feel working with NREL gave you access to data and/or expertise that you would not have otherwise had? (to be added by the researcher) AOI-1.4 In what ways have lessons been transferred to the broader USG low emissions development portfolio and what have the benefits been? 12. Are you aware of any of the following technical platforms/tools?  Climatelinks  EC-LEDS  Greening the Grid (GtG)  Renewable Energy Data Explorer (RED-E)  Geospatial Toolkit (GsT)  Development Impact Assessment (DIA)  International Jobs and Economic Development Impacts (I-JEDI) (to be added by the researcher) 13. If yes, are people using them? And to your knowledge, why? 14. What would you change about them if you had the chance? 15. NOTE: if the IP is also a “tool user” go to “tool user” questions EQ2: To what extent are the clean energy technical platforms being used effectively by their target audiences? NONE – only if ”Tool User” (to be added by the researcher) AOI-2.1 How are these technical platforms and tools being used by target audiences and where/why might they be underutilized? NONE – only if ”Tool User” (to be added by the researcher) AOI-2.2 What were the primary drivers that informed the types of technical platforms and other resources that have been developed? How are these drivers and user needs evolving? NONE – only if ”Tool User” (to be added by the researcher) PERFORMANCE EVALUATION OF THE USAID–DOE INTERAGENCY AGREEMENT | 97 AOI-2.3 What types of policies, programs, and country capacity development are being informed or advanced through these technical platforms and related technical assistance? NONE – only if ”Tool User” (to be added by the researcher) AOI-2.4 Do host country counterparts credit the platforms, tools or data derived from them with helping to drive policy or regulatory change, improve planning processes, or drive investment decisions? 16. Do you have any examples where one or more of the tools have been used in national/country-level decision-making or policy-making? (to be added by the researcher) AOI-2.5 To what extent have host country counterparts, or “owners”, of the tools committed to the long-term operational costs required to keep model output current? NONE – only if Tool User” (to be added by the researcher) EQ3: What are the strengths and weaknesses of the strategic planning, program coordination and management, communications, monitoring and reporting processes of the IAA? What are specific ways to make the program more efficient and effective? 17. Do you communicate your results/outputs to USAID leadership? How? (to be added by the researcher) 18. Do you communicate your results/outputs with NREL-DOE? How? 19. Do you share your results/outputs/outcomes/impacts with the public? How? 20. Are there other audiences for the results of your IAA activity? AOI-3.1 Does NREL have the right number of staff to meet the needs of the IAA, and do staff have the right qualifications? 21. What is your view on the effectiveness of this mechanism (IAA)? [ASK about responsiveness, relevance, and utility] (to be added by the researcher) 22. In what areas do you think the IAA would benefit from increased capacity (i.e. technical expertise or fields)? AOI-3.2 Does the broader NREL team of experts feel that they have clear direction when brought on to tasks and do they feel that their work has impact? 23. Have you had any interaction with NREL staff (and DOE)? If yes, how was that experience? (to be added by the researcher) AOI-3.3 Does NREL have a robust strategy/theory of change for how TA provided in the work plan will contribute to on the ground mitigation results? NONE (to be added by the researcher) AOI-3.4 Do the results being monitoring under the IAA reflect the overall strategy and goals of the agreement? NONE (to be added by the researcher) AOI-3.5 What barriers does NREL face in implementing their work plan on time and as planned, that USAID can address? 24. What indicators are you required to provide to USAID? (to be added by the 25. Have you conducted any assessments or researcher) evaluations for this activity? (e.g. needs assessment, baseline, PE, IE, other)? AOI-3.6 How, and to what extent, has this mechanism helped to promote consistency and continuity of program technical support under the EC-LEDS program? NONE (to be added by the researcher) AOI-3.7 How well does NREL coordinate with other USG programs and NONE (to be added by the researcher) PERFORMANCE EVALUATION OF THE USAID–DOE INTERAGENCY AGREEMENT | 98 agencies working in the same sector and country? AOI-3.8 Are gender issues sufficiently addressed in work planning or implementation? 26. In the design or implementation phase of this activity, what gender issues have you had to consider? (to be added by the researcher) 27. Have you identified any barriers for the participation of women or other groups? DEMOGRAPHICS 28. What is your highest level of education? 29. What is your field of expertise? (to be added by the researcher) Closing Remarks  “We are now at the end of the interview”  “THANK YOU for your time and for sharing your experience”  “Do you have any questions for me, for our evaluation team?”  “Your participation in this interview will contribute greatly to the evaluation team’s ability better understand the effects of the IAA and its contributions to meet Mission needs”  “The information from this interview will be analyzed with other interviews aggregated. The results will not show your response with your name.”  “Recoding data will not be shared with USAID or NREL staff. Only the evaluation team will have the raw recorded data” STOP RECORDER: PERFORMANCE EVALUATION OF THE USAID–DOE INTERAGENCY AGREEMENT | 99 SURVEY NARRATIVE AND QUESTIONS MSI Performance Evaluation of the USAID NREL Partnership 2018 Purpose USAID has commissioned Management Systems International (MSI), a TetraTech Company, to conduct an independent performance evaluation of the partnership between USAID and DOE/NREL. You are receiving this email because you were identified as an individual who attended a training/workshop implemented by NREL and funded through the USAID DOE-NREL Partnership. We, the MSI evaluation team, are conducting this survey to gather views and experiences of individuals who have taken a training or workshop related to the following tools: Greening the Grid (GtG) Renewable Energy Data Explorer (RED-E) or Geospatial Toolkits International Jobs and Economic Development Impacts (I-JEDI) Tool Development Impact Assessment (DIA) Toolkit Distributed Photovoltaic Technologies (DPV) The survey will ask questions about your expertise, background, views on training attended, application of knowledge/skills acquired, and solicit suggestions for improvements. Your participation will remain confidential and anonymous. The survey should take you between 10-15 minutes. We are grateful for your participation! Your feedback will help us improve our knowledge about the efficacy of the trainings/workshops as they relate to tool/platform-usage. Please feel free to email me should you have any questions regarding this survey and evaluation activity. Thank you for your time! Sincerely, Dr. Carolyn Fonseca E3PESurvey2018@msi-inc.com Evaluation Team Leader Management Systems International (MSI), a Tetra Tech Company [The United States (US) Agency for International Development (USAID) is a US government agency supporting international programs to help developing countries. The US Department of Energy (DOE) is the US government agency for all issues concerning energy in the United States and has several national laboratories including the National Renewable Energy Laboratory (NREL).] Consent Before you start, we are required to ask for your consent to participate in our survey:  This survey is completely voluntary  You have the right to decline participation  You have the right to stop at any point  You have the right to decline to answer any question The more honest and true your information, the more accurate the recommendations. This improves the likelihood of appropriate programmatic course corrections by USAID and NREL. Your responses will be kept confidential and anonymous. 1. Do we have your consent for you to participate in this survey? [The information you provide is only for our evaluation reports, and the identification of the respondent will be kept confidential from USAID and NREL. Participation in this survey and poses no known risks to your person or property. A unique ID will be given to each survey, and all self-identifiers - including your email - will be removed from analysis. Data collected with personal identifiers will not be shared with any third party and only used for evaluation research purposes.] (Select one option)  Yes  No PERFORMANCE EVALUATION OF THE USAID–DOE INTERAGENCY AGREEMENT | 100 Survey Questions 2. Q2. In what sector do you primarily work? [If you work in more than one sector, please select the sector in which you spend 50% or more of your time.] (Select one option)  Energy Regulation/Policy/Planning  Energy Efficiency  Energy Transmission/Distribution  Energy Generation  Telecommunication/ Utilities  Other (Please specify) __________ 3. Q3. What kind of organization is your current employer? [If you work with more than one company, please check the box of the organization for which you work 50% or more. ] (Select one option)  Government/Public Sector  Not-for-Profit Organization/Community Organization  University/College/Academic or Research Institution  Private/Industry  International Donor/Assistance Organization  Not Currently Employed (retired, between jobs, unemployed)  Other (Please specify) __________ 4. Q4. Did you ATTEND training(s)on any of the following NREL supported tools/platforms between Jan. 2016 - Oct. 2018? [For this survey, tools are defined as instruments that consist of inputs￾output software or formulas/spreadsheets, such as RED-E, I-JEDI, and DIA. This survey also includes in the tools/platforms web-based repositories such as GtG and DPV which include documents, trainings, technical assistance and other features all acting as a tool to support the user. ] Yes No I attended a training before Jan. 2016 Do not remember a. Greening the Grid (GtG) (Select one option) b. International Jobs and Economic Development Impacts (I-JEDI) (Select one option) c. Renewable Energy Data Explorer (RED-E) or Geospatial Toolkits (Select one option) d. Development Impact Assessment Toolkit (DIA) (Select one option) e. Distributed Photovoltaic Technologies (DPV) (Select one option) 5. Q5. What year did you attend/take the training(s)? [If you attended multiple training in in different years, mark all that apply. If you attended multiple trainings in the same year, mark only that year. A training can be a webinar or an in-person training.]  2016  2017  2018  I cannot remember  None of the above 6. Q6. How did you receive your training(s)? [MARK ALL that apply] PERFORMANCE EVALUATION OF THE USAID–DOE INTERAGENCY AGREEMENT | 101  In-person training/workshop  Webinar/Online training  Other (Please specify) ____________ 7. Q7. To what extent do you agree or disagree with each statement: Strongly Agree Somewhat Agree Somewhat Disagree Strongly Disagree I did not attend tool(s)/ platform(s) trainings a. The training I received helped me in my work/job/projects. (Select one option) b. The training I received showed me a new tool/data/resources I had not known about before (Select one option) c. The training I received was about a new topic/field/subject matter (Select one option) d. The training I received met my expectations (Select one option) 8. Q8. Have you USED any of the following NREL supported tools/platforms between Jan. 2016 - Oct. 2018? Yes, during and after the training No Yes, but only during the training Do not remember a. Greening the Grid (GTG) (Select one option) b. International Jobs and Economic Development Impacts (I-JEDI) (Select one option) c. Renewable Energy Data Explorer (RED￾E) or Geospatial Toolkits (Select one option) d. Development Impact Assessment Toolkit (DIA) (Select one option) e. Distributed Photovoltaic Technologies (DPV) (Select one option) PERFORMANCE EVALUATION OF THE USAID–DOE INTERAGENCY AGREEMENT | 102 9. Q9. What was your primary use of the tool? for a report to provide information to a decision maker to help set a standard/quota /code/policy/ regulation for professional growth for research and journal publications/ papers I did not use this tool Other a. Greening the Grid (GtG) (Select one option) b. International Jobs and Economic Development Impacts (I￾JEDI) (Select one option) c. Renewable Energy Data Explorer (RED-E) or Geospatial Toolkits (Select one option) d. Development Impact Assessment Toolkit (DIA) (Select one option) e. Distributed Photovoltaic Technologies (DPV) (Select one option) 10. Q10. How did you hear about the tool? [Mark all that apply.]  From a website  During the NREL training  Through a colleague  Other (Please specify) ____________ 11. Q11. What was your favorite feature(s) of the tool(s)/platform(s)? [A "feature" could be any specific aspect or characteristic of the tool/platform. ] [short answer] PERFORMANCE EVALUATION OF THE USAID–DOE INTERAGENCY AGREEMENT | 103 12. Q12. Do you think you will use the tool(s)/platform(s) again? Yes No Not Applicable (did not use the tool) Do not know a. Greening the Grid (GtG) (Select one option) b. International Jobs and Economic Development Impacts (I-JEDI) (Select one option) c. Renewable Energy Data Explorer (RED-E) or Geospatial Toolkits (Select one option) d. Development Impact Assessment Toolkit (DIA) (Select one option) e. Distributed Photovoltaic Technologies (DPV) (Select one option) 13. Q13. What additional feature or change would you recommend be added to the tool(s)/platform(s)? [short answer] 14. Q14. Is there a tool you need in your field of work that currently does not exist? [Please share the type of features you would desire, purpose or use of the tool, and any other information that could be helpful.] [short answer] 15. Q15. If you stated YES to using one of the tools/platforms in Q7, would you be willing to participate in a telephone/skype interview with our team so we can learn more about your tool experience? (Select one option)  Yes  No 16. Q15b. If you stated YES to using one of the tools/platforms in Q7, would you be willing to participate in a telephone/skype interview with our team so we can learn more about your tool experience? (skip and leave blank if you 1) did not use any of the tools/platforms or 2) if you do not wish to volunteer for an interview) [ Answer this question only if answer to Q#15 is Yes ] Title: [text field] First Name: [text field] Last Name: [text field] Email Address: [text field] Final Questions 17. Q16. In what country do you currently reside? [The word "reside" refers to where you live. For example, you may do regional work traveling to Kenya, Rwanda, Ethiopia, but live in Uganda. Please write down where your home-base is located.] (Select one option) [select country from drop-down list] 18. Q17. What is your gender? (Select one option)  Male  Female  Do not wish to say PERFORMANCE EVALUATION OF THE USAID–DOE INTERAGENCY AGREEMENT | 104 19. Q18. Are you currently married? (Select one option)  No  Yes  Do not wish to say 20. Q19. How many children do you have? (Select one option)  None (0)  1  2  3  4 or more 21. Q20. What is your age? (Select one option)  18-24  25-34  35-44  45-54  55-64  65-74  75 or older 22. Q21. What is the highest degree or level of school you have completed? [If you’re currently enrolled in school, please indicate the highest degree you have received.] (Select one option)  Less than high school degree  High school degree  Some college, but no degree  Other (please specify) __________  College degree (e.g. AA, BS, BA)  Vocational Degree  Masters Degree (e.g. MA, MS, MPA, MBA)  Doctorate degree (PhD, EdD, other)  Professional degree (e.g. MD, Law, DDM, DMV, other) 23. Q22. How long have you been with your current employer? [If you have more than one employer, please select the time reflective of your primary employer (spending 50% or greater of your time working for this employer).] (Select one option)  Less than 6 months  More than 6 months but less than 1 year  More than 1 year but less than 5 years  More than 5 years 24. Q23. Are you willing to share the name of the organization of your current employer? (Select one option)  Yes  No PERFORMANCE EVALUATION OF THE USAID–DOE INTERAGENCY AGREEMENT | 105 25. Q24. If yes, please write the name of your organization in the space below: [Answer this question only if answer to Q#24 is Yes] [short answer] Thank you for your participation! The information you have provided will be critical in helping USAID and NREL align programs to user needs. A copy of the final report including results from this survey will be available on the USAID DEC https://dec.usaid.gov/dec/home/ after the evaluation has been completed. If you have any questions or further issues regarding this survey please email E3PESurvey2018@msi-inc.com. PERFORMANCE EVALUATION OF THE USAID–DOE INTERAGENCY AGREEMENT | 106 FGD PROTOCOL: NREL GROUP 1  Paper Copy of the Protocol Document (this document)  Recorder and batteries (sound only)  Pen and paper  Confirmation of time and location of interview meeting  Note taker ________________  Facilitator ________________ DATE: Day _____ Month _____ 2018 RESEARCHER NAME: ☐ Carolyn Fonseca ☐ Isaac Morrison LOCATION/TIME: PARTICIPANTS: [see sign-in sheet] GENDER: ☐ Male _______ ☐ Female ______ Questions START HERE: 1. Introduction “Hello, my name is __________________ and I am an evaluation consultant for a Company called MSI (Management Systems International)” 2. Purpose “MSI has been contracted by USAID to conduct an independent and external evaluation of the Inter-Agency Agreement (IAA) between DOE NREL and USAID. This purpose of this evaluation is to assess the performance of the implementation of this collaboration and its activities” “We are conducting interviews and focus group discussions to collect data from stakeholders, our experiences and perspectives about the IAA activities. We will be conducting 2 FGs in Colorado with NREL” “Findings from the evaluation will provide recommendations on how to improve future IAA efforts” 3. FGD Process “Today we will ask some questions about the IAA and its activities. Together will be discuss these topics and as a team co-create data” “The focus group discussion (FGD) should take us about 2 hours” 4. Consent “Before we start with introductions and FG activities, we are required to ask for your consent”  You have the right to not participate, this is completely voluntary  You have the right to stop at any point PERFORMANCE EVALUATION OF THE USAID–DOE INTERAGENCY AGREEMENT | 107  If there is anything you do not understand, please ask me to clarify”  You have the right to decline to answer any question  The information you provide is only for our evaluation and your responses will be protected. The information will be grouped so no one can be identified Consent “Do we have your consent to participate in this Focus Group?” ☐ YES ☐ NO Consent to Record “We would like to record this interview, ONLY for accuracy of our notes, and not to share with anyone else, including USAID or NREL” “Do we have your consent to RECORD?” ☐ YES ☐ NO START RECORDER: The Inter-Agency Agreement between USAID and DOE has supported a variety of activities initiated in 2009. In 2017, the two agencies singed a new agreement directly with the National Renewable Energy Lab (NREL), a Department of Energy laboratory in Golden CO. The evaluation will investigate which activities should be prioritized going forward, identify areas for change/improvement, and provide recommendations/guidance in future efforts under the new IAA.  Through this Focus Group Discussion (FGD) we would like to learn about on your experience, role, and perspective of IAA activities/products. Focus Group Protocol: NREL Target Sample: 4-8 volunteers at NREL who are technical staff involved directly with IAA activities, technical platforms, and tools Purpose: To gather data by collectively working with each group to identify their views and perspectives Process: Audio recording, with supplementary notetaking. Consent: [see above] Ground Rules: [Will include openness, respect, and other standard rules in FGD implementation co-developed at the start with the group to ensure they feel] Illustrative examples of possible rules from the group:  Individuals will have the right to share freely and openly their views with no repercussion  Individuals will permit everyone the opportunity to share ideas and perspectives  Individuals will respect each other  There will be breaks to permit networking between the members and encourage collaboration/group building  The focus group will encourage team building to permit a collaborative environment PERFORMANCE EVALUATION OF THE USAID–DOE INTERAGENCY AGREEMENT | 108 FOCUS GROUP QUESTIONS Theme/ Area of Inquiry Matrix Match Questions Monitoring and Reporting EQ3 In your view what metrics capture impact of the tool? Are the metrics sufficient? What does a successful IAA look like? What improvements do you recommend should be made to capture more benefits and impacts of the IAA products? Platforms and Tools EQ2 Who benefits the most from the tools? Who is the target audience, and are they always clear? What does a sustainable tool/platform look like to NREL? Are the tools and platform being used effectively (and what is effective to NREL)? Technical Staff at USAID and Missions, what has been your experience communicating technical information with USAID and IPs? Are decision-making processes explicit for your tools/platforms prior to their creation? Barriers to Work (BOTH FGD groups) AOI-3.5 What are some examples of barriers affecting your ability to provide the platforms/output/products on time to Missions? What are some solutions to mitigate these obstacles? Future Technical Needs (BOTH FGD groups) For recommendations What should Missions and countries be ‘getting ready for’ in terms of products/platforms and fields of expertise? Gender Issues (BOTH FGD groups) AOI-3.8 Where does gender play a role in the provision of IAA products and tools? What types of considerations does NREL take into account when building tools for diverse users? Closing Remarks  “We are now at the end of the focus group”  “THANK YOU for your time and for sharing your experience”  “Do you have any questions for our evaluation team?”  The information from this interview will be analyzed with other interviews aggregated. The results will not show your response with your name.  Your participation will contribute greatly improving our understanding of the strengths and weaknesses of the IAA to improve future collaborative work with DOE as well as support of Missions and in-country emission goals. STOP RECORDER: PERFORMANCE EVALUATION OF THE USAID–DOE INTERAGENCY AGREEMENT | 109 FGD PROTOCOL: NREL GROUP 2  Paper Copy of the Protocol Document (this document)  Recorder and batteries (sound only)  Pen and paper  Confirmation of time and location of interview meeting  Note taker: ________________  Facilitator: ________________ DATE: Day _____ Month _____ 2018 RESEARCHER NAME: ☐ Carolyn Fonseca ☐ Isaac Morrison LOCATION/TIME: PARTICIPANTS: [see sign-in sheet] GENDER: ☐ Male _______ ☐ Female ______ Questions START HERE: 1. Introduction “Hello, my name is __________________ and I am an evaluation consultant for a Company called MSI (Management Systems International)” 2. Purpose “MSI has been contracted by USAID to conduct an independent and external evaluation of the Inter-Agency Agreement (IAA) between DOE NREL and USAID. This purpose of this evaluation is to assess the performance of the implementation of this collaboration and its activities” “We are conducting interviews and focus group discussions to collect data from stakeholders, our experiences and perspectives about the IAA activities. We will be conducting 2 FGs in Colorado with NREL” “Findings from the evaluation will provide recommendations on how to improve future IAA efforts” 3. FGD Process “Today we will ask some questions about the IAA and its activities. Together will be discuss these topics and as a team co-create data” “The focus group discussion (FGD) should take us about 2 hours” 4. Consent “Before we start with introductions and FG activities, we are required to ask for your consent”  You have the right to not participate, this is completely voluntary  You have the right to stop at any point PERFORMANCE EVALUATION OF THE USAID–DOE INTERAGENCY AGREEMENT | 110  If there is anything you do not understand, please ask me to clarify”  You have the right to decline to answer any question  The information you provide is only for our evaluation and your responses will be protected. The information will be grouped so no one can be identified Consent “Do we have your consent to participate in this Focus Group?” ☐ YES ☐ NO Consent to Record “We would like to record this interview, ONLY for accuracy of our notes, and not to share with anyone else, including USAID or NREL” “Do we have your consent to RECORD?” ☐ YES ☐ NO START RECORDER: The Inter-Agency Agreement between USAID and DOE has supported a variety of activities initiated in 2009. In 2017, the two agencies singed a new agreement directly with the National Renewable Energy Lab (NREL), a Department of Energy laboratory in Golden CO. The evaluation will investigate which activities should be prioritized going forward, identify areas for change/improvement, and provide recommendations/guidance in future efforts under the new IAA.  Through this Focus Group Discussion (FGD) we would like to learn about on your experience, role, and perspective of IAA activities/products. Focus Group Protocol: NREL Target Sample: 4-8 volunteers at NREL who are technical staff involved directly with IAA activities, technical platforms, and tools Purpose: To gather data by collectively working with each group to identify their views and perspectives Process: Audio recording, with supplementary notetaking. Consent: [see above] Ground Rules: [Will include openness, respect, and other standard rules in FGD implementation co-developed at the start with the group to ensure they feel] Illustrative examples of possible rules from the group:  Individuals will have the right to share freely and openly their views with no repercussion  Individuals will permit everyone the opportunity to share ideas and perspectives  Individuals will respect each other  There will be breaks to permit networking between the members and encourage collaboration/group building  The focus group will encourage team building to permit a collaborative environment PERFORMANCE EVALUATION OF THE USAID–DOE INTERAGENCY AGREEMENT | 111 FOCUS GROUP QUESTIONS Theme/Area of Inquiry Matrix Match Questions Strategic Planning AOI-2.1 AOI-2.2 EQ3 AOI-3.1 How are activities selected? What are the drivers Who participates/how are teams created? How are the needs evolving, how to take that into account in the work plans? Strengths and weaknesses in strategic planning? Clarity on Tasks AOI-3.2 AOI-3.3 Does the NREL team feel there is clear direction when brought on to tasks? Are teams clear about their impact? Management and Coordination AOI-3.7 How does NREL coordinate with other USG agencies in country? What would help improve coordination? Does coordination differ between TA for buy-in missions vs. non-buy missions? What additional roles/staff are needed to help coordination and management of the IAA (at NREL at USAID)? Barriers to Work (BOTH FGD groups) AOI-3.5 What are some examples of barriers affecting your ability to provide the platforms/output/products on time to Missions? What are some solutions to mitigate these obstacles? Future Technical Needs (BOTH FGD groups) For recommendations What should Missions and countries be ‘getting ready for’ in terms of products/platforms and fields of expertise? Gender Issues (BOTH FGD groups) AOI-3.8 Where does gender play a role in the provision of IAA products and tools? What types of considerations does NREL take into account when building tools for diverse users? Closing Remarks  “We are now at the end of the focus group”  “THANK YOU for your time and for sharing your experience”  “Do you have any questions for our evaluation team?” PERFORMANCE EVALUATION OF THE USAID–DOE INTERAGENCY AGREEMENT | 112  The information from this interview will be analyzed with other interviews aggregated. The results will not show your response with your name.  Your participation will contribute greatly improving our understanding of the strengths and weaknesses of the IAA to improve future collaborative work with DOE as well as support of Missions and in-country emission goals. STOP RECORDER: PERFORMANCE EVALUATION OF THE USAID–DOE INTERAGENCY AGREEMENT | 113 PROCESS MAPPING WORKSHOP PROTOCOL  Supplies for workshop  Recorder and batteries  Pen and paper  Sign-in sheet  Consent form DATE: Day _____ Month _____2018 RESEARCHER NAME: ☐ Carolyn Fonseca ☐ Isaac Morrison LOCATION/TIME: PARTICIPANTS: [see sign-in sheet] GENDER: ☐ Male _______ ☐ Female ______ Introduction START HERE: 1. Introduction “Hello, my name is __________________ and I am an evaluation consultant for a Company called MSI (Management Systems International)” 2. Purpose “MSI has been contracted by USAID to conduct an independent and external evaluation of the Inter-Agency Agreement (IAA) between DOE NREL and USAID. This purpose of this evaluation is to assess the performance of the implementation of this collaboration and its activities” “We are holding this workshop at NREL today with both USAID and NREL staff to learn map the process to create a product through the IAA mechanism, identify gaps/barriers, co-created ideas/solutions, and develop a framework for a theory of change for the IAA” “Findings from the evaluation will provide recommendations on how to improve future IAA efforts” 3. Workshop Process “Today I will ask you some simple questions about your experiences and views” “The interview should take us about 2 hours” 4. Consent “Before we start with introductions and interview questions, we are required to ask for your consent”  You have the right to not participate, this is completely voluntary PERFORMANCE EVALUATION OF THE USAID–DOE INTERAGENCY AGREEMENT | 114  You have the right to stop at any point  If there is anything you do not understand, please ask me to clarify”  You have the right to decline to answer any question  The information you provide is only for our evaluation and your responses will be protected. The information will be grouped so no one can be identified Consent “Do we have your consent for an interview?” ☐ YES ☐ NO Consent to Record “We would like to record this workshop, sound only, ONLY for accuracy of our notes, and not to share with anyone else” “Do we have your consent to RECORD?” ☐ YES ☐ NO START RECORDER: The Inter-Agency Agreement between USAID and DOE has supported a variety of activities initiated in 2009. In 2017, the two agencies singed a new agreement directly with the National Renewable Energy Lab (NREL), a Department of Energy laboratory in Golden CO. The evaluation will investigate which activities should be prioritized going forward, identify areas for change/improvement, and provide recommendations/guidance in future efforts under the new IAA.  In this process mapping workshop we would like to focus on your experience, role, and perspective of IAA activities/products. Process Mapping Workshop [DRAFT Illustrative Agenda] Target Sample: The team will invite up 15-20 people (USAID, NREL management, NREL technical staff, others involved in the IAA process) to participate Purpose: To bring together the various stakeholders from NREL and USAID for an ideation workshop. This workshop will allow both groups to work together with the evaluation to create innovative solutions to the new IAA, learn about what has worked/or not worked well in the prior IAA, and set the framework for a joint casual model/framework to guide the new IAA (informing evaluation metrics and monitoring protocols). Data collected from this session will be part of the recommendation section in the final evaluation report. Information from this workshop may also highlight areas of inquiry for the KIIs of mission staff, in-country partners, and beneficiaries. Materials (pens, markers, paper, large sticky notes in colors, large post-it notes/dry eraser board, other workshop materials) Time: 1.5-2.0 hours Agenda 1. Introductions (use new ice breaker) 2. Review the purpose of the Workshop, expectations, ground rules, and products (what should be produced at the end) PERFORMANCE EVALUATION OF THE USAID–DOE INTERAGENCY AGREEMENT | 115 3. Move everyone around to 3-5 to a table or less (depending on number of people and venue) 4. Highlight words and definitions on the wall (i.e. theory of change, etc.) 5. Start the workshop 6. Summarize the findings 7. Close out Exercise 1 – In your opinion what are the three most important activities in this IAA? [Each person will write down three things, one per sticky note, they will then be asked to each place theses on the wall, then the group will re-arrange them and arrange them into clusters] - What clusters have emerged? Does anyone have any comments about these [are they obvious, do the categories seems a surprise] - Ask people to share why they selected their top IAA activities [what does ‘most important’ mean to them, how did they measure ‘most important’ – discovery value system to understand what success might look like, what ‘most valuable’ might look like] - Work with the group and have them write up some notes/conclusions from the discussions Exercise 2 – What are the target impacts of the IAA? [Again, ask each person write down the top 3 impacts of the IAA; this will identify what types of impacts are perceived by each group, what they define as an impact]  What are the clusters that emerge? Ask the group if these targets seem a surprise?  Should these be the target impacts? If not, what should they be?  Does anyone think there are any that are easier or more difficult to achieve under the current IAA mechanism? Coffee Break (10 minutes) Exercise 3 – [send everyone to the wall/board] Roadmap to achieve the intended impacts outlined in the IAA [write on one end the impacts you want to achieve; how do you get there – backcasting] [the goal here is to elicit the mental models of the various people working on the IAAs to see if the match or differ; what are the lens/framework through which they are seeing the IAA which defines their ‘impacts’; they should be stating variables or what they think each effort/activity leads to x change or impact. This should take about 10-15 minutes]  The group will then review the entire roadmap as a team  What do people believe might be missing?  Does it reflect the current pathway? [Insert evaluation question topics to cover] o Team composition to teach target (AOI 3.1) o Clarity by teams and agencies of tasks and impacts? (AOI 3.2) o Theory of change as it relates to TA, work plan, and results (AOI 3.3)  Are there any doubts by anyone about whether this path is the real path, how would we know?  Are there alternative paths to achieving these target impacts? PERFORMANCE EVALUATION OF THE USAID–DOE INTERAGENCY AGREEMENT | 116  What do you think we would need to measure to know we have met these targets? [help people identify metrics and understand the measurement of change and impact versus compliance] o How is change monitored, how should it be monitored (AOI 3.4)  What does success look like? [Final product should be ‘results framework’ a basic causal model or roadmap to achieve the target goals] Coffee Break (5 minutes to set up the call with the panel) Exercise 4 – Ideation Time! Let’s think about creative ideas and new approaches/tools/process to would evolve the IAA. 1. List ideas [everyone has lots of ideas about new science and tools that could be built that are not yet part of the IAA, be creative outside the box] 2. List the barriers and determine whether any of these ideas help solve any of these barriers 3. Vote/select as a group the top two ideas 4. What would it take to implement them (requirements, need assessment, who would we need to talk with, timeframe, resources, other players) [The group summarizes the ideas on large post-its and organizes them by category; during this activity people should be standing and walking around; what about risk? And how each agency views that, what about incubators, and pilots, how to test ideas] [Multipurpose workshop to ensure stakeholders get a chance and medium to ideate solutions to explicit issues or barriers in the IAA effectiveness; to share with each other the various perspectives of effectiveness of the IAA; to design new innovation options/activities; to ensure participation of USAID in the evaluation; to advance CLA priorities within USAID for funded activities; to extract mental models from each group and re-align expectations and perceptions about the IAA; to create a joint-agreed upon causal model/results framework pathway to achieve the stated IAA goals; to bring in lessons from other similar platforms and IAAs] Closing Remarks  “We are now at the end of the focus group”  “THANK YOU for your time and for sharing your experience”  “Do you have any questions for our evaluation team?”  The information from this interview will be analyzed with other interviews aggregated. The results will not show your response with your name. Your participation will contribute greatly improving our understanding of the strengths and weaknesses of the IAA to improve future collaborative work with DOE as well as support of Missions and in-country emission goals. PERFORMANCE EVALUATION OF THE USAID–DOE INTERAGENCY AGREEMENT | 117 ANNEX E: EVALUATION TEAM BIOS EVALUATION SPECIALIST/TEAM LEAD – DR. CAROLYN FONSECA Dr. Fonseca has more than 10 years of research and evaluation experience in gender, environment, economic development, science and technology, policy analysis and methods. She has broad expertise in designing and implementing evaluations across Latin America, Africa, and South Asia. She has been the technical lead for several USAID-funded projects, including serving as USAID’s project manager for the $1.5 million evaluation of the joint USAID-NASA SERVIR program, and associate director for USAID’s $14 million Peanut Cooperative Research Support Program grant. As NASA’s lead monitoring and evaluation engineer for SERVIR, she designed, coordinated, and implemented evaluation activities around the 11 SERVIR applied sciences teams in three regions to determine impact and performance. Dr. Fonseca worked for USAID/E3 as the primary survey technical advisor for the Agency-wide Construction Survey project, which examined construction activities carried out for missions and other operating units for a GAO report, working directly under the E3 deputy assistant administrator. The E3 Bureau awarded her the USAID Construction Survey Award in 2014. Dr. Fonseca also brings significant gender experience, including developing gender-sensitive survey instruments and leading interviews and focus group discussions with women. She has published articles in renowned journals and served as speaker at several conferences. Dr. Fonseca is fluent in English and Spanish and holds a Ph.D. in environmental policy from the Georgia Institute of Technology and M.S. degrees in public policy and biology. RESEARCH ASSOCIATE – ISAAC MORRISON Mr. Morrison is an anthropologist with more than six years of experience as an evaluator, analyst, and educator. He is an evaluation specialist and qualitative data analyst for USAID evaluations related to disaster management, natural resource management, water quality monitoring, climate change, hydrology, economic development, and environmental protection. He recently served as field team leader and remote team coordinator for data collection teams in Central America, East and West Africa, and South Asia, supporting all phases of research design and implementation. He has worked with a range of government agencies and nonprofit organizations, doing qualitative data collection and analysis. He holds a B.A. in cultural anthropology from the University of Maryland College Park and an M.A. in anthropology with an international development focus from George Washington University. HOME OFFICE SUPPORT E3 Analytics and Evaluation Project home office team members provided support to the core evaluation team, including technical coordination and guidance, quality control assurance, research assistance, quantitative and qualitative data analysis, administrative oversight, and logistical support. U.S. Agency for International Development 1300 Pennsylvania Avenue, NW Washington, DC 20004