FINAL EVALUATION REPORT USAID CLIMATE READY AND INSTITUTIONAL STRENGTHENING IN PACIFIC ISLAND COUNTRIES TO ADAPT TO CLIMATE CHANGE (ISACC) Limited Internal Distribution Contract No. AID-486-I-14-00001 (IDIQ) Task Order No. 72048619F00001 April 2022 This publication was produced at the request of the United States Agency for International Development (USAID). It was prepared independently by Social Impact, Inc. The authors’ views expressed in this document do not necessarily reflect the views of USAID or the United States Government. DISCLAIMER This publication was produced at the request of the United States Agency for International Development (USAID). It was prepared independently by Social Impact, Inc. The authors’ views expressed in this publication do not necessarily reflect the views of the United States Agency for International Development or the United States Government. USAID CLIMATE READY AND INSTITUTIONAL STRENGTHENING IN PACIFIC ISLAND COUNTRIES TO ADAPT TO CLIMATE CHANGE (ISACC) Final Performance Evaluation Contracted under AID-486-I-14-00001 (IDIQ); Task Order 72048619F00001 Prepared for: Nancy Ebuenga (Activity Manager) United States Agency for International Development Pacific Islands (USAID/Pacific Islands) Submitted: April 20, 2022 USAID.GOV DRAFT EVALUATION REPORT | i ABSTRACT The United States Agency for International Development in the Pacific Islands (USAID/Pacific Islands) commissioned a joint performance evaluation of the Climate Ready Activity and the climate finance component of the Institutional Strengthening in Pacific Island Countries to Adapt to Climate Change (ISACC) Activity. The evaluation examined the following: 1) impact on the enabling environment required to access climate financing; 2) contributions to building the capacity of government to design, implement, and monitor climate change projects; 3) whether climate funding activities, such as proposal development and accreditation, supported country-owned climate change actions; and 4) engagement of the private sector and climate finance institutions. Support for gender, equality, and social inclusion (GESI) was also a crosscutting theme for the evaluation. Both Climate Ready and ISACC demonstrated several successes, including influencing the enabling environment by developing laws, policies, frameworks, processes, and operating plans. In addition, the evaluation found evidence of increased government capacity, particularly in project management and monitoring and evaluation (M&E). The Activities’ approach of partnering with the University of the South Pacific/Pacific Technical and Further Education to offer capacity building courses with a certificate added value and increased participation. The support that the Activities provided for accreditation of organizations to international funds and for proposal preparation for submission to those funds has improved the conditions for implementation of country-owned climate actions in future. It has led to mobilization of funds and helped organizations with management policies and processes. At the same time, Climate Ready and ISACC faced challenges, notably budget cuts that limited implementation of several interventions and hampered relationships with government. Furthermore, the complexity and long timelines for obtaining international climate adaptation funding meant that only a few country-owned climate change projects were in place at the time of the evaluation. Another key challenge centered on engagement of the private sector. Both Climate Ready and ISACC undertook a few relevant initiatives, including business continuity planning and media training. However, private sector engagement occurred at a relatively low level, and this area has been identified as a key gap in the region. Finally, results related to GESI were mixed. When it was sufficiently designed and resourced, as was the case with Climate Ready, there was progress on increasing awareness of and plans for GESI. This was not the case for ISACC, where it was not included as a priority during the planning stages. Given these findings, the evaluation team recommended continuation and expansion of several approaches that demonstrated success, such as increasing capacity building opportunities in project management and M&E, supporting fully funded government positions or focal points in future projects, and building local skills to move away from the “fly in, fly out” international consultant model. Other areas, specifically private sector engagement and integration of GESI, require new and more substantial investment (in terms of resources and in designing dedicated activities) moving forward. USAID.GOV DRAFT EVALUATION REPORT | ii TABLE OF CONTENTS Abstract........................................................................................................................................... i List of Tables................................................................................................................................. iii List of Figures ............................................................................................................................... iii Acronyms ...................................................................................................................................... iv Executive Summary...................................................................................................................... v Introduction................................................................................................................................... 1 Overview.........................................................................................................................................................................1 Background.....................................................................................................................................................................1 Methodology Overview................................................................................................................. 4 Approach.........................................................................................................................................................................4 Evaluation Questions....................................................................................................................................................4 Data Collection..............................................................................................................................................................5 Analysis and Limitations...............................................................................................................................................6 Findings and Conclusions.............................................................................................................. 7 EQ 1: Enabling Environment.......................................................................................................................................7 EQ 2: Capacity Building .............................................................................................................................................13 EQ 3: Country-Owned Implementation................................................................................................................20 EQ 4: Private Sector and Institutional Engagement ............................................................................................27 Recommendations ...................................................................................................................... 30 Annex A: Statement of Work.................................................................................................... 35 Annex B: Methodology ............................................................................................................... 49 Annex C: Timeline ...................................................................................................................... 54 Annex D: Conflict-of-Interest Disclosures ................................................................................ 55 Annex E: Documents Reviewed................................................................................................. 63 Annex F: Respondents Consulted.............................................................................................. 68 Annex G: Data Collection Instruments and Informed Consent............................................. 69 Annex H: Laws, Policies, Plans, Procedures, Etc., Introduced or Implemented ................ 110 USAID.GOV DRAFT EVALUATION REPORT | iii LIST OF TABLES Table 1. Evaluation Questions........................................................................................................................................4 Table 2. KII Respondents: Numbers by Category and Demographics.................................................................6 Table 3. Climate Ready and ISACC Key Conclusions: EQ 1 ................................................................................13 Table 4. Climate Ready and ISACC Key Conclusions: EQ 2 ................................................................................20 Table 5. Climate Ready and ISACC Key Conclusions: EQ 3 ................................................................................26 Table 6. Climate Ready and ISACC Key Conclusions: EQ 4 ................................................................................29 LIST OF FIGURES Figure 1. Types and Volume of Climate Ready Interventions in Evaluation Focus Countries........................2 Figure 2. Types and Volume of ISACC Interventions in Evaluation Focus Countries......................................3 Figure 3: KII Discussions of Climate Ready Interventions Contributing to Enabling Environment Successes......................................................................................................................................................................8 Figure 4. KII Discussions of ISACC Interventions Contributing to Enabling Environment Successes........12 Figure 5. KII Discussions of Climate Ready Capacity building Successes...........................................................14 Figure 6. Numbers of People Trained in the Skills and Expertise Necessary to Implement Climate change Adaptation through Climate Ready Activities...................................................................................................15 Figure 7. Numbers of Men and Women Trained through Climate Ready Interventions..............................16 Figure 8. Numbers of KII Discussions of ISACC Capacity building Successes.................................................17 Figure 9. Numbers of People Trained in Global Climate Change Adaptation FROM ISACC Activities....18 Figure 10. Numbers of Men and Women Trained Through ISACC ACtivities...............................................19 Figure 11. KII Discussions of Climate Ready Country-Owned Implementation Successes...........................22 Figure 12. KII Discussions of Climate Ready Interventions Contributing to Country-Owned Implementation Successes......................................................................................................................................23 Figure 13: KII Discussions of ISACC Country-Owned Implementation Successes.........................................24 Figure 14. KII Discussions of ISACC Interventions Contributing to Country Owned Implementation Successes....................................................................................................................................................................25 Figure 15. KII Discussions of Climate Ready Private Sector and Institutional Engagement Successes.......27 Figure 16. KII Discussions of Climate Ready Private Sector and Institutional Engagement Successes.......29 USAID.GOV DRAFT EVALUATION REPORT | iv ACRONYMS AF Adaptation Fund COP Conference of the Parties COVID-19 Novel Coronavirus of 2019 CSO Civil Society Organization EQ Evaluation Question ESS Environmental and Social Safeguards FCR Findings, Conclusions, and Recommendations FRDP Framework for Resilient Development in the Pacific FSM Federated States of Micronesia GCF Green Climate Fund GEF Global Environment Facility GESI Gender Equality and Social Inclusion GIZ German Agency for International Cooperation IP Implementer Partner IR Intermediate Result ISACC Institutional Strengthening in Pacific Island Countries to Adapt to Climate Change KII Key Informant Interview MSMEs Micro, Small, and Medium Enterprises M&E Monitoring and Evaluation NGO Non-governmental Organization PCCFAF Pacific Climate Change Finance Assessment Framework PIC Pacific Island Country PIFS Pacific Islands Forum Secretariat PNG Papua New Guinea SOP Standard Operating Procedure SPC Secretariat of the Pacific Community SPREP Secretariat of the Pacific Regional Environment Programme TA Technical Assistance TAFE Technical and Further Education UN United Nations UNFCCC United Nations Framework Convention on Climate Change U.S. United States USAID United States Agency for International Development USP The University of the South Pacific TAFE Pacific Technical Further Education College USAID.GOV DRAFT EVALUATION REPORT | v EXECUTIVE SUMMARY BACKGROUND AND METHODS This document outlines the results of a joint performance evaluation of the Climate Ready Activity and the climate finance component of the Institutional Strengthening in Pacific Island Countries to Adapt to Climate Change (ISACC) Activity. The two United States Agency for International Development in the Pacific Islands (USAID/Pacific Islands) Activities worked with target Pacific Island Countries (PICs) with the goal of helping them effectively access and manage climate funds to achieve self-reliance to adapt to climate change over the long term. Both ISACC and Climate Ready supported efforts to develop the capacities of PICs to plan, deliver, monitor, and report on their climate change priorities and initiatives. The evaluation’s purpose was to inform future programming on climate finance and adaptation and, more specifically, to support USAID’s Monitoring & Evaluation (M&E) Plan and Learning Agenda, which clarifies the Missions’ commitment on using empirical evidence to make programmatic and management decisions. The primary audience for the evaluation was USAID/Pacific Islands, with Climate Ready and ISACC implementing partners (IPs) consulted as key secondary audiences. The scope of the evaluation was at the national level in four focus countries selected by USAID/Pacific Islands: Fiji, the Federated States of Micronesia (FSM,) Kiribati, and Samoa. The evaluation employed a mixed-methods approach, with primary data collected via key informant interviews (KIIs) with national-government officials, nationally affiliated organizations, civil-society and nongovernmental organizations (CSOs and NGOs), regional organizations, IPs, USAID, other donors, and intergovernmental institutions. The evaluation also examined monitoring data and included a desk review of relevant reports and studies. The evaluation team used various analysis techniques to develop evaluation Findings, Conclusions, and Recommendations (FCR). This included creation of a codebook, coding of KII transcripts using a Microsoft Excel–based tally sheet, and triangulation of KII data, desk review information, and monitoring data. After coding was completed, the evaluation team held two virtual workshops to validate results and codevelop recommendations. As with all evaluations, this report must be read with limitations in mind, specifically: 1) difficulty obtaining significant sample sizes for certain respondent groups, which limited disaggregation; 2) restrictions on data collection due to the novel coronavirus of 2019 (COVID-19); and 3) a narrow scope for primary-data collection. The evaluation team aimed to mitigate these by working closely with USAID/Pacific Islands to recruit KII participants, training KII facilitators on remote platforms, seeking a diverse sample, and triangulating findings. FINDINGS AND CONCLUSIONS EQ 1: How did the Activities impact the enabling environment required to access climate financing (including governmental policies, organizational structures, standard operating procedures [SOPs], plans, frameworks, legislation)? a. What were the most significant changes to the enabling environment, and how did these come about? b. To what extent did changes to the enabling environment support GESI? The Activities positively influenced the enabling environment by helping countries (and regional partners) develop policies, plans, frameworks, etc., on climate finance. This support included direct work on policies, plans, and frameworks, such as ISACC contributing to a Climate Finance Action Plan for Samoa. The support also encompassed organizational-strengthening efforts, including Climate Ready’s collaboration with the Kiribati Climate Finance Unit to develop and implement a communication plan and SOP. A major key to success for enabling environment work was the use of local experts and locally led strategies. For example, Climate Ready used local consultants, and regional organizations implemented ISACC. However, it is too soon to determine long-term effects on the enabling environment, as many policies, plans, and USAID.GOV DRAFT EVALUATION REPORT | vi frameworks are just being put into place and need additional time to see the results of implementation. Finally, support for gender, equality, and social inclusion (GESI) was uneven between Activities. It was effective when designed into programming and resourced—as was the case with Climate Ready, but not with ISACC. Specific findings for each Activity are outlined below. Climate Ready. KIIs and the desk review both indicated that Climate Ready improved organizational structures and processes for accessing climate finance. Specifically, Climate Ready monitoring data documented the establishment of 16 laws, policies, plans, etc., from 2018 to 2021, such as assisting the Kiribati Climate Finance Unit with developing the Strategic Framework and Country Program for the Green Climate Fund (GCF). Major contributing interventions included consultant technical assistance (TA) and trainings (on topics ranging from business continuity planning to project management) that improved participants’ ability to implement policies and plans. Climate Ready faced a few different (mainly country￾specific) challenges in its work, including staff mobility, geography, and government uptake. Finally, the evaluation found evidence of GESI integration into policies, plans, and frameworks. For example, Climate Ready supported the Kiribati Finance Division to include appropriate GESI components on the Strategic Framework and Country Program for the GCF. ISACC. The revision and introduction of policies, plans, and frameworks was highlighted as a key success of ISACC across data sources. This was the most frequently discussed enabling environment change in KIIs, while ISACC monitoring data documented implementation support for 26 laws, policies, plans, etc. between 2017 and 2020. In addition, ISACC influenced the enabling environment through its support for regional collaboration on climate finance, providing a platform via meetings and events for countries to share best practices. KII respondents also noted the connection between achievement of regional successes and implementation of ISACC by organizations from the Pacific; these organizations had existing relationships with government stakeholders, which gave them credibility when organizing or participating in convenings. Other key interventions included trainings and research, such as climate finance assessments that informed policies, plans, and frameworks. Major challenges faced by ISACC centered on budget cuts, which necessitated scaling back activities and hampered government relationships. Finally, both KIIs and desk review documents noted that the integration of GESI into ISACC’s enabling environment work was relatively limited because the Activity was never designed to focus on GESI. EQ 2: How did the Activities contribute to building the capacity of government to plan, design, implement (manage), monitor, and evaluate climate change adaptation projects that will be funded through international climate finance instruments? c. What were the most significant changes to government capacity, and how did these changes come about? d. To what extent did capacity building efforts support GESI (beyond including women in capacity building activities)? The Activities contributed to increased government capacity, particularly in project management and M&E. This was mainly accomplished via trainings and workshops, with a large volume (over 2,700 total) of government officials and staff trained. The majority of training and workshop topics were highly relevant, particularly M&E—the most significant area to cover due to both the growing focus on M&E globally and the historical lack of M&E expertise in the Pacific region. The approach adopted by both Activities of partnering with The University of the South Pacific (USP)/Pacific Technical and Further Education (TAFE) to offer formal courses with a recognized certificate added value and increased participation. However, this partnership was not as well regarded by some participants, particularly in the North Pacific. The fact that different participants valued different formats (e.g., trainings with USP/Pacific TAFE, short courses implemented by in-house experts) reinforced the importance of Climate Ready and ISACC offering diverse capacity building options. At the same time, most countries have not yet implemented projects with GCF or similar funding, and thus it is too soon to see the effects of capacity building efforts on programming. While respondents USAID.GOV DRAFT EVALUATION REPORT | vii reported applying knowledge and skills gained through Climate Ready and ISACC interventions to their work, more time is needed to determine the extent to which implementation, management, and reporting for projects has improved. Finally, the extent to which the two Activities’ capacity building efforts supported GESI reflected different levels of planning and resourcing—Climate Ready integrated GESI topics, while ISACC did not because the Activity was never designed accordingly. Climate Ready. Desk review documents and KIIs described capacity building successes related to accreditation, proposal development, project management, and adaptation and disaster response. Accreditation and proposal-development skills were built mainly via consultant TA offered as part of Climate Ready’s work on country-owned implementation (see EQ 3, below). The Activity also offered trainings and workshops on climate change-specific topics, such as resilience, as well as general skills, such as project management. These interventions built capacity in government officials and staff to improve project management and monitoring. For example, a respondent from a nationally affiliated organization stated that the training “upskilled my knowledge around project management so it gave me an opportunity to discuss more detailed financial management . . . what goes into a sector plan, how do you fund it and activities . . . . We even still use materials from the training such as work plans and Gantt charts in the current work.” In addition, the Activity’s GESI work went beyond simply including women in trainings: It integrated a GESI module into a project management course. In terms of challenges, several KIIs mentioned turnover, with government staff moving to different positions with higher salaries after gaining new skills through the Activity. ISACC. KII respondents highlighted how ISACC helped build government capacity in M&E for climate change, which is significant due to both the growing focus on M&E globally and the historical lack of M&E expertise in the Pacific region. Respondents also discussed improvements in knowledge management and project management and noted how these skills were useful for climate change projects and other government work. Examples included a government-official respondent applying learnings from project management trainings to their work coordinating across multiple divisions on activity implementation, M&E, and financing and multiple KII respondents describing their use of M&E training information for climate funding proposals. ISACC’s main capacity building interventions were trainings and workshops, which engaged 880 people (including 385 women and 495 men). These were well-documented in the desk review, with reports describing short courses, workshops, formal courses at universities, etc. In addition to challenges with funding that limited the scale of capacity building activities, KII respondents noted issues with the diversity of training attendees (often the same male government representatives) and topics (with specific requests for GESI). While both men and women were included in ISACC trainings, the evaluation found less evidence of integration of GESI topics into capacity building work. EQ 3: To what extent did new USAID climate funding support countries to implement country-owned climate change actions? It is too soon to tell the extent to which the Activities supported implementation of country-owned climate change actions, as most PICs are still in the process of obtaining funding for and implementing projects. However, the Activities contributed to improving the conditions for country-owned implementation via their work on accreditation to international funds and development of specific project proposals. While Climate Ready supported direct proposal writing and engagement in accreditation processes, ISACC focused more on equipping government staff to take on these tasks. Nevertheless, structural challenges—namely, that accreditation and funding processes are slow and requirements is difficult for PICs to meet—often undercut these outcomes. Finally, Climate Ready’s focus on GESI (specifically, GESI integration into proposals) continued to be useful for stakeholders. Climate Ready. KII respondents noted how Climate Ready contributed to improving the conditions for country-owned implementation by supporting development of project proposals and helping organizations with accreditation policies and processes. Consultant TA was the main intervention used to promote country-owned implementation, with consultants providing technical expertise on areas ranging from USAID.GOV DRAFT EVALUATION REPORT | viii biodiversity to GESI and project preparation. KII respondents described TA for project proposal development as useful because the consultants were knowledgeable about the local context. However, structural challenges—namely, that funding entities’ (e.g., GCF, Adaptation Fund [AF]) processes are slow—sometimes undercut the Activity’s work, which has resulted in Climate Ready not achieving its funding and accreditation targets at the time of the evaluation. Finally, Climate Ready’s focus on GESI (specifically, GESI integration into proposals) remained useful for stakeholders. Examples provided by IP and CSO KII respondents included the United Nations Environment Programme adjusting the application form for a small-grant program to better highlight the GESI component and communities changing their approaches to project design to improve gender integration. ISACC. ISACC offered trainings and workshops that gave government the skills necessary to engage in proposal development and accreditation. In addition, KII respondents described how ISACC brought a “spotlight” to climate finance, shifting perceptions around who is responsible for activities. For example, finance and planning agencies became more involved in climate change, which was previously the domain of environment offices. Another key intervention was ISACC’s support for embedded national￾government positions, with the example of FSM cited in KIIs and the desk review. In FSM, the national coordinator helped organize ISACC implementation, supported progression of national climate change priorities, and worked on multisectoral coordination for climate finance. At the same time, budget cuts caused a reduction in the number of embedded staff, with KIIs noting ISACC interventions losing traction in some places without embedded positions. Finally, respondents highlighted that it will likely take significant time for some PIC countries to meet donor requirements; in the meantime, these countries are experiencing the negative effects of climate change and running out of time to counteract them. EQ 4: What were key successes and challenges faced by these Activities in engaging the private sector and climate financial institutions? Overall, the Activities did not significantly engage with private sector and climate finance stakeholders, which makes it difficult to determine successes and challenges. However, this is not necessarily due to issues with Activity implementation, but rather that Climate Ready and ISACC were never designed and resourced to prioritize this area of work (Climate Ready initially planned for this but changed course due to budget cuts). Nevertheless, both Activities undertook a few relevant initiatives, such as the Climate Ready business continuity training and the ISACC media training. Moving forward, engagement of the private sector in particular has been identified as a key gap and area to prioritize for future growth. Climate Ready. Desk review and KII data indicated little engagement with the private sector and climate financial institutions, with the exception of some capacity building. Specifically, Climate Ready delivered a training on business continuity planning, engaging a total of 449 people in Fiji (of which 300 were women) and 49 in Samoa (of which 30 were women). One lesson learned centered on the importance of incentivizing private sector participation; for the business continuity training, Climate Ready offered concrete tools and templates that businesses could apply to their work. The Activity’s main challenges in private sector and institutional engagement related to lack of budget and dispersed locations of various stakeholders (e.g., chambers of commerce are state based and thus dispersed in many islands, requiring more resources to engage them properly). ISACC. ISACC organized a few interventions that included the private sector and climate finance institutions. One KII respondent noted, for example, a 2019 media training designed to increase familiarity among journalists with PIC-specific climate change issues. Additionally, ISACC supported climate finance working groups that engaged the government, CSOs, and the private sector. Desk review documents discussed how these groups brought together people who were working on the same issues but without knowledge of each other’s efforts due to lack of national-level coordination strategies. Nevertheless, engagement of the private sector and climate finance institutions was not a core focus of ISACC programming. ISACC’s work with these actors did not extend beyond inviting them to specific meetings, events, and trainings, ultimately limiting the potential for significant outcomes. USAID.GOV DRAFT EVALUATION REPORT | ix RECOMMENDATIONS To build on the successes of Climate Ready and ISACC, mitigate challenges encountered, and make progress on climate action in the PICs, the evaluation makes the following recommendations: Enabling environment 1. Support implementation of policies, plans, and frameworks developed through Climate Ready and ISACC. USAID and IPs should design new Activities to build on existing policies, plans, and frameworks and monitor their effectiveness. Specific actions include: development of government or regional organization dissemination plans to prevent a policy or plan from becoming siloed; and trainings on the use of policies, plans, and frameworks. Capacity building 2. Expand capacity building opportunities and partnerships focused on project management and M&E. IP training organizers should diversify participants, expand partnerships with universities, and engage climate finance institutions so that trainings respond to their processes and standards. 3. Strengthen local ownership of capacity building. USAID should prioritize funding Activities that build local skills and move away from the “fly in, fly out” international consultant model. Country-owned actions 4. Mobilize support for project implementation. As PICs start receiving funding, USAID and IPs should increase efforts to strengthen in-country capacities for absorption and utilization of funds. Specific actions include: trainings on project management and M&E; and consultant TA on topics such as GESI and financial management. Private sector and institutional engagement 5. Design and resource activities that engage the private sector. USAID should begin with a landscape analysis exercise to identify critical private sector actors and how they could contribute to climate projects. Then, the design of new Activities should include a specific objective on private sector engagement, with indicators developed to track progress in this work. Implementation 6. Prioritize local, contextually tailored implementation of interventions. USAID should continue and expand Climate Ready and ISACC’s approach of centering local experts, national and regional organizations, and national government in project design and implementation. 7. Integrate embedded staff or focal points in future Activities. Building on the success of embedded staff, USAID and IPs should plan and budget for fully funded positions or focal points. GESI 8. Ensure GESI is integrated into future programs at the design stage and properly resourced. USAID, IPs, and government should recruit specialized staff to implement a range of GESI initiatives, including developing action plans, engaging communities, training, etc. USAID.GOV DRAFT EVALUATION REPORT | 1 INTRODUCTION OVERVIEW The United States Agency for International Development in the Pacific Islands (USAID/Pacific Islands) commissioned a joint performance evaluation of the Climate Ready Activity and the climate finance component of the Institutional Strengthening in Pacific Island Countries to Adapt to Climate Change (ISACC) Activity. The two USAID/Pacific Islands Activities, ISACC and Climate Ready, worked with target Pacific Island Countries (PICs) with the goal of helping them effectively access and manage climate funds to achieve self-reliance to adapt to climate change over the long term. Both ISACC and Climate Ready supported efforts to develop the capacities of PICs to plan, deliver, monitor, and report on their climate change priorities and initiatives. Four specific PICs were selected by USAID/Pacific Islands as the focus of the evaluation: Fiji, the Federated States of Micronesia (FSM), Kiribati, and Samoa. This evaluation aimed to understand how and whether the achievements of Climate Ready and ISACC have contributed to meeting USAID/Pacific Islands’ long-term goal of helping the PICs improve their capacity to prevent and mitigate the adverse effects of climate change. The evaluation’s purpose was to inform future programming on climate finance and adaptation and, more specifically, to support USAID’s Monitoring & Evaluation (M&E) Plan and Learning Agenda, which outlines the commitment of USAID/Pacific Islands to use empirical evidence to make effective programmatic and management decisions (see further description in Annex A: Statement of Work). The primary audience for the evaluation was USAID/Pacific Islands. Secondary audiences included Climate Ready and ISACC implementing partners (IPs), other USAID and PIC decision-makers involved in programming on climate finance and adaptation, and national and regional stakeholders. This evaluation adhered to the principles in USAID’s Evaluation Policy, including those of transparency and accountability of USAID’s program effectiveness to stakeholders and learning to improve development outcomes. BACKGROUND Climate change is a critical concern globally, and the PICs have been identified as being among those countries most vulnerable to its effects. The geographic remoteness and size of many PICs, coupled with a reliance on climate-sensitive resources and other environmental and socioeconomic challenges, underscore their susceptibility to climate change.1 The 21st session of the Conference of the Parties (COP21) of the United Nations Framework Convention on Climate Change (UNFCCC) resolved that the provision of finance, technology, and capacity building support to at-risk countries needed to be scaled significantly, and parties to the agreement arranged to provide substantial financing through international donors to fund interventions on climate change. To address these challenges, support for accessing and managing project funding–from design to implementation and M&E–is critical. PICs face numerous obstacles in accessing funding for adaptation and mitigation efforts, including technical limitations, policy challenges, and staff capacity. PIC leaders have repeatedly emphasized the difficulty of accessing climate finance, calling for the simplification of the bureaucracy surrounding funds, use of a more programmatic approach, and supporting countries’ management capacity. Morioka et al. note that there is a need for PICs to “overcome policy, governance, financial, human resources and technological barriers in information knowledge management to be successful in enabling greater adaptation and resilience outcomes.”2 Both Climate Ready and ISACC provided technical assistance (TA) to national governments to update climate-related laws, policies, and plans; the Activities also aimed to boost project management capacity 1 Kate Morioka et al. (2020), “Applying Information for National Adaptation Planning and Decision Making: Present and Future Practices in the Pacific Islands,” Regional Environmental Change, vol. 20, no. 4 (20 November 2020). 2 Ibid. USAID.GOV DRAFT EVALUATION REPORT | 2 by designing and delivering trainings. While they took different approaches, the underlying common strategies were guided by a focus on effective and meaningful participation of a wide range of stakeholders to support PICs in accessing and managing climate funding. Further detail on the Activities is provided below. CLIMATE READY Climate Ready, implemented by DT Global at a contract value of $23.9 million, commenced in November 2016 and continued through November 2021 with the objective to work with target PICs to achieve their climate change-adaptation goals and become more climate resilient in ways that are country driven, coordinated, inclusive, and equitable. The Activity’s ultimate outcome is to make progress toward the implementation of an increased number of adaptation interventions that are aligned with PICs’ National Adaptation Plans so that countries become more resilient to climate change and disaster risks.F 3 Headquartered in Suva, Fiji, with regional offices in Pohnpei, FSM, Port Moresby, Papua New Guinea (PNG), and Suva, Fiji (which is co-located with the Project Field Office), Climate Ready supports PIC governments in developing climate change-adaptation policies and creating an enhanced enabling environment for access to international finance. The Activity also provides capacity building, training, and consultant TA, seeking to improve the management and implementation of climate adaptation projects to better protect the lives and livelihoods of PIC citizens (Figure 1).4 Climate Ready aims to increase the capacity of PICs to adapt to the negative effects of climate change through the following: • Support to governments to strengthen and mainstream climate change-adaptation policies and laws and address gender and social inclusion • Support to improve the ability of the PICs to access international, multilateral climate change funds through strengthening national and regional institutions (including accreditation to sources of finance) and preparing bankable climate change-investment proposals • Building capacity and systems in both government and the private sector to coordinate, manage, monitor, and report on adaptation projects FIGURE 1. TYPES AND VOLUME OF CLIMATE READY INTERVENTIONS IN EVALUATION FOCUS COUNTRIES Due to the political transition resulting from the 2016 United States (U.S.) presidential election, there was a corresponding shift in priorities and budget reallocation among the various Climate Ready objectives. 3 DT Global (2020), USAID Climate Ready Annual Report FY20 October 2019–September 2020 (October 2020), https://pdf.usaid.gov/pdf_docs/PA00Z4HG.pdf. 4 DT Global (2021), USAID Climate Ready Quarterly Progress Report: FY21 Q3—April to June 2021 (July 2021), https://pdf.usaid.gov/pdf_docs/PA00Z534.pdf. USAID.GOV DRAFT EVALUATION REPORT | 3 Many of the initial activities on policy and mainstreaming to improve the enabling environment were defunded, with greater emphasis placed on proposal preparation and capacity building.2F 5 ISACC A regional climate change-finance project, ISACC was implemented from September 2015 to September 2020 under a cooperative agreement with the Secretariat of the Pacific Community (SPC) in partnership with the Pacific Islands Forum Secretariat (PIFS) and the Secretariat of the Pacific Regional Environment Programme (SPREP). ISACC interventions focused on development of climate change-finance policies and frameworks, capacity building, M&E, and knowledge management training, regional meetings, workshops, and forums, supporting embedded staff,6 and cultivating national and regional partnerships (Figure 2). ISACC worked in eight of USAID’s focus countries in the Pacific: FSM, Fiji, Kiribati, Palau, Samoa, Solomon Islands, Tuvalu, and Vanuatu.3F 7 FIGURE 2. TYPES AND VOLUME OF ISACC INTERVENTIONS IN EVALUATION FOCUS COUNTRIES ISACC’s goal was to strengthen the national institutional capacity of PICs to plan, coordinate, and respond to the adverse effects of climate change. To achieve this, the Activity focused on continuing multisector, whole-of-island approaches that have proved successful in other regional projects and that continue to be supported by various partners via pooling of resources and expertise. Thus, ISACC was designed to implement activities and provide technical support under three key result areas: 1. Integrated institutional frameworks and national capacity strengthened to support multisectoral approaches to climate change and disaster risks 2. Access to new climate change finance enhanced through improved capacity, systems, and tools 3. Regional cooperation and coordination strengthened through augmented national capacity, delivered through shared learning, to support national responses to climate and disaster risks 5 AECOM International Development (2018), USAID Climate Ready: Annual Report FY18: October 2017–September 2018 (October 2018), https://pdf.usaid.gov/pdf_docs/PA00Z4HD.pdf. 6 ISACC funded four national coordinator positions embedded within government ministries in Palau, FSM, Tuvalu, and Solomon Islands. While the Activity initially intended to fund embedded staff in all countries of operation, this was limited due to budget cuts. National coordinators were intended to support ISACC collaboration with governments, increase internal capacity (as they took part in capacity building interventions), and further sustainability. Two of the four embedded staff positions will continue to be supported by either PIC governments or other donors after the completion of ISACC. For further information, see Lisa Buggy and Amelia Caucau (2020), “Interim Final Report for ISACC Climate Change Financing Component,” The Pacific Community (SPC), Charles Darwin University (2019): evaluation of SPC’s capacity building. 7 The Pacific Community (SPC), “Institutional Strengthening in Pacific Island Countries to Adapt to Climate Change (ISACC) Project Brief,” n.d. USAID.GOV DRAFT EVALUATION REPORT | 4 The original budget allocation for ISACC was $5 million. Like Climate Ready, the election of a new U.S. government in 2016 resulted in shifts in funding priorities away from climate change, leading to budget cuts. As of September 2020, $3.375 million had been distributed for ISACC implementation. Areas that were scaled back from the original Activity design included: support for in-country coordinators for Fiji, Kiribati, Samoa, and Vanuatu; peer-to-peer exchange activities; and a midterm review planned for 2018.4F 8 METHODOLOGY OVERVIEW APPROACH Mixed methods. The evaluation adopted a utilization-focused, mixed-methods design. The evaluation collected qualitative primary data via key-informant interviews (KIIs), analyzed secondary quantitative data (primarily Climate Ready and ISACC monitoring data), and included a desk review of program documents and relevant independent research on climate finance in PICs. The scope of the evaluation was at the national level in four focus countries selected by USAID/Pacific Islands: Fiji, FSM, Kiribati, and Samoa. USAID selected these countries to reflect the range of experiences of ISACC and Climate Ready implementation. See Annex B: Methodology for a full description of the evaluation methodology, Annex C: Timeline for the evaluation timeline, and Annex D: Conflict-of-Interest Disclosures for the evaluation team’s disclosures of any conflicts of interest. EVALUATION QUESTIONS The evaluation focused on four specific questions in the development of findings, conclusions, and recommendations. These evaluation questions (EQs) and corresponding sub-questions are outlined in Table 1. Evaluation Questions. TABLE 1. EVALUATION QUESTIONS Improve Enabling Environment 1. How did the Activities impact the enabling environment required to access climate financing (including governmental policies, organizational structures, [SOPs], plans, frameworks, legislation)? a. What were the most significant changes to the enabling environment, and how did these changes come about? b. To what extent did changes to the enabling environment support gender equality and social inclusion (GESI)? Build Capacity 2. How did the Activities contribute to building the capacity of government to plan, design, implement (manage), monitor, and evaluate climate change adaptation projects that will be funded through international climate finance instruments? c. What were the most significant changes to government capacity, and how did these changes come about? d. To what extent did capacity building efforts support GESI (beyond including women in capacity building activities)? 8 Buggy and Caucau (2020), “Interim Final Report for ISACC Climate Change Financing Component.” USAID.GOV DRAFT EVALUATION REPORT | 5 Support Country-Led Implementation 3. To what extent did new USAID climate funding support countries to implement country-owned climate change actions? Engage the Private Sector 4. What were key successes and challenges faced by these Activities in engaging the private sector and climate financial institutions? DATA COLLECTION Fieldwork and sampling. The evaluation team collected data from October 2021 to January 2022, with respondents in Fiji, Samoa, FSM, Kiribati, and other regional and international locations (e.g., Australia, the United States). To construct the data-collection sample, the evaluation team developed a list of potential respondents from the desk review and in consultation with USAID/Pacific Islands and IPs. From this list, the team selected a purposive sample. Key stakeholder groups consulted included the following: • IPs (SPC, PIFS, and SPREP for ISACC and DT Global for Climate Ready) • Government officials (e.g., Ministry of Finance, Ministry of the Economy) and officials of nationally affiliated organizations (e.g., Fiji Development Bank) in the four focus countries • Civil society organizations (CSOs), nongovernmental organizations (NGOs), and regional or local groups, including the private sector (e.g., Micronesia Conservation Trust, Fiji Business Disaster Resilience Council) • USAID and other donors (e.g., German Agency for International Cooperation [GIZ]) • Intergovernmental institutions (e.g., Green Climate Fund [GCF]) Due to restrictions and safety protocols relating to the global pandemic of the novel coronavirus of 2019 (COVID-19), the team adapted primary data collection to require as little travel as possible. All interviews by internationally based team members were held via remote platforms (e.g., Microsoft Teams, Zoom). Local data-collection leads based in Fiji, Kiribati, and Samoa collected data either in person or remotely in the four focus countries, depending on individual country restrictions and the preferences of the interviewers and respondents. In cases of in-person data collection, the team followed safety and security processes that were reviewed and approved by Social Impact’s Institutional Review Board. DESK REVIEW The evaluation team conducted a comprehensive desk review of key documents that provided background knowledge on the processes and decisions that went into designing Climate Ready and ISACC, as well as information on implementation progress and results. The team also reviewed external documents that gave further context on climate adaptation and resilience within the four focus countries for the evaluation. Annex E: Documents Reviewed provides a bibliography. SECONDARY QUANTITATIVE DATA The evaluation team examined monitoring information from the databases of Climate Ready and ISACC to establish the level of support provided through the Activities as well as the progress made toward key government-support and capacity development objectives. The team used its analysis of indicator trends for Climate Ready and ISACC to assess information gaps and highlight issues to cover during data collection. The team disaggregated the data by geography, gender, and intervention type, where possible. USAID.GOV DRAFT EVALUATION REPORT | 6 KEY-INFORMANT INTERVIEWS The evaluation team spoke with key informants, using semi-structured interview protocols tailored by respondent type (see Annex F: Respondents Consulted for evaluation respondents and Annex G: Data Collection Instruments and Informed Consent for interview protocols). In total, the team conducted 46 KIIs (40 individual and 6 group interviews), for a total of 52 people consulted for the evaluation. Table 2 details the numbers of key informants interviewed by respondent type, gender, and location.9 TABLE 2. KII RESPONDENTS: NUMBERS BY CATEGORY AND DEMOGRAPHICS Respondent Type Male Female Fiji Samoa Kiribati FSM Other Total National Government 16 Government officials 3 7 3 3 2 2 0 10 Nationally affiliated organizations 1 5 1 3 0 2 0 6 Civil Society 12 Regional organizations 3 3 3 2 0 0 1 6 Other local organizations/NGOs 4 2 2 1 0 3 0 6 IPs 18 Climate Ready and ISACC 9 9 11 0 0 5 2 18 USAID/Other Donors 3 USAID 0 2 1 0 0 0 1 2 Other donors 0 1 1 0 0 0 0 1 Intergovernmental 3 Climate financial institutions 0 1 0 0 0 0 1 1 Other intergovernmental institutions 1 1 0 0 0 1 1 2 Grand Total 21 31 22 9 2 13 6 52 ANALYSIS AND LIMITATIONS Analysis approach. The evaluation team employed various data analysis techniques to support the development of evaluation Findings, Conclusions, and Recommendations (FCR). Throughout data collection, members of the internationally based team met regularly with local data collection leads to review notes and discuss emerging findings. The team then documented themes under each of the EQs in an FCR matrix to ensure that all questions were being adequately addressed. The team used the FCR matrix to develop a structured codebook for qualitative data coding and analysis. 9 While the evaluation team aimed to reach similar numbers of KII respondents across countries, this was not always feasible due to the location of participants or difficulties contacting with certain groups (see further discussion under Annex B: Methodology, Limitations). For example, more USAID/Pacific Islands and IP staff were located in Fiji than in other countries, increasing the sample numbers in that country; in addition, fewer respondents from Kiribati replied to evaluation-team outreach, lowering these numbers. For some EQs and respondent groups, the smaller sample numbers limited the evaluation team’s ability to provide meaningful disaggregation of KII data. USAID.GOV DRAFT EVALUATION REPORT | 7 Following data collection, KII recordings were transcribed for coding, using a Microsoft Excel–based tally sheet. The team then used the codebook to thematically analyze KII responses. To capture any divergences, coders disaggregated qualitative data by respondent type, location, and gender. The evaluation team then took the interview data, desk review information, and quantitative monitoring data to employ data triangulation. After the coding process was completed, the evaluation team held two virtual workshops with USAID/Pacific Islands, IPs, and other select stakeholders to validate evaluation results and co-develop recommendations. The themes generated from this analysis informed the key findings highlighted in this report. Limitations. As with all evaluations, this report must be read with limitations in mind: 1) difficulty obtaining significant sample sizes for certain respondent groups, which limited disaggregation; 2) COVID-19 restrictions on data collection; and 3) a narrow scope for primary data collection (i.e., four focus countries). The evaluation team worked to mitigate many of these limitations by working closely with relevant USAID staff to recruit informants, training local data collection leads to use virtual platforms (e.g., MS Teams) for remote data collection, seeking a diverse sample, and triangulating findings. FINDINGS AND CONCLUSIONS EQ 1: ENABLING ENVIRONMENT How did the Activities impact the enabling environment required to access climate financing (including governmental policies, organizational structures, [SOPs], plans, frameworks, legislation)? a. What were the most significant changes to the enabling environment, and how did these come about? b. To what extent did changes to the enabling environment support GESI? FINDINGS CLIMATE READY Successes Establishment of enabling structures. Multiple data sources indicated that Climate Ready improved organizational structures and processes for accessing climate finance. Several KII respondents (12 out of 16 total respondents who were asked the question) noted that Climate Ready interventions resulted in revised or new organizational structures, SOPs, and frameworks—the majority of which had been implemented. Climate Ready monitoring data affirmed this claim, documenting the establishment of 16 laws, policies, plans, etc., from 2018-2021 (see Annex H: Laws, Policies, Plans, Procedures, Etc., Introduced or Implemented).F 10 Three KIIs identified new SOPs, policies, and plans as the most significant enabling-environment change. The areas for support were determined through in-country consultations and examples given in the KII and desk review11 included the following: • Mainstreaming climate change and disaster risk reduction into public administration in Samoa • Assisting the Kiribati Climate Finance Unit with developing the Strategic Framework and Country Program for the GCF and with developing and implementing a communication plan and SOP • Mainstreaming climate change into the Samoa tourism sector plan According to the desk review, the outputs from Climate Ready included not only plans and policies themselves, but also capacity building on implementation and templates that could be adapted for 10 DT Global (2021), USAID Climate Ready Quarterly Progress Report: April to June 2021, https://pdf.usaid.gov/pdf_docs/PA00Z534.pdf. 11 DT Global (2021), National Consultation Reports for Samoa, FSM, Kiribati. USAID.GOV DRAFT EVALUATION REPORT | 8 numerous uses.12 According to KII respondents, the support contributed to sustained outcomes by providing tools that could be used for future activities and the capacity to implement the plans and policies that have been developed with Climate Ready support. Key Interventions Consultant TA. Consultant TA was Climate Ready’s main intervention that contributed to enabling environment successes (Figure 3). Several KIIs mentioned that the way this technical support was provided was a success factor—specifically, the close working arrangements between government and the consultant, as well as the use of local experts that provided “presence on the ground” and understanding of the local context. KII respondents also believed Climate Ready’s interventions to be complementary to work being done in the region by other donors. For example, GIZ and the Australian Department of Foreign Affairs and Trade provided support on improving the enabling environment, and Climate Ready successfully collaborated with these institutions to avoid overlap. In addition to direct support on policies, plans, and frameworks, consultants worked on research and analysis projects that the Activity used to set priorities for improving the enabling environment required to access climate financing. Examples from the desk review and KIIs included climate finance assessments in Kiribati and FSM, as well as an organizational review of the Micronesia Conservation Trust. The finance assessments (which used an approach developed by ISACC) looked at a country’s ability to manage climate change resources across seven interrelated pillars, resulting in practical options and recommendations to improve access to and management of climate change funds. Several KII respondents identified the gap analysis that came out of Climate Ready outputs in this area as a successful way to focus future support. Trainings and workshops. Trainings and workshops were the second most frequently mentioned enabling environment intervention in KIIs, although respondents often did not distinguish between general capacity building and interventions specifically tied to new plans, policies, and frameworks. Respondents in two KIIs flagged improved capacity of government officials to implement strategies or policies as being a significant change in the enabling environment. Figure 3: KII Discussions of Climate Ready Interventions Contributing to Enabling Environment Successes 12 AECOM International Development (2018), USAID Climate Ready: Annual Report FY18, https://pdf.usaid.gov/pdf_docs/PA00Z4HD.pdf, and DT Global (2019, 2020), USAID Climate Ready Annual Reports for FY 2019, FY 2020. 1 1 1 3 3 5 7 M&E, research, and publications Meetings and events Support to embed staff Other Direct support from project staff Trainings or workshops Consultant TA KII respondents (n=16) most often noted consultant TA as contributing to enabling environment successes USAID.GOV DRAFT EVALUATION REPORT | 9 Challenges Country-specific enabling-environment contexts. The desk review highlighted that the context in each country led to different challenges for the enabling environment. This meant that support had to be tailored to each country—with the consequence that it was difficult to increase efficiency by replicating activities in several countries. Some of these challenges were addressed specifically by Climate Ready (e.g., support to mainstream climate change across sector plans), while others were tackled more indirectly (e.g., through capacity building interventions that could contribute to institutional strengthening). Examples included the following: • Kiribati: National Consultation Report documents described challenges with staff mobility and a high turnover rate in organizations tasked with implementing adaptation.13 As a result, it was difficult to sustain capacity as key staff moved to other roles. This issue was not mentioned specifically in other National Consultations, but evidence from the KIIs suggests that it extends beyond Kiribati. The need to strengthen existing institutions through better identification of roles and responsibilities was also mentioned during National Consultations that Climate Ready carried out in Kiribati.14 • Samoa: KII respondents and desk review documents highlighted institutional strengthening and the need to mainstream climate change adaptation and disaster risk reduction across sector plans as key issues.F 15 • FSM: In FSM, each state formulates its own policies and legislation, which means that the approach to climate change may differ; thus, support needs are not always consistent.1F 16 Government priorities and workload. Results validation workshop participants highlighted how a change in government following elections can also alter national priorities and present a challenge to implementation. Another government-specific issue identified in the KIIs was that government counterparts have a high volume of work, which meant that they could not always fully engage in Climate Ready interventions. On occasion, this led to delays or to Climate Ready canceling some areas of work, where it appeared ineffective to continue. GESI GESI integration. A large proportion of Climate Ready enabling environment interventions included GESI components. For example, according to KIIs, GESI was one of the pillars of the climate finance assessment mentioned above and carried out for Kiribati and FSM. The climate finance assessment identified gaps with regard to GESI and project planning, development, and implementation of climate finance related initiatives. According to KIIs, this assessment was used by the governments to help plan interventions to address these gaps, including requesting support from Climate Ready. Other examples of Climate Ready interventions included supporting the Kiribati Finance Division to include appropriate GESI components on the Strategic Framework and Country Program for the GCF and mainstreaming gender and climate change into the government planning process in Samoa. KII respondents mentioned that mainstreaming helped ensure that crosscutting issues, such as gender, climate, and the needs of vulnerable groups, were considered across government, not just in the “climate change”–related departments. Broader influence of GESI work. KIIs provided examples of larger effects of Climate Ready’s GESI interventions. In Fiji, one IP interview highlighted that following initial work to develop a GESI policy with the Ministry of Economy, the government asked Climate Ready to support other departments of 13 DT Global (2021), “National Consultation Reports, Kiribati: 2021.” 14 Ibid. 15 DT Global (2021), “National Consultation Reports, Samoa: 2021.” 16 DT Global (2021), “National Consultation Reports, FSM: 2021,” and Global Environment Facility (2019), Second National Communication to the United Nations Framework Convention on Climate Change, https://www.fsmstatistics.fm/wp￾content/uploads/2019/10/2-2nd-National-Communiation-to-the-UNFCCC.pdf. USAID.GOV DRAFT EVALUATION REPORT | 10 government to develop policies for sexual harassment, women in leadership, and other issues. Additional IP KIIs cited a request from the Ministry of Economy for Climate Ready to help build capacity to operationalize the GESI policy they had developed.17 Another IP KII noted that this element of support (operationalization) in Kiribati was being provided by the Asian Development Bank,18 following development of the policy by Climate Ready. Also in Kiribati, consultants working with the Climate Finance Division were reported to be better able to include GESI considerations in project design. Finally, an other-donor KII respondent described how Climate Ready GESI work inspired other aid projects to place dedicated climate change focal points in Ministries of Women and Youth in the region. According to the same KII, these focal points increased government capacity to ensure that climate change and GESI were properly mainstreamed. ISACC Successes Establishment of policies, plans, and frameworks. The revision and introduction of government policies, organizational structures, SOPs, plans, frameworks, and legislation was highlighted as a key success of ISACC across data sources. This was the most frequently discussed enabling environment change in KIIs, cited by 10 of the 17 ISACC respondents interviewed on this topic. Respondents gave examples, including national adaptation plans; the Kiribati Joint Implementation Plan for Climate Change and Disasters, as part of the process of developing a joint national action plan; and a climate finance action plan for Samoa. This was also reflected in ISACC monitoring data, which documented the introduction or implementation of 26 ISACC-supported laws, policies, regulations, or standards on climate change adaptation between 2017 and 2020 (see Annex H: Laws, Policies, Plans, Procedures, Etc., Introduced or Implemented).19 KII respondents largely agreed that the policies, plans, SOPs, and frameworks put into place under ISACC will remain after the end of the program. This was attributed to the fact that these changes in policies and plans were spurred by requirements from the GCF, the Adaptation Fund (AF), and similar international institutions and will therefore remain important as countries apply for climate funding and implement projects. Similarly, a few respondents cited the usefulness of ISACC-sponsored assessments and studies for ongoing GCF and AF proposal development. Some respondents also believed that the experience that SPC, SPREP, and PIFS gained through ISACC implementation will carry over into future projects, as these organizations will continue to operate in the regional climate finance space. Desk review information also supported the finding that ISACC contributed to improved structures, SOPs, plans, and frameworks. For example, documents referenced ISACC’s collaboration with Climate Ready to support the creation of joint national action plans for Kiribati and Samoa that outlined processes for the planning, execution, and reporting of climate change activities.20 ISACC also provided technical input into the drafting and review of Fiji’s National Planned Relocation Guidelines and collaborated with SPREP and Griffith University’s Pacific iCLIM Project to design a national climate change and disaster 17 DT Global (2020), USAID Climate Ready Annual Report FY20, https://pdf.usaid.gov/pdf_docs/PA00Z4HG.pdf. 18 Asian Development Bank (2020), “Enhancing Gender Equality and Social Inclusion Results in South Asia Developing Member Countries” (September 4, 2020), https://www.adb.org/projects/54305-001/main. 19 USAID ISACC (2017–2020), PPR Sheets, FY 2017–FY 2020. 20 Buggy and Caucau (2020), “Interim Final Report for ISACC Climate Change Financing Component”; Government of Samoa Disaster Management Office (2019), Samoa National Action Plan for Disaster Risk Management 2017–2021, https://www.mnre.gov.ws/wp-content/uploads/2017/08/Samoa-national-action-plan-for-DRM-2017-2021-final-web.pdf; Secretariat of the Pacific Regional Environment Programme (2013), “Kiribati to Have Joint National Action Plan for Climate Change and Disaster Risk Management” (March 18, 2013), https://www.sprep.org/news/kiribati-have-joint-national-action-plan￾climate-change-and-disaster-risk-management ; Government of Kiribati (2019), Kiribati Joint Implementation Plan for Climate Change and Disaster Risk Management: 2019–2028, https://www4.unfccc.int/sites/NAPC/Documents/Parties/Kiribati-Joint￾Implementation-Plan-for-Climate-Change-and-Disaster-Risk-Management-2019-2028.pdf. USAID.GOV DRAFT EVALUATION REPORT | 11 information portal for the FSM.14F 21 Finally, the Pacific Climate Change Finance Assessment Framework (PCCFAF), developed by ISACC implementer PIFS, was noted in desk review documents as a “good practice” tool that supports countries to track and report on climate finance funds. ISACC used this framework to conduct climate finance assessments in FSM and Kiribati.22 Regional collaboration. ISACC also influenced the enabling environment through its support for regional collaboration on climate finance. KII respondents described how ISACC provided a platform via meetings and events for countries to share best practices and lessons learned related to climate finance. For example, one IP respondent viewed the regional climate finance meetings as a “very good space for countries to learn about climate finance [generally], but also to hear from countries in terms of their application to regional accreditation entities and what have been the challenges.” The desk review similarly identified ISACC meetings and events that supported regional collaboration, such as the Joint Platform for Disaster Risk Management and Pacific Climate Change Round Table, a 2017 meeting of PIC governments, NGOs and CSOs, development partners, the private sector, and community groups focused on addressing climate change and disaster risks via implementation of the Framework for Resilient Development in the Pacific (FRDP).23 Respondents noted the connection between achievement of regional (and national) successes and implementation of ISACC by organizations from the Pacific, namely SPC, SPREP, and PIFS. These organizations had existing relationships with government stakeholders, which gave them credibility when organizing or participating in convenings, such as aforementioned Joint Platform for Disaster Risk Management and Pacific Climate Change Round Table. It also helped ensure that ISACC interventions were well aligned with existing regional priorities. Key Interventions M&E and research. The M&E and research interventions that ISACC implemented helped countries establish structures, SOPs, frameworks, etc. Mentioned by 10 out of 17 ISACC respondents, these were the interventions mostly commonly cited as contributing to enabling environment changes (Figure 4). Specifically, respondents called out the climate finance assessments and integrated vulnerability assessments as especially influential research undertakings. A government-official KII respondent noted that the vulnerability assessments were particularly useful due to their integration of social dynamics as well as scientific and environmental issues, which was critical for deciding which climate change projects to prioritize. Both KIIs and desk review documents mentioned ISACC support to draft an M&E strategy for the FRDP, a regional framework established by the Pacific Resilience Partnership that provides strategic guidance to various stakeholder groups on how to enhance resilience to climate change and disasters. Trainings and workshops. Trainings and workshops were also highlighted as important enabling environment interventions by 7 out of 17 total ISACC respondents interviewed about the enabling environment. According to KII respondents, ISACC trainings, specifically those focused on M&E, gave participants technical knowledge and skills that were useful when developing and implementing structures, SOPs, plans, and frameworks. An IP respondent stated, “I think the M&E work was really important because . . . a lot of the components of having a good policy or plan in place are then not followed through by being able to actually measure the progress towards implementation of those plans and achievement of climate change goals and resilience goals.” One government-official KII respondent similarly described how capacity building on M&E and project management supported the work of the government’s main climate change coordination body, including it in its development of strategic plans on climate change and 21 Charles Nyuykonge (2020), “End of Project Evaluation for the Institutional Strengthening for Pacific Islands to Adapt to Climate Change (ISACC) Project & Design Support for a New Climate Change Adaptation Programme,” The Pacific Community (SPC) ISACC Project; Department of Environment, Climate Change & Emergency Management, FSM (2021), “FSM National Climate Change Data Portal” (July 9, 2021), https://fsm-data.sprep.org/dataset/fsm-national-climate-change-data-portal; and National Legislative Bodies/National Authorities, Fiji (2018), Planned Relocation Guidelines: A Framework to Undertake Climate Change Related Relocation (December 2018), https://www.refworld.org/docid/5c3c92204.html . 22 Buggy and Caucau (2020), “Interim Final Report for ISACC Climate Change Financing Component.” 23 Vuki Buadromo and Amelia Caucau (2017), “USAID ISACC Quarterly Report (October–December 2016),” USAID. USAID.GOV DRAFT EVALUATION REPORT | 12 disaster risk management. Both IP and government-official KII respondents noted the sustainability of these interventions, as training and workshop participants will carry over the knowledge and skills gained into their ongoing work. This was designated by KII respondents as one of the most significant outcomes of the ISACC program, as M&E expertise still represents a major gap in the Pacific region (see discussion under EQ 2: Capacity building). Other TA. A few KII respondents cited different modalities of TA—the use of consultants to help draft documents (five KIIs), direct assistance from ISACC staff (one KII), and the work of those in embedded government positions (two KIIs; see further discussion under EQ 3: Country-Owned Implementation)— as contributing to the development of policies, plans, SOPs, and frameworks. Having these extra staff, whether in the short or the longer term, was critical for supporting the sometimes under-resourced government offices. ISACC staff and consultants also brought new ideas and expertise in specific areas, such as GESI. FIGURE 4. KII DISCUSSIONS OF ISACC INTERVENTIONS CONTRIBUTING TO ENABLING ENVIRONMENT SUCCESSES Challenges Funding. KIIs and desk review documents consistently mentioned funding as a challenge to outcome achievement. As a result of funding reductions for ISACC, not all interventions could be carried out as planned, ultimately limiting the reach of the Activity (as discussed under EQ 2: Capacity building, for example, not all ISACC countries received the same volume of trainings or embedded government staff support).F 24 While cooperation with PIC governments was generally noted as a strength of ISACC (due to the fact that it was implemented by a regional organization, as mentioned above), KII participants and desk review documents described how budget cuts challenged these relationships, as the Activity previously committed to certain interventions and then had to reverse course.25 At the same time, the U.S. government proposed scaling climate finance activities to additional countries, leading to confusion among PIC governments around the availability of funding. These actions created mistrust among some respondents for U.S. government programming. Relatedly, a few KII respondents discussed challenges with slower national-government processes and having interventions approved in time to spend funds. GESI Limited GESI integration. The evaluation found a few examples of GESI integration in enabling environment research and assessments. For example, the climate finance assessments described above incorporated a dedicated pillar on GESI, which helped identify gaps related to project planning, development, and implementation of climate finance initiatives. Integrated vulnerability assessments also included considerations related to gender and marginalized groups. However, as noted in KIIs and desk 24 Buggy and Caucau (2020), “Interim Final Report for ISACC Climate Change Financing Component.” 25 Ibid. 1 2 4 5 5 6 7 10 Direct staff support Support to embed staff Knowledge management tools Other Consultant TA Meetings and events Trainings and workshops M&E/research KII respondents (n=17) most often noted M&E and research as contributing to enabling￾environment successes USAID.GOV DRAFT EVALUATION REPORT | 13 review documents, the integration of GESI into ISACC’s enabling environment work overall was relatively limited because the Activity was not designed to focus on GESI. CONCLUSIONS The Activities positively influenced the enabling environment by helping countries (and regional partners) develop policies, plans, frameworks, SOPs, etc., on climate finance. This support included direct work on policies, plans, and frameworks, such as ISACC contributing to a Climate Finance Action Plan for Samoa. It also encompassed organizational-strengthening efforts, including Climate Ready’s collaboration with the Kiribati Climate Finance Unit to develop and implement a communication plan and SOP. A major key to success for enabling environment work was the use of local experts and locally led strategies—for example, Climate Ready hiring local consultants or implementation of ISACC by regional organizations. However, it is too soon to determine long-term effect on the enabling environment, as many policies, plans, and frameworks are just being put into place and need additional time to see the results of implementation. Finally, support for GESI was uneven between Activities. It was effective only when designed into programming and sufficiently resourced—as was the case with Climate Ready, but not ISACC. Table 3 summarizes key enabling environment conclusions for Climate Ready and ISACC. TABLE 3. CLIMATE READY AND ISACC KEY CONCLUSIONS: EQ 1 Category Climate Ready ISACC Successes Climate Ready interventions focused on gaps in strategies, and SOPs led to implementation of new structures and processes. ISAAC was effective in supporting policies, plans, SOPs, etc. (many of these directly developed), and regional collaboration. Implementation of ISACC by a regional organization was an important success factor. Key Interventions Consultant TA was effective, particularly the use of local experts (or pairing local experts with international consultants), which helped ensure that support was contextually relevant. Capacity building was important to enabling environment successes. M&E/research gave necessary data, but capacity building is more sustainable. Challenges There was difficulty overall in engaging government, given the high workload for government officials, as well as country￾specific issues related to staff mobility, geography, etc. Budget cuts posed a significant challenge to both implementation and relationships. GESI GESI was well integrated into enabling environment support. GESI integration was limited, as it was not part of the initial Activity design or resourcing. EQ 2: CAPACITY BUILDING How did the Activities contribute to building the capacity of government to plan, design, implement (manage), monitor, and evaluate climate change adaptation projects that will be funded through international climate finance instruments? a. What were the most significant changes to government capacity, and how did these changes come about? b. To what extent did capacity building efforts support GESI (beyond including women in capacity building activities)? USAID.GOV DRAFT EVALUATION REPORT | 14 FINDINGS CLIMATE READY Successes Proposal development and accreditation. The capacity building successes most often mentioned by KII respondents were on accreditation and proposal development (13 out of 16 total Climate Ready respondents who were asked the question) (see Figure 5). This capacity building was delivered as part of the support provided to organizations for these interventions (discussed in more detail under EQ 3: Country-Owned Implementation). Project management capacity. Climate Ready interventions contributed to more effective development and management of adaptation projects. This included not only climate change-specific topics, such as resilience, but also more general skills, such as project management, which is vital for managing climate finance. One USAID KII respondent reported that organizations selected people for training that was relevant to their roles and responsibilities in an organization and that therefore the training should directly help them do their jobs more effectively. Other KIIs highlighted that training built capacity in staff to manage and monitor projects by equipping them with project management and monitoring, reporting, and verification knowledge and tools. One nationally-affiliated organization KII reported that the training “upskilled my knowledge around project management so it gave me an opportunity to discuss more detailed financial management . . . what goes into a sector plan, how do you fund it and activities . . . . We even still use materials from the training such as work plans and Gantt charts in the current work.” FIGURE 5. KII DISCUSSIONS OF CLIMATE READY CAPACITY BUILDING SUCCESSES Key Interventions Trainings and workshops. Climate Ready trained many of people on the skills necessary to manage finances and implement climate change actions. Almost 2,000 people were trained across the region, including more than half in the evaluation’s four target countries (see Figure 6).F 26 Training participants were drawn from a wide range of public and private sector organizations, including water and road authorities in Fiji, the Kiribati Development Bank, several chambers of commerce, NGOs, and national￾government ministries. Nine KIIs (out of 16 total interviewed on capacity building) mentioned that interventions included trainings and workshops. Respondents highlighted that some of the trainings were designed specifically for women-only groups, which encouraged participation in courses that were otherwise male dominated. Others were delivered in remote regions, ensuring participation by staff who would otherwise be excluded by the time and costs involved in traveling to a central location. 26 AECOM International Development (2018), USAID Climate Ready: Annual Report FY18, https://pdf.usaid.gov/pdf_docs/PA00Z4HD.pdf. USAID.GOV DRAFT EVALUATION REPORT | 15 FIGURE 6. NUMBERS OF PEOPLE TRAINED IN THE SKILLS AND EXPERTISE NECESSARY TO IMPLEMENT CLIMATE CHANGE ADAPTATION THROUGH CLIMATE READY ACTIVITIES In addition to general capacity building on project management or proposal development, Climate Ready delivered more specialized training, including a course on management of assets (e.g., public buildings, bridges, energy and water infrastructure), training for conservation officers on enforcement issues, and business continuity planning for the private sector (see discussion under EQ 4: Private sector and Institutional Engagement). Climate Ready also sponsored Fiji government personnel to attend a humanitarian-essentials course for disaster preparedness and risk reduction.F 27 KII respondents mentioned that Climate Ready positioned its trainings for continued outcomes. For example, the Certificate 4 in Project Management in Climate Change and Disaster Risk Resilience Course was developed and delivered with the University of the South Pacific (USP)/Pacific Technical Further Education College (TAFE) in partnership with ISACC. According to one IP KII respondent, this collaboration set up the training to continue after Climate Ready finishes (the USP/TAFE website documents the formal launch of its full Disaster Risk Management program in September 202128). Several respondents also mentioned that the fact that the training resulted in a formal certificate made it more valuable for professional development and therefore attractive to participants. This link to professional development also set up the conditions for continued improvements in enabling conditions as staff with enhanced skills progress in organizations and bring those skills to new areas. Challenges Retention. When government staff are better trained, retention can be a challenge. Several KIIs identified that retention of staff once they had gained new skills and knowledge was an issue, with staff moving on to larger offices or international organizations with higher salaries. During the findings and conclusions workshop it was mentioned that this was not specific to the trainings provided by Climate Ready but to any type of staff training. GESI GESI integration. GESI was an integral part of Climate Ready training, both to ensure that equal numbers of women and men were trained and in the course topics themselves. Around half of the people trained were women, with two women-only training courses in project management held in Fiji (see Figure 7).F 29 One USAID KII respondent reported that when women were underrepresented among the proposed participants, the Climate Ready team went back to the organization to ask for more women to 27 DT Global (2021), USAID Climate Ready Quarterly Progress Report: FY21 Q3, https://pdf.usaid.gov/pdf_docs/PA00Z534.pdf. 28 University of the South Pacific (2021), “USP Commences with Certificate 4 in Disaster Risk Management Programme” (September 27, 2021), https://www.usp.ac.fj/news/usp-commences-with-certificate-4-in-disaster-risk-management-programme/. 29 USAID Climate Ready FY 18 Annual Report. 329 592 622 313 72 2021 (Q1-3) 2020 2019 2018 2017 From 2017 to 2021, Climate Ready supported training for 1,928 people, exceeding its target of 1,193 USAID.GOV DRAFT EVALUATION REPORT | 16 be put forward. GESI was a key component in the project management training course, ensuring that managers considered GESI as an integral part of the project. The elements considered in the course were reported to relate to all stages of a project, from development to implementation. FIGURE 7. NUMBERS OF MEN AND WOMEN TRAINED THROUGH CLIMATE READY INTERVENTIONS ISACC Successes Project management capacity. KII respondents frequently cited improvements in project management capacity (7 mentions out of 20 ISACC respondents who discussed capacity building; see Figure 8). Respondents appreciated the focus on project management, because this topic is not only useful for climate change projects, but also applicable to other areas. For example, one national-government KII respondent described applying learnings to a position that requires strong management and coordination with multiple divisions on activity implementation, M&E, and financing. Knowledge management capacity.30 ISACC contributed to improved knowledge management at both regional and national levels. Out of the 20 ISACC respondents asked to describe capacity building outcomes, 7 highlighted increased capacity around knowledge management. Examples included piloting a climate finance tracking tool to assist countries in determining how much and what type of funding is available (while this was mentioned in KIIs, it was mainly implemented in countries outside of the scope of the evaluation, namely Vanuatu and Solomon Islands) and development of online portals to consolidate national climate change information.31 M&E capacity. KII respondents highlighted how ISACC helped build government capacity in M&E for climate change, an area that many were not exposed to previously. Of the 20 ISACC respondents who answered questions on capacity building, six cited improved ability to conduct M&E as a result of ISACC interventions. Government-official, nationally affiliated, and IP respondents all highlighted this as one of the most significant changes due to both the growing focus on M&E globally and the historical lack of capacity in the Pacific region. An IP respondent said that “monitoring and evaluation is one area that's really lacking in the Pacific at the moment.” According to these respondents, building capacity within PICs will also counteract the siloed manner in which M&E has been conducted in the past, with international 30 Knowledge management refers to activities to improve sharing, use, and management of information within an organization. 31 Department of Environment, Climate Change & Emergency Management, FSM (2021), “FSM National Climate Change Data Portal,” https://fsm-data.sprep.org/dataset/fsm-national-climate-change-data-portal; Ministry of Natural Resources and Environment, Samoa (2022), “National Climate Change Portal: Samoa” (February 15, 2022), https://samoa￾data.sprep.org/dataset/national-climate-change-portal-samoa; Secretariat of the Pacific Regional Environment Programme (2022), “Pacific Climate Change Portal” (February 15, 2022), https://kiribati-data.sprep.org/dataset/pacific-climate-change-portal; and Fiji Climate Change & National Designated Authority Portal (2022), https://fijiclimatechangeportal.gov.fj/. 129 320 316 179 42 184 203 276 150 31 2021 2020 2019 2018 2017 From 2017 to 2021 Climate Ready trained 985 men and 943 women Male Female USAID.GOV DRAFT EVALUATION REPORT | 17 consultants flying into the region, completing a project, and leaving behind a report that may not be used. Developing more local M&E experts will help ensure that M&E is more inclusive and contextually responsive moving forward. FIGURE 8. NUMBERS OF KII DISCUSSIONS OF ISACC CAPACITY BUILDING SUCCESSES Furthermore, several respondents noted that building M&E capacity will ultimately strengthen a country’s ability to access climate funds, with the GCF, AF, and other international institutions putting into place M&E requirements at both the project proposal and the implementation stages. A national government respondent stated: “I believe the M&E training improved our capacity to access climate financing, because it is a requirement for these projects. And without that training and that support, we wouldn't have been able to report our progress and have a better understanding of where we were going with respect to the outputs and intended outcomes.” Four KII respondents mentioned that ISACC M&E trainings increased their ability to work on climate funding proposals (see discussion under EQ 3: Country-Owned Implementation). Sustained outcomes. KII respondents believed that the knowledge and skills gained by capacity building participants will remain useful beyond the period of ISACC implementation. Several KII respondents, including some in national government positions, mentioned continuing to apply M&E and project management learnings to their work and using the tools and resources provided by ISACC. However, this sustainability is largely limited to the individual level, and some respondents questioned whether practices would continue to shift at an institutional level without further intervention. Additionally, a few respondents noted a lack of mainstreaming of certain knowledge management tools, such as climate finance dashboards, into established government processes, which will likely hamper their use moving forward. Key Interventions Trainings and workshops. ISACC monitoring data documented 880 trainings and workshops implemented from 2017 to 2020 (see Figure 9).F 32 A significant number of KII respondents linked these trainings and workshops to increased government capacity in M&E, proposal development, and project management. Sixteen out of 20 ISACC respondents named trainings and workshops as critical to capacity building, more than any other intervention offered by the Activity. These were successful for two main reasons: 1. ISACC professionalized its trainings by offering a short course through USP/Pacific TAFE: the Certificate IV in Project Management in Climate Change and Disaster Risk Resilience Course.33 As discussed by both Climate Ready (see above) and ISACC government-official and other￾organization/NGO respondents, this approach helped make the trainings more formal, particularly in contrast to other one-off professional development workshops. Participants could list training 32 USAID ISACC (2017–2020), PPR Sheets, FY 2017–FY 2020. 33 Vuki Buadromo and Amelia Caucau (2018), “USAID ISACC Quarterly Report (October–December 2017),” USAID. 1 2 4 6 7 7 1 1.5 2 2.5 3 3.5 4 4.5 5 5.5 6 KII respondents (n=20) most often noted project management, knowledge management, and M&E as key ISACC capacity-building successes Proposal development Accreditation Project management Knowledge management M&E Adaptation and disaster response USAID.GOV DRAFT EVALUATION REPORT | 18 certificates on their curricula vitae, providing additional motivation to enroll in and finish the courses. Nevertheless, results validation workshop participants noted that the partnership with USP/Pacific TAFE was not seen as contextually responsive by all, particularly those in the North Pacific who may not traditionally seek out qualifications from institutions perceived as affiliated with the South Pacific. These participants were more likely to engage with other offerings, such as the M&E for Climate Change and Disaster Resilience Training, a five-day short course on design and M&E for climate change programming implemented by SPC and the ISACC team.34 2. In addition, other-donor KII respondents noted the holistic approach adopted in many ISACC trainings, which covered not only climate finance but also the broader climate change policy landscape. This helped raise awareness among governments of the various systems and processes needed to apply for and manage climate change projects (see discussion under EQ 3: Country￾Owned Implementation). As a result, some government offices hired staff or restructured to be better placed to obtain funding. FIGURE 9. NUMBERS OF PEOPLE TRAINED IN GLOBAL CLIMATE CHANGE ADAPTATION FROM ISACC ACTIVITIES Meetings, events, and embedded staff. In addition to trainings and workshops, three national￾government and IP KII respondents cited meetings and events as contributing to capacity building. For example, the side event on information knowledge management during the 2019 Pacific Resilience Meeting in Fiji brought together different parties (government from various countries along with nongovernment actors) to hear a range of ideas and lessons learned that could then be applied to government knowledge management systems. Finally, five respondents (including IPs, nationally affiliated organizations, and USAID) discussed how ISACC support for embedded government staff contributed to capacity building; since these staff were physically present in government offices, they could provide more consistent feedback (see further discussion on embedded staff under EQ 3: Country-Owned Implementation). Challenges Diversification of topics and participants. Some respondents noted that trainings would have been strengthened by introducing more topics and different types of participants. Specifically, KII respondents mentioned the need for more training on GESI (see below). In addition, two respondents described how ISACC trainings and events often included the same participants, primarily male government officials who had established relationships with the Activity. Not only does this prevent others from accessing capacity building opportunities, but it also impedes the exchange of ideas (i.e., different types of participants will bring new voices and information). Results validation workshop participants affirmed this finding, citing the 34 Vuki Buadromo and Amelia Caucau (2019), “USAID ISACC Quarterly Report (October–December 2018),” USAID. 38 179 262 401 2020 2019 2018 2017 From 2017 to 2020, ISACC supported training for 880 people USAID.GOV DRAFT EVALUATION REPORT | 19 need to better plan for and resource strategies to expand the pool of participants; for instance, mid- or junior-level government staff, rather than just senior officials, could be invited. Funding. Respondents also noted the inherent limitations to capacity building imposed by the ISACC budget. Learning and applying new skills takes time and requires sustained (and varied) types of support. While ISACC aimed to provide this—by coupling formal courses with embedded-staff support, for example—this was not always consistent. Budget cuts prevented the Activity from supporting embedded staff in every country, and some trainings and events had to limit the number of participants to keep costs low. For example, only two countries (Fiji and Samoa) under consideration for the evaluation took part in the USP/Pacific TAFE certificate course, while only FSM had an embedded staff position. GESI Gender representation in trainings. ISACC aimed to include both men and women in capacity building activities. Several KII respondents noted representation by women in workshops, meetings, and events, while ISACC monitoring data detailed female participation in trainings (see Figure 10). FIGURE 10. NUMBERS OF MEN AND WOMEN TRAINED THROUGH ISACC ACTIVITIES Limited GESI integration. The evaluation found less evidence of GESI being integrated into capacity building efforts. Only two KII respondents were able to provide examples of GESI work: One described collaboration with the human rights and social development team at SPC to bring a GESI lens to M&E trainings, and the other noted implementation of GESI recommendations following an assessment in Kiribati. CONCLUSIONS The Activities contributed to increased government capacity, particularly in project management and M&E for climate change adaptation projects. This was mainly accomplished via trainings and workshops, with a large volume (over 2,700 total) of government officials and staff trained. The majority of training and workshop topics were highly relevant, particularly M&E—the most significant area to cover due to both the growing focus on M&E globally and the historical lack of capacity in the Pacific region. The approach adopted by both Activities of partnering with USP/Pacific TAFE to offer formal courses with a recognized certificate added value and increased participation. However, this partnership was not as well regarded by some participants, particularly in the North Pacific. The fact that different participants valued different formats (e.g., trainings with USP/Pacific TAFE, short courses implemented by in-house experts) reinforced the importance of Climate Ready and ISACC offering diverse capacity building options. The responses from the KIIs also demonstrated that both integration of capacity building in support and formal training courses can successfully build capacity. At the same time, most countries have not yet implemented projects with GCF or similar funding, and thus it is too soon to see the effects of capacity building efforts on programming. While respondents 18 95 102 170 20 84 160 231 2020 2019 2018 2017 From 2017 to 2020, ISACC trained 385 women and 495 men Female Male USAID.GOV DRAFT EVALUATION REPORT | 20 reported applying knowledge and skills gained through Climate Ready and ISACC interventions to their work, more time is needed to determine the extent to which implementation and management of projects has improved. Climate Ready ensured equal representation of women in trainings in a proactive way if submitted participant lists were not sufficiently diverse. The Activity also went beyond this by including sessions on integrating GESI into projects in the project management course; this integration is a critical element for funds, such as GCF, Global Environment Facility (GEF), and AF. The evaluation found less evidence of this for ISACC, which was not designed or resourced to focus on GESI capacity building topics. Table 4 summarizes key capacity building conclusions for Climate Ready and ISACC. TABLE 4. CLIMATE READY AND ISACC KEY CONCLUSIONS: EQ 2 Category Climate Ready ISACC Successes There was evidence of increased capacity in project management, proposal development, and accreditation. While most training topics were relevant, linkages between more specialized courses (e.g., training for conservation officers on enforcement issues) and climate finance were unclear. There was evidence of increased capacity in M&E, knowledge management, and project management. M&E capacity building efforts were particularly significant, as they filled a gap. Key Interventions Climate Ready engaged 1,928 people in trainings and workshops, including similar numbers of men and women. Partnering with USP/Pacific TAFE generally increased training value and sustainability. ISACC trained 880 people, including similar numbers of men and women. The combined approach to capacity building was effective—formal trainings (USP/Pacific TAFE), short courses, follow￾up meetings, embedded staff, etc. Challenges The main challenge centered on retention, with some government staff changing positions after being trained. Budget cuts limited trainings and volume of participants, with inconsistent implementation of activities across PICs. There is a need for training on GESI and other topics, as well as better diversification of participants to remedy the tendency to invite the same government officials, primarily males, to multiple trainings. GESI Project management courses covered GESI integration in projects. There was no major focus on GESI beyond inviting women to take part in trainings. EQ 3: COUNTRY-OWNED IMPLEMENTATION To what extent did new USAID climate funding support countries to implement country-owned climate change actions? FINDINGS CLIMATE READY Successes Proposal Development. Climate Ready also assisted organizations in developing project proposals, helping to speed up the process. A local organization/NGO KII respondent said that Climate Ready “saved USAID.GOV DRAFT EVALUATION REPORT | 21 us almost two months each time to develop funding proposals.” Twelve KII respondents specifically mentioned Climate Ready’s work on project proposal preparation, with eight reporting that the proposals for projects to improve resilience to climate change were approved to receive funding. The proposals approved have been awarded with funding from several sources, including the GEF, the GCF, and the AF. Approximately $200 million has been mobilized to project proposals that were prepared with Climate Ready’s support thus far, although this is less than half the target of $500 million. However, validation￾workshop participants pointed out that Climate Ready is still being implemented, so more projects proposals that have received support from Climate Ready for preparation may be approved and the funding of those projects may be mobilized, bringing the value closer to the target. Respondents described how Climate Ready was much quicker to mobilize and more flexible than other sources of support, such as GCF readiness funding. Furthermore, KII respondents reported the support to be more useful, as the consultants engaged by Climate Ready were knowledgeable about the local context. In addition, guidance and templates developed for proposal preparation for a particular project could then be used for other proposals, helping create the conditions for sustainability. One specific success highlighted in KIIs and the desk review was the support to communities in FSM to prepare proposals for the GEF small-grant program, resulting in six proposals for climate resilience projects (with a total value of $360,000) gaining approval and starting implementation.35 Although the value of these projects was relatively small, results validation workshop participants highlighted the effect of this support at the community level. Two examples from KIIs or desk review documents were 1) providing resources that enabled in-country contribution to a proposal on climate change and water security in the outer islands of Kiribati and 2) building SPREP capacity to develop concept notes and project proposals. Accreditation. KII respondents noted how Climate Ready contributed to improving the conditions for country-owned implementation by helping organizations fill gaps in their policies and processes related to accreditation requirements. Respondents flagged this type of support as particularly sustainable, as these policies and processes are now available for project developers to follow with other funds. In total, 10 key informants out of 18 interviewed on country-owned implementation (see Figure 11) reported that Climate Ready successfully facilitated the accreditation process, and 7 of them reported that accreditation had been achieved, with none saying that it had not been achieved. Specific examples regarding accreditation given in KIIs and from the desk review included the following: • FSM Development Bank work on anti-money laundering policy for GCF accreditation • Preparation of an updated policy and operations manual for the Micronesia Conservation Trust after accreditation to the GCF • Outside of the evaluation’s focus countries, Climate Ready conducted rapid assessments of candidates for accreditation against the standards needed, which helped define needs and priorities and was mentioned positively in some KIIs 35 Examples of community groups receiving project funding included the Madalay Community, the Island Food Community of Pohnpei, and the Dien Community Water Security Project. USAID.GOV DRAFT EVALUATION REPORT | 22 FIGURE 11. KII DISCUSSIONS OF CLIMATE READY COUNTRY-OWNED IMPLEMENTATION SUCCESSES *Other changes generally tied to project implementation and management Feedback from KIIs on the contributions of Climate Ready was generally positive, noting that Climate Ready helped accelerate work on accreditation and build capacity in the processes necessary for accreditation. The Climate Ready support often came at a crucial point in the process. One national￾government respondent highlighted: “I feel like it really helped speed up the process of accreditation. We just don’t have the in-house capacity but with the support from the international consultant and . . . being able to speak the language that works for the GCF and the AF that was really helpful.” However, according to the desk review, only three PICs have attained national implementation entity accreditation thus far— half of the Climate Ready target of six (see discussion in Challenges, below).F 36 GESI integration. Many of the Climate Ready activities on accreditation and project proposal preparation related to GESI or environmental and social safeguards (ESS). Among 8 accreditation activities, 3 had a GESI component, as did 8 of 17 proposal support activities.F 37 IP and civil society KII respondents discussed the lasting effects of this work, including adjustments by the United Nations Environment Programme (which administers the small-grant program funded by the GEF) to the small-grant program’s application form to bring out the GESI component and changes to the project design approach used by communities so that gender is integral to the process, rather than added later in the design. In addition, the approach that the Climate Ready consultant developed with the Fiji Development Bank to include GESI in its systems and strategies was shared with the Ministry of Economy to inform its approach. This expertise was recognized, as the Ministry of Economy asked the same consultant back to put the GESI plan into action. Civil society KII respondents also reported that Climate Ready experts provided very practical advice on how to integrate GESI into projects and that their local knowledge helped engage communities in an inclusive way during project proposal development. Key Interventions Consultant TA. Consultants delivered most of the Climate Ready support discussed above and provided technical expertise on areas ranging from biodiversity to ESS, GESI, and project preparation (see Figure 12).38 IP KIIs mentioned that due to the COVID-19 pandemic, Climate Ready often paired international and local consults, thereby building expertise in the local consultant pool and bringing a benefit that was not originally envisaged. Trainings and workshops. KIIs also mentioned how the Activity’s trainings and workshops helped increase organizational capacity for implementation, although one nationally-affiliated organization KII respondent also identified the risk that the capacity could be lost, as it was not lodged in a specific team. Other activities. Additional Climate Ready work mentioned in KIIs and from the desk review that supported country-owned implementation included technical studies, support for stakeholder engagement in project development in FSM, and paying for project proponents’ travel for face-to-face consultations 36 DT Global (2021), USAID Climate Ready Quarterly Progress Report: FY21 Q3, https://pdf.usaid.gov/pdf_docs/PA00Z534.pdf. 37 Ibid. 38 Ibid. 9 7 10 8 14 Other changes* to country-owned implementation Accreditation achieved Accreditation process successful Proposals approved Proposal development process successful KII respondents (n=18) most often noted proposal development and accreditation as key Climate Ready country-owned-implementation successes USAID.GOV DRAFT EVALUATION REPORT | 23 on a project proposal in the outer islands of Kiribati. Lessons from Climate Ready support to accreditation were also shared at a regional meeting organized by the GCF, and KII respondents indicated that the lessons were of interest to other organizations thinking about accreditation. Two CSO and nationally￾affiliated organization respondents reported that as a result of Climate Ready support, they had developed a peer network of experts that they could contact for specific support in the future. Climate Ready also supported the early stages of proposal preparation (e.g., supporting organizations in the development of a concept note for a GCF proposal in FSM and for the Qatar Fund in Kiribati). FIGURE 12. KII DISCUSSIONS OF CLIMATE READY INTERVENTIONS CONTRIBUTING TO COUNTRY￾OWNED IMPLEMENTATION SUCCESSES Challenges Complexity of processes. Both accreditation and project proposal approval processes of the international funds are complex, requiring investment of significant time and resources. One nationally￾affiliated organization KII respondent noted that some GCF requirements for accreditation are not appropriate in small countries—for example, complex internal audits in an organization of eight people. Respondents also indicated similar issues with the GCF project approval process; for example, GCF requires a high level of detail for each of the atolls covered by a project proposal, which an NGO KII stated are not practical within a reasonable level of resources. These and other factors such as GCF internal processes for review reported by a number of KIIs led to a lengthy and complex approval process which has affected the achievement of Climate Ready targets for both the mobilization of investments and organizations accredited. None of the KIIs attributed delays to the actions of Climate Ready. Limited implementation. KII respondents agreed that there has been very limited implementation of country-owned climate change actions to date, as countries have yet to receive anticipated proposal funding. Likewise, results validation workshop participants identified that having proposals approved is only the first step in implementation, and management of the ongoing projects is needed. The only example of direct support for implementation of country-owned implementation mentioned in KIIs was the support Climate Ready provided to update ESS policies and processes for an ongoing SPREP project. ISACC Successes Proposal development and accreditation. ISACC supported country-owned climate change actions by helping governments engage in the proposal development and accreditation processes. Of the eight KII respondents who discussed country-owned implementation for ISACC, five noted that the program successfully facilitated the proposal development process, from concept notes to full funding proposals (see Figure 13). Similarly, four respondents mentioned ISACC support for accreditation to global funding mechanisms, such as the GCF and AF, with three indicating gaining accreditation. Desk review documents also described work on proposal development and accreditation, with a particular emphasis on enabling countries to have national organizations accredited to the GCF and AF—which would allow climate finance to be channeled directly to them rather than through international or regional accredited USAID.GOV DRAFT EVALUATION REPORT | 24 organizations, increasing local ownership. However, these documents mainly discussed activities with countries outside of the scope of the evaluation (i.e., Tuvalu, Cook Islands, and Solomon Islands). FIGURE 13: KII DISCUSSIONS OF ISACC COUNTRY-OWNED IMPLEMENTATION SUCCESSES *Other changes generally tied to project implementation and management Awareness-raising. The theme of increasing awareness within national governments around managing climate funding was highlighted as contributing to country-owned implementation. Four KII respondents described how ISACC brought a “spotlight” to climate finance, which is a relatively new area of focus for some PIC governments. While this mainly centered on requirements for M&E and reporting on use of funds (see EQ 2: Capacity building discussion, above), respondents also mentioned a shift in perceptions around who is responsible for climate change activities. Agencies, such as finance and planning, became more involved in climate change, which was previously the domain of environment offices. ISACC interventions brought together officials from a variety of agencies, ultimately increasing collaboration. An IP respondent stated: “What [ISACC has] been able to establish was to . . . help governments understand the importance of situating climate change finance, or in the first instance, having a specific climate change finance staff. And secondly, situating that within a central government agency such as the Ministry of Finance.” This was cited by several national-government and other-donor respondents as a significant change that is key to strengthening country-owned implementation. They also drew linkages to sustainability, as changes in ways of working among government officials should carry over into other climate finance moving forward. Desk review documents similarly noted ISACC’s support for “countries and partners to further understand the processes needed to access climate finance as well as the complexities that come with accessing and managing the funds.”39 Key Interventions Trainings and workshops. Respondents highlighted ISACC’s M&E and project management trainings as critical to strengthening proposal development, accreditation, and management of funds (Figure 14). These trainings provided technical knowledge (particularly for M&E, which was cited in several KIIs as a gap in the region) and information on GCF and AF requirements that will continue to be useful as countries develop proposals and aim to attain direct accreditation. As discussed above in EQ 2: Capacity building, the trainings helped counteract the tendency to silo climate financing. a USAID/other-donors respondent said that the workshops brought “all the different stakeholders in to understand the climate financing landscape in the Pacific . . . this used to be just a space for only the financial people so . . . [ISACC’s] approach of engaging with other sectors was important.” Embedded staff. ISACC support for staff embedded in government offices was also mentioned as contributing to proposal development, accreditation, and management. In particular, respondents 39 Buggy and Caucau (2020), “Interim Final Report for ISACC Climate Change Financing Component.” USAID.GOV DRAFT EVALUATION REPORT | 25 highlighted how critical embedding staff was to the success of ISACC activities, with the example of FSM cited several times. In FSM, the only country under consideration for the evaluation to have a full-time embedded position, the national coordinator helped organize ISACC implementation, supported progression of national climate change priorities, and worked on specific areas of multisectoral coordination for climate finance and climate change adaptation. Several respondents described how they believed embedded government staff will continue to apply the skills and experiences gained during ISACC implementation in their respective agencies, increasing sustainability of ISACC results. Desk review documents noted that two of the four embedded positions will continue to be supported by either government or other development organizations after the end of the ISACC program.1F 40 FIGURE 14. KII DISCUSSIONS OF ISACC INTERVENTIONS CONTRIBUTING TO COUNTRY OWNED IMPLEMENTATION SUCCESSES Other activities. Desk review documents described how ISACC meetings and events, such as climate finance awareness sessions and forums, contributed to awareness-raising, ultimately leading to shifts in management of climate finance as described above. One example cited was the climate finance forum in Samoa, where a national climate finance road map was developed.41 Finally, ISACC research support (e.g., climate finance assessments) gave government staff information that guided plans for proposal development. Challenges Funding and turnover. Both desk review documents32F 42 and KIIs noted that budget cuts affected country-owned implementation, as these caused a reduction in the number of embedded staff. As a result, interventions lost traction in countries without embedded positions; for example, one respondent described how the lack of a direct government channel in Samoa sometimes complicated ISACC implementation. In addition, respondents mentioned that ISACC also experienced internal staff turnover, particularly among country coordinators. Not only did this drain time and resources for recruitment, but it further affected the Activity’s work with government—when country coordinators left, ISACC would lose the relationships built over time with government officials. Structural barriers. KII respondents described numerous structural barriers to country-owned climate change action, including the need to “have the right systems in place,” which can be difficult to achieve for small countries with limited resources. Respondents worried that it will likely take significant time for some PIC countries to meet donor requirements; in the meantime, these countries are experiencing the 40 Nyuykonge (2020), “End of Project Evaluation for the Institutional Strengthening for Pacific Islands to Adapt to Climate Change (ISACC) Project & Design Support for a New Climate Change Adaptation Programme.” 41 Ibid. 42 Buggy and Caucau (2020), “Interim Final Report for ISACC Climate Change Financing Component.” 1 1 1 3 3 M&E/research Consultant TA Knowledge management tools Support for embedded staff Trainings and workshops KII respondents (n=8) most often noted trainings and embedded staff support as contributing to country-owned implementation successes USAID.GOV DRAFT EVALUATION REPORT | 26 negative effects of climate change and running out of time to counteract these. Thus, respondents advocated for additional resources and support, particularly for smaller countries or more remote areas. CONCLUSIONS It is too soon to tell the extent to which the Activities supported implementation of country-owned climate change actions, as most PICs are still in the process of obtaining funding. However, the Activities contributed to improving the conditions for future implementation via their work on accreditation to international funds and development of project proposals. While Climate Ready supported direct proposal writing and engagement in accreditation processes, ISACC focused more on equipping government staff to take on these tasks. Nevertheless, structural challenges often undercut these outcomes: accreditation and funding processes are slow, and requirements are difficult for PICs to meet. Finally, Climate Ready’s focus on GESI (specifically GESI integration into proposals) remained useful for stakeholders. Table 5 summarizes country-owned implementation conclusions for Climate Ready and ISACC. TABLE 5. CLIMATE READY AND ISACC KEY CONCLUSIONS: EQ 3 Category Climate Ready ISACC Successes There wase evidence of effective support for accreditation and proposal development; consultants helped speed up these processes (particularly compared with support offered by international funds). Work on GESI and ESS elements of accreditation and proposal development have the potential to be sustainable, as these can be used for future projects. There was evidence of using a skill-building approach to support country-owned implementation; ISACC capacity development efforts helped PICs put systems in place to develop proposals and apply for accreditation. The “awareness-raising” effect of ISACC interventions has shifted how some government offices operate (e.g., finance and planning becoming more involved in climate change). Key Interventions Consultants, both international and local, provided human-resource support and expertise on a wide range of technical and subject-matter areas. Trainings and workshops supported capacity building in project implementation. Trainings and workshops provided technical knowledge on M&E and information on GCF and AF requirements. Embedded government staff helped facilitate implementation of ISACC interventions and contributed to sustainability, as they are likely to continue applying skills and experiences in their respective agencies. Challenges Complexity of approval processes and requirements inhibited achievement of funding ($200 million/$500 million targeted) and accreditation (3/6 organizations targeted) indicators. Given slow funding processes, there are few country-owned climate change projects in place. Budget cuts remained a significant issue that affected country-owned actions (as well as other outcome areas, such as enabling environment and capacity building). There was a tension between the time required to meet complex donor requirements and the pressing need to address climate change. USAID.GOV DRAFT EVALUATION REPORT | 27 EQ 4: PRIVATE SECTOR AND INSTITUTIONAL ENGAGEMENT What were key successes and challenges faced by these Activities in engaging the private sector and climate financial institutions? FINDINGS CLIMATE READY Successes and Key Interventions Engagement via capacity building. Desk review and KII data indicated that successful engagement with the private sector and climate financial institutions had been very limited, with the exception of some capacity building (see Figure 15). Specifically, Climate Ready delivered a training on business continuity planning, engaging a total of 449 people in Fiji (of which 300 were women) and 49 in Samoa (of which 30 were women).3F 43 One other local organization/NGO KII respondent observed that when companies engaged with climate change and disaster risk management, employees became more knowledgeable and there was a trickle-down effect on their behavior in their private life. In addition, one other local organization/NGO KII respondent reported that engagement with micro-, small, and medium enterprises (MSMEs) in this area had a broader effect on their work with the business federation, enabling the federation to better represent them with the government. The respondent reported that this engagement resulted in concessionary loans from the government for MSMEs affected by the COVID-19 pandemic.44 FIGURE 15. KII DISCUSSIONS OF CLIMATE READY PRIVATE SECTOR AND INSTITUTIONAL ENGAGEMENT SUCCESSES Overall, Climate Ready interventions contributed to a limited extent to ongoing changes to businesses’ ability to plan for disasters. In Samoa, there was a training of trainers on business continuity planning, enabling the training to continue after Climate Ready has finished.34F 45 According to participants in the results validation workshop, the Samoa Chamber of Commerce has taken ownership of this training and continues to train others in the private sector. Climate Ready developed a business continuity planning template and tools and used them in their trainings; other local organization/NGO KII respondents mentioned that these tools are easy to use and adapt to different business types, giving the template wider 43 List of training participants provided by Climate Ready to the evaluation team in November 2021. 44 Government of the Republic Fiji (2020), “Fijian Government Unveils COVID-19 Concessional Loan Packages for Fijian Micro, Small and Medium Enterprises,” https://www.fiji.gov.fj/getattachment/bba34afb-b4f6-4e6b-90f6-b0392ae4a18f/CONCESSIONAL￾LOAN-PACKAGES-FOR-MSMEs.aspx. 45 DT Global (2021), USAID Climate Ready Quarterly Progress Report: FY21 Q3, https://pdf.usaid.gov/pdf_docs/PA00Z534.pdf. 4 8 Engaged in implementation of country-owned climate action Engaged in capacity building KII respondents (n=8) most often cited capacity building as a key private sector and institutional-engagement success USAID.GOV DRAFT EVALUATION REPORT | 28 reach. The Activity also launched an infographic video to complement the template, which could be used by those who had not attended the training. Challenges Funding. The effect of budget cuts on interventions to engage the private sector continued even when the budget was restored. KII respondents agreed that long-term engagement of the private sector (particularly MSMEs) in climate change adaptation and disaster risk management had been weak in PICs, as it was not considered a priority. To address this challenge, early in the Climate Ready implementation, PIFS arranged a subregional private sector workshop on climate finance for Melanesia in partnership with Climate Ready. The workshop was one of the first to bring the private sector and government officials together on climate finance and was meant to start a more substantial initiative on private sector engagement in partnership with PIFS. Unfortunately, budget cuts meant the work did not go ahead, and once the budget was restored, PIFS had moved on to other priorities. Other challenges. KII respondents indicated specific challenges with engaging with the private sector in the PICs, arising from their geography. Generally, according to participants in the validation workshop, the small populations and large distances in PICs mean that there is a relatively low profile for the private sector and limited experience of governments and the private sector working together. Chambers of commerce, for example, are state based, and as such are dispersed in many islands, requiring more resources to engage them properly. IPs would also need to add different expertise to engage with the private sector as well as with governments. ISACC Successes and Key Interventions Engagement via trainings and meetings. ISACC organized a few interventions that brought together government and nongovernment actors, including the private sector and climate finance institutions (Figure 16). One IP KII respondent noted, for example, the 2019 Regional Climate Change and Disaster Risk Finance Forum, which served as a peer learning event for countries to exchange information on successes and challenges around accessing climate finance. The meeting included a media training designed to increase familiarity among journalists with PIC-specific climate change issues. The desk review also documented organization of this forum, which brought together 80 participants from PIC governments, NGOs, the private sector, and development partners.46 Another local organization/NGO respondent mentioned inclusion of the private sector in climate finance forums. Specifically, ISACC organized a Climate Change Finance Forum in Samoa with representatives from government, NGOs, CSOs, the private sector, regional organizations, and development partners. This provided an opportunity for discussions on challenges and opportunities with regard to climate change financing.36F 47 ISACC also supported climate finance working groups that engaged the government alongside CSOs and the private sector. Desk review documents discussed how these groups helped promote collaboration, as they brought together people who were working on the same issues but without knowledge of one another’s efforts due to lack of national-level coordination strategies.35F 48 Finally, ISACC included private sector actors in capacity building via M&E Communities of Practice, established as a follow-on to M&E short-course trainings to maintain peer learning. 49 46 Buadromo and Caucau (2019), “USAID ISACC Quarterly Report (October–December 2018),” 47 Vuki Buadromo and Amelia Caucau (2017), “USAID ISACC Quarterly Report (April–June 2017),” USAID. 48 Nyuykonge (2020), “End of Project Evaluation for the Institutional Strengthening for Pacific Islands to Adapt to Climate Change (ISACC) Project & Design Support for a New Climate Change Adaptation Programme.” 49 Buggy and Caucau (2020), “Interim Final Report for ISACC Climate Change Financing Component.” USAID.GOV DRAFT EVALUATION REPORT | 29 FIGURE 16. KII DISCUSSIONS OF CLIMATE READY PRIVATE SECTOR AND INSTITUTIONAL ENGAGEMENT SUCCESSES Challenges No design and resourcing. Engagement of the private sector and climate finance institutions was not a core focus of ISACC programming. As reflected in KIIs and desk review documents, ISACC’s work with these actors did not extend beyond inviting them to specific meetings, events, and trainings (as described above), ultimately limiting the potential for outcomes related to private sector or institutional engagement. KII respondents noted a need in the Pacific region to strengthen the role of the private sector in mobilizing climate finance and implementing projects, recommending that future initiatives reach out to these stakeholders more strategically. CONCLUSIONS Overall, the Activities did not significantly engage with private sector and climate finance stakeholders, which makes it difficult to determine successes and challenges. However, this is not necessarily due to issues with Activity implementation; Climate Ready and ISACC were never designed and resourced to prioritize this area of work (Climate Ready initially planned for this but changed course due to budget cuts). Nevertheless, both Activities undertook a few relevant initiatives, such as the Climate Ready business continuity training and the ISACC media training. Moving forward, engagement of the private sector in particular has been identified as a key gap and area to prioritize for future growth. Table 6 summarizes key private sector and institutional engagement conclusions for Climate Ready and ISACC. TABLE 6. CLIMATE READY AND ISACC KEY CONCLUSIONS: EQ 4 Category Climate Ready ISACC Successes and Key Interventions The primary method of engagement of private sector stakeholders was on business continuity planning, which included development of templates and providing training to extend outcomes beyond the Activity’s interventions. The primary method of engagement was inviting private sector stakeholders to trainings, meetings, and events. There is some (limited) evidence that including private sector and climate finance stakeholders in national initiatives (i.e., climate finance working groups) can promote collaboration and help counteract tendency to work in silos. Challenges Budget cuts and geography limited the ability of Climate Ready to reach some stakeholders. This area of work was not prioritized in terms of design and resourcing. Engagement of the private sector and climate finance institutions was not prioritized; therefore, few outcomes materialized in this area. 2 3 4 Engaged in implementation of country-owned climate action Engaged to support the enabling environment Engaged in capacity building KII respondents (n=8) most often cited capacity building as a key private sector and institutional-engagement success USAID.GOV DRAFT EVALUATION REPORT | 30 RECOMMENDATIONS Recommendations focused on the four main areas examined for the evaluation—1) enabling environment, 2) capacity building, 3) country-owned actions, and 4) private sector and institutional engagement—are presented below. Specific recommendations on the crosscutting themes of GESI and Activity implementation are also included. The recommendations provided in this report were developed via a multistep process (see further discussion in the analysis section of Annex B: Methodology) that included validation of findings and conclusions and co-development of recommendation language with results validation workshop participants. The outputs of these workshops were then examined against the evaluation team’s analysis and triangulation of KIIs, monitoring data, and desk review information. The recommendations reflect the experiences and lessons learned from Climate Ready and ISACC, while highlighting potential approaches for implementing the USAID Climate Strategy 2022–2030.50 ENABLING ENVIRONMENT 1. Support implementation of policies, plans, and frameworks developed through Climate Ready and ISACC Given the contributions of Climate Ready and ISACC to the establishment of policies, plans, and frameworks, a key next step will be continuing and deepening their implementation (including ongoing monitoring of their effectiveness to determine whether further amendments are needed). This will require coordination from a variety of actors, working together toward a long-term goal of ensuring that policies, plans, and frameworks become integral to government and organizational systems. Specifically, governments and regional organizations must identify and mitigate any barriers to implementation, such as gaps in capacity and resources. Donors, such as USAID/Pacific Islands and IPs, should design any new Activities by working on the enabling environment in a manner that builds on existing policies, plans, and frameworks and avoids duplication. Specific strategies may include the following: • Develop a dissemination plan: Governments and regional organizations that have adopted specific policies, plans, and frameworks should first determine (via departmental meetings, a survey, etc.) the extent to which these are understood by staff at different levels or departments and then develop a strategy for uptake of key information. This will help prevent the policy or plan from becoming siloed in a specific department or location. For example, in PICs, such as FSM, with dispersed geographies, it is critical that both national and local governments have the right information and systems in place to coordinate implementation of policies, plans, and frameworks. • Prioritize use: Implementation of future climate finance Activities by USAID/Pacific Islands and IPs should include capacity building on existing policies, plans, and frameworks, in line with USAID’s Climate Strategy Intermediate Result (IR) 1.2 on strengthening the capacity of national governments to enact responsive plans and policies.51 For instance, training on use of the PCCFAF for climate finance assessments would both support adoption of the framework and increase local capacity in research and M&E, which will be critical for future implementation of country-owned actions (see further discussion under 4. Reorient focus towards project implementation). CAPACITY BUILDING 2. Expand capacity building opportunities and partnerships focused on project management and M&E Climate Ready and ISACC both offered effective capacity building opportunities in project management and M&E, and these are topics of continuing demand and interest. Project management will be important 50 USAID (2021), “USAID Climate Strategy 2022–2030: Draft for Public Review” (November 2021), https://www.usaid.gov/sites/default/files/documents/USAID-Climate-Change-Strategy-2022-2030-Offical-Draft.pdf. 51 Ibid., p. 8. USAID.GOV DRAFT EVALUATION REPORT | 31 as countries begin to receive funds for implementation of projects (see further discussion under Recommendation 4, “Mobilize support for project implementation”). Similarly, M&E will be needed to help ensure that projects are on track and for reporting to donors. Moving forward, USAID/Pacific Islands should include project management and M&E capacity building in future Activities and work with IPs to design trainings that take into account the following: • Diversity of participants: To strengthen capacities and leadership opportunities of local communities as part of USAID’s Climate Strategy IR 1.4,52 IPs should expand who is included in trainings. Specifically, government officials at different levels (local and national), women, those in remote areas, and members of other relevant marginalized communities should be targeted. This will require proactive recruitment, as in the example of Climate Ready going back to organizations to ask for more women to be nominated for trainings (see discussion under EQ 2: Capacity building, above). • Formalized offerings: Based on the generally positive feedback of the partnership with USP/Pacific TAFE, formalized offerings (trainings by an established institution) should be continued. However, given that this format was not suited to all participants, USAID/Pacific Islands and IPs should seek additional partnerships with other universities in the North Pacific or elsewhere and continue to offer diverse capacity building options, such as short courses and workshops. • Expand partners: International finance organizations should be included as much as possible in the design and delivery of trainings to ensure responsiveness to their processes and standards. 3. Strengthen local ownership of capacity building Given the success of Climate Ready’s and ISACC’s approach of using local experts and locally led strategies for enabling environment and country-owned implementation initiatives (and in alignment with the USAID Climate Strategy crosscutting principle of locally led development53), it should be broadened for capacity development. USAID/Pacific Islands should prioritize funding activities that contribute to building local skills and moving away from the “fly in, fly out” international consultant model. Areas to focus on may include the following: • Training of trainers: Future Activities should include funding to develop groups of local experts in project management and M&E, who can then train others. This would also allow for trainings to be held in local languages—as needed—and encourage the development of tools and templates in local languages to be more inclusionary. Investing in local experts should help make capacity building activities more contextually relevant and increase sustainability (these experts can continue training government, community members, civil society, etc.). • Diversify training locations: KII respondents noted the utility of Climate Ready offering trainings in various locations, including in more remote areas. This approach, which helped increase access for different populations, should be continued moving forward. This will require sufficient planning and resourcing at the design stage. • Build on existing tools and templates: Although KII respondents and results validation workshop participants noted that M&E is a relatively new area of focus for PICs, Climate Ready and ISACC, among other programs, completed significant work on this topic over the past several years. Thus, when designing future M&E (or project management) trainings, IPs should aim to make the best use of tried and tested techniques and tools already circulated or developed within the region. 52 Ibid. p. 11. 53 Ibid., p. 6. USAID.GOV DRAFT EVALUATION REPORT | 32 COUNTRY-OWNED ACTIONS 4. Mobilize support for project implementation As described in EQ 3: Country-Owned Implementation (above), Climate Ready and ISACC contributed to successful proposal submission and accreditation; thus, in the short to medium term PICs will increasingly start to receive funding to implement projects. To accommodate this influx, USAID/Pacific Islands and IPs should increase efforts to strengthen management of future climate finance Activities, particularly in terms of absorption and utilization of funds (of course, accessing climate finance will remain relevant, depending on the volume of funds mobilized). This will be a critical contributing step toward USAID’s Climate Strategy IR 1.3, which emphasizes incentives and capacity development to ensure transparent and accountable governance of investments.54 M&E, GESI, ESS, and regional planning and cooperation will all be key to this work and should be integrated into the following interventions: • Trainings: As discussed above (Recommendation 2, “Expand capacity building opportunities and partnerships focused on project management and M&E”), future trainings should focus on project management and M&E to strengthen government capacity to manage, monitor, and report on project results. Specialized topics, such as GESI and ESS, should also be included. • Consultant TA: In the short term, consultant TA (using local consultants when possible or pairing local and international consultants) may be needed to develop approaches on such topics as GESI, ESS, M&E, and financial management systems in place to receive grants. This support should integrate coaching or other capacity building efforts so that government counterparts can continue this work over the long term. PRIVATE SECTOR AND INSTITUTIONAL ENGAGEMENT 5. Design and resource activities that engage the private sector The evaluation did not find evidence of strong engagement of the private sector, mainly because Climate Ready and ISACC were not designed or resourced to carry out this type of work. In line with USAID’s Climate Strategy crosscutting principle of private sector engagement,55 various data sources indicated the importance of coordinating with the private sector to diversify sources of funding. To begin this work, USAID/Pacific Islands should commission an analysis of the private sector landscape to determine the most critical private sector actors, where they are located, and their potential contributions to climate projects (identification of possible climate project implementers versus funders, and of how small, medium, and large international business can all be uniquely engaged). Then, the design of new Activities should include a specific objective on private sector engagement, with indicators developed to track progress in this work. One lesson learned from Climate Ready and ISACC that is relevant for new Activity design was around the importance of using specific incentives to bring the private sector to the table (e.g., offering business continuity planning tools and templates that could be used by businesses). IMPLEMENTATION 6. Prioritize local, contextually tailored implementation of interventions One key success factor of both Climate Ready and ISACC’s work was the use of local experts and locally led strategies (see discussions under EQ 1: Enabling Environment, and EQ 3: Country-Owned Implementation). Specifically, Climate Ready hired local consultants to perform TA, and ISACC was implemented by a regional organization. This approach should be continued and expanded moving forward, with local experts, national and regional organizations, and government at different levels 54 Ibid., p. 10. 55 Ibid., p. 6. USAID.GOV DRAFT EVALUATION REPORT | 33 involved in the design and implementation of future climate finance Activities to ensure that these are aligned to regional and national priorities. Leadership by local entities (either as main implementers or key partners) should be a key criterion for funding by USAID/Pacific Islands of any new Activities, in alignment with the Agency’s increasing commitment to and targets around localization.56 7. Integrate embedded staff or focal points in future Activities ISACC’s funding of embedded national-government positions contributed to enabling environment, capacity building, and country-owned implementation successes. The position supported coordination of ISACC interventions and served as an advocate for climate finance priorities within government. Budget cuts hampered the approach, and as a result embedded staff were not available in all countries under consideration for the evaluation. However, in FSM (where an embedded staff position was supported), this intervention facilitated ISACC implementation. Thus, USAID/Pacific Islands and IPs should plan and budget for embedded positions in future Activities, which should help increase local ownership, as called for in the crosscutting principle of locally led development in the USAID Climate Strategy.57 If fully funding these positions is not possible, another approach is to identify and recruit staff in PIC governments who can act as focal points for the governments, USAID, and IPs. GESI 8. Ensure that GESI is integrated into future programs at the design stage and properly resourced Evaluation results highlighted the importance of properly planning for and resourcing GESI integration to achieve results. With USAID/Pacific Islands (e.g., via the USAID Climate Strategy IR 1.5 on increasing women’s engagement and leadership in climate actions)58 and international finance organizations both prioritizing GESI, it will be important to incorporate this across all types of climate projects in the region and at all levels (national and local). This will require both funding and specialized staff who can deliver GESI interventions. Specific actions that USAID/Pacific Islands, IPs, and government can all take to better integrate GESI moving forward may include the following: • When designing new programming, IPs should develop a GESI action plan (based on a rigorous, context-specific GESI analysis that identifies critical issues at the outcome level) that goes beyond ensuring that similar numbers of men and women participate in trainings/workshops, with indicators to track implementation in climate finance projects. • Projects should engage more directly at the community level, particularly, but not solely, with women and those in remote areas. Integration of knowledge and experience from local communities and from disadvantaged groups at the design stage can build stronger projects that are more likely to be implemented effectively. To facilitate this, USAID/Pacific Islands should support projects in the processes needed to engage with different groups. • National governments should aim to develop dedicated climate finance officer positions in relevant offices focused on supporting women and marginalized groups (e.g., ministry of women). This can help ensure that climate change is mainstreamed into budget planning processes and that these groups have representation in national climate change and development plans. • IPs should incorporate GESI in workshops and trainings going beyond how to include GESI in proposal development, to focus on integration of GESI in project implementation and monitoring. 56 USAID (2021), “Administrator Samantha Power on a New Vision for Global Development” (November 4, 2021), https://www.usaid.gov/news-information/speeches/nov-4-2021-administrator-samantha-power-new-vision-global-development. 57 USAID (2021), “USAID Climate Strategy 2022–2030,” https://www.usaid.gov/sites/default/files/documents/USAID-Climate￾Change-Strategy-2022-2030-Offical-Draft.pdf, p. 6. 58 Ibid., pp. 11–12. USAID.GOV DRAFT EVALUATION REPORT | 35 ANNEX A: STATEMENT OF WORK SERVICE REQUEST Performance Evaluation of the Institutional Strengthening in Pacific Island Countries to Adapt to Climate Change (ISACC) and Pacific Climate Ready (READY) Projects C.1PURPOSE OF EVALUATION COP21 of the United Nations Framework Convention on Climate Change resolved that the provision of urgent and adequate finance, technology, and capacity building support by developed country Parties needed to be scaled by significantly and attendees agreed to provide substantial financing and support through international donors to fund interventions to address climate change. The USAID Climate Ready and the Institutional Strengthening in Pacific Island Countries to Adapt to Climate Change (ISACC) projects have been the spearhead of USAID pacific islands entry into the field of Climate Finance. Both the projects have tracked significant achievements over the last 5 years for example, for the FY 2020 reporting period Climate Ready helped Pacific Island countries access $54 m of international climate funds and the ISACC project closed its books with having supported Pacific Island countries propose, adopt or implement 9 laws, policies, regulations, addressing climate change adaptation supported by United States Government assistance. The purpose of this evaluation is to get a better understanding of how and if these achievements have contributed to meeting USAID/Pacific Islands' long-term goal of helping the PICs achieve self-reliance - in advancing the disaster management capacity to respond to disasters and to prevent and mitigate the adverse impacts of climate change. The hope is that this evaluation will answer questions like which capacity building activities were most successful in strengthening the capacity of PICs to access and utilize climate financing and go beyond the number of people who were trained. The conclusion of this evaluation will be instrumental not only for future programming on climate finance and adaptation but hopefully also feed into the larger policy realignment as the United States returns to the Paris Agreement. At the very least this evaluation will support USAID’s Monitoring & Evaluation Plan and Learning agenda which clarifies the commitment of USAID/Pacific Islands on using empirical evidence to make the most effective programmatic and management decisions. Transparency and accountability of USAID’s program effectiveness to stakeholders and learning to improve development outcomes are the primary purposes of all evaluation efforts of USAID as indicated in its Evaluation Policy[1]. The evaluation of ISACC and READY will run for six months, and will commence in May 2021, and end with the submission of the final report to USAID/Pacific Islands by November 2021. The implementation timeline for ISACC and READY Activities that will be evaluated are as follows: ISACC: September 29, 2015, to May 2021 READY: November 22, 2016, to May 2021 C.2BACKGROUND a) Identifying Information While USAID Climate Ready and ISACC have similar scopes, there are important differences between the two programs. They are both positioned/ designed to provide needed capacity to Pacific Island countries to access climate finance. USAID.GOV DRAFT EVALUATION REPORT | 36 ISACC was implemented by a key regional organization and it allowed USAID to work directly with Pacific regional organizations to increase the capacity of national institutions. ISACC’s scope of work was narrowly focused, specifically assessing and strengthening host country systems to prepare them to manage their own international climate change funding. ISACC works in eight of USAID’s focus countries in the Pacific including: Federated States of Micronesia, Fiji, Kiribati, Palau, Samoa, Solomon Islands, Tuvalu and Vanuatu. The focus of evaluation for ISACC will centre around undertaking processes to build capacity of government institutions involved in adaptation and disaster response: the development of a Joint National Action Plans (NAP); the creation of coordination mechanisms or new organizational structures and frameworks that are meant to enhance a countries readiness to access climate finance and management of these funds. USAID Ready, on the other hand, has a much broader scope. Its three main components focus on strengthening policy and institutional arrangements within host country governments, increasing access to international financing, and strengthening both government and private sector project implementation capacity. While ISACC is focused more tightly on building capacity within government departments, USAID Ready will partner with multilateral institutions to develop proposals for funding and build capacity of private sector actors who may be implementing climate change projects in the future. It will also support capacity building for the Pacific regional organizations themselves. USAID Ready works in 11 Pacific Island countries including: The Federated States of Micronesia, Fiji, Kiribati, Palau, Papua New Guinea, Republic of the Marshall Islands, Samoa, Solomon Islands, Tonga, Tuvalu and Vanuatu. The focus of evaluation for Ready will be on technical support provided to achieve accreditation with international climate funds such as the Green Climate Fund, Adaptation fund and support to prepare bankable project proposals that can be submitted to these institutions. The two USAID activities, ISACC and READY, implemented activities on strengthening the capacity of host governments in securing climate funds. Securing climate financing is an important element for planning and executing the various Climate Change Adaptation and Mitigation programs of the Pacific Island Countries (PICs). Accessing climate financing is one of the measures for appraising the capacities of PICs to plan for, deliver, monitor and report on their climate change priorities and initiatives, in ways that are catalytic and fully integrated with their respective national development priorities. Project Titles: Performance Evaluation of Institutional Strengthening in Pacific Island Countries to Adapt to Climate Change (ISACC); and Pacific Climate Ready (READY) Projects Activity/Project Name a. Institutional Strengthening in Pacific Island Countries to Adapt to Climate Change (ISACC) b. Pacific Climate Ready (READY) Implementing Partners a. Pacific Community (SPC) b. DT Global Cooperative Agreement/Contract # a. AID-492-10-15-00004 b. AID-492-H-17-00001 USAID.GOV DRAFT EVALUATION REPORT | 37 Total Estimated Cost a. $5,000,000. b. $24,903,456 Life of Project/Activity a. September 29, 2015 - June 30, 2021 b. November 22, 2016 – November 21, 2021 b) Project/Activity Information or Development Context ISACC The Institutional Strengthening in Pacific Island Countries to Adapt to Climate Change (ISACC) was implemented under a cooperative agreement with the Pacific Community (SPC) in partnership with the Pacific Islands Forum Secretariat (PIFS) and the Secretariat of the Pacific Regional Environment Programme (SPREP) on the climate finance component ran from September 2015 to September 2020. The geographic coverage included eight Pacific Island countries namely Fiji, Federated States of Micronesia, Kiribati, Palau, Samoa, Solomon Islands, Tuvalu and Vanuatu. As mentioned previously ISACC was a response to the need of Pacific Island countries to improve/enhance their capacity to effectively plan for, coordinate and respond to the adverse impacts of climate change. Key results areas included access to new climate change finance enhanced through improved capacity, systems and tools. Assistance delivered through mandated regional organizations presented USAID the unique opportunity to bring about country-owned awareness, recognition and solutions to system and capacity deficits that need urgent attention to bolster PICs ability to better access and benefit from the wide variety of climate change finance sources available. The Project was designed to implement activities and provide technical support under three key result areas: -1: Integrated institutional frameworks and national capacity strengthened to support multi-sectoral approaches to climate change and disaster risks. - 2: Access to new climate change finance enhanced through improved capacity, systems and tools. - 3: Regional cooperation and coordination strengthened through augmented national capacity, delivered through shared learning, to support national responses to climate and disaster risks. A summary of the key achievements from 2016 - 2019 are detailed below: 2017 USAID.GOV DRAFT EVALUATION REPORT | 38 2018 2019 The PICs recognize that their existing governance system will need to improve to meet the requirements for accessing new international funds on climate change that was flowing into the region. These requirements include the need for high level climate change plans, drafting of new policies and strategies and setting-up institutional arrangements. The ISACC project provided support to strengthen the institutional capacity of countries to address climate change based on the results of the scoping activity implemented. This activity recognized several challenges to help build these institutional capacities of PICs such as: the high labor and financial cost of supporting eight (8) PICs; investments to hold series of consultations and dialogues to bring about institutional (by default cultural changes) changes in core government ministries; and advocating for adjustments on indicators required by Donors to match the demands and needs of National governments. Climate Ready Climate Ready was designed with the objective of supporting Pacific Island countries to become more resilient to climate change and protect the lives and livelihoods of their citizens. Preparing for and responding to climate impacts place additional pressures on budgets and government structures in Pacific Island countries. As additional climate financing becomes available for the Pacific from such international sources as the Green Climate Fund and the Adaptation Fund, governments are requesting help to access these resources, and Climate Ready is positioned to assist. Climate Ready will build on existing initiatives to help national governments access climate funding and carry out their plans and priorities. • Policy: Assist countries to incorporate and operationalize climate adaptation goals into national policy. • Finance: Bolster the abilities of countries to access international climate change funds. This can be USAID.GOV DRAFT EVALUATION REPORT | 39 done by developing partnerships with a multilateral entity such as the United Nations Development Program or by supporting national governments to access international adaptation funding directly through such mechanisms as the Green Climate Fund. • Management: Build professional capacity and management systems that coordinate, manage, monitor and report on adaptation projects. This can be with national governments, as well as private sector and civil society entities that are executing climate change projects. To date, the achievements of Climate Ready Include: c) Intended Results ISACC The goal of the regional project is to strengthen the national institutional capacity of 8 Pacific Island countries (PICs) to effectively plan, coordinate and respond to the adverse impacts of climate change. PICs will have established strengthened institutions and human capacity to access and manage new sources of global climate change finance and, effectively coordinate and implement innovative, multi-sectoral approaches to combatting the adverse impacts to climate change and disaster risks. Delivery of the project will be led by SPC in close collaboration with PIFS and SPREP thereby contributing to sustainability after the completion of the project. The approach will be based on increasing adaptive capacity of PIC governments through: • Strengthened relevant national capacity in adaptation planning, policy development, project development and institutional networking including sharing of information. • Strengthened coordination mechanisms at national and regional levels to manage climate-related risks and funding opportunities in an integrated manner. • Enhanced technical, institutional and human capacity at the national level to coordinate, manage, implement and evaluate multi-sector climate change projects / programmes. USAID.GOV DRAFT EVALUATION REPORT | 40 • Development and/or expansion of tracking tools for climate change finance. • Support to PICs to access and manage international climate change finance. • Establishment of a regional process for shared learning on climate change through peer to peer exchanges, regional meetings & targeted trainings. The project will be delivered by a team of skilled professional specialists from the SPC, PIFS & SPREP in conjunction with stakeholders from national governments which will include central climate change, finance and planning agencies. The project will also engage with the United States (US Embassies in PICs to ensure that project’s activities complement new and existing Regional Programmes and US agencies. Recognizing that the institutional and capacity needs differ amongst the PICs, the project will implement selected activities in designated PICs to ensure that resources are maximized and project outcomes are achieved. Where feasible, the project will collaborate with other regional climate change programmes to support the implementation of priority activities in PICs which do not receive resources for certain activities. Climate Ready The primary objective of this project was to help 11 Pacific Island countries to achieve their climate change adaptation goals by assisting them to develop and implement country driven, coordinated climate action policies and plans. The ultimate outcome of climate ready will be an increased number of adaptation interventions that were fueled by policy and frameworks put in place that make climate finance accessible and a professional workforce that is able to manage new climate finance. The project will prioritize climate financing as the principal project driver to spark the pacific islands interest and help create new frameworks and systems that enable the Pacific Islands to attract and manage climate funds. If the funds have already been received the project will focus on improved fund management and provide capacity building assistance to ensure sustainability of efforts. The project will be focused on transforming the workforce of the Pacific islands to become proficient in world class methods and tools for project development, implementation and management by creating capacity building platforms and tools that will draw on resources both in and outside of the traditional government systems. The project will leverage e learning and explore open sources massive online courses to expand training as well. d) Approach & Implementation Both ISACC and Ready provided technical assistance that were tailor made to assist national governments to draft or update climate-related laws, policies and regulations, boosted project management capacity of government and non-government stakeholders by designing and delivering training /certification programs. The underlying common strategies were guided by a focus on effective and meaningful participation of a wide range of stakeholders as well as targeted policy analysis to support evidence-based decision making for policy makers. For fund readiness activities Climate Ready adopted a step wise approach that included raising awareness on access requirements of international climate finance institutions through meetings and country engagements. Ready worked with regional and international partners to evaluate their capabilities and options on their pursuit of accreditation. e) Geographical Coverage & Targeted Beneficiaries The two Activities have been implemented in four of the 11 Pacific Islands countries. The targeted beneficiaries are countries themselves, i.e., their climate change institutions and their staff, as well as private sector and civil society entities that execute climate change projects. f) Available Documents and Data USAID.GOV DRAFT EVALUATION REPORT | 41 Copies of the scope of work of ISACC and Ready, all progress reports including approved workplans, Monitoring, Evaluation and Learning (MEL) plan and tools, training modules and training reports prepared by the Implementing Partners and submitted to USAID will be provided to the Evaluation Team as soon as contract is awarded. C.3EVALUATION QUESTIONS The performance evaluation will answer the questions below. These questions are listed in the order of priority. 1. How did the Activities impact the enabling environment required to access climate financing (including governmental policies, organizational structures, standard operating procedures, plans, frameworks, legislation)? a. What were the most significant changes to the enabling environment, and how did these changes come about? b. To what extent did changes to the enabling environment support gender equality and social inclusion (GESI)? 2. How did the Activities contribute to building the capacity of government to plan, design, implement (manage), monitor, and evaluate climate change adaptation projects that will be funded through international climate finance instruments? a. What were the most significant changes to government capacity, and how did these changes come about? b. To what extent did capacity building efforts support GESI (beyond including women in capacity building activities)? 3. To what extent did new USAID climate funding support countries to implement country-owned climate change actions? 4. What were key successes and challenges faced by these Activities in engaging the private sector and climate financial institutions? In compliance with revised ADS 205 on Integrating Gender Equality and Female Empowerment in USAID’s Program Cycle and ADS 201 on Program Cycle Operational Policy released in 2016; the evaluation will be gender responsive. Gender-sensitive indicators, sex-disaggregated data, and attention to gender inter￾relations are required elements of USAID evaluations; therefore, the Contractor must ensure gender integration in the design and implementation of the evaluation to the extent possible. From within the evaluation team, at least one member should have expertise on gender-related project evaluation; or a part-time Gender Specialist shall be hired to support the evaluation team on gender integration throughout key steps of the evaluation. Because marginalized communities and groups (e.g., women, immigrants, the elderly, the disabled) are more exposed to climatic risk, the costs of climate change are more difficult for them. Women in particular are structurally vulnerable, and climate change can worsen existing gender-based inequities that keep them impoverished and marginalized. While the recent integration of gender considerations into key multilateral climate finance mechanisms, including the Green Climate Fund, are steps in the right direction, gender considerations have yet to be effectively mainstreamed in ongoing climate change programs and activities, and national planning. C.5) EVALUATION DESIGN AND SUGGESTED METHODOLOGIES a) Study Design, Data Collection and Analysis Methods The study is a final performance evaluation of the ISACC and Ready activities in four target countries where ISACC and READY operated: Fiji, Samoa, Federated States of Micronesia (FSM) and Kiribati. The USAID.GOV DRAFT EVALUATION REPORT | 42 evaluation will answer the four evaluation questions based on a qualitative investigation of the results and operations of the activities in all four countries. USAID expects that, while travel from international experts is not anticipated, interviews will be conducted by local experts who will be recruited by the Evaluation Team and trained virtually. Answers to the evaluation questions should be useful or forward looking to inform future USAID projects or follow-on designs as per the purpose of the evaluation described above. Prior to developing data collection instruments, collecting and analyzing data, the Evaluation Team will need to understand highly technical aspects of climate financing and the theory of change of both activities. The contractor must demonstrate that the approaches used are best practice (based on evaluation, assessment, and research literature), that they are chosen because they provide robust answers to each evaluation question, and that they are suitable to the context of the evaluation, type(s) of respondent(s), and constraints due to COVID 19. To show this, the deliverables must clearly articulate the link between each evaluation question, the data used to address it and the analysis plan for these data. This includes providing details on each planned analytical step (e.g., theoretical approach, code book development, interview protocols and data collection instruments, analytical approach for interviews and notes) for qualitative methods. The approach must demonstrate that the proposed evaluation methods and tools minimize biases or evaluator-specific judgements (meaning the same or similar findings and conclusions will be generated if and when another qualified evaluator was to undertake this same evaluation). Data Collection - What, how and when multiple information sources will be collected to answer the evaluation questions be described clearly. For primary data collection, which is anticipated to be qualitative, the proposal must identify specific categories of respondents for each question and a rationale must be provided as to why those respondents were selected. Also, the Evaluation Team must clearly describe the data collection tools that will be developed particularly its appropriateness in answering the evaluation questions. As specified in the deliverables section, the evaluation team must test and validate its data collection tools and thoroughly train enumerators/facilitators. Because of social distancing and compliance to other health protocols, interviews with key participants might be conducted through online or virtual platforms. The evaluation team must adapt the data collection protocols to mitigate undue stress to participants caused by the technology, while preserving the integrity of the approach and reliability of the data collected. The methodology description in the deliverables must be specific and name the data collection method used, explain how it will be operationalized in the context of COVID 19, the reason to use it and the motivation for the choice of respondents. For example, if conducting Focus Group Discussions (which may or may not be appropriate given the respondents and the remote data collection context) what is meant by Focus Group Discussion, what is the motivation for the selection of its participants and why would Focus Group Discussion be more appropriate than a Group Interview, an In-depth Interview, or a Key Informant Interview for a particular data source and a particular evaluation question? The evaluation will be carried out at the national level and in four target countries where ISACC and READY operated: Fiji, Samoa, Federated States of Micronesia (FSM) and Kiribati. The table below list the host government agencies assisted by ISACC and READY in these countries. The data collection and analytical approach must take into consideration the difference between the approaches of the two activities and the difference in implementation across the four countries. It is anticipated that all interviews will be conducted in English without the need for translation. USAID.GOV DRAFT EVALUATION REPORT | 43 FIJI SAMOA FSM KIRIBATI Climate Change Division (Headquarters (HQ): Suva, Fiji) Ministry of Civil Service (HQ: Suva, Fiji) Ministry of Economy (HQ: Suva, Fiji) Fiji Development Bank (HQ: Suva, Fiji) Fiji Chamber of Commerce (HQ: Suva, Fiji) Secretariat of the Pacific Regional Environment Programme (SPREP) (HQ: Apia, Samoa) Ministry of Natural Resources and Environment (HQ: Apia, Samoa) Samoa Tourism Authority (HQ: Apia, Samoa) Chamber of commerce Samoa (HQ: Apia, Samoa) Public Service Commission Samoa (HQ: Apia, Samoa) Ministry of Finance-Aid Coordination Division (HQ: Apia, Samoa) Department of Environment, Climate Change and Emergency Management (HQ: Palikir Station, Pohnpei State) Micronesia Conservation Trust (HQ: Pohnpei) Vital Energy Group (formerly FSM PetroCorp) (HQ in Pohnpei but also operates in Chuuk, Yap, Kosrae Guam, Nauru) FSM National Council on Climate Change Sustainable Development (HQ: unknown – assumed Pohnpei) Office of Te Beretitenti (HQ: Bairiki, Tarawa, Kiribati) Kiribati Climate Finance Unit, Ministry of Finance (HQ: Bairiki, Tarawa, Kiribati) National Economic and Planning Office (HQ: Bairiki, Tarawa, Kiribati) Data Analysis - Why and how multiple information collected will be coded, vetted and analyzed to answer each of the evaluation questions must be described clearly. At the minimum, the data collection will include (i) review of secondary data (internal to ISACC and READY and external sources); (ii) primary data collection of qualitative data. For the analysis of qualitative data, the Evaluation Team must clearly describe how data will be examined, compared or contrasted or how patterns can be identified. They must explicitly describe their approach to rigorous analysis of the qualitative data (for example, content analysis or thematic analysis or a combination of both) and the reason this approach meets USAID's standard of rigor. In addition to proposing a strong theoretical design, the contractor may plan on using standard empirical tools such as software standardly for qualitative analysis (for example Atlas.ti or NVIVO), as applicable. This performance evaluation must adhere to relevant quality standards as provided in the attached USAID Evaluation Policy, as well as international standards for conducting social sciences research, especially the “Do No Harm” principles such as the principles for treatment of human subjects and collection and safeguard of personally identifiable information. The Evaluation Team is required to follow USAID and/or host-country government protocols for research activity such as IRB requirements. The Evaluation Team must ensure recording and transcribing of all the data collected. Qualitative data must be coded, either by hand or using a software, systematically analyzed, and used interpretively and not just descriptively. In addition to common best practices for qualitative research, (see the attached USAID paper). b) Limitations or constraints of the design and methodologies Because of the travel and mobility restrictions brought about by the pandemic, the evaluation methodology has the following constraints and challenges that the evaluation team should mitigate with best practices: USAID.GOV DRAFT EVALUATION REPORT | 44 • The Evaluation Team must implement data collection and data analysis strategies to mitigate respondents’ possible motivation to advance their own interests. • Limited previous research studies on the topic of climate finance in the Pacific region and may restrict the theoretical foundations of the evaluation questions that will be investigated. • Access to some key informants, e.g., at the IPs, could be limited considering the evaluation timeline that is at the end stage of the Activities’ timeline. C.6 EVALUATION TEAM COMPOSITION AND QUALIFICATIONS The Offeror shall propose an evaluation team composed, at minimum of the following: 1. Team Leader, Performance Evaluation Specialist, and Facilitator 2. Technical Specialist(s) as relevant, e.g., climate change project proposal development and stakeholders’ collaboration/partnership building. 3. Local qualitative data collection teams in Fiji, Samoa, Federated States of Micronesia (FSM) and Kiribati. The Team Leader must provide leadership, data collection and data analysis expertise for the evaluation to ensure that deliverables 1-11 meet USAID’s quality expectation and are submitted to and accepted by the COR. The Team Leader is responsible for supervising the work of all members of the team including administrative personnel by providing clear directions and regular feedback on performance. He/She will lead in the preparation and submission of evaluation findings and reports and or learning or knowledge products that will be developed and synthesized. Together, the team must include the following skills: • Proven leadership in the management of similar-sized international donor-funded activities with skills in strategic planning, management, supervision and budgeting. • Extensive professional experience in designing/conducting qualitative performance evaluations • Proven skills and experience to develop methodological tools and instruments for evaluation such as interview protocols or questionnaires, data processing sheets, etc. • Demonstrated and applied expertise in climate financing • Demonstrated and applied expertise in institutional capacity building • A graduate degree in a field of study relevant to qualitative research • A graduate degree in a field of study relevant to natural resource management, environment and climate change, etc. with a focus in climate financing and/or extensive experience in climate/environmental financing program evaluation and/or design/implementation. • Demonstrated expertise in qualitative data collection instruments and protocol design and testing • Demonstrated expertise in qualitative data analysis USAID.GOV DRAFT EVALUATION REPORT | 45 • Demonstrated experience writing evaluation reports and presenting evaluation findings • Demonstrated experience facilitating participatory stakeholder meetings • Excellent oral and written communication skills in English. • Excellent oral and written communication skills in XXX C.7 TASKS, DELIVERABLES AND SCHEDULING a) Scheduling The estimated period of performance of this evaluation will last for six months. The Contractor shall work collaboratively with USAID to propose the detailed work schedules within the six-month timeframe. b) Deliverables Guidelines and Specifications 1. Evaluation design Based on the Evaluation Design Proposal, the final design for the evaluation must be approved by USAID. The required components of the Evaluation Design are the same for the Evaluation Design Proposal and the final design may be exactly the same as proposed, depending on USAID’s input. As with the Evaluation Design Proposal, the Design must also include: References to the relevant literature. The Contractor is required to complete and include the table below in the evaluation design document as parts of the document review. Evaluation Question ISACC Results READY Results • Details the approach to answer the evaluation questions including: o data collection methods (see “data collection” above), o data sources, selection criteria and sampling plan, o a description of data collection instruments, o an analysis plan (see “Data Analysis” above) USAID.GOV DRAFT EVALUATION REPORT | 46 • An Evaluation Design Matrix linking each question to relevant data source, data type and data analysis), • A discussion potential risk and limitations of the proposed approach and mitigation strategies 2. Work plan The Evaluation Team must prepare a detailed work plan which will show how to operationalize the methodologies described in the evaluation design proposal. It must include: • Draft schedule and logistical arrangements. o Evaluation milestones (including, but not limited to, the evaluation deliverables listed in this SOW) o Anticipated schedule of evaluation team data collection efforts. o Dates and arrangements for piloting data collection instruments. • Members of the evaluation team, delineated by roles and responsibilities. • Description of evaluation dissemination products, including the evaluation report, but also, presentation slides, infographics and briefs. • Dissemination plan (designed in collaboration with USAID) 3. Data collection instruments and testing protocol The Evaluation Team must submit a draft protocol of data collection that also includes instruments/tools. The tools must match the Evaluation Design and include questionnaires, forms, and/or guides for data collectors as appropriate. Before data collection begins: • Data collection instruments and protocols must meet USAID technical requirements and be approved by USAID • Data collection instruments must be tested empirically (pilot interview with interviewer and interviewee debrief afterwards, cognitive testing, or other best practices for instrument testing) • Results of data collection tests must be shared with USAID • Revised data collection tools must be approved by USAID • Interviewers must be trained in methods appropriate to the qualitative approach chosen 4. In briefing Prior to the field work the evaluation team shall meet with assigned USAID personnel for an in-briefing and planning session. 5. Initial findings virtual workshop/presentation with USAID (outbrief with USAID) Before beginning to draft the evaluation report (but after the data have been analyzed), the Evaluation Team must provide an oral briefing to the mission that includes: • reviewing findings of their evaluation, and • preparing a “Findings and Recommendation Workshop” coordinated with USAID. USAID.GOV DRAFT EVALUATION REPORT | 47 This initial findings presentation allows the Evaluation Team to demonstrate the robustness of its data collection and analysis and, given that the evaluation work was robust, to get USAID approval to conduct a findings and recommendations workshop and to begin drafting the evaluation report. 6. Findings and recommendation virtual workshop(s) with stakeholders (debriefing with IPs and other stakeholders) The Evaluation Team must organize and deliver a workshop involving key stakeholders (conditional on # 5, and to be determined by USAID) to: • present and explain key evaluation findings, • elicit feedback about the findings and • generate policy and programmatic recommendations based on the findings. Audience for the findings and recommendation workshop(s) will be determined in collaboration with USAID. 7. Draft evaluation report The Evaluation Team must provide a draft of the full report to USAID. The draft must include all the required components of the final report and must be properly copy edited. USAID and the Evaluation Team may agree to submit and review the draft in section if they jointly prefer an iterative process (for example, the Evaluation Purpose and Questions, the Project Background, and the Methods sections might be cleared first, as they should be easier to draft.) 8. Final evaluation report The final report should be no more than 30 pages and should be written for a non-evaluation audience. All technical details about the methodology should be included in the annexes. A comment log should list all the comments submitted by USAID on the draft and explain how they were addressed in the final report. 9. Draft dissemination materials The Evaluation Team must provide a draft of the briefing materials to USAID. The draft must include all the required components of the final report and must be properly copy edited. 10. Final dissemination materials The Evaluation Team must submit briefing materials to the evaluation COR in the form of a 5-10-page summary (and an accompanying set of PowerPoint slides) of key findings, conclusions and recommendations. These shall be finalized by the evaluators for approval by the evaluation COR. 11. Learning Event(s) for Evaluation Utilization. The Evaluation Team must organize at least two (2) learning events (with a maximum eight (8) hours duration for each event) for key stakeholders in the four target countries. The learning event aims to disseminate evaluation findings, explore good practices related including approaches to advance the sustainability of interventions started in ISACC and READY. c) Other Requirements and Guidance Both the draft and final evaluation reports must meet the criteria to ensure the quality of the evaluation report as outlined in the Annex 1 of the USAID Evaluation Policy as well as other ethical considerations, especially the “Do No Harm” principles. At minimum, the report shall include the following: i. Covers with correct branding and marking USAID.GOV DRAFT EVALUATION REPORT | 48 ii. Table of content iii. Acronyms list iv. Executive summary of no more than three pages v. A main report with findings, conclusions, and recommendations not to exceed 25 pages, inclusive of tables, graphs, and/or other graphics as needed. vi. Annexes including a copy of this statement of work, tools used to collect data to answer the evaluation questions including informed-consent forms, a list of informants/ respondents appropriately redacted personally identifiable information; and a signed statement from each evaluation team member attesting to a lack of conflict of interest or describing any existing conflict of interest. All deliverables that are in written format must be in plain39F 59, grammatically correct English language; be submitted in appropriate electronic format (i.e., Microsoft Word, Excel, Power Point Presentation, and PDF); and meet all the requirements. All raw data, quantitative and qualitative, collected by the Evaluation Team must be provided in machine￾readable, non-proprietary formats. All raw dataset produced for this evaluation, accompanied by a codebook, must be submitted to the COR The data shall be organized and fully documented for use by those not fully familiar with this study. USAID will retain ownership of all datasets developed and deliverables produced under this contract. The Contractor shall not use any data or information obtained through this contract, in full or in part, for purposes other than to develop the Report for USAID. The Contractor is responsible for making all logistical support in-country according to the work plan approved by USAID. A representative(s) of USAID may participate in the meetings with government officials and/or field data collections as appropriate, considering that USAID presence will not affect the evaluation quality. C.9 LIST OF ANNEXES I. SOW/PD of the Project/Activity being evaluated II. Workplan III. MEL Plan IV. Progress reports V. Recommended related literatures 59 Plain language (also called Plain English) is communication that your audience can understand the first time they read or hear it. There are many writing techniques that can help you achieve this goal. Among the most common are: logical organization with the reader in mind; active voice; short sentences; no jargons; easy-to-read design features. No one technique defines plain language. Rather, plain language is defined by results — it is easy to read, understand, and use. See more details at http://www.plainlanguage.gov/ USAID.GOV DRAFT EVALUATION REPORT | 49 ANNEX B: METHODOLOGY APPROACH Mixed-methods. The evaluation utilized a mixed-methods design. The evaluation collected qualitative primary data collected via key informant interviews (KIIs), analyzed secondary quantitative data (primarily Climate Ready and ISACC monitoring data), and included a desk review of program documents and relevant independent research on climate finance in PICs. The scope of the evaluation was at the national level in four focus countries selected by USAID/Pacific Islands: Fiji, FSM, Kiribati, and Samoa. USAID selected these countries to reflect the range of experiences of ISACC and Climate Ready implementation. Utilization-focused. The study adopted a utilization-focused approach, based on the principle that evaluations should be judged on usefulness and ability to engage with intended audiences. Thus, the evaluation was designed and implemented in ways that supported audience participation in the process itself, as well as utilization of findings to inform strategy. Specific actions taken to encourage evaluation use are outlined below: • USAID/Pacific Islands and IP meetings: During the design phase of the evaluation, the evaluation team met with both USAID/Pacific Islands and the Climate Ready and ISACC IPs to discuss overall objectives, obtain feedback on the evaluation design, and review logistics. The evaluation team continued to meet regularly with USAID/Pacific Islands to discuss evaluation progress and troubleshoot any issues encountered. • Virtual workshops: Following data collection and analysis, the evaluation team held two workshops with primary and secondary audiences (e.g., USAID/Pacific Islands, IPs, national stakeholders, etc.). The first workshop focused on validating the evaluation findings and conclusions, while the second included exercises to co-create evaluation recommendations. • Learning events: After publication of the final report, the team will hold learning events to disseminate evaluation learnings. See Annex C for the evaluation timeline and Annex D for evaluation team’s disclosures of any conflicts of interest. EVALUATION QUESTIONS The evaluation focused on four specific questions in the development of findings, conclusions, and recommendations. These questions and corresponding sub-questions are outlined in the table below: EVALUATION QUESTIONS Enabling Environment 1. How did the Activities impact the enabling environment required to access climate financing (including governmental policies, organizational structures, standard operating procedures [SOPs], plans, frameworks, legislation)? a. What were the most significant changes to the enabling environment, and how did these changes come about? b. To what extent did changes to the enabling environment support gender equality and social inclusion (GESI)? Build Capacity USAID.GOV DRAFT EVALUATION REPORT | 50 2. How did the Activities contribute to building the capacity of government to plan, design, implement (manage), monitor, and evaluate climate change adaptation projects that will be funded through international climate finance instruments? a. What were the most significant changes to government capacity, and how did these changes come about? b. To what extent did capacity building efforts support GESI (beyond including women in capacity building activities)? Support Country-lead Implementation 3. To what extent did new USAID climate funding support countries to implement country-owned climate change actions? Engage the Private Sector 4. What were key successes and challenges faced by these Activities in engaging the private sector and climate financial institutions? DATA COLLECTION Fieldwork and sampling. The evaluation team collected data from October 2021—January 2022 with respondents in Fiji, Samoa, FSM, Kiribati, and other regional and international locations (e.g., Australia, the U.S., etc.). Key stakeholder groups consulted included: 1) IPs (SPC, PIFS and SPREP for ISACC and DT Global for Climate Ready) 2) Government officials (e.g., Ministry of Finance, Ministry of the Economy, etc.) and those in nationally-affiliated organizations (e.g., Fiji Development Bank) in the four evaluation focus countries 3) Civil society organizations (CSOs), non-governmental organizations (NGOs), and regional or local groups, including the private sector (e.g., Micronesia Conservation Trust, Fiji Business Disaster Resilience Council, etc.) 4) USAID and other donors (e.g., Germany Agency for International Cooperation [GIZ]) 5) Intergovernmental institutions (e.g., Green Climate Fund [GCF]). To construct the data collection sample, the evaluation team developed a list of potential respondents from the desk review and in consultation with USAID/Pacific Islands and IPs. From this list, the team selected a purposive sample, prioritizing participants according to the following criteria: • Role: The evaluation team aimed to speak to a variety of respondent types, with particular emphasis placed on government and nationally-affiliated organizations, as these were the intended target groups for many Climate Ready and ISACC interventions. • Engagement: The team sought to prioritize respondents familiar with multiple interventions representing both ISACC and Climate Ready, and/or could speak to multiple evaluation questions. • Location: The team attempted to construct a sample with a relatively even mix of respondents in Fiji, Samoa, FSM, and Kiribati. • Gender: The team aimed to have a relatively even mix of male and female respondents. Due to restrictions and safety protocols relating to the global novel coronavirus of 2019 (COVID-19) pandemic, the team adapted primary data collection to require as little travel as possible. All interviews by internationally-based team members were held via remote platforms (e.g., Microsoft Teams or Zoom). Local data collection leads based in Fiji, Kiribati, and Samoa collected data either in-person or remotely in USAID.GOV DRAFT EVALUATION REPORT | 51 the four focus countries, depending on individual country restrictions and the preferences of the interviewer and respondent. In cases of in-person data collection, the team followed safety and security processes that were reviewed and approved by SI’s Institutional Review Board (IRB). DESK REVIEW The evaluation team conducted a comprehensive desk review of key documents that provided background knowledge on the processes and decisions that went into designing Climate Ready and ISACC, as well as information on implementation progress and results. The team also reviewed external documents that gave further context on climate adaptation and resilience within the four focus countries for the evaluation. A bibliography of documents reviewed is included in Annex E. SECONDARY QUANTITATIVE DATA The evaluation team examined monitoring information from the databases of Climate Ready and ISACC to establish the level of support provided through the Activities as well as the progress made towards key government support and capacity development objectives. The team used its analysis of indicator trends for Climate Ready and ISACC to assess information gaps and highlight issues to cover during data collection. The team disaggregated the data by geography and intervention type, where available. KEY INFORMANT INTERVIEWS The evaluation team spoke with key informants using semi-structured interview protocols tailored by respondent type (see Annex F for list of evaluation respondents and Annex G for interview protocols). The team used KIIs to gather information on stakeholders’ experiences with Climate Ready and ISACC, as well as on broader issues relating to accessing climate finance and implementing climate actions in the region. KIIs were audio recorded with consent from the participant and for use by the evaluation team for analysis. The evaluation team stored all KII recordings and transcripts in a secure, password-protected site only accessible to the evaluation team. In total, the team conducted 46 KIIs, including 40 individual interviews and six group interviews, for a total of 52 people consulted for the evaluation. The table below details the number of key informants targeted and interviewed by respondent type; information on respondents’ gender and location is also provided. While the evaluation team aimed to reach approximately even numbers of KII respondents across countries, this was not always feasible due to the location of participants or difficulties obtaining contact with certain groups (see further discussion under Limitations below). For example, a greater number of USAID/Pacific Islands and IP staff are located in Fiji, increasing the sample numbers in that country; in addition, fewer respondents from Kiribati replied to evaluation team outreach, lowering these numbers. KII RESPONDENT CATEGORIES AND DEMOGRAPHICS Gender Location Respondent Type Male Female Fiji Samoa Kiribati FSM Other Total National Government 16 Government officials 3 7 3 3 2 2 0 10 Nationally affiliated organizations 1 5 1 3 0 2 0 6 Civil Society 12 Regional organizations 3 3 3 2 0 0 1 6 Other local organizations/NGOs 4 2 2 1 0 3 0 6 IPs 18 Climate Ready and ISACC 9 9 11 0 0 5 2 18 USAID/Other donors 3 USAID.GOV DRAFT EVALUATION REPORT | 52 USAID 0 2 1 0 0 0 1 2 Other donors 0 1 1 0 0 0 0 1 Intergovernmental 3 Climate financial institutions 0 1 0 0 0 0 1 1 Other intergovernmental institutions 1 1 0 0 0 1 1 2 Grand Total 21 31 22 9 2 13 6 52 ANALYSIS Approach. The evaluation team employed various data analysis techniques to support the development of evaluation Findings, Conclusions, and Recommendations (FCR). Throughout data collection, internationally-based team members met regularly with local data collection leads to review notes and discuss emerging findings. The team then documented themes under each of the evaluation questions in an FCR matrix to ensure that all questions were being adequately addressed. When possible, preliminary findings by demographic factor (participant type, gender, etc.) were identified, enabling the team to determine any gaps where additional clarification, probing, or analysis were needed. The team used the FCR matrix to develop a structured codebook for qualitative data coding and analysis. Qualitative coding and triangulation. Following data collection, KII recordings were transcribed for coding using a Microsoft Excel-based tally sheet. The team then used the codebook to apply content and thematic analysis to KII responses. To ensure inter-coder reliability, all coders participated in training on the codebook and proper code applications. The team then conducted a blind coding exercise for several KIIs, which were then reviewed by the lead coder for alignment and further training before full coding took place. The team coded KIIs in batches, with the lead coder reviewing coded KIIs and providing feedback to coders along with any codebook adjustments to ensure consistency across team members. To capture any divergences, coders disaggregated qualitative data by respondent type (national government/nationally affiliated, civil society/regional organizations, IPs, USAID and other donors, and international/intergovernmental institutions), location (Fiji, Kiribati, FSM, Samoa, Other), and gender (male, female, group interview). The evaluation team then took the interview data, desk review information, and quantitative monitoring data to employ data triangulation, a strategy in which data are first analyzed independently, in parallel, and then cross-verified for validity. The team used the themes generated through this approach to identify the key findings highlighted in this report. Stakeholder workshops. After the coding process was completed, the evaluation team held two virtual workshops. The objective of the first workshop was to validate evaluation results and included USAID/Pacific Islands, IPs, and other KII participants. The evaluation team then facilitated a second workshop with USAID/Pacific Islands and IPs with the objective to co-develop evaluation recommendations. The team incorporated additional insights gathered from the workshops into this evaluation report. LIMITATIONS The table below describes the major limitations and mitigation strategies for the evaluation: EVALUATION LIMITATIONS AND MITIGATION STRATEGIES LIMITATIONS MITIGATION STRATEGY Certain respondent groups (particularly government) were difficult to reach, in part due to: • The evaluation team used an official letter from USAID/Pacific Islands to enhance cooperation from respondents. In addition to USAID.GOV DRAFT EVALUATION REPORT | 53 • Data collection for the evaluation included respondents who were already consulted for internal ISACC and Climate Ready M&E activities. Due to respondent fatigue, these individuals may have been less inclined to take part in interviews for the evaluation. • Data collection took place during the 2021 United National Climate Change Conference (COP 26), which several respondents attended or worked on; as a result, some were unavailable to participate in interviews. • Since the start of the COVID-19 pandemic, it has been more difficult to contact and confirm interviews with some respondents when these are held on a remote platform. As a result, sample sizes were not necessarily even for all groups (e.g., there were fewer respondents from Kiribati versus the other three focus countries). In addition, given the design of the Activities and the selected evaluation questions, not all data collection respondents could provide information about each area of inquiry (e.g., participants who had taken part in trainings could not necessarily discuss private sector engagement). Thus, for some evaluation questions and respondent groups, the smaller sample numbers limited the evaluation team’s ability to provide meaningful disaggregation of KII data. this letter, respondents received an email explaining the data collection process, the tools, and a consent form. • The evaluation team closely consulted with USAID/Pacific Islands via email and biweekly check-in meetings on any anticipated challenges related to the sample size and timeline for the evaluation. • The evaluation team avoided disaggregation of KII data by country in particular, especially for Kiribati where the respondent sample size was small. Instead, when discussing findings the team drew from the desk review and KIIs to provide country-specific examples to highlight the various experiences of ISACC and Climate Ready implementation (e.g., Climate Ready support for the Kiribati Finance Division to include appropriate GESI components on the Strategic Framework and Country Program for the Green Climate Fund (GCF) as an example of strengthening policies, plans, frameworks, etc.). The majority of data collection was conducted remotely due to COVID-19 restrictions, which can impact nonverbal and other communication during interviews. • Training of KII facilitators included sessions on use of remote platforms and practice conducting remote interviews. • Employing data triangulation helped ensure that specific interviews that may have been impacted by a remote platform would not skew study results. The majority of data collection with respondents in PICs was conducted by team members based in those countries, which allowed for use of phones (a more familiar communication platform) and time zone alignment. The evaluation team collected primary data in four PICs (ISACC was implemented in eight PICs and Climate Ready in 11). Specific operational contexts vary across countries, and those selected for primary data collection may not necessarily have reflected the overall programmatic experience (i.e., there may have been PICs included in data collection where implementation was more or less successful than others). • Within the four countries where primary data collection took place, the evaluation team aimed for a sample with a variety of respondent types to reflect as accurately as possible the range of experiences of Climate Ready and ISACC. • The evaluation team looked at document review and secondary quantitative data disaggregated by site, when available, to assess similarities and differences between countries. USAID.GOV DRAFT EVALUATION REPORT | 54 ANNEX C: TIMELINE *Note: The timeline for Phase 5 tasks remains contingent on USAID approval and sequencing of the proposed learning events. USAID.GOV DRAFT EVALUATION REPORT | 55 ANNEX D: CONFLICT-OF-INTEREST DISCLOSURES USAID.GOV DRAFT EVALUATION REPORT | 56 USAID.GOV DRAFT EVALUATION REPORT | 57 USAID.GOV DRAFT EVALUATION REPORT | 58 USAID.GOV DRAFT EVALUATION REPORT | 59 USAID.GOV DRAFT EVALUATION REPORT | 60 USAID.GOV DRAFT EVALUATION REPORT | 61 USAID.GOV DRAFT EVALUATION REPORT | 62 USAID.GOV DRAFT EVALUATION REPORT | 63 ANNEX E: DOCUMENTS REVIEWED AECOM International Development. 2017. “USAID Climate Ready Annual Report FY 17- November 2016 to September 2017.” October 2017. https://pdf.usaid.gov/pdf_docs/PA00Z4HC.pdf AECOM International Development. 2018. “USAID Climate Ready Annual Report FY18- November 2017 to September 2018.” October 2018. https://pdf.usaid.gov/pdf_docs/PA00Z4HD.pdf AECOM International Development. 2018. “USAID Climate Ready Quarterly Progress Report- October to December 2017.” January 2018. AECOM International Development. 2018. “USAID Climate Ready Quarterly Progress Report- January to March 2018.” April 2018. AECOM International Development. 2018. “USAID Climate Ready Quarterly Progress Report- April to JUNE 2018.” July 2018. https://pdf.usaid.gov/pdf_docs/PA00Z59V.pdf AECOM International Development. 2019. “USAID Climate Ready Quarterly Progress Report- October to December 2018.” January 2019. https://pdf.usaid.gov/pdf_docs/PA00Z59B.pdf AECOM International Development. 2019. “USAID Climate Ready Quarterly Progress Report- January to March 2019.” April 2019. https://pdf.usaid.gov/pdf_docs/PA00Z59D.pdf AECOM International Development. 2019. “USAID Climate Ready Quarterly Progress Report- April to June 2019.” July 2019. Asian Development Bank. 2020. “Enhancing Gender Equality and Social Inclusion Results in South Asia Developing Member Countries.” September 4, 2020. https://www.adb.org/projects/54305- 001/main Bettinger, Keith, and Ryan Longman. 2020. “USAID Climate/Ready Pacific Island Country Climate Factsheets.” DT Global. Buadromo, Vuki, and Amelia Caucau. 2017. “USAID ISACC Quarterly Report (October - December 2016).” USAID. Buadromo, Vuki, and Amelia Caucau. 2017. “USAID ISACC Quarterly Report (January - March 2017).” USAID. Buadromo, Vuki, and Amelia Caucau. 2017. “USAID ISACC Quarterly Report (April - June 2017).” USAID. Buadromo, Vuki, and Amelia Caucau. 2018. “USAID ISACC Quarterly Report (October - December 2017).” USAID. Buadromo, Vuki, and Amelia Caucau. 2018. “USAID ISACC Quarterly Report (January - March 2018).” USAID. Buadromo, Vuki, and Amelia Caucau. 2018. “USAID ISACC Quarterly Report (April - June 2018).” USAID. Buadromo, Vuki, and Amelia Caucau. 2019. “USAID ISACC Quarterly Report (October - December USAID.GOV DRAFT EVALUATION REPORT | 64 2018).” USAID. Buadromo, Vuki, and Amelia Caucau. 2019. “USAID ISACC Quarterly Report (January - March 2019).” USAID. Buadromo, Vuki, and Amelia Caucau. 2019. “USAID ISACC Quarterly Report (April - June 2019).” USAID. Buadromo, Vuki, and Amelia Caucau. 2019. “USAID ISACC Quarterly Report (July - September 2019).” USAID. Buadromo, Vuki, and Amelia Caucau. 2020. “USAID ISACC- North Pacific Food Security Governance Initiative Quarterly Report (July - September 2020).” USAID. Buadromo, Vuki, Buggy, Lisa, and Amelia Caucau. 2020. “USAID ISACC Quarterly Report (October – December 2019).” USAID. Buadromo, Vuki, Buggy, Lisa, and Amelia Caucau. 2020. “USAID ISACC Quarterly Report (January – March 2020).” USAID. Buggy, Lisa, and Amelia Caucau. 2020. “Interim Final Report for ISACC Climate Change Financing Component.” The Pacific Community (SPC). Charles Darwin University, 2019: Evaluation of SPC’s Capacity Building. Department of Environment, Climate Change & Emergency Management (DECEM), FSM. 2021. “FSM National Climate Change Data Portal.” July 9, 2021. https://fsm-data.sprep.org/dataset/fsm￾national-climate change-data-portal. DT Global. 2018. “USAID Climate Ready Annual Work Plan FY 19.” August 2018. DT Global. 2019. “USAID Climate Ready Annual Report FY19 November 2018 to September 2019.” October 2019. https://pdf.usaid.gov/pdf_docs/PA00Z4HF.pdf DT Global. 2019. “USAID Climate Ready Annual Work Plan FY 20.”. August 2019. DT Global. 2020. “USAID Climate Ready Annual Work Plan FY 21.” August 2020. DT Global. 2020. “USAID Climate Ready Quarterly Progress Report- October to December 2019.” January 2020. https://pdf.usaid.gov/pdf_docs/PA00Z57F.pdf DT Global. 2020. “USAID Climate Ready Quarterly Progress Report- January to March 2020.” April 2020. https://pdf.usaid.gov/pdf_docs/PA00Z57G.pdf DT Global. 2020. “USAID Climate Ready Quarterly Progress Report- April to June 2020.” July 2020. https://pdf.usaid.gov/pdf_docs/PA00Z57H.pdf DT Global. 2020. “USAID Climate Ready Annual Report FY20 November 2019 to September 2020.” October 2020. https://pdf.usaid.gov/pdf_docs/PA00Z4HG.pdf DT Global. 2021. “National Consultation Reports, FSM. 2021. DT Global. 2021. “National Consultation Reports, Kiribati. 2021. DT Global. 2021. “National Consultation Reports, Samoa. 2021. USAID.GOV DRAFT EVALUATION REPORT | 65 DT Global. 2021. “USAID Climate Ready Quarterly Progress Report- October to December 2020.” January 2021. https://pdf.usaid.gov/pdf_docs/PA00Z52T.pdf DT Global. 2021. “USAID Climate Ready Quarterly Progress Report- January to March 2021.” April 2021. DT Global. 2021. “USAID Climate Ready Quarterly Progress Report- April to June 2021.” July 2021. https://pdf.usaid.gov/pdf_docs/PA00Z534.pdf Fiji Climate Change & National Designated Authority Portal. 2022. https://fijiclimatechangeportal.gov.fj/. Framework for Resilient Development in the Pacific (FRDP). “An Integrated Approach to Address Climate Change and Disaster Risk Management (FDRP).” 2017. Framework for Resilient Development in the Pacific (FRDP). “Monitoring and Evaluation Strategy Pacific Resilience Partnership.” 2020. GIZ, 2017: Project evaluation: brief report- Indonesia: Policy Advice for Environment and Climate Change (PARKLIM) Global Environment Facility. 2019. “Second National Communication to the United Nations Framework Convention on Climate Change.” https://www.fsmstatistics.fm/wp-content/uploads/2019/10/2- 2nd-National-Communiation-to-the-UNFCCC.pdf. Global Resilience Partnership, 2020: Case study: learning from failure to accelerate success in resilience building initiatives Government of Kiribati. 2019. “Kiribati Joint Implementation Plan for Climate Change and Disaster Risk Management: 2019–2028.” https://www4.unfccc.int/sites/NAPC/Documents/Parties/Kiribati￾Joint-Implementation-Plan-for-Climate change-and-Disaster-Risk-Management-2019-2028.pdf Government of Samoa Disaster Management Office. 2019. “Samoa National Action Plan for Disaster Risk Management 2017-2021” 2019. https://www.mnre.gov.ws/wp-content/uploads/2017/08/Samoa￾national-action-plan-for-DRM-2017-2021-final-web.pdf. Government of the Republic Fiji. 2020. “Fijian Government Unveils COVID-19 Concessional Loan Packages for Fijian Micro, Small and Medium Enterprises.” https://www.fiji.gov.fj/getattachment/bba34afb-b4f6-4e6b-90f6-b0392ae4a18f/CONCESSIONAL￾LOAN-PACKAGES-FOR-MSMEs.aspx. ICFI, 2020: Independent evaluation of the relevance and effectiveness of the GCF’s investments in SIDS. Institutional Strengthening in Pacific Island Countries to Adapt to Climate Change (ISACC) Project Highlights 2016—2017.” 2020. Ministry of Natural Resources and Environment, Samoa. 2022. “National Climate Change Portal: Samoa.” February 15, 2022. https://samoa-data.sprep.org/dataset/national-climate change-portal-samoa. Morioka, Kate, et al., 2020: “Applying Information for National Adaptation Planning and Decision Making: Present and Future Practices in the Pacific Islands.” Regional Environmental Change, vol. 20, no. 4, 20 Nov. 2020. National Legislative Bodies/National Authorities, Fiji. 2018. Planned Relocation Guidelines: A Framework to Undertake Climate Change Related Relocation. December 2018. https://www.refworld.org/docid/5c3c92204.html. USAID.GOV DRAFT EVALUATION REPORT | 66 Nyuykonge, Charles. 2020. “End of Project Evaluation for the Institutional Strengthening for Pacific Islands to Adapt to Climate Change (ISACC) Project & Design Support for a New Climate Change Adaptation Programme.” The Pacific Community (SPC) ISACC Project. Social Impact, 2017. Secretariat of the Pacific Regional Environment Programme. 2013. “Kiribati to Have Joint National Action Plan for Climate Change and Disaster Risk Management | Pacific Environment.” March 18, 2013. https://www.sprep.org/news/kiribati-have-joint-national-action-plan-climate change-and-disaster￾risk-management. Secretariat of the Pacific Regional Environment Programme. 2022. “Pacific Climate Change Portal.” February 15, 2022. https://kiribati-data.sprep.org/dataset/pacific-climate change-portal. The Pacific Community (SPC). Regional Climate Change & Disaster Risk Finance Meetings and Media Masterclass Final Report.” 2019. The Pacific Community (SPC). “Institutional Strengthening in Pacific Island Countries to Adapt to Climate Change (ISACC) Project Brief.” n.d. The Pacific Community (SPC). “SPC Results Report on ISACC Project Activities for 2017.” 2018. UNDP, 2017: Japan-Caribbean Climate Change Partnership – Midterm Evaluation Report. UNDP, 2019: Governance for Resilient Development in Pacific (Gov4Res) – UNDP Project Design Document University of the South Pacific. 2021. “USP Commences with Certificate 4 in Disaster Risk Management Programme.” September 27, 2021. https://www.usp.ac.fj/news/usp-commences-with-certificate-4- in-disaster-risk-management-programme/. USAID Climate Ready Work Plan FY18 – Priority Sub-Activities.” 2017. https://pdf.usaid.gov/pdf_docs/PA00Z5VN.pdf. USAID ISACC. 2017. “PPR Sheet FY 2017.” USAID ISACC. 2018. “PPR Sheet FY 2018.” USAID ISACC. 2019. “PPR Sheet FY 2019.” USAID ISACC. 2020. “PPR Indicator and Narratives.” USAID ISACC. 2020. “PPR Sheet FY 2020.” The Pacific Community (SPC). USAID ISACC. 2020. “USAID ISACC COVID-19 Activity Risk Mitigation Plan.” USAID, 2010: Asia-Pacific regional climate change adaptation assessment final report: findings and Recommendations. USAID, 2016: Global Climate Change Institutional Capacity Assessment Facilitator’s Guide USAID, 2017: Final performance evaluation of the USAID Regional Climate Change Program (RCCP) USAID, 2019: Local governance and private sector opportunities for climate adaptation and sustainable USAID.GOV DRAFT EVALUATION REPORT | 67 livelihoods and biodiversity in Madagascar. Climate Economic Analysis for Development, Investment and Resilience (CEADIR) USAID. 2021. “Administrator Samantha Power on a New Vision for Global Development.” November 4, 2021. https://www.usaid.gov/news-information/speeches/nov-4-2021-administrator-samantha￾power-new-vision-global-development. USAID 2021. “USAID Climate Strategy 2022–2030: Draft for Public Review.” November 2021. https://www.usaid.gov/sites/default/files/documents/USAID-Climate change-Strategy-2022-2030- Offical-Draft.pdf. USAID/Institutional Strengthening for Pacific Island Countries to Adapt to Climate Change (ISACC) Project Monitoring & Evaluation Plan.” 2016. USAID. USAID.GOV DRAFT EVALUATION REPORT | 68 ANNEX F: RESPONDENTS CONSULTED The evaluation team interviewed respondents from the following organizations: • Climate Ready • CSIRO • Fiji Business Disaster Resilience Council (FBDRC) • Fiji Commerce and Employers Federation (FCEF) • Fiji Development Bank • Fiji Ministry of Civil Service • Fiji Ministry of the Economy / Climate Change Division • FSM Department of Environment, Climate Change and Emergency Management (DECEM) • FSM Development Bank • FSM Ministry of Finance and Administration • FSM National Council on Climate Change Sustainable Development (CCCSD) • GIZ • Green Climate Fund • Kiribati Ministry of Finance, Climate Finance Division • Kiribati National Expert Group (KNEG) • Kiribati Office of Te Beretitenti • Micronesia Conservation Trust • Micronesia Productions • Pacific Islands Private Sector Organization (PIPSO) • Pacific TAFE USP • PIFS • Samoa Chamber of Commerce • Samoa Ministry of Finance • Samoa Ministry of Natural Resources and Environment • Samoa Public Service Commission • Samoa Tourism Authority • SPC • SPREP • UNDP GEF Small Grants Program • USAID • Vital Energy Group USAID.GOV DRAFT EVALUATION REPORT | 69 ANNEX G: DATA COLLECTION INSTRUMENTS AND INFORMED CONSENT INTERVIEW PARTICIPATION CONSENT FORM The evaluation team from Social Impact, a United States-based research organization working under contract with the United States Agency for International Development (USAID), would like to request your consent to participate in an interview as part of the evaluation of the Institutional Strengthening in Pacific Island Countries to Adapt to Climate Change (ISACC) and the Pacific Climate Ready (Climate Ready) Projects. The purpose of this joint evaluation is to better understand how Climate Ready and ISACC have contributed to helping Pacific Island Countries advance the disaster management capacity and ability to mitigate the adverse impacts of climate change. You have been selected for involvement in this evaluation based on your knowledge of the climate finance sector and the work performed by ISACC and Climate Ready. In total, our team will speak to about 50 other people across multiple Pacific Island countries with similar knowledge and expertise. Themes the evaluation will explore include supporting the enabling environment to access and utilize climate finance, building the capacity of governments to implement climate change adaptation projects, engaging the private sector and climate finance institutions, and inclusion of gender and marginalized groups in climate actions. Our discussion with you is expected to take approximately one hour, though if you wish to stop the interview at any time or not answer specific interview questions, you should feel free to do so without penalty. Your involvement in the interview is completely voluntary and you are under no obligation to participate. Your decision to take part in the interview, or to leave at any point, will not impact your involvement with ISACC, Climate Ready, or USAID. There is no payment or direct benefit to you for participating in this interview, and we do not anticipate any major risks. If you agree to participate in the interview, we want to assure you that all personally identifiable responses you provide will be kept confidential to the Social Impact evaluation team. For reports we write for the evaluation, your organization may be listed in an annex to the report, but your name will not appear in any reporting. Your answers will be combined with those of other people and presented in a summary format so that none of the interview answers you give can be linked back to you.  At the start of the interview, the evaluation team will ask you for permission to audio record the conversation in order to capture your responses more accurately. Though the recordings themselves will not be shared with anyone outside of the evaluation team, de￾identified transcripts of the interview recordings that do not contain any personally identifying information will be provided to USAID, which may choose to publish the de-identified transcripts in a public or semi-public database. You may ask for the team not to record while still participating in the interview. If you have any concerns, you may contact USAID representative XXXXXX at xxxx@usaid.gov, Evaluation Team Member XXXXX at xxxx@socialipact.com, or the Social Impacts Institutional Review Board at xxx@socialimpact.com USAID.GOV DRAFT EVALUATION REPORT | 70 INTRODUCTION SCRIPT AND CONSENT TO RECORD Thank you for taking time from your day to meet with (me/us). My name is [your name]. I am part of the Social Impact team that USAID contracted with to conduct an evaluation of two of its programs: Institutional Strengthening in Pacific Island Countries to Adapt to Climate Change (ISACC) and Climate Ready (Ready). The purpose of this evaluation is to get a better understanding of how ISACC and Climate Ready contributed to meeting USAID/Pacific Islands' long-term goal of helping Pacific Island Countries (PICs) achieve self-reliance especially relating to climate change adaptation and disaster response. USAID is interested in better understanding which activities were most successful in strengthening the capacity of PICs to access and utilize climate financing and implement country-led climate actions. This information will help USAID and other interested stakeholders plan for the future. For this evaluation, we are focusing on activities in FSM, Fiji, Kiribati, and Samoa. Your opinions are important to us. To be clear, we are not evaluating you or your (official/Ministry/organization/group) in any way. Your opinions are important to us. Our reporting will be confidential and at a summary level. Our conversation will last about 60 minutes. (I/we) will take notes of our discussion. [Internal NOTE: Explain if needed]: This discussion will cover topics like: ● Basic information on your role and that of your (organization) ● Information on (ISACC/Ready) activities, especially in the four targeted countries, including key successes and challenges, increasing inclusivity of marginalized groups, and engaging the private sector. ● Coordination with other international and regional organizations relating to climate change planning ● Your opinions on future needs and priorities [Internal NOTE: (Optional) Read consent language if consent was not provided beforehand via email]. CONSENT TO PARTICIPATE: Before beginning our discussion, can you please confirm that you received and have read the interview participation consent form that was sent to you by email? I am happy to clarify any questions you might have about the information in this form. If you do not have any further questions, can you confirm that we have your permission to proceed with the interview? CONSENT TO RECORD: (I/we) would also like to record this discussion to help ensure we accurately reflect your feedback. Though the recordings themselves will not be shared with anyone outside of the evaluation team, de-identified transcripts of the interview recordings that do not contain any personally identifying information will be provided to USAID. You may ask for the team not to record while still participating in the interview. Do you have any questions? Do (I/we) have your permission to begin recording? [Internal NOTE: Record the respondent’s “Yes” or “No” answer in the table below under “Consent to Record”] USAID.GOV DRAFT EVALUATION REPORT | 71 INTERVIEW GUIDE A: IMPLEMENTING PARTNERS Interview ID number: Date: Name of Interviewer: Location of Interview (Country, City or Town): [Internal Instructions: Conduct introductions and ask for consent to record. Ask the interview participant to provide the below information] Name of Respondent: Organization: Position: Nationality: Sex: Consent to record: Yes/No [circle one] CORE QUESTIONS **Interviewer Note: Not all questions should be asked of all respondents. Tailor questions and ensure that interview can be completed within one hour. PROBES *Interviewer Note: Only use if issue is not addressed in response to core question or not already known EQ Relevant evaluation question Informant’s role 1.     What can you tell me about your involvement with (ISACC/Ready)? What activities were you directly involved with as part of the project? [Internal: Confirm focus on our 4 targeted countries and climate finance activities as needed] a. What is your role? All b. How long have you been involved? c. [If needed:] How was it decided which activities to pursue (for this workstream/country); i.e., were they ad hoc or planned requests? d. [Internal: Clarify any key details relating to activities they were involved with not already clear from the training and background materials provided by SI or if there is important relevant documentation that has not yet been shared, e.g., case studies, activity highlights] Key achievements and challenges 2.       USAID is interested in understanding more about what went well with (ISACC/Ready) as well as what was less effective. I am going to ask about some different areas in turn. USAID.GOV DRAFT EVALUATION REPORT | 72 i. First, what was (ISACC/Ready) seeking to achieve [in Country Name] related to the development of governmental policies, organizational structures, plans, frameworks, legislation, etc. Did the achievements match what was planned? Why or why not? (i.e., to what extent have (ISACC/Ready) objectives in this area of work been met?) [Internal: Ready] a. What facilitated and/or hindered achievements in this area? b. What role did Ready play relative to other programs or influencing factors? • Please provide some examples (if any) of sustainable successes in this area (i.e., lasting changes to policies or organizational structures). c. Which specific activities were successful in contributing to these achievements [seek specific examples in targeted countries]. What led you to this conclusion? d. Of the achievements you described, which, if any, would you say are the most important or relevant [to improving the enabling environment for accessing climate financing]? Why? e. [If needed, prompt using examples from documentation that were not mentioned] [Internal: ISACC] a. What facilitated and/or hindered achievements in this area? b. What role did ISACC play relative to other programs or influencing factors? c. Which specific activities were successful in contributing to these achievements [seek specific examples in targeted countries]. What led you to this conclusion? • Please provide some examples (if any) of sustainable successes in this area (i.e., lasting changes to policies or organizational structures). • [If regional conferences, events, and workshops are mentioned]: Besides production of formal outputs (conference presentations, drafted documents, etc.), what in your opinion are the main results of regional conferences, workshops, and events? How, if at all, do these activities contribute to improving the enabling environment for accessing climate finance? • In your opinion, was the level and type of support (direct funding, technical assistance, etc.) provided by ISACC to governments well matched to contribute to achievements in this area? Why or why not? Does this differ significantly by country? d. Of the achievements you described, which, if any, would you say are the most important or relevant [to improving the enabling environment for accessing climate financing]? Why? e. [If needed, prompt using examples from documentation that were not mentioned] EQ 1 + EQ 1a (probe d) USAID.GOV DRAFT EVALUATION REPORT | 73 ii. What was (ISACC/Ready) seeking to achieve [in Country Name] related to improving the capacity of government to implement climate change initiatives (this includes training government staff in monitoring and evaluation [M&E], knowledge management, etc.) Did the achievements match what was planned? Why or why not? (i.e., to what extent have (ISACC/Ready) objectives in this area of work been met?) [Internal: Ready] a. What facilitated and/or hindered achievements in this area? b. What role did Ready play relative to other programs or influencing factors? c. Which specific activities were successful in contributing to these achievements [seek specific examples in targeted countries]. What led you to this conclusion? • What, if any, training/capacity building was provided as part of the activities? • What, if any, ongoing support is there to embed the changes in the organization? If so, please describe. • In what ways, if any, have you seen the policies/procedures being implemented so far; what has helped/hindered that? d. Of the achievements you described, which, if any, would you say are the most important or relevant [to building government capacity]? Why? e. [If needed, prompt using examples from documentation that were not mentioned] [Internal: ISACC] a. What facilitated and/or hindered achievements in this area? b. What role did ISACC play relative to other programs or influencing factors? c. Which specific activities were successful in contributing to these achievements [seek specific examples in targeted countries]. What led you to this conclusion? • In your opinion, which topics (M&E, knowledge management, etc.) or formats (shorter workshops or trainings, university-sponsored courses, embedded technical assistance, short-term technical assistance/consultants, etc.) were more effective than others in supporting capacity building? What led you to this conclusion? • [If regional conferences, events, and workshops are mentioned]: Besides production of formal outputs (conference presentations, drafted documents, etc.), what in your opinion are the main results of regional conferences, workshops, and events? How, if at all, do these activities contribute to building government capacity? • What, if any, ongoing support is there to embed the changes in the organization? If so, please describe. d. Of the achievements you described, which, if any, would you say are the most important or relevant [to building government capacity]? Why? e. [If needed, prompt using examples from documentation that were not mentioned] EQ 2 + EQ 2a (probe d) USAID.GOV DRAFT EVALUATION REPORT | 74 iii. What was (ISACC/Ready) seeking to achieve [in Country Name] related to supporting funding proposals and/or accreditation to entities like the Green Climate Fund (GCF) or Adaptation Fund (AF)? Did the achievements match what was planned? Why or why not? (i.e., to what extent have (ISACC/Ready) objectives in this area of work been met?) [Internal: Ready] a. What facilitated and/or hindered achievements in this area? b. What role did Ready play relative to other programs or influencing factors? • What specific elements of the proposal did Ready contribute to? How did Ready contribute more broadly to the proposal (e.g., by taking a more convening role in developing the proposal)? If known, why was the proposal successful/unsuccessful? • In what ways did Ready contribute to speeding up the accreditation process or make the difference between being accredited or not? c. Which specific activities were successful in contributing to these achievements [seek specific examples in targeted countries]. What led you to this conclusion? • Please provide some examples (if any) of sustainable successes in this area (i.e., built capacity around proposal development/accreditation that is retained over time). d. Of the achievements you described, which, if any, would you say are the most important or relevant [to supporting proposal development and/or accreditation]? Why? e. [If needed, prompt using examples from documentation that were not mentioned] [Internal: ISACC] a. What facilitated and/or hindered achievements in this area? b. What role did ISACC play relative to other programs or influencing factors? c. Which specific activities were successful in contributing to these achievements [seek specific examples in targeted countries]. What led you to this conclusion? • Please provide some examples (if any) of sustainable successes in this area (i.e., built capacity around proposal development/accreditation that is retained over time). • In your opinion, was the level and type of support (workshops and conferences, technical assistance, etc.) provided by ISACC to governments well matched to contribute to achievements in this area? Why or why not? Does this differ significantly by country? d. Of the achievements you described, which, if any, would you say are the most important or relevant [to supporting proposal development and/or accreditation]? Why? e. [If needed, prompt using examples from documentation that were not mentioned] EQ 3 iv. What was (ISACC/Ready) seeking to achieve [in Country Name] related to support for the implementation of country-led climate finance programs and projects? Did the achievements match what was planned? Why or why not? (i.e., to what extent have (ISACC/Ready) objectives in this area of work been met?) [Internal: All] a. What facilitated and/or hindered achievements in this area? b. What role did (ISACC/ Ready) play relative to other programs or influencing factors? c. Which specific activities were successful in contributing to these achievements [seek specific examples in targeted countries]. What led you to this conclusion? • Please provide some examples (if any) of sustainable successes in this area (i.e., improvements in the way climate adaptation projects are or will be implemented, including M&E frameworks, stakeholder engagement, support to applicants for established programs, project management protocols, asset management strategies, etc.). d. Of the achievements you described, which, if any, would you say are the most important or relevant [to supporting country-led finance projects]? Why? e. [If needed, prompt using examples from documentation that were not mentioned] EQ 3 USAID.GOV DRAFT EVALUATION REPORT | 75 v. What was (ISACC/Ready) seeking to achieve [in Country Name] related to engagement of the private sector in climate adaptation and disaster preparedness? Did the achievements match what was planned? Why or why not? (i.e., to what extent have (ISACC/Ready) objectives in this area of work been met?) [Internal: Ready] a. What facilitated and/or hindered achievements in this area? b. What role did Ready play relative to other programs or influencing factors? c. Which specific activities were successful in contributing to these achievements [seek specific examples in targeted countries]. What led you to this conclusion? • Please provide some examples (if any) of sustainable successes in this area (i.e., ongoing engagement from the private sector). • What evidence are you seeing of the private sector engaging beyond the people you worked with directly? [If needed: what would you look for?] d. Of the achievements you described, which, if any, would you say are the most important or relevant [to engaging the private sector]? Why? e. [If needed, prompt using examples from documentation that were not mentioned] [Internal: ISACC] a. What facilitated and/or hindered achievements in this area? b. What role did ISACC play relative to other programs or influencing factors? c. Which specific activities were successful in contributing to these achievements [seek specific examples in targeted countries]. What led you to this conclusion? • Please provide some examples (if any) of sustainable successes in this area (i.e., private sector now participating in climate dialogues or proactively planning). • In your opinion, was the level and type of engagement (mainly workshops and conferences) with the private sector well matched to contribute to achievements in this area? Why or why not? Does this differ significantly by country? d. Of the achievements you described, which, if any, would you say are the most important or relevant [to engaging the private sector]? Why? e. [If needed, prompt using examples from documentation that were not mentioned] EQ 4 3. Are there other key successes or challenges (in this/these countries) not already covered? a. Have there been any successes and/or challenges that were unexpected? b. What were the drivers of these successes/challenges? All Facilitating GESI 4. Consideration and representation of women and marginalized groups is of particular interest to USAID. I would like to ask a few questions about this in relation to the work done by (ISACC/Ready). i. In what ways, if any, were gender and social inclusion consciously considered in the (ISACC/Ready) activities that you are familiar with? If yes, what types of activities were most effective in promoting consideration of gender and social inclusion? What led you to this conclusion? a. Were gender and social inclusion considered in all activities or in a smaller proportion? b. [Seek specific examples in targeted countries, looking particularly for examples related to: Enabling environment (governmental policies, organizational structures, standard operating procedures, plans, frameworks, legislation, etc.); and organizational capacity building, e.g., through GESI training] • [For capacity building, probe for examples (if any) beyond inclusion of women or members of other marginalized groups in trainings, workshops, and events] c. If gender and social inclusion were not consciously considered in the activities, why not? [If needed: Social inclusion can be defined as the process of improving the terms on which individuals and groups take part in society. In the context of this question, this means supporting both men and women, as well as members of marginalized groups, to take an active role in climate change initiatives that impact them.] EQ 1b + EQ 2b USAID.GOV DRAFT EVALUATION REPORT | 76 ii. Have you noted any changes stemming from the activities you mentioned that may have helped to increase inclusivity of any marginalized groups? If yes, please describe. (Please break out examples by different groups as relevant for the activities described above e.g., women, indigenous peoples, outlying islands, migrant populations, other marginalized groups) a. What facilitated inclusivity? b. What hindered inclusivity? c. What role, if any, did (ISACC/Ready) play relative to other programs or influencing factors? d. [If needed, prompt using examples from documentation] e. [If needed, prompt using examples from key activities mentioned above] 5. What suggestions do you have for the specific types of support programs like (ISACC/Ready) could provide that would help countries more deeply represent women and marginalized groups in future activities? a. What other key successes or challenges (in this/these countries) relating to promoting gender and/or social inclusion in climate change would you like to share? All Coordination and synergies 6. We recognize that there are more programs and organizations offering similar services than there were when (ISACC/Ready) began. For the next several questions, we would like to discuss coordination with some of these programs and organizations. i. To your knowledge, what services related to climate financing are others providing? How much overlap or synergies are there with what other donors and programs are offering currently? a. [If needed] probe on other US initiatives as well as of other major players like Australia, New Zealand, Germany, Japan, and UN organizations or others. All ii. Please briefly describe your active coordination efforts with other donors or implementing parties. a. Are there other organizations you are coordinating or collaborating with relating to providing support to countries we should know about? [This refers to coordination or partnerships to provide support (to PICs) not organizations that are receiving direct support.] b. In what ways has this coordination impacted your program’s ability to (if so, how?): • Influence the enabling environment (governmental policies, organizational structures, standard operating procedures, plans, frameworks, legislation, etc.) • Build capacity of government • Support implementation of country-owned climate change actions • Engage the private sector Looking ahead 7. As this will help USAID plan for the future, we are also interested in your views on how needs have evolved. i. In your opinion, what type(s) of support are most needed by PICs these days? Of the different types of services your project provided, which do you think should be prioritized going forward? And under what conditions? a. [If needed, prompt for examples from key activities mentioned above] • Examples include: funding proposal support, training, policy development support, accreditation support, workshops, conferences, and events, direct funding of government offices, etc. b. In your opinion, are there any caveats or types of support that should be avoided? If yes, please describe. All Closing USAID.GOV DRAFT EVALUATION REPORT | 77 8. We are almost finished. Do you have any other closing comments or thoughts about what we talked about today? a [Ask any remaining important questions for this informant specifically relating to the program.] b. [If needed, update them on next steps/follow up actions.] All Thank you for your time! USAID.GOV DRAFT EVALUATION REPORT | 78 INTERVIEW GUIDE B: NATIONAL GOVERNMENT AND AFFILIATED ENTITIES Interview ID number: Date: Name of Interviewer: Location of Interview (Country, City or Town): [Internal Instructions: Conduct introductions and ask for consent to record. Ask the interview participant to provide the below information] Name of Respondent: Organization: Position: Nationality: Sex: Consent to record: Yes/No [circle one] CORE QUESTIONS **Interviewer Note: Not all questions should be asked of all respondents. Tailor questions and ensure that interview can be completed within one hour. PROBES *Interviewer Note: Only use if issue is not addressed in response to core question or not already known EQ Relevant evaluation question Informant’s role and country context 1. We would like to begin by asking you about your work, particularly your involvement with climate adaptation activities and accessing climate finance in [your country]? i. What is your role (in your organization/Ministry)? a. Please provide an overview of your general portfolio and day to day responsibilities, particularly those relevant to climate finance or climate change projects All ii. Please provide a brief overview of the climate activities you have been directly involved with or know about in [country name]. [Internal: Have country notes at hand to prompt if needed; keep this section brief.] a. [As feasible, ask the informant to differentiate (ISACC/Ready) activities and other activities he/she may have participated in.] ● [If needed:] What process was used to decide what activities to pursue (for your country); i.e., were they ad hoc requests or planned annually? b. [Internal: Clarify any key details relating to the country’s progress and activities not already clear from the documentation or if there is important relevant documentation that has not yet been shared, e.g., national plans, activity highlights, etc.] iii. How would you characterize your country’s current priorities relating to accessing climate finance and implementing climate adaptation projects? a. [If needed: Use examples to prompt respondents: may be technical, e.g., developing funding proposals, managing funding secured, increasing capacity of local governments to administer projects, etc.; or topical, e.g., focusing on impacts of sea level rise.] b. In general, what would you say are the top three factors (either internally or due to external support) that contributed to work in this area so far? Key achievements and challenges 2.       USAID is interested in understanding more about what went well as well as what was less effective regarding ISACC and Ready. As we are going to cover two programs I am going to ask about some types of activities for each program in turn. * *Ask about both Activities if the stakeholder list indicates involvement in both; otherwise, only ask the questions for the relevant Activity. Use the stakeholder list and background materials to determine which outcome areas are most relevant for the stakeholder (regional collaboration, capacity building, private sector engagement, etc.); focus on the questions in this area and lightly touch the others to ensure that no information is missed. More than one outcome area may be relevant for some stakeholders. ISACC USAID.GOV DRAFT EVALUATION REPORT | 79 i. In what ways, if any, did ISACC help support the development of governmental policies, organizational structures, plans, frameworks, legislation, etc., in [country name] related to accessing climate finance? Please describe how. a. What facilitated and/or hindered achievements in this area? b. What role did ISACC play relative to other programs or influencing factors? c. Which specific activities were successful in contributing to these achievements [seek specific examples in targeted countries]. What led you to this conclusion? ● Please provide some examples (if any) of sustainable successes in this area (i.e., lasting changes to policies or organizational structures). ● [If regional conferences, events, and workshops are mentioned]: Besides production of formal outputs (conference presentations, drafted documents, etc.), what in your opinion are the main results of regional conferences, workshops, and events? How, if at all, do these activities contribute to improving the enabling environment for accessing climate finance? ● In your opinion, was the level and type of support (direct funding, technical assistance, etc.) provided by ISACC to governments well matched to contribute to achievements in this area? Why or why not? d. Of the achievements you described, which, if any, would you say are the most important or relevant [to improving the enabling environment for accessing climate financing]? Why? e. [If needed, prompt using examples from documentation that were not mentioned] EQ 1 + EQ 1a (probe d) ii. In what ways, if any, did ISACC contribute to strengthening your ability, or the ability of any of your colleagues, to implement climate change initiatives (this includes project management, monitoring and evaluation [M&E], knowledge management, etc.)? Please describe how. a. What facilitated and/or hindered achievements in this area? b. What role did ISACC play relative to other programs or influencing factors? c. Which specific activities were successful in contributing to these achievements [seek specific examples in targeted countries]. What led you to this conclusion? ● In your opinion, which topics (M&E, knowledge management, etc.) or formats (shorter workshops or trainings, university-sponsored courses, embedded technical assistance, short-term technical assistance/consultants, etc.) were more effective than others in supporting capacity building? What led you to this conclusion? ● [If regional conferences, events, and workshops are mentioned]: Besides production of formal outputs (conference presentations, drafted documents, etc.), what in your opinion are the main results of regional conferences, workshops, and events? How, if at all, do these activities contribute to building government capacity? ● What, if any, ongoing support is there to embed the changes in the organization? If so, please describe. d. Of the achievements you described, which, if any, would you say are the most important or relevant [to building government capacity]? Why? e. [If needed, prompt using examples from documentation that were not mentioned] EQ 2 + EQ 2a (probe d) USAID.GOV DRAFT EVALUATION REPORT | 80 iii. In what ways, if any, did ISACC help support [country name] to implement country-led climate finance programs and projects (this includes support for proposal submission to the Green Climate Fund (GCF) or Adaptation Fund (AF), support for the accreditation process, support for funding management, etc.)? Please describe how. a. What facilitated and/or hindered achievements in this area? b. What role did ISACC play relative to other programs or influencing factors? c. Which specific activities were successful in contributing to these achievements [seek specific examples in targeted countries]. What led you to this conclusion? ● Please provide some examples (if any) of sustainable successes in this area (i.e., built capacity around proposal development/accreditation that is retained over time, improvements in M&E frameworks, stakeholder engagements, project management protocols, etc.). ● In your opinion, was the level and type of support (direct funding, technical assistance, etc.) provided by ISACC to governments well matched to contribute to achievements in this area? Why or why not? d. Do you see improvements in the way projects are implemented? If yes, in what way? ● [if yes to the previous question]: In what ways has this improved implementation (or will) at the community level? If so, how? e. [Check stakeholders list and if the relevant activity relates to training/capacity building]: Are there plans to follow up with further training/capacity building to maintain skills and/or train new staff? f. Of the achievements you described, which, if any, would you say are the most important or relevant [to supporting country-led finance projects]? Why? g. [If needed, prompt using examples from documentation that were not mentioned] EQ 3 iv. In what ways, if any, did ISACC help support [country name] to engage the private sector in climate adaptation and disaster preparedness? Please describe how. a. What facilitated and/or hindered achievements in this area? b. What role did ISACC play relative to other programs or influencing factors? c. Which specific activities were successful in contributing to these achievements [seek specific examples in targeted countries]. What led you to this conclusion? ● Please provide some examples (if any) of sustainable successes in this area (i.e., private sector now participating in climate dialogues or proactively planning). ● In your opinion, was the level and type of engagement (mainly workshops and conferences) with the private sector well matched to contribute to achievements in this area? Why or why not? d. Of the achievements you described, which, if any, would you say are the most important or relevant [to engaging the private sector]? Why? e. [If needed, prompt using examples from documentation that were not mentioned] EQ 4 Climate Ready i. In what ways, if any, did Ready help support the development of governmental policies, organizational structures, plans, frameworks, legislation, etc., in [country name] related to accessing climate finance? Please describe how. a. What facilitated and/or hindered achievements in this area? b. What role did Ready play relative to other programs or influencing factors? ● Please provide some examples (if any) of sustainable successes in this area (i.e., lasting changes to policies or organizational structures). c. Which specific activities were successful in contributing to these achievements [seek specific examples in targeted countries]. What led you to this conclusion? d. Of the achievements you described, which, if any, would you say are the most important or relevant [to improving the enabling environment for accessing climate financing]? Why? e. [If needed, prompt using examples from documentation that were not mentioned] EQ 1 + EQ 1a (probe d) USAID.GOV DRAFT EVALUATION REPORT | 81 ii. In what ways, if any, did Ready contribute to strengthening your ability, or the ability of any of your colleagues, to implement climate change initiatives (this includes project management, monitoring and evaluation [M&E], knowledge management, etc.)? Please describe how. a. What facilitated and/or hindered achievements in this area? b. What role did Ready play relative to other programs or influencing factors? c. Which specific activities were successful in contributing to these achievements [seek specific examples in targeted countries]. What led you to this conclusion? ● What, if any, training/capacity building was provided as part of the activities? ● What, if any, ongoing support is there to embed the changes in the organization? If so, please describe. o [if needed]: What if any plans are there to follow up with further training/capacity building to maintain skills and/or train new staff? ● In what ways, if any, have you seen the policies/procedures being implemented so far; what has helped/hindered that? d. Of the achievements you described, are there any that you believe are the most important or relevant [to building government capacity]? If yes, why? e. [If needed, prompt using examples from documentation that were not mentioned] EQ 2 + EQ 2a (probe d) iii. In what ways, if any, did Ready help support [country name] to implement country-led climate finance programs and projects (this includes support for proposal submission to the Green Climate Fund (GCF) or Adaptation Fund (AF), support for the accreditation process, support for funding management, etc.)? Please describe how. a. What facilitated and/or hindered achievements in this area? b. What role did Ready play relative to other programs or influencing factors? ● What specific elements of the proposal did Ready contribute to? How did Ready contribute more broadly to the proposal (e.g., by taking a more convening role in developing the proposal)? Did other international organizations contribute to the proposal and if so, did they work with Ready or on separate elements? ● If known, why was the proposal successful/unsuccessful? ● In what ways did Ready contribute to speeding up the accreditation process or make the difference between being accredited or not? c. Which specific activities were successful in contributing to these achievements [seek specific examples in targeted countries]. What led you to this conclusion? ● Please provide some examples (if any) of sustainable successes in this area (i.e., built capacity around proposal development/accreditation that is retained over time, improvements in M&E frameworks, stakeholder engagements, project management protocols, etc.). d. Do you see improvements in the way projects are implemented? If yes, in what way? ● [if yes to the previous question]: In what ways has this improved implementation (or will) at the community level? If so, how? e. [Check stakeholders list and if the relevant activity relates to training/capacity building]: Are there plans to follow up with further training/capacity building to maintain skills and/or train new staff? f. Of the achievements you described, which, if any, would you say are the most important or relevant [to supporting country-led finance projects]? Why? g. [If needed, prompt using examples from documentation that were not mentioned] EQ 3 USAID.GOV DRAFT EVALUATION REPORT | 82 iv. In what ways, if any, did Ready help support [country name] to engage the private sector in climate adaptation and disaster preparedness? Please describe how. a. What facilitated and/or hindered achievements in this area? b. What role did Ready play relative to other programs or influencing factors? c. Which specific activities were successful in contributing to these achievements [seek specific examples in targeted countries]. What led you to this conclusion? ● Please provide some examples (if any) of sustainable successes in this area (i.e., ongoing/sustainable engagement from the private sector). ● What evidence are you seeing of the private sector engaging beyond the people you worked with directly? [If needed: what would you look for?] d. Of the achievements you described, which, if any, would you say are the most important or relevant [to engaging the private sector]? Why? e. [If needed, prompt using examples from documentation that were not mentioned] EQ 4 3. Given the ISACC and Climate Ready activities and results just discussed, to what extent have these aligned with your country’s priorities relating to accessing climate finance and implementing projects? a. Are there other key ways (ISACC/Ready) supported your country not already covered? All Facilitating GESI 4. Consideration and representation of women and marginalized groups is of particular interest to USAID. Once again, I would like to ask a few questions relating to ISACC first, then Ready. * *As above, only ask about both Activities if the stakeholder list indicates involvement in both; otherwise, ask the questions for the relevant Activity. ISACC i. In what ways, if any, were gender and social inclusion consciously considered in the ISACC activities that you are familiar with? If yes, what types of activities were most effective in promoting consideration of gender and social inclusion? What led you to this conclusion? a. Were gender and social inclusion considered in all activities or in a smaller proportion? b. [Seek specific examples in targeted countries, looking particularly for examples related to: Enabling environment (governmental policies, organizational structures, standard operating procedures, plans, frameworks, legislation, etc.); and organizational capacity building, e.g., through GESI training] ● [For capacity building, probe for examples (if any) beyond inclusion of women or members of other marginalized groups in trainings, workshops, and events] c. If gender and social inclusion were not consciously considered in the activities, why not? [If needed: Social inclusion can be defined as the process of improving the terms on which individuals and groups take part in society. In the context of this question, this means supporting both men and women, as well as members of marginalized groups, to take an active role in climate change initiatives that impact them.] EQ 1b + EQ 2b ii. Have you noted any changes stemming from the activities you mentioned that may have helped to increase inclusivity of any marginalized groups? If yes, please describe. (Please break out examples by different groups as relevant for the activities described above e.g., women, indigenous peoples, outlying islands, migrant populations, other marginalized groups) a. What facilitated inclusivity? b. What hindered inclusivity? c. What role, if any, did ISACC play relative to other programs or influencing factors? d. [If needed, prompt using examples from documentation] e. [If needed, prompt using examples from key activities mentioned above] Climate Ready USAID.GOV DRAFT EVALUATION REPORT | 83 i. In what ways, if any, were gender and social inclusion consciously considered in the Ready activities that you are familiar with? If yes, what types of activities were most effective in promoting consideration of gender and social inclusion? What led you to this conclusion? a. Were gender and social inclusion considered in all activities or in a smaller proportion? b. [Seek specific examples in targeted countries, looking particularly for examples related to: Enabling environment (governmental policies, organizational structures, standard operating procedures, plans, frameworks, legislation, etc.); and organizational capacity building, e.g., through GESI training] ● [For capacity building, probe for examples (if any) beyond inclusion of women or members of other marginalized groups in trainings, workshops, and events] c. If gender and social inclusion were not consciously considered in the activities, why not? [If needed: Social inclusion can be defined as the process of improving the terms on which individuals and groups take part in society. In the context of this question, this means supporting both men and women, as well as members of marginalized groups, to take an active role in climate change initiatives that impact them.] EQ 1b + EQ 2b ii. Have you noted any changes stemming from the activities you mentioned that may have helped to increase inclusivity of any marginalized groups? If yes, please describe. (Please break out examples by different groups as relevant for the activities described above e.g., women, indigenous peoples, outlying islands, migrant populations, other marginalized groups) a. What facilitated inclusivity? b. What hindered inclusivity? c. What role, if any, did Ready play relative to other programs or influencing factors? d. [If needed, prompt using examples from documentation] e. [If needed, prompt using examples from key activities mentioned above] 5. What suggestions do you have for specific types of support programs like (ISACC/Ready) could provide that would help countries more deeply represent women and marginalized groups in future activities? a. What other key successes or challenges (in this/these countries) relating to promoting gender and/or social inclusion in climate change would you like to share? All Coordination and synergies 6. We recognize that there are more programs and organizations offering similar services [related to climate finance] than there were when ISACC and Ready began. We would like to discuss coordination with some of these programs and organizations. i. What types of support, if any, are you receiving [or are you aware of] related to climate financing from other international donors and programs? How much overlap or synergies are there with USAID support and what other donors and programs are offering currently? a. [If needed] probe on other US initiatives as well as of other major players like Australia, New Zealand, Germany, Japan, and UN organizations or others. All USAID.GOV DRAFT EVALUATION REPORT | 84 ii. Please briefly describe your active regional coordination efforts with other PIC governments. [Probe as needed to understand the significance/importance of regional activities for their context, especially relating to ISACC supported activities. Seek to distinguish between ISACC activities and other activities by the IPs to the extent feasible (SPC, PIFS, SPREP).] ● [Probe examples as needed]: conferences, workshops and events (e.g., regional climate change finance meetings, climate finance partners’ coordination group, Pacific Resilience Meeting side event on information knowledge management, regional Monitoring, Evaluation, and Learning Convening); policy or strategy coordination (e.g., facilitated joint workshop on FRDP M&E Framework and Resilience Standards); resource development (e.g., regional climate finance tracking tool). b. In what ways has this coordination impacted the following: ● Governmental policies, organizational structures, standard operating procedures, plans, frameworks, legislation, etc. ● Building capacity of government ● Support implementation of country-owned climate change actions (i.e., accreditation and proposal development) ● Engagement of the private sector All Looking ahead 7. As this will help USAID plan for the future, we are also interested in your views on how needs have evolved. i. Of the different types of support programs like ISACC and Ready could realistically offer, which do you think should be prioritized going forward? And under what conditions? a. [If needed, prompt for examples from key activities mentioned above] ● Examples include: funding proposal support, training, policy development support, accreditation support, workshops, conferences, and events, direct funding of government offices, etc. b. In your opinion, are there any caveats or types of support that should be avoided? [e.g., only offer funding proposal support if there is sufficient political will]. If yes, please describe. All Closing 8. We are almost finished. Do you have any other closing comments or thoughts about what we talked about today? a. [Ask any remaining important questions specifically relating to the program.] b. [If needed, update them on next steps/follow up actions.] All USAID.GOV DRAFT EVALUATION REPORT | 85 INTERVIEW GUIDE B: NATIONAL GOVERNMENT AND AFFILIATED ENTITIES Interview ID number: Date: Name of Interviewer: Location of Interview (Country, City or Town): [Internal Instructions: Conduct introductions and ask for consent to record. Ask the interview participant to provide the below information] Name of Respondent: Organization: Position: Nationality: Sex: Consent to record: Yes/No [circle one] CORE QUESTIONS **Interviewer Note: Not all questions should be asked of all respondents. Tailor questions and ensure that interview can be completed within one hour. PROBES *Interviewer Note: Only use if issue is not addressed in response to core question or not already known EQ Relevant evaluation question Informant’s role and country context 1. We would like to begin by asking you about your work, particularly your involvement with climate adaptation activities and accessing climate finance in [your country]? iv. What is your role (in your organization/Ministry)? a. Please provide an overview of your general portfolio and day to day responsibilities, particularly those relevant to climate finance or climate change projects All v. Please provide a brief overview of the climate activities you have been directly involved with or know about in [country name]. [Internal: Have country notes at hand to prompt if needed; keep this section brief.] a. [As feasible, ask the informant to differentiate (ISACC/Ready) activities and other activities he/she may have participated in.] ● [If needed:] What process was used to decide what activities to pursue (for your country); i.e., were they ad hoc requests or planned annually? b. [Internal: Clarify any key details relating to the country’s progress and activities not already clear from the documentation or if there is important relevant documentation that has not yet been shared, e.g., national plans, activity highlights, etc.] vi. How would you characterize your country’s current priorities relating to accessing climate finance and implementing climate adaptation projects? a. [If needed: Use examples to prompt respondents: may be technical, e.g., developing funding proposals, managing funding secured, increasing capacity of local governments to administer projects, etc.; or topical, e.g., focusing on impacts of sea level rise.] b. In general, what would you say are the top three factors (either internally or due to external support) that contributed to work in this area so far? Key achievements and challenges 2.       USAID is interested in understanding more about what went well as well as what was less effective regarding ISACC and Ready. As we are going to cover two programs I am going to ask about some types of activities for each program in turn. * *Ask about both Activities if the stakeholder list indicates involvement in both; otherwise, only ask the questions for the relevant Activity. Use the stakeholder list and background materials to determine which outcome areas are most relevant for the stakeholder (regional collaboration, capacity building, private sector engagement, etc.); focus on the questions in this area and lightly touch the others to ensure that no information is missed. More than one outcome area may be relevant for some stakeholders. ISACC USAID.GOV DRAFT EVALUATION REPORT | 86 v. In what ways, if any, did ISACC help support the development of governmental policies, organizational structures, plans, frameworks, legislation, etc., in [country name] related to accessing climate finance? Please describe how. a. What facilitated and/or hindered achievements in this area? b. What role did ISACC play relative to other programs or influencing factors? c. Which specific activities were successful in contributing to these achievements [seek specific examples in targeted countries]. What led you to this conclusion? ● Please provide some examples (if any) of sustainable successes in this area (i.e., lasting changes to policies or organizational structures). ● [If regional conferences, events, and workshops are mentioned]: Besides production of formal outputs (conference presentations, drafted documents, etc.), what in your opinion are the main results of regional conferences, workshops, and events? How, if at all, do these activities contribute to improving the enabling environment for accessing climate finance? ● In your opinion, was the level and type of support (direct funding, technical assistance, etc.) provided by ISACC to governments well matched to contribute to achievements in this area? Why or why not? d. Of the achievements you described, which, if any, would you say are the most important or relevant [to improving the enabling environment for accessing climate financing]? Why? e. [If needed, prompt using examples from documentation that were not mentioned] EQ 1 + EQ 1a (probe d) vi. In what ways, if any, did ISACC contribute to strengthening your ability, or the ability of any of your colleagues, to implement climate change initiatives (this includes project management, monitoring and evaluation [M&E], knowledge management, etc.)? Please describe how. a. What facilitated and/or hindered achievements in this area? b. What role did ISACC play relative to other programs or influencing factors? c. Which specific activities were successful in contributing to these achievements [seek specific examples in targeted countries]. What led you to this conclusion? ● In your opinion, which topics (M&E, knowledge management, etc.) or formats (shorter workshops or trainings, university-sponsored courses, embedded technical assistance, short￾term technical assistance/consultants, etc.) were more effective than others in supporting capacity building? What led you to this conclusion? ● [If regional conferences, events, and workshops are mentioned]: Besides production of formal outputs (conference presentations, drafted documents, etc.), what in your opinion are the main results of regional conferences, workshops, and events? How, if at all, do these activities contribute to building government capacity? ● What, if any, ongoing support is there to embed the changes in the organization? If so, please describe. d. Of the achievements you described, which, if any, would you say are the most important or relevant [to building government capacity]? Why? e. [If needed, prompt using examples from documentation that were not mentioned] EQ 2 + EQ 2a (probe d) USAID.GOV DRAFT EVALUATION REPORT | 87 vii. In what ways, if any, did ISACC help support [country name] to implement country-led climate finance programs and projects (this includes support for proposal submission to the Green Climate Fund (GCF) or Adaptation Fund (AF), support for the accreditation process, support for funding management, etc.)? Please describe how. a. What facilitated and/or hindered achievements in this area? b. What role did ISACC play relative to other programs or influencing factors? c. Which specific activities were successful in contributing to these achievements [seek specific examples in targeted countries]. What led you to this conclusion? ● Please provide some examples (if any) of sustainable successes in this area (i.e., built capacity around proposal development/accreditation that is retained over time, improvements in M&E frameworks, stakeholder engagements, project management protocols, etc.). ● In your opinion, was the level and type of support (direct funding, technical assistance, etc.) provided by ISACC to governments well matched to contribute to achievements in this area? Why or why not? d. Do you see improvements in the way projects are implemented? If yes, in what way? ● [if yes to the previous question]: In what ways has this improved implementation (or will) at the community level? If so, how? e. [Check stakeholders list and if the relevant activity relates to training/capacity building]: Are there plans to follow up with further training/capacity building to maintain skills and/or train new staff? f. Of the achievements you described, which, if any, would you say are the most important or relevant [to supporting country-led finance projects]? Why? g. [If needed, prompt using examples from documentation that were not mentioned] EQ 3 viii. In what ways, if any, did ISACC help support [country name] to engage the private sector in climate adaptation and disaster preparedness? Please describe how. a. What facilitated and/or hindered achievements in this area? b. What role did ISACC play relative to other programs or influencing factors? c. Which specific activities were successful in contributing to these achievements [seek specific examples in targeted countries]. What led you to this conclusion? ● Please provide some examples (if any) of sustainable successes in this area (i.e., private sector now participating in climate dialogues or proactively planning). ● In your opinion, was the level and type of engagement (mainly workshops and conferences) with the private sector well matched to contribute to achievements in this area? Why or why not? d. Of the achievements you described, which, if any, would you say are the most important or relevant [to engaging the private sector]? Why? e. [If needed, prompt using examples from documentation that were not mentioned] EQ 4 Climate Ready v. In what ways, if any, did Ready help support the development of governmental policies, organizational structures, plans, frameworks, legislation, etc., in [country name] related to accessing climate finance? Please describe how. a. What facilitated and/or hindered achievements in this area? b. What role did Ready play relative to other programs or influencing factors? ● Please provide some examples (if any) of sustainable successes in this area (i.e., lasting changes to policies or organizational structures). c. Which specific activities were successful in contributing to these achievements [seek specific examples in targeted countries]. What led you to this conclusion? d. Of the achievements you described, which, if any, would you say are the most important or relevant [to improving the enabling environment for accessing climate financing]? Why? e. [If needed, prompt using examples from documentation that were not mentioned] EQ 1 + EQ 1a (probe d) USAID.GOV DRAFT EVALUATION REPORT | 88 vi. In what ways, if any, did Ready contribute to strengthening your ability, or the ability of any of your colleagues, to implement climate change initiatives (this includes project management, monitoring and evaluation [M&E], knowledge management, etc.)? Please describe how. a. What facilitated and/or hindered achievements in this area? b. What role did Ready play relative to other programs or influencing factors? c. Which specific activities were successful in contributing to these achievements [seek specific examples in targeted countries]. What led you to this conclusion? ● What, if any, training/capacity building was provided as part of the activities? ● What, if any, ongoing support is there to embed the changes in the organization? If so, please describe. o [if needed]: What if any plans are there to follow up with further training/capacity building to maintain skills and/or train new staff? ● In what ways, if any, have you seen the policies/procedures being implemented so far; what has helped/hindered that? d. Of the achievements you described, are there any that you believe are the most important or relevant [to building government capacity]? If yes, why? e. [If needed, prompt using examples from documentation that were not mentioned] EQ 2 + EQ 2a (probe d) vii. In what ways, if any, did Ready help support [country name] to implement country-led climate finance programs and projects (this includes support for proposal submission to the Green Climate Fund (GCF) or Adaptation Fund (AF), support for the accreditation process, support for funding management, etc.)? Please describe how. a. What facilitated and/or hindered achievements in this area? b. What role did Ready play relative to other programs or influencing factors? ● What specific elements of the proposal did Ready contribute to? How did Ready contribute more broadly to the proposal (e.g., by taking a more convening role in developing the proposal)? Did other international organizations contribute to the proposal and if so, did they work with Ready or on separate elements? ● If known, why was the proposal successful/unsuccessful? ● In what ways did Ready contribute to speeding up the accreditation process or make the difference between being accredited or not? c. Which specific activities were successful in contributing to these achievements [seek specific examples in targeted countries]. What led you to this conclusion? ● Please provide some examples (if any) of sustainable successes in this area (i.e., built capacity around proposal development/accreditation that is retained over time, improvements in M&E frameworks, stakeholder engagements, project management protocols, etc.). d. Do you see improvements in the way projects are implemented? If yes, in what way? ● [if yes to the previous question]: In what ways has this improved implementation (or will) at the community level? If so, how? e. [Check stakeholders list and if the relevant activity relates to training/capacity building]: Are there plans to follow up with further training/capacity building to maintain skills and/or train new staff? f. Of the achievements you described, which, if any, would you say are the most important or relevant [to supporting country-led finance projects]? Why? g. [If needed, prompt using examples from documentation that were not mentioned] EQ 3 USAID.GOV DRAFT EVALUATION REPORT | 89 viii. In what ways, if any, did Ready help support [country name] to engage the private sector in climate adaptation and disaster preparedness? Please describe how. a. What facilitated and/or hindered achievements in this area? b. What role did Ready play relative to other programs or influencing factors? c. Which specific activities were successful in contributing to these achievements [seek specific examples in targeted countries]. What led you to this conclusion? ● Please provide some examples (if any) of sustainable successes in this area (i.e., ongoing/sustainable engagement from the private sector). ● What evidence are you seeing of the private sector engaging beyond the people you worked with directly? [If needed: what would you look for?] d. Of the achievements you described, which, if any, would you say are the most important or relevant [to engaging the private sector]? Why? e. [If needed, prompt using examples from documentation that were not mentioned] EQ 4 3. Given the ISACC and Climate Ready activities and results just discussed, to what extent have these aligned with your country’s priorities relating to accessing climate finance and implementing projects? a. Are there other key ways (ISACC/Ready) supported your country not already covered? All Facilitating GESI 4. Consideration and representation of women and marginalized groups is of particular interest to USAID. Once again, I would like to ask a few questions relating to ISACC first, then Ready. * *As above, only ask about both Activities if the stakeholder list indicates involvement in both; otherwise, ask the questions for the relevant Activity. ISACC iii. In what ways, if any, were gender and social inclusion consciously considered in the ISACC activities that you are familiar with? If yes, what types of activities were most effective in promoting consideration of gender and social inclusion? What led you to this conclusion? a. Were gender and social inclusion considered in all activities or in a smaller proportion? b. [Seek specific examples in targeted countries, looking particularly for examples related to: Enabling environment (governmental policies, organizational structures, standard operating procedures, plans, frameworks, legislation, etc.); and organizational capacity building, e.g., through GESI training] ● [For capacity building, probe for examples (if any) beyond inclusion of women or members of other marginalized groups in trainings, workshops, and events] c. If gender and social inclusion were not consciously considered in the activities, why not? [If needed: Social inclusion can be defined as the process of improving the terms on which individuals and groups take part in society. In the context of this question, this means supporting both men and women, as well as members of marginalized groups, to take an active role in climate change initiatives that impact them.] EQ 1b + EQ 2b iv. Have you noted any changes stemming from the activities you mentioned that may have helped to increase inclusivity of any marginalized groups? If yes, please describe. (Please break out examples by different groups as relevant for the activities described above e.g., women, indigenous peoples, outlying islands, migrant populations, other marginalized groups) a. What facilitated inclusivity? b. What hindered inclusivity? c. What role, if any, did ISACC play relative to other programs or influencing factors? d. [If needed, prompt using examples from documentation] e. [If needed, prompt using examples from key activities mentioned above] Climate Ready USAID.GOV DRAFT EVALUATION REPORT | 90 iii. In what ways, if any, were gender and social inclusion consciously considered in the Ready activities that you are familiar with? If yes, what types of activities were most effective in promoting consideration of gender and social inclusion? What led you to this conclusion? a. Were gender and social inclusion considered in all activities or in a smaller proportion? b. [Seek specific examples in targeted countries, looking particularly for examples related to: Enabling environment (governmental policies, organizational structures, standard operating procedures, plans, frameworks, legislation, etc.); and organizational capacity building, e.g., through GESI training] ● [For capacity building, probe for examples (if any) beyond inclusion of women or members of other marginalized groups in trainings, workshops, and events] c. If gender and social inclusion were not consciously considered in the activities, why not? [If needed: Social inclusion can be defined as the process of improving the terms on which individuals and groups take part in society. In the context of this question, this means supporting both men and women, as well as members of marginalized groups, to take an active role in climate change initiatives that impact them.] EQ 1b + EQ 2b iv. Have you noted any changes stemming from the activities you mentioned that may have helped to increase inclusivity of any marginalized groups? If yes, please describe. (Please break out examples by different groups as relevant for the activities described above e.g., women, indigenous peoples, outlying islands, migrant populations, other marginalized groups) a. What facilitated inclusivity? b. What hindered inclusivity? c. What role, if any, did Ready play relative to other programs or influencing factors? d. [If needed, prompt using examples from documentation] e. [If needed, prompt using examples from key activities mentioned above] 5. What suggestions do you have for specific types of support programs like (ISACC/Ready) could provide that would help countries more deeply represent women and marginalized groups in future activities? a. What other key successes or challenges (in this/these countries) relating to promoting gender and/or social inclusion in climate change would you like to share? All Coordination and synergies 6. We recognize that there are more programs and organizations offering similar services [related to climate finance] than there were when ISACC and Ready began. We would like to discuss coordination with some of these programs and organizations. iii. What types of support, if any, are you receiving [or are you aware of] related to climate financing from other international donors and programs? How much overlap or synergies are there with USAID support and what other donors and programs are offering currently? a. [If needed] probe on other US initiatives as well as of other major players like Australia, New Zealand, Germany, Japan, and UN organizations or others. All USAID.GOV DRAFT EVALUATION REPORT | 91 iv. Please briefly describe your active regional coordination efforts with other PIC governments. [Probe as needed to understand the significance/importance of regional activities for their context, especially relating to ISACC supported activities. Seek to distinguish between ISACC activities and other activities by the IPs to the extent feasible (SPC, PIFS, SPREP).] ● [Probe examples as needed]: conferences, workshops and events (e.g., regional climate change finance meetings, climate finance partners’ coordination group, Pacific Resilience Meeting side event on information knowledge management, regional Monitoring, Evaluation, and Learning Convening); policy or strategy coordination (e.g., facilitated joint workshop on FRDP M&E Framework and Resilience Standards); resource development (e.g., regional climate finance tracking tool). b. In what ways has this coordination impacted the following: ● Governmental policies, organizational structures, standard operating procedures, plans, frameworks, legislation, etc. ● Building capacity of government ● Support implementation of country-owned climate change actions (i.e., accreditation and proposal development) ● Engagement of the private sector All Looking ahead 7. As this will help USAID plan for the future, we are also interested in your views on how needs have evolved. ii. Of the different types of support programs like ISACC and Ready could realistically offer, which do you think should be prioritized going forward? And under what conditions? b. [If needed, prompt for examples from key activities mentioned above] ● Examples include: funding proposal support, training, policy development support, accreditation support, workshops, conferences, and events, direct funding of government offices, etc. b. In your opinion, are there any caveats or types of support that should be avoided? [e.g., only offer funding proposal support if there is sufficient political will]. If yes, please describe. All Closing 8. We are almost finished. Do you have any other closing comments or thoughts about what we talked about today? a. [Ask any remaining important questions specifically relating to the program.] b. [If needed, update them on next steps/follow up actions.] All USAID.GOV DRAFT EVALUATION REPORT | 92 INTERVIEW GUIDE C: CIVIL SOCIETY AND REGIONAL ORGANIZATIONS Interview ID number: Date: Name of Interviewer: Location of Interview (Country, City or Town): [Internal Instructions: Conduct introductions and ask for consent to record. Ask the interview participant to provide the below information] Name of Respondent: Organization: Position: Nationality: Sex: Consent to record: Yes/No [circle one] CORE QUESTIONS **Interviewer Note: Not all questions should be asked of all respondents. Tailor questions and ensure that interview can be completed within one hour. PROBES *Interviewer Note: Only use if issue is not addressed in response to core question or not already known EQ Relevant evaluation question Informant’s role and country context 1. We would like to begin by asking you about your work, particularly your involvement with climate adaptation activities and accessing climate finance in [your organization/group]? i. What is your role (in your organization/group)? a. Please provide an overview of your general portfolio and day to day responsibilities related to climate finance or climate change projects All ii. Please provide a brief overview of the climate activities you have been directly involved with or know relating to your [organization/group]. [Internal: Have organization/group notes at hand to prompt if needed; keep this section brief.] [As feasible, ask the informant to differentiate (ISACC/Ready) activities and other activities he/she may have participated in.] ● [If needed:] What process was used to decide what activities to pursue (for your organization/group); i.e., were they ad hoc requests or planned annually? b. [Internal: Clarify any key details relating to the organization’s/group’s progress and activities not already clear from the documentation or if there is important relevant documentation that has not yet been shared, e.g., mission statements, activity highlights, case studies, etc.] iii. How would you characterize your [organization’s/group’s] current priorities relating to accessing climate finance and implementing climate adaptation projects? a. [If needed: Use examples to prompt respondents: may be technical, e.g., developing funding proposals, increasing capacity of local governments to administer projects, etc.; or topical, e.g., focusing on impacts of sea level rise.] b. In general, what would you say are the top three factors (either internally or due to external support) that contributed to your [organization’s/group’s] work in this area so far? Key achievements and challenges 2.       USAID is interested in understanding more about what went well as well as what was less effective regarding ISACC and Ready. As we are going to cover two programs I am going to ask about some types of activities for each program in turn. * *Ask about both Activities if the stakeholder list indicates involvement in both; otherwise, only ask the questions for the relevant Activity. Use the stakeholder list and background materials to determine which outcome areas are most relevant for the stakeholder (regional collaboration, capacity building, private sector engagement, etc.); focus on the questions in this area and lightly touch the others to ensure that no information is missed. More than one outcome area may be relevant for some stakeholders. USAID.GOV DRAFT EVALUATION REPORT | 93 ISACC i. In what ways, if any, did ISACC contribute to improving the capacity of government to implement climate change initiatives (this includes project management, monitoring and evaluation [M&E], knowledge management, etc.)? Please describe how. a. What facilitated and/or hindered achievements in this area? b. What role did ISACC play relative to other programs or influencing factors? c. Which specific activities were successful in contributing to these achievements [seek specific examples in targeted countries]. What led you to this conclusion? ● In your opinion, which topics (M&E, knowledge management, etc.) or formats (shorter workshops or trainings, university-sponsored courses, embedded technical assistance, short-term technical assistance/consultants, etc.) were more effective than others in supporting capacity building? What led you to this conclusion? ● [If regional conferences, events, and workshops are mentioned]: Besides production of formal outputs (conference presentations, drafted documents, etc.), what in your opinion are the main results of regional conferences, workshops, and events? How, if at all, do these activities contribute to building government capacity? ● What, if any, ongoing support is there to embed the changes in the organization? If so, please describe. d. Of the achievements you described, are there any that you believe are the most important or relevant [to building government capacity]? If yes, why? e. [If needed, prompt using examples from documentation that were not mentioned] EQ 2 + EQ 2a (probe d) ii. In what ways, if any, did ISACC help support [country name] to implement country-led climate finance programs and projects (this includes support for proposal submission to the Green Climate Fund (GCF) or Adaptation Fund (AF), support for the accreditation process, support for funding management, etc.)? Please describe how. a. What facilitated and/or hindered achievements in this area? b. What role did ISACC play relative to other programs or influencing factors? c. Which specific activities were successful in contributing to these achievements [seek specific examples in targeted countries]. What led you to this conclusion? ● Please provide some examples (if any) of sustainable successes in this area (i.e., built capacity around proposal development/accreditation that is retained over time, improvements in M&E frameworks, stakeholder engagements, project management protocols, etc.). ● In your opinion, was the level and type of support (direct funding, technical assistance, etc.) provided by ISACC to governments well matched to contribute to achievements in this area? Why or why not? Does this differ significantly by country? d. Do you see improvements in the way projects are implemented? If yes, in what way? ● [if yes to the previous question]: In what ways has this improved implementation (or will) at the community level? If so, how? e. [Check stakeholders list and if the relevant activity relates to training/capacity building]: Are there plans to follow up with further training/capacity building to maintain skills and/or train new staff? f. Of the achievements you described, which, if any, would you say are the most important or relevant [to supporting country-led finance projects]? Why? g. [If needed, prompt using examples from documentation that were not mentioned] EQ 3 USAID.GOV DRAFT EVALUATION REPORT | 94 iii. In what ways, if any, did ISACC help support [country name] to engage the private sector in climate adaptation and disaster preparedness? Please describe how. a. What facilitated and/or hindered achievements in this area? b. What role did ISACC play relative to other programs or influencing factors? c. Which specific activities were successful in contributing to these achievements [seek specific examples in targeted countries]. What led you to this conclusion? ● Please provide some examples (if any) of sustainable successes in this area (i.e., private sector now participating in climate dialogues or proactively planning). ● In your opinion, was the level and type of engagement (mainly workshops and conferences) with the private sector well matched to contribute to achievements in this area? Why or why not? Does this differ significantly by country? d. Of the achievements you described, which, if any, would you say are the most important or relevant [to engaging the private sector]? Why? e. [If needed, prompt using examples from documentation that were not mentioned] EQ 4 Climate Ready i. In what ways, if any, did Ready help support countries seeking accreditation to entities like the Green Climate Fund (GCF) or Adaptation Fund (AF)? a. What facilitated and/or hindered achievements in this area? b. What role did Ready play relative to other programs or influencing factors? ● In what ways did Ready contribute to speeding up the accreditation process or make the difference between being accredited or not? c. Which specific activities were successful in contributing to these achievements [seek specific examples in targeted countries]. What led you to this conclusion? ● Please provide some examples (if any) of sustainable successes in this area (i.e., built capacity around accreditation processes that is retained over time). d. Of the achievements you described, which, if any, would you say are the most important or relevant [to supporting accreditation]? Why? e. [If needed, prompt using examples from documentation that were not mentioned] EQ 3 ii. In what ways, if any, did Ready help support countries seeking to submit funding proposals to entities like the Green Climate Fund (GCF) or Adaptation Fund (AF)? a. What facilitated and/or hindered achievements in this area? b. What role did Ready play relative to other programs or influencing factors? ● What specific elements of the proposal did Ready contribute to? How did Ready contribute more broadly to the proposal (e.g., by taking a more convening role in developing the proposal)? Did other international organizations contribute to the proposal and if so, did they work with Ready or on separate elements? ● If known, why was the proposal successful/unsuccessful? c. Which specific activities were successful in contributing to these achievements [seek specific examples in targeted countries]. What led you to this conclusion? ● Please provide some examples (if any) of sustainable successes in this area (i.e., built capacity around proposal development that is retained over time). d. Of the achievements you described, which, if any, would you say are the most important or relevant [to supporting proposal development]? Why? e. [If needed, prompt using examples from documentation that were not mentioned] EQ 3 USAID.GOV DRAFT EVALUATION REPORT | 95 iii. In what ways, if any, did Ready help support [country name] to implement country-led climate finance programs and projects? If yes, please describe how. a. What facilitated and/or hindered achievements in this area? b. What role did Ready play relative to other programs or influencing factors? c. Which specific activities were successful in contributing to these achievements [seek specific examples in targeted countries]. What led you to this conclusion? ● Please provide some examples (if any) of sustainable successes in this area (i.e., improvements in the way climate adaptation projects are or will be implemented, including M&E frameworks, stakeholder engagement, support to applicants for established programs, project management protocols, asset management strategies, etc.). d. Do you see improvements in the way projects are implemented? If yes, in what way? ● [if yes to the previous question]: In what ways has this improved implementation (or will) at the community level? If so, how? e. [Check stakeholders list and if the relevant activity relates to training/capacity building]: Are there plans to follow up with further training/capacity building to maintain skills and/or train new staff? f. Of the achievements you described, which, if any, would you say are the most important or relevant [to supporting country-led finance projects]? Why? g. [If needed, prompt using examples from documentation that were not mentioned] EQ 3 iv. In what ways, if any, did Ready help support engagement of the private sector the private sector in climate adaptation and disaster preparedness? If yes, please describe how. a. What facilitated and/or hindered achievements in this area? b. What role did Ready play relative to other programs or influencing factors? c. Which specific activities were successful in contributing to these achievements [seek specific examples in targeted countries]. What led you to this conclusion? ● Please provide some examples (if any) of sustainable successes in this area (i.e., ongoing/sustainable engagement from the private sector). ● What evidence are you seeing of the private sector engaging beyond the people you worked with directly? [If needed: what would you look for?] d. Of the achievements you described, which, if any, would you say are the most important or relevant [to engaging the private sector]? Why? e. [If needed, prompt using examples from documentation that were not mentioned] EQ 4 3. Given the ISACC and Climate Ready activities and results just discussed, to what extent have these aligned with your [organization’s/group’s] priorities relating to accessing climate finance and implementing projects? a. Are there other key ways (ISACC/Ready) supported your country not already covered? All Facilitating GESI 4. Consideration and representation of women and marginalized groups is of particular interest to USAID. Once again, I would like to ask a few questions relating to ISACC first, then Ready.40F 60 ISACC 60 As above, only ask about both Activities if the stakeholder list indicates involvement in both; otherwise, ask the questions for the relevant Activity. USAID.GOV DRAFT EVALUATION REPORT | 96 i. In what ways, if any, were gender and social inclusion consciously considered in the ISACC activities that you are familiar with? If yes, what types of activities were most effective in promoting consideration of gender and social inclusion? What led you to this conclusion? a. Were gender and social inclusion considered in all activities or in a smaller proportion? b. [Seek specific examples in targeted countries, looking particularly for examples related to: Enabling environment (governmental policies, organizational structures, standard operating procedures, plans, frameworks, legislation, etc.); and organizational capacity building, e.g., through GESI training] ● [For capacity building, probe for examples (if any) beyond inclusion of women or members of other marginalized groups in trainings, workshops, and events] c. If gender and social inclusion were not consciously considered in the activities, why not? [If needed: Social inclusion can be defined as the process of improving the terms on which individuals and groups take part in society. In the context of this question, this means supporting both men and women, as well as members of marginalized groups, to take an active role in climate change initiatives that impact them.] EQ 1b + EQ 2b ii. Have you noted any changes stemming from the activities you mentioned that may have helped to increase inclusivity of any marginalized groups? If yes, please describe. (Please break out examples by different groups as relevant for the activities described above e.g., women, indigenous peoples, outlying islands, migrant populations, other marginalized groups) a. What facilitated inclusivity? b. What hindered inclusivity? c. What role, if any, did ISACC play relative to other programs or influencing factors? d. [If needed, prompt using examples from documentation] e. [If needed, prompt using examples from key activities mentioned above] EQ 1b + EQ 2b Climate Ready i. In what ways, if any, were gender and social inclusion consciously considered in the Ready activities that you are familiar with? If yes, what types of activities were most effective in promoting consideration of gender and social inclusion? What led you to this conclusion? a. Were gender and social inclusion considered in all activities or in a smaller proportion? b. [Seek specific examples in targeted countries, looking particularly for examples related to: Enabling environment (governmental policies, organizational structures, standard operating procedures, plans, frameworks, legislation, etc.); and organizational capacity building, e.g., through GESI training] ● [For capacity building, probe for examples (if any) beyond inclusion of women or members of other marginalized groups in trainings, workshops, and events] c. If gender and social inclusion were not consciously considered in the activities, why not? [If needed: Social inclusion can be defined as the process of improving the terms on which individuals and groups take part in society. In the context of this question, this means supporting both men and women, as well as members of marginalized groups, to take an active role in climate change initiatives that impact them.] EQ 1b + EQ 2b ii. Have you noted any changes stemming from the activities you mentioned that may have helped to increase inclusivity of any marginalized groups? If yes, please describe. (Please break out examples by different groups as relevant for the activities described above e.g., women, indigenous peoples, outlying islands, migrant populations, other marginalized groups) a. What facilitated inclusivity? b. What hindered inclusivity? c. What role, if any, did Ready play relative to other programs or influencing factors? d. [If needed, prompt using examples from documentation] e. [If needed, prompt using examples from key activities mentioned above] USAID.GOV DRAFT EVALUATION REPORT | 97 5. What suggestions do you have for specific types of support programs like (ISACC/Ready) could provide that would help countries more deeply represent women and marginalized groups in future activities? a. What other key successes or challenges (in this/these countries) relating to promoting gender and/or social inclusion in climate change would you like to share? EQ 1b + EQ 2b Coordination and synergies 6. We recognize that there are more programs and organizations offering similar services [related to climate finance] than there were when ISACC and Ready began. We would like to discuss coordination with some of these programs and organizations. i. To your knowledge, what services related to climate financing are others providing? How much overlap or synergies are there with what other donors and programs are offering currently? a. [If needed] probe on other US initiatives as well as of other major players like Australia, New Zealand, Germany, Japan, and UN organizations or others. All ii. Please briefly describe your active coordination efforts with other donors or implementing parties. a. Are there other organizations you are coordinating or collaborating with relating to providing support to countries we should know about? [This refers to coordination or partnerships to provide support (to PICs) not organizations that are receiving direct support.] b. In what ways has this coordination impacted your organization’s ability to (if so, how?): ● Influence the enabling environment (governmental policies, organizational structures, standard operating procedures, plans, frameworks, legislation, etc.) ● Build capacity of government ● Support implementation of country-owned climate change actions ● Engage the private sector Looking ahead 7. As this will help USAID plan for the future, we are also interested in your views on how needs have evolved. i. Of the different types of support programs like ISACC and Ready could realistically offer, which do you think should be prioritized going forward? And under what conditions? a. [If needed, prompt for examples from key activities mentioned above] ● Examples include: funding proposal support, training, policy development support, accreditation support, workshops, conferences, and events, direct funding of government offices, etc. b. In your opinion, are there any caveats or types of support that should be avoided? [e.g., only offer funding proposal support if there is sufficient political will]. If yes, please describe. All Closing 8. We are almost finished. Do you have any other closing comments or thoughts about what we talked about today? a [Ask any remaining important questions specifically relating to the program.] b. [If needed, update them on next steps/follow up actions.] All Thank you for your time! USAID.GOV DRAFT EVALUATION REPORT | 98 INTERVIEW GUIDE D: U.S./BILATERAL GOVERNMENT Interview ID number: Date: Name of Interviewer: Location of Interview (Country, City or Town): [Internal Instructions: Conduct introductions and ask for consent to record. Ask the interview participant to provide the below information] Name of Respondent: Organization: Position: Nationality: Sex: Consent to record: Yes/No [circle one] CORE QUESTIONS **Interviewer Note: Not all questions should be asked of all respondents. Tailor questions and ensure that interview can be completed within one hour. PROBES *Interviewer Note: Only use if issue is not addressed in response to core question or not already known EQ Relevant evaluation question Informant’s role and country context 1. We would like to begin by asking you about your work, particularly your involvement with climate adaptation activities and accessing climate finance in [your organization/group]? i. What is your role (in your agency/organization)? a. Please provide an overview of your general portfolio and day to day responsibilities related to climate finance or climate change projects All ii. Please provide a brief overview of any (ISACC/Ready) climate activities you have been directly involved with or know relating to your [agency/organization]. [Internal: Have organization/group notes at hand to prompt if needed; keep this section brief.] a. [As feasible, ask the informant to differentiate (ISACC/Ready) activities and other activities he/she may have participated in.] b. [Internal: Clarify any key details relating to the organization’s/group’s progress and activities not already clear from the documentation or if there is important relevant documentation that has not yet been shared, e.g., mission statements, activity highlights, case studies, etc.] iii. How would you characterize your [agency/organization] activities relating to helping countries access climate finance and implement climate adaptation projects? a. [If needed: Use examples to prompt respondents: may be technical, e.g., providing finance for projects, technical support to develop funding proposals, increasing capacity of local governments to administer projects, etc.; or topical, e.g., focusing on impacts of sea level rise.] b. In general, what would you say are the top three factors (either internally or due to external support) that contributed to your [agency/organization] work in this area so far? Key achievements and challenges USAID.GOV DRAFT EVALUATION REPORT | 99 2.       USAID is interested in understanding more about what went well as well as what was less effective regarding ISACC and Ready. As we are going to cover two programs I am going to ask about some types of activities for each program in turn. * *Ask about both Activities if the stakeholder list indicates involvement in both; otherwise, only ask the questions for the relevant Activity. Use the stakeholder list and background materials to determine which outcome areas are most relevant for the stakeholder (regional collaboration, capacity building, private sector engagement, etc.); focus on the questions in this area and lightly touch the others to ensure that no information is missed. More than one outcome area may be relevant for some stakeholders. ISACC i. In what ways, if any, did ISACC help support the development of governmental policies, organizational structures, plans, frameworks, legislation, etc., in [country name] related to accessing climate finance? Please describe how. a. What facilitated and/or hindered achievements in this area? b. What role did ISACC play relative to other programs or influencing factors? c. Which specific activities were successful in contributing to these achievements [seek specific examples in targeted countries]. What led you to this conclusion? ● Please provide some examples (if any) of sustainable successes in this area (i.e., lasting changes to policies or organizational structures). ● [If regional conferences, events, and workshops are mentioned]: Besides production of formal outputs (conference presentations, drafted documents, etc.), what in your opinion are the main results of regional conferences, workshops, and events? How, if at all, do these activities contribute to improving the enabling environment for accessing climate finance? ● In your opinion, was the level and type of support (direct funding, technical assistance, etc.) provided by ISACC to governments well matched to contribute to achievements in this area? Why or why not? Does this differ significantly by country? d. Of the achievements you described, which, if any, would you say are the most important or relevant [to improving the enabling environment for accessing climate financing]? Why? e. [If needed, prompt using examples from documentation that were not mentioned] EQ 1 + EQ 1a (probe d) ii. In what ways, if any, did ISACC contribute to improving the capacity of government to implement climate change initiatives (this includes project management, monitoring and evaluation [M&E], knowledge management, etc.)? Please describe how. * *Ask for USAID interviews only; omit for other US government or bilateral stakeholders. a. What facilitated and/or hindered achievements in this area? b. What role did ISACC play relative to other programs or influencing factors? c. Which specific activities were successful in contributing to these achievements [seek specific examples in targeted countries]. What led you to this conclusion? ● In your opinion, which topics (M&E, knowledge management, etc.) or formats (shorter workshops or trainings, university-sponsored courses, embedded technical assistance, short-term technical assistance/consultants, etc.) were more effective than others in supporting capacity building? What led you to this conclusion? ● [If regional conferences, events, and workshops are mentioned]: Besides production of formal outputs (conference presentations, drafted documents, etc.), what in your opinion are the main results of regional conferences, workshops, and events? How, if at all, do these activities contribute to building government capacity? ● What, if any, ongoing support is there to embed the changes in the organization? If so, please describe. d. Of the achievements you described, are there any that you believe are the most important or relevant [to building government capacity]? If yes, why? e. [If needed, prompt using examples from documentation that were not mentioned] EQ 2 + EQ 2a (probe d) iii. In what ways, if any, did ISACC help support [country name] to implement country-led climate finance programs and projects (this includes support for proposal submission to the Green Climate Fund (GCF) or a. What facilitated and/or hindered achievements in this area? b. What role did ISACC play relative to other programs or influencing factors? c. Which specific activities were successful in contributing to these achievements [seek specific examples in targeted countries]. What led you to this conclusion? EQ 3 USAID.GOV DRAFT EVALUATION REPORT | 100 Adaptation Fund (AF), support for the accreditation process, support for funding management, etc.)? Please describe how. ● Please provide some examples (if any) of sustainable successes in this area (i.e., built capacity around proposal development/accreditation that is retained over time, improvements in M&E frameworks, stakeholder engagements, project management protocols, etc.). ● In your opinion, was the level and type of support (direct funding, technical assistance, etc.) provided by ISACC to governments well matched to contribute to achievements in this area? Why or why not? Does this differ significantly by country? d. Do you see improvements in the way projects are implemented? If yes, in what way? ● [if yes to the previous question]: In what ways has this improved implementation (or will) at the community level? If so, how? e. [Check stakeholders list and if the relevant activity relates to training/capacity building]: Are there plans to follow up with further training/capacity building to maintain skills and/or train new staff? f. Of the achievements you described, which, if any, would you say are the most important or relevant [to supporting country-led finance projects]? Why? g. [If needed, prompt using examples from documentation that were not mentioned] iv. In what ways, if any, did ISACC help support [country name] to engage the private sector in climate adaptation and disaster preparedness? Please describe how. a. What facilitated and/or hindered achievements in this area? b. What role did ISACC play relative to other programs or influencing factors? c. Which specific activities were successful in contributing to these achievements [seek specific examples in targeted countries]. What led you to this conclusion? ● Please provide some examples (if any) of sustainable successes in this area (i.e., private sector now participating in climate dialogues or proactively planning). ● In your opinion, was the level and type of engagement (mainly workshops and conferences) with the private sector well matched to contribute to achievements in this area? Why or why not? Does this differ significantly by country? d. Of the achievements you described, which, if any, would you say are the most important or relevant [to engaging the private sector]? Why? e. [If needed, prompt using examples from documentation that were not mentioned] EQ 4 Climate Ready i. In what ways, if any, did Ready help support the development of governmental policies, organizational structures, plans, frameworks, legislation, etc., in [country name] related to accessing climate finance? Please describe how. a. What facilitated and/or hindered achievements in this area? b. What role did Ready play relative to other programs or influencing factors? ● Please provide some examples (if any) of sustainable successes in this area (i.e., lasting changes to policies or organizational structures). c. Which specific activities were successful in contributing to these achievements [seek specific examples in targeted countries]. What led you to this conclusion? d. Of the achievements you described, which, if any, would you say are the most important or relevant [to improving the enabling environment for accessing climate financing]? Why? e. [If needed, prompt using examples from documentation that were not mentioned] EQ 1 + EQ 1a (probe d) ii. In what ways, if any, did Ready help support countries seeking accreditation to entities like the Green Climate Fund (GCF)or Adaptation Fund (AF)? a. What facilitated and/or hindered achievements in this area? b. What role did Ready play relative to other programs or influencing factors? ● In what ways did Ready contribute to speeding up the accreditation process or make the difference between being accredited or not? c. Which specific activities were successful in contributing to these achievements [seek specific examples in targeted countries]. What led you to this conclusion? EQ 3 USAID.GOV DRAFT EVALUATION REPORT | 101 ● Please provide some examples (if any) of sustainable successes in this area (i.e., built capacity around accreditation processes that is retained over time). d. Of the achievements you described, which, if any, would you say are the most important or relevant [to supporting accreditation]? Why? e. [If needed, prompt using examples from documentation that were not mentioned] iii. In what ways, if any, did Ready help support countries seeking to submit funding proposals to entities like the Green Climate Fund (GCF) or Adaptation Fund (AF)? a. What facilitated and/or hindered achievements in this area? b. What role did Ready play relative to other programs or influencing factors? ● What specific elements of the proposal did Ready contribute to? How did Ready contribute more broadly to the proposal (e.g., by taking a more convening role in developing the proposal)? Did other international organizations contribute to the proposal and if so, did they work with Ready or on separate elements? ● If known, why was the proposal successful/unsuccessful? c. Which specific activities were successful in contributing to these achievements [seek specific examples in targeted countries]. What led you to this conclusion? ● Please provide some examples (if any) of sustainable successes in this area (i.e., built capacity around proposal development that is retained over time). d. Of the achievements you described, which, if any, would you say are the most important or relevant [to supporting proposal development]? Why? e. [If needed, prompt using examples from documentation that were not mentioned] EQ 3 iv. In what ways, if any, did Ready help support [country name] to implement country-led climate finance programs and projects? If yes, please describe how. a. What facilitated and/or hindered achievements in this area? b. What role did Ready play relative to other programs or influencing factors? c. Which specific activities were successful in contributing to these achievements [seek specific examples in targeted countries]. What led you to this conclusion? ● Please provide some examples (if any) of sustainable successes in this area (i.e., improvements in the way climate adaptation projects are or will be implemented, including M&E frameworks, stakeholder engagement, support to applicants for established programs, project management protocols, asset management strategies, etc.). d. Do you see improvements in the way projects are implemented? If yes, in what way? ● [if yes to the previous question]: In what ways has this improved implementation (or will) at the community level? If so, how? e. [Check stakeholders list and if the relevant activity relates to training/capacity building]: Are there plans to follow up with further training/capacity building to maintain skills and/or train new staff? f. Of the achievements you described, which, if any, would you say are the most important or relevant [to supporting country-led finance projects]? Why? g. [If needed, prompt using examples from documentation that were not mentioned] EQ 3 v. In what ways, if any, did Ready help support engagement of the private sector the private sector in climate adaptation and disaster preparedness? If yes, please describe how. a. What facilitated and/or hindered achievements in this area? b. What role did Ready play relative to other programs or influencing factors? c. Which specific activities were successful in contributing to these achievements [seek specific examples in targeted countries]. What led you to this conclusion? ● Please provide some examples (if any) of sustainable successes in this area (i.e., ongoing/sustainable engagement from the private sector). ● What evidence are you seeing of the private sector engaging beyond the people you worked with directly? [If needed: what would you look for?] d. Of the achievements you described, which, if any, would you say are the most important or relevant [to engaging the private sector]? Why? e. [If needed, prompt using examples from documentation that were not mentioned] EQ 4 USAID.GOV DRAFT EVALUATION REPORT | 102 3. Given the ISACC and Climate Ready activities and results just discussed, to what extent have these aligned with your [agency/organization] priorities relating to accessing climate finance and implementing projects? a. Are there other key ways (ISACC/Ready) supported your [agency/organization] not already covered? b. Please describe any support that your [agency/organization] has provided that is similar to (ISACC/Ready) activities related to: development of governmental policies, organizational structures, plans, frameworks, legislation, etc.,; improving the capacity of government; support countries seeking to submit funding proposals or seeking accreditation; supporting the implementation of country-led climate finance programs and projects; or engagement of the private sector. ● In general, does your work overlap with or is more distinct from (ISACC/Ready) activities? All Facilitating GESI 4. Consideration and representation of women and marginalized groups is of particular interest to USAID. Once again, I would like to ask a few questions relating to ISACC first, then Ready. * * As above, only ask about both Activities if the stakeholder list indicates involvement in both; otherwise, ask the questions for the relevant Activity. ISACC i. In what ways, if any, were gender and social inclusion consciously considered in the ISACC activities that you are familiar with? If yes, what types of activities were most effective in promoting consideration of gender and social inclusion? What led you to this conclusion? a. Were gender and social inclusion considered in all activities or in a smaller proportion? b. [Seek specific examples in targeted countries, looking particularly for examples related to: Enabling environment (governmental policies, organizational structures, standard operating procedures, plans, frameworks, legislation, etc.); and organizational capacity building, e.g., through GESI training] ● [For capacity building, probe for examples (if any) beyond inclusion of women or members of other marginalized groups in trainings, workshops, and events] c. If gender and social inclusion were not consciously considered in the activities, why not? [If needed: Social inclusion can be defined as the process of improving the terms on which individuals and groups take part in society. In the context of this question, this means supporting both men and women, as well as members of marginalized groups, to take an active role in climate change initiatives that impact them.] EQ 1b + EQ 2b ii. Have you noted any changes stemming from the activities you mentioned that may have helped to increase inclusivity of any marginalized groups? If yes, please describe. (Please break out examples by different groups as relevant for the activities described above e.g., women, indigenous peoples, outlying islands, migrant populations, other marginalized groups) a. What facilitated inclusivity? b. What hindered inclusivity? c. What role, if any, did ISACC play relative to other programs or influencing factors? d. [If needed, prompt using examples from documentation] e. [If needed, prompt using examples from key activities mentioned above] EQ 1b + EQ 2b Climate Ready USAID.GOV DRAFT EVALUATION REPORT | 103 i. In what ways, if any, were gender and social inclusion consciously considered in the Ready activities that you are familiar with? If yes, what types of activities were most effective in promoting consideration of gender and social inclusion? What led you to this conclusion? a. Were gender and social inclusion considered in all activities or in a smaller proportion? b. [Seek specific examples in targeted countries, looking particularly for examples related to: Enabling environment (governmental policies, organizational structures, standard operating procedures, plans, frameworks, legislation, etc.); and organizational capacity building, e.g., through GESI training] ● [For capacity building, probe for examples (if any) beyond inclusion of women or members of other marginalized groups in trainings, workshops, and events] c. If gender and social inclusion were not consciously considered in the activities, why not? [If needed: Social inclusion can be defined as the process of improving the terms on which individuals and groups take part in society. In the context of this question, this means supporting both men and women, as well as members of marginalized groups, to take an active role in climate change initiatives that impact them.] EQ 1b + EQ 2b ii. Have you noted any changes stemming from the activities you mentioned that may have helped to increase inclusivity of any marginalized groups? If yes, please describe. (Please break out examples by different groups as relevant for the activities described above e.g., women, indigenous peoples, outlying islands, migrant populations, other marginalized groups) a. What facilitated inclusivity? b. What hindered inclusivity? c. What role, if any, did Ready play relative to other programs or influencing factors? d. [If needed, prompt using examples from documentation] e. [If needed, prompt using examples from key activities mentioned above] 5. What suggestions do you have for specific types of support programs like (ISACC/Ready) could provide that would help countries more deeply represent women and marginalized groups in future activities? a. What other key successes or challenges (in this/these countries) relating to promoting gender and/or social inclusion in climate change would you like to share? EQ 1b + EQ 2b Coordination and synergies 6. We recognize that there are more programs and organizations offering similar services [related to climate finance] than there were when ISACC and Ready began. We would like to discuss coordination with some of these programs and organizations. i. To your knowledge, what services related to climate financing are others providing? How much overlap or synergies are there with what other donors and programs are offering currently? a. [If needed probe on other US initiatives as well as of other major players like Australia, New Zealand, Germany, Japan, and UN organizations or others.] All USAID.GOV DRAFT EVALUATION REPORT | 104 ii. Please briefly describe your active coordination efforts with the (ISACC/Ready) team as well as with other donors. a. Are there other organizations you are coordinating or collaborating with relating to providing support to countries we should know about? [This refers to coordination or partnerships to provide support (to PICs) not organizations that are receiving direct support.] b. In what ways has this coordination impacted your agency’s/organization’s ability to (if so, how?): ● Influence the enabling environment (governmental policies, organizational structures, standard operating procedures, plans, frameworks, legislation, etc.) ● Build capacity of government ● Support implementation of country-owned climate change actions ● Engage the private sector Looking ahead 7. As this will help USAID plan for the future, we are also interested in your views on how needs have evolved. i. Of the different types of support programs like ISACC and Ready could realistically offer, which do you think should be prioritized going forward? And under what conditions? a. [If needed, prompt for examples from key activities mentioned above] ● Examples include: funding proposal support, training, policy development support, accreditation support, workshops, conferences, and events, direct funding of government offices, etc. b. In your opinion, are there any caveats or types of support that should be avoided? [e.g., only offer funding proposal support if there is sufficient political will]. If yes, please describe. All Closing 8. We are almost finished. Do you have any other closing comments or thoughts about what we talked about today? a [Ask any remaining important questions specifically relating to the program.] b. [If needed, update them on next steps/follow up actions.] All Thank you for your time! USAID.GOV DRAFT EVALUATION REPORT | 105 INTERVIEW GUIDE E: INTERNATIONAL/INTERGOVERNMENTAL INSTITUTIONS Interview ID number: Date: Name of Interviewer: Location of Interview (Country, City or Town): [Internal Instructions: Conduct introductions and ask for consent to record. Ask the interview participant to provide the below information] Name of Respondent: Organization: Position: Nationality: Sex: Consent to record: Yes/No [circle one] CORE QUESTIONS **Interviewer Note: Not all questions should be asked of all respondents. Tailor questions and ensure that interview can be completed within one hour. PROBES *Interviewer Note: Only use if issue is not addressed in response to core question or not already known EQ Relevant evaluation question Informant’s role and country context 1. We would like to begin by asking you about your work, particularly your involvement with climate adaptation activities and accessing climate finance in [your organization/group]? i. What is your role (in your agency/organization)? a. Please provide an overview of your general portfolio and day to day responsibilities related to climate finance or climate change projects All ii. How familiar are you with (ISACC/Ready) climate activities? Please provide a brief overview of any (ISACC/Ready) activities you have been directly involved with or know relating to your [agency/organization]. [Internal: Have organization/group notes at hand to prompt if needed; keep this section brief.] a. [As feasible, ask the informant to differentiate (ISACC/Ready) activities and other activities he/she may have participated in.] ● [If needed:] How familiar are you with US government support for climate change adaptation and accessing climate finance in the region? b. [Internal: Clarify any key details relating to the agency’s/organization’s progress and activities not already clear from the documentation or if there is important relevant documentation that has not yet been shared, e.g., program briefs, activity highlights, etc.] iii. How would you characterize your [agency/organization] activities relating to helping countries access climate finance and implement climate adaptation projects? a. [If needed: Use examples to prompt respondents: may be technical, e.g., providing finance for projects, technical support to develop funding proposals, increasing capacity of local governments to administer projects, etc.; or topical, e.g., focusing on impacts of sea level rise.] Key achievements and challenges 2.       USAID is interested in understanding more about what went well as well as what was less effective regarding ISACC and Ready. As we are going to cover two programs I am going to ask about some types of activities for each program in turn. * *Ask about both Activities if the stakeholder list indicates involvement in both; otherwise, only ask the questions for the relevant Activity. Use the stakeholder list and background materials to determine which outcome areas are most relevant for the stakeholder (regional collaboration, capacity building, private sector engagement, etc.); focus on the questions in this area and lightly touch the others to ensure that no information is missed. More than one outcome area may be relevant for some stakeholders. ISACC* *For ISACC questions, only ask if the stakeholder indicated familiarity with the program(s) with from previous questions. If the representative is focused on a particular country or only a certain country or type of activity, it is acceptable to rephrase questions accordingly, e.g., for the GCF you might focus on accreditation and funding proposals. USAID.GOV DRAFT EVALUATION REPORT | 106 i. In what ways, if any, did ISACC help support [country name(s)] to implement country-led climate finance programs and projects (this includes support for proposal submission to the Green Climate Fund (GCF) or Adaptation Fund (AF), support for the accreditation process, support for funding management, etc.)? Please describe how. a. What facilitated and/or hindered achievements in this area? b. What role did ISACC play relative to other programs or influencing factors? c. Which specific activities were successful in contributing to these achievements [seek specific examples in targeted countries]. What led you to this conclusion? ● Please provide some examples (if any) of sustainable successes in this area (i.e., built capacity around proposal development/accreditation that is retained over time, improvements in M&E frameworks, stakeholder engagements, project management protocols, etc.). ● In your opinion, was the level and type of support (direct funding, technical assistance, etc.) provided by ISACC to governments well matched to contribute to achievements in this area? Why or why not? Does this differ significantly by country? d. Do you see improvements in the way projects are implemented? If yes, in what way? ● [if yes to the previous question]: In what ways has this improved implementation (or will) at the community level? If so, how? e. [Check stakeholders list and if the relevant activity relates to training/capacity building]: Are there plans to follow up with further training/capacity building to maintain skills and/or train new staff? f. Of the achievements you described, which, if any, would you say are the most important or relevant [to supporting country-led finance projects]? Why? g. [If needed, prompt using examples from documentation that were not mentioned] EQ 3 Climate Ready* * For Climate Ready questions, only ask if the stakeholder indicated familiarity with the program(s) with from previous questions (i.e., ask the questions on proposal development or accreditation only if the stakeholder has indicated familiarity with these). If the stakeholder is focused on a particular country or only a certain country or type of activity, it is acceptable to rephrase questions accordingly, e.g., for the GCF you might focus on accreditation and funding proposals. i. In what ways, if any, did Ready help support countries seeking accreditation to entities like the Green Climate Fund (GCF) or Adaptation Fund (AF)? a. What facilitated and/or hindered achievements in this area? b. What role did Ready play relative to other programs or influencing factors? ● In what ways did Ready contribute to speeding up the accreditation process or make the difference between being accredited or not? c. Which specific activities were successful in contributing to these achievements [seek specific examples in targeted countries]. What led you to this conclusion? ● Please provide some examples (if any) of sustainable successes in this area (i.e., built capacity around accreditation processes that is retained over time). d. Of the achievements you described, which, if any, would you say are the most important or relevant [to supporting accreditation]? Why? e. [If needed, prompt using examples from documentation that were not mentioned] EQ 3 USAID.GOV DRAFT EVALUATION REPORT | 107 ii. In what ways, if any, did Ready help support countries seeking to submit funding proposals to entities like the Green Climate Fund (GCF) or Adaptation Fund (AF)? a. What facilitated and/or hindered achievements in this area? b. What role did Ready play relative to other programs or influencing factors? ● What specific elements of the proposal did Ready contribute to? How did Ready contribute more broadly to the proposal (e.g., by taking a more convening role in developing the proposal)? Did other international organizations contribute to the proposal and if so, did they work with Ready or on separate elements? ● If known, why was the proposal successful/unsuccessful? c. Which specific activities were successful in contributing to these achievements [seek specific examples in targeted countries]. What led you to this conclusion? ● Please provide some examples (if any) of sustainable successes in this area (i.e., built capacity around proposal development that is retained over time). d. Of the achievements you described, which, if any, would you say are the most important or relevant [to supporting proposal development]? Why? e. [If needed, prompt using examples from documentation that were not mentioned] EQ 3 iii. In what ways, if any, did Ready help support [country name(s)] to implement country-led climate finance programs and projects? If yes, please describe how. a. What facilitated and/or hindered achievements in this area? b. What role did Ready play relative to other programs or influencing factors? c. Which specific activities were successful in contributing to these achievements [seek specific examples in targeted countries]. What led you to this conclusion? ● Please provide some examples (if any) of sustainable successes in this area (i.e., improvements in the way climate adaptation projects are or will be implemented, including M&E frameworks, stakeholder engagement, support to applicants for established programs, project management protocols, asset management strategies, etc.). d. Do you see improvements in the way projects are implemented? If yes, in what way? ● [if yes to the previous question]: In what ways has this improved implementation (or will) at the community level? If so, how? e. [Check stakeholders list and if the relevant activity relates to training/capacity building]: Are there plans to follow up with further training/capacity building to maintain skills and/or train new staff? f. Of the achievements you described, are there any that you believe are the most important or relevant [to supporting country-led finance projects]? If yes, why? g. [If needed, prompt using examples from documentation that were not mentioned] EQ 3 Facilitating GESI 4. Consideration and representation of women and marginalized groups is of particular interest to USAID. Once again, I would like to ask a few questions relating to ISACC first, then Ready. * *As above, only ask about both Activities if the stakeholder list indicates involvement in both; otherwise, ask the questions for the relevant Activity. ISACC USAID.GOV DRAFT EVALUATION REPORT | 108 i. In what ways, if any, were gender and social inclusion consciously considered in the ISACC activities that you are familiar with? If yes, what types of activities were most effective in promoting consideration of gender and social inclusion? What led you to this conclusion? a. Were gender and social inclusion considered in all activities or in a smaller proportion? b. [Seek specific examples in targeted countries, looking particularly for examples related to: Enabling environment (governmental policies, organizational structures, standard operating procedures, plans, frameworks, legislation, etc.); and organizational capacity building, e.g., through GESI training] ● [For capacity building, probe for examples (if any) beyond inclusion of women or members of other marginalized groups in trainings, workshops, and events] c. If gender and social inclusion were not consciously considered in the activities, why not? [If needed: Social inclusion can be defined as the process of improving the terms on which individuals and groups take part in society. In the context of this question, this means supporting both men and women, as well as members of marginalized groups, to take an active role in climate change initiatives that impact them.] EQ 1b + EQ 2b ii. Have you noted any changes stemming from the activities you mentioned that may have helped to increase inclusivity of any marginalized groups? If yes, please describe. (Please break out examples by different groups as relevant for the activities described above e.g., women, indigenous peoples, outlying islands, migrant populations, other marginalized groups) a. What facilitated inclusivity? b. What hindered inclusivity? c. What role, if any, did ISACC play relative to other programs or influencing factors? d. [If needed, prompt using examples from documentation] e. [If needed, prompt using examples from key activities mentioned above] EQ 1b + EQ 2b Climate Ready i. In what ways, if any, were gender and social inclusion consciously considered in the Ready activities that you are familiar with? If yes, what types of activities were most effective in promoting consideration of gender and social inclusion? What led you to this conclusion? a. Were gender and social inclusion considered in all activities or in a smaller proportion? b. [Seek specific examples in targeted countries, looking particularly for examples related to: Enabling environment (governmental policies, organizational structures, standard operating procedures, plans, frameworks, legislation, etc.); and organizational capacity building, e.g., through GESI training] ● [For capacity building, probe for examples (if any) beyond inclusion of women or members of other marginalized groups in trainings, workshops, and events] c. If gender and social inclusion were not consciously considered in the activities, why not? [If needed: Social inclusion can be defined as the process of improving the terms on which individuals and groups take part in society. In the context of this question, this means supporting both men and women, as well as members of marginalized groups, to take an active role in climate change initiatives that impact them.] EQ 1b + EQ 2b ii. Have you noted any changes stemming from the activities you mentioned that may have helped to increase inclusivity of any marginalized groups? If yes, please describe. (Please break out examples by different groups as relevant for the activities described above e.g., women, indigenous peoples, outlying islands, migrant populations, other marginalized groups) a. What facilitated inclusivity? b. What hindered inclusivity? c. What role, if any, did Ready play relative to other programs or influencing factors? d. [If needed, prompt using examples from documentation] e. [If needed, prompt using examples from key activities mentioned above] USAID.GOV DRAFT EVALUATION REPORT | 109 5. What suggestions do you have for specific types of support programs like (ISACC/Ready) could provide that would help countries more deeply represent women and marginalized groups in future activities? a. What other key successes or challenges (in this/these countries) relating to promoting gender and/or social inclusion in climate change would you like to share? EQ 1b + EQ 2b Coordination and synergies 6. We recognize that there are more programs and organizations offering similar services [related to climate finance] than there were when ISACC and Ready began in 2015. We would like to discuss coordination with some of these programs and organizations. i. To your knowledge, what services related to climate financing are others providing? How much overlap or synergies are there with what other donors and programs are offering currently? a. [If needed probe on other US initiatives as well as of other major players like Australia, New Zealand, Germany, Japan, and UN organizations or others.] ii. Please briefly describe your active coordination All efforts with the (ISACC/Ready) team as well as other international entities or donors. a. Are there other organizations you are coordinating or collaborating with relating to providing support to countries we should know about? [This refers to coordination or partnerships to provide support (to PICs) not organizations that are receiving direct support.] b. In what ways has this coordination impacted your agency’s/organization’s ability to (if so, how?): ● Influence the enabling environment (governmental policies, organizational structures, standard operating procedures, plans, frameworks, legislation, etc.) ● Build capacity of government ● Support implementation of country-owned climate change actions ● Engage the private sector Looking ahead 7. As this will help USAID plan for the future, we are also interested in your views on how needs have evolved. i. Of the different types of support programs like ISACC and Ready could realistically offer, which do you think should be prioritized going forward? And under what conditions? a. [If needed, prompt for examples from key activities mentioned above] ● Examples include: funding proposal support, training, policy development support, accreditation support, workshops, conferences, and events, direct funding of government offices, etc. b. In your opinion, are there any caveats or types of support that should be avoided? [e.g., only offer funding proposal support if there is sufficient political will]. If yes, please describe. All Closing 8. We are almost finished. Do you have any other closing comments or thoughts about what we talked about today? a [Ask any remaining important questions specifically relating to the program.] b. [If needed, update them on next steps/follow up actions.] All Thank you for your time! USAID.GOV DRAFT EVALUATION REPORT | 110 ANNEX H: LAWS, POLICIES, PLANS, PROCEDURES, ETC., INTRODUCED OR IMPLEMENTED Climate Ready (2018-2021) Tonga 1. Support the Department of Climate Change, MEIDECC to edit the JNAP II and facilitate its passage to Cabinet for endorsement (Indicator 1.1, D1) (TON-17-11001) – Tonga Joint National Action Plan II. PNG 2. Support to CCDA with implementation of GCF Readiness Grant (Indicator 1.1, 1.3 & 2.3, D4 & D5) (PNG-17-17002) – CCDA Corporate Plan. 3. USAID Climate Ready provided assistance to PNG’s CCDA in preparing the final draft and supporting the adoption of the CCDA Communications Strategy (Indicator 1.1, 1.3 & 2.3, D4 & D5) (PNG-17-17002) – CCDA Communications Strategy. 4. Finalize assistance to CCDA following submission of GCF Readiness Grant Proposal and provide follow-on support to CCDA with implementation of GCF Readiness Grant (PNG-17-12002) – CCDA Annual Operational Plan 2020. 5. Finalize assistance to CCDA following submission of GCF Readiness Grant Proposal and provide follow-on support to CCDA with implementation of GCF Readiness Grant (PNG-17-12002) – PNG GCF Country Programme. 6. Support CCDA to prepare the Adaptation Chapter and climate change adaptation targets for the Nationally Determined Contributions (NDC) to the UNFCCC (PNG-21-11004) – PNG Nationally Determined Contributions (Adaptation) Report. FSM 7. PCCFAF Climate Finance Assessment Report (Indicator 1.1, D1) (FSM-18-11001). 8. Support FSM NDA to develop a National Climate Change Communication Strategy and build capacity to implement it (FSM-21-11004). Kiribati 9. PCCFAF Climate Finance Assessment Report (KIR-17-11001). 10. Assist with establishment of Climate Finance Unit (KIR-17-12002) – Kiribati Climate Finance Division Standard Operating Procedures. 11. Support Kiribati Climate Finance Division to develop the Kiribati Strategic Framework and Country Programme (GCF) (Indicator 1.1 and 1.3, D1 & D4) (KIR-20-11003). Vanuatu 12. Provide support to NAB conducting a gap analysis of the Government of Vanuatu’s status in regard to GCF requirements for environmental and social safeguards (ESS) (VAN-18-11003) – ESS Report. 13. Support DoCC to draft the Implementation Plan for the National Climate Change and Disaster Risk Reduction Policy (VAN-20-11004). Solomon Islands USAID.GOV DRAFT EVALUATION REPORT | 111 14. Support the MECDM in undertaking Integrated Vulnerability Adaptation (IVA) Assessment and Reporting in provinces in support of the National Adaptation Plans (NAP) and the Solomon Islands Integrated Vulnerability Assessment Database (SIIVA) (Indicator 1.1 & 1.3, D1 & D4) (SOL-21- 11003) - Solomon Islands Integrated Vulnerability and Adaptation Assessment Report (Temotu Province). Tuvalu 15. Support Tuvalu CCD to prepare a National Adaptation Plan Framework (Indicator 1.1 and 1.3, D1 & D4) (TUV-20-11001) - National Adaptation Plan Framework. Regional 16. Support to MCT to prepare and submit the project proposal 'Preparing the Enabling Environment and Building Organizational Capacity for the Implementing and Executing Agencies in Micronesia' through GCF's Enhanced Direct Access facility (FSM, Palau and RMI) (GCF, US$10m) (REG-19- 12006) – MCT Strategic Plan ISACC (2017-2020) Fiji: 1. Implementation support for the Fiji National Climate Change Policy; supporting activities conducted in 2018 2. Implementation support for the Greengrowth Framework for Fiji; supporting activities conducted in 2018 FSM: 3. Implementation support for the Nationwide Climate Change Policy (2009); supporting activities conducted in 2017 and 2018 Kiribati: 4. Implementation support for the National Development Plan (2016 – 2019); supporting activities conducted in 2017 and 2018 5. Implementation support for the Kiribati Joint Implementation Plan (KJIP); supporting activities conducted in 2017 and 2018 Palau: 6. Implementation support for the Palau Climate Change Policy (2015); supporting activities conducted in 2017 and 2019 7. Implementation support for the Palau Nationally Determined Contributions (NDCs); supporting activities conducted in 2019 8. Implementation support for the Palau Voluntary National Report (VNR); supporting activities conducted in 2019 Regional: 9. Implementation support for SPC and PIFS Finance and Procurement Policies; supporting activities conducted in 2018 10. Implementation support for the PIFS Procurement Policy 11. Implementation support for the Framework for Resilient Development in the Pacific (FRDP); supporting activities conducted in 2019 and 2020 USAID.GOV DRAFT EVALUATION REPORT | 112 12. Implementation support for the Framework for Resilient Development in the Pacific (FRDP) M&E Strategy Samoa: 13. Drafting support for Strategy for Development of Samoa (2016-19/20); supporting activities conducted in 2017 and 2019 14. Implementation support for the Samoa National Policy of Combating Climate Change (2007); supporting activities conducted in 2017 and 2019 15. Implementation support for the Samoa Nationally Determined Contributions (NDC); supporting activities conducted in 2019 16. Implementation support for the, the National Environment Sector Plan (2017-2021); supporting activities conducted in 2019 17. Implementation support for the Energy Sector Plan (2017-2022); supporting activities conducted in 2019 Solomon Islands: 18. Implementation support for National Climate Change Policy (2012-2017); supporting activities conducted in 2017, 2018, and 2020 19. Implementation support for the National Development Strategy; supporting activities conducted in 2020 20. Implementation support for the National Disaster Management Plan; supporting activities conducted in 2020 21. Implementation support for the Solomon Islands Nationally Determined Contribution (NDC); supporting activities conducted in 2020 Tuvalu: 22. Implementation support for the Tuvalu Climate Change Policy (Te Kaniva) (2012-2021); supporting activities conducted in 2017 and 2018 23. Implementation support for the Tuvalu National Strategic Action Plan for Climate Change and Disaster Risk Management (2012-2016); supporting activities conducted in 2017 and 2018 Vanuatu: 24. Implementation support for Climate Change and Disaster Risk Reduction Policy (2016- 2030); supporting activities conducted in 2017 and 2018 25. Implementation support for the Vanuatu Climate Finance Roadmap; supporting activities conducted in 2017 and 2018 26. Implementation support for the Department of Climate Change Strategic Plan; supporting activities conducted in 2020