Web-basedShortfall FINAL PERFORMANCE EVALUATION USAID ² ADAPTASI PERUBAHAN IKLIM DAN KETANGGUHAN (APIK) USAID Contract No. AID-497-C-16-00006 Indonesia Monitoring & Evaluation Support Project July 2020 Prepared by: Richard Volk Saut Sagala Retno Handini Indradi Soemardjan David Callihan Irma Sitompul Front cover photos: ‹ USAID-APIK Top right: Corn farmer in Malang District, East Java Bottom left: Fisherman in Maluku Bottom right: Disaster response exercises in Malang District, East Java i | FINAL PERFORMANCE EVALUATION USAID-APIK USAID.GOV ACKNOWLEDGMENT In pulling together this evaluation we are grateful for the generous contributions of time and effort from so many people. Foremost among these are the numerous program stakeholders³private citizens and government officials alike³who gave freely of themselves to share with us the challenges they face and their ongoing efforts to build a better, more resilient Indonesia. We know that with the human spirit and dedicated pursuit that we witnessed there is much to be optimistic about for the countr\·s future. We would also like to thank the APIK staff who provided invaluable help throughout this journey. Paul Jeffery and Agus Hernadi were most gracious in responding to our many questions, and in facilitating our access to their entire staff both in Jakarta and in the provinces. It has been refreshing to observe the enthusiasm and commitment to excellence from this outstanding group of professionals. Not least, we wish to thank the staff of USAID/Indonesia for their vision and support of improved climate and disaster resilience in Indonesia, and for their commitment to an evaluation of this type on an issue so critical to the future of the country. With the growing list and magnitude of climate impacts and vulnerabilities, it becomes more urgent and important with each passing year to make these opportunities available to learn and share best practices with the global community. The USAID APIK program has made valuable contributions and at a time when action on climate adaptation in Indonesia had yet to gain momentum. There are many to thank, but immediately deserving of our gratitude are Yusak Oppusunggu, Endah Shofiani, Amin Budiarjo, Matthew Burton, Jason Seuc, Fitria Wahid and Marunga Manda. Finally, we wish to thank Emmy La Imu and her team at MESP for their attention to every detail in keeping our team moving and on time. The logistics for a team of six can be challenging in the best of circumstances, but made even more so with the onset of the COVID-19 pandemic in the middle of our five weeks of in-country field work. In the midst of these trying times, we are perhaps most thankful for the humor, resolve, and positive outlook of Indonesians at every turn. ~ The Evaluation Team USAID.GOV FINAL PERFORMANCE EVALUATION USAID-APIK | ii TABLE OF CONTENTS Acknowledgment................................................................................................................................ i List of Tables..................................................................................................................................... iii List of Figures.................................................................................................................................... iii List of Boxes..................................................................................................................................... iii Acronyms.......................................................................................................................................... iv Executive Summary.......................................................................................................................... vi 1. Introduction................................................................................................................................... 1 Global and National Context...................................................................................................... 1 APIK Program................................................................................................................................ 3 2. Evaluation Purpose and Methods............................................................................................... 8 Evaluation Questions.................................................................................................................... 9 Evaluation Team............................................................................................................................ 9 Methodology................................................................................................................................10 Limitations....................................................................................................................................11 3. Findings and Conclusions...........................................................................................................12 4. Recommendations......................................................................................................................34 5. References ...................................................................................................................................39 Annex A. Scope of Work ..............................................................................................................40 Annex B. Evaluation Questions and Sub-Questions..................................................................51 Annex C. Evaluation Matrix...........................................................................................................53 Annex D. Data Collection Tools..................................................................................................59 Annex E. List of Respondents.......................................................................................................66 Annex F. Resilience Fund Grants..................................................................................................72 Annex G. List of Documents and Data Sources........................................................................74 Annex H. Conflict of Interest Disclosures..................................................................................75 Annex I. Indonesian Version of the Executive Summary and Recommendations.................81 iii | FINAL PERFORMANCE EVALUATION USAID-APIK USAID.GOV LIST OF TABLES Table 1: Evaluation questions............................................................................................................. 9 Table 2: Number of Interviewees.................................................................................................... 11 Table 3: GCF proposals initiated through APIK............................................................................ 12 Table 4: APIK's high-level targets and achievements.................................................................... 13 Table 5: List of APIK training manuals and materials ................................................................... 16 Table 6: Number of persons trained by APIK............................................................................... 16 Table 7: APIK'S private sector partnerships.................................................................................. 23 Table 8. Respondents by Gender.................................................................................................... 70 Table 9. Respondents by Type......................................................................................................... 71 LIST OF FIGURES Figure 1: Benefits and Costs of Illustrative Investments in Adaptation ....................................... 3 Figure 2: APIK Geographic Targets................................................................................................... 7 Figure 3: APIK·S place-based resilience TOC.................................................................................. 7 LIST OF BOXES Box 1: CWIS for optimal planting time .......................................................................................... 27 Box 2: Community-based early warning systems ......................................................................... 28 Box 3: A systems approach to mangrove restoration ................................................................. 32 Box 4: Thinking about resilience...................................................................................................... 35 Box 5: Design principles for scaling up........................................................................................... 38 USAID.GOV FINAL PERFORMANCE EVALUATION USAID-APIK | iv ACRONYMS AmCham American Chamber of Commerce and Industry in Indonesia APIK Adaptasi Perubahan Iklim dan Ketangguhan/ Climate Change Adaptation and Resilience Bappeda Regional Development Planning Agency/ Badan Perencanaan Pembangunan Daerah Bappenas National Development Planning Agency/ Badan Perencanaan Pembangunan Nasional BMKG Meteorology, Climatology, and Geophysics Agency/ Badan Meteorologi, Klimatologi dan Geofisika BNPB National Disaster Management Agency/ Badan Nasional Penanggulangan Bencana BPBD Local Disaster Management Agency/ Badan Penanggulangan Bencana Daerah CCA Climate Change Adaptation CSR Corporate Social Responsibility CSV Corporate Shared Value CVA Climate Vulnerability Assessment CWIS Climate and Weather Information Services DRR Disaster Risk Reduction EWS Early Warning System FGD Focus Group Discussion GHG Greenhouse Gas GIS Geographic Information System GoI Government of Indonesia Kadin Indonesia Chamber of Commerce and Industry/ Kamar Dagang dan Industri Indonesia IPCC Intergovernmental Panel on Climate Change Kemendes Ministry of Villages, Development of Disadvantaged Regions and Transmigration/ Kementerian Desa, Pembangunan Daerah Tertinggal, dan Transmigrasi KII Key Informant Interview KLHK Ministry of Environment and Forestry/ Kementerian Lingkungan Hidup dan Kehutanan KSB Village Disaster Alert Group/ Kelompok Siaga Bencana v | FINAL PERFORMANCE EVALUATION USAID-APIK USAID.GOV LAPAN National Institute of Aeronautics and Space/ Lembaga Penerbangan dan Antariksa Nasional M&E Monitoring and Evaluation MOU Memorandum of Understanding NAP National Adaptation Plan NCAR National Center for Atmospheric Research NGO Non-Government Organization OPD Local Government Working Unit/ Organisasi Perangkat Daerah Pokja Working Group/ Kelompok Kerja PSE Private Sector Engagement PY Project Year RAN-API National Action Plan for Climate Change Adaptation/ Rencana Aksi Nasional Adaptasi Perubahan Iklim ROI Return on investment RPJMD Local Medium-Term Development Plan/ Rencana Pembangunan Jangka Menengah Daerah RPJMN National Medium-Term Development Plan/ Rencana Pembangunan Jangka Menengah Nasional SIDIK Vulnerability Index Data Information System/ Sistem Informasi Data Indeks Kerentanan USAID United States Agency for International Development USAID.GOV FINAL PERFORMANCE EVALUATION USAID-APIK | vi EXECUTIVE SUMMARY PROJECT BACKGROUND USAID·s APIK (Adaptasi Perubahan Iklim dan Ketangguhan / Climate Change Adaptation and Resilience) Program is a 4.5 year, US$15.4 million program launched in January 2016 with the goal of assisting the Government of Indonesia (GoI) to strengthen climate and disaster resilience. Originally planned as a five-year program funded with US$19.3 million, a funding shortfall forced a six-month early close out. The program utilized an integrated, place-based approach to engage stakeholders from government, private sector, and local communities to achieve three principal objectives: (1) Mainstream climate change adaptation (CCA) and disaster risk reduction (DRR) into national and sub-national governance frameworks; (2) Build the capacity of local communities and the private sector to address climate change and weather-related natural hazards; and (3) Support the use of climate and weather information services for managing risks and vulnerabilities caused by extreme weather events and shifting climate patterns. Climate change presents an existential threat to human health, safety and welfare in countries around the globe³and Indonesia is no exception. Temperature increase, changes in the frequency and intensity of rainfall, more extreme weather, and sea-level rise already impose risks and vulnerabilities on citizens and sectors throughout the country. Although the countr\·s 34 proYinces Zill be affected by climate change in different ways and to varying degrees, future impacts are likely to be increasingly significant. Indonesia has the tenth largest economy in the world and is considered an emerging lower middle-income country.1 But anal\sis funded b\ USAID in 2016 projects that Indonesia·s econom\ ma\ be exposed to annual losses of as much as 1.4 percent of GDP by the year 2050, involving major impacts to the agriculture and health sectors, and damages from sea-level rise.2 APIK was designed to raise the evidence-based profile of climate risk and vulnerability, to coordinate GoI actions between central and sub-national entities, and to engage the private sector and local communities in the co-design and implementation of adaptation measures to strengthen resilience and provide best practice models for replication and scale-up. This report describes the findings, conclusions, and recommendations from a final performance evaluation of the USAID APIK Program scheduled for close out in June 2020. EVALUATION PURPOSE AND METHODS USAID/Indonesia commissioned this study to evaluate program progress, relevance, and effectiveness and to identify lessons learned that will help guide the design of future programming for improved climate and disaster resilience. The evaluation was designed to collect data and information from a broad range of stakeholders and beneficiaries while ensuring independence of the evaluation process, as well as accuracy and completeness of the subsequent findings, conclusions, and recommendations. An evaluation team comprising six members convened in Jakarta in mid-February 2020 and carried out an extensive schedule of interviews, focus group discussions (FGD), and consultations over a five￾1 World Bank, 2020. 2 USAID ATLAS, 2016. vii | FINAL PERFORMANCE EVALUATION USAID-APIK USAID.GOV week period. Nearl\ 170 program stakeholders, including 52 Zomen, from Jakarta and the program·s three target provinces (East Java, Southeast Sulawesi, and Maluku) were interviewed during the data collection phase. SUMMARY FINDINGS AND CONCLUSIONS APIK·s oYerall approach³to achieve place-based resilience through addressing institutional, socioeconomic, and landscape issues³is a highly appropriate theory of change, and especially so in a highl\ decentrali]ed countr\. The program·s efforts to build resilience through national and sub￾national governments, private sector and community-based actions, and to increase the availability and use of climate and weather information by end-users, are equally appropriate and logical as part of any efforts to increase resilience. APIK is to be commended for helping to raise the profile and urgency of climate adaptation by helping to elucidate the many vulnerabilities that must be faced today and in preparation for an uncertain future. The university connections that were made should be considered an essential component of future programming, especially given the ongoing need to link science to policy across multiple disciplines. APIK·s most important achieYements relate to improYed understanding of climate risks and vulnerabilities, and new commitments by sub-national authorities to increase their own investments in, and attention to, CCA and DRR. A number of policies and regulations now incorporate climate resilience concerns and actions, as do planning and budgeting documents at national, sub-national and village levels. There is potential for a strong legacy achievement for APIK in the assistance it provided to each of the three provinces in developing applications for funding from the Green Climate Fund (GCF). Prior to APIK, the country had not utilized GCF funding for adaptation, and the fact that these three applications have successfully advanced to the full proposal stage with GCF-accredited organizations is a significant achievement for APIK and an important development for the country. In many respects, APIK has been a unique project in its approach to climate adaptation in Indonesia, in that it has supported the GoI at all levels to improve communication and coordination, while simultaneously introducing a broad range of pilot activities to enhance community and private sector practices and attitudinal shifts toward improved resilience. The program achieved five of six high-level results targets (Table 4). As with any large, multi-component program working across several geographies, the evaluation team identified shortcomings. The sustainability of some activities is in question, and the technical rationale and approach for other activities can be challenged. But these are important opportunities for learning and it should be highlighted that the six-month early closure of the program did not allow APIK staff time to properly evaluate, reflect upon and share their own lessons learned as they had hoped. Universally, interviewees were laudatory of the dedication, enthusiasm and technical support demonstrated by APIK staff. EQ1: How has APIK built the resilience and adaptation of different levels of government and the commXniWieV WhaW iW ZoUkV in, and hoZ haV APIK·V naWional ZoUk influenced or been influenced by the district and community levels? APIK invested heavily and from the beginning to engage stakeholders at all levels of government and with communities and the private sector. The analyses undertaken to inform initial discussions regarding current and emerging climate risks and vulnerabilities were essential to the foundation of USAID.GOV FINAL PERFORMANCE EVALUATION USAID-APIK | viii trust, commitment and a spirit of collaboration that is eYident across APIK·s three target provinces. An APIK staff member was embedded in the National Development Planning Agency (Bappenas) to help lead the National Action Plan for Climate Change Adaptation (RAN-API) Secretariat in an intergovernmental and public process to revise and update the RAN-API which is moving toward approYal in 2020. It is fair to sa\ that APIK·s coordinating and convening role, especially at the sub￾national level, has been one of its most important achievements and one that has played an important role in multiple successes to incorporate climate adaptation into policies, plans and budgets at all levels. Indonesia·s RPJMN 2020-2024 (National Medium-Term Development Plan) includes US$3.02 billion (IDR 43.45 trillion) in funding for climate adaptation, a clear indication of the increased understanding and urgency of the issue that has developed over the past five years. The significance and success of adaptation measures implemented in the three provinces has been somewhat mixed. While there have been literally tens of thousands of people involved in program activities³including trainings, Climate Field Schools, demonstration projects, and many other activities³it is questionable whether the various adaptation measures, when viewed in the aggregate, represent a level of output and outcome that will be effective and sustainable in improving resilience for a sizable portion of the target populations in the three provinces. However, it should also be said that the testing of a wide range of interventions provides valuable lessons for how to improve the focus and effectiveness of future efforts. APIK allowed local priorities to largely drive the selection of adaptation measures, and while the approach makes sense from a local ownership and commitment perspective, it raises questions about strategic relevance, scale, replicability, and sustainability. Further reflection on this is found in the recommendations below. EQ2: HoZ ZaV APIK·V ZoUk ZiWh Whe pUiYaWe VecWoU able Wo incUeaVe aZaUeneVV aboXW adaptation and disaster risk reduction? Early efforts by APIK to encourage private sector involvement in resilience activities were based primarily on outreach events and meetings where climate risk and vulnerability was discussed and APIK had the opportunity to develop contacts with companies. Staff attended and presented at business association meetings, including those at the Indonesian Chamber of Commerce and Industry (Kadin) and the American Chamber of Commerce and Industry in Indonesia (AmCham). More direct contact with local businesses began in Project Year 3 (PY3) after a review of about 40 companies was narrowed down to a dozen companies willing to collaborate with APIK, local governments and communities on specific demonstration projects and using Corporate Social Responsibility (CSR) funds. APIK·s role Zas principall\ as an initiator and conYener in bringing these groups together, while helping to develop MOUs and design local activities. The implementation of activities had generally positive results in terms of success and producing benefits, although the activities involving the private sector were small-scale and with few direct beneficiaries. The evaluation team found little evidence that activities were likely to be replicated or scaled up as, for example, there did not appear to be any plans or commitments from local governments to provide continued support. The exception may be an investment by PT Cargill aimed at increasing egg production through the replication of improved poultr\ sheds. OYerall, APIK·s priYate sector engagement (PSE) efforts were not found to be optimal for creating market-based solutions of the type involving serious market analysis to create value propositions and convince companies to embed best practices into operations. On a local scale, however, several of the efforts did bring measurable benefits to those who participated and offer models that could be considered for broader scaling up under future programs. ix | FINAL PERFORMANCE EVALUATION USAID-APIK USAID.GOV EQ3: To what extent are targeted beneficiaries better prepared and ready to adapt as a UeVXlW of APIK·V climaWe and ZeaWheU infoUmaWion VeUYiceV? Completion of an assessment in PY1 of climate and weather information services (CWIS) needs and opportunities led to a ´roadmap reportµ that helped APIK and the Meteorology, Climatology and Geophysics Agency (BMKG) develop work plans to improve those services. The Ministry of Environment and Forestry (KLHK) and the National Disaster Management Authority (BNPB), local governments, communities and Climate Field Schools (CFS) were identified as beneficiaries for improved CWIS. APIK helped to improve the main climate vulnerability and disaster risk tools being used by KLHK and BNPB, respectively. However, the integration of these tools could not be completed. The integration of CCA and DRR objectives into a single set of guidelines and tools is an outstanding task for the country to address. At the local level, APIK provided downscaled climate projections at provincial scale that were used in making RAD-API (local action plan for climate change adaptation). APIK also helped BMKG to develop OtoKlim software which enables web-based spatial display of weather information, plus various communication tools (e.g. social media, WhatsApp groups) to improve access to weather data at the sub-national and community levels. Stakeholders were appreciative of the attempts to improve CWIS, as well as the handful of early warning systems (EWS) for flooding and landslides that the program helped to install in several villages in the three provinces. Training has been a large component of APIK tasks to improve CWIS, with each of the Climate Field Schools now equipped to incorporate weather data into agriculture and fisheries sector operations. Communities were positive regarding APIK·s CFS actiYities, although the usage of neZl\ proYided weather information is sporadic for reasons related to the lack of technical capacity and funding for the operation and maintenance of equipment. The CWIS component of APIK has played an important role in helping to focus attention on the science and evidence-base that will hopefully sustain and increase GoI attention on CCA/DRR in the years ahead. EQ4: What are the key lessons learned from application of the landscape approach to the three areas of watershed, coastal, and small islands? As one of the three pillars of APIK·s theor\ of change, the landscape approach Zas an important contribution to discussions of climate vulnerability and disaster risk. APIK primed those discussions with information from completed Climate Vulnerability Assessments (CVA) and detailed site selection assessments that brought science, based on multiple data sets, into the discussions, and the planning and budgeting processes of the various stakeholders. CVAs were completed at the provincial, district/city, and community scales using the same hazard/threat factors but with increasing granularity at each scale. APIK trained local authorities that then led CVA trainings for other districts with the assistance of APIK staff. Stakeholders learned that the vulnerability of a location is interconnected with the ecological, hydrological, and meteorological characteristics of the surrounding landscape, and that sound land use decisions that safeguard ecosystem services are critical for protecting urban, rural, and island communities from extreme and changing weather. APIK performed a valuable service in bringing these concepts and analyses to discussions, and attempting to assist in the implementation of landscape-scale activities. Examples include planned storm water drainage improvements in Southeast Sulawesi, groundwater recharge in East Java, mangrove restoration at four locations, and water-source protection b\ planting bamboo. Unfortunatel\, due to the program·s early closure, APIK was not able to conduct follow-up monitoring to determine the success, effectiveness and sustainability of these interventions. In terms of lessons learned, while the landscape approach is not an entirely new concept in Indonesia, there are few examples of implementation, and the term can benefit from greater clarification of key concepts, definitions and practical guidance on how to implement a landscape approach as a foundational component of climate resilience. USAID.GOV FINAL PERFORMANCE EVALUATION USAID-APIK | x EQ5: What was the impact of the funding challenges and what opportunities were missed as a result? The program·s original budget and timeframe Zas approYed at US$19.3 million over five years. But in PY2, APIK was informed that there would likely be a budget shortfall and that it should limit monthly expenditures accordingly. At the beginning of PY5, the program was informed by USAID that the program would receive total funding limited to US$15.4 million, which forced APIK to a decision to end the program six months early. In other words, APIK had to complete the program in 88 percent of the original timeframe with 79 percent of the original budget. The most significant result of the reduced funding was that many local activities could not be implemented or fully completed. Also, a Resilience Fund (small grants fund) originally budgeted at US$2 million was scaled back to US$650,000. Importantl\, APIK·s oZn plans for using the final \ear of the program to evaluate, reflect upon and share lessons did not play out as hoped. Despite expressed disappointments, when asked during interviews and FGDs if there were any recommendations for future programming, stakeholders were universally positive about APIK and expressed strong enthusiasm and hope for a possible program extension or a new similar program to continue the important work. SUMMARY RECOMMENDATIONS There is urgenc\ in the task of helping Indonesia·s citi]ens and sectors adapt to the coming impacts of climate change. Embedded in that urgency is the need for actions that are more strategic, synergistic, and transformative at scale for greater numbers of people. Many of the objectives of adaptation, mitigation, and ecological resilience are synergistic and should be designed to be. Mitigating greenhouse gas (GHG) emissions has the potential to reduce the need for, and cost of, adaptation measures in the long-term. Similarly, efforts to increase ecological integrity will enable Indonesia to be more resilient to the adverse effects of climate change. Effective preparation for an uncertain future can only be achieved through the collective action of government, individuals, businesses, communities and citizens, and their collective recognition of the linkages inherent in these objectives and the multiple benefits of doing so. The evaluation team offers the following list of recommendations for future programming of development assistance toward improved CCA/DRR. Intergovernmental, community engagement, and resilience Continue to strengthen vertical and horizontal integration of GoI entities and improved planning and budgeting for CCA/DRR. Build upon the valuable coordination efforts started by APIK, beginning with high-level dialogue between Bappenas, KLHK and BNPB on a co-designed, collaborative approach for replication and scale-up within targeted geographies. Help build GoI capacity and responsibility for monitoring and evaluation reporting specific to resilience. Assist with the co-design of a country-scale Adaptation Theory of Change, and an M&E system with identified roles and responsibilities of GoI actors. The objective is to increase transparency, accountability and overall public/private investment in climate resilience. Focus on one or two key sectors and specific outcomes at meaningful scale. The goal is to accelerate adaptation in key sectors or systems and where wide-scale adoption of improved practices or behavioral change can have a transformative impact. Consider the following focused objectives that could have beneficial impacts for more people: agro-forestry for smallholder farmers at landscape scale; conservation and management of water resources at basin-scale; co-management of artisanal xi | FINAL PERFORMANCE EVALUATION USAID-APIK USAID.GOV fisheries at seascape scale; flood mitigation through basin-scale application of gray/green infrastructure; and mangrove conservation and management at landscape scale. Use every opportunity to design adaptation measures that also contribute to mitigation. The global community is calling for greater integration of adaptation and mitigation efforts. There is plenty of scope for contributing to both objectives in almost any efforts involving landscape restoration or nature-based solutions. That is the case as well for climate smart agriculture at the landscape scale. Private sector mobilization Incorporate tasks and sufficient funding devoted to stimulating public and private investment. There are a growing number of climate financing opportunities worldwide and USAID can strategically leverage its own resources through support for a facility-type platform that can help link investors (e.g. Green Climate Fund; Adaptation Fund) with bankable projects that utilize GoI funds (at all levels from central level down to the village level). Give the highest priority to market-based solutions and Corporate Shared Value.3 Help to build a shared narrative between the public and private sectors based on mutual interests and shared responsibility. Give the highest priority to market-based solutions and Corporate Shared Value to integrate key principles and practices into operations. Use a co-design process for early identification of the mutual interests of all parties. Data and information needs Help Indonesia to re-think its overall data and information needs related to climate and disaster resilience. Think of this as a two-level process. First, build upon APIK·s efforts to improYe the quality and accessibility of climate and weather information by taking the next step of assessing the data/information needs of end-users in targeted sectors. Second, assist Indonesia to augment its acquisition, analysis, and application of remotely sensed data. Remotely sensed data, and the technical products and services derived therefrom, offer strong potential for addressing CCA/DRR issues at temporal and spatial scales that are transformative. Landscape scale solutions Design future programs to be outcome-oriented and transformative at scale. The restoration of key landscapes and their hydrological functions can pay huge dividends for improved resilience. USAID can play an important role in helping to clarify key principles, guidelines, outcome measures, and the cost-benefit rationale for applying the landscape approach related to CCA/DRR. 3 Corporate Shared Value (or Creating Shared Value) is a new generation business thinking term that enhances the competitive position of a company while at the same time advancing the society in which it operates. Michael Porter and Mark Kramer argue that businesses can help in this process by creating shared value. In a 2011 HBR article ´Creating Shared Value,µ Porter and Kramer argued that companies can moYe be\ond corporate social responsibility (CSR) and gain competitive advantage by including social and environmental considerations in their strategies. 1 | FINAL PERFORMANCE EVALUATION USAID-APIK USAID.GOV 1. INTRODUCTION This report describes the findings, conclusions and recommendations from a final performance evaluation of the USAID APIK Program in Indonesia. APIK aims to strengthen climate and disaster resilience, with a particular focus on three of the countr\·s 34 proYinces. The eYaluation Zas designed and implemented by a team of independent evaluators under an agreement with the Indonesia Monitoring and Evaluation Support Project (MESP). APIK and this external evaluation are funded by USAID/Indonesia. The report provides indications of program progress, relevance, effectiveness, and the likelihood that activities can be sustained. In addition, lessons learned and recommendations are offered to help guide future programming of development assistance pursuing similar objectives related to climate change adaptation (CCA) and disaster risk reduction (DRR) in Indonesia and elsewhere. GLOBAL AND NATIONAL CONTEXT The world is not on track to limit the potential worst impacts of climate change. As the global community gears up to hold the increase in global temperature to 1.5 degrees Celsius, the world remains on a pathway toward 3 degrees Celsius or more by 2100. 4 Regardless of the eventual magnitude of climate change, temperature increase, changes in the frequency and intensity of rainfall, more extreme weather and sea-level rise are already exerting a growing number of negative impacts on human health, safety and welfare in countries around the globe. These impacts and others are exacerbated by deforestation, destruction of mangroves and weak governance to implement and enforce sustainable land and water use. Untold numbers of lives and livelihoods are being lost, with people displaced, and social and economic development gains being dragged down by competing and repeated expenditures on disaster relief. With its large population and high dependence on natural resources, Indonesia is among the most vulnerable countries to climate change. The 14th largest country in the world, it has over 17,500 islands and the third longest coastline. Sea-level rise is thus especially threatening to this archipelagic country and deserves special attention. The Intergovernmental Panel on Climate Change (IPCC) in its Fifth Assessment Report issued in 2014 predicted global sea levels would rise 28 inches (71 centimeters) by 2100. Recent analysis published b\ NASA·s Jet Propulsion Laborator\ indicates that Greenland and Antarctica ice is melting at a rate six times greater than during the 1990s, and that sea-level is on track to rise in line with IPCC·s Zorst case scenario.5 It is not difficult to imagine how catastrophic that would be for a large portion of Indonesia·s population, Zith commercial and residential propert\ losses, flooding, and saline intrusion affecting both agriculture and aquaculture. Even greater impacts from climate change are projected for the health and agriculture sectors. Health and employment impacts from dengue fever and malaria are already significant and increasing. While 4 UN SDG Group, 2020. 5 NASA, 2020. USAID.GOV FINAL PERFORMANCE EVALUATION USAID-APIK | 2 the incidence of dengue fever may decline in some provinces, malaria incidence is expected to rise throughout the country.6 Impacts to human health from climate change are many and generally fall into the following four categories: heat-related illness; cardiopulmonary illness; food, water and vector￾borne disease; and mental health consequences from stress.7 Similarly for agriculture, some areas may experience a rainfall increase while others will most certainly see production declines as a result of water scarcity. On a national scale, irrigated rice, soy and sugar cane production are likely to decline.8 In terms of water security, drought, flooding, erosion, and reduced groundwater recharge will jeopardize surface and ground water quantity and quality, leading to shortages for irrigation, drinking water and sanitation.9 Fires are likely to become more intense as a result of rising temperatures and drought.10 The poorest people³and especially women³are often the hardest hit by climate impacts, either because they already live on marginal lands, such as riverbanks, deforested, areas, or flood plains, or because they have the lowest adaptive capacity due to lack of education, financial resources, and/or government support. A ke\ message in all of this is that the countr\·s 34 proYinces Zill be affected b\ climate change in different ways and to a varying degree, but that overall the impacts are likely to be significant. Indonesia currently enjoys GDP growth of around 5 percent. 11 HoZeYer, the countr\·s econom\ ma\ be exposed to annual losses of as much as 1.4 percent of GDP by 2050 as a result of climate change. It is projected that the heaviest economic toll will be related to agriculture (53 percent), followed by health (34 percent) and sea-level rise (13 percent).12 Sea-level rise economic losses will be largely due to commercial and residential property losses, together with flooding and saline intrusion in agriculture and aquaculture production systems. The good news is that scaled-up investments in adaptation can pay large social, economic, and environmental dividends. The Global Commission on Adaptation finds that³at the global scale³ investing US$1.8 trillion in five areas from 2020 to 2030 could generate US$7.1 trillion in total net benefits.13 The Commission identifies the following five areas of adaptation investment as having potential to produce the greatest benefit: early warning systems, climate-resilient infrastructure, improved dryland agriculture, mangrove protection, and investments in water resources management (Figure 1). Indonesia is making important strides in national efforts to address CCA, and completed the first version of its National Action Plan for Climate Change Adaptation (Rencana Aksi Nasional Adaptasi Perubahan Iklim, or RAN-API) in 2014. The RAN-API identifies some of the countr\·s main vulnerabilities to climate change and charts out short-, medium- and long-term actions to strengthen resilience. Decentralization in Indonesia has given significant authority to local governments for many of the key functions and services that influence climate and disaster resilience. However, in many cases 6 USAID ATLAS, 2016. 7 U.S. Global Change Research Program, 2016. 8 Ibid. 9 ICCSR, 2009. 10 BNBP, 2014. 11 A figure certain to decline following the COVID-19 pandemic. 12 USAID ATLAS, 2016. 13 Global Commission on Adaptation, 2019. 3 | FINAL PERFORMANCE EVALUATION USAID-APIK USAID.GOV the capacity and resources of local government are insufficient to fully execute those duties. There is no clear guidance from the central government on which unit between the provincial and local governments should take the lead in coordinating adaptation planning, analysis, and budgeting. The awareness and implementation of the RAN-API is limited. The country is now moving toward significantly enhanced budgetary allocations for climate adaptation in its RPJMN (Rencana Pembangunan Jangka Menengah Nasional), or National Medium-Term Development Plan. And as this report describes, the USAID APIK Program has helped spur the increased attention being given to climate adaptation in the planning and budgetary processes of the countr\. FiYe \ears ago Zhen APIK Zas getting started in 2015, the countr\·s climate-related focus Zas largel\ on the mitigation of greenhouse gas (GHG) emissions, and ´adaptationµ Zas a new concept for many, especially at the sub-national level. APIK has played an important role as convener, facilitator, and technical assistance provider, and achieved some significant results along the way toward improving climate resilience for its stakeholders. As APIK prepares to close out this month (April 2020), the program leaves the country with many lessons learned, and much to build upon, for the ne[t important phase of Indonesia·s efforts to improYe CCA and DRR for its 273 million citizens. Figure 1: Benefits and Costs of Illustrative Investments in Adaptation APIK PROGRAM Adaptasi Perubahan Iklim dan Ketangguhan (APIK) is a five-year USAID-funded program to assist Indonesians to strengthen their ability to manage and adapt to climate and disaster risk. The program is implemented by Development Alternatives, Inc. (DAI) with an in-country program head office in Jakarta where support was provided to GoI national ministries. APIK also maintained satellite program offices in three provinces (East Java, Southeast Sulawesi, and Maluku) where sub-national and community level assistance was focused. The program launched in late 2015 and is scheduled to close in June 2020. APIK·s strategic objectiYe is to improYe the abilit\ of Indonesians to manage and respond to climate change and disaster risk. The program utilized an integrated, place-based approach to engage From Global Commission on Adaptation, 2019. USAID.GOV FINAL PERFORMANCE EVALUATION USAID-APIK | 4 stakeholders from government, the private sector and local communities to achieve the three intermediate results, as described below. 1. Integration of climate change adaptation and disaster risk reduction into national and sub-national governance frameworks improved Currentl\, the central goYernment·s state budget is the largest source of climate finance. The GoI·s climate finance flows were US$6.37 billion between 2011 and 2014. Interventions under this result focus on identifying and influencing existing planning and budgeting processes within the GoI that are most relevant for strengthening resilience to climate change and weather-related natural disasters at the local level. APIK applies a development-first adaptation strategy to heighten national government awareness of the risks posed to critical development goals by climate change and hydro-meteorological disasters, and to strengthen the GoI response by mainstreaming CCA/DRR tools and approaches. APIK works with and through the Secretariat of RAN-API and the National Development Planning Agency (Bappenas) to elevate the visibility of the RAN-API and equip key line ministries to support its implementation. APIK also works with the National Disaster Management Authority (BNPB) and the National Platform for Disaster Risk Reduction (PLANAS PRB) to accelerate the understanding of climate and weather-related issues. APIK works with intra-governmental working groups to strengthen horizontal coordination and share lessons learned between government ministries and departments. Given the cross-cutting nature of CCA, this requires collaboration with several ministries, including Bappenas, BNPB, the Ministry of Finance (MOF), the Ministry of Marine Affairs and Fisheries (KKP), the Ministry of Environment and Forestry (KLHK), the Ministry of Public Works and Housing (PUPERA), the Ministry of Agrarian and Spatial Planning, the Ministry of Agriculture (Kementan), and others. In support of Indonesia·s decentrali]ation process, APIK proYides support to sub-national government entities that exercise authority to achieve climate and disaster resilience. APIK focuses on local government and community capacity-building, using entry points such as local government working groups and community-based vulnerability mapping to convene stakeholders and facilitate planning and actions that strengthen landscape, socioeconomic and institutional resilience. APIK works with sub￾national entities in selected districts on planning, budgetary and operational processes, including regional adaptation strategies, local development plans (RPJMDs), annual local government plans (RKPDs), Musrenbangda,14 local disaster management plans, spatial management plans (RTRWP/K, RTDR), and village budget allocations under the ´Village LaZµ (No. 6/2014 on Villages). The place￾based interventions are linked to, and enabled by, a strengthened national enabling environment, relevant climate and disaster risk information at the sub-national and community levels, and the engagement of the private sector around shared risks. 2. Capacity of local communities and the private sector to address climate change and weather-related natural hazards strengthened Hydro-meteorological disasters can significantly impact the private sector by damaging businesses, the impacts of which are often amplified through supply chains that affect entire sectors. Often, the 14 Musrenbangda are public consultation events held at the local level. 5 | FINAL PERFORMANCE EVALUATION USAID-APIK USAID.GOV benefits of integrating CCA/DRR considerations are not readily understood by the private sector. APIK·s strateg\ emphasi]es the engagement and mobili]ation of business associations and business networks around shared risks in key economic sectors, such as the fishing industry in the Maluku Islands, or cocoa and rice production in Southeast Sulawesi. Cutting across the national and sub￾national levels, APIK aims to improve private sector awareness about climate and weather-related natural disaster risks while strengthening the capacity of private sector partners to mitigate risks in their business models and carry out sector-specific risk reducing activities. The program seeks opportunities to maximize synergies between this and other project interventions by, for example, encouraging cooperation with relevant business associations, such as the Indonesian Chamber of Commerce and Industry (Kadin) and the American Chamber of Commerce and Industry in Indonesia (AmCham), to build awareness among Indonesian business networks on CCA/DRR. 3. Utilization of information for climate and disaster risk management among key stakeholders improved In Indonesia, climate impacts and their effect on the variability and intensity of disasters are of growing concern, although the understanding of the best local adaptation strategies is nascent. Data are often lacking, and the translation, relevance and communication of information is weak. A cross-cutting activity focuses on the collection, packaging, and dissemination of climate and weather information services (CWIS). Better CWIS are fundamental to fostering place-based resilience across the archipelago, saving lives in the near term (i.e. disaster risk reduction) while supporting better planning and public investment in the medium- to long-term (i.e. climate change adaptation). Improved information is not an end in itself, but must be linked to institutional capacity-building, pro-poor programming for sustainable livelihoods, and transboundary ecosystem management. The Resilience Fund: The Resilience Fund is an important component of APIK. This fund is used to support projects that build climate and disaster resilience in partnership with communities, government and the private sector. The fund encourages participation and collaboration of local non￾governmental and private sector entities that will spur innovations in new products and services. Through provision of small grants or subcontracts APIK supports activities by non-governmental organizations (NGOs), civil society organizations (CSOs) and research institutions and universities, as well as co-investments with the private sector in CCA/DRR efforts. Geographic coverage APIK undertook a comprehensive process to select project sites within the three provinces (East Java, Southeast Sulawesi, and Maluku) that were identified by USAID as target geographies. The process incorporated local data sets, established government indicators for climate vulnerability and disaster risk, and stakeholder engagement. APIK applied four spatial lenses at the provincial level to conduct the initial macro-scale analyses. The four lenses include the following analyses: (1) Landscape (e.g. hydrological, meteorological, and land use characteristics); (2) Disaster and climate risk (e.g. indices to understand key vulnerabilities across the landscape); (3) Socioeconomic (e.g. demographic data to focus on reaching the most vulnerable populations); and USAID.GOV FINAL PERFORMANCE EVALUATION USAID-APIK | 6 (4) Institutional (e.g. governance analyses to assess evidence of local government demand).15 The following is a brief description of the three project sites and key issues to be addressed: East Java: APIK worked in the Brantas Watershed and its seven districts (Sidoarjo District, Mojokerto District, Jombang District, Blitar District, Malang District, Malang City and Batu City). The Brantas Watershed is one of the GoI·s priorit\ Zatersheds for conserYation and rehabilitation. About half of East JaYa·s 38 million people liYe in the basin, which contains a concentration of critical infrastructure and is a center of economic activity. APIK project sites include locations in the upper, middle, and lower watershed, which is highly vulnerable to natural hazards, especially flooding and landslides. The Brantas Watershed is degraded by deforestation, industrial pollution, inadequate sanitation and wastewater treatment, and seasonal water scarcity. A lack of sufficient water storage adds to the basin·s recurring floods, droughts and falling water tables, and many riverbanks are occupied by illegal settlements as a result of weak/absent enforcement of zoning regulations. Water scarcity remains the greatest climate change impact of concern in coming years for not only the Brantas but East Java as a whole. The selection of the Brantas Watershed provides opportunity to work in a watershed landscape having strong hydrological linkages to densely populated areas. Southeast Sulawesi: APIK worked in Kendari City and the Konawe-Selatan District of Southeast Sulawesi having a combined population of 595,000 people. The region has a long coastline as well as dense but rapidly degrading forests in the center of the province. Land degradation is at least partially ascribed to two of the main industries in the province: nickel mining and palm oil production. Increasingly erratic weather patterns are resulting in significant impacts to local economies and livelihoods. Droughts, floods, and warming sea temperatures are taking a toll on the agriculture, livestock, fishing and seaweed production sectors. Upstream deforestation is resulting in more rapid runoff, downstream flooding, and erosion, especially in the Wanggu Watershed, which is a GoI priority watershed. The selection of Southeast Sulawesi provides APIK with the opportunity to work in a coastal zone landscape/seascape, with potential to demonstrate ridge-to-reef linkages. There are 117 coastal villages and 44 villages in the Wanggu Watershed. Maluku: APIK worked in three districts in the eastern province of Maluku, which comprises several island clusters spread over 47,000 km2. These islands are primarily steep, mountainous, and forested with limited available land for human settlement. As a result, land clearance for housing in upland forested areas has worsened flooding and landslide potential. Weather patterns (wet and dry season) in the province are reversed from those found elsewhere in Indonesia, and becoming increasingly erratic. This in turn has resulted in labor shifts between farming and fishing, and between rural and urban areas as traditional farming and fishing practices are disrupted. The predominant hazard and climate risks for the 425,000 people in these three districts are landslides, flooding, and sea level rise. The remoteness of Maluku is a challenge to natural disaster preparedness and response. The selection of this geography provides APIK with the opportunity to demonstrate activities and approaches within a small islands landscape. 15 APIK, 2016. 7 | FINAL PERFORMANCE EVALUATION USAID-APIK USAID.GOV Figure 2: APIK Geographic Targets Theory of change APIK·s Yision for the program is that ´after fiYe \ears, the prioritized geographies³including local governments, businesses, and households therein³will have achieved measurable gains in resilience to natural disasters and climate change, with concrete examples of adaptation in practice serving as replicable models for DRR and CCA integration across the archipelago. Further, such local improvements in resilience will be enabled, cemented, and sustained through the confluence of more accessible, user￾focused climate information services and the mainstreaming of disaster preparedness and climate adaptation into core goYernance processes at all leYels.µ16 Translating this Yision into action, APIK·s theory of change utilizes a technical approach centered on place-based resilience. This approach recognizes that it is first necessary to understand current risks and vulnerabilities within each locale, and the likely additional impacts related to climate change, 16 APIK, 2016. Figure 3: APIK·S place-based resilience TOC USAID.GOV FINAL PERFORMANCE EVALUATION USAID-APIK | 8 if Indonesia is to make progress on gaining local ownership, commitment and sustained leadership toward improved climate and disaster resilience. Place-based resilience helps communities to move beyond local issues and priorities through linkages with adjacent communities and with all levels of government having expertise and resources to be mobilized. APIK describes this place-based resilience approach as having three principal components, namely: institutional resilience, socioeconomic resilience and landscape resilience (Figure 3). Institutional resilience requires that all levels of government mainstream CCA/DRR principles into planning and budgetary processes and to be prepared with the capacity to adapt and evolve policies, tools, and services over time. Socioeconomic resilience requires that these same principles be incorporated into business operations across all sectors, and within the livelihood practices of individuals exposed to climate and weather risks. Landscape resilience requires an understanding of the ecological, hydrological, and meteorological linkages that underpin ecosystem integrity and that provide physical buffer and other safeguards against extreme and changing climate and weather. In addition, APIK recognizes that CWIS are essential to inform sound, evidence-based decision-making for governments, businesses, communities and individuals. CWIS must be accurate, trusted, widely available and tailored to the needs of different sectors, actors, and locales throughout the country. 2. EVALUATION PURPOSE AND METHODS With this final performance evaluation of APIK, USAID/Indonesia aims to achieve accountability and learning to inform future efforts on the integration of CCA/DRR into national and sub-national government structures and processes. The Mission wishes to continue efforts to enhance the capacity of public and private sector institutions to systematically integrate CCA/DRR into their planning and investment plans. This evaluation will provide information to USAID, the GoI, and others on the challenges faced, opportunities, and lessons learned during implementation which will help to improve the design of future programs working to strengthen the capacity of the country to address climate and disaster risks. The specific objectives of the evaluation are as follows: 1) Accountability ² Assess program progress, relevance, and effectiveness to allow transparency and accountability for the stakeholders involved; and 2) Learning ² Identify lessons for future programming of development assistance aimed at improving climate and disaster resilience in Indonesia and similar settings elsewhere. Coastal erosion at Batu Jaya Village, Java. Photo: ‹ USAID-APIK 9 | FINAL PERFORMANCE EVALUATION USAID-APIK USAID.GOV EVALUATION QUESTIONS The evaluation addressed the following questions: Table 1: Evaluation questions 1) How has APIK built the resilience and adaptation of different levels of government and the communities that it Zorks in and hoZ has APIK·s national Zork influenced or been influenced by the district and community levels? 2) HoZ Zas APIK·s Zork Zith the priYate sector able to increase awareness about climate change adaptation and disaster risk reduction? 3) To what extent are targeted beneficiaries better prepared and ready to adapt as a result of APIK·s climate and Zeather information serYices? 4) What are the key lessons learned from application of the landscape approach to the three areas of: watershed; coastal; and small islands. 5) What was the impact of the funding challenges and what opportunities were missed as a result? In addition, 37 sub-questions were developed to further elucidate these five primary questions and to guide the development of data collection tools. A list of the sub-questions can be found in Annex B. EVALUATION TEAM The team comprised six independent evaluators with a diversity of relevant skills and experience. A short biograph of each team member follows. Richard Volk served as Team Leader and oversaw all aspects of evaluation design, data collection, analysis, and report writing. Richard has extensive experience in natural resources management, particularly in the areas of water security, coastal management, fisheries and climate resilience. He has more than 35 years of international development experience, including 20 years at USAID where he provided worldwide technical and managerial support as a senior technical adYisor to the Agenc\·s Water Office. Richard has traveled throughout Indonesia on more than a dozen professional assignments over the course of the past 25 years. Saut Sagala is cluster chair of Disaster and Climate Resilience as part of the Resilience Development Initiative (RDI) of Indonesia. He is also a tenured professor in the Regional and City Planning Department of the Institute of Technology Bandung (ITB). Saut has extensive experience in disaster risk and climate resilience work, including for the Asian Development Bank, the World Bank, SNV, UNDP, the American Red Cross, and several GoI ministries and agencies. His recent work includes a series of assignments for the Ministry of Agrarian and Spatial Planning on flood risk reduction, climate adaptation, and climate vulnerability risk assessment in the WASH sector. Retno (Dini) Handini is a Senior Evaluation Specialist with MESP with extensive governance experience, including sub-national planning and budgeting. Ms. Handini is highly experienced in evaluation design, research, implementation, analysis, and report writing. She has been a team member for seYeral eYaluations and preYiousl\ serYed for fiYe \ears as and M&E Specialist for the World Bank·s Public Sector Governance Practice in Jakarta. Dini holds advanced degrees in economics and has in￾depth knowledge of Indonesian governance from assignments carried out for the World Bank, GTZ, CIDA and AusAID. USAID.GOV FINAL PERFORMANCE EVALUATION USAID-APIK | 10 Indradi (Indi) Soemardjan is a Private Sector Engagement (PSE) Specialist. He is an innovative and highly accomplished regional business developer/entrepreneur with a strong background in sustainable agriculture, including coffee production. Indi was a key member of the recent MESP/USAID Private Sector Engagement Assessment and has an in-depth knoZledge of Indonesia·s business and regulator\ environment. David Callihan serYes as MESP·s Chief of Part\, and proYided for this eYaluation a range of support, including assistance for evaluation design, data collection and data analysis. David has led more than a dozen major evaluations, including many of natural resource and climate-focused activities for organizations that include USAID, the U.S. Fish and Wildlife Service, and the World Wildlife Fund. He previously served as Team Leader for the Midterm Performance EYaluation of USAID/Vietnam·s Forests and Delta Project, which has a significant focus on climate adaptation. Irma Sitompul is an MESP Project Officer with several years of experience in project and office operations and management on a range of community development projects funded by international humanitarian organizations. She provided support to programs on activity planning, logistics, financial management, compliance, and administrative functions. For this evaluation, Irma was chiefly responsible for logistics, scheduling and data management. METHODOLOGY The evaluation was designed to collect data and information from a broad range of stakeholders and beneficiaries while ensuring independence of the evaluation process, as well as accuracy and completeness of the subsequent findings, conclusions and recommendations. To collect data, the team used a number of data sources that balance each other including: quantitative and qualitative data; individual and group responses; analysis of existing documents and data; and direct observation. Two types of data were collected: primary data collected through key informant interviews (KIIs), focus group discussions (FGDs), and direct observations; and secondary data compiled from existing documents. The Evaluation Matrix that helped guide evaluation design can be found in Annex C. The evaluation team conducted KIIs with government, private sector, and civil society representatives at the national, provincial, district and city levels, and FGDs at the community level. To conduct interviews, the team used structured but flexible interview guides designed to ensure consistency across interviewees and locations. The various data collection tools used in KIIs and FGDs can be found in Annex D. Interviews and FGDs were mostly conducted in Bahasa, utilizing professional translators as needed. Secondary data was collected through extensive desk review of program documents (e.g. work-plans, performance reports, M&E plan, etc.), plus other documents provided by USAID/Indonesia and APIK. A list of documents and data sources can be found in Annex G. Following data collection, and during curation and quality-checks of the notes obtained from KIIs and FGDs, the evaluation team coded the responses with pre-determined thematic categories related to the five evaluation questions. The curated and coded notes were uploaded into Dedoose software files to facilitate the comparison of responses and to define response patterns between the various interviewees. In-country data collection took place during a five-week period in February-March 2020. The evaluation team visited nine of the 12 districts in the three provinces where APIK has field activities. A total of 167 individuals (31 percent female) participated in KIIs and FGDs, as shown in Table 2. 11 | FINAL PERFORMANCE EVALUATION USAID-APIK USAID.GOV Table 2: Number of Interviewees No. of Key Informants TOTAL Male Female Central Government 7 1 8 APIK staff in Jakarta 8 1 9 APIK staff in the provinces 4 1 5 Local Government 43 31 74 Academia 6 1 7 NGOs 2 0 2 Community/Desa 41 17 58 Private Sector 4 0 4 TOTAL 115 52 167 LIMITATIONS It is important to acknowledge some potential limitations of the methodology. The qualitative methods of the evaluation provide valuable insights related to stakeholder perceptions of program activities and attitudinal or behavioral changes as a result of the program, but the evaluation team was not able to rigorously quantify all of the reported achievements, nor was that the intent of the evaluation. Among the many possible limitations, response and recall bias are two of the largest potential limitations for any qualitative data collection effort. To mitigate response bias, the evaluation team rigorously tested the facilitation and interview guides and protocols to ensure that the purpose of each evaluation question was clear, there were no leading questions, respondents were not primed with information that could skew their responses, and respondents could feel comfortable speaking truthfully. Recall bias may lead to exaggerated negative or positive perceptions of past experiences, as people tend to remember only key aspects of feelings over time. Follow-up interviews, well-crafted data collection guides, appropriate follow-up questions, and the use of secondary data helped the evaluation team mitigate some of the challenges of recall bias. Additional possible limitations to the data collection methods include, but are not limited to: ‡ Lack of data, including baseline data ‡ Potential selection bias as a result of purposive sampling ‡ Possible loss of meaning and data richness due to the use of translators ‡ Silent participants in FGD and/or peer influence ‡ Availability of key respondents during field-based data collection ‡ Respondents· and eYaluators· indiYidual biases ‡ Potential error in generalizing findings to other sites not directly visited. The evaluation team mitigated the above limitations through: ‡ Triangulation and validation of information across different data sources ‡ Rigorous training and testing of data collection instruments and translations thereof ‡ Careful design of data collection instruments to avoid leading questions or other bias USAID.GOV FINAL PERFORMANCE EVALUATION USAID-APIK | 12 ‡ Ensuring that all FGD participants have opportunity to speak ‡ Confirming with ample time in advance the data and time of community visits, FGDs and interviews. 3. FINDINGS AND CONCLUSIONS General Reflections Given the many challenges facing Indonesian communities and their elected officials, APIK has made significant achieYements that can be built upon, and there is urgenc\ to do so. Indonesia·s Yulnerabilit\ to climate change deserves serious ongoing concern, and APIK is to be commended for helping to raise the profile and urgency of climate adaptation by helping to elucidate the many vulnerabilities that must be faced today and in preparation for an uncertain future. In many respects, APIK has been a unique project in the country in its approach to climate adaptation in that it has supported the GoI at all levels to improve communication and coordination, while simultaneously introducing a broad range of pilot activities to enhance community and private sector practices and attitudinal shifts toward improved resilience. The program achieved five of six high-level results targets (Table 4). APIK·s oYerall approach³to achieve place-based resilience through addressing institutional, socioeconomic, and landscape issues³is a highly appropriate theory of change, especially so in the conte[t of the countr\·s decentrali]ation process. The program·s efforts to build resilience through private sector and community-based actions, and to increase the availability and use of climate and weather information by end-users, are equally appropriate and logical as part of any efforts to increase resilience. The eYaluation team finds that some of APIK·s most important achieYements relate to an improYed understanding of climate vulnerabilities and new commitments by sub-national authorities to increase their own investments in, and attention to, climate adaptation and disaster issues. A number of policies and regulations now incorporate climate resilience concerns and actions, as do planning and budgeting documents at national, sub-national and village levels. These are important achievements and set the stage for action by GoI entities to fully fund and implement a number of measures, many of which were co-designed with the help of the project. Another important and significant achievement of the program was the participatory process and preparation of three concept notes in pursuit of Green Climate Fund (GCF) funding to implement additional CCA activities in each of the three provinces. Project preparation for GCF funding is an elaborate process and APIK introduced the requirements to sub-national stakeholders through several socialization workshops which led to multiple concept notes and eventually to the selection of one proposed activity from each province. APIK then assisted local and provincial government officials to develop and submit three proposals for consideration and accreditation by the Fiscal Policy Agency (BKF) of the Ministry of Finance, the administrative unit in Indonesia with responsibility for coordinating GCF proposals. As of early January 2020, all three proposals have been approved by BKF for submission of a full proposal by accredited entities. These hurdles have now been accomplished, and the three proposals are slated to be submitted in 2020 for funding and implementation³a potentially strong legacy achievement of APIK (Table 3). Table 3: GCF proposals initiated through APIK 13 | FINAL PERFORMANCE EVALUATION USAID-APIK USAID.GOV Location Project Title Funds requested (USD) Implementing Organization East Java Community based adaptation through capacity building of local government and stakeholders in 20 districts in East Java $1.8 million Kemitraan (NGO) Southeast Sulawesi Build the adaptive capacity and resilience of communities to the risks of climate change $1.1 million IUCN Maluku Climate change mitigation and adaptation through conservation and restoration of mangrove ecosystems in Aru Islands District $1.3 million IUCN Source: Personal communication with APIK staff As with any large, multi-component program working across several geographies, the evaluation team identified shortcomings that will be addressed in the findings and conclusions below. The sustainability of some of the activities are in question, and the technical rationale and approach for other activities can be challenged. But these are important opportunities for learning and it should be highlighted that the six-month early closure of the program did not allow APIK staff to properly evaluate, reflect upon, and share their own lessons learned as they had hoped. Universally, interviewees were laudatory of the dedication, enthusiasm, and technical support demonstrated by APIK staff, and especially so given the shortened timeframe and budgetary constraints with which they had to contend. In short, the evaluation team finds the overall program approach of APIK to be sound and relevant to the Indonesian context, and notes several significant achievements that can be built upon. Foremost among these is the program·s approach to inter-sectoral coordination and planning. Despite mixed results with respect to a few activities at the sub-national and community levels, strong support and appreciation for the program were expressed by those stakeholders who were interviewed. Table 4: APIK's high-level targets and achievements RESULT NO. INDICATOR UNIT LOP TARGET APIK·s achievements High-Level Result 1 Number of people trained in climate change adaptation or disaster risk reduction (with at least 30% demonstrating increased CCA/DRR capacity) Number of people 30,000 people trained 9,000 people with increased capacity 22,210 people trained 7,733 people with increased capacity High-Level Result 2 Number of institutions with improved capacity to assess or address climate change and natural disaster risks Number public of institutions 40 174 High-Level Result 3 Number of laws, policies, regulations, or standards addressing climate change adaptation or disaster risk reduction formally proposed, adopted, or implemented Number of laws, policies, etc. 50 151 USAID.GOV FINAL PERFORMANCE EVALUATION USAID-APIK | 14 RESULT NO. INDICATOR UNIT LOP TARGET APIK·s achievements High-Level Result 4 Amount of investment mobilized (in USD) for climate change or disaster risk reduction as supported by USG assistance USD equivalent 3,000,000 3,707,30 High-Level Result 5 Number of people supported by the USG to adapt to the effects of climate change or be more resilient to natural disasters (with at least 40% women) Number of people 30,000 people supported 12,000 are women 42,197 people supported 16,246 for women High-Level Result 6 Number of people using climate information or implementing risk-reducing actions to improve resilience to climate change or natural disasters Number of people 4,000 people 6,130 people Adapted from: USAID/APIK, 2019. EQ1: How has APIK built the resilience and adaptation of different levels of government and Whe commXniWieV WhaW iW ZoUkV in, and hoZ haV APIK·V naWional ZoUk influenced or been influenced by the district and community levels? Findings APIK dedicated much of its first two years to building understanding, trust and relationships with program stakeholders. In order to inform discussions and engage stakeholders, program staff facilitated the completion of several assessments, including Climate Vulnerability Assessments, a needs and opportunities assessment related to CWIS, Zhich led to a ´roadmap reportµ, and a comprehensiYe process of assessing multiple data sets to inform site selection. APIK played an important coordinating and convening role in these early discussions by bringing together government agencies, communities and the private sector to better understand the principles of climate adaptation. This, in turn, afforded the opportunity for APIK to assess and learn of the interests and priorities as expressed by the various entities and stakeholders at all levels. In fact, APIK played an important coordinating and convening role throughout its 4.5 years of operations. GoI stakeholders, especially at the sub-national level, expressed strong appreciation for this role that APIK played, and many expressed concern that the coordination and convening of in￾person eYents Zould diminish in the absence of APIK·s presence. An e[ample of this concern relates to the need for ongoing discussions to improve the integration of CCA and DRR tools, guidance and designations across the country. KLHK is responsible for CCA, and uses its own vulnerability assessment tool (SIDIK) to assess vulnerability and adaptive capacity in a given locale. BNPB is responsible for DRR, and uses its own risk assessment tool (InaRISK) to assess vulnerability and risk reduction capacit\. The KLHK process leads to ´Climate Resilient Villageµ designations, Zhile the BNPB process leads to ´Disaster Resilient Villageµ designations. On top of these, there are other designations b\ other goYernment agencies (e.g. ´smart cit\µ, ´child friendl\µ, etc.). Each of these 15 | FINAL PERFORMANCE EVALUATION USAID-APIK USAID.GOV designations have their own criteria, monitoring and reporting, and the result of this stove-piping is inefficiency and confusion. APIK assisted both KLHK and BNPB with their respective tools, guidance, and designations, and continued with efforts that had already begun to integrate CCA and DRR, including with the establishment of a national CCA-DRR working group. But there is still much to be done to improve coordination, consolidation and integration efforts for CCA/DRR. A ke\ feature of APIK·s success at improYing intergoYernmental coordination Zas its permanent presence at the sub-national and community levels. Central ministries and agencies do not have the capacity and resources to provide regular/ongoing assistance to local governments, resulting in large capacity gaps within the decentralization process. For example, BNPB has thus far facilitated about 200 districts/cities with their disaster risk assessments and management plans, but that is only some 39 percent of the total of 514 districts/cities nationZide. The eYaluation team heard praise for APIK·s approach of establishing strong field teams, presence and regular interaction with sub-national stakeholders. Numerous stakeholders stated hoZ APIK had helped to change ´mindsetsµ in terms of understanding the importance and urgency of CCA/DRR. Some said they had previously only heard about mitigation, and that adaptation and resilience were new concepts for them five years ago. During the first two years, APIK staff, in concert with universities, engaged in high-level discussions about risk and vulnerability, while exploring these issues in the context of regional and local priorities. An APIK staff member was embedded in Bappenas and tasked to lead the RAN-API Secretariat in revising the RAN-API, the countr\·s National Action Plan for Climate Adaptation (NAP). The RAN￾API was unveiled at COP 25 in Madrid and, as of now (April 2020), is still under review and awaiting approval in Indonesia. The delay is due to the fact that there remains differing views on whether the RAN-API should be the formal Indonesian NAP. Bappenas is responsible for the RAN-API while KLHK is technically responsible for the NAP. Despite the delay in approving the RAN-API, APIK worked with the Secretariat and in partnership with Bappenas to include climate adaptation in the RPJMN 2020-2024. The evaluation team was told that this was the first time for Indonesia to recognize CCA as a national priority in its RPJMN. APIK reports that about US$3.02 billion (IDR 43.45 trillion17) has now been allocated to climate resilience in four sectors, including agriculture, water, coastal/marine and health. As the program closes out, APIK staff are working with government on the identification of concrete actions and details on how these funds will be budgeted in the annual work plan (RKP). At the provincial level, APIK reports that Southeast Sulawesi allocated about US$4.3 million (IDR 67.5 trillion) for CCA/DRR actions in the RPJMD 2018-2023. At the district/city level, APIK reports that the Central Maluku District allocated about US$27.3 million (Rp. 428.7 trillion) for CCA/DRR for 2017-22, where it had allocated zero dollars for such activities in the previous RPJMD. APIK worked with the Ministry of Villages to develop and adopt tools and guidelines for supporting CCA/DRR actions with Village Funds. APIK also helped Southeast Sulawesi pass a provincial regulation 17 Based on exchange rate of 14,387.41 Rupiah to 1 USD, as posted on June 2, 2020. USAID.GOV FINAL PERFORMANCE EVALUATION USAID-APIK | 16 (Perda No. 11/2019) that now requires local agencies to include CCA in local action plans. The previous Perda was reportedly generic and did not include specific criteria for eligible funding. APIK designed and delivered numerous trainings for GoI and community stakeholders on a range of topics, including but not limited to CVA, M&E for Resilience, GIS and spatial mapping of hazard areas, disaster preparedness and response, disaster resilient schools, use of CWIS, improving water access through spring protection and infiltration, and community-based CCA. In many cases, APIK developed new training materials, and in most cases it made training materials available on its program web site (Table 5). Pre/post testing was applied to assess participant knowledge and learning. Stakeholders were generally positive about such training, and APIK ensured gender diversity in all stakeholder activities. APIK reported that a total of 20,640 people attended training, of which 7,455 (36 percent) were women (Table 6). Table 5: List of APIK training manuals and materials Table 6: Number of persons trained by APIK An APIK-managed Resilience Fund provided grants and a few small contracts totaling about US$650,000 to 12 NGOs, universities and village groups (see Annex F). The Resilience Fund was initiall\ budgeted at US$2 million but Zas pared doZn as a result of the program·s budget shortfall. Activities undertaken focused mainly on village level activities (both livelihood and disaster-related). In addition to activities funded by the Resilience Fund, APIK provided support (e.g., technical assistance, materials and equipment) for various community-scale activities, including Climate Field Schools, disaster preparedness for schools, designations of Climate Resilient Villages and Disaster Resilient Villages, early warning systems, mangrove restoration, water infiltration wells, and more. CWIS was incorporated into virtually all of these activities in one way or another. 17 | FINAL PERFORMANCE EVALUATION USAID-APIK USAID.GOV APIK supported existing Pokmas (village working groups) or in many cases helped to establish new ones as the principal vehicle for community-level implementation. The program also worked with Village Disaster Alert Groups (kelompok siaga bencana, or KSB). APIK helped to establish 28 new Pokmas during the course of the program, including 20 new groups established in PY4. The evaluation team visited 11 communities/villages and found that three of the Pokmas in these villages were either already disbanded or expressed that they did not feel they would continue without APIK support. The evaluation team visited nine of the 12 districts/cities where APIK implemented pilot activities, and was not able to visit all field sites. It is clear from discussions, obserYations and APIK·s oZn reporting that the program supported (through technical assistance and/or with funding) a broad range of implementation measures for the purpose of demonstrating ways to improve climate resilience for communities, individuals and private sector partners. What is not clear to the evaluation team is whether the measures, when looked at in the aggregate, represent a significant level of output and outcome that will be effective and sustainable in improving resilience for a sizable portion of the target populations in the three provinces. Moreover, certain measures were found to be poorly implemented or examples of maladaptation. As an example of poor implementation, of the four mangrove restoration sites (for which APIK provided technical assistance), two of the sites that were planted have failed. Stakeholders at those sites claim various reasons for the failures, including animal herbivory and planting during the dry season. The larger issues of dealing with the drivers of mangrove loss and degradation, and their conservation and management, were not addressed. In another example, LED monitors were provided to five communities for the purpose of receiving BMKG weather information via internet. At least one of these monitors (in Haruku) was no longer working, and in fact had worked for only a couple months before falling into disrepair. Examples of maladaptation include efforts in Haruku to use a fish finder to map the most productive fishing grounds and efforts in Southeast Sulawesi to install Fish Aggregation Devices to increase fish production. These findings will be addressed in recommendations. The evaluation team notes that the above shortcomings are relatively minor in scope or consequence, but nevertheless worth making note of for lessons learned. Conclusions The various early assessments completed and discussions at all levels (central, provincial, district/city and community) were critical in helping to build the interest, trust, willingness and commitment of stakeholders. Overall, this investment in time and resources was necessary and well spent, and in fact should be seen as essential tasks that led eventually to site selection and agreed upon points of entry for program activities. Some of APIK·s most significant achieYements relate to polic\ coordination, planning and budgeting. By taking an evidence-based approach to inform discussions about risk and vulnerability, GoI entities at all levels are more convinced that additional investments and actions are needed to address climate and disaster resilience. That said, there still remain some significant coordination and integration challenges, as evidenced by the struggle to integrate CCA and DRR in a more efficient manner. Ministries and agencies have their own mandates, authorities and roles, and the harmonization of policies, laws and regulations will take several years beginning with discussions among technical counterparts and increased advocacy by local governments for harmonization. USAID.GOV FINAL PERFORMANCE EVALUATION USAID-APIK | 18 APIK has used the same definition of resilience (ketangguhan) that is used by BNPB and that originates from the UN Office for Disaster Risk Reduction (UNDRR18). In future discussions, and to help integrate the separate concepts of CCA and DRR, it may be advantageous to begin using a definition for resilience that incorporates key concepts of both climate and disaster resilience. That will require definitional clarification, and its requirements pertaining to criteria, monitoring and reporting. APIK played a valuable coordinating and convening role that will likely not continue beyond the program. There is no other entity (governmental or non-governmental) that has been identified and capacitated to fulfill this important function. This is seen as a lost opportunity at sustainability and deserves more careful consideration in future programming. APIK did attempt to encourage and build capacity of Bappeda to play this role, and there has been improved interaction between Bappeda and some local elected and civil authorities. Sustained implementation of a convening role for Bappeda, as formalized under Bappenas decree, might be considered as a workable option for future efforts. But as mentioned previously, any solution will likely require institutional capacity, commitment, funding and designated personnel with defined responsibilities. The strong and continuous field presence by APIK staff is an important feature of the overall approach. Program offices and staff in the three provinces enhanced GoI communication and coordination at all levels, and the regular presence of staff has been important in raising the profile and urgency to take action on climate and disaster resilience. Moreover, APIK staff working in collaboration with universities to build capacity and understanding of CCA/DRR issues, and then using these universities as sources of ongoing expertise for local governments, is a good approach and should help to foster sustainability. The academic connections should be considered an essential component of future programming, especially given the ongoing need to link science to policy across multiple disciplines. The RPJMN drives budget allocations for the key central ministries involved in CCA/DRR, but also the development of the sub-national RPJMDs and, in turn, priorities for Village Fund expenditures. The new and increased budget allocations at all levels are important achievements, and the evaluation team believes that the science and technical assistance generated through APIK played an important role in influencing these decisions. The achievement of Perda No. 11/2019 in Southeast Sulawesi deserves special recognition for the potential model it provides for the incorporation of CCA in local planning throughout the country. These are all important achievements given competing priorities, and deserving of post-project monitoring in a few years to see if and how effectively the allocated funds were actually planned, budgeted and spent. That said, follow-on work is needed to cement these gains. Village leadership needs more training and technical assistance on how to incorporate CCA/DRR projects into Village Fund budgets based on Permendes No. 11/2019 (guidelines for Village Fund allocation). Further socialization of key concepts and further strengthening the responsibility of citizens at the village level still needs to happen if there is to be any successful scaling up. Kemendes (Ministry of Villages) has the responsibility for this and it will need considerable assistance and resources as it typically has only one staff member available to 18 UNDRR resilience definition: The ability of a system, community or society exposed to hazards to resist, absorb, accommodate, adapt to, transform and recover from the effects of a hazard in a timely and efficient manner, including through the preservation and restoration of its essential basic structures and functions through risk management. 19 | FINAL PERFORMANCE EVALUATION USAID-APIK USAID.GOV assist four villages. Although local guidelines and some capacity on how to complete CVAs now exist, further help will be needed to replicate and update the assessments and socialize them within local communities. APIK·s numerous capacit\-building efforts received high-marks by all who participated in the training. Workshops appear to have been well organized, supported with the appropriate level and kind of technical expertise, and were highly releYant to stakeholder needs. The program·s success at ensuring maximum participation by women in the various workshops will be an important signal and building block for future efforts. Related to training, it should be noted that GoI staff (at all levels) are generally eager to receive documents and tools (e.g., methods, guidelines, protocols, case-studies, etc.) that can help strengthen their technical capacity and assist with their own self-initiated efforts. One request in particular is for information on the various global funding opportunities in support of CCA, and how to access those funds. Pokmas (community group) are a well-established mechanism for community (village) scale activities in Indonesia, and thus pla\ed an appropriate role in APIK·s approach to implementation. It is not clear why the majority of the work to establish new Pokmas by APIK was done in the penultimate year of the program, but the effort could have benefited from more careful consideration of sustainability. It appears that the more successful and long-standing Pokmas are well integrated into planning and budgeting processes for the use of Village Funds. The aforementioned issue of an ongoing need for an entity to perform the coordinating/convening role that has been served by APIK could perhaps also be tied to this question of finding the right structure and process to ensure that Pokmas remain incentivized and sustained. This may point to the need for some form of institutional anchor at the provincial level with acknowledged responsibility and capacity to sustain the convening/facilitating role that APIK has played so well. Another option to consider for increasing the likelihood of sustainability is to channel local activity support through existing village fund structures where a village government can pass certain accreditation criteria. The selection of local scale implementation measures could have been done in a more strategic, outcome-oriented manner. APIK staff speak about the Yalue of the ´open and fle[ibleµ implementation approach of the program that allowed for opportunism and for sub-national governments and community voices to have a strong say in the selection of implementation measures. But the evaluation team feels that this openness is at the root of questions concerning a level of outcome that will be effective and sustainable in improving resilience for a sizable portion of the target population. The team feels that a narrower set of implementation measures that meet co-designed but prescribed eligibility criteria would allow for more concerted efforts at building the technical skills, learning, replication and sustainability needed to achieve outcomes at greater scale. In consideration of the foregoing, the evaluation team notes that APIK was not designed to focus on a small number of strategically selected adaptation needs. Rather, the program was designed to develop a range of interventions specific to the three different landscapes. According to program staff, the main purpose of the small grants and activities at the community level was to build an understanding of the types of activities that government and communities can do to build resilience and to stimulate government funding or to catalyze private sector investment. Finally, more careful selection, design and implementation of adaptation measures are needed in future programming. Mangrove restoration, for example, is a much more technically demanding endeavor than most people realize, especially if it is to be successfully accomplished and sustained at a scale and in a manner that has significance for improved resilience. There are far more failed mangrove restoration efforts in the world than there are successful ones, but with careful design and technical USAID.GOV FINAL PERFORMANCE EVALUATION USAID-APIK | 20 input they can be successful (see EQ4 below). Similarly, interventions in the fisheries sector should be undertaken only by experienced fisheries managers. Efforts to increase fish production in areas of declining fish catch³but in the absence of measures to manage the fishery itself³should be considered as counter to the goals of increased long-term resilience. EQ2: HoZ ZaV APIK·V ZoUk ZiWh Whe pUiYaWe VecWoU able Wo incUeaVe aZaUeneVV aboXW adaptation and disaster risk reduction? Findings APIK was requested to engage with the private sector to increase awareness of climate and disaster risks, and identif\ actions that can mitigate risks to businesses and their communities. APIK·s SOW notes that hydro-meteorological disasters can significantly impact the private sector by causing losses to business, disrupting suppl\ chains and damaging economic sectors important to Indonesia·s economy, including in the agricultural, transport and extractive industry sectors. Illustrative activities cited in the SOW mentioned enhancing cooperation with relevant business associations, such as Kadin and AmCham; strengthening tools and analysis to help the private sector understand climate risks and integrate risk reduction into their operations; and helping to build resilience into community and private sector operations across target geographies. Private sector engagement also featured as a cross-cutting theme in APIK·s Zork. The focus of this work was to engage the private sector to support activities that can enhance community resilience. In its PY4 Work Plan, APIK indicated it would work toward encouraging private sector partners to move beyond Corporate Social Responsibility (CSR) to support sustainable resilience activities. Examples of work in this area included working with the poultry industry in Blitar, supporting clean water protection in Mojokerto, and supporting commercial seaweed and fishery activity in South Konawe. PriYate sector coordination Zas principall\ managed b\ APIK·s Jakarta-based PSE Advisor. Field coordinators helped to carry out activities developed from Jakarta and worked to develop local actiYities. An e[ample of the latter is APIK·s Zork in Southeast SulaZesi to help Kampung Salo Zrite a proposal to secure funds for a ´Zaste bankµ (plastics rec\cling center) from Pertamina·s CSR fund. Over the first tZo \ears, APIK·s approach inYolYed outreach eYents and meetings to deYelop contacts with a variety of companies to identify opportunities for collaboration. For example, presentations were delivered at business association meetings (e.g., Kadin, AmCham and Indonesia Business Links). This approach resulted in some expressed interest to collaborate but did not lead to the development of specific activities. According to program staff, despite regular follow-up, it was learned that these associations are important conveners but not implementers. The evaluation team believes that not enough was done to stimulate analysis and interest in market-based solutions that would lead to positive return-on-investment (ROI) for the companies. APIK changed its personnel and approach in PY3 and, with a new PSE Advisor, began to work directly with a dozen or so companies on conceptualizing and developing specific small-scale partnership activities. These activities, such as a collaboration with PT Cargill to improve egg production in East Java, were funded through corporate CSR funds and developed under MOU arrangements. APIK·s role in deYeloping PSE actiYities Zas to act as an initiator and conYener to bring together local government and communities in ways that encouraged the private sector to contribute to project goals, mainly around resilience and livelihood activities. The program facilitated a range of activities to encourage involvement of private companies, including for example bamboo planting, chicken shed 21 | FINAL PERFORMANCE EVALUATION USAID-APIK USAID.GOV construction, and support for BMKG Climate Field Schools to improve the resilience and production of cacao, corn, sugar cane, and other commodities.19 In some cases, support for Climate Field Schools involved private sector participation, but this was not always the case. Support was also provided for seaweed cultivation and facilitation of community proposals for CSR funding. APIK private sector activities have produced generally positive results in terms of implementation success and in producing benefits. For example, the Climate Field School for Corn in Southeast Konawe helped raise the income of the 24 farmers who participated. The collaboration with PT Multi Bintang resulted in bamboo seedlings being planted to protect an upland groundwater protection area. The PT Cargill collaboration resulted in two improved chicken sheds being completed as demonstration projects. APIK·s projects Zere small-scale and involved relatively few direct beneficiaries. Typically, a training activity would involve a single training session for 20 to 30 farmers, and land rehabilitation involved a few hectares or less of bamboo planting. All of the implemented private sector partnerships relied on CSR funding as opposed to investments from corporate operational budgets that actually change the way of doing business. It is not possible to determine to what degree there has been replication and scaling of the private sector activities. Program reporting is not sufficiently detailed, but it may also be that projects ended too early to conduct such measurements. While activity approaches are documented, the performance of the activities themselves are not fully reported. For example, under the Climate Field School for Corn, activity reporting mentions that ACA was to provide crop insurance and Bank Sultra was to provide loans, but reporting did not mention that neither loans nor insurance were ultimately provided. The team learned from stakeholders, however, that in fact Bank BRI provided loans to farmers since the loan process managed by Bank Sultra was too cumbersome and time consuming. The PT Cargill activity appears to be the most likely of the private sector activities to be sustained, especially given that the National Layers Association (PPRN) has indicated its interest to assist replication of the poultry sheds to other parts of the country. For other activities, there does not appear to be similar commitments or plans from local organizations. In some cases, such as the corn and cacao Climate Field Schools, the activities appear to have been designed as one-off interventions, premised on assumptions that the training and PSE would continue through a process of self-interest by involved participants. Presumably, BMKG will continue to offer Climate Field School training, but it is not clear if there will be anyone to encourage broader PSE. According to APIK reporting, by the end of PY4, APIK had successfully engaged with 13 companies in four sectors, namely agriculture, poultry, agro-forestry and 19 The Climate Field Schools program has been operated by BMKG since 2010. Improved egg-laying shed, East Java Photo: ‹ USAID-APIK USAID.GOV FINAL PERFORMANCE EVALUATION USAID-APIK | 22 water (Table 7). APIK had contacted additional companies about developing other activities, but these activities failed to gain traction. A total of US$99,203 in leveraged contributions from private companies was reported in association with the activities presented in Table 7. The businesses themselves (mainly PT Cargill and Corn SLI) indicated that their engagement with APIK is expected to create around US$9.3 million in value over the coming five-year period. The formulas and assumptions behind this number were not available for the evaluation team to analyze. Conclusions APIK played a valuable role in convening the participation of companies, communities and sub-national governments in developing local climate mitigation and resilience activities and starting pilot projects with a dozen or so companies. Such co-creation activities are a good starting point toward building more long-lasting and sustainable market-driven solutions. An example of this would be PT Cargill·s effort to work with chicken farmers to develop better chicken shed solutions to increase production. Program activities were appreciated by the communities and stakeholders who participated. The projects appear to have mostly succeeded in meeting their individual objectives. With respect to the Climate Field School activities, it is clear that farmers were pleased with results of their first season of planting after participating in CFS training. Some farmers even reported a doubling of production (yield/hectare), and were eager to share their new techniques with other farmers. APIK assisted this process through a training-of-trainers effort to help replicate and scale up. The farmers are pleased to now be able to get BMKG weather reports through WhatsApp. But with these positives in mind, the evaluation team believes that further analysis is warranted regarding the use of CFS as a replication/scaling strategy. BMKG will likely continue the CFS on its own, as it is their program. It is therefore not clear to the evaluation team exactly how future USAID-funded efforts might best engage with CFS activities; there needs to be better understanding of the APIK approach and how it can be improved. The evaluation team feels that private sector activities cannot likely be sustained unless market-based solutions are implemented, or counterpart organizations take ownership of the activities. Most of the PSE activities do not have a counterpart committed to continuing the work that was begun. APIK was successful at helping communities engage with the private sector, for example by guiding communities in the use of Village Funds. These are a source of funds that can be used to leverage private sector CSR funds to support small-scale projects. This was the case, for example, with the development of the plastics recycling center in Southeast Sulawesi. APIK·s design, hoZeYer, Zas not optimal for creating market-based partnerships, as the timeframe, personnel and financial resources were inadequate to undertake the level of market analysis and value propositions required. Convincing companies to redesign their approaches generally requires well designed efforts over several years that need to be underpinned by clear business value proposition demonstration and analysis. Market-based solutions to change a compan\·s business practices are more difficult to develop than are CSR-funded programs, but offer greater promise in terms of scale and sustainability. Private sector activities were small scale and usually only involved a few dozen direct participants. Corporate investment has been minimal³about US$99,000 in total³a significant portion of which was from in-kind contributions. Most of the private sector activities have been one-off and based on CSR rather than Corporate Shared Value (CSV). 23 | FINAL PERFORMANCE EVALUATION USAID-APIK USAID.GOV Table 7: APIK'S private sector partnerships Partners Role & Approach Results 1. Climate Smart Chicken Shed, Blitar, East Java 1a PT Cargill Indonesia APIK was the convener of this activity to build and demonstrate a chicken shed that will improve climate resilience by increasing commercial egg production. Funding: CSR contribution for design and construction Two demonstration chicken sheds (for 2,000 chickens) were constructed by Cargill. The effectiveness will be evaluated over time and PPRN will encourage replication by egg producers in other locations. 1b National Layers Association - PPRN Role is to increase outreach to expand participation by egg producers. 2. Conservation-based Tourism & Disaster Risk Mitigation, Mojokerto, East Java 2a Yayasan Sahabat Multi Bintang APIK prepared a CVA, which identified risk of landslides in an upstream area. This area also protects water supply for Bintang (for beer production) and urban use. Funding: US$ 22,600 to co-finance CVA and for technical assistance on tourism development. 0.75 acres of bamboo planted (4,400 bamboo seedlings) in upstream watershed to provide water source protection and tourism opportunities. The tourism village was not built as the project ended early. 3. Climate Field School for Corn Cultivation, Konawe, Southeast Sulawesi 3a PT Syngenta Indonesia APIK brought together private sector companies to support the BMKG Climate Field School for corn. Syngenta provided free hybrid seeds for demo plots and technical assistance; Bank Sultra was to provide Rp$11 million loans to farmers for agricultural inputs; ACA Asuransi was to provide crop protection insurance; PT Pupuk Kaltim sold/provided fertilizer. 24 farmers were trained and farmed a demonstration plot. The demonstration plot increased corn yields by about 50%, and increased income. No bank loans were approved or issued. No crop insurance was provided. APIK reported that around 85 farmers have used the knowledge gained in planting 170 hectares of land during the planting seasons until August 2019. 3b PT Bank Sultra 3c PT ACA Asuransi USAID.GOV FINAL PERFORMANCE EVALUATION USAID-APIK | 24 Partners Role & Approach Results 3d PT Pupuk Kaltim Funding: Donation of seeds for demo plot by Syngenta 4. Climate Field School for Cacao Cultivation, Konawe, Southeast Sulawesi 4a PT Kalla Kakao Industri (KKI) APIK initiated Kalla Kakao·s inYolYement in the BMKG Climate Field School for cacao. One training was held for 20 or so farmers, which focused on using climate and weather information, and improving crop management. Funding: $16,226 for venue, logistics and training (in-kind) 30 KKI Farmers were trained as master trainers. There are no current plans to continue the initiative as Kalla Kakao is struggling with processing issues and thus does not plan further CSR support. The impact on farmer productivity is unclear (has not been measured as program has stopped). Source: APIK PY4 Annual Report. It was reported that the above activities generated US$ 99,203.36 in contributions from the private sector, including for materials and in-kind contributions for technical expertise and the use of facilities. 25 | FINAL PERFORMANCE EVALUATION USAID-APIK USAID.GOV EQ3: To what extent are targeted beneficiaries better prepared and ready to adapt as a UeVXlW of APIK·V climaWe and ZeaWheU infoUmaWion VeUYiceV? Findings APIK invested much of PY1 to undertake an assessment of needs and opportunities for improving CWIS as proYided b\ BMKG. This led to the preparation of a ¶roadmap report· that identified seYeral actions upon which to focus APIK efforts. Many of those actions were targeted at KLHK and BNPB as the two agencies that could benefit the most from improved CWIS. The roadmap also noted the importance of improved CWIS for the Climate Field Schools by making weather information more available to farmers and fishers. APIK also supported BNPB during PY1 to improve and roll-out the InAware system and the use of its risk/vulnerability data for the development of early warning systems (EWS). APIK assisted BMKG to develop OtoKlim, a web-based tool to display local weather data and information in spatial form to make the information more understandable and accessible at the sub￾national and community levels. OtoKlim was developed to be used in the three APIK provinces initially, and at this time it has not yet been replicated or shared elsewhere. APIK worked with BMKG to develop a website for each province to display provincial scale weather information and provided training to increase local government understanding and use of the information in conjunction with the installation of flood EWS. Unfortunately, the provincial websites were no longer accessible at the time of the evaluation. APIK provided technical assistance to local governments (e.g. Bappeda Kendari City, Bappeda Ambon City) to integrate the Resilient City Index and climate information into the RPJMDs. A key component of APIK assistance to BMKG was capacity building for improved downscaling of climate data. APIK helped to provide downscaled climate projection data for each of the provinces to help with strategic environmental assessment and mid-term development planning. Particularly in Maluku Province, APIK helped to provide the downscaled climate data since that province in particular had no such data. APIK brought an expert from the National Center for Atmospheric Research (NCAR) who prepared a report on needs and opportunities. The expert was initially supposed to return to deliver training on downscaling, but then NCAR determined that it would be too resource￾intensive to do downscaling properly in Indonesia. Rather, they advised that APIK could make a valuable contribution by helping to build BMKG·s capacit\ to undertake statistical Yerification and hindcasting for improved seasonal forecasting, and this is the type of training ultimately provided by NCAR. NCAR continued to provide remote support to BMKG to help staff better understand the seasonal forecasting and storm monitoring technologies. The evaluation team heard only positive feedback from participants involved in the training. Project beneficiaries are generally very appreciative of the improved access to BMKG weather information (either through WhatsApp groups, other social media, hydro-met gauges, or via the monitors provided by APIK to display weather information). All of the Climate Field Schools have benefited from improved weather information. However, the ongoing operation and maintenance costs for local weather information in display monitors is an unresolved issue that has resulted in systems that falling into disrepair (e.g. Haruku, since September 2019). USAID.GOV FINAL PERFORMANCE EVALUATION USAID-APIK | 26 APIK supported many efforts to improve community-scale EWS and to link those to information provided by government entities and improved weather information. For example, APIK provided communities on two rivers in East Java (Kalikatir and Panguluran) with flood EWS. Flow sensors and rain gauges were installed at Dillem Village crossed by the Klorak River, and data from these devices were made accessible to BPBD via the internet (which does not work during times of electrical blackout). Water discharge and rain gauges were installed in Sukodono and Kedung Banteng villages, as well as two display monitors in Kalikatir Village. APIK installed a rainfall gauge and provided training for farmers in Konawe Selatan. APIK later uninstalled the gauge to take to another location, but the weather data are ostensibly still available through the BMKG weather app (BMKG sultra) via smartphone. With help from a university in Southeast Sulawesi, landslide EWS components were installed at Salo Village, but stakeholders indicate the system has not yet worked. While some of these systems are still working and others are not, stakeholders are very appreciative of these local EWS and weather gauges now made available to local governments and communities. There is, however, an expectation that the local EWS installed by APIK will be connected with the larger provincial EWS, so that BPBD can also monitor local conditions. Open Street Map (Ushahidi system 20 ) was introduced by APIK in PY4 to encourage citizen involvement in disaster risk mapping. The mapping is intended to provide local governments and citizens with near-real time information regarding hydro￾meteorological disasters for preparedness and response. The evaluation team heard that more villages are proposing to become a Disaster Resilient Village in 2020 as a result of their involvement in this training and their new awareness of CCA/DRR issues. Training has been a large part of APIK·s Zork plan each year, and much of that training has focused on the access and use of CWIS for CCA/DRR. Each of the Climate Field Schools had a strong CWIS component to aid decisions about the timing of planting, and even about appropriate seed varieties. The Haruku island community was trained in using CWIS (especially wind and wave data) for improved safety at sea. In fact, there have been elements of CWIS in all aspects of the training. The key challenge remains less the reliability and accuracy of weather information (although there are improvements to be made), but rather dissemination and timely access to the information by all communities and citizens, and in ways that the information is easily understandable. 20 https://www.ushahidi.com/ Flood Early Warning System equipment, Sitiarjo Village, East Java. Photo: ‹ USAID-APIK 27 | FINAL PERFORMANCE EVALUATION USAID-APIK USAID.GOV Box 1: CWIS for optimal planting time APIK partnership with BMKG Climate Field School for Corn leads to impressive yield increases for farmers in Southeast Sulawesi. Working together Zith Indonesia·s Agenc\ for Meteorolog\, Climatolog\ and Geophysics (BMKG), the Ministry of Agriculture and private sector agricultural input suppliers, APIK provided training to farmers on how to use climate and weather information to determine the optimal time for crop planting. The training included building skills and knowledge in understanding climate and weather variability, accessing weather and climate forecasts through a BMKG weather app, and using practical tools such as rain gauges. This knowledge was combined with training provided by private sector agricultural input supply firms on how to use and apply fertilizer and pesticides. The farmers who participated in planting demonstration plots indicated that yields increased from about 8 tons per hectare to nearly 12 tons per hectare. The Climate Field School for Corn training in Southeast Sulawesi was provided to more than 20 farmers who represented a total of 11 farmer groups, including two all-female groups. Each member who participated in the training went on to train members of their respective farmer groups, which will allow the program and benefits to expand. The private sector agricultural input suppliers, and BMKG climate specialists, are positioned to continue to provide planting advice and services to area farmers. The farmers who were trained indicated they have continued to access and use the weather and climate information provided by BMKG, and said they were very pleased with the advice they had received, as it had resulted in significant increases in income. Conclusions The needs and opportunities assessment carried out in PY1 that led to the ¶roadmap report· Zas an important task that helped delineate work plan objectives and tasks for the following years. The report helped to focus work on improving the InAware system, OtoKlim system, and a website that houses location-specific weather information for each province. Unfortunately, the BMKG website has not been fully functional for all provinces and districts/cities. It is possible that some of the problems could have been rectified had it not been for the early closure and budget cuts of APIK. With respect to the SIDIK and InaRISK tools, stakeholders suggested that there was still work to be done to improve the climate data incorporated into these tools. This relates largely to the view of many in Indonesia that more accurate climate-model downscaling is needed at greater resolution for all locales. Although this remains an ongoing debate within GoI circles, the evaluation team is of the opinion that climate-model downscaling, while useful, is a resource-intensive and sophisticated technical challenge for the ocean/atmosphere complexity of Indonesia. Certain parameters, such as temperature, will be more accurately downscaled than others, such as precipitation. While many GoI officials, including those at BMKG, would like to see more support for improved climate-model downscaling, other stakeholders feel it is even more important to strengthen the capacity of end-users to put to effective use the data and information that are currently available. This view argues that it is more important for local governments and communities to understand the simple basics of climate change for their locale, in other words, the seasonal variability and the direction (increasing/decreasing) and magnitude of change, while recognizing that the absence of more accurate climate projections should not be a constraint to action-planning and implementation. In this regard, more support is needed for BMKG·s Climate Application Center to deYelop simple communication language and tools to help local governments and communities identify their particular CWIS needs and how best to apply the climate and weather information. For example, it is probably best to speak about specific events/impacts of concern to local citizens (e.g. related to planting, safety at sea, or potential flooding). In this regard, BMKG at the national level expressed interest in the idea of a USAID.GOV FINAL PERFORMANCE EVALUATION USAID-APIK | 28 ´dashboardµ that is both public-facing and internal to BMKG needs (for something like a ´situation roomµ). The tools, devices and gauges installed in communities to aid hazard early warning or simply to provide near-real time weather information are appreciated by the communities and can provide a useful demonstration effect for replication elsewhere. But more attention needs to be given to the details of ongoing O&M of equipment, and alternative energy sources during blackout. Linkage of community scale EWS with provincial EWS is advantageous for efficiency, consistent messaging and O&M objectives. The use of social media to disseminate climate and weather information can have serious drawbacks in the event that the information being shared is not timely, accurate, credible or complete. BMKG has standard operating procedures with respect to public dissemination of weather information, and has concerns regarding the use of social media as an alternative to official communications coming from BMKG. There were times when BMKG had to correct wrong or misleading information conveyed through social media. This is especially relevant to EWS-related information and the requirements that those alerts come through official channels and with a time-stamp. Overall, APIK stakeholders today have improved understanding of CCA/DRR and the data and information needed to remain alert to, and take action on, climate impacts. Local governments and communities, at the very least, should now be in a better position to advocate on their own behalf for continuing efforts and funding to improve disaster preparedness, and for the necessary tools and information that will help safeguard their livelihoods. The CWIS component of APIK has played an important role in helping to focus attention on the science and evidence-base that will hopefully sustain and increase GoI attention to CCA/DRR in the years ahead. Box 2: Community-based early warning systems Early warning systems (EWS) are important tools for DRR. The installment of EWS are important to provide warning to a community prone to disaster. APIK focuses on hydro-meteorological disasters, especially those related to flood and landslide risk. On flood EWS, APIK installed several EWS in East Java to ensure that communities are informed and prepared to evacuate during floods. Thes EWS include low cost rain gauges. EWS were installed in some spots and can be accessed by internet and used by the local BPBD. The aim is to integrate community EWS with the information provided by government entities. The assisted villages are also involved in several activities related to flood risk assessment. The activities among others are the establishment of Disaster Alert Community (Kelompok Siaga Bencana, or KSB), community training and disaster simulation. APIK collaborated with Yayasan Pattiro to develop a contingency plan EWS (for flooding) in Sitiarjo (East Java) and worked with Aplikasi Cerdas Indonesia to implement a community based EWS alongside the Klorak River. In Southeast Sulawesi, APIK and BPBD Kendari conducted a joint assessment for flood mitigation measures in Kampung Salo. EWS for landslides were installed with help from UGM, but the system (involving sensors and loudspeaker warnings) is currently not functioning. APIK also conducted training sessions on climate and weather literacy in the Aru Islands in Maluku where APIK and the Aru government agreed to conduct follow-up activities such as continuing to raise CWIS awareness and develop a flood EWS. APIK with BPBD Kendari supported EWS in Amohulo and Lepo-Lepo Village for river water level monitoring to avoid floods and Salo Village for landslide monitoring. In Maluku, APIK provided training and installed an Automatic Rain Gauge in Aru to collect rainfall data. The rainfall data are then linked with the EWS to inform and provide warning of excessive rainfall. As for landslide monitoring, the system aims to predict rainfall-induced shallow landslides. It has several main components: (i) a susceptibility mapping and hotspot identification component based on a land surface geospatial database (e.g., topographical information, maps of soil properties, and local landslide inventory, etc.); and (ii) a physically based, rainfall-induced landslide prediction model. ´APIK taught us that Ze must consider the weather when planting. APIK climate school alumni are now very smart about the Zeather.µ ~ Southeast Sulawesi farmer 29 | FINAL PERFORMANCE EVALUATION USAID-APIK USAID.GOV EQ4: What are the key lessons learned from application of the landscape approach to the three areas of: watershed; coastal; and small islands? Findings APIK undertook early assessments at the landscape/provincial scale in all three provinces, including the aforementioned Climate Vulnerability Assessments and site selection analyses involving several datasets describing disaster risk and hazard vulnerability. The CVAs used the KRAPI methodology developed by the Bandung Institute of Technology (ITB), under guidelines issued by KLHK in 2016 and 2018, and under guidelines on disaster risk assessment issued by BNPB in 2012. KRAPI requires the use of scientific data to underpin the hydro-met portion of the analysis; data which are not always readily available for every landscape in Indonesia. KLHK is the GoI entity responsible for periodic updates to the CVAs, and they have adopted a new ministerial guideline to update and undertake new CVAs based on methodology that was developed with APIK assistance. In the case of the three APIK provinces, the climate projections were available for East Java and Southeast Sulawesi, but not Maluku. In Maluku, this initially presented some credibility issues. The CVAs that were completed were well received and appreciated by the various sub-national governments. When APIK learned that provincial scale CVAs had limited use at the local scale (as their resolution is too coarse), they prepared community-scale CVAs to address smaller-scale contexts, while involving stakeholders to drive local understanding and commitment. APIK conducted training for local government staff, who later led CVA training for other districts with the assistance of APIK staff. As for PSE in the landscape approach, most (if not the only) interest was demonstrated by companies that rely on water as a key input to their operations. For example, the beverage company PT Multi Bintang Indonesia engaged with districts in East Java to clean and restore functions of the Cumpleng River, a tributary to the Brantas River. Building upon the evidence-base from the landscape analyses, APIK then used the information to influence government (at all levels), and private sector planning and budgeting processes, and decisions. APIK prepared detailed vulnerability and hazard maps for each of the three provinces to inform local action planning and budgeting. Stakeholders learned that the vulnerability of a place is interconnected with the ecological, hydrological and meteorological characteristics of the surrounding landscape, and that sound land use decisions that safeguard ecosystem services are critical for protecting urban, rural and island communities from extreme and changing weather. The landscape approach is not a new concept in Indonesia, but its application is limited. For Maluku, we heard that the landscape approach had been in use for spatial planning, but until APIK it was not used for adaptation or disaster management. With respect to implementing the landscape approach, APIK was able to influence governments in Southeast Sulawesi and East Java to undertake landscape scale improvements. The collaboration in Southeast Sulawesi was focused on better management of the Wanggu Watershed in South Konawe District and Kendari City with planned construction of storm water drainage improvements. In East Java, vulnerability assessments of springs in the Upper Brantas were completed in partnership with the IUWASH Plus program. Infiltration wells to improve ground water recharge and reduce flooding were constructed in multiple locations in Batu City, Malang District and Malang City. APIK also worked on improved water access in Malang Raya (on spring water protection). USAID.GOV FINAL PERFORMANCE EVALUATION USAID-APIK | 30 In addition, reforestation efforts involving bamboo, fruit trees and mangrove were carried out in several watersheds and coastal areas. APIK reports that 21 hectares of upper watershed land were rehabilitated with bamboo and six hectares of coastal mangroves planted in PY4. Due to early closure of the program, APIK staff indicated that they were unable to return to these restoration sites to conduct follow-up monitoring. The evaluation team learned from stakeholders that in at least two locations mangrove seedlings had died or been destroyed by animal herbivory (e.g., Haruku, Maluku and Segoro Tambak, East Java). Undertaking implementation measures to redress landscape scale issues were described by sub￾national government entities as a challenge. Administrative boundaries define planning and budgeting processes, and decisions to collaborate across government entities require MOUs and thus decisions by various levels of elected leadership. Funds from different administrative jurisdictions cannot be co￾mingled into a single contract, hampering implementation. The three provinces selected for implementation were described by APIK as representing different types of landscapes and associated issues, including watershed, coastal zone and small-island landscapes. In East Java, APIK activities are centered in the population dense Brantas Watershed; in Southeast Sulawesi on coastal landscapes; in Maluku on remote small island landscapes. In Maluku, APIK based its landscape approach on the idea that the connectivity of the islands is being affected by climate change. For example, with more extreme weather events and unpredictable storms, travel between islands can be more difficult and prevent people accessing markets or getting to doctors or hospitals. Conclusions The landscape scale assessments (in particular the CVAs and site selection assessments) were important precursors for stakeholder engagement. Analyses at that scale helped to inform dialogues about the macro-scale issues that might otherwise not be dealt with, and to bring stakeholders together around a shared understanding of their mutual interests and priorities for planning purposes. APIK·s Zillingness to reassess climate Yulnerabilities Zith greater resolution at the district/city/community scale was important for stakeholder trust and acceptance of proposed local action plans. The various Dinas agencies at the sub-national level were especially appreciative of the landscape approach concepts and analyses, as they help to justify and support some of the work they feel is needed but otherwise do not have sufficient information about. The landscape approach is an important unifying principle for efforts in both adaptation and mitigation. It recognizes that sustainable ecosystem services, and thus resilient systems, depend on healthy ecosystems with their natural hydrological functions intact. APIK performed a valuable service in bringing these concepts to discussions at all levels of government, and by including landscape issues, including hazards and other contextual factors, in its analyses for site selection and assessments of disaster risk and climate vulnerability. The approach to focus on particular vulnerabilities and hazards of each target geography helped to instill commitment in local stakeholders to own and make decisions that will underpin landscape resilience. Stakeholders (including both government and private citizens) now have a better understanding of the ecosystem services and connectivity issues at landscape scale as they relate to climate and disaster related vulnerability. APIK performed a valuable service in bringing these concepts to discussions at all levels of government, and by including landscape issues, including hazards and other contextual factors, into analyses for site selection and assessments of disaster risk and climate vulnerability. 31 | FINAL PERFORMANCE EVALUATION USAID-APIK USAID.GOV Implementing a landscape approach requires that actions be taken to redress land degradation, to relocate or adapt poorl\ sited or constructed infrastructure, and otherZise ensure that a landscape·s natural defenses against extreme weather events and geophysical hazards are as strong as possible. In other words, it requires action to restore natural functions with the use of ecosystem-based adaptation (EbA) tools and techniques. We might more appropriately call this EbA and Mitigation given what we know about the carbon sequestration value of forests, mangroves, other wetlands and healthy soils. APIK·s efforts are an important foundation for future efforts to address landscape-scale planning and restoration. Achieving such action is challenging anywhere but especially in Indonesia. As APIK has experienced, gaining the political will and budgetary commitments to tackle cross-administrative boundary challenges at the appropriate scale is anything but straight-forward. Administrative boundaries define local budgets and actions, and inter-governmental cooperation requires MOUs that must be approved by various levels of elected leadership. And as with all local plans, such actions must directly support the strategies and plans spelled out in planning and budgeting documents that ultimately link with the RPJMN. The term ´landscape approachµ can benefit from greater clarification of key concepts, definitions and practical guidance on how to implement a landscape approach as a foundational component of CCA/DRR. The term has greater intuitive sense when used with the features of a landscape or landscape/seascape that can be restored and managed for ecological integrity. Thus, restoration efforts involving a watershed, river basin or coastal zone can be undertaken and monitored over time to evaluate progress toward improved resilience. However, the idea of restoration efforts inYolYing ´a small island clusterµ is conceptuall\ different. The eYaluation team suggests that with respect to the small island cluster as approached by APIK, concepts of a ´s\stems approachµ Zould be more appropriate. In this regard, resilience efforts were focused on the linkages between human communities and the services offered (e.g., transportation, communications, medical services), as opposed to issues of ecological integrity. Should the focus of resilience efforts be on improving the ecological integrity of the marine environment³such as through a system of marine protected areas and/or fisheries management measures³then such an approach can legitimately be called a seascape approach. Regardless of terminology, the ultimate objective of a landscape approach is to implement measures that restore ecosystem services and functions at a scale that is significant in redressing the climate risks and disaster hazards identified. Further efforts are needed to demonstrate the actual implementation, effectiveness, and value of the landscape approach, and to institutionalize responsibility and capacity for doing so in the appropriate GoI entity, with perhaps technical assistance from academia. An example of what this might look like for mangrove restoration efforts in Indonesia can be found in Box 3. ´I think the landscape approach has several benefits, as it induces stakeholder collaboration, integrates ecological benefits in protecting the Brantas Watershed, and it makes it easier for us to coordinate all the interests within the Watershed.µ ~ Local stakeholder in Batu City, East Java USAID.GOV FINAL PERFORMANCE EVALUATION USAID-APIK | 32 Box 3: A systems approach to mangrove restoration Restoring mangroves and coastal resilience: Lessons from North Java The multiple benefits of mangrove restoration at scale and in a holistic manner can be witnessed along the coast of Semarang and Demak in North Java where stakeholders have restored a 20-km belt of mangroves, introduced sustainable aquaculture and reduced groundwater extraction. More than 70,000 people now enjoy increased resilience together with benefits for carbon storage, biodiversity and fisheries. Through the efforts of an innovative program called Building with Nature in Indonesia, local communities have participated in a planning and implementation process to restore a 20-km stretch of receding coastline and mangroves that now provide significant and sustainable benefits for coastal protection, aquaculture and improved quality of groundwater. The approach acknowledges that to restore the mangrove system it is not possible to simply plant mangroves, as the wave exposure, submersion time and sediment conditions in the eroded area are not optimal anymore. A temporary semi-permeable barrier was first built from poles and brushwood to dampen the waves and capture sediment, creating sheltered areas near the coast for accretion. Once the nearshore bed level rises enough, mangroves will regenerate naturally, developing a natural defense that will protect the hinterland from further erosion. Improved practices for sustainable aquaculture were then adopted to ensure mangrove protection and other environmental and economic benefits. Control measures are now based on community bylaws and rooted in community development plans. Building with Nature is innovative and site-specific, and it operates in areas where the understanding of systems is limited and where changes are dynamic. The approach demonstrated in Demak can be replicated in other parts of Indonesia prone to coastal degradation and is an important adaptation measure against sea-level rise. For more information: https://www.ecoshape.org/en/projects/building-with￾nature-indonesia/ Mangrove restoration efforts inside an aquaculture pond. Awunio Village, Southeast Sulawesi. Photo: ‹ USAID-APIK 33 | FINAL PERFORMANCE EVALUATION USAID-APIK USAID.GOV EQ5: What was the impact of the funding challenges and what opportunities were missed as a result? Findings APIK began its work in late 2015 with approved total funding for a US$19.3 million five-year program. In 2017, APIK was told by USAID that it was likely that the program would not receive the full funding amount and asked APIK to limit monthly expenditures to US$300,000. Ultimately, and after a six￾month earlier than planned program close out, total funding equaled US$15.4 million, close to a 20 percent reduction from the approved contract. In more specific terms, APIK had to complete the program in 88 percent of the original timeframe and 79 percent of the original budget. APIK was told of the likely budget shortfall in PY2, but there was no official contract modification and the early closure and budget shortfall was not made publicly known until the beginning of PY5 (October 2019). With official notice finally given, APIK field staff had to scramble to complete ongoing activities and inform many stakeholders that their planned activities could not be funded. Stakeholders expressed to the evaluation team that there was confusion and frustration during this period. The most significant result of the reduced funding was that many local activities could not be implemented or fully completed. Some examples of activities curtailed or canceled include: (i) a weather information monitor that was promised in Maluku but not installed; (ii) planned activities for mangrove and watermelon planting in East Java were canceled; (iii) activities of the Climate Village Program in Maluku were scaled back, and a planned integrated water management and clean water project was canceled; (iv) after a completed participatory CVA in four villages of Mojokerto (East Java), there were no implementation or follow-up activities; and (v) mangrove restoration efforts in Kepetingan (East Java) were canceled. Significantly, the Resilience Fund, originally budgeted at US$2 million was scaled back to US$650,000, a budget shortfall of two-thirds. Many villages had to be told by APIK field staff that their expectations could not be met. Importantl\, APIK·s oZn plans for using PY5 to eYaluate, reflect upon and share lessons Zere not able to play out as hoped, and APIK staff claimed that some replication efforts were also cut short. Because community projects could not be implemented to the extent planned, APIK was limited in its ability to assess the effectiveness of its approach, or to generate and share lessons as planned. Nevertheless, when asked during interviews and FGDs if they had any recommendations for future programming, stakeholders were universally positive about APIK, and expressed strong enthusiasm and hope for a possible program extension or a new similar program to allow them to continue this important work. The evaluation team heard no condescension or lack of trust about USAID-funded activities; in fact, quite the opposite was heard in support and encouragement of renewed funding and technical assistance for CCA/DRR in their locales. Conclusions APIK·s strateg\ called for balancing planning and institutional support Zith implementing small, community-scale CCA/DRR activities. That is an important approach and should be a key principle of future programming. But USAID and APIK could have been more proactive in managing and communicating to stakeholders the mandated budget cuts which curtailed many of the private sector and community activities. Full clarity on the extent of budget cuts was provided by USAID only at the beginning of the final year of implementation, and although this did not allow time for APIK to adjust staffing patterns or work plans, a more concerted communication strategy to inform stakeholders of USAID.GOV FINAL PERFORMANCE EVALUATION USAID-APIK | 34 the curtailed activities³one including involvement and messaging from USAID³would have allowed greater acceptance of the cuts by sub-national stakeholders. As it was, APIK Regional Coordinators and Field Coordinators were asked to inform sub-national stakeholders of the cuts, and the evaluation team sensed during field visits a certain degree of consternation about how it had all unfolded. Demonstrating effectiveness and sharing lessons are critical to persuading stakeholders to adopt or replicate the program·s approach. Because communit\ projects could not be implemented to the extent planned, APIK was limited in its ability to assess the effectiveness of its approach, or to generate and share lessons to the extent that was planned. This opportunity was largely missed. There does not appear to be any residual or long-term erosion of trust in USAID as a funder of these important CCA/DRR activities. However, if a future similar situation arises, it would be more suitable to notify stakeholders at the earliest signal of even the chance that programmatic changes might disrupt planned activities. Transparency would help considerably in alleviating any confusion or frustration. 4. RECOMMENDATIONS There is urgenc\ in the task of helping Indonesia·s citi]ens and sectors adapt to the coming impacts of climate change. Embedded in that urgency is the need for actions that are more strategic, synergistic and transformative at scale for greater numbers of people. While it is important to mainstream adaptation efforts across all sectors, achieving that objective demands focused effort on developing the necessary data and information, policies and laws, public and private financing, and institutional capacity to support implementation measures at all levels of government and society. Many of the objectives of adaptation, mitigation and ecological resilience are synergistic and should be designed to be. Mitigating GHG emissions has the potential to reduce the need for, and cost of, adaptation measures in the long term. Similarly, efforts to increase ecological integrity will enable Indonesia to be more resilient to the adverse effects of climate change. Effective preparation for an uncertain future can only be achieved through the collective action of government, individuals, businesses, communities and citizens, and their collective recognition of the linkages inherent in these objectives and the multiple benefits of doing so. The evaluation team offers the following list of recommendations for future programming of development assistance toward improved CCA/DRR. Many of the ideas herein stem from challenges highlighted and lessons learned during APIK, but are not meant to be indicative of shortcomings in the design or execution of APIK. The USAID APIK program was simply not designed or resourced to incorporate all of the recommendations that follow. Intergovernmental actions, community engagement and resilience 1. Continue to strengthen vertical and horizontal integration of GoI entities and improved planning and budgeting for CCA/DRR. Build upon the valuable coordination efforts started by APIK and help to institutionalize responsibility for cross-sectoral coordination within the appropriate GoI entity (e.g., Bappeda). This role will require the close support of technical experts, such as university scientists, and strengthened skills in strategic planning and facilitation to work across ministries. At the central level, begin with high-level dialogue between Bappenas, KLHK and BNPB on a co-designed, collaborative approach for replication and scale-up within targeted geographies (see Box 5). Continue as well with efforts to incorporate CCA/DRR within local action plans and budgets at the community level, but in a more focused and outcome- 35 | FINAL PERFORMANCE EVALUATION USAID-APIK USAID.GOV oriented manner (see Recommendation 3). USAID can catalyze more optimal use of GoI funding for CCA/DRR, including Village Funds, and should consider ways to do so while exploring ways to take advantage of the BPDLH (Pooling Fund for Environment) established by KLHK and the Ministry of Finance. Horizontal integration should include the private sector as appropriate (see Recommendations 5 and 6). 2. Help build GoI capacity and responsibility for M&E reporting specific to resilience. Assist Bappenas with the co-design of a country-scale Adaptation Theory of Change, Results Framework, and an M&E system with identified roles and responsibilities of GoI actors. Help to develop specific indicators and targets, baselines, data collection procedures and reporting requirements. Connect this M&E system Zith responsibilities for ¶adaptation communication· under the countr\·s Enhanced Nationall\ Determined Contributions (NDCs). The objectiYe is to increase transparency, accountability and overall public/private investment in climate resilience. Help KLHK and BNPB to collaborate on developing a new definition for the term ¶resilience· that effectively integrates CCA and DRR. Recall that the current definition in use is one that grew out of the DRR community. A new definition for resilience is needed that addresses climate change, and socio-ecological systems that must be prepared to respond to a changing climate. A new definition should also be the focus for M&E efforts that track both CCA and DRR objectives. An example follows of some of the key questions to be answered in monitoring for improved resilience. Box 4: Thinking about resilience Source: Grafton, Q.R., 2019. 3. Focus on one or two key sectors and specific outcomes attainable in a five-year program. Future programs on climate adaptation should continue to involve multiple tiers of stakeholders, but be more narrowly focused on prescribed key sectors or systems based on priorities identified in the RAN-API or the new RPJMN. Consider the following focused objectives that can have beneficial impact for more people: agro-forestry for smallholder farmers; USAID.GOV FINAL PERFORMANCE EVALUATION USAID-APIK | 36 conservation and management of water resources; co-management of artisanal fisheries; flood mitigation through basin-scale application of gray/green infrastructure; and mangrove conservation and management. The goal is to accelerate adaptation in key sectors or systems, and where wide￾scale adoption of improved practices or behavioral change can have transformative impact. Expand the CVA methodology to include a focus on specific commercial risks, and use the information to engage industries and companies to design collaborative solutions. A more focused set of CCA/DRR objectives will require involvement of the appropriate line ministries (e.g., agriculture, fisheries, public works, spatial planning, etc.), with co-design and contributing investments of their own. USAID can provide valuable assistance with technical design, if not the actual collaborative funding and construction, of gray/green infrastructure. Nature-based solutions can offer the biggest impact per dollar spent for improved climate resilience. Given that flooding is central to so many of the challenges facing Indonesia, consider a program having multiple benefits that is built around nature-based flood management.21 4. Use every opportunity to design adaptation measures that also contribute to mitigation. The global community is calling for greater integration of adaptation and mitigation efforts. There is plenty of scope for contributing to both objectives in almost any effort involving landscape restoration or nature-based solutions. That is the case as well for climate smart agriculture at the landscape scale. Integrating adaptation and mitigation in the agricultural landscape can achieve multiple benefits, including food security, poverty alleviation and biodiversity conservation. A key objective is to incorporate crop residues, composting and low-tillage to increase soil carbon, improve soil moisture and minimize sheet erosion. Agro-forestry and other agro-ecological practices to build and conserve soil structure are win-win solutions for both adaptation and mitigation. It is necessary to bear in mind, however, that achieving and sustaining these objectives will require accompanying policy, legal, and institutional changes. Private sector mobilization 5. Support communities to develop and fund local resilience activities, including helping communities to engage more actively with the private sector. These activities have tangible benefits and help to build the goodwill and knowledge required to initiate broader support and action for disaster preparedness and climate resilience at a local level. CSR funds are available in many locations and provide a valuable source of support for small-scale resilience and disaster preparedness activities. Likewise, providing communities (and district and provincial government representatives) with tools to encourage increased interaction with the private sector can help to create and sustain the types of activities that have been successfully developed by APIK, for example by proactively involving the private sector in Climate Field School activities. As possible, devise approaches to localize and scale such support efforts, for example by running regional-level workshops for Village Fund managers to bolster skills in project design and fundraising and using early adoption villages to disseminate best practice examples. 21 See for example: Natural and Nature-based Flood Management: A Green Guide. www.worldwildlife.org/publications/natural-and-nature-based-flood-management-a-green-guide 37 | FINAL PERFORMANCE EVALUATION USAID-APIK USAID.GOV 6. Focus on stimulating public and private investment. There are a growing number of climate financing opportunities worldwide and USAID can strategically leverage its own resources through support for a facility-type platform that can help link investors (e.g., Green Climate Fund, Adaptation Fund, etc.) with bankable projects that utilize GoI funding at all levels.22 Implementing partners can provide technical assistance and help develop the science needed to underpin rationale and commitment. More attention is being given to blended finance (combining public and private funding) in support of climate actions globally, and USAID is well-positioned to assist this model for Indonesia, especially given its experience with, and access to, the Development Credit Authority. Consider the Convergence model that uses blended finance to support landscape improvements related to renewable energy and smallholder livelihoods.23 7. Increase the attention given to market-based solutions and Corporate Shared Value. Having the tools and capacity to cope with an uncertain future requires the coordinated input of everyone, not the least of which are the businesses and investors that are the key engines of economic growth. In most developing countries, the private sector accounts for 60 percent or more of GDP, 80 percent of capital flows and 90 percent of jobs. Future programming needs to recognize this reality and to invest substantially in effective PSE while insisting that specific objectives, eligibility requirements and outcome-oriented results drive the entire PSE agenda. Integrate the principles and practices of market-based solutions and Corporate Shared Value into operations. Pay attention to the enabling conditions for private sector engagement and self￾replicating investment, including information, financing, institutional structures and capacity building. Recognize that specific private sector actors have specific climate data and information needs, and that many of them will need help in articulating the business-case and incorporating a business approach for adaptation into operations. Speak the language of profit and loss, revenues, and market share. Help to build a shared narrative between the public and private sectors based on mutual interests and shared responsibility. Finally, consider the use of a dedicated business support unit to lead the development of private sector partnerships, either internal to an adaptation program or as an available external resource. Ensure that PSE efforts receive sufficient funding to support effective MERL (monitoring, evaluation, research, and learning). Data and information needs 8. Help Indonesia re-think its overall data and information needs related to climate and disaster resilience. Think of this as a two-level process. First, build upon APIK·s efforts to improve the quality and accessibility of climate and weather information by taking the next step of assessing the data/information needs of end-users in targeted sectors. AVViVW BMKG·V ClimaWe Applications Center to prepare a report on data/information requirements for specific end-users, and the dissemination vehicles appropriate to each sector. Social media can be used for informal communication, but should not substitute for the timely, reliable and accurate forecasts, advisories/alerts and other information provided through the official channels of BMKG. Second, assist the National Institute of Aeronautics and Space (LAPAN) to augment its acquisition, analysis and application of remotely sensed data. Satellite data of different 22 See for example: Deploying Adaptation Finance for Maximum Impact.WRI, 2018. https://wriorg.s3.amazonaws.com/s3fs-public/wri-commentary-adaptation-finance.pdf 23 Convergence web site: https://www.convergence.finance/about USAID.GOV FINAL PERFORMANCE EVALUATION USAID-APIK | 38 types is quickly becoming available in Analysis Ready Data (ARD) form, which have many applications to assist CCA/DRR. Help LAPAN and the relevant line ministries to develop technical products and tools to assist with a range of needs, including, for example, regional land cover monitoring, regional drought and crop yield monitoring, and water resources management applications, including satellite-radar derived rainfall and river flow gauging for flood forecasting. Satellite data can play an important role in disaster forecasting, preparedness and response. BNPB is beginning a five-year project to incorporate artificial intelligence (AI) processing of satellite data to monitor and understand trends in land use/land cover in five river basins for improved flood forecasting. USAID is well positioned to help LAPAN partner with SERVIR-Mekong (located at the Asian Disaster Preparedness Center in Bangkok) to take advantage of the wealth of decision support tools and services already available for the Mekong Basin. These and other digital technologies offer strong potential for addressing CCA/DRR issues at temporal and spatial scales that are transformative. Landscape scale solutions 9. Design future programs to be outcome-oriented and transformative at scale. Considering the urgency of climate and disaster resilience challenges, focus on the human drivers of risk and vulnerability and work to reverse those drivers. The restoration of key landscapes and their hydrological functions can pay huge dividends for improved resilience. For example, mangrove conservation and management to protect remaining mangroves is in most places more important and transformative than a few small-scale mangrove restoration efforts that, even if successful, are still losing ground to mangrove loss and degradation. Similarly, basin-scale or watershed-scale restoration through an appropriate combination of gray and green infrastructure, together with reforestation and land management, should be given elevated priority. USAID can play an important role in helping to clarify key principles, guidelines, outcome measures and the cost-benefit rationale for applying the landscape approach related to CCA/DRR. The GoI has the resources and technical capacity to design and implement mixed (gray and green) infrastructure and nature-based solutions. Strengthen and involve the academic institutions of the country to increase the value of their technical capacity and their role in identifying innovative solutions to a range of sector-specific adaptation and disaster risk reduction needs. Box 5: Design principles for scaling up Application of scaling-up principles can aid the next phase of climate change adaptation work in Indonesia. APIK has achieved impressive success on a range of small-scale pilot activities; however, the number of beneficiaries reached has been modest. To increase the impact of future efforts, CCA programming should focus on methodically applying scaling-up principles from the outset. Among the factors that have been shown to increase the prospects for successfully scaled programs: 1) Pilots should include an emphasis on clearly defining and documenting approaches; 2) Attention should be given to rigorous measurement to Yalidate an approach·s efficac\; 3) Budgets and unit costs should be considered to ensure the effort can be adopted by counterpart institutions; 4) Complexity should be reduced to the degree possible, as simple and easily understood interventions have a higher prospect for scaling success; and 5) Support partners need to be identified early on and advocacy efforts undertaken to secure their support. 39 | FINAL PERFORMANCE EVALUATION USAID-APIK USAID.GOV 5. REFERENCES APIK, 2016. Final Site Selection Memorandum. BNPB, 2014. Rencana Nasional Penanggulangan Bencana (2015-2019). https://bnpb.go.id/documents/buku-renas-pb.pdf Global Commission on Adaptation, 2019. Adapt Now: A Global Call for Leadership on Climate Resilience. https://gca.org/global-commission-on-adaptation/adapt-our-world Grafton, Q.R., 2019. Realizing resilience for decision-making. In Nature Sustainability, published October 9, 2019. https://www.nature.com/articles/s41893-019-0376-1 ICCSR, 2009. Indonesia Climate Change Sectoral Roadmap. Synthesis Report. https://www.adaptation￾undp.org/sites/default/files/downloads/indonesia_climate_change_sectoral_roadmap_iccsr.pdf NASA, 2020. Greenland, Antarctica Melting Six Times Faster than in the 1990s. https://www.nasa.gov/feature/jpl/greenland-antarctica-melting-six-times-faster-than-in-the-1990s Singapore National Climate Change Secretariat, 2020. Charting Singapore·s LoZ-Carbon and Climate Resilient Future. https://unfccc.int/sites/default/files/resource/SingaporeLongtermlowemissionsdevelopmentstrategy.pdf USAID/APIK, 2019. Quarterly Report No. 1, Project Year 5. USAID ATLAS, 2016. Indonesia: Costs of Climate Change 2050. Policy Brief. https://www.climatelinks.org/sites/default/files/asset/document/Indonesia%20Costs%20of%20CC%202 050%20Policy%20Brief.pdf U.S. Global Change Research Program. 2016. The Impacts of Climate change on Human Health in the United States: A Scientific Assessment. https://health2016.globalchange.gov/ UN SDG Group, 2020. Shared Responsibility, Global Solidarity: Responding to the socio-economic impacts of COVID-19. https://www.un.org/sites/un2.un.org/files/sg_report_socio economic_impact_of_covid19.pdf World Bank, 2020. Indonesia Overview. https://www.worldbank.org/en/country/indonesia/overview World Resources Institute, 2018. Deploying Adaptation Financing for Maximum Impact. https://wriorg.s3.amazonaws.com/s3fs-public/wri-commentary-adaptation-finance.pdf USAID.GOV FINAL PERFORMANCE EVALUATION USAID-APIK | 40 ANNEX A. SCOPE OF WORK January 13, 2020 Final Performance Evaluation of the USAID Adaptasi Perubahan Iklim dan Ketangguhan (APIK) Project 1. Purpose of Evaluation With this final evaluation of APIK, the Mission intends to capture emerging results to inform current and future programming. USAID works closely with the Government of Indonesia (GOI) in the integration of climate change adaptation (CCA) and disaster risk reduction (DRR) into national and sub-national government structures. The Mission will continue efforts to enhance the capacity of public and private sectors to systematically integrate CCA and DRR into their planning and investment plans. This APIK evaluation will provide information to the Mission on the challenges faced, opportunities, and lessons-learned during implementation, which will support future design activities aimed at building the capacity of the GOI to address climate and disaster risks. With the exclusion of any procurement sensitive sections, USAID intends to disseminate the report widely with stakeholders such as government agencies and NGOs, USAID implementing partners, donors, and environment and climate change professional associations. The evaluation report will contribute to evidence, as well collaboration and learning priorities of the Mission, the GOI, and the private sector. Other users of the report include USAID/Washington, implementing partners, and other donors. 2. Summary Information Project Information Project Name Adaptasi Perubahan Iklim dan Ketangguhan (APIK) Award Number Contract No. AID-497-C-16-00003 Project Date Start Date November 20, 2015 Funding $15,402,305 Implementing Partner DAI Global, LLC Special Study Most Significant Change (Survey) COR Yusak Oppusunggu 3. Background A. Description of the Problem and Context Indonesia has long been vulnerable to both geographical and hydro-meteorological natural disasters, and climate change is further escalating the countr\·s risks to natural disasters. Indonesia is among the Zorld·s top fiYe countries Yulnerable to global coastal flood exposure, with more than 17 million 41 | FINAL PERFORMANCE EVALUATION USAID-APIK USAID.GOV exposed to the expected sea level rise this century24. In terms of human health and water security: drought, flooding, erosion, and reduced groundwater recharge will jeopardize surface and ground water quantity, and quality likely leading to shortages for irrigation, drinking water, and sanitation25. Fires are likely to become more intense as a result of rising temperatures and drought26. The poorest people are often the hardest hit by climate change, either because they already live on marginal lands, such as riverbanks, deforested, areas, or flood plains, or because they have the lowest adaptive capacity due to lack of education, financial resource, and/or government support. The policy environment in Indonesia for both climate change adaptation and disaster risk reduction is in its early stage, particularly at the sub-national levels. The Disaster Management Law No. 24/2007 represented a significant milestone in shifting from response-only approach to a wider risk management perspective involving other stakeholders such as the private sector, communities. The National Action Plan on Climate Change Adaptation (RAN-API) identifies some of the countr\·s main vulnerabilities to climate change and charts out short, medium and long-term actions to strengthen resilience. It represents the GOI·s first concrete effort to incorporate climate change adaptation into the national development process. Decentralization in Indonesia has given significant authority to local governments for many of the key functions and services that influence climate and disaster resilience. However, in many cases the capacity and resources of local government are insufficient to fully execute those duties. There is no clear guidance from the central government on which unit between the provincial and local government should take the lead in coordinating adaptation planning, analysis, and budgeting. The awareness and implementation of the RAN-API is limited. Communities and local level officials are the first, and often the only risk managers and responders to climate change impacts and natural disasters and yet their capacity is limited. Equally important is the role that can be played by local universities, research institutions, NGOs and the private sector in building resilience. B. Description of the USAID Activity Theory of Change: USAID·s theor\ of change for APIK is: By strengthening GOI capacity at the national, provincial, and local levels to integrate CCA and DRR into decision-making processes, by working directly with communities and the private sector to manage risk proactively, and by improving the capacity of Indonesian stakeholders to understand and communicate climate information, USAID expects to improve the ability of Indonesians to manage climate and disaster risk. Key Results: The overall objective of APIK is to improve the ability of Indonesians to manage climate change and disaster risk. The objective hinges on the following intermediate results being achieved: 24 Jongman et. al, 2012 Global exposure to river and coastal flooding: Long-term trends and changes 25 Ratag 2007 Climate Variability and Climate Change Scenario in Indonesia. BAPPENAS 2010b. Indonesia Climate Change Sectoral Roadmap (ICCSR), Synthesis Report 26 BNBP, 2014 Rencana Nasional Penanggulangan Bencana (renas PB 2015-2019). BAPPENAS, 2010b USAID.GOV FINAL PERFORMANCE EVALUATION USAID-APIK | 42 1. Integration of climate change adaptation and disaster risk reduction into national and sub-national governance frameworks improved Currentl\, the central goYernment·s state budget is the largest source of climate finance. GOI·s climate finance flows were USD $6.37 billion between 2011 and 2014. Therefore, interventions under this result focus on identifying and influencing existing planning and budgeting processes within the GOI that have the most relevance for strengthening resilience to climate change and weather-related natural disasters at local levels. APIK applies a development-first adaptation strategy to heighten national government awareness of the risks posed to critical development goals by climate change and hydro-meteorological disasters and to strengthen the goYernment·s response by mainstreaming climate change adaptation and disaster risk reduction (CCA-DRR) tools and approaches. APIK works with and through the Secretariat of Climate Change Adaptation National Action Plan (RAN API) and the Ministry of National Development Planning (BAPPENAS) to elevate the visibility of the RAN API and equip key line ministries to support its implementation. APIK also works with BNPB and PLANAS PRB to accelerate understanding of climate/weather issues. Additionally, APIK works with intra-governmental working groups to strengthen horizontal coordination and share lesson learned from Task 1 activities between government ministries and departments. Given the cross-cutting nature of climate change adaptation, this requires collaborative work with a range of ministries including BAPPENAS, Finance (MOF), and the National Disaster Management Agency (BNPB), as well as technical ministries like Marine Affairs and Fisheries (KKP), Environment and Forestry (KLHK), Energy and Mineral Resources (MEMR), Public Works and Public Housing (PUPERA), Agrarian and Spatial Planning, Agriculture (Kementan), and others. FolloZing Indonesia·s decentrali]ation, USAID also focuses on support to sub-national government entities that are central in exercising authority that impacts climate and resilience. Under Task 2, APIK focuses on local government and community capacity building, using entry￾points such as local government working groups and community-based vulnerability mapping to convene stakeholders and facilitate planning and actions that strengthen landscape, socioeconomic, and institutional resilience. APIK works with sub-national entities in selected districts on planning, budgetary and operational processes (regional adaptation strategies, local development plans (RPJMDs), annual local government plans (RKPDs), Musrenbangda, local disaster management plans, spatial management plans (RTRWP/K, RTDR), and village budget allocations under the Village Law (No.6/2014). The place-based interventions under Task 2 are linked to and enabled by a strengthened national enabling environment (Task 1), relevant climate and disaster information at the sub-national and community level (Task 3), and the engagement of the private sector around shared risks (Task 4). 2. Capacity of local communities and the private sector to address climate change and weather-related natural hazards strengthened Hydro meteorological disasters can significantly impact the private sector by damaging individual businesses, the impacts of which are often then amplified through supply chains to affect entire sectors. Often, the benefits of integrating CCA/DRR considerations are not readily understood b\ the priYate sector. APIK·s Task 4 strateg\ emphasi]es the engagement and mobili]ation of business associations and business networks around shared risks in key economic sectors, such as the fishing industry in the Maluku Islands or cocoa and rice production in Southeast Sulawesi. Cutting across the national and sub-national levels, APIK aims to improve awareness of business 43 | FINAL PERFORMANCE EVALUATION USAID-APIK USAID.GOV about climate and weather-related natural disaster risks while strengthening the capacity of private sector partners to mitigate risks in their business models and carry out sector-specific risk reducing activities. APIK works with the private sector to integrate CCA/DDR into their risk analyses and to identify resilience building measures. APIK maximizes opportunities for synergy between this intervention and other interventions under the activity. APIK engages and enhances cooperation with relevant business associations, such as the Indonesian Chamber of Commerce (KADIN), or American Chamber of Commerce (AmCham), to build awareness among Indonesian business networks on climate and disaster risk management. 3. Utilization of information for climate and disaster risk management among key stakeholders improved. In Indonesia, awareness of climate change impacts and their effect on the variability and intensity of natural disasters remains low, particularly at the local level. Often data is lacking, and the translation, relevance, and communication are weak. Task 3 focuses on the collection, packaging, and dissemination of climate and weather information (CWI) services. Better climate and weather information systems are fundamental to fostering place-based resilience across the archipelago, saving lives in the near term (i.e. disaster risk reduction) while supporting better planning and public investment in the medium to long term (i.e. climate change adaptation). Critically, improved information is not an end in and of itself but must be linked to institutional capacity building (Task 1 and Task 2), pro-poor programming for sustainable livelihoods (Task 2 and Task 4), and trans-boundary ecosystem management (Task 2). The Resilience Fund: The Resilience Fund is an important component of APIK. This fund is used to support projects that build climate or disaster resilience in partnership with communities, government and the private sector. The fund encourages participation and collaboration of local non-governmental and private sector entities that will spur innovations in new products and services. Through provision of grants or subcontracts APIK supports activities by NGOs, CSO, research institutions/ universities and co-invest with the private sector in climate or disaster risk￾reduction efforts. Geographic Coverage x East Java: APIK works in the Brantas Watershed as its priority landscape (Sidoarjo District, Mojokerto District, Jombang District, Blitar District, Malang District, Malang City and Batu Cit\). The Brantas Watershed is one of the GOI·s priorit\ Zatersheds for conserYation and rehabilitation. x Southeast Sulawesi: Southeast Sulawesi has a long coastline as well as dense but rapidly degrading tropical forests in the center of the province. The degradation is caused by one of the main industries in the province, nickel mining. Hydro-meteorological disasters are common in the region. x Maluku: Climate change is aggravating livelihood security and producing shifts in labor patterns ² from fishing to farming and back, as well from rural work to urban employment. The remoteness of Maluku is a challenge to natural disaster preparedness and response. USAID.GOV FINAL PERFORMANCE EVALUATION USAID-APIK | 44 Other Complementary Activities: APIK was designed to complement and builds upon other ongoing and completed USAID and other donors· projects. These include: the Indonesia Urban Water, Sanitation and Hygiene, Penyehatan Lingkungan Untuk Semua (IUWASH PLUS); Sustainable Ecosystems Advanced (SEA); ADAPT ASIA-Pacific; Institutionalizing Disaster Preparedness and Management Capacity of BPBDs in Indonesia through Technical Assistance and Training Teams (TATTs); the Global Resilience Partnerships (GRP); Indonesia Marine and Climate Support (IMACs); the Indonesia Climate Adaptation and Disaster Resilience (CADRE), and the Stakeholder Coordination, Advocacy, Linkages and Engagement for Resilience (SCALE-R). Projects by other donors include: the Policy Advice for Environment Change (PAKLIM) by GIZ; Project of Capacity Development for Climate Change Strategies in Indonesia by JICA; several climate change and adaptation projects b\ the Asian DeYelopment Bank; the Rockefeller·s Asian Cities Climate Change Resilience Network (ACCCRN); and work by the World Bank in supporting mainstreaming of DRR and CCA in the national development plan. APIK M&E Plan: The APIK M&E Plan lays out a range of performance indicators that track the achievement of the key results. 4. Evaluation Questions The evaluation should address the following questions: 1. How has APIK built the resilience and adaptation of different levels of government and the communities that it Zorks in and hoZ has APIK·s national Zork influenced or been influenced by the district and community levels? DINI 2. HoZ Zas APIK·s Zork Zith the priYate sector able to increase awareness about climate change adaptation and disaster reduction? INDI 3. To what extent are targeted beneficiaries better prepared and ready to adapt as a result of APIK·s climate Zeather information serYices? SAUT 4. What are the key lessons learned from application of the landscape approach to the three areas of: watershed; coastal; and small islands? RICHARD 5. What was the impact of the funding challenges and what opportunities were missed as a result? 5. Evaluation Methodology It is recommended that the evaluation team consider a mixed-method evaluation approach to the extent possible. The methodology should combine a review of quantitative data and application of qualitative evaluation techniques to obtain information, opinions, and data from counterparts, contractors, partners, GOI entities, beneficiaries, and other donors as appropriate. The approach should be participatory and should involve the use of appropriate data collection tools. By using a mixed approach, the evaluation team will gain insight on the impact of APIK interventions (mostly from quantitative data collected by the activity and others) and the processes (mostly qualitative information provided by the project staff and key informants) that lead to those impacts. Sequential and iterative approaches should be used to integrate the mixture of methods at various stages of the evaluation. 45 | FINAL PERFORMANCE EVALUATION USAID-APIK USAID.GOV The Evaluation Team will draft an evaluation methodology/design for USAID approval. The detailed methodology of this evaluation will be described by the evaluation team in the Work Plan; this will include presentation of an evaluation matrix that will explicitly link evaluation questions to particular data collection approaches and data sources. The team should start deciding on specific methodologies for the Yarious actiYities during the EYaluation Team·s initial planning meetings. The following essential elements should be included in the methodology as well as the additional methods proposed by the team: Document Review: USAID/Indonesia will provide the Team Leader with a core list and/or copies of the Agreement, performance reports, relevant assessment reports, and other key Mission, GOI and implementing partner documentation before the evaluation begins. The Team Leader will be responsible for expanding this background documentation as appropriate, review, prioritize, and distribute it to other team members for their review. All team members will review relevant documentation before their initial team meetings. Key informant interviews: The team will conduct interviews to obtain feedback from a representative number of stakeholders including (list is not exhaustive): o Various USAID offices and other USG offices in Indonesia; o APIK Implementing partner at both HQ and field level; o Key GOI representatives across national, and sub-national levels including field level staff engaged in CCA/DDR; o Private Sector: representatives of business associations and business networks; individual businesses; o Other stakeholders, e.g. major donors, involved in CCA/DDR. o Staff from other relevant USAID implementing organizations Focus Group Discussions: The team will conduct group discussions with a range of representatives across the APIK beneficiaries and stakeholders to obtain feedback on the impact of the activity. Discussion groups will include balanced numbers of men and women; in addition, as appropriate to local circumstances sex- or age-segregated discussion groups will be used to promote free discussion by women, men, and youth. Site visits: In addition to offering to interview stakeholders to solicit views on the overarching evaluation question, the Evaluation Team will conduct direct observations on selected activity locations in the three areas of East Java, Southeast Sulawesi, and Maluku, to collect and/or validate data and inform findings related to the evaluation questions. Staff from Monitoring and Evaluation Support Project (MESP) and APIK with support from USAID/Indonesia will assist in organizing logistics for all site visits for the Evaluation Team. Data Analysis: Team members will analyze APIK documents and information gained from key informant interviews and site visits in order to inform their findings and recommendations. The team will keep a record of meetings that take place and record the summaries of each meeting. Some quantitatiYe anal\ses ma\ be featured, for e[ample, in the reYieZ of APIK·s performance monitoring data or in the anal\sis of the actiYit\·s efficienc\. The EYaluation Team Zill anal\]e the information collected to establish credible answers to the questions and provide major trends and issues. USAID requires that evaluations explore issues of gender; thus, the evaluation should examine gender issues within the context of the evaluation of APIK activities. USAID.GOV FINAL PERFORMANCE EVALUATION USAID-APIK | 46 Limitation: Since key informant interview will be a major source for validation of information available from the activity, chances of bias are likely. The evaluation should carefully decide on the methodology and select interviewees in a way that the possibility of bias is avoided or reduced to a minimum. The Evaluation Team should also take systematic actions to counter any biases in interpretations of collected data by the team. The methodology narrative of the evaluation report should discuss the merits and limitations of the final evaluation methodology. The evaluation team will design appropriate tools for collecting data from various units of analysis. The tools will be shared with USAID during the evaluation and as part of the evaluation report. 6. Existing Sources of Information USAID/Indonesia will provide documents for the desk review. The Evaluation Team will be responsible for identifying and reviewing additional materials relevant to the evaluation. USAID/Indonesia will also help the Evaluation Team with contact information for relevant interviewees. 7. Deliverables and Reporting Requirements 1. Evaluation Work Plan, to include evaluation methodology and instruments that will be applied to answer evaluation questions; evaluation schedule, list of people/groups to be interviewed, timeframe, and draft schedule of field activities. 2. In-briefing, to present and discuss draft work plan. 3. Mid-term Briefing, update on progress, opportunities, challenges, and any requested revisions to schedule. 4. Debriefing, PowerPoint presentation of findings with bulleted response to evaluation questions, and discussion on development of recommendations 5. Draft Evaluation Report, clearly describing findings, conclusions, and recommendations. USAID will provide comment on the Draft Evaluation Report within ten working days of submission. 6. Final Evaluation Report following USAID standard evaluation report format and branding guidelines, within 10 working days of receiving Mission comments on the draft report. The format of the final report is provided below. The report will be submitted in English, electronically. 47 | FINAL PERFORMANCE EVALUATION USAID-APIK USAID.GOV 8. Evaluation Schedule Performance Evaluation Schedule Scheduling and Logistics Funding and Logistical Support The proposed evaluation will be funded through MESP. MESP will be responsible for all off-shore and in-country administrative and logistical support, including identification and fielding appropriate consultants. They will take care of arranging and scheduling meetings, international and local travel, hotel bookings, working/office spaces, computers, printing, and photocopying. The evaluation team should be able to make all logistic arrangements, including the vehicle arrangements, for travel within and outside Jakarta and should not expect any logistic support from the Mission. The team should also make their own arrangements on space for team meetings, and equipment support for producing the report. Date or Duration Activities Considerations Jan 6 ² Jan 24 Budget approved; recruitment finalized; employment agreements finalized January 20 is a U.S. holiday Jan 24 ² Feb 15 Preparation of the work plan and evaluation design Feb 17 ² Feb 21 Team begins work in Jakarta. Briefings are held with USAID, IP and GoI; final workplan is presented to USAID for approval. Availability in the Mission Feb 24 ² Mar 14 Data collection/interviews in Jakarta and at project sites Local holidays (none), season/weather, transport availability Mar 16 ² Mar 20 Data analysis and summary findings presentation development Mar 23 Debriefing ² findings/conclusions/lessons learned and collaborative discussion of recommendations with USAID Mar 23/24 Team Leader departs Apr 16 Submission of draft report to USAID Length of time for all relevant stakeholders to read and provide feedback. TBD Three weeks to incorporate of USAID comments, including MESP internal review of final report + editing. Final report submitted to USAID. Length of time to reconcile feedback from varying stakeholders and comply with formatting requirements. May 2020 USAID Review and Approval of Final Report Evaluation Team fully addresses USAID comments USAID.GOV FINAL PERFORMANCE EVALUATION USAID-APIK | 48 9. Final Report Format The total pages, excluding the executive summary, references and annexes, should not exceed 35-40 pages, with general guidance on the number of pages per section included below. The following content (and suggested length) should be included in the report: Table of Contents 1. Abstract - The evaluation abstract of no more than 250 words should describe what was evaluated, evaluation questions, methods, and key findings or conclusions. 2. Executive Summary - concisely state the project purpose and background, key evaluation questions, methods, most salient findings and conclusions (plus recommendations and lessons learned as appropriate) (2-3 pp.); 3. Evaluation Purpose - purpose, audience, and synopsis of task (1 pp.); 4. The Development Problem - highlight the development problem and the context in which intervention took place, including a summary of any relevant history, demography, socio-economic status etc. (1 pp.); 5. USAID·V ReVponVe - brief overview of APIK project, USAID program strategy and activities implemented in response to the problem, (2-3 pp.) 6. Evaluation Questions ² state the evaluation questions as identified in the SOW (0.5 pp) 7. Methodology - describe evaluation methods in detail, including limitations (strengths, constraints and gaps related to issues such as data availability; any potential bias; such as measurement, interviewer, response etc., must be disclosed and implications on conclusions drawn (2-3 pp.). 8. Findings and Conclusions - describe and analyze findings for each objective area using graphs and tables, as applicable, and also include data quality and reporting system that should present verification of spot checks, issues, and outcome (20 pp.) 9. Recommendations - prioritized for each key area or evaluation focus; should be separate from conclusions and be supported by clearly defined set of findings and conclusions (3-4pp.) 10. Lessons Learned- provide a brief of key technical and/or administrative lessons that could be used for future project or relevant program designs (2-3pp.); 11. References (including bibliographical documentation, meetings, interviews and focus group discussions); 12. Annexes ²to include statement of work, documents reviewed, evaluation methods, data generated from the evaluation, tools used, interview lists and tables, conflict of interest forms for each evaluation team member, and summary information about evaluation team members including qualifications, experience and role on the team. Annexes should be succinct, pertinent and readable. The final report should have the following criteria to ensure the quality of the report: The evaluation report should represent a thoughtful, well-researched and well-organized effort to objectively evaluate what worked in the project, what did not and why. Evaluation reports shall address all evaluation questions included in the scope of work. The evaluation report should include the scope of work as an annex. All modifications to the scope of work, whether in technical requirements, evaluation questions, evaluation team composition, methodology or timeline need to be agreed upon in writing by the COR. Evaluation methodology shall be explained in detail and all tools used in conducting the evaluation such as questionnaires, checklists and discussion guides will be included in an Annex in the final report. 49 | FINAL PERFORMANCE EVALUATION USAID-APIK USAID.GOV Limitations to the evaluation shall be disclosed in the report, with particular attention to the limitations associated with the evaluation methodology (selection bias, recall bias, etc.). Evaluation findings should be presented as analysed facts, evidence and data and not based on anecdotes, hearsa\ or the compilation of people·s opinions. Findings should be specific, concise and supported by strong quantitative or qualitative evidence. Sources of information need to be properly identified and listed in an annex. Recommendations need to be supported by a specific set of findings. Recommendations should be action-oriented, practical and specific, with defined responsibility for the action. The Final Evaluation Report should be submitted in Word and PDF formats and 12-point type font should be used throughout the body of the report, with page margins one-inch top/bottom and left/right. The Mission should receive 5 hard copies of the final version of the report and an electronic copy of the final report. MESP will submit the Final Evaluation Report to the Development Experience Clearinghouse. 10. Evaluation Team Composition The Evaluation Team shall consist of four (4) individuals with substantial experience in CCA and DRR in mid or low-income countries with USAID and/or other donors. At a minimum, the evaluation team will consist of a team leader and three evaluation/technical experts. 1. Team Leader (International Staff) - The Team Leader will be responsible for direct liaison with USAID, managing evaluation team personnel and assignment responsibilities, and producing high￾quality evaluation deliverables, including the evaluation methodology and work plan, evaluation presentations and the draft and final reports. The Team Leader will work under the direct superYision of Indonesia MESP·s Chief of Part\ and Zill coordinate Zith, and be supported b\, MESP evaluation specialists and research assistants. Qualifications: - Advanced degree in a relevant discipline, such as natural resource management, public administration, public policy, rural economic development or a similar field. - Minimum of ten years of experience in development, with a significant experience in one or more of the following areas: climate change management, rural development, natural disaster risk reduction, institutional capacity building. - Significant performance evaluation experience, including prior experience designing data collection instruments. - Experience analyzing and synthesizing large amounts of data and information into meaningful conclusions and recommendations for strategic input and implementation. - Have a proven track record in terms of leadership, coordination of evaluation of development projects. - Have excellent writing, oral presentation, and organizational skills. - Prior work experience in Southeast Asia is required, and prior experience in Indonesia is preferred. - Familiarity with USAID climate change adaptation programming approaches is an advantage. 2. Evaluation/Climate Change Specialists (Indonesian Staff - three positions). These positions will work under the supervision of the Evaluation Team Leader and will be responsible for USAID.GOV FINAL PERFORMANCE EVALUATION USAID-APIK | 50 contributing to all aspects of the evaluation, including supporting the development of the eYaluation·s methodolog\, data collection and management, data anal\sis and Zriting the draft and final reports. Qualifications: Evaluation team members will be selected based on a mix of the relevant skills that will be required for the assignment: - Advanced degree in a relevant discipline, such as: public administration/policy; research skills; international development; natural resource management; agriculture or a similar field. - Minimum of ten years of experience in development, with a significant experience in one or more of the following areas: community disaster management; climate change management; rural development; natural disaster risk reduction; institutional capacity building. - Minimum of five years of performance evaluation experience, including designing data collection instruments, conducting structured interviews and managing qualitative data. - Knowledge of climate change and disaster risk reduction issues and programming approaches. - Familiarity with approaches to engage the private sector to achieve development objectives. - Public sector management experience, including in development planning and budgeting processes at the sub-national level in Indonesia. 11. Budget and Level of Effort for the Evaluation Team MESP will submit a budget and personnel level of effort chart to USAID/Indonesia PRO once the SOW has been approved. 51 | FINAL PERFORMANCE EVALUATION USAID-APIK USAID.GOV ANNEX B. EVALUATION QUESTIONS AND SUB-QUESTIONS EQ1:How has APIK built the resilience and adaptation of different levels of government and the communities that it works in, and how haV APIK·V naWional ZoUk inflXenced oU been inflXenced b\ Whe diVWUicW and commXniW\ leYelV? To understand community perception 1.1 What do community members say are the most important climate-related impacts that they must face? 1.2 How have communities come to understand and prioritize those impacts? Are the impacts understood as short-term or long-term? 1.3 How has GOI (at all levels) helped communities to take action to prepare for those impacts? 1.4 How has APIK helped communities to take action to prepare for those impacts? 1.5 What activities do communities believe will be the most effective at reducing risk? In the short-term? In the long-term? 1.6 What additional activities are needed to reduce risk of climate-related impacts? 1.7 What are the challenges faced by the communities to carry out measures for improved climate resilience? 1.8 What are the reasons/factors affecting community participation in climate change adaptation? 1.9 Does the community utilize the village fund mechanism to improve CCA/DRR? If yes, how has APIK helped this process? 1.10 What challenges does APIK face in helping communities to improve climate resilience? To understand GOI perception 1.11 What do GOI officials say have been the most important contributions of APIK to improve climate and disaster resilience in Indonesia? 1.12 What methods did APIK use in helping GOI entities to fulfill their responsibilities? Which were most effective? Least effective? Why? 1.13 HaYe APIK·s communit\ and district-level efforts influenced how national GOI entities approach climate and disaster resilience? How? 1.14 How has APIK helped the GOI (all levels) on identifying and influencing the planning and budgeting process for improved CCA/DRR? Which activities have been most/least effective? Why? EQ2:`HoZ ZaV APIK·V ZoUk ZiWh Whe pUiYaWe VecWoU able Wo incUeaVe aZaUeneVV aboXW adapWaWion and diVaVWeU UiVk UedXcWion? 2.1 Do private sector entities have improved awareness and understanding of climate and disaster resilience as a result of APIK? In what ways has their awareness and understanding improved? Any differences with regard to Indonesian vs Multi-national companies? 2.2 What activities did APIK undertake with the private sector to increase this awareness and understanding? Which were most effective? Least effective? Why? 2.3 What climate resilience actions have the private sector undertaken as a result of their engagement with APIK? Are these actions seen (by the private sector entity) as effective and worthwhile? Why? 2.4 What reasons do private sector entities give for wanting to undertake climate resilience actions? Will they be sustained post-APIK? 2.5 Are there market-based approaches involved, and if so how do they work? 2.6 What additional activities have potential for private sector involvement to reduce risk of climate-related impacts? USAID.GOV FINAL PERFORMANCE EVALUATION USAID-APIK | 52 2.7 Are there factors constraining the private sector from engaging/investing in climate resilience actions in Indonesia? 2.8 How has APIK PSE helped to achieve/improve overall project objectives? Increased efficiency/effectiveness/outreach? EQ3:To ZhaW e[WenW aUe WaUgeWed beneficiaUieV beWWeU pUepaUed and Uead\ Wo adapW aV a UeVXlW of APIK·V climaWe and ZeaWheU information services? 3.1 Do project beneficiaries feel that APIK·s climate and Zeather information serYices haYe benefitted them? HoZ? 3.2 Have project beneficiaries changed their behavior as a result of the improved climate and weather information? How? 3.3 Do project beneficiaries feel they have the information needed to be better prepared for future climate and disaster risks? For all/specific risks? Somewhat/fully prepared? 3.4 What additional climate and weather information services do project beneficiaries wish to see in the future? 3.5 Have beneficiaries received guidance on how to access CWI services? How is this done and is it effective? 3.6 What challenges do beneficiaries face in gaining access to CWI services? What additional support is needed? EQ4:What are the key lessons learned from application of the landscape approach to the three areas of: watershed; coastal; and small islands? 4.1 What has been the APIK approach related to defining and understanding a geography within which to apply climate resilience efforts? 4.2 Has a landscape approach been useful for promoting climate resilience? In what ways? 4.3 What could be done to improve the landscape approach as implemented by APIK? 4.4 Do project beneficiaries see themselves and their assets as part of a larger landscape? How does that stimulate/alter their own climate resilience efforts? EQ5: What was the impact of the funding challenges and what opportunities were missed as a result? 5.1 How did the timing of reduced funding affect project planning and delivery? 5.2 How did the amount of reduced funding affect project planning and delivery? 5.3 Were project beneficiaries made aware of these impacts and did they have concerns? 5.4 Did the reduced funding affect APIK staffing pattern? How? 5.5 What project activities were eliminated or scaled-back as a result of the reduced funding? 53 | FINAL PERFORMANCE EVALUATION USAID-APIK USAID.GOV ANNEX C. EVALUATION MATRIX Evaluation Question Sub-question Data Collection Method Data Sources Sampling or Selection Criteria Data Analyses Methods 1. How has APIK built the resilience and adaptation of different levels of government and the communities that it works in, and hoZ has APIK·s national work influenced or been influenced by the district and community levels? 1.1 What do community members say are the most important climate change impacts that they must face? Document review, FGDs, KIIs Work Plans, Annual Reports, technical reports, interviews with project beneficiaries Exhaustive review of project documents; Purposive and quota sampling Content Analysis, Descriptive Analysis, Thematic Analysis 1.2 How have communities come to understand and prioritize those impacts? Are the impacts understood as short-term or long-term? Document review, FGDs, KIIs Work Plans, Annual Reports, technical reports, interviews with project beneficiaries Exhaustive review of project documents; Purposive and quota sampling Content Analysis, Descriptive Analysis, Thematic Analysis 1.3 How has GOI (at all levels) helped communities to take action to prepare for those impacts? Document review, FGDs, KIIs Work Plans, Annual Reports, technical reports, interviews with project beneficiaries Exhaustive review of project documents; Purposive and quota sampling Content Analysis, Descriptive Analysis, Thematic Analysis 1.4 How has APIK helped communities to take action to prepare for those impacts? Document review, FGDs, KIIs Work Plans, Annual Reports, technical reports, interviews with project beneficiaries Exhaustive review of project documents; Purposive and quota sampling Content Analysis, Descriptive Analysis, Thematic Analysis 1.5 What activities do communities believe will be the most effective at reducing risk? In the short-term? In the long-term? Document review, FGDs, KIIs Work Plans, Annual Reports, technical reports, interviews with project beneficiaries Exhaustive review of project documents; Purposive and quota sampling Content Analysis, Descriptive Analysis, Thematic Analysis 1.6 What additional activities are needed to reduce risk of climate-related impacts? Document review, FGDs, KIIs Work Plans, Annual Reports, technical reports, interviews with project beneficiaries, government reports Exhaustive review of project documents; Purposive and quota sampling Content Analysis, Descriptive Analysis, Thematic Analysis 1.7 What are the challenges faced by the communities to carry out measures for improved climate resilience? Document review, FGDs, KIIs Work Plans, Annual Reports, technical reports, interviews with project beneficiaries, government reports Exhaustive review of project documents; Purposive and quota sampling Content Analysis, Descriptive Analysis, Thematic Analysis USAID.GOV FINAL PERFORMANCE EVALUATION USAID-APIK | 54 1.8 What are the reasons/factors affecting community participation in climate change adaptation? Document review, FGDs, KIIs Work Plans, Annual Reports, technical reports, interviews with project beneficiaries, government reports Exhaustive review of project documents; Purposive and quota sampling Content Analysis, Descriptive Analysis, Thematic Analysis 1.9 Does the community utilize the village fund mechanism to improve CCA/DRR? If yes, how has APIK helped this process? Document review, FGDs, KIIs Work Plans, Annual Reports, technical reports, interviews with project beneficiaries, government reports Exhaustive review of project documents; Purposive and quota sampling Content Analysis, Descriptive Analysis, Thematic Analysis 1.10 What challenges does APIK face in helping communities to improve climate resilience? Document review, FGDs, KIIs Work Plans, Annual Reports, technical reports, interviews with project beneficiaries, government reports Exhaustive review of project documents; Purposive and quota sampling Content Analysis, Descriptive Analysis, Thematic Analysis 1.11 What do GOI officials say have been the most important contributions of APIK to improve climate and disaster resilience in Indonesia? Document review, KIIs Work Plans, Annual Reports, technical reports, interviews with project beneficiaries, government reports Exhaustive review of project documents; Purposive and quota sampling Content Analysis, Descriptive Analysis, Thematic Analysis 1.12 What methods did APIK use in helping GOI entities to fulfill their responsibilities? Which were most effective? Least effective? Why? Document review, KIIs Work Plans, Annual Reports, technical reports, interviews with project beneficiaries, government reports Exhaustive review of project documents; Purposive and quota sampling Content Analysis, Descriptive Analysis, Thematic Analysis 1.13 HaYe APIK·s communit\ and district￾level efforts influenced how national GOI entities approach climate and disaster resilience? How? Document review, KIIs Work Plans, Annual Reports, technical reports, interviews with project beneficiaries, government reports Exhaustive review of project documents; Purposive and quota sampling Content Analysis, Descriptive Analysis, Thematic Analysis 1.14 How has APIK helped the GOI (all levels) on identifying and influencing the planning and budgeting process for improved CCA/DRR? Which activities have been most/least effective? Why? Document review, KIIs Work Plans, Annual Reports, technical reports, interviews with project beneficiaries, government reports Exhaustive review of project documents; Purposive and quota sampling Content Analysis, Descriptive Analysis, Thematic Analysis 55 | FINAL PERFORMANCE EVALUATION USAID-APIK USAID.GOV 2. HoZ Zas APIK·s work with the private sector able to increase awareness about adaptation and disaster risk reduction? 2.1 Do private sector entities have improved awareness and understanding of climate and disaster resilience as a result of APIK? In what ways has their awareness and understanding improved? Any differences with regard to Indonesian vs Multi-national companies? Document review, KIIs Work Plans, Annual Reports, technical reports, interviews with project beneficiaries Exhaustive review of project documents; Purposive and quota sampling Content Analysis, Descriptive Analysis, Thematic Analysis 2.2 What activities did APIK undertake with the private sector to increase this awareness and understanding? Which were most effective? Least effective? Why? Document review, KIIs Work Plans, Annual Reports, technical reports, interviews with project beneficiaries Exhaustive review of project documents; Purposive and quota sampling Content Analysis, Descriptive Analysis, Thematic Analysis 2.3 What climate resilience actions have the private sector undertaken as a result of their engagement with APIK? Are these actions seen (by the private sector entity) as effective and worthwhile? Why? Document review, KIIs Work Plans, Annual Reports, technical reports, interviews with project beneficiaries Exhaustive review of project documents; Purposive and quota sampling Content Analysis, Descriptive Analysis, Thematic Analysis 2.4 What reasons do private sector entities give for wanting to undertake climate resilience actions? Will they be sustained post-APIK? Document review, KIIs Work Plans, Annual Reports, technical reports, interviews with project beneficiaries Exhaustive review of project documents; Purposive and quota sampling Content Analysis, Descriptive Analysis, Thematic Analysis 2.5 Are there market-based approaches involved, and if so how do they work? Document review, KIIs Work Plans, Annual Reports, technical reports, interviews with project beneficiaries Exhaustive review of project documents; Purposive and quota sampling Content Analysis, Descriptive Analysis, Thematic Analysis 2.6 What additional activities have potential for private sector involvement to reduce risk of climate-related impacts? Document review, KIIs Work Plans, Annual Reports, technical reports, interviews with project beneficiaries Exhaustive review of project documents; Purposive and quota sampling Content Analysis, Descriptive Analysis, Thematic Analysis 2.7 Are there factors constraining the private sector from engaging/investing in climate resilience actions in Indonesia? Document review, KIIs Work Plans, Annual Reports, technical reports, interviews with project beneficiaries Exhaustive review of project documents; Purposive and quota sampling Content Analysis, Descriptive Analysis, Thematic Analysis USAID.GOV FINAL PERFORMANCE EVALUATION USAID-APIK | 56 2.8 How has APIK PSE helped to achieve/improve overall project objectives? Increased efficiency/effectiveness/outreach? Document review, KIIs Work Plans, Annual Reports, technical reports, interviews with project beneficiaries Exhaustive review of project documents; Purposive and quota sampling Content Analysis, Descriptive Analysis, Thematic Analysis 3. To what extent are targeted beneficiaries better prepared and ready to adapt as a result of APIK·s climate weather information services? 3.1 Do project beneficiaries feel that APIK·s climate and Zeather information services have benefitted them? How? Document review, FGDs, KIIs Work Plans, Annual Reports, technical reports, interviews with project beneficiaries Exhaustive review of project documents; Purposive and quota sampling Content Analysis, Descriptive Analysis, Thematic Analysis 3.2 Have project beneficiaries changed their behavior as a result of the improved climate and weather information? How? Document review, FGDs, KIIs Work Plans, Annual Reports, technical reports, interviews with project beneficiaries Exhaustive review of project documents; Purposive and quota sampling Content Analysis, Descriptive Analysis, Thematic Analysis 3.3 Do project beneficiaries feel they have the information needed to be better prepared for future climate and disaster risks? For all/specific risks? Somewhat/fully prepared? Document review, FGDs, KIIs Work Plans, Annual Reports, technical reports, interviews with project beneficiaries Exhaustive review of project documents; Purposive and quota sampling Content Analysis, Descriptive Analysis, Thematic Analysis 3.4 What additional climate and weather information services do project beneficiaries wish to see in the future? Document review, FGDs, KIIs Work Plans, Annual Reports, technical reports, interviews with project beneficiaries Exhaustive review of project documents; Purposive and quota sampling Content Analysis, Descriptive Analysis, Thematic Analysis 3.5 Have beneficiaries received guidance on how to access CWI services? How is this done and is it effective? Document review, FGDs, KIIs Work Plans, Annual Reports, technical reports, interviews with project beneficiaries Exhaustive review of project documents; Purposive and quota sampling Content Analysis, Descriptive Analysis, Thematic Analysis 3.6 What challenges do beneficiaries face in gaining access to CWI services? What additional support is needed? Document review, FGDs, KIIs Work Plans, Annual Reports, technical reports, interviews with project beneficiaries Exhaustive review of project documents; Purposive and quota sampling Content Analysis, Descriptive Analysis, Thematic Analysis 57 | FINAL PERFORMANCE EVALUATION USAID-APIK USAID.GOV 4. What are the key lessons learned from application of the landscape approach to the three areas of: watershed; coastal; and small islands. 4.1 What has been the APIK approach related to defining and understanding a geography within which to apply climate resilience efforts? Document review, FGDs, KIIs Work Plans, Annual Reports, technical reports, interviews with project beneficiaries Exhaustive review of project documents; Purposive and quota sampling Content Analysis, Descriptive Analysis, Thematic Analysis 4.2 Has a landscape approach been useful for promoting climate resilience? In what ways? Document review, FGDs, KIIs Work Plans, Annual Reports, technical reports, interviews with project beneficiaries Exhaustive review of project documents; Purposive and quota sampling Content Analysis, Descriptive Analysis, Thematic Analysis 4.3 What could be done to improve the landscape approach as implemented by APIK? Document review, FGDs, KIIs Work Plans, Annual Reports, technical reports, interviews with project beneficiaries Exhaustive review of project documents; Purposive and quota sampling Content Analysis, Descriptive Analysis, Thematic Analysis 4.4 Do project beneficiaries see themselves and their assets as part of a larger landscape? How does that stimulate/alter their own climate resilience efforts? Document review, FGDs, KIIs Work Plans, Annual Reports, technical reports, interviews with project beneficiaries Exhaustive review of project documents; Purposive and quota sampling Content Analysis, Descriptive Analysis, Thematic Analysis 5. What was the impact of the funding challenges and what opportunities were missed as a result? 5.1 How did the timing of reduced funding affect project planning and delivery? Document review, FGDs, KIIs Work Plans, Annual Reports, technical reports, interviews with project beneficiaries Exhaustive review of project documents; Purposive and quota sampling Content Analysis, Descriptive Analysis, Thematic Analysis 5.2 How did the amount of reduced funding affect project planning and delivery? Document review, FGDs, KIIs Work Plans, Annual Reports, technical reports, interviews with project beneficiaries Exhaustive review of project documents; Purposive and quota sampling Content Analysis, Descriptive Analysis, Thematic Analysis 5.3 Were project beneficiaries made aware of these impacts and did they have concerns? Document review, FGDs, KIIs Work Plans, Annual Reports, technical reports, interviews with project beneficiaries Exhaustive review of project documents; Purposive and quota sampling Content Analysis, Descriptive Analysis, Thematic Analysis USAID.GOV FINAL PERFORMANCE EVALUATION USAID-APIK | 58 5.4 Did the reduced funding affect APIK staffing pattern? How? Document review, FGDs, KIIs Work Plans, Annual Reports, technical reports, interviews with project beneficiaries Exhaustive review of project documents; Purposive and quota sampling Content Analysis, Descriptive Analysis, Thematic Analysis 5.5 What project activities were eliminated or scaled-back as a result of the reduced funding? Document review, FGDs, KIIs Work Plans, Annual Reports, technical reports, interviews with project beneficiaries Exhaustive review of project documents; Purposive and quota sampling Content Analysis, Descriptive Analysis, Thematic Analysis 59 | FINAL PERFORMANCE EVALUATION USAID-APIK USAID.GOV ANNEX D. DATA COLLECTION TOOLS Community/Village FGD ² Introductory Statements Upon arrival at each community/village, the evaluation team will convene APIK stakeholders (men and women) together for introductions and general messaging. The FGD Lead will acknowledge the leaders and welcome and thank everyone in attendance. Following the welcome, the FGD Lead will say: We are here to gather this communit\·s input to help assess progress and lessons learned from APIK. We are an independent team ² that is, we are not part of the APIK program or USAID ² and we have various expertise related to climate and disaster resilience. This is one of many communities we will visit, and we will also speak with government and other officials at the national, provincial, and district levels. Our main goals are to learn if APIK has met its objectives, to identify lessons learned, and to recommend ways to similar future programs that aim to assist communities facing climate adaptation and disaster risk reduction challenges. Following brief introductions by the evaluation team members, the FGD Lead will say: My colleague (standing here) and I will serve as translators for the team. They have asked that I first briefly explain the process today. Following our discussion here as a group, and answering any questions that you may have. We will then begin to ask our questions. We ask for your full and honest answers to the questions. Remember please that our purpose is to learn and be able to improve these types of projects in the future. The group discussion will last for approximately 1 hour. The FGD Lead will pause and ask if everyone understands. The evaluation team has invited each of you to participate as an individual today, and want to hear about your own involvement and experience with APIK. We want to learn what you think has worked well (and why), and what has not worked so well (and why). They want to learn if the program is meeting your expectations, and if you are seeing benefits to you and your family. It is important for everyone to be part of the discussion ² Zomen, \outh, and men. EYer\one·s opinion is important to us. The FGD Lead will pause and ask if everyone understands. The evaluation team also wants you to know that your participation is voluntary. You do not have to participate, or to answer every question. Comments received will not be attributed to any individual. Your comments will be merged with the comments received from the other communities, and that will allow the team to make some overall conclusions and recommendations related to each question. At conclusion of the group discussion you may leave, but some of you may be asked to stay and participate in a short interview to allow for some more detailed questioning by the team. Once again, this will be voluntary, and should not take more than an additional 15-20 minutes. If you agree to participate, we ask that you sign the attendance sheet. Are there any questions? USAID.GOV FINAL PERFORMANCE EVALUATION USAID-APIK | 60 FOCUS GROUP DISCUSSION GUIDE ² COMMUNITY LEVEL Community FGD Lead Date of FGD FGD Co-Lead Start Time End Time No. Adults Males Females Total No. Youth Males Females Total EQ# QUESTION 1 Is climate change affecting your community, and if so, how? 1 Which impacts (and why) are of most concern to you? 1 How have you come to better understand and prioritize the impacts? (S-T/L-T) 1 What actions are your community taking to prepare for climate and disaster impacts? 1 What kind of help are you getting from government? 1 What kind of help are you getting directly from APIK? 1 What actions do you believe will be most effective at reducing risk? (S-T/L-T) 1 What additional activities are needed to improve climate and disaster resilience? 1 What kind of training have you received during the project? 1 Was the training useful, or could it have been made more useful? 1 What challenges do you face to implement measures for improved resilience? 1 Does everyone participate to implement measures? If not, why? 1 Has your community used the Village Fund mechanism? Did APIK help you to do so? 3 Has your community benefitted from the improved CWIS as a result of APIK? If so, how? 3 Does your community have any new policies/practices in place as a result of improved CWIS? 3 Does your community have all the climate and weather information it needs to be better prepared for future climate and disaster risks? 3 Would you say that your community is not well prepared, somewhat prepared, or fully prepared to address CCA/DRR? 3 What additional CWIS would you like to have available to your community in the future? 3 Has your community received training/guidance on how to make the most use of the available CWIS? 3 What challenges does your community face in gaining access to CWIS? What additional support is needed? 4 Does your community see itself as part of a larger landscape (e.g., watershed, or coastal zone)? If so, does that affect the way your community approaches its own efforts at CCA/DRR? Are there lessons learned from APIK that are important to include in future projects? 61 | FINAL PERFORMANCE EVALUATION USAID-APIK USAID.GOV INTERVIEW DISCUSSION GUIDE ² VILLAGE LEVEL Name of interviewee Male / Female / Youth Role within APIK Interviewed by Village Translator Date EQ QUESTION Which activity (or activities) have you been most involved with? Have you seen your community come together in support of APIK, or are there some differences or concerns among members? As a result of APIK, have there been any noticeable changes in the attitudes of community members towards climate and disaster risk? If so, how? Do you believe your own attitude has changed towards these activities? If so, how? Did you participate in all of APIK activities, or are there reasons why you did not participate more? Have women been encouraged to participate in APIK activities, or are there any constraints to female participation? Have youth been encouraged to participate in APIK activities, or are there any constraints to youth participation? Does the community discuss how it will keep the activities going after APIK is finished? Do you believe that community members are willing to invest their own time and effort to sustain the activities beyond the project? Do government officials and offices have the capacity and tools needed to help sustain activities? Do you have any suggestions for how APIK could have improved the way it provides assistance to your community? Do you feel that the purpose of APIK has been well explained to the community? Did the project make efforts to understand the priorities and organization of this community before it offered assistance? What do you think are the most important lessons learned from APIK? Do you have any additional comments or recommendations you would like to make? USAID.GOV FINAL PERFORMANCE EVALUATION USAID-APIK | 62 INTERVIEW DISCUSSION GUIDE ² PRIVATE SECTOR Name of interviewee Male / Female Company Name Indonesian or Multi-national Location Interviewed by Date EQ # QUESTION 2 How is climate change affecting Indonesia and specifically your business? 2 Which impacts are of most concern and why? 2 In what way does your company have improved awareness and understanding of CCA/DRR as a result of working with APIK? 2 What activities did APIK undertake with your company to increase understanding? Were some activities more effective than others? Why? 2 What CCA/DRR measures has your company implemented since working with APIK? Does your company see these as worthwhile and effective actions? Why? 2 Will your company continue to sustain these measures after APIK project closes down? 2 Are there any market-based approaches used by your company to improve CCA/DRR? If so, please describe how they work. 2 Are there any additional activities or support you would like to see to improve your compan\·s climate and disaster resilience? 2 What factors constrain your company from engaging or investing further to improve CCA/DRR in Indonesia? 3 Has your company benefitted from the improved CWIS as a result of APIK? If so, how? 3 Does your company have any new policies/practices in place as a result of improved CWIS? 3 Does your company have all the climate and weather information it needs to be better prepared for future climate and disaster risks? 3 Do you feel that your company is not well prepared, somewhat prepared, or fully prepared? 3 What additional CWIS would you like to have available to your company in the future? 3 Has your company received training/guidance on how to make the most use of the available CWIS? 3 What challenges does your company face in gaining access to CWIS? What additional support is needed? 4 Does your company see itself as part of a larger landscape (e.g., watershed, or coastal zone)? If so, does that affect the way your company approaches its own efforts at CCA/DRR? Are there lessons learned from APIK that are important to include in future projects? 63 | FINAL PERFORMANCE EVALUATION USAID-APIK USAID.GOV INTERVIEW DISCUSSION GUIDE ² NATIONAL/PROVINCIAL/DISTRICT LEVEL Name of interviewee Male / Female GOI Ministry/Office Interviewed by Date Translator EQ # QUESTION 1 How is climate change affecting Indonesia and specifically this province/district? 1 Which impacts are of most concern and why? 1 How have you come to better understand and prioritize the impacts? (S-T/L-T) 1 What actions are you taking to help communities improve climate and disaster resilience? 1 In what ways has APIK helped your ministry/office to fulfill your responsibility for improved climate and disaster resilience? 1 What actions/methods are most important/effective at reducing risk? (S-T/L-T) 1 What kind of training has been provided by APIK? 1 Was the training useful, or could it have been made more useful? 1 Is there any feedback loop learning going on between national, provincial, district, and community levels? 1 If so, how does this occur and how could it be improved? 1 Has APIK helped your ministry/agency with the planning and budgeting process for CCA/DRR? What has been most effective? 2 Do you feel that APIK has effectively engaged the private sector for improved CCA/DRR in Indonesia? 2 What activities would you like to see future projects undertake with regard to PSE? 3 Have the improved CWIS (as a result of APIK support) benefitted your ministry/agency? If so, in what ways? 3 Does your ministry/agency have any new policies/practices in effect as a result of the improved CWIS? 3 Does your ministry/agency have all the climate and weather information that it needs to carry out its functions? 3 If not, what additional CWIS would you like to see made available in the future? 3 Has your ministry/agency received training/guidance on accessing and using CWIS? Has this been effective? 3 Are there any particular challenges facing your ministry/agency in accessing CWIS? What additional support is needed? 4 According to your understanding, what has been the APIK approach to define and understand the CCA/DRR issues to address and the spatial scale to address them? 4 Has a landscape approach been useful for promoting CCA/DRR? If so, in what ways? 4 Are there any measures you can suggest to improve the landscape approach to CCA/DRR? Are there lessons learned from APIK that are important to include in future projects? USAID.GOV FINAL PERFORMANCE EVALUATION USAID-APIK | 64 INTERVIEW DISCUSSION GUIDE ² APIK STAFF Name of interviewee Male / Female Role within APIK Interviewed by Location Date EQ # QUESTION Has the APIK theory of change been modified at all during the course of the project? If so, how? Have project targets and indicators changed? If so, why and in what way? 5 How did the timing of reduced funding affect project planning and delivery? 5 How did the amount of reduced funding affect project planning and delivery? 5 Were project beneficiaries made aware of these impacts and did they have concerns? 5 Did the reduced funding affect APIK staffing pattern? How? 5 What project activities were eliminated or scaled-back as a result of the reduced funding? 1 What challenges does APIK face in helping communities to improve climate resilience? 2 How has APIK PSE helped to achieve/improve overall project objectives? Increased efficiency/effectiveness/outreach? 3 What additional climate and weather information services do project beneficiaries wish to see in the future? 4 What has been the APIK approach related to defining and understanding a geography within which to apply climate resilience efforts? 4 Has a landscape approach been useful for promoting climate resilience? In what ways? What criteria were used to select field sites for implementing CCA/DRR measures? What criteria were used to select CCA/DRR implementation measures within target sites? Please explain the process and methodology for conducting the vulnerability analyses. Does it include ¶shock measurement·? That is, the relatiYe seYerit\ of the shock or stress? Was consideration given to the complex, interactive effects of shocks/stresses (e.g. flooding, health risk, household/community impacts, education, livelihoods, etc.)? Was consideration given to demographic/cultural factors (wealth, gender, religion, disenfranchised groups)? Or characteristics of resilience (presence of basic services, access to electricity, markets, infrastructure, urban/rural composition, etc.? Were gender dimensions fully explored and incorporated? Were short-term (acute) and long-term (chronic) considerations incorporated? Were idiosyncratic (individual/household) or covariate (larger community) considerations incorporated? Regarding PSE, were both business (enterprises) and financiers (lenders, insurance providers, etc.) involved? 65 | FINAL PERFORMANCE EVALUATION USAID-APIK USAID.GOV Does the GOI have incentives (Policies, legislation, tax breaks) in place to promote private sector involvement? Are there lessons learned from APIK that are important to include in future projects? USAID.GOV FINAL PERFORMANCE EVALUATION USAID-APIK | 66 ANNEX E. LIST OF RESPONDENTS DAY/ DATE ACTIVITY GENDER NAME OF KEY INFORMANTS POSITION/ INSTITUTION Tue, 18-Feb-20 Group Interview F Kuntarti Sri Rejeki Bappeda East Java F Wisye Souhuwat Bappeda Maluku M LM Ali Said Bappeda Southeast Sulawesi Group Interview F Ardanti APIK Regional Manager of East Java M Willy Wicaksono APIK Regional Manager of Maluku M Buttu Ma'dika APIK Regional Manager of SE Sulawesi Thu, 20-Feb-20 National Interview F Sri Tantri Arundhat Director for Climate Change Adaptation - Ministry of Environment and Forestry M Kardono Deputy Director of Vulnerability Identification and Analysis - Ministry of Environment and Forestry Group Interview M Paul Jeffery COP USAID-APIK M Agus Hernadi DCOP USAID-APIK M Raphael Anindito Climate Change Advisor USAID-APIK M Ari Mochamad Climate Governance Advisor USAID-APIK M Hidayatullah Al Banjari Private Sector Engagement Advisor USAID￾APIK M Putra Dwitama RAN-API USAID-APIK Mon, 24-Feb-20 Provincial Interview M Zuhdi Head of Provincial Enviromental Office, Surabaya F Sulityawati Climate Village Progra - Provincial Environmental Office, Surabaya F Rusdiana P Provincial Environmental Office, Surabaya Community/ Village Interview M Zainul Arifin Head of Dilem Village, Mojokerto M Chandra Yudha Secretary of Dilem Village Office, Mojokert M Wahyudi Dilem Village Office Staff, Mojokerto M Surejo Dilem Village Office Staff, Mojokerto F Woro Analupin Dilem Village Office Staff, Mojokerto F Duri Indah Pertiwi Dilem Village Office Staff, Mojokerto F Unun Maspupah Dilem Village Office Staff, Mojokerto Interview M Adi Suprayitno PT. Multi Bintang Indonesia, Mojokerto Community/ Village Interview M Termidi K Head of Wonokerto Village, Malang M Zaenal Abidin SLI of Sugarcane - Farmer Group, Malang M Suyanto Hedianto SLI of Sugarcane - Farmer Group, Malang M Paiman SLI of Sugarcane - Farmer Group, Malang M Mardianto SLI of Sugarcane - Farmer Group, Malang M Purwanto SLI of Sugarcane - Farmer Group, Malang F Ida Herniati SLI of Sugarcane - Farmer Group, Malang F Mujiyah SLI of Sugarcane - Farmer Group, Malang M Hari Suwarno SLI of Sugarcane - Farmer Group, Malang District Group Interview M Bambang Istiawan Head of BPBD Malang District M Ikhsan Hadi BPBD Malang District M Sadono Irawan BPBD Malang District 67 | FINAL PERFORMANCE EVALUATION USAID-APIK USAID.GOV M Anung Head of BMKG Karangploso, Malang District F Yulia Kresnawati Natural Resource Planning, Environment, and Water Resources, Bappeda Malang District M Tri S Bappeda Malang District F Sulin R. Bappeda Malang District F Anis Sulistyowati Enviromental Office Malang District Tue, 25-Feb-20 Interview M Dwidjo Prawito Head of BPBD Sidoarjo District District Group Interview M Dika Gunawan Fishery Agency Sidoarjo District F Fesan Eradianty Fishery Agency Sidoarjo District F Dewi Ratih Fishery Agency Sidoarjo District M Freegerd S Fishery Agency Sidoarjo District District Group Interview F Dwi S Bappeda Malang City F Dewi Alfiah Bappeda Malang City F Arum Pawestri Bappeda Malang City District Interview M Pandu Zanuar Settlement Area Division, Public Works Office, Malang City Group Interview M Moch. Syaiful Rizal Field Coordinator Malang District, USAID￾APIK M Erstayudha Hayyu Nur Rizqi GIS Specialist Malang District, USAID-APIK Wed, 26-Feb-20 Interview M Dwi Sukamto Academia of Marine and Fishery Polytechinc Academy, Sidoarjo District Community/ Village Interview M Koko Prayito Secretary of Sawahan Village, Sidoarjo District M Edi Iswanto Head of Kepetingan Hamlet, Sidoarjo District City Group Interview M Gatot Noegroho Head of BPBD Batu City F Ferry Setyaningrum Environmental Office Batu City F Nur Wiwit Puji Mahastiti Ika Bappelitbangda Batu City M Fatah Hayat Bappelitbangda Batu City Interview M Arif Dermawan Gensalim dan Himpunan Pegiat Adiwiyata Indonesia (HPAI), Malang City Thu, 27-Feb-20 District Group Interview M Haryono Head of Bappeda Mojokerto District M Heru Nugroho Bappeda Mojokerto District Provincial Interview M Gatot S BPBD East Java Province F Pinky Hidayati BPBD East Java Province Community/ Village Interview M Suryanto Segoro Tambak Village, Sidoarjo District M Kodero Segoro Tambak Village, Sidoarjo District F Aisyah Minarsih Segoro Tambak Village, Sidoarjo District Community/ Village Interview M Suwiji Bumiaji Village, Batu City M Nur Kholiq Bumiaji Village, Batu City Community/ Village Interview F Mamikh Misniati Head of Sitiharjo Village, Malang District M Dwi Pujo Santoso Sitiharjo Village Staff, Malang District M Yusak Krismanto Head FPRB Sitiharjo Village, Malang District Community/ Village Interview F Dyah Pitaloka Academia of Universitas Islam Raden Rahmat, Malang M Joko Utomo Manisa Farmer Group, Tulungrejo, Batu City USAID.GOV FINAL PERFORMANCE EVALUATION USAID-APIK | 68 M Sukamto Manisa Farmer Group, Tulungrejo, Batu City M Moch. Ridwan Manisa Farmer Group, Tulungrejo, Batu City M Moch. Yasin Manisa Farmer Group, Tulungrejo, Batu City M Nurali Manisa Farmer Group, Tulungrejo, Batu City M Hartono Manisa Farmer Group, Tulungrejo, Batu City F Elok Manisa Farmer Group, Tulungrejo, Batu City M Sulianto Manisa Farmer Group, Tulungrejo, Batu City M Aripin Manisa Farmer Group, Tulungrejo, Batu City F Yuni Manisa Farmer Group, Tulungrejo, Batu City M Sriah Manisa Farmer Group, Tulungrejo, Batu City Mon, 2-Mar-20 Provincial Interview M Ansar Head of SES Provincial Environmental Office, Kendari F Hamrilah SES Provincial Environmental Office, Kendari M Untung Ratu SES Provincial Environmental Office, Kendari M Aminoto K SES Provincial Environmental Office, Kendari F Yuliana Ulfah SES Provincial Environmental Office, Kendari M Takdir SES Provincial Environmental Office, Kendari City Group Interview M Ridwansyah Raridala Head of Bappeda Kendari City F Minarni Ajusa Bappeda Kendari City F Isria Bappeda Kendari City F Athikasari Bappeda Kendari City M Ridwan Saleh Bappeda Kendari City M Daeng Pasorong BPBD Kendari City M Aris Yunatas BMKG Maritim Kendari City F Arniaty DK Environmental Office Kendari City F Wd. Murniati S Public Work Office Kendari City Interview M Jamhir Safani Academia of Universitas Halu Oleo/ Head of FPRB Province Tue, 3-Mar-20 Interview M Denis Priyanto PT. Syngenta Indonesia, Kendari Interview M Dwi Larompong Kalla Kakao Industri, Kendari Community/ Village Interview M Budi Utomo Head of Kampung Salo Village, Kendari M Sardin Head of KSB Kampung Salo Village, Kendari M Burhan Odo Community Kampung Salo, Kendari M Wahyudi Arif Community Kampung Salo, Kendari M Sulaiman Community Kampung Salo, Kendari F Asni Waste Bank Kampung Salo, Kendari F Rinawati Waste Bank Kampung Salo, Kendari M Sutamis Waste Bank Kampung Salo, Kendari Provincial Interview M Roy C. Sianta Provincial Environmental Office, Ambon F Elma N. Ririhena Provincial Environmental Office, Ambon M Haikal Baadola Provincial Environmental Office, Ambon F Rithasya T. Wenno Provincial Environmental Office, Ambon City Group Interview F Feberien Maail Bappeda Ambon City F Andreyanni Kiriwenno Bappeda Ambon City F Jetkel Farfar Bappeda Ambon City 69 | FINAL PERFORMANCE EVALUATION USAID-APIK USAID.GOV F Elda E. Silanno Bappeda Ambon City F Wilhelmina Paays BMKG Pattimura, Ambon City M Juan E. Kayadoe Bappelitbang Ambon City M A. J. Hehamahua Environmental Office Ambon City F Eva Tuhumury BPBD Ambon City Provincial Interview M John Hursepuny BPBD Maluku Province M Fretha Julian K BPBD Maluku Province Wed, 4-Mar-20 Group Interview M Aminuddin Mani Kandori Dean Faculty of Enviroment and Forestry, Univ Halu Oleo, Kendari M La Baco Sudia Academia of Unviversitas Halu Oleo, Kendari/ Head of Forum DAS Province SES M Safril Kasim Academia of Universitas Halu Oleo, Kendari/ Head of FPRB Kendari City Interview M Solihin Program Coordinator of LepMIL, Kendari Community/ Village Interview M Zesnat Ferdinanders Raja Negeri Haruku, Maluku M Yopi Salmon Staf Negeri Haruku Village office, Maluku M J. Lappy Staf Negeri Haruku Village office, Maluku M Clif H. Kissya Head of Pokmas API-PRB Negeri Haruku, Maluku M Feri Lappy Member of Pokmas API-PRB Negeri Haruku, Maluku F Johana Selanno Member of Pokmas API-PRB Negeri Haruku, Maluku M Will Tuahatu Kaur of Negeri Haruku, Maluku M Yohanes Ferdinandes Secretary of Saniri of Negeri Haruku, Maluku M Andrianus Mantouw Saniri of Negeri Haruku, Maluku M Markus Souissa Community Negeri Haruku, Maluku F Lina Amahoru Community Negeri Haruku, Maluku M Paulus Mustaru Community Negeri Haruku, Maluku Community/ Village Interview F Amalia Sialana Pokmas API-PRB Negeri Morella, Cental Maluku M Hairun Sialana Pokmas API-PRB Negeri Morella, Cental Maluku M M. Saleh Sasole Pokmas API-PRB Negeri Morella, Cental Maluku Thu, 5-Mar-20 District Group Interview M Assruddin ASM Secretary of Bappeda South Konawe District M Mutakhir Hidayat Bappeda South Konawe District M Edy Amiruddin Bappeda South Konawe District M Riswan Mahgidi Environmental Office South Konawe District M Ober Adewitra Environmental Office South Konawe District M Sumartono BPBD South Konawe District Community/ Village Interview F Fatkuriyawachidah Head of Female Farmer Group, SLI Corn, South Konawe District M Yahya Head of Ombu Jaya Farmer Group, SLI Corn, South Konawe District M Junaid Head of Laeya Farmer Group, SLI Corn, , South Konawe District M Sunarji Member of Farmer Group, SLI Corn, South Konawe District USAID.GOV FINAL PERFORMANCE EVALUATION USAID-APIK | 70 F Suwarni Ningsih Member of Farmer Group, SLI Corn, South Konawe District F Iluh Saniasih Member of Farmer Group, SLI Corn, South Konawe District M Fadlan Member of Farmer Group, SLI Corn, South Konawe District Interview M Agustinus Kastanya Academia of Universitas Pattimura, Ambon Thu, 12-Mar-20 Interview M Yohanes Agung Baskoro PT. Cargill, Jakarta Fri, 13-Mar-20 National Interview M Dodo Gunawan Head of Center for Climate Change Information, BMKG Mon, 16-Mar-20 Interview F Irmia Fitryah Gender Specialist USAID-APIK Tue, 17-Mar-20 National Interview M M. Rokhis Khomarudin Head of Utilization Remote Sensing Center, LAPAN, Jakarta M Syarif Budhiman Utilization Remote Sensing Center, LAPAN, Jakarta National Interview M Fachri Labalado Director of Community Village Empowerment, Ministry of Village Tue, 31-Mar-20 Interview M Yacub Nugraha ACA Insurance Sat, 4-Apr￾20 Interview M Raditya Jati Director of Disaster Risk Reduction, National Disaster Management Agency (BNPB) Wed, 8- Apr-20 Interview M Medrilzam Director of Environmental, National Development Planning Agency (Bappenas) Respondents by Gender Table 8. Respondents by Gender Female; 52; 31% Male; 115; 69% Gender of Respondents 71 | FINAL PERFORMANCE EVALUATION USAID-APIK USAID.GOV Respondents by Type 8 74 58 9 5 7 2 4 Type of Respondents National Government Local Government Community APIK Jakarta APIK Field Staff Academia NGO Private Sector Table 9. Respondents by Type USAID.GOV FINAL PERFORMANCE EVALUATION USAID-APIK | 72 ANNEX F. RESILIENCE FUND GRANTS NO GRANTEES WORKING AREA ACTIVITY 1 Yayasan PATTIRO Timeline: July 2016 to March 2017 East Java Province, Malang District Mainstream climate adaptation and disaster risk reduction in village budget and planning documents. 2 Halu Oleo University Research Institution Timeline: August 2016 to April2017 SE Sulawes Province South Konawe District Improve fisheries through integration of Fish Aggregating Devices (FADs) and bio-reef technology. 3 Yayasan Walang Perempuan Timeline: May 2017 to October 2018 Maluku Province, Central Maluku District Align local wisdom (Nanaku and Sasi) with climate and weather forecast produced by BMKG. 4 Yayasan PATTIRO Timeline: June 2017 to August 2018 East Java Province, Blitar District Improve community and local government resilience in disaster impact (flood and landslide), and improve capacity to access CWIS. 5 Destructive Fishing Watch (DFW) Indonesia Timeline: August 2017 to December 2018 SE Sulawesi Province, South Konawe District Integrate climate and disaster resilience in village planning and budgeting, mangrove ecosystem and seaweed farming 6 Lembaga Pengembangan Masyarakat Pesisir dan Pedalaman (LePMIL) Timeline: August 2017 to January 2019 SE Sulawesi Province, South Konawe District Rehabilitate Wanggu Watershed and improve management to reduce flood and landslide risk through social forestry and collaboration of key stakeholders using a landscape approach. 7 Pusat Kajian Teknologi Terapan (PKTT) Fakultas Teknik dan Ilmu Komputer Timeline: March to November 2018 East Java Province, Batu District Implement conservation agriculture through chayote farming and drip irrigation methods as adaptation efforts to climate impact and maintaining landscape condition and land use. 73 | FINAL PERFORMANCE EVALUATION USAID-APIK USAID.GOV NO GRANTEES WORKING AREA ACTIVITY 8 PATTIRO Malang Timeline: April to November 2018 East Java Province, Malang District Reduce flood risk in Sitiarjo Village through EWS installation and good practices for improved coffee production. 9 Wahana Edukasi Harapan Alam Semesta Timeline: August 2018 to December 2019 East Java Province, Mojokerto District Develop rain water harvesting (RWH) units as a measure to reduce vulnerability to drought and build climate resilience. 10 Lembaga Partisipasi Pembangunan Masyarakat Timeline: August 2018 to November 2019 Maluku Province, Central Maluku District Provide weather and climate information display and introduce permaculture methods as adaptation actions. 11 Perkumpulan Sulawesi Institute (SI) Timeline: October 2018 to October 2019 SE Sulawesi Province, South Konawe Farming integration system implementation as a strategy for strengthening community livelihood. 12 Yayasan Walang Perempuan Timeline: May 2019 to December 2019 Maluku Province, Central Maluku District Improve community resilience through sustainable permaculture and improved water access during drought. USAID.GOV FINAL PERFORMANCE EVALUAITON USAID-APIK | 74 ANNEX G. LIST OF DOCUMENTS AND DATA SOURCES APIK CVAs 1. Downscaling Proyeksi Perubahan Iklim di Maluku 2. Kajian Kerentanan Blitar Malang Raya 3. Kajian Kerentanan Jombang Mojokerto Sidoarjo 4. Kajian Kerentanan Kab Kepulauan Aru 5. Kajian Kerentanan Kendari Konsel 6. Kajian Kerentanan Kepulauan Ambon and Lease 7. Kajian Kerentanan Provinsi Jawa Timur 8. Kajian Kerentanan Provinsi Maluku 9. Kajian Kerentanan Provinsi Sulawesi Tenggara APIK WORKPLANS & REPORTS 1. 2015_11_24 APIK Technical Statement of Work 2. APIK M&E Plan June 25 2019_rev 3. APIK_PY1_Workplan 4. APIK_PY2_Workplan_Ver4_Nov 2016_FINAL 5. APIK_PY3_Workplan_final_Nov 9 6. APIK_PY4_Workplan 7. APIK_PY5_Workplan 8. APIK Annual Report PY1 9. APIK Annual Report PY2 10. APIK Annual Report PY3 11. APIK Annual Report PY4 12. APIK Quarterly Report, No. 1, PY5 APIK TECHNICAL DOCUMENTS 1. MSC APIK USAID 2. MSC Selected Stories PY4 3. Fact-sheet-SLI-Tebu_Sugar Cane Field School Apr 2018 4. APIK-CWIS-Assessment-Report-Dec 2016-weather services 5. Fact-sheet-APIK Climate Field School 6. Fact-sheet-Landskap-Pacet-Final-Merged Apr 2019 7. Fact-sheet-Profil-Jatim_English-14 Mar 2019 East Java Profile 8. Final Site Selection Memorandum USAID EVAL REFERENCE MATERIALS 1. USAID Evaluation Policy 2. How_to-note_Preparing_Evaluation_Reports ANNEX H. CONFLICT OF INTEREST DISCLOSURES USAID.GOV FINAL PERFORMANCE EVALUAITON USAID-APIK | 76 USAID.GOV FINAL PERFORMANCE EVALUAITON USAID-APIK | 78 USAID.GOV FINAL PERFORMANCE EVALUAITON USAID-APIK | 80 ANNEX I. INDONESIAN VERSION OF THE EXECUTIVE SUMMARY AND RECOMMENDATIONS EVALUASI AKHIR KINERJA USAID ² ADAPTASI PERUBAHAN IKLIM DAN KETANGGUHAN (APIK) USAID Contract No. AID-497-C-16-00006 Indonesia Monitoring & Evaluation Support Project Juni 2020 Disusun oleh: Richard Volk Saut Sagala Retno Handini Indradi Soemardjan David Callihan Irma Sitompul Foto sampul depan: ‹ USAID-APIK Kanan atas: Petani jagung di Malang, Jawa Timur Kiri bawah: Nelayan di Maluku Kanan bawah: Pelatihan tanggap darurat di Malang, Jawa Timur USAID.GOV RINGKASAN EKSEKUTIF DAN REKOMENDASI EVALUASI AKHIR KINERJA USAID-APIK | 2 1. RINGKASAN EKSEKUTIF LATAR BELAKANG PROYEK Proyek APIK (Adaptasi Perubahan Iklim dan Ketangguhan / Climate Change Adaptation and Resilience) yang berdurasi 4,5 tahun, diluncurkan pada bulan Januari 2016 dan didanai oleh USAID sebesar US$15,4 juta. Tujuan dari Proyek ini adalah untuk membantu Pemerintah Indonesia dalam memperkuat ketangguhan terhadap bencana dan perubahan iklim di Indonesia. Awalnya program ini dirancang sebagai program lima tahun dengan dana US$19,3 juta, namun pengurangan dana menyebabkan penutupan proyek enam bulan lebih cepat dari jadwal semula. Proyek ini menggunakan pendekatan terpadu berbasis ruang yang melibatkan berbagai pemangku kepentingan dari pemerintah, sektor swasta, dan masyarakat lokal untuk mencapai tiga tujuan utama: (1) Pengarusutamaan adaptasi perubahan iklim (API), dan pengurangan risiko bencana (PRB) ke dalam kerangka tata kelola nasional dan daerah; (2) Membangun kapasitas masyarakat lokal dan sektor swasta untuk menangani perubahan iklim dan ancaman bencana alam terkait cuaca; dan (3) Mendukung penggunaan layanan informasi iklim dan cuaca dalam menangani risiko dan kerentanan yang diakibatkan oleh peristiwa cuaca ekstrem dan perubahan pola iklim. Perubahan iklim merupakan ancaman bagi kesehatan, keselamatan, dan kesejahteraan manusia di seluruh dunia³dan Indonesia tidak luput dari ancaman ini. Peningkatan suhu, perubahan frekuensi dan intensitas curah hujan, cuaca ekstrem, dan kenaikan permukaan laut telah membawa risiko dan kerentanan terhadap penduduk serta sektor usaha di seluruh negeri. Meskipun pengaruh perubahan iklim dirasa berbeda pada intensitas yang juga berbeda di 34 provinsi di Indonesia, dampak yang dirasakan di masa depan kemungkinan akan semakin signfikan. Indonesia merupakan negara dengan ekonomi terbesar ke sepuluh di dunia dan digolongkan sebagai negara berpenghasilan menengah ke bawah yang sedang mengalami pertumbuhan.27 Namun analisis yang didanai oleh USAID di tahun 2016 memprediksi bahwa ekonomi Indonesia berpotensi mengalami kerugian tahunan hingga 1,4 persen dari PDB pada tahun 2050, yang berdampak pada sektor pertanian dan kesehatan, dan kerusakan yang disebabkan oleh kenaikan permukaan laut.28 APIK dirancang untuk meningkatkan profil risiko iklim dan kerentanan yang berbasis bukti di Indonesia. Dalam menjalankan kegiatannya, APIK berkoordinasi dengan Pemerintah Indonesia di tingkat pusat dan daerah, dengan melibatkan sektor swasta dan masyarakat setempat dalam perancangan, dan pelaksanaan adaptasi untuk memperkuat ketangguhan, dan membagikan model praktik-praktik terbaik untuk dapat direplikasi dan diperluas. Laporan ini menggambarkan temuan, kesimpulan, dan rekomendasi dari evaluasi kinerja akhir Program USAID APIK yang awalnya dijadwalkan selesai pada bulan Juni 2020. 27 World Bank, 2020. 28 USAID ATLAS, 2016. 3 | RINGKASAN EKSEKUTIF DAN REKOMENDASI EVALUASI AKHIR KINERJA USAID-APIK USAID.GOV TUJUAN DAN METODE EVALUASI USAID/Indonesia melaksanakan kajian ini untuk mengevaluasi kemajuan, relevansi, dan efektivitas program dan mengidentifikasi pembelajaran yang dapat dipetik dari Proyek APIK. Temuan dan rekomendasi dari evaluasi ini akan memberikan masukan untuk rancangan program di masa depan dan untuk perbaikan program ketangguhan perubahan iklim dan bencana di Indonesia. Evaluasi akhir ini dirancang untuk mengumpulkan data dan informasi dari berbagai pemangku kepentingan dan penerima manfaat. Proses evaluasi telah mengikuti standar evaluasi yang telah ditetapkan, untuk memastikan independensi, akurasi dalam menghasilkan temuan, kesimpulan, dan rekomendasi. Tim evaluasi yang terdiri dari enam orang berada di Jakarta pada pertengahan Februari 2020 dan telah melaksanakan sejumlah wawancara, diskusi kelompok terarah/focus group discussions (FGD), dan konsultasi. Pengumpulan data dilakukan selama lima minggu dengan mewawancarai 170, 52 diantaranya perempuan, pemangku kepentingan program, dari Jakarta dan tiga provinsi target program (Jawa Timur, Sulawesi Tenggara, dan Maluku). RINGKASAN TEMUAN DAN KESIMPULAN Pendekatan APIK³untuk mencapai ketangguhan berbasis tempat dengan mengatasi permasalahan kelembagaan, sosial, ekonomi, serta lanskap³ merupakan teori perubahan yang sangat sesuai, terutama untuk negara yang sangat terdesentralisasi. Upaya program untuk membangun ketangguhan melalui aksi bersama antara pemerintah nasional dan daerah, sektor swasta serta masyarakat, dan meningkatkan ketersediaan dan penggunaan informasi iklim dan cuaca oleh pengguna akhir, merupakan upaya peningkatan ketangguhan yang sesuai dan logis. Upaya APIK ini perlu diapreasiasi karena telah membantu meningkatkan profil dan urgensi dari adaptasi iklim, dengan membantu menjelaskan beragam kerentanan yang harus dihadapi saat ini, serta persiapan untuk menghadapi ketidakpastian di masa depan. Kerja sama dengan perguruan tinggi yang terjalin merupakan komponen yang penting untuk program ke depan, terutama mengingat perlunya menghubungkan ilmu pengetahuan dengan kebijakan di berbagai bidang secara berkelanjutan. Pencapaian terpenting yang telah dilakukan olek APIK adalah terkait dengan peningkatan pemahaman akan risiko dan kerentanan iklim, serta komitmen-komitmen baru yang dibuat oleh otoritas daerah. Hal ini merupakan upaya APIK untuk meningkatkan investasi dan perhatian terhadap API dan PRB di Indonesia. Sejumlah kebijakan dan peraturan yang ada saat ini telah mengaplikasikan isu-isu dan tindakan-tindakan yang terkait dengan ketangguhan iklim, seperti terlihat dalam dokumen perencanaan dan penganggaran di tingkat nasional, daerah dan desa. Bantuan APIK kepada tiga provinsi dalam penyusunan pengajuan pendanaan dari Green Climate Fund (GCF), berpotensi menjadi peninggalan yang sangat berarti bagi Indonesia. Sebelum APIK, Indonesia tidak pernah memanfaatkan dana GCF untuk adaptasi iklim. Saat ini, tiga pengajuan pendanaan tersebut telah berhasil maju ke tahap proposal lengkap dengan organisasi yang terakreditasi GCF. Hal ini merupakan pencapaian yang signifikan untuk APIK dan kemajuan yang penting untuk Indonesia. Dalam banyak hal, APIK telah menjadi proyek unik dalam pendekatannya terhadap adaptasi iklim di Indonesia, dimana APIK telah mendukung Pemerintah Indonesia di semua tingkatan. Dukungan dilakukan melalui peningkatan komunikasi dan koordinasi, bersamaan dengan memperkenalkan beragam kegiatan percontohan untuk memperbaiki pergeseran sikap, serta praktik sektor swasta serta masyarakat menuju ke arah peningkatan ketangguhan. Program APIK telah mencapai lima dari enam target tingkat outcome dari proyek. USAID.GOV RINGKASAN EKSEKUTIF DAN REKOMENDASI EVALUASI AKHIR KINERJA USAID-APIK | 4 Seperti program besar lainnya yang memiliki multi-komponen dan beroperasi di beberapa lokasi, tim mengidentifikasi beberapa kekurangan dalam evaluasi ini. Keberlanjutan beberapa kegiatan dipertanyakan, dan alasan teknis serta pendekatan untuk kegiatan lainnya juga diperdebatkan. Namun ini merupakan peluang penting untuk pembelajaran dan perlu digarisbawahi bahwa penutupan program yang enam bulan lebih awal dari jadwal, tidak memberi cukup waktu bagi staf APIK untuk mengevaluasi, merefleksikan serta berbagi pembelajaran sebagaimana yang mereka harapkan. Secara umum, pemangku kepentingan yang diwawancara memuji dedikasi, antusiasme dan dukungan teknis yang diberikan oleh staf APIK. PE1: Di wilayah dimana APIK beroperasi, sejauh manakah APIK telah membangun aspek ketangguhan dan adaptasi di berbagai tingkatan pemerintah serta masyarakat? Dan bagaimana upaya APIK di tingkat nasional telah mempengaruhi atau dipengaruhi oleh pemerintah di tingkat kabupaten dan masyarakat? APIK berinvestasi banyak dan dari awal telah melibatkan pemangku kepentingan di semua tingkat pemerintahan, masyarakat dan sektor swasta. Analisis yang dilaksanakan oleh APIK dengan beberapa pemangku kepentingan merupakan upaya yang sangat penting, untuk membentuk diskusi awal terkait risiko dan kerentanan iklim saat ini, dan di saat mendatang. Upaya ini telah membangun landasan kepercayaan, komitmen dan semangat kolaborasi yang nyata di ketiga provinsi sasaran APIK. Salah satu staf APIK telah diperbantukan di Badan Perencanaan Pembangunan Nasional (Bappenas) untuk membantu Sekretariat Rencana Aksi Nasional Adaptasi Perubahan Iklim (RAN-API) dalam merevisi dan memperbarui RAN-API yang bergerak menuju persetujuan di tahun 2020. Dapat dikatakan bahwa peran koordinasi dan fasilitasi dari APIK, terutama di tingkat daerah, telah menjadi salah satu pencapaian terpenting. Hal ini dapat dilihat dengan program-program adaptasi iklim ke dalam kebijakan, perencanaan dan anggaran di semua tingkatan. RPJMN Indonesia tahun 2020-2024 menyertakan dana US$3,02 milyar (IDR43,45 triliun) untuk adaptasi iklim, yang merupakan bukti jelas adanya peningkatan pemahaman serta urgensi terhadap isu yang telah berkembang dalam lima tahun terakhir ini. Signifikansi dan kesuksesan langkah-langkah adaptasi yang diterapkan di ketiga provinsi cukup beragam. Meskipun ada puluhan ribu orang yang terlibat dalam kegiatan program³ termasuk pelatihan, Sekolah Lapang Iklim, proyek percontohan, dan banyak kegiatan lainnya³namun perlu dipertanyakan apakah berbagai tindakan adaptasi ini, jika dilihat secara agregat, sudah mewakili tingkat keluaran (output) dan hasil (outcome), yang secara efektif dan berkelanjutan dapat meningkatkan ketangguhan untuk porsi populasi target yang cukup besar di tiga provinsi. Namun, perlu juga diketahui bahwa pengujian berbagai program intervensi memberikan pelajaran yang berharga dalam peningkatan fokus dan efektivitas program-program di masa depan. APIK telah memberikan arahan terhadap pemerintah daerah terkait dengan prioritas lokal dalam langkah-langkah adaptasi. Walaupun upaya ini merupakan upaya yang baik dilihat dari sisi kepemilikan dan komitmen lokal, namun sekaligus menimbulkan pertanyaan terkait relevansi strategis, skala, replikasi, dan keberlanjutan. Pemikiran lebih lanjut dapat ditemuan di bagian rekomendasi. PE2: Bagaimana kerja sama APIK dengan sektor swasta, apakah hal ini dapat meningkatkan kesadaran atas adaptasi perubahan iklim dan pengurangan risiko bencana? Upaya awal APIK dalam melibatkan sektor swasta diawali dengan kegiatan penjangkauan (outreach) dan mengikuti pertemuan-pertemuan di mana risiko dan kerentanan iklim didiskusikan. APIK juga memiliki banyak peluang untuk menjalin kontak dengan perusahaan. Staf APIK telah banyak terlibat dalam memberikan paparan di pertemuan asosiasi bisnis, termasuk di Kamar Dagang dan Industri Indonesia (Kadin) dan Kamar Dagang dan Industri Amerika (AmCham). Di tahun ketiga proyek (PY3) kontak 5 | RINGKASAN EKSEKUTIF DAN REKOMENDASI EVALUASI AKHIR KINERJA USAID-APIK USAID.GOV langsung dengan bisnis lebih banyak dilakukan, setelah kajian terhadap 40 perusahaan yang mengerucut kepada 12 perusahaan yang bersedia bekerja sama dengan APIK, pemerintah daerah dan masyarakat pada beberapa proyek tertentu dengan menggunakan dana Corporate Social Responsibility (CSR). Peran APIK utamanya adalah sebagai penggagas dan fasilitator dalam menghubungkan kelompok-kelompok tersebut, sambil membantu menyusun MOU serta merancang kegiatan lokal. Walaupun pelaksanaan kegiatan pada umumnya memberikan hasil yang positif dan bermanfaat, namun demikian, kegiatan APIK yang melibatkan sektor swasta ini masih berskala kecil dan penerima manfaat langsungnya juga masih sedikit. Tim Evaluasi hanya menemukan sedikit bukti bahwa kegiatan yang melibatkan swasta akan direplikasi atau diperluas, sebagai contoh, tidak adanya rencana atau komitmen dari pemerintah daerah untuk memberikan dukungan yang berkelanjutan. Pengecualiannya mungkin terjadi dalam kasus investasi oleh PT Cargill yang bertujuan meningkatkan produksi telur melalui replikasi kandang ayam petelur yang lebih baik. Secara keseluruhan, upaya pelibatan sektor swasta (PSE) yang dilakukan oleh APIK kurang optimal untuk menciptakan solusi berbasis pasar, yang didasari analisis pasar yang serius untuk menghasilkan proposisi nilai (value propositions) dan meyakinkan perusahaan untuk menerapkan praktik-praktik terbaik ke dalam operasi mereka. Namun, pada skala lokal, beberapa upaya APIK memang membawa manfaat yang terukur bagi pihak yang berpartisipasi dan menawarkan sebuah model yang layak dipertimbangkan untuk diperluas dalam program-program mendatang. PE3: Sampai sejauh manakah penerima manfaat proyek APIK menjadi lebih siap dan dapat beradaptasi sebagai hasil dari layanan informasi iklim dan cuaca APIK? Perampungan kajian kebutuhan dan peluang layanan informasi iklim dan cuaca di Tahun Proyek1(PY1) telah menghasilkan ´rancangan peta jalan (roadmap)µ. Roadmap ini membantu APIK serta Badan Meteorologi, Klimatologi dan Geofisika (BMKG) dalam menyusun rencana kerja untuk meningkatkan layanan informasi iklim. Kementerian Lingkungan Hidup dan Kehutanan (KLHK) dan Badan Nasional Penanggulangan Bencana (BNPB), pemerintah daerah, masyarakat dan Sekolah Lapang Iklim (SLI CFS) diidentifikasi sebagai penerima manfaat untuk perbaikan layanan informasi iklim dan cuaca (CWIS). APIK telah membantu peningkatan perangkat-perangkat pendeteksi kerentanan iklim dan risiko bencana utama yang dipakai oleh KLHK dan BNPB. Namun, pengintegrasian perangkat-perangkat ini tidak dapat diselesaikan. Pengintegrasian tujuan API dan PRB kedalam satu rangkaian pedoman dan perangkat adalah tugas penting yang harus ditangani oleh pemerintah nasional. Di tingkat lokal, APIK telah memberikan proyeksi iklim yang di-downscale pada tingkat provinsi, dan selanjutnya digunakan dalam menyusun RAD-API. APIK juga membantu BMKG dalam penyusunan perangkat lunak OtoKlim, yang memungkinkan tampilan spasial informasi cuaca berbasis web, dan di berbagai media komunikasi (mis. media sosial, grup WhatsApp). OtoKlim bertujuan untuk meningkatkan akses terhadap data cuaca di tingkat daerah dan masyarakat. Para pemangku kepentingan mengapresiasi upaya APIK untuk meningkatkan layanan informasi iklim dan cuaca (CWIS), serta beberapa sistem peringatan dini (EWS) untuk bencana banjir dan longsor yang dipasang di beberapa desa di ketiga provinsi. Pelatihan telah menjadi komponen besar dari proyek APIK untuk meningkatkan layanan informasi iklim dan cuaca (CWIS), sebagai hasilnya masing-masing Sekolah Lapang Iklim sekarang dapat menggunakan data cuaca ke dalam kegiatan sektor pertanian dan perikanan. Masyarakat menyambut positif kegiatan Sekolah Lapang Iklim APIK, walaupun penggunaan informasi cuaca yang baru disediakan masih belum terstruktur, dikarenakan kurangnya kapasitas teknis serta pendanaan untuk pengoperasian dan pemeliharaan alat. Komponen layanan informasi iklim dan cuaca (CWIS) dari APIK yang juga penting adalah dengan memusatkan perhatian pada sains dan berbasis bukti, yang diharapkan dapat meningkatkan perhatian Pemerintah Indonesia pada isu API/PRB di tahun-tahun mendatang. USAID.GOV RINGKASAN EKSEKUTIF DAN REKOMENDASI EVALUASI AKHIR KINERJA USAID-APIK | 6 PE4: Apa saja pelajaran utama yang dapat dipetik dari penerapan pendekatan bentang alam bagi tiga aspek berikut: daerah aliran sungai; wilayah pesisir; dan kepulauan kecil? Sebagai salah satu dari tiga pilar teori perubahan APIK, pendekatan bentang alam (landscape approach), merupakan kontribusi penting terhadap diskusi kerentanan iklim dan risiko bencana. Dalam diskusi￾diskusi tersebut, APIK memberikan informasi terkait dengan Kajian Kerentanan Iklim (Climate Vulnerability Assessments, CVA) yang telah dilakukan dan kajian pemilihan lokasi yang terperinci. Kajian tersebut telah memberikan unsur ilmu pengetahuan dan kumpulan data ke dalam diskusi dan proses perencanaaan dan penganggaran. Kajian Kerentanan Iklim yang dilakukan pada skala provinsi, kabupaten/kota, dan masyarakat menggunakan faktor bahaya/ancaman yang sama, namun dengan tingkat rincian yang semakin tinggi di setiap skalanya. APIK juga melatih staf-staf lokal, yang kemudian memimpin pelatihan-pelatihan Kajian Kerentanan Iklim untuk kabupaten lainnya dengan bantuan staf APIK. Dari proyek APIK, pemangku kepentingan mempelajari bahwa kerentanan sebuah tempat sangat berhubungan dengan karakteristik ekologis, hidrologis, dan meteorologis lanskap sekitarnya. Sehingga keputusan tata guna lahan yang baik adalah yang melindungi layanan ekosistem, komunitas perkotaan, pedesaan, dan pulau dari cuaca ekstrem dan perubahan cuaca. APIK memberi kontribusi penting dalam membawa konsep-konsep serta analisis semacam ini dalam diskusi, dan mencoba membantu dalam pelaksanaan kegiatan yang berskala lanskap. Contohnya termasuk perbaikan drainase air hujan yang direncanakan di Sulawesi Tenggara, sumur resapan di Jawa Timur, restorasi hutan bakau di empat lokasi, dan perlindungan mata air dengan penanaman pohon bambu. Sayangnya, karena program harus ditutup lebih awal, APIK tidak dapat melakukan pemantauan lebih lanjut dan melihat kesuksesan, efektivitas dan keberlanjutan dari program-program tersebut. Dalam hal pembelajaran, meskipun pendekatan bentang alam bukan konsep yang sepenuhnya baru di Indonesia, namun hanya terdapat sedikit contoh pelaksanaannya. Pendekatan bentang alam akan sangat bermanfaat bila konsep utamanya dapat diklarifikasi terlebih dulu, termasuk definisi serta panduan praktis tentang cara mengimplementasi pendekatan ini sebagai komponen dasar dari ketangguhan iklim. PE5: Apa dampak dari kendala pendanaan dan peluang apa saja yang terlewatkan sebagai akibatnya? Anggaran serta jangka waktu program awalnya disetujui pada US$19,3 juta selama lima tahun. Namun di Tahun Proyek 2 (PY2), APIK mendapatkan informasi bahwa terdapat potensi terjadi pengurangan anggaran dan oleh sebab itu APIK perlu mengurangi pengeluaran bulanannya. Di awal Tahun Proyek 5 (PY5), program APIK diinformasikan oleh USAID bahwa hanya akan menerima total dana sebesar US$15,4 juta. Hal ini memaksa APIK untuk menyudahi program enam bulan lebih awal. Dengan kata lain, APIK harus menyelesaikan programnya dalam waktu 88 persen dari jangka waktu awal dengan dana 79 persen dari anggaran awal. Dampak yang paling signifikan dari pengurangan anggaran ini adalah banyaknya kegiatan lokal yang tidak dapat dilaksanakan atau tidak diselesaikan secara penuh. Juga, Dana Ketangguhan (dana hibah kecil) yang awalnya dianggarkan sebesar US$2 juta dikurangi menjadi US$650.000. Sangat disayangkan lagi, rencana APIK untuk menggunakan tahun terakhir program untuk tujuan evaluasi, refleksi dan berbagi pembelajaran tidak berjalan sesuai harapan. Terlepas dari kekecewaan tersebut, saat ditanya selama wawancara dan diskusi kelompok terarah (FGD) terkait rekomendasi dan masukan untuk program di masa depan, para pemangku kepentingan merespon positif terhadap proyek APIK dan menunjukkan antusiasme dan harapan yang kuat untuk potensi perpanjangan program atau program serupa di masa depan untuk melanjutkan pekerjaan penting yang telah dicapai APIK. 7 | RINGKASAN EKSEKUTIF DAN REKOMENDASI EVALUASI AKHIR KINERJA USAID-APIK USAID.GOV 2. REKOMENDASI Terdapat urgensi dalam tugas membantu masyarakat dan sektor bisnis Indonesia beradaptasi terhadap dampak perubahan Iklim di masa yang akan datang. Perlu adanya tindakan-tindakan yang lebih strategis, sinergis, dan transformatif pada skala besar. Meskipun penting untuk mengarusutamakan upaya adaptasi di semua sektor, namun dalam mencapai tujuan tersebut, dituntut upaya yang terfokus pada penyusunan data dan informasi, kebijakan dan undang-undang yang diperlukan, pendanaan publik dan swasta, serta kapasitas kelembagaan untuk mendukung langkah-langkah implementasi di semua tingkat pemerintahan dan masyarakat. Harus dirancang sedemikian rupa agar tujuan dari adaptasi, mitigasi, dan ketangguhan ekologi saling bersinergi. Penurunan emisi GRK memiliki potensi untuk mengurangi kebutuhan, biaya, dan langkah￾langkah adaptasi dalam jangka panjang. Demikian pula, upaya untuk meningkatkan keutuhan ekologi akan memungkinkan Indonesia menjadi lebih tahan terhadap dampak buruk perubahan iklim. Persiapan yang efektif untuk menghadapi ketidakpastian di masa depan hanya bisa dicapai melalui tindakan kolektif dari pemerintah, individu, pelaku bisnis, masyarakat dan warga negara, serta kesadaran bersama akan keterkaitan antar tujuan tersebut serta berbagai potensi manfaat yang akan didapat. Tim evaluasi menawarkan daftar rekomendasi berikut untuk program bantuan pembangunan di masa depan demi tujuan perbaikan API/PRB di Indonesia. Banyak dari rekomendasi berikut berasal dari tantangan kedepan dan pembelajaran yang diperoleh selama proyek APIK, tapi tidak dimaksudkan untuk menunjukkan kekurangan dalam desain atau pelaksanaan program APIK. Program USAID APIK sama sekali tidak dirancang atau dilengkapi sumber daya untuk memenuhi semua rekomendasi di bawah ini. Tindakan antar pemerintah, keterlibatan masyarakat dan ketangguhan 10. Terus memperkuat integrasi vertikal dan horisontal lembaga Pemerintah Indonesia dan perbaikan perencanaan dan penganggaran API/PRB. Lanjutkan upaya koordinasi yang dimulai oleh APIK dan bantu melembagakan tanggung jawab dalam hal koodinasi lintas sektor pada lembaga Pemerintah Indonesia yang sesuai (mis. Bappeda). Peran ini akan membutuhkan dukungan penuh pakar teknis, seperti ilmuwan universitas, dan keterampilan yang diperkuat dalam perencanaan dan fasilitasi strategis untuk bekerja lintas kementerian. Di tingkat pusat, mulailah dengan dialog tingkat tinggi antara Bappenas, KLHK dan BNPB tentang pendekatan kolaboratif yang dirancang bersama untuk direplikasi dan diperluas dalam lokasi yang ditargetkan (lihat Kotak 2). Lanjutkan juga dengan upaya untuk memasukkan API/PRB kedalam rencana aksi dan anggaran daerah di tingkat masyarakat, namun dalam cara yang lebih terfokus dan berorientasi pada hasil (lihat Rekomendasi 3). USAID dapat mendorong penggunaan dana Pemerintah Indonesia yang lebih optimal untuk API/PRB, termasuk Dana Desa, dan perlu mempertimbangkan caranya sambil mencari cara untuk memanfaatkan dana BPDLH yang diberikan oleh KLHK dan Kementerian Keuangan. Integrasi horisontal harus menyertakan sektor swasta sebagaimana mestinya (lihat Rekomendasi 5 dan 6). 11. Bangun kapasitas Pemerintah Indonesia dan tanggung jawab untuk pelaporan Monitoring & Evaluasi khusus terkait ketangguhan. Membantu Bappenas dengan merancang bersama sebuah Teori Perubahan Adaptasi berskala nasional, Kerangka Kerja Hasil, dan sebuah sistem Monitoring & Evaluasi dengan peran serta dan tanggung jawab yang teridentifikasi secara jelas dari Kementerian/Lembaga Pemerintah Indonesia. Membantu menyusun indikator dan target, baseline, prosedur pengumpulan data dan persyaratan pelaporan yang spesifik. Menghubungkan sistem Monitoring & Evaluasi ini dengan tanggung jawab untuk ¶komunikasi adaptasi· di bawah komitmen Kontribusi yang Ditetapkan secara Nasional yang disempurnakan atau Enhanced USAID.GOV RINGKASAN EKSEKUTIF DAN REKOMENDASI EVALUASI AKHIR KINERJA USAID-APIK | 8 Nationally Determined Contributions (NDCs). Tujuannya adalah untuk meningkatkan transparansi, akuntabilitas, dan investasi publik/swasta secara keseluruhan dalam hal ketangguhan iklim. Membantu KLHK dan BNPB untuk berkolaborasi menyusun definisi baru untuk istilah ¶ketangguhan· yang menggabungkan API dan PRB secara efektif. Definisi yang dipakai saat ini adalah definisi yang tumbuh dari komunitas PRB. Diperlukan definisi baru untuk ketangguhan yang membahas perubahan iklim, dan sistem sosio-ekologis yang harus dipersiapkan untuk menanggapi perubahan iklim. Definisi baru tersebut harus menjadi fokus dalam upaya Monitoring & Evaluasi untuk melacak kedua tujuan API maupun PRB. Berikut ini adalah contoh dari beberapa pertanyaan kunci yang harus dijawab dalam monitoring untuk ketangguhan yang lebih baik. Kotak 1: Berpikir tentang ketangguhan TUJUH PERTANYAAN TENTANG KETANGGUHAN: 1. Ketangguhan apa yang dikelola? 2. Untuk siapa (pemangku kepentingan) ketangguhan ini dikelola? 3. Satuan pengukuran apa yang diterapkan untuk mengukur kinerja pemangku kepentingan? 4. Bagaimana kemungkinan (atau keamanan) dari sasaran pemangku kepentingan (dan pengukurnya) untuk suatu sistem kunci berikut variabelnya, yang dapat membuat sistem tersebut mempertahankan identitasnya? 5. Bencana atau penyebab apa terkait ketangguhan yang menjadi pertimbangan? 6. Bagaimana sifat-sifat ketangguhan (daya tahan, daya bangkit dan ketangguhan) diukur terkait dengan kinerja dalam menghadapi bencana kritis? 7. Manfaat apa yang diharapkan pada saat ini dan di masa mendatang dari tata kelola ketangguhan? Sumber: Grafton, Q.R., 2019. 12. Fokus pada satu atau dua sektor utama dan hasil spesifik yang dapat dicapai dalam sebuah program lima tahun. Program-program masa depan tentang adaptasi iklim harus terus melibatkan berbagai tingkatan pemangku kepentingan, tetapi lebih fokus pada sektor atau sistem utama yang ditentukan berdasarkan prioritas yang diidentifikasi dalam RAN-API atau RPJMN yang baru. Pertimbangkan tujuan terfokus berikut yang dapat memiliki dampak menguntungkan bagi lebih banyak orang: wanatani untuk petani kecil; konservasi dan pengelolayaan sumber daya air; pengelolaan bersama perikanan tradisional (artisanal); mitigasi banjir melalui penerapan infrastruktur abu-abu/hijau berskala DAS; dan konservasi dan pengeolaan hutan bakau. Tujuannya disini adalah untuk mempercepat adaptasi di sektor atau sistem utama, dimana adopsi skala besar dari perubahan perilaku atau praktik yang lebih baik memiliki dampak yang transformatif. Perluas metodologi Kajian Kerentanan Iklim (CVA) untuk menyertakan fokus pada risiko komersial tertentu, dan gunakan informasi tersebut untuk melibatkan sektor industri dan perusahaan untuk merancang solusi kolaboratif. Serangkaian tujuan API/PRB yang lebih terfokus akan memerlukan keterlibatan kementerian terkait (mis, pertanian, perikanan, pekerjaan umum, perencanaan tata ruang, dst.), dengan desain bersama serta kontribusi investasi masing-masing. USAID dapat memberikan bantuan berharga dengan desain teknis, jika bukan pendanaan kolaboratif dan pembangunan yang sesungguhnya, terkait infrastruktur abu-abu/hijau. Solusi berbasis alam menawarkan dampak terbesar per dolar yang dihabiskan untuk meningkatkan ketangguhan iklim. Mengingat bahwa banjir merupakan salah satu permasalahan terbesar di Indonesia, pertimbangkan 9 | RINGKASAN EKSEKUTIF DAN REKOMENDASI EVALUASI AKHIR KINERJA USAID-APIK USAID.GOV sebuah program yang memiliki beragam manfaat yang dirancang seputar pengelolaan banjir berbasis alam.29 13. Gunakan setiap kesempatan untuk merancang langkah-langkah adaptasi yang juga berkontribusi terhadap mitigasi. Komunitas global menyerukan integrasi yang lebih luas dari upaya adaptasi dan mitigasi. Ada banyak ruang untuk berkontribusi pada kedua tujuan dalam hampir semua upaya yang melibatkan restorasi bentang alam atau solusi berbasis alam. Sama halnya dengan pertanian cerdas iklim pada skala bentang alam. Mengintegrasikan adaptasi dan mitigasi dalam lanskap pertanian dapat mencapai banyak manfaat, termasuk ketangguhan pangan, pengentasan kemiskinan dan konservasi keanekaragaman hayati. Tujuan utamanya adalah untuk menggabungkan residu tanaman, pengomposan dan persiapan lahan rendah untuk meningkatkan karbon tanah, meningkatkan kelembaban tanah dan meminimalisir erosi lapisan. Wanatani dan praktik agro￾ekologis lainnya untuk membangun dan melestarikan struktur tanah adalah solusi saling menguntungkan baik untuk adaptasi maupun mitigasi. Namun perlu diingat, bahwa mencapai dan mempertahankan tujuan-tujuan ini akan memerlukan perubahan kebijakan, hukum, dan kelembagaan yang menyertainya. Mobilisasi sektor swasta 14. Dukung masyarakat untuk menyusun dan mendanai kegiatan ketangguhan lokal, termasuk membantu masyarakat untuk terlibat lebih aktif dengan dengan sektor swasta. Kegiatan-kegiatan ini memiliki manfaat nyata dan membantu membangun niat baik dan pengetahuan yang dibutuhkan untuk memulai dukungan dan tindakan yang lebih luas untuk kesiapsiagaan bencana dan ketangguhan iklim di tingkat lokal. Dana CSR tersedia di banyak lokasi dapat memberikan sumber dukungan berharga untuk kegiatan ketangguhan dan kesiapan bencana berskala kecil. Berikan masyarakat (dan perwakilan pemerintah kabupaten dan provinsi) alat untuk mendorong peningkatan interaksi dengan sektor swasta. Hal ini dapat membantu untuk menciptakan dan mempertahankan jenis kegiatan yang telah berhasil dikembangkan oleh APIK, sebagai contoh dengan melibatkan sektor swasta secara proaktif dalam kegiatan Sekolah Lapang Iklim. Sedapat mungkin, rancang pendekatan untuk melokalisasi dan menskalakan upaya dukungan tersebut, sebagai contoh dengan menyelenggarakan lokakarya tingkat regional untuk pengelola Dana Desa. Lokakarya ini bertujuan untuk meningkatkan keterampilan dalam merancang program dan penggalangan dana, serta menggunakan desa adopsi awal untuk menyebarluaskan contoh￾contoh praktik terbaik. 15. Fokus merangsang investasi publik dan swasta. Ada semakin banyak peluang pendanaan iklim di seluruh dunia dan USAID dapat secara strategis memanfaatkan sumber daya tersebut melalui dukungan menciptakan platform, jenis fasilitas yang dapat membantu menghubungkan investor (mis. Green Climate Fund, Adaptation Fund, dst.) dengan proyek bankable yang menggunakan dana Pemerintah Indonesia di semua tingkatan.30 Mitra pelaksana dapat menyediakan bantuan teknis dan membantu mengembangkan ilmu pengetahuan yang diperlukan untuk mendukung alasan 29 Untuk contoh, lihat: Natural and Nature-based Flood Management: A Green Guide. www.worldwildlife.org/publications/natural-and-nature-based-flood-management-a-green-guide 30 See for example: Deploying Adaptation Finance for Maximum Impact.WRI, 2018. https://wriorg.s3.amazonaws.com/s3fs-public/wri-commentary-adaptation-finance.pdf USAID.GOV RINGKASAN EKSEKUTIF DAN REKOMENDASI EVALUASI AKHIR KINERJA USAID-APIK | 10 serta komitmennya. Saat ini lebih banyak perhatian diberikan terhadap dana campuran, atau blended finance (menggabungkan pendanaan publik dan swasta) untuk mendukung aksi iklim secara global. USAID berada di posisi yang tepat untuk membantu model ini untuk Indonesia, terutama mengingat pengalamannya dengan, dan akses ke, Development Credit Authority. Pertimbangkan model Konvergensi yang menggunakan blended finance untuk mendukung perbaikan lanskap terkait dengan energi terbarukan dan mata pencaharian petani kecil.31 16. Tingkatkan perhatian yang diberikan pada solusi berbasis pasar dan Nilai Bersama Perusahaan (Corporate Shared Value). Memiliki alat dan kapasitas untuk menghadapi masa depan yang tidak pasti, membutuhkan input yang terkoordinasi dari semua pihak, tidak terkecuali bisnis dan investor yang merupakan mesin utama pertumbuhan ekonomi. Di sebagian besar negara berkembang, sektor swasta menyumbang 60 persen atau lebih besar dari PDB, 80 persen aliran modal dan 90 persen lapangan kerja. Program di masa depan perlu menyadari kenyataan ini dan berinvestasi secara substansial dalam Keterlibatan Sektor Swasta (PSE) yang efektif, dengan tujuan yang jelas dan hasil yang berorientasi outcome. Gabungkan prinsip dan praktik solusi berbasis pasar dan Nilai Bersama Perusahaan ke dalam operasi. Perhatikan kondisi yang memungkinkan keterlibatan sektor swasta dan investasi yang swa-replikasi, termasuk informasi, pembiayaan, struktur kelembagaan, dan pembangunan kapasitas. Sektor swasta tertentu memiliki kebutuhan data dan informasi iklim spesifik, dan banyak dari mereka akan butuh bantuan dalam mengartikulasikan business-case dan menggabungkan pendekatan bisnis untuk adaptasi ke dalam operasi. Komunikasi dalam bahasa untung dan rugi, pendapatan, dan pangsa pasar sangat diperlukan. Membangun narasi bersama antara sektor publik dan swasta berdasarkan kepentingan dan tanggung jawab bersama sangat diperlukan. Akhirnya, pertimbangkan penggunaan unit pendukung bisnis khusus untuk memimpin pengembangan kemitraan sektor swasta, baik internal terhadap program adaptasi tersebut atau sebagai sumber daya eksternal yang tersedia. Pastikan bahwa upaya PSE menerima pendanaan yang memadai untuk mendukung monitoring, evaluasi, penelitian dan pembelajaran, atau MERL (monitoring, evaluation, research, and learning) yang efektif. Kebutuhan data dan informasi 17. Bantu Indonesia memikirkan ulang kebutuhan data dan informasi secara keseluruhan terkait ketangguhan iklim dan bencana. Proses ini terjadi pada dua tingkatan. Pertama, lanjutkan upaya APIK meningkatkan kualitas dan aksesibilitas informasi iklim dan cuaca dengan mengambil langkah selanjutnya dalam mengkaji kebutuhan data/informasi pengguna akhir di sektor yang ditargetkan. Bantu Pusat Aplikasi Iklim BMKG untuk menyiapkan laporan tentang persyaratan data/informasi untuk pengguna akhir (end-user) tertentu, serta media penyebarannya yang sesuai untuk masing-masing sektor. Media sosial dapat digunakan untuk komunikasi informal, tetapi tidak boleh menggantikan ramalan, peringatan/status siaga, dan informasi lainnya yang tepat waktu, andal, serta informasi lain yang disediakan melalui saluran resmi BMKG. Kedua, bantu Lembaga Penerbangan dan Antariksa Nasional (LAPAN) untuk memperkuat akuisisi, analisis, dan penerapan data penginderaan jauh (remotely sensed data). Data satelit dari berbagai jenis semakin cepat tersedia dalam bentuk Analysis Ready Data (ARD), yang memiliki banyak aplikasi dalam membantu API/PRB. Membantu LAPAN dan 31 Convergence web site: https://www.convergence.finance/about 11 | RINGKASAN EKSEKUTIF DAN REKOMENDASI EVALUASI AKHIR KINERJA USAID-APIK USAID.GOV kementerian teknis terkait untuk mengembangkan produk dan alat teknis untuk membantu berbagai kebutuhan, misalnya, pemantauan tutupan lahan regional, pemantauan kekeringan dan hasil panen regional, dan aplikasi manajemen sumber daya air, termasuk curah hujan yang berasal dari radar satelit dan pengukuran daerah aliran sungai untuk prakiraan banjir. Data satelit dapat memainkan peran penting dalam peramalan, kesiapsiagaan, dan respon bencana. BNPB sedang memulai proyek lima tahun untuk menggabungkan pemrosesan data intelijen satelit (AI) untuk memantau dan memahami tren penggunaan lahan/tutupan lahan di lima DAS untuk prakiraan banjir yang lebih baik. USAID berada di posisi yang tepat untuk membantu LAPAN bermitra dengan SERVIR-Mekong (berlokasi di Asian Disaster Preparedness Center di Bangkok) untuk memanfaatkan kayanya perangkat dan layanan pendukung keputusan (decision support tools and services) yang telah tersedia untuk DAS Mekong Basin. Hal ini serta teknologi digital lainnya menawarkan potensi kuat untuk mengatasi masalah API/PRB pada skala temporal dan spasial yang tansformatif. Solusi skala bentang alam 18. Rancang program di masa depan agar berorientasi pada hasil (outcome) dan transformatif pada skala. Mengingat urgensi tantangan ketangguhan iklim dan bencana, fokuslah pada faktor pendorong risiko dan kerentanan manusia, dan upayakan untuk membalikkan faktor pendorong tersebut. Restorasi lanskap kunci dan fungsi hidrologisnya dapat menghasilkan dividen besar untuk meningkatkan ketangguhan. Sebagai contoh, konservasi dan pengelolaan bakau untuk melindungi hutan bakau yang tersisa di sebagian besar lokasi lebih penting dan transformatif daripada beberapa upaya restorasi mangrove berskala kecil, yang bahkan jika berhasil, masih kalah penting dari kehilangan dan degradasi bakau. Demikian pula, restorasi berskala DAS gemuk atau DAS kurus melalui kombinasi infrastruktur abu-abu dan hijau yang tepat, bersamaan dengan reboisasi dan pengelolaan lahan, harus diberi prioritas tinggi. USAID dapat berperan penting dalam membantu mengklarifikasi prinsip utama, pedoman, ukuran hasil dan justifikasi biaya-manfaat untuk menerapkan pendekatan bentang alam terkait API/PRB. Pemerintah Indonesia memiliki sumber daya dan kapasitas teknis untuk merancang dan mengimplementasikan infrastruktur campuran (abu-abu dan hijau) dan solusi berbasis alam. Perkuat dan lembaga-lembaga akademik negara untuk meningkatkan nilai kapasitas teknis mereka serta peran mereka dalam mengidentifikasi solusi inovatif untuk serangkaian adaptasi sektor tertentu dan kebutuhan pengurangan risiko bencana. Kotak 2: Prinsip desain untuk perluasan (scale-up) Penerapan prinsip perluasan (scale-up) harus dilakukan dalam kegiatan adaptasi perubahan iklim di Indonesia. APIK telah mencapai keberhasilan yang mengesankan pada serangkaian kegiatan uji coba skala kecil; namun, jumlah penerima manfaat masih tergolong sedikit. Untuk meningkatkan dampak upaya di masa depan, program Adaptasi Perubahan Iklim harus fokus pada penerapan prinsip peningkatan secara metodis sejak awal. Di antara faktor-faktor yang telah terbukti meningkatkan prospek untuk program yang berhasil diskalakan: 1) Uji coba harus didefinisikan dan didokumentasikan secara jelas; 2) Pengukuran yang ketat untuk memvalidasi efektifitas sebuah pendekatan harus diberi perhatian khusus; 3) Anggaran dan biaya unit harus dipertimbangkan untuk memastikan upaya yang ada dapat diadopsi oleh lembaga rekanan; 4) Kompleksitas harus ditekan sebisa mungkin, karena program yang sederhana dan mudah dimengerti memiliki prospek lebih tinggi untuk meningkatkan kesuksesan; dan 5) Mitra pendukung perlu diidentifikasi sejak dini dan upaya advokasi harus dilakukan untuk mendapatkan dukungan mereka.