This publication was produced at the request of the United States Agency for International Development. It was prepared independently by an independent evaluation consultant, Mildred Guerrero Echegaray, hired by Miyamoto International, Inc. FINAL EVALUATION PREPARE COSTA RICA APRIL 2023 FINAL EVALUATION OF PREPARE COSTA RICA April 2023 Award Number: 720FDA20GR00052 DISCLAIMER: The author’s views expressed in this publication do not necessarily reflect the views of the United States Agency for International Development of the United States Government. TABLE OF CONTENTS ACRONYMS.............................................................................................................................2 EXECUTIVE SUMMARY.........................................................................................................3 EVALUATION PURPOSE AND EVALUATION QUESTIONS .........................................1 EVALUATION PURPOSE ........................................................................................................................................1 EVALUATION QUESTIONS...................................................................................................................................1 BACKGROUND.......................................................................................................................3 EVALUATION METHODS AND LIMITATIONS................................................................5 LIMITATIONS .............................................................................................................................................................6 FINDINGS, CONCLUSIONS, AND RECOMMENDATIONS ............................................7 FINDINGS....................................................................................................................................................................7 RELEVANCE.......................................................................................................................................................7 EFFICIENCY........................................................................................................................................................8 PARTICIPATION ............................................................................................................................................10 EFFECTIVENESS ..............................................................................................................................................12 SUSTAINABILITY............................................................................................................................................17 IMPACT .............................................................................................................................................................19 CONCLUSIONS ......................................................................................................................................................20 RELEVANCE.....................................................................................................................................................20 EFFICIENCY......................................................................................................................................................20 PARTICIPATION ............................................................................................................................................20 EFFECTIVENESS ..............................................................................................................................................21 SUSTAINABILITY............................................................................................................................................21 IMPACT .............................................................................................................................................................22 LESSONS LEARNED ...............................................................................................................................................22 RECOMMENDATIONS..........................................................................................................................................23 FOR DRM SYSTEM AUTHORITIES: ..........................................................................................................23 FOR MIYAMOTO: ..........................................................................................................................................23 FOR USAID/BHA: ...........................................................................................................................................23 ANNEXES.................................................................................................................................1 ANEXX I: SOURCES OF INFORMATION.........................................................................................................1 ANEXX II: INFORMATION GATHERING TOOLS .........................................................................................2 2 | FINAL EVALUATION PREPARE COSTA RICA ACRONYMS BHA Bureau of Humanitarian Assistance CFIA Colegio Federado de Ingenieros y Arquitectos de Costa Rica (Spanish acronym for Federated School of Engineers and Architects of Costa Rica) CNE Comite Nacional de Prevencion de Riesgos y Atención de Emergencias (Spanish acronym for National Committee for Risk Prevention and Emergency Care) DRM Disaster Risk Management IMAS Instituto Misto de Ayuda Social (Spanish acronym for Mixed Institute of Social Aid) MI Miyamoto International MINSA Ministerio de Salud (Spanish acronym for Health Ministry) MIVAH Ministerio de Vivienda y Asentamientos Humanos (Spanish acronym for Ministry of Housing and Human Settlements) NRMS National Risk Management System OFDA Office of Foreign Disaster Assistance DAC/OECD Development Assistance Committee/Organization for Economic Cooperation and Development PDNA Post Disaster Needs Assessment PDA Preliminary Damage Assessment PREPARE Strengthening Deployable Damage Assessment Capacity in Costa Rica RDA Rapid Damage Assessment SIDEVI Sistema de Información para la Atención y Seguimiento de las Emergencias en Vivienda (Spanish acronym for Information System for Attention and Monitoring of Housing Emergencies) TWG Technical Working Group UCR University of Costa Rica USAID U.S. Agency for International Development FINAL EVALUATION PREPARE COSTA RICA | 3 EXECUTIVE SUMMARY The final external evaluation of the PREPARE Costa Rica program, Strengthening the Deployable Capacity for Damage Assessments in Costa Rica, seeks to assess the progress achieved based on the expected results and the objectives foreseen in its planning, and to identify and document the lessons learned, the unplanned results, the opportunities for improvement and its sustainability measures. The intervention evaluated in this study, implemented between 2020 and 2022, is the culmination of a process that began in 2017, during which the foundation was laid that allowed the results and objectives planned in this stage to be achieved, with a high level of effectiveness. Miyamoto's experience in its years of previous interventions in Costa Rica led to the establishment of sufficient knowledge of the conditions of the institutional context to remain aligned with the capacities, priorities, and legal mandates of the emergency response system and its actors, in order to respond exactly to the weaknesses and gaps that limited the effectiveness and efficiency of its damage assessment capacity, which was identified by all the parties that participated in the previous assessment of needs, as one of the aspects that most limited the effectiveness of disaster response in the country. The overall planned objective was to strengthen multi-hazard disaster resilience in Costa Rica through improved disaster response coordination, planning, and deployable damage assessment capacity in order to effectively inform policies and post-disaster decision-making, which will be achieved through two strategies, formulated as specific objectives: the development of operational protocols to coordinate the flow of disaster information and facilitate the implementation of damage assessment activities, and the updating of Preliminary Damage Assessment (PDA) tools and related data/information management system. In this way, the main changes achieved were: the unification of post-disaster damage assessment tools and procedures; the increase in the deployment capacity to carry out these assessments by incorporating resources that were not previously mapped and articulating the different actors in a logic sequence, and eliminating obstacles and duplications that made it lose efficiency and effectiveness; the strengthening of the capacities of DRM system institutions; and the collaborative and complementary work between institutions that previously acted in an uncoordinated manner. Among the actions that most contributed to the achievement of these results, was fundamental the work of the Technical Working Group, a participatory space of a transdisciplinary and inter￾institutional nature, made up of representatives of public and private entities involved in disaster risk management, local governments and by program management. In its formation, special care was taken in the selection of the representatives of the institutions and organizations among people with extensive experience in emergency care, in order to take advantage of and enhance this local knowledge. This group, with a horizontal and collaborative operation, contributed to the program design, and reviewed and validated all the tools, systems and protocols developed, ensuring that the requirements and visions of the diversity of participating actors were incorporated into them, and generating synergies and strategic associations that enhance the capabilities of each of them and at the same time of the entire system. The successful leadership by Miyamoto managed to overcome the difficulties of a participatory process with such heterogeneous actors, achieving their active and committed participation in the production of high-quality technical tools that meet the expectations of all institutions. Other activities of the program very valued for their contribution to the achievement of the objectives sought, were the trip to California to exchange experiences between the state, the United States, and Costa Rica, where the participants were able to learn directly about the experience of various emergency management centers at different levels of government and academic institutions, and the simulation exercise of an earthquake, carried out as a test of the operation of the new protocols and tools developed under the program, which made it possible to identify some gaps that still need to be corrected. 4 | FINAL EVALUATION PREPARE COSTA RICA It is worth noting that this program began during the COVID 19 pandemic, when several of the key participating institutions were focused on attending this emergency. This produced a notable delay in the start of the intervention, which influenced some of its final products, such as the systematization of the simulation exercise, to be implemented at the end of the execution period, with which the socialization and monitoring of its results were affected. However, the pandemic also accelerated the migration process towards digital procedures that the public administration institutions had already started, helping the officials involved in their practice to accept the digital tools and systems generated within the framework of the program. The sustainability of the changes and results obtained is high, due to the strong appropriation and empowerment generated in the interested parties through their participation both in the formulation of the intervention and in the co-creation of its main products, such as the damage evaluation forms, which have been formalized by the corresponding authorities, ensuring their use in emergency response at the national level. However, another important product for the continuity and sustainability to the results obtained, was the Critical Path Analysis which establishes a map of the actions that DRM institutions must follow in order to continue strengthening their damage assessment capacity, prepared through an external consultancy, but it did not have the same appropriation by these institutions, so it requires an additional effort of socialization among the authorities involved. Some of the economic, social, and political impacts obtained by the program are: • More efficient management of aid resources for the population affected by disasters, due to the elimination of duplication in the assessment of damages, and the generation of reliable information for decision-making. • Elimination of the re-victimization of the population due to the multiplication of damage assessment processes. • Increased effectiveness of humanitarian assistance, by reducing aid management periods. • Increased transparency in the management of humanitarian aid leading to the improved confidence of the population in state institutions. • Improved capacity of public institutions to fulfill their social responsibilities. • A strengthened political culture of systemic work. • The implementation of public policies for emergency care in a way that is more in line with the reality of the country. The main recommendations to improve the results of the program, give the program continuity, and make it replicable in other contexts, are the following: 1. Implement an earthquake drill as a practical and face-to-face test of the real operational capabilities and response times of the institutions involved for the deployment of damage assessments in the field, and to verify some aspects of the designed tools and protocols that could not be evaluated through the simulation carried out in offices. 2. Include in the DRM system universities that have careers related to damage assessments in the built environment, and jointly implement ongoing training plans for advanced students, private sector professionals, and institution officials of the system, to ensure the availability of sufficient and trained damage assessment personnel at the time of a disaster. 3. Design and implement a strategy for disseminating the results of the Critical Path Analysis and other products of interest, which includes abbreviated versions and graphical displays of the reports for DRM authorities, presentations of results that may be the subject of discussion in professional and academic forums, and pieces of information about their main findings, in non-technical language, to be disseminated by the media to broader audiences. 4. Start any initiative/program by financing an identification and formulation phase in which all the interested parties participate, to incorporate their conditions, needs and priorities in the design of the actions to be implemented. FINAL EVALUATION PREPARE COSTA RICA | 5 5. Work on the production of communication pieces to raise awareness about the importance of disaster prevention and preparation, which can be used in catastrophic event scenarios, when public opinion is more sensitive and receptive to this type of message. FINAL EVALUATION PREPARE COSTA RICA | 1 EVALUATION PURPOSE AND EVALUATION QUESTIONS EVALUATION PURPOSE The main purpose of this final external evaluation is to assess the progress achieved in the implementation of the PREPARE Costa Rica program (Strengthening the Deployable Capacity for Damage Assessments in Costa Rica), based on the expected results and the objectives foreseen in its planning. It is also expected that the evaluation process will identify impacts, lessons learned, and strategic and operational recommendations that will contribute to the sustainability of the results achieved. The specific objectives of the evaluation, as specified in the Terms of Reference, are as follows: • Evaluate the effectiveness and impact of program activities according to established objectives and indicators. • Identify and document lessons learned from the various activities and processes undertaken, as well as the impact of the program and its unplanned results, both positive and opportunities for improvement, and identify measures for its sustainability. EVALUATION QUESTIONS The evaluation questions that are answered in this study are: • To what extent does the program respond to the priorities and mandates of the institutions and organizations that are part of the Costa Rican DRM system? • Is there alignment of the program's objectives and activities with the strategic priorities and mandates of the institutions that are part of the DRM system? • To what extent were program activities adapted to the baseline findings? • To what extent has the Covid-19 crisis had an impact on the results achieved, and what adaptation strategies were adopted? • Was the distribution of human and material resources of the program sufficient and adequate to achieve its objectives? • To what extent has the program taken advantage of the internal experience and knowledge of the country's organizations and institutions? • How has MI's role as a catalyst in the articulation of the different approaches of the participating organizations work? • To what extent has the involvement of national stakeholders shaped and helped develop the program's outputs and outcomes? • To what extent have outputs and outcomes included participation and accountability to the affected population? • To what extent were stakeholders' priorities and needs incorporated into the design of program activities? • To what extent have the overall program goal and objectives been achieved? • How has damage assessment deployment capacity increased to effectively inform disaster decision making? • Has the coordination and communication between organizations, institutions and agencies increased as a result of their participation in the project? • Have the tools developed under the program been put into practice by stakeholders? • What program activities have been most effective in achieving the stated objectives? • What have been the main challenges in achieving the goals and objectives of the program? • How did the specific intended and unintended changes positively or negatively affect program activities? • What lessons can be learned from PREPARE Costa Rica that can be replicated in other cases in the country and the region? • To what extent have program activities contributed to sustainable changes or results, and what factors have contributed to achieving sustainability? 2 | FINAL EVALUATION PREPARE COSTA RICA • Have mechanisms been established to maintain public-private collaboration beyond the life of the program? • To what extent has the program contributed to improving the self-sufficiency of participating organizations and institutions? • How was the exit strategy and sustainability plan established? • To what extent have the changes achieved caused or may cause direct or indirect social, economic and/or political effects in the intervention area? FINAL EVALUATION PREPARE COSTA RICA | 3 BACKGROUND The combination of high seismicity and rapid and disorderly urban expansion with the lack of proactive territorial planning has increased the country's vulnerability to seismic events in recent decades, despite scientific advances in monitoring seismic movements. The inability of municipalities to absorb new inhabitants causes growing proportions of the urban population to settle in high-risk areas throughout the country's central valley, which is subject to a series of risks in addition to seismic, such as cyclones that transit annually through the Caribbean Sea, producing recurring social and economic impacts. The mobilization of a timely and effective response to the impacts of these events requires a well￾coordinated evaluation and gathering of situational information system, in which the planning, organization, and division of responsibilities between the components of the system is fundamental. In the previous phases of the PREPARE program implemented in Costa Rica between 2015 and 2019, important advances were made, which were focused on three main results: Understanding seismic risk and strengthening the political will to face it through the preparation of a seismic risk study of the built environment of the city of San José (which quantifies human and material losses and the volume of debris to be managed in probable earthquake scenarios); Developing a debris management strategy aligned with the mandates, roles, and responsibilities of the institutions; and Developing a rapid assessment system for multi-origin damage to homes. Based on these results, the current phase of the program was formulated, which reviews the entire disaster response infrastructure, in order to develop and institutionalize plans and procedures that establish a clear division of roles and responsibilities and that are integrated into the structure of the agencies that make up the system, to improve its operation and ensure the success of damage assessment missions. The current phase of the PREPARE Costa Rica program was implemented between September 2020 and March 31, 2023, and was financed by USAID/BHA, under grant number 720FDA20GR00252. The program aimed to strengthen multi-hazard disaster resilience in Costa Rica through improved response coordination, planning, and deployable damage assessment capacity in order to effectively inform policies and post-disaster decision-making. The direct beneficiaries of the intervention were a group of authorities and decision-makers estimated at 50 people; representatives of national and municipal disaster risk management organizations, and key organizations and institutions from the public, academic, and private sectors, including: the National Emergency Commission (CNE), the Ministry of Housing and Human Settlements (MIVAH), the Ministry of Health (MINSA), the Mixed Institute of Social Aid (IMAS), the Federated College of Engineers and Architects (CFIA) , and the Municipalities of San José, Alajuela and Heredia. By working at the level of national public policies, the program sought to have an impact on the entire population of the country, and its indirect beneficiaries were estimated at approximately 2.5 million people, which represent the urban population estimated to be affected by a major event, such as an earthquake. The program was based on the principles and strategic approaches of Working in inter-institutional groups, which seeks the inclusion of local technical experts to incorporate their knowledge of the context and their understanding of local needs and capacities, Integration and Co-design, to generate synergies between the components of the program that enhance its impact, Strengthening the autonomy of the participating institutions to give sustainability to the changes achieved, and Decision￾making based on data. To contribute to the achievement of its general objective, the program was structured around two specific objectives, framed within the Policy and Planning Subsector, which were: SO1: Develop operational protocols, through a critical path analysis and planning exercises, to coordinate the flow of information and facilitate the implementation of damage assessment activities. 4 | FINAL EVALUATION PREPARE COSTA RICA SO2: Strengthen intelligence and coordination mechanisms for disaster response, by updating preliminary damage assessment tools and related information management systems. The main activities carried out within the framework of these objectives were: 1. A critical path analysis, response planning, and response infrastructure timeline mapping, including a Technical Study Tour to California, to share experiences and learnings from both country’s disaster emergency response systems. 2. The integration and rationalization of disaster information and intelligence in coordination with the needs and mandates of MINSA and MIVAH, through the creation of a Technical Working Group to develop common tools for both agencies. 3. The development of field operational protocols to facilitate the implementation of damage assessments, which make it possible to overcome the deficiencies and weaknesses identified in past disaster response scenarios. 4. An earthquake simulation as a practical exercise to validate the operational protocols designed. 5. The updating of the Preliminary Damage Assessment form. 6. The development of an App to manage the information gathered from the Preliminary Damage Assessments. 7. The preparation of training materials for the use of the tools developed. FINAL EVALUATION PREPARE COSTA RICA | 5 EVALUATION METHODS AND LIMITATIONS The theoretical-methodological framework of the evaluation was guided by the criteria evaluation model 1, complemented by two evaluative approaches. On the one hand, the systemic evaluation, which describes the way in which the design and implementation of any intervention or system is translated into results through a chain of causal relationships. And on the other hand, the democratic evaluation2 which is a participatory process centered on organizations and is highly in tune with the transversal intervention approaches. From this triple approach and based on the secondary documentation, it was possible to define the logical skeleton of the project, the assumptions that composed it, the factors that influenced it, and its sequential logic of cause-effect. The evaluation questions were the backbone of the final evaluation because they delimited the dimensions to be investigated and evaluated. These were taken from the information provided in the terms of reference, and were complemented with additional questions, arising from the documentary review and the needs and expectations of the program's technical team, collected through consultations. With them, the evaluation matrix was elaborated, relating each question with the sources of information to answer them, and with the methods of collection and analysis of that information. The evaluation was based on a mixed, qualitative and quantitative methodology, using, for the collection of information, social science techniques such as semi-structured interviews, and documentary review and analysis of secondary sources. The phase of collection and analysis of information from primary and secondary sources was carried out between the months of January and February 2023. Among the program documents reviewed in this evaluation were the original technical proposal, the baseline report, the semi-annual performance reports, the earthquake simulation systematization report, the technical products produced, such as the critical path study, action protocols and post-disaster damage assessment forms and product use manuals, written agreements with partners and local authorities, among others. Additional documents included the PREPARE II final evaluation and the lessons learned report. The evaluator, in conjunction with the Miyamoto representative, identified a list of program participants to be part of the evaluation process. From these individuals, a qualitative sample of 11 people, representatives of the project's technical implementation team which included public and private organizations that are members of the national DRM system and participating municipalities, were sent requests for interviews. It was only possible to carry out 9 of the 11 interviews requested. The information obtained from the analysis of the secondary sources that were made available to the evaluation team was triangulated with the verbalizations issued by the key informants, who had a committed participation and thus provided high-quality information. This crossing of both logics (discourses with facts) led to the identification of gaps in the discourses issued in terms of what "should be". Both qualitative information and quantifiable data, although not statistically representative, were extracted from the set of actors and documentary sources consulted. At a qualitative level, the analytical process was carried out by contrasting the descriptive narratives of the informants with secondary information capable of ordering milestones, stages and actors, and by superimposing this descriptive story with intersubjective verbalizations, coming from the (explicit or underlying) evaluations of the people interviewed. In addition, the descriptive or content analysis of the numerical data obtained from secondary sources of the project (monitoring and evaluation), allowed for the understanding of the final state of 1 The evaluation criteria of the Development Assistance Committee of the Organization for Economic Cooperation and Development were agreed in 1991. This document contains the five evaluation criteria that have been used during the last 25 years: relevance, effectiveness, efficiency, impact and sustainability. These criteria have serve to standardize the evaluation practice, contributing to learning and improvement in international cooperation for sustainable development. 2 Evaluative model created by Barry Mac Donald (1967) or "democratic approach", that fosters the informed participation and responsability of the differenet audiences, by democratizing the use of knowledge. 6 | FINAL EVALUATION PREPARE COSTA RICA achievement of the indicators of the project, its coverage and scope. This analysis makes it possible to establish causal hypotheses of an introductory nature, whose value and interpretive sense acquire the dimension of an evaluative finding in its triangulation with qualitative methods. LIMITATIONS Although the evaluation had the receptivity and collaboration of an important group of representatives of the organizations and institutions that participated in the program, it was not possible to contact the person who acted as the program manager from its beginning through October 2022. This absence constitutes a significant limitation for the understanding of the entire process implemented, since it is the person who had the direct and close knowledge of the processes launched and the reasons for the achievements or failures of the initiative and leaves some gaps in the reconstruction of these processes that have not been able to be filled from the other sources of information available. Another limitation was the little utility of the indicators of the program to measure its advances towards its specific and general objectives. While one of them, indicator number 3 “Availability of local and national risk evaluations, data on threats and information on vulnerability”, refers to products already prepared in the previous phases of the program and not in this phase, indicators number 1 and number 2 remain at the level of activities and products, leaving out some of the most important activities, and do not properly evaluate the level of results, nor the measure in which the program advanced towards the achievement of its objectives. However, despite these limitations, it was possible to collect a wide variety of informative supplies, with which a solid base was built to check the identified findings and elaborate a complete and well￾founded evaluation report that, in line with the democratic evaluation approach, widely uses the words of the actors consulted to illustrate the value judgments issued. FINAL EVALUATION PREPARE COSTA RICA | 7 FINDINGS, CONCLUSIONS, AND RECOMMENDATIONS FINDINGS RELEVANCE The evaluation of the relevance assesses the adaptation of the objectives and actions of the program to the context in which it was carried out. Through the evaluation, the quality of the diagnosis that supports the program was studied, judging its correspondence with the needs observed in the beneficiary population3. In assessing the relevance, the needs analysis presented in the technical proposal to USAID/OFDA4 in 2019 was considered to answer to following evaluation question: To what extent does the program respond to the priorities and mandates of the institutions and organizations that are part of the DRM system in Costa Rica? The PREPARE Costa Rica program, arose as a continuation of the previous phases of PREPARE implemented in the country, to operationalize and give sustainability to the results achieved, at the request of the CNE, MIVAH and MINSA as key actors of the DRM system, who participated in the identification and formulation of its objectives, actions and strategies. In strict alignment with the legal order that governs the operation of the system, the program proposed to analyze, with the participation of its protagonists, the complex network of actors and activities involved in the management of disaster emergencies, to identify and solve the gaps that weaken its ability to effectively implement the damage assessments necessary to provide timely care to the affected population. "What was going on here? That the Ministry of Housing had its form, the emergency commissions had their form, the local committee had its form, the firefighters, the Red Cross... But when you put all that together you didn't get anything, you couldn't conclude anything because each form collected different information, and then you couldn't make informed decisions.” According to Law Nª 84885, once a state of emergency has been declared in the country due to a disaster, the institutions that participate in the DRM system, with the CNE at the head, must carry out the pertinent damage assessments and present their results to the central government within a maximum period of two months. However, the lack of a clear distribution of roles and responsibilities among the actors in the system and of tools and mechanisms common to all, generated gaps and duplication of efforts that wasted available resources, weakened the effectiveness of aid and hindered the proper functioning of a systematic response. The Baseline study carried out at the beginning of the intervention identified the needs and priorities of all interested parties to move towards the planned objective, and which were addressed by the different activities of the program, such as the: • Need to standardize preliminary damage assessment tools • Need for an automated information storage and management system • Need to improve inter-institutional coordination in post-disaster field operations It is precisely on these identified needs that the PREPARE Costa Rica program focused its efforts, planning activities and products to address each of them, with the participation of all concerned institutions and public and private organizations. And it did so at a historic moment in which the budgetary restrictions that all these organizations face make external aid necessary to carry out 3 OECD DAC Assessment Management Manual. 4 USAID/OFDA PREPARE Costa Rica, April 2020: Technical Proposal. Strengthening Deployable Damage Assessment Capacity in Costa Rica. 5 National Law of Emergencies and Risk Protection in Costa Rica. MIYAMOTO INFORMANT 8 | FINAL EVALUATION PREPARE COSTA RICA MINSALUD INFORMANT initiatives like this one, which are among their mandates and functions for ordinary times, but that attention to contingencies of extraordinary situations does not allow prioritizing these needs, despite its great impact and strategic value. In this way, the design of the program fully responds to the most urgent needs identified by the DRM actors to unblock the operation of the system and improve its standards and results, making it possible to comply with its legal mandate, and taking advantage of the interest and willingness expressed by government institutions at the national and municipal levels to appropriate the results and give them continuity and operability. “PREPARE fits precisely within the areas of Preparedness and Response and Education and Knowledge Generation of the national risk management policy. It is how we do all these studies, how we equip ourselves, how we are trained, how we share other experiences.” EFFICIENCY The analysis of the efficiency of development cooperation projects and actions refers to the study and assessment of the results achieved in comparison with the resources used and of the compliance with the scheduled workplan of activities, and answers the following questions: Was the distribution of human and material resources of the program sufficient and adequate to achieve its objectives? To what extent has the program taken advantage of the experience and internal knowledge of the country's organizations and institutions? The PREPARE Costa Rica program was able to take full advantage of the experience of local stakeholders, relying on their knowledge of the shortcomings and needs within their territorial and institutional context to identify gaps and overlaps in the DRM system, and to collaboratively build protocols and tools to overcome these irregularities and improve the effectiveness of disaster response actions. To do this, program management carefully identified the participating institutions, so as not to leave any of the nodes that make up the system out of the process. Program management then chose the representatives of these institutions that were going to make up the technical working group based on their years of experience and knowledge of the operational mechanisms with direct experience in emergency management, so that all this wealth of knowledge could be used in the development of protocols, tools and systems adapted to their operational contexts. "The participation of all the parties that were involved was fully used in the development of the products, because none of us were specialists in everything or had all the knowledge, but we all had knowledge to contribute, and effectively, everything was consolidated and given a national vision.” The financial resources available for the program were used very efficiently. One of the most expensive planned actions, which was the trip to California to exchange experiences and learn first￾hand about the operating strategies of the risk prevention and emergency response organizations in the state, was greatly valued by all the people consulted. The experience served as a source of knowledge and innovation and allowed attendees to compare local systems with international standards for prevention and emergency care. There was also enough flexibility to adapt the budget to the specific needs of the partners, which arose in the process. This was the case of the funds destined to develop the application for the preliminary evaluation of damages, which were redirected, at the request of the CNE, to extend the MIVAH INFORMANT FINAL EVALUATION PREPARE COSTA RICA | 9 license of the Arc GIS program, while the PDA computer experts themselves assumed the task of digitizing the PDA tool. 6 Regarding the human resources destined to manage the program, the person who managed it from the beginning until almost the end, had the necessary characteristics and abilities to successfully manage a complex participatory process, made up of very different individuals, both in their academic and professional backgrounds, that came from institutions between which there were even some disagreements. “At the beginning, it was a little difficult to agree among all the institutions because each one has its character and we don't like to expose how badly we do (…) The MINSA colleague was the one who began to let go, to pass on information, to speak frankly, and a beautiful group of trust was created, which was not that we were going to steal information, but rather that we wanted to release a unique product. We set that goal, but the person who helped us set that goal was Diana Ubico.” The project manager knew how to articulate all the different visions and languages to make them coincide in a common purpose that managed to overcome their differences. Miyamoto International, as the managing organization, also efficiently used the strength that its transnational character gives it, taking advantage of the wide variety of specialties and experiences that it counts on (among its human resources), to advise the work group, integrating a necessary global perspective within the systems and products produced. The collaboration and horizontal exchange between the technical teams of the PREPARE program in the different countries of the region, which materialized in the Lessons Learned Workshop held in Costa Rica in April 20227, also allowed for the sharing between them of the methodologies and strategies that had produced the best results. The management of the entire participatory process was very efficient and made the most of the scarce time of the representatives of the partner institutions. Meetings were never more than two hours, during which topics were presented and discussed, and the drafts of the products were brought in already prepared for their collective validation. This allowed the participating representatives to reconcile their attendance at these meetings with the rest of their obligations and maintain participation throughout the two years of program implementation, despite a large number of meetings over that time, as shown in the table below. Virtuality as a work modality, to which the public administration was already migrating, consolidated with the effects of the pandemic, was the appropriate channel that made possible the intensity of this interaction. Table 1. Number of Discussion Meetings and Number of Participants 8 nº of participants nº of meetings in 2021 nº of meetings in 2022 Total meetings 1 - 4 15 9 24 5 - 9 23 4 27 ≥ 10 7 3 10 Total 45 16 61 The table shows that most of the work meetings (83%) had less than ten participants. This is consistent with the opinion expressed by some of the informants, that small work groups are more efficient than large ones in moving towards the objectives sought, since they promote a deeper exchange of ideas and information between the people present. 6 USAID/BHA PREPARE Costa Rica: Semi-Annual Performance Report, october 2021-march 2022 7 Miyamoto International, April 2022: LESSONS LEARNED PREPARE and EBRP Program in Latin America and the Caribbean. 8 Own elaboration based on: USAID/BHA PREPARE Costa Rica: Participation Tracking Tool MUNICIPALITY OF ALAJUELA INFORMANT 10 | FINAL EVALUATION PREPARE COSTA RICA The start of the program in 2020 was influenced by the COVID 19 emergency given that several of the actors directly involved in the program, such as the Ministry of Health, the National Emergency Commission, and local governments were fully focused on the response to the pandemic. This delayed the start of the scheduled activities, due to the fact that the representatives that these institutions had assigned to participate in the PREPARE Costa Rica program were dedicating all their time to issues related to the pandemic. This delay, coupled with the slowness of the participatory processes, which perhaps was not adequately calibrated in the initial workplan, meant that the scheduled period of two years was insufficient to adequately implement all the planned actions. An example of this is the earthquake simulation which was carried out at the end of the program without enough time to socialize and discuss the results among the participants, as would have been preferred, given its importance as an activity and its nature as a pilot test. PARTICIPATION The analysis of participation focuses on determining the actors that have were involved in the different stages of planning and evaluation, assessing their impact on decision-making, and answering the following questions: To what extent has the participation of national stakeholders determined and helped to develop the products and results of the program? To what extent were stakeholder priorities and needs incorporated into the design of program actions? To carry out the work of the program, two spaces for interaction with local partners were formed. A Steering Committee in charge of maintaining the strategic orientation of the program, made up of project management and high-level officials from the CNE, and a Technical Working Group, made up of representatives of all interested sectors, from the CFIA, MINSA, the MIVAH, the IMAS, the CNE, the Municipal Emergency Committees of the mayors and municipalities and the donor. This technical group contributed to the design, review, and endorsement of all the documents and tools created. To form the group, all the agencies and departments that have some kind of involvement or responsibility within the DRM system were mapped out, and then within these agencies, individuals with experience in the institutions and who knew their operational mechanisms well and had experience in emergency management were identified, with the hopes that they could contribute their knowledge to the program’s processes and products. Despite the fact that a strong attempt was made to seek gender equality when forming the TWG, men were the majority among the representative participants, which is representative of the reality of the context and of local institutions. The need for the individuals who made up the TWG to have a specific background did not make it possible to push for the participation of more women in it. In order to achieve the planned results, it was very important to work with representatives of local governments, the first to attend to the population in their territories in cases of emergency. Likewise, it was also important to include the Federated College of Engineers and Architects, which by law forms part of the DRM system but it is not usually taken into account in these initiatives, despite the fact that it has a large proportion of the professionals who are called upon to participate in damage assessments after a disaster and exercises other roles of great responsibility within the system. The participation of the Mixed Institute of Social Aid (IMAS) also helped highlight the human resources that this institution has, both in the social and technical fields, which were not mapped in the DRM system but are an important asset for emergency care. This technical team was formed as a multidisciplinary and horizontal work group in which it was ensured that there was no leadership from any of the interested parties, so that the concerns, needs, opinions and experiences of all could be collected to enrich the process and the results. Miyamoto had the role of mediator, and of providing follow-up, technical support, and accountability. Despite the difficulty involved in sharing very different visions and language, it was possible to establish a FINAL EVALUATION PREPARE COSTA RICA | 11 collaborative and productive work dynamic due to the close, empathetic, and assertive leadership of the program manager, who knew how to generate trust and enthusiasm and found the precise balance between democracy and productivity to allow the fruitful participation of all in the development of the planned results: “We ended up developing everything, with the contribution of the specialized knowledge that the other colleagues had. Luckily, they called us and we have actively participated in the whole process, and we became part of the team, even though we were computer engineers. We work really well, and Diana was a great coordinator in that sense.” Like any participatory process, this one was slow and difficult. Initially, certain rivalries arose between institutions whose powers were not clearly defined, such as MINSA and MIVAH, but these were overcome, and the objective of the common good prevailed over private interests and each agency shared with the others the information that it handled which was vital for fulfilling the objectives sought. The frank and open discussions between representatives of the technical group were conducive to achieving the complementarity of knowledge in the design of the damage assessment instruments that took into account the different dimensions of the material, psychological and social impact that a catastrophic phenomenon can have on a population and its territory. The meetings for discussion, presentation and validation of products had strong participation of representatives of all the organizations involved. A number of these participants attended only very few meetings, in a more informative than cooperative manner, but there was also a significant core of people who remained present throughout the entire program, as illustrated in the following table: Table 2. Frequency of Participation in Discussion Meetings 9 Number of meetings attended 1 2-4 5-9 10-14 15-19 ≥20 Total nº of participants nº of people 32 11 8 4 5 4 64 It is important to note that the TWG experienced several changes in representatives of the organizations, which resulted in setbacks from having to explain to the new participants the context and differences of visions that had been reconciled, which interrupted the work dynamics and delayed progress. This situation was also evidenced in the exchange trip to California, to which some institutions did not send the people who had participated in the entire program, and instead sent representatives who did not know the contents discussed or the work dynamics. As the technical group was made up of a single representative from each institution and organization, the workload on that person was very heavy and difficult to reconcile with other obligations, and these substitutions or occasional absences produced a loss of generated knowledge and a slowed down the learning curve, both for that institution in particular and for the group as a whole. But, despite this, the group was able to continue working in a participatory and productive manner until the set goals were achieved, and the group also overcame the rearrangement caused by the departure of the program manager near the end of the program, and her replacement by another person who did not closely know the background and details of the group’s experience. “There was a very active participation. All the institutions were very responsible and committed, despite the Covid situations, and this allowed them to develop high-quality products that met the expectations of all the institutions.” 9 Own elaboration based on: USAID/PREPARE Costa Rica: Participation Tracking Tool MIVAH INFORMANT IMAS INFORMANT 12 | FINAL EVALUATION PREPARE COSTA RICA EFFECTIVENESS The evaluation of effectiveness tries to measure and assess the degree of achievement of the initially planned objectives, that is, it seeks to judge the intervention based on its orientation to results, also taking into account the results and effects that are not foreseen but that emerged from the processes launched. In the analysis of this criterion, the components of the program that contributed most effectively to the achievement of the planned results and objectives were assessed, as well as those that worked the least, or even hindered progress, and that constitute areas of improvement for future interventions. Progress towards formulated objectives In this section the following evaluation questions will be answered: To what extent have the goal and general objectives of the program been achieved? Has the coordination and communication between organisms, institutions and agencies increased as a result of the participation in the project? How has damage assessment deployment capacity increased to effectively inform disaster decision-making? Have the tools developed under the program been put into practice by stakeholders? The consultations with key informants and the documentary review carried out in this evaluation allowed us to conclude that the program fulfilled, to a great extent, its specific planned objectives, and contributed significantly to the achievement of the general objective of strengthen multi-hazard disaster resilience in Costa Rica through improved disaster response coordination, planning, and deployable damage assessment capacity in order to effectively inform policies and post-disaster decision-making. These achievements, however, cannot be entirely attributed to this phase of intervention, but rather are the result of the progress made in the implementation of all stages of the PREPARE program in Costa Rica. Although the two specific objectives formulated are reiterative, since both refer to improving coordination in the management of information for damage assessments, it is in the means to achieve it where they differ, since SO1 focuses on operational protocols and the SO2 in updating the tools and systems, which are the two major areas that make up the program10, which has fully complied in achieving the goals of both components. The main achievement of the program has been to articulate and homogenize the post-disaster response of all the national agencies and institutions that are involved in the response from the different fields of action and levels of government, and that previously did so individually, partially , heterogeneous and disjointed, generating a great waste of the resources and efforts used and an inappropriate attention to the needs of the affected populations. The elaboration and institutionalization at the national level of a single response mechanism, which articulates the actions of each of the participating actors in a rational and coherent way, as a result of the program, is a great advance towards the general objective of strengthening resilience against multi-hazard disasters, since it ensures that the means and resources of the public administration will be used in a more efficient and transparent way in the response to disasters, contributing more effectively to the social and economic recovery of the population and the country. “Communication and coordination between everyone has improved. Now we know exactly what the communication channels are, with which people we have to coordinate. We have been making agreements with the municipalities so that they tell us exactly which is the technical link in the 10 SO1: Through critical path analysis and planning exercises, develop operational protocols to coordinate the flow of information and facilitate the implementation of damage assessment activities. SO2: Strengthen disaster response and intelligence mechanisms, by updating the preliminary damage assessment tool and related information and data management system. FINAL EVALUATION PREPARE COSTA RICA | 13 engineering and architecture part, and which is the social link for emergency management, since we have to be renewing these contacts due to changes in local governments, also with the Emergency Commission, but we already know what the link is in the area or canton where we are going to work.” Coordination and communication between the participating organizations, institutions and agencies has improved significantly, not only due to the legal mandate that requires the use of the protocols, systems and tools generated to articulate their actions, but also, to a very important extent, the approaches of inter-institutional work, integration and co-design that were implemented in its development, as all the interested parties consulted affirmed. These strategies have managed to generate fluid communication and work synergies between entities that previously did not communicate with each other, achieving the coordination of their resources and different possibilities in the search for a common objective. Thus, for example, the importance of the strategic relationship established with a private entity such as the Federated College of Engineers and Architects can be pointed out. The college has specialized human resources that are scarce in public administration organizations and now their deployment capacity for damage assessments has been increased. This deployment capacity has also been increased by the mapping of the human resources of the different institutions of the DRM system, which makes it possible to integrate these resources in a response, in a coordinated manner, at the time of an emergency. Despite the fact that the tools developed by the program have been institutionalized, there are still some important steps missing to ensure they can be used effectively by response institutions. Some of these gaps and problems with the functioning of the manufactured products were detected during the earthquake simulation carried out in November 2022. The simulation detected articulation and communication problems in the implementation of the first stage of the inter-institutional coordination protocol for damage assessment, the need for prior training of the actors in the use of the designed tools and systems, the need to be able to make queries and extract information in any phase of the emergency by the DRM system actors, among other observations11. However, since the simulation was implemented at the end of the program period, it did not allow time for adjusting the tools and systems to solve these areas of concern. The simulation also identified aspects of the operation of the tools that can only be tested in person, and therefore were not included in the exercise, which was carried out in the offices of the participating organizations, completely virtual, as agreed upon by all participating stakeholders. “Time is up and I hoped to have finished with the inter-institutional arrangements so that the Ministry of Health is already using the tool, the CNE is using it and the others too, but we haven't gotten there yet. There is a protocol that says that we use the instrument, but for us to be able to use it, technological developments have to come. There must be web services, licenses, cybersecurity policies, that is, a job is needed to be able to make this mandate operational.” Although, as previously mentioned, the program indicators are not very useful for measuring the progress made towards the formulated objectives, their measurement does translate into very high compliance with the planned activities, as shown in the following table: Table 3. Compliance with the Formulated Indicators Indicator Final degree of compliance Number of hazard risk reduction plans, strategies, policies, disaster preparedness, and contingency plans developed and in place 100% achieved 11 USAID/BHA PREPARE Costa Rica, diciembre 2022: Informe de sistematización del ejercicio de simulación. MIVAH INFORMANT MINSA INFORMANT 14 | FINAL EVALUATION PREPARE COSTA RICA Number of people participating in discussions regarding national risk reduction strategies as a result of the program Goal exceeded at 128% National and local risk assessment, hazards data and vulnerability information is available within targeted areas (Y/N) Does not apply Percentage of disaster risk management and response actors citing improvements in municipal-level preparedness at program end 100% achieved For the first indicator, a goal of four products was established, which were prepared and delivered to the relevant institutions. These were: critical path protocol, updating of the preliminary damage assessment (PDA) form, incorporation of the information needs of MINSA and IMAS in the MIVAH rapid damage assessment (RDA) form, and inter-institutional coordination protocol for on-site damage assessments. For the second indicator, the number of people who participated in the discussions on national risk reduction strategies that took place in the TWG and other spaces was 64, which represents 128% of the goal of 50 people. As already explained, the third indicator does not apply to this intervention as these products and data were developed in earlier phases of the PREPARE program. And regarding the last indicator, 100% of the people consulted in this evaluation stated that the program produced tangible improvements in disaster response preparedness, both at the municipal and national levels. Perceived changes and more effective interventions There have been numerous and very positive changes perceived from the PREPARE Costa Rica program expressed by the consulted informants, who cite not only those that were achieved as a result of this implementation phase but also the previous ones, which many of them see as a whole program difficult to separate. Among these changes it is mentioned that the authorities can now make decisions on emergency care based on documented, timely and reliable information, and can integrate the capacities and resources of the different actors in a rational manner, which is a fundamental change with respect to what they had been doing, which was to respond in an erratic, uncoordinated and blind way. It is a paradigm shift that is already being seen in the way institutions have been working, such as the Ministry of Housing and the Ministry of Health, which now use data collection instruments that integrate the needs of the different actors in the DRM system, and they share among themselves all the information necessary to manage the aid in a comprehensive manner. “Then the CNE took away our powers and said that they were the ones who were going to take care of it. That caused the relationship between the municipality and the CNE to break a little. And we, for our part, were trying to distribute the aid that IMAS gives, in an uncoordinated way with other bodies such as MIVAH, when there were people who had lost their homes and needed housing (...) At the end of the program we already began to have more relationships, to collaborate on things like a temporary aid regulation for people who have lost their belongings, and we began to give all the information we had to the other organizations, because we began to think more about the common good than about our own interests. A radical change in what were our institutional relations.” The coordination and communication between institutions, beyond the new legal guidelines, is also the result of the intense experience of inter-institutional exchange that was lived during these years of work in the PREPARE program, which produced new synergies and strategic associations between organizations that previously worked independently or even had rivalries among themselves. The availability of accessible and reliable information on the effects of disasters that have already occurred will allow for work on the prevention of the impacts that can be avoided, and on the preparation to face them. The informants also point out the importance of having figures on the MUNICIPALITY OF ALAJUELA INFORMANT FINAL EVALUATION PREPARE COSTA RICA | 15 magnitude of the risk, the seismic risk, which until now had been suspected to be very large but is now much more visible due to the damage quantifications revealed in the study carried out in the previous phase of PREPARE in Costa Rica. This quantification has led to the mobilization of resources and determination that has contributed to making possible the committed participation of the country's institutions in this phase of the program. The operational change in the way the DRM system deals with emergencies has been radical and has had very positive consequences. Previously, damage assessments were done using different paper forms, which then had to be assembled and transcribed into data management systems. Now the data collection instruments are implemented through digital devices, as common as a smartphone, which transmit the information directly to a centralized system, at the same time as it is recorded in the field and is made available to decision makers. These decision makers now know what kind of impact each family and house has suffered, and based on this, if a more specific evaluation is necessary and what technically trained human resources are available to carry it out, what kind of help is required, and which institution has to intervene in each case, since they have the protocols established to implement the corresponding aid. “It also implies the requirement of personnel to carry out damage assessment in the municipalities. That is why the CFIA got involved, because of the lists of professionals they have and to be able to call on them to get involved in the damage assessment, to make sure that it is not just anyone who is going to do that assessment.” The new response systems and protocols represent not only a very important saving of resources and time to be able to timely assist people who are going through critical and urgent situations, but also a minimization of human error in the processes and corruption in the management of available resources, which are always scarce. What activities and aspects of the program have been most effective in achieving the stated objectives? Among the activities of the program valued as most effective, are the TWG meetings, spaces where a true exchange of knowledge and experiences was generated for the joint development of tools that would meet the information needs of all the entities concerned. The success of this space was due in large part to the selection process of the institutional representatives as well as the management of the group by Miyamoto. The selection process ensured that individuals were chosen that had extensive experience in the field, that represented different professional areas of practice, and that all shared a genuine interest in the well-being of the public. These criteria made it so all the representatives complimented each other and could take advantage of the different visions of one another to create more robust procedures and tools. The transdisciplinarity of this group boosted its strength to generate products that are technically and culturally appropriate to the context of both the institutions and the population they serve: "The country has a high-level professional human resource, what is required is to have a space for communication and articulation like this, assertive, for dialogue, and for consensus-based alliances to be made, strategic alliances between the actors." The strengthening of the capacities of the institutions and the actors of the DRM system was accomplished not only with the development of the new tools and action protocols, but also by demonstrating how these systems can be well used. Hence the importance that all the people consulted gave to the California exchange trip, where representatives of the participating organizations were able to see first-hand an emergency management system that has faced large catastrophic events in a successful way. MUNICIPALITY OF SAN JOSÉ INFORMANT CNE INFORMANT 16 | FINAL EVALUATION PREPARE COSTA RICA The group, made up of representatives of the CNE, MIVAH, MINSA, IMAS, CFIA, the municipalities of San José and Alajuela, the donor, and Miyamoto's technical team, visited emergency management agencies in the northern Californian region at different levels of government, the Red Cross emergency management center, and the Center for Disaster Risk Management at the University of California at Berkeley. The representatives were able to see how these centers are organized and how they incorporate the use of technology, not only to address the damage already produced but to make damage projections to prepare to face future emergencies. They were also able to see how they integrate into the response parts of civil society such as pensioners and retirees, who are given the necessary training during blue sky times to be able to intervene adequately during emergencies. Another interesting takeaway from the visit was how these organizations and agencies take advantage of emergencies to sensitize public opinion about the importance of prevention and preparedness efforts. “Being able to see other emergency operations centers work, see how post-disaster evaluations are carried out, see how tools and resources are used, that we are going in the same direction and are developing it, is very beneficial. It is not only that they come to give you a tool or to give an orientation, but that they allow you to see the experiences of other actors.” The simulation exercise, despite some of its challenges, was effective in identifying the weaknesses of the emergency response systems and tools of the participating entities. It identified areas to be strengthened as well as areas for collaboration with other actors to ensure the proper execution of the established protocols. The Provincial Emergency Committees, the CNE, the MIVAH, the MINSA, and the IMAS participated in the simulation. Each one worked from their own offices on their designated responsibilities according to the protocols, and on the evaluation of the protocols themselves. “It seemed very important to me to do it before, because putting it to the test at the time of a true emergency could be very complicated. The exercise allowed us to realize a number of things that we have to improve, and it will allow us to prepare to be able to apply the protocol and the tools in a more efficient way.” An important reason for the achievement of the program objectives was the participatory process that was followed for its formulation, in which the specific needs and requirements of the institutions were identified with great precision. Likewise, Miyamoto's previous experiences with some of the actors in the DRM system created a level of confidence in Miyamoto resulting in the decision of the authorities to allocate human and material resources to the implementation of this stage of the PREPARE program and commit to giving institutional support for its results. Main challenges to achieve the objectives of the program The main obstacle that arose in achieving the goals of the program was the delay in the start of the activities, due to the COVID 19 emergency, which took away the focus and attention of several key program actors, such as the CNE and MINSA. This made it difficult to obtain the commitment of their officials and authorities, especially the CNE, to dedicate time and human resources to the intense participatory work that had to be undertaken to advance the objectives of the program. Approximately one year of the two planned to implement the program was invested in achieving the participation of the representatives of the key partner institutions. Changes in the representatives of some of the institutions throughout the participatory process in the TWG also caused some delays. Every time a representative changed, the program had to take the time to inform the new representative of the progress made and discuss with them the suitability of the work approach used to work. MINSA INFORMANT IMAS INFORMANT FINAL EVALUATION PREPARE COSTA RICA | 17 To address these obstacles, program management implemented measures such as maintaining very close and regular communication with the participating members of the steering committee, regularly sharing information about the work sessions carried out and the products produced, and setting the delivery dates of the products with the participants and having the flexibility to make changes to the work plan according to the especially busy periods of the organizations’ officials. However, the accumulated delay was significant, and made the programmed period of two years insufficient to adequately complete all the actions necessary to give sustainability to the results achieved, therefore a no cost extension was requested and approved. How did the specific intended and unintended changes positively or negatively affect program activities? The project team asked the donor for a one year no cost extension, which was approved. Due to this extension, it was decided to delay the implementation of the earthquake simulation exercise for several months, to allow time for the participants to familiarize themselves with the protocols and tools that were going to be implemented/tested during the exercise. However, funds began to run low, which impacted certain activities. The earthquake simulation exercise was one of the most affected activities. The simulation was intended as a test of the application of the tools, protocols and systems developed for the evaluation of post-earthquake damages, to identify possible gaps or inconsistencies, in order to make a final adjustment to all these products and leave them ready to be used. The simulation was carried out virtually in the most realistic way possible, with each actor working from their office, just as they would during a disaster. However, this simulation was held in November 2022, and the report on its systematization was delivered in December, the closing month for program activities, with little opportunity to discuss the results among the participants or to work with the TWG to resolve the problems identified. There were a couple of weaknesses identified in the articulation of the different actors of the response system, such as the link with the Municipal Emergency Committees, which due to their large number (since there is one in each canton of the country), required a subsequent process of capacity building and adaptation to the action protocols that was not able to be carried out. According to the opinion of several individuals consulted, the simulation exercise was implemented without the previous training and certification of its participants, which were necessary steps to guarantee the best results. The pandemic did also have a positive impact on the program. For example, it accelerated the migration of work to digital platforms in public administration bodies, which, although had already started before COVID 19, was definitely promoted as a result of this health emergency. The PREPARE program approach has always been the generation of digital tools and systems for disaster risk management, but this change would have been much more difficult to accept by officials who were used to collecting information on paper, had the pandemic not occurred during this program. SUSTAINABILITY The assessment of sustainability focuses on assessing the continuity over time of the positive impacts generated by the program once the aid has been withdrawn. In the field of cooperation, this concept is linked to the empowerment of key development factors and the appropriation of the process by the aid recipients. The sustainability analysis answers the following questions: To what extent have program activities contributed to sustainable changes or results, and what factors have contributed to achieving sustainability? Due to the collaborative and inclusive work carried out with the key actors of the emergency response system throughout the entire implementation period, from incorporating their needs and requirements into the design of the program to co-creating the program products, the actors have 18 | FINAL EVALUATION PREPARE COSTA RICA taken ownership of the changes achieved and have already been able to institutionalize some of the products within the regulations that govern them. The participating actors, both from the public and private sectors, some of which had not previously been taken into account for these types of activities, have incorporated the practices and strategies of interdisciplinary, interinstitutional and multisectoral work into their way of operating, both professionally and personally. This represents a significant social change that gives the program high institutional sustainability, within a public administration that, exceptionally in the Central American region, is characterized by the administrative continuity of its institutions and governing bodies. The incorporation of civil society organizations and the valuing of their potential in emergency response strategies and interventions, such as the Federated College of Engineers and Architects, into these new dynamics for response operations gives added sustainability to the results of the program. However, this can still be built upon by incorporating other actors with great potential, for examples universities, which have technical careers and research institutes linked to the assessment of damages to buildings and the reinforcement of structures to increase their seismic resistance. Proof of the sustainability achieved is that the new tools developed have already begun to be used and have demonstrated their usefulness, in recent events that have taken place in the country: “Precisely this week there was an earthquake that did not cause casualties, but it was an opportunity to put into practice all the new procedures and tools, because the organizations and community committees uploaded the information, the photos, because they learned to use it, they were part of the drill, and they put it into practice without anyone asking them.” However, some elements are also mentioned that can weaken the sustainability of the changes and results obtained, such as the following: • The lack of political will, or the lack of understanding on the part of politicians and heads of institutions, about the importance of respecting and maintaining the protocols designed for disaster response, which leads them to prioritizing other types of actions more immediate but less effective in the medium and long term. • The lack of a forecast for resources for emergency management in the regular budgets of the institutions of the system. Prevention and mitigation, as well as preparation for disaster response, are efforts that require constant actions in normal times, but in some institutions they are only contemplated in the additional budgets that are implemented only in disaster scenarios to face crisis situations. • Political changes in public administration institutions, such as the one that is currently taking place to merge various agencies, including IMAS, and suppress some programs, can affect the designed protocols, leading to the need for restructuring the protocols. Have mechanisms been established to maintain public-private collaboration beyond the life of the program? The program was very effective in generating synergies and strategic associations between the different participating organizations, which have been incorporated into the newly established joint action protocols, where the times and scope of the authority assigned to each of them are well defined. These synergies are also very present in the awareness and will of the people who have participated as representatives of key emergency response agencies, which gives hopeful signs that these collaborations will be maintained and strengthened beyond the life of the program, with the daily practice of its participants: “One of the weaknesses we have is that we are a centralized institution, with no presence in the territories, and we have few engineers and architects to face a large event. This has helped us to know how far we can go and where we have to ask for support from others, and to prepare MIYAMOTO INFORMANT FINAL EVALUATION PREPARE COSTA RICA | 19 trainings so that engineers and architects, in the event of a need, can help us in the field. Thanks to the CFIA comrade, they have been thinking about forming groups of volunteers by region so that they do not have to mobilize. That is another area of coordination that we have been improving with the program.” How was the exit strategy and sustainability plan established? The program's exit strategy was based on three elements. The first was the work approach that was mainstreamed throughout the program’s implementation, which focused on achieving the commitment of the entities that make up the DRM system and favored the appropriation of the products and results through stakeholder empowerment. An example of this is the formalization by the CNE of the Preliminary Assessment and Rapid Damage Assessment forms as the only tools to be used at the national level, which ensures that from now on the procedures carried out for post￾disaster damage assessments will be standardized and the information collected will be centralized to inform decision-making authorities on the necessary assistance measures. The second was to leave the authorities with a clear understanding of the next steps to continue strengthening and improving the system's damage assessment capacity. These steps are specified in the Critical Path Analysis, which was carried out by an external consultancy, and presented in April 2022, which maps out the steps/actions required to continue strengthening damage assessment capacities and provides key data for future efforts to be followed by institutions. This third and perhaps the weakest, was to ensure the stakeholders were aware of the key program recommendations for future actions. However, the stakeholders consulted did not appear to be aware, which allows us to infer that the results of this study have not been sufficiently disseminated among the actors involved in the program’s implementation and among the concerned authorities, which may compromise the sustainability of all the achievements made by the program. IMPACT In this section we value the effects and impacts generated by the project, expected or not, direct and indirect, collateral or induced. In theory, the impact analysis focuses on determining the net effects attributed to the activity, but this requires another type of intervention design that isolates the influence of external factors, to determine if the changes generated are direct results of the actions of the project. However, under a contribution to results approach, which recognizes that changes are generated as a result of many combined factors, we can identify the following impacts deriving from program activities, in combination with other context factors that were out of the program’s control: Economic impacts: • Having clear processes and homogeneous tools for post-disaster damage assessments will eliminate the duplication of efforts that until now have been wasting administration resources and will free up more resources to allocate to direct care for families. • Having reliable and standardized information will allow aid to be allocated to people who really need it, helping to return their lives to normal more quickly, which will in turn contribute to the economic recovery of the affected territories. Social impacts: • Having a well-articulated damage assessment system will lead to a more human and respectful treatment of individuals and families affected by natural disasters, eliminating their re-victimization through repeated damage and loss assessments by different institutions that act in an uncoordinated manner. • The acceleration of the social permit processes will allow humanitarian assistance to reach in MIVAH INFORMANT 20 | FINAL EVALUATION PREPARE COSTA RICA a timelier manner people who are in crisis situations and cannot wait for long periods of time for aid to be implemented. • The centralization of information and transparency in its collection and management will generate greater confidence by society in the role of government institutions. Political impacts: • The efficiency of the systems and protocols created will allow public institutions to fulfill the purposes and social responsibilities for which they were created. • A strengthened political culture of systemic and collaborative management among the different state agencies. • Having reliable information to make decisions with will allow for public policies to be directed in a way that is more in line with the reality of the country. “This information allows the system to make decisions, and those decisions are priorities in social aid, in aid that has to do with housing, and in reorienting the state investment plan towards the affected needs. The important aspect is: How a reliable information system allows making political, strategic and operational decisions that benefit and prioritize and have a positive impact at a social and economic level.” CONCLUSIONS RELEVANCE The PREPARE Costa Rica program was highly relevant due to the process followed for its formulation, in which key institutions within the DRM system participated, prioritizing their specific needs in the design of the activities. Miyamoto's experience, with several years of previous program intervention in Costa Rica, gave it sufficient knowledge about the institutional context to remain aligned with the legal mandates of the system and its actors, in order to respond exactly to the weaknesses and gaps that limited the effectiveness and efficiency of its damage assessment capacity. Those same weaknesses and gaps were also identified by all parties that participated in the prior needs assessment as the aspects that most limited the effectiveness of disaster response in the country. EFFICIENCY Program leadership knew how to take advantage of and enhance the knowledge and experience of local actors, through a careful selection of the people chosen to represent the participating organizations and institutions, and by successful managing collaborative inter-institutional work spaces. The financial resources of the project were adequately used in the implementation of the actions necessary to achieve the planned results, and with sufficient flexibility to adapt to unexpected situations and requirements. The implementation plan was temporarily altered by the COVID 19 health emergency, which at the beginning of the program kept several of the program's key partners busy responding to it and unable to start participating actively and continuously until well into the year 2021. Although measures were taken to lessen the effects of attention to this emergency on the participation of key actors in program activities, this delay affected the implementation of the last planned activities. PARTICIPATION The work was structured around two spaces for participation. The Steering Committee, for coordination and strategic monitoring, made up of Miyamoto's team and authorities from the National Emergency Commission, and the Technical Working Group (TWG), of an operational, transdisciplinary and inter-institutional nature, made up of representatives of public and private entities that make up the DRM system, local governments and program management. The TWG, with a horizontal and collaborative operation, contributed to the design, review, and validation of all MUNICIPALITY OF SAN JOSÉ INFORMANT FINAL EVALUATION PREPARE COSTA RICA | 21 the tools, systems and protocols developed, ensuring that the requirements and visions of the diverse participating actors were incorporated into them, generating synergies and associations that enhance the capabilities of each of them and at the same time of the entire system. The successful leadership by Miyamoto managed to overcome the difficulties of a participatory process with such heterogeneous and particular actors, achieving their active and committed participation in the production of high-quality technical tools that meet the expectations of all institutions. EFFECTIVENESS The PREPARE Costa Rica program effectively met its planned specific objectives, and significantly contributed to the achievement of its overall objective of strengthening multi-hazard disaster resilience in Costa Rica through improved disaster response coordination, planning, and deployable damage assessment capacity in order to effectively inform policies and post-disaster decision-making. All the planned actions were satisfactorily carried out and contributed to the achievement of the two specific objectives of the program, that were: the development of operational protocols to coordinate the flow of disaster information and facilitate the implementation of damage assessment activities, and the updating of Preliminary Damage Assessment (PDA) tools and related data/information management system. Regarding compliance with the established indicators, although these are of little use for measuring progress towards the achievement of the objectives, in their final measurement they met the programmed goals and even exceeded them. The main results achieved were: the unification of post-disaster damage assessment tools and procedures; the increase in the deployment capacity to carry out these assessments by incorporating resources that were not previously mapped, and articulating the different actors and resources in a logical sequence, eliminating the obstacles and duplications that made them lose efficiency and effectiveness; the strengthening of the capacities of the DRM institutions; and the collaborative and complementary work between institutions that previously acted in an uncoordinated individual manner. The actions that most contributed to the achievement of these results were the work of the technical group, the experience exchange trip to California with the DRM institutions and organizations in the United States, and the simulation exercise of an earthquake, despite the fact that it did not have the planned scope due to the accumulated delay in the implementation schedule. The obstacles that arose for the achievement of the results and the fulfillment of the programmed objectives were the COVID 19 emergency, the difficulty of involving officials of the institutions in work processes that required high time demand and dedication, and the slowness of the participatory co-creation process of the planned products. All these factors delayed the implementation of the actions, affecting the preparation and scope of the last ones. SUSTAINABILITY The results and changes achieved by the program are highly sustainable, due to the strong sense of appropriation and empowerment generated in the interested parties through their participation, both in the formulation of the program to be implemented and in the inter-institutional process of co-creation of the products elaborated. This participatory process produced synergies and associations between diverse actors that will foreseeably last beyond the life of the program. The formalization of the damage assessment tools by the CNE authority, in Agreement No. 206-11- 2022 signed on November 3, 2022, gives strong institutional sustainability to the program results, ensuring their use by all the parties interested in responding to emergency situations at the national level. The inclusion of private sector organizations and the strengthening of their role and importance within the operating protocols designed for the system gives social sustainability to the results, although this must still be strengthened by incorporating other social agents with great potential to contribute to damage assessments, such as universities. The Critical Path Analysis carried out within the framework of the program establishes a map of the actions that must be followed by the DRM institutions to continue strengthening their damage 22 | FINAL EVALUATION PREPARE COSTA RICA assessment capacity. However, the results of the analysis need to be socialized among the authorities of the institutions involved in their compliance, in order to mobilize the essential political will to allocate the necessary resources to give continuity and sustainability to this effort. IMPACT We can affirm that the results of the project can contribute, together with other factors of the context, to the achievement of the following economic, social, and political impacts: • Making the management of aid resources for the population affected by disasters more efficient, by eliminating duplication in the assessment of damages, and generating reliable information for decision-making. • Eliminating re-victimization of the population due to the multiplication of damage assessment processes. • Increasing the effectiveness of humanitarian assistance, by reducing aid management periods. • Increasing transparency in the management of humanitarian aid to improve the confidence of the population in state institutions. • Improving the capacity of public institutions to fulfill their social responsibilities. • Strengthening a political culture of systemic work. • Guiding public policies in a way that is more in line with the reality of the country. LESSONS LEARNED 1. Participatory work processes with public institutions can be very effective in generating appropriation of the products produced and empowering the participants to implement them, but they are also time demanding, due to the irregular availability of public officials to maintain a continuous and active participation in these processes. 2. Setting up medium-sized technical work teams that incorporate different disciplines and sectors that are united by common interests is a very effective strategy to break down prejudices and disagreements, and to generate products that integrate the diversity of visions and needs, as well as synergies and robust strategic partnerships. 3. A close, empathetic and proactive leadership of the inter-institutional participatory work groups is essential to achieve the desired results. 4. The joint work between the private sector and the public sector is very important to complement the shortcomings, experiences and resources of each other. 5. Activities that involve work with the public administration and seek the institutionalization of their results are long-term activities, since they require the generation of trust between the organizations that finance and implement them and the high-level authorities to achieve their political support, the demonstration of the technical and operational capacity of the implementing partners, and the deep knowledge of the institutional and legal context that will make these results sustainable. 6. The products that are elaborated through external consultancies do not have the same potential to generate awareness, appropriation and empowerment as those that are elaborated with the direct involvement of the interested parties. 7. The participation of beneficiaries in the identification of needs and priorities to be addressed in a program, greatly contributes to the effectiveness of the planned actions. 8. Actively choosing people with experience in the field and a long history of public service as representatives of the institutions in the technical working group was decisive for the achievement of program objectives. The inclusion of decision makers, that is, high authorities of the participating entities, was also important. 9. Involving local governments was crucial for the development of valid and useful products, FINAL EVALUATION PREPARE COSTA RICA | 23 since they are the ones who know and directly attend to the territories and the population in disaster events. 10. Carrying out a regional process/event to identify lessons learned at the mid-term of the program helped in decision-making and allowed for the application of methodologies and strategies implemented successfully in other countries. RECOMMENDATIONS FOR DRM SYSTEM AUTHORITIES: 1. Implement an earthquake drill as a practical and face-to-face test of the operational capabilities and real response times of the institutions involved in the deployment of damage assessors in the field, and to verify some aspects of the designed tools and protocols that could not be evaluated through the virtual simulation performed. 2. Elaborate, with the collaboration of the Technical Group participants, a project proposal to give continuity to this phase of PREPARE Costa Rica, that addresses the findings of the simulation exercise and the guidelines and road maps of the Critical Path Analysis, to strengthen the necessary capacities for the deployment of damage assessments and the management of humanitarian assistance. Present this proposal to potential donors and partners who may be interested in funding and implementing it. 3. Include universities that have careers related to damage assessments in the built environment, in the DRM system and jointly implement ongoing trainings for advanced students, private sector professionals, and DRM institution officials, to ensure the availability of sufficient and trained damage assessment personnel during a disaster. 4. Work on the production of communication pieces to raise awareness about the importance of disaster prevention and preparation, which can be used in scenarios of catastrophic events, when public opinion is more sensitive and receptive to this type of message. FOR MIYAMOTO: 1. Design and implement a strategy for disseminating the results of the Critical Path Analysis, and other products of interest, which includes abbreviated versions and graphical displays of the reports for DRM authorities, presentations of results that may be the subject of discussion in professional and academic forums, and pieces of information about their main findings, in non-technical language, to be disseminated by the media to broader audiences. 2. Work in the design phase of projects, to more precisely formulate the theory of change of the program, and include indicators appropriate to that design, that go beyond the contractual indicators established by the donor, and address the level of planned results and objectives, and not only the activities carried out. 3. Establish a system to regularly record and share the experiences and key information of the processes carried out in the interventions, so that the accumulated knowledge is not lost when changes are made to the technical personnel assigned to the projects. 4. When working with public institutions, it is recommended that there are at least two representatives of each institution in the technical working groups, so as not to overburden a single person with work, and so that the absence of one of them does not interrupt the learning curves and delay collective processes. FOR USAID/BHA: 1. Consider the possibility of giving financial support to a continuation phase of the PREPARE Costa Rica program, to address the findings of the Critical Path Analysis and the results of the simulation exercise, in order to give sustainability to the achievements, and deepen the impact of the previous phases of intervention. 24 | FINAL EVALUATION PREPARE COSTA RICA 2. Start any intervention with the financing of an identification and formulation phase in which all interested parties participate, to incorporate their conditions, needs and priorities in the design of the actions to be implemented. 3. Continue proactively cultivating relationships with authorities and decision makers, to count on their commitment to participation and the institutionalization of the results of the intervention. ANNEXES ANEXX I: SOURCES OF INFORMATION Table 1. Semi-structured Interviews INFORMANT ORGANIZATION OR INSTITUTION Roberto Brito (Director for LAC) Miyamoto International Bertalía Vega (Social Integration Coordinator) Municipalidad de Alajuela Eduardo Morales Quiroz Ministerio de Vivienda y Asentamientos Humanos MIVAH Javier Castillo Chien (Professional in Socio-Productive and Community Development) Instituto Mixto de Asistencia Social IMAS Keylor Castro (Focal Point for Risk Management) Ministerio de Salud MINSA Liliana Reyes Piña Departamento de Tecnologías de Información MIVAH Lorena Romero Vargas (Disaster Risk Management Coordinator) Municipalidad de San José Marcial Rivera (Head of Process Engineering Department) Colegio Federado de Ingenieros y Arquitectos CFIA Walter Fonseca Comisión Nacional de Emergencias CNE Table 2. Consulted Documents Technical Proposal USAID/OFDA PREPARE Costa Rica. Strengthening Deployable Damage Assessment Capacity in Costa Rica USAID/BHA PREPARE Costa Rica, February 2022. Critical Path Analysis for Increased Capacity for Post-Disaster Housing Damage Assessor Deployment, Data Collection, and Decision Making USAID/BHA PREPARE Costa Rica, January 2022. Diagnosis od the Current Capability of the Post￾Disaster Damage Assessment System for Housing MIVAH. Rapid Housing Damage Assessment Form. USAID/BHA PREPARE Costa Rica. Diagnosis for the Development of a Protocol for the Evaluation of Post-Disaster Housing Damage and Loss in Costa Rica and its Monitoring System USAID/BHA PREPARE Costa Rica, August 2021. Monitoring System for the Protocol for the Evaluation of Post-Disaster Housing Damage and Losses in Costa Rica USAID/BHA PREPARE Costa Rica, May 2022. Methodological process and concept note for the execution of the simulation of the “Post-Disaster Housing Damage and Loss Assessment Protocol” USAID/BHA/CNE/MIVAH/IMAS/Miyamoto. Housing damage assessment form USAID/BHA PREPARE Costa Rica, August 2022. Manual for the use of the form to record housing damages USAID/BHA PREPARE Costa Rica, December 2022. Simulation Exercise Systematization Report 2 | FINAL EVALUATION PREPARE COSTA RICA Miyamoto Internacional, April 2022. LESSONS LEARNED. PREPARE and EBRP Program in Latin America and the Caribbean (LAC) USAID/BHA PREPARE Costa Rica. Participation Indicators Tracking Tool USAID/BHA PREPARE Costa Rica. Baseline Report USAID/BHA, January 2021. Final evaluation of PREPARE II USAID/BHA PREPARE Costa Rica. Semi-Annual Performance Report, April-September 2021 USAID/BHA PREPARE Costa Rica. Semi-Annual Performance Report, September-March 2021 USAID/BHA PREPARE Costa Rica. Semi-Annual Performance Report, October 2021-March 2022 USAID/BHA PREPARE Costa Rica. Semi-Annual Performance Report, April-September 2022 ANEXX II: INFORMATION GATHERING TOOLS INTERVIEW SCRIPT A. Greeting and reading of informed consent B. Questions: 1. To what extent does the program respond to the priorities of institutions and organizations working in risk reduction in Costa Rica? 2. To what extent has the participation of national stakeholders shaped and helped develop the program’s outputs and outcomes? Did you participate in the first phase of the program? Did you participate in the design and formulation of the second phase? 3. To what extent has the program taken advantage of the internal experience and knowledge of organization and institutions in the country? 4. How has Miyamoto’s role been a catalyst in the articulation of the different approaches of the participating organizations? 5. To what extent has the overall program objective of strengthening resilience to multi-hazard disasters through improved response coordination, planning, and damage assessment deployment capacity to effectively inform policy and decision making in the aftermath of a disaster been achieved? 6. What were the most effective actions or strategies to achieve this objective? 7. What were the main difficulties or challenges encountered in achieving this objective? 8. Has coordination and communication among agencies, institutions and organizations interested in disaster risk reduction increased as a result of participation in the project? 9. To what extent has the program contributed to improving the self-sufficiency of participating organizations and institutions? 10. To what extent have program activities contributed to sustainable changes or results over time, and what factors have contributed to achieving this sustainability? 11. To what extent have the changes achieved caused or may cause direct or indirect social, economic and/or political effects in the country? 12. What lessons can be learned from PREPARE Costa Rica that can be replicated in other cases in the country and the region? 13. What recommendations could you identify for program improvement in a future program expansion phase? C. Farewell and Thank You ANNEX III: EVALUATION TEAM MEMBER Mildred Guerrero Echegaray Evaluator Social Investigator with two master’s degrees from the University of Valencia (Spain) in Promotion and Management of Local Development and Development Policies and Procedures, and vast experience in various types of assessments.