This publication was produced at the request of the United States Agency for International Development. It was prepared independently by an independent evaluation consultant, Mildred Guerrero Echegaray, hired by Miyamoto International, Inc. FINAL EVALUATION OF PREPARE GUATEMALA MARCH 2023 ii | FINAL EVALUATION OF PREPARE GUATEMALA FINAL EVALUATION OF PREPARE GUATEMALA March 2023 Award Number: 720FDA20GR00246 DISCLAIMER: The author’s views expressed in this publication do not necessarily reflect the views of the United States Agency for International Development of the United States Government. iii | FINAL EVALUATION OF PREPARE GUATEMALA TABLE OF CONTENTS ACRONYMS...........................................................................................................................IV EXECUTIVE SUMMARY.........................................................................................................1 EVALUATION PURPOSE AND EVALUATION QUESTIONS .........................................3 EVALUATION PURPOSE ........................................................................................................................................3 EVALUATION QUESTIONS...................................................................................................................................3 BACKGROUND.......................................................................................................................5 EVALUATION METHODS AND LIMITATIONS................................................................7 LIMITATIONS .............................................................................................................................................................7 FINDINGS, CONCLUSIONS, AND RECOMMENDATIONS ............................................9 FINDINGS....................................................................................................................................................................9 CONCLUSIONS ......................................................................................................................................................24 PROGRAMMATIC RECOMMENDATIONS.....................................................................................................26 ANEXXES...............................................................................................................................28 ANEXX I: INFORMATION SOURCES .............................................................................................................28 ANEXX II: DATA COLLECTION TOOLS .......................................................................................................30 FINAL EVALUATION OF PREPARE GUATEMALA | iv ACRONYMS AGIES Spanish acronym for Guatemalan Association of Structural and Seismic Engineering BHA Bureau of Humanitarian Assistance CESEM Spanish acronym for Center for Higher Studies in Energy and Mines COMRED Spanish Acronym for Municipal Coordinator for Disaster Reduction CONRED Spanish Acronym for National Coordinator for Disaster Reduction DDA Detailed Damage Assessment DRM Disaster Risk Management DRR Disaster Risk Reduction ESEGIR Spanish acronym for School for Higher Education in Comprehensive Risk Management EOC Emergency Operations Center FIUSAC Spanish acronym for School of Engineering of the San Carlos University of Guatemala INSIVUMEH Spanish acronym for National Institute of Seismology, Volcanology, Meteorology, and Hydrolgy MI Miyamoto International OFDA Office for Foreign Disaster Assistance PCI Project Concern International RDA Rapid Damage Assessment SE-CONRED Spanish acronym for Executive Secretariat of the National Coordinator for Disaster Reduction SEGEPLAN Spanish acronym for Secretariat of Planning and Programming of the Presidency USAC Spanish acronym for San Carlos University of Guatemala USAID U.S. Agency for International Development 1 | FINAL EVALUATION OF PREPARE GUATEMALA EXECUTIVE SUMMARY The PREPARE Guatemala program was implemented between 2020 and 2023 by Miyamoto International in partnership with Global Communities, and with funding from USAID/BHA, with the overall goal of reducing vulnerability to seismic risk in the Guatemala City Metropolitan Area through targeted technical and capacity building assistance closely aligned with existing DRR strategy and investments. The final evaluation of the program, carried out between January and March 2023, seeks to assess the progress achieved with the implementation of the program, according to the expected results and the objectives foreseen in its planning, identifying lessons learned and possible impacts, and proposing strategic and operational recommendations that collaborate with the sustainability of these results. The evaluation was done by reviewing and analyzing the main products of the intervention and its monitoring system and conducting 13 semi-structured interviews with key program stakeholders. The main findings of the evaluation were: • The program has been effective in achieving the specific objectives planned, but the contribution of these to the achievement of the overall goal formulated is still very incipient. This goal, although aligned with the attributions that disaster risk management institutions have by law in Guatemala, turned out to be very ambitious, given the distance that exists between these legal mandates and the current reality of these institutions, which lack the resources and mechanisms to appropriate the products developed and use them to improve disaster risk management, and the lack of perception of seismic risk among Guatemalan society and its authorities, who do not allocate resources to work on its prevention. • The program's implementation period, reduced as a consequence of the COVID-19 pandemic, was insufficient to give continuity to some of the program's most interesting products, which were presented almost at the end of the stipulated period. This did not allow enough time for the program to take advantage of its potential to mobilize consciences and wills to work on seismic risk prevention and reduction. • The participatory and integrative approach used in the development of the program's products achieved a broad use of the knowledge and experience of local partners, the adaptation of these products, of a high technical quality, to the Guatemalan context, and the generation of synergies and promising strategic partnerships between social actors that had not previously worked in an articulated manner, opening new opportunities for complementary efforts. • The institutional sustainability of the results of the training and capacity building component is high, since it relies on the ESEGIR, an organization that participated in the design and implementation of the trainings and has the interest and capacity to promote them and give them continuity. The participation of universities in this component is also critical to the sustainability of the results as they are interested in including these pedagogical tools in the academic course catalog of construction-related careers. However, the sustainability of the results of the policy and planning component is not guaranteed, since the institutions that in theory should take ownership of them and incorporate them into their processes do not have the mechanisms, resources, or capacity to do so. • The results obtained have a high potential for impact, since they highlight, in a novel way and based on reliable data, the importance and urgency of addressing an issue that Guatemalan society has kept forgotten despite its social and economic relevance for the country. This impact depends on its dissemination among public opinion and civil society, on mobilizing the political will of the authorities in the risk management sector to make the necessary changes and investments in the system, and on the materialization and institutionalization of the alliances that have emerged among the various participating sectors. FINAL EVALUATION OF PREPARE GUATEMALA | 2 In response to these findings, the following program recommendations and strategic guidelines were formulated to improve this and future interventions: • Expand the program to a second phase where inter-institutional collaborations are deepened to work on specific issues and needs derived from the diagnostics conducted, such as debris management strategies. • Use the products and data generated to design an awareness-raising strategy to increase the perception of seismic risk among the Guatemalan population and civil society, including abbreviated and graphically visualized versions of the reports for the authorities, presentations of results that can be the subject of discussion in professional and academic forums, and information pieces on their main findings, in non-technical language, to be disseminated by the media to wider audiences. • Deepen and institutionalize the relationship between DRR entities and universities, in order to take advantage of their potential to compensate for the lack of qualified personnel in structural damage assessment due to disasters that the public administration lacks, and to venture into some of the lines of research that result from the diagnoses carried out within the framework of the program. • Work on political advocacy and humanitarian diplomacy with high-level officials from the institutions of the DRR system and the national government, for the institutionalization of the instruments and processes generated. • In the project design phase, it is important to work on developing a theory of change that takes into account the threats and weaknesses of the context and local partners, in order to formulate realistic objectives that can be achieved with the available resources and timeframe. Similarly, indicators should be formulated that go beyond the contractual indicators established by the donor, and that address the level of planned results and objectives, and not only the level of activities carried out, so that they are useful for assessing progress achieved. • Local partners should participate in the design and formulation of interventions to ensure that their conditions, needs, and priorities are incorporated and that they can appropriate the results obtained to make them sustainable. 3 | FINAL EVALUATION OF PREPARE GUATEMALA EVALUATION PURPOSE AND EVALUATION QUESTIONS EVALUATION PURPOSE The main purpose of this final external evaluation is to assess the progress achieved in the implementation process of the PREPARE Guatemala program, based on the expected results and the objectives foreseen in its planning. It is also expected that the evaluation process will identify impacts, lessons learned, and propose strategic and operational recommendations that will contribute to the sustainability of the results achieved. The specific objectives, as specified in its Terms of Reference, are as follows: • Evaluate the effectiveness and impact of program activities in accordance with established objectives and indicators. • Identify and document lessons learned from the various activities and processes undertaken, as well as the impact of the program and its unplanned results, both positive and opportunities for improvement, and identify measures of program sustainability. EVALUATION QUESTIONS The evaluation questions answered in this study are: • To what extent does the program respond to the priorities and mandates of the institutions and organizations that are part of the DRM system in Guatemala? • To what extent were the program activities adapted to the baseline findings? • To what extent has the cross-cutting Neighborhood Approach been incorporated into the program? • To what extent has the COVID-19 crisis had an impact on the results achieved, and what adaptation strategies were adopted? • Was the program's allocation of human and material resources sufficient and adequate to achieve its objectives? • To what extent has the program taken advantage of the internal experience and knowledge led by the country's organizations and institutions? • To what extent has the participation of national stakeholders shaped and helped develop the program's outputs and outcomes? • To what extent have the outputs and outcomes included participation and accountability to the affected population? • To what extent were stakeholder priorities and needs incorporated into the design of program actions? • To what extent have the overall program goal and objectives been achieved? • What program activities have been most effective in achieving the stated objectives? • What have been the main challenges to achieving the program's goals and objectives? • Has coordination and communication between organizations, institutions and agencies increased as a result of participation in the project? • Have the tools developed under the program been put into practice by stakeholders? • How have specific intended and unintended changes affected program activities positively or negatively? • What lessons can be learned from PREPARE Guatemala that can be replicated in other cases in the country and the region? • To what extent have program activities contributed to sustainable changes or results, and what factors have contributed to achieving sustainability? • Have mechanisms been established to maintain public-private collaboration beyond the life of the program? FINAL EVALUATION OF PREPARE GUATEMALA | 4 • To what extent has the program contributed to improving the self-sufficiency of participating organizations and institutions? • How was the exit strategy and sustainability plan established? • To what extent have the changes achieved led, or are likely to lead, to direct or indirect social, economic and/or political effects in the intervention area? 5 | FINAL EVALUATION OF PREPARE GUATEMALA BACKGROUND The combination of high seismicity and accelerated and disorderly urban expansion converging in the metropolitan area of Guatemala City makes it particularly vulnerable to large-scale earthquakes1. This seismic risk is aggravated by the risk of landslides on its volcanic slopes, densely populated by informal housing settlements. National government projections predict that current migration patterns will intensify over the next decade, leading 75% of the country's population to reside in cities by 2032 and increasing the country's vulnerability to seismic events. Since 1976, when an earthquake measuring 7.6 on the Richter scale shook Guatemala socially and economically2, resulting in 23,000 fatalities, 76,000 injured and the loss of housing for 1.2 million people, according to official figures, it is expected that at some point a major telluric event will happen again, with the concern of whether the country is better prepared than then to face and mitigate the impacts of such a phenomenon. The PREPARE Guatemala program was implemented between October 2020 and December 2022, in partnership with PCI (now Global Communities) and funded by USAID/OFDA, under grant number 720FDA20GR00246. The program worked in alliance with municipal and central government authorities through institutions such as CONRED, INSIVUMEH and SEGEPLAN, as well as other strategic partners from academia and the private sector. Its overall objective is to Reduce vulnerability to seismic risk in the Metropolitan Area of Guatemala City, through capacity building and specific technical assistance, aligned with existing DRR strategies and investments in the country, and was designed to directly benefit 126 people from the public and private sectors of DRR and institutions responsible for disaster response, and indirectly to 1.5 million people who are considered to live in areas of high risk to natural disasters in Guatemala City. The theory of change underpinning the PREPARE Guatemala program is that effective and sustainable disaster risk reduction among vulnerable populations requires a systems approach. This implies broad stakeholder consultation in disaster risk reduction planning and implementation, an integrated design that addresses gaps in public policy and service delivery, and the generation of solid knowledge about conditions and needs based on scientific data and technical expertise. In this sense, the program shares the principles and good practices of the Neighborhood Approach applied by PCI in the Barrio Mío program implemented in Guatemala City since 2012, a model for humanitarian assistance and upgrading of high-risk informal settlements. 3 To plan and develop risk mitigation and preparedness programs for such scenarios, it is necessary to have timely and reliable information on risk exposure and the gaps and needs of the management and preparedness system. This information will be vital to strengthen the regulatory framework for the application of building codes and municipal land use planning efforts that the Government of Guatemala is now promoting, so the PREPARE program focuses on facilitating planning and decision￾making processes based on data through the development of models and methodologies for the production of this type of information, the strengthening of the actors of the DRR system´s capacities to use these methodologies, and the generation of strategic alliances among them. The latter involves facilitating inter-institutional collaboration between the different levels of government and cultivating links between governance institutions and the private sector, including the participation of representatives of academia, the professional sectors of engineering, architecture, and construction, and other areas of expertise relevant to the program. To contribute to the achievement of its general objective, the program was structured around three specific objectives, the first two framed within the Policy and Planning Subsector, which are: 1 USAID/BHA PREPARE Guatemala: Monitoring and Evaluation Plan. 2 USAID/BHA PREPARE Guatemala: Estudio para la formulación de estrategias ante sismos de gran magnitud en Guatemala. 3 Miyamoto International Inc. 2020: Revised Technical Proposal USAID/OFDA Guatemala Earthquake Risk Reduction Program (PREPARE Guatemala). FINAL EVALUATION OF PREPARE GUATEMALA | 6 SO1: Develop a risk exposure model of the built environment in Guatemala City and Mancomunidad Gran Ciudad del Sur to inform municipal disaster risk reduction, preparedness and land use planning. SO2: Assess the seismic vulnerability of prioritized critical infrastructure to guide decision-makers in planning and resource allocation for retrofits and other structural upgrades. The third specific objective, framed under the Capacity Building Subsector was: SO3: Strengthen the disaster risk reduction and response capacity of national and municipal disaster representatives and the private sector through targeted training and technical assistance informed by the vulnerability analysis. The following graphic summarizes the theory of change of the PREPARE Guatemala program: Table 1. Program Theory of Change 7 | FINAL EVALUATION OF PREPARE GUATEMALA EVALUATION METHODS AND LIMITATIONS The evaluation’s theoretical and methodological framework was guided by the criteria-based evaluation model 4, complemented by two evaluative approaches. A systemic evaluation, which describes the way in which the design and implementation of any intervention translate into results through a chain of causal relationships, and a democratic evaluation5, which is a participatory process centered on the organizations and which is highly attuned to the intervention cross-cutting approaches. Based on this triple approach and on secondary documentation analysis, it was possible to define the logical skeleton of the project, the assumptions that comprise it, the factors that influenced it and its sequential cause-effect logic. The evaluation questions are the backbone of the final evaluation because they determine the areas to be investigated and assessed. The questions were taken from the information provided in the terms of reference but were also complemented with other questions arising from the desk review and from the needs and expectations of the program's technical team, gathered through consultation. With them, the evaluation matrix was prepared, relating each question with the sources of information to answer them, and with the methods for collecting and analyzing this information. The evaluation was based on a qualitative methodology, using social science techniques such as semi￾structured interviews and documentary review and analysis of secondary sources. The phase of collecting and analyzing information from primary and secondary sources was carried out between January and February 2023. A total of 13 semi-structured interviews were carried out, out of 15 that were requested, representative of the different types of actors participating in the program. The informants were members from implementing partner organizations, and representatives of public and private organizations involved in the DRM system, municipalities, universities and public and private research institutes. The information obtained from the analysis of the secondary sources that were made available to the evaluation team was triangulated with the verbalizations made by the key informants, who had a committed participation, providing high quality information. This cross examination (speeches with facts), led to the identification of argumentative ruptures in the speeches issued in terms of what "should be", especially present in the evaluation processes. From the set of actors and documentary sources consulted, the evaluation team extracted both qualitative information and quantifiable data, although not statistically representative. At the qualitative level, the analytical process was carried out by contrasting the descriptive narratives of the informants with secondary information capable of ordering milestones, stages, and actors, and by superimposing this descriptive account with intersubjective verbalizations, coming from the explicit or underlying assessments of the interviewees. In addition, it was possible to obtain quantitative data from the project's secondary sources (monitoring and evaluation system), to which a descriptive or content analysis of the numerical data was carried out to determine the final state of achievement of the logical framework indicators, their coverage and scope. This analysis lead to the establishment of causal hypotheses of an introductory nature, whose value and interpretative sense acquires the dimension of an evaluative finding in its triangulation with qualitative methods. LIMITATIONS 4 The evaluation criteria of the Development Assistance Committee of the Organisation for Economic Co-operation and Development (OECD) were agreed in 1991. This document contains the five evaluation criteria that have been used over the last 25 years: relevance, effectiveness, efficiency, impact and sustainability. These criteria have served to homogenize evaluation practice, contributing to learning and improvement in international cooperation for sustainable development. 5 Evaluative model created by Barry Mac Donald (1967) or "democratic approach", which, by democratizing the use of knowledge, encourages informed participation and the responsibility of the different audiences. FINAL EVALUATION OF PREPARE GUATEMALA | 8 Although the evaluation has counted on the receptivity and collaboration of a significant group of representatives of the organizations and institutions participating in the PREPARE program, it was difficult to contact some key informants, who were not able to participate. For example, some of the representatives of the municipalities, whose multiple activities in their agendas apparently do not leave them time for this type of space, or who do not value their importance. However, despite this, a wide variety of informative inputs were successfully compiled which made it possible to build a solid basis for the ascertainment of the findings identified and the development of a comprehensive and well-founded evaluation report that, in keeping with the democratic evaluative approach, makes extensive use of the consulted actors' own words to illustrate the value judgments made. 9 | FINAL EVALUATION OF PREPARE GUATEMALA FINDINGS, CONCLUSIONS, AND RECOMMENDATIONS FINDINGS RELEVANCE AND ADAPTABILITY The evaluation of relevance consists of assessing the appropriateness of the objectives and actions of the intervention to the context in which it is carried out. Its analysis studies the quality of the diagnosis that supports it, judging its correspondence with the needs observed in the target population.6. Adaptability, on the other hand, is the ability to respond adequately to the changing demands of the environment in which the intervention is implemented. In assessing the relevance of the program, the evaluation team took into account the analysis of the problems to which the program responds, described in the technical proposal submitted in 2019 to USAID/BHA 7, and will answer the following evaluation question: To what extent does the program respond to the priorities and mandates of the institutions and organizations that are part of the DRRS in Guatemala? The program was designed in conjunction with Project Concern International (PCI), building on the partnerships and strategies developed by them in the Barrio Mío program implemented in Guatemala City since 2012, and maintaining its Neighborhood Approach in broad stakeholder participation, joint planning and coordination with the activities and investments that underpin DRR work. For the identification and design of the PREPARE program, a needs assessment was conducted in 2019 based on a series of meetings with authorities from different agencies part of the National Security System of Guatemala that have among their legal attributions the management and prevention of disaster risks, agencies and authorities at the municipal level, a unndiversities and private sector organizations, to review the priorities of DRR policies and programs and identify opportunities to support their initiatives. The program is aligned with the laws enacted to regulate urban growth and address disaster risks due to natural events, such as the National Disaster Risk Reduction Law, which develops the norms and protocols of the National Risk Management System, and with the efforts that Guatemala City has made at the municipal level to develop a national construction code to regulate urban land use, dictate earthquake-resistant construction standards, and plan investments in modernization and seismic upgrading of infrastructure. According to the regulations of the Law of the National Coordinator for the Reduction of Disasters of Natural or Provoked Origin (CONRED), it "(...) is constituted as a system integrated by agencies and entities of the public sector, the private sector and the civil society. It makes the decisions within its competence through its decision-making bodies, which are executed by the Executive Secretariat". The Law establishes as a priority function of the CONRED System "the assistance to the population during emergencies and disasters, providing immediate attention through the timely application of intervention procedures aimed at reducing the effects of the event and facilitating recovery, as well as the maximum leverage of all resources, avoiding duplication of efforts." However, the actual practice of the institutions and organizations working in risk reduction and emergency response is far from what is contained in the laws governing their functions. What on paper is conceived as a system, lacks mechanisms for the articulation of its parts, which operate separately, with no instruments to facilitate or oblige them to follow guidelines, coordinate efforts or share information. This weakens their capacity for risk prevention and post-disaster response. The municipalities, for example, have their own Emergency Operations Committees, which coordinate the prevention activities that some of the local emergency centers carry out, such as the drill 6 OECD/CAD Evaluation Management Manual. 7 Miyamoto International Inc. 2020: Revised Technical Proposal USAID/OFDA Guatemala Earthquake Risk Reduction Program (PREPARE Guatemala). FINAL EVALUATION OF PREPARE GUATEMALA | 10 implemented annually by the Municipality of Guatemala in commemoration of the 1976 earthquake, but without articulation with the national level coordination entities such as CONRED. “The Municipal Coordinators for Disaster Management are the entities of the municipality to attend to emergencies. It is difficult for us to attract them to join us because they always have a lot of work, they don´t give importance to preparatory issues or capacity building (…) Even many of our colleagues arrive at the municipalities and they don´t attend them, and even less in this elections year.” CONRED INFORMANT The meetings held with CONRED actors and leaders of local agencies revealed that the national legal framework for DRR is very weakly aligned with the strategies and resource allocation at the municipal levels, where CONRED does not have the legal mandate to enforce the implementation of building codes and land use regulation policies. In addition, the development of these codes and plans, in which efforts have been invested in recent years, is limited by the lack of reliable information on structural vulnerability in the built environment, which is vital for planning actions to strengthen and update critical infrastructure in emergency scenarios. On the other hand, the agencies called upon to implement this system do not have the human resources technically trained to fulfill the responsibilities attributed to them by law, and although these capacities do exist in the country, they are mostly concentrated in academia and the private sector, and there are no channels of communication between them and the public sector to ensure that these resources are used for risk prevention, vulnerability reduction and emergency response. The seismic risk, although very high in Guatemala City, is little perceived by both the public and by the institutions, which concentrate their human and technical resources on attending the emergencies that affect the country every year, such as floods, droughts and storms, and do not prioritize prevention in the face of seismic events whose periodicity is much longer, as expressed in the following testimonies of the actors consulted: “PREPARE fully responds to the mandate of the institutions that work in disaster risk reduction. I think that for many of them, it meant a relief from their day to day life, where what happens are generally meteorological phenomena, rain, storm. But a seismic event of great magnitude happens every fifty years, and them nobody worries, and even less in a Latin culture, where everything is to improvise, attend to the immediate and we will see when it happens, because there are thousand problems to solve before.” GLOBAL COMMUNITIES INFORMANT In the needs assessment conducted for the formulation of the program, the lack of sufficient, timely and reliable information to guide planning and decision making was identified as one of the most important shortcomings of the system, so it was decided to concentrate efforts on the creation of methodologies, tools and capacities for the elaboration of such information, which the participants valued very positively: "Having those models helps us to have more knowledge of the territory to better prepare ourselves and see where we can focus when foreseeing an emergency." MUNICIPALITY OF GUATEMALA INFORMANT However, this study did not identify the importance of other weaknesses that would continue to hinder the production of the necessary information and its use for timely decision making, even with these tools, such as those noted in the Baseline study: (1) lack of specialized personnel in the institutions of the DRR system, and even more in the municipalities, which have the responsibility to apply seismic resistant construction standards but not the technical capacity to do so; (2) CONRED does not assign personnel to seismic monitoring and prioritizes emergency response over risk management; and (3) the information that should flow between institutions is not being systematically captured or shared, which hinders the functioning of the system. 11 | FINAL EVALUATION OF PREPARE GUATEMALA The Baseline proposes some adjustments to the program to address the deficiencies identified, despite which its design did not undergo substantial modifications to adapt to these findings, but remained at a very theoretical level, poorly adapted to the local reality in some respects, as expressed in the following testimony: "This first phase was formulated by an outside consultant, it was very theoretical and idealized things very much, and for the second phase we tried to formulate it participatively with the technical group through several virtual and one face-to-face consultation spaces, and adjust the proposal, based on the needs, to make it more in line with reality." MIYAMOTO INFORMANT During the implementation of the program, important efforts were made to adapt the planned actions to the reality of its beneficiaries. To address the weakness and lack of institutional continuity, a considerable number of organizations from the private and academic sectors were included in the actions implemented, such as the Universities San Carlos de Guatemala and Rafael Landívar, the Guatemalan Institute of Cement and Concrete, the Guatemalan Association of Insurance Institutions, the Guatemalan Association of Structural and Seismic Engineering, the Guatemalan Association of Engineers and Architects, and public organizations, but with a technical profile, such as the Institute of Seismology, Volcanology, Meteorology and Hydrology, as well as representatives of different local governments, which are directly responsible for implementing emergency and disaster response in the territories. The inclusion of this wide variety of social actors was intended to generate a network of diverse stakeholders interested in disaster risk reduction, which could be a counterweight to the lack of continuity and articulation in public administration and at the same time could complement with its own resources the lack of qualified personnel in government institutions. Another modification made to improve the relevance of the program were the adjustments implemented in conjunction with ESEGIR officials to adapt the Course for the Certification and Evaluation of Damage to Infrastructure to the characteristics of the technicians of the risk management and emergency response agencies. This is a highly technical course, originally aimed at engineering, architecture and construction professionals who work in structural damage assessment in buildings, when the reality of state agencies is that they have very few professionals of these characteristics within their staff. The following testimony illustrates this case: "Many times, municipalities assign risk management tasks to personnel who do not have the appropriate professional profile, but who do have the responsibility to carry out certain actions. So we tried to ensure that the acceptance requirements for these courses did not limit their scope, because many of the engineering and architecture professionals are not linked to the administrations. That is why we included students in the last years of these careers, and we mapped, to participate for the agencies, people who had at least the experience of knowing the general evaluation methodologies that we already had." CONRED INFORMANT The course was adapted, both in its content and methodology, to the profile of the participants, and the students who successfully completed the course were officially certified as evaluators for two years. In spite of this, among some of the actors consulted, doubts persist as to whether this certification can be sufficient to make decisions on reinforcement measures or even demolition of structures, without having the academic and legal training and accreditation to do so. The relevance of the program was also affected by the period from its formulation in 2019 to its implementation at the end of 2020, which weakened the articulation planned with the Barrio Mio project and the contacts that had been made with officials and authorities who in that course were removed from their positions due to the change of government. Similarly, restrictions due to the COVID-19 pandemic prevented Miyamoto International engineers from traveling to work on site with local engineers to adapt their post-earthquake scenario calculation models to the FINAL EVALUATION OF PREPARE GUATEMALA | 12 characteristics of the buildings in Guatemala City. The remote work that was done with them was not conducive to deepening the specifics of these constructive forms, or generating the trust and exchange necessary to provide feedback to their methodologies and adapt them to the local reality. One strategy used to overcome the difficulties of teamwork through virtual means, due to the restrictions of the pandemic, was to create a technical group to work on specific issues, which also served to sensitize the personnel of the institutions on the importance of seismic risk management and reduction: "The technical group was created because we had only had virtual meetings, which are very bad when they go beyond 5 people, who get lost and do not talk. So the WG was created, and because they are technicians they are more involved, and we had regular meetings with them. Strategically this group was also convenient, because when we started nobody remembered the seismic issue, because two years had passed since the Miyamoto director came and met with authorities and key people, but two years later all those people had changed because of the elections, and we had to go little by little to each of the institutions, and to start having the delegates appointed took us a good time." MIYAMOTO INFORMANT EFFICIENCY The analysis of the efficiency of development cooperation actions refers to the study and assessment of the results achieved in comparison with the resources used, and the compliance with the schedule and planning of activities, and answers questions such as: To what extent has the program taken advantage of the internal experience and knowledge led by the organizations and institutions of the country? Was the program's distribution of human and material resources sufficient and adequate to achieve its objectives? As already noted, at the beginning of the program, a strong delay was generated that spanned almost the entire year of 2021, while the relationship with the institutions that had started in 2019 was recovered due to the interruption caused by the change of government and by the COVID-19 pandemic. An important factor that affected the execution and distribution of the budget was the inability of carrying out the scheduled trips of the MI engineers to Guatemala, to work face-to-face with local professionals in the adjustment of the probabilistic seismic risk assessment methodology. This freed up resources that were used in the generation of other products, not initially planned, whose objective was to strengthen the effectiveness of the results produced. These products included two additional training courses: (1) a short course on damage assessments in one- and two-story houses, which was requested by the municipality of Guatemala, and (2) a course on seismic risk assessment methodology in urban environments. In addition to these training courses, the need for a more comprehensive assessment of the existing gaps and needs in the DRR system became evident, in order to establish plans and strategies to overcome them. The program implemented a qualitative study called “Study for the formulation of strategies for large earthquakes in Guatemala”, which as mentioned above, recovers the experience and knowledge of more than 50 local professionals who have worked in risk management and emergency response, to identify the numerous lines of action and research that need to be addressed in order to move towards the goal of reducing seismic vulnerability in Guatemala City. As these cases show, the PREPARE program was very efficient in taking advantage of the existing knowledge and experience of the country's institutions and organizations, as is also unanimously affirmed by the key actors consulted, who consider that their contributions and those of others involved were at all times listened to, respected, and valued throughout all the processes implemented. Other examples of the use of existing resources are the use of information on hazards 13 | FINAL EVALUATION OF PREPARE GUATEMALA contained in a study financed by the Spanish cooperation for all of Central America, called RESIS 2, and the saving of a database of structural evaluation of buildings from 2001: "We wanted to make more progress in the inventory of critical infrastructure, and we were able to rescue a database from 2001, when the Presidency of the Republic carried out a project to evaluate 800 buildings throughout the country. It was stored in a very old database, in text only, in a computer at CONRED, and that was another extra product. On February 6 we are going deliver everything that was digitalized, and geo referenced." MIYAMOTO INFORMANT Regarding the human resources assigned to the program, the evidence confirms that they were sufficient and adequate for the execution of the planned actions, and beyond that, that the project management did a good job in adapting the original design to the conditions of the local context and to the prioritized requirements to advance towards the proposed objectives, maintaining the strategic orientation and the working principles and approaches that guide the program, throughout its implementation. In order to take advantage of the financial resources that were not used in activities that the pandemic restrictions did not allow to be executed, and to be able to complete processes that had been significantly delayed due to the slow start of the work, the program management and the Director of International Development and Humanitarian Affairs, based on the recommendations of the financial department, requested a no-cost extension until June 2023. The requested period would be used to link with a second phase of the program scheduled to begin in the fourth quarter of 2023, while maintaining the participation and commitment of key stakeholders, giving continuity to the results obtained and implementing other complementary processes necessary to ensure the sustainability of the actions carried out. Some of these processes, which were already in progress, were the meetings of the technical group to work on issues identified in the qualitative study and jointly formulate the actions and strategies to be followed in the second phase of the program, and the follow-up and strengthening of the volunteer group formed by the graduates of the course on rapid damage assessment in buildings, among others. But the unexpected way in which, for reasons external to the program, the planning of its second phase was paralyzed and funds were cut to finish what was executed in the first phase, seriously compromised the efficiency achieved, as the deficiencies and weaknesses identified in this phase of the program could not be addressed, nor the necessary processes for the sustainability of the products and the progress towards the general objective of the intervention could not be deepened. PARTICIPATION The participation analysis focuses on identifying the actors that have been involved in the different stages of planning and evaluation, assessing their impact on decision making, answering the following questions: To what extent has the participation of national stakeholders determined and helped develop program outputs and outcomes? To what extent were stakeholders' priorities and needs incorporated into the design of program actions? The PREPARE program incorporated in its design and implementation the Neighborhood Approach, which seeks to strengthen participatory processes for governance, catalyze local resources, share expertise, and promote accountability to beneficiaries. To this end, it was proposed to establish spaces for consultation, participation, co-creation, and planning with a wide and varied group of local partners interested in the subject and key actors of the DRM system. One of these spaces was the Technical Group, created in 2022 to energize discussion and joint reflection after a period of very little interaction due to the limitations of COVID-19 and the difficulty of getting institutions to FINAL EVALUATION OF PREPARE GUATEMALA | 14 participate. This space functioned as a working table to reflect, discuss, and contribute to specific technical issues, where the results obtained and the reports of the meetings held were shared with all participants and with full transparency, and which thus managed to arouse interest and overcome the existing reluctance among government institutions and private and civil society organizations to share information and openly discuss DRM needs and strategies. A wide range of social agents participated in this technical group, from authorities of several municipalities and national institutions such as SEGEPLAN, the National Security Council, the Ministries of Infrastructure and Housing, Environment and Natural Resources, to university and scientific research institutions, and professional and business organizations, such as the Guatemalan Association of Structural and Earthquake Engineering. Another aspect that stimulated active participation was that the methodologies and models used to generate the planned results were not used as finished models, but as pilot projects to be updated and adjusted to the context, with the contributions of the participating actors. This awakened the interest and commitment of the key organizations and institutions of the program, which worked altruistically in the construction of products of high technical quality, which required a strong dedication of time and effort. "When we developed the seismic risk model, we had 15 architecture students from Landívar and 12 structural engineers from the two universities, the College of Engineers and AGIES. We generated a nice synergy because we transcended the typical presentation where people assume a passive attitude, and because here we did concrete work. The students collected videos of all the polygons that were chosen, and the engineers worked voluntarily with the 1,700 sample buildings. This was very significant, because people are enthusiastic when there is a well-defined goal.” MIYAMOTO INFORMANT Under this same integration approach, in defining the contents, access profiles and teaching methodologies of the “Course for Certification in Damage Identification in Structures due to Natural Events”, the program maintained a total openness to incorporating the contributions and strategies devised by the ESEGIR of SECONRED. The contributions led to good results, the completion of the course by and certification of students, and a strong appropriation of this institution of this course/product, in whose co-creation it actively participated in: "We did a collective construction work, they as specialists and us in educational mediation, where we suggested the most suitable way to teach this course. And the project agreed to join the guidelines we already had here, in terms of course structure, modalities, evaluation and control mechanisms, even supporting documentation, because we asked the teacher to adjust their training package and develop a program following our templates, highlighting certain points that we did not see in the materials they brought." ESEGIR INFORMANT This collaborative work included the development of additional products, such as guides and manuals for course facilitators or instructors, requested by local partners to adapt the course to the context and ensure the effectiveness and sustainability of the results achieved. These guides are necessary so that the course can be taught by people who have already successfully completed it, but who are not professionals in careers related to construction, since these professionals are not abundant in the agencies working in risk management and preparedness for response to natural disasters in Guatemala. The course was loaned by Miyamoto to ESEGIR, and was hosted in its system, to be taught to future cohorts. Similarly, CONRED and the municipalities, as strategic partners of the project, participated in the development of the risk exposure model for the built environment, not only by providing the basic information, but also in the definition of the specific area and the critical infrastructure to be used in the study: 15 | FINAL EVALUATION OF PREPARE GUATEMALA "We chose diversity, that there would be community halls, schools, industry, commerce, because that is what would be worth having active in a risk scenario, but also that there would be a hillside area, which are mostly occupied by precarious settlements, and we did incorporate a small area of these characteristics." MUNICIPALITY OF GUATEMALA INFORMANT EFFECTIVENESS The evaluation of EFFECTIVENESS seeks to measure and assess the degree of achievement of the initially planned objectives. It seeks to judge the intervention in terms of its results orientation, also taking into account results and effects that were not foreseen but emerged from the processes put in place. The analysis of this criterion will also assess the components of the intervention that contributed most effectively to the achievement of the planned results and objectives, and those that worked less well, or even hindered progress, and that constitute possible areas of improvement for future interventions. Progress towards the formulated objectives To answer the first evaluation question of the effectiveness criterion: To what extent have the overall program goal and objectives been achieved? the evaluation team analyzed the following table that summarizes the degree of compliance with the outcome indicators for each of the specific planned objectives. Table 2. Compliance with program indicators General Objective: REDUCE VULNERABILITY TO SEISMIC RISK IN THE METROPOLITAN AREA OF GUATEMALA, THROUGH TECHNICAL ASSISTANCE AND STRENGTHENING OF SPECIFIC CAPACITIES ALIGNED WITH EXISTING STRATEGIES AND INVESTMENTS IN DISASTER RISK REDUCTION Subsector POLICY AND PLANNING SO1: Develop a risk exposure model of the built environment of Guatemala City and Mancomunidad Gran Ciudad del Sur, to provide information for disaster risk preparedness and reduction, and land use planning. SO2: Assess the seismic vulnerability of prioritized critical infrastructure to guide decision makers in planning and allocating resources for upgrades and other structural improvements. INDICATOR FINAL VALUE 1.1 Number of people participating in discussions on national disaster risk reduction strategies as a result of the program. 150 % 1.2 Available information on risk, hazard and vulnerability assessment in selected areas. ACHIEVED 1.3 Risk information is incorporated into risk management and response plans and policies at the departmental and municipal levels. NOT ACHIEVED Subsector CAPACITY STRENGTHENING SO3: Strengthen the disaster risk reduction and response capacity of national and municipal decision￾makers and the private sector through training and targeted technical assistance informed by vulnerability analysis INDICATOR FINAL VALUE 3.1 Number of people trained in disaster preparedness and management and risk reduction. 204 % 3.2 Number of people who pass final exams or receive certificates. 204 % 3.3 Percentage of trainees who retain knowledge and skills two months after the course. 91% FINAL EVALUATION OF PREPARE GUATEMALA | 16 The first indicator of Specific Objectives 1 and 2 had a target of 40 people participating in discussions on national disaster risk reduction strategies. The program held several meetings with different objectives and participants. Of these, those that included space for discussion were the five meetings of the Technical Group and a coordination meeting held in December 2021, which were attended by 60 people in total, representing 150% over the target. Among the main activities planned to achieve these two objectives is the probabilistic seismic risk assessment of the built environment in a selected area of Guatemala City, with the purpose of providing risk management authorities with information that will allow engineers to model a range of scenarios, estimating the possible impacts of earthquakes of different magnitudes. With the active participation of CONRED and the municipalities of Guatemala and Mixco, a polygon of approximately 20 km² was defined, distributed between the two municipalities, in a central sector of the city with a high population density and comprising housing areas, some of which are old and have had problems of land settlement, as well as commercial and industrial areas, local and metropolitan facilities and important road arteries. A sample of 4% of the existing buildings within this polygon was taken to study their structures, in which a typology of the buildings was elaborated based on their structural and construction systems, the materials used, and the number of floors. Given the health security measures due to the COVID-19 pandemic, the buildings of the selected sample were not visited internally, but photographs and films were taken of their facades and exterior areas, which were used, together with aerial photographs and cadastral information, for subsequent analysis by the engineers. Engineering and architecture students, who were trained for this purpose, participated in the collection of information in the field, and professionals from academia and professional associations participated in the analysis of the information, working altruistically for a month. Based on this information, demographic data taken from the 2018 census, and land use patterns in the city, it was possible to model the behavior of infrastructure in the event of a large magnitude earthquake, using a probabilistic analysis program for seismic risk assessment, and determine what impacts this event would have on the city of Guatemala, in terms of number of deaths and injuries in daytime and nighttime scenarios, number of people displaced by loss of their homes, and volume of debris generated by collapsed structures. For rapid assessment of critical infrastructure8, in conjunction with the Department of Infrastructure Works of CONRED's Mitigation Directorate, 25 buildings were selected for their functionality and importance to the municipality and their viability to be included in a response action in the event of an adverse event, such as schools, churches, hospitals, etc. The evaluation was implemented as a pilot exercise, based on a format suggested by Miyamoto International, which was tested with the participation of engineers and other actors from the program's partner organizations, whose suggestions for changes and adjustments were collected and incorporated into the final evaluation report. This instrument, adjusted to the Guatemalan context, together with the manual that was prepared as a protocol for its use, was delivered to CONRED and became part of the Rapid Damage Assessment Preparation Course, which is part of Specific Objective 3. The evaluation of these 25 buildings was carried out through on-site visits to 21 of them and virtual visits to the remaining 4, complemented with information from the universities and the institutions themselves. The study showed their state and degree of vulnerability, and proposed actions for their structural reinforcement and to improve the conditions for their articulation within the mechanisms and protocols for disaster response. With the preparation and delivery of these two products, the Risk Exposure Model of the Built Environment of Guatemala City and the Seismic Vulnerability Assessment of the prioritized critical infrastructure, the program fulfilled the first part of Specific Objectives 1 and 2, achieving a high 8 Critical infrastructure is defined as the set of systems and facilities whose destruction or incapacity would have a debilitating impact on the economy, security, health or welfare, ultimately affecting the ability to govern. 17 | FINAL EVALUATION OF PREPARE GUATEMALA measurement of indicator 1.2. and a very positive evaluation by all the participants consulted, both regarding their technical quality and the processes followed to prepare them. However, although awareness-raising work was carried out with the highest-level authorities, who were presented with the products that were developed that showed the existing vulnerabilities and risks, it has not yet been possible to mobilize the political will to make the necessary changes so that these products are appropriated by the institutions and used in response planning and risk management. This is evidenced by the measurement of indicator 1.3, which refers to the usefulness of these products. "Although the specific objective was met, there is still a gap, because the methodology should be considered as a resource to be implemented in other areas, which is not the case today. That is still up in the air because, although we are part of these activities, our day-to-day activities do not allow us to take care of this issue (...) We have just returned from a series of evaluations of an area affected by seismic swarms, and we saw the need for a tool that would facilitate the evaluation. The one that was built in the program could do it, but we were not allowed to use it. We had to use one that was already implemented by other management, but with this one the results are going to be a little biased by the tool, which doesn't evaluate the real aspects of a seismic hazard." CONRED INFORMANT As for Specific Objective 3, the main product planned to achieve it was the Course for certification in the identification of damage to structures due to natural events, with emphasis on pre- and post-earthquake assessment. The general objective of this course is to "(...) generate competencies for the identification of structural damage or potential damage to structures before the occurrence of an earthquake and other events (...) The certification will provide knowledge on the behavior of structures in disaster events, thus strengthening the community in terms of its capacity to respond to incidents, especially those related to earthquakes.”9 The course was originally intended for engineers, architects and other construction professionals, who are the ones who may be academically and legally empowered to make binding judgments about the condition of building structures and recommendations based on those judgments, as specified in the Teaching Guide developed to support the course: "The RDA aims to identify which buildings are safe and can be occupied, or on the contrary, their use is restricted or they are advised to be evacuated and then evaluated in detail by means of an DDA. In general, carrying out damage assessments requires good judgment on the part of the inspection brigade, since it is very difficult to develop procedures and guidelines for damage assessment without sound criteria. That is why the RDA should be carried out by professionals in architecture, construction engineering, civil engineering and especially structural engineering. It is not recommended that the process be performed by the general population or specialists from other professions, as they lack the necessary technical knowledge about structures." However, given that its main target audience, which are governmental institutions and municipalities as actors of the DRR system, do not have these types of professionals among their staff, it was necessary to make important adjustments to the course. The first was to expand the entry profile to allow access to officials without specific training but with experience in incident response and basic knowledge about the measures to take during the occurrence of an emergency, and to students in their last year of architecture and engineering careers, who are the ones who worked as volunteers in the collection of information to perform structural evaluations of the buildings that made up the sample analyzed in the risk study. In the graduation profile it is specified, contradictorily to what is stated in the aforementioned Teaching Guide, that "the student (who successfully completes the course) will possess skills to participate in official qualification processes as an evaluator before official entities." The course was implemented over five weeks, with virtual sessions, both synchronous and asynchronous, and an 8-hour face-to-face practice. Participants came from public agencies such as 9 USAID/BHA Prepare Guatemala: Course for certification in identification of damage to structures due to natural events, with emphasis in pre- and post-earthquake evaluation. FINAL EVALUATION OF PREPARE GUATEMALA | 18 COMRED, COLRED, SSO brigades and risk management offices of several municipalities and mayors' offices. Despite the aforementioned warnings about the academic training that evaluators should have, all those who completed and passed this training have been accredited as officially recognized structural damage evaluators for 2 years. It has been a forced landing of the original design of the course in the context of Guatemalan institutions that, in spite of the adjustment efforts made, still leaves some contradictions and questions in the air. "We did have to make adaptations in the rapid assessment course. We made a report card that was adjusted to Guatemalan standards, to those of AGIES, and we also lowered the level a little. It was difficult, because of the participants in the course, those who were structural engineers, who came from the university or the private sector, they did not need the course. What they needed was a methodology, to review processes, feedback meetings... Meanwhile, the people who came from the institutions, from the municipalities, many are not even engineers, there were lawyers, accountants, firefighters, and although we tried to find a middle ground, I have the impression that the course was too high for many of the people who participated." MIYAMOTO INFORMANT CONRED's Higher School of Studies in Comprehensive Risk Management (ESEGIR) actively participated in the adaptation of the course to the Guatemalan context, both in its content and access requirements, as well as in the methodologies and pedagogical modalities implemented, and in the elaboration of a teaching guide to support future facilitators. In this way, mechanisms were established that allowed the participation of people from places far from the capital, and that were adapted to the time availability of students who were also full-time workers. "We saw the need to have a virtual space where students could host all the support documents, an asynchronous learning classroom where they could have access to all the class recordings. In the same way, the tests were taken and the grade book was updated by module and the student could have access to it and know their progress. A group chat was generated where the students and the facilitator were, and for any doubts there was an instant messaging where they were answered and sent to everyone." ESEGIR INFORMANT The course is hosted on the ESEGIR´s platform as a permanent course offered open to all CONRED organizations, and for this year (2023) the launching of two new cohorts is already being planned, led by several of the graduates of the first course. The 53 graduates have formed a volunteer group to act at the request of the DRM organizations in cases of need. Regarding the measurement of the indicators of Specific Objective 3, the target established for indicator 2.1 was 70 people trained in disaster preparedness and management and risk reduction. The following graph summarizes the number of people trained in four courses implemented under the project: Table 3. Number of participants in implemented training courses Course Name nº of Registered Participants nº of Certificates Absolute Relative Course for certification in the identification of damage to structures due to natural events. 69 53 76% Course on damage identification in one- and two-story dwellings 41 41 100% Damage Assessment 42 42 100% Methodology for seismic risk assessment in the built environment. 7 7 100% TOTAL 159 143 89.93% 19 | FINAL EVALUATION OF PREPARE GUATEMALA As can be seen, the number of people who completed and were certified in the courses implemented was 143, which is 204% of the target. In the first of these courses, the most important of the program, 53 of the 69 people enrolled (76%) successfully completed it, a very good result considering that it was a demanding course in terms of difficulty and dedication, that endorses the adequacy of the pedagogical strategies and resources designed to teach it. This good result is reinforced by the measurement of indicator 3.3, which exceeds the established goal of 80% of certified students (in this course only) having at least 70% correct answers in an evaluation survey conducted two months after the end of the course, the result of which showed that 91% of the respondents met this expectation. Specific Objective 3 also sought to provide technical assistance to national and municipal authorities and the private sector to overcome the gaps and deficiencies in the system identified in the vulnerability analysis. In this search, a technical group was created which held working meetings on the most problematic aspects arising from this assessment, and another detailed qualitative diagnosis was prepared, which was not foreseen in the formulation of the program but which became necessary in the course of the intervention. In this study, over 50 local professionals working in the field of risk management and who have participated in processes of attention to seismic events were interviewed. This diagnosis made it possible to "identify limitations and strengths of the current CONRED system, as well as 36 specific issues that are important to take into account in the generation of strategies, plans, programs and projects that address the impact scenario of a large magnitude earthquake in a Guatemalan metropolitan area." Despite the great importance of the results of this study, its effectiveness in contributing to the achievement of specific objective 3, to strengthen the disaster risk reduction and response capacity of national and municipal decision makers and the private sector, was limited because it was conducted at the end of the project period (it was presented in December 2022). Although, its results constitute basic inputs to start working on strengthening the DRR system. Perceived changes and more effective interventions Has coordination and communication between organizations, institutions and agencies increased as a result of their participation in the project? Some of the people consulted pointed out that a better coordination and collaboration between organizations, institutions and agencies working in DRM was achieved, especially between private sector organizations and public sector organizations, which maintained prejudices and mistrust that prevented a fruitful interaction for the achievement of common objectives. Due to the implementation of the program, some promising strategic alliances were established and strengthened among diverse actors and novel collaborative dynamics were promoted, but they are still based on the interest and will of the participants, since they lack institutionalized protocols and mechanisms to be maintained over time and beyond the change of the people who occupy key positions in risk management and emergency response institutions. The technical working group was the space where this change in behavior was generated, but most of the informants consider that without a strong leadership as the PREPARE program had, it will be very difficult for this working group to be maintained, since it is very recent in its creation, and there is currently no organization that assumes such leadership. What program activities have been most effective in achieving the established objectives? One of the conditions that most contributed to the achievement of the program's specific objectives was the extensive network of contacts and the convening capacity that had been built within the framework of the Barrio Mío project of Global Communities, which were the basis on which the participatory work between the different public and private actors in the DRM sector was structured. The relationship with the universities, for example, was very fruitful, since their contribution in terms of work, time and knowledge was decisive in the elaboration of all the FINAL EVALUATION OF PREPARE GUATEMALA | 20 program's products. Likewise, the collaboration with CONRED, which was actively involved in the activities, allowed the adaptation of methodologies to expand their coverage, as in the case of the course implemented jointly with ESEGIR. A determining factor in this collaborative relationship was the practice of working together on specific topics, with an appreciative approach that stimulated interest, contributions, and enriching discussions among all the participants, generating a climate of trust that overcame the prejudices that previously existed in some of them. "The aspects that collaborated the most were the work tables where the topics were discussed and each entity contributed from its knowledge to contribute to the topic. I think there was a very important part of learning from others and accepting tools created by others and that they are not only the managers, but we can all be." ACADEMIC SECTOR INFORMANT The conceptualization of the tools and methodologies used as pilot projects to be tested and adjusted with the suggestions of the participants was key to their adaptation to the specific context and the use of local knowledge and experiences. Similarly, incorporating international standards and having foreign professionals contribute their experiences in other countries was vital for these tools and methodologies to transcend the local context and allow for foreign assistance in disaster events that require it. The rapid damage assessment of structures course was taught in a mixed modality, where the virtual sessions were complemented with classroom practice, visiting the buildings and facilities to directly observe the risks suffered by their structures. This made it possible to reach people in areas far from the capital city who would not have been able to participate due to travel and accommodation costs, while at the same time stimulating the understanding and learning of the concepts studied through their application in practical exercises. A very important factor in the success of this course was the mapping carried out in the institutions to select the most suitable people to participate due to their experience in risk management and field response, as shown by the high percentage of passing grades obtained. The detailed analysis of the area's infrastructure addressed in the risk exposure model of the built environment, combining documentary information on them with extensive and exhaustive field work, was an important added value, since it is usually done using satellite imaging techniques. By quantifying the risks and bringing to the forefront issues that had not been previously highlighted, such as the volume and management of the debris that could be produced by a major earthquake, the study managed to arouse the interest and commitment of the various stakeholders to continue collaborating on the work. The results of this study, were striking and innovative. Have the tools developed within the framework of the program been put into practice by the stakeholders? The appropriation and use of the tools generated by the program is inconsistent, depending on the specific interest and the existence of institutional channels and technical and operational capacities in each of the organizations that are called upon to use them. "The course will continue to be used by the institution. Even the first cohort we worked with was a pilot plan, where the students evaluated the methodology, the synchronous and asynchronous issues, the application itself, how they felt when they applied for it, whether it was user-friendly, whether it generated doubts, etc. We compiled all this by means of instruments and at this moment we are making the corresponding improvements to present the definitive one and replicate it under the same scheme: a hybrid program that incorporates virtual moments, the exercise to be developed and all the platforms we use attached." ESEGIR INFORMANT 21 | FINAL EVALUATION OF PREPARE GUATEMALA The course for certification in the identification of damage to structures due to natural events was adopted by ESEGIR, which has the mechanisms and technical resources to incorporate it into its permanent platform and to continue teaching it. However, the other products, despite the existing interest in them and their technical quality, and the fact that they were also adapted to the Guatemalan context with the active participation of the various stakeholders, do not have an institutional insertion structure that can provide them with use, continuity, and development. It is notable that CONRED, which was a fundamental actor in the implementation of this intervention, does not currently implement risk prevention and disaster preparedness processes where these products can be used, despite its legal mandate, which prefigures an ideal situation that is still far from reality: "We as the Mitigation Directorate, who should be generating these analyses and methodologies, are busy with other activities, not doing what would actually generate greater benefits, instead, we are attending to the many requests that come in daily, preparing field visits, preparing reports, issues that are not the responsibility of the secretariat but of other agencies and institutions. But since they require a report from CONRED, then we have to do it here". CONRED INFORMANT Of the different actors that participated in the program, it is the universities, research centers and public organizations of a more technical than political nature, such as INSIVUMEH, that have the capacity and interest to take advantage of the program's products such as methodologies and courses on seismic risk and vulnerability assessment of infrastructures, and to deepen some of the multiple lines of research and actions resulting from the diagnoses carried out. However, although there is an intention to continue working on the alliances built, the institutional channels for these research and academic programs to improve the DRM and seismic event response system have not yet been established: "From the research institute we have a specific line of research on the topic of risk. We monitored two volcanoes in Guatemala, and the topic of earthquakes was a debt. Now everything we did in the program allowed us to see how we can link up to support monitoring, how we can help in giving the virtual platform that we already manage to make these processes easier at the time of an event." ACADEMIC SECTOR INFORMANT Main challenges in achieving the program's objectives One of the main challenges was the lack of articulation, coordination and exchange of information and resources among the different actors of the DRM system and the different levels of government involved. CONRED as coordinator does not have the power to establish homologated norms and procedures to be followed by these institutions, who are not working as part of a whole, as stipulated in the law that defines the system, but separately, each with its own resources, mechanisms, and tools, which produces duplication of efforts, inconsistencies and little effectiveness in prevention and response. The lack of trained human resources in DRM organizations also weakens efforts to improve their performance, and although the private sector does have these professionals, there are no mechanisms for their work to be integrated into risk management and emergency response. The lack of continuity in the technical staff and authorities of public agencies, who are removed from their posts with each change of government, hindered and delayed the joint work processes, and is an obstacle for the mobilization and training efforts on disaster risk management to bear lasting fruit. "The most complicated thing is working with public institutions, there is always a constant change among those who are involved in the processes, because a boss changes and the whole team changes, and then we start from scratch. In the program it happened with some of the entities, that one was the representative and then changed and we had to start all over again. So public management is very complicated for us because, in whose hands is it left?” FINAL EVALUATION OF PREPARE GUATEMALA | 22 ACADEMIC SECTOR INFORMANT The low perception of seismic risk in comparison with other risks that are much more present due to their actuality and recurrence, makes it difficult to allocate resources and efforts to seismic prevention and preparedness, in an institutional culture that is mostly reactive and focused on emergency response rather than prevention. Another obstacle was that there is a lack of access to information or secrecy among various institutions and levels of government, which makes it difficult to share among them the information that is important and necessary to develop strategies and plans based on reliable data: "It is very difficult to access the census data disaggregated by area, which is information that should be public but is not really accessible. If it were accessible, we could have come to identify how much real population is exposed in the area that was delimited in the study, because we did it by extrapolating, but up to that level we could not reach because we only have population data at the municipality level." ACADEMIC SECTOR INFORMANT A fundamental obstacle was the long delay at the beginning of the project, because its key actors were the agencies that were focused on the COVID-19 emergency and because of the slow process of incorporating the institutions, almost two years and a change of government having passed since the first contacts were made by MI. This substantial delay within the short period of program implementation made it necessary to request an extension of the program and the implementation of a second phase, in order to put into operation, the products developed, according to the objectives sought. SUSTAINABILITY The sustainability assessment focuses on the evaluation of the continuity over time of the positive effects generated by the program intervention once the aid has been withdrawn. In the field of cooperation, this concept is linked to the enhancement of key development factors and ownership of the process by aid recipients. The sustainability analysis answers the following questions: To what extent have program activities contributed to sustainable changes or outcomes, and what factors have contributed to achieving sustainability? Have mechanisms been established to maintain the public-private partnership beyond the life of the program? How was the exit strategy and sustainability plan established? The component of the project that has a greater promise of sustainability is capacity building, since the Course for the certification on damage assessments of structures due to natural events was institutionalized within the ESEGIR as part of its permanent teaching platform and this organization is already planning the next cohorts for the year 2023, where the tools and methodologies developed within the framework of the PREPARE program will be applied. Likewise, the universities that participated are interested in incorporating these courses and workshops into the curricula of the architecture and engineering careers: "Also, from the perspective of academia it was very important to have professors participating in the program, because in the courses for future architects and civil engineers who could participate in these topics, they really did not go deeper into the seismic topic because there were no technical capabilities to be able to explain more, and the program did allow it." ACADEMIC SECTOR INFORMANT 23 | FINAL EVALUATION OF PREPARE GUATEMALA Regarding the products and results of the Policies and Planning component, its sustainability is weak, since it relies on an organization such as CONRED, which has neither the human and technical resources, nor the articulation mechanisms, nor the legal tools, to take advantage of these products and unify the actions of all the entities working in DRM, so that they can optimize their disaster preparedness and response capabilities. The sustainability of these results would then depend on the national political will to make the necessary changes and allocate the necessary resources to introduce a culture of prevention in a DRM system that does not function as such and that acts in a reactive manner, giving partial and isolated responses to the natural events that year after year impact the country. But this political will shows no signs of existing, much less in the case of seismic hazards, whose recurrence periods are very long in comparison with the short-term goals that characterize government management, within a context of lack of institutional continuity, where officials and technicians of public administration agencies are replaced with each change of government. Additionally, the low perception of seismic risk in society in general does not contribute to the sustainability of the implemented actions, since it does not create pressure on the government to promote the necessary transformations to implement a well-articulated DRM system, which could take advantage of the technical tools generated by the program in the reduction of vulnerability to seismic events, which is the objective for which they were created. IMPACT In this section we will assess the effects and impacts generated by the project, whether expected or not, direct or indirect, collateral or induced. In theory, impact analysis focuses on determining the net effects attributable to the action, and therefore requires another type of design that isolates the influence of other factors to determine whether the changes generated are direct results of the project's actions. However, under a contribution to results approach, which recognizes that changes are generated as a result of many factors in combination, we will seek to understand the contribution to changes from program actions in combination with other contextual factors that are outside of control. We start from the fact that the conditions of the context and the time of execution of the program were not conducive to generating the expected impact from the General Objective, which was to reduce vulnerability to seismic risk in the Metropolitan Area of Guatemala, through technical assistance and strengthening of specific capacities aligned with existing strategies and investments in disaster risk reduction. Impacts at this level are only generated with a longer-term vision, which would justify the implementation of a second phase of the program. However, the PREPARE program did manage to establish well-founded starting points to advance towards the achievement of this objective. It can be affirmed that the results obtained in the program have a high potential impact, since they highlight the importance and urgency of addressing an issue that the Guatemalan society has kept forgotten despite its social and economic relevance for the country. It also does so in a novel way, bringing to the surface problems that had never been considered within the management of seismic risk in the country, involving all the relevant sectors, and demonstrating with the implemented practice the need and possibility that all of them contribute to its management, each one from their knowledge and experience. In this way, the following can be identified as possible impacts of the program: • Establishing a new awareness of the issue of seismic risk: The data and findings revealed as a result of the vulnerability assessments of the built environment and the diagnosis of the weaknesses of the DRM system were able to sensitize the representatives of the participating organizations and entities, and are powerful instruments to draw the attention of public opinion to the urgency of seriously addressing this issue, so that it can press for the necessary political changes. • Updating the curricula of construction-related careers to include seismic risk FINAL EVALUATION OF PREPARE GUATEMALA | 24 and structural damage assessment among their contents: "In terms of education, this could have a great social impact, that each student, who between civil engineering and architecture at our university graduates about 200 per year, each one of them with this information could make these evaluations in their own community and consider the measures that can be taken. This would have a great impact on social awareness, because the generation that lived through the 1976 earthquake is already very old, and the current generations do not perceive this risk in the same way. ACADEMIC SECTOR INFORMANT • Establishing strategic alliances between academia and institutions of the DRR system: so that advanced students, through their certification as structural damage assessors, can integrate into volunteer groups and contribute the professional knowledge that these organizations do not have. • Establishing strategic alliances between research institutes and institutions of the DRM system: to implement information systems and lines of research on technical issues necessary for seismic risk management in the country's cities. • Incorporation of the data-driven planning approach within public agencies at the national and local levels to improve land use management and disaster prevention and response capacity. The program left in the institutions the tools and methodologies to do so, but there has to be the vision and political will to allocate the resources and make the necessary changes. • Influencing a shift towards replacing the current reactive approach with a preventive approach in the DRM system. CONCLUSIONS RELEVANCE Although the general objectives of the program are perfectly aligned with the mandates and attributions that the disaster risk management system in Guatemala has by law, and that respond to a certain need for timely and reliable information to improve the quality of prevention and emergency response, its relevance is weakened by the great distance that exists between the ideal functioning that this law prefigures and the current situation of the institutions that make up the system. The program set itself a very ambitious general objective, which depends on the operation of institutions without the necessary structures, resources, or mechanisms to take ownership of its products, and on the political will of authorities whose short-term vision prevents them from making the long-term changes that would be necessary to make it possible. The lack of perception of seismic risk in Guatemalan society results in a lack of pressure on authorities to work on its prevention. On the other hand, the long period elapsed from the identification to the implementation of the program weakened its articulation with the Barrio Mío project, in the framework of which it was planned to give use to its products and continuity to its actions. EFFECTIVENESS The program has been effective in achieving the specific objectives planned, but the contribution of these to the achievement of the overall objective formulated is still very incipient. The short space of two years foreseen for the program proved to be even more insufficient due to the delays in its initiation, the pandemic, and the change of government that forced to repeat the whole process of contact with the new authorities of the competent agencies. The products generated, of high technical quality, were adapted to the context with the participation of all the local actors involved and made available to the relevant institutions. However, the use of 25 | FINAL EVALUATION OF PREPARE GUATEMALA these products within institutional processes of risk prevention and emergency preparedness has not been achieved and is not guaranteed. The intervention's strategic approach, which, although not included in its logical framework, was the driving force behind all the actions and processes undertaken, emphasizes the need to strengthen participatory processes for governance with the inclusion of all stakeholders and the different levels of government, the use of local knowledge and expertise, and accountability to beneficiaries. This cross-cutting approach led to an unplanned but very important result for the sustainability of the program, which was the generation of multiple and novel alliances between previously unlinked sectors that, although not yet institutionalized, open new opportunities for the articulation of efforts. Due to this strategic approach, it was possible to undertake processes and elaborate products in addition to those planned, which were important to consolidate and give continuity to the results achieved. Among the best practices that contributed the most to the results achieved are: the quantification of losses and the economic and social impacts of a major earthquake, which is a powerful tool for social awareness and political advocacy; group work on specific technical issues and with total transparency in the results achieved, which overcame prejudices and mistrust among different actors; the demonstration of the importance of prevention actions and planning based on reliable and timely information, which opens new perspectives in a mainly reactive DRM system. EFFICIENCY The program was able to take advantage of and leverage the knowledge and experience of local partners in the formulation and implementation of actions. There was an efficient management of available resources, characterized by close monitoring and flexibility that allowed taking advantage of items not executed due to changes in the context, for the development of unplanned but important products to improve the effectiveness and sustainability of the implemented actions. However, the long delay in the start of activities, due to the health emergency caused by the pandemic, in which several of the main project partners were involved in, and the difficulty in the process of sensitizing the authorities of the participating institutions, influenced some of the most important products and processes generated at the end of the program, which reduced their possibilities of impact. PARTICIPATION The participation of national stakeholders was fundamental in defining and developing program outputs and outcomes and in adapting methodologies to the local context. Beneficiary consultation and accountability mechanisms functioned satisfactorily and were perceived as secure and accessible. Their needs and priorities were incorporated more effectively in training and capacity building actions, but not to the same extent in policy and planning, where incorporating the constraints of local partners would have meant making major changes to the original program design. SUSTAINABILITY The institutional sustainability of the results of the training and capacity building component is high, since it relies on ESEGIR, an organization that participated in its design and implementation and has the interest and capacity to promote and give the training continuity. Likewise, among the universities that participated in the intervention there is a new awareness of the importance of seismic risk management and an interest in including these pedagogical tools in the academic content of construction-related courses. However, the results and products of the policy and planning component are not guaranteed to be sustainable, since the institutions that in theory should take ownership of them and incorporate them into their processes do not have the mechanisms, resources, or capacities to do so. The strategic alliances that were formed during implementation could be an element of sustainability, but they are still very incipient and require direction and follow-up in order to be strengthened. The additional products and processes that were being generated to provide continuity and improve the FINAL EVALUATION OF PREPARE GUATEMALA | 26 appropriation of the results achieved were interrupted indefinitely, thus weakening the sustainability of the entire intervention. IMPACT The results obtained in the program have a high potential for impact, since they highlight, in a novel way and based on reliable data, the importance and urgency of addressing an issue that Guatemalan society has kept forgotten despite its social and economic relevance for the country. However, the impact of the program's results depends on its dissemination among public opinion and civil society, on mobilizing the political will of the authorities in the risk management sector to make the necessary changes and investments in the system, and on the materialization and institutionalization of the alliances that emerge among the various participating sectors. LESSONS LEARNED 1. Working on the public perception of seismic risk is essential to mobilize the political will necessary for prevention and preparedness efforts. 2. In order to formulate effective and sustainable actions, it is necessary to identify and incorporate the real limitations of the actors destined to appropriate them. 3. Face-to-face and participatory work on specific technical issues can break down mistrust and generate collaborative links between actors with different perspectives. 4. Combining theoretical distance training with practical on-site workshops enhances the learning, scope and coverage of training actions. PROGRAMMATIC RECOMMENDATIONS 1. Extend the program through a second phase where inter-institutional collaborations are deepened to work on specific issues and needs derived from the diagnoses made, such as, for example, debris management strategies. 2. Extend the vulnerability assessment study of the built environment to areas occupied by informal settlements, since the behavior of the structures existing in these areas is unknown, and include in the study the effects of landslides, a very important determinant in Guatemala City, where unstable slopes are increasingly occupied by this type of settlements. 3. Use the products and data generated in the PREPARE program to design an awareness￾raising strategy to increase the perception of seismic risk among the Guatemalan population and civil society, including abbreviated and graphically visualized versions of the reports for the authorities, presentations of results that can be the subject of discussion in professional and academic forums, and information pieces on their main findings, in non-technical language, to be disseminated by the media to wider audiences. 4. Deepen and institutionalize the relationship between DRR institutions and universities, in order to take advantage of their potential to compensate for the lack of qualified personnel that the public administration has to assess risks and damages caused by disasters, and to venture into some of the lines of research that result from the diagnoses carried out within the framework of the program. 5. Work with local engineers in the revision of the seismic risk model, so that it is adjusted to the structural and constructive characteristics of the buildings in Guatemala. 6. Work on political advocacy and humanitarian diplomacy with high-level officials from institutions of the DRM system and the national government, to institutionalize the instruments and processes created. 7. Deepen relations with local universities so that they include the building damage assessment courses implemented in the program into their construction and design courses. 8. Extend the vulnerability assessment of critical infrastructure to main roads, water reservoirs, electrical installations and other urban services that are indispensable for the functioning and governance of cities. STRATEGIC GUIDELINES 1. Capacity building actions should include the possibility of providing associated equipment for 27 | FINAL EVALUATION OF PREPARE GUATEMALA partner institutions, to ensure that the acquired capacities have the necessary support to put them into practice, and to generate greater ownership of the actions implemented. 2. Local partners should participate in the design and formulation of interventions to ensure that their conditions, needs, and priorities are incorporated so that they can appropriate the results obtained to make them sustainable. 3. In the project design phase, it is important to work on the development of a theory of change that takes into account the threats and weaknesses of the context and local partners, so that the objectives set are realistic and can be achieved within the stipulated resources and timeframe. Similarly, work must be done on the formulation of indicators that are adjusted to the program design, that go beyond the contractual indicators established by the donor, and that address the level of planned results and objectives, and not only the level of activities carried out, so that they are truly useful for the evaluation of progress achieved. 4. It is necessary to know the local partner institutions very well, not only in terms of their attributions and legal mandates on paper, but also in terms of their true capacities, resources, interests, and operating mechanisms. FINAL EVALUATION OF PREPARE GUATEMALA | 28 ANEXXES ANEXX I: INFORMATION SOURCES Table 4. Semi-Structured Interviews INFORMANT INSTITUTION OR ORGANIZATION National Program Manager Miyamoto International Consultant Global Communities Deputy Director of Preparedness School for Higher Education in Comprehensive Risk Management (ESEGIR) Researcher Institute of Natural Sciences and Technology, University Rafael Landívar Actuarial Advisor Guatemalan Association of Insurance Institutions Vice President Federation College of Engineers and Architects of Guatemala Head of the Infrastructure Department Department of Infrastructure Works, Mitigation Directorate, CONRED Professor School of Engineering of the San Carlos of Guatemala University Academic Coordinator School for Higher Education in Comprehensive Risk Management (ESEGIR) Professor School of Architecture, Rafael Landívar University Security Advisor Technical Secretariat of the National Security Council Head of Risk Management Municipality of Guatemala Head of Geophysics Department National Institute of Seismology, Volcanology, Meteorology and Hydrology of Guatemala (INSIVUMEH) Table 5. Documents Consulted Revised Technical Proposal USAID/OFDA Guatemala Earthquake Risk Reduction Program (PREPARE Guatemala) USAID/BHA PREPARE Guatemala Participant/Indicators Tracking Tool USAID/BHA PREPARE Guatemala Baseline Report USAID/BHA PREPARE Guatemala Semi-Annual Performance Report (October 2020-March 2021) USAID/BHA PREPARE Guatemala Semi-Annual Performance Report (April-September 2021) USAID/BHA PREPARE Guatemala Semi-Annual Performance Report (October 2021-March 2022) USAID/BHA PREPARE Guatemala Semi-Annual Performance Report (April-September 2022) USAID/BHA PREPARE Guatemala Evaluación del Riesgo Sísmico en la Ciudad de Guatemala 29 | FINAL EVALUATION OF PREPARE GUATEMALA USAID/BHA PREPARE Guatemala Risk Study Factsheet Spanish USAID/BHA PREPARE Guatemala, diciembre 2022. Informe de Evaluación Estructural Rápida de Posibles Riesgos Sísmicos en Edificios de Ciudad de Guatemala USAID/BHA PREPARE Guatemala, diciembre 2022. Manual de llenado para la Boleta de Evaluación Rápida USAID/BHA PREPARE Guatemala, diciembre 2022. Manual Docente Certificación en identificación de daños en estructuras ante eventos naturales con énfasis en la evaluación pre y post-sismo PREPARE Guatemala PREPARE Course participant list Gobierno de Guatemala/CONRED: Curso para certificación en identificación de daños en estructuras ante eventos naturales con énfasis en la evaluación pre y post-sismo PREPARE Guatemala Phase Two Concept Note PREPARE Guatemala Technical Working Group Work Plan FINAL EVALUATION OF PREPARE GUATEMALA | 30 ANEXX II: DATA COLLECTION TOOLS QUESTION SCRIPT FOR INSTITUTIONAL INFORMANTS 1. To what extent does the program respond to the priorities and mandate of institutions and organizations working in disaster risk reduction in Guatemala? 2. Was the distribution of human and material resources of the program sufficient and adequate to achieve its objectives? 3. To what extent has the program taken advantage of the internal experience and knowledge of organization and institutions in the country? 4. To what extent were stakeholders’ priorities and needs incorporated into the design of program activities? 5. Do you consider that the PREPARE program has contributed to achieving its overall objective of reducing vulnerability to seismic risk in Guatemala City? 6. What program actions have been the most effective in achieving the established objective? 7. What have been the main obstacles to achieving the program’s goals and objectives? 8. Has coordination and communication among agencies, institutions and organizations interested in disaster risk reduction increased as a result of participation in the project? 9. Have the tools developed by the project been put into practice by the stakeholders? Have they been or will they be used by the organization to which you belong? 10. What lessons can be learned from PREPARE Guatemala that can be replicated in other cases in the country and the region? 11. To what extent have program activities contributed to sustainable changes or results over time, and what factors have contributed to achieving this sustainability? 12. Have mechanisms been established to maintain the public-private partnership beyond the life of the program? 13. To what extent has the program contributed to improving the self-sufficiency of participating organizations and institutions? 14. To what extent are the changes achieved likely to have direct or indirect social, economic or political effects in the intervention area? 15. What recommendations could you identify for program improvement in a future program expansion phase? PROGRAM PROJECT MANAGER INTERVIEW SCRIPT 1. What is the role of the different stakeholders in this program? 2. How is the Neighborhood Approach incorporated into the ations of this program? 3. What was the reason for the low percentage of female participation (25.19%) in discussion spaces? 4. How did the limitations imposed by COVID influence the program and how were those limitations overcome in the field data collection activities for the development of the risk models? How did the limitations affect the training and capacity building activities? 5. What safe and accessible spaces and mechanisms for feedback and information from participants and partners were established, both informal and regular throughout the program? 31 | FINAL EVALUATION OF PREPARE GUATEMALA 6. Were modifications made to the work plan based on the results of the baseline report? 7. Was the end-line survey conducted as outlined in the program’s Monitoring and Evaluation Plan? 8. How did the political environment affect program implementation? 9. Was the Project extended through 2023 or did it end in 2022? 10. In objective 1, was work carried out in two zones or only in one zone? 11. Was the technical group created to follow up on the completion of the three specific Project objectives or for one in particular? What was the scope of the WG? Did the WG continue to meet after the closing of the program? 12. Rapid Damage Assessment (RDA) is not the same as Rapid Visual Screening (RVS). How do they differ, which corresponds to which OE, and who was trained in each? Does each type of assessment correspond to a different degree of structural qualification, higher, equal, or lower than 1.5? 13. Is the Rapid Visual Assessment Manual, which is among the documents from the 3rd WG meeting, different from the RDA? Who was trained in RVS? 14. Were formal cooperation agreements signed between government institutions and the private sector as advised by the baseline? 15. Were changes made to the training courses to adapt them to the low level of preparedness of officials from key program institutions identified in the baseline? 16. Are there any other follow-up reports after March 2022?