This publication was produced at the request of the United States Agency for International Development. It was prepared independently by an independent evaluation consultant, Spencer Salime, hired by Miyamoto International, Inc. FINAL EVALUATION PREPARE EJE CAFETERO JUNE 2023 FINAL EVALUATION PREPARE EJE CAFETERO | i FINAL EVALUATION OF PREPARE EJE CAFETERO June 2023 Award Number: 720FDA20GR00265 DISCLAIMER: The author’s views expressed in this publication do not necessarily reflect the views of the United States Agency for International Development of the United States Government. ii | FINAL EVALUATION OF PREPARE EJE CAFETERO TABLE OF CONTENTS ACRONYMS...........................................................................................................................IV EXECUTIVE SUMMARY........................................................................................................ V EVALUATION PURPOSE AND EVALUATION QUESTIONS......................................................................V BACKGROUND ........................................................................................................................................................V EVALUATION QUESTIONS, DESIGN, METHODS, AND LIMITATIONS.............................................. VI FINDINGS, CONCLUSIONS, AND RECOMMENDATIONS ..................................................................... VI EVALUATION PURPOSE AND EVALUATION QUESTIONS .........................................1 EVALUATION PURPOSE ........................................................................................................................................1 EVALUATION QUESTIONS...................................................................................................................................1 BACKGROUND.......................................................................................................................2 CONTEXT...................................................................................................................................................................2 RELEVANCE – PRE-PROGRAM PHASE ..............................................................................................................2 LOGIC MODEL ..........................................................................................................................................................2 PROGRAM DETAILS.................................................................................................................................................3 EVALUATION METHODS AND LIMITATIONS................................................................4 DATA COLLECTION...............................................................................................................................................4 DESK REVIEW OF SECONDARY SOURCES ...........................................................................................4 DATA COLLECTION FROM PRIMARY SOURCES................................................................................4 KEY INFORMANT INTERVIEWS.................................................................................................................4 FOCUS GROUPS ..............................................................................................................................................4 DATA ANALYSIS.......................................................................................................................................................5 CRITERIA .....................................................................................................................................................................5 LIMITATIONS .............................................................................................................................................................6 FINDINGS, CONCLUSIONS, AND RECOMMENDATIONS ............................................7 FINDINGS....................................................................................................................................................................7 IMPACT ...............................................................................................................................................................7 EFFECTIVENESS ..............................................................................................................................................20 EFFICIENCY......................................................................................................................................................25 SUSTAINABILITY............................................................................................................................................26 COORDINATION AND HARMONIZATION ......................................................................................31 LESSONS LEARNED ......................................................................................................................................33 LESSONS APPLIED .........................................................................................................................................35 CONCLUSIONS ......................................................................................................................................................37 RECOMMENDATIONS..........................................................................................................................................40 FINAL EVALUATION PREPARE EJE CAFETERO | iii RECOMMENDATIONS FROM PARTICIPANTS ...................................................................................41 ANNEXES...............................................................................................................................43 ANNEX I: TIMELINE...............................................................................................................................................43 ANNEX II: EVALUATION STATEMENT OF WORK....................................................................................44 ANNEX III: EVALUATION METHODS AND LIMITATIONS .....................................................................44 ANNEX IV: DATA COLLECTION AND ANALYSIS TOOLS.....................................................................44 ANNEX V: SOURCES OF INFORMATION.....................................................................................................53 ANNEX VII: EVALUATION TEAM MEMBERS.................................................................................................56 iv | FINAL EVALUATION OF PREPARE EJE CAFETERO ACRONYMS APR Administrative and Planning Region BHA Bureau for Humanitarian Assistance CAEF Colombian Association of Engineering Faculties CGS Colombian Geological Survey DRM Disaster Risk Management DRR Disaster Risk Reduction DRMDC Disaster Risk Management Departmental Council DRMMC Disaster Risk Management Municipal Council ICS Incident Command System INSARAG International Search and Rescue Advisory Group LoC Letter of Commitment Miyamoto Miyamoto International MoU Memorandum of Understanding NCE No Cost Extension S&R Search and Rescue UNGRD National Unit for Disaster Risk Management USAID United States Agency for International Development USAR Urban Search and Rescue VSOM Vulcanological and Seismological Observatory of Manizales FINAL EVALUATION PREPARE EJE CAFETERO | v EXECUTIVE SUMMARY This evaluation report examines the PREPARE program in the Eje Cafetero region of Colombia, which aimed to strengthen regional risk management and response capacity of departmental and municipal governments and the private sector. The program was implemented by Miyamoto International in collaboration with political entities, first responder groups and the private sector. EVALUATION PURPOSE AND EVALUATION QUESTIONS In line with the original purpose of the external evaluation of the PREPARE Eje Cafetero program, the evaluation intended to measure and document the results and impact of the program, including lessons learned to inform future risk management and response planning in Colombia and elsewhere. The evaluation team took into consideration the objectives provided by the Miyamoto program team as follows: 1) Evaluate the effectiveness and impact of program activities in accordance with established objectives and indicators, 2) Identify and document lessons learned, as well as unplanned positive outcomes and opportunities for improvement, identification of program impact and measures of program sustainability, and 3) Assess the level of involvement of response organizations with the aim of securing sustainability of the program. Considering the above, the document answers more detailed questions such as: • To what extent were the overall goal and objectives of the program achieved? • What were the main challenges in achieving the program's goals and objectives? • What program activities were most effective in achieving the stated objectives? • To what extent did outputs and outcomes include participation and accountability to the affected population, including gender considerations? • To what extent did activities contribute to sustainable changes or outcomes and what factors contributed to achieving sustainability? • Did coordination and communication between agencies and institutions increase because of participation in the program? • Were the tools developed under the program put into practice by the stakeholders? BACKGROUND Miyamoto International implemented the PREPARE Program in the Eje Cafetero region of Colombia, which sought to strengthen the policies and practices of risk management, risk reduction and emergency response institutions, and to strengthen community, inter-institutional and inter-regional relations. The program assisted departmental and municipal governments with the collection, organization, analysis, and application of disaster risk data to improve risk management and response policy and planning, specifically in relation to seismic hazard. The program goal was to strengthen the regional risk management and response capacity of departmental and municipal governments and the private sector in the Eje Cafetero. After thorough, highly participative, and long-term preparatory work during which all relevant stakeholders in the region were consulted and relevant documents analyzed, the program logic was planned to respond to the identified needs around three sectors: 1) Policy and Planning, 2) Capacity Building and Training, and 3) Public Private Partnerships. The following were the identified needs: 1) Risk management had not been adequately incorporated into local policy or planning, 2) There was insufficient knowledge of seismic risk to effectively prepare for and respond to the next earthquake, and 3) Government had not adequately engaged with academia, the business community, NGOs, and community groups, in risk management and response planning, Responding to the above needs and to the regions’ vulnerability due to high seismicity and multi-threat events such as landslides, floods, volcanic activities, windstorms, etc. the following objectives were set: vi | FINAL EVALUATION OF PREPARE EJE CAFETERO Objective 1: Enhance the availability and coverage of seismic exposure, vulnerability and risk data for risk management policy and planning, Objective 2: Strengthen the capacity of disaster authorities to understand and apply seismic risk data for improved policy, planning and implementation outcomes, Objective 3: Strengthen the capacity of municipal agencies, USAR and other first responders to plan and implement effective response operations, and Objective 4: Strengthen regional cooperation in risk management and response through joint planning, information and resource sharing. EVALUATION QUESTIONS, DESIGN, METHODS, AND LIMITATIONS In line with the original purpose of the external evaluation of the PREPARE Eje Cafetero program, the evaluation intended to measure and document the results and impact of the program, including lessons learned, to inform future risk management and response planning in Colombia and elsewhere. The evaluation team took into consideration the objectives provided by the Miyamoto program team as follows: 1) Evaluate the effectiveness and impact of program activities in accordance with established objectives and indicators, 2) Identify and document lessons learned, as well as unplanned positive outcomes and opportunities for improvement, identification of program impact and measures of program sustainability, and 3) Assess the level of involvement of response organizations with the aim of securing sustainability of the program. The evaluation methodological design focused on the results obtained and conducting an assessment according to the ToR criteria around impact, effectiveness, efficiency, sustainability, participation, gender aspects, and lessons learned. A comparison of qualitative data from both primary and secondary sources of information was carried out. For data collection, a desk review of secondary sources was followed by data collection from primary sources through face-to-face and online key informant interviews and focus group discussions. These semi-structured interviews followed a scripted set of questions (attached in the annex), encouraging interviewees to add openly other aspects and allowing them to complement each other. The results of the interviews, focus groups and review of secondary source documents were analyzed according to the given criteria. The different sources supported the triangulation of the information to draw objective conclusions. FINDINGS, CONCLUSIONS, AND RECOMMENDATIONS The report highlights that the program had a significant impact in the region, enhancing the preparedness capacity of participants and providing high-quality equipment tailored to their needs. The professionalism, expertise, and dedication of the program team were positively acknowledged, along with the significant improvement of regional and inter-institutional collaboration. Among the program achievements are the unification and levelling of risk management knowledge, improved communication and collaboration among response teams, and the preparation of important technical documents, such as seismic exposure models and a repository of all relevant data/documents. Additionally, a plan was developed to deliver training using an innovative methodology and an important achievement was the preparation of the knowledge strengthening plan. The program showed good results with regards to the indicators in terms of efficiency, effectiveness, impact, inclusiveness, and cooperation, notwithstanding some objectives were easier to achieve than others. While there were significant achievements in some indicators, such as participation in discussions on national risk reduction strategies and training of people in specific areas, there were some targets which were more challenging and time consuming to reach, such as the inclusion of the updated risk management plans and response strategies in departmental and municipal level policies. There are letters of commitment from mayors, governors, and office directors to conclude such. The report also emphasizes the importance of addressing risks beyond seismic events, such as mass movements, floods, and windstorms. It is recommended to continue the program in a second phase, FINAL EVALUATION PREPARE EJE CAFETERO | vii broadening the focus to community preparedness and levelling different groups in terms of technical equipment. Some recommendations include strengthening the psychosocial component through additional training, facilitating knowledge exchange at national and international levels, and improving communication infrastructure for more adequate response capabilities. The need for a detailed exit strategy and ensuring sustainability of program results through commitments at national, departmental, and municipal levels is also highlighted. The program successfully achieved the overall goal of strengthening disaster response and risk management in the Eje Cafetero region. Activities that were most effective included needs assessment at the initial phase, risk data repository creation, capacity building both at operational, knowledge and logistical level, knowledge strengthening strategy drafting and joint planning within the region. The involvement of national stakeholders shaped program outputs and outcomes, improving coordination and communication between agencies and institutions. Mechanisms for maintaining public-private partnerships were established, but engaging the private sector remained challenging. The implementation of the PREPARE Eje Cafetero program provided valuable lessons that can be applied to future programs. The COVID-19 pandemic highlighted the importance of flexibility and adaptability in program planning and implementation, emphasizing the need for contingency plans and alternative scenarios. Collaboration and coordination between government entities, emergency groups, and other actors were key lessons learned, fostering a regional working environment, and improving connection between stakeholders. Selecting the right participants and people for positions was crucial, as the success of the program often relied on the leaders' commitment and expertise. Adaptability, collaboration, effective communication, and building on existing frameworks were highlighted as essential for addressing seismic risk and improving disaster preparedness and response. The program's impact was demonstrated through the progress made by USAR groups in the national accreditation process, leading to a more effective emergency response. Lessons also included the need for improved monitoring, regional procurement policy, and hiring external consultants with specific ToRs. Participation, accountability, and gender considerations were emphasized as positive results. There was effective coordination between the Volcanological and Seismic Observatory of Manizales, part of the Colombian Geological Service, academia, and partners, ensuring that relevant information for decision-making was generated and disseminated. For example, carrying out periodic drills that contribute to improving response capacity and coordination between different institutions. In conclusion, the PREPARE program made a significant impact on disaster risk management in the Eje Cafetero region, enhancing emergency preparedness and response. Despite the achievements, challenges such as resource constraints, lack of political support in some respects, high rotation of staff, conflicts of interest and need for more training are recognized. Continued work in the region is recommended, addressing these areas of improvement to ensure more effective risk management in the longer term. Recommendations for improvement from the evaluation team and the participants include adapting technical language, widening the program's focus, levelling different groups in terms of equipment, and addressing additional risks beyond seismic risk. Requests were made for additional training, particularly in the psychosocial component, and knowledge exchange opportunities. Continuing the program in a second phase was recommended to address the region's high level and variety of risks, with a focus on strengthening the psychosocial component, providing more trainings, and facilitating knowledge exchange for long-term success and sustainability. It is important to highlight that the program was originally developed to have a logical multi-year sequence of activities to be covered in around 5 years, that is, during two or three phases of the program. Hence in this first phase the highest priority activities were developed considering resources and time assigned by the donor. FINAL EVALUATION PREPARE EJE CAFETERO | 1 EVALUATION PURPOSE AND EVALUATION QUESTIONS EVALUATION PURPOSE In line with the original purpose of the final external evaluation of the PREPARE Eje Cafetero program, the evaluation intends to measure and document the results and impact of the program, including lessons learned, to inform future risk management and response planning in Colombia and elsewhere. The evaluation team took into consideration the following objectives provided by Miyamoto program team: 1. Evaluate the effectiveness and impact of program activities in accordance with established objectives and indicators. 2. Identify and document lessons learned, as well as unplanned positive outcomes and opportunities for improvement, identification of program impact and measures of program sustainability. 3. Assess the level of involvement of responding organizations with the aim of securing sustainability of the program. The audience can be both USAID and Miyamoto staff as well as participants and direct beneficiaries of the program and any other stakeholders. EVALUATION QUESTIONS Considering the above, the document will answer the following questions: 1. To what extent were the overall goal and objectives of the program achieved? 2. What were the main challenges in achieving the program's goals and objectives? 3. What program activities were the most effective in achieving the stated objectives? 4. To what extent did the involvement of national stakeholders shape and help to develop program outputs and outcomes? 5. To what extent did the outputs and outcomes include participation and accountability to the affected population, including gender considerations? 6. How did specific intended and unintended changes in program activities positively or negatively affect the program? 7. To what extent did program activities contribute to sustainable changes or outcomes and what factors contributed to achieving sustainability? 8. How was the exit strategy and Sustainability Plan set up? 9. Has coordination and communication between agencies and institutions increased because of the program? 10. Have mechanisms been established to maintain the public-private partnership beyond the life of the program? 11. Have the tools developed under the program been put into practice by the stakeholders? 2 | FINAL EVALUATION OF PREPARE EJE CAFETERO BACKGROUND CONTEXT The Eje Cafetero region of Central Colombia is characterized by high seismicity due to its location at the convergence of the Nazca, Cocos, and Pacific tectonic plates as well as highly vulnerable to multi￾threat events such as landslides, floods, volcanic activities, windstorms, etc. Although after the 1999 Eje Cafetero Earthquake some authorities have made progress with the documentation of seismic risks and started to prepare contingency plans to map zones based on risk exposure of the population for future earthquakes, there remain significant gaps in risk management knowledge and capacity. The following findings were highlighted in 2019 during a conference commemorating the 20-year anniversary of the Eje Cafetero Earthquake, as well as in a “State of the Art” review of 167 scientific studies: • Risk management had not been adequately incorporated into local policy or planning, • There is insufficient knowledge of seismic risk to effectively prepare for and respond to the next earthquake; and, • The government had not adequately engaged with academia, the business community, NGOs, and community groups, in risk management and response planning. Due to the finding above, the PREPARE program intended to respond to two major issues in the region: 1) Disaster risk management and response authorities often lack access to essential seismic hazard and risk data and the technical knowledge and skills to use existing data needed for effective planning, and 2) Urban Search and Rescue (USAR) personnel and other first responders also require training, equipment and access to effectively carry out response operations. RELEVANCE – PRE-PROGRAM PHASE Prior to program implementation, Miyamoto started to organize and facilitate regional and municipal discussions and planning workshops to explore gaps and deficiencies in the risk management and emergency response system. Representatives from the Departments and the Municipalities, risk management specialists from academia, USAR and other first responders, community-based organizations and the private sector took part in these workshops which were in line with the risk management strategic framework developed by the National Disaster Risk Management Unit (UNGRD) in accordance with National Law 1523 of 2012. A detailed list of variables was collected that define and measure gaps in risk management and response capacity, as well as capacity building priorities and resource requirements by Department. As a next step Miyamoto worked with partners in Eje Cafetero to develop a framework to address the above issues. The program itself was developed together with the regional Colombian government counterparts, first responder entities and the private sector, and in line with the national level work and legal framework. One of the most important elements of the success of the program is based exactly on the above approach: a highly detailed, timely and inclusive preparation process done together with all the stakeholders. Several participants mentioned that already during the previous program implemented in Pasto there had been initial conversation with the region and the different future participants lobbied to have the program implemented in the region. “We got to know the program in Pasto in 2017, they told us about the Pasto microzoning program, and there we met Miyamoto. In 2018 we already started to work with Jaime, to see if the program could be brought to our region” - Juliana Escandon, Departmental Disaster Risk Management Unit of Quindío. LOGIC MODEL In line with the above, the program logic was planned to respond to the identified needs around three sectors: FINAL EVALUATION PREPARE EJE CAFETERO | 3 1) Policy and Planning; 2) Capacity Building and Training and 3) Public Private Partnerships With the following objectives: Objective 1: Enhance the availability and coverage of seismic exposure, vulnerability and risk data for risk management policy and planning. Objective 2: Strengthen the capacity of disaster authorities to understand and apply seismic risk data for improved policy, planning and implementation outcomes. Objective 3: Strengthen the capacity of municipal agencies, USAR teams and other first responders to plan and implement effective response operations. Objective 4: Strengthen regional cooperation in risk management and response through joint planning, information, and resource sharing. PROGRAM DETAILS The United States Agency for International Development’s Bureau of Humanitarian Assistance (USAID/BHA) PREPARE Eje Cafetero Program (Award # 720FDA20GR00265) officially launched September 30, 2020, with an original end date of September 29, 2022. However, it received a 9- month no-cost extension with a new end date of June 30, 2023. Miyamoto International implemented activities in the Eje Cafetero region corresponding to the departments of Caldas, Risaralda, and Pereira. The total program budget for PREPARE Eje Cafetero was $1,500,000. The program affected 619 beneficiaries directly and 1,306,026 beneficiaries indirectly. The direct beneficiaries of the program included USAR members, municipal and departmental government staff involved in DRR/M, and the private sector. The indirect beneficiaries correspond to 50% of the estimated population in the Eje Cafetero at the end of the program, a population at high seismic risk. 4 | FINAL EVALUATION OF PREPARE EJE CAFETERO EVALUATION METHODS AND LIMITATIONS To facilitate a rigorous and independent evaluation, the team has used different methods, as outlined below, thus making use of multiple evidence that facilitated a comparison of qualitative data from both primary and secondary sources of information. The evaluation was based on the given questions from the ToR and the findings were structured around the following main criteria: Impact, Effectiveness, Efficiency, Sustainability, Participation, Gender aspects and Lessons Learned. DATA COLLECTION DESK REVIEW OF SECONDARY SOURCES The review and analysis of relevant documents provided insight into the implementation logic of the program as well as evidence on the results. Several products of the program were documents, which were the specific outputs of some of the objectives. The revision of the documents also supported the planning phase and creation of interview questions as well as the identification of the findings. The list of documents revised and respective answers to evaluation questions according to each will be detailed below. DATA COLLECTION FROM PRIMARY SOURCES A hybrid data collection methodology was used: some interviews were done online, and some interviews and focus groups were conducted face-to-face during field visits. The sample: the group of interviewees was established according to the suggestions of the program team, considering their major role in the program. For the purpose of recording the information obtained, there were two consultants present in the interviews/focus groups, one conducting the interview and one recording (taking notes, making recordings). Online calls/videos were recorded, and sound was recorded during face-to-face interviews. On every occasion prior consent was obtained from the participants and the material was only for the exclusive use of the consultants to write the evaluation. All material was returned to Miyamoto at the end of the contract. KEY INFORMANT INTERVIEWS Following a semi-structured protocol, it was expected to obtain relevant information from the different actors on their role, the effectiveness, impact, relevance, and efficiency, as well as to determine from their experiences, the degree of articulation between the different program actors. The interviews followed a script of questions, but at the same time the interviewers were open to new information and additional aspects shared by the interviewees. Responding to the design of the interview guide, the answers of the different actors to the same question facilitated the triangulation of information and the respective validation of preliminary findings, as well as the identification of unforeseen results. FOCUS GROUPS The application of this method enabled the evaluation team to collect information on the experiences, perceptions, and opinions that the interviewees had about the training, products received and their role in the activities. These conversations also helped to achieve a better understanding of the process and impact of program implementation. By using a semi-structured questionnaire and depending on the components of the intervention, as well as the different roles played by each focus group member, different opinions were heard which fed into the answers searched for the evaluation questions. Therefore, focus groups proved to be a useful method to promote analysis of program effectiveness, thus facilitating the identification of impacts and/or unintended outcomes and sustainability. FINAL EVALUATION PREPARE EJE CAFETERO | 5 The topics presented to the focus group followed a scripted set of questions, but participants were encouraged to freely express their opinions, which gave a much broader understanding of the successes and difficulties of the program. Interviewers attempted to have an honest and open dialogue, encouraging interaction between group members, allowing them to complement each other. DATA ANALYSIS The results of the interviews, focus groups and review of secondary source documents were analyzed according to the criteria described below. The different sources supported the triangulation to determine the specific activities and strategies that achieved definable results (positive or negative), or that participants or key informants and documented results reached similar conclusions from different perspectives. For a rigorous analysis a tabulation method was used in which all the data gathered was included into tables for easier analysis and quantification of the results. CRITERIA The criteria used was in line with the ToR for the evaluation and was based on the following concepts: 1. Impact: measures the results and impact of the program according to the program objectives to identify positive or negative changes, lessons learned and best practices. 2. Effectiveness: reviews the level of achievement of specific objectives and expected results compared to the original plan. 3. Efficiency: measures the efficiency of the program in terms of resource utilization and costs. 4. Sustainability: reviews the sustainability of the program and the ability of the participants to continue with the results in the long term. 5. Participation: measures the coordination between the different actors and the participation, empowerment, and accountability of the target population during the program implementation process. 6. Gender aspects: assesses the extent to which the program processes, products and trainings have included gender aspects. 7. Lessons learned: collects lessons learned, suggestions for improvement and feedback for future similar programs. STAKEHOLDER METHODOLOGY DATE TIME Realizado Posicion MANIZALES, CALDAS 29 Mayo Project Management Team Interview 29 Mayo 9:00-10:30 Jaime Eraso OGR CALDAS Focus group 29 Mayo 11:00-12:30 Felix Giraldo Director Gestión del Riesgo Caldas OGR MANIZALES, Univ Focus group 29 Mayo 14:00-15:30 Alexa Morales Directora Gestión del Riesgo Manizales William Gaviria Universidad de Manizales Marta Yara Trabajadora social Bomberos Comandante Alvaro Vasquez Ingeniero civil USAR BV MANIZALES Interview 29 Mayo 16:00-17:00 Nathalia Valencia Comandante/Lider Grupo USAR 30 Mayo USAR CR CALDAS Interview 30 Mayo 9:00-10:00 Daniel Fernando Cano Mejia Gestor de Conocimiento de Reducción de Riesgo de Desastres PEREIRA, RISARALDA USAR DOSQUEBRADAS Interview 30 Mayo 11:30-13:30 Comandante Lina Marín Comandante/Lider Grupo USAR USAR OFICIALES PEREIRA Interview 30 Mayo 16:30 - 17:30 Alejandro Arango Director BO Pereira OGR PEREIRA, Univ. Focus group 30 Mayo 15:30-17:00 Alexander Galindo and team Director Gestión del Riesgo Pereira DIGER 31 Mayo USAR DC RISARALDA Interview 31 Mayo 9:00-10:00 Alfredo Muñoz Director Defensa Civil Risaralda USAR DC RISARALDA Jairo Rojas Operativo Defensa Civil Risaralda USAR BV CHINCHINA Interview VIRTUAL31 Mayo Andres Gomez Lider USAR OGR RISARALDA Focus group 31 Mayo 11:00 - 12:00 John Jairo de la Pava Coordinador operativo CDGRD Risaralda ARMENIA OGR Quindio Focus group 31 Mayo 14:00-15:00 Monica Camacho Directora Gestión del Riesgo Quindio Juliana Escandon Operativa Gestión del Riesgo Quindio Pablo Cesar Defensa Civil Ariel Ospina Cruz Roja Project Team Faber Mosquera Project Coordinator OGR Armenia Focus group 31 Mayo 17:00 -18:00 Javier Velez Director OMGERD Universidad Gran Colombia Ximena Cifuentes Decana de la Facultad de Ingeneria 1Junio USAR DC QUINDIO Interview 1 Junio 9:00-10:00 Mauricio Ruiz Rodriguez Director Defensa Civil Quindío Pablo Cesar Integrante grupo USAR DC Quindío USAR CR QUINDIO Interview 1 Junio 10:30-11:30 Cristhian Martinez Director seccional Jacobo Franco Lider grupo USAR CR Quindío 6 | FINAL EVALUATION OF PREPARE EJE CAFETERO LIMITATIONS Financial evaluation – hence efficiency of revision - was beyond the scope of this consultancy and the evaluator team did not see the budget or financial reports. The evaluator team could only interview the leaders of the USAR groups and did not have the possibility to talk to USAR members, hence the views were more from the management. MAIN ACTIVITIES TEAM SCHEDULE Level of Effort Anna Wrochna Alejandro Gómez Total Days Days Days Meetings and consultation process/evaluation design 3 1 4 May 15-22 Design of data collection tool and work plan 3 1 4 May 15-22 Preparation of interview schedule 0 1 1 May 15-22 Documentary review and data collection: Program agreement, award document, M&E plan, all reports submitted to USAID/BHA and specific output reports, and partner meeting minutes 2 4 6 May 23-28 Interviews and/or online secondary data collection 2 2 4 May 23-28 Individual Interviews 4 4 8 May 29 – June 1 Draft of Final Evaluation Report 3 3 6 June 2 – 12 PowerPoint presentation of preliminary key facts 1 1 2 June 13 Feedback and adjustments to draft report 2 2 4 June 13-20 Translation 3 3 June 21-23 Final version of Final Evaluation Report 2 2 4 June 23 Total 25 21 46 FINAL EVALUATION PREPARE EJE CAFETERO | 7 FINDINGS, CONCLUSIONS, AND RECOMMENDATIONS FINDINGS IMPACT The impact section aims to identify the results and impact of the main activities according to the program objectives to measure positive or negative changes, lessons learned and best practices. In addition, this section responds to the respective questions of the Terms of Reference such as: • To what extent have the overall goal and objectives of the program been achieved? • What have been the main challenges in achieving the program's goals and objectives? • Have the tools developed under the program been put into practice by the stakeholders? To identify the level of impact of the program, both secondary sources such as reports and documents were assessed as well as the information from the interviews and participant feedback. Both primary and secondary sources confirmed that the impact of the program is high. This is based on the revision of the program products and the information received from the 24 interviews/focus groups conducted. The list of documents can be found in the Annexes. The interviews and individual personal observations revealed that all participants were satisfied with the program’s result and most of the interviewees expressed a very high level of satisfaction. According to the questionnaire, 82% of the respondents qualified the impact achieved as 10 and 11% as 8 and 9 both. “PREPARE has been the largest program that has strengthened the most of our department in the last 20 to 25 years, a great impact and we feel fortunate to be part of it. The program’s strength, that regardless of the color of the uniform, it prepared all institutions, hence increased the response capacity of the department itself. PREPARE came to give the final push to the teams, tools that we did not have and important training. It increased the resilience of the teams; all the relevant programs of the Red Cross got strengthened.” - Daniel Fernando Cano Mejía, Red Cross Caldas Objective 1: Enhance the availability and coverage of seismic exposure, vulnerability, and risk data for risk management policy and planning The first objective is part of the Policy and Planning Sub-sector of the program, which was essential for the first phase of the program as it built the base for a well-planned, successful program. The high-level impact of the program strongly relates to a very detailed and well-planned preparatory work. Dedicating the first phase of the program to a detailed analysis on needs and current capacities 78% 11% 11% To what extent the main objectived and goal of the program has been reached from 1 to 10? 1 9 8 8 | FINAL EVALUATION OF PREPARE EJE CAFETERO of the participants and building up the program’s specific trainings, documents, and program activities according to this information gave an enormous value to the program itself as it responded to exact needs. The evaluation team considers this approach as one of the key-success factors of the program. Another important factor was the selection of those participants who were committed and showed previous results. With this approach it became possible to avoid losing resources on investments which would not give the expected results (e.g., groups of participants not completing the activities as required) and strongly increased the effectiveness of the program. This objective in the initial phases collected, organized, and classified information on the current capacities and needs of possible program participants in a database, which was finalized in a baseline study. Gaps and weaknesses in risk knowledge were identified, which was followed by identifying a strategy to strengthen knowledge of seismic risk and DRM based on the analysis of the information collected. The strategy included detailed steps and activities necessary to fill those gaps with roles and responsibilities of the entities involved identified with schedules of activities and budgets. The first objective also included the preparation of a variety of documents, such as plans, strategies, policies, for example the knowledge repository. The value of this platform is that it has gathered the available data from several different sources which was not accessible to everyone. “PREPARE has provided us with information, knowledge, experiences, documents, such as through the repository. It compiled information from many entities that we might not have had access to as well as the work of the academia, which we might have, but would never have had the time to read 500 pages. PREPARE made an adjustment of what was required, with accessible info, and now that we are updating the departmental plan for risk management we can include those documents, e.g., seismology analysis” – John Jairo, Departmental Council for Disaster Risk Management, CDGRD, Risaralda The work was well prepared and methodologically planned out as the first documents were drafted, which gave the base for the repository itself: • “Research on national, international and regional repositories in the coffee-growing region” • “Plan for the administration and management of the repository of seismic risk knowledge and seismic risk management in the coffee-growing region of Colombia” • “Report on the design of the repository and database” The repository was linked to the national platform of UNGRD and is available for anyone at the following link: http://repositorio.gestiondelriesgo.gov.co/page/partners. This had a very important impact on the program as well as its sustainability. Due to the turnover of UNGRD staff, it has not been possible to notify the public yet. The main objective was a regional level repository of knowledge, which will be managed, maintained, and updated by the municipal and departmental risk management offices, as well as by the RAP Eje Cafetero in its regional risk management plan. In addition, the plan for the management and administration of the repository was provided to facilitate its integration and operation, and the linkage at the national level was an important added value. “The importance of an articulated work and the gathering of all technical resources, documentation and others in one place, facilitates decision making.” – Javier Velez, Municipal Office for Disaster Risk Management of Armenia In addition, the Program managed to create Seismic Hazard Exposure Models for Armenia and Calarca, with the support and collaboration of the Gran Colombia University of Armenia, which is a good example of inter-institutional collaboration. Students were trained by the program in basic concepts of seismic risk, structural vulnerability and techniques for the collection and generation of inventories for seismic exposure models. The information on the structural characteristics of 10,000 buildings in Armenia was collected by groups of architecture and environmental engineering students. FINAL EVALUATION PREPARE EJE CAFETERO | 9 In Pereira, information on the structural characteristics of buildings to develop the Pereira exposure model was collected with the support of 30 civil engineer students from the Libre University of Pereira. A seismic microzonification study was identified in the initial needs analysis and the program supported the Colombian Geological Service in its preparation to improve building design parameters and update existing information. These studies are of great relevance for understanding and preventing the effects of earthquakes on structures, which demonstrates an adequate identification of the needs and challenges in seismic risk management. The impact and sustainability of these two activities will be shown when these are added to the national repository, and the exposure models to the National Seismic Risk Models of ACOFI which is understood to be in progress. The cooperation with the Colombian Geological Service (CGS) and the UNGRD were instrumental to the success of the program. There were long-term relationships with these institutes which started in Pasto, which shows the professional work and efficiency of Miyamoto. The collaboration between the GSC and Miyamoto enabled the generation of specialized information and practical tools, such as the calculation of debris volumes, demonstrating effective knowledge transfer and practical applications. The studies related to seismic risk management carried out in collaboration made progress in building damage assessment and early response capacity, showing effective implementation of preventive measures. During their interview, the CGS praised how useful the cooperation with Miyamoto was, as it strengthened collaborations, helped make progress in assessing damage to buildings, gave professional support in the microzonification study process involving the universities and also supported its efforts to fulfill Law 1523. “This law establishes the management of information, but there is currently no public system to ensure its proper storage and access. Miyamoto's participation in the PREPARE program is highly valued, as it shows that it is possible to move forward in this field.” – Miguel Mora, Colombian Geological Service Under this Objective, 14 disaster risk management and emergency response instruments were modified and updated as a result of the program: 3 municipal disaster risk management plans (Manizales, Pereira, and Armenia); 3 departmental disaster risk management plans (Caldas, Risaralda, and Quindío); 3 municipal emergency response strategies (Manizales, Pereira, and Armenia); 3 departmental emergency response strategies (Caldas, Risaralda, and Quindío); the RAP regional risk management plan; and the national S&R plan (UNGRD). The regional RAP plan included the knowledge repository and the protocols for damage assessment, psychosocial assistance, and radio communications. The table below shows the fulfillment of the activities in relation to the knowledge strengthening strategy, where “x” is planned and “ ” is completed. This table is part of the regional Knowledge Strengthening Strategy created by the program. The program collected and analyzed information, as part of the plans. In the table there are activities, which were not part of the program, however, will be fundamental for the authorities to implement to have an adequate knowledge of risk and risk management. The program provided a step-by-step description of how to design and implement these, a chronogram, an approximate budget for each one, and the Letters of Commitment from the directors of the institutions, and municipal and departmental mayors. The program supported the region with exposure modelling and damage assessments during the NCE (including the exposure models for Armenia and Calarca), however, as mentioned above the responsibility to develop them in the short, medium, or long term lies with the authorities. Several of the products below were included in a proposal to the donor for a follow up phase. IMPLEMENTATION COMPLEMENTATION ACTUALIZATION ID ACTIVITIES Q-A R-P C-M Q-A R-P C-M Q-A R-P C-M 10 | FINAL EVALUATION OF PREPARE EJE CAFETERO 1. Exposure model x x 2. Damage assessment x x x 3. Debris management x x x 4. Evaluation of essential buildings x x x x x x x x 5. Regional, departmental, and municipal response protocols x x x 6. Updating and institutionalization of hazard and risk information and data x x x Q-A : Quindío-Armenia R-P: Risaralda - Pereira C-M: Caldas - Manizales Challenges: One of the most difficult objectives and indicators to reach in the program was: “Risk data is incorporated into municipal and departmental risk management and response policy and plan” for different reasons. First, it depended on the political leadership and their commitment, secondly these processes take quite a long time, which would exceed the established program timeline. The CMGRD of Pereira and the RAP Eje Cafetero have already implemented it. The other Departmental and Municipal Councils have letters of intent where their inclusion is corroborated. “We are in the process of updating our emergency response policies, we will have specific protocols for each type of scenario, and a municipal risk management plan. We have one of the products already in place, as for the area of seismic knowledge, seismic risk zoning – results from PREPARE - has been included in the municipal plan.” - Pereira Municipal level The CGS also confirmed that cooperation with political entities on these topics is often difficult as they do not recognize the importance of the issue for their political agenda. They also highlighted that the existence of conflicting interests between different universities can lead to conflicts and hinder the effectiveness of collaboration in the implementation of studies and activities. However, thanks to PREPARE, the cooperation with University Gran Colombia, University of Pereira and University of Quindío was successful in their provision of students trained by the program and GCS to gather data and prepare the Exposition model of Armenia, Calarca and Pereira which will feed the national model. Moreover, the lack of technical knowledge at these levels is always a serious issue they are struggling to deal with, hence cooperation with universities and the training of students was helpful to overcome this obstacle. GCS confirmed that they requested to continue this work in other cities in the Eje Cafetero region, however, Miyamoto had to reject this request due to limited resources as this activity took place during the no-cost extension period. During the NCE, the program conducted a process of homogenization and standardization at the regional level of the existing damage assessment tools and methodologies, responding to the assessed gap identified in the initial diagnosis, that there was a great quantity and diversity of tools and methodologies that exist in the Eje Cafetero region for the evaluation of damage to buildings caused FINAL EVALUATION PREPARE EJE CAFETERO | 11 by natural phenomena. Joint regional workshops were held to develop a joint design of a homogenized tool with standardized parameters for the entire region. The program also generated user manuals for training and field work, and training plans for permanent replication by the authorities. Objective 2: Strengthen the capacity of disaster authorities to understand and apply seismic risk data for improved policy, planning and implementation outcomes forms part of the Policy and Planning Sub-sector. Successes and challenges: This objective was more challenging to measure by specific indicators. It took considerable effort and time for the team to inform all the relevant actors constantly and rigorously, especially at the political level, before, during and after each respective activity. This was important for the relevance, impact, and sustainability of the program as the approval and buy-in of the political decision makers was needed to plan for the next phase based on their feedback. This effort can be well observed by the high % of achievement of this specific indicator (161% at the final report): “Number of people participating in discussions regarding national risk reduction strategies as a result of the program”. One of the most important elements of the impact, tangible result and sustainability of the program, and specifically this objective, is that some of the risk data is in the process of being incorporated into municipal and departmental risk management and response policies and plans. While two were already signed, the others are committed to be included by letters of commitment. In addition, activities from other components, such as different documents e.g., repository, databases, regional risk management plan are all related to this objective as these were planned, approved, and hopefully will be used by the municipal and departmental level political decision makers. One main challenge that this program had to face, was the high rotation of the personnel in these positions as well as the originally identified risk of lack of political will to fulfil the required tasks. There was a lot of support and political will at the regional level with support and commitment from the authorities, however at the national level more was needed. This had already been included in the possible risk factors within the program proposal itself and it materialized partially when presidential elections in 2022 led to major changes in the structure of the national membership. Local elections are coming up in October 2023, which will probably result in the change of several officials, unfortunately those civil servants who were trained, committed, and involved in the program. To avoid loss of continuity, the program team came up with two solutions: 1) rigorous hand-over process, with specific templates, documents; and 2) signing of MoUs or including results into public policies, making them legally binding. While commitment and political will could be seen from most of the entities, it was clear that not all of them were equally committed and while it was rare, it was observed that some of the politicians seemed to have different views on the same subject and lack of communication among one other. The evaluation team considers this as an external factor, which was also included in the original risk analysis and the program team did everything possible to overcome these obstacles or to find different solutions. “We have given dozens of suggestions to the management for risk management program, but the current administration has not even looked at these. There is no will to work on a disaster management policy, we have tried everything.” – Javier Velez, Director OMGERD Some regions were less focused on by the national authorities. For example, Armenia, since it is more distant from the Nevado de Ruiz, it is not prioritized by the Colombian leadership. However, the ash which would reach the town in case of an eruption would have such a weight that all roofs could collapse. Moreover, there is the Volcano Cerro Machin, which has not been active for a long time, but could activate anytime as well. 12 | FINAL EVALUATION OF PREPARE EJE CAFETERO Some activities which have been achieved according to the reports: • Knowledge transfer related to hazard, vulnerability and seismic risk to authorities, decision makers and rescue groups • Planning meetings for the construction of exposure and vulnerability models; presentations, monitoring and follow-up of activities • Creation of a structured database with information related to seismic hazard, vulnerability, seismic risk, and disaster risk management in the Eje Cafetero region • Work plan and a schedule of activities, instruments, categorization matrix and a sectorization of actors • Plans for a regional risk management plan for the Eje Cafetero Planning Office; a regional post￾disaster psychosocial assistance protocol; the national search and rescue plan; individual development plans for each rescue group supported in the region; the knowledge strengthening strategy; and the knowledge transfer plan • Seismic risk and DRM information compiled as a basis for the development of the knowledge strengthening strategy was integrated into the Eje Cafetero regional knowledge repository, linked to the platform nationwide Objective 3: Strengthen the capacity of municipal agencies, USAR and other first responders to plan and implement effective response operations Objective 3 was part of the Capacity Building and Training Sub-Sector and mainly focused on increasing the preparation and response capacities of First Responder groups. The field visit and data collection from primary sources as well as the results from the indicator table confirmed that the most visible results were related to the objective of building capacities of the first responder groups. According to the interviews, most of the participants mentioned the following factors which resulted in high level impact: • The program substantially increased their technical knowledge on seismic risks and response strategies as well as incremented their capacities to respond adequately in case of an earthquake by providing them with 1) the necessary technical knowledge through the specific technical trainings, and 2) the highly needed technical equipment for successful responses “The most important result of the program at the institutional level, was that the operational capacity increased from 28 rescuers to 35 rescuers. As well as the improvement of the level of response with the delivery of tools, equipment, and accessories.” – Carlos Gomez, Chinchina Voluntary Firefighters group • The above resulted in the possibility of applying for and receiving the nationally recognized Urban Search and Rescue (USAR) training and accreditation by the International Search and Rescue Advisory Group (INSARAG). While at the beginning there were only 11 USAR teams in Colombia, by the end of the program there were 20. Of the 9 supported USAR teams in the Eje Cafetero, 4 achieved national accreditation at the Light or Medium levels, and 5 additional groups are expected to receive this accreditation or re-accreditation this year. Participants explained that without the professional technical training and financial support (through high quality equipment) of the PREPARE program, they would not have had the possibility to comply with the requirements of INSARAG. “The program is very useful not only for the firefighters but for the region, without PREPARE it would have taken much longer to certify, without the necessary equipment it can last forever. PREPARE provided us a level that helped us push it through much faster.” – Alejandro Arango, Official Firefighters Pereira • The procurement process of the high-quality specialized equipment (e.g. S&R helmets and gloves, specialized tents, specialized medical kits, high altitude rescue equipment, FINAL EVALUATION PREPARE EJE CAFETERO | 13 administrative supplies, and tools) ensured that not only were they of high quality (that the local teams would have never been able to purchase), but the program also supported with the complex and cumbersome international procurement process in some cases, which would have been highly difficult for a local team. The departments' response capacity and resilience have been increased through the provision of equipment and training. “The provision of resources managed directly by the donor entities up to the final recipient reduces time, costs, optimizes employment and gives full assurance that the program will be successful, as was the case with this program.” - Mauricio Ruiz, Quindio Civil Defense Thirteen entities received equipment and goods as part of the logistical and operational capacity building process, of a value of 237,879 USD. In the below section the comparison of the initial requests to the received goods is explained, however, it is important to mention that most of the equipment received was in line with the requirements for the USAR accreditation process and in accordance with the capacity diagnostics carried out jointly with UNGRD. “There are tools and equipment that are not so easy to get such as equipment for structural collapse, sensors, scanners, shoring sensors, the tents purchased are not of the same quality that PREPARE delivered. In emergencies you need command posts, in communications, everything that you have taught us.” – Lina Marin, Official Firefighters Dosquebradas • Passing the accreditation and receiving the necessary equipment in most cases had a psychological impact on the first responders, especially the volunteer groups (Volunteer Firefighters, Red Cross volunteers, Civil Defense volunteers) as they now feel highly motivated to give better service considering that their work has been acknowledged. The leaders expressed that high-quality equipment, or specialized training gives an increased motivation to a volunteer who feels better recognized and this results in higher commitment. “The program would need to continue, but for the time it lasted, many things were achieved, ask a volunteer what the impact was like for him/her. Paying a volunteer doesn't make sense, providing them training and equipment motivates them. For us it was an important transformation, there is a before and after the PREPARE program. As the program was at departmental level, the impact will last for a long time. Although it is over, we have now more capacities and financial investment for maintenance of equipment has been received. Five community first response committees have been formed and women leaders have been trained in various processes. They will continue to form other volunteers.” – Lina Marin, Volunteer Firefighters Dosquebradas • Motivation and commitment were also increased by the fact that participants knew that only first responder groups that passed a certain percentage of readiness were supported. During the initial evaluation phase supported by UNGRD, each group had to be at 70% or above in terms of preparation in order for their accreditation to be supported and after other visits their commitment to continue with the program was assessed. The program team rigorously assessed and reassessed their level of readiness and commitment to provide support. • According to the level of each group and a very detailed needs assessment at the first phase of the program, the support plan and program activities were planned. This meant that each group received exactly the kind of training, logistical and operational support which responded to their specific needs and thus supported the leveling of their responding capacities. The evaluators could observe in person the gratefulness of several participants – leaders of first responder groups – as they became emotional while explaining their gratitude to the program that gave much more than tools, it increased their preparedness to another level. 14 | FINAL EVALUATION OF PREPARE EJE CAFETERO "We only had a pick and shovel.... and now we have a warehouse full of the highest-level equipment. I started crying when we had the ceremony." - Nathalia Valencia, Commander, Manizales Volunteer Fire Brigade. Capacity Building Trainings There were 619 participants trained in 38 trainings. As it can be seen from the indicator table, the number of completed trainings are much higher that the number of planned trainings, achieving 433% of the indicator: “Number of people trained in disaster preparedness, risk reduction and management”. While this could mean a not well-planned program, the evaluators understood from the program team that the training modality changed to online from in person during the first phase of the program as it adapted to the COVID-19 pandemic restrictions. This meant saving on costs and being able to train many more participants. Moreover, many of the initially planned expenses could be reduced with good management with the suppliers, and therefore, reinvested, obtaining greater coverage. On the other hand, some program partners provided training at no cost, such as the SGC. The above is a good result for effectiveness. The interviewees who took part in the training confirmed that these technical trainings were of very high quality (also USAID approved), highly professional and very useful for their everyday work. Of the respondents, 86% gave a value of 10 to the training, while 14% gave a value of 8. The following are the trainings provided to USAR groups, staff, and other beneficiaries: • 6 trainings on Incident Command System (ICS) basic level • 1 training on Incident Command System (ICS) Intermediate level • 1 workshop in Incident Command System (ICS) basic level • 1 training on Pre-Hospital Assistance (PHA) or Basic Vital Support (BVS) • 4 Atmospheres Detection and Ventilation workshops • 3 Structural Pathology workshops • 2 trainings on GPS and Cartography Management • 3 Knowledge Transfer Workshops • 1 training on Assembly and Use of Specialized Equipment • 1 training on Legislation in Risk Management and Response • 1 training on Emergency Communication • 1 training on Operational Analysis and Safety in Rescue Operations • 1 training on First Response to Incidents with Hazardous Materials • 1 training of Trainers • 2 training for Medical Units for Search and Rescue Groups • 1 training on the Elaboration of Protocols and Procedures • 1 training on Rescue Operations in Landslides • 1 training on Psychosocial assistance and Recuperation in Emergencies • 2 training on Building Information Collection Tools • 2 trainings on Risk Estimation • 2 trainings for medical units of USAR teams FINAL EVALUATION PREPARE EJE CAFETERO | 15 While feedback for improvement of the training was asked from everyone, the main suggestion was to continue with the training in the next phase. The request was to have more members trained and except in two cases, everyone said that they had no further suggestions for improvement. One person suggested to prioritize the topics according to the context and the indigenous volunteer firefighters, adapting the technical language to a more colloquial one for everyone to understand. “Capabilities have been enhanced in areas such as GPS management, structure assessment and incident command. - Alejandro Arango, Official Firefighters Pereira Increasing knowledge of municipal and departmental officials was also an important element of the program, with several training courses conducted to support the preparation of officials for their decision-making tasks in this subject. The training included the latest developments at global level in methodology, pedagogy, and tools for the transmission of technical knowledge to non-technical or non-scientific users. The program also delivered educational elements and role-playing games. Challenges: The medical component was the last one developed. With the support of UNGRD, the USAR medical training started during the last phase of the program. From the UNGRD, 70% of medical units were trained, 30% are left as they require a permit. The psychosocial component was not known to some participants, however hopefully with the trained trainers this will continue. The commitment of authorities and instructors is to replicate this knowledge in their respective entities and regions, which How do you rate your level of technical knowledge and understanding of seismic hazards and vulnerability from 1 to 10? 18% 27% 18% 36% 9 7 10 86% 14% 10 8 16 | FINAL EVALUATION OF PREPARE EJE CAFETERO is outside the scope of the program. With the rotation of the volunteers and officials, the training cycle should be continuous. Objective 4: Strengthen regional cooperation in risk management and response through joint planning, information, and resource sharing Objective 4 was part of the Public-Private Partnerships sub-sector. Before the program started, the risk management and response work of the first responders of the three departments of the Eje Cafetero region (Caldas, Risaralda and Armenia) were mostly working in an independent way, especially at the political level. Although the region is relatively small in Colombian terms and the three departments are geographically close to each other, there was historically a lack of cooperation. This was also due to the fact, that the preparation and knowledge level of the first responders of the three departments were quite different. “The main impact of the program for us was that we managed to work in coordination with the other departments of the Eje Cafetero, which has strengthened the technical, operational, administrative and coordination capacities for the benefit and reduction of risk in the territories. All the results and created documents of the program were useful, as these provide us with knowledge of DRM and seismic risk to include in our planning instruments. An important result was the significant strengthening of the operational entities of Quindío, in terms of knowledge and logistics for a response action.” – Juliana Escandon, Departmental Disaster Risk Management Unit of Quindío. The program focused on making equal the capacity and knowledge of the three departments so they can cooperate at a regional level, when needed. To achieve this, during the first phase of the program (Obj.1), the current capacities and necessities of each department were researched and analyzed. According to this baseline, the program team planned the necessary activities, training, and equipment of each first responder group, to increase their level of readiness with the goal of reaching a similar level. The main activities and objectives were planned during the workshops with authorities before the implementation of the program in 2019. “We support Rio Sucio firefighters when needed, our nearby units are activated, our units can cover Dosquebradas, Santa Rosa and Pereira. We have a mutual aid agreement; this allows us to call our colleagues in Santa Rosa and we can get there. Respect and cooperation between entities is strong.” - Alejandro Arango, Official Firefighters Pereira Another focus of this objective was to ensure that the teams from the different departments met each other in person. Thus, several trainings were organized at a regional level, always in a different location. This allowed the participants to make personal connections, which – according to the interviews – makes a whole difference when responding to a disaster. These connections also helped them to be fully aware of the strengths, capacities, and equipment of the other groups, hence they know whom to reach out to in specific cases. Exchanges at regional level were part of the main philosophy of the program and trainings in different locations were used as a tactic to have participants from the different regions together for the other objectives as well. “To know how the other response entities work and operate in a more articulated manner helps a lot. Now I can just pick up the phone for example when I need an expert in vertical rescue operations, as I know who the best would be to support us in this, and we are like friends now.” - Carlos Gomez, Voluntary Firefighters Chinchina In addition, the participants participated in a national level USAR team meeting, MOBEX 2022, a symposium to share the different experiences with 900 first responders, organized in Pereira, which shows the high-level preparedness of this region. It was a catalysator and gave national level recognition. FINAL EVALUATION PREPARE EJE CAFETERO | 17 "Growth and recognition: The PREPARE program has been instrumental in the growth and recognition of our work in national risk management. The program placed our team on the map at a national level and enabled us to obtain resources to implement initiatives. Alliances were established with different entities and relief organizations, such as universities, police, fire brigades, Red Cross, Civil Defense, etc. Effective coordination and capacity building on risk management issues was achieved. The importance of establishing good relationships with different entities and actors involved in risk management is highlighted. Effective communication and coordination are essential for an efficient response." - Alexander Galindo, Risk Management Director Pereira DIGER Psychosocial Component The psychosocial component of the program can also be mentioned as the start of a success story for better regional integration, which also strongly supported the objective of inter-institutional cooperation. Not being originally planned, but identified as a need of the initial diagnosis, it can also be considered as an unintended positive result. As several first responder groups received psychosocial training for the first time, we can consider this a very important and essential element which has a high impact on the work of the rescuers. Some organizations, such as Red Cross have their psychosocial components, but most of the first respondent groups have not had this focus before. Finally, the sustainability element of these activities is also considerable, because since the protocol on psychosocial assistance and recovery developed is a regional one, the program supported the creation of an interdepartmental working group to have ownership and continue the work. The working group functions as an advisory body to local and regional authorities. The training was conducted for 30 instructors, who are committed to disseminating the knowledge acquired in each of their cities and departments following a work plan that is included in the risk management plan of each city and department. “The psychosocial part is very important, we have a psychosocial support process, we have had a psychologist in the entity, we talked with the team about the emergency afterwards, we have learned to know the partner. When there is blood, confined spaces, screams, it is difficult to tell someone calmly that everything is going to be alright. We managed to focus on understanding other people, to make the boys lower the stress, we also have a personal trainer, we are like a family. “ – Lina Marin, Official Firefighter Dosquebradas It is important to mention that the protocol faces a few challenges such as the complexity and multidimensionality of the psychosocial phenomena that the protocol mentions but does not delimit in detail. It would be suggested to include informed consent of participants. video Formación de Formadores en Autocuidado Emocional y Primeros Auxilios Psicológicos.mp4 During the last phase of the program the development of local and regional radio communication protocols was started, which supports the strong need of communication among the region during emergencies. This need was expressed during several interviews, as the topography of the region challenges life-saving communication during a disaster response operation. Some participants expressed the need for high-level communication equipment, which would need to be homogenized, especially as some organizations have equipment which works with a different system, meaning that they cannot communicate with others. “The communication system is not articulated, a program must be presented to UNGRD, the program presented a program of communication protocols, radio amateurs. We also need a more modern crisis room, here some equipment does not work anymore. As we are a vulnerable region, if we do not have the right equipment to communicate, it is a risk, it is important that the program continues.” – Maurizio Ruiz, Civil Defense Quindío 18 | FINAL EVALUATION OF PREPARE EJE CAFETERO Inter-Institutional Cooperation The real strengthening of the capacities and the inter-institutional cooperation of the first responder groups was proved with the tangible result of applying their knowledge and equipment in practice in a real situation. The interviewees have mentioned several such cases, with most of them stating that they have been applying the knowledge acquired every day. "With the Incident Command System (ICS) we have been successful in working with other institutions in emergencies, carrying out the processes very well according to our role.” - Christian Martinez, Colombian Red Cross Quindío Section. It was interesting to hear that the same story of how the different entities cooperated in the rescue operations after an avalanche covered a bus was told in a very similar way by different organizations. All of them explained how well they managed to work together in this case. “There was an avalanche and a bus got trapped under it. Almost forty people died. The rescue operation took a long time. All the first responder groups acted according to the Incident Command System. Everyone knew their role and acted accordingly. There was no chaos, and the response was efficient. This was different before.” – Lina Marin, Dosquebradas Firefighter Cooperation with UNGRD, Academia and Geological Services According to the interview with the Colombian Geological Service and the Seismic and Vulcanological Observatory of Manizales as well as some universities, the cooperation was successful, which is a good result as it was a new element. The cooperation was a win-win-win partnership as each party was supported in their work. The universities managed to provide professional practice and practical work experience to their students. The Geological Service had well-prepared professionals supporting it in their work and finally they have also received the studies needed from the region while they provided their professional support. And the UNGRD managed to increase the number of USAR teams in the region, while it supported in the different needs assessment as well as in the analysis of the level of readiness of the first responder groups. The collaboration between these entities and Miyamoto proved to be sustainable, ensuring the transfer of knowledge and technical resources necessary for the effective implementation of the program. “Thanks to the program we now have an organized group in the region that has the necessary elements and knowledge to respond to disasters. Before we had to go to attend to the emergency nowadays, I just accompany as they are organized among themselves. We are not worried now, because whatever is happening in the coffee-growing region, we know that they can handle it.” – Diana Corrales, UNGRD Challenges: Raising awareness within the affected communities is a challenge as there is a high level of ignorance on the technical part among the general population and sometimes misinformation from other organizations. Objective 2 was dedicated to increasing the knowledge of political and technical staff and with the knowledge transfer plan to pass down this knowledge. With regards to specific issues, it is important to leave the specific roles to those responsible for these aspects, which normally function very well at the regional level. “It is important to establish an implicit agreement on who the spokesperson is and to maintain a good relationship with entities such as the Red Cross to avoid misunderstandings and misinformation.” - Lina Castaño, Volcanological and Seismic Observatory of Manizales FINAL EVALUATION PREPARE EJE CAFETERO | 19 Inter-Regional Cooperation The evaluation team considers that one of the biggest results of the program is the inter-regional cooperation which was detailed above. In short, the program not only implemented inter-regional products, such as the psychosocial protocol, but organized training in different locations, so that the participants got to know each other. As already explained, personal relationships, the leveling of the capacities, and being aware of the equipment and capacities of other groups, allows for a much better articulated coordination effort. Successful inter-regional cooperation was shown during the preparatory work for the Nevado de Ruiz operation, which can be considered as one of the biggest proofs that the PREPARE Eje Cafetero program was successful. Several interviewees confirmed that they used their knowledge acquired during the program in the face of the orange alert due to the threat of the eruption of the Nevado del Ruiz volcano. More details about this cooperation can be found under the unintended positive results part, at 4.3 Efficiency. The program supported the first Technical Committee to formulate the Regional Emergency Response Strategy promoted by RAP Eje Cafetero. "We can work as a team, the system can work with all the relief agencies, before this change in the alert of the Nevado del Ruiz volcano, the plan was already being updated" One of the most important elements of the success and high impact of the program was the right people chosen for the Program Team. Everyone has praised their expertise, professional management, local relationships, and personal attitude. Challenges: It was shared that working with the private sector was quite challenging. With regards to professional associations, such as engineering or architecture related associations, they were interested at the beginning, but were not ready to get involved as they feared their independence. As for the private companies, these would not commit or sign any MoU to support getting their equipment prior to an emergency, as they required confirmation to receive in-kind support, such as fuel. 20 | FINAL EVALUATION OF PREPARE EJE CAFETERO EFFECTIVENESS This section reviews the level of achievement of the specific objectives and expected results against the original plan. This section will also answer to the following questions of the ToR: • What program activities have been most effective in achieving the stated objectives? • To what extent has the involvement of national stakeholders shaped and helped to develop program outputs and outcomes? • How did specific intended and unintended changes in program activities positively or negatively affect the program? o The COVID-19 pandemic had both negative and positive impacts: it has caused delay in the implementation of some of the components, e.g., face-to-face trainings, however the online training module has allowed to increase the number of participants of the workshops, hence more there is higher capacity in the region than originally planned CASE STUDY OF THE IMPACT OF THE PROGRAM ON VOLUNTEER FIREFIGTHERS Certification and strengthening: Interviewees mentioned the importance of certification and strengthening of the Fire Brigade through the PREPARE program. They highlighted how specialized training and accreditation contributed to improving processes, optimizing resources, and accelerating the achievement of objectives. Equipment and tools: The relevance of the equipment provided by the program was highlighted, such as tents, GPS, structure assessment, incident command, among others. Firefighters highlighted the importance of having adequate tools to carry out their work efficiently and safely. Cooperation and mutual support: The importance of cooperation and support between the different fire brigades in the region was mentioned. The ability to work in an articulated manner and provide mutual assistance was highlighted as a key factor for effective emergency response. Training and education: Firefighters highlighted the relevance of continuous training and the acquisition of new knowledge. Topics such as Incident Command, first aid, confined space response, hazardous materials, among others, were mentioned. Training was considered essential to improve operational capacity and provide quality services. Psychosocial and well-being: Reference was made to the importance of addressing the psychosocial aspect for both firefighters and the community. Psychosocial clearance activities, attention to the emotional needs of those affected, and the importance of personal and team well-being were mentioned. History and trajectory: The trajectory and experience of the Fire Brigade was highlighted, mentioning its foundation, years of service and participation in events both locally and in other cities in the country. The commitment and willingness of the firefighters in their work was also highlighted. Risk management and awareness-raising: Activities related to risk management, awareness-raising programs, and community support activities on issues such as recycling, and risk prevention were mentioned. FINAL EVALUATION PREPARE EJE CAFETERO | 21 Indicator LOP Target LOP Achieved % Target Achieved Number of hazard risk reduction plans, strategies, policies, disaster preparedness, and contingency plans developed and in place 9 14 156% Number of people participating in discussions about national risk reduction strategies as a result of the program 200 322 (189 Males, 133 Females) 161% National and local risk assessment, hazards data and vulnerability information is available within targeted areas (Y/N) Y Y Y Risk data is incorporated into municipal and departmental risk management and response policy and plan Y Y Y Number of people trained in disaster preparedness, risk reduction and management 135 619 (456 Males, 163 Females) 459% Number of people passing final exams or receiving certificates 115 583 (436 Males, 147 Females) 507% Percentage of people trained who retain skills and knowledge after two months 80% 81% 101% Number of people trained in First Aid, Search and Rescue, or health related DRR activities 135 619 (456 Males, 163 Females) 459% Training accepted by UNGRD for national USAR accreditation Y Y Y Number of private sector businesses directly engaged in response or DRR-related activities as a result of the program 6 8 133% Estimated total number of individuals indirectly benefitting from DRR-related program activities 1,268,305 1,306,026 103% Number of government disaster contingency plans that incorporate private-sector aspects as a result of the program 1 3 300% Public-private partnerships included in the regional, departmental or municipal emergency response protocols Y Y Y As for the Policy and Planning sub-sector, the program exceeded the objective of “participation of men and women in discussions on national risk reduction strategies” with 159% of the participation target achieved. The target for the availability of data on risks and vulnerabilities in selected areas was achieved, indicating a positive impact on risk management and risk reduction. Several program outputs were, and some are being incorporated into municipal, departmental, and regional risk management plans and their respective response strategies. Although some of these are not finalized yet, according to the certified LoCs signed by mayors, governors, and risk management directors this will be guaranteed and hopefully maintained with the new political leadership after local elections. 22 | FINAL EVALUATION OF PREPARE EJE CAFETERO Progress update Objective 1. Progress update Obj. 2. As for Objective 1 and 2, the progress level graph shows the finalization of the objectives as it was planned, and the divergence is low. As for the Capacity Building and Training sub-sector, the program far exceeded the training target, with 433% reached. A significant number of men and women were trained in disaster preparedness, risk reduction and risk management. Under the same objective, the target for passing exams and issuing certificates, exceeded the original goal, reaching 489%. In the same objective the program exceeded the skills and knowledge retention target with 101% of the target achieved. Progress of Objective 3. April 2023 This indicates a high-level success in the training provided which was effective in maintaining skills and knowledge in the long term. The program also exceeded the target in specific areas related to health and risk reduction, with 433% of the target achieved. The progress level graph shows a very small divergence from the original plan (in orange the implementation progress and in black the planning curve), which is understood to be repeatedly adapted to the needs of the participants. The rigorousness in the follow-up of the planning and the final compliance of 100% is highlighted. USAR Accreditation The number of USAR groups accredited or reaccredited was not included in the original indicator table, therefore does not correspond to a program commitment. However, the statistics show a positive impact for the region. Of the 9 supported USAR teams in the Eje Cafetero, 4 achieved national accreditation at the Light or Medium levels, and five additional groups are expected to receive this accreditation or re-accreditation this year. The evaluation team compared the initial evaluation % from 2021 and the level of improvement from 2022 and 2023, which shows significant development of the first responder groups which were supported by the PREPARE program. It is an effective approach to FINAL EVALUATION PREPARE EJE CAFETERO | 23 support the groups which are not only at a good level of preparation but have the motivation to fulfill it. For example, we can see significant increase from Dosquebradas (60% increase), Pereira (20% increase) and Riosucio firefighters (47% increase). To see the efficiency of the program, we took one example to demonstrate it in practice. Riosucio Volunteer Firefighters during the initial needs assessment phase had requested the items in yellow. According to the donation list, the elements with a tick show partial fulfilment of the same or similar elements. We understand that this was an initial wish list and with the support of the program team, the list has been further negotiated according to the program’s possibilities and the principle of levelling the different first responder groups. The equipment had to be in line with USAR requirement and firstly the basic list of equipment had to be secured. It is considered as a strength of the approach of PREPARE that elements donated were on the obligatory list for the USAR accreditation and this had to be in order of level (starting with basic elements). VIDEO ENTREGA DE EQUIPOS A GRUPOS USAR (1).mp4 Denomination Name Level of documented advancement % of advancement in 2021 % of advancement in 2022 % of advancement in 2023 COL – 14 GRUPO DE RESCATE BOMBEROS CHINCHINA Reacreditación II Semestre 2022 97 100 99 COL – 15 USAR -DEFENSA CIVIL CALDAS Reacreditación II Semestre 2022 83 100 84 COL – 16 USAR CRUZ ROJA CALDAS Reacreditación II Semestre 2022 84 100 98 COL – 17 USAR CRUZ ROJA QUINDIO Reacreditación II Semestre 2023 99 100 99 COL -21 USAR BOMBEROS VOLUNTARIOS MANIZALES 98% 83 98 84 COL-25 CUERPO OFICIAL DE BOMBEROS DOSQUEBRADAS 79% 32 79 95 COL-22 USAR BOMBEROS OFICIALES PEREIRA 67% 54 67 95 COL-20 USAR BOMBEROS RIOSUCIO 52% 28 52 92 DEFENSA CIVIL SECCIONAL QUINDIO 14% 5 14 80 DEFENSA CIVIL SECCIONAL RISARALDA 14% 5 14 70 CRUZ ROJA SECCIONAL RISARALDA 5% 0 5 CUERPO OFICIAL DE BOMBEROS DE MANIZALES 0% 0 0 62 CUERPO OFICIAL DE BOMBEROS DE ARMENIA 0% 0 0 61 24 | FINAL EVALUATION OF PREPARE EJE CAFETERO “PREPARE has provided trainings for those capacities and donated those elements which are necessary for USAR accreditation and based on the current needs and capacities of each group.” – Diana Corrales, UNGRD One of their members, a female firefighter, received the highest number of trainings (6), in the following areas: Atmosphere Detection and Ventilation Workshop, Structural Pathology Virtual Workshop, Strategy for Seismic Risk and Risk Management Communication, Basic APH Course, Workshop on the Elaboration of P&P, and Knowledge Sharing (Caldas) Comp. 2. With regards to the Public Private Partnerships sub-sector, the program partially succeeded in engaging four private sector companies in response or risk reduction activities, reaching 67% of the target. This shows moderate effectiveness. As for inclusion and cooperation, although no contingency plans incorporating aspects of the private sector were developed, several attempts were made and the objective of including public-private partnerships in emergency response protocols at regional, departmental, and municipal levels was achieved. The most difficulties were had to achieve Objective 4 due to the lack of interest of the private sector to be involved, as well as the long-term process and political will of including documents in public policies. This can be also seen on the progress graph. However, by the end of the program most of these difficulties had been overcome and the objectives achieved. Progress of Objective 4. June 2023 FINAL EVALUATION PREPARE EJE CAFETERO | 25 Successes: The original training plan had to be changed to adapt to the needs of the groups. Another important factor that impacted the increased effectiveness was the selection of participants who were committed and had shown previous results. With this approach it became possible to avoid losing resources on investments which would not give the expected results (e.g., groups of participants not completing the activities as required). The feedback from participants was that the meetings were efficient and useful, dynamic, did not waste time, and gave results. Challenges: A Non-Cost Extension of 9 months (from September 29, 2022, to June 30, 2023) was requested from the donor. Once the extension was approved, the additional time was used for the development of new activities, such as exposure models for Armenia and Calarcá, communication protocols, psychosocial assistance protocols, homogenization of damage assessment tools, training of medical units, among others. This additional time was also used to strengthen relations with the private sector and achieve the institutionalization of program results. Unintended Positive Results: The PREPARE Eje Cafetero program addressed the escalation of the activity of the Nevado del Ruiz volcano which changed from yellow to orange. The team formulated a proposal to prevent the loss of lives and material goods in case of a new eruption. The impact of Nevado del Ruiz in the region corresponds to a multi-hazard scenario, which could cause landslides, floods, atmospheric emissions, and a potential increase in seismic activity (average magnitude 5.0 on the Richter scale), among others. The program worked on the development of response protocols, which were added in the NCE phase. With the Nevado de Ruiz activation, the formulation of the first Regional Emergency Response Strategy of the Technical Committee of the RAP was recently included in Tolima among its departments, which is not part of the Eje Cafetero region and possibly future proposals. Institutions were working to preemptively evacuate residents settled in the foothills of the volcano, and their initiatives had to consider georeferenced damages as identified by the Geological Service, sanitary treatment of water sources, and structural collapse due to ash accumulation. Several sectoral groups were activated with a unified directive to be able to coordinate when needed. “We are equipped thanks to the program we have participated in; we have an advantage; we are a small department, and we can work as a team. We know how we can support each other, how to strengthen ourselves in the face of the threat.” - Juliana Escandon This result is strongly related to the cooperation with the Volcanic and Seismic Observatory of Manizales and Caldas department, which happened during the last phase of the program because of the activation of the Nevado del Ruiz Volcano. This demonstrates the program’s capacity to adapt as well as the result of its impact. The exact activities implemented in this regard will be shared in the final report. A significant result of the cooperation is the implementation of a radio communications protocol that improved the speed and effectiveness of the transmission of relevant information in volcanic risk situations. In detail, the establishment of a call chain in volcanic risk situations that guarantees agile and efficient communication with the corresponding authorities is only working with the support of the necessary telecommunications equipment, which has been donated by Miyamoto. Important to highlight is that the first female commander of a USAR team was accredited during and thanks to the PREPARE Eje Cafetero program. This is a significant result considering she is the only one in the whole of Latin America. There were two female commanders, which is also important and several women were prioritized in the trainings, even in the indigenous firefigthers group. EFFICIENCY 26 | FINAL EVALUATION OF PREPARE EJE CAFETERO This section usually measures efficiency in terms of resource utilization and costs. As the ToR did not include financial evaluation of the program and the evaluators have not received the budget or financial reports, the analysis below only includes feedback from interviews and what could be understood from the narrative reports. Based on the feedback from the program team it was understood that some of the original costs were reallocated, and spending was decreased adapting to the needs and external impacts, such as the COVID-19 pandemic. Providing courses online instead of in person lowered the costs and increased the number of direct beneficiaries reached. The team managed to receive in-kind support from the institutions as well, such as a location for the training, and sometimes snacks, coffee, etc. according to the possibilities of each. This support also decreased the costs and gave ownership to the participating entities. In these cases when costs were reduced, they were reallocated to other training or activities, so that more people could benefit from the capacity building. These changes were allowed at the management level as these had a good rationale. Originally the budget was led by the regional management, but later the budget was shared with the country team, so that they could adapt it to the reality in the field. More savings could have been made with regional level equipment purchases as the availability of these items was low in the region and the costs were high. The program team also focused strongly on levelling the logistical and operational capacities of each USAR team, hence allocated the resources in a way that each team received the equipment they were missing, and if one team was missing a piece, another team nearby had it. This was important considering that according to the direct observation of the evaluation team, the preparation levels, location, equipment, etc. was quite different among some of the USAR groups. Based on the above, the evaluation team considers that the cost-efficiency was high, however it needs to be mentioned that neither the budget nor financial reports were received or reviewed. With regards to efficiency of human resources, the three program team members were the biggest asset to achieving the impact, efficiency, efficacy, and high results, due to their professionalism, personality, and commitment - hence they were of high value. As for the administrative processes, the program was very well coordinated from the LAC regional level, in both administrative and financial terms, with periodic meetings (weekly and then bi-weekly to save time) with regular monitoring reports which provided the necessary information to make informed decisions and adapt to changing circumstances. Giving the program team access and control of the program budget was a very positive achievement as it gave the team the flexibility to make decisions and adapt the budget to the current needs. The budget itself was authorized by the donor to amend up to 10%, hence as the changes suggested were reasonable and increased efficiency. An example of this was the reallocation of funds for the Nevado del Ruiz response operations. At the coordination level, there was a well-structured and inclusive system which allowed continuous monitoring and feedback. 1) Regional committee (Steering Committee): heads of institutions, coordinators of the 6 risk management offices, BHA representative from national level, UNGRD, and National Fire Brigade of Colombia; 2) Local committee - department and municipality level; 3) Working groups according to Objectives – e.g., psychosocial assistance, Univ. Manizales; local - Armenia, Calarca; Damage assessment - regional. SUSTAINABILITY The following section reviews the sustainability of the program and the ability of the participants to continue with the results in the long term. In line with the questions of the ToR, this section focuses on the following aspects: • To what extent have program activities contributed to sustainable changes or outcomes and what factors contributed to achieving sustainability? FINAL EVALUATION PREPARE EJE CAFETERO | 27 • How was the exit strategy and Sustainability Plan set up? • Have mechanisms been established to maintain the public-private partnership beyond the life of the program? The program logic itself was based on the principle of sustainability from the very beginning as it had a long pre-phase where the specific needs, interest and commitment from the future participants were confirmed. This was the basis of the proposal itself, which was followed by the first phase of the program solely focusing on the selection of those participants who would be committed to continue with the work when the program finalized. During this phase their logistical, operational capacities and current knowledge, needs and gaps were identified, and the program activities were based on these analyses. This process ensured that the resources would not be spent wrongly or lost, and participants would be enabled with stronger capacities to continue after the program as well. With regards to capacity building activities, the team ensured that knowledge was not lost by examining the knowledge retention of participants two months after the training. With the latest report, the indicator of “Percentage of people trained who retain skills and knowledge after two months with at least 80%” was at 101%. “The training was according to the needs and agreement of the participants, the USAID approved training model is technical and professional, as it is standardized it supports inter-regional work. The workshops generated capacities also among the instructors of the Red Cross, like for example in the Incident Command System. The training was received by the USAR teams (40), their capacitation of rescue lasts 2 years, they are specialized then to replicate information to the volunteers throughout the department, they must guarantee continuity of the process.” - Daniel Fernando Cano Mejía, Red Cross Caldas. In addition, Letters of Commitment and Memorandums of Understanding were signed and a few documents were included in public policies making them a binding document. This ensures that even with a change of government the achievements are included in local policies and the achieved public￾private partnerships will continue. “The PREPARE program gave us tools that are of great importance for the region, such as the seismic exposure model, the strategy for seismic risk communication, the risk management risk communication strategy, the risk management repository, methodologies for the development of protocols, the development of protocols, the tool for the homogenization of damage assessment, the diverse assessment tools, the various technical trainings for the staff of our relief agencies and risk management offices, among others. These are permanently reflected in the more comprehensive vision when dealing with emergencies and disasters and addressing the issues of emergencies and disasters that affect the territory.” – Letter of Commitment Armenia, OMGERD Armenia The high-level rotation characteristic of the political sphere in Colombia was identified as risk at the beginning and a detailed hand-over plan was prepared to avoid loss of continuity. Moreover, several products, e.g., repository, seismic exposure models, etc. are already or will be included into national databases, hence will continue their impact after the program. Important to mention is the intra-regional and inter-institutional cooperation that was achieved, especially with regards to personal relations, which allows a more efficient response in case of a disaster and ensures sustainability. With the goal of increasing the sustainability, during the last phase of the program a strengthening plan for the USAR groups was created, detailing their level of progress in the national accreditation process and a new UNGRD capacity assessment was prepared to determine their level of progress in the national USAR group accreditation. Regarding challenges for sustainability, the following should be mentioned: 28 | FINAL EVALUATION OF PREPARE EJE CAFETERO • The volunteer organizations often have volunteers that leave the institution due to change of life situation, work, etc. Several interviewees mentioned that some of the participants of the workshops had already left the organization. • Although a hand-over plan was set for the case of change of government officials in the positions involved in the program, it was often not fulfilled. • While the indicator for “Risk data is incorporated into municipal and departmental risk management and response policy and plan” was a long process due to its political nature. Two were signed and the others committed with a signed letter, which hopefully will be maintained with the change of government. As these policies and commitment to the issue depend on the political will of each government, this is considered as an external factor which is hard to control. • Regarding the indicator “Public-private partnerships included in the regional, departmental or municipal emergency response protocols,” some partnerships were successful, mainly with the academia, however most of the private sector companies were not ready to sign any commitment document or get involved in preparedness work. The Engineering and Architecture Societies of Caldas were included in the Departmental Disaster Risk Management Plan of Caldas and the Association of Engineers of Caldas signed a letter of commitment for the damage assessment studies. • The constant change of leadership of UNGRD and the different approach to DRM of the current government made it difficult for the local entities to continue a good collaboration, which impacts their work at the local level as well. The program team continuously cooperated with UNGRD, also with the change of POC, and they are aware of this challenge. Again, this is out of the control of the program team. “ The program could not control the political issue; the last donation was at high risk to be finalized. Donation of equipment for USAR medical units for training. The last training was done in the last month as there was no one to authorize us.” – Diana Corrales, UNGRD • Regarding the Exit Strategy, the issue of sustainability was included in the proposal, and was mostly implemented. As there were plans to further continue the program with a next phase including the capacity and knowledge building of the communities and all participants expressed their high interest and request to further continue the program, the information of not supporting the next phase by the donor came in the late phase. Some activities can be continued by the program participants as the program was specifically planned to be sustainable, but the activities of the planned next phase (e.g., community training) would not be. Hence the lack of funds for the future is a risk for the continuation of the activities. • An endline study is also missing – this was not budgeted – which would give a real picture of the impact compared with the baseline. The current final evaluation is partially responding to this need; however, it is not meant to be a rigorous comparison with the baseline study. • As USAID is focusing on other geographical areas, the program management is in the process of looking for other donors, specifically with the activation of the Nevado del Ruiz for emergency programs, as well for the continuation of the program with a second phase to address the needs identified during the first phase. • As the USAR accreditation is annual and most of the institutions would struggle to get the necessary fund for reaccreditation, the program management is in the process of convincing UNGRD to commit to supporting the accreditation process for next year. “We will continue to strengthen skills and abilities by making use of the equipment and tools that allow for better institutional programming through the application of established procedures and protocols.” – Oscar Mejia, Riosucio Voluntary Firefighters FINAL EVALUATION PREPARE EJE CAFETERO | 29 Transversal Aspects: Gender Equity This section assesses the extent to which the program processes, products and trainings included gender aspects, answering the following question of ToR: • To what extent have the outputs and outcomes included participation and accountability to the affected population, including gender considerations? To evaluate transversal issues including gender equity and inclusion of vulnerable population, the evaluators used different methods as the program did not directly work with vulnerable populations or had the opportunity to choose the participants based on their gender. However, several conclusions can be made based on different interesting discussions with the interviewees. Gender Equity: First, a limitation to this question was the fact that the direct beneficiaries were public officers or first responders, not communities. Their gender was a fact, and the program team could not choose to increase the gender parity. Second, in these specific groups, such as firefighters or civil defense groups, traditionally males are overrepresented, as some interviewee said, it is “traditionally a male job”. Third, the USAR training participants were chosen by the institutions themselves and not the program team. For example, in the Red Cross group of Quindío, from the 60 USAR members, between 10 and 20 are women, including health professionals, psychologists, etc. Nevertheless, there were efforts made to have more women in the capacity building workshops and according to the indicator table, this reached 26%, while for the people participating in discussions regarding national risk reduction strategies, 41% were women. The evaluator team considers this a good result based on the above limitations. Success stories: It is the first time in Latin-America that an accredited USAR team leader is a woman and it happened under the PREPARE Eje Cafetero program. The commander of the Manizales Volunteer Firefighters is a role model for everyone, not only within her own community and firefighter team, but at the regional and possibly national level as well. She has been highly committed to being a volunteer firefighter for decades and started the USAR accreditation process with almost no equipment as she said with a “pick and a shovel”. Still, she worked hard with her team to get all the documentation done (70% of the process) and when the program came to provide them the operational and logistical capacity needed, they were the most committed participants. She and her team burst into tears during the accreditation process and when talking about what the program gave to them, they had teary eyes. Her high-level of professionalism was impressive, considering the conditions they work in (without their own place). She told stories of how they cooperate with other entities and showed excellent leadership in difficult situations e.g., when there was a big operation and the official firefighters could not manage the situation and it was becoming chaotic, she stayed in her position according to the rules of the Incident Command System. While they did not let her team in, she set up the tent with her team as a simulation exercise and requested to let her help them several times. They finally gave her the command and she managed the situation till the regional body came in and asked her to leave. “To be a commander as a female firefighter in Colombia is very difficult, in the Officer Corps, among 23 commanders, there is only one woman. The fire brigade is a professional career with a high responsibility which is difficult if one has children. “ – Lina Marin, Official Firefighters Dosquebradas Challenges: Both women leaders of the voluntary firefighters group shared how difficult it was for them to reach to a managerial level. They had to confront criticism and misbelief of their capacities. The commander of the Manizales Volunteer Firefighters at first did not want to be a commander, but the previous commander asked her to replace him on his death bed and the day the evaluator team interviewed 30 | FINAL EVALUATION OF PREPARE EJE CAFETERO her she shared that it was her last day, but at the next interview she said she will need to continue, as there is no one within the team who wants to replace her. Both women had personal stories and motivation to become a firefighter and shared that they needed to prove themselves three times as much to get to their positions. The perception of gender equity still has a less gender sensitive approach within most of the interviewed people, stating that they do have several women within their teams, and they are “the most tough or badass women”, as if being tough is a criterion to be a leader or part of a USAR team. The women leaders also expressed that they are a bit tired of being considered “tough or badass” as their personal qualities are the ones that helped them to reach their position which should not have to do with their gender. Some participants also expressed that female qualities are more than useful in USAR and first responder teams, stating that women are more organized, sensitive, supportive in teamwork activities or during operations. Other people also mentioned the psychosocial aspect which according to them is a quality the women bring to the teams. According to some, a woman can carry the same heavy equipment, as it depends on the body position they use, while others said that woman can have other roles and do not need to carry the heavy tools. “Eres mas valiente de lo que crees, mas fuerte de lo que pareces y mas inteligente de lo que piensas.” – “You are braver than you believe it, stronger than you look like and more intelligent than what you think”. – a painted citation on the wall of the Pereira Official Firefighters female dorm, which shows how much harder it is for women to reach this position. Within Civil Defense groups there are more women, although often in administrative roles, while in the Red Cross gender equity is not really an issue as volunteers come from both genders at an equal level (USAR team members are more predominantly male). In brief, it considered as a good result that almost all USAR teams trained under the program had at least a few (up to 20%) women members. However, this was out of the control of the program. The general opinion was that by now anyone can be a firefighter; the public perception is changing, but it still much more difficult for woman. Indigenous Groups The evaluation team had the chance to conduct an online interview with the Riosucio Volunteer Firefighters, which can be considered as the best example of an indigenous volunteer firefighter group in the country and in the region. It is the first accredited indigenous firefighters USAR group and its history goes back decades. Success story: The evaluation team felt that the success of the accreditation of the indigenous firefighter USAR group was strongly linked to the Commander of the Riosucio Voluntary Firefighter group, who has been a firefighter for 46 years. The commander shared how he has worked together, supporting, and learning from indigenous firefighters for more than two decades. He started to work together with the indigenous communities in their territories where there were often forest fires, which they managed to prevent with their joint work. Today there are 140 indigenous firefighters in the region in 4 different stations. Thanks to their high level of preparation, they now have a preparation center for indigenous firefighters, they have travelled to 15 other departments of Colombia to train other groups, they have had three national meetings, travelled also to other countries in the region to train them, had a knowledge exchange workshop with indigenous groups from the United States, and they will have 50 European Union representatives visit them this year to learn about their example and bring it to other parts of the world. Their success has even prevented young individuals from joining the illegal armed groups in earlier decades as it happened in many different areas of Colombia, as they claimed they are part of the volunteer firefighters and not interested in joining different groups A good example of how they learned best practices from each other while becoming safer in nature is that now they are not burning their fields anymore after use, and they instead reuse the material as natural fertilizer. FINAL EVALUATION PREPARE EJE CAFETERO | 31 The approach of the commander seemed to be highly inclusive as he shared how they have learned from the indigenous populations how to use ancestral proceedings and harmonization with the nature (e.g., before entering a fire, they request their natural gods to calm down, they invoke the natural elements of water, fire, air, and earth). Living together with nature and using these techniques was suggested that could also be included in the training. Even though they were already in the USAR accreditation process, PREPARE gave them fundamental support by providing them with tools and machinery that they were missing to be accredited and this increased their motivation to continue with the complex accreditation process. When asked to rate the impact of the program from 1 to 10, the Commander said 20; “The accreditation gave us confidence that we are doing it well”. With regards to the gender perspective within the indigenous groups, the Commander shared that due to traditional roles within these groups it has not been easy, but he has put a lot of emphasis on including women in the firefighter groups and they often hold pre-hospitalization, administrative, etc. roles. Challenges: The Commander explained that the language, methodology and requirements of the USAR accreditation and some trainings are too technical and difficult to understand for the indigenous groups. It intimidates them and they think they will not be able to pass, and this is demotivational. They have managed to translate it to simpler terms, adapt it to the indigenous dialect and use games to share the same knowledge, so that the indigenous groups can also fulfill the requirements. Even though some firefighters might not have the same level of education, they can be even more professional in this work, hence it is important to adapt to the different levels of people as well. Accountability To The Affected Population The Program did not have any activities directly involving the communities and affected population, which would be the second phase of the program. However, the different first respondent groups did have community-based training and educational activities to prepare them for the disasters. "In the new station of Villa Santa Ana, we are giving training to mothers, we have six fire stations, we do activities with the people, sharing everything we have learned" – Lina Marin, Firefighters Dosquebradas With regards to accountability with the direct beneficiaries of the program – members of USAR and OGR teams – the structure and buildup of the whole process was based on involving the beneficiaries from the very beginning, starting with the pre-phase, the planning, implementation, continuous feedback, and monitoring. Therefore, we can comfortably state that the program itself was designed and implemented together with the direct beneficiaries. This ensured sustainability as well as accountability. COORDINATION AND HARMONIZATION While this section measures the coordination between the different actors and the participation, empowerment, and accountability of the target population during program implementation, it also strongly focuses on Objective 4 of the program to answer the following question from the ToR: • Has coordination and communication between agencies and institutions increased because of participation in the program? While these criteria can be considered as successful with regards to cooperation among the participants, first responders and at inter-regional level (detailed under Impact Obj.4), the results have not reached the planned level of cooperation with the private sector with regards to corporations and companies (other than academia and a few associations). This was expressed as difficulty by the program team as well. 32 | FINAL EVALUATION OF PREPARE EJE CAFETERO Success Stories: • The support of the program for inter-institutional cooperation with regards to the first responder organizations was detailed above, however it is important to mention that the joint workshops and more personal connections have also helped to decrease the rivalry among the organizations. This is also in part due to the program’s focus on making equal the different organizations which helps cooperation during disaster response. With standardized training, all the organizations know and act according to their roles: Civil Defense secures the area, the Red Cross provides first aid support, and the firefighters rescue trapped victims. • The Regional Administrative and Planification entity, RAP Eje Cafetero was also actively involved in the process of generating a regional plan. The program helped to potentialize it also in risk management aspects. • There were two agreements signed with private universities: 1) with the University of Manizales to develop a regional protocol on psychosocial assistance and recovery after a disaster and a training for trainers on the same topic; and 2) with the University Gran Colombia for support with the collection of information to feed the Eje Cafetero repository and collect building information for the Armenia and Calarca seismic exposure model. • Two additional Memorandums of Understanding were signed; one with a private university and one with a technical association: Alexander Von Humboldt University and the Colombian Association of Engineering Schools. • In Pereira, 30 students from the Libre University of Pereira supported the collection of information on the structural characteristics of buildings in Pereira to develop the Pereira exposure model. • The coordination with the Colombian Geological Service and its local counterpart, the Seismic and Vulcanological Observatory of Manizales supported the professional and technical aspects of the program, and the institutions also praised the cooperation for being helpful in fulfilling their job in the region. • With UNGRD the collaboration was of mutual support and as explained earlier it was a win￾win cooperation among the different institutions. Challenges: • Private companies were not ready to sign any commitment document or get involved in preparedness activities stating that they can provide their equipment, machinery, and support when a disaster occurs, however they cannot sign any legal document as they would require support of inputs, such as petrol, etc. to operate the machinery. As a solution, the program prepared an inventory list, which can be used during a response. • The program team, after a lot of effort and considerable time, finally managed to get signed a few commitment letters with professional entities, such as the Association of architects of Caldas, for the damage assessment work. The feedback was that, although at the early talks they expressed high interest to the program, they mostly wanted to maintain their independence. • In this phase, the program did not have direct cooperation with the communities (it was planned for the next phase); however, several institutions (beneficiaries of the program itself) are volunteer organizations, and their members are from the community (Red Cross, Voluntary Firefighters or Civil Defense) or they themselves have programs/workshops at the community level. For example, Civil Defense is approaching each family and conducting house visits to raise awareness on risk preparedness. “We at the Red Cross prepare the community in different topics, such as first aid, health and vulnerability, risk management. We seek to work together with them, so that they understand how to respond, and we give them tools so that they can manage these processes” – Jacobo Franco, Red Cross Quindio FINAL EVALUATION PREPARE EJE CAFETERO | 33 LESSONS LEARNED This section collects lessons learned, suggestions for improvement and feedback for future similar programs both in implementation as in general. Throughout the implementation of the PREPARE Eje Cafetero program, several valuable lessons were gathered. First, the COVID-19 pandemic highlighted the importance of flexibility and adaptability in program planning and implementation. The team quickly adjusted to the methodology of virtual events, which allowed activities to be carried out remotely and progress to be maintained despite social gathering constraints. The team also adjusted to other external variables when needed, which emphasized the need for contingency plans and alternative scenarios to ensure effective coordination and interaction among Eje Cafetero stakeholders. Another key lesson was the importance of collaboration and coordination between government entities, emergency groups and other actors such as private enterprises, like academia at the local and regional level. It was shown that by involving people from across the region in each training and course, generated a regional working environment, fostering collaboration and connection between members of rescue groups from different entities and cities. The importance of strong coordination and collaboration with local, regional, and national entities cannot be underestimated. Working closely with authorities, partner institutions and decision-makers facilitated effective planning, decision-making and implementation of program activities. It is a highly impactful approach to support groups that are committed and produce results. In most cases, the good results and commitment depended on one person or the leader of the institution. Where there was no support from leadership, the program could not thrive. Choosing the right people for each position is 50% of the success. Almost all participants mentioned and highly praised all the three program team members for their professionalism, expertise, knowledge, commitment, continuous communication, accompaniment, and human quality. “The Miyamoto Team has great human, technical and administrative capacity, as well as all the willingness to support and cooperate. Growth in technical, administrative and/or operational aspects of the Deptal DRM system, which strengthens territorial DRM is all added value from Miyamoto.” – Juliana Escandon, Departmental Disaster Risk Management Unit of Quindío. “The Miyamoto team had its personal commitment and high involvement for the program, not all development workers have that, one feels accompanied.” - UNGRD The program experiences highlighted the importance of adaptability, collaboration, effective communication and building on existing frameworks to effectively address seismic risk and improve disaster preparedness and response in the Eje Cafetero region. The assessment conducted by Colombia's National Unit for Disaster Risk Management identified and selected USAR groups that demonstrated significant progress in the national accreditation or reaccreditation process, which ensured a more effective response in emergency situations and a highly motivated team. It is important to ensure that training courses are also given using less technical language. A plan was developed to convey key technical concepts and words in a language and with methods more accessible to people of all educational backgrounds. “The socialization with the strategy was very interesting through playfulness, games, techniques and grounded in the language that one can handle with the same community.” – Juliana Escandon, Operational Risk Management Quindío 34 | FINAL EVALUATION OF PREPARE EJE CAFETERO The unintended positive result of signing a contract with the University of Manizales to provide a diploma for a training of trainers course in psychosocial assistance and disaster recovery at the regional level was an important step in strengthening the psychosocial component within first responders. The importance of developing communication and knowledge transfer strategies on seismic risk was recognized and responded to, as well as the need for post-disaster psychosocial assistance protocols. The creation of regional protocols in damage assessment, psychosocial assistance, and radio communications at the last phase of the program improved the articulation of different entities in the region under the same standardized structure, increasing the efficiency of emergency responses. Continuity of work was affected by the high rotation of municipal and departmental officials, which is a common issue and was identified as a risk at the beginning. An important lesson was the identification of the need to have staff formally assigned to the program partners to ensure continuity of work and the use of acquired knowledge. To mitigate the possible impacts of administrative changes and high￾level rotation, at least two people per entity were asked to be assigned to the program. In addition, a transition plan and transfer of responsibilities in case of staff changes was designed and proved to be an effective strategy. Although limitations were identified in the capacity of institutions and a general lack of knowledge, progress was made in the inclusion of different actors due to the program, such as contractors and universities, in seismic risk management. Overcoming the historical issue of political independence and hence lack of collaboration fostered joint work which helped to improve risk awareness, management of protocols and emergency response processes. The program's emphasis on public-private partnerships was also valuable, facilitating knowledge sharing, cooperation in response and the integration of important data into policies and plans. The experience of the program highlighted the importance of building on existing legal frameworks and mandates for disaster risk management. Aligning with national laws and leveraging risk management councils as main implementing partners provided a solid basis for program activities. Underlining the importance of institutionalizing program results and outputs within the planning instruments of local, regional, and national authorities was crucial to ensure their improvement, sustainability of their support, and relevance of the program. By becoming official and binding, the program's achievements are ensured to be used and developed by responsible actors and authorities. The importance of sustainability was also highlighted and was considered beginning with the design of the program activities, through the setting up of regional and local committees, which held regular meetings to provide program updates and reaffirm the commitment of stakeholders. In addition, the inclusion of authorities and entities outside of the Eje Cafetero area at the national and international level not only allowed for the dissemination of the program's objectives and successes but also provided valuable feedback and potential support for future collaborations. The logical framework tool was missing (other tools were used to explain the logics of the program and a detailed monitoring system was set up), and the indicator table does not reflect properly all the important achievements of the program, like for example the accreditation of USAR groups. The need to improve the monitoring and follow-up of indicators that were possibly designed with a lower expectation in terms of program beneficiaries was recognized. The procurement and delivery of specialized equipment to USAR teams and other entities involved in emergency response was an important milestone. Careful planning and proper monitoring of procurement processes ensured the timely provision of the necessary equipment; however it was suggested that it would be helpful to develop a regional procurement policy and put it into practice to save costs. The emergency response protocols and procedures and the microzonification and seismic risk studies at local and regional levels was essential for effective risk management and response, but many more should be developed. FINAL EVALUATION PREPARE EJE CAFETERO | 35 The hiring of specialized external consultants was fundamental to support the technical development of models and studies and it is important to have very detailed and specific ToRs to optimize the use of the consultant’s work. The importance of participation and accountability to the affected population was emphasized. Although the program had no direct interaction with the indirect beneficiary population, feedback mechanisms for the direct beneficiaries were established through local working groups and regional committees. These spaces allowed us to receive information on the impact of the program activities, which facilitated adaptive management and the re-evaluation of processes when necessary. Voluntary service is a special kind of work and commitment which had opposite impacts on the program itself. Volunteer organizations were much more committed to the processes and their results were highly impressive, thus their involvement was considered very positive. The volunteers usually come from the same community where they give their service, hence they are motivated to “save their own people”. As they do not expect financial recognition, receiving high quality tools and useful and professional training gave them a lot of motivation. In contrast, the official firefighters were salaried and worked within fixed schedules and would not want to join, for example trainings on the weekend. It was also mentioned that they have different trade unions, which support them in not giving their extra time for the organization. It could be observed directly by the evaluation team the difference in the approach, motivation, and attitude of the voluntary and official groups, which can also be seen in the USAR percentage. The negative aspect of volunteers is that there can be a higher rotation, as they might need to leave their service to fulfill jobs, family duties, etc. and their availability is mostly out of working hours. Hence in some cases the training given to the volunteers who left, can be considered lost. However, these were only a few cases and in different organizations the volunteers are committed for a “lifetime” and already involved from a young age alongside their parents. Another negative aspect is that volunteers can be individuals without prior education and training, hence they often lack the necessary knowledge and expertise to attend high risk events. To overcome this issue, these organizations have their own training process. Another challenge with volunteer organizations is the lack of new volunteers, as young individuals are interested in other types of activities. Before it was possible to engage children, hence it helped to plan the seed of volunteerism and several staff today have been a part of the organization since they were young. The only exception is the Red Cross, which – for its different type of work and high reputation globally – does not struggle to find new volunteers. “The PREPARE Eje Cafetero program has achieved significant progress in seismic risk management in the region. The positive aspects identified, such as the efficiency in the implementation of preventive measures, the effectiveness in damage assessment, the impact on early response capacity, the relevance of seismic microzonation, the sustainability of the program, the inclusion of different actors and the cooperation between the SGC and Miyamoto, demonstrate the importance and value of this initiative. However, the negative aspects identified need to be addressed, such as lack of resources, lack of political support, conflicts of interest, the need to strengthen psychosocial awareness and training, the implementation of seismic standards, the lack of knowledge in local administrations, the need for a public information management system, the lack of capacities in academia and the need for greater cooperation between different actors.” – Miguel Mora, Colombian Geological Service These lessons learned will serve as a guide for future programs and programs in the field of risk reduction and emergency management, promoting greater coordination among actors, effective participation of the direct beneficiaries and the integration of results into planning instruments. LESSONS APPLIED Evaluation of the fulfillment of the “Lessons Learned: PREPARE and EBRP Programs in Latin America and the Caribbean (LAC)” report. 36 | FINAL EVALUATION OF PREPARE EJE CAFETERO Continuously engage local, regional, and national stakeholders at the political and technical levels Fulfilled Have ongoing communication with local partners to generate program activities based on their needs Fulfilled Have regular meetings with staff from all levels of partner organizations Fulfilled Have regular scheduled meetings with community leaders and partners to ensure accountability N/A Involve academic institutions in the programs to achieve sustainability Fulfilled Implement programs in collaboration with partner organizations that work at the local level, such as international organizations and NGOs Fulfilled; local partners Do not create programs without a logical framework that breaks down objectives into outputs and activities with specific budgets N/A Do not develop new program proposals without involving local staff and partners Fulfilled Do not hire local consultants without properly defining and establishing the service they should provide Fulfilled Appropriately use institutional branding Fulfilled High quality communication materials Partially, the quality was high prepared by an interdisciplinary group, but the program was not that visible. It is understood that it was not a program for public campaigning. Formally present and socialize the program with the local authorities, the technical representatives of the institutions, the local leaders, and community participants. Fulfilled Use different resources for the socialization and promotion of the program Partially; Throughout the program, several videos and materials were produced highlighting the progress and impact of the program; however, given that it is a BHA guideline that the management of social networks is done only through the communications office, all information related to the Program was been channeled through them. Also, the Departmental and Municipal Councils response groups contributed to the dissemination of events facilitated by the Program through their accounts. Ensure strict compliance with all offers made Fulfilled Have a “hook” to ensure the attendance of participants in the post￾training follow-up and evaluation processes Information not available FINAL EVALUATION PREPARE EJE CAFETERO | 37 Work closely with governmental and academic institutions Fulfilled Do not intervene in a community that is not interested Fulfilled Respect set meetings times Fulfilled Do not cancel meetings without prior notice Fulfilled Involve and inform stakeholders of all program activities Fulfilled Maintain transparent and clear communication Fulfilled Promote continued improvement of administrative and financial processes Information not available Capitalize on knowledge and expertise of individuals to make processes more efficient. Fulfilled Use appropriate program planning tools, such as work plans, M&E trackers, etc. Fulfilled Promote program visibility Partially Give Program Managers access to the budgets Fulfilled Have monthly BvA and programmatic review meetings Fulfilled Have clear terms of reference when hiring a consultant Fulfilled Properly execute the budget according to the plan/programming Information not available Deliver payment documents on time and without errors Information not available Not to have limited integration among the country teams in the region Information not available Do not require too much and frequent additional reporting Fulfilled Not to have inflexible procurement processes Information not available Not to hire staff without a minimal onboarding process Information not available CONCLUSIONS 1. To what extent have the overall goal and objectives of the program been achieved? Detailed analysis can be found under the Impact section of the report, structured by each objective. In summary: The overall goal: “Strengthen the capacity for disaster response and risk reduction in the Eje Cafetero region of Colombia” was achieved to a significantly high level. Objective 1: Enhance the availability and coverage of seismic exposure, vulnerability and risk data for risk management policy and planning. This objective was achieved, as several documents, analyses, and studies were developed. The initial needs assessment which was the base for deciding the most useful and needed activities was highly useful. The repository that gathered all the existing information and is available to all, including officials responsible for planning, and is integrated into the UNGRD repository was an important accomplishment. http://repositorio.gestiondelriesgo.gov.co/page/partners. Also, the seismic exposure models developed for Armenia and Calarca are very useful. During the NCE phase more and new activities were additionally implemented such as the homogenization and standardization at the regional level of the existing damage assessment tools and methodologies. 38 | FINAL EVALUATION OF PREPARE EJE CAFETERO Objective 2: Strengthen the capacity of disaster authorities to understand and apply seismic risk data for improved policy, planning and implementation outcomes. While several authorities were strongly involved in the process and constantly included in the processes as well as trained (hence the objective being achieved), the high rotation of these positions might hinder these results down the line. As a solution the program prepared a knowledge strengthening plan and emphasized including the developed documents into government policies, which in some cases are already implemented and in others are in progress of being so. Objective 3: Strengthen the capacity of municipal agencies, USAR teams and other first responders to plan and implement effective response operations. This objective was not only achieved but exceeded the targets by more than 4.5 times. The strengthened capacities were seen in practice in different operations and the USAR accreditation results show the real impact. Almost all participants interviewed were extremely grateful for the PREPARE Eje Cafetero program and expressed that the impact was of a very high level as it substantially increased their level of preparation through training and the provision of high-quality equipment according to their needs. Objective 4: Strengthen regional cooperation in risk management and response through joint planning, information, and resource sharing. This objective was also achieved at a high level, as cooperation has increased within the region. The efficient joint work of the different entities from the three departments during the Nevado de Ruiz Volcano activation showed the program’s real impact. Evening out the knowledge and capacities within the region and within the different institutions, as well as the homogenization and unification are important results. 2. What have been the main challenges in achieving the program's goals and objectives? Several risks which were originally identified materialized, such as the high rotation of personnel, lack of political will in some cases, change of leadership of local and national authorities, difficulty to engage the private sector and other entities, and long administrative processes at the governmental level to include the documents into policies. Respective information can be found under the challenge sections. 3. What program activities have been most effective in achieving the stated objectives? Objective 1: Enhance the availability and coverage of seismic exposure, vulnerability and risk data for risk management policy and planning. • The initial needs assessment, which was the basis for the planning of the most useful and needed activities. • The knowledge repository which gathered all the existing information and is available to all including officials responsible for planning. • The seismic exposure model is useful; however these models should be developed for more locations in the region in the future per request of the interviewees. Objective 2: Strengthen the capacity of disaster authorities to understand and apply seismic risk data for improved policy, planning and implementation outcomes. • The knowledge strengthening strategy which was a very rigorous and detailed product involving several partners and with a high impact and usefulness. It includes a specific knowledge transfer plan for non-technical people which was developed by an interdisciplinary professional group. • The different workshops, meetings and documents which increased the capacities of policy makers. Objective 3: Strengthen the capacity of municipal agencies, USAR teams and other first responders to plan and implement effective response operations. FINAL EVALUATION PREPARE EJE CAFETERO | 39 • The capacity building component including the training as well as the donations had the biggest impact on the first responder groups, several of which managed to achieve their national USAR accreditation due to the support of the program. Objective 4: Strengthen regional cooperation in risk management and response through joint planning, information, and resource sharing. • Regional damage assessments, regional response protocols, regional psychosocial protocol and Training of Trainers course, and other regional products. • Joint trainings, events, and work (e.g., psychosocial) at the regional level helped to get awareness of the capacities of other groups as well as to develop direct relationships between members which resulted in increasing effectiveness during response operations. • Joint response preparation for the Nevado de Ruiz activation. 4. To what extent has the involvement of national stakeholders shaped and helped to develop program outputs and outcomes? All stakeholders (local governments, national entities, and first responders) were part of the program planning, implementation, and monitoring. Therefore, the program itself was shaped by national stakeholders as well. This strong collaboration is highlighted in several parts of the report and was also mentioned by the interviewed individuals. 5. To what extent have the outputs and outcomes included participation and accountability to the affected population, including gender considerations? The direct beneficiaries (officials, first responders, and universities) were significantly involved in the whole process, starting from the planning, hence the accountability of the program was high. The indirect population and the affected communities would be involved in a next phase. This part is detailed in the participation and accountability section. As for the gender aspect, within the limitations of the program to include the gender aspect being it technical and considering that the list of beneficiaries was externally set and not under the control of the program, the evaluation considers that the program achieved good results. Details of gender equity can be found under the gender section. 6. How did specific intended and unintended changes in program activities positively or negatively affect the program? Unintended positive outcomes were the following: 1) psychosocial component, Training of Trainers, and the protocol, 2) regional and inter-institutional cooperation put in practice in real life in such a big event as the preparation for the activation of the Nevado Ruiz volcano, 3) first female USAR commander, and 4) radiocommunication aspect. During the NCE, the program conducted a process of homogenization and standardization at the regional level of the existing damage assessment tools and methodologies. During this period new training for medical USAR units was developed as well as further evaluation of damages. There have not been unintended negative outcomes as the challenges and weaknesses were related to risks identified prior to the program. 7. To what extent have program activities contributed to sustainable changes or outcomes and what factors contributed to achieving sustainability? The new capacity built, the increased understanding of authorities, officials, and rescue groups of the issues related to seismic risk, the USAR accreditations, the received donations, the documents which will be included in local legally binding policies, the knowledge repository including all relevant information needed and the trained policy makers as well as the trained trainers prepared to share their knowledge are all elements of sustainability. Respective part can be found under the sustainability section. 40 | FINAL EVALUATION OF PREPARE EJE CAFETERO 8. How was the exit strategy and Sustainability Plan set up? The program itself had the approach of installing capacities, tools, and knowledge within individuals, institutions, and tools so that the beneficiaries could continue the work achieved when the program was over. The program had its exit strategy and continuity processes planned out in the different phases of the program design. However, there has not been a detailed activity specific exit strategy and the stakeholders are eager to continue the program in the next phase. Respective part can be found under Sustainability section. 9. Has coordination and communication between agencies and institutions increased because of participation in the program? Yes, this objective was achieved at a high-level, the details can be found under Impact/Objective 4 as well as under section “Coordination and harmonization”. 10. Have mechanisms been established to maintain the public-private partnership beyond the life of the program? A few MoUs were signed with universities and national entities, and in some cases (e.g., psychosocial component), the collaboration will continue after the program. However, the private sector was difficult to get commitment from (e.g., the Association of the Engineers of Caldas signed a letter of compromise for the damage assessment studies) and with the upcoming local elections the officials most probably will change. 11. Have the tools developed under the program been put into practice by the stakeholders? The equipment provided to the USAR groups as well as the trainings were used in different emergencies and the teams’ efficiency increased accordingly (e.g., Firefighters Pereira, Manizales, Dosquebradas and Volunteer Firefighters Manizales, Red Cross Caldas). Further examples were shared by participants. One good example is the preparation for the Nevado Ruiz activation. These can be found in the personal stories within apostrophe. RECOMMENDATIONS This section focuses on suggestions for improvement. • The continuation of the program is recommended at two levels: o Widen the focus to prepare communities, as they are the real first responders o Standardize the preparation of different groups, especially with regards to technical equipment (e.g., communication equipment which is essential during emergencies as several groups have outdated equipment which is not compatible with the equipment of other groups) • A second phase is important, due to the high level and variety of risks the region is prone to. The suggestion is not only to work on seismic risk, but on multi-hazard risk, hence also including other scenarios such as mass movements, floods, torrential floods, etc. “In the Eje Cafetero region the only threat not present are tsunamis, all the rest are here. We are in constant response at a daily level and with more complex situations which we need preparation for as well” – Alexander Galindo, DIGER Pereira • Continue with trainings for new USAR members to ensure they are properly prepared • Even though some firefighters might not have the same level of education, especially within the indigenous groups, they could be even more professional in this work, hence it is important to adapt the technical language and methodology to the different education levels of people as well • Strengthen the psychosocial component with more trainings, as several institutions have not received any or even heard about it yet FINAL EVALUATION PREPARE EJE CAFETERO | 41 • Some of the interviewed participants requested knowledge exchange opportunities at national or international level “While the actions met the expectations, some components demonstrated the need for continuity. For example, the Department does not have an optimal communication network for the articulation with the Departmental DRM system, and it does not have an operational and administrative center for emergency - Departmental DRM.” – Juliana Escandon, Departmental Disaster Risk Management Unit of Quindío. • Structure the information contained in the repository, especially that which contain technical data necessary for critical decision-making, considering, among other important variables, those contained in Law 1523 of 2012. • Include the roles and responsibilities of the trained instructors in the risk management plans as it provides greater coordination and coherence in the response actions, as well as the protocols in the chapter on communication with the public • Establish more detailed program exit strategy of specific activities to ensure sustainability as lack of funding might impede continuation of next phase • Administratively, it is suggested to have a specific logistical framework as well as a tool for planning and monitoring future programs with more relevant indicators where possible • Increase the number of participants from the private sector, for example it might be helpful to talk to road concession companies, as well as associations of representative sectors as well as the corporates. This could be facilitated by the Caldas Association of Engineers, who participated with representatives in the regional damage assessment workshops • Continue feeding and developing the Eje Cafetero repository, and include it within the regional risk management plan • Provide the latest set of specialized equipment and training to search and rescue groups (USAR) and municipal and departmental entities • Continue with inter-departmental work and programs to enhance collaboration • Search for new ways of overcoming the risk of losing built capacities due to high overturn • Promote volunteerism as an effective strategy for maintaining committed groups of people who are based in their communities • Create exposure models for residential buildings in the urban area of further cities as well (in the program Manizales had the actualization of its exposure model while Pereira, Armenia and Calarca had received the first one). RECOMMENDATIONS FROM PARTICIPANTS • To continue the program in a next phase and address issues such as risk awareness and strengthening the geographic information system – Caldas Red Cross “We need higher quality communication equipment and capacities to see the information in real time and be able to give a more adequate response. It would be also useful to have exchanges with other countries and international trainers. We would need this to be able to set up a complete telecommunications system, solve transport issues, improve logistical capacity, establish a specialized rescue unit, and have mobile utility equipment” – Volunteer Firefighters Dosquebradas • Continue the process of strengthening the response groups and maintaining high standards. – Official Firefighters, Pereira • Challenges and needs: The need for more resources to meet the multiple needs of the territory is essential and the importance of the continuity of the process to maintain the achievements is highlighted. – DIGER, Pereira • Continue to prepare microzonification studies in other cities as well – CGS 42 | FINAL EVALUATION OF PREPARE EJE CAFETERO • Adapt equipment/materials to the reality of the terrain, as some are not very useful in real situations, e.g., the big boxes with wheels containing the heavy tents are not possible to transport in the mountainous terrain of Colombia, would need handles as in the military. The tents are too heavy for deployment when there is not adequate transportation and not enough people to carry them. The helmets would need to have lights on them. “The achievement of other complementary elements that would optimize the capabilities of the equipment Light USAR teams as now some equipment hinders its full use.” – Mauricio Ruiz, Quindío Civil Defense • Miyamoto and PREPARE should be implemented in other regions of the country as well, for example at the Northern part of Colombia – UNGRD FINAL EVALUATION PREPARE EJE CAFETERO | 43 ANNEXES ANNEX I: TIMELINE MAIN ACTIVITIES TEAM SCHEDULE Level of Effort Anna Wrochna Alejandro Gómez Total Days Days Days Meetings and consultation process/evaluation design 3 1 4 May 15-22 Design of data collection tool and work plan 3 1 4 May 15-22 Preparation of interview schedule 0 1 1 May 15-22 Documentary review and data collection: Program agreement, award document, M&E plan, all reports submitted to USAID/BHA and specific output reports, and partner meeting minutes 2 4 6 May 23-28 Interviews and/or online secondary data collection 2 2 4 May 23-28 Individual Interviews 4 4 8 May 29 – June 1 Draft of Final Evaluation Report 3 3 6 June 2 – 12 PowerPoint presentation of preliminary key facts 1 1 2 June 13 Feedback and adjustments to draft report 2 2 4 June 13-20 Translation 3 3 June 21-23 Final version of Final Evaluation Report 2 2 4 June 23 Total 25 21 46 44 | FINAL EVALUATION OF PREPARE EJE CAFETERO ANNEX II: EVALUATION STATEMENT OF WORK ANNEX III: EVALUATION METHODS AND LIMITATIONS ANNEX IV: DATA COLLECTION AND ANALYSIS TOOLS Final Evaluation of the PREPARE Eje Cafetero Program Focus Group Interview Guide Interview Method: In Person Names and Last Names: Institution/Organization: USAR Position/Title: Age: Gender: Contact Information: Date: Time: Interview Duration (max. 90 min): Important Note: Before starting the interview, a brief presentation is made to contextualize the interview and to request permission to record the interview under the commitment of confidentiality of its content. At the beginning of the interview, the purpose of the interview is presented and rules of confidentiality and the importance of listening to other participants are established. STAKEHOLDER METHODOLOGY DATE TIME Realizado Posicion MANIZALES, CALDAS 29 Mayo Project Management Team Interview 29 Mayo 9:00-10:30 Jaime Eraso OGR CALDAS Focus group 29 Mayo 11:00-12:30 Felix Giraldo Director Gestión del Riesgo Caldas OGR MANIZALES, Univ Focus group 29 Mayo 14:00-15:30 Alexa Morales Directora Gestión del Riesgo Manizales William Gaviria Universidad de Manizales Marta Yara Trabajadora social Bomberos Comandante Alvaro Vasquez Ingeniero civil USAR BV MANIZALES Interview 29 Mayo 16:00-17:00 Nathalia Valencia Comandante/Lider Grupo USAR 30 Mayo USAR CR CALDAS Interview 30 Mayo 9:00-10:00 Daniel Fernando Cano Mejia Gestor de Conocimiento de Reducción de Riesgo de Desastres PEREIRA, RISARALDA USAR DOSQUEBRADAS Interview 30 Mayo 11:30-13:30 Comandante Lina Marín Comandante/Lider Grupo USAR USAR OFICIALES PEREIRA Interview 30 Mayo 16:30 - 17:30 Alejandro Arango Director BO Pereira OGR PEREIRA, Univ. Focus group 30 Mayo 15:30-17:00 Alexander Galindo and team Director Gestión del Riesgo Pereira DIGER 31 Mayo USAR DC RISARALDA Interview 31 Mayo 9:00-10:00 Alfredo Muñoz Director Defensa Civil Risaralda USAR DC RISARALDA Jairo Rojas Operativo Defensa Civil Risaralda USAR BV CHINCHINA Interview VIRTUAL31 Mayo Andres Gomez Lider USAR OGR RISARALDA Focus group 31 Mayo 11:00 - 12:00 John Jairo de la Pava Coordinador operativo CDGRD Risaralda ARMENIA OGR Quindio Focus group 31 Mayo 14:00-15:00 Monica Camacho Directora Gestión del Riesgo Quindio Juliana Escandon Operativa Gestión del Riesgo Quindio Pablo Cesar Defensa Civil Ariel Ospina Cruz Roja Project Team Faber Mosquera Project Coordinator OGR Armenia Focus group 31 Mayo 17:00 -18:00 Javier Velez Director OMGERD Universidad Gran Colombia Ximena Cifuentes Decana de la Facultad de Ingeneria 1Junio USAR DC QUINDIO Interview 1 Junio 9:00-10:00 Mauricio Ruiz Rodriguez Director Defensa Civil Quindío Pablo Cesar Integrante grupo USAR DC Quindío USAR CR QUINDIO Interview 1 Junio 10:30-11:30 Cristhian Martinez Director seccional Jacobo Franco Lider grupo USAR CR Quindío FINAL EVALUATION PREPARE EJE CAFETERO | 45 Areas of Analysis Key Issues Answers 1 Introduction and Background Ask for name and surname, position, organization/institution 2 Role and involvement in the program How long have you been involved in the project? 3 What has been your role (or that of the organization you present) in the project? 4 What events, activities, products have you participated in? 5 What do you believe has been your “value added” to the program (or the organization you represent)? 6 Main achievements and changes What do you consider to be the most outstanding results and achievements of the project? Which activities have been the most useful for you in your work/for your organization? 7 Would you recommend other institutions/regions to participate in another similar project? Why? 8 From 1 to 10 how would you rate the impact of the project? 9 Participation in trainings and events Have you participated in any trainings? Which ones? 10 How many trainings have you participated in/attended? 11 On a scale of 1 to 10, how would you rate the quality of the training? 12 Do you have any suggestions for the organizers/trainers? 13 Specific questions on program results Can you give us some examples of the acquired capabilities and results of the program activities? 14 How would you assess your level of knowledge and understanding of seismic hazards and vulnerability in the region compared to your knowledge prior to taking part in the program? 15 Can you give us some examples of the acquired capabilities and results of the program activities? 16 How would you rate your level of technical knowledge and understanding of seismic hazards and vulnerability from 1 to 10? 17 How would you rate your level of preparedness in emotional self-care and psychological first aid after training on the topic if you have taken it? Rate from 1 to 10. 18 Can you give us some examples of how you have effectively applied risk knowledge in your planning and capacity building activities? 46 | FINAL EVALUATION OF PREPARE EJE CAFETERO Areas of Analysis Key Issues Answers 19 Pending/Not achieved What results or activities were delayed or could not be achieved? Explain possible reasons. 20 On the future How will you continue with the activities/how will you maintain program results after program completion? Do you have the managerial, financial and technical capabilities to implement and ensure program sustainability? 21 What are the challenges for continuation? 22 Specific questions on program results To what extent do you feel that your institution/members are empowered after the trainings? In which area/topic/subject? 23 Have USAR teams and other first responders used the results of the capacity assessment to prioritize and address capacity gaps and weaknesses? Which of the tools developed under the program have been implemented by you? 24 Cooperation Has coordination and communication between agencies and institutions from different sectors and departments increased as a result of participation in the program? Can you give some specific examples? 25 How many organizations have you collaborated with during the program? 26 Have lessons learned been shared between regions, and has dialogue between partners been encouraged? 27 What strategies have been developed to raise public awareness of risks? What tools have been developed to improve public knowledge of seismic risk and to facilitate citizen participation in risk management actions? What are the communication tools? 28 Cross-cutting issues In the activities in which you have participated, to what extent have vulnerable populations been considered, including women, individuals with disabilities, LGBTQ+ groups, and others? 29 What kind of specific measures have been taken within the program activities to encourage the participation of women? 30 Miyamoto’s role and coordination How has the relationship with Miyamoto been? 31 What has been Miyamoto’s main value added? 32 What would you recommend to Miyamoto for the future? 33 Main Lessons What lessons can be learned from PREPARE Eje Cafetero for a possible replication of the project in other parts of Colombia or in the region? What would you have done FINAL EVALUATION PREPARE EJE CAFETERO | 47 Areas of Analysis Key Issues Answers differently and how? What lessons have you personally learned? 34 Other Final Issues Is there any other feedback or suggestions you would like to share regarding the overall project? EVALUATION MATRIX Questions Interviewees Additional Resources IMPACT What are the most outstanding results and achievements of the project? - Miyamoto Team - Municipal OGR - Departmental OGR - USAR - UNGRD - University Representatives - Colombian Geological Service - Volcanological and Seismological Observatory - Review of program documents (monitoring reports, donor reports, factsheets) - Schedule (Gantt chart) What indicators were used to evaluate the impact of the program? Do you consider that they have appropriately demonstrated the achievements and impact of the program? - Miyamoto Team To what extent has the program achieved its intended impact: “Institutions responsible for disaster risk management and response have improved risk data, training and cooperative strategies that address existing capacity gaps and deficiencies.” - Miyamoto Team Have you participated in any training? How would you rate the training and why? How would you rate the level of the final tests and the knowledge retention test? Do you have any suggestions to the organizers/trainers for modifications? - Municipal OGR - Departmental OGR - Universities - USAR - Workshop reports If you have provided/conducted a training, what was your perception of 1) the level of knowledge, 2) interest, 3) the level of ability after the training? - Universities To what extent does your institution feel empowered after the training? In which area/topic/subject? - Municipal OGR - Departmental OGR - USAR 48 | FINAL EVALUATION OF PREPARE EJE CAFETERO How would you rate your knowledge and ability to adequately interpret seismic risk data information? Do you have examples of your success in applying and implementing seismic risk data and information in related policies and plans? - Municipal OGR - Departmental OGR Obj.2. Have training participants demonstrated a better understanding of seismic hazards and vulnerability in the region? How would you assess the capacities of the participants after the trainings compared to the initial assessment? - UNGRD - Universities - Miyamoto Team Knowledge retention test How would you assess your level of knowledge and understanding of seismic hazards and vulnerability in the region compared to your knowledge prior to taking part in the program? - Municipal OGR - Departmental OGR - USAR Can you give us some examples of the skills acquired and the results of the program activities? - Municipal OGR - Departmental OGR - USAR Can you give us an example of how you have affectively applied risk knowledge in your planning and capacity building activities? - Municipal OGR - Departmental OGR How do you assess the availability of and access to risk and vulnerability data in your region and its inclusion in policies after the program compared to before? - Municipal OGR - Departmental OGR Public policies Have the Project products and their results (the information acquired) been included in risk and disaster reduction plans and policies? - Municipal OGR - Departmental OGR Have USAR teams and other first responders used the results of the capacity assessment to prioritize and address capacity gaps and weaknesses? - USAR Which of the tolos developed under the program have been implemented by you? What has been their level of adoption? USAR: Have you been able to use the knowledge and disaster response tools? (Feb.2022.Landslide) - Municipal OGR - Departmental OGR - USAR If you have supported the development of any program deliverable/product (study, repository, training, evaluations, plans, risk study, tools, information collection, etc.) how was: 1) the development process, 2) the cooperation among those involved, 3) the - Universities - UNGRD - Observatories Product Documents FINAL EVALUATION PREPARE EJE CAFETERO | 49 usefulness of the final result, and 4) your roles? EFFECTIVENESS To what extent were the planned results achieved, or not? According to the levels of implementation observed, what are the possibilities of reaching the established goals? - Miyamoto Team Monthly and Semi￾Annual Reports What are the most outstanding results obtained so far in relation to the main objectives? - Miyamoto Team In your experience, what specific program activities have been most effective in achieving the objectives? - Miyamoto Team Which activities have been the most useful for your preparation/role/job? - Municipal OGR - Departmental OGR - USAR - Universities What were the main internal and external factors that influenced the achievement, or not, of the initially planned results? - Miyamoto Team Were there any unplanned results? Which ones? - Miyamoto Team What interventions and outputs were not delivered according to schedule? Have any of the delays affected the implementation of the results? How have the intended and unintended changes affected program activities, either positively or negatively? - Miyamoto Team What were the main challenges faced in achieving the program’s goals and objectives? - Miyamoto Team Did any of the planned risks materialize and were there any other risks that occurred? - Miyamoto Team EFFICIENCY Were the expenses in line with the budget? If not, what were the reasons? - Miyamoto Team - Budget - Financial Reports Were there any excessive savings/expenses (underspending/overspending)? How have they been documented and managed? - Miyamoto Team - Budget - Financial Reports Were the selected activities and processes cost-effective? - Miyamoto Team 50 | FINAL EVALUATION OF PREPARE EJE CAFETERO How did the program coordinate with other initiatives and organizations in the same area of work in the region? Did this allow for cost of activities to be reduced? - Miyamoto Team SUSTAINABILITY To what extent have program activities contributed to sustainable changes or outcomes and what factors have contributed to achieving sustainability? - Miyamoto Team How will you continued with the activities/how will you maintain the results of the program after its completion? - Municipal OGR - Departmental OGR - USAR - Universities Letters of Commitment How was the exit strategy and sustainability plan established? - Miyamoto Team What activities, initiatives or strategies were specifically undertaken to ensure the financial sustainability of the program? - Miyamoto Team Do the implementing partners have the managerial, financial, and technical capabilities to execute and ensure the sustainability of the program? - Miyamoto Team What is the level of ownership of the program among implementing partners, participants, allies, community members and civil society, and private sector? - Municipal OGR - Departmental OGR - USAR - Universities Were mechanisms established to sustain the public-private partnership beyond the life of the program? - Miyamoto Team Letters of Commitment What is your plan to continue the collaboration between public and private entities? - Municipal OGR - Departmental OGR - Universities Have mechanisms been established to maintain the public-private partnerships beyond the life of the program? - Miyamoto Team - Municipal OGR - Departmental OGR - Universities PARTICIPATION AND COOPERATION Have mechanisms been established to maintain the public-private partnership beyond the life of the program? - Miyamoto Team - Municipal OGR - Departmental OGR - USAR - Universities - UNGRD - Observatories FINAL EVALUATION PREPARE EJE CAFETERO | 51 Is the program coherent and complementary to other programs or projects of other organizations in the same region? - Miyamoto Team What was the participation of civil society organizations and the private sector in the program and can you give specific examples? - Miyamoto Team - Municipal OGR - Departmental OGR - USAR - Universities - UNGRD - Observatories Have lessons learned been shared between regions, and has dialogue between partners been encouraged? - Miyamoto Team - Municipal OGR - Departmental OGR - USAR - Universities - UNGRD - Observatories What strategies were developed to raise public awareness of risks? What tools have been developed to improve public knowledge of seismic risk and to facilitate citizen participation in rick management actions? Tell us, if you know, more about the knowledge repository! - Miyamoto Team - Municipal OGR - Departmental OGR - USAR To what extent have program outputs and outcomes included participation and accountability to the target population? Indicator: Though disaster risk analysis will be largely limited to technical experts and risk management authorities and planners, sessions will be facilitated to address the needs of traditionally excluded or marginalized groups, particularly in impoverished, high-risk communities - Miyamoto Team Broader stakeholder engagement and planning activities, particularly those designed to encourage active private sector participation, will enlist community leaders and others with strong standing in vulnerable communities, to advocate for their interests in the formulation of risk management policy and plans; GENDER How well have gender equity issues been incorporated into the planning, implementation, and monitoring of the program? - Miyamoto Team - Proposal - Reports To what extent have the plans, policies, documents, strategies, and trainings developed and implemented considered the vulnerable - Miyamoto Team - Proposal - Reports 52 | FINAL EVALUATION OF PREPARE EJE CAFETERO population, including gender, women, individuals with disabilities, LGBTQ+ groups and others? What were the challenges in achieving gender equity results and how were they addressed? - Miyamoto Team What kind of specific measures were taken within the program activities to encourage the participation of women? - Miyamoto Team How (work methodologies, training approaches, technical solutions, etc.) did program activities take into account the differences between vulnerable groups at the national and local level? - Miyamoto Team QUESTIONS TO WOMEN PARTICIPANTS How did the program activities include you in the program and address gender issues in the different activities? Women from: - Miyamoto Team - Municipal OGR - Departmental OGR - USAR - Universities - UNGRD - Observatories LESSONS LEARNED What lessons can be learned from the Eje Cafetero program for possible replication in other parts of Colombia or the region? What would you have done differently and how? - Miyamoto Team - Municipal OGR - Departmental OGR - USAR - Universities - UNGRD - Observatories Did you receive the necessary support and resources to carry out your work effectively, and was there anything that could have improved your performance? - Miyamoto Team What things about the program or the program team do you think could be improved for future similar projects? - Miyamoto Team Is there any other feedback or suggestions you would like to share regarding the overall program? - Municipal OGR - Departmental OGR - USAR - Universities - UNGRD - Observatories How has the relationship with Miyamoto been? What has been Miyamoto’s main added value? - Municipal OGR - Departmental OGR - USAR FINAL EVALUATION PREPARE EJE CAFETERO | 53 What would you recommend to Miyamoto for the future? - Universities - UNGRD - Observatories ANNEX V: SOURCES OF INFORMATION • Miyamoto (2020 June): Annex F Revised Safety and Security Plan Eje Cafetero • Miyamoto (2020 March): Annex H PSEA Implementation Plan Eje Cafetero • Miyamoto (2020 March): Annex Ja Monitoring and Evaluation Eje Cafetero • Miyamoto (2020 March): Annex Jb Monitoring Table Eje Cafetero • Miyamoto (2020 June): Annex M Logistics Plan Original Eje Cafetero • Miyamoto (2021a). Reporte de recolección de información, Actividad Uno, Componente Uno - PREPARE Eje Cafetero. Colombia. • Miyamoto (2021b). Reporte de diseño de repositorio y base de datos, Actividad Dos, Componente Uno - PREPARE Eje Cafetero. Colombia. • Miyamoto (2022, Feb): Investigación de repositorios Nacionales e internacionales y Regionales del eje cafetero • Componente 1 - PREPARE Eje Cafetero. Colombia. • Miyamoto (2022, Feb): Plan para la administración y gestión del repositorio del conocimiento del riesgo sísmico y gestión del riesgo sísmico en el Eje Cafetero - PREPARE Eje Cafetero. Colombia. • Miyamoto (2023, May): Acta de entrega de documentos Modelo de Exposición Sísmica de Armenia y Calarcá (Quindío). • Miyamoto (2020, Jun): USAID/OFDA PREPARE Eje Cafetero Miyamoto International, Inc. – Technical Proposal. • Miyamoto (2020, Jun): PLANEACIÓN Y DESCRIPCIÓN GENERAL DE COMPONENTES PREPARE Eje Cafetero. • Miyamoto (2020, Dec): USAID-BHA PREPARE Eje Cafetero Baseline Report • Miyamoto (2022, Jan): USAID-BHA PREPARE Eje Cafetero Estrategia fortalecimiento conocimiento. • Miyamoto (2023, Feb): Modelo de exposición de edificaciones residenciales del área urbana de Armenia y Calarcá (Quindío) Informe final. • Miyamoto (2021, Jun): Diagnóstico de capacidades proyecto PREPARE Eje Cafetero equipo USAR o Diagnostico grupos USAR - UNGRD o Anexo 5. Capacidad Operativa Estándar Mínimo EHA’s por Nivel de Acreditación o Anexo 8. Matriz de Capacitación por Función Equipo USAR Nacional V2 (2019) o Anexo 9. Modelo Plan de trabajo documental – NAP – V1 o Anexo 11. Formato Lista de Chequeo evaluación V2 • Miyamoto (2022, May): Matriz de priorizacion grupos USAR • Miyamoto (2023, Jun): PROTOCOLO Respuesta psicosocial en situaciones de emergencia o desastres en el Eje Cafetero • Miyamoto (2021): 1. Psychosocial R&R Training Contracting Agreement • Miyamoto (2021): Estrategia interdepartamental de respuesta psicosocial v1 • Miyamoto (2021): Propuesta formación psicosocial Miyamoto • Miyamoto: Formulario regional para evaluación rápida de daños en edificaciones • Miyamoto (2021-2023, Oct - Apr): PREPARE Eje Cafetero USAID Monthly Updates, 20 o Miyamoto (2021 Sept): PREPARE Eje Cafetero USAID Monthly Updates o Miyamoto (2021 Oct): PREPARE Eje Cafetero USAID Monthly Updates o Miyamoto (2021 Nov): PREPARE Eje Cafetero USAID Monthly Updates o Miyamoto (2021 Dec): PREPARE Eje Cafetero USAID Monthly Updates o Miyamoto (2022 Jan): PREPARE Eje Cafetero USAID Monthly Updates o Miyamoto (2022 Feb): PREPARE Eje Cafetero USAID Monthly Updates 54 | FINAL EVALUATION OF PREPARE EJE CAFETERO o Miyamoto (2022 March): PREPARE Eje Cafetero USAID Monthly Updates o Miyamoto (2022 April): PREPARE Eje Cafetero USAID Monthly Updates o Miyamoto (2022 May): PREPARE Eje Cafetero USAID Monthly Updates o Miyamoto (2022 June): PREPARE Eje Cafetero USAID Monthly Updates o Miyamoto (2022 July): PREPARE Eje Cafetero USAID Monthly Updates o Miyamoto (2022 Aug): PREPARE Eje Cafetero USAID Monthly Updates o Miyamoto (2022 Sept): PREPARE Eje Cafetero USAID Monthly Updates o Miyamoto (2022 Oct): PREPARE Eje Cafetero USAID Monthly Updates o Miyamoto (2022 Nov): PREPARE Eje Cafetero USAID Monthly Updates o Miyamoto (2022 Dec): PREPARE Eje Cafetero USAID Monthly Updates o Miyamoto (2023 Jan): PREPARE Eje Cafetero USAID Monthly Updates o Miyamoto (2023 Feb): PREPARE Eje Cafetero USAID Monthly Updates o Miyamoto (2023 March): PREPARE Eje Cafetero USAID Monthly Updates o Miyamoto (2023 April): PREPARE Eje Cafetero USAID Monthly Updates • Miyamoto (2020 Sep- 2021 Mar): USAID-BHA PREPARE Eje Cafetero, 1 Semi-annual Updates • Miyamoto (2021 Apr - Sept): USAID-BHA PREPARE Eje Cafetero, 2 Semi-annual Updates • Miyamoto (2021 Oct – 2022 March): USAID-BHA PREPARE Eje Cafetero, 3 Semi-annual Updates • Miyamoto (2022 Apr - Sept): USAID-BHA PREPARE Eje Cafetero, 4 Semi-annual Updates o Success Stories: USAR ACCREDATION GROUPS OF MANIZALES VOLUNTEER FIREFIGHTERS & PEREIRA OFFICIAL FIREFIGHTERS o Success Stories: TRAINING OF EJE CAFETERO USAR GROUPS PERSONNEL AND AUTHORITIES o Success Stories: REGIONAL WORKSHOP ON THE INSTITUTIONALIZATION OF THE PROGRAM'S RESULTS AND PRODUCTS WITHIN PLANNING INSTRUMENTS o Success Stories: FIRST DELIVERY OF SPECIALIZED EQUIPMENT AND TOOLS FOR USAR GROUPS IN EJE CAFETERO o Success Stories: COMPLETION OF REGIONAL TRAINING FOR INSTRUCTORS IN PSYCHOSOCIAL ASSISTANCE AND RECOVERY IN EMERGENCIES • Miyamoto (2022 Oct – 2023 March): USAID-BHA PREPARE Eje Cafetero, 5 Semi-annual Updates • Miyamoto (2023). Equipment donation acts, CDGRD Caldas • Miyamoto (2023). Equipment donation acts, CD Risaralda • Miyamoto (2023). Equipment donation acts, DRMDC Risaralda • Miyamoto (2023). Equipment donation acts, RC Caldas • Miyamoto (2023). Equipment donation acts, CD Quindío • Miyamoto (2023). Equipment donation acts, RC Quindío • Miyamoto (2023). Equipment donation acts, OVSM - SGC • Miyamoto (2023). Equipment donation acts, OF Manizales • Miyamoto (2023). Equipment donation acts, UNGRD • Miyamoto (2023). Equipment donation acts, VF Manizales • Miyamoto (2023). Equipment donation acts, VF Chinchiná • Miyamoto (2023). Equipment donation acts, VF Riosucio • Miyamoto (2017-2019): Previous work Eje Cafetero (2017-2019). o Evaluación de necesidades de las Entidades, Municipio, Departamento, Bomberos voluntarios Riosucio o Evaluación de necesidades de las Entidades, Municipio, Departamento, Defensa Civil Seccional Caldas o Evaluación de necesidades de las Entidades, Municipio, Departamento, Bomberos voluntarios Riosucio o Evaluación de necesidades de las Entidades, Municipio, Departamento, Bomberos FINAL EVALUATION PREPARE EJE CAFETERO | 55 voluntarios Chinchiná o Evaluación de necesidades de las Entidades, Municipio, Departamento, Listado de Asistencia o Evaluación de necesidades de las Entidades, Municipio, Departamento, Alianza SUMA Universidades o Evaluación de necesidades de las Entidades, Municipio, Departamento, Bomberos voluntarios Manizales o Evaluación de necesidades de las Entidades, Municipio, Departamento, Cruz Roja Seccional Caldas o Evaluación de necesidades de las Entidades, Municipio, Departamento, Defensa Civil Seccional Caldas o Evaluación de necesidades de las Entidades, Municipio, Departamento, Jefatura Gestión del Riesgo Caldas o Evaluación de necesidades de las Entidades, Municipio, Departamento, Universidad Autónoma de Manizales o Clasificación y propuesta de secuencia lógica de componentes y actividades para crear o potenciar habilidades y capacidades en gestión del riesgo de desastres y respuesta a emergencias en Quindío, Colombia o Evaluación de necesidades - Consolidado Quindío Matriz PREPARE o Evaluación de necesidades - Lista Actividades Quindío o Evaluación de necesidades – Actividades Propuestas – Mayo, Junio • Miyamoto (2023). Letters of Commitment Sustainability o Miyamoto (May 2023). Letters of Commitment Sustainability, Armenia o Miyamoto (June 2023). Letters of Commitment Sustainability, Damage Assessment Eje Cafetero Entities o Miyamoto (May 2023). Letters of Commitment Sustainability, Quindío o Miyamoto (May 2023). Letters of Commitment Sustainability, ACOFI o Miyamoto (May 2023). Letters of Commitment Sustainability, UGC o Miyamoto (May 2023). Letters of Commitment Sustainability, Risaralda o Miyamoto (May 2023). Letters of Commitment Sustainability, Universidad de Manizales • Miyamoto (2023). Letters of Commitment Sustainability, Quindío • Miyamoto (2023). Letters of Commitment Sustainability, ACOFI • Miyamoto (2023). Letters of Commitment Sustainability, UGC • Miyamoto (2023). Letters of Commitment Sustainability, Risaralda • Miyamoto (2023). Letters of Commitment Sustainability, Damage Assessment Eje Cafetero entities • Miyamoto (2023). Letters of Commitment Sustainability, Carta Miyamoto International University Manizales • Miyamoto (2021): LoC - MoC - Agreements. o 021-Convenio Marco Miyamoto - UGC o Prorroga Convenio UGC o Carta de compromiso Miyamoto Departamento de Caldas o Carta de compromiso Miyamoto Alcaldía de Manizales o Carta de compromiso Miyamoto UNGRD o Convenio Marco Miyamoto UAVH INC o Convenio Marco Miyamoto Universidad Gran Colombia o Convenio Marco Miyamoto Uni Manizales o Convenio Marco Practicas y Pasantías entre Uni Quindío y Miyamoto o Memorando de Entendimiento ACOFI Miyamoto o Letter of Commitment RAP – 207 Miyamoto o Letter of Commitment Municipality of Armenia – Miyamoto o Letter of Commitment Municipality of Caldas – Miyamoto o Letter of Commitment Municipality of Manizales – Miyamoto 56 | FINAL EVALUATION OF PREPARE EJE CAFETERO o Letter of Commitment Municipality of Pereira – Miyamoto o Letter of Commitment Municipality of Risaralda – Miyamoto o Letter of Commitment Municipality of Quindío – Miyamoto • USAID – Miyamoto Contract 720FDA20GR00265 PREPARE Eje Cafetero • https://www.instagram.com/p/Cs_nPE0oX9_/?igshid=MTc4MmM1YmI2Ng== https://www.instagram.com/p/CmW22U1ucWC/?igshid=MTc4MmM1YmI2Ng== https://www.instagram.com/p/CoxrIHwrIsI/?igshid=MTc4MmM1YmI2Ng== • https://instagram.com/miyamoto_international?igshid=MzRlODBiNWFlZA= ANNEX VII: EVALUATION TEAM MEMBERS The lead consultant, Anna Wrochna’s professional DRR and MEAL experience of the last 15 years have covered several final evaluations in the field of DRR. She led Oxfam's regional MEL work in Colombia and in South Asia, regional MEAL work for the Norwegian Red Cross in Latin America, developed MEL strategies for the Peace Brigades in Colombia and lead several related consultancies on systematization processes, knowledge management, quantitative and qualitative evaluations, MEL integration at local, national, and international levels with UNICEF, UNISDR, UNHCR, Red Cross, ECHO, and Save the Children in 4 continents. She has two MA in International Relations and Humanitarian studies. The local expert, Alejandro Gomez has been involved in rural community led processes in Colombia for the last ten years being environmental ombudsman and hence leading the environment protection work in rural Colombia. He supports the consultancy both with detailed local background knowledge, local political, cultural, and social experience as well as critical points of view. His experience in business administration, information management and visual arts adds useful approaches to the consultancy. U.S. Agency for International Development 1300 Pennsylvania Avenue, NW Washington, DC 20523