1 INCIDENT COMMAND SYSTEM (ICS) PERFORMANCE EVALUATION FINAL EVALUATION REPORT This publication was produced at the request of the United States Agency for International Development. It was prepared independently by DevTech Systems, Inc. 2 ABSTRACT This report is a performance evaluation of the Incident Command System (ICS) program in Indonesia, the Philippines, and Thailand managed by the United States Office of Foreign Disaster Assistance (USAID/OFDA). The evaluation examined the program’s effectiveness and sustainability in building the capacity of local government agencies in the three countries to respond to natural disaster using ICS. The evaluation applied a mixed methods design, collecting data from document reviews, key informant interviews, focus group discussions, and an online survey for ICS master trainers in the Philippines. Results show that the programs have identified the appropriate local government agencies to work with and ICS capacities are being built to varying degrees. Demand for training is high but USAID/OFDA and the United States Forest Service (USFS) trainers must consider contextualizing ICS principles and training content to make the system relevant to local agencies and communities. USAID/OFDA and USFS trainers also should consider conditions unique to each country, such as governance structures, to solicit the support of the local first responders’ community. The effectiveness and sustainability will also be better understood if local government agencies develop performance management plans to track the program’s effectiveness and sustainability. 3 INCIDENT COMMAND SYSTEM (ICS) PERFORMANCE EVALUATION FINAL EVALUATION REPORT A Performance Evaluation of the ICS Program in East Asia and Covering the Assistance Provided by USAID/OFDA from 2012 August 14, 2017 [Contract No. AID-OAA-I-15-00018; Task Order No. AID-OAA-TO-16-00037] DISCLAIMER The author’s views expressed in this publication do not necessarily reflect the views of the United States Agency for International Development or the United States Government. 4 CONTENTS Abstract .........................................................................................................................................................................2 Acronyms.......................................................................................................................................................................5 I. Evaluation Purpose .............................................................................................................................................11 II. Evaluation Questions..........................................................................................................................................11 III. Project Background.............................................................................................................................................12 IV. Evaluation Methods & Limitations......................................................................................................................16 V. Findings...............................................................................................................................................................20 VI. Conclusions.........................................................................................................................................................36 VII. Recommendations..........................................................................................................................................47 Annexes .......................................................................................................................................................................50 Annex I: Evaluation Statement of Work ..............................................................................................................51 Annex II: List of Indicators.....................................................................................................................................59 Annex III: Data Collection Instruments..................................................................................................................64 Annex IV: Sources of Information ..........................................................................................................................89 Annex V: Field Photos ...........................................................................................................................................96 Annex VI: Summary Information on the Evaluation Team.....................................................................................99 5 ACRONYMS AAR After Action Review AC Area Command ACDM ASEAN Committee on Disaster Management AHIMT All-Hazard Incident Management Team ADPC Asia Disaster Preparedness Center AFP Armed Forces of the Philippines ASEAN Association of Southeast Asian Nations BFP Bureau of Fire Prevention BNPB Badan Nasional Penanggulangan Bencana/National Agency for Disaster Management BPBD Badan Penanggulangan Bencana Daerah/Provincial Agency for Disaster Management COR Contracting Officer Representative DDPM Department of Disaster Prevention and Mitigation DEC Development Experience Clearinghouse DILG Department of Interior and Local Government DOH Department of Health DOS Department of State DPMRC Disaster Prevention and Mitigation Regional Center DRR Disaster Risk Reduction DSWD Department of Social Welfare and Development EAP East Asia and the Pacific EOC Emergency Operations Center ESF Emergency Support Function FGDs Focus group discussions FY Fiscal Year GIs Group interviews GOI Government of Indonesia GOP Government of the Philippines ICP Incident Command Post ICS Incident Command System IMATs Incident Management Assistance Teams IMTs Incident Management Teams KIIs Key informant interviews MOU Memorandum of Understanding 6 MT Master Trainer NDRF National Risk Reduction Framework NDRMF National Disaster Risk Management Framework NDRMP National Disaster Risk Management Plan (2015) NDRRMC National Disaster Risk Reduction and Management Council NGO Non-Governmental Organization NIEM National Institute of Emergency Medicine (Thailand) NIMS National Incident Management System OCD Office of Civil Defense OFDA Office of U.S. Foreign Disaster Assistance PMI Palang Merah Indonesia / Indonesian Red Cross PNP Philippine National Police POC Point of Contact RFTOP Request for Task Order Proposal RTG Royal Thai Government SEA Southeast Asia SGLG Seal of Good Local Governance SOW Statement of Work THAIMAT Thailand Incident Management Assistance Team ToC Theory of Change TOT Training-of-Trainers UN United Nations USAID United States Agency for International Development USD United States Dollar USG United States Government USFS United States Forest Service UFE Utilization-focused evaluation WFP World Food Program WVI World Vision Indonesia 7 EXECUTIVE SUMMARY EVALUATION PURPOSE AND EVALUATION QUESTIONS The goal of this performance evaluation is to improve the understanding of the Incident Command System (ICS) program supported by the United States Agency for International Development Office of U.S. Foreign Disaster Assistance (USAID/OFDA) in the East Asia and the Pacific (EAP) region. The evaluation focused on the effectiveness and sustainability of the USAID/OFDA funded ICS programs in Indonesia, the Philippines, and Thailand. The findings of this evaluation will inform future programming decisions and adjustments to ongoing ICS programming. Program effectiveness and efficiency was evaluated along the following lines of inquiry: Objective 1: Effectiveness 1. To what extent has the ICS program built the capacity of national governments to respond to disasters effectively, particularly at the initial stages of the emergency? 2. Which aspects of the ICS training program are most effective? Which aspects of the ICS training program are least effective? 3. Is the ICS targeting actors within the national government that are best positioned to effectively implement elements of the ICS system? 4. What country-level factors influence the effectiveness of the ICS program? 5. To what extent do training participants—including the different levels of the cascade training system—retain knowledge and skills transferred through the ICS trainings? The second objective, to evaluate the sustainability of the program, has the following four lines of inquiry: Objective 2: Sustainability 6. To what extent are countries able to integrate the ICS into their national frameworks or other institutionalization models? 7. What barriers to utilization of the ICS exist? 8. How much adaptation (translation of core concepts and revision of key elements to better reflect local conditions) of the ICS is required before the materials can be utilized at the local government￾level? 9. What factors contribute to sustainability of the ICS? Will existing investments remain viable without external donor support? PROJECT BACKGROUND Countries in the EAP region face extremely high disaster risks from typhoons, monsoon rains, volcanoes, earthquakes, tsunamis, and other natural phenomena. To mitigate these risks, USAID/OFDA has funded a range of Disaster Risk Reduction (DRR) activities in the region. In the USAID/OFDA context, DRR is a broad term that includes initiatives to prevent or reduce the damage caused by natural hazards like earthquakes, floods, droughts, and storms. Investments in DRR save lives, not just after the disaster occurs, but even as disaster strikes. While DRR programming by USAID/OFDA in EAP focuses on context-specific activities designed to meet the particular DRR needs of each individual country, host country human and institutional capacity building has been a consistent theme throughout. Developing a capability to respond to disasters more effectively using the ICS has been a key element in these efforts. 8 ICS is a standardized incident management approach, applicable to any disaster scenario, and designed to improve coordination and communication among the various actors and agencies involved in disaster response. ICS comprises a set of basic principles, an organizational structure, and an operational planning process, and includes a clear chain of command, common terminology, interoperable communications, standardized training, and consistent certification requirements for disaster response staff. The DRR strategy of USAID/OFDA in the EAP region focuses on increasing the capacity of national disaster management agencies to effectively respond to disasters, among other objectives. As part of this effort, USAID/OFDA has funded U.S. Forest Service (USFS) ICS capacity building activities across the EAP, providing training in Brunei, Indonesia, the Philippines, Thailand, Vietnam, Mongolia, the Solomon Islands, Timor Leste, and Vanuatu, recently expanding activities to China, Malaysia, and Palau, and supported the Emergency Operations Center in Burma and regional activities. EVALUATION QUESTIONS, DESIGN, METHODS, AND LIMITATIONS The evaluation was initially designed as a mixed methods design to evaluate the three country programs, combining qualitative and quantitative data collection techniques. The qualitative techniques were based on a purposive approach in collecting field data to get the most comprehensive information available from the sources. Specifically, the team identified 84 individuals from the local ICS implementing agencies within the national governments of the three countries, partner agencies within the national governments that use ICS, and private organizations that provide disaster relief and humanitarian assistance operations. The evaluation team also designed a retained knowledge survey to collect quantitative data on retained knowledge of ICS principles and to collect verifiable data on measures of effectiveness and sustainability. However, the programs in Indonesia and Thailand did not have project records of trainees to draw a sample from for the survey. A database of trainees in Indonesia was said to have been developed between the country program and the University of Indonesia, but the evaluation team did not have the opportunity to access it. The OCD in the Philippines maintains a database of trainees, but the office was migrating their database to a new system at the time that the evaluation team was in-country. The OCD provided a list of all 341 MTs as of January 2017, and the evaluation team used it to conduct the retained knowledge survey. A total of 89 responses were collected, which was below the 181 needed for statistical significance1. Therefore, results of the survey have been presented, but only in the context of the respondent sample and are not representative of the entire MT cadre population. FINDINGS AND CONCLUSIONS The evaluation methods that the team conducted yielded the following findings and conclusions: • Multiple stakeholder agencies in all country programs have adopted ICS, but the degrees in usage vary. The Philippines has been the relatively more developed in terms of capacity. • The ICS program helps trainees understand the effectiveness of having an organized management structure with common terminologies when responding to a disaster. This understanding has been the most effective aspect of the ICS program for all countries. 1 Assuming a 95% confidence level and a 5% confidence interval. 9 • The country programs have targeted the right actors for capacity building from an operational perspective; i.e., the appropriate first responder individuals and units within the national government. But, from a policy perspective, the programs in Indonesia and Thailand currently have to solicit the support of high level officials to obtain the buy-in from staff at the implementing agencies. There is a “top-down” mentality, where operational staff will buy-in to ICS (or any program) if senior-level officials support it. • ICS is generally accepted within the leadership and operational staff of the implementing agency and partner organizations. ICS has been fully accepted as a best practice within the national government and other partner agencies, but there are some cultural barriers to overcome. • The evaluation team identified evidence of knowledge retention of ICS principles, particularly in the Philippines and Thailand. However, the inability to conduct the retained knowledge survey in Thailand and Indonesia forced the evaluation team to collect qualitative data on retained knowledge outcomes. • ICS has been integrated in national disaster frameworks in the Philippines and Thailand, while Indonesia is anticipating the inclusion of ICS in the anticipated National Disaster Response Framework. Institutionalization of ICS in local laws and plans will help ensure the sustainability of ICS, even after USAID funding concludes. • Barriers to the utilization of ICS exist, and the reasons are country-specific. In Indonesia and Thailand, a “top-down” structure of governance limits the progress of ICS without the support of higher level officials in the government. In the Philippines, substantially more trainings have been conducted, but the program needs to address “next level” questions, such as reaching more barangays (villages) or expanding the program’s focus beyond training to developing incident management teams (IMTs). • In all three countries, ICS resources have been translated and adapted; doctrine and guiding documents have been provided to key implementers who received training. However, each program will need to further provide these resources to scale up the program’s reach and meet current demands, such as requests for training for local governments. • Funding is available from the country governments to continue the use of ICS if external support ends. The funding will be able to support the delivery of trainings, either by the USFS or the local cadre. The current pace of training delivery in all countries does not meet the local demands. RECOMMENDATIONS With the findings and conclusions gathered, the evaluation team recommends the following courses of action for USAID/OFDA to consider: Recommendation 1: Accelerate Basic and Intermediate Training. USAID, USFS, and the partner nation governments have collaborated well to establish fully functioning ICS programs in each country with the full-suite of courses and certified master trainers to provide all courses. However, basic and intermediate trainings must be accelerated to meet local demands. Recommendation 2: Strengthen MT Cadres in All Countries. Related to the first recommendation, the program should strengthen and expand the MT cadres in all countries. “Strengthen” has a local context. 10 The cadre in Indonesia, for example, only has 49 members with an additional 25 MT to be added by the end of 2017 with three courses to still be delivered this fall. The requests for training are coming from all 34 provinces. But strengthening the Indonesia program means improving retention, aside from expanding training. The Philippines has the largest cadre with 431 trainers, but ICS courses have not reached all local government units and the development of IMTs at the local levels should be considered a possible way of strengthening. The Thailand program has experienced significant attrition in the cadre due to promotions, retirements, and concurrent duties, so accelerating trainings should be a consideration. Recommendation 3: Provide Implementation-Focused Support and Tracking. All programs must address key programmatic areas heavily influenced by local conditions. Indonesia must find a champion at the highest levels of the government agencies involved in disaster response due to the “top-down” structure of the government. The Philippines has trained approximately 27,000 participants, but stakeholders should review the program objectives and determine if it should evolve from training delivery to the establishment of local IMTs. As the Department of Disaster Prevention and Mitigation (DDPM) transitions ICS training responsibilities to the DDPM Academy in Thailand, changes are expected in the way courses are managed. However, the program’s focus should continue to be on the accelerated delivery of trainings. Recommendation 4: Develop Intermediate-Term Performance Targets. So that country initiatives can be achieved and measured objectively, the local government agencies in all countries should develop a performance monitoring plan over the next two years with indicators that have ambitious but achievable performance targets. While a plan is not a requirement under the agreement with the local government agencies, Indonesia and the Philippines have built monitoring systems for trainings. The Indonesia ICS program has collaborated with the University of Indonesia to develop a database, while the program in the Philippines has a database of an estimated 27,000 trainees that also tracks post-training performance. The Thailand program is in its early stages, and USFS should make a recommendation to the DDPM to develop a system. These resources can be used to develop performance monitoring to track how inputs meet higher level program objectives to build ICS capacity. Recommendation 5: Assist in Developing a Participant Database. Related to Recommendation 4 above, the ICS program in each country should encourage all local government agencies to develop or maintain a database that will track training recipients. Programs should have a resource that collects and reports meaningful data with which to measure effectiveness, capacity, or targeting of current training effort. Recommendation 6: Establish a performance management system within the country programs. Related to Recommendations 4 and 5 above, the local government agencies should establish a performance management system and go through the complete cycle to: 1) develop a performance management plan, 2) construct a country-level results framework, 3) define indicators to track performance, and 4) conduct periodic assessments of outcomes. A performance management system will allow the country team to effectively monitor activities and manage results. As stated in Recommendation 4, foundational resources in Indonesia and the Philippines have been built with databases, and Thailand should develop a similar system. These databases can serve as primary data sources for information across the different levels of the results framework to monitor the program. 11 I. EVALUATION PURPOSE The goal of this performance evaluation is to improve USAID/OFDA’s understanding of the performance of the ICS program supported by USAID/OFDA in the EAP region. The evaluation focused on the effectiveness and sustainability of the USAID/OFDA funded ICS programs in Indonesia, the Philippines and Thailand. The findings of this evaluation will inform future programming decisions and adjustments to ongoing ICS programming. After five years of supporting the ICS programming in a variety of countries, USAID/OFDA seeks a deeper understanding of the successes and areas for improvement in this capacity-building model. An evaluation of the performance of the ICS program will allow USAID/OFDA decision-makers to apply the information in future decisions related to DRR programming and the ICS program model specifically. As with many capacity-building initiatives, contextual factors can influence the results of the intervention. The introduction of ICS into a variety of countries—all with their own disaster risks, government structures, and socio-economic factors—has led USAID/OFDA to seek more information about the factors for success and the potential for sustainability. II. EVALUATION QUESTIONS This evaluation has two objectives: understanding effectiveness and sustainability. Each objective has several lines of inquiry that shall inform the evaluation design. Data was collected for all lines of inquiry in the Philippines, Indonesia, and Thailand, and was analyzed to compare results from the different contexts. These objectives and their respective lines of inquiry are listed below. Objective 1: Effectiveness 1. To what extent has the ICS program built the capacity of national governments to respond to disasters effectively, particularly at the initial stages of the emergency? 2. Which aspects of the ICS training program are most effective? Which aspects of the ICS training program are least effective? 3. Is the ICS targeting actors within the national government that are best positioned to effectively implement elements of the ICS system? 4. What country-level factors influence the effectiveness of the ICS program? 5. To what extent do training participants—including the different levels of the cascade training system—retain knowledge and skills transferred through the ICS trainings? Objective 2: Sustainability 6. To what extent are countries able to integrate the ICS into their national frameworks or other institutionalization models? 7. What barriers to utilization of the ICS exist? 8. How much adaptation (translation of core concepts and revision of key elements to better reflect local conditions) of the ICS is required before the materials can be utilized at the local government￾level? 9. What factors contribute to sustainability of the ICS? Will existing investments remain viable without external donor support? 12 III. PROJECT BACKGROUND ICS Program Background Countries in the EAP region face extremely high risks from typhoons (hurricanes), monsoon rains, volcanoes, earthquakes, tsunamis, and other natural phenomena. To mitigate these risks, USAID/OFDA has funded a range of DRR activities in the region. In the OFDA context, DRR is a broad term that includes initiatives to prevent or mitigate the impact of hazards like earthquakes, floods, droughts, and storms. Investments in disaster risk reduction help make communities more resilient to disasters and better able to manage and mitigate their impacts, saving lives, and reducing suffering before, during, and after events occur. While DRR programming by USAID/OFDA in the EAP region focuses on context specific activities designed to meet the particular DRR needs of each individual country, host country human and institutional capacity building has been a consistent theme throughout. Developing a capability to respond to disasters more effectively using the ICS has been a key element in all of these efforts. ICS is a standardized incident management approach, applicable to any disaster scenario, which is designed to improve coordination and communication among the various actors and agencies involved in disaster response. ICS was originally developed in the U.S. in the 1970s, following a series of catastrophic southern California fires, when analysis revealed that inadequate management, rather than lack of resources or failure of tactics, was the main reason for ineffective response. ICS was intended to address these management weaknesses, which had caused problems with accountability, communication, planning, management structures, and integration of interagency requirements. ICS comprises a set of basic principles, an organizational structure, and an operational planning process, and includes a clear chain of command, common terminology, interoperable communications, standardized training, and consistent certification requirements for disaster response staff. ICS expanded throughout the U.S. in the 1980s and 1990s, and has since the early 2000s become a primary component of the United States’ overall, presidentially mandated National Incident Management System (NIMS), where it is codified in the Command and Management section. More recently, spearheaded by the U.S., the ICS approach has also spread internationally. One of several components of USAID/OFDA’s DRR strategy in EAP focuses on increasing the capacity of national disaster management agencies to effectively respond to disasters. As part of this effort, USAID/OFDA has funded USFS ICS capacity building activities across the EAP, providing training in Brunei, Indonesia, the Philippines, Thailand, Vietnam, Mongolia, the Solomon Islands, Timor Leste, and Vanuatu, recently expanding activities to China, Malaysia, and Palau, and supported the Emergency Operations Center in Burma and regional activities. Indonesia Country Program Background The combined threat to humans from natural disasters and climate change is nowhere more severe than in Indonesia. Volcanoes, earthquakes, and tsunamis have gained media attention, but floods, droughts, storms, landslides, and forest fires have posed the greatest threat to the country over the past three decades; according to the Badan Nasional Penanggulangan Bencana (BNPB), the National Disaster Management Agency, the incidence of natural disasters has increased by almost 350 percent during that time. This increase is largely due to the effects of climate change, population growth, the accelerated extraction of resources, and other factors, which have exacerbated the frequency and intensity of disasters. Natural disasters affect the lives and livelihoods of millions of Indonesians, and have a substantial impact on the Indonesian economy. From 2004 to 2010, BNPB estimates that the economic impact of natural disasters totaled approximately $14 billion. An estimated 70 percent of disasters in Indonesia are hydro-meteorological in nature; therefore, the 13 incidence and intensity of disasters is expected to continue to grow due to climate change and associated effects. Natural disasters and climate change impacts will increase vulnerabilities for tens of millions of Indonesians. There are 17,000 islands in Indonesia - approximately 6,000 of which are inhabited - with a majority of the 240 million-population living near the coast. Climate change will affect a large segment of the population, threatening livelihoods, economic growth and stability of this important country and strategic ally to the United States. The ICS program in Indonesia was launched to build on the previous training and technical support provided by the United States Government (USG). During Phase 1 (2004-2006) of the Association of Southeast Asian Nations (ASEAN)-U.S. Disaster Management Cooperation Program funded by the Department of State (DOS), Government of Indonesia (GOI) disaster managers were introduced to ICS through a series of workshops, a regional Basic/Intermediate ICS course conducted in Hanoi, Vietnam, and an ICS hosted study tour to the U.S. that included two GOI officials. Following the Association of Southeast Asian Nations (ASEAN)-U.S. Phase I activities, the USFS conducted a Basic/Intermediate ICS course for GOI disaster managers in Jakarta in 2008 through the Indian Ocean Tsunami Warning System (IOTWS). USAID funded this activity. From 2008 to present, the Department of Justice International Criminal Investigation and Technical Assistance Program (ICITAP) has conducted a series of Basic/Intermediate/Advanced ICS courses at the national and provincial level. The primary audience for these trainings is the Indonesian National Police (INP); however, at the request of the USAID/Indonesia Mission, ICITAP has included BNPB and other organizations in trainings when space is available. ICITAP has also provided direct training and technical support on ICS to BNPB staff and some personnel from the Badan Penanggulangan Bencana Daerahs (BPBDs) or Provincial Agencies for Disaster Management in response to requests for assistance. In 2008, the BNPB issued Regulation 10, which mandates that GOI response agencies use ICS. BNPB requested USG assistance in integrating ICS and other relevant NIMS components into the newly established Indonesian disaster management system, and ICS is a primary component of BNPB’s curriculum for GOI disaster managers. Regulation 10 was updated in 2016 by Regulation 3 to further set the framework for ICS application and the basis for the provision of training courses. USAID/OFDA provided $450,000 and USAID/Indonesia provided $1.085 million to support the implementation of ICS in Indonesia from FY 2012 to 2015. The funding supported seven program activities, namely the establishment of an ICS Steering Committee, the development of a training plan and the adaptation of course materials, the completion of ICS Training-Of-Trainer (TOT) courses, monitoring initial course offerings, supporting BNPB in implementing new systems, the completion of a U.S. study tour, and the completion of program evaluation workshops. Batch 1 MT cycle was completed in 2014 and evaluated during the Mentawai Megathrust Disaster Relief Exercise conducted in Pandang, West Sumatra. In FY 2017, USAID/OFDA provided $200,000 to continue ICS training in Indonesia. The Philippines Country Program Background USAID/OFDA provided more than $1.4 million to support ICS programming in the Philippines from FY 2010 to FY 2015. USAID/OFDA support has funded TOT courses, as well as technical support for the adaptation of course materials, the establishment of Incident Management Teams (IMTs) in focal areas, the development of national ICS policy and guidelines, the mentoring of MTs, the conduct of exercises, the establishment of training and certification standards for ICS trainers, and the development of a website to track ICS training nation-wide. In FY 2012, USAID/OFDA supported the USFS to bring ICS training to the local government unit level, with a focus on local government units and provinces in four regions (Cagayan, Laguna, Sorsogon and Benguet) that USAID/OFDA targeted for a capacity-building and technical support program 14 implemented through the United Nations (UN) World Food Program (WFP). USFS worked with WFP and the Philippine Office of Civil Defense (OCD) to conduct an ICS course for participants from selected local government units and provincial Disaster Risk Reduction Management Committees (DRRMCs) in each of the four regions, as well as team-based All-Hazard Incident Management Team (AHIMT) courses and exercises for newly established Incident Management Teams in Cagayan and Benguet. In April 2011, the OCD selected three pilot areas for testing ICS at different geographic levels in the country, selecting one location in each of the three main island regions in the country. In the Luzon region, OCD selected Olongapo City while Negros Occidental and Surallah Municipality were selected for the Visayas and Mindanao regions, respectively. In May 2011, OCD and the USFS jointly conducted three sessions of the basic and intermediate ICS courses with members of the DRRMCs in each pilot area. In November 2011, USFS conducted the AHIMT course. For the purposes of this evaluation, the trainings conducted in the three areas were the bases for sustained ICS use and practice in the country, while Manila and Olongapo were the focus areas of the site visits due to logistical considerations. In Manila, the team met with beneficiary organizations such as the Bureau of Fire Protection (BFP), Philippine National Police (PNP), Armed Forces of the Philippines (AFP), and the Department of Health (DOH). Thailand Country Program Background Thailand is exposed to disasters, primarily from hydro-meteorological events, such floods, landslides, storms, and droughts. Floods present the greatest hazard, with the Great Flood of 2011 affecting 64 out of 77 provinces, over 5 million households, and costing 1.44 billion Baht ($45.7 billion) and over 1,000 lives. Thailand is also affected by earthquakes and tsunamis, with over 8,000 people either dead or missing after the 2004 Indian Ocean Tsunami. The Thailand Department of Disaster Prevention and Mitigation (DDPM) was established in 2003 as the lead government agency for disaster management and coordination. DDPM was introduced to ICS during Phase 1 of the ASEAN-US Disaster Management Cooperation Program, conducted between 2004 and 2006. DDPM received further basic-level ICS training in 2009 from the Washington State National Guard through the Department of Defense State Partnership Program. Following this training, DDPM decided to incorporate ICS into the Thailand disaster management system and established an ICS Working Committee, which developed the conceptual framework for ICS integration. Framework components include the development of course materials and broad ICS training for responders and decision-makers, development of manuals, guidelines and standard operating procedures, adjustment of relevant laws and regulations, and selection of pilot project areas/provinces to experiment and adjust the system based on lessons learned. DDPM also developed an ICS TOT cadre, selecting the top 20 participants from prior ICS training sessions. DDPM contacted the USFS in April 2010 to request assistance in ICS implementation, including providing training to the TOT cadre and technical support in the adaptation and implementation process. The DDPM Director General submitted a request to the ASEAN Secretariat to become a third Pilot Country under the ASEAN-US Disaster Management Cooperation Program, and the request was approved by the ASEAN Committee on Disaster Management ICS Task Force on June 23, 2010. USFS began working with DDPM in 2010 through the ASEAN-U.S. Disaster Management Cooperation Program, which aims to enhance the capacity of ASEAN for disaster management through integration of ICS into regional and national disaster management systems. In FY 2011-2013, USAID/OFDA provided funding for a series of ICS TOT courses for the 1st Generation of MTs. In FY 2013-2014, USAID/OFDA provided funding to USFS to provide training to support a 2nd Generation of MTs. DDPM and the MTs have subsequently cascaded ICS training to operational responders in a variety of agencies and regions, and have applied ICS during the conduct of tabletop and full-scale multi-agency exercises. Starting in 15 2012, USFS assisted in selecting and initiating pilot programs in six provinces: Phuket, Songkhla, Khon Kaen, Ayutthaya, Rayong, and Chiang Mai. In 2014 and 2015, USAID/OFDA funded USFS for the training and development of a 3rd Generation cadre to serve as Thailand Incident Management Assistance Team (THAIMAT) members and facilitated a U.S. based study visit to support the THAIMAT program. The THAIMATs are intended to be field deployable teams that utilize ICS to support provincial-level disaster response and improve coordination with the national government. DDPM has continued to improve their disaster management systems in 2016-17, with priority focus on the THAIMAT (development of a 4th Generation cadre), national Emergency Operations Center (including developing Emergency Support Functions) and a national exercise program. From FY2011-FY2016, USAID/OFDA provided $910,0002 to support the implementation of ICS in Thailand. The funding supported various program activities, including the 1st and 2nd Generation MT Cadres, THAIMAT development, establishment of an ICS Working Committee, the development of a training plan and the adaptation of course materials, the completion of ICS TOT courses, monitoring initial course offerings, supporting DDPM in implementing new systems, the completion of a U.S. study tour, and the completion of program evaluation workshops. In FY 2017, USAID/OFDA provided $150,000 to continue support for ICS implementation in Thailand. Concurrently and in concert with these efforts, the Royal Thai Government (RTG) has undertaken initiatives to create “disaster immunity” by improving its ability to manage disasters. The Eleventh National Economic and Social Development Plan (2012) and the National Preparedness Strategy (2014) aimed to further this goal by encouraging inclusive participation and synergizing stakeholder activities for comprehensive disaster management. The National Disaster Risk Management Plan (2015), developed under the Disaster Prevention and Mitigation Act B.E. 2550 (2007) supplements the National Disaster Prevention and Mitigation Plan (2010-2014). It provides descriptions of roles and responsibilities across disaster response stakeholders, a reference of applicable laws, and top-level guidance for using ICS in response operations. Table 1 presents the historical funding streams of the ICS program for each country. Table 1: ICS Funding by Country Period Indonesia Thailand (FY11-16) Philippines (FY10- FY14) FY10 $150,000 FY11 $150,000 $400,000 FY12 $90,000 $170,000 FY13 $450,000 a $150,000 $250,000 FY14 $0 b $0 d FY15 $50,000 $250,000 FY16 $0 c $0 FY17 $200,000 Total $440,000 Total: $1,220,000 e a. FY2012-2015 b. Planned was $150,000 but used existing DASP Agreement. Not counted in the total. c. Planned was $150,000 but used existing DASP Agreement. Not counted in the total 2 Source: DASP Program documents for USFS contracts/SOWs. 16 d. Planned was $250,000 but used existing DASP Agreement. Not counted in the total e. Including through FY16; $970,000 from FY10-14 IV. EVALUATION METHODS & LIMITATIONS Qualitative Approach During the development of the Inception Report, the DevTech team developed a mixed methods design to evaluate the three country programs, combining qualitative and quantitative data collection techniques. The qualitative techniques were based on a purposive approach in collecting field data to get the most comprehensive information available from the sources. Specifically, the team identified individuals from the local ICS implementing agencies within the national governments of the three countries, partner agencies within the national governments that use ICS, and private organizations that provide disaster relief and humanitarian assistance operations. Table 2 provides a list of the organizational affiliations of all respondents. Table 2: Organizational Affiliations of Respondents Indonesia: • Badan Nasional Penanggulangan Bencana (BNPB), National Disaster Management Agency • Badan Penanggulangan Bencana Daerahs (BPBDs), Provincial Disaster Management Agencies • Indonesian National Police (INP), Disaster Victim Identification (DVI) Unit • United Nations Office for the Coordination of Humanitarian Affairs (UN OCHA) • World Vision Indonesia • Palang Merah Indonesia (PMI), Indonesian Red Cross • TNI (Indonesian Military) The Philippines: • Office of Civil Defense (OCD), including the Central Office (Manila), Region III, and Region V offices • Metro Manila Development Authority (MMDA) • Olongapo City Disaster Risk Reduction Management Office (DRRMO) • Quezon City DRRMO • Sorsogon City DRRMO • Armed Forces of the Philippines (AFP), relevant units • Department of Interior and Local Government (DILG) • Department of Social Welfare and Development (DSWD) • Department of Health (DOH) • Bureau of Fire Protection (BFP) • Philippine National Police (PNP) • UN World Food Programme Thailand: • Department of Disaster Prevention and Mitigation (DDPM) • National Institute for Emergency Medicine (NIEM) • DDPM Ayutthaya regional office • DDPM Phuket regional office 17 • Thai Red Cross The qualitative data collection activities included Key Informant Interviews (KIIs) and group discussions using guided interview instruments and discussion agendas (see Annex III) that were designed to address all the lines of inquiry from the evaluation objectives. The interviews and group discussions were supplemented by a review of country program documents, such as time-bound (e.g., annual and quarterly) reports, DRR assessments, and annual plans. The evaluation team also traveled to specific locations to see facilities and physical resources used in ICS and to observe actual ICS operations that were taking place. The locations included: Indonesia: • Indonesia Disaster Relief Training Ground, Sentul. The evaluation team observed a Basic and Intermediate ICS TOT course that was conducted. Philippines: • ASEAN IMT Staging Area. The field visit was planned partly to coincide with the staging of an Incident Management Team for the 50th ASEAN meetings. The evaluation team observed activities during Day One of the staging. • Quezon City Disaster Risk Reduction Management Office (QC DRRMO). The office is one of the most developed DRR units for ICS operations in Metro Manila. The evaluation team visited monitoring centers and inspected physical resources. • OCD Region V, including Sorsogon City and Sorsogon province. USAID/Philippines provided significant funding to the region and has partnered with the UN WFP to implement ICS in Sorsogon City and Sorsogon Province. • OCD Region III, including Olongapo City. The city was one of the four selected pilot locations after the ASEAN regional meetings in 2010. Thailand: • Chulchomklao Royal Military Academy in Nakhon Nayok. The evaluation team observed the 4th Generation THAIMAT course that was being conducted in the facility. • Incident Command Post (ICP) in Surat Thani. The team observed the Area Command (AC) set up in response to the flood situation affecting southern provinces. • DDPM in Phuket, a former pilot province. • DDPM in Ayutthaya, a current pilot province. A main limitation of the use of qualitative methods as an evaluation approach is the inability to state generalized findings that can be objectively or empirically measured. This limitation was mitigated in the evaluation design by consistently collecting data along all lines of inquiry as outlined in the evaluation Statement of Work (SOW). For example, to explore the effects of country-level factors on evaluating the effectiveness of the program (Objective 1), the team gathered feedback regarding this factor from all respondents. The evaluation team anticipated some variation in the information provided.; e.g., a local national would expectedly have more in-depth feedback compared to an expat or foreign national on the example regarding the inquiry on country-level factors. By consistently applying the lines of inquiry across all respondents, the evaluation team could draw outcomes and identify patterns in responses that allowed the team to conclude that findings are valid and reflective of actual conditions. The approach also directly answers the evaluation lines of inquiry using this approach. Capacity Building Index 18 The consistent application of the lines of inquiry also allowed the evaluation team to construct the ICS capacity building index, as proposed in the evaluation work plan. The team developed the index by examining a series of factors or elements that lead to the effectiveness and sustainability of the Indonesia ICS program, and rating each factor along a 1-4 scale with the following working definitions: 1. This element is Present but in a condition that contributes only to a Very Limited extent in achieving its intended purpose. Considerable External Assistance is required to develop or improve this function. 2. This element is Present but in a condition that makes a minimum, but somewhat Adequate contribution to achieving its intended purpose. Some External Assistance is required relating to this function. 3. This element is Present and in a condition that makes a Generally Satisfactory contribution to achieving its intended purpose. Some External Assistance is required to ensure that the purpose intended is achieved and sustained. 4. This element is Present and in a condition that makes a Clearly Satisfactory contribution to achieving its intended purpose. No External Assistance is required to ensure that the intended purpose is achieved and/or sustained. Results of the index are presented in more detail in Section VI: Conclusions. Quantitative Approach The evaluation team also designed a retained knowledge survey to assess the level of ICS lessons gained and maintained using objective and observable data. The survey questionnaire was developed and submitted to USAID/OFDA. In Indonesia and Thailand, stakeholders managing the ICS program did not have comprehensive records of trainees that the evaluation team could draw respondents from to have a sample. A database of trainees in Indonesia was said to have been developed between the country program and the University of Indonesia, but the evaluation team did not have the opportunity to access it. In the Philippines, the OCD NCR office maintains a database of approximately 27,000 past trainees with relevant data, including completed courses, certification numbers, and organizational affiliation. However, the OCD was in the process of migrating the database into an electronic system at the time the evaluation team was in Manila, and files could not be made available to the evaluation team. The only complete and available list that could be provided then was the MT Cadre as of January 2017. The list has 341 MTs and the evaluation team sent a web-based survey to all of them from June 5 to April 11. The survey response did not meet the required sample size of 181 respondents3 to achieve statistical significance. Therefore, the results cited throughout this report are within the context of the respondent sample and are not representative of the MT cadre population. Where the questions addressed lines of inquiry within the qualitative portion of the evaluation, response data and summaries have been included as an added value. Score summaries have been provided, and, in addition, average scores on the retained knowledge portion were compared against self-identified levels of ICS usage. The evaluation team includes these scores and comparisons to add insight to the evaluation; however, limitations in the sample size should be kept in mind when drawing conclusions from this data. Other limitations include: 1. Attrition. Some past trainees might have been reassigned or might have left their units during the time of their ICS training. This limited the ability of the evaluation teams to contact them for interviews or surveys. 3 The 181 respondents needed were based on the standard parameters of a 95% confidence level and 5% confidence interval. 19 2. Country selection. The three countries were selected by USAID/OFDA beforehand and thus the context when reporting findings will only be in relation to these countries, individually or collectively. Findings are not representative of the program in the entire EAP region. 3. Hawthorne bias. This effect is the tendency of a respondent to provide information based on their awareness of being observed or that an evaluation is being conducted. This effect often occurs in externally-funded assistance programs and beneficiaries respond with the intent of not being critical or with the expectation of receiving further support rather than providing candid assessments of the program. The evaluation team has experience in identifying this effect during data collection when trends in responses became evident. The team applied proven social science techniques (e.g., probes) to mitigate this effect. Where possible, the team also cross￾referenced possible biased responses with other sources of data, such as trainee records, as a confirmation measure. As much as possible, the team used discrete and observable data when doing this triangulation exercise and not a different set of information from another respondent to avoid the “he said, she said” dilemma that can happen in KIIs and FGDs. 4. Non-representative sample. The retained knowledge survey was conducted on a population of MTs. This group differs from the full population of ICS training recipients in various ways, including (1) higher and longer-term exposure to ICS material, (2) considerable real world and exercise experience with ICS usage, and (3) higher formal education levels. The evaluation team cannot draw significance from the overall scores of this population in terms of retained knowledge across the full population of ICS training recipients. Respondent Groups The respondents that provided information and feedback for this evaluation can be categorized into five groups. The first group includes the operational personnel of the government agencies that are implementing ICS. This group consists of trainee participants who work for the implementing government agency primarily as first responders. The second group includes higher-level management and policy personnel of the implementing government agencies, such as directors or chiefs of disaster management agencies. While they may not necessarily have received ICS training, they are overseeing the implementation of the country program within their organization. The third group consists of individual from government and non-governmental organizations that are working with the government implementers in the integration of ICS, such as the DOH in the Philippines and the UN Office for the Coordination of Humanitarian Affairs (UN OCHA) in Indonesia. The fourth group includes U.S. government personnel that work in the ICS program as advisors and/or trainees, such as the USFS disaster management specialists. Lastly, the fifth group includes the USAID/OFDA Regional Advisers, who represent USAID as the funding agency. 20 V. FINDINGS The findings, conclusions, and recommendations of the evaluation report and the individual country evaluations need to be guided by an analytical framework, so that the information obtained will be evidence-based and utilization-focused to inform future programming decisions. The analytical framework the evaluation team applied is illustrated in Figure 1, which shows the hypothesized logic model of the ICS program along the following levels: • Inputs. The main input of the program is the design and delivery of the ICS training courses: basic, intermediate, position courses, and advanced courses. For a trainee to be eligible to become a master trainer, s/he should also complete the Training for Instructors (TFI) course. • Outputs. The delivery and completion of courses results in the increase in the number of trained personnel within the partner nation government agencies, as well as some private sector emergency responders. The advancement of some participants to become an MT also creates a cadre of future ICS trainers in the country. • Outcomes. Over time, the increase in the number of additional trainees as well as the advancement of earlier trainees lead to the expansion of the application of the ICS practices among the community of emergency responders. Expansion is accelerated at this level through the TOT program where MTs lead ICS trainings. Spillover effects can also be expected as trained personnel apply lessons learned back at their respective units and through interactions with non-ICS trained colleagues. • Objectives. Ultimately, the objective of the program is to increase the capacity to respond to disasters using the ICS model. While USAID/OFDA defines capacity as the ability to carry out the stated program objectives “to effectively and efficiently respond to disasters using the ICS model,” the understanding of capacity needs to be contextualized for each country in the broader evaluation because of differences between the 21 three countries in local conditions. USAID/OFDA also recognizes that “the objectives of each of the iterations of ICS programming vary depending on the disaster profile and baseline capacity of the country in which the program is implemented.” Thus, capacity as understood for the Indonesia program will be different from how it is understood in the Philippines and Thailand for the broader evaluation. Capacity For all countries, the evaluation team defined capacity in three dimensions. First, capacity is recognized as the usage of ICS/NIMS concepts and procedures at multiple levels of the partner government agencies. The second dimension of capacity is the perceived improvements in disaster response using ICS. The third dimension of ICS capacity is the level of existing support for the continued use of the ICS model nationwide. The causal pathway that leads to the program objectives of the increased capacity to respond to disasters using the ICS model also needs to control for other factors not directly related to the inputs of training delivery but that nonetheless influence program objectives. In Figure 1, these are identified as external factors, such as governance frameworks and policies at the national or local levels. In the Philippines, for example, the Local Government Code decentralized governance and empowered different local government units. The code led to the increased responsibility and accountability in operations, including disaster response. Another external factor to consider is the presence of other disaster relief mechanisms or programs that might have confounding effects on the ICS program in the country. For example, the integration of ICS and the support provided by the Government of New Zealand can be seen in BNPB Regulation 3, the initiative of the GOI to institutionalize disaster management at the local level. Regulation 3 is expected to increase capacity in disaster relief operations but the effect of ICS on this outcome will be difficult to disentangle from the effects of the New Zealand government as well as other factors. These effects have to be taken into consideration, if not controlled for, in an evaluation for a more precise assessment of the effects of the direct interventions. Objective 1: Effectiveness 1. To what extent has the ICS program built the capacity of national governments to respond to disasters effectively, particularly at the initial stages of the emergency? Capacity as defined in the SOW is the ability of the program to carry out the stated objectives. The SOW also states that the objectives of each of the iterations of ICS programming varies depending on the disaster profile and baseline capacity of the country in which the program is implemented. In general, however, the objectives of ICS programs are to increase the capacity of trainees to understand the ICS model and operate effectively within it once a disaster occurs. The evaluation team further refined the definition of capacity to include three dimensions. The first dimension is the use of ICS concepts and procedures at multiple levels of government. At the national level, the program has partnered with the appropriate government agencies in the three countries to effectively implement the ICS program. In Indonesia, the program is implemented by the BNPB, or the National Disaster Management Agency, while the Office of Civil Defense (OCD) and the Department of Disaster Prevention and Mitigation (DDPM) are the national government implementers for the Philippines and Thailand, respectively. More importantly, the three implementing ICS Country Program Snapshot Indonesia • 49 Master Trainers • 200 individuals per year Philippines • 431 Master Trainers • 27,000 trained (estimate) Thailand • 15-20 active instructors • 400-500 trained per year 22 agencies have also expanded the use of ICS within their organizations (i.e., vertically) and through partnerships with other national government agencies (i.e., laterally). In the Philippines, for example, the OCD has established operational arrangements with key line departments (e.g., health, interior, and local government, and social welfare) and the defense/law enforcement agencies (e.g., Philippine National Police, AFP) in the Government of the Philippines (GOP) in integrating ICS in disaster operations and establishing incident management assistance teams (IMATs). In Thailand, the DDPM implements ICS throughout the country through the regional DDPM offices. The same approach is used in Indonesia between the BNPB and BPBDs. Generally, all national government implementers simultaneously carry out the vertical and lateral implementation of ICS through the trainings and exercises. The second dimension of capacity is the perceived improvements in disaster response using ICS. Capturing “improvements” presented a challenge in the evaluation because of the absence of (historical) quantitative data to identify trends or patterns over time. The evaluation team addressed this limitation by directly asking respondents for verifiable information on actual events, planned or unplanned, where ICS was applied with observable results. While the evaluation team asked all respondents about perceived improvements, the team also focused the inquiry on stakeholders who: 1) have been involved in a country ICS program in a management and/or operational role(s) since the beginning or over several years, and 2) have significant technical functions, such as a trainer or MT cadre. The overall finding showed that there have been improvements, with 70 percent of respondents saying that disaster response has been more effective with the implementation of ICS. The most common changes cited were the strengthening of the command chain/structure and the improvements in communication among teams. Prior to ICS, respondents stated the systems were “chaotic” and lacked any systematic order. Teams were unclear on the chain of command and did not have systems for planning and managing resources. In the Philippines, for example, the assistance requested by local officials in the morning during Typhoon Ondoy in 2009 was not provided until the evening. A retrospective examination concluded that there could have been fewer victims if there was a clear system to provide assistance. Since the implementation of ICS, coordination lines have been clearer and thresholds for requesting assistance have been institutionalized. Respondents also cited improved communications as another distinct improvement following ICS. Response organizations have communicated better due to the ICS program and information sharing across agencies has been more effective and efficient. This finding was provided by 17 out of 28 respondents. The third dimension of capacity is the level of existing support for the continued use of the ICS model nationwide. While stakeholders interviewed by the evaluation team generally stated that there has been support for the continued use of ICS, there were some distinctions among the three countries. In Indonesia and Thailand, nationwide support has been increasing, but there were some challenges in how to expand ICS. In Indonesia, trainings have been conducted at a pace of approximately 200 trained individuals per year, largely from the BNPB and BPBDs but also among agencies. However, the program has experienced challenges in providing more training because the expansion of the MT cadre in the country is in its early stages. There are 49 MTs in the country and another batch will complete training by March 2017, mostly from the fire-prone provinces of Sumatra and Kalimantan, but more courses are needed to cover the 34 provinces in the country. The BNPB and BPBDs also experienced administrative issues in releasing budgets for disaster response, but a letter from the Ministry of Home Affairs (MOHA) codified the requirement for local governments to provide budgets for disaster response. “Our mission is to give the understanding to the people that every single incident must have one commander. So now when incidents happen, we appoint one incident commander. This is the best achievement for us - the most significant.” -Indonesia respondent 23 In Thailand, ICS trainings have been conducted regularly at DDPM and province levels, as well as within partner agencies, resulting in a steady expansion of ICS usage within the RTG and its partners, largely within DDPM, but also among other agencies. Although an accurate total of those trained in ICS in Thailand is unavailable, the information provided above gives a minimum of 2,500 trained countrywide in both disaster and public health sectors. The actual number of trainees is higher, as this estimate does not reflect all cascaded training, which was not tracked. However, there have been challenges in cascading the trainings mainly because the MT Cadre is insufficient to meet the needs of all the provinces and partner agencies. Although ICS is gaining traction in Thailand, interviews indicated that full acceptance is limited to DDPM and select partners (e.g., military and public health agencies). Outside of DDPM, organizations have limited knowledge of the Emergency Support Function (ESF) construct, and do not necessarily perform their roles as laid out in ESFs or in line with ICS. There was consensus among those interviewed that more training is needed at the provincial levels and across partner organizations. The ICS program in the Philippines is relatively further along in terms of expanding ICS nationwide. There have been approximately 27,000 individuals trained in at least the Basic course and, as of January 2017, there are 431 MT cadre members. Apart from it being the oldest among the three country programs, the expansion of ICS nationwide in the Philippines has been the result of institutional developments in the governance system. Specifically, the 1991 Local Government Code decentralized the government system and gave greater authority and accountability to local governments. The result has been a relatively faster and more localized integration of ICS. Trainings have been conducted regularly throughout the Philippines and the system is in use heavily at the national and regional levels, led by OCD and the National Disaster Risk Reduction and Management Council (NDRRMC) at the national level and Disaster Risk Reduction Management Offices (DRRMOs) at the regional levels. Stemming from the decentralization of governance, each regional DRRMO has been empowered to create its own strategy to meet regional training needs. For example, Region 5 DRRMO has focused on creating its own MT cadre first, so that it can cascade the training independently. Other regions have opted to invest in large scale Basic/Intermediate ICS courses in order to raise awareness and basic competency quickly. Regional and local Incident Management Teams (IMTs) have been formed and regularly respond to incidents or manage planned events. Regions, and the local government units (LGUs) under each region, have taken different approaches to building capacity. Some municipalities, like Olongapo City (a former pilot city in Region 3), have a large batch of MTs and have worked intensely with regional partners, first responders, and barangays (villages) to build capacity, having provided 200 people with Basic ICS training and 150 individuals with a “crash course.” They and their partners have used ICS during a variety of planned events (e.g., APEC meetings) and disasters (e.g., Typhoon Lando), and cite many improvements stemming from establishing a well-organized management capability. On a smaller scale, Sorsogon City has included a one- to two-hour ICS overview in their general disaster and emergency response training for barangays and first responders, but the city government feels most of the benefits of ICS have not yet materialized at that level. Although ICS was cited as improving response capabilities during Typhoon Nina, the city leadership felt the improvement was due to use of ICS at the CDRRMO, and not at the barangay level. 2. Which aspects of the ICS training program are most effective? Which aspects of the ICS training program are least effective? 24 For the respondents from Philippines and Thailand programs, the most effective aspect of ICS training is its application in real world events4. The practical application exercises during trainings were regularly cited by 70 percent of respondents in the Philippines as helpful; however, trainees understood the process best once they had responded to an actual event. After the course, instructors encourage trainees to practice it during planned events, such as festivals, concerts, and other good venues for practice. Trainees tend to appreciate the courses more through that practice, and in turn that makes them more effective during a disaster. They also integrate ICS into pre-planning processes for floods, and use it for drills and exercises. There is no formal requirement for them to conduct these exercises or planning processes, but MTs encourage them to do so and consider it not only practice but a chance to showcase capabilities. During the retained knowledge survey in the Philippines, participants were asked about most and least effective aspects of the training. Those surveyed mostly interpreted the question as addressing which modules were least and most effective, and 29 percent of respondents indicated that all of the modules were “most” effective, while 48 percent of respondents felt none of the modules were “least” effective. The All Hazards Incident Management Team was cited as the most effective course by 25 percent of the respondents, often citing this as the course where the rest of the information “comes together.” The course methods, including interactive lectures and in particular the practical exercises, were cited by 16 percent of respondents as being the most effective aspect. While specific courses were not commonly cited as the least effective parts of the training, the position courses, (in particular the Finance/Admin Section course), as well as the Integrated Planning Course were cited by some respondents as the course modules that needed improvement. Interestingly, the command and control aspect of the ICS program was recognized as effective by only 3 percent of the respondents. Tables three and four provide breakdowns of all qualitative responses on most and least effective aspects of the program, as categorized by the evaluation team. Table 3: Most effective aspects of the ICS program as ranked by the percentage of survey responses All 29% AHIMT 25% Basic ICS 17% Integrated Planning 12% Practical Exercises 11% Position 7% TFI 5% Unity of Effort 5% Methods 4% Ops/Planning 3% Lecture 3% Command and Control 3% Sharing of Experiences 1% Resource Management 1% Table 4: Least effective aspects of the ICS Program as ranked by the percentage of survey responses 4 During the interviews, respondents understood “program” to mean the overall presence of ICS operations in the country, not just the delivery of training. 25 Nothing 48% Position Courses/Integrated Planning 15% Finance/Admin 8% Forms 5% Executive Training 5% Basic ICS 5% Not enough time in class 3% Not enough practical exercises 3% Demobilization 3% Redundant Content 3% Inappropriate target audience 3% Handouts 3% Among 63 percent of respondents in Thailand, the common theme around what makes training effective is the ability to apply the knowledge to real world events. The practical application exercises during training were cited as helpful in course reports describing participant feedback, and this perception was also reflected by IMAT training participants, deployed THAIMAT personnel, and other informants. However, two practitioners interviewed in the Surat Thani Area Command and in provincial DDPMs indicated that they understood the process best once they had responded to an actual event. Several ICS concepts were cited as the most effective aspects of ICS: • Management by Objectives provides clear guidelines of management, priorities (life safety, strategy, effective use of resources) and a systematic process of thinking and implementing (i.e., map objective to tactic and assign appropriate equipment). In Indonesia, for example, the development of an actual Incident Action Plan (IAP) for the next operational period has been BNBP’s primary objective since 2012.5 • Check-in System allows ICP staff to monitor resource use and availability. • Interoperable Communication allows for more coordinated communication and sharing of information among relevant organizations and more effective use of communication equipment and resources to share information, with greater accuracy. • Joint Information Center fosters accurate information-sharing among relevant stakeholders and media. For stakeholders in the Indonesia program, another effective aspect of the ICS program is its impact on behavioral change. ICS led to an institutionalized change in behavior on multiple levels, even just in the way responders think about managing emergencies. The community of responders with ICS training now has a greater appreciation for planned meetings and briefings to discuss what to do. Prior to ICS, responders also had perceptions that response failures were caused by limited resources for relief operations. Because of ICS, for example, two BNPB respondents said that their units later understood that failures were due largely to the absence of an organized structure and that the model provided a system to allocate resources more effectively and efficiently. For example, a responder in the Pidijaya earthquake stated that coordination among groups was challenging and it was later realized that the situation was due to a “lack of organizational structure” among officials who were either new or not very experienced in ICS. 5 Presently, the plan has still not been accomplished either during a planned event or during disaster response. However, it was tested during the Mentawai Megathrust Disaster Relief Exercise in 2014. 26 3. Is the ICS targeting actors within the national government that are best positioned to effectively implement elements of the ICS system? In answering this line of inquiry, the evaluation team determined that the concept of “targeting actors” within the national government should be understood from two perspectives to effectively implementing elements of the ICS system. First, targeting actors from an operational perspective means the implementing government agency has the institutional capabilities to manage the program and expand the use of ICS through the delivery of training services at the national level. The program should target the actors who are responsible for responding to disasters from the initial stages to national-level response. Using this perspective, the country programs have targeted the right actors in all three countries. In Indonesia, the program has targeted the right actors within the BNPB, BPBDs, and other GOI agencies such as the Ministry of Agriculture. The engagement by the program with key BNPB and BPBD individuals established a growing training environment where courses are effectively planned in terms of identifying participants, developing curricula, scheduling, and selecting instructors. The training environment has been growing since 2012 even though internal funding requests have not increased over the same period. In the Philippines, the target actors include OCD, NDRRMC, and other departments/agencies at the national level, regional DRRMOs, LGUs and their DRRMOs, first responders (e.g., fire and police), and partner organizations, such as volunteer organizations and hospitals. In Thailand, the target actors include DDPM at national and provincial levels, local government units, first responders (e.g., fire and police), ministries with roles in disaster response, and partner organizations, such as volunteer organizations. Second, from a policy perspective, the program should target actors who are positioned to build an effective ICS program, which includes influencing leadership, creating buy-in across all levels of government, and developing laws, regulations, policies, and procedures that guide ICS implementation from both a training and operational perspective. The evaluation team found that targeting actors for policy purposes was a key element more in Indonesia and Thailand than in the Philippines due to more localized cultural factors. Indonesia and Thailand function on a “top-down” structure in professional settings, particularly in the government. In Indonesia, the program must recognize the “top-down” structure in the GOI and in the overall working environment in the country. In general, any initiative implemented in the country will be accepted by the broader beneficiary community if it has the support of the community leaders or highest officials. In Thailand, although DDPM does not have enough MTs to train other government agencies to fully implement/integrate ICS, there is support at high levels, evidenced by ICS influencers’ ability to make ICS usage required by law. There is good political will, an available budget for ICS, and there is a policy in place to support ICS implementation. Although there is little resistance, low levels of knowledge impede full institutionalization. The decentralized system of governance in the Philippines lessened the need for support from high-level public officials to encourage buy-in from officials and staff in the different government agencies. The ICS program has targeted a wide variety of stakeholders, and training has reached actors at every level, from the national government to first responders, though not every level is fully saturated; that is, training levels vary across agencies and across LGUs, depending on leadership priorities and availability of trainers. However, a concerted effort has been made by DILG in particular to support expansion of the ICS program by targeting and incentivizing leadership within LGUs. Program advocates have also built the legal framework for ICS implementation nationwide as well as strategies to incentivize LGU adoption. National level partners such as the armed forces, Bureau of Fire Protection, and Department of Health have received training and have advocates placed well within their agencies to propagate the program. Overall, support for continued ICS is high, not solely because it is codified into law but because of a genuine commitment from the government agencies, and because the benefits have been 27 recognized nationwide. According to one regional OCD respondent, the successful use of ICS in their region draws them closer as a community of first responders, and s/he “has also been hearing success stories on ICS that happened in other regions.” 4. What country-level factors influence the effectiveness of the ICS program? There are two main factors at the country-level that influence the effectiveness of the ICS program in Indonesia. First, according to background research the evaluation team conducted, the overall approach to disaster response in the country generally uses the UN cluster approach, where UN and non-UN organizations are grouped into clusters, e.g., shelter or health, under a coordination mechanism to establish clear points of contact and make communication and coordination more effective and efficient. The model is also applied where international assistance is required and where the UN has an active presence. The UN cluster system was rolled out in Indonesia following the ASEAN decision to use it as the on-the-ground system for disaster operations, although it is also recognized as a global initiative. In Indonesia, there was a need to understand how the two systems can work together, given that the two systems target different actors in disaster response operations. Second, one of the more notable feedbacks from respondents on country-level factors that influence ICS effectiveness was the need for ICS and its resources to be more contextualized to the Indonesian setting. This feedback was shared by 67 percent of interview respondents in Indonesia. The idea of “contextualization” was very broad. At a higher programmatic level, interview respondents said that ICS needs to understand the dynamics of the local culture and working environment, specifically the “top-down” structure mentioned in Evaluation Question 3 earlier. On more specific levels, ICS resources such as forms, manuals, and other documents not only need to be translated to Bahasa, but also need to be revised to use terminologies that would be easier for stakeholders to accept. For example, the term “command” in an incident command system can mean a hierarchical structure, particularly to personnel at the operational levels of the GOI. The word “command” had the same problems in the U.S. when ICS was first developed and used in the 1970’s in the wildland fires. In the Philippines, the main country-level factors influencing the effectiveness of the ICS program that the evaluation team identified from background research are: (1) the Local Government Code, which decentralized governance and empowered different local government units; (2) presence of regional DRRMOs that coordinate and support regional LGUs with disaster risk reduction planning; (3) the Republic Act 10121, which promoted preparedness activities and mandated spending on DRR activities; and (4) frequent disasters. Decentralization of government often makes promotion of central government initiatives more difficult. However, in the Philippines, the Local Government Code empowers LGUs and fosters accountability. This motivates LGU leaders to adopt best practices and has helped propagate ICS nationwide. DILG has encouraged this adoption through their role in coordinating local governance. This gave the ICS program a mechanism for incentivizing ICS adoption, which the DILG has fostered (and continues to foster) through Operation Listo and the Seal of Good Local Governance (SGLG) program. Regional DRRMOs have been effective in planning and supporting their regions’ ICS training strategies and implementation. These offices work with LGUs and OCD, providing a link to local governance that OCD would not be able to effectively coordinate. Passage of RA 10121 shifted the GOP approach from a relief mindset to preparedness. Previously, LGUs did not have a good response organization; RA 10121 put structures in place, so that there is a system now for organizing capabilities. It also empowered DRRM Officers, giving them more control and authority over other departments. The delegation gave them the power to lead. The main country level factors influencing effectiveness of the ICS program in Thailand that the evaluation team discovered from interviews are: (1) executive level authority due to the disaster management law that codifies ICS; (2) DDPM presence in provincial offices; and (3) perception that ICS 28 adoption is low-risk. These factors stem from the centralized nature of Thai government. Like Indonesia, Thailand has adapted a “top-down” approach in integrating ICS. Thailand is institutionalizing ICS by integrating it into the National Master Plan on Disaster Management 2015. Although DDPM does not have enough MTs to train other government agencies to fully implement/integrate ICS, there is support at high levels, evidenced by ICS influencers’ ability to make ICS usage required by law. There is good political will; i.e., a commitment from DDPM leadership, an available budget for ICS, and there is a policy in place to support ICS implementation. Although there is little resistance, low levels of awareness and knowledge of ICS among partner agencies and local government officials impede full institutionalization. The system has been able to work to the extent that it does because the DDPM has provincial offices. These provincial offices are key factors for developing awareness, knowledge of, and implementation of ICS concepts. Many of the gains Thailand has made stem from ICS being well received. Although it has not been used extensively, the times it has have been relatively successful. According to one ICS trainer, if ICS were perceived as a risk, officials would not use it. However, Thai leaders realize that ICS can make them look professional and competent through a few exposures. 5. To what extent do training participants—including the different levels of the cascade training system—retain knowledge and skills transferred through the ICS trainings? The most effective way to retain knowledge gained from ICS trainings is the continued application of what was learned. According to the literature on ICS best practices, the application can be done best through actual incidents, planned events, and exercises. Classroom-based tasks such as refresher courses and tabletop exercises also help retain knowledge but do not have the practical, hands-on component. In Indonesia, opportunities to apply ICS knowledge and skills have been more limited compared to the Philippines and Thailand. The Indonesia program is relatively younger and has smaller pools of MTs and trained personnel. Thus, while ICS has been applied during various natural disasters, there has been no regular use of it during planned events and exercises and the retention of knowledge has been challenging. For example, the MT cadre currently has only 49 members. The program aims to have an additional 25 members in the cadre by the end of 2017. However, according to one MT respondent, the number of trainers is low because 50 percent of the trained group failed the retention test administered two months after the training. Trainers observed a higher passing rate of 84 percent when the retention test was administered five days after training, which means the decline suggests that some form of fadeout occurs among the MT cadre over time. However, the individuals that need refresher training could still be used as unit leader instructors under the supervision of certified MTs. Indonesia has a copy of the NWCG Instructor Guide, which defines these criteria along with the FEMA NIMS Guide. In the Philippines, the OCD is working with the regional units to develop systems for tracking knowledge retention. OCD is developing a diagnostic exam tool to determine if participants have sufficient knowledge to attend. Currently, instructors collect pre- and post-tests, but pre-test results are not tracked by OCD. A central theme around knowledge retention was that application of knowledge in practical exercises, planned events (e.g., drills, transit strikes, elections), and disaster responses greatly enhances the ability to retain and use the knowledge gained. Among those interviewed, 61percent indicated they have many opportunities to use the information, although 96 percent of interviewees were also involved in ICS application in their primary job, either as an instructor, disaster response professional, or DRRM Officer. For example, the MT cadre in a city DRRMO indicated that when they provide lectures or instruction at the barangay level or elsewhere, they review course materials from OCD, which improves their knowledge retention. They acknowledged that they might “get rusty” on the forms because they are not using them every day. During response, they have also taken the initiative to go through refreshers on roles as they introduce sections, and put guidelines in briefing templates so that staff had a reference for what they needed to accomplish. Much of this was 29 accomplished during tactical planning at the city level, during which they prepared all the documents needed for disaster response. Their goal was to prepare everything in advance, so they do not have to do any thinking during an event. In the retained knowledge survey in the Philippines, 71 percent of the MTs answered at least seven out of 10 questions correctly, and only 8 percent answered fewer than five questions correctly (Figure 2). These questions were drawn from Basic and Advanced ICS course exams, and were selected to test both general knowledge and details that might be easily forgotten. These results show that knowledge is retained at a considerable level among the respondent sample. Figure 2: Distribution of scores among survey respondents Survey results also showed experience did tend to correlate with improved test scores (see Figure 3). Those self-reporting experience using ICS in real world events on average answered 1.1 more questions correctly (out of 10) than those who self-reported lack of real world experience (7.2 vs. 6.1). Likewise, those who reported that their organization frequently or regularly used ICS performed better than those whose organizations used ICS only at a minimum or occasionally (7.2 vs. 6.3). It should be noted that because the required sample sizes for data fidelity were not attained, these results were not tested for significance and should not be considered definitive due to the low sample size. However, they do indicate that a larger-scale quality monitoring program is likely to find that training recipients’ usage of ICS principles after training helps foster improved knowledge retention. 0 5 10 15 20 25 10 20 30 40 50 60 70 80 90 100 Number of Respondents Score (percent) Retained Knowledge Survey Score Distribution 30 Figure 3: Test score averages showed correlation with self-reported experience levels For Thailand, THAIMAT staff at the Area Command in Surat Thani indicated that the knowledge obtained from the trainings was reinforced by its extensive application to the real-world flood situation in January-February 2017. Three DDPM interview respondents said the incident was a learning-by-doing opportunity for some organizations that tried to implement, or at least understand, the ICS/National Plan on Disaster Management. They were also able to access course materials and handbooks to refresh knowledge on an as-needed basis. They reported no significant gaps in knowledge that prevented efficient operations. Interviews in Ayutthaya reinforced these findings. However, the 2017 flood was also an opportunity to learn lessons on what can be done better. According to two DDPM respondents, the event highlighted a disparity between more experienced high-level Emergency Support Function (ESF) members who knew how to apply ICS and the local ESF members who did not fully understand their roles in the response structure. Leaders reported that knowledge is retained well in disaster response organizations in Ayutthaya because there are frequent incidents (flooding and industrial responses, in particular), hence many opportunities to practice. For the Ayutthaya Volunteer Association, the knowledge is reportedly retained well, as there are real incidents that require the use of ICS integrated disaster management plans on a frequent basis. However, for volunteers who have not often been in the field response team, the Association requests participation in exercises conducted by DDPM. In 2016, the DDPM provincial office conducted two exercises. Because Thailand does not experience major disasters as frequently as some other countries in the region (e.g., the Philippines and Indonesia), opportunities to apply ICS at the national level6 and retain knowledge through that mechanism are limited without using ICS to manage planned events or instituting an exercise program. DDPM is planning to request USAID support for a refresher course and MT course, to support the DDPM five￾year plan (which is currently being drafted). Thailand has institutionalized ICS into their National Plan, 6 Local and regional responses, which are more common in Thailand, provide opportunities for district and province levels to apply ICS, but frequency differs based on location. 5.4 5.6 5.8 6 6.2 6.4 6.6 6.8 7 7.2 7.4 Real World Experience Yes (n=49) vs. No (n=11) Organizational Usage Regular/Frequent (n=44) vs. Minimal/Occasional (n=16) Score (out of 10) Test Score Averages Sorted by Experience More Experience Less Experience Total Population 31 but they recognize the need to implement it on a wider scale to retain the knowledge. This is in part the focus of training modules of the DDPM Academy, with plans to have at least two MT courses per year with up to 30 participants per course to meet the scale needed to retain knowledge. Objective 2: Sustainability 6. To what extent are countries able to integrate the ICS into their national frameworks or other institutionalization models? There have been key milestone events in Indonesia that integrated ICS into national disaster response frameworks. In 2008, the BNPB incorporated ICS principles in Regulation 10, the agency-wide regulation document for disaster response. It was updated in Regulation 3 in 2016 and ICS principles were retained. But the decentralized structure of governance limits the enforcement of the regulations within the BNPB only (see the findings for Inquiry 7 below). This reinforces the need for the “top-down” approach so that regional governments and other agencies outside of BNPB will buy-in to the integration of ICS. The BNPB, with funding from the New Zealand government, is currently developing the National Disaster Response Framework (NDRF). Its development currently involves multiple stakeholders in an iterative and consultative process to develop a comprehensive response plan. However, none of the members of the NDRF team have ICS training and only two of the MTs have been engaged with the team during the development process. The hope is that, once the NDRF is completed and accepted, the Indonesian conceptualization of ICS can be adapted to their country. This outcome was seen in the U.S. when ICS became the national standard. In the Philippines, ICS has been codified in Republic Act 10121, which is an act to strengthen the Philippine disaster risk reduction and management system, providing for the National Disaster Risk Reduction and Management Framework (NDRMF) and institutionalizing the National Disaster Risk Reduction and Management Plan (NDRMP), and appropriating funds for these purposes. RA 10121 shifted the policy environment and the way the country deals with disasters from response to preparedness. The act provides a comprehensive, all-hazard, multi-sector, inter-agency, and community￾based approach to disaster risk management through the formulation of the NDRMF. The NDRMP was formulated, developed, and implemented as the master plan, which will provide the strategies, organization, tasks of concerned agencies and local government units, and other guidelines in dealing with disasters or emergencies. Thailand has integrated ICS into its top-level guidance with the publication of the National Disaster Risk Management Plan (NDRMP) in 2015. Upon its approval by the Cabinet on March 31, 2015, “all relevant agencies are required to use this plan as a blueprint, framework and guideline in handling national disaster management actions.” ICS development is also integrated in the 20-year National Strategic Plan approved by the Cabinet in 2016, as part of its disaster management strategy. DDPM has further institutionalized these processes by publishing a National Incident Management System guidebook that places ICS in the context of how Thailand responds to incidents, and an ICS planning handbook, which serves as a reference guide for ICS users in the field. It includes guidance for operating an ICP, as well as translated ICS forms adapted to the Thai context. The country programs have not been directly involved in the development of national plans, as the mandate of ICS is to provide capacity through training delivery and not primarily to conduct policy formulation. However, the program has contributed to the discussions in the development of these plans and stakeholders have advocated for the inclusion of ICS in these initiatives. In Indonesia, for example, two of the MTs have been invited to consult and provide feedback on the NDRF. 7. What barriers to utilization of the ICS exist? 32 There are two major barriers to the utilization of ICS among emergency responders in Indonesia. The first is the decentralization of government functions. In 1999, the GOI implemented a decentralized system of government that provided local government officials – governors, mayors, and village heads – with more authority and autonomy in governing their administrative areas. This outcome seemed counterintuitive since decentralization meant increased independence and authority to implement ICS. However, a negative outcome of the plan was that local officials did not feel bound to enforce policies and regulations coming from the national level. The absence of a high-level authority as an ICS champion also meant there was no incentive for officials to apply the practice. Thus, regulations created by the BNPB, which included the application of ICS principles, were not always enforced at the local level, and thus limited the utilization of ICS. This was commonly known among the disaster response units within the GOI. The second barrier to utilization is the turnover of personnel as a result of the standard reassignments and rotations in the GOI. Each province is estimated to need 10 ICS-trained individuals; hence, approximately 340 local officials will need ICS training. Additionally, there are 10 agencies in the GOI and certain units such as the BNPB, BPBDs, INP, and military are involved in disaster relief operations. Thus, the planning and logistics to deliver courses at the national and local levels need to be at a considerably fast pace to reach all the targeted individuals at the shortest possible time. However, according to interview respondents from the BNPB and BPBDs, the utilization of ICS as well as the retention and transfer of knowledge among the trained personnel are negatively affected when trained individuals are transferred or rotated to other units. In the Philippines, the main barrier to full utilization of ICS is developing a concept for how barangays can realistically implement the system. Generally, one barangay is too small to field an IMT, and barangays cannot easily be combined to form a team because of jurisdictional and political incompatibilities. This means IMTs have to be set up at the municipal level, but municipal governments do not have enough trained staff, and they would have to manage multiple barangays (maybe 10 or more). This level of granularity on ICS implementation has been addressed by some municipalities, for example Olongapo City and Sorsogon City, but their solutions are driven by local actors, not an overarching OCD vision on how barangays should implement ICS. In addition, despite the large cadre of MTs, there are still limitations to how many the cadre can train, nationwide. Over1400 municipalities in the Philippines need training, but trainer availability, time schedules, and coordination are challenges. Because of accreditation requirements, only MTs can train even the Basic level, which is required by a large amount of people throughout the country. Even then, LGUs need more than the Basic level to fully implement ICS; they need a full team, with more than just one or two trainees with advanced ICS training. Although training capacity can reasonably be expected to take time, delays in providing training to LGUs can ultimately impact support for ICS implementation. LGUs can generally find the required funds for training if they are motivated, but if the training is not scheduled quickly, the money and the motivation both go away. The ladderized basis of the courses is an understandable requirement, but it can exacerbate training availability problems, as there is just one of each type of class offered per year. If staff trainees miss one of the courses, they have to wait a full year for the next iteration. For some LGUs in the Philippines, a main impeding factor is budget. Levels 2-4 are five-day courses, so it takes more funding to provide that level of training. The requirement to spend five percent of their budget on DRR ensures a dedicated funding stream that could be applied to ICS training, but there are other DRR measures that LGU leadership could also prioritize. Because of the DILG incentive programs, most LGUs manage to find the required funds. Also, 43 percent of interviewees noted that much of the ICS program seems to be focused on training, and not on implementation. As a general rule, training is run once in the LGU, and then it is up to them to take it on. Although LGUs are 33 generally receptive to training, they need to have a confidence level to try to approximate what happened in training. If they want more assistance or mentorship after the training, the MT cadre is not available for that type of support. The primary obstacle to ICS utilization in Thailand is training capacity. From the first two generations of MTs, there are currently only 15-20 active as instructors. While many of the MT cadre have moved into positions where they have positively affected the program (e.g., helping draft the NDRMP), the reduction in training capability leaves a significant gap in the country program to use ICS nationwide. As a result, ICS concepts remain not well understood by all relevant stakeholders. Apart from public health agencies and the military, other agencies involved in responding to disasters (e.g., Ministry of Interior, Ministry of Transport, Royal Irrigation Department, Thai Red Cross) have limited understanding. There are currently 53 trained personnel authorized by the Head of DDPM to work as IMAT staff. During the flood emergency in 2017, these individuals were divided into: (i) Area Command in Bangkok, (ii) lead trainers in Nakhon Nayok for the IMAT training, and (iii) Area Command in Surat Thani. DDPM staff reported that English language capability was a barrier to their continued expansion of the MT cadre7. USFS instructors represent the gold standard in instruction; however, trainees must understand English well enough to gain value from the instruction. Translated materials assist trainees in following along, but cannot fully compensate for inadequate language understanding. Since English capabilities had been a requirement for the 1st and 2nd Generations, DDPM’s best English speakers have already received training, and there are not enough DDPM staff with the requisite language skills to produce another cadre (this is one of the factors that led DDPM to focus on developing IMAT capabilities instead of a 3rd/4th generation of MTs). Integration of budgeting systems has also been a major challenge due to Thai laws and processes. There is no legal framework for consistently and responsively funding interagency disaster response; in fact, there are conflicting definitions of “disaster.” In the relevant Thai law, some authorities and funds are only available for use for natural disasters, and not man-made events. DDPM ICS advocates are acting in two ways to overcome this challenge: (1) change the regulations through lobbying, and (2) educate agencies about using exceptions to the legal framework for emergency situations. Challenges also remain with respect to integrating resource providers through the ESF concept. Most ESFs do not know their roles in ICS, and, importantly, many of them have no ESF group leader to coordinate resources within their ESF groups. All the different organizations within ESFs have their own bosses and lines of command, so it is challenging to determine who will lead the coordination among these agencies. In addition, most ESFs, except for communication, have their headquarters in Bangkok and do not have a regional director. It is therefore difficult for IMAT staff to coordinate military resources locally. Additional barriers to utilization of ICS involve culture. A high-level official commented that there may be challenges in adopting a systematic way of thinking that is outside of the norm of Thai culture, and this viewpoint was supported by comments made by participants in focus groups. In addition, the provincial government often defaults to what they know, so building trust is an important factor. This barrier has, to some extent, been mitigated by leveraging the credibility of the U.S. Government, which adds a level of trust to the in-country team. There is confidence that when the U.S. Government is present, the team will not fail. Some of this cultural transformation has been seen with the THAIMAT – mid-level government staff (IMAT members), who would not normally be involved in decision making, 7 Although this was reported by only one staff member, that person was well positioned to have insight into this information, and was considered credible on the subject. 34 are presenting data and recommendations to authorities. This is gaining acceptance now as generals and senior leadership recognize that the team can actually help them perform their duties well. 8. How much adaptation (translation of core concepts and revision of key elements to better reflect local conditions) of the ICS is required before the materials can be utilized at the local government-level? In Indonesia, as with other country programs, ICS resources, such as curricula and manuals, were modified from their original versions by the USFS. Compared to Thailand and the Philippines, there was relatively more adaptation required prior to implementation in Indonesia. For example, local government leaders found the standard ICS forms to be irrelevant to their community; an MT in the evaluation team interviewed stated that he had to revise and simplify the forms. The evaluation team also observed an emphasis on making resources more “Indonesian,” although there was no unified and consistent response from local interviewees on what this meant. The most significant point of contextualization made by local respondents was the leadership structure of assigning the most senior official, not the most experience in ICS, as the Incident Commander (IC). This was shared by approximately 44 percent of respondents and is considered more as a preparedness issue rather than an ICS issue. Due to the high-risk activity that occurs in Indonesia, some assigning of individuals has to occur. In May 2017, USFS conducted 1CS 100 and IC course workshops with the appointed ICs for the upcoming dry season wildfires. BNBP Preparedness has planned to send a core of ICS MT to the ICP to do hands on training on the ICS planning process if incidents occur. In a general sense, the Philippines is more familiar with American culture, concepts, and contexts in communication, so the ICS program required the least amount of adaptation here. Overall, the core ICS concepts, documents, and materials have been adapted and are currently in use at the local level. OCD has provided guidebooks, though some have commented that a Filipino version of the U.S. ICS Pocket guide would be useful. Although most of the materials do not require further adaptation, some would benefit from more fine tuning to fit the local systems; for example, financial instruction needs to better reflect Philippine laws and regulations. But according to some MTs, ICS forms, though generally useful, continue to pose problems for local practitioners. LGUs have found that the forms are not applicable to the Philippines at their level. For example, numbers for resources may not be applicable, and any remaining references to resource typing cause confusion since that is not used in the Philippines. Practitioners noted that in an attempt to maintain standardization, they do not adapt the forms; they just use the ones they find useful and not the others. Respondents from the Bicol region, for example, commented that the course needs to be adapted to make it available/accessible to first responders. Although some municipalities have provided simplified instruction to barangays, OCD has not issued a formal module for the barangays that standardizes training and implementation at that level. NGOs have adapted ICS principles for their operations. DOH officials have also adapted ICS principles to meet public health procedures and practices, including forms and functions. Having good scenarios contextualized to health would be better for their purposes. DOH may need to apply training examples in health specific ways. All ICS training materials and forms have been translated into Thai language and context, adjusting certain boxes as needed. The major adaptation made for application in the Thai context was the combination of the Logistics and Finance/Admin sections into one section. This was done to better match Thai budgeting regulations that would have limited the ability of the Logistics section to act (e.g., order supplies) without the appropriate finance/admin. personnel overseeing. Further adaptations could address the ICS term “command,” which interviewees indicated Thais might find imposing. DDPM is reportedly considering a “softer,” alternative term, like “management.” Thai planners also have considered the importance of developing financial reporting methods, tracking qualifications and certification. Thailand does not have such systems yet, but is on the path to develop one for a systematic 35 way to certify trainers and track participants. Additional adaptations may be needed to ease local use of forms if widespread training is not provided. Further adaptations for use of forms and planning processes in EOCs vs. ICPs may clarify the right processes, depending on the focus of the ICS organization. 9. What factors contribute to sustainability of the ICS? Will existing investments remain viable without external donor support? The evaluation team considers sustainability along several factors that demonstrate: (1) country commitment to funding; (2) institutionalization; and (3) capability built. The decentralization plan has affected the ICS program in the Indonesian government budgeting process. During the early stages of implementation, activities were rolled out at the GOI without a Memorandum of Understanding (MOU) between agencies. For example, the ADPC operated a training center at the BNPB facility in Sentul without an agreement, but instead of finding a solution to the situation, leadership at BNPB stopped all trainings and the progress of the operations was affected. Since the mistakes in the processes were revealed, agencies became very careful and slow in moving forward with planned activities. This affected the ICS program because the Sentul facility was a planned training location. Since the early stages of implementation, there have been improvements in the decentralization process as it relates to the funding of ICS training. MOHA has released an official letter requiring governors to provide funding for disaster response training, including ICS. Provincial governments - such as the ones in Central Java, East Java, and Jakarta - have used their own funds to conduct trainings as well as fund ICS implementation at the district and sub-district levels. With the combination of internal funding and the expansion of the MT cadre from the current pool of 49 trainers, the sustainability of ICS without external support will be achievable. The GOP has contributed considerable funding to developing the current ICS training and implementation program in the Philippines. For example, the regional level has to allocate five percent of its overall budget to DRR activities. Within the DRR funds, 70 percent goes to preparedness and capacity building, including training, equipment, and drills. ICS activities are imbedded within the 70 percent allocation. USAID/OFDA funded USFS activities, but the majority of funding for technical assistance has come from the host government and its national, regional, and LGU partners. This funding is expected to continue regardless of USAID/OFDA inputs, although many LGUs have improved their ICS through OFDA-sponsored, WFP-implemented DRR initiatives that might not be continued without continued funding. The Philippine ICS program is fairly well institutionalized. It is codified by law, guided by policies, documents, and forms, and fostered by programs that incentivize adoption and require LGU spending on DRR programs. It enjoys support from high level officials throughout government agencies, and there are many applications to both planned events and disasters that demonstrate its usefulness to LGU leadership and their constituents. The primary output of USAID/OFDA for this program is cadres of MTs; outcomes in capability improvements are then cascaded from the MT cadre to achieve expansion outcomes, independent from USAID/OFDA involvement. Therefore, when assessing sustainability, the capability built should be viewed in terms of whether the number and quality of MTs is sufficient to meet the needs of the country. Interviews with 53 percent of respondents reinforced the view that the current MT cadre (431 trainers as of July 2017) is insufficient to provide the training needed to fully implement ICS. The Philippine MT cadre provides high-quality training that is standardized across the country and is considered the most advanced cadre in the EAP region. They are “DDPM Bangkok has provided basic ICS training to local authorities, yet our ability to provide such training is limited due to small number of available trainers.” -Thailand respondent 36 able to create new MTs without additional USFS personnel - a key component to sustainability - as a country must be able to replace MTs lost to promotion, attrition, retirement, or other factors. The MT cadre in general is also involved in responding to incidents, which means they have regular opportunities to apply the system and maintain their skills, though some regions and members are more “sought after” for this experience than others. Thailand has contributed considerable funding to developing the current ICS training and implementation program. Although USAID/OFDA funded USFS activities, most of the support for trainings, institutionalizing the program, and implementing ICS programs has come from the host government and its provincial partners. This funding, authorized by law, is expected to continue regardless of USAID/OFDA inputs. The Thai ICS program has been institutionalized at the top government levels, and this is expected to continue even without USAID/OFDA involvement. However, it may not mature to the point that all provincial governors are aware of and using ICS, ESFs are fully functional, and the program has international credibility. More capability must be built to achieve that level of sustainability, and USFS trainers are still needed to provide high quality instruction to create more MTs. To support the growth of the program, an ICS Working Committee has been established with 18 representatives from the highest levels of DDPM, including the Deputy Director General. Although it currently consists of only DDPM representatives, they are working on incorporating other agencies. The ICS Working Committee was a key driver in developing the NDRMP. They are also preparing a five-year roadmap and elevating the ICS Working Committee to include active stakeholders; i.e., Bangkok Metropolitan, Department of National Parks, Wildlife and Plant Conservation, and public health stakeholders. This is expected to foster sustainability despite potential changes in high-level personnel and administrations. The primary output of USAID/OFDA for this program is cadres of MTs. Outcomes in capability improvements are then cascaded from these MT cadres to achieve expansion outcomes, independent from USAID/OFDA involvement. Therefore, similar to the Philippines, when assessing sustainability, the capability built should be viewed in terms of whether the number and quality of MTs is sufficient to meet the needs of the country. Multiple interviews reinforced the view that 15-20 MTs (the number currently available) is insufficient to provide the training needed to fully implement ICS. While the Thai MT cadre provides high-quality training that is standardized across the country, their experience level and credibility are not to the same standards as the trainers from USFS. Therefore, while they can provide instruction to in-country ICS practitioners, they cannot themselves create new MTs without additional USFS personnel. Without this capability, any program would lack the ability to sustain itself as it could not replace MTs lost to promotion, attrition, retirement, or other factors. As the program matures, one would expect that increased opportunities to apply ICS in the Thai context will grow MT capabilities. Access to professional development and certification would also contribute to nurturing the current and future MT cadre. VI. CONCLUSIONS The conclusions of the evaluation team presented in this section are tied to the findings in Section V through the establishment of benchmarks to reflect an “if-then” linkage. Per the evaluation SOW, the evaluation team collected findings along the evaluation questions or lines of inquiry. With the collected findings, the team assessed current conditions and mapped the information against capacity benchmarks that were developed in the evaluation design stage and submitted to USAID/OFDA. This approach responds to the SOW, which specifically states that “the evaluation team must work with USAID/OFDA and USFS to map benchmarks related to the ICS programming that link to levels of capacity-building. This mapping should help to understand what essential inputs and processes are required at each level 37 to achieve capacity-building benchmarks in various country contexts.” The evaluation team scored each benchmark on a 1-4 scale to calculate the capacity building index. As an example, the evaluation team collected field data to see if findings showed the usage of ICS/NIMS at various levels of each country’s government, perceived ICS effectiveness, and support for implementation for the evaluation question, “To what extent has the ICS program built the capacity of national governments to respond to disasters effectively, particularly at the initial stages of the emergency?” With the collected findings, the team then mapped the three elements against the following benchmarks to assess the level of capacity: • ICS has been used successfully to manage events, either planned or no-notice. • Multiple stakeholder agencies have adopted and used ICS. • Formal training programs are in place and funded. The results of this benchmarking exercise to identify conclusions are discussed below and a summary of the index scoring is provided in Table 5. The rows list the benchmarks identified to evaluate each line of inquiry and the “Score” column is the ranking of the evaluation team for that benchmark using the 1-4 scale. The “Indicator” column lists the indicator number using the list of indicators the evaluation team proposed to USAID/OFDA for assessment. The full indicator list is presented in Annex II. The conclusions presented here are also summaries and syntheses of the conclusions formed for each country program. Country-specific conclusions are discussed in greater detail in each of the country evaluation reports. 1. To what extent has the ICS program built the capacity of national governments to respond to disasters effectively, particularly at the initial stages of the emergency? Table 5: Benchmarks scores Benchmarks Indonesia Philippines Thailand Indicator ICS has been used successfully to manage events, either planned or no-notice. 1 3 2 8 Multiple stakeholder agencies have adopted and used ICS. 2 3 2 7 Formal training programs are in place and funded. 3 3 3 6,37 Among the three countries, the Philippine program has applied ICS the most during planned and unplanned events. ICS has been applied at every level of government and within partner agencies throughout the Philippines. It was used to manage high-profile events like APEC and ASEAN meetings, and is regularly applied at mass gatherings like the yearly Black Nazarene festival. It is also used in disaster responses, for example Typhoon Nina, as well as smaller scale incidents. In Thailand, DDPM had its first widespread ICS application when it set up an Area Command in Surat Thani using the IMAT concept. ICS has been applied at the local level in Ayutthaya and by first responders in the public health/emergency medicine community, but examples of usage are still isolated. Except for the internal operations of the Disaster Victims Identification (DVI) unit of the INP, the evaluation team found no compelling examples of meaningful applications of ICS during planned events in Indonesia. The DVI sets up an ICS post when called in to respond to an incident, but even this process significantly limits ICS use since other INP units apply the response system familiar to them. However, all country programs have 38 not fully implemented ICS, as training has yet to cascade down to the local government levels and other first responder agencies (e.g., fire departments, police). Multiple stakeholder agencies in all country programs have adopted ICS but the degrees in usage vary. In Indonesia, ICS principles have been implemented in the BNPB, BPBD, DVI (INP), PMI, World Vision Indonesia (WVI), and the different provinces (e.g., Yogyakarta) but the level of adoption has been limited due to the “top-down” approach. The country program hopes to get an Incident Action Planning Process in the wildfire community and active and interested provinces like Jakarta and Yogyakarta will lead this first. Both areas have large amounts of response capability and incidents. By focusing limited budgets on certain key areas and hopefully with success on disasters, this will cause other provinces to apply the system. In the Philippines, ICS principles have been implemented in OCD at the national level and in regions and LGUs, shepherded by regional DRRMOs and supported by other agencies such as the BFP, DILG, and DOH, among others, though levels of implementation are uneven and have not generally filtered down to the barangay level. Stakeholder agencies have high awareness and use ICS to coordinate with incident managers, if not for their own operations. BFP, the armed forces, and DOH in particular have embraced ICS. Still, LGUs vary in their adoption and use of ICS, in part due to their own priorities and in part due to challenges in scheduling training. In Thailand, ICS principles have been implemented in DDPM at the national level and in some provinces through the DDPM provincial office, though levels of implementation are uneven. Stakeholder agencies have limited awareness and use of ICS outside of the military and public health sectors, hindering the ability to integrate resources using the ESF approach. The third benchmark is the presence of a formal training program with sufficient funding to ensure continued operations. This element was clearly present in all countries with the full suite of ICS courses being provided, including the TOT and TFI tracks, as well as the funding to provide the courses. The programs are also integrated into nationwide disaster response frameworks, as in the case of the Philippines and Thailand, and are anticipated to be in the NDRF for Indonesia. However, the current pace of course delivery in all three countries has generally not met the need at the national and provincial levels. 2. Which aspects of the ICS training program are most effective? Which aspects of the ICS training program are least effective? Table 6: Benchmarks Benchmarks Indonesia Philippines Thailand Indicator Training courses are effectively improving ICS knowledge and organizational capacity. 1 3 3 15,16,17 A Lessons Learned program is in place to capture best practices for the country and remedy shortfalls. 1 2 2 44 The ICS program helps trainees understand the effectiveness of having an organized management structure with common terminologies when responding to a disaster. This understanding has been the most effective aspect of the ICS program for all countries. But seeing the effectiveness of the management structure in actual, real-world incidents has been limited in Thailand and, to a greater extent, Indonesia. The link between training effectiveness and real-world events is that skills gained in training make it an effective program, but gains are not maximized because there are limited opportunities to apply the skills. This was evident among the members of the MT cadre that experienced fadeout during the retention test. 39 Additionally, the programs in all countries do not have a sufficient “lessons learned” mechanism to capture best practices and address program shortfalls. Lessons learned programs exist in the Philippines, but these are not formalized, centralized, or focused directly on ICS program improvement. In Olongapo City, for example, the DRRMO team members have met to discuss operations after an event or exercise. But there is no set structure to conduct these after-action meetings. There is a lack of program-level data tracking training effectiveness, making it difficult to judge actual effectiveness of training. Based on trainee perceptions, IMAT training courses have been fairly effective – no significant gaps in knowledge were reported to present serious challenges during deployment to 2017 flood operations. 3. Is the ICS targeting actors within the national government that are best positioned to effectively implement elements of the ICS system? Table 7: Benchmarks Benchmarks Indonesia Philippines Thailand Indicator ICS course offerings are diverse, include training of Senior Officials as well as Position-Specific sessions. 2 4 2 6,20 Program Directors have the authority to implement ICS activities at national and local levels. 1 4 3 19 Women receive ICS training at comparable levels to men. 2 3 3 1,2,3,4,5 Personnel at multiple levels of government and stakeholder agencies have received training. 2 3 2 20 At the operational level, personnel at multiple levels of government agencies, local government units, and stakeholder organizations have received training in all three countries. The programs have established fully-functioning training programs and are able to schedule courses, identify trainers, select participants, and plan logistics. Past trainees have provided positive feedback on the delivery of the courses. However, each country faces local circumstances that make the expansion or acceleration of training a challenge. In Indonesia, the courses are not offered nationwide and providing trainings at 6,000 inhabited islands presents logistical challenges. The OCD in the Philippines has not been able to fully meet the demands of the LGUs for training at the barangay level. In Thailand, DDPM has established course offerings through its MT cadre and is now transitioning this responsibility to the DDPM Academy to be a formalized program. However, it is not clear which classes they will offer or how they will implement the training. Personnel from all provinces can attend the courses, with significant cost subsidies from DDPM. In the Philippines and Thailand, program directors have the authority to implement ICS activities at the national and local levels by virtue of a national framework or strategy. Although the decentralized government in the Philippines means that mayors and other local officials have the final word on what response system will be used, the system has generally been socialized well enough that responsible officials empower ICS-trained staff to use the system to support response operations. In Thailand, DDPM officials at the national and local levels have the authority to implement ICS. But in practice, they still need a mechanism for engaging partners meaningfully through the ESF system, and to gain the trust and understanding of governors and other local officials. In Indonesia, the program still has to solicit buy￾in from the higher levels of the GOI and the provinces. Executive courses are available but the program will need to be more active in obtaining support at both the national level and among governors. 40 Women are well represented in the programs in all countries. Women occupied positions at all levels of training and have influential roles within the ICS programs. However, in terms of participation in courses and during observed response operations, men outnumber women significantly and tend to occupy leadership roles at a higher rate. The reviews by the evaluation team of available rosters and records also show that more men receive ICS training. 4. What country-level factors influence the effectiveness of the ICS program? Table 8: Benchmarks Benchmarks Indonesia Philippines Thailand Indicator No major cultural barriers to ICS acceptance. 1 4 3 33 Acceptance of ICS as a best practice within national government. 2 4 3 26 Acceptance of ICS as a best practice within regional / local governments. 2 3 2 27 ICS is generally accepted within the leadership and operational staff of the implementing agency and partner organizations. The leadership and operational staff of BNPB and the BPBDs, some GOI partner agencies, and some provinces, such as Jakarta and Central Java, indicate an acceptance of ICS in Indonesia, but the program will need a champion or a core group of key supporters to accelerate the application of the model. The evaluation team considers this a major barrier to the country program because the removal of this single element can potentially lead to greater acceptance by relevant GOI agencies and more accelerated and scaled-up trainings. In the Philippines, ICS has been fully accepted as a best practice within the national government, and many (but not all) LGUs are aware of ICS, have some level of trained staff, and would like to implement it fully if training is available. Although there are some cultural barriers in the Thailand program, most have been addressed in the adaptation of ICS to the Thai context or can be addressed with additional training. For example, interviewees highlighted that while it is unusual for mid-level officials to assume roles such as “incident commander,” in Thai culture, the approach is gaining traction with increased awareness of the system and examples of successful use. Some of the bureaucratic barriers (e.g., the funding issues discussed under the line of inquiry 7. What barriers to utilization of the ICS exist?), and how to get districts to use ICS will be more difficult to address, but ICS advocates are actively working on solutions to these issues through lobbying and executive-level training. The national government has accepted ICS as a best practice, codified in their law. The main challenge to full acceptance is a lack of awareness, as the centralized Thai government and national policies in place reduce the likelihood of resistance to implementation. 5. To what extent do training participants—including the different levels of the cascade training system—retain knowledge and skills transferred through the ICS trainings? Table 9: Benchmarks Benchmarks Indonesia Philippines Thailand Indicator Training recipients retain knowledge of ICS concepts following training. 2 3 3 15 41 In-country Master Trainer Cadre has the knowledge, skills, and experience to provide high quality training. 2 4 3 42 Knowledge retention is hindered in Indonesia by the limited ability to use ICS in practical situations, particularly during planned events. In real-world scenarios, the potential to mix ICS principles with other response systems exists. The program does not leverage planned events in the country. There are many events, including the annual Ramadan holiday, that could be used for the ICS Planning cycle. The program in the Philippines did not show evidence of issues with knowledge retention, though more steps could be taken to gather data on whether training is effective after participants conclude training. The Philippine MT Cadre provides high quality instruction and OCD maintains a robust instructor monitoring program to ensure quality and standardization, with a course monitor attending each session and performance data submitted to OCD. The MT Cadre can produce additional MTs, and opportunities to apply knowledge to planned events and incidents are sufficient to maintain and develop the knowledge and skills of the cadre. IMAT staff in Thailand indicated that they had retained sufficient knowledge to be effective in the 2017 flood operations. However, outside of the active MT and IMAT cadres, knowledge retention may be hindered by the limited ability to use ICS in practical situations, as use of ICS is not widespread throughout the country. Because of the lack of a tracking mechanism for participants, data on how well cascaded training recipients retained knowledge could not be obtained. The team of 15-20 active MTs out of the original 60-person 1st and 2nd Generation cadres provides high quality training, but is insufficient to meet all training needs. Current focus on developing IMAT members could hinder the ability of the country program to produce more MTs. The knowledge and skills have been developed but they need more practical experience and professional development. Objective 2: Sustainability 6. To what extent are countries able to integrate the ICS into their national frameworks or other institutionalization models? Table 10: Benchmarks Benchmarks Indonesia Philippines Thailand Indicator Laws codifying the use of ICS in disaster management are in place. 2 4 3 9 Doctrine guiding the integration of ICS with the disaster management system of the country is developed at the strategic level. 1 3 3 9 ICS has been successfully integrated and used at multiple levels of government. 1 3 2 8,10,30 42 For this evaluation, sustainability is understood not only to mean whether the program will continue, but also whether it will be able to mature and reach a point of complete implementation, without continued USAID financial/technical support. ICS has been incorporated in Indonesia on two iterations in Regulation 10 (2008) and Regulation 3 (2016) in the BNPB. But enforcement has been limited due to the decentralized structure of governance. The NDRF will be a nationwide disaster response strategy, funded by the government of New Zealand. The current efforts to develop the framework involve various stakeholders in a consultative and iterative process and the significant advancement of ICS in the country will be greatly influenced by this framework. The Philippines has a robust ICS training program in place that is already mostly self-funded, with targeted USAID technical assistance and LGU support through WFP. Laws codifying ICS use have been enacted and have helped institutionalize the program: ICS is required by law, there is a strong cohort of ICS-trained professionals in influential positions, and they have developed a suite of laws, policies, and memos circular, as well as supporting documents that guide implementation. However, implementation has not reached all LGUs from a practical standpoint (due to insufficient trainer availability) as well as a strategic shortfall (lack of standardized guidance for implementing ICS at the barangay level). Thailand has in place the foundational pieces needed to institutionalize the program: here too, ICS is required by law, there is a strong cohort of ICS-trained professionals in influential positions, and they have developed a NDRMP. Still, the pieces are not completely in place to fully mature the program. Partner awareness of their obligations under the law is low, and stakeholders outside of DDPM are not yet able to function effectively as ESFs or local partners. ICS has been successfully used in some locations and agencies, but not throughout the national and local governments. 7. What barriers to utilization of the ICS exist? Table 11: Benchmarks Benchmarks Indonesia Philippines Thailand Indicator Few reports of opposition to adopt ICS practices. 2 4 3 12,31,32,33 Country has established a committee to further ICS acceptance and use. 3 3 3 39 Opportunities exist to train on, practice, and apply ICS regularly. 1 3 2 29 As mentioned earlier, the adoption of ICS processes in Indonesia has been affected by institutional factors such as the “top-down” structure. This has resulted in some situations where the BNPB remains unable to enforce regulations in other government units, specifically on the provincial level. In the Philippines, there has been a few reports of opposition by national, regional, or local agencies to use ICS. Not all LGUs have adopted ICS despite DILG’s program incentives, but most feedback reflected that this is due more to lack of training availability or other funding priorities, rather than a negative “You can only run training once; need to make sure the LGUs take it on. They were receptive, but need to have a confidence level to try to approximate what happened in training. If they want more training or mentorship, after the training the cadre is not available for that type of support. It’s up to them to take it and run with it.” -Philippines respondent 43 impression of ICS. There are few reports of significant opposition to adopting ICS practices in Thailand, but widespread acceptance of ICS is hindered by a lack of awareness outside of DDPM. This may be tied to the development of an ICS Working Committee that only includes DDPM representation. This shortfall has been noted and DDPM is taking steps to rectify the issue; but lack of external stakeholders has likely contributed to the lack of broader awareness and implementation. In Indonesia, there is an ICS Steering Committee that is being operationally led by Ms. Regina Rahadi, one of the earlier MTs in the Indonesian ICS community. The steering committee regularly meets to discuss how ICS adoption can be expanded locally and to discuss what needs to be done for future expansion. This committee will need to continue its operations and be more engaged with GOI stakeholders in exploring opportunities to expand ICS use. There have been limited opportunities in Indonesia for training and application and not much can be expected in the near term until the higher levels of leadership in the BNPB and key agencies provide support for exercises and more courses. Opportunities to practice and apply ICS in the Philippines are plentiful. However, training opportunities need to expand in order to meet LGU demand. 8. How much adaptation (translation of core concepts and revision of key elements to better reflect local conditions) of the ICS is required before the materials can be utilized at the local government-level? Table 12: Benchmarks Benchmarks Indonesia Philippines Thailand Indicator Country-specific ICS Forms, Doctrine, and guiding documents have been provided to practitioners. 2 3 3 34,35 Country-specific examples have been used in ICS Training Courses. 1 4 3 36,35 In all three countries, ICS Forms have been translated and adapted; doctrine and guiding documents have been provided to key implementers who received training. However, each program will need to further provide these resources to targeted groups that have not yet received them. In Indonesia, course materials, manuals, and other resource documents were translated prior to the delivery of courses. Standard USFS models are also being used for training, but these resources have to be disseminated further among the stakeholder organizations. In the Philippines, concerns about the appropriate usage of forms at the barangay level, as well as use of ICS in general by barangays, remain, and guiding documents are needed for training and implementation to be successful and standardized at that level. Country-specific examples are used in ICS training in addition to the U.S. examples in the original course material. This combination is considered successful because it includes best practices and real-world examples from Philippine, and often region-specific, experiences. In Thailand, those who have received training still account for a small subset of all disaster management professionals, and do not cover all practitioners from first responders to national level agencies. 9. What factors contribute to sustainability of the ICS? Will existing investments remain viable without external donor support? Table 13: Benchmarks Benchmarks Indonesia Philippines Thailand Indicator Local/Regional/Country Governments have funded ICS training and/or exercise events. 2 3 3 38 44 ICS Master Trainer Cadre is able to meet training demands nation-wide. 2 3 2 5,41,42 Training and implementation are standardized across the country. 2 3 3 43 In Indonesia, USAID has provided funding to WVI and some provinces have provided funding for trainings, but internal funding has grown since the implementation of the decentralization plan. The MOHA requirement for provincial governments to provide funding for disaster response, including ICS, will help achieve the use of ICS without external donor support. This funding will have to be monitored. In the Philippines, training at LGU locations is funded in large part by LGUs, with some cost sharing by OCD. LGUs can also send representatives to OCD-based trainings, which are more heavily subsidized by OCD, but do not provide training to as many LGU staff members (often just 2-3, depending on space availability). Few of the courses offered are provided through USAID/OFDA and USFS at this point; however, USAID/OFDA has coordinated with the WFP to fund DRR programs, including ICS training, at LGUs in various regions of the Philippines. This funding has helped foster more widespread ICS implementation at the LGU level, and it is possible that LGUs will not fund ICS training as robustly without continued funding of this program. In Thailand, DDPM has provided funding for most aspects of the ICS training and implementation outside of the salaries and expenses of USFS personnel serving as trainers, mentors, and technical advisors. They have committed to funding all ICS training out of the DDPM Academy moving forward, and have a budget for ICS training and implementation. Without USAID/OFDA funding, the ICS program would continue in the near term; however, the MT cadre will not be able to be sustained or grow without further USFS (or other subject matter expert) instruction. With the current MT cadre, provinces can slowly train personnel over the course of several years, but it is not clear when they would be able to get a “critical mass” of personnel trained to actually implement ICS in any meaningful way in all or even a majority of the 34 provinces. The current rate of training is insufficient to build capabilities at the province level, and has not begun to address cascading to the district level, a necessary component of improving response capabilities in the “initial stages” of emergencies. Long term, without additional external funding, technical assistance, and expansion of the MT cadre, the program is likely to decline due to attrition of experienced MT cadre through retirement and/or promotion, and even modest training needs will no longer be met. The current MT cadre in Indonesia does not have enough members to meet training demands nationwide and that is one of the main reasons why USFS personnel have continued to play an active training role in the country. Despite its 431 members, the ICS MT cadre in the Philippines is not sufficient to meet the demand. As LGUs work to implement ICS, they are struggling to access timely training through OCD and regional MTs. Regions have adopted different strategies to meet the training demands of their LGUs, and some have had more success than others cascading training to LGUs and barangays. Overall, the training program has been successful, but this has created more demand as LGUs wish to replicate successful programs. In Thailand, the MT cadre is not able to meet training demands nation-wide. Currently, cascaded training is primarily provided by the MT cadre using standardized curriculum, and guidance for implementation is standardized across provinces. Though some provinces are implementing ICS fully, most do not have a sufficient amount of trained staff to meaningfully implement the system, leading to losses in standardization. Standardization of training has been most evident in the Philippines. OCD has worked diligently to ensure standardization of training throughout the country. They have developed a formal accreditation and tracking program and a database of trainees that gives each participant a unique identifier code. When they discovered that another organization was providing non-standardized training in the regions, they identified and engaged with the group to rectify the situation and re-establish standardized training practices. Course monitors attend every class to ensure high quality and standardized instruction. 45 However, not as much focus has been put on standardized implementation. After training, regions and LGUs adapt the curriculum to match their office’s capabilities, and lean forward to determine the best way to train and integrate barangays into municipality DRRMO operations. While this is an important part of the process that is needed to determine the appropriate roles of barangays in ICS and involve local leaders in preparedness and response activities, there is a risk of processes becoming non￾standardized across the Philippines. Table 14 below presents the overall scores of each country program based on the evaluation team’s findings and conclusions. Table 14: Country programs overall scores Country Score Objective 1: Effectiveness Indonesia Philippines Thailand Indicator To what extent has the ICS program built the capacity of national governments to respond to disasters effectively, particularly at the initial stages of the emergency? ICS has been used successfully to manage events, either planned or no-notice. 1 3 2 8 Multiple stakeholder agencies have adopted and used ICS. 2 3 2 7 Formal training programs are in place and funded. 3 3 3 6,37 Which aspects of the ICS training program are most effective? Which aspects of the ICS training program are least effective? Training courses are effectively improving ICS knowledge and organizational capacity. 1 4 3 15,16,17 A Lessons Learned program is in place to capture best practices for the country and remedy shortfalls. 1 2 2 44 Is the ICS targeting actors within the national government that are best positioned to effectively implement elements of the ICS system? ICS course offerings are diverse, and include training of Senior Officials as well as Position￾Specific sessions. 2 4 2 6,20 Program Directors have the authority to implement ICS activities at national and local levels. 1 4 3 19 Women receive ICS training at comparable levels to men. 2 3 3 1,2,3,4,5 Personnel at multiple levels of government and stakeholder agencies have received training. 2 3 2 20 What country-level factors influence the effectiveness of the ICS program? No major cultural barriers to ICS acceptance. 1 4 3 33 46 Acceptance of ICS as a best practice within national government. 2 4 3 26 Acceptance of ICS as a best practice within regional / local governments. 2 3 2 27 To what extent do training participants—including the different levels of the cascade training system—retain knowledge and skills transferred through the ICS trainings? Training recipients retain knowledge of ICS concepts following training. 2 3 3 15 In-country Master Trainer Cadre has the knowledge, skills, and experience to provide high quality training. 2 4 3 42 Effectiveness Total: 24 47 36 out of 56 Objective 2: Sustainability Indonesia Philippines Thailand Indicator To what extent are countries able to integrate the ICS into their national frameworks or other institutionalization models? Laws codifying the use of ICS in disaster management in place. 2 4 3 9 Doctrine guiding the integration of ICS with the disaster management system of the country is developed at the strategic level. 1 3 3 9 ICS has been successfully integrated and used at multiple levels of government. 1 3 2 8,10,30 What barriers to utilization of the ICS exist? Few reports of opposition to adopt ICS practices. 2 4 3 12,31,32,33 Country has established a committee to further ICS acceptance and use. 3 3 3 39 Opportunities exist to train on, practice, and apply ICS regularly. 1 3 2 29 How much adaptation (translation of core concepts and revision of key elements to better reflect local conditions) of the ICS is required before the materials can be utilized at the local government-level? Country-specific ICS Forms, Doctrine, and guiding documents have been provided to practitioners. 2 3 3 34,35 Country-specific examples have been used in ICS Training Courses. 1 4 3 36,35 What factors contribute to sustainability of the ICS? Will existing investments remain viable without external donor support? 47 Local/Regional/Country Governments have funded ICS training and/or exercise events. 2 3 3 38 ICS Master Trainer Cadre is able to meet training demands nation-wide. 2 3 2 5,41,42 Training and implementation are standardized across the country. 2 3 3 43 Sustainability Total: 19 36 30 out of 44 ICS Capacity Index: 43 83 66 out of 100 VII. RECOMMENDATIONS Based on the finding and conclusions presented in Sections VI and VII, respectively, this section presents the recommendations of the evaluation team for the consideration of USAID/OFDA. The recommendations presented here are relevant to all the three countries, although country-specific approaches are briefly discussed here, and in the specific country evaluation reports in greater detail. Recommendation 1: Accelerate Basic and Intermediate Training Indonesia is a logistically challenging place to conduct nationwide trainings in, with over 6,000 inhabited islands. However, these conditions, along with the fact that the country experiences numerous natural disasters in a year, also make the case to accelerate trainings. The Basic and Intermediate courses do not necessarily require a classroom-structure where trainees have to attend. The country USAID/OFDA team had discussions with the Asian Disaster Preparedness Center (APDC) and the BNPB to explore the possibility of providing online learning courses for the Basic and Intermediate levels and the USAID/OFDA team should revisit this idea. In the Philippines, there is a significant need for Basic/Intermediate ICS training, particularly at the barangay level, that could be satisfied with online or distance learning coursework, or an expedited “Basic-only” training certification. These approaches would free up the MT cadre to provide additional advanced training courses that could greatly increase LGUs’ abilities to operationalize the training they receive. USAID/OFDA should work with OCD to develop online or distance learning courses that can reliably train LGU and other partner practitioners, and/or develop an expedited “Basic-only” trainer certification that can be earned by personnel outside of the MT cadre. Recommendation 2: Strengthen MT Cadres in All Countries The MT cadre in Indonesia is small with 49 members and should be expanded to meet the training demands of the 34 provinces in the country. The program is also challenged by the recently low retention rates from the cadres’ pre-/post-tests. USAID/OFDA should support BNPB to accelerate the number of trainings to expand the MT pool. The program also needs to utilize MTs as unit leaders delivering certain portions of the training under the supervision of a fully qualified MT. While the OCD in the Philippines retains the highest number of MTs among the three countries, the OCD is regularly faced with the challenge of meeting training demands by the LGUs. In Thailand, the MT cadre has suffered significant attrition due to promotions, retirements, and concurrent duties. Thailand will need to develop additional highly qualified MTs, provide professional development opportunities, and ensure trainers are available to provide training. USAID/OFDA should 48 help Thai officials set target training benchmarks and determine the number of MTs needed to meet these goals. Recommendation 3: Provide Implementation-Focused Support and Tracking As a relatively newer program, Indonesia has to address key programmatic areas to strengthen the foundations of ICS operations in the country. Apart from accelerating training (Recommendation 1), the program should find a champion at the highest levels of the government agencies involved in disaster response to encourage the acceptance of ICS among all staff, further support adaptation and contextualization efforts of ICS, and apply ICS to planned and unplanned events. A specific area that the Philippine country program should consider is the creation of IMTs at the municipal levels as an added output of the program. With eight years into the program, 431 MTs, and approximately 27,000 individuals trained, the OCD and USAID/OFDA should answer the question, “Does the objective of the program to increase capacity mean purely providing training, or does it mean building teams at the local levels to be more responsive to disasters?” The ICS program has been focused on providing training, with little to no post-training support to LGUs and other partners as they implement ICS. Increased focus on implementation, accompanied by an ICS-focused lessons learned program would help partners implement more effective, and standardize ICS-based disaster management programs. USAID/OFDA should provide technical assistance to help OCD develop a program that meets these objectives. As DDPM transitions ICS training responsibilities to the DDPM Academy in Thailand, changes are expected in the way courses are managed. USAID/OFDA should consider offering course monitors who can ensure quality and fidelity of training from an external, trusted source. Steps should also be taken to expand course offerings to ensure diversity of course offerings (e.g., inclusion of executive level, advanced, and position-specific courses), and that response practitioners at all levels are targeted. USAID/OFDA should work with Thai officials to determine additional training requirements needed to offer these courses at the required level. Recommendation 4: Develop Intermediate-Term Performance Targets So that country initiatives can be achieved and measured objectively, OCD, BNPB, and DDPM should develop a performance monitoring plan over the next two years with indicators that have ambitious but achievable performance targets. • At the training or input level, the programs should set the higher course completion targets to meet the demands for trained personnel in both the local government and partner agencies. • At the output level, the program should not just increase the number of trained personnel but also ensure knowledge is retained through the continued application of pre/post testing and the regular practice of ICS during planned events, which relates to the outcome level below. Targets at the input level should also be set to what is needed locally for outputs; e.g., to meet LGU demands in the Philippines, to provide the needed trainings on geographically disparate islands in Indonesia, and to ensure the diversity of course offerings in Thailand. • At the outcome level, practices should include planned events and the continued documentation of ICS in real-world scenarios. While a performance monitoring plan is not a requirement under the agreement with the local government agencies, Indonesia and the Philippines have built monitoring systems for trainings. The Indonesia ICS program has collaborated with the University of Indonesia to develop a database, while the program in the Philippines has a database of an estimated 27,000 trainees that also tracks post￾training performance. The Thailand program is in its early stages and USFS should make a 49 recommendation to the DDPM to develop a system. These resources can be used to develop a performance monitoring system to track how inputs meet higher level program objectives to build ICS capacity. Recommendation 5: Assist in Developing a Participant Database Related to Recommendation 4 above, USAID/OFDA should encourage all local government agencies to develop a database that tracks training recipients. For example, the OCD in the Philippines has a database that tracks not only trainee participation, but also advancement to higher course levels and even into the MT cadre. It could however be improved with more robust pre/post data collection mechanisms. Programs should have a resource that collects and reports meaningful data to measure effectiveness, capacity, or targeting of current training efforts. Recommendation 6: Establish a performance management system within the country programs Related to Recommendations 4 and 5 above, the local government agencies should establish a performance management system and go through the complete cycle to: 1) develop a performance management plan, 2) construct a country-level results framework, 3) define indicators to track performance, and 4) conduct periodic assessments of outcomes. A performance management system will allow the country team to effectively monitor activities and manage results. As stated in Recommendation 4, foundational resources in Indonesia and the Philippines have been built with databases, and Thailand should develop a similar system. These databases can serve as primary data sources for information across the different levels of the results framework to monitor the program. The foundations to develop a performance management mechanism exist. While performance management is not a requirement and OFDA is exempted from complying with USAID’s Automated Directives System (ADS) requirements, the local government agencies can leverage free and readily available resources from the agency, such as the TIPS series on developing PMPs, results frameworks, indicators, reference sheets, data collection, and analysis. The individual country reports also discuss more targeted recommendations to address country-specific issues. 50 ANNEXES 51 ANNEX I: EVALUATION STATEMENT OF WORK RFTOP SECTION C – STATEMENT OF WORK C.1 PURPOSE USAID/OFDA seeks to award a contract to evaluate the effectiveness and sustainability of the Incident Command System (ICS) programming in the East Asia and Pacific (EAP) region2. The aim of this evaluation is to improve USAID/OFDA’s understanding of the performance of the ICS program supported by USAID/OFDA in the EAP region. The evaluation shall inform future decision-making and contribute to the evidence-base on Disaster Risk Reduction (DRR) programming. The evaluation will focus on the ICS programming in the Philippines, Indonesia, and Thailand. C.2 BACKGROUND: USAID/OFDA DRR programming in EAP Many countries in the EAP region undergo seasonal periods of increased hydrometeorological activity, experiencing cyclones and monsoon rains, which can increase the risk of floods and landslides and result in significant damage. Several EAP countries situated along the Pacific Ring of Fire experience volcanic activity, as well as earthquakes and associated tsunamis. To decrease the potential risks from these disasters, USAID/OFDA funds a range of disaster risk reduction (DRR) programming throughout the EAP region. USAID/OFDA has focused its DRR activities in the region on strategic, context-specific programs designed to meet particular risk reduction needs in each country, with capacity building as a consistent theme throughout all programs. USAID/OFDA provided nearly $34 million in FY 2013 and approximately $38.3 million in FY 2014 for DRR projects throughout EAP, including programs that integrate DRR with disaster response. More information on USAID/OFDA’s DRR programming can be found in Section J. Background to the ICS model ICS is a standardized incident management concept that is applicable to any hazard or disaster scenario. It is designed to improve coordination and communication among various national government agencies and actors with the aim of expediting emergency response. ICS was developed in the 1970s following a series of catastrophic fires in southern California that resulted in deaths, injuries, and considerable property damage. The personnel assigned to determine the causes of this disaster discovered that response problems could rarely be attributed to lack of resources or failure of tactics; instead, response problems more frequently resulted from inadequate management than from any other single reason. Weaknesses in incident management were often due to lack of accountability, poor communication, lack of an orderly planning process, absence of a predesigned yet flexible management structure, and the absence of a way to integrate interagency requirements into the response effectively. To respond to these failures, the ICS was created; it comprises a set of basic principles, an organizational structure, and an operational planning process. Principles include a clear chain of command, the use of common terminology and interoperable communication systems, standardized training, and certification for positions. Although initially used by firefighters, response agencies began to use the system to manage other types of incidents in the 1980s. Following the September 11, 2001 attacks, the use of ICS at the federal, state, and local level was mandated through a Presidential 52 Directive, and ICS is now a primary component of the U.S. National Incident Management System (NIMS). USAID/OFDA ICS programming in EAP Part of USAID/OFDA’s DRR strategy in EAP is to support programs aimed at increasing the capacity of national incident command systems to effectively respond to disasters. Overall, the objectives of ICS training programs are to enhance the capacity of countries to manage disaster response operations particularly during the initial emergency phase prior to the arrival of outside assistance. The training programs are also designed to ensure vertical integration of disaster management from the local to national level and to encourage the sustainability of the ICS program following the cessation of program activities. USAID/OFDA supports ICS development by funding the U.S Forest Service (USFS) to conduct ICS capacity building activities across the EAP region. USAID/OFDA has provided funding for ICS trainings in the ASEAN member countries of Brunei, Burma, Indonesia, the Philippines, Thailand, and Vietnam, as well as to non-ASEAN member nations Mongolia, Solomon Islands, Timor Leste, and Vanuatu. Phase one of the program introduced ICS to ASEAN members through trainings and a study tour in the United States. Under Phase two, which is ongoing, USFS is continuing regional training activities, including basic and intermediate ICS courses and the development of online ICS training modules. C.3 BACKGROUND: ICS PROGRAMS TO BE EVALUATED C.3.1 Overview of Programs This evaluation will examine the effectiveness and sustainability of USAID/OFDA funded ICS programs in the Philippines, Indonesia, and Thailand. This section summarizes the program’s activities in these three countries. Philippines USAID/OFDA has provided $1,170,000 to support ICS programming in the Philippines from FY 2010 to FY 2014. USAID/OFDA support has funded TOT courses, as well as technical support for the adaptation of course materials, the establishment of Incident Management Teams (IMTs) in focal areas, the development of national ICS policy and guidelines, the mentoring of Master Trainers, the conduct of exercises, the establishment of training and certification standards for ICS trainers, and the development of a website to track ICS training nation-wide. In FY 2012, USAID/OFDA supported the USFS to bring ICS training to the local government unit level, with a focus on local government units and provinces in four regions (Cagayan, Laguna, Sorsogon and Benguet) that USAID/OFDA targeted for a capacity-building and technical support program implemented through the World Food Program (WFP). The USFS worked with WFP and the Philippine Office of Civil Defense to conduct an ICS course for participants from selected local government units and Provincial Disaster Risk Reduction Management Committees (PDRRMCs) in each of the four regions, as well as team-based All-Hazard Incident Management Team (AHIMT) courses and exercises for newly established Incident Management Teams in Cagayan and Benguet. Indonesia USAID/OFDA has provided $520,000 to support the implementation of ICS in Indonesia since FY 2012. In 2008, National Agency for Disaster Management (BNPB) issued Regulation 10, which mandates that Government of Indonesia (GOI) response agencies use the ICS. BNPB requested USG assistance in integrating ICS and other relevant NIMS components into Indonesia’s newly established disaster management system, and ICS is a primary component of BNPB’s curriculum for GOI disaster managers. 53 Thailand USAID/OFDA has provided $861,500 for ICS programming in Thailand from FY 2010 to FY 2014 through its interagency agreement with the USFS. The USFS began working with Thailand’s Department of Disaster Prevention and Mitigation (DDPM) in 2010. In FY 2012 and 2013, USAID/OFDA provided funding for a series of ICS TOT courses. DDPM and the Master Trainers have subsequently cascaded ICS training to operational responders in a variety of agencies and regions, and have applied ICS during the conduct of tabletop and full-scale multi-agency exercises. C.3.2 Goal, Objectives, and General Theory of Change of ICS programs The goal of USAID/OFDA’s support to ICS programs is to increase the capacity of countries within in EAP to respond to disasters effectively and efficiently using the ICS model. The objectives of each of the iterations of ICS programming varies depending on the disaster profile and baseline capacity of the country in which the program is implemented. In general, however, the objectives of ICS programs are to increase the capacity of trainees to understand the ICS model and operate effectively within it once a disaster occurs. The theories of change behind this type of program are both institutional and individual. By implementing ICS training programs targeting civilian government officials involved in disaster response, the capacity of the individuals trained will increase and the effectiveness of the disaster response institutions in which they work will improve. This evaluation is designed to examine this model by studying its effectiveness and sustainability in the Philippines, Indonesia, and Thailand. C.3.3 Existing performance data The contractor will be able to access periodic performance reports from the ICS programs implemented with USAID/OFDA funding. The contractor will not have access to evaluation data or detailed quantitative results from performance monitoring. C.4 THE EVALUATION C.4.1 Purpose & Use The goal of this evaluation is to improve USAID/OFDA’s understanding of the performance of the ICS program supported by USAID/OFDA in the EAP region. The evaluation shall focus on the effectiveness and sustainability of the USAID/OFDA funded ICS programs in the Philippines, Indonesia, and Thailand. The findings of this evaluation will inform future programming decisions and adjustments to ongoing ICS programming. After five years of supporting the ICS programming in a variety of regions, USAID/OFDA seeks a deeper understanding of the successes and areas for improvement in this capacity-building model. An evaluation of the performance of the ICS program in FY 2015 will allow USAID/OFDA decision-makers to apply the information from this evaluation in future programming decisions related to DRR programming and the ICS program model specifically. With more than $4 million invested in the ICS program over the last five years, the program is a significant investment in resources from USAID/OFDA. As with many capacity-building initiatives, contextual factors can influence the results from the capacity intervention. The introduction of ICS into a variety of countries—all with their own disaster risks, government structures, and socio-economic factors—has led USAID/OFDA to seek more information about the factors for success and the potential for sustainability. C.4.2 Evaluation Objectives & Lines of Inquiry This evaluation has two objectives: understanding effectiveness and sustainability. Each objective has several lines of inquiry that shall inform the evaluation design. Data must be collected for all lines of 54 inquiry in the Philippines, Indonesia, and Thailand and will be analyzed to compare results from the different contexts. USAID/OFDA does not regard these lines of inquiry as final and could modify the lines of inquiry through the design phase of the evaluation through conversations with the evaluation team. The lines of inquiry will be considered final in the approved version of the inception report deliverable of this contract. While exact wording of the lines of inquiry could be modified through the design phase, the objectives of the evaluation as well as the intent and focus of the evaluation will not change. Objective 1: Effectiveness 1. To what extent has the ICS program built the capacity3 of national governments to respond to disasters effectively, particularly at the initial stages of the emergency? 2. Which aspects of the ICS training program are most effective? Which aspects of the ICS training program are least effective? 3. Is the ICS targeting actors within the national government that are best positioned to effectively implement elements of the ICS system? 4. What country-level factors influence the effectiveness of the ICS program? 5. To what extent do training participants—including the different levels of the cascade training system—retain knowledge and skills transferred through the ICS trainings? Objective 2: Sustainability 6. To what extent are countries able to integrate the ICS into their national frameworks or other institutionalization models? 7. What barriers to utilization of the ICS exist? 8. How much adaptation (translation of core concepts and revision of key elements to better reflect local conditions) of the ICS is required before the materials can be utilized at the local government￾level? 9. What factors contribute to sustainability of the ICS? Will existing investments remain viable without external donor support? C.4.3 Evaluation Type This performance evaluation will be summative in nature. It will evaluate programs that have closed in order to draw conclusions about their effectiveness and sustainability so that USAID/OFDA can adjust future implementation plans and strategies. C.4.4 Evaluation Approach This evaluation will take a utilization-focused evaluation approach, meaning the evaluation will be designed through extensive consultation with USAID/OFDA and the USFS. USAID/OFDA seeks to gain feedback directly from the client countries who have participated in ICS activities. The evaluation will be designed to ensure that findings from the evaluation can directly inform future programming decisions related to ICS. The evaluation team will need to tailor the evaluation approaches to the specific context of country and analyze findings accordingly. The evaluation should be designed to allow readers to draw comparisons across country locations and provide analysis of the findings for the different contexts. C.4.5 Evaluation Audience The users of this evaluation will be USAID/OFDA’s EAP team and USFS International Programs Department. More broadly, this evaluation will be read by USAID/OFDA staff in Washington, D.C., and regional field offices. This evaluation will also add to the evidence base on ICS and DRR programming. The evaluation will be available to all USAID/OFDA staff and will be made publicly available, in accordance with specific contractual requirements. 55 C.4.6 Evaluation Methods & Data Sources General guidance This evaluation must employ mixed methods: both qualitative and quantitative methods are required. Primary data must form the majority of the data collected for this evaluation; secondary data review alone will not suffice to inform the deliverables for this evaluation. The evaluation must use primary data collected in Thailand, the Philippines, and Indonesia to inform responses to the lines of inquiry of the evaluation. Any quantitative data collection that takes place as part of this evaluation must include a representative sample of the survey population and must use rigorous methods for data collection and analysis. The survey populations for any large-scale survey conducted as part of this evaluation are the individual countries (Thailand, the Philippines, and Indonesia) or groups of stakeholders within those countries, not the total population of survey populations in all three. Therefore, if a large-scale survey is conducted, there will be three separate surveys. The data must be representative at a sub-national level, though the exact level of representation will be determined during the design stage of the evaluation. USAID/OFDA prefers data with no more than a 5% margin of error and at least a 95% confidence level. However, the exact parameters of the survey design will be determined during the initial stages of the evaluation process. If less rigorous quantitative methods are needed, the Contractor must submit a justification to the COR for review and approval. Only after the COR has approved the less rigorous quantitative methods may the Contractor use them. The qualitative methods proposed should consider the various approaches to evaluating training and capacity building programs in development contexts. The evaluation plan must incorporate methods suitable to evaluating training, capacity-building, and DRR. The qualitative inquiry should be designed to analyze data from multiple levels along the capacity-building results chain. The evaluation will use multiple data collection instruments, aiming to capture data from multiple angles and data sources. Having analysis from a variety of levels will inform the findings of the evaluation. Multiple-level analysis is important to understand where and how the interventions are influencing the government’s capacity to respond to disasters. This evaluation should consider the relationships between capacity interventions and capacity outcomes, and the links between capacity and performance variables. In the selection of methods, the contractor must include ethical considerations, do no harm precautions, and informed consent. Specific methods The exact methods to be used in this evaluation will be determined through the submission and acceptance of the Inception Report and Evaluation Plan deliverables. However, USAID/OFDA requires the set of methods described in the rest of this section to be used for certain lines of inquiry in this evaluation. If, through the design process, the evaluation team finds that these methods are not suitable for responding to the identified lines of inquiry, the Contractor must provide a clear justification. Changes to these methods requirements are subject to the review and approval of the COR. • To what extent do training participants—including the different levels of the cascade training system—retain knowledge and skills transferred through the ICS trainings? o How much adaptation (translation of core concepts and revision of key elements to better reflect local conditions) of the ICS is required before the materials can be utilized at the local government-level? 56 o What factors contribute to sustainability of the ICS? Will existing investments remain viable without external donor support? o To what extent has the ICS program built the capacity4 of national governments to respond to disasters effectively, particularly at the initial stages of the emergency? There are several types of methods that USAID/OFDA requires for the above lines of inquiry. In general, the contractor must use both quantitative and qualitative methods to gather data related to the lines of inquiry above. First, the evaluation team must work with the USAID/OFDA and USFS to map benchmarks related to the ICS programming that link to levels of capacity-building. This mapping should help to understand what essential inputs and processes are required at each level to achieve capacity-building benchmarks in various country contexts. For example, one benchmark that indicates a level of capacity has been reached is the introduction of a policy that mandates the use of ICS for disaster management. This mapping must be an ongoing process during the evaluation. Second, the contractor must use a survey of training participants to measure retained knowledge over time. Third, the contractor must conduct structured or unstructured individual interviews with training participants, government officials, and other relevant stakeholders. Fourth, this evaluation must include self-assessment techniques that have the added value of allowing participants in the evaluation to reflect and learn. Self-assessment techniques must be balanced with other objective measures and data collection. Fifth, this evaluation must include focus group discussions with relevant stakeholders, community groups, and government officials. Finally, the contractor must develop a capacity building index for the ICS system must be an overall tool for evaluators to utilize in data collection. The capacity building index must be based on the ICS curriculum and ICS program objectives. • Is the ICS targeting actors within the national government that are best positioned to effectively implement elements of the ICS system? • To respond to the line of inquiry stated above, the contractor must conduct individual interviews and focus group discussions with relevant stakeholders. • To what extent are countries able to integrate the ICS into their national frameworks or other institutionalization models? • What barriers to utilization of the ICS exist? • How much adaptation (translation of core concepts and revision of key elements to better reflect local conditions) of the ICS is required before the materials can be utilized at the local government￾level? • What factors contribute to sustainability of the ICS? Will existing investments remain viable without external donor support? To respond to the lines of inquiry stated above, the contractor must use qualitative methods. If the contractor believes that quantitative methods would be useful to use for these questions, the contractor may use them. The qualitative methods must include individual interviews with training participants and government officials at a minimum. 57 Data Sources The evaluation team must interview the most relevant respondents for each of the lines of inquiry. While each line of inquiry will have a different set of respondents deemed to be most relevant, the evaluation team must collect data from the following groups of individuals, (contingent upon the receipt of informed consent): 1. Training participants 2. Government officials 3. USAID/OFDA staff 4. USFS staff 5. ICS trainers 6. Community members 7. Other members of the USG that have worked with training participants during a disaster response 8. Training curricula 9. Periodic programmatic reports The above list of respondents is not exhaustive. The contractor is encouraged to add more groups to respondents to ensure that the list is comprehensive and adequately covers all lines of inquiry. This list of respondents may include individuals who have departed the countries in question or who no longer work for the organizations that employed them during the ICS training or disaster response. As such, the evaluation team must find as many of these individuals as possible to interview. C.4.7 Limitations Most ICS programs may not have baseline data and may not have consistent performance monitoring data. There have not been previous evaluations of the programs and documentation of progress, successes, and challenges is very limited. It may take time to contact respondents who have changed jobs or job location since the training program began. C.4.8 Geographic Scope The geographic scope of this evaluation is Indonesia, the Philippines, Thailand, and the United States. Primary data collection for this evaluation – and thus travel for relevant members of the evaluation team – must occur in Indonesia, the Philippines, and Thailand. Travel to countries other than the aforementioned countries for the purposes of this contract is allowable only if the evaluation plan identifies a clear need for primary data collection to occur and if the COR approves the need. The evaluators must also prepare to interview respondents via the telephone or internet-enabled communication in countries outside of the aforementioned countries. The Contractor is responsible for accessing all countries and regions within the countries in the geographic scope of this contract. C.4.9 Programming Period Covered by Evaluation This evaluation will cover USAID/OFDA-funded ICS programs implemented between FY 2010 and FY2015. C.4.10 Period of Performance of the Evaluation This evaluation must be carried out under an agreed upon schedule of work, determined through the work plan. The period of performance of this contract is eight (8) months. C.5 DELIVERABLES 58 Each of the deliverables must be submitted to USAID/OFDA for review and approval. USAID/OFDA has the following general quality standards for each of the deliverables submitted as part of this contract: 1. Deliverables must contain no typographical or grammatical errors. 2. Deliverables must be submitted at or before the deadline established in the contract. For deliverables with deadlines established in the work plan, the Contractor must submit the deliverables at or before the deadline approved in the work plan. 3. The deliverables must be written using concise and direct language. Deliverable Contract Reference Planned Completion Date Post-Award Conference Call C.5.1 NLT 7 days after Contract Award Work Plan C.5.2 NLT 14 days after Contract Award Kick-Off Meeting C.5.3 NLT 21 days after Contract Award Inception Report C.5.4 To be determined in the work plan 59 ANNEX II: LIST OF INDICATORS No. Indicators Data Sources Data Collection Locations Objective 1: Effectiveness To what extent has the ICS program built the capacity of national governments to respond to disasters effectively, particularly at the initial stages of the emergency? 1 Number of government officials trained annually Program Plans, Records, and Reports including: ICS Training Participant rosters from program Implementers (OFDA, USFS) Training provider records Government records Course Curricula US and In-country (some can be 2 Number of government collected remotely) officials trained total 3 Number of first responders trained annually 4 Number of first responders trained total 5 Number of Master Trainers trained 6 Number of training events by type 7 Number of organizations who have adopted ICS (in plans, proclamations, etc.) 8 ICS/NIMS used successfully in responses or exercises After action reports / Incident Reviews In-Country (can be collected remotely) 9 References to ICS/NIMS components in plans, policies, procedures, and laws Government disaster response agencies plans, policies, procedures, and laws Local response agencies plans, policies, procedures 10 Usage of ICS/NIMS concepts and procedures at multiple levels Response Officials In-Country 11 Perceived effectiveness of ICS program in improving disaster response capabilities 12 Support for implementation 60 Which aspects of the ICS training program are most effective? Which aspects of the ICS training program are least effective? 13 Positive Trainee Feedback Training Feedback Forms (if available) US and In-Country (can be collected remotely) 14 Negative Trainee Feedback 8 ICS/NIMS used successfully in responses or exercises After Action Reports / Incident Reviews US 15 Retained Knowledge Survey: Amount of knowledge retained by training participants (did position, number of opportunities to apply knowledge, etc. impact retained knowledge) Training Participants In-Country 16 Impact of training (self￾assessment) 17 Organizational capacity and/or performance improvements Master Trainers Response Officials In-Country 18 Unsuccessful ICS/NIMS use in responses or exercises 16 Impact of training (self￾assessment) Is the ICS targeting actors within the national government that are best positioned to effectively implement elements of the ICS system? 12 Support for implementation Response Officials In-Country 19 Authority to implement 11 Perceived effectiveness of ICS program in improving disaster response capabilities 20 Distribution of training participants' levels of authority/activity Training records Government records Organizational charts In-Country (can be collected remotely) What country-level factors influence the effectiveness of the ICS program? 21 Existing culture of coordination/collaboration Open source reports 22 Presence of good governance indicators - accountability, transparency, rule of law 23 Resource availability 61 24 Geography 25 Frequency of disasters 26 Acceptance by national governments Response Officials In-Country 27 Acceptance by local governments 28 Perceived contributions of country level data listed above or other factors 29 Opportunities to use gained knowledge in exercises or real world events Training Participants In-Country To what extent do training participants—including the different levels of the cascade training system—retain knowledge and skills transferred through the ICS trainings? 15 Retained Knowledge Survey: Amount of knowledge retained by training participants (did position, number of opportunities to apply knowledge, etc. impact retained knowledge) Training Participants In-Country Objective 2: Sustainability To what extent are countries able to integrate the ICS into their national frameworks or other institutionalization models? 8 ICS/NIMS used successfully in responses or exercises After Action Reports/Incident Reviews In-Country (can be collected remotely) 9 References to ICS/NIMS components in plans, policies, procedures, and laws National, Regional, Local plans, policies, procedures 30 Success of integration at multiple levels of government Response Officials In-Country What barriers to utilization of the ICS exist? 31 Failed attempts to integrate Response Officials In-Country 32 National/Regional level challenges 62 33 Personal experiences of challenges at various levels of implementation (e.g., didn't understand terminology; culture not congruent with principles; didn't have country-specific examples of usage) Response Officials In-Country How much adaptation (translation of core concepts and revision of key elements to better reflect local conditions) of the ICS is required before the materials can be utilized at the local government-level? 34 Changes / adaptations to ICS forms, structures, or protocols that have been made for the country Response Officials In-country 35 Suggested adaptations / areas for improvement Response Officials 36 Changes / adaptations to ICS Training Curricula that have been made for the country USFS Trainers Master Trainers US and In-country 35 Suggested adaptations / areas for improvement 37 Training delivery challenges 37 Training delivery challenges Course evaluations (if they exist, using IOTWS evaluations as a benchmark) Course Reports US What factors contribute to sustainability of the ICS? Will existing investments remain viable without external donor support? 38 Availability of other sources of Funding - internal or international (e.g., national level funding, CDC International Fellowship Programs) Response Officials In-Country 39 Perceived sustainability of country ICS programming Response Officials In-Country 40 Establishment of Structures, such as an ICS Steering Committee, to continue ICS capability development Response Officials In-Country 63 41 Cost of additional training given availability of Master Trainers reasonable Master Trainers Training Participants Response Officials US and In-Country 42 Master Trainers provide high quality training 43 Standardized training and implementation 44 Programs to capture and share lessons learned are in place Response Officials In-Country 12 Support for implementation Response Officials In-Country 9 References to ICS/NIMS components in plans, policies, procedures, and laws Government disaster response agencies plans, policies, procedures, and laws Local response agencies plans, policies, procedures US (can be collected remotely) 38 Availability of other sources of Funding - internal or international (e.g., national level funding, CDC International Fellowship Programs) Program Activity Reports, SOWs, Open Source 12 Support for implementation Local, Regional, National plans, policies, laws, etc. 64 ANNEX III: DATA COLLECTION INSTRUMENTS Annex Table 1: Instrument for Response Officials Methods No. Indicators Questions Responses Notes Objective 1: Effectiveness To what extent has the ICS program built the capacity of national governments to respond to disasters effectively, particularly at the initial stages of the emergency? Interviews / Focus Group 10 Usage of ICS/NIMS concepts and procedures at multiple levels How many levels of government and agencies are currently using ICS? 11 Perceived effectiveness of ICS program in improving disaster response capabilities Has the ICS programming provided improved disaster response capabilities? 12 Support for implementation How much support is there for continued use of ICS nationwide? Which aspects of the ICS training program are most effective? Which aspects of the ICS training program are least effective? Interviews / Focus Group 17 Organizational capacity and/or performance improvements Have ICS training programs improved partner nation(s) organizational capacity and performance? Can you cite any specific, tangible gains? 18 Unsuccessful ICS/NIMS use in responses or exercises Are you aware of any uses of ICS/NIMS in our partner nations that have been unsuccessful or ineffective? 65 16 Impact of training (self-assessment) What have been the most effective aspects of the ICS Training program? Which aspects have been least effective? Is the ICS targeting actors within the national government that are best positioned to effectively implement elements of the ICS system? Interviews / Focus Group 12 Support for implementation Has the partner nation been supportive of implementing ICS? Has there been a specific champion? 19 Authority to implement Do the people receiving ICS training have the authority to implement ICS at their levels? 11 Perceived effectiveness of ICS program in improving disaster response capabilities How effective has the ICS programming been in improving disaster response capabilities? What country-level factors influence the effectiveness of the ICS program? Interviews / Focus Group 26 Acceptance by national governments How well is ICS accepted at the National level? 27 Acceptance by local governments How well is ICS accepted at the local level? 28 Perceived contributions of country level data listed above or other factors What factors in this partner nation have influenced the effectiveness of the ICS program, whether negative or positive? To what extent do training participants—including the different levels of the cascade training system—retain knowledge and skills transferred through the ICS trainings? Objective 2: Sustainability To what extent are countries able to integrate the ICS into their national frameworks or other institutionalization models? 66 Interviews / Focus Groups 30 Success of integration at multiple levels of government Have ICS processes, documents, and strategies been successfully integrated into response processes at local and national levels? What barriers to utilization of the ICS exist? Interviews/ Focus Groups 31 Failed attempts to integrate Are you aware of any times officials have tried to integrate ICS concepts into their frameworks, but were not successful? 32 National/Regional level challenges What challenges exist to full and effective use of ICS in the partner nation? Are there challenges at both the national and regional levels? 33 Personal experiences of challenges at various levels of implementation (e.g., didn't understand terminology; culture not congruent with principles; didn't have country￾specific examples of usage) What challenges have you encountered when trying to implement ICS? (e.g., didn't understand terminology; culture not congruent with principles; didn't have country￾specific examples of usage) How much adaptation (translation of core concepts and revision of key elements to better reflect local conditions) of the ICS is required before the materials can be utilized at the local government￾level? 67 Interviews / Focus Groups 34 Changes / adaptations to ICS forms, structures, or protocols that have been made for the country What changes have been made to standard ICS forms, structures, and protocols to meet country requirements? 35 Suggested adaptations / areas for improvement What changes should be made to enhance ICS implementation? What factors contribute to sustainability of the ICS? Will existing investments remain viable without external donor support? Interviews / Focus Groups 38 Availability of other sources of Funding - internal or international (e.g., national level funding, CDC International Fellowship Programs) Have ICS initiative been funded by organizations other than USAID/OFDA? What funding have the national or local governments provided? 39 Perceived sustainability of country ICS programming Is the country's ICS programming sustainable? i.e., if OFDA no longer sponsored it, would the country still use ICS? 40 Establishment of Structures, such as an ICS Steering Committee, to continue ICS capability development Has the partner nation established a committee responsible for advocating for and guiding the adoption of ICS practices at national and local levels? 41 Cost of additional training given availability of Master Trainers reasonable Is the cost of cascaded training reasonable and accessible to a variety of local responder organizations? 68 42 Master Trainers provide high quality training Do the in-country trainers provide high quality training that is comparable to that offered by the USFS cadre? If not, what can be done to ensure quality? 43 Standardized training and implementation Is ICS training standardized across the nation? Is implementation standardized? 44 Programs to capture and share lessons learned are in place Are programs in place to capture and share lessons learned nationally and internationally? 12 Support for implementation Asked Previously Annex Table II: Instrument for Training Cadres Methods No. Indicators Question Responses Notes Objective 1: Effectiveness To what extent has the ICS program built the capacity of national governments to respond to disasters effectively, particularly at the initial stages of the emergency? Which aspects of the ICS training program are most effective? Which aspects of the ICS training program are least effective? Interviews / Focus Group 17 Organizational capacity and/or performance improvements Have ICS training programs improved partner nation(s) organizational capacity and performance? Can you cite any specific, tangible gains? 18 Unsuccessful ICS/NIMS use in responses or exercises Are you aware of any uses of ICS/NIMS in our partner nations that have been 69 unsuccessful or ineffective? 16 Impact of training (self-assessment) What have been the most effective aspects of the ICS Training program? Which aspects have been least effective? Is the ICS targeting actors within the national government that are best positioned to effectively implement elements of the ICS system? Interviews / Focus Group 12 Support for implementation Has the partner nation been supportive of implementing ICS? Has there been a specific champion? 11 Perceived effectiveness of ICS program in improving disaster response capabilities How effective has the ICS programming been in improving disaster response capabilities? What country-level factors influence the effectiveness of the ICS program? Interviews / Focus Group 26 Acceptance by national governments How well is ICS accepted at the National level? 27 Acceptance by local governments How well is ICS accepted at the local level? 28 Perceived contributions of country level data listed above or other factors What factors in this partner nation have influenced the effectiveness of the ICS program, whether negative or positive? To what extent do training participants—including the different levels of the cascade training system—retain knowledge and skills transferred through the ICS trainings? Objective 2: Sustainability To what extent are countries able to integrate the ICS into their national frameworks or other institutionalization models? What barriers to utilization of the ICS exist? 70 Interviews/ Focus Groups 31 Failed attempts to integrate Are you aware of any times officials have tried to integrate ICS concepts into their frameworks, but were not successful? 32 National/Regional level challenges What challenges exist to full and effective use of ICS in the partner nation? Are there challenges at both the national and regional levels? 33 Personal experiences of challenges at various levels of implementation (e.g., didn't understand terminology; culture not congruent with principles; didn't have country￾specific examples of usage) What challenges have you encountered when trying to implement ICS? (e.g., didn't understand terminology; culture not congruent with principles; didn't have country￾specific examples of usage) How much adaptation (translation of core concepts and revision of key elements to better reflect local conditions) of the ICS is required before the materials can be utilized at the local government￾level? Interviews / Focus Groups 36 Changes / adaptations to ICS Training Curricula that have been made for the country What changes have been made to the ICS Training curriculum to adapt the content and/or presentation to the target country? 35 Suggested adaptations / areas for improvement What changes should be made to enhance ICS implementation? 71 37 Training delivery challenges What are the main challenges you encounter when delivering training? What factors contribute to sustainability of the ICS? Will existing investments remain viable without external donor support? Interviews / Focus Groups 38 Availability of other sources of Funding - internal or international (e.g., national level funding, CDC International Fellowship Programs) Have ICS initiative been funded by organizations other than USAID/OFDA? What funding have the national or local governments provided? 39 Perceived sustainability of country ICS programming Is the country's ICS programming sustainable? i.e., if OFDA no longer sponsored it, would the country still use ICS? 40 Establishment of Structures, such as an ICS Steering Committee, to continue ICS capability development Has the partner nation established a committee responsible for advocating for and guiding the adoption of ICS practices at national and local levels? 41 Cost of additional training given availability of Master Trainers reasonable Is the cost of cascaded training reasonable and accessible to a variety of local responder organizations? 42 Master Trainers provide high quality training Do the in-country trainers provide high quality training that is comparable to that offered by the USFS cadre? If not, 72 what can be done to ensure quality? 43 Standardized training and implementation Is ICS training standardized across the nation? Is implementation standardized? 44 Programs to capture and share lessons learned are in place Are programs in place to capture and share lessons learned nationally and internationally? 73 DevTech Systems, Inc. Independent Evaluation of the USAID/OFDA Incident Command Systems (ICS) Training Program Focus Group Discussion Questions RESPONSE OFFICIALS You have received this request to join us for this group discussion because of your participation or involvement in the Incident Command System (ICS) training program funded by the United States Agency for International Development Office of Foreign Disaster Assistance (USAID/OFDA). DevTech Systems, Inc. has been commissioned by OFDA to conduct an external evaluation of the ICS program and we are conducting this survey as part of the evaluation tasks. We would like to get your thoughts and opinions on the effectiveness and sustainability of the ICS program in building the capacity of local first responder teams to events. The information you will share will be kept strictly confidential and there will be no disclosure of any individual response. The only information that will be used for disclosure to third parties will be aggregates and summaries of the information from all participants, and a selection of comments made, without attribution to any individual who made them. We are grateful for your participation in this evaluation. The information you provide will contribute significantly to the evaluation and the work of the ICS program as a whole. Thank you very much. Sincerely, The DevTech Evaluation Team 74 Objective 1: Effectiveness The FGD moderator should attempt to elicit responses to all of the Primary Questions using Primary Questions or secondary/probe questions to elicit perception- and experience-based responses from the focus group participants. Indirect responses that come up in discussions that follow any of the moderator’s prompts should be recorded. If, in the course of the FGD, participants do not answer a Primary Question, use the secondary or probe questions to ensure that all relevant indicators have been addressed. If, in answering one Primary Question, an interviewee provides an answer for another, the interviewer may alter the original order of Primary Questions to best maintain a natural conversational flow. The moderator should use the Primary Questions as a guide and select appropriate secondary or probe questions to ask that align with the focus group’s particular set of experiences or expertise. Primary Question 1: Have you noticed a difference in the effectiveness of disaster response(s) in your country since the OFDA/USFS ICS program began? 1a. How many levels of government and agencies are currently using ICS? 1b. Has the ICS programming provided improved disaster response capabilities? Have any aspects of disaster response become less effective as a result of the training? If so, do you think these are temporary setbacks as part of the change toward ICS, or is ICS incompatible with any part of your country’s disaster response system? 1c. How much support is there for continued use of ICS nationwide? Primary Question 2: Which aspect(s) of the ICS training program have you found to be the most effective? Which aspect(s) have you found to be the least effective? 2a. Have ICS training programs improved your country’s disaster response organizational capacity and performance? Have you noticed a difference in skill level and response performance between responders who have and have not received ICS training? 2b. Are you aware of any uses of ICS/NIMS in our partner nations that have been unsuccessful or ineffective? 2c. In your opinion, what have been the most effective aspects of the ICS Training program? 2d. Which aspects have been least effective? Primary Question 3: Is the ICS targeting actors within the national government that are best positioned to effectively implement elements of the ICS system? Which actors, if any, have been left out of ICS planning, training and implementation that should have been included? 3a. Has the national government been supportive of implementing ICS? Has any particular branch, agency, organization or person (local or foreign) been a champion for ICS implementation? 3b. Do the people receiving ICS training have the authority to implement ICS at their levels? At what point did they receive this authority (i.e.: how long into the ICS training and implementation did this change occur)? 3c. How effective has the ICS programming been in improving disaster response capabilities? Can you provide a specific example – in training or in actual disaster response – to support your view? Primary Question 4: What country-level factors influence the effectiveness of the ICS program? 75 4a. How well is ICS accepted at the National level? 4b. How well is ICS accepted at the local level? 4c. What factors in this partner nation have influenced the effectiveness of the ICS program, whether negative or positive? Objective 2: Sustainability Primary Question 1: To what extent has your country integrated ICS into its national framework or other institutional models? 1a. Have ICS processes, documents, and strategies been successfully integrated into response processes at local and national levels? Primary Question 2: What barriers to utilization of the ICS exist? 2a. Are you aware of any times officials have tried to integrate ICS concepts into their frameworks, but were not successful? 2b. What challenges exist to full and effective use of ICS in your country? Are there challenges at both the national and regional levels? 2c. What challenges did you encounter or notice in integrating ICS principles into the national framework? (e.g.: didn't understand terminology; culture not congruent with principles; didn't have country-specific examples of usage) Primary Question 3: How much adaptation (translation of core concepts and revision of key elements to better reflect local conditions) of the ICS is required before the materials can be utilized at the local government-level? 3a. What changes have been made to standard ICS forms, structures, and protocols to meet country requirements? 3b. Did you encounter any challenges specific to the ICS training and training materials that hindered the integration of the ICS into your national framework? If so, what changed – or should have changed – to enhance and speed up the integration process? Who first noticed the issues, and who made the appropriate changes? 3b. What changes should be made to enhance ICS implementation? Primary Question 4: What factors contribute to sustainability of the ICS? Will existing investments remain viable without external donor support? 4a. Have ICS initiative been funded by organizations other than USAID/OFDA? If so, have these trainings helped or hindered the integration and standardization of ICS in your country? 4b. What funding have the national or local governments provided? 4b. Do you believe that the ICS program in your country would remain viable and sustainable without external donor (i.e.: USAID) funding or training? Why or why not? 76 4c. Has the government established a committee responsible for advocating for and guiding the adoption of ICS practices at national and local levels? 4d. Have you heard of any issues with cascaded training related to cost and/or accessibility? Are local responder organizations able to access training, and is this access sustainable in the long-term? 4e. Do the in-country trainers provide high quality training that is comparable to that offered by the USFS cadre? What differences have you noticed or been told about between local trainers and USFS trainers? What can be done to ensure quality? 4f. Is ICS training standardized across the nation? Is implementation standardized? 4g. Are programs in place to capture and share lessons learned nationally and internationally? Who (what government branch/agency or individual) spearheaded the development of these programs? 77 DevTech Systems, Inc. Independent Evaluation of the USAID/OFDA Incident Command Systems (ICS) Training Program Focus Group Discussion Questions TRAINER CADRES You have received this request to join us for this group discussion because of your participation or involvement as a trainer in the Incident Command System (ICS) training program funded by the United States Agency for International Development Office of Foreign Disaster Assistance (USAID/OFDA). DevTech Systems, Inc. has been commissioned by OFDA to conduct an external evaluation of the ICS program and we are conducting this survey as part of the evaluation tasks. We would like to get your thoughts and opinions on the effectiveness and sustainability of the ICS program in building the capacity of local first responder teams to events. The information you will share will be kept strictly confidential and there will be no disclosure of any individual response. The only information that will be used for disclosure to third parties will be aggregates and summaries of the information from all participants, and a selection of comments made, without attribution to any individual who made them. We are grateful for your participation in this evaluation. The information you provide will contribute significantly to the evaluation and the work of the ICS program as a whole. Thank you very much. Sincerely, The DevTech Evaluation Team 78 Objective 1: Effectiveness The FGD moderator should attempt to elicit responses to all of the Primary Questions using Primary Questions or secondary/probe questions to elicit perception- and experience-based responses from the focus group participants. Indirect responses that come up in discussions that follow any of the moderator’s prompts should be recorded. If, in the course of the FGD, participants do not answer a Primary Question, use the secondary or probe questions to ensure that all relevant indicators have been addressed. If, in answering one Primary Question, an interviewee provides an answer for another, the interviewer may alter the original order of Primary Questions to best maintain a natural conversational flow. The moderator should use the Primary Questions as a guide and select appropriate secondary or probe questions to ask that align with the focus group’s particular set of experiences or expertise. Primary Question 1: Have you noticed a difference in the effectiveness of disaster response(s) in your country since the OFDA/USFS ICS program began? 1a. If so, what changes that have occurred? Can you give an example of these changes? At what level – national, provincial, organizational, etc. – have these changes been the most profound? 1b. Would you say that the national disaster response effectiveness has improved, worsened, or remained the same as before? Can you provide specific examples? 1c. Do you believe that your country could effectively respond to a disaster without foreign assistance? Why or why not? 1d. If you could use foreign assistance for one aspect of a disaster response, which aspect would that be? Why? 1e. Has the ICS program changed the capacity of your country to respond to a disaster? Primary Question 2: Which aspect(s) of the ICS training program have you found to be the most effective? Which aspect(s) have you found to be the least effective? 2a. In your experience, how did the ICS training programs change the way in which your country’s disaster response functioned? 2b. Have you found that any of the uses of ICS/NIMS have been unsuccessful, ineffective or unnecessary? Can you please provide an example? Probe: Other than the USFS ICS training, have you received similar training? If so, when, and who provided it? In what ways was the training similar? In what ways was the training different? 2c. What aspect of ICS training has been the most useful at improving national organizational capacity and performance? 2d. What aspect of ICS training has been the least useful? Primary Question 3: Which actors (agencies, individuals, offices) work or have worked with the ICS programmers? Why do you believe these actors were targeted? Note for moderator: Probe to find out at what level – national, provincial, organizational, or other – the focus group participants have noticed the changes being the most profound. Make sure to note in which level(s) of ICS training participants have had direct experience. Note for moderator: Find out at what point along the post-disaster timeline (i.e.: immediately following a disaster or later) the participants view their countries’ response strengths and weaknesses. 79 3a. Is there a more appropriate actor or set of actors that should have been targeted instead? Probe: Where (geographically, provincially) is the ICS program lacking? 3b. Has the national government supported the implementation of ICS? If yes, was this support given quickly, or did it take time for a strong relationship between USFS/OFDA and the national government to form? 3c. Has there been a person or agency that has been particularly supportive of ICS (in other words: has there been a champion who has worked very hard to ensure that ICS programming is incorporated into the national system)? Primary Question 4: What factors have influenced the effectiveness of the ICS program, in the country overall and in your province/organization/agency specifically? 4a. How well is ICS accepted at the National level? 4b. How well is ICS accepted at the local level? 4c. Do you think that your country has incorporated ICS effectively? Why or why not? 4d. Compared to other countries in the region, how well do you believe your country has incorporated and utilized ICS? What factors have contributed to your country’s relative success or slower pace? What challenges have you notices or experienced in the process of integrating ICS into your country’s national framework? Objective 2: Sustainability Primary Question 1: To what extent has your country integrated ICS into its national framework (or other relevant institutional models)? What barriers to the full utilization of ICS exist? 1a. Are you aware of any times officials have tried to integrate ICS concepts into their frameworks, but were not successful? 1b. What challenges have been overcome since ICS implementation and training began? What challenges still exist, at both the national and regional levels? Probe: What needs to take place for ICS to be fully integrated and utilized? What changes need to occur, and who should be in charge? 1c. When you were first trained in ICS, were there any aspects of the training that you found to be especially challenging? Probe: Could the training have been changed to provide a more fluid, intuitive explanation of ICS? Did the concepts (terminology, principles, and country-specific examples of usage) seem relevant to local users or did they seem foreign? Primary Question 2: Is further adaptation (translation of core concepts and revision of key elements to better reflect local conditions) of the ICS is required before the materials can be utilized at the local government-level? 80 2a. What changes have been made to standard ICS forms, structures, and protocols to meet country requirements? 2b. What changes should be made to enhance ICS implementation? 2c. What are the main challenges you encounter when delivering training? Primary Question 3: Do you think that ICS will continue to grow and become central in your country’s disaster response system? Do you foresee ICS remaining sustainable in the long-term without external donor support? 3a. Have you been trained – or heard of trainings – in ICS by organizations other than USAID/OFDA (USFS)? 3b. Has your local or national government provided funding for ICS training that you know of? 3b. Is the country's ICS programming currently sustainable? i.e. if OFDA no longer sponsored it, would the country still use ICS? If not, what would need to change to make it sustainable? 3c. Has the partner nation established a committee responsible for advocating for and guiding the adoption of ICS practices at national and local levels? Has your local region/province/town had a voice in steering ICS practices? 3d. Did your training end up costing you money, or was it supported? Who (if anyone) supported it? As far as you know, is the cost of cascaded training reasonable for local responders and organizations, or is it cost-prohibitive? Is it geographically accessible? What obstacles to giving and receiving training exist? 3e. Is the cascaded training of comparable quality as the training offered by the USFS cadre? Do you believe the training has changed in any way without the USFS cadre? If so, are these changes improvements or impediments to the full utilization of ICS? 3f. Is ICS training standardized across the nation? Is implementation standardized? 3g. Are programs in place to capture and share lessons learned nationally and internationally? Have you participated in these programs? 81 DevTech Systems, Inc. Independent Evaluation of the USAID/OFDA Incident Command Systems (ICS) Training Program Retained Knowledge Survey You have received this request to complete this survey because of your participation or involvement in the Incident Command System (ICS) training program funded by the United States Agency for International Development Office of Foreign Disaster Assistance (USAID/OFDA). DevTech Systems, Inc. has been commissioned by OFDA to conduct an external evaluation of the ICS program and we are conducting this survey as part of the evaluation tasks. You may access the survey by clicking on the link provided below. This link is uniquely tied to this survey and your email address. Please do not forward this message. The survey is estimated to take approximately 10-15 minutes to complete. You may also save your responses at any point of the survey and continue answering it at a later time if you wish. We would appreciate it if you could complete the survey by July 11. The survey results will be kept strictly confidential and there will be no disclosure of any individual survey response. The only information that will be used for disclosure to third parties will be aggregates and summaries of the survey results from all participants, and a selection of comments made, without attribution to any individual who made them. We are grateful for your participation in this survey. The information you provide will contribute significantly to the evaluation and the work of the ICS program as a whole. Thank you very much. Sincerely, The DevTech Evaluation Team 82 Respondent Information: Name: Age: Occupation: [_] 1 Manager [_] 2 Doctor/Nurse [_] 3 Teacher [_] 4 Lawyer/ [_] 5 Engineer [_] 6 Other (professional) ________ [_] 7 Technician [_] 8 Office worker (e.g. secretary) [_] 9 Service worker (e.g. mechanic) [_] 10 Care worker [_] 1 1 Religious / Community leader [_] 1 2 Police / security [_] 1 3 Military [_] 1 4 Other (skilled) [_] 98 Refusal [_] 99 Do not know Organization/Professional affiliation: Organization/Professional title or position: Country” District or region City: Years in organization: Email address: Education (highest degree completed): 83 [_] 1 Elementary/grade school [_] 2 High school [_] 3 Technical/Vocational [_] 4 College/Bachelors [_] 5 Graduate degree (MA, MS, Ph.D.) [_] 6 MD/Medical degree [_] 7 JD/Law degree [_] 98 Refusal [_] 99 Do not know ICS training(s) received Please check (√) and provide the month and year the course was taken and instructor. √ Training course name Month Year Instructor USFS In-country Master Trainer Basic: ICS For Senior Officials/Executives Introduction to NIMS Introduction to ICS Intermediate: Emergency Operations Center (EOC) Multi-Agency Coordination (MAC) ICS Position Courses: Integrated Planning Section Operations Logistics Finance/Admin Operations Command Advanced: Train-the-Trainer Developing Emergency Plans Exercise Design and Evaluation All-Hazards Incident Management Team (AHIMT) Incident Commander Training (ICT) 84 Section I: ICS Effectiveness. “Effectiveness” is defined to mean that project activities met stated goals. For ICS, the program is considered effective if trainings and other activities met the stated goal of increasing the country’s capacity to efficiently respond to disasters using the ICS model. 1. Which aspects of the ICS training program(s) you participated in are most effective? 2. Which aspects of the ICS training program are least effective? 3. Have you used the knowledge you gained from ICS training in an exercise? a. No ____. If no, skip to the next question. b. Yes ___. If yes, please explain below ___________________________________________________________________ ___________________________________________________________________ ___________________________________________________________________ 4. Have you used the knowledge you gained from ICS training in a real-world event? a. No ____. If no, skip to the next question. b. Yes ___. If yes, please explain below ___________________________________________________________________ ___________________________________________________________________ ___________________________________________________________________ 5. Do you see the use of ICS/NIMS concepts and procedures within your organization? a. Not at all b. Only at a minimum c. Occasionally d. Frequently e. Regularly Please explain. ___________________________________________________________________ 85 ___________________________________________________________________ ___________________________________________________________________ 6. Did you see any challenges in your organization’s use of ICS/NIMS concepts and procedures? a. Not at all b. Only at a minimum c. Occasionally d. Frequently e. Regularly Please explain. ___________________________________________________________________ ___________________________________________________________________ ___________________________________________________________________ 7. Did you see any challenges in your use of ICS/NIMS concepts and procedures within your organization? a. Not at all b. Only at a minimum c. Occasionally d. Frequently e. Regularly Please explain. ___________________________________________________________________ ___________________________________________________________________ ___________________________________________________________________ 8. What suggestions do you have to improve the effectiveness of ICS Programming in your country? Please explain. ___________________________________________________________________ ___________________________________________________________________ ___________________________________________________________________ Section II: Retained knowledge 86 Please select an answer for each question below. This section is a not a test and is designed to determine which parts of ICS training needs to be strengthened for future courses. As mentioned previously, answers in this section will remain confidential. 9. Which Section Chief is responsible for ensuring that assigned incident personnel are fed and have communications, medical support, and transportation as needed to meet the operational objectives? a. Operations b. Planning c. Command Staff d. Finance/Admin e. Logistics f. Liaison Officer 10. Command is: a. Directing, ordering, or controlling by virtue of explicit statutory, regulatory, or delegated authority. b. The ability to control information exchange within and across organizations involved in an incident. c. Based on the number of individuals or resources that one supervisor can manage effectively during an incident. d. Assumed by the individual who is the highest ranking person on the scene regardless of experience or training level. 11. Interoperability means: a. Surrounding jurisdictions all purchase the same type of communications hardware and software. b. Personnel from different jurisdictions can all perform the same tasks using the same protocols. c. Communication equipment, procedures, and systems can operate together during a response. d. A single plan is used to direct the tactical assignments with the Operations Section. 12. ICS defines a manageable span of control as: a.No more than 8 individuals reporting to a single manager b.5 to 9 resources, with 7 being optimum c. No fewer than 4 direct reports, or the structure is inefficient d.3 to 7 resources, with 5 being optimum 13. During initial response actions, the person currently in charge (Incident Commander) must: 1 ___________________________________(Select from the list below.) 2 Determine if life is at immediate risk. 3 Ensure that personnel safety factors are taken into account. 4 Determine if there are any environmental issues that need to be addressed. 87 a. Assert authority when assuming command. b. Help execute critical tactical activities. c. Gather input from stakeholders. d. Size up the situation. 14. Which of the statements is correct regarding the use of ICS in expanding incidents? a. An Incident Commander always directly manages the incident. b. ICS Supervisors should delegate a task only when they cannot perform the task effectively themselves. c. When a Unified Command is in place, there is still a single Incident Commander in charge. d. Designated Agency Incident Commanders must have the authority to act on behalf of their agency in order to perform as part of a Unified Command. 15. Which of the following is an example of an objective that follows the SMART characteristics? a. As needed, provide assistance to those who might have damage from the fire. b. Enable public works teams to locate downed power lines or set up generators for 48 hours. c. Clear all debris and reopen roadways no later than 0700 tomorrow morning. d. Facilitate the provision of assistance to the victims of the airplane crash. 16. Which of the following is a true statement? a. Area Command replaces the incident-level ICS organizations or functions. b. Area Command should be collocated with one of the Incident Command Posts. c. When Area Command is established, the Incident Commander(s) report to the Area Commander. d. Area Command establishes the Incident Action Plan for the individual Incident Commanders. 17. An incident is geographically dispersed, and the incident base cannot support the incident logistical needs. Assuming that you do not want to divide the incident in two separate incidents, what is another option for managing this incident? a. Collocate the logistics base with the staging areas, with the Staging Area Manager assuming responsibility. b. Divide the Logistics Section into Divisions that represent each geographic area reporting to a Supervisor. c. Establish a second Logistics Section reporting to a Deputy Incident Commander (Logistics). d. Have the Emergency Operations Center establish a Logistics Section and assume command responsibility for that function. 18. What type of ICS structure is illustrated below? Figure 1: ICS organization with Incident Command, Operations Section, Planning Section, Logistics Section, and Finance/Admin Sections. Reporting to the Operations Section are Branch I (Incident 1), Branch II (Incident 2), and Branch III (Incident 3). 88 a. Area Command b. Incident Complex c. Unified Command d. Incident Command Post Thank you very much for your participation. END OF SURVEY 89 ANNEX IV: SOURCES OF INFORMATION Annex Table III: List of Program Documents Reviewed Program Documents 4th Quarter FY10 Report - OFDA Philippines Planning 1st Quarter FY11 Report - OFDA ASEAN ICS 2nd Quarter FY11 Report - OFDA ASEAN ICS 3rd Quarter FY11 Report - OFDA ASEAN ICS 4th Quarter FY11 Report - OFDA ASEAN ICS 1st Quarter FY12 Report - OFDA ASEAN ICS 2nd Quarter FY12 Report - OFDA ASEAN ICS 3rd Quarter FY12 Report - OFDA EAP ICS Activities 4th Quarter FY12 Report - OFDA EAP ICS Activities 1st Quarter FY13 Report - OFDA EAP ICS Activities 2nd Quarter FY13 Report - OFDA EAP ICS Activities 3rd Quarter FY13 Report - OFDA EAP ICS Activities 4th Quarter FY13 Report - OFDA EAP ICS Activities 1st Quarter FY14 Report - OFDA EAP ICS Activities 2nd Quarter FY14 Report - OFDA EAP Activities 3rd Quarter FY14 Report - OFDA EAP Activities 4th Quarter FY14 Report - OFDA EAP ICS Activities 1st Quarter FY15 Report - OFDA EAP Activities 2nd Quarter FY15 Report - OFDA EAP Activities 3rd Quarter FY15 Report - State ASEAN ICS 2012 Funds 1st Quarter FY16 Report - OFDA EAP Activities 2nd Quarter FY16 Report - OFDA EAP Activities 3rd Quarter FY16 Report - OFDA EAP Activities 90 Annex Table IV: List of Program Documents Reviewed Indonesia 201211 Indonesia ICS SOW with Budget 20150803 Indonesia ICS_3 Yar_SOW_final_w_budgets Attachment C - DRR Assessment Attachment D - ICS SOW - 60 TOT 20120321 Indonesia ICS SOW - USAID Mission Indonesia Summary Email - Oct 18 2016 Indonesia USFS Disaster Management - FY 2012 OFDA Philippines 2011_03 Philippine LSC FSC OSC IC Course Report 201106 Philippines Pilot Project ICS Training Report 201111 Philippines AHIMT Course Report 201405 USFS Typhoon Yolanda_AAR_Trip_Report 20140504 FOG Workshop Trip Report 20141215 Philippines Trip Report Phi-ICS Accomplishment Report 2012 Philippines USFS Disaster Management - FY 2012 DASP Philippines USFS Disaster Management - FY 2013 DASP Philippines USFS Disaster Management - FY 2014 DASP Philippines USFS Disaster Management SOW- FY 2015 2010 Philippines Planning - OFDA SOW 2011 Philippines ICS - OFDA SOW 201010 Philippines Plans Course Report Final Report - May 2012 ICS Courses - WFP Provinces OCD Pilot Program Proposal - Negros Occidental Region OCD Pilot Program Proposal - Olongapo PDMO OCD Pilot Program Proposal - Surallah Municipality Mindanao Thailand 201103 Thailand Plans Course Report 201105 LSC FSC OSC IC Course Report 201111 Thailand ICS Concept Paper 201202 Thailand TOT Course Report 2011 Thailand ICS - OFDA SOW 91 201102 Thailand Basic Intermediate ICS Course Report Thailand USFS Disaster Management - FY 2016 Thailand USFS Disaster Management - FY 2015 Thailand USFS Disaster Management - FY 2014 DASP Thailand USFS Disaster Management - FY 2013 DASP Thailand USFS Disaster Management - FY 2012 DASP ICSapplicationThailand - 2013 Trip Report_Thailand_5.2012 Trip Report_Thailand_8.2012 National Disaster Risk Management Plan (2015) NIMS National Incident Management System (Thai Language) ICS Planning Handbook (Thai Language) Annex Table V: List of Interview and Group Discussion Participants Indonesia Name Organization and Designation Endang Achadiat SKM, DKI Jakarta Ms. Alsinta Master Trainer (MT), Central Java Harlan Hale OFDA Regional Advisor Mujtahiddin Indonesian Red Cross, Master Trainer Mr. Nugroho Disaster Victim Identification Unit (DVI) staff member, Indonesia National Police (INP) Marg Olson USFS Disaster Management Specialist Yusak Oppusunggu USAID OFDA Program Managers Regina Rahadi USAID OFDA Program Managers Mindaraga Rahardja (Iwan) Humanitarian Affairs Analyst/Reporting, UNOCHA Joe Reyes USFS, Master Trainer Danang Samsu Master Trainer Efraim Sitinjak World Vision Ms. Supriyati BNPB, Master Trainer Dr. Bagus Tjahjono, MPH Pusdiklat Penanggulangan Bencana, BNPB Rusty Witwer USFS, Master Trainer 92 Philippines Name Organization (and designation) Dr. Joseph Bacareza Department of Interior and Local Government (DILG) Original MT Cadre Aubrey Bautista Department of Social Welfare and Development (DSWD) Disaster Response Assistance & Management Bureau (DREAMB), Development Officer ICS Level 3 trained Wenceslao Blas, MD DOH, Assistant Regional DRRMH Rodel Cabaddu DSWD DREAMB Office, ICS Level 3 trained Maj. Gen. Romulo H. Cabantac (Ret.) Regional Director, OCD NCR Senior Vice-Chairperson, MMDRRMC Ryan Castaneda Metro Manila Deveklopment Authority (MMDA) Public Safety Division Planning Section Chief Patrick C. Co, MD DOH, Epidemiology and Health Emergency Cluster Raden D. Dimaano, C.E., PDGH Sorsogon Province DRRMO Bim Dineiros Sorsogon City DRRMO Glenn Diwa OCD, Region 3 Capacity Building and Training Service (CBTS) Carlo Elepongga Olongapo DRRMO Research and Planning Ivanhoe Escardin, MD DOH Cindy Garcia, PSC, Adminstration and Training Chief, Quezon City DRRMO Esther Geraltoy DSWD DREAMB Office, Division Chief IMT 1st Batch Juan Blenn Huelgas National Programs Officer, UN World Food Programme Jeff Lapid Olongapo City DRRM Official; MT Cadre Instructor Ronald Law, MD DOH, Chief of Preparedness Division Angelito Layug DRRM Officer, Olongapo City; original MT Cadre Mayor Sally Lee Sorsogon City Mike Marasigan Acting Disaster Control Division Chief/Head, QCDRRMO Glenly Monje ADRRMO – Olongapo City 93 Julia Nebrija MMDA, Operations John Palmo Sorsogon City DRRMO Joe-Mar Perez OCD NCR, CBTS Susan Quiambao OCD, OIC-Logistics Artel Rivera, MD DOH, Chief of Response Division; original MT Cadre Ramon Santiago MMDA Consultant and DRRM Advisor Director Vicente F. Tomazar OCD, Operations Service GOP partner agencies presenters at Bangladesh study tour ASEAN National IMT: Col. PJ Bondoc Philippine Army Lt. Joven Aclan ASEAN IMT Operations Section Chief Lt. Esther Cirunay Philippine Navy 1st Lt. Gillbert Bengao Philippine Army Jello Mangaoang OCD Fred Vargas OCD Ops Division Joe-Mar Perez OCD CBTS Liaison Officer Darius Vallejos ASEAN IMT Asst Liaison Officer Monique Jacob OCD Sr Supt Randolph Bides Bureau of Fire Prevention Capt. Lyndon del Rosario AFP: Chief, HADR Brigade, ISOP OJ3 Robin Lim DILG Ivanhoe Escardin, MD DOH Thailand Title First name Last name Designation Organization Mr. Somboon Malak Trainer Disaster Prevention and Mitigation Academy (DPMA), DDPM Ms. Paoraumpai Janya Director DPMA, DDPM Mr. Santi Bussabongthong Plan and Policy Analyst, Professional Level DDPM Mr. Udomsak Kawnuna Chief of Disaster Prevention and Mitigation Provincial DDPM Ayutthaya province 94 Title First name Last name Designation Organization Ms. Angsumalin Angsusingha Plan and Policy Analyst, Senior Professional Level Department of Disaster Prevention and Mitigation Mr. Chalermrat Chansivanont Plan and Policy Analyst, Professional Level Department of Disaster Prevention and Mitigation Ms. Nichanan Teerakamtorn Nurse Red Cross Surin province Ms. Teetat Taotong Victim Relief Assistance DDPM Bangkok Ms. Marisa Iamsila Forensic Scientist Central Institute of Forensic Science Ms. Kwanchai Tongaatong Plan and Policy Analyst DDPM Pathumthani regional office no.1 Ms. Wantipa Chanpong Forester, Professional level Department of National Park, and Wildlife Preservation Mr. Boonprasong Nuansai Chief DDPM Khon Kean Mr. Krisda Dabpol-on Chief Fire Department, Sakol Nakorn Mr. Rittichai Thammasang Victim Relief Assistance DDPM Nakorn Pathom Mr. Mitri Sriya Deputy, Emergency Monitoring, Preparedness and Respponse Office of Atom for Peace Mr. Chaloemsak Junklang Fire instructor Nakorn Ratchasima Municipality Ms. Chakreeya Setthaseree Plan and Policy Analyst, Senior Professional Level DDPM Uttaradit Mr. Suwat Plubplaeng Plan and Policy Analyst DDPM Suphanburi Ms. Paweena Thongsagulphan Plan and Policy Analyst, Senior Professional Level DDPM Nakorn Naiyok Ms. Tuntiva Rangrongtanin Plan and Policy Analyst, Professional Level DDPM Bangkok Ms. Wongduan Mangmee Plan and Policy Analyst, Professional Level DDPM Nakorn Sawan Mr. Thomsan Chaisri Mechanic Experienced Level DDPM Nan Mr. Khamron Imnei Plan and Policy Analyst, Senior Professional Level DDPM Sukhothai 95 Title First name Last name Designation Organization Mr. Kaweewat Sukkasame Mechanic Senior Level DDPM Lampang Mr. Patcharin Luangkham Communicaiton Specialist DDPM Chiang Rai Mr. Suppapimit Paorik Director DPMRC 2 Suphanburi Mr. Rachit Sudpum Deputy Governor Yala Province Mr. Somsak Sontarapornpol Director of water management and maintainance Devision Regional Irrigation Office 16 Ms. Suttima Sanyawong Director News and Program Division Public Relation Center Region 5 Suratthani Mr. Warat Surawadee Plan and Policy Analyst, Professional Level DDPM Bangkok Mr. Dusit Pongsapipat Plan and Policy Analyst, Professional Level DDPM Bangkok Ms. Pannapa Na Nan Plan and Policy Analyst, Professional Level DDPM Bangkok Mr. Pongsatorn Sirisakorn Director Disaster Mitigation Directing Center DDPM Bangkok Actin g Sub Lt. Trakul Totham Songkla Area Command Chief Songkla Area Command/DDPM Phuket Mr. Ammat Chaitaweewong Songkla Area Command staff DDPM Regional Office No. 12 Mr. Wasan Chaitaweewong Acting Head DDPM Songkla Ms. Pastreeya Koomban Plan and Policy Analyst DDPM Phuket Ms. Aimon Emsakul Administration Officer Red Cross Phuket Mr. Chaknarong Naimolee Secretariat Ayutthaya Ruamjai Volunteer Association Ms. Supreeporn Utsaha Administration Officer, Professional Level DDPM Ayutthaya Ms. Supitchaya Sirsanroongreung Manager National Institute of Emergency Medicine 96 ANNEX V: FIELD PHOTOS 1. Responsible Official (RO) briefing during Day 1 of ICS staging for ASEAN meetings (Philippines) 2. Supply of temporary shelters in Quezon City DRRMO (Philippines) 97 3. ICS Forms from 2012 Holy Week (Semana Santa) Festivities (Philippines) 4. Basic/Intermediate TOT Course in Sentul (Indonesia) 98 5. National Command Center at DDPM Headquarters in Bangkok (Thailand) 99 ANNEX VI: SUMMARY INFORMATION ON THE EVALUATION TEAM Reuben R. Hermoso, Team Leader Reuben R. Hermoso has experience in designing and implementing quantitative monitoring and evaluation (M&E) methods for projects under USAID, the World Bank, the United States Department of State, and other international donor agencies and organizations. He has developed various M&E components such as framework construction, indicator development, survey design and administration, sampling, data collection and analysis, and evaluation methods design. He has technical qualifications in conducting more rigorous experimental and quasi-experimental impact evaluation methods such as random assignment, double difference, regression discontinuity, and matching. He has done M&E work for projects i n microfinance and rural credit, entrepreneurship and small business development, food and agriculture, health, youth and education, and other sectors. He has a Ph.D. in Economics and he serves as the Director of Monitoring and Evaluation for DevTech Systems, Inc. Frances Christine Fisher Veasey, ICS Subject Matter Expert Frances Veasey is an accomplished project manager, facilitator, and principal analyst with over 14 years in the field of emergency management/response and the National Incident Management System (NIMS). She is an exercise practitioner with experience leading, coordinating, facilitating, and evaluating exercises focused on assessing organizational capacities for responding to disasters. She has evaluated and led evaluation teams for civilian and military operations centers implementing Incident Command System (ICS) structures and processes. She also has experience and specialized knowledge in Asia-Pacific issues, domestic and international disaster response, and public health and infectious diseases. Ms. Veasey served as the Deputy Director of Banyan Analytics, an ANSER Board of Trustees initiative focused on the Asia-Pacific region. In addition to managing the overall operations of the institute, she led all analytic activities, implementing strict quality controls for all documents, including the institute’s bi￾weekly newsletter, the Banyan Analytics Brief. She led the institute’s disaster response / emergency preparedness research vector, and contributed subject matter expertise to all analytical works. 100 For inquiries regarding this report, please contact ofdainquiries@ofda.gov U.S. Agency for International Development 1300 Pennsylvania Avenue, NW Washington, DC 20523