INCIDENT COMMAND SYSTEM (ICS) PERFORMANCE EVALUATION THAILAND COUNTRY REPORT This publication was produced at the request of the United States Agency for International Development. It was prepared independently by DevTech Systems, Inc. i INCIDENT COMMAND SYSTEM (ICS) PERFORMANCE EVALUATION THAILAND COUNTRY REPORT A Performance Evaluation of the ICS System in Thailand Covering the Assistance Provided by USAID/OFDA from 2011 August 14, 2017 USAID Contract No AID-OAA-I-15-0008 DISCLAIMER The author’s views expressed in this publication do not necessarily reflect the views of the United States Agency for International Development or the United States Government. Cover photo: Incident Management Assistance Team (IMAT) Training in Nakorn Naiyok i Contents Acronyms.......................................................................................................................................................................ii I. Evaluation Purpose ...............................................................................................................................................1 II. Evaluation Questions............................................................................................................................................1 III. Project Background...............................................................................................................................................2 IV. Evaluation Methods & Limitations........................................................................................................................4 V. Findings.................................................................................................................................................................7 VI. Conclusions.........................................................................................................................................................18 VII. Recommendations..........................................................................................................................................26 Annex I: Evaluation Statement of Work ......................................................................................................................29 Annex II: List of Indicators ...........................................................................................................................................37 Annex III: Data Collection Instruments........................................................................................................................43 Annex IV. Sources of Information ..........................................................................................................................60 Annex V. Field Photos..................................................................................................................................................64 ii Acronyms AAR After Action Review/Report AC Area Command ACDM ASEAN Committee on Disaster Management AHIMT All-Hazard Incident Management Team ADPC Asia Disaster Preparedness Center ASEAN Association of Southeast Asian Nations COR Contracting Officer Representative DASP Disaster Assistance Support Program DDPM Department of Disaster Prevention and Mitigation DEC Development Experience Clearinghouse DOH Department of Health DPMRC Disaster Prevention and Mitigation Regional Center DRR Disaster Risk Reduction DOS Department of State EAP East Asia Pacific EOC Emergency Operations Center ESF Emergency Support Function FGDs Focus group discussions FY Fiscal Year GIs Group interviews ICP Incident Command Post ICS Incident Command System IMATs Incident Management Assistance Teams IMTs Incident Management Teams KIIs Key informant interviews MT Master Trainer NDRMP National Disaster Risk Management Plan (2015) NGO Non-Governmental Organization NIEM National Institute of Emergency Medicine (Thailand) NIMS National Incident Management System OFDA Office of U.S. Foreign Disaster Assistance POC Point of Contact RFTOP Request for Task Order Proposal RTG Royal Thai Government SEA Southeast Asia iii SOW Statement of Work THAIMAT Thailand Incident Management Assistance Team ToC Theory of Change TOT Training-of-Trainers UN United Nations USAID United States Agency for International Development USD United States Dollar USG United States Government USFS United States Forest Service UFE Utilization-focused evaluation WFP World Food Program 1 I. Evaluation Purpose The goal of this evaluation is to improve U.S. Agency for International Development Office of U.S. Foreign Disaster Assistance (USAID/OFDA)’s understanding of the performance of the Incident Command System (ICS) program supported by USAID/OFDA in the East Asia Pacific (EAP) region. The evaluation focuses on the effectiveness and sustainability of the USAID/OFDA funded ICS programs in Thailand, as well as in Indonesia and the Philippines. The findings of this evaluation will inform future programming decisions and adjustments to ongoing ICS programming. After five years of supporting the ICS programming in a variety of regions, USAID/OFDA seeks a deeper understanding of the successes and areas for improvement in this capacity-building model. An evaluation of the performance of the ICS program in FY 2017 will allow USAID/OFDA decision-makers to apply the information from this evaluation in future programming decisions related to DRR programming and the ICS program model specifically. With more than $4 million invested in the ICS program over the last five years, the program is a significant investment in resources from USAID/OFDA. As with many capacity-building initiatives, contextual factors can influence the results from the capacity intervention. The introduction of ICS into a variety of countries—all with their own disaster risks, government structures, and socio-economic factors—has led USAID/OFDA to seek more information about the factors for success and the potential for sustainability. II. Evaluation Questions This evaluation has two objectives: understanding effectiveness and sustainability. Each objective has several lines of inquiry that shall inform the evaluation design. Data was collected for all lines of inquiry in the Philippines, Indonesia, and Thailand, and was analyzed to compare results from the different contexts. These objectives and their respective lines of inquiry are listed below. Objective 1: Effectiveness 1. To what extent has the ICS program built the capacity of national governments to respond to disasters effectively, particularly at the initial stages of the emergency? 2. Which aspects of the ICS training program are most effective? Which aspects of the ICS training program are least effective? 3. Is the ICS targeting actors within the national government that are best positioned to effectively implement elements of the ICS system? 4. What country-level factors influence the effectiveness of the ICS program? 5. To what extent do training participants—including the different levels of the cascade training system—retain knowledge and skills transferred through the ICS trainings? Objective 2: Sustainability 6. To what extent are countries able to integrate the ICS into their national frameworks or other institutionalization models? 7. What barriers to utilization of the ICS exist? 8. How much adaptation (translation of core concepts and revision of key elements to better reflect local conditions) of the ICS is required before the materials can be utilized at the local government-level? 9. What factors contribute to sustainability of the ICS? Will existing investments remain viable without external donor support? 2 III. Project Background ICS Program Background Countries in the EAP region face extremely high risks from typhoons (hurricanes), monsoon rains, volcanoes, earthquakes, tsunamis and other natural phenomena. To mitigate these risks, USAID/OFDA has funded a range of disaster risk reduction (DRR) activities in the region. While OFDA’s DRR programming in EAP focuses on context specific activities designed to meet the particular DRR needs of each individual country, host country human and institutional capacity building has been a consistent theme throughout. Developing a capability to respond to disasters more effectively using the ICS has been a key element in all of these efforts. ICS is a standardized incident management approach, applicable to any disaster scenario, which is designed to improve coordination and communication among the various actors and agencies involved in disaster response. ICS was originally developed in the U.S. in the 1970s, following a series of catastrophic southern California fires, when analysis revealed that inadequate management, rather than lack of resources or failure of tactics, was the main reason for ineffective response. ICS was intended to address these management weaknesses, which had caused problems with accountability, communication, planning, management structures, and integration of interagency requirements. ICS comprises a set of basic principles, an organizational structure, and an operational planning process and includes a clear chain of command, common terminology, interoperable communications, standardized training, and consistent certification requirements for disaster response staff. ICS expanded throughout the U.S. in the 1980s and 1990s, and since the early 2000s has become a primary component of the United States’ overall, presidentially mandated National Incident Management System (NIMS), where it is codified in the Command and Management section. More recently, spearheaded by the U.S., the ICS approach has also spread internationally. OFDA’s DRR strategy in EAP focuses on increasing the capacity of NIMS to effectively respond to disasters. As part of this effort, OFDA has funded U.S. Forest Service (USFS) ICS capacity building activities across the EAP, providing training in Brunei, Burma, Indonesia, the Philippines, Thailand, Vietnam, Mongolia, the Solomon Islands, Timor Leste and Vanuatu, and supported U.S. study tours and regional activities. Thailand Country Program Background Thailand is exposed to disasters, primarily from hydro-meteorological events such floods, landslides, storms, and droughts. Floods present the greatest hazard, with the Great Flood of 2011 affecting 64 out of 77 provinces, over 5 million households, and costing 1.44 billion Baht (USD$45.7 billion) and over 1,000 lives. Thailand is affected by earthquakes and tsunamis, with over 8,000 people either dead or missing after the 2004 Boxing Day Tsunami. Thailand is also vulnerable to fires, haze, and man-made hazards like industrial chemical spills. Thailand’s Department of Disaster Prevention and Mitigation (DDPM) was established in 2003 as the lead government agency for disaster management and coordination. DDPM was introduced to ICS during Phase 1 of the ASEAN (Association of Southeast Asian Nations)-US Disaster Management Cooperation Program, conducted between 2004 and 2006. DDPM received further basic-level ICS training in 2009 from the Washington State National Guard through the Department of Defense State Partnership Program. Following this training, DDPM decided to incorporate ICS into Thailand’s disaster management system and established an ICS Working Committee that developed the conceptual framework for ICS integration. Framework components include the development of course materials and broad ICS training for responders and decision-makers, development of manuals, guidelines and standard operating procedures, adjustment of relevant laws and regulations, and selection of pilot 3 project areas/provinces to experiment and adjust the system based on lessons learned. DDPM also developed an ICS training-of-trainers (TOT) cadre, selecting the top 20 participants from prior ICS training sessions. DDPM contacted the Forest Service in April 2010 to request assistance in ICS implementation, including providing training to the TOT cadre and technical support in the adaptation and implementation process. The DDPM Director General submitted a request to the ASEAN Secretariat to become a third Pilot Country under the ASEAN-US Disaster Management Cooperation Program, and the request was approved by the ASEAN Committee on Disaster Management ICS Task Force on June 23, 2010. The U.S. Forest Service (USFS) began working with DDPM in 2010 through the ASEAN – U.S. Disaster Management Cooperation Program, which aims to enhance ASEAN’s capacity for disaster management through integration of ICS into regional and national disaster management systems. In FY 2011-2013 USAID/OFDA provided funding for a series of ICS TOT courses for the 1st Generation of Master Trainers (MTs). In FY 2013-2014, USAID/OFDA provided funding to USFS to provide training to support a 2nd Generation of MTs. DDPM and the MTs have subsequently cascaded ICS training to operational responders in a variety of agencies and regions, and have applied ICS during the conduct of tabletop and full-scale multi-agency exercises. Starting in 2012, USFS assisted in selecting and initiating pilot programs in six provinces: Phuket, Songkhla, Khon Kaen, Ayutthaya, Rayong, and Chiang Mai. In 2014 and 2015, USAID/OFDA funded USFS for the training and development of a 3rd Generation cadre to serve as Thailand Incident Management Assistance Team (THAIMAT) members and facilitated a U.S. based study visit to support the THAIMAT program. The THAIMATs are intended to be field deployable teams that utilize ICS to support Provincial level disaster response and improve coordination with the national government. DDPM has continued to improve their disaster management systems in 2016-17, with priority focus on the THAIMAT (development of a 4th Generation cadre), national Emergency Operations Center (including developing Emergency Support Functions) and a national exercise program. From FY2011-FY2016, USAID/OFDA provided $910,0001 to support the implementation of ICS in Thailand. The funding supported various program activities, including the 1st and 2nd Generation MT Cadres, THAIMAT development, establishment of an ICS Working Committee, the development of a training plan and the adaptation of course materials, the completion of ICS train-the-trainer courses, monitoring initial course offerings, supporting DDPM in implementing new systems, the completion of a US study tour, and the completion of program evaluation workshops. Concurrently and in concert with these efforts, the Royal Thai Government (RTG) has undertaken initiatives to create “disaster immunity” by improving its ability to manage disasters. The Eleventh National Economic and Social Development Plan (2012) and the National Preparedness Strategy (2014) aimed to further this goal by encouraging inclusive participation and synergizing stakeholder activities for comprehensive disaster management. The National Disaster Risk Management Plan (2015), developed under the Disaster Prevention and Mitigation Act B.E. 2550 (2007) supplements the National Disaster Prevention and Mitigation Plan (2010-2014). It provides descriptions of roles and responsibilities across disaster response stakeholders, a reference of applicable laws, and top-level guidance for using ICS in response operations. ICS development is integrated in Thailand’s 20-year National Strategic Plan approved by the Cabinet in 2016, as part of its disaster management strategy. 1 Source: Program documents for USFS contracts/Statements of Work. 4 IV. Evaluation Methods & Limitations Qualitative Approach The evaluation team applied a purposive approach in collecting data in the field to get the most comprehensive data available from the sources. Specifically, the team identified individuals from DDPM, OFDA, USFS, and partner organizations (; e.g., Thai Red Cross, National Institute for Emergency Medicine (NIEM)) who have in-depth knowledge of the ICS program in the country particularly during USAID’s involvement from FY 2011. Because there was a targeted group of stakeholders identified, the most appropriate technique to collect data was through qualitative methods such as key informant interviews and group discussions using guided interview instruments and discussion agendas (see Annexes II and III). The interviews and group discussions were supplemented by a review of country program documents, such as time-bound (; e.g., annual and quarterly) reports, DRR assessments, and annual plans. The evaluation team also visited multiple locations: • Chulchomklao Royal Military Academy in Nakorn Naiyok, where they observed the 4th Generation THAIMAT course that was being conducted in the facility • Area Command (AC) in Surat Thani, where they observed response to the flood situation affecting southern provinces, • DDPM in Phuket, a former pilot province, and • DDPM in Ayutthaya, a current pilot province. The complete list of interview and group discussion participants and a list of documents reviewed are provided in Annex IV. A main limitation of qualitative methods is the inability of the approach to state generalized findings that can be objectively or empirically measured. This limitation was mitigated in the evaluation design by consistently collecting data along all lines of inquiry as outlined in the evaluation SOW. For example, to explore the effects of country-level factors on evaluating the effectiveness of the program (Objective 1), the team gathered feedback regarding this factor from all respondents. The evaluation team anticipated some variation on the information provided, where a local national would expectedly have more depth in the feedback compared to an expat or foreign national in the example for the inquiry on country-level factors. But consistently applying the lines of inquiry across all respondents was the only way for the evaluation team to draw significant outcomes or identify patterns in responses that would allow the team to conclude that findings are valid, reflective of actual conditions, and that the information gathered directly answers the specific evaluation question. Capacity Building Index The consistent application of the lines of inquiry also allowed the evaluation team to construct the ICS capacity building index, as proposed in the evaluation work plan. The team developed the index by examining a series of factors or elements that lead to the effectiveness and sustainability of the Indonesia ICS program, and rating each factor along a 1-4 scale with the following working definitions: 1. This element is Present but in a condition that contributes only to a Very Limited extent in achieving its intended purpose. Considerable External Assistance is required to develop or improve this function. 2. This element is Present but in a condition that makes a minimum, but somewhat Adequate contribution to achieving its intended purpose. Some External Assistance is required relating to this function. 5 3. This element is Present and in a condition that makes a Generally Satisfactory contribution to achieving its intended purpose. Some External Assistance is required to ensure that the purpose intended is achieved and sustained. 4. This element is Present and in a condition that makes a Clearly Satisfactory contribution to achieving its intended purpose. No External Assistance is required to ensure that the intended purpose is achieved and/or sustained. Results of the index are presented in more detail in Section VI: Conclusions. Additional limitations include: 1. Attrition. Some past trainees might have been reassigned or might have left their units during the time of their ICS training. This limited the evaluation team’s ability to contact them for interviews or surveys. 2. Country selection. The three countries were selected by OFDA beforehand and thus the context when reporting findings is only in relation to these countries, individually or collectively. Findings will not be representative of the program in the entire EAP region. 3. Hawthorne bias. This effect is the tendency of a respondent to provide information based on their awareness of being observed or that an evaluation is being conducted. This effect often occurs in externally-funded assistance programs and beneficiaries respond with the intent of not being critical or with the expectation of receiving further support rather than providing candid assessments of the program. The evaluation team has experience in identifying this effect during data collection when trends in responses become evident. The team applied proven social science techniques (; e.g., probes) to mitigate this effect. Where possible, the team also cross￾referenced possible biased responses with other sources of data such as trainee records as a confirmation measure. As much as possible, the team used discrete and observable data when doing this triangulation exercise and not a different set of information from another respondent to avoid the “he said, she said” dilemma that can happen in KIIs and FGDs. Program Record Limitation The evaluation team was unable to access program records or course lists of participants in cascaded training due to lack of record keeping for these trainings in DDPM. Because there was no population from which to draw a representative sample, we were unable to deploy a quantitative survey method that would have yielded additional insight into retained knowledge. The team substituted a qualitative method of engaging interviewees on perceived ability to retain and apply knowledge, as well as barriers to retaining knowledge and recommendations for improving trainee’s ability to retain and utilize the skills learned during ICS training. Respondent Groups The respondents that provided information and feedback for this evaluation can be categorized into five groups. The first group includes the operational personnel of the government agencies that are implementing ICS. This group consists of trainee participants who work for the implementing government agency primarily as first responders. The second group includes higher-level management and policy personnel of the implementing government agencies, such as directors or chiefs of disaster management agencies. While they may not necessarily have received ICS training, they are overseeing the implementation of the country program within their organization. The third group consists of individual from government and non-governmental organizations that are working with the government implementers in the integration of ICS. The fourth group includes U.S. government personnel that work in the ICS program as advisors and/or trainees, such as the USFS disaster management specialists. Lastly, the fifth group includes the USAID/OFDA Regional Advisers, who represent USAID as the funding agency. 6 7 V. Findings The findings, conclusions, and recommendations of this country evaluation report and the larger evaluation of the ICS EAP program needs to be guided by an analytical framework so that the information obtained will be evidence-based and utilization-focused to inform future programming decisions. The analytical framework the evaluation team applied is illustrated in Figure 1, which shows the hypothesized logic model of the ICS program along the following levels: Objectives Increased Capacity Inputs Training Outputs Trainees/ Cadres Outcomes Expansion Figure 1: ICS Program Logic Model • Inputs. The program’s main input is the design and delivery of the ICS training courses: basic, intermediate, position courses, and advanced courses. For a trainee to be eligible to become a MT, s/he should also complete the Training for Instructors (TFI) course. • Outputs. The delivery and completion of courses results in the increase in the number of trained personnel within the partner nation government agencies, as well as some private sector emergency responders. The advancement of some participants to become a MT also creates a cadre of future ICS trainers in the country. • Outcomes. Over time, the increase in the number of additional trainees as well as the advancement of earlier trainees lead to the expansion of the application of the ICS practices among the community of emergency responders. Expansion is accelerated at this level through the Training-of-Trainers (TOT) program where MTs lead ICS trainings. Spillover effects can also be expected as trained personnel apply lessons learned back at their respective units and through interactions with non-ICS trained colleagues. • Objectives. Ultimately, the ICS program’s objective is to increase the capacity to respond to disasters using the ICS model. While USAID/OFDA defines capacity as the ability to carry out the stated program objectives “to effectively and efficiently respond to disasters using the ICS model,” the understanding of capacity needs to be contextualized for each country in the broader evaluation because of differences among the three 8 countries in local conditions. USAID/OFDA also recognizes that “the objectives of each of the iterations of ICS programming vary depending on the disaster profile and baseline capacity of the country in which the program is implemented.” Thus, capacity as understood for the Thailand program will be different from how it is understood in the Philippines and Indonesia for the broader evaluation. Capacity For Thailand, the evaluation team defined capacity in three dimensions. First, ICS has been used successfully to manage events, either planned or no-notice. The second dimension of capacity is the adoption and use of ICS by stakeholder agencies. The third dimension of ICS capacity is that formal training programs are in place and meet the needs of stakeholder agencies. Underlying these dimensions is the development, use, and retention of knowledge and skills within ICS. Training programs build the skills, but without a framework for successful application, knowledge and skills alone will not increase capacity to respond to events. If the framework has been built, successful use (first dimension) will be evident, as will adoption and use by stakeholders (second dimension). We see this as particularly critical for capacity, as an emergency management organization does not respond to incidents alone; they primarily organize resources, which usually come from other agencies, departments, and organizations. If ICS capacity is restricted to the emergency management agency, outside organizations’ ability to integrate into response will be limited, and capacity will remain low. The causal pathway that leads to the program objectives of the increased capacity to respond to disasters using the ICS model also needs to control for other factors not directly related to the inputs of training delivery but that nonetheless influence program objectives. In Figure 1, these are identified as external factors such as governance frameworks and policies at the national or local levels. In the Philippines, for example, the Local Government Code decentralized governance and empowered different local government units. The code led to the increased responsibility and accountability in operations, including disaster response. Another external factor to consider is the presence of other disaster relief mechanisms or programs that might have confounding effects on the ICS program in the country. For example, ICS-related training provided by other countries or NGOs may complement in￾country efforts or introduce non-standardized instruction that limits effectiveness. These effects have to be taken into consideration, if not controlled for, in an evaluation to have more precise assessments of the effects of the direct interventions. Findings Objective 1: Effectiveness 1. To what extent has the ICS program built the capacity of national governments to respond to disasters effectively, particularly at the initial stages of the emergency? As stated earlier, capacity is defined along the three dimensions of: usage at national, provincial, and district levels, stakeholder adoption and use, and effective training programs. ICS trainings have been conducted regularly at DDPM and province levels, as well as within partner agencies, so there has been a steady expansion of ICS usage within the RTG and its partners, largely within DDPM, but also among other agencies. However, challenges were reported broadly in cascading the trainings, mainly because the MT Cadre is insufficient to meet the needs of all the provinces and partner agencies. The program has been cascaded to multiple levels of DDPM, provinces, and within stakeholder agencies. DDPM has not kept a formal record of all trainings provided or a database/registry of trainees, so an accurate total or a reliable estimate of the number or trainings or trainees is not available; however, partners provided snapshots that provide an idea of the level of capacity being built: 9 • 60 attended intensive MT courses (1st and 2nd Generation), with 58 receiving some level of certification as a MT, either in all subjects or in some subjects. An estimated 18 MTs are currently active as trainers. • DDPM has certified 53 IMAT-qualified personnel, some from the MT Cadre and some from the 3rd Generation IMAT cohort, which trained 30. In addition, 30 more are currently in the 4th Generation IMAT training cohort. • DDPM estimates that 500-600 of their staff have received some level of ICS training. • NIEM provides Disaster Medical Assistance Teams with ICS-integrated training annually, including for public foundations. In 2014, NIEM provided training to three (3) provinces, 100 participants each (300 total participants); in 2015, 4 provinces (400 total); in 2016, 9 provinces (900 total trained). • Ayutthaya province’s DDPM office provides Basic/Intermediate ICS and abbreviated courses to its local Volunteer Association, local authorities, and those who work on the ground as response teams during incidents as part of the pilot program they have run since 2012. In addition, all courses in Ayutthaya’s Disaster Management program have ICS integrated, even if they are not formal ICS courses. Their database of all participants in sponsored training currently numbers 400-500 people. Although an accurate total of those trained in ICS in Thailand is unavailable, the information provided above gives a minimum of 2,500 trained countrywide in both disaster and public health sectors. The actual number is higher, as this estimate does not reflect all cascaded training, which was not tracked. Although ICS is gaining traction in Thailand, interviews indicated that full acceptance is limited to DDPM and select partners (; e.g., military and public health agencies). Outside of DDPM, organizations have limited knowledge of the Emergency Support Function (ESF) construct, and do not necessarily perform their roles as laid out in ESFs or in line with ICS. There was consensus among those interviewed that more training is needed at the provincial levels and across partner organizations. The study team was able to interview multiple THAIMAT staff members deployed to Surat Thani as part of the first major emergency response using the IMAT concept (see text box below for more information). The primary perceived improvements the THAIMAT staff identified resulting from ICS implementation are: (1) having one commander improves clarity, (2) clear roles and responsibilities for response agencies reduce duplication of effort, and (3) a more systematic and organized way to allocate human and other resources expedites response. Before ICS adoption, response management was cited as chaotic and confusing due to unclear chain of command, no coordinated planning, and no systematic way to manage resources. One example presented reflected on a flood response in Ayutthaya (prior to ICS), during which many organizations offered help, but “piled up” at the incidents, not knowing what to do next. The improvements identified by the THAIMAT staff were consistent with benefits described by others who had implemented ICS in Thailand, including practitioners in Ayutthaya and Songkhla. Case Study: THAIMAT Deployment to Flooding in South Thailand In November 2016, flooding incidents began in three to four Southern provinces in Thailand, constituting a Level 2 urgency (multiple provinces affected). Response was managed at the provincial level by Governors supported by their provinces’ DDPMs. The floods temporarily alleviated, and operations returned to normal. Then, on December 1, 2016, the situation worsened. Provincial authorities reported increased severity and requested assistance from DDPM Bangkok. Following that response, the situation then improved by the end of December. On January 1, 2017, the situation again worsened due to heavy rainfall, causing floods in 14 provinces that was beyond the capabilities of provincial authorities to address. On January 5, RTG announced an increase of the level of urgency to Level 3, and ordered the establishment of an Area Command in Surat Thani, with the objective to 10 coordinate between Area Command in Bangkok and affected southern provinces. This Area Command was responsible for 12 of the affected provinces. The Incident Commander named for this response was the Minister of the Interior, located in Bangkok. The Deputy Permanent Secretary of the Ministry of the Interior was appointed as the Area Commander, and was located in Surat Thani. All THAIMAT staff are civil servants from DDPM, and were pulled from their regular positions in different provinces across Thailand. They had all received ICS 100, 200, and 300 courses, as well as IMAT 1. They participated in national exercises in May 2016 (Bangkok) and July 2016 (Lopburi). They had planned to participate in another exercise in December 2016, but the flooding situation required that exercise be canceled. The Area Command was staffed by 15 total THAIMAT staff, which allowed for approximately five staff per section. Every day, DDPM conducted a 10:00am teleconference meeting out of DDPM Bangkok, with the 12 affected provinces. Attendees at the Area Command included but were not limited to: Deputy IC – Vice Governor Surat Thani, Director of DDPM, Surat Thani, Irrigation Agency of Thailand (ESF – Recovery), Police (ESF), and Social Welfare Agency (ESF), which was collecting data on effects to population. Provinces updated DDPM Bangkok on their status, while DDPM Bangkok requested statistics and gave updates on the funding being released from the Office of the Prime Minister. Media was present at the Area Command in Surat Thani so they could receive updates concurrently. While a full after action review is in progress and not available for this report, the evaluation team was able to interview members of the deployed THAIMAT team as well as DDPM Directors from some of the affected provinces. The data gained from these interviews was used throughout this report to gain insights on most and least effective aspects of the program and training, sustainability, as well as to develop recommendations for the way forward. One local provincial DDPM Director mentioned that while the province uses ICS in their plan, they hadn’t yet had an opportunity to use it full scale because they had not had emergency situations that would warrant ICS usage. In their province, one team member had received training in ICS and had been training others and using the knowledge gained to set up provincial response processes and structures. The Director noted that ICS offers a more systematic response and has been helpful; furthermore, there is support for its continued use and expansion within the province. Another provincial DDPM Director commented that the use of one incident commander and one line of command helped expedite ordering and mustering of supplies. He also noted that ICS helped facilitate inclusion of academic resources, such as weather forecasting. While there were not many ESF managers deployed to the Area Command, requiring THAIMAT staff to act as ESF managers in their stead, one deployed manager felt it was a good decision to have an Area Command closer to the affected area because the area was so vast, and the closer proximity was helpful for coordination. There were some initial hiccups in coordination between the representatives in Surat Thani and those in Bangkok due to separate reporting, but eventually the reporting protocols were aligned and became more efficient. Another ESF Manager felt there was more work to be done as much of the reporting from provinces was going to DDPM Bangkok instead of to the Area Command in Surat Thani. He felt roles, lines of authority, and responsibilities were still unclear. Notably, just prior to the short interview, the evaluation team overheard someone explaining to this ESF Manager what ICS is, so his level of knowledge on the system’s benefits or disadvantages is unclear. However, his lack of awareness of ICS despite participating in a leadership capacity during a major THAIMAT deployment demonstrates that there is still work to be done by DDPM Bangkok to raise awareness levels countrywide. At the time of the interviews at Surat Thani, the team had been deployed for over one month. While they had anticipated demobilization shortly thereafter, another round of storms had recently been forecast and flooding was anticipated to continue, along with sustained Area Command operations. 11 2. Which aspects of the ICS training program are most effective? Which aspects of the ICS training program are least effective? A common theme around what makes training effective is the ability to apply the knowledge to real world events. The practical application exercises in training were cited as helpful in course reports describing participant feedback, and this perception was also reflected by IMAT training participants, deployed THAIMAT personnel, and other informants. However, practitioners in the Surat Thani Area Command and in provincial DDPMs indicated they understood the process best once they had responded to an actual event. Several ICS concepts were cited as the most effective aspects of ICS: • Management by Objectives provides clear guidelines of management, priorities (life safety, strategy, effective use of resources) and a systematic process of thinking and implementing (i.e. map objective to tactic and assign appropriate equipment). • Check-in System allows IMAT staff to monitor resource use and availability. • Interoperable Communication allows for more coordinated communication and sharing of information among relevant organizations and more effective use of communication equipment and resources to share information, with greater accuracy. • Joint Information Center fosters accurate information-sharing among relevant stakeholders and media. Few areas of weakness were identified in the current USFS-based training curriculum outside of instructor-specific feedback. Focus group discussions with training participants revealed few perceived weaknesses in the training provided; it is unclear if this was (1) a cultural hesitation to provide negative feedback, (2) a language barrier in which they perceived the question to address ICS itself rather than the training, (3) an indication that there were few if any shortfalls in training, or (4) if the training audience was not knowledgeable enough to identify areas of weakness. Following deployment to Surat Thani, however, THAIMAT participants suggested that instructors should focus more time on practical exercises in order to fully prepare future IMAT staff. In addition, there is a demonstrated need more trainers, as only approximately 18 remain as trainers out of the original pool of 60 who entered training as part of the 1st and 2nd Generation MT Cadres. The rest have retired, been promoted, or no longer provide training due to other job requirements. 3. Is the ICS targeting actors within the national government that are best positioned to effectively implement elements of the ICS system? The concept of “targeting actors” within the national government should be understood in two perspectives important for countries to effectively implement elements of the ICS system. First, from an operational perspective, the program should target the actors who are responsible for responding to disasters from the initial stages to national-level response. In Thailand, this includes DDPM at national and provincial levels, local government units, first responders (; e.g., fire and police), ministries with roles in disaster response, and partner organizations such as volunteer organizations. Second, from a policy perspective, the program should target actors who are positioned to build an effective ICS program, which includes influencing leadership, creating buy-in across all levels of government, and developing laws, regulations, policies, and procedures that guide ICS implementation from both a training and operational perspective. The program has primarily targeted actors within DDPM, but also included key response partners. In the 1st Generation MT Cadre, most of the trainees came from DDPM, but representatives from Public Sector Development Group, Department of Energy, Bureau of Health Administration, Bangkok Fire and 12 Rescue Department, NIEM, Thai Red Cross, Royal Thai Police, and Thailand Institute of Nuclear Technology were also included. A similar approach was followed in subsequent generations. As a result, DDPM has built a MT Cadre capable of cascading training not only to DDPM but also within key partners, and has fostered interagency understanding of ICS benefits to a limited extent. The approach has created “champions” in DDPM and partner organizations who have cascaded training; for example in Ayutthaya and within NIEM. Some of the 1st Generation MT Cadre have conducted 50-100 training sessions each, building capacity at national and provincial levels. An estimated 500-600 DDPM Officers at provincial level emergency operations centers (EOCs) have received EOC-focused training from the MT Cadre. Some organizations, such as DDPM Bangkok, DDPM Ayutthaya, and NIEM, have targeted local authorities and actors for basic ICS training. However, ability to provide enough training at the first￾responder level is limited by the small number of available trainers. Experienced DDPM personnel and recently deployed THAIMAT staff concurred that full implementation is also hindered by ESF providers’ limited knowledge of their roles in ICS, or a fundamental understanding of how ICS works. 4. What country-level factors influence the effectiveness of the ICS program? Key informants with high-level program knowledge indicated the main country level factors influencing effectiveness of the ICS program in Thailand are: (1) executive level authority due to the disaster management law that codifies ICS; (2) DDPM presence in provincial offices; (3) perception that ICS adoption is low-risk. These factors stem from the centralized nature of Thai government. Thailand has taken a top-down approach in integrating ICS. Thailand is institutionalizing ICS by integrating it into the National Master Plan on Disaster Management 2015. Although DDPM does not have enough MTs to train other government agencies to fully implement/integrate ICS, there is support at high levels, evidenced by ICS influencers’ ability to make ICS usage required by law. There is good political will, an available budget for ICS, and there is a policy in place to support ICS implementation. Although there is little resistance, low levels of awareness and knowledge of ICS among partner agencies and local government officials impede full institutionalization. The system has been able to work to the extent that it does because the DDPM has provincial offices. These provincial offices are key factors for developing awareness, knowledge of, and implementation of ICS concepts. Many of the gains Thailand has made stem from ICS being well received. Although it hasn’t been used extensively, the times it has been used (; e.g., in Ayutthaya at the local level, and during 2010 oil spill and 2017 flooding at the national level, discussed below) have been relatively successful. Key informants indicated that if ICS were perceived as a risk, officials would not do it; however through a few exposures, Thai leaders at the national and provincial levels realize that ICS can make them look professional and competent. There have been three foundational incidents that have contributed to this perception. In 2010, Thailand deployed a small team of MTs to help respond to an oil spill. The governor of the affected province let them lead the planning cell, and was impressed with how they did. This was a turning point, as prior to the oil spill response, there were only about 80 “believers,” and this number expanded with increased awareness following the effective response. In 2011, however, during a major flood response the new Prime Minister (who was not aware of or comfortable with NIMS/ICS) opted to place trusted military advisors in lead roles and went back to old procedures instead of using the ICS￾trained cadre (approximately 80 people) and the ICS-based procedures DDPM had developed. Dissatisfaction with that response resulted in lawsuits being filed against the Prime Minister and relevant public organizations for failing to perform according to the National Master Plan on Disaster Management. This ended up hastening acceptance of ICS, as the “old way” now posed a greater risk. In early 2017, concurrent with field work on this evaluation, Thailand undertook the first large-scale deployment of ICS to respond to flooding in southern provinces. This included setup of an Area Command in Surat Thani and a smaller Incident Command Post in Songkhla province in addition to the standard Area Command at DDPM headquarters in Bangkok. While After Action Reviews (AARs) are 13 still ongoing, early reports indicate that the success of this flood response will further accelerate Thailand’s acceptance and implementation of ICS. 5. To what extent do training participants—including the different levels of the cascade training system—retain knowledge and skills transferred through the ICS trainings? For practical purposes, there are three categories of training recipients in Thailand. First are participants in cadre/cohort training that is intensive and generation-based (1st and 2nd Generation MTs, then 3rd and 4th Generation IMAT trainees). The second category is the recipients of the formal ICS courses provided primarily to DDPM Provincial representatives, but also other partners, through annual DDPM courses (now being transferred to DDPM Academy). The third category is recipients of cascaded training through both formal (as in Ayutthaya or through MTs delivering official curriculum) and informal (; e.g., one DDPM provincial staff member teaches colleagues what was learned in ICS 100/200, or MTs deliver abbreviated courses). Levels of knowledge retention will be different across these categories, with retention highest in the first category as the quality and intensity of courses is highest. Currently, Thailand does not track ICS training participants in a centralized database, or award certifications beyond general certificates of participation. This makes any systematic attempt to measure knowledge retention quantitatively impossible, because there is no representative population from which to pull a statistically meaningful sample. Similarly, levels of knowledge retention cannot currently be compared across categories. Complicating this measurement further, the third category of trainees primarily receive non-standardized instruction, so any attempt to measure knowledge retention would be hampered by lack of a baseline of understanding for what was in the original course of instruction. While MT Cadre members may have cascaded ICS 100/200 curriculum as it was taught to them in some courses, there are also reports that they and others have delivered a “Basic” ICS course of 3-4 hours. This terminology further hampers measurement because the term “Basic” corresponds to a specific ICS course (ICS 100) that is generally taught over 3-5 days. The use of the term “Basic” interchangeably between these very different levels of instruction impedes the formation of a complete understanding of how much knowledge was transferred at a given course. With this understanding, the evaluation team pursued a qualitative approach through which training recipients performed self-assessments on how well they retained knowledge, and what factors contributed to this retention. Provincial- and national-level DDPM leaders were also asked if trainees retained knowledge, and if failures to do so critically hampered response activities. Multiple THAIMAT Focus Groups as well as individually interviewed staff at the Surat Thani Area Command and a pilot province2 reported no significant gaps in knowledge that prevented efficient operations. They agreed that the most effective way to retain knowledge gained from ICS trainings is the continued practical application of what was learned. The application can only be done in three scenarios: in actual incidents, planned events, and exercises. THAIMAT staff at the Area Command in Surat Thani indicated that the knowledge obtained from the trainings was reinforced by its extensive application to the real-world flooding situation in January-February 2017. They were also able to access course materials and handbooks to refresh knowledge on an as-needed basis. Interviews in Ayutthaya reinforced these findings. Provincial and volunteer organization leaders perceived that knowledge is retained well in Ayutthaya’s disaster response organizations because there are frequent incidents (flooding and industrial responses, in particular), hence many opportunities to practice. For the Ayutthaya Volunteer Association, the knowledge is reportedly retained well, as there are real incidents that require the use of ICS integrated disaster management plans on a frequent basis. However, if volunteers have not been regularly deployed in the field response team, the Association requests they participate in exercises conducted by DDPM. In 2016, the DDPM provincial office conducted two exercises. 2 The province is not identified to maintain the confidentiality of the persons who provided the feedback. 14 Because Thailand does not experience major disasters as frequently as some of its regional partners (; e.g., Philippines and Indonesia), opportunities to apply ICS at the national level3 and retain knowledge through that mechanism are limited without using ICS to manage planned events or instituting an exercise program. DDPM is planning to ask USAID support for a refresher course and MT course, in order to support the DDPM 5-year plan (which is currently being drafted). Thailand has institutionalized ICS into its National Plan, but DDPM leadership recognizes the need to implement it on wider scale to retain the knowledge. This is in part the focus of training modules of DDPM Academy. Focus group participants at IMAT training and in the deployed THAIMAT also emphasized the benefits of international site visits and a formal lessons learned program that would allow them to share best practices across countries. Importantly, the evaluation team found that while retaining knowledge on the individual level is important, another key consideration is the ability of organizations to retain knowledge. This was evidenced by the lack of a functioning ICS program in Phuket in 2017, even though it had been selected as a pilot province in May 2012. Key ICS influencers in DDPM Phuket had been reassigned and departed the province, taking the ICS knowledge and advocacy needed to develop a robust program with them. Objective 2: Sustainability 6. To what extent are countries able to integrate the ICS into their national frameworks or other institutionalization models? Thailand has integrated ICS into its top level guidance with the publication of the National Disaster Risk Management Plan (2015) (NDRMP). Upon its approval by the Cabinet on 31 March, 2015, “all relevant agencies are required to use this plan as a blueprint, framework and guideline in handling national disaster management actions.” ICS development is also integrated in Thailand’s 20-year National Strategic Plan approved by the Cabinet in 2016, as part of its disaster management strategy. DDPM has further institutionalized these processes by publishing a National Incident Management System guidebook that places ICS in the context of how Thailand responds to incidents, and an ICS planning handbook, which serves as a reference guide for ICS users in the field. It includes guidance for operating an Incident Command Post (ICP), as well as translated ICS forms adapted to the Thai context. Operationally, the extent of ICS integration is best demonstrated by its use in the 2017 flood emergency, where 16 out of 18 ESFs were active. After action reviews are being conducted by DDPM and each ESF to assess the level of success of those operations. Evaluation team research indicated that while the response was overall a success, ESF organizations outside of a few key partners (; e.g., armed forces, public health) are not fully knowledgeable on ICS in terms of structures, processes, forms, and how to integrate into the structure. These agencies will need to receive more training and guidance to fully institutionalize ICS. The deployed THAIMAT team reported that their scope of work expanded dramatically due to partner unfamiliarity with ICS and the ESF framework. Instead of having ESF representatives coordinating resources, THAIMAT team members had to act as ESF leads for some functions, creating a significant drain on their staff. In addition, they were unsure of where to access the information required for situational awareness and resource identification, indicating a further need for institutionalizing the processes required for sharing the information needed to make ICS work. Personnel working in other provinces confirmed that partners’ limited ESF knowledge significantly inhibited ICS operations, and will need to improve for ICS to be fully institutionalized. Of the 77 provinces, only six were selected for pilot programs, and even within these, capacity built varies. The rest will need expanded awareness and access to training in order to bring them to the level where they can implement the program effectively. Although DDPM offices in provinces have helped expand ICS operational capability, ability to actually implement ICS is not standardized across regions or 3 Local and regional responses, which are more common in Thailand, do provide opportunities for district and province levels to apply ICS, but frequency differs based on location. 15 provinces, and in most cases has not reached the district level. Notably, the legal framework that requires ICS usage at national and provincial levels does not mandate that local authorities adopt ICS or provide training. DDPM and ICS proponents are working to fix this issue in order to further institutionalize the system. 7. What barriers to utilization of the ICS exist? The primary obstacle to ICS utilization in Thailand is training capacity. From the first two generations of MTs, there are currently only 15-20 active as instructors. While many of the MT Cadre have moved into positions where they have positively affected the program (; e.g., helping draft the NDRMP), the degradation of training capability leaves a significant gap in Thailand’s ability to use ICS nationwide. As a result, ICS concepts remain not well understood by all relevant stakeholders. Apart from public health agencies and the military, other agencies involved in responding to disasters (; e.g., Ministry of Interior, Ministry of Transport, Royal Irrigation Department, Thai Red Cross) have limited understanding. Relatedly, there is a need for more IMAT training in order to support the existing team, which is limited, in times of emergency. There are currently 53 trained personnel authorized by the Head of DDPM to work as IMAT staff. During the flooding emergency in 2017, these individuals were divided into: (i) Area Command in Bangkok, (ii) lead trainers in Nakhon Nayok for the IMAT training, and (iii) Area Command in Surat Thani. Those who work in the affected provinces are not used, as they need to remain in place to lead responses there. IMAT staff deployed to Surat Thani were based in the field for over one month without a chance to visit their families. There was only enough staff for one operational period per day, so the entire staff was active from early morning until late evening hours, which staff reported was a grueling schedule. While these conditions are not unusual in disaster response operations worldwide, insufficient training and staffing could further degrade Thailand’s ability to utilize ICS if chronic overwork of staff results in heightened attrition. DDPM staff reported that English language capability was a barrier to Thailand’s continued expansion of the MT Cadre. Although this was reported by only staff member, that person was well positioned to have insight into this information, and was considered credible on the subject. It is well established that USFS instructors represent the gold standard in instruction; however, trainees must understand English well enough to gain value from the instruction. Translated materials assist trainees in following along, but cannot fully compensate for inadequate language understanding. The DDPM staff member reported that previous efforts to provide translators to help close the language barrier had not been successful. Since English capabilities had been a requirement for the 1st and 2nd Generations, DDPM’s best English speakers have already received training, and there are not enough DDPM staff with the requisite language skills to produce another cadre (this is one of the factors that led DDPM to focus on developing IMAT capabilities instead of a 3rd/4th Generation of MTs). DDPM is currently working on identifying and recruiting personnel with English language skills, in order to build another generation of MTs. Integration of budgeting systems has also been a major challenge due to Thai laws and processes. There is no legal framework for consistently and responsively funding interagency disaster response, and in fact there are conflicting definitions of “disaster.” In the relevant Thai law, some authorities and funds are only available for use for natural disasters, not man-made events. DDPM ICS advocates are acting in two ways to overcome this challenge: (1) change the regulations through lobbying, and (2) educate agencies about using exceptions to the legal framework for emergency situations. Challenges also remain with respect to integrating resource providers through the ESF concept. As discussed in the previous line of inquiry, most ESFs do not know their roles in ICS, and, importantly, many of them have no ESF group leader to coordinate resources within their ESF groups. All the different organizations within ESFs have their own bosses and lines of command, so it is challenging to determine who will lead the coordination among these agencies. In addition, most ESFs, except for communication, have their headquarters in Bangkok and do not have a regional director. For example, 16 the Thai military’s services, each have their own bosses. The Director of the Royal Thai Army sits in Bangkok, separate from the Directors of the Royal Thai Navy, Royal Thai Air Force, and other military elements. There is no regional head who is authorized to command all these groups. Therefore it is difficult for IMAT staff to coordinate military resources locally. Additional barriers to utilization of ICS involve culture. A high level official commented that there may be challenges in adopting a systematic way of thinking that is outside of the norm for Thai culture, and this viewpoint was supported by comments made by participants in focus groups. In addition, the provincial government often defaults to what they know, so building trust is an important factor. One source with extensive program knowledge commented that this barrier has to some extent been mitigated by leveraging the credibility of the US Government, which adds a level of trust to the in￾country team. This official noted that there is confidence that when the US Government is there, the team won’t fail. Some of this cultural transformation has been seen with the THAIMAT – mid-level government staff (IMAT members) who would not normally be involved in decision making are presenting data and recommendations to authorities. This is gaining acceptance now as generals and senior leadership recognize that the team can actually help them perform their duties well. 8. How much adaptation (translation of core concepts and revision of key elements to better reflect local conditions) of the ICS is required before the materials can be utilized at the local government-level? All ICS training materials and forms have been translated into Thai language and context, adjusting certain boxes as needed. The major adaptation made for application in the Thai context was the combination of the Logistics and Finance/Admin Sections into one section. This was done to better match Thai budgeting regulations that would have limited the Logistics Section’s ability to act (; e.g., order supplies) without the appropriate finance/admin personnel overseeing. This change to a core ICS concept has been applied to all ICS documentation, handbooks, and forms reviewed by the evaluation team. In addition, the term “Safety Officer” was replaced with “Advisor and Expert” because high-level decision makers perceived the term “Safety Officer” as lower level and referring to a medical team only. In reality, this could apply to more disciplines, ; e.g., engineer in the case of building collapse. Changing the name ensures a higher level of respect for the position. Additional adaptations that may be made could address the ICS term “Command,” which interviewees indicated Thais might find imposing. DDPM is reportedly considering an alternative term, which is “softer,” like management. Thai planners also have considered the importance of developing bookkeeping methods, and tracking qualifications and certification. Thailand does not have such systems yet, but is on the path to developing them in order to have a systematic way to certify the trainers and track participants. IMAT staff in Surat Thani indicated it was difficult for local provincial officials to complete the ICS 201 because they were not trained and the form was new to them. IMAT staff then needed to complete the forms for them. Additional adaptations may be needed to ease local use of forms if widespread training is not provided. In addition, the deployed IMAT staff did not use the Incident Action Plan at the Area Command in Surat Thani because they were responsible for many incidents at the same time in 12 different provinces. Further adaptations for use of forms and planning processes in EOCs vs. ICPs may clarify the right processes depending on the focus of the ICS organization. 9. What factors contribute to sustainability of the ICS? Will existing investments remain viable without external donor support? The evaluation team considers sustainability along several factors that demonstrate: (1) country commitment to funding; (2) institutionalization; and (3) capability built. These three factors underlie each of the benchmarks identified for sustainability-related lines of inquiry, which will be discussed in the conclusions section. 17 Commitment to Funding. Thailand has contributed considerable funding to developing the current ICS training and implementation program. Although USAID/OFDA funded USFS activities, the majority of funding for hosting trainings, institutionalizing the program, and implementing ICS programs has come from the host government and its provincial partners. This funding, authorized by law, is expected to continue regardless of USAID/OFDA inputs. Institutionalization. The Thai ICS program has been institutionalized at the top levels of government, and this is expected to continue even without USAID/OFDA involvement. However; it may not mature to the point that: all provincial governors are aware of and using ICS; ESFs are fully functional; and the program has international credibility. More capability must be built to achieve that level of sustainability, and USFS trainers are still needed to provide high quality instruction to build more MTs. To support growing the program, an ICS Working Committee has been established with 18 representatives from the highest levels of DDPM, including the Deputy Director General. Although it currently consists of only DDPM representatives, they are working on incorporating other agencies. The ICS Working Committee was a key driver in developing the NDRMP. They are also preparing a 5-year roadmap and elevating the ICS Working Committee to include active stakeholders, i.e. Bangkok Metropolitan, Department of National Parks, Wildlife and Plant Conservation, and public health stakeholders. This is expected to foster sustainability despite potential changes in high-level personnel and administrations. Capability Built. USAID/OFDA’s primary output for this program is cadres of MTs. Outcomes in capability improvements are then cascaded from these MT cadres to achieve expansion outcomes, independent from USAID/OFDA involvement. Therefore, when assessing sustainability, the capability built should be viewed in terms of whether the number and quality of MTs is sufficient to meet the needs of the country. Every headquarters-level DDPM representative interviewed commented that that 15-20 MTs (the estimated number of MTs currently available) is insufficient to provide the training needed to fully implement ICS, and province-level DDPM staff confirmed this view. While the Thai MT cadre provides training to provincial DDPM staff and other stakeholders across the country, their experience level and credibility are not to the same standards as the trainers from USFS. Therefore, while they can provide instruction to in-country ICS practitioners, they have not themselves created new MTs without additional USFS personnel, and questions remain on whether they are qualified to do so. Without this capability, any program would lack the ability to sustain itself as it could not replace MTs lost to promotion, attrition, retirement, or other factors. As the program matures, one would expect that increased opportunities to apply ICS in the Thai context will grow MT capabilities. Access to professional development and certification would also contribute to nurturing the current and future MT cadre. 18 VI. Conclusions The evaluation team’s conclusions presented in this section are tied to the findings in Section V through the establishment of benchmarks to reflect an “if-then” linkage. Per the evaluation SOW, the evaluation team collected findings along the evaluation questions or lines of inquiry. With the collected findings, the team assessed current conditions and mapped the information against capacity benchmarks that were developed in the evaluation design stage and submitted to OFDA. This approach responds to the SOW, which specifically states that “the evaluation team must work with the USAID/OFDA and USFS to map benchmarks related to the ICS programming that link to levels of capacity-building. This mapping should help to understand what essential inputs and processes are required at each level to achieve capacity￾building benchmarks in various country contexts.” The evaluation team scored each benchmark on a 1-4 scale to calculate the capacity building index. As an example, the evaluation team collected field data to see if findings showed the usage of ICS/NIMS at various levels of the RTG, perceived ICS effectiveness, and support for implementation for the evaluation question, “To what extent has the ICS program built the capacity of national governments to respond to disasters effectively, particularly at the initial stages of the emergency?” With the collected findings, the team then mapped the three elements against the following benchmarks to assess the level of capacity: • ICS has been used successfully to manage events, either planned or no-notice. • Multiple stakeholder agencies have adopted and used ICS. • Formal training programs are in place and funded. The results of this benchmarking exercise to identify conclusions are discussed below and a summary of the index scoring is provided in Table 1. The rows list the benchmarks identified to evaluate each line of inquiry and the “Score” column is the evaluation team’s ranking of that benchmark using the 1-4 scale. The “Indicator” column lists the indicator number using the list of indicators the evaluation team proposed to OFDA for assessment. The full indicator list is presented in Annex VI. Objective 1: Effectiveness 1. To what extent has the ICS program built the capacity of national governments to respond to disasters effectively, particularly at the initial stages of the emergency? Benchmarks Score Indicator ICS has been used successfully to manage events, either planned or no-notice. 2 8 Multiple stakeholder agencies have adopted and used ICS. 2 7 Formal training programs are in place and funded. 2 6,37 The first benchmark to assess capacity is the successful application of ICS in planned and unplanned events. DDPM had its first widespread ICS application when it set up an Area Command in Surat Thani using the IMAT concept. ICS has been applied at the local level in Ayutthaya and by first responders in the public health/emergency medicine community, but examples of usage are still isolated. Overall in Thailand, there are limited improvements in the initial stages of disaster because training has not cascaded down to the first responders (fire departments, police, etc.). The two main exceptions to this are Ayutthaya Province and NIEM, both of which have provided training at the actual first responder level. 19 The second benchmark is the adoption of ICS across multiple stakeholder agencies. ICS principles have been implemented in DDPM at the national level and in some provinces through the DDPM provincial office, though levels of implementation are uneven. Stakeholder agencies have limited awareness and use of ICS outside of the military and public health sectors, hindering the ability to integrate resources using the ESF approach. The third benchmark is the presence of a formal training program with sufficient funding to ensure continued operations. This is clearly present with ICS courses being provided and the formalization of courses through the DDPM Academy. However, the current pace of course delivery has not met the need at national and provincial levels, and it is unclear if the full suite of ICS courses will be offered at the DDPM Academy or how they will be monitored for effectiveness. Training program effectiveness is also hindered by the lack of a formal tracking and credentialing system for trainers and participants. 2. Which aspects of the ICS training program are most effective? Which aspects of the ICS training program are least effective? Benchmarks Score Indicator Training courses are effective at improving ICS knowledge and organizational capacity. 2 15,16,17 A Lessons Learned program is in place to capture best practices for the country and remedy shortfalls. 2 44 There is a lack of program-level data tracking training effectiveness, making it difficult to judge actual effectiveness of training. Based on trainee perceptions, IMAT training courses have been fairly effective – no significant gaps in knowledge were reported to present serious challenges during deployment to 2017 flood operations. Trainees reported that the most effective parts of training were the practical exercises; however they also felt that more time effort should be dedicated to these sections, as they did not yet fully prepare them for real-world response. The evaluation team was unable to assess the effectiveness of cascaded training due to lack of a centralized database of trainees from which to pull a representative sample. From an overall program perspective, there is a significant need for more training of stakeholder agencies at all levels of government, including district and provincial officials as well as ESF providers. Contributions of current training to organizational capacity, regardless of how effective the actual classroom learning is, will be limited until organizations have a “critical mass” of trained staff who can interface effectively with ICS organizations. More opportunities for training and applying knowledge are needed through exercises, planned, and no-notice events. A lessons learned program is in place and AARs developed, but it is not focused directly on ICS program improvement. This limits DDPM’s ability to monitor effectiveness of training courses and whether ICS is being applied effectively when it is used in exercises or real world events. An AAR is underway for the 2017 flood response, and is expected to yield considerable insights into ICS use during a major disaster. Its findings were not available for this report. 3. Is the ICS targeting actors within the national government that are best positioned to effectively implement elements of the ICS system? Benchmarks Score Indicator ICS course offerings are diverse, include training for Senior Officials as well as Position-Specific sessions. 2 6,20 20 Program Directors have the authority to implement ICS activities at national and local levels. 3 19 Women receive ICS training at comparable levels to men. 3 1,2,3,4,5 Personnel at multiple levels of government and stakeholder agencies have received training. 2 20 DDPM has established course offerings through its MT Cadre, and is now transitioning this responsibility to the DDPM Academy so it will be a formalized program. However, it is not clear which classes they will offer or how they will implement the training. Personnel from all provinces can attend the courses, with significant cost subsidies from DDPM, but there will be limited space in these classes and they are unlikely to meet the nationwide need. More engagement with senior officials at the local, provincial, and national levels is needed; although there is an executive training briefing there was no evidence of a strategy for systematically training senior officials. DDPM officials at the national and local levels have the authority to implement ICS, given by law. In practice, they still need a mechanism for engaging partners meaningfully through the ESF system, and also to gain the trust and understanding of governors and other local officials. Women were represented at all levels of training and have influential roles at DDPM; the Director of Training is female as was the Incident Commander at the Surat Thani Area Command. Trainees perceive a high level of gender parity and opportunity, however a review of class rosters indicates that more men than women are receiving ICS training. For example, the 2nd Generation of MTs (2011) had only five females out of a class of 30 total. The program still needs to work toward building the capacity to provide adequate training to local and national partners. While hundreds of staff at DDPM have received training, there are still significant training needs within both the department and stakeholder agencies. 4. What country-level factors influence the effectiveness of the ICS program? Benchmarks Score Indicator No major cultural barriers to ICS acceptance. 3 33 Acceptance of ICS as a best practice within national government. 3 26 Acceptance of ICS as a best practice within regional / local governments. 2 27 Although there are some cultural barriers, most have been addressed in the adaptation of ICS to the Thai context or can be addressed with additional training. For example, interviewees highlighted that while it is unusual for mid-level officials to assume roles such as “incident commander,” in Thai culture, the approach is gaining traction with increased awareness of the system and examples of successful use. Some of the bureaucratic barriers (; e.g., the funding issues discussed in line of inquiry #7’s findings discussion, and how to get districts to use ICS) will be more difficult to address, but ICS advocates are actively working on solutions to these via lobbying and executive-level training. The national government has accepted ICS as a best practice, codified in their law. The main challenge to full acceptance is lack of awareness, as the centralized Thai government and national policies in place reduce the likelihood of resistance to implementation. Province and local governments have not had much exposure outside of pilot provinces, necessitating increased outreach to improve awareness and acceptance by officials there. 5. To what extent do training participants—including the different levels of the cascade training system—retain knowledge and skills transferred through the ICS trainings? 21 Benchmarks Score Indicator Training recipients retain knowledge of ICS concepts following training. 2 15 In-country Master Trainer Cadre has the knowledge, skills, and experience to provide high quality training. 2 42 Program-level monitoring has not been performed to a level where a full assessment of how well trainees gain and retain knowledge is possible. Anecdotally, THAIMAT staff interviewed at the Surat Thani Area Command indicated they had retained sufficient knowledge to be effective in the 2017 flood operations, and could reference training documents when they needed additional fidelity. Because of the lack of a tracking mechanism for participants, data on how well cascaded training recipients retained knowledge could not be obtained. Outside of the active MT and IMAT cadres, knowledge retention may be hindered by: (1) low contact time, as cascaded courses may include only a 3-4 hour basic ICS orientation, (2) low absorptive capacity if trainees had not been previously exposed to disaster response or ICS principles, and (3) limited ability to use ICS in practical situations, as use of ICS is not widespread throughout the country. Additionally, the evaluation team found that the inability of organizations to retain trained individuals (; e.g., Phuket pilot program example) may be a more impactful gap than gaps in individually retained knowledge. The team of 15-20 active MTs out of the original 60-person 1st and 2nd Generation cadres provides formal ICS training to provincial and other partner agencies, but the current cadre is insufficient to meet all training needs. Current focus on developing IMAT members hinders the country’s ability to produce more MTs. The knowledge and skills have been developed within the realm of curriculum content and training delivery, but they need more practical experience and professional development, either in more response operations or planned events, to be a more effective cadre of responders and trainers. Objective 2: Sustainability 6. To what extent are countries able to integrate the ICS into their national frameworks or other institutionalization models? Benchmarks Score Indicator Laws codifying the use of ICS in disaster management in place. 3 9 Doctrine guiding the integration of ICS with country's disaster management system is developed at the strategic level. 3 9 ICS has been successfully integrated and used at multiple levels of government. 2 8,10,30 For this evaluation, sustainability is understood not only to mean whether the program will continue without continued USAID financial/technical support, but also whether it will be able to mature and reach a point of complete implementation.. Thailand has in place the foundational pieces needed institutionalize the program: ICS is required by law, there is a strong cohort of ICS-trained professionals in influential positions, and they have developed a NDRMP as well as supporting documents that guide implementation. Still, the pieces are not completely in place to fully mature the program. Partner awareness of their obligations under the law is low, and stakeholders outside of DDPM are not yet able to function effectively as ESFs or local partners. ICS has been successfully used in some locations and agencies, but not throughout the national and local governments. 7. What barriers to utilization of the ICS exist? 22 Benchmarks Score Indicator Few reports of pushback to adoption of ICS practices. 3 12,31,32,33 Country has established a committee to further ICS acceptance and use. 3 39 Opportunities exist to train on, practice, and apply ICS regularly. 2 29 There are not many reports of significant pushback to adoption of ICS practices, but widespread acceptance of ICS is hindered by lack of awareness outside of DDPM. This may be tied to the development of an ICS Working Committee that only includes DDPM representation. This shortfall has been noted and DDPM is taking steps to rectify the issue; but lack of external stakeholders has likely contributed to the lack of broader awareness and implementation. However, the critical limiting factor to sustainability of the program at this juncture is the insufficiency of the MT Cadre to meet the training needs of all stakeholders. DDPM made the decision to focus on IMAT training vs. development of additional MTs; while this accelerated Thailand’s ability to operationalize ICS and support provincial governments, it did not address the shortage in qualified MTs. Some cross-benefit could be achieved if the IMAT-trained staff go on to complete ToT/TFI courses; however this will only be effective if a conscious effort is made to maintain trainers’ availability for actual instruction. One major barrier identified to building sufficient opportunities for practitioners to train on ICS was the difficulty in populating new MT courses with personnel with sufficient English language comprehension to gain maximum benefit from USFS trainers. This was cited as one of the drivers of the decision to shift focus to IMAT development rather than expanding the MT Cadre. Thailand will likely need to solve this language barrier, either by identifying additional English-language speaking staff, modifying course materials and delivery so that a mixed-instructor approach and/or targeted translation can be used effectively, or building the capabilities and credibility of the current MT Cadre such that they can reliably produce new MT generations. In addition, Thailand will need to expand opportunities to practice and apply ICS through two vehicles: (1) enhancing exercise programs so they can effectively offer ICS practical experience, and (2) identifying opportunities such as planned events and low-level emergencies, where ICS can be applied in both a capacity-building and proof-of-concept manner. It should be noted that the ICS Working Committee is considering many of the options described above. 8. How much adaptation (translation of core concepts and revision of key elements to better reflect local conditions) of the ICS is required before the materials can be utilized at the local government-level? Benchmarks Score Indicator Country-specific ICS Forms, Doctrine, and guiding documents have been provided to practitioners. 3 34,35 Country-specific examples have been used in ICS Training Courses. 3 36,35 ICS Forms have been translated and adapted; doctrine and guiding documents have been provided to key implementers who received training. However; this is still a small subset of all disaster management professionals that does not cover all practitioners from first responders to national level agencies. Country-specific examples have been integrated into ICS Training Courses, making the modules more effective. This aspect will continue to improve training as more examples of successful ICS use in Thailand occur in real-world events and can be used in training modules. 9. What factors contribute to sustainability of the ICS? Will existing investments remain viable without external donor support? 23 Benchmarks Score Indicator Local/Regional/Country Governments have funded ICS training and/or exercise events. 3 38 ICS Master Trainer Cadre is able to meet training demands nation-wide. 2 5,41,42 Training and implementation are standardized across the country. 3 43 DDPM has provided funding for most aspects of the ICS training and implementation outside of the salaries and expenses of USFS personnel serving as trainers, mentors, and technical advisors. They have committed to funding all ICS training out of DDPM Academy moving forward, and have a budget for ICS training and implementation. Without USAID/OFDA funding, the ICS program would continue in the near term; however, the MT cadre will not be able to be sustained or grow without further USFS (or other subject matter expert) instruction. With the current MT Cadre, provinces can slowly train personnel over the course of years, but it is not clear when they would be able to get a “critical mass” of personnel trained to actually implement ICS in any meaningful way in all or even a majority of the 76 provinces. The current rate of training is insufficient to build capabilities at the province level, and has not begun to address cascading to the district level, a necessary component to improving response capabilities in the “initial stages” of emergencies. Long term, without additional external funding, technical assistance, and expansion of the MT Cadre, the program is likely to decline due to attrition of experienced MT Cadre through retirement and/or promotion, and even modest training needs will no longer be met. Expanding the MT Cadre is critical as the MT Cadre is not able to meet training demands nation-wide. Currently, cascaded training is primarily provided by the MT Cadre using standardized or abbreviated curriculum, and guidance for implementation is standardized across provinces (though guidelines could be more robust). Though some provinces are implementing ICS fully, most do not have sufficient numbers of trained staff to meaningfully implement the system, leading to losses in standardization. Table 1 below presents the overall score of the Thailand country program based on the evaluation team’s findings and conclusions. The Effectiveness score is 33 out of a highest possible score of 56 while the Sustainability score is 30 out of 44. This leads to an overall index score of 63 out of 100. Table 1: Capacity Building Index Results Objective 1: Effectiveness To what extent has the ICS program built the capacity of national governments to respond to disasters effectively, particularly at the initial stages of the emergency? Score Indicator ICS has been used successfully to manage events, either planned or no-notice. 2 8 Multiple stakeholder agencies have adopted and used ICS. 2 7 Formal training programs are in place and funded. 3 6,37 Which aspects of the ICS training program are most effective? Which aspects of the ICS training program are least effective? Score Indicator Training courses are effective at improving ICS knowledge and organizational capacity. 2 15,16,17 A Lessons Learned program is in place to capture best practices for the country and remedy shortfalls. 2 44 24 Is the ICS targeting actors within the national government that are best positioned to effectively implement elements of the ICS system? Score Indicator ICS course offerings are diverse, include training for Senior Officials as well as Position-Specific sessions. 2 6,20 Program Directors have the authority to implement ICS activities at national and local levels. 3 19 Women receive ICS training at comparable levels to men. 3 1,2,3,4,5 Personnel at multiple levels of government and stakeholder agencies have received training. 2 20 What country-level factors influence the effectiveness of the ICS program? Score Indicator No major cultural barriers to ICS acceptance. 3 33 Acceptance of ICS as a best practice within national government. 3 26 Acceptance of ICS as a best practice within regional / local governments. 2 27 To what extent do training participants—including the different levels of the cascade training system—retain knowledge and skills transferred through the ICS trainings? Score Indicator Training recipients retain knowledge of ICS concepts following training. 2 15 In-country Master Trainer Cadre has the knowledge, skills, and experience to provide high quality training. 2 42 Effectiveness Total: 33 out of 56 Objective 2: Sustainability To what extent are countries able to integrate the ICS into their national frameworks or other institutionalization models? Score Indicator Laws codifying the use of ICS in disaster management in place. 3 9 Doctrine guiding the integration of ICS with country's disaster management system is developed at the strategic level. 3 9 ICS has been successfully integrated and used at multiple levels of government. 2 8,10,30 What barriers to utilization of the ICS exist? Score Indicator Few reports of pushback to adoption of ICS practices. 3 12,31,32,33 Country has established a committee to further ICS acceptance and use. 3 39 Opportunities exist to train on, practice, and apply ICS regularly. 2 29 How much adaptation (translation of core concepts and revision of key elements to better reflect local conditions) of the ICS is required before the materials can be utilized at the local government-level? Score Indicator Country-specific ICS Forms, Doctrine, and guiding documents have been provided to practitioners. 3 34,35 Country-specific examples have been used in ICS Training Courses. 3 36,35 What factors contribute to sustainability of the ICS? Will existing investments remain viable without external donor support? Score Indicator Local/Regional/Country Governments have funded ICS training and/or exercise events. 3 38 25 ICS Master Trainer Cadre is able to meet training demands nation-wide. 2 5,41,42 Training and implementation are standardized across the country. 3 43 Sustainability Total: 30 out of 44 ICS Capacity Index: 66 out of 100 26 VII. Recommendations Based on the finding and conclusions presented in Sections V and VI, respectively, the evaluation team proposes the following recommendations for OFDA’s consideration: Recommendation 1: Develop a Dedicated Master Trainer Cadre 1st and 2nd Generation MTs were instrumental in developing an institutionalized, sustainable ICS program in Thailand. However, the MT Cadre has suffered significant attrition due to promotions, retirements, and concurrent duties. Thailand will need to develop additional highly qualified MTs, provide professional development opportunities, and ensure trainers are available to provide training. USFS should help Thai officials set target training benchmarks and determine the number of MTs needed to meet these goals. If English language capabilities continue to restrict the ability to train additional MTs, DDPM should consider looking outside of DDPM for candidates, either in partner organizations or outside of the disaster response community, as proficiency with ICS material and disaster response is easier to build than second language proficiency. They may also want to revisit ways to make courses accessible across languages, perhaps with a combined Thai-USFS instructor team so that not all instruction is given in US, and translations can be targeted to reduce fatigue or loss of curriculum fidelity. Recommendation 2: Provide Professional Development for Master Trainer Cadre In order to create a truly sustainable program, the MT Cadre must be able to credibly create new Thai MTs. Currently, USFS provides gold-standard MT instruction; however, with additional professional development and response capabilities, the current MT Cadre could provide Thai-language MT instruction that is comprehensive enough to build the next generation internally. USFS should assist Thai officials in developing a credentialing program that tracks trainer competencies and monitors performance to ensure quality and consistency, and in identifying and providing professional development that nurtures the MT force (; e.g., study tours, performing as Incident Commander in real world or exercise events). Recommendation 3: Assist in Developing Participant Database Currently, DDPM does not have a database that tracks training recipients. While MTs and IMAT￾qualified staff are tracked, those receiving cascaded Basic/Intermediate, Executive, or other training are not tracked in any meaningful way. This causes two immediate impacts: (1) DDPM does not have visibility of trained personnel who could be called upon during an incident; and (2) DDPM does not have meaningful data with which to measure effectiveness, capacity, or targeting of current training efforts. Future issues likely to arise include an inability to track professional development, and potential introduction of non-standardized training as external operators may begin to provide ICS training that is not sanctioned by DDPM, as has occurred in the Philippines. Without a centralized credentialing and tracking system, DDPM will be unable to verify the qualifications of its own and partners’ staffs. USFS should consider providing technical assistance or mentoring to help Thai officials design and implement such a system. In addition, cross-country exchanges should be facilitated so DDPM can learn from the Philippines’ experience and lessons learned setting up the system they currently use to designate unique tracking codes to participants and track all trainees in a central database. Recommendation 4: Use Planned Events and Exercises to Practice ICS All trainers and disaster response specialists interviewed by the evaluation team throughout the region stated that practical application opportunities are crucial to maintaining the knowledge gained in ICS, but that there is insufficient exercise design knowledge in Thailand to design and execute exercises that really support learning, and in particular ICS training. USFS should encourage DDPM to revisit the idea of using ICS to manage mass gatherings like public holidays, large-scale sporting events, and other events 27 where response systems might be needed and can be applied. USFS should also provide technical instruction in exercise design and execution so ICS-focused drills and exercises can help trainees retain knowledge and gain practical experience. Recommendation 5: Monitor DDPM ICS Courses to Ensure Quality and Variety of Offerings As DDPM transitions ICS training responsibilities to DDPM Academy, changes are expected in the way courses are managed. USFS should consider offering course monitors who can ensure quality and fidelity of training from an external, trusted source. Steps should also be taken to expand course offerings to ensure diversity of course offerings (; e.g., inclusion of executive level, advanced, and position-specific courses), and that response practitioners at all levels are targeted. USFS should work with Thai officials to determine additional training requirements needed to offer these courses at the required levels, including expansion of training to meet provincial and district training needs. Recommendation 6: Encourage More Effective Pilot Province Programs The success of Ayutthaya’s ICS Pilot Program was due to two key factors: (1) long-term strong leadership and (2) frequent incidents. While this success may not be replicable in every province, particularly in provinces where motivation is low due to lack of regular incidents or disasters, ICS implementation can be accelerated through targeted province-oriented pilot programs. Strong leaders within the MT Cadre, supported by other MTs within DDPM or regional stakeholder groups, should be assigned for a long enough time period to institute sustainable programming at the provincial, district, and first responder levels. USFS should encourage these programs in higher-risk provinces where motivation will be greater and opportunities to practice and implement will help solidify ICS concepts through implementation. Recommendation 7: Provide guidance and assistance to help RTG institute a formal lessons learned program, with ICS as a key focus. After Action Reviews are regularly conducted, but no formal lessons learned program is in place to review ICS-related areas for improvement or success stories. USFS should provide guidance on establishing a formal lessons learned program with the intent to foster capability building at all levels of government. Lessons learned programming should cover monitoring, quality assurance, and continual improvement of training courses; planning, executing, and evaluating exercises; as well as monitoring and evaluating use of ICS in planned events and no-notice responses. Lessons learned should be reported regularly in a standardized format and used to inform future planning. The evaluation team acknowledges that some of these functions are happening individually, but processes are not focused on ICS and a formal program is not funded and administered as a stand-alone capability. Recommendation 8: Establish a performance management system within the country programs Related to Recommendations 4 and 5 above, the local government agencies should establish a performance management system and go through the complete cycle to: 1) develop a performance management plan, 2) construct a country-level results framework, 3) define indicators to track performance, and 4) conduct periodic assessments of outcomes. A performance management system will allow the country team to effectively monitor activities and manage results. As stated in Recommendation 4, foundational resources in Indonesia and the Philippines have been built with databases, and Thailand should develop a similar system. These databases can serve as primary data sources for information across the different levels of the results framework to monitor the program. The foundations to develop a performance management mechanism exist. 28 While performance management is not a requirement and OFDA is exempted from complying with USAID’s Automated Directives System (ADS) requirements, the local government agencies can leverage free and readily available resources from the agency, such as the TIPS series on developing PMPs, results frameworks, indicators, reference sheets, data collection, and analysis. 29 Annex I: Evaluation Statement of Work RFTOP SECTION C – STATEMENT OF WORK C.1 PURPOSE USAID/OFDA seeks to award a contract to evaluate the effectiveness and sustainability of the Incident Command System (ICS) programming in the East Asia and Pacific (EAP) region2. The aim of this evaluation is to improve USAID/OFDA’s understanding of the performance of the ICS program supported by USAID/OFDA in the EAP region. The evaluation shall inform future decision-making and contribute to the evidence-base on Disaster Risk Reduction (DRR) programming. The evaluation will focus on the ICS programming in the Philippines, Indonesia, and Thailand. C.2 BACKGROUND: USAID/OFDA DRR programming in EAP Many countries in the EAP region undergo seasonal periods of increased hydrometeorological activity, experiencing cyclones and monsoon rains, which can increase the risk of floods and landslides and result in significant damage. Several EAP countries situated along the Pacific Ring of Fire experience volcanic activity, as well as earthquakes and associated tsunamis. To decrease the potential risks from these disasters, USAID/OFDA funds a range of disaster risk reduction (DRR) programming throughout the EAP region. USAID/OFDA has focused its DRR activities in the region on strategic, context-specific programs designed to meet particular risk reduction needs in each country, with capacity building as a consistent theme throughout all programs. USAID/OFDA provided nearly $34 million in FY 2013 and approximately $38.3 million in FY 2014 for DRR projects throughout EAP, including programs that integrate DRR with disaster response. More information on USAID/OFDA’s DRR programming can be found in Section J. Background to the ICS model ICS is a standardized incident management concept that is applicable to any hazard or disaster scenario. It is designed to improve coordination and communication among various national government agencies and actors with the aim of expediting emergency response. ICS was developed in the 1970s following a series of catastrophic fires in southern California that resulted in deaths, injuries, and considerable property damage. The personnel assigned to determine the causes of this disaster discovered that response problems could rarely be attributed to lack of resources or failure of tactics; instead, response problems more frequently resulted from inadequate management than from any other single reason. Weaknesses in incident management were often due to lack of accountability, poor communication, lack of an orderly planning process, absence of a predesigned yet flexible management structure, and the absence of a way to integrate interagency requirements into the response effectively. To respond to these failures, the ICS was created; it comprises a set of basic principles, an organizational structure, and an operational planning process. Principles include a clear chain of command, the use of common terminology and interoperable communication systems, standardized training, and certification for positions. Although initially used by firefighters, response agencies began to use the system to manage other types of incidents in the 1980s. Following the September 11, 2001 attacks, the use of ICS at the federal, state, and local level was mandated through a Presidential Directive, and ICS is now a primary component of the US National Incident Management System (NIMS). 30 USAID/OFDA ICS programming in EAP Part of USAID/OFDA’s DRR strategy in EAP is to support programs aimed at increasing the capacity of national incident command systems to effectively respond to disasters. Overall, the objectives of ICS training programs are to enhance the capacity of countries to manage disaster response operations particularly during the initial emergency phase prior to the arrival of outside assistance. The training programs are also designed to ensure vertical integration of disaster management from the local to national level and to encourage the sustainability of the ICS program following the cessation of program activities. USAID/OFDA supports ICS development by funding the U.S Forest Service (USFS) to conduct ICS capacity building activities across the EAP region. USAID/OFDA has provided funding for ICS trainings in the ASEAN member countries of Brunei, Burma, Indonesia, the Philippines, Thailand, and Vietnam, as well as to non-ASEAN member nations Mongolia, Solomon Islands, Timor Leste, and Vanuatu. Phase one of the program introduced ICS to ASEAN members through trainings and a study tour in the United States. Under Phase two, which is ongoing, USFS is continuing regional training activities, including basic and intermediate ICS courses and the development of online ICS training modules. C.3 BACKGROUND: ICS PROGRAMS TO BE EVALUATED C.3.1 Overview of Programs This evaluation will examine the effectiveness and sustainability of USAID/OFDA funded ICS programs in the Philippines, Indonesia, and Thailand. This section summarizes the program’s activities in these three countries. Philippines USAID/OFDA has provided $1,170,000 to support ICS programming in the Philippines from FY 2010 to FY 2014. USAID/OFDA support has funded train-the-trainer courses, as well as technical support for the adaptation of course materials, the establishment of Incident Management Teams (IMTs) in focal areas, the development of national ICS policy and guidelines, the mentoring of MTs, the conduct of exercises, the establishment of training and certification standards for ICS trainers, and the development of a website to track ICS training nation-wide. In FY 2012, USAID/OFDA supported the USFS to bring ICS training to the local government unit level, with a focus on local government units and provinces in four regions (Cagayan, Laguna, Sorsogon and Benguet) that USAID/OFDA targeted for a capacity-building and technical support program implemented through the World Food Program (WFP). The USFS worked with WFP and the Philippine Office of Civil Defense to conduct an ICS course for participants from selected local government units and Provincial Disaster Risk Reduction Management Committees (PDRRMCs) in each of the four regions, as well as team-based All-Hazard Incident Management Team (AHIMT) courses and exercises for newly established Incident Management Teams in Cagayan and Benguet. Indonesia USAID/OFDA has provided $520,000 to support the implementation of ICS in Indonesia since FY 2012. In 2008, National Agency for Disaster Management (BNPB) issued Regulation 10, which mandates that Government of Indonesia (GoI) response agencies use the ICS. BNPB requested USG assistance in integrating ICS and other relevant NIMS components into Indonesia’s newly established disaster management system, and ICS is a primary component of BNPB’s curriculum for GoI disaster managers. Thailand USAID/OFDA has provided $861,500 for ICS programming in Thailand from FY 2010 to FY 2014 through its interagency agreement with the USFS. The USFS began working with Thailand’s Department 31 of Disaster Prevention and Mitigation (DDPM) in 2010. In FY 2012 and 2013, USAID/OFDA provided funding for a series of ICS Train-the-Trainer courses. DDPM and the MTs have subsequently cascaded ICS training to operational responders in a variety of agencies and regions, and have applied ICS during the conduct of tabletop and full-scale multi-agency exercises. C.3.2 Goal, Objectives, and General Theory of Change of ICS programs The goal of USAID/OFDA’s support to ICS programs is to increase the capacity of countries within in EAP to effectively and efficiently respond to disasters using the ICS model. The objectives of each of the iterations of ICS programming varies depending on the disaster profile and baseline capacity of the country in which the program is implemented. In general, however, the objectives of ICS programs are to increase the capacity of trainees to understand the ICS model and operate effectively within it once a disaster occurs. The theories of change behind this type of program are both institutional and individual. By implementing ICS training programs targeting civilian government officials involved in disaster response, the capacity of the individuals trained will increase and the effectiveness of the disaster response institutions in which they work will improve. This evaluation is designed to examine this model by studying its effectiveness and sustainability in the Philippines, Indonesia, and Thailand. C.3.3 Existing performance data The contractor will be able to access periodic performance reports from the ICS programs implemented with OFDA funding. The contractor will not have access to evaluation data or detailed quantitative results from performance monitoring. C.4 THE EVALUATION C.4.1 Purpose & Use The goal of this evaluation is to improve USAID/OFDA’s understanding of the performance of the ICS program supported by USAID/OFDA in the EAP region. The evaluation shall focus on the effectiveness and sustainability of the USAID/OFDA funded ICS programs in the Philippines, Indonesia, and Thailand. The findings of this evaluation will inform future programming decisions and adjustments to ongoing ICS programming. After five years of supporting the ICS programming in a variety of regions, USAID/OFDA seeks a deeper understanding of the successes and areas for improvement in this capacity-building model. An evaluation of the performance of the ICS program in FY 2015 will allow USAID/OFDA decision-makers to apply the information from this evaluation in future programming decisions related to DRR programming and the ICS program model specifically. With more than $4 million invested in the ICS program over the last five years, the program is a significant investment in resources from USAID/OFDA. As with many capacity-building initiatives, contextual factors can influence the results from the capacity intervention. The introduction of ICS into a variety of countries—all with their own disaster risks, government structures, and socio-economic factors—has led USAID/OFDA to seek more information about the factors for success and the potential for sustainability. C.4.2 Evaluation Objectives & Lines of Inquiry This evaluation has two objectives: understanding effectiveness and sustainability. Each objective has several lines of inquiry that shall inform the evaluation design. Data must be collected for all lines of inquiry in the Philippines, Indonesia, and Thailand and will be analyzed to compare results from the different contexts. 32 USAID/OFDA does not regard these lines of inquiry as final and could modify the lines of inquiry through the design phase of the evaluation through conversations with the evaluation team. The lines of inquiry will be considered final in the approved version of the inception report deliverable of this contract. While exact wording of the lines of inquiry could be modified through the design phase, the objectives of the evaluation as well as the intent and focus of the evaluation will not change. Objective 1: Effectiveness 1. To what extent has the ICS program built the capacity3 of national governments to respond to disasters effectively, particularly at the initial stages of the emergency? 2. Which aspects of the ICS training program are most effective? Which aspects of the ICS training program are least effective? 3. Is the ICS targeting actors within the national government that are best positioned to effectively implement elements of the ICS system? 4. What country-level factors influence the effectiveness of the ICS program? 5. To what extent do training participants—including the different levels of the cascade training system—retain knowledge and skills transferred through the ICS trainings? Objective 2: Sustainability 6. To what extent are countries able to integrate the ICS into their national frameworks or other institutionalization models? 7. What barriers to utilization of the ICS exist? 8. How much adaptation (translation of core concepts and revision of key elements to better reflect local conditions) of the ICS is required before the materials can be utilized at the local government￾level? 9. What factors contribute to sustainability of the ICS? Will existing investments remain viable without external donor support? C.4.3 Evaluation Type This performance evaluation will be summative in nature. It will evaluate programs that have closed in order to draw conclusions about their effectiveness and sustainability so that USAID/OFDA can adjust future implementation plans and strategies. C.4.4 Evaluation Approach This evaluation will take a utilization-focused evaluation approach, meaning the evaluation will be designed through extensive consultation with USAID/OFDA and the USFS. USAID/OFDA seeks to gain feedback directly from the client countries who have participated in ICS activities. The evaluation will be designed to ensure that findings from the evaluation can directly inform future programming decisions related to ICS. The evaluation team will need to tailor the evaluation approaches to the specific context of country and analyze findings accordingly. The evaluation should be designed to allow readers to draw comparisons across country locations and provide analysis of the findings for the different contexts. C.4.5 Evaluation Audience The users of this evaluation will be USAID/OFDA’s EAP team and USFS International Programs Department. More broadly, this evaluation will be read by USAID/OFDA staff in Washington, D.C., and regional field offices. This evaluation will also add to the evidence base on ICS and DRR programming. The evaluation will be available to all USAID/OFDA staff and will be made publicly available, in accordance with specific contractual requirements. C.4.6 Evaluation Methods & Data Sources General guidance 33 This evaluation must employ mixed methods: both qualitative and quantitative methods are required. Primary data must form the majority of the data collected for this evaluation; secondary data review alone will not suffice to inform the deliverables for this evaluation. The evaluation must use primary data collected in Thailand, the Philippines, and Indonesia to inform responses to the lines of inquiry of the evaluation. Any quantitative data collection that takes place as part of this evaluation must include a representative sample of the survey population and must use rigorous methods for data collection and analysis. The survey populations for any large scale survey conducted as part of this evaluation are the individual countries (Thailand, the Philippines, and Indonesia) or groups of stakeholders within those countries, not the total population of survey populations in all three. Therefore, if a large scale survey is conducted, there will be three separate surveys. The data must be representative at a sub-national level, though the exact level of representation will be determined during the design stage of the evaluation. USAID/OFDA prefers to have data with no more than a 5% margin of error and at least a 95% confidence level. However, the exact parameters of the survey design will be determined during the initial stages of the evaluation process. If less rigorous quantitative methods are needed, the Contractor must submit a justification to the COR for review and approval. Only after the COR has approved the less rigorous quantitative methods may they be used by the Contractor. The qualitative methods proposed should consider the various approaches to evaluating training and capacity building programs in development contexts. The evaluation plan must incorporate methods suitable to evaluating training, capacity-building, and DRR. The qualitative inquiry should be designed to analyze data from multiple levels along the capacity-building results chain. The evaluation will use multiple data collection instruments, aiming to capture data from multiple angles and data sources. Having analysis from a variety of levels will inform the findings of the evaluation. Multiple-level analysis is important to understand where and how the interventions are influencing the government’s capacity to respond to disasters. This evaluation should consider the relationships between capacity interventions and capacity outcomes, and the links between capacity and performance variables. In the selection of methods, the contractor must include ethical considerations, do no harm precautions, and informed consent. Specific methods The exact methods to be used in this evaluation will be determined through the submission and acceptance of the Inception Report and Evaluation Plan deliverables. However, USAID/OFDA requires the set of methods described in the rest of this section to be used for certain lines of inquiry in this evaluation. If, through the design process, the evaluation team finds that these methods are not suitable for responding to the identified lines of inquiry, the Contractor must provide a clear justification. Changes to these methods requirements are subject to the review and approval of the COR. • To what extent do training participants—including the different levels of the cascade training system—retain knowledge and skills transferred through the ICS trainings? o How much adaptation (translation of core concepts and revision of key elements to better reflect local conditions) of the ICS is required before the materials can be utilized at the local government-level? o What factors contribute to sustainability of the ICS? Will existing investments remain viable without external donor support? 34 o To what extent has the ICS program built the capacity4 of national governments to respond to disasters effectively, particularly at the initial stages of the emergency? There are several types of methods that USAID/OFDA requires for the above lines of inquiry. In general, the contractor must use both quantitative and qualitative methods to gather data related to the lines of inquiry above. First, the evaluation team must work with the USAID/OFDA and USFS to map benchmarks related to the ICS programming that link to levels of capacity-building. This mapping should help to understand what essential inputs and processes are required at each level to achieve capacity-building benchmarks in various country contexts. For example, one benchmark that indicates a level of capacity has been reached is the introduction of a policy that mandates the use of ICS for disaster management. This mapping must be an ongoing process during the evaluation. Second, the contractor must use a survey of training participants to measure retained knowledge over time. Third, the contractor must conduct structured or unstructured individual interviews with training participants, government officials, and other relevant stakeholders. Fourth, this evaluation must include self-assessment techniques that have the added value of allowing participants in the evaluation to reflect and learn. Self-assessment techniques must be balanced with other objective measures and data collection. Fifth, this evaluation must include focus group discussions with relevant stakeholders, community groups, and government officials. Finally, the contractor must develop a capacity building index for the ICS system must be an overall tool for evaluators to utilize in data collection. The capacity building index must be based on the ICS curriculum and ICS program objectives. • Is the ICS targeting actors within the national government that are best positioned to effectively implement elements of the ICS system? • To respond to the line of inquiry stated above, the contractor must conduct individual interviews and focus group discussions with relevant stakeholders. • To what extent are countries able to integrate the ICS into their national frameworks or other institutionalization models? • What barriers to utilization of the ICS exist? • How much adaptation (translation of core concepts and revision of key elements to better reflect local conditions) of the ICS is required before the materials can be utilized at the local government￾level? • What factors contribute to sustainability of the ICS? Will existing investments remain viable without external donor support? To respond to the lines of inquiry stated above, the contractor must use qualitative methods. If the contractor believes that quantitative methods would be useful to use for these questions, the contractor may use them. The qualitative methods must include individual interviews with training participants and government officials at a minimum. Data Sources 35 The evaluation team must interview the most relevant respondents for each of the lines of inquiry. While each line of inquiry will have a different set of respondents deemed to be most relevant, the evaluation team must collect data from the following groups of individuals, (contingent upon the receipt of informed consent): 1. Training participants 2. Government officials 3. USAID/OFDA staff 4. USFS staff 5. ICS trainers 6. Community members 7. Other members of the USG that have worked with training participants during a disaster response 8. Training curricula 9. Periodic programmatic reports The above list of respondents is not exhaustive. The contractor is encouraged to add more groups to respondents to ensure that the list is comprehensive and adequately covers all lines of inquiry. This list of respondents may include individuals who have departed the countries in question or who no longer work for the organizations that employed them during the ICS training or disaster response. As such, the evaluation team must find as many of these individuals as possible to interview. C.4.7 Limitations Most ICS programs may not have baseline data and may not have consistent performance monitoring data. There have not been previous evaluations of the programs and documentation of progress, successes, and challenges is very limited. It may take time to contact respondents who have changed jobs or job location since the training program began. C.4.8 Geographic Scope The geographic scope of this evaluation is Indonesia, the Philippines, Thailand, and the United States. Primary data collection for this evaluation – and thus travel for relevant members of the evaluation team – must occur in Indonesia, the Philippines, and Thailand. Travel to countries other than the aforementioned countries for the purposes of this contract is allowable only if the evaluation plan identifies a clear need for primary data collection to occur and if the COR approves the need. The evaluators must also prepare to interview respondents via the telephone or internet-enabled communication in countries outside of the aforementioned countries. The Contractor is responsible for accessing all countries and regions within the countries in the geographic scope of this contract. C.4.9 Programming Period Covered by Evaluation This evaluation will cover USAID/OFDA-funded ICS programs implemented between FY 2010 and FY2015. C.4.10 Period of Performance of the Evaluation This evaluation must be carried out under an agreed upon schedule of work, determined through the work plan. The period of performance of this contract is eight (8) months. C.5 DELIVERABLES Each of the deliverables must be submitted to USAID/OFDA for review and approval. USAID/OFDA has the following general quality standards for each of the deliverables submitted as part of this contract: 1. Deliverables must contain no typographical or grammatical errors. 36 2. Deliverables must be submitted at or before the deadline established in the contract. For deliverables with deadlines established in the work plan, the Contractor must submit the deliverables at or before the deadline approved in the work plan. 3. The deliverables must be written using concise and direct language. Deliverable Contract Reference Planned Completion Date Post-Award Conference Call C.5.1 NLT 7 days after Contract Award Work Plan C.5.2 NLT 14 days after Contract Award Kick-Off Meeting C.5.3 NLT 21 days after Contract Award Inception Report C.5.4 To be determined in the work plan 37 Annex II: List of Indicators No. Indicators Data Sources Data Collection Locations Objective 1: Effectiveness To what extent has the ICS program built the capacity of national governments to respond to disasters effectively, particularly at the initial stages of the emergency? 1 Number of government officials trained annually Program Plans, Records, and Reports including: ICS Training Participant rosters from program Implementers (OFDA, USFS) Training provider records Government records Course Curricula US and In-country (some can be collected remotely) 2 Number of government officials trained total 3 Number of first responders trained annually 4 Number of first responders trained total 5 Number of Master Trainers trained 6 Number of training events by type 7 Number of organizations who have adopted ICS (in plans, proclamations, etc.) 8 ICS/NIMS used successfully in responses or exercises After action reports / Incident Reviews In-Country (can be collected remotely) 9 References to ICS/NIMS components in plans, policies, procedures, and laws Government disaster response agencies plans, policies, procedures, and laws Local response agencies plans, policies, procedures 10 Usage of ICS/NIMS concepts and procedures at multiple levels Response Officials In-Country 38 11 Perceived effectiveness of ICS program in improving disaster response capabilities 12 Support for implementation Which aspects of the ICS training program are most effective? Which aspects of the ICS training program are least effective? 13 Positive Trainee Feedback Training Feedback Forms (if available) US and In-Country (can be 14 Negative Trainee Feedback collected remotely) 8 ICS/NIMS used successfully in responses or exercises After Action Reports / Incident Reviews US 15 Retained Knowledge Survey: Amount of knowledge retained by training participants (did position, number of opportunities to apply knowledge, etc. impact retained knowledge) Training Participants In-Country 16 Impact of training (self-assessment) 17 Organizational capacity and/or performance improvements Master Trainers Response Officials In-Country 18 Unsuccessful ICS/NIMS use in responses or exercises 16 Impact of training (self-assessment) Is the ICS targeting actors within the national government that are best positioned to effectively implement elements of the ICS system? 12 Support for implementation Response Officials In-Country 19 Authority to implement 11 Perceived effectiveness of ICS program in improving disaster response capabilities 39 20 Distribution of training participants' levels of authority/activity Training records Government records Organizational charts In-Country (can be collected remotely) What country-level factors influence the effectiveness of the ICS program? 21 Existing culture of coordination/collaboration Open source reports 22 Presence of good governance indicators - accountability, transparency, rule of law 23 Resource availability 24 Geography 25 Frequency of disasters 26 Acceptance by national governments Response Officials In-Country 27 Acceptance by local governments 28 Perceived contributions of country level data listed above or other factors 29 Opportunities to use gained knowledge in exercises or real world events Training Participants In-Country To what extent do training participants—including the different levels of the cascade training system—retain knowledge and skills transferred through the ICS trainings? 15 Retained Knowledge Survey: Amount of knowledge retained by training participants (did position, number of opportunities to apply knowledge, etc. impact retained knowledge) Training Participants In-Country Objective 2: Sustainability To what extent are countries able to integrate the ICS into their national frameworks or other institutionalization models? 8 ICS/NIMS used successfully in responses or exercises After Action Reports/Incident Reviews In-Country (can be collected remotely) 40 9 References to ICS/NIMS components in plans, policies, procedures, and laws National, Regional, Local plans, policies, procedures 30 Success of integration at multiple levels of government Response Officials In-Country What barriers to utilization of the ICS exist? 31 Failed attempts to integrate Response Officials In-Country 32 National/Regional level challenges 33 Personal experiences of challenges at various levels of implementation (; e.g., didn't understand terminology; culture not congruent with principles; didn't have country-specific examples of usage) Response Officials In-Country How much adaptation (translation of core concepts and revision of key elements to better reflect local conditions) of the ICS is required before the materials can be utilized at the local government-level? 34 Changes / adaptations to ICS forms, structures, or protocols that have been made for the country Response Officials In-country 35 Suggested adaptations / areas for improvement Response Officials 36 Changes / adaptations to ICS Training Curricula that have been made for the country USFS Trainers Master Trainers US and In-country 35 Suggested adaptations / areas for improvement 37 Training delivery challenges 41 37 Training delivery challenges Course evaluations (if they exist, using IOTWS evaluations as a benchmark) Course Reports US What factors contribute to sustainability of the ICS? Will existing investments remain viable without external donor support? 38 Availability of other sources of Funding - internal or international (e.g., national level funding, CDC International Fellowship Programs) Response Officials In-Country 39 Perceived sustainability of country ICS programming Response Officials In-Country 40 Establishment of Structures, such as an ICS Steering Committee, to continue ICS capability development Response Officials In-Country 41 Cost of additional training given availability of Master Trainers reasonable Master Trainers Training Participants Response Officials US and In-Country 42 Master Trainers provide high quality training 43 Standardized training and implementation 44 Programs to capture and share lessons learned are in place Response Officials In-Country 12 Support for implementation Response Officials In-Country 9 References to ICS/NIMS components in plans, policies, procedures, and laws Government disaster response agencies plans, policies, procedures, and laws Local response agencies plans, policies, procedures US (can be collected remotely) 42 38 Availability of other sources of Funding - internal or international (e.g., national level funding, CDC International Fellowship Programs) Program Activity Reports, SOWs, Open Source 12 Support for implementation Local, Regional, National plans, policies, laws, etc. 43 Annex III: Data Collection Instruments Instrument for Response Officials Methods No. Indicators Questions Responses Notes Objective 1: Effectiveness To what extent has the ICS program built the capacity of national governments to respond to disasters effectively, particularly at the initial stages of the emergency? Interviews / Focus Group 10 Usage of ICS/NIMS concepts and procedures at multiple levels How many levels of government and agencies are currently using ICS? 11 Perceived effectiveness of ICS program in improving disaster response capabilities Has the ICS programming provided improved disaster response capabilities? 12 Support for implementation How much support is there for continued use of ICS nationwide? Which aspects of the ICS training program are most effective? Which aspects of the ICS training program are least effective? Interviews / Focus Group 17 Organizational capacity and/or performance improvements Have ICS training programs improved partner nation(s) organizational capacity and performance? Can you cite any specific, tangible gains? 18 Unsuccessful ICS/NIMS use in responses or exercises Are you aware of any uses of ICS/NIMS in our partner nations that have been unsuccessful or ineffective? 16 Impact of training (self￾assessment) What have been the most effective aspects of the ICS Training program? Which aspects have been least effective? 44 Is the ICS targeting actors within the national government that are best positioned to effectively implement elements of the ICS system? Interviews / Focus Group 12 Support for implementation Has the partner nation been supportive of implementing ICS? Has there been a specific champion? 19 Authority to implement Do the people receiving ICS training have the authority to implement ICS at their levels? 11 Perceived effectiveness of ICS program in improving disaster response capabilities How effective has the ICS programming been in improving disaster response capabilities? What country-level factors influence the effectiveness of the ICS program? Interviews / Focus Group 26 Acceptance by national governments How well is ICS accepted at the National level? 27 Acceptance by local governments How well is ICS accepted at the local level? 28 Perceived contributions of country level data listed above or other factors What factors in this partner nation have influenced the effectiveness of the ICS program, whether negative or positive? To what extent do training participants—including the different levels of the cascade training system—retain knowledge and skills transferred through the ICS trainings? Objective 2: Sustainability To what extent are countries able to integrate the ICS into their national frameworks or other institutionalization models? Interviews / Focus Groups 30 Success of integration at multiple levels of government Have ICS processes, documents, and strategies been successfully integrated into response processes at local and national levels? 45 What barriers to utilization of the ICS exist? Interviews/ Focus Groups 31 Failed attempts to integrate Are you aware of any times officials have tried to integrate ICS concepts into their frameworks, but were not successful? 32 National/Regional level challenges What challenges exist to full and effective use of ICS in the partner nation? Are there challenges at both the national and regional levels? 33 Personal experiences of challenges at various levels of implementation (e.g., didn't understand terminology; culture not congruent with principles; didn't have country-specific examples of usage) What challenges have you encountered when trying to implement ICS? (e.g., didn't understand terminology; culture not congruent with principles; didn't have country-specific examples of usage) How much adaptation (translation of core concepts and revision of key elements to better reflect local conditions) of the ICS is required before the materials can be utilized at the local government-level? Interviews / Focus Groups 34 Changes / adaptations to ICS forms, structures, or protocols that have been made for the country What changes have been made to standard ICS forms, structures, and protocols to meet country requirements? 35 Suggested adaptations / areas for improvement What changes should be made to enhance ICS implementation? What factors contribute to sustainability of the ICS? Will existing investments remain viable without external donor support? Interviews / Focus Groups 38 Availability of other sources of Funding - internal or international (e.g., national level funding, CDC Have ICS initiative been funded by organizations other than USAID/OFDA? What funding have the 46 International Fellowship Programs) national or local governments provided? 39 Perceived sustainability of country ICS programming Is the country's ICS programming sustainable? i.e. if OFDA no longer sponsored it, would the country still use ICS? 40 Establishment of Structures, such as an ICS Steering Committee, to continue ICS capability development Has the partner nation established a committee responsible for advocating for and guiding the adoption of ICS practices at national and local levels? 41 Cost of additional training given availability of Master Trainers reasonable Is the cost of cascaded training reasonable and accessible to a variety of local responder organizations? 42 Master Trainers provide high quality training Do the in -country trainers provide high quality training that is comparable to that offered by the USFS cadre? If not, what can be done to ensure quality? 43 Standardized training and implementation Is ICS training standardized across the nation? Is implementation standardized? 44 Programs to capture and share lessons learned are in place Are programs in place to capture and share lessons learned nationally and internationally? 47 12 Support for implementation Asked Previously Instrument for Training Cadres Methods No. Indicators Question Responses Notes Objective 1: Effectiveness To what extent has the ICS program built the capacity of national governments to respond to disasters effectively, particularly at the initial stages of the emergency? Which aspects of the ICS training program are most effective? Which aspects of the ICS training program are least effective? Interviews / Focus Group 17 Organizational capacity and/or performance improvements Have ICS training programs improved partner nation(s) organizational capacity and performance? Can you cite any specific, tangible gains? 18 Unsuccessful ICS/NIMS use in responses or exercises Are you aware of any uses of ICS/NIMS in our partner nations that have been unsuccessful or ineffective? 16 Impact of training (self￾assessment) What have been the most effective aspects of the ICS Training program? Which aspects have been least effective? Is the ICS targeting actors within the national government that are best positioned to effectively implement elements of the ICS system? Interviews / Focus Group 12 Support for implementation Has the partner nation been supportive of implementing ICS? Has there been a specific champion? 11 Perceived effectiveness of ICS program in improving How effective has the ICS programming been in 48 disaster response capabilities improving disaster response capabilities? What country-level factors influence the effectiveness of the ICS program? Interviews / Focus Group 26 Acceptance by national governments How well is ICS accepted at the National level? 27 Acceptance by local governments How well is ICS accepted at the local level? 28 Perceived contributions of country level data listed above or other factors What factors in this partner nation have influenced the effectiveness of the ICS program, whether negative or positive? To what extent do training participants—including the different levels of the cascade training system—retain knowledge and skills transferred through the ICS trainings? Objective 2: Sustainability To what extent are countries able to integrate the ICS into their national frameworks or other institutionalization models? What barriers to utilization of the ICS exist? Interviews/ Focus Groups 31 Failed attempts to integrate Are you aware of any times officials have tried to integrate ICS concepts into their frameworks, but were not successful? 32 National/Regional level challenges What challenges exist to full and effective use of ICS in the partner nation? Are there challenges at both the national and regional levels? 33 Personal experiences of challenges at various levels of implementation (e.g., didn't understand terminology; culture not What challenges have you encountered when trying to implement ICS? (e.g., didn't understand terminology; culture not congruent with 49 congruent with principles; didn't have country-specific examples of usage) principles; didn't have country-specific examples of usage) How much adaptation (translation of core concepts and revision of key elements to better reflect local conditions) of the ICS is required before the materials can be utilized at the local government-level? Interviews / Focus Groups 36 Changes / adaptations to ICS Training Curricula that have been made for the country What changes have been made to the ICS Training curriculum to adapt the content and/or presentation to the target country? 35 Suggested adaptations / areas for improvement What changes should be made to enhance ICS implementation? 37 Training delivery challenges What are the main challenges you encounter when delivering training? What factors contribute to sustainability of the ICS? Will existing investments remain viable without external donor support? Interviews / Focus Groups 38 Availability of other sources of Funding - internal or international (e.g., national level funding, CDC International Fellowship Programs) Have ICS initiative been funded by organizations other than USAID/OFDA? What funding have the national or local governments provided? 39 Perceived sustainability of country ICS programming Is the country's ICS programming sustainable? i.e. if OFDA no longer sponsored it, would the country still use ICS? 50 40 Establishment of Structures, such as an ICS Steering Committee, to continue ICS capability development Has the partner nation established a committee responsible for advocating for and guiding the adoption of ICS practices at national and local levels? 41 Cost of additional training given availability of Master Trainers reasonable Is the cost of cascaded training reasonable and accessible to a variety of local responder organizations? 42 Master Trainers provide high quality training Do the in-country trainers provide high quality training that is comparable to that offered by the USFS cadre? If not, what can be done to ensure quality? 43 Standardized training and implementation Is ICS training standardized across the nation? Is implementation standardized? 44 Programs to capture and share lessons learned are in place Are programs in place to capture and share lessons learned nationally and internationally? 51 DevTech Systems, Inc. Independent Evaluation of the USAID/OFDA Incident Command Systems (ICS) Training Program Focus Group Discussion Questions RESPONSE OFFICIALS You have received this request to join us for this group discussion because of your participation or involvement in the Incident Command System (ICS) training program funded by the United States Agency for International Development Office of Foreign Disaster Assistance (USAID/OFDA). DevTech Systems, Inc. has been commissioned by OFDA to conduct an external evaluation of the ICS program and we are conducting this survey as part of the evaluation tasks. We would like to get your thoughts and opinions on the effectiveness and sustainability of the ICS program in building the capacity of local first responder teams to events. The information you will share will be kept strictly confidential and there will be no disclosure of any individual response. The only information that will be used for disclosure to third parties will be aggregates and summaries of the information from all participants, and a selection of comments made, without attribution to any individual who made them. We are grateful for your participation in this evaluation. The information you provide will contribute significantly to the evaluation and the work of the ICS program as a whole. Thank you very much. Sincerely, The DevTech Evaluation Team 52 Objective 1: Effectiveness The FGD moderator should attempt to elicit responses to all of the Primary Questions using Primary Questions or secondary/probe questions to elicit perception- and experience-based responses from the focus group participants. Indirect responses that come up in discussions that follow any of the moderator’s prompts should be recorded. If, in the course of the FGD, participants do not answer a Primary Question, use the secondary or probe questions to ensure that all relevant indicators have been addressed. If, in answering one Primary Question, an interviewee provides an answer for another, the interviewer may alter the original order of Primary Questions to best maintain a natural conversational flow. The moderator should use the Primary Questions as a guide and select appropriate secondary or probe questions to ask that align with the focus group’s particular set of experiences or expertise. Primary Question 1: Have you noticed a difference in the effectiveness of disaster response(s) in your country since the OFDA/USFS ICS program began? 1a. How many levels of government and agencies are currently using ICS? 1b. Has the ICS programming provided improved disaster response capabilities? Have any aspects of disaster response become less effective as a result of the training? If so, do you think these are temporary setbacks as part of the change toward ICS, or is ICS incompatible with any part of your country’s disaster response system? 1c. How much support is there for continued use of ICS nationwide? Primary Question 2: Which aspect(s) of the ICS training program have you found to be the most effective? Which aspect(s) have you found to be the least effective? 2a. Have ICS training programs improved your country’s disaster response organizational capacity and performance? Have you noticed a difference in skill level and response performance between responders who have and have not received ICS training? 2b. Are you aware of any uses of ICS/NIMS in our partner nations that have been unsuccessful or ineffective? 2c. In your opinion, what have been the most effective aspects of the ICS Training program? 2d. Which aspects have been least effective? Primary Question 3: Is the ICS targeting actors within the national government that are best positioned to effectively implement elements of the ICS system? Which actors, if any, have been left out of ICS planning, training and implementation that should have been included? 3a. Has the national government been supportive of implementing ICS? Has any particular branch, agency, organization or person (local or foreign) been a champion for ICS implementation? 53 3b. Do the people receiving ICS training have the authority to implement ICS at their levels? At what point did they receive this authority (i.e.: how long into the ICS training and implementation did this change occur)? 3c. How effective has the ICS programming been in improving disaster response capabilities? Can you provide a specific example – in training or in actual disaster response – to support your view? Primary Question 4: What country-level factors influence the effectiveness of the ICS program? 4a. How well is ICS accepted at the National level? 4b. How well is ICS accepted at the local level? 4c. What factors in this partner nation have influenced the effectiveness of the ICS program, whether negative or positive? Objective 2: Sustainability Primary Question 1: To what extent has your country integrated ICS into its national framework or other institutional models? 1a. Have ICS processes, documents, and strategies been successfully integrated into response processes at local and national levels? Primary Question 2: What barriers to utilization of the ICS exist? 2a. Are you aware of any times officials have tried to integrate ICS concepts into their frameworks, but were not successful? 2b. What challenges exist to full and effective use of ICS in your country? Are there challenges at both the national and regional levels? 2c. What challenges did you encounter or notice in integrating ICS principles into the national framework? (e.g.: didn't understand terminology; culture not congruent with principles; didn't have country-specific examples of usage) Primary Question 3: How much adaptation (translation of core concepts and revision of key elements to better reflect local conditions) of the ICS is required before the materials can be utilized at the local government-level? 3a. What changes have been made to standard ICS forms, structures, and protocols to meet country requirements? 3b. Did you encounter any challenges specific to the ICS training and training materials that hindered the integration of the ICS into your national framework? If so, what changed – or should have changed – to enhance and speed up the integration process? Who first noticed the issues, and who made the appropriate changes? 54 3b. What changes should be made to enhance ICS implementation? Primary Question 4: What factors contribute to sustainability of the ICS? Will existing investments remain viable without external donor support? 4a. Have ICS initiative been funded by organizations other than USAID/OFDA? If so, have these trainings helped or hindered the integration and standardization of ICS in your country? 4b. What funding have the national or local governments provided? 4b. Do you believe that the ICS program in your country would remain viable and sustainable without external donor (i.e.: USAID) funding or training? Why or why not? 4c. Has the government established a committee responsible for advocating for and guiding the adoption of ICS practices at national and local levels? 4d. Have you heard of any issues with cascaded training related to cost and/or accessibility? Are local responder organizations able to access training, and is this access sustainable in the long-term? 4e. Do the in-country trainers provide high quality training that is comparable to that offered by the USFS cadre? What differences have you noticed or been told about between local trainers and USFS trainers? What can be done to ensure quality? 4f. Is ICS training standardized across the nation? Is implementation standardized? 4g. Are programs in place to capture and share lessons learned nationally and internationally? Who (what government branch/agency or individual) spearheaded the development of these programs? 55 DevTech Systems, Inc. Independent Evaluation of the USAID/OFDA Incident Command Systems (ICS) Training Program Focus Group Discussion Questions TRAINER CADRES You have received this request to join us for this group discussion because of your participation or involvement as a trainer in the Incident Command System (ICS) training program funded by the United States Agency for International Development Office of Foreign Disaster Assistance (USAID/OFDA). DevTech Systems, Inc. has been commissioned by OFDA to conduct an external evaluation of the ICS program and we are conducting this survey as part of the evaluation tasks. We would like to get your thoughts and opinions on the effectiveness and sustainability of the ICS program in building the capacity of local first responder teams to events. The information you will share will be kept strictly confidential and there will be no disclosure of any individual response. The only information that will be used for disclosure to third parties will be aggregates and summaries of the information from all participants, and a selection of comments made, without attribution to any individual who made them. We are grateful for your participation in this evaluation. The information you provide will contribute significantly to the evaluation and the work of the ICS program as a whole. Thank you very much. Sincerely, The DevTech Evaluation Team 56 Objective 1: Effectiveness The FGD moderator should attempt to elicit responses to all of the Primary Questions using Primary Questions or secondary/probe questions to elicit perception- and experience-based responses from the focus group participants. Indirect responses that come up in discussions that follow any of the moderator’s prompts should be recorded. If, in the course of the FGD, participants do not answer a Primary Question, use the secondary or probe questions to ensure that all relevant indicators have been addressed. If, in answering one Primary Question, an interviewee provides an answer for another, the interviewer may alter the original order of Primary Questions to best maintain a natural conversational flow. The moderator should use the Primary Questions as a guide and select appropriate secondary or probe questions to ask that align with the focus group’s particular set of experiences or expertise. Primary Question 1: Have you noticed a difference in the effectiveness of disaster response(s) in your country since the OFDA/USFS ICS program began? 1a. If so, what changes that have occurred? Can you give an example of these changes? At what level – national, provincial, organizational, etc. – have these changes been the most profound? 1b. Would you say that the national disaster response effectiveness has improved, worsened, or remained the same as before? Can you provide specific examples? 1c. Do you believe that your country could effectively respond to a disaster without foreign assistance? Why or why not? 1d. If you could use foreign assistance for one aspect of a disaster response, which aspect would that be? Why? 1e. Has the ICS program changed the capacity of your country to respond to a disaster? Primary Question 2: Which aspect(s) of the ICS training program have you found to be the most effective? Which aspect(s) have you found to be the least effective? 2a. In your experience, how did the ICS training programs change the way in which your country’s disaster response functioned? 2b. Have you found that any of the uses of ICS/NIMS have been unsuccessful, ineffective or unnecessary? Can you please provide an example? Probe: Other than the USFS ICS training, have you received similar training? If so, when, and who provided it? In what ways was the training similar? In what ways was the training different? Note for moderator: Probe to find out at what level – national, provincial, organizational, or other – the focus group participants have noticed the changes being the most profound. Make sure to note in which level(s) of ICS training participants have had direct experience. Note for moderator: Find out at what point along the post-disaster timeline (i.e.: immediately following a disaster or later) the participants view their countries’ response strengths and weaknesses. 57 2c. What aspect of ICS training has been the most useful at improving national organizational capacity and performance? 2d. What aspect of ICS training has been the least useful? Primary Question 3: Which actors (agencies, individuals, offices) work or have worked with the ICS programmers? Why do you believe these actors were targeted? 3a. Is there a more appropriate actor or set of actors that should have been targeted instead? Probe: Where (geographically, provincially) is the ICS program lacking? 3b. Has the national government supported the implementation of ICS? If yes, was this support given quickly, or did it take time for a strong relationship between USFS/OFDA and the national government to form? 3c. Has there been a person or agency that has been particularly supportive of ICS (in other words: has there been a champion who has worked very hard to ensure that ICS programming is incorporated into the national system)? Primary Question 4: What factors have influenced the effectiveness of the ICS program, in the country overall and in your province/organization/agency specifically? 4a. How well is ICS accepted at the National level? 4b. How well is ICS accepted at the local level? 4c. Do you think that your country has incorporated ICS effectively? Why or why not? 4d. Compared to other countries in the region, how well do you believe your country has incorporated and utilized ICS? What factors have contributed to your country’s relative success or slower pace? What challenges have you notices or experienced in the process of integrating ICS into your country’s national framework? Objective 2: Sustainability Primary Question 1: To what extent has your country integrated ICS into its national framework (or other relevant institutional models)? What barriers to the full utilization of ICS exist? 1a. Are you aware of any times officials have tried to integrate ICS concepts into their frameworks, but were not successful? 58 1b. What challenges have been overcome since ICS implementation and training began? What challenges still exist, at both the national and regional levels? Probe: What needs to take place for ICS to be fully integrated and utilized? What changes need to occur, and who should be in charge? 1c. When you were first trained in ICS, were there any aspects of the training that you found to be especially challenging? Probe: Could the training have been changed to provide a more fluid, intuitive explanation of ICS? Did the concepts (terminology, principles, and country-specific examples of usage) seem relevant to local users or did they seem foreign? Primary Question 2: Is further adaptation (translation of core concepts and revision of key elements to better reflect local conditions) of the ICS is required before the materials can be utilized at the local government-level? 2a. What changes have been made to standard ICS forms, structures, and protocols to meet country requirements? 2b. What changes should be made to enhance ICS implementation? 2c. What are the main challenges you encounter when delivering training? Primary Question 3: Do you think that ICS will continue to grow and become central in your country’s disaster response system? Do you foresee ICS remaining sustainable in the long-term without external donor support? 3a. Have you been trained – or heard of trainings – in ICS by organizations other than USAID/OFDA (USFS)? 3b. Has your local or national government provided funding for ICS training that you know of? 3b. Is the country's ICS programming currently sustainable? i.e. if OFDA no longer sponsored it, would the country still use ICS? If not, what would need to change to make it sustainable? 3c. Has the partner nation established a committee responsible for advocating for and guiding the adoption of ICS practices at national and local levels? Has your local region/province/town had a voice in steering ICS practices? 3d. Did your training end up costing you money, or was it supported? Who (if anyone) supported it? As far as you know, is the cost of cascaded training reasonable for local responders and organizations, or is it cost-prohibitive? Is it geographically accessible? What obstacles to giving and receiving training exist? 3e. Is the cascaded training of comparable quality as the training offered by the USFS cadre? Do you believe the training has changed in any way without the USFS cadre? If so, are these changes improvements or impediments to the full utilization of ICS? 59 3f. Is ICS training standardized across the nation? Is implementation standardized? 3g. Are programs in place to capture and share lessons learned nationally and internationally? Have you participated in these programs? 60 Annex IV. Sources of Information Table 2: List of Program Documents Reviewed Program Documents 4th Quarter FY10 Report - OFDA Philippines Planning 1st Quarter FY11 Report - OFDA ASEAN ICS 2nd Quarter FY11 Report - OFDA ASEAN ICS 3rd Quarter FY11 Report - OFDA ASEAN ICS 4th Quarter FY11 Report - OFDA ASEAN ICS 1st Quarter FY12 Report - OFDA ASEAN ICS 2nd Quarter FY12 Report - OFDA ASEAN ICS 3rd Quarter FY12 Report - OFDA EAP ICS Activities 4th Quarter FY12 Report - OFDA EAP ICS Activities 1st Quarter FY13 Report - OFDA EAP ICS Activities 2nd Quarter FY13 Report - OFDA EAP ICS Activities 3rd Quarter FY13 Report - OFDA EAP ICS Activities 4th Quarter FY13 Report - OFDA EAP ICS Activities 1st Quarter FY14 Report - OFDA EAP ICS Activities 2nd Quarter FY14 Report - OFDA EAP Activities 3rd Quarter FY14 Report - OFDA EAP Activities 4th Quarter FY14 Report - OFDA EAP ICS Activities 1st Quarter FY15 Report - OFDA EAP Activities 2nd Quarter FY15 Report - OFDA EAP Activities 3rd Quarter FY15 Report - State ASEAN ICS 2012 Funds 1st Quarter FY16 Report - OFDA EAP Activities 2nd Quarter FY16 Report - OFDA EAP Activities 3rd Quarter FY16 Report - OFDA EAP Activities 61 Table 3: List of Country Documents Reviewed Thailand 201103 Thailand Plans Course Report 201105 LSC FSC OSC IC Course Report 201111 Thailand ICS Concept Paper 201202 Thailand TOT Course Report 2011 Thailand ICS - OFDA SOW 201102 Thailand Basic Intermediate ICS Course Report Thailand USFS Disaster Management - FY 2016 Thailand USFS Disaster Management - FY 2015 Thailand USFS Disaster Management - FY 2014 DASP Thailand USFS Disaster Management - FY 2013 DASP Thailand USFS Disaster Management - FY 2012 DASP ICSapplicationThailand - 2013 Trip Report_Thailand_5.2012 Trip Report_Thailand_8.2012 National Disaster Risk Management Plan (2015) NIMS National Incident Management System (Thai Language) ICS Planning Handbook (Thai Language) Table 4: List of Interview and Group Discussion Participants Title First name Last name Designation Organization Mr. Somboon Malak Trainer Disaster Prevention and Mitigation Academy (DPMA), DDPM Ms. Paoraumpai Janya Director DPMA, DDPM Mr. Santi Bussabongthong Plan and Policy Analyst, Professional Level DDPM Mr. Udomsak Kawnuna Chief of Disaster Prevention and Mitigation Provincial DDPM Ayutthaya province Ms. Angsumalin Angsusingha Plan and Policy Analyst, Senior Professional Level Department of Disaster Prevention and Mitigation 62 Title First name Last name Designation Organization Mr. Chalermrat Chansivanont Plan and Policy Analyst, Professional Level Department of Disaster Prevention and Mitigation Ms. Nichanan Teerakamtorn Nurse Red Cross Surin province Ms. Teetat Taotong Victim Relief Assistance DDPM Bangkok Ms. Marisa Iamsila Forensic Scientist Central Institute of Forensic Science Ms. Kwanchai Tongaatong Plan and Policy Analyst DDPM Pathumthani regional office no.1 Ms. Wantipa Chanpong Forester, Professional level Department of National Park, and Wildlife Preservation Mr. Boonprasong Nuansai Chief DDPM Khon Kean Mr. Krisda Dabpol-on Chief Fire Department, Sakol Nakorn Mr. Rittichai Thammasang Victim Relief Assistance DDPM Nakorn Pathom Mr. Mitri Sriya Deputy, Emergency Monitoring, Preparedness and Respponse Office of Atom for Peace Mr. Chaloemsak Junklang Fire instructor Nakorn Ratchasima Municipality Ms. Chakreeya Setthaseree Plan and Policy Analyst, Senior Professional Level DDPM Uttaradit Mr. Suwat Plubplaeng Plan and Policy Analyst DDPM Suphanburi Ms. Paweena Thongsagulphan Plan and Policy Analyst, Senior Professional Level DDPM Nakorn Naiyok Ms. Tuntiva Rangrongtanin Plan and Policy Analyst, Professional Level DDPM Bangkok Ms. Wongduan Mangmee Plan and Policy Analyst, Professional Level DDPM Nakorn Sawan Mr. Thomsan Chaisri Mechanic Experienced Level DDPM Nan Mr. Khamron Imnei Plan and Policy Analyst, Senior Professional Level DDPM Sukhothai Mr. Kaweewat Sukkasame Mechanic Senior Level DDPM Lampang 63 Title First name Last name Designation Organization Mr. Patcharin Luangkham Communicaiton Specialist DDPM Chiang Rai Mr. Suppapimit Paorik Director DPMRC 2 Suphanburi Mr. Rachit Sudpum Deputy Governor Yala Province Mr. Somsak Sontarapornpol Director of water management and maintainance Devision Regional Irrigation Office 16 Ms. Suttima Sanyawong Director News and Program Division Public Relation Center Region 5 Suratthani Mr. Warat Surawadee Plan and Policy Analyst, Professional Level DDPM Bangkok Mr. Dusit Pongsapipat Plan and Policy Analyst, Professional Level DDPM Bangkok Ms. Pannapa Na Nan Plan and Policy Analyst, Professional Level DDPM Bangkok Mr. Pongsatorn Sirisakorn Director Disaster Mitigation Directing Center DDPM Bangkok Actin g Sub Lt. Trakul Totham Songkla Area Command Chief Songkla Area Command/DDPM Phuket Mr. Ammat Chaitaweewong Songkla Area Command staff DDPM Regional Office No. 12 Mr. Wasan Chaitaweewong Acting Head DDPM Songkla Ms. Pastreeya Koomban Plan and Policy Analyst DDPM Phuket Ms. Aimon Emsakul Administration Officer Red Cross Phuket Mr. Chaknarong Naimolee Secretariat Ayutthaya Ruamjai Volunteer Association Ms. Supreeporn Utsaha Administration Officer, Professional Level DDPM Ayutthaya Ms. Supitchaya Sirsanroongreung Manager National Institute of Emergency Medicine 64 Annex V. Field Photos 1 IMAT Trainees work on a practical application during training at Chulchomklao Royal Military Academy in Nakorn Naiyok 65 2 2nd Generation IMAT Trainee Class in Nakorn Naiyok 66 3 THAIMAT Staff at the Staging Area next to the Area Command in Surat Thani 67 4 THAIMAT Staff manage response operations at Area Command in Surat Thani 68 5 National Command Center at DDPM Headquarters in Bangkok