INCIDENT COMMAND SYSTEM (ICS) PERFORMANCE EVALUATION INDONESIA COUNTRY REPORT This publication was produced at the request of the United States Agency for International Development. It was prepared independently by DevTech Systems, Inc. i INCIDENT COMMAND SYSTEM (ICS) PERFORMANCE EVALUATION INDONESIA COUNTRY REPORT A Performance Evaluation of the ICS System in Indonesia Covering the Assistance Provided by USAID/OFDA from 2012 August 14, 2017 USAID Contract No AID-OAA-I-15-0008 DISCLAIMER The author’s views expressed in this publication do not necessarily reflect the views of the United States Agency for International Development or the United States Government. Cover photo: Monitoring center of the Indonesia National Agency for Disaster Management in Jakarta i Contents Acronyms.......................................................................................................................................................................ii I. Evaluation Purpose ...............................................................................................................................................1 II. Evaluation Questions............................................................................................................................................1 III. Project Background...............................................................................................................................................2 IV. Evaluation Methods & Limitations........................................................................................................................3 V. Findings.................................................................................................................................................................6 VI. Conclusions.........................................................................................................................................................12 VII. Recommendations..........................................................................................................................................18 Annex I: Evaluation Statement of Work ......................................................................................................................20 Annex II: List of Indicators ...........................................................................................................................................28 Annex III: Data Collection Instruments........................................................................................................................34 Annex IV. Sources of Information ..........................................................................................................................46 Annex V. Field Photos..................................................................................................................................................48 ii Acronyms AAR After Action Review AHIMT All-Hazard Incident Management Team ADPC Asia Disaster Preparedness Center ASEAN Association of Southeast Asian Nations BFP Bureau of Fire Protection BNPB Badan Nasional Penanggulangan Bencana/National Agency for Disaster Management BPBD Badan Penanggulangan Bencana Daerah/Provincial Agency for Disaster Management DDPM Disaster Prevention and Mitigation DEC Development Experience Clearinghouse DOH Department of Health DRR Disaster Risk Reduction DOS Department of State DVI Disaster Victims Identification, a unit under the Indonesian National Police EAP East Asia Pacific FGDs Focus group discussions FY Fiscal Year GoI Government of Indonesia GIs Group interviews ICS Incident Command System IMATs Incident Management Assistance Teams IMTs Incident Management Teams INP Indonesian National Police KIIs Key informant interviews NIMS National Incident Management System OFDA Office of Foreign Disaster Assistance POC Point of Contact RFTOP Request for Task Order Proposal SEA Southeast Asia SOW Statement of Work ToC Theory of Change TOT Training-of-Trainers UN United Nations USAID United States Agency for International Development USG United States Government USFS United States Forest Service iii UFE Utilization-focused evaluation WFP World Food Program 1 I. Evaluation Purpose The goal of this evaluation is to improve USAID/OFDA’s understanding of the performance of the ICS program supported by USAID/OFDA in the EAP region. The evaluation shall focus on the effectiveness and sustainability of the USAID/OFDA funded ICS programs in Indonesia, as well as in the Philippines and Thailand. The findings of this evaluation will inform future programming decisions and adjustments to ongoing ICS programming. After five years of supporting ICS programming in a variety of regions, USAID/OFDA seeks a deeper understanding of the successes and areas for improvement in this capacity-building model. An evaluation of the performance of the ICS program in FY2017 will allow USAID/OFDA decision-makers to apply the information from this evaluation in future programming decisions related to DRR programming and the ICS program model specifically. With more than $4 million invested in the ICS program over the last five years, the program is a significant investment in resources from USAID/OFDA. As with many capacity-building initiatives, contextual factors can influence the results from the capacity intervention. The introduction of ICS into a variety of countries—all with their own disaster risks, government structures, and socio-economic factors—has led USAID/OFDA to seek more information about the factors for success and the potential for sustainability. II. Evaluation Questions This evaluation has two objectives: understanding effectiveness and sustainability. Each objective has several lines of inquiry that shall inform the evaluation design. Data was collected for all lines of inquiry in the Philippines, Indonesia, and Thailand and was analyzed to compare results from the different contexts. The first objective to evaluate effective has the following five lines of inquiry: Objective 1: Effectiveness 1. To what extent has the ICS program built the capacity of national governments to respond to disasters effectively, particularly at the initial stages of the emergency? 2. Which aspects of the ICS training program are most effective? Which aspects of the ICS training program are least effective? 3. Is the ICS targeting actors within the national government that are best positioned to effectively implement elements of the ICS system? 4. What country-level factors influence the effectiveness of the ICS program? 5. To what extent do training participants—including the different levels of the cascade training system—retain knowledge and skills transferred through the ICS trainings? The second objective to evaluate the sustainability of the program has the following four lines of inquiry: Objective 2: Sustainability 6. To what extent are countries able to integrate the ICS into their national frameworks or other institutionalization models? 7. What barriers to utilization of the ICS exist? 8. How much adaptation (translation of core concepts and revision of key elements to better reflect local conditions) of the ICS is required before the materials can be utilized at the local government￾level? 9. What factors contribute to sustainability of the ICS? Will existing investments remain viable without 2 external donor support? III. Project Background ICS Program Background Countries in the EAP region face extremely high climate-related disaster risks from typhoons (hurricanes), monsoon rains, volcanoes, earthquakes, tsunamis and other natural phenomena. To mitigate these risks, USAID/OFDA has funded a range of disaster risk reduction (DRR) activities in the region. In the OFDA context, DRR is a broad term that includes initiatives to prevent or mitigate the impact of hazards like earthquakes, floods, droughts, and storms. Investments in disaster risk reduction help make communities more resilient to disasters and better able to manage and mitigate their impacts, saving lives and reducing suffering before, during, and after events occur. While OFDA’s DRR programming in EAP focuses on context specific activities designed to meet the particular DRR needs of each individual country, host country human and institutional capacity building has been a consistent theme throughout. Developing a capability to respond to disasters more effectively using the ICS has been a key element in these efforts. ICS is a standardized incident management approach, applicable to any disaster scenario, which is designed to improve coordination and communication among the various actors and agencies involved in disaster response. ICS was originally developed in the U.S. in the 1970s, following a series of catastrophic southern California fires, when analysis revealed that inadequate management, rather than lack of resources or failure of tactics, was the main reason for ineffective response. ICS was intended to address these management weaknesses, which had caused problems with accountability, communication, planning, management structures, and integration of interagency requirements. ICS comprises a set of basic principles, an organizational structure, and an operational planning process and includes a clear chain of command, common terminology, interoperable communications, standardized training, and consistent certification requirements for disaster response staff. ICS expanded throughout the U.S. in the 1980s and 1990s, and since the early 2000s has become a primary component of the United States’ overall, presidentially mandated National Incident Management System (NIMS), where it is codified in the Command and Management section. More recently, spearheaded by the U.S., the ICS approach has also spread internationally. One of several components of OFDA’s DRR strategy in EAP focuses on increasing the capacity of national disaster management agencies to effectively respond to disasters. As part of this effort, OFDA has funded U.S. Forest Service (USFS) ICS capacity building activities across the EAP, providing training in Brunei, Burma, Indonesia, the Philippines, Thailand, Vietnam, Mongolia, the Solomon Islands, Timor Leste and Vanuatu, recently expanding activities to China, Malaysia, and Palau, and supported U.S. study tours and regional activities. Indonesia Country Program Background The combined threat to humans from natural disasters and climate change is nowhere more severe than in Indonesia. Volcanoes, earthquakes and tsunamis have gained the media’s attention but floods, droughts, storms, landslides and forest fires have posed the greatest threat to the country over the past three decades; according to the Badan Nasional Penanggulangan Bencana (BNPB), the National Disaster Management Agency, the incidence of natural disasters has increased by almost 350 percent during that time. This increase is largely due to the effects of climate change, population growth, the accelerated extraction of resources, and other factors, which has exacerbated the frequency and intensity of disasters. In 2009 there were 1,954 disaster events while 2,232 events were recorded in 2010. Natural disasters affect the lives and livelihoods of millions of Indonesians, and also have a substantial impact on 3 Indonesia’s economy. From 2004 to 2010, BNPB estimates that the economic impact of natural disasters totaled approximately $14 billion. An estimated 70 percent of Indonesia’s disasters are hydro-meteorological in nature; therefore, the incidence and intensity of disasters is expected to continue to grow due to climate change and associated effects. Natural disasters and climate change impacts will increase vulnerabilities for tens of millions of Indonesians. There are 17,000 islands in Indonesia with a majority of the 240 million￾population living near the coast. Climate change will affect a large segment of the population, threatening livelihoods, economic growth and stability of this important country and strategic ally to the United States. The ICS program in Indonesia was launched to build on the previous training and technical support provided by the USG. During Phase 1 (2004-2006) of the ASEAN-US Disaster Management Cooperation Program funded by the Department of State (DOS), GoI disaster managers were introduced to ICS through a series of workshops, a regional Basic/Intermediate ICS course conducted in Hanoi, Vietnam, and an ICS hosted study tour to the U.S. that included two GoI officials. Following the ASEAN-US Phase I activities, the USFS conducted a Basic/Intermediate ICS course for GoI disaster managers in Jakarta in 2008 through the Indian Ocean Tsunami Warning System (IOTWS). This activity was funded by USAID. From 2008 to present, the Department of Justice’s International Criminal Investigation and Technical Assistance Program (ICITAP) has conducted a series of Basic/Intermediate/Advanced ICS courses at the national and provincial level. The primary audience for these trainings is the Indonesian National Police (INP); however, at the request of the USAID Mission, ICITAP has included BNPB and other organizations in trainings when space is available. ICITAP has also provided direct training and technical support on ICS to BNPB staff and some personnel from the Badan Penanggulangan Bencana Daerahs (BPBDs) or Provincial Agencies for Disaster Management in response to requests for assistance. In 2008, the BNPB issued Regulation 10, which mandates that Government of Indonesia (GoI) response agencies use ICS. BNPB requested USG assistance in integrating ICS and other relevant NIMS components into Indonesia’s newly established disaster management system, and ICS is a primary component of BNPB’s curriculum for GoI disaster managers. Regulation 10 was updated in 2016 by Regulation 3 to further set the framework for ICS application and the basis for the provision of training courses. USAID/OFDA provided $450,000 and USAID/Indonesia provided $1.085 million to support the implementation of ICS in Indonesia from FY 2012 to 2015. The funding supported seven program activities, namely the establishment of an ICS Steering Committee, the development of a training plan and the adaptation of course materials, the completion of ICS train-the-trainer courses, monitoring initial course offerings, supporting BNPB in implementing new systems, the completion of a US study tour, and the completion of program evaluation workshops. Batch 1 MT cycle was completed in 2014 and evaluated during the Mentawai Megathrust Disaster Relief Exercise conducted in Pandang, West Sumatra. In FY 2017, USAID/OFDA provided $200,000 to continue ICS training in Indonesia. IV. Evaluation Methods & Limitations Qualitative Approach The country program is relatively new so the evaluation team applied a purposive approach in collecting data in the field to get the most comprehensive data available from the sources. Specifically, the team identified individuals from BNPB, BPBD, OFDA, USFS, and partner organizations (e.g., UN OCHA, 4 World Vision Indonesia) who have in-depth knowledge of the ICS program in the country particularly during USAID’s involvement from 2012. Because there was a targeted group of stakeholders identified, the most appropriate technique to collect data was through qualitative methods such as key informant interviews and group discussions using guided interview instruments and discussion agendas (see Annex II). The interviews and group discussions were supplemented by a review of country program documents, such as time-bound (e.g., annual and quarterly) reports, DRR assessments, and annual plans. The evaluation team also visited the Indonesia Disaster Relief Training Ground in Sentul and observed the Basic and Intermediate TOT course that was being conducted in the facility. The complete list of interview and group discussion participants is provided in Annex III while the list of documents reviews is provided in Annex IV. A main limitation about the use of qualitative methods is the inability of the approach to state generalized findings that can be objectively or empirically measured. This limitation was mitigated in the evaluation design by consistently collecting data along all lines of inquiry as outlined in the evaluation SOW. For example, to explore the effects of country-level factors on evaluating the effectiveness of the program (Objective 1), the team gathered feedback regarding this factor from all respondents. The evaluation team anticipated some variation on the information provided, where a local national would expectedly have more depth in the feedback compared to an expat or foreign national in the example for the inquiry on country-level factors. But consistently applying the lines of inquiry across all respondents was the only way for the evaluation team to draw significant outcomes or identify patterns in responses that would allow the team to conclude that findings are valid, reflective of actual conditions, and that the information gathered directly answers the specific evaluation question. Capacity Building Index The consistent application of the lines of inquiry also allowed the evaluation team to construct the ICS capacity building index, as proposed in the evaluation work plan. The team developed the index by examining a series of factors or elements that lead to the effectiveness and sustainability of the Indonesia ICS program and rating each factor along a 1-4 scale with the following working definitions: 1. This element is Present but in a condition which contributes only to a Very Limited extent in achieving its intended purpose. Considerable External Assistance is required to develop or improve this function. 2. This element is Present but in a condition that makes a minimum, but somewhat Adequate contribution to achieving its intended purpose. Some External Assistance is required in connection with this function. 3. This element is Present and in a condition which makes a Generally Satisfactory contribution to achieving its intended purpose. Some External Assistance is required to ensure that the purpose intended is achieved and sustained. 4. This element is Present and in a condition which makes a Clearly Satisfactory contribution to achieving its intended purpose. No External Assistance is required to ensure that the intended purpose is achieved and/or sustained. Results of the index are presented in more detail in Section VI: Conclusions. Retained Knowledge Survey During the evaluation design development stage, DevTech proposed doing a retained knowledge survey in all three countries to gather objective and observable measures of how ICS lessons are kept. During the field work, the evaluation team inquired with the ICS implementing partner agency in each country if trainee records are kept to determine if the team could administer the survey. Unfortunately none of 5 the IP agencies in Indonesia (BNPB) and Thailand (DDPM) kept comprehensive lists of past trainees so the survey could not be done. Other limitations include: 1. Attrition. Some past trainees might have been reassigned or might have left their units during the time of their ICS training. This limited the evaluation team’s ability to contact them for interviews or surveys. 2. Country selection. The three countries were selected by OFDA beforehand and thus the context when reporting findings will only be in relation to these countries, individually or collectively. Findings are not representative of the program in the entire EAP region. 3. Hawthorne bias. This effect is the tendency of a respondent to provide information based on their awareness of being observed or that an evaluation is being conducted. This effect often occurs in externally-funded assistance programs and beneficiaries respond with the intent of not being critical or with the expectation of receiving further support rather than providing candid assessments of the program. The evaluation team has experience in identifying this effect during data collection when trends in responses become evident. The team will apply proven social science techniques (e.g., probes) to mitigate this effect. Where possible, the team will also cross-reference possible biased responses with other sources of data such as trainee records as a confirmation measure. As much as possible, the team used discrete and observable data when doing this triangulation exercise and not a different set of information from another respondent to avoid the “he said, she said” dilemma that can happen in KIIs and FGDs. 6 V. Findings The findings, conclusions, and recommendations of this country evaluation report and the larger evaluation of the ICS EAP program needs to be guided by an analytical framework so that the information obtained will be evidence-based and utilization-focused to inform future programming decisions. The analytical framework the evaluation team applied is illustrated in Figure 1, which shows the hypothesized logic model of the ICS program along the following levels: Objectives Increased Capacity Inputs Training Outputs Trainees/ Cadres Outcomes Expansion Figure 1: ICS Program Logic Model • Inputs. The program’s main input is the design and delivery of the ICS training courses: basic, intermediate, position courses, and advanced courses. For a trainee to be eligible to become a master trainer, s/he should also complete the Training for Instructors (TFI) course. • Outputs. The delivery and completion of courses results in the increase in the number of trained personnel within the partner nation government agencies, as well as some private sector emergency responders. The advancement of some participants to become a master trainer (MT) also creates a cadre of future ICS trainers in the country. • Outcomes. Over time, the increase in the number of additional trainees as well as the advancement of earlier trainees lead to the expansion of the application of the ICS practices among the community of emergency responders. Expansion is accelerated at this level through the Training-of-Trainers (TOT) program where MTs lead ICS trainings. Spillover effects can also be expected as trained personnel apply lessons learned back at their respective units and through interactions with non-ICS trained colleagues. • Objectives. Ultimately, the ICS program’s objective is to increase the capacity to respond to disasters using the ICS model. While USAID/OFDA defines capacity as the ability to carry out the stated program objectives “to effectively and efficiently respond to disasters using the ICS model,” the understanding of capacity needs to be contextualized for each country in the broader evaluation because of differences among the three 7 countries in local conditions. USAID/OFDA also recognizes that “the objectives of each of the iterations of ICS programming vary depending on the disaster profile and baseline capacity of the country in which the program is implemented.” Thus, capacity as understood for the Indonesia program will be different from how it is understood in the Philippines and Thailand for the broader evaluation. Capacity For Indonesia, the evaluation team defined capacity in three dimensions. First, capacity is recognized as the usage of ICS/NIMS concepts and procedures at multiple levels of the GoI. The second dimension of capacity is the perceived improvements in disaster response using ICS. The third dimension of ICS capacity is the level of existing support for the continued use of the ICS model nationwide. The causal pathway that leads to the program objectives of the increased capacity to respond to disasters using the ICS model also needs to control for other factors not directly related to the inputs of training delivery but nonetheless influence program objectives. In Figure 1, these are identified as external factors such as governance frameworks and policies at the national or local levels. In the Philippines, for example, the Local Government Code decentralized governance and empowered different local government units. The code led to the increased responsibility and accountability in operations, including disaster response. Another external factor to consider is the presence of other disaster relief mechanisms or programs that might have confounding effects on the ICS program in the country. For example, the integration of ICS and the support provided by the Government of New Zealand can be seen in BNPB Regulation 3, the GoI’s initiative to institutionalize disaster management at the local level. Regulation 3 is expected to increase Indonesia’s capacity in disaster relief operations but the effect of ICS on this outcome will be difficult to disentangle from the effects of the New Zealand government as well as other factors. These effects have to be taken into consideration, if not controlled for, in an evaluation to have more precise assessments of the effects of the direct interventions. Findings Objective 1: Effectiveness 1. To what extent has the ICS program built the capacity of national governments to respond to disasters effectively, particularly at the initial stages of the emergency? As stated earlier, capacity is defined along the three dimensions of usage at various GoI levels, perceived improvements in disaster response, and continued support for nationwide implementation. ICS trainings have been conducted at a pace of approximately 200 participants per year so there has been a steady expansion of ICS usage within the GoI, largely within the BNPB and BPBDs but also among other agencies. However, there have been challenges in cascading the trainings mainly because the financial support to provide these courses has not been included in the BNPB budget. The expansion of the country’s master trainer cadre is also in its very early stages. Under the “Building Capacity for Capacity Building” initiative, 25 out of the first batch of 60 trainees graduated as MTs. The second batch had 24 MTs and the third batch will start in March 2017. This upcoming batch will include trainees from three of the more fire-prone provinces of Sumatra and four from the most fire-prone provinces in Kalimantan. The batch will also represent various sectors including the military, police, private industry (palm oil), forestry, and the environment. The courses will be co-led by the Indonesian MTs and USFS training officers. There has been an increase in the perceived effectiveness of ICS among the disaster response community, particularly when information was shared on how it was applied in certain disaster response operations. For example, information about the application of ICS during the 2014 East Java volcano 8 eruption was immediately shared among all response institutions in the country. The perception was stronger in the BNPB and BPBDs but was also evident across all stakeholder groups, such as the Indonesian army and local government of Yogyakarta. As mentioned earlier, the budget needed to provide trainings has not been included in the overall BNPB budget and this can potentially weaken the support needed to expand training and build capacity. The situation is due to the GoI’s decentralization efforts, which has presented challenges to the local ICS program and other initiatives within the government. Under the decentralization plan, the GoI budget development process is part of policy discussions at the national level of the government. But because ICS is recognized as an operational initiative, the needed support will have to come from the local levels of government. 2. Which aspects of the ICS training program are most effective? Which aspects of the ICS training program are least effective? In a general sense, the most effective aspect of the ICS training program was that it provided an organized structure on how individuals and agencies should respond to a disaster event. Prior to the implementation of the ICS model, stakeholders felt disaster response was disorganized and communication systems among agencies were poor. This perception was stated by 33 percent of respondents but the evaluation team assumes this sentiment is shared by the majority of respondents as other shared conceptually similar perceptions, such as improvements in response times and ICS use among partner GoI agencies. ICS provided an effective method to communicate among responders, establish roles and responsibilities, and capitalize on resources. Additionally, ICS led to an institutionalized change in behavior even just in the way responders think about managing emergencies. The community of responders with ICS training now has a greater appreciation for planned meetings and briefings to discuss what to do. Prior to ICS, responders also had perceptions that response failures were caused by limited resources for relief operations. Because of ICS, they later understood that failures were due largely to the absence of an organized structure and that the model provided a system to allocate resources more effectively and efficiently. For example, a responder in the Pidijaya earthquake stated that coordination among groups was challenging and it was later realized that the situation was due to a “lack of organizational structure” among officials who were either new or not very experienced in ICS. While there is no specific aspect of the ICS program that can be identified as least effective, there are parts of it that cannot be fully implemented mainly because those are not really reflective of country￾specific conditions. The most obvious element is the assignment of the Incident Commander or IC. Generally, the ICS model advocates for assigning the IC position to the most qualified individual and/or the most experienced in disaster response because operations and tasks are very specific. In Indonesia, however, there is the general practice of assigning the leadership position to the most senior official or the individual with the highest designation. Generally, the governor would have the authority to appoint the IC and in past cases, some governors have selected ICs who do not have ICS training. In other cases, ICs were selected from the BPBD, INP, or the military and they would apply the response systems they learned from their agencies. There have been successful operations under this command structure because ICS management and communication systems were followed. In the 2014 volcano eruption at East Java, for example, the responding team consisted of previous trainees who applied ICS principles to communicate data and coordinate operations. The data allowed the governor to make informed decisions and coordinate efforts with the BPBD, military, and other responding agencies. Though ICS was not fully applied, the event led to an appreciation of ICS among other stakeholders. But generally there can be potential challenges in response operations if an IC has not received ICS training. 9 3. Is the ICS targeting actors within the national government that are best positioned to effectively implement elements of the ICS system? The concept of “targeting actors” within the national government should be understood in two perspectives to effectively implementing elements of the ICS system. First, from an operational perspective the program has targeted the right actors within the BNPB, BPBDs, and other GoI agencies such as the Ministry of Agriculture. The program’s engagement with key BNPB and BPBD individuals established a growing training environment where courses are effectively planned in terms of identifying participants, developing curricula, scheduling, and selecting instructors. Half of the stakeholders the evaluation team interviewed mentioned the importance of champions during the early stages of ICS implementation. Specifically, Pak Wisnu was a strong advocate of integrating ICS in the nationwide response system. The importance of individual champions or proponents can still be seen today. In Yogyakarta, Mr. Danang, an ICS-trained MT, has been leading implementation efforts through training and exercise events in the small region. The governor has also provided support by issuing regulations that ICS principles should be adopted in operational planning so the emergency community in Yogyakarta has developed incident action plans (IAPs) for the province. From a policy perspective, the Indonesia program has to recognize the “top-down” structure in the GoI and in the country’s overall working environment. In general, any initiative implemented in the country will be accepted by the broader beneficiary community if it has the support of the community’s leaders or highest officials. Thus, the ICS program will need to engage with individuals at the highest possible levels of the bureaucracy so that the implementation of the ICS system will be accelerated. The BNPB chief can implement regulations integrating the use of ICS in response operations. But these regulations are only effective within the BNPB; local government leaders, such as governors, mayors, and village heads (lurah) are not required to integrate it within their response system. The “top-down” structure along with a decentralized system of government (see Evaluation Question 7 below) has to be considered in accelerating the application of ICS in the country. 4. What country-level factors influence the effectiveness of the ICS program? There are two main factors at the country-level that influence the effectiveness of the ICS program. First, the overall approach to disaster response in the international community uses the United Nations (UN) cluster approach, where UN and non-UN organizations are grouped into clusters, e.g., shelter or health, under a coordination mechanism to establish clear points of contact and make communication and coordination more effective and efficient. The model is also applied where international assistance is required and where the UN has an active presence. The UN cluster system was rolled out in Indonesia following the ASEAN decision to use it as the on-the-ground system for disaster operations, although it is also recognized as a global initiative. In Indonesia, there was a need to understand how the two systems can work together given that the two systems target different actors in disaster response operations. Two years ago, regional stakeholders held a meeting in Bangkok to explore where there are “crossover areas” between the two systems. For example, the use of the multi-agency coordination (MAC) system to replace clusters was discussed since MACs provide the guidance for the allocation of resources, prioritization, and agency coordination. These types of discussions have been ongoing and stakeholders hope to find a resolution soon, particularly as ICS has now become the standard for ASEAN and other stakeholders (e.g., Red Cross) have adopted the system. One of the more significant feedbacks from respondents was the need for ICS and its resources to be more contextualized to the Indonesian setting. This feedback was shared by 67 percent of respondents. The idea of “contextualization” was very broad. At a higher programmatic level, respondents said that 10 ICS needs to understand the dynamics of the local culture and working environment, specifically the “top down” structure mentioned in Evaluation Question 3 earlier. On more specific levels, ICS resources such as forms, manuals, and other documents not only need to be translated to Bahasa, but also need to be revised to use terminologies that would be easier for stakeholders to accept. For example, the term “command” in incident command system can mean a hierarchical structure, particularly from personnel at the operational levels of the GoI. 5. To what extent do training participants—including the different levels of the cascade training system—retain knowledge and skills transferred through the ICS trainings? The most effective way to retain knowledge gained from ICS trainings is the continued application of what was learned. The application can only be done in two scenarios: in actual incidents and planned events. In terms of actual incidents, there have been identified cases where ICS was successfully applied. Some units within the DVI have applied at least certain principles of ICS. The DVI is a unit within the INP and ICS principles are applied where possible in police operations, not just natural disasters. One planned event, the Mentawi Megathrust Disaster Relief Exercise (MMDRIRex) was a planned event but regular exercises have not been conducted. The MT training has been more challenging, with only 49 members right now. The program aims to have an additional 25 members in the cadre by the end of 2017. The logistics needed to provide high-quality MTs to 6,000 inhabited islands for Basic and Intermediate courses are also challenging because of the efforts to convene trainees, send trainers, and arrange travel and accommodation, among other things. According to one MT respondent, the number of trainers is low because 50 percent of the trained group failed the retention test administered two months after the training. Trainers observed a higher passing rate of 84 percent when the retention test was administered five days after training so the decline suggests some form of fadeout occurs among the MT Cadre over time. The decline also suggests that a range of resources might be necessary, from the availability of materials and course documents to ICS exercises in planned events. Objective 2: Sustainability 6. To what extent are countries able to integrate the ICS into their national frameworks or other institutionalization models? In 2008, the BNPB incorporated ICS principles in Regulation 10, the agency-wide regulation document for disaster response. It was updated in Regulation 3 in 2016 and ICS principles were retained. However, the decentralized structure of governance limits the enforcement of the regulation within the BNPB only. The BNPB, with funding from the New Zealand government, is developing the National Disaster Response Framework (NDRF). Its development currently involves multiple stakeholders in an iterative and consultative process to develop a comprehensive response plan. However, none of the members of the NDRF team have ICS training and only two of the MTs have been engaged with the team during the development process. 7. What barriers to utilization of the ICS exist? There are two significant barriers to the utilization of ICS among emergency responders. The first is the decentralization of government functions. In 1999, the GoI implemented a decentralized system of government that provided local government officials – governors, mayors, and village heads – with more authority and autonomy in governing their administrative areas. Local officials have more knowledge 11 about needs and issues in their areas so decentralization was seen as an effective way to improve service delivery to the people. However, a negative outcome of the plan was that local officials were not bound to enforce policies and regulations coming from the national level. Thus, regulations created by the BNPB, which included the application of ICS principles, were not always enforced at the local level and thus limited the utilization of ICS. This was commonly known among the disaster response units within the GoI. For country comparison purposes, it is worth noting that decentralization in Indonesia had the opposite effect of what resulted in the Philippines. In 1991 the Local Government Code of the Philippines established a decentralized form of government and emphasized greater accountability to local officials along with the autonomy in governance. Respondents from the evaluation of the ICS program in the Philippines (forthcoming) suggested that the emphasis on accountability served as an incentive for local officials to address disaster preparedness including, by extension, the adoption of the ICS model in some cases. The second most significant barrier to utilization is the turnover of personnel as a result of the standard reassignments and rotations in the GoI. There are 34 provinces and over 500 districts in Indonesia. Each province is estimated to need 10 ICS-trained individuals so approximately 340 local officials that will need ICS training. Additionally, there are 10 agencies in the GoI and certain units such as the BNPB, BPBDs, INP, and military are involved in disaster relief operations. Thus, the planning and logistics to deliver courses at the national and local levels need to be at a considerably fast pace to reach all the targeted individuals at the shortest possible time. However, according to interview respondents from the BNPB and BPBDs, the utilization of ICS as well as the retention and transfer of knowledge among the trained personnel are negatively affected when trained individuals are transferred or rotated to other units. According to one respondent, some trained employees have stayed for as short as three to six months before being transferred to another office. 8. How much adaptation (translation of core concepts and revision of key elements to better reflect local conditions) of the ICS is required before the materials can be utilized at the local government-level? As with other country programs, ICS resources, such as curricula and manuals, were modified from their original versions from the USFS. Compared to Thailand and the Philippines, there was relatively more adaptation required prior to implementation in Indonesia. For example, local government leaders found the standard ICS forms to be irrelevant to their community because they understood it to be applied at the national level so an MT the evaluation team interviewed stated he had to revise and simplify the forms. The evaluation team also observed the emphasis on making resources more “Indonesian” although there was no unified and consistent response from local interviewees on what this meant. But the most significant point of contextualization made by local respondents was that the leadership structure of assigning the most senior official, not the most experience in ICS, as the IC. This was shared by approximately 44 percent of respondents. 9. What factors contribute to sustainability of the ICS? Will existing investments remain viable without external donor support? The decentralization plan has also affected the ICS program in the budgeting process. During the early stages of implementation, activities were rolled out at the GoI without an MOU between agencies. For example, the ADPC operated a training center at the BNPB facility in Sentul without an agreement. But instead of finding a solution to the situation, leadership at BNPB stopped all trainings and the progress of the operations was affected. Now that mistakes in the processes were revealed, agencies have been very careful and slow in moving forward with planned activities. Since the early stages of implementation, there have been improvements in the decentralization process 12 as it relates to the funding of ICS training. The Ministry of Home Affairs (MOHA), has released an official letter requiring governors to provide funding for disaster response training, including ICS. Provincial governments – such as the ones in Central Java, East Java, and Jakarta - have used their own funds to conduct trainings as well as fund ICS implementation at the district and sub-district levels. With the combination of internal funding and the expansion of the MT cadre, the sustainability of ICS without external support will be achievable. VI. Conclusions The evaluation team’s conclusions presented in this section are tied to the findings in Section V through the establishment of benchmarks to reflect an “if-then” linkage. Per the evaluation SOW, the evaluation team collected findings along the evaluation questions or lines of inquiry. With the collected findings, the team assessed current conditions and mapped the information against capacity benchmarks that were developed in the evaluation design stage and submitted to OFDA. This approach responds to the SOW, which specifically states that “the evaluation team must work with USAID/OFDA and USFS to map benchmarks related to the ICS programming that link to levels of capacity-building. This mapping should help to understand what essential inputs and processes are required at each level to achieve capacity￾building benchmarks in various country contexts.” The evaluation team scored each benchmark on a 1-4 scale to calculate the capacity building index. As an example, the evaluation team collected field data to see if findings showed the usage of ICS/NIMS at various levels of the GoI, perceived ICS effectiveness, and support for implementation for the evaluation question, “To what extent has the ICS program built the capacity of national governments to respond to disasters effectively, particularly at the initial stages of the emergency?” With the collected findings, the team then mapped the three elements against the following benchmarks to assess the level of capacity: • ICS has been used successfully to manage events, either planned or no-notice. • Multiple stakeholder agencies have adopted and used ICS. • Formal training programs are in place and funded. The results of this benchmarking exercise to identify conclusions are discussed below and a summary of the index scoring is provided in Table 1. The rows list the benchmarks identified to evaluate each line of inquiry and the “Score” column is the evaluation team’s ranking of that benchmark using the 1-4 scale. The “Indicator” column lists the indicator number using the list of indicators the evaluation team proposed to OFDA for assessment. The full indicator list is presented in Annex VI. Objective 1: Effectiveness 1. To what extent has the ICS program built the capacity of national governments to respond to disasters effectively, particularly at the initial stages of the emergency? Benchmarks Score Indicator ICS has been used successfully to manage events, either planned or no-notice. 1 8 Multiple stakeholder agencies have adopted and used ICS. 2 7 Formal training programs are in place and funded. 3 6,37 13 The first benchmark to assess capacity is the successful application of ICS in planned and unplanned events. Except for the internal operations of the Disaster Victims Identification (DVI) unit of the Indonesian National Police (INP), the evaluation team found no compelling examples of meaningful applications of ICS in planned events. The DVI sets up an ICS post when called in to respond to an incident but even this process significantly limits ICS use since other INP units apply the response system familiar to them. The second benchmark is the adoption of ICS across multiple stakeholder agencies. ICS principles have been implemented in the BNPB, BPBD, DVI (INP), Red Cross, World Vision Indonesia (WVI), and the different provinces (e.g., Yogyakarta) but level of adoption has been limited due to the different factors discussed in Section V such as “top-down” approach, and limited budget, among other things. The third benchmark is the presence of a formal training program with sufficient funding to ensure continued operations. This is clearly present with the full suite of ICS courses being provided including the TOT and TFI tracks as well as the funding at BNPB that is then released for disaster response operations at the local government level by virtue of the MOHA letter. The funding is available at the national government and the improvements within the mechanisms for disbursement indicate that the program can be sustained with little to no external support. However, the current pace of course delivery has not met the need at the GoI and provincial levels. 2. Which aspects of the ICS training program are most effective? Which aspects of the ICS training program are least effective? Benchmarks Score Indicator Training courses are effective at improving ICS knowledge and organizational capacity. 1 15,16,17 A Lessons Learned program is in place to capture best practices for the country and remedy shortfalls. 1 44 The ICS program helps trainees understand the effectiveness of having an organized management structure with common terminologies when responding to a disaster. This understanding has been the most effective aspect of the ICS program. But seeing the effectiveness of the management structure in actual, real-world incidents has been limited. The inability of the program to address the practice of selecting an individual as an IC without ICS training might also limit the effectiveness of future response operations. Additionally, the program does not have a “lessons learned” mechanism to capture best practices and address program shortfalls. 3. Is the ICS targeting actors within the national government that are best positioned to effectively implement elements of the ICS system? Benchmarks Score Indicator ICS course offerings are diverse, include training for Senior Officials as well as Position-Specific sessions. 2 6,20 Program Directors have the authority to implement ICS activities at national and local levels. 1 19 Women receive ICS training at comparable levels to men. 2 1,2,3,4,5 Personnel at multiple levels of government and stakeholder agencies have received training. 2 20 14 At the operational level, personnel at multiple levels of government agencies and stakeholder organizations have received training. The program has also established a fully-functioning training program and is able to schedule courses, identify trainers, select participants, and plan logistics. But these courses are not available nationwide and are only available in certain areas depending on where trainings have been planned. The program still has to solicit buy-in from the higher levels of the GoI and the provinces. Executive courses are available but the program will need to be more active in obtaining support at both the national level and among governors. BNPB regulations exist, but local and agency leadership is needed to fully implement it. 4. What country-level factors influence the effectiveness of the ICS program? Benchmarks Score Indicator No major cultural barriers to ICS acceptance. 1 33 Acceptance of ICS as a best practice within national government. 2 26 Acceptance of ICS as a best practice within regional / local governments. 2 27 Contextualization will have to be addressed with wide-ranging initiatives since it is an issue that spans multiple facets of the disaster response community, from the policy level to the development of contextually appropriate forms, materials and resources. ICS is generally accepted within the leadership and operational staff of BNPB, among some GoI partner agencies, and in some provinces such as Jakarta and Central Java. But the program will need a champion or a core group of key supporters to accelerate the application of ICS. 5. To what extent do training participants—including the different levels of the cascade training system—retain knowledge and skills transferred through the ICS trainings? Benchmarks Score Indicator Training recipients retain knowledge of ICS concepts following training. 2 15 In-country Master Trainer Cadre has the knowledge, skills, and experience to provide high quality training. 2 42 Knowledge retention is hindered by the limited ability to use ICS in practical situations, particularly in planned events. In real-world scenarios, the potential to mix ICS principles with other response systems exists. The program also does not leverage planned events in the country. There have been previous discussions about using ICS and IMT teams prior to the Eid al-Fitr holiday, similar to how ICS has been applied to planned events in the Philippines like the Feast Day of the Black Nazarene. There is a team of 49 MTs and the program should see an increase in the MT cadre with additional trainings. The knowledge and skills have been developed but they need more practical experiences, either in more response operations or planned events, to be a more effective cadre of responders and trainers. Objective 2: Sustainability 15 6. To what extent are countries able to integrate the ICS into their national frameworks or other institutionalization models? Benchmarks Score Indicator Laws codifying the use of ICS in disaster management in place. 2 9 Doctrine guiding the integration of ICS with country's disaster management system is developed at the strategic level. 1 9 ICS has been successfully integrated and used at multiple levels of government. 1 8,10,30 ICS has been incorporated in two iterations in Regulation 10 (2008) and Regulation 3 (2016) in the BNPB. But enforcement has been limited due to the decentralized structure of governance. The NDRF will be a nationwide disaster response strategy that was funded by the government of New Zealand. The current efforts to develop the framework involve various stakeholders in a consultative and iterative process. However, while there are two MTs that have been involved in the framework process, most members of the NDRF team do not have ICS training. Within the multiple levels of the GoI, ICS integration has been strongest in the BNPB and in Jakarta in general. But efforts to implement ICS in Yogyakarta because of the governor’s support and the local ICS team’s active implementation can potentially lead to an increase in the awareness about the ICS model. 7. What barriers to utilization of the ICS exist? Benchmarks Score Indicator Few reports of pushback to adoption of ICS practices. 2 12,31,32,33 Country has established a committee to further ICS acceptance and use. 3 39 Opportunities exist to train on, practice, and apply ICS regularly. 1 29 As mentioned earlier, the adoption of ICS processes has been affected by institutional factors such as the “top-down” structure. This has resulted to some situations where the BNPB remains unable to enforce regulations to other government units, specifically the provincial level. There is an ICS Steering Committee that is being operationally led by Ms. Regina Rahadi, one of the earlier MTs in the Indonesian ICS community. The steering committee regularly meets to discuss how ICS adoption can be expanded locally and to discuss what needs to be done for future expansion. This committee will need to continue to its operations and be more engaged with GoI stakeholders in exploring opportunities to expand ICS use. As mentioned earlier, there have been limited opportunities for training and application and not much can be expected in the near term until the higher levels of leadership in the BNPB and key agencies provide support for exercises and more courses. 8. How much adaptation (translation of core concepts and revision of key elements to better reflect local conditions) of the ICS is required before the materials can be utilized at the local government-level? Benchmarks Score Indicator 16 Country-specific ICS Forms, Doctrine, and guiding documents have been provided to practitioners. 2 34,35 Country-specific examples have been used in ICS Training Courses. 1 36,35 Course materials, manuals, and other resource documents have already been translated prior to the delivery of courses. Standard USFS models are also being used for training. But these resources have to be disseminated further among the stakeholder organizations. 9. What factors contribute to sustainability of the ICS? Will existing investments remain viable without external donor support? Benchmarks Score Indicator Local/Regional/Country Governments have funded ICS training and/or exercise events. 2 38 ICS Master Trainer Cadre is able to meet training demands nation-wide. 2 5,41,42 Training and implementation are standardized across the country. 2 43 USAID has provided funding to World Vision Indonesia and some provinces have provided funding for trainings. But internal funding has grown since the implementation of the decentralization plan. The national government funded the construction of the DRTG in Sentul and the BNPB has allocations to provide BPBDs with funding for ICS training. The MOHA requirement for provincial governments to provide funding for disaster response, including ICS, will help achieve the use of ICS without external donor support. The current MT cadre does not have enough members to meet training demands nationwide and that is one of the main reasons why USFS personnel have continued to play an active training role in the country compared to other country programs like the Philippines where USFS trainers have scaled back their training delivery functions. The evaluation team concludes that the combination of internal funding and the expansion of the MT cadre will achieve ICS sustainability without external support. But while the team saw indications of improvements from initial conditions, significant progress needs to be seen from both elements, particularly the training. Table 1 below presents the overall score of the Indonesia country program based on the evaluation team’s findings and conclusions. The Effectiveness score is 24 out of a highest possible score of 56 while the Sustainability score is 19 out of 44. This leads to an overall index score of 43 out of 100. Table 1: Capacity Building Index Results Objective 1: Effectiveness To what extent has the ICS program built the capacity of national governments to respond to disasters effectively, particularly at the initial stages of the emergency? Score Indicator ICS has been used successfully to manage events, either planned or no-notice. 1 8 Multiple stakeholder agencies have adopted and used ICS. 2 7 Formal training programs are in place and funded. 3 6,37 17 Which aspects of the ICS training program are most effective? Which aspects of the ICS training program are least effective? Score Indicator Training courses are effective at improving ICS knowledge and organizational capacity. 1 15,16,17 A Lessons Learned program is in place to capture best practices for the country and remedy shortfalls. 1 44 Is the ICS targeting actors within the national government that are best positioned to effectively implement elements of the ICS system? Score Indicator ICS course offerings are diverse, include training for Senior Officials as well as Position-Specific sessions. 2 6,20 Program Directors have the authority to implement ICS activities at national and local levels. 1 19 Women receive ICS training at comparable levels to men. 2 1,2,3,4,5 Personnel at multiple levels of government and stakeholder agencies have received training. 2 20 What country-level factors influence the effectiveness of the ICS program? Score Indicator No major cultural barriers to ICS acceptance. 1 33 Acceptance of ICS as a best practice within national government. 2 26 Acceptance of ICS as a best practice within regional / local governments. 2 27 To what extent do training participants—including the different levels of the cascade training system—retain knowledge and skills transferred through the ICS trainings? Score Indicator Training recipients retain knowledge of ICS concepts following training. 2 15 In-country Master Trainer Cadre has the knowledge, skills, and experience to provide high quality training. 2 42 Effectiveness Total: 24 out of 56 Objective 2: Sustainability To what extent are countries able to integrate the ICS into their national frameworks or other institutionalization models? Score Indicator Laws codifying the use of ICS in disaster management in place. 2 9 Doctrine guiding the integration of ICS with country's disaster management system is developed at the strategic level. 1 9 ICS has been successfully integrated and used at multiple levels of government. 1 8,10,30 What barriers to utilization of the ICS exist? Score Indicator Few reports of pushback to adoption of ICS practices. 2 12,31,32,33 Country has established a committee to further ICS acceptance and use. 3 39 Opportunities exist to train on, practice, and apply ICS regularly. 1 29 How much adaptation (translation of core concepts and revision of key elements to better reflect local conditions) of the ICS is required before the materials can be utilized at the local government-level? Score Indicator Country-specific ICS Forms, Doctrine, and guiding documents have been provided to practitioners. 2 34,35 18 Country-specific examples have been used in ICS Training Courses. 1 36,35 What factors contribute to sustainability of the ICS? Will existing investments remain viable without external donor support? Score Indicator Local/Regional/Country Governments have funded ICS training and/or exercise events. 2 38 ICS Master Trainer Cadre is able to meet training demands nation-wide. 2 5,41,42 Training and implementation are standardized across the country. 2 43 Sustainability Total: 19 out of 44 ICS Capacity Index: 43 out of 100 VII. Recommendations Based on the finding and conclusions presented in Sections V and VI, respectively, the evaluation team proposes the following recommendations for OFDA’s consideration: Recommendation 1: Accelerate Basic and Intermediate Training Indonesia is a logistically challenging place to conduct nationwide trainings with over 6,000 inhabited islands. However, these conditions also make the case to accelerate trainings along with the fact that the country experiences numerous natural disasters in a year. The Basic and Intermediate courses do not necessarily require a classroom-structure where trainees have to attend. The country OFDA team had discussions with the Asian Disaster Preparedness Center (APDC) and the BNPB to explore the possibility of providing online learning courses for the Basic and Intermediate levels and the OFDA team should revisit this idea. Recommendation 2: Continue the ICS-Cluster Discussions The application of the ICS model along with the UN cluster approach is a challenging situation to face but the OFDA team should continue its discussions with OCHA and other stakeholders on identifying areas where (components of) both systems can be applied and possibly support the GoI to develop a response structure for Indonesia where both systems are co-implemented because each system is targeting different response roles with different response agencies or actors. The discussions need to be framed with a common understanding of the purpose and function of ICS and Cluster Coordination mechanisms, their unique roles, and how they can work together among agencies, as there has been confusion. There have been misunderstandings of what ICS is especially in comparison to the Cluster system. ICS is not meant to be a replacement of the Cluster system. Recommendation 3: Engage Higher Authorities The “top-down” structure is an institutional barrier that can be addressed by engaging the highest possible authorities in the GoI and soliciting their buy-in to the ICS model. Finding a champion at the highest levels of the government agencies involved in disaster response will encourage the acceptance of ICS among all staff. Their buy-in can be gained through various mechanisms. The Program has Executive-level courses but more engaged discussions and dialogues with key officials might be more effective. The evaluation team acknowledges this is a complicated issue and it would this require a coordinated effort where senior staff from BNPB and the BPBDs can initiate communications with high officials from the GoI. USFS and OFDA can then present high-level discussions on the advantages of ICS 19 application. Recommendation 4: Conduct Exercises with Planned Events All trainers and disaster response specialists interviewed by the evaluation team throughout the region stated that exercises are crucial to maintaining the knowledge gained in ICS. The OFDA team in Indonesia should encourage the USFS trainers and the BNPB to revisit the idea of conducting ICS exercises in public holidays, large-scale sporting events, and other events where response systems might be needed and can be applied. Recommendation 5: Develop Intermediate-Term Performance Targets The country ICS program is reaching its eighth year in implementation. While there have been institutional issues that have affected operations up to this point, there are feasible plans mentioned above to expand the adoption of ICS in the country and increase capacity at a more accelerated pace. So that these plans can be achieved and measured objectively, OFDA should develop a performance monitoring plan over the next two years with indicators that have ambitious but achievable performance targets. • At the training or Input level, the program should set the higher course completion targets to meet the demands for trained personnel at both the local government and GoI agencies. • At the Output level, the program should not just increase the number of trained personnel but also ensure knowledge is retained through the continued application of pre/post testing and the regular practice of ICS in planned events, which relates to the Outcome level below. • At the Outcome level, practices should include planned events and the continued documentation of ICS in real-world scenarios. Lastly, the program should dedicate resources to regularly track performance, from information of individuals trained at the Basic courses to the use of ICS in large-scale, real-world scenarios. This evaluation was limited in the ability to examine previously collected project data because a performance plan was not available. The development and implementation of a plan will contribute significantly to the documentation of capacity building and sustainability of ICS in the future. 20 Annex I: Evaluation Statement of Work RFTOP SECTION C – STATEMENT OF WORK C.1 PURPOSE USAID/OFDA seeks to award a contract to evaluate the effectiveness and sustainability of the Incident Command System (ICS) programming in the East Asia and Pacific (EAP) region2. The aim of this evaluation is to improve USAID/OFDA’s understanding of the performance of the ICS program supported by USAID/OFDA in the EAP region. The evaluation shall inform future decision-making and contribute to the evidence-base on Disaster Risk Reduction (DRR) programming. The evaluation will focus on the ICS programming in the Philippines, Indonesia, and Thailand. C.2 BACKGROUND: USAID/OFDA DRR programming in EAP Many countries in the EAP region undergo seasonal periods of increased hydro-meteorological activity, experiencing cyclones and monsoon rains, which can increase the risk of floods and landslides and result in significant damage. Several EAP countries situated along the Pacific Ring of Fire experience volcanic activity, as well as earthquakes and associated tsunamis. To decrease the potential risks from these disasters, USAID/OFDA funds a range of disaster risk reduction (DRR) programming throughout the EAP region. USAID/OFDA has focused its DRR activities in the region on strategic, context-specific programs designed to meet particular risk reduction needs in each country, with capacity building as a consistent theme throughout all programs. USAID/OFDA provided nearly $34 million in FY 2013 and approximately $38.3 million in FY 2014 for DRR projects throughout EAP, including programs that integrate DRR with disaster response. More information on USAID/OFDA’s DRR programming can be found in Section J. Background to the ICS model ICS is a standardized incident management concept that is applicable to any hazard or disaster scenario. It is designed to improve coordination and communication among various national government agencies and actors with the aim of expediting emergency response. ICS was developed in the 1970s following a series of catastrophic fires in southern California that resulted in deaths, injuries, and considerable property damage. The personnel assigned to determine the causes of this disaster discovered that response problems could rarely be attributed to lack of resources or failure of tactics; instead, response problems more frequently resulted from inadequate management than from any other single reason. Weaknesses in incident management were often due to lack of accountability, poor communication, lack of an orderly planning process, absence of a predesigned yet flexible management structure, and the absence of a way to integrate interagency requirements into the response effectively. To respond to these failures, the ICS was created; it comprises a set of basic principles, an organizational structure, and an operational planning process. Principles include a clear chain of command, the use of common terminology and interoperable communication systems, standardized training, and certification for positions. Although initially used by firefighters, response agencies began to use the system to manage other types of incidents in the 1980s. Following the September 11, 2001 attacks, the use of ICS at the federal, state, and local level was mandated through a Presidential Directive, and ICS is now a primary component of the US National Incident Management System (NIMS). 21 USAID/OFDA ICS programming in EAP Part of USAID/OFDA’s DRR strategy in EAP is to support programs aimed at increasing the capacity of national incident command systems to effectively respond to disasters. Overall, the objectives of ICS training programs are to enhance the capacity of countries to manage disaster response operations particularly during the initial emergency phase prior to the arrival of outside assistance. The training programs are also designed to ensure vertical integration of disaster management from the local to national level and to encourage the sustainability of the ICS program following the cessation of program activities. USAID/OFDA supports ICS development by funding the U.S Forest Service (USFS) to conduct ICS capacity building activities across the EAP region. USAID/OFDA has provided funding for ICS trainings in the ASEAN member countries of Brunei, Burma, Indonesia, the Philippines, Thailand, and Vietnam, as well as to non-ASEAN member nations Mongolia, Solomon Islands, Timor Leste, and Vanuatu. Phase one of the program introduced ICS to ASEAN members through trainings and a study tour in the United States. Under Phase two, which is ongoing, USFS is continuing regional training activities, including basic and intermediate ICS courses and the development of online ICS training modules. C.3 BACKGROUND: ICS PROGRAMS TO BE EVALUATED C.3.1 Overview of Programs This evaluation will examine the effectiveness and sustainability of USAID/OFDA funded ICS programs in the Philippines, Indonesia, and Thailand. This section summarizes the program’s activities in these three countries. Philippines USAID/OFDA has provided $1,170,000 to support ICS programming in the Philippines from FY 2010 to FY 2014. USAID/OFDA support has funded train-the-trainer courses, as well as technical support for the adaptation of course materials, the establishment of Incident Management Teams (IMTs) in focal areas, the development of national ICS policy and guidelines, the mentoring of Master Trainers, the conduct of exercises, the establishment of training and certification standards for ICS trainers, and the development of a website to track ICS training nation-wide. In FY 2012, USAID/OFDA supported the USFS to bring ICS training to the local government unit level, with a focus on local government units and provinces in four regions (Cagayan, Laguna, Sorsogon and Benguet) that USAID/OFDA targeted for a capacity-building and technical support program implemented through the World Food Program (WFP). The USFS worked with WFP and the Philippine Office of Civil Defense to conduct an ICS course for participants from selected local government units and Provincial Disaster Risk Reduction Management Committees (PDRRMCs) in each of the four regions, as well as team-based All-Hazard Incident Management Team (AHIMT) courses and exercises for newly established Incident Management Teams in Cagayan and Benguet. Indonesia USAID/OFDA has provided $520,000 to support the implementation of ICS in Indonesia since FY 2012. In 2008, National Agency for Disaster Management (BNPB) issued Regulation 10, which mandates that Government of Indonesia (GoI) response agencies use the ICS. BNPB requested USG assistance in integrating ICS and other relevant NIMS components into Indonesia’s newly established disaster management system, and ICS is a primary component of BNPB’s curriculum for GoI disaster managers. Thailand 22 USAID/OFDA has provided $861,500 for ICS programming in Thailand from FY 2010 to FY 2014 through its interagency agreement with the USFS. The USFS began working with Thailand’s Department of Disaster Prevention and Mitigation (DDPM) in 2010. In FY 2012 and 2013, USAID/OFDA provided funding for a series of ICS Train-the-Trainer courses. DDPM and the Master Trainers have subsequently cascaded ICS training to operational responders in a variety of agencies and regions, and have applied ICS during the conduct of tabletop and full-scale multi-agency exercises. C.3.2 Goal, Objectives, and General Theory of Change of ICS programs The goal of USAID/OFDA’s support to ICS programs is to increase the capacity of countries within in EAP to effectively and efficiently respond to disasters using the ICS model. The objectives of each of the iterations of ICS programming varies depending on the disaster profile and baseline capacity of the country in which the program is implemented. In general, however, the objectives of ICS programs are to increase the capacity of trainees to understand the ICS model and operate effectively within it once a disaster occurs. The theories of change behind this type of program are both institutional and individual. By implementing ICS training programs targeting civilian government officials involved in disaster response, the capacity of the individuals trained will increase and the effectiveness of the disaster response institutions in which they work will improve. This evaluation is designed to examine this model by studying its effectiveness and sustainability in the Philippines, Indonesia, and Thailand. C.3.3 existing performance data The contractor will be able to access periodic performance reports from the ICS programs implemented with OFDA funding. The contractor will not have access to evaluation data or detailed quantitative results from performance monitoring. C.4 THE EVALUATION C.4.1 Purpose & Use The goal of this evaluation is to improve USAID/OFDA’s understanding of the performance of the ICS program supported by USAID/OFDA in the EAP region. The evaluation shall focus on the effectiveness and sustainability of the USAID/OFDA funded ICS programs in the Philippines, Indonesia, and Thailand. The findings of this evaluation will inform future programming decisions and adjustments to ongoing ICS programming. After five years of supporting the ICS programming in a variety of regions, USAID/OFDA seeks a deeper understanding of the successes and areas for improvement in this capacity-building model. An evaluation of the performance of the ICS program in FY 2015 will allow USAID/OFDA decision-makers to apply the information from this evaluation in future programming decisions related to DRR programming and the ICS program model specifically. With more than $4 million invested in the ICS program over the last five years, the program is a significant investment in resources from USAID/OFDA. As with many capacity-building initiatives, contextual factors can influence the results from the capacity intervention. The introduction of ICS into a variety of countries—all with their own disaster risks, government structures, and socio-economic factors—has led USAID/OFDA to seek more information about the factors for success and the potential for sustainability. C.4.2 Evaluation Objectives & Lines of Inquiry This evaluation has two objectives: understanding effectiveness and sustainability. Each objective has several lines of inquiry that shall inform the evaluation design. Data must be collected for all lines of 23 inquiry in the Philippines, Indonesia, and Thailand and will be analyzed to compare results from the different contexts. USAID/OFDA does not regard these lines of inquiry as final and could modify the lines of inquiry through the design phase of the evaluation through conversations with the evaluation team. The lines of inquiry will be considered final in the approved version of the inception report deliverable of this contract. While exact wording of the lines of inquiry could be modified through the design phase, the objectives of the evaluation as well as the intent and focus of the evaluation will not change. Objective 1: Effectiveness 1. To what extent has the ICS program built the capacity3 of national governments to respond to disasters effectively, particularly at the initial stages of the emergency? 2. Which aspects of the ICS training program are most effective? Which aspects of the ICS training program are least effective? 3. Is the ICS targeting actors within the national government that are best positioned to effectively implement elements of the ICS system? 4. What country-level factors influence the effectiveness of the ICS program? 5. To what extent do training participants—including the different levels of the cascade training system—retain knowledge and skills transferred through the ICS trainings? Objective 2: Sustainability 6. To what extent are countries able to integrate the ICS into their national frameworks or other institutionalization models? 7. What barriers to utilization of the ICS exist? 8. How much adaptation (translation of core concepts and revision of key elements to better reflect local conditions) of the ICS is required before the materials can be utilized at the local government￾level? 9. What factors contribute to sustainability of the ICS? Will existing investments remain viable without external donor support? C.4.3 Evaluation Type This performance evaluation will be summative in nature. It will evaluate programs that have closed in order to draw conclusions about their effectiveness and sustainability so that USAID/OFDA can adjust future implementation plans and strategies. C.4.4 Evaluation Approach This evaluation will take a utilization-focused evaluation approach, meaning the evaluation will be designed through extensive consultation with USAID/OFDA and the USFS. USAID/OFDA seeks to gain feedback directly from the client countries who have participated in ICS activities. The evaluation will be designed to ensure that findings from the evaluation can directly inform future programming decisions related to ICS. The evaluation team will need to tailor the evaluation approaches to the specific context of country and analyze findings accordingly. The evaluation should be designed to allow readers to draw comparisons across country locations and provide analysis of the findings for the different contexts. C.4.5 Evaluation Audience The users of this evaluation will be USAID/OFDA’s EAP team and USFS International Programs Department. More broadly, this evaluation will be read by USAID/OFDA staff in Washington, D.C., and regional field offices. This evaluation will also add to the evidence base on ICS and DRR programming. The evaluation will be available to all USAID/OFDA staff and will be made publicly available, in accordance with specific contractual requirements. 24 C.4.6 Evaluation Methods & Data Sources General guidance This evaluation must employ mixed methods: both qualitative and quantitative methods are required. Primary data must form the majority of the data collected for this evaluation; secondary data review alone will not suffice to inform the deliverables for this evaluation. The evaluation must use primary data collected in Thailand, the Philippines, and Indonesia to inform responses to the lines of inquiry of the evaluation. Any quantitative data collection that takes place as part of this evaluation must include a representative sample of the survey population and must use rigorous methods for data collection and analysis. The survey populations for any large scale survey conducted as part of this evaluation are the individual countries (Thailand, the Philippines, and Indonesia) or groups of stakeholders within those countries, not the total population of survey populations in all three. Therefore, if a large scale survey is conducted, there will be three separate surveys. The data must be representative at a sub-national level, though the exact level of representation will be determined during the design stage of the evaluation. USAID/OFDA prefers to have data with no more than a 5% margin of error and at least a 95% confidence level. However, the exact parameters of the survey design will be determined during the initial stages of the evaluation process. If less rigorous quantitative methods are needed, the Contractor must submit a justification to the COR for review and approval. Only after the COR has approved the less rigorous quantitative methods may they be used by the Contractor. The qualitative methods proposed should consider the various approaches to evaluating training and capacity building programs in development contexts. The evaluation plan must incorporate methods suitable to evaluating training, capacity-building, and DRR. The qualitative inquiry should be designed to analyze data from multiple levels along the capacity-building results chain. The evaluation will use multiple data collection instruments, aiming to capture data from multiple angles and data sources. Having analysis from a variety of levels will inform the findings of the evaluation. Multiple-level analysis is important to understand where and how the interventions are influencing the government’s capacity to respond to disasters. This evaluation should consider the relationships between capacity interventions and capacity outcomes, and the links between capacity and performance variables. In the selection of methods, the contractor must include ethical considerations, do no harm precautions, and informed consent. Specific methods The exact methods to be used in this evaluation will be determined through the submission and acceptance of the Inception Report and Evaluation Plan deliverables. However, USAID/OFDA requires the set of methods described in the rest of this section to be used for certain lines of inquiry in this evaluation. If, through the design process, the evaluation team finds that these methods are not suitable for responding to the identified lines of inquiry, the Contractor must provide a clear justification. Changes to these methods requirements are subject to the review and approval of the COR. • To what extent do training participants—including the different levels of the cascade training system—retain knowledge and skills transferred through the ICS trainings? o How much adaptation (translation of core concepts and revision of key elements to better reflect local conditions) of the ICS is required before the materials can be utilized at the local government-level? 25 o What factors contribute to sustainability of the ICS? Will existing investments remain viable without external donor support? o To what extent has the ICS program built the capacity4 of national governments to respond to disasters effectively, particularly at the initial stages of the emergency? There are several types of methods that USAID/OFDA requires for the above lines of inquiry. In general, the contractor must use both quantitative and qualitative methods to gather data related to the lines of inquiry above. First, the evaluation team must work with the USAID/OFDA and USFS to map benchmarks related to the ICS programming that link to levels of capacity-building. This mapping should help to understand what essential inputs and processes are required at each level to achieve capacity-building benchmarks in various country contexts. For example, one benchmark that indicates a level of capacity has been reached is the introduction of a policy that mandates the use of ICS for disaster management. This mapping must be an ongoing process during the evaluation. Second, the contractor must use a survey of training participants to measure retained knowledge over time. Third, the contractor must conduct structured or unstructured individual interviews with training participants, government officials, and other relevant stakeholders. Fourth, this evaluation must include self-assessment techniques that have the added value of allowing participants in the evaluation to reflect and learn. Self-assessment techniques must be balanced with other objective measures and data collection. Fifth, this evaluation must include focus group discussions with relevant stakeholders, community groups, and government officials. Finally, the contractor must develop a capacity building index for the ICS system must be an overall tool for evaluators to utilize in data collection. The capacity building index must be based on the ICS curriculum and ICS program objectives. • Is the ICS targeting actors within the national government that are best positioned to effectively implement elements of the ICS system? • To respond to the line of inquiry stated above, the contractor must conduct individual interviews and focus group discussions with relevant stakeholders. • To what extent are countries able to integrate the ICS into their national frameworks or other institutionalization models? • What barriers to utilization of the ICS exist? • How much adaptation (translation of core concepts and revision of key elements to better reflect local conditions) of the ICS is required before the materials can be utilized at the local government￾level? • What factors contribute to sustainability of the ICS? Will existing investments remain viable without external donor support? To respond to the lines of inquiry stated above, the contractor must use qualitative methods. If the contractor believes that quantitative methods would be useful to use for these questions, the contractor may use them. The qualitative methods must include individual interviews with training participants and government officials at a minimum. 26 Data Sources The evaluation team must interview the most relevant respondents for each of the lines of inquiry. While each line of inquiry will have a different set of respondents deemed to be most relevant, the evaluation team must collect data from the following groups of individuals, (contingent upon the receipt of informed consent): 1. Training participants 2. Government officials 3. USAID/OFDA staff 4. USFS staff 5. ICS trainers 6. Community members 7. Other members of the USG that have worked with training participants during a disaster response 8. Training curricula 9. Periodic programmatic reports The above list of respondents is not exhaustive. The contractor is encouraged to add more groups to respondents to ensure that the list is comprehensive and adequately covers all lines of inquiry. This list of respondents may include individuals who have departed the countries in question or who no longer work for the organizations that employed them during the ICS training or disaster response. As such, the evaluation team must find as many of these individuals as possible to interview. C.4.7 Limitations Most ICS programs may not have baseline data and may not have consistent performance monitoring data. There have not been previous evaluations of the programs and documentation of progress, successes, and challenges is very limited. It may take time to contact respondents who have changed jobs or job location since the training program began. C.4.8 Geographic Scope The geographic scope of this evaluation is Indonesia, the Philippines, Thailand, and the United States. Primary data collection for this evaluation – and thus travel for relevant members of the evaluation team – must occur in Indonesia, the Philippines, and Thailand. Travel to countries other than the aforementioned countries for the purposes of this contract is allowable only if the evaluation plan identifies a clear need for primary data collection to occur and if the COR approves the need. The evaluators must also prepare to interview respondents via the telephone or internet-enabled communication in countries outside of the aforementioned countries. The Contractor is responsible for accessing all countries and regions within the countries in the geographic scope of this contract. C.4.9 Programming Period Covered by Evaluation This evaluation will cover USAID/OFDA-funded ICS programs implemented between FY 2010 and FY2015. C.4.10 Period of Performance of the Evaluation This evaluation must be carried out under an agreed upon schedule of work, determined through the work plan. The period of performance of this contract is eight (8) months. C.5 DELIVERABLES 27 Each of the deliverables must be submitted to USAID/OFDA for review and approval. USAID/OFDA has the following general quality standards for each of the deliverables submitted as part of this contract: 1. Deliverables must contain no typographical or grammatical errors. 2. Deliverables must be submitted at or before the deadline established in the contract. For deliverables with deadlines established in the work plan, the Contractor must submit the deliverables at or before the deadline approved in the work plan. 3. The deliverables must be written using concise and direct language. Deliverable Contract Reference Planned Completion Date Post-Award Conference Call C.5.1 NLT 7 days after Contract Award Work Plan C.5.2 NLT 14 days after Contract Award Kick-Off Meeting C.5.3 NLT 21 days after Contract Award Inception Report C.5.4 To be determined in the work plan 28 Annex II: List of Indicators No. Indicators Data Sources Data Collection Locations Objective 1: Effectiveness To what extent has the ICS program built the capacity of national governments to respond to disasters effectively, particularly at the initial stages of the emergency? 1 Number of government officials trained annually Program Plans, Records, and Reports including: ICS Training Participant rosters from program Implementers (OFDA, USFS) Training provider records Government records Course Curricula US and In-country (some can be collected remotely) 2 Number of government officials trained total 3 Number of first responders trained annually 4 Number of first responders trained total 5 Number of Master Trainers trained 6 Number of training events by type 7 Number of organizations who have adopted ICS (in plans, proclamations, etc.) 8 ICS/NIMS used successfully in responses or exercises After action reports / Incident Reviews In-Country (can be collected remotely) 9 References to ICS/NIMS components in plans, policies, procedures, and laws Government disaster response agencies plans, policies, procedures, and laws Local response agencies plans, policies, procedures 10 Usage of ICS/NIMS concepts and procedures at multiple levels Response Officials In-Country 29 11 Perceived effectiveness of ICS program in improving disaster response capabilities 12 Support for implementation Which aspects of the ICS training program are most effective? Which aspects of the ICS training program are least effective? 13 Positive Trainee Feedback Training Feedback Forms (if available) US and In-Country (can be 14 Negative Trainee Feedback collected remotely) 8 ICS/NIMS used successfully in responses or exercises After Action Reports / Incident Reviews US 15 Retained Knowledge Survey: Amount of knowledge retained by training participants (did position, number of opportunities to apply knowledge, etc. impact retained knowledge) Training Participants In-Country 16 Impact of training (self-assessment) 17 Organizational capacity and/or performance improvements Master Trainers Response Officials In-Country 18 Unsuccessful ICS/NIMS use in responses or exercises 16 Impact of training (self-assessment) Is the ICS targeting actors within the national government that are best positioned to effectively implement elements of the ICS system? 12 Support for implementation Response Officials In-Country 19 Authority to implement 11 Perceived effectiveness of ICS program in improving disaster response capabilities 30 20 Distribution of training participants' levels of authority/activity Training records Government records Organizational charts In-Country (can be collected remotely) What country-level factors influence the effectiveness of the ICS program? 21 Existing culture of coordination/collaboration Open source reports 22 Presence of good governance indicators - accountability, transparency, rule of law 23 Resource availability 24 Geography 25 Frequency of disasters 26 Acceptance by national governments Response Officials In-Country 27 Acceptance by local governments 28 Perceived contributions of country level data listed above or other factors 29 Opportunities to use gained knowledge in exercises or real world events Training Participants In-Country To what extent do training participants—including the different levels of the cascade training system—retain knowledge and skills transferred through the ICS trainings? 15 Retained Knowledge Survey: Amount of knowledge retained by training participants (did position, number of opportunities to apply knowledge, etc. impact retained knowledge) Training Participants In-Country Objective 2: Sustainability To what extent are countries able to integrate the ICS into their national frameworks or other institutionalization models? 8 ICS/NIMS used successfully in responses or exercises After Action Reports/Incident Reviews In-Country (can be collected remotely) 31 9 References to ICS/NIMS components in plans, policies, procedures, and laws National, Regional, Local plans, policies, procedures 30 Success of integration at multiple levels of government Response Officials In-Country What barriers to utilization of the ICS exist? 31 Failed attempts to integrate Response Officials In-Country 32 National/Regional level challenges 33 Personal experiences of challenges at various levels of implementation (e.g., didn't understand terminology; culture not congruent with principles; didn't have country-specific examples of usage) Response Officials In-Country How much adaptation (translation of core concepts and revision of key elements to better reflect local conditions) of the ICS is required before the materials can be utilized at the local government-level? 34 Changes / adaptations to ICS forms, structures, or protocols that have been made for the country Response Officials In-country 35 Suggested adaptations / areas for improvement Response Officials 36 Changes / adaptations to ICS Training Curricula that have been made for the country USFS Trainers Master Trainers US and In-country 35 Suggested adaptations / areas for improvement 37 Training delivery challenges 32 37 Training delivery challenges Course evaluations (if they exist, using IOTWS evaluations as a benchmark) Course Reports US What factors contribute to sustainability of the ICS? Will existing investments remain viable without external donor support? 38 Availability of other sources of Funding - internal or international (e.g., national level funding, CDC International Fellowship Programs) Response Officials In-Country 39 Perceived sustainability of country ICS programming Response Officials In-Country 40 Establishment of Structures, such as an ICS Steering Committee, to continue ICS capability development Response Officials In-Country 41 Cost of additional training given availability of Master Trainers reasonable Master Trainers Training Participants Response Officials US and In-Country 42 Master Trainers provide high quality training 43 Standardized training and implementation 44 Programs to capture and share lessons learned are in place Response Officials In-Country 12 Support for implementation Response Officials In-Country 9 References to ICS/NIMS components in plans, policies, procedures, and laws Government disaster response agencies plans, policies, procedures, and laws Local response agencies plans, policies, procedures US (can be collected remotely) 33 38 Availability of other sources of Funding - internal or international (e.g., national level funding, CDC International Fellowship Programs) Program Activity Reports, SOWs, Open Source 12 Support for implementation Local, Regional, National plans, policies, laws, etc. 34 Annex III: Data Collection Instruments Instrument for Response Officials Methods No. Indicators Questions Responses Notes Objective 1: Effectiveness To what extent has the ICS program built the capacity of national governments to respond to disasters effectively, particularly at the initial stages of the emergency? Interviews / Focus Group 10 Usage of ICS/NIMS concepts and procedures at multiple levels How many levels of government and agencies are currently using ICS? 11 Perceived effectiveness of ICS program in improving disaster response capabilities Has the ICS programming provided improved disaster response capabilities? 12 Support for implementation How much support is there for continued use of ICS nationwide? Which aspects of the ICS training program are most effective? Which aspects of the ICS training program are least effective? Interviews / Focus Group 17 Organizational capacity and/or performance improvements Have ICS training programs improved partner nation(s) organizational capacity and performance? Can you cite any specific, tangible gains? 18 Unsuccessful ICS/NIMS use in responses or exercises Are you aware of any uses of ICS/NIMS in our partner nations that have been unsuccessful or ineffective? 16 Impact of training (self￾assessment) What have been the most effective aspects of the ICS Training program? Which aspects have been least effective? 35 Is the ICS targeting actors within the national government that are best positioned to effectively implement elements of the ICS system? Interviews / Focus Group 12 Support for implementation Has the partner nation been supportive of implementing ICS? Has there been a specific champion? 19 Authority to implement Do the people receiving ICS training have the authority to implement ICS at their levels? 11 Perceived effectiveness of ICS program in improving disaster response capabilities How effective has the ICS programming been in improving disaster response capabilities? What country-level factors influence the effectiveness of the ICS program? Interviews / Focus Group 26 Acceptance by national governments How well is ICS accepted at the National level? 27 Acceptance by local governments How well is ICS accepted at the local level? 28 Perceived contributions of country level data listed above or other factors What factors in this partner nation have influenced the effectiveness of the ICS program, whether negative or positive? To what extent do training participants—including the different levels of the cascade training system—retain knowledge and skills transferred through the ICS trainings? Objective 2: Sustainability To what extent are countries able to integrate the ICS into their national frameworks or other institutionalization models? Interviews / Focus Groups 30 Success of integration at multiple levels of government Have ICS processes, documents, and strategies been successfully integrated into response processes at local and national levels? 36 What barriers to utilization of the ICS exist? Interviews/ Focus Groups 31 Failed attempts to integrate Are you aware of any times officials have tried to integrate ICS concepts into their frameworks, but were not successful? 32 National/Regional level challenges What challenges exist to full and effective use of ICS in the partner nation? Are there challenges at both the national and regional levels? 33 Personal experiences of challenges at various levels of implementation (e.g., didn't understand terminology; culture not congruent with principles; didn't have country-specific examples of usage) What challenges have you encountered when trying to implement ICS? (e.g., didn't understand terminology; culture not congruent with principles; didn't have country-specific examples of usage) How much adaptation (translation of core concepts and revision of key elements to better reflect local conditions) of the ICS is required before the materials can be utilized at the local government-level? Interviews / Focus Groups 34 Changes / adaptations to ICS forms, structures, or protocols that have been made for the country What changes have been made to standard ICS forms, structures, and protocols to meet country requirements? 35 Suggested adaptations / areas for improvement What changes should be made to enhance ICS implementation? What factors contribute to sustainability of the ICS? Will existing investments remain viable without external donor support? Interviews / Focus Groups 38 Availability of other sources of Funding - internal or international (e.g., national level funding, CDC Have ICS initiative been funded by organizations other than USAID/OFDA? What funding have the 37 International Fellowship Programs) national or local governments provided? 39 Perceived sustainability of country ICS programming Is the country's ICS programming sustainable? i.e. if OFDA no longer sponsored it, would the country still use ICS? 40 Establishment of Structures, such as an ICS Steering Committee, to continue ICS capability development Has the partner nation established a committee responsible for advocating for and guiding the adoption of ICS practices at national and local levels? 41 Cost of additional training given availability of Master Trainers reasonable Is the cost of cascaded training reasonable and accessible to a variety of local responder organizations? 42 Master Trainers provide high quality training Do the in -country trainers provide high quality training that is comparable to that offered by the USFS cadre? If not, what can be done to ensure quality? 43 Standardized training and implementation Is ICS training standardized across the nation? Is implementation standardized? 44 Programs to capture and share lessons learned are in place Are programs in place to capture and share lessons learned nationally and internationally? 38 12 Support for implementation Asked Previously Instrument for Training Cadres Methods No. Indicators Question Responses Notes Objective 1: Effectiveness To what extent has the ICS program built the capacity of national governments to respond to disasters effectively, particularly at the initial stages of the emergency? Which aspects of the ICS training program are most effective? Which aspects of the ICS training program are least effective? Interviews / Focus Group 17 Organizational capacity and/or performance improvements Have ICS training programs improved partner nation(s) organizational capacity and performance? Can you cite any specific, tangible gains? 18 Unsuccessful ICS/NIMS use in responses or exercises Are you aware of any uses of ICS/NIMS in our partner nations that have been unsuccessful or ineffective? 16 Impact of training (self￾assessment) What have been the most effective aspects of the ICS Training program? Which aspects have been least effective? Is the ICS targeting actors within the national government that are best positioned to effectively implement elements of the ICS system? Interviews / Focus Group 12 Support for implementation Has the partner nation been supportive of implementing ICS? Has there been a specific champion? 11 Perceived effectiveness of ICS program in improving How effective has the ICS programming been in 39 disaster response capabilities improving disaster response capabilities? What country-level factors influence the effectiveness of the ICS program? Interviews / Focus Group 26 Acceptance by national governments How well is ICS accepted at the National level? 27 Acceptance by local governments How well is ICS accepted at the local level? 28 Perceived contributions of country level data listed above or other factors What factors in this partner nation have influenced the effectiveness of the ICS program, whether negative or positive? To what extent do training participants—including the different levels of the cascade training system—retain knowledge and skills transferred through the ICS trainings? Objective 2: Sustainability To what extent are countries able to integrate the ICS into their national frameworks or other institutionalization models? What barriers to utilization of the ICS exist? Interviews/ Focus Groups 31 Failed attempts to integrate Are you aware of any times officials have tried to integrate ICS concepts into their frameworks, but were not successful? 32 National/Regional level challenges What challenges exist to full and effective use of ICS in the partner nation? Are there challenges at both the national and regional levels? 33 Personal experiences of challenges at various levels of implementation (e.g., didn't understand terminology; culture not What challenges have you encountered when trying to implement ICS? (e.g., didn't understand terminology; culture not congruent with 40 congruent with principles; didn't have country-specific examples of usage) principles; didn't have country-specific examples of usage) How much adaptation (translation of core concepts and revision of key elements to better reflect local conditions) of the ICS is required before the materials can be utilized at the local government-level? Interviews / Focus Groups 36 Changes / adaptations to ICS Training Curricula that have been made for the country What changes have been made to the ICS Training curriculum to adapt the content and/or presentation to the target country? 35 Suggested adaptations / areas for improvement What changes should be made to enhance ICS implementation? 37 Training delivery challenges What are the main challenges you encounter when delivering training? What factors contribute to sustainability of the ICS? Will existing investments remain viable without external donor support? Interviews / Focus Groups 38 Availability of other sources of Funding - internal or international (e.g., national level funding, CDC International Fellowship Programs) Have ICS initiative been funded by organizations other than USAID/OFDA? What funding have the national or local governments provided? 39 Perceived sustainability of country ICS programming Is the country's ICS programming sustainable? i.e. if OFDA no longer sponsored it, would the country still use ICS? 41 40 Establishment of Structures, such as an ICS Steering Committee, to continue ICS capability development Has the partner nation established a committee responsible for advocating for and guiding the adoption of ICS practices at national and local levels? 41 Cost of additional training given availability of Master Trainers reasonable Is the cost of cascaded training reasonable and accessible to a variety of local responder organizations? 42 Master Trainers provide high quality training Do the in-country trainers provide high quality training that is comparable to that offered by the USFS cadre? If not, what can be done to ensure quality? 43 Standardized training and implementation Is ICS training standardized across the nation? Is implementation standardized? 44 Programs to capture and share lessons learned are in place Are programs in place to capture and share lessons learned nationally and internationally? 42 DevTech Systems, Inc. Independent Evaluation of the USAID/OFDA Incident Command Systems (ICS) Training Program Focus Group Discussion Questions RESPONSE OFFICIALS You have received this request to join us for this group discussion because of your participation or involvement in the Incident Command System (ICS) training program funded by the United States Agency for International Development Office of Foreign Disaster Assistance (USAID/OFDA). DevTech Systems, Inc. has been commissioned by OFDA to conduct an external evaluation of the ICS program and we are conducting this survey as part of the evaluation tasks. We would like to get your thoughts and opinions on the effectiveness and sustainability of the ICS program in building the capacity of local first responder teams to events. The information you will share will be kept strictly confidential and there will be no disclosure of any individual response. The only information that will be used for disclosure to third parties will be aggregates and summaries of the information from all participants, and a selection of comments made, without attribution to any individual who made them. We are grateful for your participation in this evaluation. The information you provide will contribute significantly to the evaluation and the work of the ICS program as a whole. Thank you very much. Sincerely, The DevTech Evaluation Team 43 Objective 1: Effectiveness The FGD moderator should attempt to elicit responses to all of the Primary Questions using Primary Questions or secondary/probe questions to elicit perception- and experience-based responses from the focus group participants. Indirect responses that come up in discussions that follow any of the moderator’s prompts should be recorded. If, in the course of the FGD, participants do not answer a Primary Question, use the secondary or probe questions to ensure that all relevant indicators have been addressed. If, in answering one Primary Question, an interviewee provides an answer for another, the interviewer may alter the original order of Primary Questions to best maintain a natural conversational flow. The moderator should use the Primary Questions as a guide and select appropriate secondary or probe questions to ask that align with the focus group’s particular set of experiences or expertise. Primary Question 1: Have you noticed a difference in the effectiveness of disaster response(s) in your country since the OFDA/USFS ICS program began? 1a. How many levels of government and agencies are currently using ICS? 1b. Has the ICS programming provided improved disaster response capabilities? Have any aspects of disaster response become less effective as a result of the training? If so, do you think these are temporary setbacks as part of the change toward ICS, or is ICS incompatible with any part of your country’s disaster response system? 1c. How much support is there for continued use of ICS nationwide? Primary Question 2: Which aspect(s) of the ICS training program have you found to be the most effective? Which aspect(s) have you found to be the least effective? 2a. Have ICS training programs improved your country’s disaster response organizational capacity and performance? Have you noticed a difference in skill level and response performance between responders who have and have not received ICS training? 2b. Are you aware of any uses of ICS/NIMS in our partner nations that have been unsuccessful or ineffective? 2c. In your opinion, what have been the most effective aspects of the ICS Training program? 2d. Which aspects have been least effective? Primary Question 3: Is the ICS targeting actors within the national government that are best positioned to effectively implement elements of the ICS system? Which actors, if any, have been left out of ICS planning, training and implementation that should have been included? 3a. Has the national government been supportive of implementing ICS? Has any particular branch, agency, organization or person (local or foreign) been a champion for ICS implementation? 44 3b. Do the people receiving ICS training have the authority to implement ICS at their levels? At what point did they receive this authority (i.e.: how long into the ICS training and implementation did this change occur)? 3c. How effective has the ICS programming been in improving disaster response capabilities? Can you provide a specific example – in training or in actual disaster response – to support your view? Primary Question 4: What country-level factors influence the effectiveness of the ICS program? 4a. How well is ICS accepted at the National level? 4b. How well is ICS accepted at the local level? 4c. What factors in this partner nation have influenced the effectiveness of the ICS program, whether negative or positive? Objective 2: Sustainability Primary Question 1: To what extent has your country integrated ICS into its national framework or other institutional models? 1a. Have ICS processes, documents, and strategies been successfully integrated into response processes at local and national levels? Primary Question 2: What barriers to utilization of the ICS exist? 2a. Are you aware of any times officials have tried to integrate ICS concepts into their frameworks, but were not successful? 2b. What challenges exist to full and effective use of ICS in your country? Are there challenges at both the national and regional levels? 2c. What challenges did you encounter or notice in integrating ICS principles into the national framework? (e.g.: didn't understand terminology; culture not congruent with principles; didn't have country-specific examples of usage) Primary Question 3: How much adaptation (translation of core concepts and revision of key elements to better reflect local conditions) of the ICS is required before the materials can be utilized at the local government-level? 3a. What changes have been made to standard ICS forms, structures, and protocols to meet country requirements? 3b. Did you encounter any challenges specific to the ICS training and training materials that hindered the integration of the ICS into your national framework? If so, what changed – or should have changed – to enhance and speed up the integration process? Who first noticed the issues, and who made the appropriate changes? 45 3b. What changes should be made to enhance ICS implementation? Primary Question 4: What factors contribute to sustainability of the ICS? Will existing investments remain viable without external donor support? 4a. Have ICS initiative been funded by organizations other than USAID/OFDA? If so, have these trainings helped or hindered the integration and standardization of ICS in your country? 4b. What funding have the national or local governments provided? 4b. Do you believe that the ICS program in your country would remain viable and sustainable without external donor (i.e.: USAID) funding or training? Why or why not? 4c. Has the government established a committee responsible for advocating for and guiding the adoption of ICS practices at national and local levels? 4d. Have you heard of any issues with cascaded training related to cost and/or accessibility? Are local responder organizations able to access training, and is this access sustainable in the long-term? 4e. Do the in-country trainers provide high quality training that is comparable to that offered by the USFS cadre? What differences have you noticed or been told about between local trainers and USFS trainers? What can be done to ensure quality? 4f. Is ICS training standardized across the nation? Is implementation standardized? 4g. Are programs in place to capture and share lessons learned nationally and internationally? Who (what government branch/agency or individual) spearheaded the development of these programs? 46 Annex IV. Sources of Information Table __: List of Program Documents Reviewed Program Documents 4th Quarter FY10 Report - OFDA Philippines Planning 1st Quarter FY11 Report - OFDA ASEAN ICS 2nd Quarter FY11 Report - OFDA ASEAN ICS 3rd Quarter FY11 Report - OFDA ASEAN ICS 4th Quarter FY11 Report - OFDA ASEAN ICS 1st Quarter FY12 Report - OFDA ASEAN ICS 2nd Quarter FY12 Report - OFDA ASEAN ICS 3rd Quarter FY12 Report - OFDA EAP ICS Activities 4th Quarter FY12 Report - OFDA EAP ICS Activities 1st Quarter FY13 Report - OFDA EAP ICS Activities 2nd Quarter FY13 Report - OFDA EAP ICS Activities 3rd Quarter FY13 Report - OFDA EAP ICS Activities 4th Quarter FY13 Report - OFDA EAP ICS Activities 1st Quarter FY14 Report - OFDA EAP ICS Activities 2nd Quarter FY14 Report - OFDA EAP Activities 3rd Quarter FY14 Report - OFDA EAP Activities 4th Quarter FY14 Report - OFDA EAP ICS Activities 1st Quarter FY15 Report - OFDA EAP Activities 2nd Quarter FY15 Report - OFDA EAP Activities 3rd Quarter FY15 Report - State ASEAN ICS 2012 Funds 1st Quarter FY16 Report - OFDA EAP Activities 2nd Quarter FY16 Report - OFDA EAP Activities 3rd Quarter FY16 Report - OFDA EAP Activities 47 Table __: List of Country Documents Reviewed Indonesia 201211 Indonesia ICS SOW with Budget 20150803 Indonesia ICS_3 Yar_SOW_final_w_budgets Attachment C - DRR Assessment Attachment D - ICS SOW - 60 TOT 20120321 Indonesia ICS SOW - USAID Mission Indonesia Summary Email - Oct 18 2016 Indonesia USFS Disaster Management - FY 2012 OFDA Table __: List of Interview and Group Discussion Participants Name Organization and Designation Endang Achadiat SKM, DKI Jakarta Ms. Alsinta Master Trainer (MT), Central Java Harlan Hale OFDA Regional Advisor Idan Muhktijahidden Red Cross Indonesia Mr. Nugroho Head of Disaster Victim Identification Unit (DVI), Indonesia National Police (INP) Marg Olson USFS Disaster Management Specialist Yusak Oppusunggu USAID OFDA Program Managers Regina Rahadi USAID OFDA Program Managers Mindaraga Rahardja (Iwan) Humanitarian Affairs Analyst/Reporting, UNOCHA Joe Reyes USFS, Master Trainer Danang Samsu Master Trainer Efraim Sitinjak World Vision Ms. Supriyati BNPB, Master Trainer Dr. Bagus Tjahjono, MPH Pusdiklat Penanggulangan Bencana, BNPB Rusty Witwer USFS, Master Trainer 48 Annex V. Field Photos 1 Basic/Intermediate TOT at the Indonesia Disaster Relief Training Grounds in Sentul 2 Earthquake Monitoring System