EVALUATION OF USAID/ OFDA/ LAC RISK MANAGEMENT PROGRAM Prepared for: USAID/OFDA Prepared by: Richard J. Moore Nicholas F. Colmenares Checchi and Co. Consulting/ Louis Berger Joint Venture 1899 L Street, NW Suite 800 Washington, DC 20036 Contract No. AEP-I-011-00-00022-00 Task Order # 12 Final Report December 2004 CHECCHI AND COMPANY CONSULTING, INC The Louis Berger Group, INC Evaluation: USAID/OFDA/LAC December 2004 Risk Management Program TABLE OF CONTENTS Page No. List of Acronyms ......................................................................................................................ii Executive Summary.................................................................................................................iv Evaluation Report .................................................................................................................... 1 I. Scope of Work of OFDA/LAC RMP Program............................................................ 1 II. Evaluation Methodology ............................................................................................. 1 III Evolution of OFDA/LAC Model................................................................................. 2 IV. Principal Organizing Themes for Evaluation............................................................... 4 A. Coverage of Programs and Country Target Populations/Actors...................... 5 B. OFDA’s Role in Building National Capacity and Coordination................... 10 C. Overall Program Impact................................................................................. 20 D. Program Management Themes ...................................................................... 27 V. General Conclusions .................................................................................................. 34 VI. New Horizons and the Future .................................................................................... 37 ANNEXES Annex A: Contact List ........................................................................................................A-1 Annex B: Countries Visited By the Evaluation Team........................................................ B-1 Annex C: Interview Protocols.............................................................................................C-1 Annex D: Scope Of Work: .................................................................................................D-1 CHECCHI AND COMPANY CONSULTING, INC The Louis Berger Group, INC Evaluation: USAID/OFDA/LAC December 2004 Risk Management Program i LIST OF ACRONYMS ARC American Red Cross CAMI Central American Mitigation Initiative CAPRADE Comité Andino para la Prevención y Atención de Desastres CDB Caribbean Development Bank CDERA Caribbean Disaster Relief Agency CENAPRED Centro Nacional de Prevención de Desastres CEPREDNAC Centro de Coordinación para la Prevención de los Desastres Naturales en América Central CNE Comision Nacional de Prevención de Riesgos y Atención de Emergencias CONRED Comision Nacional para la Reducción de Desastres CRS Catholic Relief Services FEMA Federal Emergency Management Agency GIS Geographic Information System GRP Gobierno Regional de Piura HAP Humanitarian Assistance Program (USSOUTHCOM) IADB Inter-American Development Bank IMF International Monetary Fund INAB Instituto Nacional de Bosques INDECI Instituto de Defensa Civil INFOR Instituto Nacional Forestal INS Instituto Nacional de Seguros INSARAG International Search and Rescue Group (UN) IRC International Red Cross IRG International Resources Group, Ltd. MINED Ministerio de Educación MDRO Mission Disaster Relief Officer MININTER Ministerio del Interior MINSA Ministerio de Salud MFI Multilateral Financial Institution NEMA/NEMO National Emergency Management Association/Organization NGO Non-government Organization OAS Organization of American States ODPEM Office of Disaster Preparedness and Emergency Management OFDA Office of U.S. Foreign Disaster Assistance PAHO Pan American Health Organization PIP Plan Internacional Piura PP Plan País RA Regional Advisor RMP Risk Management Program SINAPRED Sistema Nacional Para la Prevención, Mitigación y Atención de Desastres UN United Nations UNAN Universidad Nacional Autónoma de Nicaragua USM Universidad de San Marcos USC Universidad de San Carlos USAID United States Agency for International Development CHECCHI AND COMPANY CONSULTING, INC The Louis Berger Group, INC Evaluation: USAID/OFDA/LAC December 2004 Risk Management Program ii EXECUTIVE SUMMARY The following report provides an independent evaluation of the USAID’s Office of Foreign Disaster Assistance in the Latin America and the Caribbean region (USAID/OFDA/LAC) activities in risk management and other sectors implemented via the contract with the International Resources Group (IRG) in the LAC region from 1998 to the present. This evaluation focuses on documenting and assessing overall program and country coverage, program management, client/OFDA relations and coordination, effects and impact. It reflects the work of a two-person team of outside professionals who examined OFDA’s and IRG´s ability to deliver on strategic objectives, have discernable impact, demonstrate efficiency in reaching its target audiences in the region, coordinate complex activities with multiple key players, and realize an element of sustainability through its interventions. During the course of the field-work for this report, the team was able to visit USAID/OFDA/W for initial interviews and debriefing and the USAID/OFDA/LAC office in San José, Costa Rica for a period of approximately ten days. During that time the team interviewed both the OFDA team and the resident IRG team in substantial depth. In addition, a number of teleconference interviews were held with one Regional Advisor (Caribbean), IRG consultants and National Disaster Coordinators in Perú, St. Lucia, St. Kitts and Nevis and the Bahamas).. The team also conducted interviews with principal OFDA clients in Costa Rica, both public and non￾governmental. A fourth source of information for this report consists of documentation gathered in the OFDA and IRG offices. Finally, the team carried out an extensive set of field visits to Jamaica, Perú, Guatemala and Nicaragua over a period of three weeks. Persons interviewed and offices visited are listed in Annex A. The following conclusions and recommendations for future direction are at the heart of this evaluation. While we present generalized conclusions, the reader must bear in mind that the reality of countries and the OFDA/LAC-IRG approach place great emphasis on different country contexts. q Evolution. The overall experience of OFDA’s engagement in Latin America and the Caribbean has been a highly positive one. The “model” for OFDA’s engagement throughout the region has evolved over the past two decades in response to very practical and grounded perception of needs and opportunities. This evolving approach clearly is the result of the overarching influence of Mr. Paul Bell until his unfortunate demise in 2003. The approach is unique, contextual, and historical in its evolution, and this may affect its replicability. o From the early days (mid to late 1980s) through the present we see at least three critical junctures. In the period from 1986 through the middle of the 1990s, the approach was characterized by highly personal relationships, an emphasis on first response, a somewhat reactive and sporadic effort to strengthen individual stakeholder response skill-sets, and a limited sense of accountability in a programmatic or budget sense. o A second period or juncture (mid-to-late 1990’s) is characterized by a deepening of the network through national long-term field consultants, a stronger emphasis on CHECCHI AND COMPANY CONSULTING, INC The Louis Berger Group, INC Evaluation: USAID/OFDA/LAC December 2004 Risk Management Program iii refining course offerings, deeper relationships with national institutions (not just individuals), and some integration of course content into client-institutional practices. However, course delivery remained the essence, while inter-agency coordination remained a long way away. o A third juncture or period (1999/2000 until very recently) brings order and management to a relatively loose institutional structure, both in OFDA/LAC central and in the IRG consultant network. Programmatic and budget accountability, regular and systematic client contact toward stronger institutional relationships, and an incipient thrust toward linking TA to the OFDA/IRG “in-house” process is evident, although response and training remained (remains) the bread and butter of the approach. o Finally, we are at another critical juncture. Many of the observations and recommendations that follow address what might be done and what should or could be new horizons for the program and the model. q Program Development. The past four years have seen the emergence of a critical integrating tool for the RMP: the Plan País. From its inception in 2000 as a means for bringing increasing order to the training and technical assistance functions, Plan Pais has become the key instrument for client-country planning, programming, and budgeting, and improving in-house communication and coordination with the USAID missions and OFDA/W. The success of the development of this tool is recognized in the 2003 contract itself. Two key findings merit attention: o Significant advances have been made in utilizing Plan País as an evaluation tool; there is a need to strengthen these ongoing efforts. While the TTS and training reporting, along with the quarterly reports, provide a basis for M&E, there is greater need to move from process indicators to those that might point to tangible results, and more importantly, tangible impact. It is often difficult to separate results from training and technical assistance because they are both part of an integrated process of the transfer of risk management capacity. In the second year of the contract there has been significant internal discussion in this direction, and in some cases initial impact indicators have been developed (e.g., impact measures in forest fires) o Ironically, the tool has a substantive institutional vulnerability diagnostic component, but as a basis for strategic interventions, it is not shared with national counterparts, or with the MDRO at the Mission level in at least one field case. Clearly, the PP has sensitive content. However, with some editorial work, this instrument could serve as a basis for national dialogue, and not just as an IRG internal tool. It needs to be shared with countries. q In a sense, OFDA/LAC is a victim of its own success. This is intended as a positive sign, because real transfer has taken place. The institutional maturation of a number of countries in the region has resulted in less need for OFDA involvement in course/based training. The multiplier effects of initial efforts through the network of national consultants have created a substantial cadre of national clients trained in OFDA techniques, and national instructors to impart this training to other stakeholder groups. While this is not true for all countries, in at least two of the countries visited (Peru and Guatemala, and possibly Costa Rica) this is clearly the case. CHECCHI AND COMPANY CONSULTING, INC The Louis Berger Group, INC Evaluation: USAID/OFDA/LAC December 2004 Risk Management Program iv q However, the course content of most of the training remains response oriented. In all of the country visits, the emphasis on the need for a stronger presence in risk mitigation and prevention was proffered by client respondents. A shift toward risk reduction is evident in some of the country programs, but by far the institutional framework - and OFDA training – remains response driven. While this is in keeping with OFDA´s principal role (SO) as risk response, ironically the shift from two strategic objectives (response and mitigation/risk management) to a single response oriented SO by OFDA is short-sighted. q In some cases, there was concern expressed about course content updating to current realities and the need for new response modalities (e.g., chemical disasters). In addition, there is a greater need for OFDA to play a role in technical assistance beyond its current scope, and in areas where to date partial efforts exist (legal framework, regulatory structure, land use planning, etc.). Evidence of movement in this direction can be seen in the support provided for the 2001 Hemispheric Summit on Risk Reduction, support for CAPRADE, the Caribe Risk Management Inventory project. q In addition to upgrading and adapting of course content, a common concern expressed among clients was the need to alter the approach to institutional coverage and target groups. Traditionally, the key institutions of client interaction have been the lead national government institutions (CNE, INDECI, SINAPED, CONRED, ODPEM, NEMAs) and first response organizations (firefighters, and to a lesser extent, Red Cross). The need to extend both TA (mitigation, vulnerability) and (simplified) response and assessment courses to the real first responders (those directly affected) is paramount. “OFDA/IRG understands that risk reduction and mitigation activities reduce potential losses from disasters and reduces the impact on the response systems in place. Therefore, OFDA/IRG should continue to vigorously pursue all possible client groups in its risk prevention activities, with added focus on community-based organizations. While the multiplier role of national instructor training, in part, addresses this issue, OFDA should continue to explore new technical assistance functions, specifically new roles and methods for consultants to introduce risk prevention priorities into existing institutional programs and outreach to community-based organizations.” Community organizations, parish councils, parent-run school boards already have substantial organization at the local level, and are prime candidates for inclusion in mitigation, prevention, greater response, and even assessment. For example, in Central America, school related community and parent organizations are extremely well organized (juntas escolares in Honduras, Guatemala and Nicaragua, PRONADE in Guatemala, EDUCO in El Salvador, etc.) and closely linked to localized NGOs. q The impact of Hurricane Mitch on the entire OFDA (and national) portfolio of activities cannot be underestimated. The “storm of the century” provided a laboratory for understanding (a) the need for deepening the response capabilities of national institutions; and (b) the need to shift emphases increasingly toward mitigation and risk management. The Central American Mitigation Initiative (CAMI) not only provided a substantial funding base for national governments to respond, but it informed the foci for the initial IRG contract (albeit the contract preceded the event). These emphases included: o A need to focus greater attention on risk assessment and vulnerabilities. o The need to address weakness (vulnerability) in institutions and the absence of inter￾agency coordination. CHECCHI AND COMPANY CONSULTING, INC The Louis Berger Group, INC Evaluation: USAID/OFDA/LAC December 2004 Risk Management Program v o The CAMI initiative led to the development of the Plan País, OFDA/IRG’s more comprehensive and (inter-)institutionally focused risk management effort. o The need to focus on various actors and jurisdictional levels in response, and not just key national institutions. o An increased focus on more comprehensive vulnerability analyses that did not exist prior to this time, with a better understanding of economic, social, and physical relationships before, during, and after a disaster. q Coordination with USAID Missions. The early decision (1988) to move activities from USAID MDROs as a cumbersome mechanism was probably correct, but one result is a continuing image of OFDA/LAC as a loner within the USAID structure. In the case of OFDA/IRG, this image is also evident and consultant relations with MDROs is a mixed bag, and is a function of both the consultant and the MDRO perception of their roles. In some cases (Jamaica) the relation is strong and interaction is constant. In others (Peru), the IRG consultant has had little interaction directly. In part, this is also a function of the MDRO as primarily a response mechanism, not one involved deeply in risk management or mitigation. In addition, it is through the OFDA Regional Advisors that this relationship is most evident. The RA structure provides the OFDA/IRG consultants with greater access to USAID, is important well beyond the IRG contractual role in areas more related to policy and political interaction, reflects the authority of OFDA, and provides a needed liaison role to other USG agencies and international organizations. In addition, MDROs often are not OFDA trained. Further, the interest q Management. The overall management modality or style that has evolved works reasonably well and provides a more effective and accountable institutional mechanism for the “model” that did not exist in the earlier years of OFDA engagement in the region. However, there is some concern that the shift from the initial flexible contractual arrangement (similar to an IQC in the reimbursable nature of costs) to a fixed sum contract may have reduced the flexibility of action on the ground in response to country level needs. q The “seamless” nature of the IRG/OFDA partnership is evident in the relationship between consultants and OFDA staff, in singular strategies, and in the lack of an OFDA/IRG distinction among client nationals. Seamlessness seems to be a function of the long-term relationships, but does not pertain to management functions as it does to operations. However, that is a positive outcome of the contractual relationship, because it has brought a clear sense of management accountability, something absent in earlier years (pre-2000). q However, there is some concern that IRG has stepped into strategic programmatic decisions that are the appropriate purview of OFDA management. This concern has been expressed in particular in relation to strategic decisions with regard to areas to be cut initiated by IRG as a result of budget shortfalls in 2003 and beyond (discussed in detail below). In this case, the “tail wagging the dog” may have interrupted some of the seamless nature of the relationship q In the current (2003+) period of the new contract with IRG, there has been a problem and a concern about the reason for, and consequences of what appears to be a lack of funding commensurate with a pre-designed work plan and program. It is not the role of the evaluation team to enter the question of why this occurred. However, it important to understand the consequences for Year 1 of the new contract, and the perspective for Year 2 (already in implementation). CHECCHI AND COMPANY CONSULTING, INC The Louis Berger Group, INC Evaluation: USAID/OFDA/LAC December 2004 Risk Management Program vi q There have been consequences that relate to cutbacks in programmed activities, particularly those involving travel for national consultants. The result? Substantially fewer training exercises conducted when compared to initial programming; reduction in the transmission of some new course material that has been designed; some reduction in the face-to-face interaction among consultants to plan and impart experiences; postponement to Year 2 of activities planned in Year 1. However, even with budget reductions from work plan prescriptions, much of the initial work plan has been carried out. The question becomes how to incorporate lessons learned from this period of budget tightening to assure continued financial and programmatic accountability. In addition, two particular circumstances have exacerbated the funding for key activities of the program: the termination of the participation of Miami-Dade County and related start-up costs for the new contracts with Fairfax and Los Angeles, and the absence of (expected) contributions for training to the training portfolio. Both the evaluation team and OFDA/IRGrecognize that changes in strategy and client relationships are part of the cost of doing business. In order to lower this cost as much as possible, OFDA/IRG is investing time and money in establishing these new contract relationships which may not yield tangible results well into the next contract year. q All aspects of the program have been affected, and it is the evaluation team’s estimation that the program cannot operate fully at the current levels of funding. While we have a number of observations about the what and the how of line items and activities, it is clear that insufficient funds are available. q Specific recommendations in the text relate to several themes emerging to the budget crisis of 2003-2004. o Travel cuts. The increase in travel over the past several years needs to be reconciled with the nature of the decentralized model: if national consultants are to be used, if national instructors increasingly replace consultants, and if self-reliance is evident in some countries, why should the travel budgets increase rather than decrease? The evaluation team is cognizant of the increased costs of travel and that in South America distances are great. However, per diem costs have doubled between 2001/02 and 2003/04 while the LOE has increased by 50% and the number of consultants by 3. o Move to Increasingly Full-Time Consultants. The LOE for national IRG consultants has increased to virtual full-time in most cases (from 135 days in 2000/01 to 173 in 2001/02 to 203 in 2003/04). Participation of IRG consultants in training is increasingly less cost-effective as the strength of transfer improves, and their participation in training should reflect this decreased reliance. However, the increased LOE reflects in part a shift to TA and institutional strengthening, an area mush more difficult to measure in cost effectiveness terms. Transfer of responsibilities and funding to nationals implies a changed IRG role, and a shift to other activities, or an overall decrease in presence. Success means shift in strategy and/or shift in resources. o It is important to note is that, distinct from the vast majority of course offerings, BREC and ICS require far more extensive LOE and funding to reinforce existing client capacity, and expand coverage into new client groups. OFDA/IRG must continue to closely examine its cost-efficiencies as a function of the up-front costs required to shift the RMP from training toward increased “program building”. CHECCHI AND COMPANY CONSULTING, INC The Louis Berger Group, INC Evaluation: USAID/OFDA/LAC December 2004 Risk Management Program vii o Training. While overall training was cut by a substantial amount from the original WP, the eventual $737,420 was not fully executed and funds were transferred to cover a portion of travel expenditures not allocated in the budget. Only 58% of the original WP, and only 78% of the revised WP, were actually expended. Limits on travel may have contributed to this shortfall, but clearly other factors are at work. As we note earlier, the effectiveness of the use of funds is clear because while 58% of the funds originally proposed in the Work Plan were executed, 71% of the activities were executed. This cost-effectiveness needs to be viewed in light of these figures. o Decentralization and IRG/W Participation. While the amounts are not particularly significant, it is not clear substantively why IRG/W should increase its travel as part of the contract. IRG/SJO has been and is self-sufficient and reporting requirements are the thrust of interaction with IRG/W. o Small Grants and INSARAG. It is not clear why the IRG contract is the appropriate venue for these items. In the former case, the impact is minimal and the mechanism is cumbersome. In the latter case, these were funds not allocated nor reimbursed, but the activity was requested by OFDA/W. q Monitoring and Evaluation. One of the areas that still requires substantial effort is that of monitoring and evaluation (M&E). The Plan País provides an excellent framework for M&E, but more can be done. OFDA/IRG is stretching the bounds of how to measure results and impacts. This is an enormous and innovative undertaking that has never been attempted before in the LAC”. This is especially true given the difficulties in identifying tangible measures of vulnerability reduction through a program of this scope. Also, it is important to take into account the disaster management professional community’s historical difficulties with interpreting risk management concepts, OFDA’s evolving processes are inherently qualitative in terms of measuring effectiveness. q There are four levels of evaluation evident and in various stages of development: o During the courses in classroom evaluations: From discussions with those trained and Instructors in the field, there is a sense that the tool needs strengthening. This is less a criticism of IRG and perhaps more relevant as a comment on how OFDA evaluates classroom performance. o Internal to the program itself (training package, program as a whole) o Institutional Evaluations (how trained perform in the job, performance evaluation, use of course in work). In part, the Institutional Diagnosis of the PP fills this role, but again a more systematic set of evaluation instruments need to be developed. o Organizational culture and values, organizational change (has a culture of response been superceded by a focus on mitigation, degree of intra and inter-institutional focus on the broader context of risk reduction and response) To date, the RMP/IRG PP primarily deal at levels one and two, and to some degree at level three in the diagnostics. Indeed, contractually IRG is fulfilling its M&E mandate (levels 1 and 2) and perhaps beyond as it moves specifically into the area of establishing new baseline data, the tremendous effort in integrating training and outcome through SCI which is a means for evaluating impact, the preliminary impact measures in forest fires, and some country specific institution strengthening measures. Again, the issue is not IRG performance; rather it reflects the limits of OFDA´s strategic objective. CHECCHI AND COMPANY CONSULTING, INC The Louis Berger Group, INC Evaluation: USAID/OFDA/LAC December 2004 Risk Management Program viii q Replicability. The issue of the replicability of the LAC “model” or approach is a difficult one to answer in the abstract. The essence of the approach can be divided into four key areas – Planning, Training, Technical Assistance, and Management, and includes the following key elements: o The existence of a set of contextually developed course materials, primarily oriented toward response, and increasingly, as an integrated set of materials, part of a risk management process; o A network of national “consultants” who have developed close bonds to OFDA, and in most cases to the original set of the OFDA team. In most cases these consultants were participants in the original course and training program. The “seamless” nature of the IRG/OFDA relationship is an outgrowth of this close knit group of consultants; o National consultants and OFDA Regional Advisors play a key role in first response, providing immediate assistance and damage assessment from a well-nurtured relationship with national institutions of first response and a link to USAID and embassy response; o An evolutionary and demand driven approach to responding to national training needs; o An institutional multiplier effect as the original sets of trainees have become a set of trainers, both nationally and regionally, reducing the need and role for direct training through the OFDA/IRG consultants who take on a stronger role as quality control, supervision, and technical assistance on the ground; o A requirement of some minimal institutional development on the part of the national government in disaster response and preparedness. The approach is clearly contextual and evolutionary, and in this sense may be difficult to replicate. The long-standing relationship among consultants and the OFDA team is unique, and may be difficult to replicate without similar interpersonal dynamics and collegiality built up over time. The existence of the original OFDA team, oftentimes referred to as the “four horsemen” is purely contextual, and probably not replicable. However, some elements of the model can and should be adopted by other regions. Future Directions q A common theme among client organizations is a need for OFDA to extend its grasp toward organizations and groups at the local and commmunity level. The closest associate or client of OFDA is and will remain the lead national institution and NDC within countries. However, at both the first response and mitigation level, these is a felt need that much more can be done. As is described in the text of this report, several clear needs and efforts in this direction emerge: the shortcomings of the Jamaican Parish response to Hurricane Ivan; the successful and impressive cohesion that has emerged in the case of Piura’s Disaster Network; the strong relationship of CONRED with some 17 NGOs in Guatemala. q The Piura Regional Government and disaster network provides a potential model for new directions and an understanding of real trade-offs in risk management. Recent coordination efforts build upon risk mitigation opportunities and their cost-benefit with respect to the region’s history of economic losses from disasters. However, economic loss has caused a selective acceptance of risk with respect to zoning and residential construction in the city of CHECCHI AND COMPANY CONSULTING, INC The Louis Berger Group, INC Evaluation: USAID/OFDA/LAC December 2004 Risk Management Program ix Piura, because the cost to build in less risky zones is reported to be four times higher than in risky ones. In addition, a growing population has led to public demand for affordable housing, a related governmental willingness to side-step hazard mapping, building code enforcement standards, and the allowance of building in flood and fluvial soil (earthquake) zones. In communities throughout the region, this same scenario is repeating itself. OFDA/IRG is well-positioned to use its existing base of resources and relationships to build a risk prevention mindset at the community level. q Linked to the above is a need for OFDA to form closer alliances with the NGO community. OFDA/LAC has matured in its institutional relationship with countries both in training and in technical assistance. So too has the NGO community moved from a “first generation” of foreign relief and direct humanitarian assistance to a “second generation” of more institutional relationships with national personnel, to a “third generation” of NGOs with sustainable developmental partnerships within countries linking governmental institutions and the community. These third generation NGOs provide clear opportunities to deepen not only first response, but strengthen development-oriented efforts at risk and vulnerability reduction. OFDA needs to strengthen the involvement of these NGOs in their program. (The cases of Piura and Guatemala suggest these new opportunities.) q The evaluation team would like to suggest another innovative area for activity, one that goes well beyond the purview of the OFDA/IRG contract, but is important as new strategies emerge. National institutions aimed at risk management, relief and reconstruction find themselves searching for funding in a world of dwindling resources. A common complaint among national agencies is that programmatic budgets are virtually non-existent, and even national counterpart funding for training and workshops is minimal. A similar recognition is part of USAID’s development strategy and strategic objectives. One answer is an increasing recognition of the need to form strategic “Public/Private Alliances” in the face of dwindling resources. A similar recognition is part of USAID’s development strategy and strategic objectives (SO). A recent pilot project in Kingston, Jamaica highlights some of these possibilities where vulnerability assessment and mapping could be linked to private sector development, property insurance and re-insurance interests. Involvement of university students in vulnerability assessment and training is an important part of this pilot exercise. A recent USAID project initiative in Guatemala with public private alliances in the social sectors (health, education) also suggests the possibility of incorporating risk management and assessment efforts to developmental purposes. Finally, it is the opinion of the team that the “service delivery” model and the current OFDA/IRG relationship bring a great deal of value added to OFDA’s capacity to accomplish a great deal in a very cost effective manner. The “bang for the buck” is evident. Whether one measures OFDA’s impact from a process perspective (numbers trained, courses delivered, etc.), from the perception of the client (“OFDA has created the national institutional capacity,”), from an impact perspective (national institutions and programs bear the OFDA stamp in virtually all of their training program, response time has been reduced and coordination increased, overall vulnerability has been reduced and bears some relationship to the presence of OFDA), there is a clear message: it works and works quite well. While some improvement along lines suggested above will make the model better, as it currently stands, it works and well. CHECCHI AND COMPANY CONSULTING, INC The Louis Berger Group, INC Evaluation: USAID/OFDA/LAC December 2004 Risk Management Program 1 EVALUATION: USAID/OFDA/LAC RISK MANAGEMENT PROGRAM I. Scope of Work of OFDA/LAC RMP Program This report provides an independent evaluation of the USAID¨s Office of Foreign Disaster Assistance in the Latin America and the Caribbean region (USAID/OFDA/LAC) activities in risk management and other sectors implemented via the contract with the International Resources Group (IRG) in the LAC from 1998 to the present. This evaluation focuses on documenting and assessing overall program and country coverage, program management and coordination, client/OFDA relations, effects and impact. It reflects the work of a two-person team of outside professionals who examined OFDA’s and IRG´s ability to deliver on strategic objectives, have discernable impact, demonstrate efficiency in reaching its target audiences in the region, coordinate complex activities with multiple key players, and realize an element of sustainability through its interventions. In pursuit of its global mandate to save lives, alleviate suffering, and reduce the economic impact of disasters, OFDA has a contract with International Resources Group (IRG) to provide various programmatic and administrative services in the LAC region. This set of two contracts, which began in June 1998, was originally used primarily to conduct disaster response-related activities, and especially training. In May 2003, OFDA shifted its overall approach in the LAC region, now placing a greater emphasis on risk management. An examination of the Operational Plan reveals an important change of direction, although the overall OFDA strategic objective is focused primarily on response. 1 The focus of the IRG contract changed accordingly (minimally), with its activities increasingly focused on reducing regional vulnerabilities while simultaneously building local capacity to respond to disasters. IRG support contract activities continue to be training and evaluation focused in keeping with the thrust of OFDA/W but now emphasize technical assistance and training, although response remains the primary contract focus. Overall, the IRG support contract offers training, technical assistance, disaster mitigation, and disaster response support to twenty six countries in the LAC region. Activities carried out under the contract are tailored to the particular needs of each country. This report focuses on IRG, but in the context of an approach that has emerged in the period that is OFDA/LAC/IRG. The effort documents the types of assistance provided under the IRG support contract, assesses the effectiveness and sustainability of the activities undertaken, identifies areas of strength and potential improvement, and offers recommendations on new directions and the scope of a future institutional support contract in the LAC region. 1 A review of the Operational Plan for OFDALAC, the IRG 2003 contract and overall OFDA SO and IRs reveals some important distinctions in scope. CHECCHI AND COMPANY CONSULTING, INC The Louis Berger Group, INC Evaluation: USAID/OFDA/LAC December 2004 Risk Management Program 2 II. Evaluation Methodology This evaluation is a bit different from traditional Project or Program Evaluations. That is because the OFDALAC program is not a typical USAID program. The evaluation team has opted for a more grounded approach, possibly anthropological in tone at times, given the nature of the program. During the course of the field-work for this report, the team was able to visit USAID/OFDA/W for initial interviews and debriefing and the USAID/OFDA/LAC office in San José, Costa Rica for a period of approximately ten days. During that time the team interviewed both the OFDA team and the resident IRG team in substantial depth. In addition, a number of teleconference interviews were held with one Regional Advisor, IRG consultants and National Disaster Coordinators in Perú and various countries in the Caribbean.. The team also conducted interviews with principal OFDA clients in Costa Rica, both public and non-governmental. A fourth source of information for this report consists of documentation gathered in the OFDA and IRG offices. Finally, the team carried out an extensive set of field visits to Jamaica, Perú, Guatemala and Nicaragua over a period of three weeks. The joint (IRG, OFDA/LAC, Evaluation team) decision as to what countries would be visited as part of the field research was guided by several criteria, although an element of subjectivity is also evident: o Budget limitations that required a limit to three and possibly four countries in addition to Costa Rica; o Relative levels of complexity of coordination among institutions and country size; o Relative degree of success or failure of operations; o Relative levels of participation of local actors and (senior, in particular) IRG consultants for better scope or view of operations. Persons interviewed and offices visited are listed in Annex A. Interview Protocols that served as guides to information gathering are provided in Annex D. Itinerary of visits are provided in Annex C. Of importance as well is a debriefing exercise held with both OFDALAC and IRG staff where the effort was to elicit a better understanding of a very complex and unique USAID program. The evaluation is able to draw conclusions on the impact of the RMTP in the LAC by framing and directly relating impact to the strengths and weaknesses in OFDA/IRG’s identification of key stakeholder organizations (clients), assessment of client needs and vulnerabilities, delivery of course content and technical assistance, and client management. Impacts identified in this report are tangible products, tangible measures and outcomes, and indicators of risk management processes or “process indicators”. III. The Evolution of the OFDA/LAC Approach: Understanding the “Model” While the essence of this evaluation is in the period from 1998 to the present and the initiation of the IRG contract with OFDA, it is impossible to view the more recent period without reference to an emerging paradigm that began prior to the current contractual relationship. Indeed, the CHECCHI AND COMPANY CONSULTING, INC The Louis Berger Group, INC Evaluation: USAID/OFDA/LAC December 2004 Risk Management Program 3 “model” itself is the result of evolution, trial-and-error, and a growing internal sense of lessons learned from the past. The overall experience of OFDA’s engagement in Latin America and the Caribbean has been a highly positive one. The “model” for OFDA’s engagement throughout the region has evolved over the past two decades in response to very practical and grounded perception of needs and opportunities. This evolving approach clearly is the result of the overarching influence of Mr. Paul Bell until his unfortunate demise in 2003.2 The approach is unique, contextual, and historical in its evolution, and this may affect its replicability. From the early days (mid to late 1980s) through the present we see at least three critical junctures. In the period from 1986 through the middle of the 1990s, the approach was characterized by highly personal relationships, an emphasis on first response, a somewhat reactive and sporadic effort to strengthen individual stakeholder response skill-sets, and a limited sense of accountability in a programmatic or budget sense. It is in this period that the initial informal network of national colleagues (now, many of the national consultants) began.3 However, even at this early stage, institutional presence and action began. In Costa Rica, the CNE credits Mr. Bell and USAID/OFDA for efforts to move out of simple reconstruction efforts and for promoting the creation of the Directorate for Mitigation and Prevention in the early 1990s. Other national lead institutions reflect the same emerging relationship with national experts (later consultants), OFDA and the institutions.. Similarly, initial contacts between national firefighters and training and the physical contact with Miami-Dade County Firefighters began for a number of Latin American countries. Because the approach was disaster relief, the role of USAID and the MDRO were more significant. In 1987, that was manifest as almost pure disaster response and interaction and through the MDRO in AID missions that they supported. That relationship would change quickly as the realization that the institutional relationship with USAID missions was too cumbersome, wrapped in red tape and bureaucracy and that an alternative quick response had to be set up. The decision to use a contractor for training emerged (Partners of the Americas). The early efforts in the Eastern Caribbean (EC) began somewhat later but with a similar pattern. In 1989, training was conducted directly by OFDA/W (through U.S. consultants) and courses were primarily in TFI, disaster program management and relief, later in shelter management, and still later other courses were added. But a program did not exist A second period or juncture (mid-to-late 1990’s) is characterized by a deepening of the network through national long-term field consultants, a stronger emphasis on refining course offerings, deeper relationships with national institutions (not just individuals), and some integration of course content into client-institutional practices. However, course delivery remained the essence, while inter-agency coordination remained a long way away. Perhaps more importantly, the management style under the rather open-ended (grant) contract with Partners of the Americas left a significant gap in administration and management practices, and in any real sense of 2 There is/was a strong cult of personality in the persona of Paul Bell, Ricardo Bermudez, Alejandro James and René Carrillo, the “four horsemen” as more than one interview suggested. The links to the International Red Cross in those early years is also evident. 3 All but five of the current consultants were already part of the network prior to the arrival of IRG and the 1998 contract. CHECCHI AND COMPANY CONSULTING, INC The Louis Berger Group, INC Evaluation: USAID/OFDA/LAC December 2004 Risk Management Program 4 financial and cost-effective accountability. In the words of one of the participants, “It worked so well we became victims of our own success. We became overwhelmed by the demand for training to which we couldn’t yet respond.” The growth of demand overwhelmed the meager administrative capabilities and number of the personal and active leadership and even the small network. A third juncture or period (1999/2000 until very recently) is perhaps the most significant shift. The impact of Hurricane Mitch on the entire OFDA (and national) portfolio of activities cannot be underestimated. The “storm of the century” provided a laboratory for understanding (a) the need for deepening the response capabilities of national institutions; and (b) the need to shift emphases increasingly toward mitigation and risk management. It is here where OFDA begins to bring order and management to a relatively loose institutional structure, both in OFDA/LAC central and in the (new) IRG consultant network. Programmatic and budget accountability, regular and systematic client contact toward stronger institutional relationships, and an incipient thrust toward linking TA to the OFDA/IRG “in-house” process is evident, although response and training remained (remains) the bread and butter of the approach. The Central American Mitigation Initiative (CAMI) not only provided a substantial funding base for national governments to respond, but it informed the foci for the initial IRG contract (albeit the contract preceded the event). These emphases included: o A need to focus greater attention on risk assessment and vulnerabilities. o The need to address weakness (vulnerability) in institutions and the absence of inter￾agency coordination. o The CAMI initiative led to the development of the Plan País, OFDA/IRG’s more comprehensive and (inter-)institutionally focused risk management effort. o The need to focus on various actors and jurisdictional levels in response, and not just key national institutions. o An increased focus on more comprehensive vulnerability analyses that did not exist prior to this time, with a better understanding of economic, social, and physical relationships before, during, and after a disaster. This period also brought the most significant change in management and administration. Among the most significant changes were: o A business-like approach to service delivery (e.g., the incorporation of IRG South as a legal entity, staff-management relations, new legal parameters introduced in Costa Rica) o Strengthening of the financial management, budget and control systems with the advent of a financial manager with strong USAID experience o The clear separation of management from technical service delivery (the arrival of a permanent Senior Technical Manager) o A program approach to the design of activities (work plans, and eventually the Plan País) Finally, we are at another critical juncture. In part, this is driven by OFDA concerns with new contract relations and the constraints imposed by the current contract. In part, this juncture is one where one might question the relative emphases in the overall program (e.g., among response, mitigation, training and TA). And, finally, the current milieu provides an opportunity to reflect on future directions. Many of the observations and recommendations that follow CHECCHI AND COMPANY CONSULTING, INC The Louis Berger Group, INC Evaluation: USAID/OFDA/LAC December 2004 Risk Management Program 5 address what might be done and what should or could be new horizons for the program and the model. IV. Principal Evaluation Themes and Country Case Examples Four principal themes emerge from the evaluation: A. What has been the coverage of risk management activities under the IRG/OFDALAC contract in terms of types of activities, country coverage, and the client targeted institutions and populations? B. How has IRG/OFDALAC contributed to building national capacity and strengthening national institutions in response and risk reduction, and how has coordination evolved among national institutions, other donors, USAID, and NGOs? C. What has been the overall impact of IRG/OFDA’s program? D. How has the management and administration of the IRG contract evolved and what issues remain to address in management and administration as these relate to operational outcomes of the program? In each theme, the report begins with an overview of major observations and findings and follows with a discussion of key observations in each of the country site visits of the evaluation team. A. Coverage of Programs and Country Target Populations/Actors The IRG country portfolio has grown substantially over the course of the past five years, and now includes 26 countries in the LAC region, and operates through a network of 19 field consultants. Until recently, the program in the Eastern Caribbean has received less attention, following a lull in activities in the late 1990’s, and into early 2000. However, the East Caribbean represents the largest growth area for the RMP. The number of trained personnel in the client countries has grown steadily over the life of the IRG/OFDALAC relationship, although determining the number of participants still active is less clear. As can be seen in the following graphic, the numbers of participants and instructors trained under the program has grown, albeit with some variation among years. From a modest level of participants/instructors in 1998 of some 1300, the program has grown to a consistent average of more than 3100 participants and instructors over the remaining years of the program to date. Relationship Participants/Instructors by Year 1998 1999 2000 2001 2002 2003 Participants 1200 1896 2952 3642 2604 2745 % 0.92 0.95 0.87 0.80 0.82 0.86 Instructors 100 100 440 937 582 446 % 0.08 0.05 0.13 0.20 0.18 0.14 Total 1300 1996 3392 4579 3186 3191 The dominance of the Latin American presence in the program is evident. But in recent years, and as the program deepens, the emphasis on the Eastern Caribbean is clear. The numbers are small to date, but the growing adaptation of the course content to emerging English-speaking Caribbean needs is apparent. From an initial course (TFI) in 1998, and 8 participants/instructors, in 2003 there were 11 courses and 233 participants. Much more needs to be done in this region, particularly as some Latin American countries “graduate” from direct participation of IRG CHECCHI AND COMPANY CONSULTING, INC The Louis Berger Group, INC Evaluation: USAID/OFDA/LAC December 2004 Risk Management Program 6 consultants in course delivery. In particular, the problem of the need for more instructors is clear in the face of problems of retention. One other important lesson that emerges from the Caribbean case is the answer to the question of replicability. Th past several years appear almost as a new beginning after a substantial lull in OFDA/LAC relations and the presence of OFDA/W directly. But the application of the Latin American experience to the EC shows elements of the replicability OFDA/W is looking for. Courses by Year and Type: Caribbean Course Year 1 Year 2 Year 3 Year 4 Year 5 Year 6 Total TFI 1 2 0 0 1 0 4 DANA 0 0 4 2 0 2 8 SSM 0 0 0 1 7 4 12 IDA 0 0 4 6 2 5 17 Total 1 2 8 9 10 11 41 # Participants/ Instructors 48 24 192 216 201 233 Course Content and Changing Strategy. The second IRG contract coincides with a dramatic shift in strategy away from the existing emphasis on courses and response. However, the course content of much of the training remains response oriented. This is in keeping with OFDA´s strategic objective. As can be seen in the following graphic, first response and forest fires still represent the majority of training. In all of the country visits, the emphasis on the need for a stronger presence in risk mitigation and prevention was proffered by client respondents. While a shift toward risk reduction is evident in some of the country programs and in the graphic, by far the institutional framework - and OFDA training – remains response driven. In some cases, there was concern expressed about course content updating to current realities and the need for new response modalities (e.g., chemical disasters). Technical assistance has become an increasingly evident part of the strategic portfolio. However, there is a greater need for OFDA/IRG to play a role in technical assistance beyond its current scope, and in areas where to date little has been done (legal framework, regulatory structure, land use planning, etc.). A further area of some discussion in the field and with IRG and OFDALAC staff has been the need for management and programmatic decisions about the “success” of transfer of technology and the passing of responsibilities to national institutions and staff. Some countries (in the Latin American case) may have matured to the point of “graduation,” particularly in terms of course instruction Two of the five country visits reflect this issue. In both Peru and Guatemala, the level of institutional maturation is very high, and one might question the role of IRG consultants in terms of the level of effort nationally and in terms of the kinds of activities pursued. This is not the case in the Caribbean countries, although in the short period of “re-engagement” substantial advances have been made. In training, the role for senior consultants in at least these two countries has diminished substantially: even with problems of rates of retention of instructors, both countries have graduated with qualified national instructors. But, the implication also is that new emphases and roles should be emphasized (and to some degree these are becoming evident in strategy): new courses responding to new technologies and events, greater institutionally focused TA, more intense work with entities outside the national institutional framework (NGOs, local community organizations). And, perhaps, a lower level of effort nationally. CHECCHI AND COMPANY CONSULTING, INC The Louis Berger Group, INC Evaluation: USAID/OFDA/LAC December 2004 Risk Management Program 7 T/TA Distribution by Year 0 20 40 60 80 1998 1999 2000 2001 2002 2003 1998 16 19 0 6 11 0 3 0 1999 27 24 0 15 16 0 2 0 2000 49 38 2 46 20 0 4 4 2001 46 40 7 71 27 0 0 5 2002 32 27 5 56 6 1 4 15 First Response Forest Fires School Safety Risk Management Methods of Instruction Facilitation D y D Technical Assistance Targeting RMP Clients in the LAC. Key to IRG activities in every country is the national emergency organization, and the firefighter organization(s) as first response institutions. From the IRG perspective, the Plan Pais and its diagnostic tool provides the key input to determining institutional strengths and weaknesses, and potential programmatic objectives. The work plan and programming exercises are conducted by each consultant, and the program review is conducted in unison with other IRG consultants and management. Because the country diagnostic must take into account numerous positive and negative factors, OFDA/IRG seeks to identify the broadest possible audience. For example, it is sometimes the case that political sensitivities to intervention training make it preferable for IRG to work at the functional or operational levels. Other times, it is preferable to court upper-level decision makers and political leaders in order to establish a sustainable program. Specifically, the criteria used to select clients and assess program needs relate to organizational weaknesses found throughout the LAC region: small client budgets, lack of continuity/high rate of attrition among recipients of training, institutional turn-over, institutional and geographic territorialism, and little institutional or external support for risk prevention and mitigation programs. These factors should be considered “contextual constraints” to the RMP’s overall effectiveness and impact. In response to these limitations or obstacles, IRG consultants have difficulty monitoring not only course training, but also the utilization of the information itself. To counter the difficulties in capacity monitoring, IRG consultants rely on each other and share experiences with client selection and country diagnostics through annual regional meetings. The institutionalization of this form of internal communication is critical for programming individually and collectively. This approach enables consultants to collaborate in problem-solving and targeting new client organizations as candidates for course instruction toward building additional risk management capacity in the LAC. There are several of “internal constraints” (OFDA/IRG) that affect the reach of the country diagnostic, identification of clients, and client needs assessment activities. First, the shift from an emphasis on response to risk management is a recent one. Given OFDALAC’s 20-year legacy of response-oriented field-work, operationalizing risk management principles will take more time to achieve. Second, OFDA/IRG engages the LAC region through 19 consultants covering CHECCHI AND COMPANY CONSULTING, INC The Louis Berger Group, INC Evaluation: USAID/OFDA/LAC December 2004 Risk Management Program 8 26 countries, and produces strong results as measured in absolute terms or in cost-benefit term; however, the resources are stretched. The focus on key stakeholders at the national and institutional level is – and will remain - a priority for program continuity. As a result, the RMP has had less opportunity to reach out to local level institutions such as community leadership, action committees, and parent groups. These populations and groups make up the first-impact and first-responder landscape and require greater attention in the program. In the field visits by the evaluation team, some of these factors are apparent. In the following sections, these country contexts are highlighted. One of the more poignant findings of this evaluation has been the overwhelmingly positive perception of IRG field consultants, and their role in establishing strong client relations and program content in conjunction with national institutions. A clear lesson from the model emerges: being on the ground counts. Costa Rica. Costa Rica is somewhat unique in its risk management capacity profile because of comparatively few disaster events, and its position as the long-standing centerpiece and headquarters for OFDALAC’s risk reduction and response program. Today, OFDA/IRG has a smaller operational role in its client’s activities, but is greatly valued by clients for its high credibility in furthering the country’s disaster management interests with national, regional, and international development entities. OFDA’s main client, the National Emergency Commission (CNE), is able to meet a variety of risk management challenges. Ironically, the relationship with CNE is strong but intermittent. While the CNE actively develops (inter) institutional programs and coordination competencies, it is the firefighters who appear to be the active recipients of OFDA activity in recent years. The presence of direct OFDA activity in training is less apparent today because in many ways Costa Rica has also “graduated”: self-sufficiency is evident. The strong institutional capacity here is a function of earlier OFDA work, and in particular the presence and participation of firefighters from the Miami-Dade County contract. In the case of the national Red Cross, OFDA interaction has waned dramatically from earlier years. The loss of a strong historical personal relationship has changed the interaction with OFDA. Relationship with OFDA seems to be limited to the provision of equipment, logistics during relief efforts but not as strong in training today. Since 2003 (and the arrival of the IRG consultant for the country) there has been some renewed training (SCI, BREC) but more often with didactic materials rather than courses. Jamaica. OFDA’s approach to client selection and coverage for Jamaica focuses on its national disaster authority, the Office of Disaster Preparedness and Emergency Management (ODPEM). OFDA/IRG presence is strengthened immeasurably by the presence of a very active senior IRG consultant who has developed strong relations with both national institutions and USAID. OFDA works with ODPEM to conduct profiles of ODPEM partners in emergency management, like the Red Cross. For Jamaica, the strength of the RMP is limited, in large part, by the contextual weaknesses found throughout the Caribbean: Cabinet-level and client budget constraints, lack of continuity/high rate of attrition among training recipients, a highly centralized (ODPEM) disaster management structure, and little institutional, legislative, or external support for risk prevention and mitigation programs. Respondents indicate that attrition rates are likely owing to the fact that between 1991 and 1994, OFDA offered instructor training but no courses, and from 1995-2000 offered courses, but no instructor training. Qualified personnel simply move on. Only since 2000 has OFDA provided both capacity building products. The appropriate “clients” for action emerge here clearly. In response to Hurricane Ivan, the level of disaster preparedness and damage assessment capability appeared limited at the CHECCHI AND COMPANY CONSULTING, INC The Louis Berger Group, INC Evaluation: USAID/OFDA/LAC December 2004 Risk Management Program 9 level of local institutions; the parish councils were not as successful as expected either in assessment or in communication. Peru. While the size of the country, its geographic challenges, cultural variations, and range of hazards (earthquakes, floods, landslides, forest fires, drought, and El Nino) seem daunting, the challenges are well matched by Peru’s institutional resources. Peru has a dynamic and effective risk management profile. OFDA/IRG RMP country diagnostic and client needs assessment divides northern, central and southern regions according to the unique contextual factors particular to each. In Lima, client groups who have benefited most from the RMP are INDECI, the Ministry of Health, Ministry of Education, Ministry of the Interior, and the Volunteer Fire Department. Secondary beneficiaries are the National Police, the Peruvian Army, and International Red Cross. Overall, Peru’s legacy of military command and control over civil defense issues, and over the public sector, is evident. In the south (Lima), the military’s tendency toward a self- insulation and self-protection is particularly strong. IRG has tried to circumvent such forces by promoting its consultative process of assessing client needs as “an opportunity to improve its own methods and course content.” The IRG senior consultant indicates that this tactic has met with limited success. One message that emerges from the Peru context is that client responsiveness requires new efforts with changing partners. There is a need to strengthen strategic alliances for the longer term. For example, opportunities present themselves to solidify ties with universities to strengthen formation for risk management in a more formal way. IRG consultants also point to the need to create a more integrated and forward looking training program with the private sector and NGOs, something that these groups have asked for. Peru’s NGO community exhibits particularly strong organizational maturity, and provides an ideal partner for future OFDA/IRG activity. Municipal development and decentralization, or regionalization, are major thrusts of the government and OFDA can and should move to support these efforts. Today, we see this effort in the case of regional efforts in Piura, and to a lesser degree, in Cuzco. The case of regional efforts in Piura were visited by the evaluation team as a possible model for promoting interaction among various types on actors in risk management and response. In Piura, the RMP’s diagnostic and needs assessments differ from those in Lima, because of the complexity and extensiveness of risk vulnerability, that is, exposure to five types of risk on a continuous basis, and frequent impacts from combined hazards including El Niño, earthquakes, floods, drought, and landslides. As such, Piura’s risk management community and its citizenry have a conscious and heightened perception of risk which, according to respondents, “permeates daily thinking and concerns.” However, the geography/hazard complex creates great differences in community risk management capabilities, and OFDA and its clients struggle to build equal capacity across communities far from the regional capital. Moreover, Piura, with an agro￾industrial (cotton) and fishing economy, is impacted by flood and drought damage in the near￾term, and economically depressed by flood, drought, and El Niño in the long-term term. Finally, the prevalence of flooding has brought about frequent outbreaks of dengue throughout the region. Client groups most benefiting from the RMP are the Piura Regional Government (PRG) the Ministry of Health, Ministry of Education, Ministry of Agriculture, INDECI’s regional division, Plan Piura (formerly Plan International, locally-based disaster management NGO), CARE, and the University of Piura. Secondary beneficiaries include municipal (sector) commissions, and local (mayoral and citizen) committees. CHECCHI AND COMPANY CONSULTING, INC The Louis Berger Group, INC Evaluation: USAID/OFDA/LAC December 2004 Risk Management Program 10 Guatemala. Guatemala appears to be another country close to “graduation.” In selecting appropriate RMP clients and terms of engagement, OFDA/IRG takes into account numerous factors: its coastal and mountain landscapes, its risks to earthquakes, floods, landslides, forest fires, and hurricanes, differences in risk perception and cultural practices among its numerous Mayan communities, Guatemala’s long history of military command and control influences on the public sector, and the fact that, as of 1999, its national disaster management authority (CONRED), is civilian-led. The country diagnostic is particularly informative in the identification of key socio-cultural and institutional factors. Assumed in the country diagnostic and client needs assessments are the ubiquitous challenges to maintaining institutional capacity, such as institutional turn-over, shifting priorities, inter-agency political gamesmanship, and funding constraints. The key to Guatemala’s success has been the intense development of CONRED and a very well seasoned and trained firefighting team (public and voluntary), in part due to the earlier and continued presence of OFDA and now IRG. The RMP has built capacity built within CONRED (National Disaster Response Council), Ministries of Health and Education, and among both Volunteer and Municipal Fire Departments. Secondary beneficiaries are the National Police and the Guatemalan Army trained by OFDA-certified client instructors. Finally, because CONRED has built strong relationships with 17 municipal and community-based NGO’s, OFDA/IRG is well positioned to build new relationships and RMP capacity among first-impact populations. Discussions with members of the NGO community point again to the need to strengthen OFDA/IRG relations with alternative clients and institutions to the traditional national institutions. This thrust is important in Guatemala because of the maturity of NGOs such as CARE, Caritas, and others. Nicaragua. Nicaragua recently (2000) transitioned from a Civil Defense led emergency response authority to a national risk management coordinating authority (SINAPRED), driven by an Executive Secretariat and an Operational Commission. Overall, OFDA/IRG clients are pleased with the shift, indicating that OFDA has been instrumental in gaining institutional support. In particular, client organizations point to a growing role that OFDA/IRG has played in providing technical assistance for sectoral planning and even legislation. However, the effort remains principally response focused on the part of OFDA/IRG interventions (courses). OFDA/IRG’s main client institutions (SINAPRED, Ministries of Health and Education) are strong at the national level (Managua), but little capacity at the departmental or community levels exists. In the sector ministries and in government efforts to decentralize, the opportunity to reach beyond traditional clients is evident. OFDA is well-positioned to continue to work with SINAPRED to develop strategies for extending the RMP to Nicaragua’s rural community-based organizations. Groups benefiting most from the RMP are SINAPRED, Volunteer and Municipal Fire Departments, Ministries of Education and Health, and Civil Defense. Secondary beneficiaries include the National Police and the Nicaraguan Army. B. OFDA’s Role in Building National Capacity and Coordination A primary objective of the RMP is to empower national risk management stakeholders with disaster preparedness and response capacity (self-sufficiency). The secondary objective is to enable “graduated” disaster organizations to meet the next set of challenges, namely risk prevention and mitigation programs that link risk management practices to national development plans. Building national capacity is the essence of the program. Regardless of the frequency of crisis events, decentralized client (response) capacity enables OFDA/IRG to move a client and/or a country toward reduced vulnerability. In general, the Incident Command System (ICS) course CHECCHI AND COMPANY CONSULTING, INC The Louis Berger Group, INC Evaluation: USAID/OFDA/LAC December 2004 Risk Management Program 11 and the Facilitation Skills course embody OFDA’s skills and services package for building risk management capacity both within and across organizations respectively. Capacity Building among National Institutions. The OFDA/IRG team has produced a high level of service delivery in the countries targeted by the evaluation. OFDA/IRG has the greatest success building capacity within national disaster authority and first-responder institutions at the national level. Certified client-instructors are relied upon to reach out to regional and municipal divisions. Because of the endemic institutional problems with retention and turn-over, the RMP strategy is to reinforce capacity at the national level where retention is less of a problem. To a lesser degree, the RMP has built capacity with local-level governments and community groups based on its current resource-use priorities. Activity at this level is increasing, is clearly the direction to move, and is based on interaction with the NGO community. Technical Assistance and Responsiveness to Client Capacity Needs. OFDA-IRG is perceived generally as effective in building capacity through technical assistance as well as courses. A telling indicator of this responsiveness and cultural affinity to clients is seen in Guatemala’s CONRED where choices between a “FEMA approach” and an “OFDA approach” divide the training of a number of staff members (see below in Guatemala). With respect to technical assistance, IRG consultants personally monitor client programs, though, for example, the damage and needs assessment (DANA) program’s effectiveness requires an emergency to evaluate. While consultants interact with clients on program design (disaster program administration, damage and needs assessment, and shelter management), they rarely jointly design training course-content, and only minimally conduct follow-up needs assessments. Client groups indicate a need for OFDA/IRG to increase the continuity of consultant contact with clients, and extend the RMP’s consultative methods for technical assistance into its course design process. Respondents perceive this move as a way for OFDA to become more flexible and responsive to client needs, and for trained clients need to better equip themselves to apply these skill-sets to changing demands and conditions within organizations. Two areas in particular have been noted: courses and training in chemical and biological material and simplified courses and materials for local level training. Finally, given its current human and financial resources, OFDA/IRG builds inter-agency coordination by identifying and creating opportunities for institutional clients to communicate with each other and coordinate through forums, workshops, conferences, and planning cells within countries, and replicate or regionalize the developmental process of inter-agency coordination in other countries. As such, OFDA/IRG’s true value-added in the LAC region is found in the range of close relationships it maintains with key decision-makers across stakeholder institutions. This evaluation team believes that OFDA/IRG is well positioned to increase political liaison activities, and develop a platform for linking national development and disaster management authorities at the Ministerial and Cabinet levels. It is the key to any interest it might have in building the relief-to-development linkage. One area that has seen increasing support for capacity has been in forest fire prevention and support. OFDA/IRG’s forest fire prevention training has been extended to stakeholders throughout the LAC. Although country officials often differ with OFDA on course content, timing of implementation, and dollars needed, with limited funds, OFDA proactively follows up on course planning, and monitors launched programs, which has led to a positive impact on capacity building. There are some 170 volunteer, community-based educators and 750 volunteer firefighters in the LAC. The Forest Service’ (OFDA) course content and objectives are integrated into the policies that guide the National Development Plan (CNE), which integrates all CHECCHI AND COMPANY CONSULTING, INC The Louis Berger Group, INC Evaluation: USAID/OFDA/LAC December 2004 Risk Management Program 12 risk management stakeholder organizations. Today, BREC courses are standardized in 17 Central and South American countries. OFDA and Coordination. OFDA/IRG¨s role in coordination takes a number of forms. First, OFDA has played an important role in strengthening inter-agency coordination within national contexts. This role has a long history in some of the case countries (Peru, Guatemala, Costa Rica), and is more incipient in the case of the EC. The move toward TA activities in the recent past provides significant opportunities for expanding that role to more than simply inter￾institutional coordination in the formal sense. Activities in the legislative/regulatory sphere, activities in strengthening linkages with non-governmental and local organizations, activities in the sub-national levels of government, and even with private sector organizations in both response and mitigation are windows of opportunity. In addition, recent course development (SCI, for example, or “light” courses in response) provide real opportunities to both strengthen inter-institutional relationships and to reach to new partners. One of the areas of concern and interest is the need to strengthen regional (cross-national) and international coordination efforts. In each LAC sub-region (South America, Central America and the Caribbean) there are ongoing OFDA/IRG efforts in courses and in communications. The strengthening of relations with CAPRADE in the Andean region has been very effective in giving greater regional flavor to risk management activities. In the Caribbean, the incipient relations with national disaster organizations and the Caribbean Development Bank (CDB) provide improved mitigation and response capabilities, although strained relations with CDERA give some room for pause. However, severe financial limitations hinder these regional efforts, especially on the part of nationals. The cases in Central America of both the forest fire and firefighters attempts to coordinate regionally deserve mention. Closer interaction with existing regional organizations is a priority and should be strengthened. Current interactions through UN INSARAG, CAPRADE, UNDAC, PAHO and others are important for OFDALAC and IRG, albeit it is not clear that they are a part of the IRG contractual relation but go beyond formal obligations. Coordination with USAID Missions. While the team’s contact with missions was limited, several observations are warranted. The early decision (1988) to move activities from USAID MDROs as a cumbersome mechanism was probably correct, but one result is a continuing image of OFDA/LAC as a loner. In the case of OFDA/IRG this image is also evident and consultant relations with MDROs is a mixed bag, and is a function of both the consultant and the MDRO perception of their roles. In some cases (Jamaica) the relation is strong and interaction is constant. In others (Peru), the IRG consultant has had little interaction directly. In part, this is also a function of the MDRO as primarily a response mechanism, not one involved deeply in risk management or mitigation. In addition, it is through the OFDA Regional Advisors that this relationship is most evident. The RA structure provides the OFDA/IRG consultants with greater access to USAID, is important well beyond the IRG contractual role in areas more related to policy and political interaction, reflects the authority of OFDA, and provides a needed liaison role to other USG agencies and international organizations. In addition, MDROs often are not OFDA trained. Further, the interest of MDROs in this added role to an already full work agenda is dependent upon individuals. With respect to measuring institutional capacity and inter-agency coordination, one final observation should be taken into account toward improving client relations and capacity building efforts: while nearly 34,000 people in the LAC have been trained, it has proven difficult for OFDA/IRG to track the existence and the use of such capacity within countries, though new CHECCHI AND COMPANY CONSULTING, INC The Louis Berger Group, INC Evaluation: USAID/OFDA/LAC December 2004 Risk Management Program 13 tracking programs are under development. In spite of advances in the TTS, there remain difficulties in determining the numbers of active and inactive course recipients actually on the ground. 4 Costa Rica. OFDA/IRG’s main client, the National Emergency Commission (CNE) has “graduated” to manage emergency preparedness and response activities without much need for external support. In part, this is a function of earlier OFDA support for the institution. However, much of that earlier interaction was of a highly personal nature, a relationship that is only now beginning to reemerge with the appointment of the IRG consultant. OFDA’s reduced role also reflects a reduction in the nation’s institutional vulnerability to natural hazards. Through the National Emergency Commission, OFDA/IRG has built a solid and decentralized risk management profile for Costa Rica, and reduced the need for OFDA/IRG’s vigilant reinforcement and expansion of client capacity. Clients act independently, and OFDA/IRG personnel provide mostly political and some programmatic support to clients as needed and determined by the clients themselves. The Red Cross exhibits a high level of risk management capacity, achieved through OFDA training, and consistent interaction with the CNE. The Red Cross’ public approval rating is 97.5%. The Ministry of Education oversees its course (DANA, ICS, TFI) certification needs, and was empowered by OFDA to perform these functions. As such, OFDA has not directly built the overall capacity of the Red Cross, especially when compared to OFDA’s ongoing commitment to building capacity for the Fire Department. Red Cross leadership expressed a great deal of frustration in describing its interaction with OFDA, saying that frequent requests for training have not been met. But, because the Fire Department is a legislatively mandated first-responder, priority is placed on addressing its training needs. Another potential source of weakness in OFDA-Red Cross relations, ironically, may be due to the success of the Red Cross: The Red Cross has internally constructed a network of some 6,000 volunteers through its community outreach programs – volunteers range from members of rural communities to doctors, assembled across the country through 120 committees The Fire Department exhibits the highest degree of capacity among all fire departments studied in this evaluation and has benefited from long-standing relations with OFDA.. This unique status is the result of a strong and permanent flow of funds to the department, a product of its “siting” within the public monopoly of the INS, the National Insurance Institute. In addition, this location has pushed the firefighters in the direction of mitigation and risk management more rapidly. Because it is the most “advanced” fire department in the LAC, there is little current need for traditional training and TA. OFDA/IRG focuses almost exclusively on client relations and management, and less on capacity building. In fact, both volunteers and paid firefighters have undergone the APAA, SCI, HAZMAT, BREC, and EDAN courses and serve as regional trainers. Prevention efforts have grown since 1999, especially through advances in chemical analysis and fire detection systems. Specifically, its engineering department links fire prevention and preparedness, and so firefighters directly reduce the potential need for crisis response situations. The Fire Department has 400 full-time firefighters and 1000 volunteers located in 71 stations throughout Costa Rica. Here, graduation has occurred. The Fire Department, with many positive lessons learned from its contract with the Miami-Dade Fire Department, recently (2002) expanded its collaborative agreements to work with Fairfax County, VA and Los Angeles County, CA. Impacts on capacity building and coordination have 4 The TTS has an impressive list of over 5,800 participants in the variety of training activities, by gender by country by training and whether active of inactive. CHECCHI AND COMPANY CONSULTING, INC The Louis Berger Group, INC Evaluation: USAID/OFDA/LAC December 2004 Risk Management Program 14 been extremely positive by all accounts, both in terms of program development and management. The year 2003 marked a milestone for inter-agency coordination between OFDA and the department with efforts toward regionalization in Central America. With momentum from cooperative agreements, the Bomberos and OFDA began an effort to increase communication and coordination between fire departments in Central America and the Caribbean. The Confederation of Central American and Caribbean Firefighters communicates and coordinates capacity building through conferences, workshops, web pages, and publications like the OFDA￾funded International Firefighter quarterly periodical, published in Miami, FL. For the Forest Service, the ICS and BREC courses, first offered in Costa Rica in 1998, have been the key to its internal capacity, and to its efforts to build forest fire response capacity within the Fire Department, and within municipal and local-level governments and volunteers Strong volunteer response has led to the development of a forest-fire response network comprised of more than 300 citizens from a variety of backgrounds and communities throughout Costa Rica. Additionally, many (unknown number) citizens are instructed in the basics of forest fire awareness, which also supports emergency response and capacity. Retention, Continuity and Follow-up. But, IRG consultants estimate that while 70% of people within key client organizations who have received course training do not use the information on a regular basis, it has little to no impact on institutional capacity. However, while emphasizing client independence, they conclude that the heavy work-load for client functionaries makes IRG’s response to clients’ changing priorities and coordination with tasking difficult as a matter of clients’ needs versus wants. Clearly, the RMP’s success should be closely guarded. If this is a trend within this client sector, it runs contrary to what we know about the relationship of skill retention and utilization. In addition, it is vital for the RMP to maintain vigilance over past successes, if, in the future, it chooses to develop further its risk prevention and mitigation programs. The RMP has supported strong inter-agency coordination in Costa Rica. The National Emergency Commission independently manages a comprehensive set of multi-hazard, multi￾sector risk management activities. In preparedness and response, it leads planning and operations with the Fire Department, Forest Service, Ministry Officials, Embassies, Red Cross, and others. Its prevention and mitigation activities integrate meteorological scientists on flood and hurricane risk prevention, (university) earthquake researchers on GIS mapping on at-risk populations, Ministry of Education on hazards awareness in primary and secondary education, and fortifying the two largest hospitals in San Jose. Jamaica. Among the countries of the Eastern Caribbean, Jamaica exhibits the most advanced levels of institutional capacity and inter-agency coordination. But, like most of the EC, OFDA/IRG has lagged behind the LA region.5 OFDA/IRG have provided Jamaica’s Office of Disaster Preparedness and Emergency Management (ODPEM) with the institutional capacity to deliver training to institutional stakeholders throughout the country. Because ODPEM is relatively new as the lead institution, capacity building and coordination remain principal 5 In teleconferences with other IRG consultants and NDCs, the recognition that there is some “catching up” in both course and TA work with less developed institutions is clear. A significant lull in activity until the end of 2002, and an apparent loss of some credibility for OFDA for a time period, have made catching up even more important. National Emergency Management Organizations and Agencies (NEMO and NEMA) are relatively new in several countries, a sign of the need for capacity building activities. St. Lucia appears to have substantial advance, but issues such as sufficient trainers locally and regionally remain prevalent. CHECCHI AND COMPANY CONSULTING, INC The Louis Berger Group, INC Evaluation: USAID/OFDA/LAC December 2004 Risk Management Program 15 concerns, and the relative politicization of disaster response activities has hampered some action. OFDA/IRG has consistently engaged ODPEM to develop shelter management and damage and needs assessment programs, and instructor training. In terms of ongoing RMP support, the IRG Senior Consultant, consistently interacts with ODPEM toward RMP improvements, and is called upon to serve in a variety of advisory roles, including serving as a panel member on the upcoming National Program Review. ODPEM has two individuals with the capacity to train throughout the Caribbean, but do so very infrequently. ODPEM’s regional coordinators (4 regions) exhibit a moderate degree of variation in their responsiveness, commitment, and transparency with OFDA/IRG. Inter-Agency Coordination and Retention of Instructors. While ODPEM clearly has the responsibility for motivating and managing its regional and parish-level personnel, they frequently look to IRG for such improvements. However, retaining qualified instructors in-house is a serious difficulty: only about 2 out of 24 individuals who undergo training will qualify to become trainers. As a result, proper implementation of strategic objectives for damage assessment and shelter management programs is greatly hindered. The aftermath of Hurricane Ivan made these shortcomings evident, as damage assessment data was never received by ODPEM or the AID Mission Disaster Relief Officer. But, “contextual constraints” greatly undermine OFDA/IRG’s efforts to build additional capacity: ODPEM’s limited ability to oversee its regional and parish-level personnel, high turn-over of trained personnel at the parish level; difficulties identifying new instructors at the parish level; Parish Council members individually represent Ministry sectors, and, therefore, compete for funding sector-specific issues. The single greatest obstacle to building internal, sustainable capacity to assume many of OFDA’s response functions is lack of adequate funding. ODPEM cannot internally fund basic program activities, including didactic materials. ODPEM is structured to interact with Ministries, first￾responders, and the Red Cross on emergency preparedness and response planning and operations. But, as in the rest of the Caribbean, limited interaction with regional stakeholders takes place, as little to no funding exists within ODPEM or OFDA for regional coordination activities, regional response exercises, or follow-up training. Moreover Jamaica’s Cabinet will not fund regional coordination activities for ODPEM, as line-item funds are withheld for sector emergency response. An area for improved possibilities of coordination with non-traditional actors is part of an informal agenda expressed by some members of the ODPEM team. Private sector participation, greater strengthening of municipal or parish councils, work with the university, and other local groups could provide a venue for increased OFDA/IRG action. The NGO community generally is not strong in the EC. The Red Cross has been a long time actor, especially in the absence of a national coordinating institution, and continues to interact with OFDA in setting priorities for its damage and needs assessment program. Shelter management is also a core area of joint program planning, implementation, and post-event program evaluation. Retaining instructors at the level of parish committees (made up of community-based first responder agency personnel) and community (citizen) councils is difficult. Trained personnel often move away or into other fields. Also, changes in risk perception affect levels of continued interest: immediately after Ivan the Red Cross was inundated with requests for training, but as time passed, the level of interest in training and the sense of urgency faded. Red Cross sees OFDA as invested in the concept of risk management and for developing increased risk management capacity in its national counterpart CHECCHI AND COMPANY CONSULTING, INC The Louis Berger Group, INC Evaluation: USAID/OFDA/LAC December 2004 Risk Management Program 16 institutions. OFDA has assisted in identifying vulnerabilities and mitigation opportunities; however, no funding exists either within Red Cross, OFDA, or ODPEM for implementing projects such as land-use and flood-plain management, and retrofitting critical facilities or schools. Peru. Peru’s institutional capacity and level of coordination among national, regional, local and non-governmental institutions is extremely well developed. The array of organizations involved, the depth of OFDA interaction with these organizations over a number of years, and the high degree of in-house training and TA capacity. Peru has “graduated” in an institutional capacity sense, which has implications for OFDA/IRG and IRG consultant roles. The existence of a senior and two “junior” IRG consultants may seem to be “overkill.” The National Institute for Civil Defense (INDECI), by legislative mandate, integrates disaster preparedness and response activities through all sector ministries and first-responder groups, including the National Police, the Peruvian Army, and, of course, the Fire Department. OFDA has played a key role not only in strengthening INDECI, but has been credited with legitimating its activities. According to INDECI’s Executive Director, OFDA’s course content, including its methods for coordinating all stakeholders, from Embassies to first-responders, directly shapes its (planning) commission and committee structures, its (response) operational structure, and its program administration practices. OFDA has guided INDECI toward the establishment of multi-sector (Ministerial) commissions, charged with integrating its working groups from national to municipal levels. Working groups are based on subject areas: mayoral governance, communications, education and capacity building, public health, law/order/technical inspections, science and technology, planning, and inter-departmental coordination. The commissions reflect the multi-sector Civil Defense Committee structure at national and regional levels. However, in spite of the formal mechanisms for coordination and integration, INDECI is not seen as an effective mechanism for fulfilling this role by some institutional members of the disaster mitigation and relief effort.. In part, this is a result of a continuing (military oriented) response mentality, and an effort to centralize control. The issue of inter-institutional jealousies, and the reaction to INDECI´s long-standing military posture, has hampered the role of OFDA. During the 1990s, and until 1999, INDECI basically shut the door on OFDA and claimed “national security” as the reason, and USAID generally did not play a role. Ironically, CARE and CARITAS played a more significant liaison role in those years, in spite of deep suspicions about NGOs in government (especially military) circles.. In response terms, INDECI has expanded its interaction with several institutions, again suggesting the depth of institutional capacity and coordination. In 2004, INDECI, in conjunction with the Ministry of the Interior (MININTER), initiated a training cell with the Army in order to multiply damage and needs assessment (DANA) capabilities. In building on the military’s training in force protection (civil unrest), the project will train 2000 personnel over the next 2 years. The Ministry of the Interior consults with INDECI through a multi-sector panel to determine annual budget requirements. Budget focuses on CPI, APD, and EDAN training in regions throughout Peru. Because of limited human and financial resources, EDAN training is conducted on a continuous basis because of institutional turn-over, and on a rotating basis based on seasonal shifts in the country’s risk profiles (floods, forest fires). Disaster response simulations are formal components of EDAN training after completion of coursework. Volunteer Fire Department. OFDA has built great capacity within Lima’s Volunteer Fire Department over a number of years, and the image of this relationship and its contribution to CHECCHI AND COMPANY CONSULTING, INC The Louis Berger Group, INC Evaluation: USAID/OFDA/LAC December 2004 Risk Management Program 17 evident capacity is strong. There are approximately 1300 OFDA course-trained members, and concrete plans to expand the number to 2500 by 2008 reflect this commitment. In most cases, the transfer is quite complete and self-sufficiency is evident. Courses include TFI, APAA, APAA-A, BREC, EDAN, APD, and PRIMAP. The department’s commitment to capacity is obvious in light of limited resources characteristic of volunteer departments. In spite of very strong institutional presence in Lima, the inter-institutional coordination among key actors is affected by contention between INDECI and the firefighters. OFDA/IRG consultants have not been able to assuage these jealousies, in spite of strong relations with both institutions, yet it is a role that they can play. This contention stems from two factors or reasons. First, as a volunteer department, they are perceived by INDECI as an employee rather than a valued partner. Second, the Fire Department allegedly does the vast majority of disaster-related work on a daily basis, responding to “every conceivable type of event”. INDECI, by contrast, goes into action only during catastrophic events, is “undisciplined, and “poorly managed” (according to the firefighters and others). In response, INDECI and MININTER officials indicate that the Fire Department has created the conflict by refusing to provide instructor training for the PRIMAP (level 3) course. Given the civil protection, site containment, and forensic analysis priorities addressed by PRIMAP (chemical spill events), and, that adequate response requires input from Fire Fighters, as well as Police and others, INDECI believes that the Fire Departments’ unwillingness to share its capacity in this area is detrimental to the entire risk management system. In some sense, the IRG consultant needs to address these inter-institutional contentious relations to assure the flow of action and information. Inter-ministerial coordination is also reflected in the sometime participation of the Ministry of Health (MINSA) in first responder activities and simulations. Again, OFDA/IRG have been instrumental especially in course training (CPI, APD) in line with the growing concern in MINSA in extending its training capabilities.. Turn-over among trained individuals and trainers is a recurring obstacle to capacity building continuity within MINSA. Nevertheless, MINSA has been able to “contextualize” the CPI and APD courses according to Lima’s current risk management trends and conditions. Finally, the maturity of institutional capacity is reflected in the existence of specialization in disaster medicine at the post-graduate level at the University of San Marcos, School of Medicine. The program integrates elements of emergency medical response, trauma/emergency surgery and triage skills, and OFDA’s Incident Command System. The program is developing long-term health risk management understanding, by combining current foci with epidemiological course work. In addition, through a signed MOU, the university meets three times per year with the Ministry of Health and INDECI to share developments in risk management practices. Piura again presents an exciting advance in inter-institutional coordination and capacity building, and in programmatic terms for this year, represents a significant investment for the IRG consultant. This coordination is even more relevant because it is decentralized, involves the NGO community, and builds on community recognition of extreme disaster vulnerability in the region. OFDA began to achieve capacity building success in Piura in 1996, through the Piura Regional Government (PRG). EDAN and APD continue to be the centerpiece of the PRG’s in￾house risk management efforts. But, the PRG ‘s Disaster Management Division recently formed the Disaster Management Group (RMG), linking the Ministry of Health, Ministry of Education, Ministry of Agriculture, INDECI’s regional office, the PRG’s Planning Department, Plan Piura (formerly Plan International, a locally-based disaster management NGO), CARE, and the University of Piura. CHECCHI AND COMPANY CONSULTING, INC The Louis Berger Group, INC Evaluation: USAID/OFDA/LAC December 2004 Risk Management Program 18 OFDA/IRG’s reinforcement of its capacity building activities has grown steadily within regional institutions, and through increased efforts to expand capacity into rural communities: Ministries of Health and Education, Plan International, and the PRG receive ongoing technical assistance from the IRG consultant, who serves as an advisors and RMG members. OFDA/IRG and the PRG initiated an Information, Education, and Coordination campaign with local municipalities to reach out to citizens on the risk of dengue; In 2004, OFDA and the PRG began community￾based simulations and response exercises; As of August, 2004, OFDA and the PRG completed a pilot-project, multi-community disaster preparedness and response plans between Piura, Tambo Grande, and Curamori. OFDA and the PRG are beginning the next pilot in 2005. The RMP’s success at building capacity has led to increased inter-organizational coordination beyond institutional frameworks. Plan International leadership has expanded OFDA’s core efforts through an “action committee” for addressing internal challenges to capacity building, and to enhance the OFDA curriculum – political, technical/technological, institutional, and financial. The evaluation team consistently found that the “sub-text” within each of these in sues is both the greatest threat to, and opportunity for risk management improvements, which shows clear insight on the part of Plan International. Five action committee members are OFDA￾certified instructors, and teach both courses and CPI (instructor training) in regions throughout Peru, including Lima. Finally, it should be noted that participation in the Risk Management Group is entirely voluntary and informal, and no one receives any compensation or incentive to participate, a sign of commitment.. Guatemala. The strength of OFDALAC in improving internal capacity within the institutions of a country is more than apparent in the case of the central institution for emergency preparedness and risk management: the National Commission for Disaster Reduction (CONRED). But the success of OFDALAC and IRG goes much deeper, other principal institutions for first response have been beneficiaries and their deep capacity and coordination reflect well on the overall program. In addition, OFDA and IRG have played a key role in providing a core around which other international agencies (OPS, for example) have revolved. As is true in all of the countries visited, the role of the IRG senior consultant in providing continuing support has been critical. The Director of the CONRED refers to CONRED as the house that OFDA built. As noted earlier, CONRED has a complete and graduated capacity profile, a long-standing relationship with OFDA, is equipped with OFDA’s complete range of core capacities, and is the focal point for OFDA’s capacity reinforcement and expansion across institutions, including fire departments and 17 NGO’s. Its work with FEMA, NOAA, USAID, and the USGS is a clear indication of its maturity, and reflects its interest in expanding its risk prevention and mitigation efforts. It has led regional simulation exercises for several years, with the next simulation scheduled for December 6-10, 2004. CONRED’s executive committee provided four suggestions for OFDA/IRG to improve the RMP: (1) assist CONRED to establish leadership on regulatory issues, (2) improve standardization of inter-agency coordination methods, (3) provide additional technical assistance with structural vulnerability analysis and risk prevention activities; and (4) move more intensively to other areas of TA, such as the legal/regulatory framework, greater emphasis on integrated mitigation, a clearer focus on development and mitigation, and a focus on cause. Each of these areas are seen as incipient on the OFDA/IRG strategy, but more is needed. To these ends, CONRED sees OFDA’s current task is to stimulate demand at the ministry level. Anecdotal evidence from CONRED suggests the depth of the influence. Staff is jokingly referred to as “Femites” or “Ofdites” based upon training received. During the past four years, the former Director of CONRED opted for close association with FEMA and as a result, much of CHECCHI AND COMPANY CONSULTING, INC The Louis Berger Group, INC Evaluation: USAID/OFDA/LAC December 2004 Risk Management Program 19 the technology utilized in CONRED bears a FEMA stamp. Current management suggests that opting for high technology rather than grounded technology will come to haunt the institution in the future (high maintenance costs, etc.). The credibility of OFDA resides in this groundedness in capacity building. It is seen as culture and region sensitive in a way that FEMA is not. The depth of institutional capacity and coordination does not stop with CONRED. The Municipal Fire Department has been a long time partner, and course material such as APAA is reported to be the most effective tool for meeting daily demands. As a result, this training module is prevalent throughout the system: the department’s Commander estimates that 10% of all fire fighters in Guatemala have APAA training. The department has not received training in SCI, but is aware of it, and interested in participating. Introducing SCI is essential, since staff indicate that no standardized command and control system exists for first-responders, each organization uses its own system, and all the systems are different. The TFI course is well￾integrated, though staff expresses a need for “new blood” among trainers. Graduation, with a new agenda for OFDA/IRG participation, is clear again in the case of Guatemala. The department has 56 members trained in BREC. Staff indicates that it is the only BREC￾trained first-responder, so it responds to BREC-based events independently. The department indicates a strong need for OFDA to increase PRIMAP training: No regulatory enforcement exists for chemical (especially fuel) transport, and spill incidents are frequent. Especially troubling is the assertion that the department’s frequent requests to discuss chemical hazard issues with private (transport) companies are ignored. Finally, it should be mentioned that the department credits OFDA/IRG with bringing together Municipal and Volunteer departments through joint training, which has resolved traditional tensions. The department offered the following suggestions for improving the RMP: (a) increase BREC coverage; (b) increase PRIMAP coverage, and clarify its target audience (regulatory); and, finally, (c) increase HAZMAT training. OFDA/IRG is credited with adding a high level of professionalism to the Volunteer Fire Department as well. APAA, BREC, and some PRIMAP training and capacity is at the center of this effort. 26 members have been re-certified in APAA. The department is providing follow-up training for fire fighters in other departments throughout the country, although they express a need for OFDA/IRG guidance and funding. The department offered the following suggestions for improving the RMP: (a) additional didactic materials and audio-visual equipment; (b) update course content in line with client experiences as some of the material is somewhat outdated; (c) rural fire department training in PRIMAP customized to local department needs; (d) increased funding and geographic coverage for BREC; and, (e) create a HAZMAT course geared toward identifying risk. The National Institute of Forestry (INAB) is another beneficiary of OFDA/IRG training and support, not only for national support but for regional support as well. However, as is true in other country contexts, there is a perception of “favoritism”: OFDA works more closely and pays more attention to firefighters than the forest protection. Because of the extreme difficulties resulting from (particularly Peten) forest fires in the late 1990s, forest fire training and improved response have been critical for all of Central America. In the Guatemala case, 300 staff members have been trained in the CBF course. INAB has 34 TFI-certified instructors for the COPSI course, and 10 regional instructors are certified. INAB has trained 100 fire fighters as (TFI) instructors for the CBF course, and 100 military personnel in CBF. INAB regularly communicates with the regional IRG consultant in forest fire training directly, whose presence is seen as critical for the success of the program on forest-specific issues, and indicates that OFDA CHECCHI AND COMPANY CONSULTING, INC The Louis Berger Group, INC Evaluation: USAID/OFDA/LAC December 2004 Risk Management Program 20 training and technical assistance is methodologically strong. INAB staff also indicates that OFDA/IRG has assisted with community outreach toward identifying vulnerabilities, an area that this evaluation team supports strongly in all areas of preparedness and risk management in a much more intensive effort. INAB offered suggestions for further improving the RMP: (a) decentralize course administration to the country level (as found in Costa Rica and Nicaragua); (b) customize courses per local needs/community-based training; (c) produce a simplified, one￾day CBF course (from 2.5 days); and (d) add one INAB support specialist for IRG consultant. Finally, the support of the OFDA/IRG program is seen in the role that it has played to some degree with the NGO community. The team’s experience with Catholic Relief Services (CRS) in Guatemala reveal that part of the success in capacity building has been flexibility, particularly in the interactions with the resident IRG senior consultant who has been most responsive to inter￾agency coordination needs, and CRS’ need to target funding sources. OFDA’s funding and flexible technical assistance during the CAMI initiative (post-Mitch) continues today. An increased role in inter/agency coordination for OFDA/IRG was expressed by CRS, where there is a perceived need to play a role in resolving to resolve its tensions with CONRED; allegedly CONRED is unwilling to assist CRS with the installation and maintenance of flood early warning systems. CRS sees OFDA as the key to facilitating political tensions in general, a role best evidenced through CRS’ recent loss of dollars from the NOAA due to inter-agency (committee) conflicts over the use of funds. Nicaragua. Overall, Nicaragua is a relative new-comer to the RMP, and most activities have come as a result of Hurricane Mitch. Here, OFDA is credited with playing a key role in the institutionalization of a national system and in providing incipient strengthening to create the national “system”. The national disaster authority, the National System for Disaster Prevention, Mitigation, and Response - SINAPRED - was formed in 2000, and is legislatively mandated (Law 337) to act as the lead coordination body among 10 Ministries. The new organizational framework is a critical advance toward risk management and not just response. This shift, supported strongly by OFDA/IRG has resulted in a shift to remove lead coordination responsibility from Civil Defense (response). The country’s stakeholders have taken OFDA’s lead to significantly improved preparedness and response, although funding remains an obstacle to further advances. Finally, the stakeholders have implemented a comparatively high level of risk prevention activities, going beyond OFDA core training initiatives. Here, OFDA/IRG has, and can increasingly, play a fundamental TA role. The agency indicates that OFDA’s biggest contribution to capacity building and inter-agency coordination has been, and continues to be its successful influence on the Ministries to accept the leadership of SINAPRED’s Executive Secretariat. In addition, OFDA assisted SINAPRED with the drafting of the National Response Plan (2001), Ministry (sectoral) response plan revisions (2003), forest fire (technical assistance) prevention planning workshop (2004), and a baseline needs assessment for ICS training (2004). With respect to OFDA/IRG’s client management functions, and ongoing opportunities to improve the RMP, SINAPRED offered several suggestions: (1) work with SINAPRED to identify a risk prevention pilot project at the community level. Because Nicaragua’s affected (disaster) areas are largely rural, with a subsistence agricultural economy, SINAPRED emphasized that a prevention pilot should operate through non-tradition (local) stakeholder groups; (2) the need to strengthen early alert capabilities at the local level; (3) closer relations and capacity building at the municipal level. CHECCHI AND COMPANY CONSULTING, INC The Louis Berger Group, INC Evaluation: USAID/OFDA/LAC December 2004 Risk Management Program 21 As is true in all of the case country analyses, OFDA and IRG are seen as real partners with the firefighters in first response and capacity building. The Municipal Fire Department recognized the contribution of OFDA/IRG’s capacity building/training and follow-up technical assistance/client management activities. BREC and ICS training is most central to their capacity, with 103 fire fighters trained through SINAPRED/directed OFDA courses today. While commitment to advances is evident, the department shares a common dilemma with other response and preparedness institutions: an extreme lack of resources both for equipment and for firefighters themselves. Although this falls well beyond the purview of OFDA/IRG, fire fighters are the lowest paid in all of Central America ($70.00/mo. compared to $600.00/mo. in Costa Rica), and all of their equipment is donated from other countries (Spain, Germany, Russia, US, etc.). Staff members offered 5 suggestions for improving the RMP: (1) provide liaison functions to identify dollars or manpower to reach rural fire departments. Building national capacity requires traveling to, or bringing in, trainees. The process is slow and cost-prohibitive; (2) community-based risk prevention programs which reduces the potential burden to response capacity; (3) simulation exercises to enhance ICS training (an activity that clearly is at the core of new OFDA/IRG initiatives; (4) added emphasis on PRIMAP and HAZMAT training; (5) the provision of audiovisual and bibliographic materials. The Volunteer Fire Department is even more effusive in its support for the contributions made through the OFDA/IRG relationship and the earlier OFDALAC partnership as well. The 300- member department credits OFDA with improving professionalism and effectiveness: it has 124 trained members across Nicaragua’s 10 districts. However, 78 are located in Managua (53) and Masaya (25). Emphasized is training that has strengthened capacity in APAA, PRIMAP, and MACOE, with 10 TFI instructors nation-wide. The volunteer firefighters point to important roles played by OFDA/IRG in recent years in moving into non-traditional entities to strengthen local capacity. In 2004, the OFDA consultants led an effort to educate children in rural schools (Esteli) on fire prevention. However, more inter-agency coordination is needed, and OFDA can play a role. While, the volunteer firefighters interact with the municipal firefighters on an “as needed basis”, there is a separatist attitude, which may call for increased OFDA/IRG liaising toward improved fire fighting capacity nationally. In the case of both Municipal and Volunteer Fire Departments, strong relations with Miami Dade County had been established. There is a perceived sense of loss in the absence of this relationship. To date, there is no sense of alternative to this hands-on relationship. The evaluation team also visited the National Institute of Forestry (INAFOR) where capacity building has been a key for OFDA/IRG. Over the last three years, 2000 personnel throughout the country have undergone COPSIF training. Forest fires occur most often from December - April, and in rural areas. INAFOR has trained (COPSIF) 1500 multi-sector field operatives to handle the high hazard period with OFDA/IRG assistance. OFDA has provided follow-up guidance (technical assistance) on INAFOR’s coordination with the Army. INAFOR offered two suggestions to improve the RMP: increase public education campaigns in rural areas and outside of traditional institutions with a focus on local populations practicing subsistence agriculture (slash and burn); emphasize the CDF course over COPSIF. CDF is “more practical and relevant at the community level”, and costs $200.00 vs. $6,000.00. Finally, OFDA/IRG presence is felt in the Ministry of Education (CUSE). Since 1999, OFDA has provided course materials and technical assistance for introducing hazards issues into primary and secondary schools, and training for education and housing administrative personnel. In 2003-2004, follow-up training has been “minimal”, but is planned for 2005. The Ministry staff CHECCHI AND COMPANY CONSULTING, INC The Louis Berger Group, INC Evaluation: USAID/OFDA/LAC December 2004 Risk Management Program 22 offered one suggestion to enhance RMP activities: build bridges to alternative (AID, grant) funding sources. C. Overall Program Impact In the previous sections of this chapter the evaluation team has focused attention on the internal workings and actions of the OFDA/IRG relations with client countries and institutions. However, key to this evaluation is the overall theme of impact. What is the overall impact of OFDA/IRG programs in the LAC? 1. What types of impacts have been produced through client programs? 2. What is the (perceived) degree of vulnerability reduction overall? However, the evaluation of the theme of impact must be related both the contractual relations of IRG with OFDA, and as a consequence of the strategic objectives (SO) expressed in overall OFDA strategy. The reduction of two broad SOs (response and mitigation) to a single SO (focused still on response for the 2004-2009 period has a direct bearing on OFDA/IRG. Further, limits in the contract bear on what can be expected. The OFDA Strategic Objective (SO), or impact goal, remains principally driven by response: Critical needs met of targeted vulnerable groups in emergency situations. This SO incorporates four intermediate results (IR), of which one goes beyond the principal response objective: IR 1.1. Emergency assistance, meeting recognized standards, received by disaster victims in a timely manner; IR 1.2. Improved targeting of emergency assistance to the most vulnerable groups; IR 1.3. Capacities for livelihoods protected or restored; and, IR 1.4. Increased adoption of risk management measures in countries at greatest risk to natural and human-caused disasters. As seen from the perspective of the IRG contract, the principal activities focus on administrative support to OFDALAC, service and commodity procurement and small grants during disasters, short term response assistance, but more importantly and principally training, TA and an M&E system. In each of these areas, the performance of OFDA/IRG has provided critical support to the overall objective of OFDA/LAC, and the overall impact has been very positive. We have noted these impacts in various sections of this report, and here we simply summarize by category. Administrative support. Beyond the obvious day-to-day administrative support that IRG provides through the contract, a number of recent (new contract) events point to the efficacy of this aspect of the overall contract. It is clear that IRG provides a means for OFDALAC to carry out its overall mandate through this support, and relieves OFDALAC of “in-house” administration. But, support through the contract for such activities as the INSARAG conference (not reimbursed by OFDA) and the administration and oversight of security upgrades clearly impact directly on OFDALAC performance. Procurement and Grants Activity. In the former, IRG provision or facilitation (for example, of donations of forest fire equipment) has been effective, especially in response mode. In the case of the latter (and as described elsewhere in this report), it is not clear why this contract is an appropriate venue nor how effective this activity is. CHECCHI AND COMPANY CONSULTING, INC The Louis Berger Group, INC Evaluation: USAID/OFDA/LAC December 2004 Risk Management Program 23 Short Term Response Support. IRG consultants have been extremely successful and have had a major initial impact because they are located on the ground. This has been true throughout the association with IRG and prior to it, but a number of recent (2003-2004) events point again to the efficacy of this role. Flooding in Peru, Mexico, Nicaragua, the Dominican Republic; earthquake responses in Mexico in 2003; forest fire responses in Guatemala; the dramatic mall fire in Uruguay; any number of additional less dramatic examples all point to the effectiveness of the immediacy of the IRG presence and coordination with USG personnel. But nowhere is this more evident than in the recent hurricane season in the Caribbean. From all on the ground accounts and discussions with NDCs and USG personnel, the critical role of the IRG team and the OFDA RA team has been effective in both initial communication and assessment of needs.6 Immediate and short term provision of funds to organizations, such as the Red Cross, has strengthened these organizations’ capacity to reduce immediate consequences of risk vulnerability. Training and TA. This category is the bread and butter of the historical and ongoing activities of the IRG relationship with OFDALAC, and the evaluation team’s comments throughout point to its effectiveness and impact. In recent years, the maturation of the LA program and the potential “graduation” of some countries is evidence of impact. The strong, albeit still incipient, efforts throughout the Caribbean region to adapt the current training program to their needs have already begun to bear fruit. More importantly, perhaps, this Caribbean “push” provides OFDA with clear evidence of the replicability of many elements of the training and networking of the model. While time is needed to reflect more fully on this impact, the mere creation of NEMOS, stronger roles for NDCs, positive perceptions of courses and the role of IRG-trained instructors, the demand for new and improved course material all point to impact. Again, it is a bit early to evaluate the impact, the very recent development and pilot exercises in two new training courses are evidence of the culmination of much of the integration of the training and TA program: the Incident Command Course (SCI) and the Facilitation Skills course. In both cases, and especially in the former, OFDALAC and OFDA/W will be able to evaluate the integration of the overall training program. Pilot exercises and initiation of instructor and participant courses in SCI should be carefully monitored for the immediacy of the impact they will reveal in overall training and TA. SCI integrates the entire OFDALAC portfolio and aims at first response for the combined institutional forces, both public and private. Ironically, in Guatemala a coordinated national effort to carry out an exercise in simulated response (based on much of the OFDA/IRG portfolio) is scheduled for the first week of December and will provide national and regional authorities with insight on future needs and impact. Monitoring and Evaluation. Throughout this report the evaluation team has pointed to the need to strengthen impact evaluation and a move beyond measurement of inputs and process, or even intermediate results. This is not meant to suggest that evaluation is weak. Rather, it is because the strength of existing evaluation methodologies in Plan Pais, TTS, annual reviews, and course instruction in process and results measures have paved the way for more sophisticated impact measurement. We simply highlight this area as work to strengthen for the near term. Impact is difficult to measure under any circumstances. As the contract notes, in training impact measures provide the view to the future in terms of training investments. Impact measurement is 6 During the two-month period of this evaluation (October-December) there were at least three continuing or new events that point to the role and the effectiveness of this immediate capacity to respond. The evaluation team visit to Jamaica coincided wit continuing efforts to respond to Ivan, where the IRG consultant was praised for his contributions. The severe consequences in Haiti have occupied several of the RAs on an extended and repetitive basis. Finally, the most recent events in Dominica prompted the RA to become an immediate responder to provide evaluation of needs. Without these supports, consequences clearly would have been more severe. CHECCHI AND COMPANY CONSULTING, INC The Louis Berger Group, INC Evaluation: USAID/OFDA/LAC December 2004 Risk Management Program 24 also difficult, but necessary, in terms of vulnerabilities to disaster. How much reduced is the risk or how much lower is the vulnerability in actual events is the essence of risk management as well as immediate response. OFDA/IRG is totally cognizant of this need, especially as the entire program deepens mitigation and management components of the portfolio. In discussions with IRG personnel (and in particular with the Technical Manager), evidence of proposed impact indicators - and not just process and efficacy measures - for course instruction is being developed. In at least nine areas of training, initial impact indicators are being developed and instituted. More difficult perhaps is the challenge of introducing impact indicators in TA activities not directly related to training. In the following pages we refer to several overall aspects of OFDA/IRG impact, but more importantly we attempt to view impact in the grounded cases of countries visited. How does the process of building inter-agency coordination lead to vulnerability reduction? The Risk Management Program, in the absence of massive financial investment (infrastructure retrofitting and modernization), must rely on intellectual capital, and the efficacy of inter-organizational coordination for disaster preparedness, response, and prevention/mitigation programs. And, integrating human capital hinges on assessing and understanding stakeholder differences in organizational structure, activities, methods and goals. (Some physical hardening projects have taken place in Costa Rica (schools, hospitals), and low￾technology prevention (roof straps for wind damage) measures exist in the Caribbean, but these seem to be tangential to the RMP’s developmental process. In order to achieve positive impacts, the RMP focuses on institutional vulnerability, moving from client assessment activities, to building capacity, to institutional strengthening aimed at inter-institutional coordination and regionalization across the LAC. In so doing the RMP targets the following objectives: Incorporation of risk management issues into legislation and into national development plans; standardization of key concepts, activities, goals, and methods consistent across stakeholder organizations; Formation of networks among stakeholder organizations; and, introduction of development issues into risk management practices and plans. As such, OFDA/IRG measures the RMP’s achievement in these areas according to “process indicators” (new client activities resulting from the RMP) “effect indicators” (changes client behavior and processes directly resulting from RMP activities), and “Impact Indicators” (changes in the condition of life in target populations resulting from changes in client behavior). In general, “contextual constraints” to the Risk Management Program’s impact include small client budgets, lack of continuity/high rate of attrition among recipients of training, institutional turn-over, institutional territorialism, and little institutional or external support for risk prevention and mitigation programs. These factors hinder the RMP’s ability to place greater weight on community-based emergency preparedness, response, and risk prevention activities. The largest internal constraint to measuring impact is found in the RMP’s educational programs’ tendency toward “long-term developmental processes that do not yield hard data today”. According to OFDA/IRG consultants and client-respondents, by its nature, (OFDA’s) introduction of hazards issues into standard course content in primary and secondary schools is expected to gradually increase perception of risk, and, slowly lead to a “cultural mindset of risk prevention”. Toward this end, OFDA/IRG consultants report that certain “process indicators” exemplify progress: consensus on program goals, methods, and content; consensus on necessary changes; and, political legislation or mandates prioritizing risk management initiatives. CHECCHI AND COMPANY CONSULTING, INC The Louis Berger Group, INC Evaluation: USAID/OFDA/LAC December 2004 Risk Management Program 25 The goal of the RMP in the LAC is to positively impact vulnerability reduction by framing, and directly relating its impact to its core functions: identification of key stakeholder organizations (clients), assessment of client needs and vulnerabilities, delivery of course content and technical assistance, and client management. Impacts identified in this report are tangible products, tangible measures (instructors), tangible outcomes (legislation), and indicators of risk management processes in motion or “process indicators”. A small number of structural retrofitting projects have taken place, but due to cost, coverage is minimal. Hardening projects do not fall under the RMP for OFDALAC. Costa Rica. The CNE has graduated to move toward developing prevention and mitigation programs. OFDA, through the RMP’s influence, has assisted the CNE with four key mitigation programs in four different contexts: 1. Meteorological science has produced a GIS mapping system applied to community vulnerability prediction models for flooding and hurricanes. 2. University of Costa Rica conducted an earthquake study in Guanacaste province (Northwest part of the country) on event probability of occurrence, intensity, and duration with associated risks, i.e., impact (social, physical) prediction models. 3. OFDA and the Ministry of Education partnered on course content and format to include hazard preparedness issues within standard coursework in primary education. 4. The Ministry of Health has physically fortified 2 hospitals: Hospital Mexicano and Children’s Hospital. With respect to OFDA’s direct impact on the Fire Department’s crisis response capacity and its investment in people and resources, the Fire Department has 400 full-time firefighters and 1000 volunteers located in 71 stations throughout Costa Rica. In 2002, it expanded its collaborative agreements to work with Fairfax County, VA and Los Angeles County, CA. 2003 marked a milestone for OFDA’s relationship with the department – regionalization. The department is a founding member of the Confederation of Central American and Caribbean Firefighters, which enables departments to communicate through conferences, workshops, web pages, and publications like the OFDA-funded International Firefighter quarterly periodical, published in Miami, FL. But, the most long-term and profound impact of OFDA’s influence on the Fire Department is found in its departmental structure: in order to produce budgetary and operational efficiency and improvements, the department operates at 2 levels of risk management: the engineering division conducts materials/chemical research, and structural building plan assessments in order to address fire prevention issues, administration, while operations and volunteer programs address fire rescue and forest fire response. The Forest Service’s (OFDA) course content and objectives are integrated into the policies that guide Costa Rica’s National Development Plan. As such, OFDA (course) methods for coordinated preparedness and response guides how the Plan integrates risk management stakeholder organizations. Jamaica: With a limited budget, ODPEM’s relationship to the RMP is mostly preparedness and response based. The limited budget negatively impacts existing priorities, and largely prevents ODPEM from exploring risk prevention and mitigation opportunities with OFDA/IRG. Risk management roughly equates to preparedness and response training. Despite its problems, CHECCHI AND COMPANY CONSULTING, INC The Louis Berger Group, INC Evaluation: USAID/OFDA/LAC December 2004 Risk Management Program 26 ODPEM has worked closely with OFDA to innovate and recent efforts point to a rowing impact overall of OFDA/IRG presence. Because of the rapid damage assessment manpower and coordination gaps that emerged post-Ivan, ODPEM decided to create a rapid response simulation mirroring Ivan’s impact characteristics. 8 university students volunteered to serve as the assessment team. Students attended a one-day training module, and immediately began field￾work. The results were positive and three-fold: Students consistently provided ample and relevant data; ODPEM verified that its rapid training model is viable; ODPEM’s volunteer training empowers locals with skill-sets, vertical relationships with disaster and development professionals, and encourages generational interest in disaster management and development issues. Because of the success of the student project, OFDA is now working closely with ODPEM to use the methodology of the Initial Damage Assessment (IDA) as an operational framework for developing ODPEM’s new Rapid Damage Assessment training, designed to gather data within 48 hours to 1 week post-disaster. With volunteer citizens trained through local workshops to rapidly gather information on types of damages, ODPEM can prioritize reconstruction and cost estimates. Baseline data is channeled to Parish Councils equipped with DANA (needs assessment) capabilities. In addition, ODPEM is beginning to qualify certain members of local disaster response committees to receive DANA training in order to act as a filter and conduit for what is most relevant for the Parish Councils to consider. While promising, the rapid assessment capability is still in its infancy. Red Cross staff provides two examples of the positive impact of OFDA-course training: Hurricane Ivan evacuation planning and operations was far superior to Gilbert (1988); Public participation in preparedness and response planning: in 2003, the 8 most at-risk communities participated in drafting and implementing disaster preparedness and response plans. Peru. One preeminent sign of impact is that in 2004, EDAN was legislated to be the standard for disaster response information flow, equipment mobilization, and coordination led by INDECI. INDECI and the Ministry of the Interior developed a training cell with the Army in order to multiply damage and needs assessment (DANA) capabilities. In building on the military’s training in force protection (civil unrest), the project will train 2000 personnel over the next 2 years. Similar evidence of impacts in MINSA where the completion of a database integrating natural hazard risks, environmental resources, and demographics has occurred. Based on EDAN, the database is a decision-making tool which empowers decision-makers to set priorities for risk prevention, mitigation, response, and reconstruction. The tool is implemented at national and regional levels. Staff training in the Fire Department suggests impact as well: the department estimates that over 90% of staff has OFDA skills in BREC, EDAN, PRIMAP, APD, and APAA. The Department has 6 people TFI-certified to train other departments in EDAN, BREC, and APD. Instructors have conducted training in Colombia, Ecuador, Bolivia, Paraguay, and Chile. Other evidence of the tremendous impact in the case of Peru has been the success in Piura in integrating, coordinating, extending and deepening risk response and management capacity through the assistance of IRG consultants and the use of the OFDA/IRG portfolio beyond the response led modality. Here, IRG consultant TA has been critical in terms of impact. Several examples are relevant. CHECCHI AND COMPANY CONSULTING, INC The Louis Berger Group, INC Evaluation: USAID/OFDA/LAC December 2004 Risk Management Program 27 o In 1999, the Ministry of Agriculture and OFDA jointly developed a disaster prevention plan focusing on the combined impact of floods and drought. El Nino cyclically exacerbates the threats to cotton production, and, in 2001-2002, OFDA helped to establish risk management committees made up of cotton producers and other large land-owners, educated to assist OFDA in risk diagnostics. Today, these highly motivated committees have expanded their prevention efforts into disaster response, mitigation, and recovery. o OFDA and the PRG have established 8 community-based commissions based on subject areas: mayoral governance, communications, education and capacity building, public health, law/order/technical inspections, science and technology, planning, and inter-departmental coordination. The commissions reflect the multi-sector Civil Defense Committee structure at the national and regional levels. Most importantly, the commissions have prioritized risk factors, linking them with natural resource and environmental concerns. Positive results include “barrier zones for production zones” (cotton), and inclusion of multi-hazard risk factors in the new Regional Economic Development Plan. The multi-sector, multi-hazard commissions’ operational structure and hazards-development linkage is unique in Peru. o In 2003-2004, OFDA and the University of Piura examined the ability of 6 communities to conduct damage and needs assessments. The majority exhibits a low level of capacity. Community first responders are key to improving the RMP, and have proven to be an interested and captive audience for OFDA to expand its client base. o The Risk Management Group is completing a multi-hazard risk map and contingency plan to take to INDECI, in order to energize improvements in Piura’s land-use planning practices, and to inform the development of a formal land-use plan: due to recent population pressure (beginning in 2001), housing development projects have taken place in fluvial soil and flood high hazard zones against the recommendations of OFDA, INDECI, and the RMG.. While the hazard map and contingency plan are essential tools for reducing risk, OFDA/IRG’s challenge is to influence the region’s stakeholders to adopt land-use practices that will supersede the financial and cultural disincentives. Guatemala. OFDA/IRG impact in Guatemala is evident in the depth of training, institutional capacity, inter-agency coordination and the overall level of disaster response and risk management. Again, TA in recent years has been the key to a renewed impact on pre-response, has achieved administrative, financial, and legislative support from Guatemala’s public sectors, However, training and the multiplier effect further demonstrate impact. In this case, recertification has been very important. In 2003-2004, approximately 24 Ministry personnel underwent follow-up/ re-certification training in APAA, and the Ministry recently made APAA training an official in-house requirement. APAA is a requirement for departmental certification as a “technical expert”, and over 90% of personnel are trained in APAA. Additional APAA￾based certification in emergency medical response (paramedics) has been developed through a partnership with the Ministry of Health and the hospital system. The three-way accord has a four-year plan to certify fire fighters throughout the country, and establish a re-certification protocol. The RMP’s impact on the department is evident through the following 6 activities and programs: a partnership with the University of San Carlos to update APAA course materials, a partnership with the university to distribute the APAA course via the internet to universities throughout the LAC region; a partnership to develop an emergency medicine program at the CHECCHI AND COMPANY CONSULTING, INC The Louis Berger Group, INC Evaluation: USAID/OFDA/LAC December 2004 Risk Management Program 28 university; BREC-based simulation exercises twice a year with 4 other departments selected for geographic coverage; customization of the BREC course for the Army; course training for departments in El Salvador, Honduras, and Nicaragua. CONRED´s increasing sophistication and reach in the national institutional system is in large part a result of many years of OFDLAC and OFDA/IRG participation. The CNE is moving the RMP beyond preparedness and response through three innovative risk reduction programs: o In conjunction with the Red Cross and Catholic Relief Services, CONRED is educating local community groups on ways to reduce risk exposure in homes and schools; o CONRED has partnered with the University of San Carlos to develop a disaster management 4-course curriculum through Engineering and Earth Sciences departments; o CONRED and OFDA are assessing structural vulnerabilities within communities through risk management profiling (community resources, past event data). Twenty earthquake scenarios are applied to the design of a simulation model to predict impacts, and target necessary changes in preparedness and response strategies. These are all examples of a new generation of activities beyond the more traditional role that OFDA has played. Finally, CONRED staff indicates a future growth area for OFDA lies in assessments of risk perception as related to cultural (agricultural, homestead) practices among the Maya, toward identifying prevention opportunities. Nicaragua. The challenge in Nicaragua has been national coordination and expansion beyond Managua to other municipalities and to important target groups outside the first response community. Again, innovation in OFDA/IRG interventions has been the key to new impacts in pre-response actions, and evidence of direct response impact is also clear. o With OFDA technical and course assistance, SINAPRED quickly has made its mark, especially in prevention and mitigation. In 2003, OFDA provided technical assistance to SINAPRED to develop Masters-level courses “Natural Hazard Risk Prevention”, and “Natural Hazard Management” with the Autonomous University of Nicaragua (UNAN), and has introduced hazard prevention into existing course content in primary and secondary schools in Managua. o SINAPRED’s goal is to create a “culture of prevention”: staff report that the RMP “inspired” SINAPRED to air a weekly, 30-minute radio show on risk prevention, broadcast nationally, and designed to make people consider their contribution to improving quality of life through individual risk prevention measures at home and work. o During the floods of 2004, OFDA training directly produced more efficient and more rapid evacuation. Ministry of Education staff indicated that no loss of life occurred, and attributed the successful evacuation to OFDA training. Related, the Ministry established “teacher auxiliary response brigades”, in conjunction with, and trained by SINAPRED, and partnered with the OFDA/IRG education consultant on the introduction of risk prevention issues into Managua’s primary and secondary school courses. o Finally, in the case of INAFOR, existing capacity is the result of earlier strong OFDALAC presence. Ironically, INAFOR staff note that from 2000-2002, contact with the OFDA/IRG forest fire consultant was less evident than earlier, yet existing instructor capacity with OFD training allowed them expand training to some 1500 field operatives, including 700 military personnel. More importantly, INAFOR reports that from 1998- CHECCHI AND COMPANY CONSULTING, INC The Louis Berger Group, INC Evaluation: USAID/OFDA/LAC December 2004 Risk Management Program 29 2000, Nicaragua had 500 forest fire events, compared to 2001-2003, when forest fire events were reduced to 86. INAFOR credits OFDA completely. While this set of contextual examples in the countries visited is not a systematic statement on impact, the message for the evaluation team is clear. OFDA/IRG has had a profound impact whether measured in process, result and more importantly, real world impact terms. Shifts to greater presence and innovation in the entire risk management cycle through training (SCI, etc.), TA, and increased - albeit still incipient - efforts to reach to new actors and targeted populations all point in the right direction. D. Program Management Issues As noted earlier, one of the major successes of the IRG contract with OFDA/LAC has been the emergent administrative and financial management since 2000 and its separation from the technical management of the program. While it is difficult to separate programmatic content from management style in some cases, the gains for both are clear. The overall management modality or style that has evolved works well and provides a more effective and accountable institutional mechanism for the “model” that did not exist earlier. IRG and OFDA/LAC should be commended for this increased transparency, management, order, accounting and accountability. The very fact that planning, budgeting, programming and execution data exists in an orderly fashion permits this portion of this evaluation. There are a number of sub-themes to explore in the area of program management and strategic management decisions on program content. Seamlessness in Operational and Management Relationships. The “seamless” nature of the IRG/OFDA partnership is evident in the relationship between consultants and OFDA staff, in the field, in singular strategies, and in the lack of an OFDA/IRG distinction among client nationals. In every field visit, the answer to the question of whether national consultants were IRG or OFDA employees invariably was that they were OFDA employees. In some cases, the IRG name was not even known! That has served the program well, especially due to the high credibility that OFDA shares in the field. Seamlessness seems to be a function of the long-term relationships, but does not pertain to management functions as it does to operations. In general, that is a positive outcome of the contractual relationship, because it has brought a clear sense of management accountability, something absent in earlier years (pre-2000). However, there is some concern that IRG has stepped into strategic programmatic decisions that are the appropriate purview of OFDA management. This concern has been expressed in particular in relation to strategic decisions with regard to areas to be cut initiated by IRG as a result of budget shortfalls in 2003 and beyond (discussed in detail below). In this case, the “tail wagging the dog” may have interrupted some of the seamless nature of the relationship. Trade-offs: Administrative Strengthening and Field Flexibility. This shift in management and administrative style and content has not come without some costs. There is some concern that the shift from the initial flexible contractual arrangement (similar to an IQC in the reimbursable nature of costs) to a fixed sum (CPFF) contract may have reduced the flexibility of action on the ground in response to country level needs. Several consultants referred to the need to assure that management improvements do not come at the cost of field flexibility. There is a need to retain greater contractual flexibility, albeit with clear accountability, because of the very nature of the objectives and mission of OFDA and the external contractor: disaster relief and risk CHECCHI AND COMPANY CONSULTING, INC The Louis Berger Group, INC Evaluation: USAID/OFDA/LAC December 2004 Risk Management Program 30 reduction. These are uncertain areas, and innovative approaches to contract finance mechanisms need to be explored without violating USAID rules. The circumstances of the current contract exemplify the need for clearer and more flexible relationships. Plan País. The past four years have seen the emergence of this critical integrating tool for the RMP. From its inception in 2000 as a means for bringing increasing order to the training and technical assistance functions, Plan Pais has become the key instrument for client-country planning, programming, and budgeting, and to a lesser extent, monitoring and evaluation. The information is updated quarterly, and it contains detailed programmatic activities (plans and execution), diagnostic information on demography, socio-economic factors, political realities, the history of disasters responded to by IRG/OFDA, and an evaluation of institutional capacity in risk management. The PP incorporates the OFDALAC Strategic Plan for 2001/2006, and presents objectives and results expected for each of the objectives (preparedness/response, institution strengthening and coordination, training and information).7 Two key findings merit attention: o Significant advances have been made in utilizing Plan País as an evaluation tool; there is a need to strengthen these ongoing efforts. While the TTS and training reporting, along with the quarterly reports, provide a basis for M&E, there is greater need to move from process indicators to those that might point to tangible results, and more importantly, tangible impact. It is often difficult to separate results from training and technical assistance because they are both part of an integrated process of the transfer of risk management capacity. In the second year of the contract there has been significant internal discussion in this direction, and in some cases initial impact indicators have been developed (e.g., impact measures in forest fires), and initial proposals for distinctions among process, result and impact measures.8 This strengthening has been discussed thoroughly with IRG, and there is every indication that this is the direction in which they are headed. o Ironically, the tool has a substantive institutional vulnerability diagnostic component, but as a basis for strategic interventions, it is not shared with national counterparts, or with the MDRO at the Mission level in at least one field case. Clearly, the PP has sensitive content. However, with some editorial work, this instrument could serve as a basis for national dialogue, and not just as an IRG internal tool. It needs to be shared with countries. Operational Efficiency and Cost Effectiveness. There is no question that the operational model is cost-effective because of the reliance on national consultants and increasing passing of instructional roles to national instructors. The improved administrative capacity has had a direct impact on the capacity to measure and improve operational efficiency. Nowhere is this more evident than in the dramatic reductions that have occurred in the IRG costs of field training activities as counterparts pick up larger portions of total costs and total responsibilities. This “changing of the operational guard” has implications not only for costs per activity, but also for the need for the intensity of consultant participation in training-related activities. 7 A more recent update of the Strategic Plan for FY2004-2009 provides insight into the increasing move from response to risk management, although it remains as one IR (1.4) in the single overall SO of response: “critical needs met of targeted vulnerable groups in emergency situations”. 8 In several internal documents these efforts can be seen, and are intended to be incorporated in course material: J.P. Sarmiento, “Propuesta de Indicadores: María Luisa Alfaro, “Propuesta de Indicadores para Programa de Incendios Forestales OFDA/LAC”. CHECCHI AND COMPANY CONSULTING, INC The Louis Berger Group, INC Evaluation: USAID/OFDA/LAC December 2004 Risk Management Program 31 As can be seen in the following graphic, the average cost for the most prominent training courses has declined over the first five years of the program. Data for 2003-2004 support this general trend, with average daily costs in these five courses remaining much the same. Approximate values for 2003-2004 are: COPCIF ($880, slightly higher); CPI ($950, substantially higher); EDAN ($345); APD ($350); APAA $500, lower).9 Average cost per day (5 courses) 0.00 200.00 400.00 600.00 800.00 1,000.00 1,200.00 1,400.00 Year 1 Year 2 Year 3 Year 4 Year 5 Year Average cost per day COPCIF CPI EDAN APD APAA Monitoring and Evaluation. One of the areas that still requires substantial effort is that of monitoring and evaluation (M&E). The Plan País provides an excellent framework for M&E, but more can be done. OFDA/IRG is stretching the bounds of how to measure results and impacts. This is an enormous and innovative undertaking that has never been attempted before in the LAC”. This is especially true given the difficulties in identifying tangible measures of vulnerability reduction through a program of this scope. Also, it is important to take into account the disaster management professional community’s historical difficulties with interpreting risk management concepts, OFDA’s evolving processes are inherently qualitative in terms of measuring effectiveness. There are four levels of evaluation evident and in various stages of development: o During the courses in classroom evaluations: From discussions with those trained and Instructors in the field, there is a sense that the tool needs strengthening. This is less a criticism of IRG and perhaps more relevant as a comment on how OFDA evaluates classroom performance. o Internal to the program itself (training package, program as a whole) o Institutional Evaluations (how trained perform in the job, performance evaluation, use of course in work). In part, the Institutional Diagnosis of the PP fills this role, but again a more systematic set of evaluation instruments need to be developed. o Organizational culture and values, organizational change (has a culture of response been superceded by a focus on mitigation, degree of intra and inter-institutional focus on the broader context of risk reduction and response) To date, the RMP/IRG PP primarily deal at levels one and two, and to some degree at level three in the diagnostics. Indeed, contractually IRG is fulfilling its M&E mandate (levels 1 and 2) and perhaps beyond as it moves specifically into the area of establishing new baseline data, the 9 The costs associated with TA/Workshops and Instructor Training tends to be substantially higher, in part because of the greater number of regional courses which involves greater travel and per diem costs. CHECCHI AND COMPANY CONSULTING, INC The Louis Berger Group, INC Evaluation: USAID/OFDA/LAC December 2004 Risk Management Program 32 tremendous effort in integrating training and outcome through SCI which is a means for evaluating impact, the preliminary impact measures in forest fires, and some country specific institution strengthening measures. Again, the issue is not IRG performance; rather it reflects the limits of OFDA´s strategic objective. Regional Advisors and Senior IRG Consultants. An area that remains unclear to the evaluation team is the role and relationship of OFDA’s Regional Advisors (RAs) and the IRG (senior) consultants. National (and especially senior) consultants seem to fulfill many roles except that of liaison and funding with the national Missions, especially in times of response. In addition to response and coordination roles, RAs are also charged with broader political and policy roles that fall well outside the purview of IRG consultants. In addition, RAs have official government status which broadens their authority to interact with government and regional and international organizations.10 The recent addition of new Regional Advisors provides a real opportunity to examine this relationship. The evaluation team sees a role for each, but it is incumbent upon OFDALAC to take this opportunity to further clarify the roles of each. Funding Levels and the Second IRG Contract. In the current (2003+) period of the new contract with IRG, there has been a problem and a concern about the reason for, and consequences of what appears to be a lack of funding commensurate with a pre-designed work plan and program. It is not the role of the evaluation team to enter the question of why this occurred. However, it important to understand the consequences for Year 1 of the new contract, and the perspective for Year 2 (already in implementation). There have been consequences that relate to cutbacks in programmed activities, particularly those involving travel for national consultants. The result? Substantially fewer training exercises conducted when compared to initial programming; reduction in the transmission of some new course material that has been designed; some reduction in the face-to-face interaction among consultants to plan and impart experiences; postponement to Year 2 of activities planned in Year 1. However, even with budget reductions from work plan prescriptions, much of the initial work plan has been carried out. The question becomes how to incorporate lessons learned from this period of budget tightening to assure continued financial and programmatic accountability. While this evaluation is not intended to be a financial or management audit, because of the concerns surrounding the circumstances of the first year budget in the new IRG contract (2003- 2004), its impact on the program, and the continuing uncertainty as to amount of the second year contract, the evaluation team feels it incumbent to offer some guidance and observations. The following observations are based upon interviews/conversations with San Jose IRG senior management, San Jose OFDA management and staff, IRG work plan and budget and quarterly reports, and interviews with field consultants. There are substantial disparities between the original IRG Work Plan, the eventual WP, the first year budget and the contract. The modified Work Plan (WP) of July, 2003 incorporates decisions taken to accommodate the eventual contract for Year 1 with OFDA. An examination of the WP suggests a number of critical decisions made that have both current and future impact on the outcomes of IRG and OFDA/LAC work in the region. In addition, a review of the July (2003) WP and July (2004) Quarterly Report reveal several strategic characteristics that may well affect present and future activities under the contract.11 Finally, a review of Years 1 and Year 2 of the Plan Pais CD and discussions add to a formidable amount of transparent and indicative information for this analysis. 10 See OFDALAC’s operational plan for 2004-2009 for a discussion of RA roles in response. 11 These tables are not included but are readily available in IRG documentation. CHECCHI AND COMPANY CONSULTING, INC The Louis Berger Group, INC Evaluation: USAID/OFDA/LAC December 2004 Risk Management Program 33 The nature of cutbacks or adjustments to contract requirements. The major decision in adjusting budgets was to cut back of both the travel and the training to be delivered under the contract. One clear area affected by the differential has been the overall level of training and TA activities. The following WP and eventual expenditures reflect the pattern for Year 2003- 2004 in terms of program activities. Training and TA are at the core of the program. From an initial programming of 197 activities, the execution for Year 1 reflects 140 activities, a diminution of 29% of training and 32% of TA activities, and a budget diminution of almost 43%. Activity and Budget Execution: Year 1 Activities Work Plan 2003 Budget First Year 2/29 Executed % of WP 5/31 Executed % of WP Training: # 150 72 48% 109 71% TA: # 47 24 51% 31 68% Total ($000) $988 $737 $375 38% $568 58% Total # Activities 197 96 49% 140 71% Sources: Quarterly Reports, Work Plan Year 1, TTS In addition, two particular circumstances have exacerbated the funding for key activities of the program: the termination of the participation of Miami-Dade County and related start-up costs for the new contracts with Fairfax and Los Angeles, and the absence of (expected) contributions for training to the training portfolio. In the case of the former, the result has been the need to bring the new firefighter agencies “up to speed” on how and what OFDA and IRG do in the context of different cultural milieus. This represents an initial cost to IRG, although the longer term benefit of the contribution that these agencies can make to OFDA and client countries is undeniable. In the case of the second, the opportunity cost is evident: no longer are funds available from CAMI, the American Red Cross nor the Southern Command. During the last five years of IRG participation, the proportion of OFDA financing (co-financing with national contributions) of activities has increased steadily. In Year one, all (55) activities were financed directly by OFDA. In Years two and three the contribution of the American Red Cross was significant; in Year four and five, the Southern Command funding was relevant. However, in 2003 and 2004 all funding for training and technical assistance has been financed through the IRG contract, in spite of expected funding from Southern Command. The following graphic provides evidence of these changes. Participation in Training Activities (%) Year IRG ARC SC 1998-1999 100 0 0 1999-2000 58 42 0 2000-2001 62 38 0 2001-2002 67 5 28 2002-2003 92 0 8 2003-2004 100 0 0 All aspects of the program have been affected, and it is the evaluation team’s estimation that the program cannot operate fully at the current levels of funding. While we have a number of observations about the what and the how of line items and activities, it is clear that insufficient funds are available. CHECCHI AND COMPANY CONSULTING, INC The Louis Berger Group, INC Evaluation: USAID/OFDA/LAC December 2004 Risk Management Program 34 In response to these changing budgetary circumstances, specific decisions and actions were taken in the revised Work Plan for Year 1: o Travel in the budget particularly for IRG consultants was cut to a minimum, albeit in terms of expenditures it remained substantial; o As noted above, training and TA were cut. In some cases this meant the postponement of courses under development (SCI, revised English language IDA and DANA) were postponed for delivery until Year 2; o The principal travel and training activities for the year were cut to a 10 month period; o Some activities were reprogrammed for Year 2 of the contract. However, a number of questions remain in the view of the evaluation team about the Work Plans for Years 1 and 2 in terms of potential savings and efficiencies that may be obtained.12 We share these here in the spirit of constructive criticisms. Travel Budget. In the (July) WP for Year 1, there was no allocation for travel, although in the budget some $39,000 was allocated. In the EOY budget report, a substantial deficit is reported ($475 thousand) and in Year 2, a request for $600 thousand is presented. Travel for the program is essential: for senior consultants with multiple countries, for regional TA and training, for group meetings. However, the increase in travel over the past several years needs to be reconciled with the nature of the decentralized model: if national consultants are to be used, if national instructors increasingly replace consultants, and if self-reliance is evident in some countries, why should the travel budgets increase rather than decrease? The increase in travel over the past several years needs to be reconciled with the nature of the decentralized model: if national consultants are to be used, if national instructors increasingly replace consultants, and if self-reliance is evident in some countries, why should the travel budgets increase rather than decrease? The evaluation team is cognizant of the increased costs of travel and that in South America distances are great. However, per diem costs have doubled between 2001/02 and 2003/04 while the LOE has increased by 50% and the number of consultants by 3. Move to Increasingly Full-Time Consultants. In a similar vein, an examination of the Plan Pais documentation reveals that the LOE for national IRG consultants has increased to virtual full-time in most cases. (from 135 days in 2000/01 to 173 in 2001/02 to 203 in 2003/04). Participation of IRG consultants in traditional training is increasingly less cost￾effective as the strength of transfer improves, and their participation in training should reflect this decreased reliance. However, the increased LOE reflects in part a shift to TA and institutional strengthening, an area mush more difficult to measure in cost effectiveness terms. Transfer of responsibilities and funding to nationals implies a changed IRG role, Transfer of responsibilities and funding to nationals implies less IRG role, and a shift to other activities, or an overall decrease in presence. Success means shift in strategy and/or shift in resources. It is important to note is that, distinct from the vast majority of course offerings, BREC and ICS require far more extensive LOE and funding to reinforce existing client capacity, and expand coverage into new client groups. OFDA/IRG must continue to closely 12 The evaluation team is a bit confused with the reported G&A for Year 1. The reported sum of $1,132,996 (Quarterly Report for April-May, 2004) is not reflective of the flat rate (24.5%), nor does it show as an element in the overall calculation of expenditures. CHECCHI AND COMPANY CONSULTING, INC The Louis Berger Group, INC Evaluation: USAID/OFDA/LAC December 2004 Risk Management Program 35 examine its cost-efficiencies as a function of the up-front costs required to shift the RMP from training toward increased “program building”. Training. While overall training was cut by a substantial amount from the original WP, the eventual $737,420 was not fully executed. Only 58% of the original WP, but only 78% of the revised WP, were actually expended. Limits on travel may have contributed to this shortfall, but clearly other factors are at work. As we note earlier, the effectiveness of the use of funds is clear because while 58% of the funds originally proposed in the Work Plan were executed, 71% of the activities were executed. This cost-effectiveness needs to be viewed in light of these figures. Decentralization and IRG/W Participation. While the amounts are not particularly significant, it is not clear substantively why IRG/W should increase its travel as part of the contract. IRG/SJO has been and is self-sufficient and reporting requirements are the thrust of interaction with IRG/W. Small Grants and INSARAG. It is not clear why the IRG contract is the appropriate venue for these items. Again, while the amounts are small, they complicate the contractual relationship of IRG with OFDA and raise funding levels. In the case of grants, IRG itself has opined that it may no be appropriate: little flexibility of use in disasters, cumbersome rules and regulations make them less timely, size is too small. Our understanding is that these funds have been placed under the commodity line (correctly). In the case of INSARAG it simply isn’t clear why this is the appropriate venue. In this case, these were funds not allocated nor reimbursed, but the activity was requested by OFDA/W. V. Principal Conclusions q Coverage. The IRG country portfolio has grown substantially over the course of the past five years, and now includes 26 countries in the LAC region, and operates through a network of 19 field consultants. Until recently, the program in the Eastern Caribbean has received less attention, following a lull in activities in the late 1990’s, and into early 2000. However, the East Caribbean represents the largest growth area for the RMP. q The second IRG contract coincides with a dramatic shift in strategy away from the existing emphasis on courses and response. However, the course content of much of the training remains response oriented. This is in keeping with OFDA´s strategic objective. First response and forest fires still represent the majority of training. In all of the country visits, the emphasis on the need for a stronger presence in risk mitigation and prevention was proffered by client respondents. The recent emergence of SCI as the culmination of the training package is a very positive addition to the OFDA/IRG portfolio. q The focus on key stakeholders at the national and institutional level is – and will remain - a priority for program continuity. As a result, the RMP has had less opportunity to reach out to local level institutions such as community leadership, action committees, and parent groups. These populations and groups make up the first-impact and first-responder landscape and require greater attention in the program. q A further area of some discussion in the field and with IRG and OFDALAC staff has been the need for management and programmatic decisions about the “success” of transfer of CHECCHI AND COMPANY CONSULTING, INC The Louis Berger Group, INC Evaluation: USAID/OFDA/LAC December 2004 Risk Management Program 36 technology and the passing of responsibilities to national institutions and staff. Some countries (in the Latin American case) may have matured to the point of “graduation,” particularly in terms of course instruction Two of the five country visits reflect this. In both Peru and Guatemala, the level of institutional maturation is very high, and one might question the role of IRG consultants (in training) in terms of the level of effort nationally and in terms of the kinds of activities pursued. Graduation in course training roles means pursuing increasing TA and coordination roles. This is not the case in the Caribbean countries, although in the short period of “re-engagement” substantial advances have been made. q The implication also is that new emphases and roles should be emphasized (and to some degree these are becoming evident in strategy): new courses responding to new technologies and events, greater institutionally focused TA, more intense work with entities outside the national institutional framework (NGOs, local community organizations). And, perhaps, a lower level of effort nationally. q Building National Capacity and Coordination. The OFDA/IRG team has produced a high level of service delivery in the countries targeted by the evaluation. OFDA/IRG has the greatest success building capacity within national disaster authority and first-responder institutions at the national level. Certified client-instructors are relied upon to reach out to regional and municipal divisions. Because of the endemic institutional problems with retention and turn-over, the RMP strategy is to reinforce capacity at the national level where retention is less of a problem. To a lesser degree, the RMP has built capacity with local-level governments and community groups based on its current resource-use priorities. Activity at this level is increasing, is clearly the direction to move, and is based on interaction with the NGO community. q OFDA/IRG has been successful in promoting internal institutional coordination and in strengthening regional and international linkages in disaster response. In all three regions, regional organizations have been a principal focus for OFDA/IRG activities. Coordination with USG (USAID) has been a principal role for RA actions, and in a number of country cases the role of IRG consultants in providing immediate communication and coordination through the MDRO has been successful. q Management. The overall management modality or style that has evolved works reasonably well and provides a more effective and accountable institutional mechanism for the “model” that did not exist in the earlier years of OFDA engagement in the region. However, there is some concern that the shift from the initial flexible contractual arrangement (similar to an IQC in the reimbursable nature of costs) to a fixed sum contract may have reduced the flexibility of action on the ground in response to country level needs. q The “seamless” nature of the IRG/OFDA partnership is evident in the relationship between consultants and OFDA staff, in singular strategies, and in the lack of an OFDA/IRG distinction among client nationals. Seamlessness seems to be a function of the long-term relationships, but does not pertain to management functions as it does to operations. However, that is a positive outcome of the contractual relationship, because it has brought a clear sense of management accountability, something absent in earlier years (pre-2000). q However, there is some concern that IRG has stepped into strategic programmatic decisions that are the appropriate purview of OFDA management. This concern has been expressed in particular in relation to strategic decisions with regard to areas to be cut initiated by IRG CHECCHI AND COMPANY CONSULTING, INC The Louis Berger Group, INC Evaluation: USAID/OFDA/LAC December 2004 Risk Management Program 37 as a result of budget shortfalls in 2003 and beyond (discussed in detail below). In this case, the “tail wagging the dog” may have interrupted some of the seamless nature of the relationship q In the current (2003+) period of the new contract with IRG, there has been a problem and a concern about the reason for, and consequences of what appears to be a lack of funding commensurate with a pre-designed work plan and program. It is not the role of the evaluation team to enter the question of why this occurred. However, it important to understand the consequences for Year 1 of the new contract, and the perspective for Year 2 (already in implementation). q There have been consequences that relate to cutbacks in programmed activities, particularly those involving travel for national consultants. Substantially fewer training exercises conducted when compared to initial programming; reduction in the transmission of some new course material that has been designed; some reduction in the face-to-face interaction among consultants to plan and impart experiences; postponement to Year 2 of activities planned in Year 1. However, even with budget reductions from work plan prescriptions, much of the initial work plan has been carried out. The question becomes how to incorporate lessons learned from this period of budget tightening to assure continued financial and programmatic accountability. q In addition, two particular circumstances have exacerbated the funding for key activities of the program: the termination of the participation of Miami-Dade County and related start￾up costs for the new contracts with Fairfax and Los Angeles, and the absence of (expected) contributions for training to the training portfolio. q All aspects of the program have been affected, and it is the evaluation team’s estimation that the program cannot operate fully at the current levels of funding. While we have a number of observations about the what and the how of line items and activities, it is clear that insufficient funds are available. q Replicability. The issue of the replicability of the LAC “model” or approach is a difficult one to answer in the abstract. The essence of the approach can be divided into four key areas – Planning, Training, Technical Assistance, and Management, and includes the following key elements: o The existence of a set of contextually developed course materials, primarily oriented toward response, and increasingly, as an integrated set of materials, part of a risk management process; o A network of national “consultants” who have developed close bonds to OFDA, and in most cases to the original set of the OFDA team. In most cases these consultants were participants in the original course and training program. The “seamless” nature of the IRG/OFDA relationship is an outgrowth of this close knit group of consultants; o National consultants and OFDA Regional Advisors play a key role in first response, providing immediate assistance and damage assessment from a well-nurtured relationship with national institutions of first response and a link to USAID and embassy response; o An evolutionary and demand driven approach to responding to national training needs; CHECCHI AND COMPANY CONSULTING, INC The Louis Berger Group, INC Evaluation: USAID/OFDA/LAC December 2004 Risk Management Program 38 o An institutional multiplier effect as the original sets of trainees have become a set of trainers, both nationally and regionally, reducing the need and role for direct training through the OFDA/IRG consultants who take on a stronger role as quality control, supervision, and technical assistance on the ground; o A requirement of some minimal institutional development on the part of the national government in disaster response and preparedness. The approach is clearly contextual and evolutionary, and in this sense may be difficult to replicate. The long-standing relationship among consultants and the OFDA team is unique, and may be difficult to replicate without similar interpersonal dynamics and collegiality built up over time.. The existence of the original OFDA team, oftentimes referred to as the “four horsemen” is purely contextual, and probably not replicable. However, some elements of the model can and should be adopted by other regions. Proof of replicability comes in the form of the recent “re-emergence” of OFDALAC and IRG in the Caribbean. The adaptation of the “model”, course material and TA is taking root, albeit with a time lag. But this time lag is significantly shorter than the two decades over which the model has been built! While not the same as the Latin American experience, the EC shares many of the characteristics. VI. Future Directions and New Horizons: Innovative Opportunities for OFDA/LAC and Possible Strengthening of OFDA/IRG Catastrophes, by definition, exceed human investment and efforts to reduce risk. In this sense, the cost effectiveness or cost-benefit of risk management programs does not center on a positive net effect financially, but focuses on reducing the level of potential loss. It is not without surprise that the losses from Mitch disheartened many, to the extent that new opportunities to mitigate risk are overshadowed by the awareness that original efforts to do so were easily overwhelmed. Given that preparedness and response will remain the centerpiece of the OFDA mission, the evaluation team, nevertheless, has identified new risk management opportunities, centering on risk reduction and hazard mitigation, and designed to complement OFDALAC’s mission and its current level of human and financial resources. Some of these new directions fall beyond the scope of the IRG contractual relationship. The following statements are based on OFDA’s mandates and experience with catastrophes in the region, and are designed to give context to the New Directions discussed below: ß The commitment to response is the foundational role. ß Frequent events force an emphasis on response, which dissipates mitigation activities and the attention given to concepts like relief-to-development. ß Mitigation and other forward-thinking programs tend to gain priority in countries/regions/locals with less catastrophic losses. ß Commitment to mitigation and other expanded OFDA roles is conditional; there is a threshold to risk reduction efforts (risk acceptance). q The Organization of American States (OAS) funded the Caribbean Hazard Mitigation Initiative, a comparatively low-cost mitigation project that produced flood mapping and vulnerability indexing for Kingston, Jamaica. Though successful, no funds have been found to implement the data through engineering projects for reducing flood vulnerabilities. Given the funding limits within ODPEM (and virtually all national disaster agencies), the CHECCHI AND COMPANY CONSULTING, INC The Louis Berger Group, INC Evaluation: USAID/OFDA/LAC December 2004 Risk Management Program 39 budget priorities within Ministries and Cabinets, Jamaica’s national stakeholders bodies, OFDA/IRG should consider assisting Jamaica’s stakeholders in targeting demonstration projects by using the vulnerability data from the OAS Initiative. With a data-base on flood vulnerability, engineering firms are equipped to conduct a design-specific and site-specific cost-benefit analysis of its interventions. Prior to this initiative, the insurance industry has not been equipped with this data traditionally used in risk underwriting procedures. In addition, Cabinet and Ministry-level personnel are with quantitative data to perform the following project-support functions: target funding sources for projects prioritized according vulnerability and critical needs; use flood-plain data on site-specific vulnerabilities to negotiate terms and conditions for insurance coverage from insurance and/or re-insurance industries. q A common theme among client organizations is a need for OFDA to extend its grasp toward organizations and groups at the local and commmunity level. The closest associate or client of OFDA is and will remain the lead national institution and NDC within countries. However, at both the first response and mitigation level, these is a felt need that much more can be done. As is described in the text of this report, several clear needs and efforts in this direction emerge: the shortcomings of the Jamaican Parish response to Hurricane Ivan; the successful and impressive cohesion that has emerged in the case of Piura’s Disaster Network; the strong relationship of CONRED with some 17 NGOs in Guatemala. q Linked to the above is a need for OFDA/LAC to form closer alliances with the NGO community. OFDA/LAC has matured in its institutional relationship with countries both in training and in technical assistance. So too has the NGO community moved from a “first generation” of foreign relief and direct humanitarian assistance to a “second generation” of more institutional relationships with national personnel, to a “third generation” of NGOs with sustainable developmental partnerships within countries linking governmental institutions and the community. These third generation NGOs provide clear opportunities to deepen not only first response, but strengthen development-oriented efforts at risk and vulnerability reduction. OFDALAC and IRG need to strengthen the involvement of these NGOs in their program. (The cases of Piura in Peru and Guatemala suggest these new opportunities.) q The Piura Regional Government and disaster network provides a potential model for new directions and an understanding of real trade-offs in risk management.. Recent coordination efforts build upon risk mitigation opportunities and their cost-benefit with respect to the region’s history of economic losses from disasters. However, economic loss has caused a selective acceptance of risk with respect to zoning and residential construction in the city of Piura, because the cost to build in less risky zones is reported to be four times higher than in risky ones. In addition, a growing population has led to public demand for affordable housing, a related governmental willingness to side-step hazard mapping, building code enforcement standards, and the allowance of building in flood and fluvial soil (earthquake) zones. q In communities throughout the region, this same scenario is repeating itself. OFDA/IRG is well-positioned to use its existing base of resources and relationships to build a risk prevention mindset at the community level. The following OFDA/IRG activities will not only improve the RMP, but will enhance the relief-to-development linkage by empowering communities to reduce their risk profiles: Identify funding sources to bring local, regional, CHECCHI AND COMPANY CONSULTING, INC The Louis Berger Group, INC Evaluation: USAID/OFDA/LAC December 2004 Risk Management Program 40 national stakeholders together for workshops, and planning cells; establish consistent contact with communities by leading the development of (OFDA/PRG) vulnerability assessment and risk reduction work plans with the PRG and community commissions, including providing guidance and methods for community-based monitoring and evaluation; increase pilot projects in communities, with a focus on building code “minimum standards” and the provision of “minimum equipment” to carry out preparedness and hazard response efforts; increase efforts to involve the PRG’s Scientific Advisory Group in the region’s formal risk prevention and crisis response activities. q National institutions aimed at risk management, relief and reconstruction find themselves searching for funding in a world of dwindling resources. A common complaint among national agencies is that programmatic budgets are virtually non-existent, and even national counterpart funding for training and workshops is minimal. A similar recognition is part of USAID’s development strategy and strategic objectives. One answer is an increasing recognition of the need to form strategic “Public/Private Alliances” in the face of dwindling resources. A similar recognition is part of USAID’s development strategy and strategic objectives (SO). A recent pilot project in Kingston, Jamaica highlights some of these possibilities where vulnerability assessment and mapping could be linked to private sector development, property insurance and re-insurance interests. Involvement of university students in vulnerability assessment and training is an important part of this pilot exercise. A recent USAID project initiative in Guatemala with public private alliances in the social sectors (health, education) also suggests the possibility of incorporating risk management and assessment efforts to developmental purposes. Finally, it is the opinion of the team that the “service delivery” model and the current OFDA/IRG relationship bring a great deal of value added to OFDA’s capacity to accomplish a great deal in a very cost effective manner. The “bang for the buck” is evident. Whether one measures OFDA’s impact from a process perspective (numbers trained, courses delivered, etc.), from the perception of the client (“OFDA has created the national institutional capacity,”), from an impact perspective (national institutions and programs bear the OFDA stamp in virtually all of their training program, response time has been reduced and coordination increased, overall vulnerability has been reduced and bears some relationship to the presence of OFDA), there is a clear message: it works and works quite well. While some improvement along lines suggested above will make the model better, as it currently stands, it works and well. CHECCHI AND COMPANY CONSULTING, INC The Louis Berger Group, INC Evaluation: USAID/OFDA/LAC December 2004 Risk Management Program 41 ANNEX A: CONTACT LIST The following names form a partial listing of interviews undertaken by the evaluation team. The list is divided by geographic locale. USAID/OFDA, Washington, D.C. Rob Thayer Regional Coordinator for Asia, Latin America and Caribbean Gilbert Collins Team Leader, Planning and Evaluation Marcella Michaud Program Operations Assistant Barbara Howald Training Advisor James Fleming Disaster Operations Specialist Greg Gottleib Deputy Director Michael Marx Disaster Response Team Leader OFDA Stephanie Savolaine OFDA Eileen Simoes OFDA Milena Popp OFDA Carol Chan OFDA Washington, IRG Katherine McNeil Senior Manager Relief and Reconstruction Division IRG Timothy Knight Corporate Vice President IRG USAID/OFDA/LAC/IRG Costa Rica Tim Callaghan Senior Regional Advisor OFDA Costa Rica Rene Carrillo Regional Advisor OFDA Phil Gellman Regional Advisor OFDA John Taylor Chief of Party IRG Juan Pablo Sarmiento Señor Technical Manager IRG Fernando Calderón IRG Consultant, Costa Rica Nelly Segura IRG Training Specialist María Luisa Alfaro IRG Consultant in Forestry Manuel Ramirez IRG Consultant in Education CHECCHI AND COMPANY CONSULTING, INC THE Louis Berger Group, INC Evaluation of USAID/OFDA/LAC December, 2004 Risk Management Program Costa Rica Sr. Luis Diego Morales Presidente, Comisión Nacional de Emergencia Sr. Hector Chávez Director Nacional Cuerpo de Bomberos Sr. Luis Fernando Salas Subjefe, Departamento de Operaciones Cuerpo de Bomberos Sr. Miguel Carmona Presidente Consejo Nacional Cruz Roja, CR Sr.. Guillermo Arroyo Director Nacional Socorros y Operaciones Cruz Roja, CR Sr. Jorge Rovira SubDirector Socorros y Operaciones Cruz Roja, CR Jamaica and Eastern Caribbean Carl Herbert St. Kitts and Nevis National Disaster Coordinator Dr. Kevin Rushing MDRO and Deputy Director USAID – Jamaica Howard Basset Assistant MDRO USAID-Jamaica Alan Ross IRG Consultant- Jamaica Dawn French National Disaster Coordinator Saint Lucia Beryl Armbrister IRG Consultant Bahamas Dr. Barbara Carby Director ODPEM - Jamaica Ron Jackson Deputy Director ODPEM - Jamaica Julie Leonard Regional Advisor, OFDA Barbados Yvonne Clark Director, Red Cross Jamaica Ruth Chisholm Assistant Director, Red Cross Jamaica Perú Carlos Cordova IRG Senior Consultant Lima Dante Torres IRG Consultant Lima Dr. Jorge Grande IRG Senior Consultant, Southern Cone Contralmirante Juan Luis Podesta Llosa Instituto Nacional de Defensa Civil (INDECI) CHECCHI AND COMPANY CONSULTING, INC THE Louis Berger Group, INC Evaluation of USAID/OFDA/LAC December, 2004 Risk Management Program Dr. Alejandro León Paxes Secretario Permanente de los Consejos Consultivos INDECI Tulio Nicolini, Comandant General Cuerpo General de Bomberos Voluntarios del Perú (CGBVP) José Aguirre, Vice Comandante Cuerpo General de Bomberos Voluntarios del Perú (CGBVP) Julio Méndez Director de Capacitación e Instrucción (DIGECIN) Cuerpo General de Bomberos Voluntarios del Perú Augusto Zegarra Secretario Técnico del Gobierno Regional de Piura Norma Ordinola Concejal de la Municipalidad de Piura Manuel Javier Moizer INDECI-Piura Rolando Sosa Alzamora Ministerio Regional de Salud German Coreza Durand Gobierno Regional Piura Lucy Harmam CARE Piura Ing Hugo Ruíz Director de Operaciones I Región de Defensa Civil Dr. Celso Banbarem Director General, División de Desastres Ministerio de Salud (MINSA) Luis Honorio Director de Operaciones MINSA Lic. Abel Aliaga Director de Planificación MINSA Dr. José Ráez Universidad Nacional Mayor de San Marcos UNMSM Ing. Roberto Segura Director General Ministerio del Interior (MINTER) Santiago Valero Coordinador de Capacitación y Movilización Oficina de Defensa Nacional MINTER Guatemala Ricardo Berganza IRG Senior Consultant Guatemala Oficial Mayor Jorge García Molina Comandante Ejecutivo Cuerpo de Bomberos Municipales Guatemala Ing. Miguel López, Coordinador Nacional de Protección Forestal Instituto Nacional de Bosques (INAB) Lic. Jessica Solano Departamento de Cooperación CONRED CHECCHI AND COMPANY CONSULTING, INC THE Louis Berger Group, INC Evaluation of USAID/OFDA/LAC December, 2004 Risk Management Program Lic. Hugo Hernández, Secretario Ejecutivo CONRED Ing. Miguel Cotero SubSecretario Ejecutivo CONRED Lic. Walter Wintzer Gerente de Organización y Capacitación CONRED Lic. Manuel Pinelo Asesor CONRED Lic. Eber García, Departamento de Educación. CONRED Ing. Ovidio García, Gerente de Manejo de Emergencias. CONRED Lic. César González, Comandante Primer Jefe. Cuerpo Voluntario de Bomberos Oficial Juan José Ruiz Cuerpo Voluntario de Bomberos Oficial Mario Quintanilla Cuerpo Voluntario de Bomberos Mayor Marco Antonio Ramos 2ndo Comandante Cuerpo Voluntario de Bomberos Mayor Cesar Augusto González 1er Comandante Cuerpo Voluntario de Bomberos Oficial Daniel Antonio Méndez Cuerpo Voluntario de Bomberos Oficial Héctor Mauricio Sicaza Sub Jefe, Patrulla de Rescate Cuerpo Voluntario de Bomberos Mayor Fulgencio Marcelonio Angel Lopez Cuerpo Voluntario de Bomberos Mayor José Carlos Sierra Vocal II Directorio Nacional Cuerpo Voluntario de Bomberos Mayor Juan Carlos Cortés Vocal IV Directorio Nacional Cuerpo Voluntario de Bomberos Mayor Rolando Ruiz Vocal I Directorio Nacional Cuerpo Voluntario de Bomberos Dr. Pablo Estacuy Cirujano General Cuerpo de Bomberos Municipales Ing. José Luís Loarca Asesor Regional de Emergencias LACHO, CRS Nicaragua Sergio David Gutiérrez IRG Consultant Ingeniero Cristóbal (Tito) Sequeira Secretario Ejecutivo de SINAPRED Ingeniera Ivonne Velásquez Directora Sectorial, SINAPRED Licenciada Lesbia Centeno/Silvia Ulloa Directora Gestión del Riesgo Ministerio de Educación Ing. Miguel Alemán Sub Director Dirección General de Bomberos Ingeniero Ricardo Selva CHECCHI AND COMPANY CONSULTING, INC THE Louis Berger Group, INC Evaluation of USAID/OFDA/LAC December, 2004 Risk Management Program Director Nacional Federación de Bomberos Voluntarios Ingeniera Zaida Zúñiga Instituto Nacional Forestal (INAFOR) Coronel Mario Pérez Cassar 1er. Jefe Defensa Civil Estado Mayor Defensa Civil CHECCHI AND COMPANY CONSULTING, INC THE Louis Berger Group, INC Evaluation of USAID/OFDA/LAC December, 2004 Risk Management Program ANNEX B: COUNTRIES VISITED AND ITINERARY Washington, D.C.: OFDA/W: October 12-16 San Jose, Costa Rica: OFDA/LAC, IRG and Costa Rican governmental and non-governmental institutions: October 17-31 Kingston, Jamaica: IRG, USAID and governmental and non-governmental institutions: November 1-5 Lima and Piura, Peru: November 6-11 Guatemala City, Guatemala: November 12-16 Managua, Nicaragua: November 17-20 San Jose, Costa Rica: November 21-30 Washington, D.C.: December 6-8 CHECCHI AND COMPANY CONSULTING, INC THE Louis Berger Group, INC Evaluation of USAID/OFDA/LAC December, 2004 Risk Management Program ANNEX C: INTERVIEW PROTOCOLS As part of the preparation and conduct of this evaluation the evaluation team has established a set of interview instruments or protocols that serve as a semi-structured methodology for gathering information from different groups of actors. While these instruments are formalized, in most cases they were utilized simply as a guide to a more conversational interview aimed at eliciting specific information, but not limiting commentary of relevance. In the cases of the two regions – LAC and the Eastern Caribbean – interviews were conducted in the language appropriate to the interviewee, although the protocols as presented reflect only one of the two languages utilized. In the cases of the Senior OFDA Regional Advisor and IRG Technical Manager no such protocols were developed because of the multiple interactions necessary, the need to procure documentation as well as information and impressions, and the overall need to reinforce images and impressions of the evaluation team throughout this exercise. These different protocols apply to the following: A. Chief of Party for IRG B. Regional Advisors for OFDA C. Field Consultants for IRG (teleconference interviews in particular) D. Country Client Organizations (Firefighters, National Emergency Institutions, Red Cross, etc.) and National Disaster Coordinators E. Ranking Questionnaire for National Institutions Below we provide these protocols. A. Interview Protocol for Chief of Party IRG 1. History: Arriving in February 2000, what did you find from a management and administrative side. 2. How did the administrative and technical sides interact? What changed? 3. Data from the first two years seems to have been reconstructed ex-post. How were you and your predecessor able to measure efficiency, effectiveness, cost/benefit etc. at that stage? 4. How does the Plan Pais work as a management and administrative tool, not just a program tool? 5. Effect of the period of extensions in 2002-2003 on work program? Shift from grants to reimbursable IQC to a competitive fixed cost contract. How does the change affect the work program from both a technical and administrative perspective? Flexibility? 6. What advantages and disadvantages for folding this more directly into the USAID Mission portfolio? CHECCHI AND COMPANY CONSULTING, INC THE Louis Berger Group, INC Evaluation of USAID/OFDA/LAC December, 2004 Risk Management Program 7. How do you see relationships on the ground (CR) between OFDA and IRG. Concept of a “seamless” organization has often been mentioned. Are there advantages and disadvantages to that? B. Interview Protocol for Regional Advisors 1. What are the principal roles for Regional Advisors today? 2. How has that changed over three periods: pre-1998; 1999-2003; 2003 – today? 3. Has the changed administrative and management style changed the way in which OFDA operates? For example, Paul Bell and the RA system and the consultants versus post-Paul Bell and the RA and IRG and consultant system. 4. Is there some conflict between the roles of the RAs and IRG leadership in relation to consultants? 5. Delicate but important question: why does OFDA need both a support contract and the RA structure? If not both necessary, which ought to go? 6. What about OFDA-AID? Can the seamlessness in IRG/OFDA/LAC be said of OFDA/LAC and USAID Missions? Can the role of the MDRO as the point person in the Mission be strengthened and need it be? How, if at all? Can OFDA/USAID relationships be altered and need they be? How? 7. Other thoughts relating to both programmatic content and management/administrative structure? C. Interview Protocol for IRG Consultants and National Disaster Coordinators (Teleconference in countries not visited) 1. ¿Qué es la historia de la relación entre OFDA/IRG y sus clientes en el país? ¿Qué son Fuerzas y/o debilidades con la relación?? 2. ¿Que son los programas principales de OFDA/IRG? 3. ¿Cómo se comparten las prioridades de planificación, gestión, y/o evaluación de los programas con sus clientes? 4. ¿Que son las contribuciones de OFDA/IRG hacia la capacitación de sus clientes? ¿Hay ejemplos de impacto positivo? 5. ¿e puede distinguir las actividades de OFDA y las de IRG? 6. En general, ¿se puede caracterizar el impacto de los programas de OFDA/IRG? ¿Hay ejemplos tangibles de una reducción de vulnerabilidad en el país? CHECCHI AND COMPANY CONSULTING, INC THE Louis Berger Group, INC Evaluation of USAID/OFDA/LAC December, 2004 Risk Management Program 7. ¿Cree UD. que se puede replicar elementos o el modelo de OFDALAC/IRG en otras regiones? D. Interviews Protocol for Country Client Organizations and (potentially) National Disaster Coordinators 1. Coverage: a. Un poco de la historia de la relación con el programa de OFDA/IRG: cuando comenzó, que actividades, como llegaron a identificar necesidades, que instituciones han participado, quienes han sido los beneficiarios del programa b. The national institution: what is the scope of its role? How does it interact with other institutions? Public/private? Level of coordination among national institutions and is it a result of OFDA? 2. Perceptions of OFDA/IRG and OFDA/IRG Impact a. Perceptions of OFDA/IRG Activities: ¿Como ven Uds. la contribución de la presencia de OFDA/IRG y como han afectado el comportamiento de su agencia/institución? ¿Fortalezas y debilidades en la relación? ¿Que se puede mejorar? b. Como ha afectado el establecimiento de sus prioridades la presencia de OFDA? Que criterio? ¿Compartido el proceso? c. ¿Puede describir una o varias instancias donde el efecto de la capacitación, TA, u otras actividades de OFDA es evidente? Describe. d. ¿Que pasará si el contrato o la intervención de OFDA terminara mañana? ¿Las actividades son sostenibles en términos e las actividades de las instituciones nacionales? e. Que rol ha tomado personal de OFDA-IRG en la respuesta a desastres aquí, tanto antes como después. f. Como distinguen entre actividades de OFDA y actividades de IRG? g. Que ha sido el énfasis de las actividades de OFDA/IRG con Uds.? ¿El ciclo ponderado de 1-5? ¿Que nivel de impacto ha alcanzado en estas áreas? ¿En ciertas áreas ha sido mayor el impacto que en otros? i. Planificación: ii. Capacitación iii. Mitigación/prevención iv. Asistencia Técnica v. Respuesta vi. Monitoreo y evaluación vii. Gestión de riesgo h. Ha habido un énfasis en la integración de las varias actividades en una forma holística? O sea, ¿contribuye a la coordinación e integración de esfuerzos nacionales? CHECCHI AND COMPANY CONSULTING, INC THE Louis Berger Group, INC Evaluation of USAID/OFDA/LAC December, 2004 Risk Management Program i. Que impacto ha habido de las actividades de OFDA/IRG? ¿Como se puede medir este impacto? j. Ha habido mucha gente entrenado o capacitado en el programa. ¿Que es su percepción del impacto a nivel individual? ¿Retención de personal capacitado? ¿Efectos en su desempeño? 3. Problems/Opportunities a. ¿Ha habido problemas en la capacidad de OFDA/IRG de responder a las necesidades de Uds.? ¿O en la coordinación de acciones o en prioridades? b. ¿Cómo han resuelto estos problemas en términos institucionales? E. Subsequent Questionnaire of Ranking for Client Institutions and Organizations Favor de responder a las frases siguientes con el nivel que corresponde a su opinión: 1 (no acuerdo); 2 (acuerdo mínimo); 3 (acuerdo moderado); 4 (acuerdo fuerte); 5 (acuerdo excepcional). 1. Los programas de OFDA son efectivos porque satisfacen las necesidades de su organización. 2. OFDA inició un proceso consultivo con su organización para establecer el contenido y formato de los programas y prioridades. 3. Los programas de OFDA son efectivos porque han capacitado a personal de su organización. 4. Los programas de OFDA han producido impactos positivos que son tangibles. 5. OFDA exhibe un proceso suficiente en el seguimiento de sus programas. 6. Se encuentra debilidades en los programas a través del proceso de evaluación. Cómo se puede mejorar la retroalimentación de sus programas. 7. OFDA exhibe una metodología sostenible y transferible. 8. En general, los programas de OFDA han logrado una reducción de la vulnerabilidad en su país. 9. En general, OFDA continuará a ser al centro de la gestión de riesgo porque es una organización que responde al cambio de asuntos. Comentario: Hay otros asuntos o programas que deban ser considerados debajo de la misión de OFDA? CHECCHI AND COMPANY CONSULTING, INC THE Louis Berger Group, INC Evaluation of USAID/OFDA/LAC December, 2004 Risk Management Program ANNEX D: SCOPE OF WORK: EVALUATION OF USAID/OFDA/LAC RISK MANAGEMENT PROGRAM Evaluation of USAID/OFDA Risk Management Program for Latin America and the Caribbean Overview The U.S. Agency for International Development’s Office of U.S. Foreign Disaster Assistance (USAID/OFDA) seeks to evaluate activities in risk management and other sectors implemented via contract with the International Resources Group (IRG) in Latin America and the Caribbean from 1998 to the present. This evaluation will focus on documenting and assessing overall achievement, program management, effects and impact. OFDA seeks a team of two experienced professionals to conduct research in the field and Washington over an estimated period of 47 days. Background In pursuit of its global mandate to save lives, alleviate suffering, and reduce the economic impact of disasters, OFDA has a contract with International Resources Group (IRG) to provide various programmatic and administrative services in the LAC region. This contract, which began in June 1998, was originally used primarily to conduct disaster response-related activities. In May 2003, OFDA shifted its overall approach in the LAC region, now placing a primary emphasis on risk management. The focus of the IRG contract changed accordingly, with its activities increasingly focused on reducing regional vulnerabilities while simultaneously building local capacity to respond to disasters. IRG support contract activities now emphasize technical assistance and training, though response remains of significant importance. Overall, the IRG support contract offers training, technical assistance, disaster mitigation, and disaster response support to twenty six countries in the LAC region. Activities carried out under the contract are tailored to the particular needs of each country. OFDA is now looking to learn lessons from its experience with the IRG support contract. OFDA seeks to document the types of assistance provided under the IRG support contract, assess the effectiveness and sustainability of the activities undertaken, identify areas of strength and potential improvement, and obtain recommendations on the future shape and scope of a future institutional support contract in the LAC region. Evaluation Questions The evaluation will address the following series of questions. CHECCHI AND COMPANY CONSULTING, INC THE Louis Berger Group, INC Evaluation of USAID/OFDA/LAC December, 2004 Risk Management Program Overview & Coverage What types of risk management activities and projects have been carried out in the Latin America & Caribbean region by OFDA under the IRG contract? Which groups have benefited from activities under the IRG contract? Were the demographic groups with greatest needs targeted by contract activities? Subjectively, how are OFDA’s risk management contributions in the LAC region generally perceived by the intended beneficiaries? What process was used by OFDA and/or IRG to determine the sectors to be targeted by contract activities? What process was used to determine the geographic areas targeted by contract activities? What were the causes of any sectoral, geographic, or demographic differences in coverage? Were they external or a product of program design and implementation? Outcomes and Impacts What are the most significant outcomes and impacts of activities undertaken by OFDA through the IRG contract on beneficiaries over the duration of the contract? How do the various types of risk management activities undertaken differ in impact? Do OFDA’s interventions through the IRG contract contribute to enhancing the coordination capacity of regional, national, and local authorities and organizations? Will the effects of IRG’s ongoing contract activities be sustainable without continued external financial input? What factors will contribute to this possible sustainability? Is the range of risk management activities undertaken by OFDA through the IRG contract generally cost-efficient? Are some types of activities more cost-efficient than others? Have there been any significant problems in the design or implementation of activities under the IRG contract? Are there any examples of risk management projects carried out under the IRG contract which can be demonstrated to have a measurable positive impact on communities? Are there any instances where IRG-implemented risk management activities did not pay dividends as expected? Coordination How did OFDA risk management priorities and subsequent activities implemented under the IRG contract affect strategies developed by other donors and implementers? Did other actors meet critical risk management needs that were unmet by OFDA? In what ways were OFDA-supported risk management activities coordinated with USAID Missions in the LAC region? Looking ahead, what are the prospects for increased OFDA-Mission risk management program integration, and for possible handoff of OFDA-sponsored risk management programs to USAID Missions? What role has IRG contract staff played in coordinating various relief actors, both before and during crises? In coordinating with USAID Missions? Program Management Is the IRG contract budget sufficient to meet OFDA’s operational requirements? If not, what are the causes of any budgetary shortfalls? Describe the responsiveness of IRG contract staff to action requests from OFDA Regional Advisors. CHECCHI AND COMPANY CONSULTING, INC THE Louis Berger Group, INC Evaluation of USAID/OFDA/LAC December, 2004 Risk Management Program Is there enough programmatic flexibility within the contract scope of work to allow IRG to carry out all of the tasks required by OFDA? Do regional authorities, relief organizations and beneficiaries tend to attribute IRG contract activities to OFDA or to IRG? Do OFDA-funded IRG consultant staff tend to view themselves as affiliated primarily with OFDA or with IRG? Looking Forward Are there additional risk management activities that could be undertaken under a future risk management institutional support contract? Is it preferable to have a broad or narrow sectoral focus for risk management activities under an operational strategy for the LAC region? Geographic focus? Demographic focus? Should certain types of risk management activities (training, technical assistance, etc.) be emphasized to a greater extent in a future LAC regional support contract? Should any be de-emphasized? Are there any other issues OFDA should consider in planning for a future support contract? Evaluation Team & Estimated Level of Effort The two-person evaluation team will consist of a team leader and one project specialist. To provide a broader perspective and better facilitate data collection, prospective evaluation teams are strongly encouraged to include at least one current or former national from a country in the LAC region, and also to include a mix of genders. OFDA staff will assist as necessary with the facilitation of meetings and procurement of documents. The team should collectively possess the following set of skills: Experience carrying out two or more humanitarian evaluations for a major donor, international NGO, or international organization, experience managing and/or implementing disaster risk management activities (including preparedness, training, and other capacity building initiatives) in various geographic regions around the world, preferably from several perspectives - United Nations (UN), international organization (IO), non-governmental organization (NGO), and/or donor. Familiarity with the political and humanitarian context in Latin America and the Caribbean. Proficiency in Spanish is preferred. The team leader will be a Senior Level Institutional Analyst. The second team member will be a Mid-Level Humanitarian and Crisis Analyst. Both members will participate for the entire duration of the evaluation. The primary point of contact for the evaluation team will be the OFDA Senior Regional Advisor in San Jose, Costa Rica. Methodology and Estimated Timeline The evaluation team will conduct the evaluation and complete the report in CHECCHI AND COMPANY CONSULTING, INC THE Louis Berger Group, INC Evaluation of USAID/OFDA/LAC December, 2004 Risk Management Program approximately 47 days. Key informant interviews and document review in Washington, DC and San Jose, Costa Rica (7 days). The team will meet with staff from OFDA, including the OFDA Senior Management Team and Training Advisor in Washington, and the OFDA/LAC Regional Team in San Jose. The team will review strategic assessments, contract files, and other relevant documents. OFDA staff will assist with facilitation of meetings and procurement of documents as necessary. Field work and data collection in LAC Region (24 days). The team will conduct research throughout the LAC region, meeting with IRG staff, government representatives at all levels (as appropriate), donors, beneficiaries, regional coordination bodies, international and local NGOs, UN organizations, U.S. Embassy staff, and other relevant groups and individuals. OFDA staff in the region will assist with facilitation as necessary, but the evaluation team is expected to be as independent as possible. Writing report (10 days). The team will draft the report over 10 days in San Jose, Costa Rica. Briefing staff (3 days, not including travel). The team brief OFDA staff in San Jose, Costa Rica, and will then travel to Washington to brief OFDA and IRG staff on findings and obtain feedback. Final report revisions and printing (3 days). Following the final oral briefings and taking into account any new information obtained, the evaluation team will prepare and publish a final version of the evaluation report. Deliverables The evaluation team will produce the following deliverables: Work Plan: Prior to departure to the field from San Jose, Costa Rica, the evaluation team will produce (with OFDA input) a 2-3 page written strategy detailing how the evaluation will be completed. The work plan will include a list of countries and locations to be visited, potential interviewees, a draft list of interview questions, and a description of any other data collection instruments (e.g., surveys) to be used. The questions and instruments should be tailored to individual categories of respondents such as implementing partners, beneficiaries, government officials, and other donors. This plan will be submitted to the OFDA Senior Regional Advisor in San Jose, Costa Rica for approval. Written Report: The team shall write and present for review a first draft of the evaluation report at least one week prior to the final oral briefings (below). The report will include an executive summary, brief overview of the humanitarian context in the LAC region the focus period (as pertains to preparedness and response), description of methodology, and a detailed description of the evaluation’s findings and recommendations. Additional information including team itinerary, interviewee lists, questionnaires, surveys, and CHECCHI AND COMPANY CONSULTING, INC THE Louis Berger Group, INC Evaluation of USAID/OFDA/LAC December, 2004 Risk Management Program bibliography should be included in annexes. The report should be no more than 40 pages, excluding annexes. No more than ten business days following the final oral briefings and taking into account any new information obtained, the evaluation team will prepare and print a final version of the evaluation report, with the number of printed copies to be determined. Final Oral Briefings: At least one week after distribution of the written report to OFDA, the full evaluation team will conduct four oral debriefs to present findings, one to OFDA staff in San Jose, Costa Rica, one with OFDA senior management in Washington, DC, one to a broader audience of OFDA staff, and one to a joint meeting of IRG staff and select OFDA representatives, to present study findings and obtain feedback.