EVALUATION OF THE RAPID ENVIRONMENTAL IMPACT ASSESSMENT PROJECT A REPORT PREPARED FOR CARE USA David Stone, Livelihood and Environmental Security (LIVES) Advisory Group, 1272 Genolier, Switzerland ACKNOWLEDGEMENTS The author of this report is grateful to CARE USA for the opportunity to undertake this evaluation. Particular thanks to Jock Baker, CARE USA, for steering this work in a very careful and appropriate manner: your wisdom and support are very much appreciated. Special thanks also to Charles Kelly who helped guide the author to pertinent reports and provided much additional information, without which this evaluation would have been a far more difficult undertaking. Paul Thompson kindly supplied reference materials and much additional insight to the training element of this project, while Colin Reynolds kindly assisted with technical aspects of the process. At CARE Atlanta, Lisa Smith helped greatly with administrative and logistical arrangements and my thanks for that. Finally, many thanks to all those who responded to questionnaires or phone calls: your insights to this project have been most useful, as selected comments in this report highlight. iii EXECUTIVE SUMMARY BACKGROUND CARE USA and the London-based Benfield Hazard Research Centre have been pursuing the development and application of a Rapid Environmental Impact Assessment (REA) tool for use in emergency and disaster situations. The results of the REA are expected to be used in relief operations assessment and planning. This should result in a discernible improvement in how relief operations identify and deal with environmental issues which, if it succeeds, would serve as an important indicator of the project’s success. The first phase of the project began in August 2001 with initial funding from the joint programme of the UN Environmental Programme (UNEP) and the UN Office for the Co￾ordination of Humanitarian Affairs (OCHA). Additional funding was secured from USAID’s Office of Foreign Disaster Assistance (OFDA), the Royal Norwegian Ministry of Foreign Affairs, and CARE International. This first phase was set to end in February 2004. A range of activities has been undertaken and outputs produced in accordance with the project description. Now, at the end of the project period, CARE USA has requested an evaluation of the REA process. This has been conducted by an independent consultant who has had no formal links with the REA process and who has had no recent contact with the project or experience with the outputs. The purpose of the evaluation was to: 1. document actual outcomes against performance measurement criteria stated in the relevant project and sub-project documents; 2. assess the effectiveness of the REA process as a “best practice” tool in disaster management; 3. consolidate and summarise perceptions of project participants and interested parties about the REA methodology and training materials, and 4. identify successes – matched against preset indicators – and improvements in the project implementation process. The evaluation was carried out on a part time basis from 17 January to 10 March 2004. In all, 15 days were allocated for this evaluation. Work was to have been undertaken on the basis of a desk study: no field authentication was required. KEY FINDINGS The following achievements can be highlighted: • development of comprehensive Guidelines for Rapid Environmental Impact Assessment in Disasters, and an associated Quick Guide to the same process; • field tests organised and conducted in Afghanistan, Ethiopia and Indonesia; • Participant’s Handbook and Trainer’s Guide produced as standard training materials; • pilot training events conducted in Norway, Central America and India: among those trained has been a small group of CARE staff who could possibly now undertake additional training; • REA concepts integrated into Sphere and (proposed for) the latest revision of OFDA’s Field Operations Guide; iv • mainstreaming of REA guidelines and principles into ongoing training programmes. Introductory modules on REA are now routinely included in OFDA’s standard assessment training and UNEP/OCHA’s emergency management training; • localisation of REA – CARE Ethiopia is developing local versions of the training modules for partners; REA principles have also been integrated into university curricula for environmental students in Honduras; • a web site (http://www.benfieldhrc.org/SiteRoot/disaster_studies/rea/rea_index.htm) which contains key REA resources – from this project as well as related themes and issues; and • development and submission of a proposal to donors for a second phase to this project. While the evaluation recognises the many positive outcomes of this phase of activities, it also sheds light on a number of specific issues which urgently need to be addressed. Key among these is the logic and methodology applied in the REA process, as described in the Guidelines – the core of the whole project and process – the presentation of these materials, and the need to create ownership for this process which might help ensure that recommendations from REAs are translated into action. LESSONS LEARNED Specific lessons can be drawn from the reports from field tests, training events and, to a lesser extent, feedback from contacts made during the course of this evaluation. These are summarised below – with particular reference given to the application of the REA Guidelines, management of the project thus far and the use of end results – and expanded upon in Section 3 of this report. Preparing to Use the REA Guidelines • Good preparation is essential for all field tests and training events. • The REA Guidelines proved easier to use if adapted to the country/local conditions. • Language and language skills needed particular attention. • Lack of information on ways to address environmental issues hampered the decision￾making process and may flaw recommendations. • An independent (REA-based) collection of data at the community level is not necessary if other disaster impact assessments are conducted. • The assessment process can serve a double role in terms of assessment and education. • Sensitivity needs to be shown to gender-related issues throughout the whole process. Applying the REA Guidelines • Use of the Guidelines can help identify critical environmental issues. • Information collected and issues identified during the assessment provided useful input into formulating relief project proposals, but are not the only inputs required. • The REA Guidelines can produce useable results without extensive training or support. • A REA assessment can take a considerable amount of time for a tool intended to be used in an emergency. • But, it is not an overtly expensive undertaking. • Conducting a REA at different organisational levels – Head Office, Field Office, community – can help harmonise views as to disaster impacts and response needs and priorities. • Validation of assessment results is important and can provide additional insight into environmental and emergency conditions. v • The Guidelines-based assessment process is more difficult to accomplish for multi-sector and geographically diverse assistance operations than for a geographically limited and highly focused activity. Managing the Process • Local institutional responsibility is required for the REA process to be managed and results used. • Participation in the REA process will detract staff from other disaster response activities. • Sharing lessons and experiences is an important part of this process. Applying the Results • Translating the issues identified into action can be difficult where an assistance programme is already well established. • Further guidance is needed in the Guidelines on how to use the assessment results. RECOMMENDATIONS The main recommendations drawn from this evaluation are shown below and expanded in the main text and Section 5. Recommendation 1. Strengthen the Institutional Structure and Commitment behind this Project. Recommendation 2. Enhance the Technical Integrity of the REA Process. Recommendation 3. Enhance the Quality of the Project’s Outputs to Encourage Use and Application. Recommendation 4. Identify and/or Allocate Resources to Encourage and Enable Follow-up to Past and Future REA Field Tests. Recommendation 5. Continue to Establish Key Partnerships and Focus Resources on Getting these Agencies to Use or Customise the REA for their Own Benefits. Recommendation 6. Produce a Short, Sharp Training Module on the REA. Recommendation 7. Focus Attention on Training Potential REA Leaders and Other Trainers. Recommendation 8. Revitalise or Abandon the Advisory Group. Recommendation 9: Improve the Visibility and Outreach of the REA Process. vi TABLE OF CONTENTS ACKNOWLEDGEMENTS II EXECUTIVE SUMMARY III 1. INTRODUCTION 1 1.1 The Context: People, the Environment and Disasters 1 1.2 A Need for Assessments? 1 1.3 Rapid Environmental Impact Assessment 2 1.3.1 Background to this REA Project 2 1.3.2 Goal of Project 3 1.3.3 Implementation Arrangements 4 2. EVALUATION OF THE RAPID ENVIRONMENTAL IMPACT ASSESSMENT 6 2.1 Purpose 6 2.2 Approach Taken and Methodology 6 3. MAIN FINDINGS I: PROCESS AND OUTPUTS 8 3.1 The REA Process and Guidelines 8 3.1.1 The REA Process 8 3.1.2 The REA Guidelines 9 3.2 Field Tests 11 3.2.1 Comparison of Main Findings from the Three Field Tests 12 3.3 Lessons Learned 16 3.3.1 Preparing to Use the REA Guidelines 17 3.3.2 Applying the REA Guidelines 18 3.3.3 Managing the Process 19 3.3.4 Applying the Results 19 3.4 Training Materials 19 3.4.1 Background 19 3.4.2 Training Guides 20 3.4.3 Training Workshops 21 4. MAIN FINDINGS II: MANAGEMENT AND MONITORING 23 4.1 Administrative Aspects 23 4.2 Donors and Funding Support 24 4.3 The REA Advisory Group 24 5. RECOMMENDATIONS AND CONCLUSTIONS 26 5.1 Introduction 26 5.2 Has This Project Achieved its Objectives? 27 5.3 What Could Be Improved? 29 5.3.1. Some Technical Needs 29 5.3.2. Some Content/Presentation Needs 29 5.4 Recommendations 30 5.5 Suggested Next Steps 32 vii ANNEXES ANNEX I REA Chronology 34 ANNEX II Sample Questionnaire Used in This Evaluation 37 ANNEX III Terms of Reference 38 ANNEX IV People Contacted in the Evaluation 42 ANNEX V Materials Consulted 43 1 1. INTRODUCTION 1.1 THE CONTEXT: PEOPLE, THE ENVIRONMENT AND DISASTERS Experience shows that the environment is often marginalised during relief operations – those responding to humanitarian needs, as well as natural disasters. This has clear impacts, most of which are either not recognised or overlooked at the time. Meeting short-term needs can, however, result in longer and protracted involvement of international and national agencies and is invariably more complex and expensive to sort out. Environmental rehabilitation, for example, following the hosting of people displaced by a crisis can be costly and time consuming but much of the damage can be avoided or lessened if certain actions are taken early enough in the response. Several reasons can be identified as to why environmental issues are not systematically included in disaster response. Priority is quite naturally given to securing the basic needs of affected people. The links between the environment and human well-being, however, are often not fully realised and a failure to see the wider picture – as water, sanitation, shelter and food security needs are addressed, all of which have direct links with the environment – can have serious and lasting repercussions for people and the environment. Environmental issues are also often viewed as being the domain of specialists and thus too complicated for the “average” relief worker. In reality, these are not complex issues: common sense together with a little training in or exposure to some of the most critical issues that might need attention are often all that is required. 1.2 A NEED FOR ASSESSMENTS? But why go to the bother of assessing environmental issues when more pressing needs might be apparent? Environmental impact assessments (EIAs) are becoming mandatory in more and more countries, for an increasing number of activities. While once the focus of primarily construction and large-scale development activities, attention is increasingly being given to almost any activity that has the potential of affecting the environment, albeit to very different standards, and for different purposes. The growing need to tackle this requirement head-on has meant than many organisations and donors have developed specific EIAs for their own purposes. At the same time, however, it must be recognised that a “standard” EIA is largely inappropriate for an emergency situation. An EIA’s validity rests in the comprehensive collection and weighing of data and options. This deliberative process is incompatible with the often chaotic and difficult operating conditions encountered in a disaster. Short-cutting the EIA process risks creating an assessment that misses or misrepresents critical considerations. This can lead to focusing on less important environmental problems, and may even result in more harm than if the assessment was not done at all. Thus, there is a clear need – and often obligation – to carry out some form of impact assessment even in an emergency situation. While the need for an assessment is becoming increasingly appreciated, a number of issues remain, including uncertainty as to: • how to do it? • whom to involve? • when to do it? • how to finance it? and 2 • what to do with the results? Clearly a response to these concerns in any emergency/relief setting/operation should take the following into consideration: • it should provide decision-makers in such operations with an analysis and decision-making framework based primarily on saving lives and reducing damage; • it must clearly identify the relevance and implications – direct and indirect – of environmental issues to these objectives; • it must be dynamic and responsive, providing real-time operations input on environment￾related factors; • it should be applicable over a wide range of agro-ecological, geographic and socio-economic situations; and • above all else perhaps, it must be simple and straightforward, imposing the least additional workload on people engaged with the relief operation. An important step towards addressing this gap has come from the patient development of a Rapid Environmental Impact Assessment (REA) tool by CARE USA and the Benfield Hazard Research Centre (BHRC), UK. Designed with these, and other, concerns in mind, the REA’s point of departure was to produce a simple guide to identifying what are or what might emerge to be some of the main environmental concerns during a given, and changing, situation, as relief operations are planned, unfold and implemented. It looks at the short- and longer term needs of communities, as well the physical environment. Information is generated from various sources using a range of tried and tested tools, and is pulled together in a set of standard appraisal forms. Used systematically, its potential is enormous and the outputs can clearly overshadow any doubts or concerns as to why time and resources should be spent dealing with this issue. 1.3 RAPID ENVIRONMENTAL IMPACT ASSESSMENT 1.3.1 Background to this REA project The idea for this project grew from a concern – by the current REA Lead Researcher, Charles Kelly – in West Africa in the late 1980s and early 1990s that locusts and grasshoppers would devastate food crops, leading to famine and a consequent need for large-scale food aid. Standard environmental impact assessment procedures were waived – possibly as they were seen as being cumbersome, expensive, time consuming and required expert guidance and involvement – given the need for urgent action to avoid a disaster. This, however, proved to be an inappropriate, expensive and environmentally damaging option as substantial amounts of pesticides were required year after year. A simple lesson learned was that considering environmental issues at the beginning of the anti-grasshopper campaign would have made for a better overall campaign with less negative environmental impacts. The notion of incorporating environmental issues into disaster operations was first considered in relation to population displacements, by the REA Lead Researcher and other individuals and institutions. At the time, consideration was being given to possible modification of standard EIA procedures to fit disaster conditions, but this was proving too laborious to be manageable in field situations. There followed various presentations of ideas/concepts at conferences, discussions with potentially interested agencies, and continued background work, culminating in the incorporation of an REA project in the BHRC’s programme (a complete chronology of the process is given in Annex 1). 3 Discussions were held with a number of potential partners and donors: it was felt that a partner with experience in either training or disaster operations was important to ensure that the development of the REA was not simply an academic exercise. Such interest was expressed by CARE USA which, at the time, was starting to give more attention to the links between the environment and disasters. Funding from UNEP/OCHA allowed practical work to begin. Following extensive discussion and a field visit to Orissa, India – the latter to collect first hand input on environment-disaster linkages and how to better visualise how an REA process might feasibly be accomplished in a disaster setting – a draft REA process1 was completed in January 2002. From the initial focus on disaster survivor impacts on the environment, the scope and breadth of the REA had expanded considerably. Additions included a section to: • frame the disaster (the “Context Statement”) which also served to focus attention on special environmental considerations (e.g. environmental concerns from before the disaster); • address the potential environmental impacts of disaster events; and • consider the potential negative consequences of relief assistance. Procedurally, each section was designed to use a simple rating table/check list approach to identify and prioritise the issues covered in each topical section of the process. Once issues had been identified and prioritised in each section of the assessment process, they were consolidated and further ranked to generate a prioritised list of issues requiring immediate action. This process (further details of which are given in Section 1.3.4) has remained basically the same during the evolution of the REA and the Guidelines. Throughout this whole process, it is important to note that the main driving force has been the Lead Researcher, with some support and input from what can only be described as a “limited number” of individuals. This perhaps has implications for the future use, application and further development of the REA and is discussed further in Section 4.1. 1.3.2 Goal of Project The intention of this project was to develop an assessment tool for use in emergencies and disasters, based on the following three elements: • a review of current conditions at a disaster to identify victim needs not being met and which can result in negative environmental impacts; • an evaluation of disaster-related factors that can have a direct and immediate impact on the environment; and • an evaluation of the potential negative impacts of external assistance. The goal of this project, as stated in the Project Summary document was to “reduce disaster impact by improving the comprehensiveness of disaster response efforts by including the consideration of environmental impacts in needs assessments, planning and relief operations”. This was to be achieved through work towards two objectives: Objective 1: Establish a rapid environmental impact assessment process for disaster situations. This was to be accomplished by: • transforming an existing REA concept to a formal methodology, through consultations and a comprehensive review of literature and experiences; 1 Guidelines for Rapid Environmental Impact Assessment in Disasters 4 • testing the REA procedures during actual response operations to at least three different types of disasters in different locations; • integrating the REA process and outputs into disaster management procedures; • establishing an advisory group of disaster management and environmental professionals; and • assessing the methodology, procedures, process and test applications of the REA. The expected output from this work was a formal process for completing an environmental impact assessment during disasters, clearly documented with appropriate guidelines, tables and matrices. Objective 2: Assure the adoption of the REA and environmental considerations in disaster response as best practices by international organisations and non-governmental organisations (NGOs). This was to be accomplished through two related actions: a) Developing a training syllabus and course material for the REA, which would involve: • drafting course work and a training plan for teaching the REA. The course would be designed as a stand alone unit, as a module in other training programmes and as a self￾study course; • undertaking an external review of the syllabus/draft course book (which was to have included the advisory group) and test presentation of the final draft materials in two training courses; and • publication of the training and background materials (including access through the World Wide Web). b) Training those involved in responding to disasters in how to conduct a REA by: • conducting two ‘test’ training events to solicit critical comments on the training materials and REA process; and • conducting a formal training on REA for non-governmental organisations. Anticipated outputs from this were: • a training course and manual for environmental impact assessment during disasters; • general access to this material through hard and electronic media and a standard Facilitator and Participants’ manuals; and • a cadre of training disaster management personnel able to carry out the REA and train others in this process. 1.3.3 Implementation Arrangements From the outset, the project has been managed by the Benfield Hazard Research Centre, University College London, primarily through facilities (including administrative assistance), provided to the Lead Researcher. Collaboration with CARE USA was initially ensured through the Senior Environmental Adviser (Mario Pareja) until this post was closed in March 2002. Since then, CARE, through its Emergency and Humanitarian Assistance Unit, has continued to oversee the completion of this phase of work, while the BHRC provided a part-time Consultant (Mario Pareja) to assist with further development of the REA tool, field testing and training. Development of the training materials and the actual test training events was contracted to InterWorks, USA. 5 The initial time required for this project was estimated at 22 months. The first phase began in August 2001 with initial funding from UNEP/OCHA. Additional funding was secured from USAID’s Office of Foreign Disaster Assistance (OFDA), the Royal Norwegian Ministry of Foreign Affairs, and CARE International (see Section 4.2). The current phase of development and funding is scheduled to end 29 February 2004. Throughout the process, informal links have been established with other relief agencies, many UN agencies and environmental NGOs. As part of the management process, an Advisory Group was set up at the beginning of the project: members were expected to “provide a check and balance to developments and provide technical advice regarding the development of the REA”. Starting with four members, at the time this evaluation was carried out, five additional people had joined the Group. Members were invited on an ad hoc, personal choice basis, the resulting Board representing a considerable body of knowledge and expertise, from many different angles and a broad geographical representation. Among these members was one representative from CARE and three from agencies partially funding the initiative – UNEP/OCHA and OFDA/USAID. 1.3.4 REA Development Process The REA process is described in a 109 page document entitled “Guidelines for Rapid Environmental Impact Assessment in Disasters”. This has gone through many revisions with input provided from a number of individuals and institutions. The Guidelines have now been tested in three situations – Afghanistan, Ethiopia and Indonesia, with practical on-site community application being carried out in the two latter countries – used in the preparation and delivery of training sessions (Norway, India and Guatemala), and the preparation of training materials. A series of PowerPoint presentations have also been developed around these materials. During these activities, feedback was continuously being provided to the further refinement and elaboration of the REA Guidelines. The most common changes noted were in relation to: • wording and the language used in the assessment forms and training materials; • introduction of and subsequent refinement to the community level assessment; and • identification of the relevance and usefulness of the green procurement processes. Various other presentations of the Guidelines have been made in the past two years, feedback from which has also been considered in the revision of the REA Guidelines. In January 2004, a Quick Guide to Rapid Environmental Impact Assessment in Disasters (pp40) was produced, mainly in response to comments on the bulk of the REA Guidelines. 6 2. EVALUATION OF THE RAPID ENVIRONMENTAL IMPACT ASSESSMENT 2.1 PURPOSE With funding from USAID/OFDA, this evaluation was requested by CARE USA to undertake the following: • document actual outcomes against performance measurement criteria stated in the relevant project and sub-project documents; • assess the effectiveness of the REA process as a “best practice” tool in disaster management; • consolidate and summarise perceptions of project participants and interested parties about the REA methodology and training materials, and • identify successes and improvements in the project implementation process. Findings from this evaluation are expected to influence the future direction of the REA tool – its structure, content and application – if a second phase of this project finds support. 2.2 APPROACH TAKEN AND METHODOLOGY The evaluation was carried out on a part time basis from 17 January to 10 March 2004. In all, 15 days were allocated for this evaluation. Three methods were applied in the evaluation, in keeping with the Terms of Reference (Annex III): • a review of project documents available on the project web site2 in addition to others provided by CARE, the BHRC, InterWorks and the Lead Researcher in particular. These documents included background information on the project, the project proposal to donors, reports on the REA field tests and training events, and training materials; • feedback via specific questionnaires from individuals who had participated in some stage of the REA development; and • direct contact – meetings or phone calls – with selected individuals who again had direct links with the development or implementation of the REA during this period. For both of the latter points, a list of contact names and phone and/or e-mail addresses was provided by CARE and BHRC. The evaluation was therefore primarily desk-based using materials accessed from the project’s web site or requested from one of the individuals or agencies working on this initiative. Review of the contact list suggested five broad categories of people identified according to their broad experience with the project. These categories were those involved primarily in: • project design and implementation; • training; • trainees; • Advisory Group members; and • other contacts who had been involved with the project. While the majority of such cases were straightforward – someone had participated only as a trainee or was on the Advisory Group, for example – several people (at least on paper) fulfilled 2 www.benfieldhrc.org/SiteRoot/disaster_studies/rea/rea_index.htm 7 several roles. For the latter, the evaluator’s own judgement was used as to which group the person was placed, and which questionnaire s/he was sent. Simple questionnaires, each having around 12 questions, were then sent to a total of 77 individuals (Annex II). A separate questionnaire in Spanish was prepared and sent to all trainees from the Antigua workshop. The same gesture should have been made for the participants in the Kalimantan (Indonesian) field test but it was not until late in the course of the evaluation that the difficulties experienced with language and interpretation of some of the REA elements became apparent, by which time this was too late to take appropriate action. A response was requested from those contacted within two weeks. One reminder note was sent. Of the 86 people contacted in total, 20 written responses were received, while 7 direct interviews were carried out (31 per cent overall return rate). Selected quotes from this process are used in the following report to highlight particular opinions or general comments/ impressions on the project. For the sake of anonymity, however, such quotes have deliberately not been attributed to a particular person or persons. Apart from the poor response to the questionnaires – which is to some degree expected, particularly in this case where it was later learned that language was a major obstacle to both field tests and training events – this approach is thought to have worked well. A clearer orientation through the project would, however, have been useful at the outset of the consultancy, in particular direct access to the most significant materials relevant to this evaluation. Time was lost in retracing steps as later drafts of key documents emerged on a different web site. This was compounded by the actual volume of paper to be analysed, much of which was later found be relevant to the evaluation. A draft final report was circulated for comment to 25 individuals on 27 February. By the requested cut off date, just three replies had been received so a decision was taken by CARE USA to extend the deadline by another week, to 9 March. Contacts were duly notified, but this action only resulted in two additional sets of comments being received. Difficulties were also encountered with accessing and downloading information from the main web site, cited above, a point which should be kept in mind if this is to serve as one of the main means of dissemination of the REA tool and training materials. Had more time been available it would have been expedient to have somehow reached a larger number of people with direct experience of a field test or training event – or even to have participated as an evaluator at one of these. Nonetheless, it is felt that the following account and recommendations captures the broad essence of peoples’ opinions and experiences with relevant aspects of this project. 8 3. MAIN FINDINGS I: PROCESS AND OUTPUTS 3.1 THE REA PROCESS AND GUIDELINES 3.1.1 The REA Process The basic output from this project has been a tool to conduct a rapid environmental assessment through a determined process, as described in the REA Guidelines (Section 3.1.2). While the remit of this Evaluation was not to investigate the technical rigour, integrity or the ease of applying the REA process, it was unavoidable that some degree of investigation had to be carried out in order to understand the working within the process and links between the various activities such as training (manuals and events) and field tests. Further comments from one reviewer provided more substantial detail on this aspect, which is included in this particular section. A clear interest exists at the individual and institutional levels in using the REA as a tool. Those who have been exposed to the tool have, in general, appreciated it for the needs and information it revealed. With few exceptions, however, it is doubtful whether any of those trained in its use could at this stage undertake or lead a rapid environmental assessment on their own, without qualified external assistance. This relates to one of four main points on the REA as a tool – its complexity. Many people contacted during this evaluation referred to the “complex” and/or “obtuse” nature of the process – comments with which this evaluation fully concurs. Multiple readings of the guidelines are necessary to get a clear understanding of the mechanical steps associated with the approach and, even then, it would probably not be clear to a first time user how s/he might proceed. The risk is therefore that people will not be eager to carry out an environmental assessment or that, if they do, it will not be carried out correctly. A second, and fundamental, critique applies to the integrity of the current REA tool. From a design basis, the approach lacks conceptual clarity. This is compounded by the use of a series of rating scales and rating forms as the main means of qualitative data collection and the subsequent processing of these data – some of which may not be mathematically correct. Many of the questions posed are also multi-faceted which could elicit differing answers depending on the respondent’s interpretation. This only adds to possible errors in the data collection process, overlaying a possibly flawed design. A third point of concern addresses the concept that the REA is a rapid undertaking. From the experience of three field tests, however, it is clearly demanding on a few people’s presence and input, while also requiring consultation with a much broader group of people. While situations differ considerably, it must be assumed that a tool requiring low level input from personnel involved in relief operations will be more appreciated and thus perhaps more widely used that one which places clear time demands on these individuals. A final point relates to the seemingly independent nature of the REA tool at present. Given the difficulties experienced in implementing the recommendations stemming from the field tests (see Section 3.2), it might be appropriate to also try and integrate this REA with other assessment tools, so that environmental issues are treated as cross-cutting or interlinked requirements rather than being seen as stand alone concerns. Reference is made to this later in the report through awareness raising and institutionalisation of the REA process as well as the development of a specific small REA training module for inclusion in the training programmes of other agencies. 9 While the REA concept appears to be well appreciated, there appears therefore a need for some backtracking and consolidation of the current tool before other activities are advanced. Technical input or guidance to the actual concept and structure of the REA process appears to have been limited thus far, but this is the time to try and improve the integrity and rigour of this tool as well as the Guidelines which describe its application (see below). 3.1.2 The REA Guidelines The REA Guidelines have undergone repeated and extensive revision since first elaborated in 2002 (see Section 1.3.1 for a quick overview and Annex I for a detailed chronology). The Guidelines actually outline the path to follow when preparing for and conducting the REA (as referred to in Section 3.1.1) but are treated as a separate issue here as some of the issues need to be addressed on their own. In reality though, both are of course inextricably linked. Funding from UNEP/OCHA for the period August-December 2001 (later extended to the end of January 2002) enabled a first draft of the Guidelines to be completed. Funds were also obtained from the Royal Norwegian Ministry of Foreign Affairs, through CARE Norway, specifically for one training workshop. No time line was attached to the latter: this component of the project was to have been completed by December 2002. Additional funds were later secured from OFDA for an 18-month period to enable some field testing, the development of training materials and two training workshops to be carried out. With the exception of funds of course, donor input to the project thus far has been relatively low, the exception perhaps being OFDA which provided quite detailed feedback to the original proposal, requesting certain changes be made to the REA tool and Guideline. Table 1 shows how concerns expressed by OFDA in April 2002 have seemingly been taken into account. Table 1. Concerns Expressed to CARE USA by OFDA (Letter to the Senior Adviser, EHAU on 9 April 2002) ISSUES TO ADDRESS RESPONSE TAKEN 1. “The indicators listed are all process indicators. Please add several appropriate impact indicators that will help track the success of the programme. For example: the number of environmental problems identified in a given assessment that are addressed by the humanitarian community as a result of the REA. An additional monitoring/ evaluation component may be necessary to determine such impact.” Two specific indicators were elaborated (see Table 3) but proved to be difficult to meet. 2. “…what is the protocol for follow-up once problems have been identified? How will REA findings be communicated to the rest of the humanitarian community? How will the findings be linked to specific sector programmes?” During the three field tests, attempts were made to integrate at least some of the key recommendations into existing projects/ programmes or new project proposals. This has not been entirely successful but relates more to an institutional issue than being any fault of the REA process per se. 10 ISSUES TO ADDRESS RESPONSE TAKEN (contd) CARE Ethiopia is developing local versions of the training modules for partners; REA principles have also been integrated into university curricula for environmental students in Honduras. All outputs from the process to date have been made freely available on a dedicated web site at the BHRC. 3. “…While OFDA recognises that the questions [in the assessment sheets] need to be somewhat general, the units for “Range” in Rating Form Number One, for example “high” and “low” and “short” and “long” need to be quantified for the assessor. Otherwise results cannot be compared over time, or even from assessment to assessment with the same tester. ….. The forms should be designed to be readily understood on their own, and without having to refer constantly to the guidelines.” The scales and rating forms have been a source of much debate, part of which is still ongoing. Experience from the field tests and training workshops show this to be an area where much explanation and discussion is needed: the Trainer’s Guide, for example, encourages those preparing to lead the REA process to “be ready to discuss the metrics you shall use…”, “be ready to define Small, Medium and Large..”, etc. Many improvements have, however, been made to the rating forms in line with feedback from participants: details of these are not given here but can be viewed in the original reports. The Guidelines, in a section on “A Note on Rating Metrics” suggests that “the rating methods or scales can be changed to reflect local preferences"” which is probably preferable, but it is questioned as to whether REA leaders will have the experience necessary to complete this correctly. A second, remaining, concern relates to differences in opinion or interpretation if a second group conducts a later REA, since they might have different viewpoints or experience. This concern is pointed out in the Guidelines and is perhaps unavoidable. 4. “OFDA suggests that the guidelines associated with the assessment, once finalised, be transformed at some point into field guide format for easier transport and use.” A Quick Guide to Rapid Environmental Impact Assessment in Disasters (40 pages) was produced in January 2004. It is intended mainly for people who have already undergone some formal training in the REA workshops or who have participated in a field test. It is not intended for use by a generalist without training or experience in environmental matters or disaster response/management. The OFDA and UNEP/OCHA were also on the Advisory Group (see Section 4.3) and provided input to the early reformulation of the REA Guidelines. Much of the change to the Guidelines, in 11 fact, has come as a result of direct input from a few Advisory Group members, and especially following direct experience and additional feedback from field tests and training workshops. 3.2 FIELD TESTS The REA Guidelines were tested in three different countries during 2002 and 2003 (Box 1), each taking place over a period of approximately one month. In each case, the national CARE office served as the focal point, with some CARE staff being involved in the process. Conditions in the three countries varied, but all signs indicate that the operating environment in which the REA was undertaken was suitable for testing this process and its evolving outputs. These conditions included: 1. In Afghanistan: • a country emerging from 23 years of warfare and experiencing continued security problems, was experiencing its fifth year of drought, which was contributing to severe water problems and poor food security and health conditions; • local and external sources of standardised environmental conditions were limited; • communications – local and international – were poor and haphazard; • CARE was operating five major programmes, so time pressure existed; and • pressure existed to develop or initiate new projects with minimal design input and lead time. 2. In Ethiopia: • the Awash-based test site lacked the institutional capacity for support or implementation which was present at other CARE-assisted projects in Ethiopia; • the general situation changed mid-way through the REA process as senior and disaster management staff began to focus on a rapid degradation in food security in east and west Hararghe; and • the lack of water for public consumption emerged as a serious problem in some areas. 3. In Indonesia: • prevailing conditions related to problems caused by annual forest fires, haze and, in some parts of the province, drought and flooding; Box 1. Field Tests Undertaken Afghanistan 14 February – 16 March 2002: an assessment was carried out with 10 CARE staff completing the REA on the basis of information available within this group, supplemented by the limited information otherwise available. No practical field activities were undertaken. Ethiopia 14 August – 13 September 2002: this test involved the CARE country office in Addis Ababa, the CARE project office in Awash and four communities in Fentale and Awash Fentale districts, Oromiya and Afar regions. Indonesia 8 January – 1 February 2003: 5 CARE and 3 NGO (Yayasan Cakrawala Indonesia) staff were involved in the initial group assessment in Central Kalimantan, with the involvement of 12 communities. 12 • CARE was in the process of opening a new office to support two new projects, both intended to address the consequences of recent disasters (fire and conflict) and significant hazards (fire and haze). Information from the REA was intended to feed into project implementation plans and options for future programmes; • CARE staff were unfamiliar with the test environment; • none of the participants had extensive experience in emergency operations: all CARE team members had other duties and several had to handle non-related tasks as well as participate in the REA; • only the Team Leader was fluent in English; and • access to senior management and officials was generally possible. That such conditions and differences existed at the time of field testing is important since this exposes the REA tool and its users to a range of valid conditions and circumstances, giving further credibility to the outcomes of each field test assignment and the process overall. A final point which should be noted was that there was no in-house CARE environmental advisor or officer responsible for this concern in any of the three countries. As can be seen, however, the conditions in each situation varied but the above shows that the REA was being tested in many of the conditions likely to be experienced in emergencies and disasters. The test process was generally similar in all three situations – but particularly in the case of Ethiopia and Indonesia as only limited field visits were made in Afghanistan – with some minor adjustments occurring as experience was gained. The last test in Indonesia comprised of eight stages, summarised as follows: 1. discussions at CARE Head Office level on the purpose of the test; 2. preparation for field test and discussion with team leader; 3. group assessment with CARE, counterpart personnel, local NGOs and government representatives; 4. preparation for community assessment; 5. test application of community questionnaires, followed by data collection: 6. data analysis; 7. presentation of findings and discussions; and 8. preparation and review of field test report. Rather than being discrete steps, it appears, again increasingly as experience was gained, that these stages flowed more seamlessly from one to another. 3.2.1 Comparison of Main Findings from the Three Field Tests Before starting, each field test set a series of questions linked to the test results in an attempt to identify difficulties, identify positive outcomes and generally revise the REA Guidelines as needed. Although some additional questions were asked in the second and third tests, enough similarities exist to allow some observations to be drawn, as presented below. These comments are based on feedback expressed in the reports from the field tests and may be subject to some degree of misinterpretation as these comments have been compiled and condensed by a third party. However, information obtained directly from feedback invited through this evaluation will, in general, have helped balance any such misinterpretations. 13 1. Are the Guidelines sufficiently detailed to accurately identify critical environmental issues during a disaster operation? Results show that the large numbers of environmental issues were identified through the group assessment process – sometimes so many it seemed like a shopping list: 15 in Afghanistan, 34 related to drought alone in Ethiopia, and 19 in Indonesia. The clear omission of the Koli Hashnat Xan wetland south of Kabul – an area of high social and environmental value which could be impacted by CARE activities – from the first test assignment suggests that more care might be needed at this fundamental stage of the assessment. This is supported by the tendency that group assessments may focus on less than critical issues, especially noticeable in Ethiopia. One way around this might be to revisit this issue throughout the REA process. Distinctions have also emerged regarding issues identified in the group and community assessments, the latter generally appearing more complete. This point may, however, relate to the amount of information available to the respective participants as this might have been an obstacle to group assessment participants. In this respect, it is not the Guidelines which are “at fault”, but identifying or accessing suitable information. It might, however, be helpful if the Guidelines were to include a list of what type of information is typically likely to be the most useful and cite some possible sources for this? Many changes have taken place to the basic REA Guidelines since they were first circulated and overall it would appear as though the level of detail in the Guidelines is adequate – perhaps too much at times. Many people spoken to during this evaluation commented favourably on the aspect of revisions, as viewpoints, opinions and concerns were largely – and efficiently – taken into account. The latest text (Version 4.2, December 2003) includes many additional refinements. This, together with the new 40-page “Quick Guide” address many of the concerns raised in later tests. 2. Is the Guidelines document an appropriate assessment tool for the time compressed, information limited, high workload demand environment found in disaster situations? It is unlikely that the Guidelines will find application in a disaster situation unless there is someone skilled in their use or, as a minimum, if someone who has attended a training event and is knowledgeable about part of the technique and its application, is available to drive the process. At present though, there is too much detail required which, in turn, takes time and commitment. All three tests exceeded the amount of time estimated for initial preparation: initially it was thought that two hours preparation and two hours consolidation would be sufficient. Observations showed that the time required ranged from half a day to as much as two days, depending on how well participants had prepared for the task, and the prior availability of materials in an appropriate language. While conditions will obviously vary from one situation to another, following the instructions in the Guidelines takes time and the process cannot be rushed. The third field test in Indonesia, for example, required 14 days of field work for completion, although 10 days were dedicated to collecting community input. While this test experienced particular difficulties with language and the level of prior preparation, these constraints must be borne in mind as it is not feasible to have the Guidelines adapted to every possible situation or culture. Experience in managing the whole process is therefore clearly indispensable as summarised in the following comments from one 14 field test: “The REA tools and Guidelines may lead into lots of details and complex issues, if the participants are prepared to deal with them and so desire. It is a matter of deciding where to draw the line, when to stop”. The Guidelines seemed to work well in the group settings. Having to discuss formerly unfamiliar terms within the group or community also served to help people reach a better level of understanding of the purpose of the activity, but this was at the expense of time. Accessing relevant information did not seem to hinder the process of using the Guidelines, but the point previously made concerning identification of potential environmental issues which might be impacted by relief operations needs to kept in mind. This process is, however, aided by having the combination of an “internal” group assessment and experiences from the community somehow feeding into the process. Two major limitations which should therefore be borne in mind are the time required and management of the process, the latter which should include follow-up on findings and recommendations. This issue is well summarised in the observations of one of the two consultants overseeing this test, with the comment that the REA may “face two different extreme situations”. With a rapid onset disaster the REA should be completed within at least four days. In this situation, a strong REA manager will be needed to head the assessment, someone who can manage a process in the context of a disaster, who has a good command of English and the local language(s), a certain level of understanding of the local conditions and familiarity with the REA process. The other extreme is that of a slow onset or protracted disaster in which a more participatory REA can be implemented over a longer period of time – 5-10 days. A REA leader in this scenario could be a local mid-level staff with skills in participatory rural appraisal (or similar social techniques) and with certain minimum management skills for processes and personnel. Similar concerns were voiced by others: “…the REA itself is too time consuming which limits its usefulness during an emergency because it means that when the REA results are presented to programmers (after two weeks), most proposals are written and submitted. …no time is left to make use of the recommendations while they are easily outdated due to often rapid changes during emergencies”. While the Guidelines may contain good information and guidance, much will clearly depend on the REA leader and his/her ability to organise the group and steer it on the right path. For this reason, some improvements are therefore seen to be still necessary to the existing materials and approaches (see Section 5). 3. Were the Guidelines outputs integrated into relief and recovery planning and operations and did they have any discernible or perceived positive impact on disaster assistance operations? It has proven difficult to determine whether the action items identified through the three field tests have been addressed, e.g. through inclusion in or revision of normal planning and operations, or the results used. In Afghanistan, no specific changes were made to programmes, projects or activities, but Senior CARE Afghanistan staff indicated that some assessment findings would have an impact on current or future projects – generally to make them more environmentally appropriate and sustainable. In Ethiopia the findings were used to design a relief project proposal but this has never been submitted for funding, while in Indonesia, changes had been anticipated in the composition and approach of two new CARE projects, but this likewise did not take place. 15 The level of interest expressed by other agencies contacted during the field tests has been generally positive. Many have indicated their desire to use the REA Guidelines as a tool for screening and ongoing activities. To date, however, there has been no indication that any such agency has tried to use the Guideline. A fundamental question to ask at this stage of the process relates to the future application of this REA tool, particularly why should an individual or agency use it? Similar type assessments or evaluations are carried out on occasion as a face-saving strategy: there is never any intention to use the results, but it is deemed sufficient to have followed the process. As one of the consultants guiding the Indonesian field test observes “in order for the REA to be consistently used, the system (NGO structures, donors, etc) has to provide some rewards for the staff that use it”. There is little benefit in individuals going though this process if their organisation does nothing with the outputs. Certain elements of the REA Guidelines have already been incorporated into the Sphere Handbook, OFDA’s standard assessment training and UNEP/OCHA’s emergency management training, which is already a milestone achievement for this project. A welcomed next step, however, would be that CARE and other relief agencies and humanitarian agencies make it a policy requirement that a REA is routinely conducted in all operations. Feedback from CARE USA suggests that this is unlikely given the decentralised nature of the organisation and the fact that this tool will have more relevance and appeal to some but not all offices. Several respondents made comments to the effect that they would see their “country office benefiting from this initiative when we have more staff familiarised and skilled in how to carry out the assessment, and including it into our regular assessment toolkit”. An appropriate strategy might thus be to make offices aware of this tool and its potential and to continue to update them on developments and achievements through, perhaps, a simple project newsletter, if there is a second phase of work. 4. Could the Guidelines be used by local staff who do not have extensive environmental or disaster management backgrounds? The greatest difficulty encountered with the Guidelines by inexperienced local staff relates to language. Certain unfamiliar terms needed explanation, posing not only linguistic but conceptual challenges. In addition, non-English speaking participants experienced particular difficulties in following discussions (similar observation were naturally noted regarding training materials in other situations, see below). More careful guidance would appear necessary in this respect if the Guidelines are intended to be used as a stand alone tool, i.e. where no experience exists or external assistance cannot be provided. A much higher level of support to participants must be expected where language difficulties are likely to occur. Feedback from workshop trainees indicated that a few would feel confident in participating in an REA application as a co-leader, as long as an experienced assistant was on hand. For the Guidelines to find broader use by local staff who lack environmental or disaster management exposure, however, it is essential that much clearer guidance is provided on certain issues and that the manual(s) are made more user friendly. Among the recommended changes (see also Section 5) would be: • clearer introductions to some of the issues; • a more user friendly glossary of main terms and concepts; 16 • clearer guidance on how to prepare for a REA (perhaps similar to that in the training materials); • more helpful tips on group organisation and management (which currently features as Annex I in the Guidelines); • guidance on how to adapt the rating scales (or similar) to local needs; • guidance on formulating or adapting appropriate case study scenarios; • help with formulating raw assessment results into clear actions and objectives; and • guidance on how to apply and institutionalise the outputs. A concern with providing any additional information is clearly that of bulk and the extra reading this will require. One way to perhaps address this would be to repackage the Guidelines into a small number of “User Guides” each one having a different, but mutually supporting, purpose, e.g. “Background Information to REA”, “Steps to Follow when Conducting a REA” and “Reference Material and Technical Details”. Such a collection, while possibly adding to the overall bulk of paper would streamline documentation and allow intending users to refer to specific themes or steps and not the whole bulky document. These handbooks would naturally be linked with the training materials. 5. Could the Guidelines be used at the community level? As with the previous question, language, again, is one of the first barriers to overcome in enabling the Guidelines to work at the community level. In Ethiopia and Indonesia, the community questionnaire needed to be translated but it emerged that in so doing the clarification provided by participants helped overall understanding of the issues and questions, as well as the purpose of the REA. This was, however, at the expense of time. Two other points of particular interest should be noted. First, the personal assessment made by the consultants managing the REA in Indonesia was that the community assessment went far smoother than the group assessment work, possibly on account of language difficulties experienced among the latter, but also perhaps because the community work was more straightforward through the use of a questionnaire. At the same time, however, it is important to note that USAID and CARE management involved in the Indonesia test indicated a strong inclination to not conduct community level REA assessments in future, but to collect information for use in the REA process through other disaster impact tools. If this is to be adopted in future, then it is important that the comment made under Question 1 above regarding the different quality of information emerging from the group and community assessment be taken into account. 3.3 LESSONS LEARNED Each of the three field tests contains a set of lessons learned from that particular experience. An attempt to synthesise these into a small number of topics is presented below to demonstrate some of the – mainly positive – benefits that an organisation might expect from applying this tool. It should also be noted, however, that each of these tests carried specific recommendations on how to improve various aspects of the REA process and Guidelines, particularly in relation to terminology used, the rating schemes, prioritisation of issues, and the need for standard formatting. With a few exceptions, these are not described in the following as, to the extent where it can be determined, they have already been taken into account in the latest version of the REA Guidelines. 17 3.3.1 Preparing to Use the REA Guidelines Good preparation is essential for all field tests and training events. This ranges from pre￾event logistics (per diem and venue arrangements) to translation and provision of documents, and agreed co-ordination by the event leaders. Event leaders must themselves be fully versant with the materials/modules they are responsible for delivering. The REA Guidelines should be read and understood by all those undertaking a rapid environmental impact assessment. The REA Guidelines prove easier to use if adapted to the country/local conditions: There is a clear benefit to having the Guidelines at least partly adapted to country/local/situation specifics as participants in the process instantly feel more comfortable with the examples being discussed. Undertaking this, however, needs to be balanced against other needs and time constraints as it would be impossible to prepare such generic materials for use in all situations. Where such modifications are made, however, awareness should be raised of their existence so that they might find wider application. Language and language skills need particular attention: Clear communications are a prerequisite for a good and lasting understanding of the REA process. Having all relevant materials available in the local language(s), as well as competent facilitators/presenters is essential. Otherwise, time will be lost in the group sessions with translating or explaining terms, phrases or approaches. Lack of information on ways to address environmental issues hamper the decision making process and may flaw recommendations: reliable information may not always be at hand in an emergency. Common problems might include: poor communications, scarcity of reference materials on environmental issues, and difficulties in identifying appropriate experts on environmental issues in that particular context. An independent (REA-based) collection of data at the community level may not be necessary if other disaster impact assessments have already been conducted: data on environmental issues might have been compiled in the course of other disaster assessments, e.g. water and sanitation, nutrition or food security. If so, there should be no need to repeat the exercise but to simply access and assess the information. Input from the community level is, however, essential as this can be more informative and practical than the views and needs identified by people not as familiar with the local situation. The REA Team must therefore, at an early stage, decide whether they will devote time to this directly or try to glean information from other sources, accepting perhaps that the latter decision may not produce the breadth and scope of input from the community level questionnaire. The assessment process can serve a double role in terms of assessment and education: discussions during the formal assessment sessions and follow-up exchanges can help raise questions about the environment which have not previously been considered. New ways might then be found to address existing and new environmental concerns. Sensitivity needs to be shown to gender-related issues throughout the whole process: although groups are expected to include representation for both women and men, attention can easily focus more on one than the other. REA facilitators in particular should take note and ensure that particular gender focused activities are considered at the appropriate time and in the correct manner. This prevents the need to draw up a specific checklist of gender-specific environmental issues which is unlikely to find widespread application across cultures and social systems. 18 3.3.2 Applying the REA Guidelines Use of the Guidelines can help identify critical environmental issues: field tests in Afghanistan identified 15 critical environmental issues, most of which related directly to CARE’s ongoing project/programme activities in the country. Results from Ethiopia identified significant environmental issues which could be addressed through relief and development activities. Although some of these issues were already known, application of the REA enabled users to identify and agree on ways to address these. Information collected and issues identified during the assessment provide useful input into formulating relief project proposals, but are not the only inputs required: health, nutrition, shelter, water and sanitation needs must also be taken into account when considering the possible and most appropriate responses in disaster situations. Exclusion of one could lead to unnecessary impacts in others. A mechanism should be established early on how links can be assured between the various sectors. The REA Guidelines can produce useable results without extensive training or support: the Central Kalimantan field test, for example, was led and conducted by persons with little environmental or disaster management experience, who received minimal training or support. According to CARE Indonesia, the assessment provided useful input into ongoing project planning and management. For best results, however, it would be preferred if at least one person who has previously carried out a REA, or has had training experience in this, was also present. A REA assessment can take a considerable amount of time for a tool intended to be used in an emergency: few situations are likely to be the same but experience shows that the time needed to complete the assessment will depend on the size and complexity of the programme and disasters/crises involved. Completion of a REA will take at least eight working days – unless completed concurrently with other assessments – excluding initial data collection, travel and logistical arrangements. But it is not an overtly expensive undertaking: best estimates from the REA field tests in Indonesia suggest that the REA can be competed for less than US$8,000. Costs will increase if an international expert/consultant is required. Conducting a REA at different organisational levels – Head Office, Field Office, community – can help harmonise views as to disaster impacts and response needs and priorities: focusing on different levels links community participation with decision-making on relief and rehabilitation issues, and allows community input to the selection of ways to respond to the disaster. Validation of assessment results is important and can provide additional insight into environmental and emergency conditions: the views expressed by group members may not be the same as those of communities. Any such disagreement, when discussed with the relevant parties can help overall understanding of the process, lead to a better appreciation of the outcome of the assessment and increase buy-in to the next steps. The Guidelines-based assessment process is more difficult to accomplish for multi-sector and geographically diverse assistance operations than for a geographically limited and highly focused activity: it may be better to address projects/programmes individually after an 19 initial assessment meeting and use group meetings to validate issues and propose actions from a broader perspective than each project/programme. 3.3.3 Managing the Process Local institutional responsibility is required for the REA process to be managed and results used: a position dedicated to at least part time environmental assessment and monitoring would have a positive impact on disaster responses. Development projects, for example, if routinely screened for environmental impact, are likely to be more appropriate and contribute to people’s livelihoods. At present this is done on a hit-or-miss basis. Effective follow-up to recommendations is also difficult to assure if no focal point is made available. Participation in the REA process will detract staff from other disaster response activities: individuals participating in the assessment phases will need to invest around eight hours of their time in the process. An alternative – having a consultant carry out the assessment – might provide more comprehensive results but the buy-in is unlikely to be as strong and the individual staff time still required to work with the consultant would likely equal or exceed the time spent in group sessions and follow-up. Sharing lessons and experiences is an important part of this process: many useful lessons and experiences have emerged from the development of this REA. Other agencies have started to express interest in using this for their own purposed of relief planning but, as far as is known, this has not yet been adopted. There are, however, clear benefits from other agencies using this tool. 3.3.4 Applying the Results Translating the issues identified into action can be difficult where an assistance programme is already well established: the ideal use of the REA process is in the relief phase of a disaster and with activities being implemented without an established framework or management infrastructure. Making changes to existing structures to address environmental issues can be difficult because of a natural resistance to change the “established” way of doing things and/or a lack of capacity or funding to make the changes. Further guidance is needed in the Guidelines on how to use the assessment results: results from the assessments fall into two categories – issues which can be addressed by direct action (e.g. environmental degradation) or conceptual issues such as environmental resilience or sustainable resource use. Linking actions to the conceptual issues was found to be difficult: more guidance and consideration needs to be given to how these issues can be addressed in more practical terms. 3.4 TRAINING MATERIALS 3.4.1 Background The task of formulating a training syllabus and course development for the REA was assigned to InterWorks, Madison, USA. Three tasks were identified to: • develop a training syllabus and course material for the REA (see Section 3.4.2); • hold three training sessions on how to conduct an REA (see Section 3.4.3); and 20 • develop a eLearning module.3 In the brief to undertake this work, it is stated that “The trainee level of knowledge at the start of a training course is expected to include a good understanding of development and a general awareness of environmental issues. Trainees are expected to have only a minimal knowledge of disaster management concepts and procedures and no specific technical knowledge in environmental management or relief assistance skills areas”. Two products were prepared, based primarily on the REA Guidelines – a Trainer’s Guide and a Participant’s Workbook. 3.4.2 Training Guides The 100-page Trainer’s Guide is presented in a clear manner with a simple structure and presentation. Three different workshop formats can be designed on the basis of the contents, depending on the training needs assessment of the audience or the objectives of the training event. These range from a three-day agenda for individuals with little or no experience in disaster management, environmental management and/or disaster assessment, to a 1.5-day workshop for those only interested in the organisational level methodology. A number of useful practical tips are presented up front for REA Leaders – those who will be responsible for applying the REA Guidelines – much of which appears to be based on feedback from training sessions, given the level of detail. Individual sessions are clearly designed and well organised to guide prospective trainers through the various steps. An adequate number of case studies are included, of varying situations, although Leaders are invited to prepare their own examples based on the local situation. The Participant’s Handbook is likewise well developed, with a clear open format. Some independent comments on how this might be improved are to: • provide an introduction, with details of who at CARE/BHRC might be contacted for further information or assistance; • introduce the REA process diagram or, if layout permits, have this on a fold-out sheet so that participant’s have constant access to this as a navigation tool for the whole process; • redevelop and greatly simplify the “Key Terms” which has abundant technical terms and jargon: even some native English speakers will have difficulty with this as it stands; and • provide expanded and perhaps clearer case studies, if possible, on some key criteria that participants, or potential trainers, might use when elaborating local scenarios. In general, it would appear that those who have seen and used the training materials have been very satisfied with them. Two major concerns remain, however, as to whether these will now be used outside of the training workshops and whether those who have already participated in the training events will put their new found experience to practical application. Feedback from people who have seen and used the training materials has been quite positive. One observer notes that “it is a very well thought out and documented approach… I do feel that it will be a good guide, but will need adaptation in accordance with local sensitivities of the place of application”. This point was raised by several people and cannot be over emphasised. Future 3 This aspect was not examined as the module was undergoing final development and testing at the time of this evaluation. 21 revisions of the training materials should, perhaps, consider how guidance might be provided for this purpose? 3.4.3 Training Workshops Three training workshops were staged in relation to this project, two (Norway and India) of which had a national focus, while the Antiguan event drew participants from the Central American region4 . The timing of these events was as follows: • Oslo, Norway 8-10 April 2003 – chosen at the request of the Ministry to train Norwegian NGOs that would be operational in developing countries; • Antigua, Guatemala 23-25 April 2003 – selected as a venue on account of CARE preparing a Disaster Preparation Plan, which offered a good opportunity for a regional training event; and • Bhubaneswar, India 12-14 November 2003 – a practical example of where there had been a recent disaster. While the approaches to each workshop differed slightly and were presented by different people, the purpose remained more or less consistent – at the end of the workshop, participants were expected to be able to: • describe the purpose and rationale of the REA; • describe how disasters and the environment are interconnected; • be able to implement all four modules of an REA in an emergency situation; and • be able to make recommendations on disaster response programming that takes the REA results into consideration. From the workshop reports it appears that at two of the events the lack of timely co-ordination and communication led to confusion, especially regarding the participants and their language proficiencies, as well as to misunderstandings, delay and duplication of efforts regarding the translations. Improvements would appear necessary regarding administration, planning and co￾ordination of these events in future. Comparison of the evaluation scores from participants at the three training events show that their appreciation of the sessions was generally high: from 3.5-4.5 in Norway, 4-4.4 in Guatemala and 3.9-4.4 in India (range was 1 low to 5 high). The sessions on “Unmet Basic Needs” and “Participant’s Experiences” were the most highly appreciated in Norway, with less enthusiasm been shown for “Green Procurement”. Similar observations were that “Environmental Threats of Disasters” and “REA implementation issues” were the most appreciated in Guatemala, and that on “Community Level Assessment” the least. In India, sessions on “Disaster Management Context” and “REA Implementation Issues” both got the lowest scores, while at the opposite end of the scale, high appreciation was shown for “REA Conceptual Framework”, “Unmet Basic Needs” and “ Green Review and Relief Procurement”. In terms of relevance of the sessions to the individual’s work, apart from the welcome and objectives, on average those participating in the Norway workshop found “Participant’s Experiences” to be the least relevant and the “REA Implementation Issues” the most. IN contrast, participants at the India workshop found the session on “REA Implementation Issues” to be the least relevant, and “The Environment-Disaster Connection” to be the most useful. Similar 4 An additional half-day training was organised on 12 December 2003 in conjunction with a regional disaster preparedness training workshop organised by the Lutheran World Federation at Konark (Orissa), India. 22 reflections from Guatemala showed that the “Context Statement” and “Unmet Basic Needs” sessions were most appreciated, and “Participant Experience” again the least. Different evaluation forms were used in the different events which unfortunately limits analysis of the workshops to some extent. In Norway, however, two participants rated the workshop as “Excellent”, with a further seven opting for the “Very Good” choice. In Guatemala, two participants again rated the workshop as “Excellent”, nine as “Very Good” and four as “Good”. When asked “If I feel adequately prepared to conduct an REA”, the average score was 4.1 (range is 1 Strongly Disagree to 5 Strongly Agree). Interestingly though the two lowest scores were 3.9 and 3.8 to questions “The programme was relevant to my job” and “The programme met my individual needs”, respectively. Of the small number of trainees responding to questionnaires, most expressed doubt as to whether they would able to use the tool – “I would not be ready to use the tool in a situation where it might be useful due to my general lack of experience in environment/emergencies”. Some, however, expressed more confidence in what they observed and would be ready and confident to try and use the tool if they had some additional expertise on standby, perhaps as a co-facilitator. Common feedback from the training workshops was that materials did not reach them before the workshop so there was a great deal to absorb at the events. This was compounded by the language and comprehension issues already touched upon. A number of respondents, however, complemented the training teams on their approach and input which appears to have made up for at least some of the seemingly poor preparation. The use of case studies was singled out as being a particularly useful and helpful feature of the training, which probably merits more attention being given to these in the manuals (some are quite sketchy in detail) and courses, again perhaps with a slant also on how national/local REA Leaders might go about designing those of local interest and relevance. ` The ideal balance would seem to be workshops like that in India, where three days of theory are then put into practise, giving people a chance to apply some of their learning and enabling further informal exchange with the trainers. A number of the trainees from this workshop in particular highlighted the relevance of Module 3 of the REA process – consolidation and analysis – as they found it “very helpful in prioritising the issues and actions required in a disaster situation in such a short period of time”. Another pertinent comment was that the experience helped project staff “think through issues that otherwise they might not have done, in terms of how to focus an emergency intervention through the lens of longer term environmental impacts”. 23 4. MAIN FINDINGS II: MANAGEMENT AND MONITORING 4.1 ADMINISTRATIVE ASPECTS Day-to-day administration for this project has been mainly handled by the BHRC, which was responsible for hiring the Lead Researcher and additional expertise for field tests and training. Funds from the OFDA were transferred from CARE USA to the BHRC for this purpose. Overall management of the project appears to have gone quite smoothly, despite the involvement of two “overseeing” institutions, three sources of funding and contractual links to other partners, e.g. InterWorks. Some comments received, however, were that specific tasks would possibly have worked better had they been specifically assigned to an individual or group. This relates in particular with training and it is strongly recommended that a focal point is nominated/hired to undertake this responsibility in any subsequent phases. Since the departure of the Senior Environmental Adviser from CARE USA, CARE’s role in the process has been one mainly of management. The Senior Adviser of CARE’s Emergency and Humanitarian Unit, however, participated in the last training event in India in November 2003. As with the training comment mentioned above, there is also a clear need for either CARE and/or the BHRC to appoint someone responsible for taking this project and its outputs further forward, including follow-up to the outcomes of past and future REAs. Appreciative comments were expressed by those close to the whole development process in terms of the flexibility they were permitted to take different approaches towards developing the REA tool. It was, by all accounts, despite some teething troubles regarding institutional administrative issues, a “very positive, open experience”. This aspect of management has certainly helped deliver a product that corresponds more to user needs than it might otherwise have done. While not meshing together terribly well with the rest of he REA process at present, the addition of the module on green procurement is seen as having a valid purpose but was not foreseen at the outset, being added only following demand. Another positive outcome of this flexible attitude was the inclusion of more of a community focus in the assessment process than had been seen from the outset. Similar commendation should be given for the open, transparent nature of the web site, to which draft reports were often posted, as well as each of the field test and workshop reports. Many organisations or partnerships would not have been prepared to do this. Throughout the whole process, however, one must note that the driving force has been the Lead Researcher, with some support and assistance from partner organisations. This is unlikely to prove sustainable in the longer term and a strong argument is made for this activity to now be institutionalised – not in regards of ownership per se but in terms of finding a home for this product from where conscious and consistent efforts will be made to translate it further from a paper document to a series of practical and much needed tools. In the Evaluation’s view, this is the critical time for institutions to commit themselves fully to this project by: • acknowledging the value and appropriateness of this tool; • revising the REA in accordance with concerns expressed in Section 3.1; • revising training materials to match the new product; • securing additional funds to enable a successful roll-out of the tool; • integrating the main messages and processes into existing institutional policies and guidelines; • raising awareness of what this tool offers and encouraging partnerships to further spread the use of this tool; and 24 • continuing to support and monitor its implementation. 4.2 DONORS AND FUNDING SUPPORT It should be remembered that although reference is made to the “REA Project”, this in fact consisted of several distinct phases, as funding was obtained from different donors. This aspect may on occasion have been disruptive but overall it has been on schedule. A review of the financial aspect of this project was not specified for this evaluation, so little attention has been given to this. Table 2, however, presents the funding background for the project. Comments from those spoken to though on the issue of project management were favourable in terms of the manner in which funds were spent and, to a higher degree even, to the value of the output from this process. Table 2. Overview of Project Timeframe and Funding Situation Funding Source Activity Timeframe Amount (US$) UNEP/OCHA Initial REA development August-December 2002; extended to end-January 2003 25,000 Royal Norwegian Ministry of Foreign Affairs REA field tests (Afghanistan and Ethiopia) and training (Norway) December 2001 – December 2002 49,490 OFDA/USAID REA field test (Indonesia) training (Norway, Guatemala and India) July 2002 – February 2004 206,305 CARE USA REA development and co￾ordination (staff time) Total 280,795 4.3 THE REA ADVISORY GROUP Most people contacted through this review confirmed the value of having an Advisory Group in theory, but those with direct experience of it admit that it did not fulfil its role, as was intended. One response to the evaluation questionnaire perhaps sums this up: “The 'Advisory' board was… not well handled. There was never a face-to-face meeting to discuss the tool, we do not know each other, and we never had a chance to directly speak to those who used the tool. There was no direct two-way communication of any kind – not even a telephone call”. As stated in the Project Summary, the Advisory Board was to have been involved in the external review of the training syllabus. This, however, did not take place and there has been only minimal input to this aspect of the project outside of the training events. Likewise, feedback on the initial REA was received from only 3-4 members – chiefly those closely engaged in the process from the funding or management side, but there has been little input since this time. As far as can be deduced, members were not officially requested to play an 25 active role in the REA process. There was, apparently, an initial Terms of Reference, although there is some confusion surrounding this document. Responsibility does not lie on the individual members alone, however, as it appears as though the role of the board was never made clear which has added some confusion and probably did not generate much interest among the members more remotely involved in this work, i.e. those outside of the donor agencies. The Lead Researcher had face-to-face meetings with some of the Group members. In addition, 12 communications were sent from the Lead Researcher to the Group from 13 October 2001 to 1 February 2004, informing them of updates (including a follow-up to all training sessions), events and inviting feedback on any of these developments. The last note even mentioned this current evaluation with notification that they might be contacted. Response, however, was sporadic and limited. This is unfortunate and perhaps could have been overcome through either convening some Group meeting, a teleconference call or an invitation to Group members to participate in specific training events or field tests. Consideration should therefore be given as to the future role and structure of this Group. 26 5. RECOMMENDATIONS AND CONCLUSIONS 5.1 INTRODUCTION What has been described on several occasions as being a “small project” has, notwithstanding this notion, created a noticeable impact. Our awareness of how an environmental assessment might be conducted during an emergency or disaster has been considerably heightened, and the resources and other means needed to carry this out carefully recorded. Appreciation of this can be traced from community members, through a broad range of NGOs (including environmental and relief organisations) to some of the main UN agencies, and selected governments and donors. This in itself has been quite an achievement. Equally important, however, is the product, or more correctly the products, which have evolved and emerged from this process. The structure behind the current REA process has been modified many times following individual opinion or practical experience from direct field testing as well as feedback from a number of training events. While the process itself may appear to be time and labour intensive, the fact that a particular element or approach is now included in the process may be because it was requested or has been found worthwhile, perhaps even essential, to have it there. On the one hand, the combined approach of the development process, field testing, training and feedback has shaped the outputs, but serious concern has also emerged as to whether some of the underlying principles of the process are in fact adequate. This issue needs to be addressed as a priority, unless the institutions promoting its use – primarily CARE – are willing to accept this tool in its present state as a “Quick and Dirty” REA and not the ultimate Rolls Royce version that some may see as being essential. As stated in Section 1.2 “the REA’s point of departure was to produce a simple guide to identifying what are or what might emerge to be some of the main environmental concerns during a given, and changing, situation, as relief operations are planned, unfold and implemented.” The current REA tool/Guidelines is certainly not simple but limited experience from field tests suggests that it can work if resources are made available. A few observations may assist future guidance: • first, this REA is not intended to be a once-off exercise, nor it is expected to obviate the need for a full EIA should this be seen necessary. • expectations from the use of this tool are, it appears, sometimes too high and there is a need to revisit some of the basics before advancing further; • there has, perhaps, been too much readiness to adapt the process/Guidelines to individual comment, which might explain the disjointed impression one gets when reading the materials, and which might also explain some of the current design weaknesses. While a decision clearly needs to be taken in relation to the future of the present REA tool, it is perhaps meanwhile worthwhile to ask “Does this tool bring added value to relief and disaster assessments? This question was posed to most people contacted during this evaluation, and drew almost unanimous support. People like it as “it brings [to the fore] issues very often neglected in crisis situations”, one said, while other comments were that it “highlights the participation of communities”, “complements other disaster assessment tools”, “can make a significant contribution to having the environment considered in disaster response”, “gives importance to a critical area often ignored”, and that “with [this tool] CARE can advocate for the inclusion of environmental issues in the implementation of relief aid”. Thus, people on the working or receiving end have appreciated its worth, but may not necessarily be aware that they were 27 possibly drawing false assumptions from the observations on account of the tool’s current analytical structure. It is clear that additional work is required on the basic process and the Guidelines before any additional application or promotion is undertaken. While the time or resources required for carrying out a REA may not necessarily change as a result of the suggestions provided below, the applicability of the process might. In particular, more people might be interested in testing the approach for themselves or, more likely, would feel more confident to try and do so. This should remain one of the ultimate goals if this project continues. Once these concerns have been resolved, focus should shift to translating the final outputs into some key languages, refining the training materials, and designing and enabling successful roll￾out of the REA tool, primarily through additional specialist training and the training of trainers. Added consideration must also, however, be given to ways to try and encourage/ensure that the findings of REAs are taken into account and that they are seen to have a positive impact. That this has not happened as smoothly as one would like to have seen, should not be strictly seen as any fault with the REA tool or process, but rather highlights the need for institutional commitment and perhaps additional resources to support this process. 5.2 HAS THIS PROJECT ACHIEVED ITS OBJECIVES? Looking at some issues in more detail, eight specific indicators, addressing both the process and output, were selected from the outset to measure performance in the OFDA-funded activities. These actually form a useful means of summarising the achievements of this project and are shown in Table 3, together with additional comments on the extent to which each has been accomplished. Table 3. Progress Achieved with Regards Specified Project Indicators Indicator Comments Degree to which the Indicator has been Met Completion of the REA revision following the field test. The REA has been consistently revised after each field test and training event. To say it was “complete”, however, would be inaccurate as the REA process continues to undergo some change with additional input from reviewers. Full (but additional requirements have been identified). Publication of the REA, with hard and soft copies available. The REA is available on a web site dedicated to this project. Work is continuing to improve access to and the clarity of this site. It is not certain whether a printed version of the Guidelines will, or was ever meant to, be effected. If funds are available, however, then this should be carried out and copies widely distributed – once revisions have been made to the existing Guidelines. Considerable, but in the present context this is probably adequate. 28 Table 3 contd Indicator Comments Degree to which the Indicator has been Met Completion of one field test funded under this project (of three total). The Indonesian field test was successfully carried out with eight CARE and local NGO staff forming he core assessment team. Full. Training materials produced: syllabus, facilitator and participant manuals. Trainer’s Guide and Participant’s Manual produced. Full. Completion of two test training courses and one final training course. Three training workshops were organised, enabling further testing of the training materials Full. Training of relief assistance cadres in REA procedures. From among the 59 people who attended the training workshops, a small number of these (including those responsible for developing training the current materials as well as those for other relief and humanitarian organisations) are “routinely” engaged in relief operations so have benefited from this experience. Several instances of where this experience has been put into practise have been cited above. Partial – as no figure was specified for this. The number of priority issues identified in the test assessment which are addressed through changes to relief or rehabilitation activities within 30 days of completing the assessment. A number of “general” priority issues were identified in each field test. Feedback, however, suggests that none of these have been addressed. None. The number of priority environmental issues identified in the assessment which are resolved within 30 days of completion of the assessment. Among the priority environmental issues addressed, moves were made to integrate these into project design/redesign. From what information can be gathered, however, none has reached the level of having resources assigned to dealing with them, thus none have been resolved. None. Table 3 shows that the majority of the indicators framed at the outset have been met. Many, in fact, many have even been surpassed, e.g. through the unintended production of a separate Quick Guide to REA which is aimed primarily at those who have undergone training, or might otherwise be familiar with REA/EIA approaches and principles. The least successful aspect of implementation relates to the poor level to which the recommendations from the assessments have been picked up. This is again not a particular problem with this project’s outputs but clearly opens up a new area which will need further thought, resources and, above all else, institutional commitment if it is to be fulfilled. A highly 29 relevant comment was made in this context by one respondent who particularly appreciated the Green Procurement section of the REA but noted that: “it is hard to convince the finance personnel about the cost-effectiveness of the green products. But it is not impossible”. By extension, one could also pose the question “What is the purpose of conducting an REA if the commitment or resources to act on recommendations are not there?” 5.3 WHAT COULD BE IMPROVED? There is always room for improvement in even in the best projects, but if one need emerges from this evaluation it is in relation to the REA tool and accompanying Guidelines. The history, time and effort which has been invested in this product is well recognised. However, the current tool needs some basic attention while the Guidelines document suffers from a series of features which unfortunately detract from its appearance, readability and ease of comprehension, all of which work against it being a widely read document and thus becoming a broadly applied tool. Needed improvements falls into two categories: technical structure and content/presentation. The following comments are offered with a view to helping improve this worthwhile tool. 5.3.1 Some Technical Needs Although this particular aspect was not part of the Terms of Reference, four particular comments are made as these clearly relate to the structure, use and application of this tool. These are: • a need to revisit the design criteria, in particular the type of information that may be or is required, as well as certain design analytical requirements and data collection methodologies; • the rating scales and forms, which need to be revisited as these can be subject to much misinterpretation and misrepresentation, as well as being difficult to apply in certain instances; • additional guidance on consolidating observations and recommendations with particular emphasis on translating these into action; and • further guidance needs to be given that would help assure consistency between an initial and subsequent REAs which, although involving the same location, might be carried out by entirely different teams. 5.3.2 Some Content/Presentation Needs The need to get into the actual body of the REA has been addressed through the very recent provision of a Quick Guide which, as it points out in the Introduction, is intended for “those with a basic understanding of the REA process either through reviewing the Guidelines or from participating in training on the REA”. It is still questionable, however, as to whether this document alone is sufficient for someone to fully organise and carry out a REA: this should be tested before too long. A comment from one respondent captures this: “The main challenge I foresee with the tool is that it is very bulky and not user-friendly and as a result it has not been used to analyse the situation in our areas of emergency operation nor support emergency project design”. Specific suggestions made are that: • the Guidelines need a thorough edit and redesign; • the Guidelines could be re-organised to improve clarity and logic/flow; • the Guidelines are repetitive: much of the background and introductory material is repeated again in the modules; 30 • the use of terms is often not consistent, e.g. “Section” in the Modules and “Steps” in the REA diagram. Likewise, standard titles are required for modules and corresponding annexes, cf Module 1 and Annex B and Module 2 and Annex D; • there is too much cross referencing – causing the reader’s mind to jump around – and use of bold as a supposed visual aid; • a clearer step-wise guide to each module would be useful; • Module 4 should be enlarged and made more relevant and responsive; • Section 3 is perhaps the most important part of the whole process, assuming that the assessment has been planned and carried out to a high level of quality, but needs clearer instructions on how to amalgamate and prioritise; and • changes are needed to the Annexes: Annex C should be Annex I, references should be placed at the end; Annex J should be earlier; and Annex F and Annex G should either be dropped or kept in a reference companion volume (see below); and • it would be helpful if the Guidelines (or perhaps the proposed User Guide) included a list of what type of information is likely to be the most useful and the possible sources of this? The need to get into the actual body of the REA has in part been addressed through the provision of a Quick Guide which, as it points out in the Introduction, is intended for “those with a basic understanding of the REA process either through reviewing the Guidelines or from participating in training on the REA”. It is still questionable, however, as to whether this document alone is sufficient for someone to fully organise and carry out a REA: this should be tested before too long. A comment from one respondent captures this: “The main challenge I foresee with the tool is that it is very bulky and not user-friendly and as a result it has not been used to analyse the situation in our areas of emergency operation nor support emergency project design”. It is suggested that serious consideration be given to repackaging the REA Guidelines once agreement has been reached regarding its structure. A suitable structure could be a series of three handbooks arranged as follows: • “Background Information to REA”– an introduction to the REA process, what it is, what it can do and what resources are required to carry it out; • “Steps to Follow when Conducting a REA” – clearly describing suggested steps to guide a reader through the REA process. This would also outline the first steps to take when considering a training event, including how to set up a training workshop; and • “Reference Material and Technical Details”– a technical handbook containing annexes from the current Guidelines and any additional information which may need occasional referencing. Once a user has read through parts I and II, future reference should be mainly in relation to Part II only, with occasional consultation with Part III. Other suggestions for improvements to the management, training and practical application of the REA tool have been indicated elsewhere in the previous sections and will not be repeated here. 5.4 RECOMMENDATIONS The following recommendations are made on the basis of the Evaluator’s observations during the course of this evaluation, but are largely shaped on the comments and concerns raised by people contacted during this short exercise. 31 Recommendation 1. Strengthen the Institutional Structure and Commitment behind this Project. To make proper use of the materials thus far developed will require a significant shift in gear, and the lead agencies in this initiative CARE and/or the BHRC must be willing to commit to supporting continuation of this work, to the extent of institutionalising the REA process in their respective agencies work. Much of the “salesmanship” of this process has been at the individual level but further development of the REA, and in particular the uptake of its recommendations, will only be possible if this institutional commitment is made. This is therefore the critical time for institutions to commit themselves fully to this project by: • acknowledging the value and appropriateness of this tool; • securing additional funds to enable a successful roll-out of the tool; • integrating its main messages into existing institutional policies and guidelines; • raising awareness of what this tool offers and encouraging partnerships to further spread the use of this tool; and • continuing to support and monitor its implementation. Recommendation 2. Enhance the Technical Integrity of the REA Process. Before any other work is carried out it is essential that differences of opinion and concerns over some of the methodological and analytical approaches be sorted out. A small, active, working group should be established for a short period of time to overhaul the current process where needs have been identified. Recommendation 3. Enhance the Quality of the Project’s Outputs to Encourage Use and Application. It is strongly recommended that the manuals and guidelines produced thus far are revised and repackaged, following which they should be translated (or reworked in the case of the Spanish text) and disseminated – even if they are still evolving. The following in particular should be noted: • following the above-mentioned technical revision, the Guidelines should receive a thorough edit for structure, content and language, with practical steps to follow more clearly described; • present the information as a three-part guide to conducting an REA: Part I – “Background Information to REA”, Part II – “Steps to Follow when Conducting a REA” and Part III – “Reference Material and Technical Details”; • all outputs should have a common format and appearance; and • if resources allow, development of a computer-based “How to Conduct a REA” for ease of data capture. Once materials have been repackaged, an official launch of the process should be organised to raise awareness of its existence. Recommendation 4. Identify and/or Allocate Resources to Encourage and Enable Follow-up to Past and Future REA Field Tests. Unless practical uptake of the REA’s recommendations happens, there will be little reason to continue with the development and dissemination of this tool. Leading by example, CARE, in particular, should identify how it might enable locally recognised priorities to be integrated into ongoing projects and programmes. Many people are convinced of the outputs of the REA assessments but not enough attention has been given to ensure that they are implemented and monitored. As this is the fundamental purpose of engaging in an REA process, it seems important that some of these findings are Recommendation 5. Continue to Establish Key Partnerships and Focus Resources on Getting these Agencies to Use or Customise the REA for their Own Benefits. Attention should concentrate on getting the tool used with a select number of agencies outside CARE, as 32 well as within. The examples started by UNEP/OCHA, OFDA and others in integrating REA approaches into their own assessment and training systems should be highlighted and built upon. Recommendation 6. Produce a Short, Sharp Training Module on the REA. The current training materials, while comprehensive, are seemingly too large and detailed for quick and easy uptake by institutions. If a short, single stand alone module was available, this might encourage use of the tool by other agencies in their respective training programmes, including environmental tools in their emergency assessments rather than dealing with it as an add on. Recommendation 7. Focus Attention on Training Potential REA Leaders and Other Trainers. Priority attention should be given to training individuals who are currently in a position to use and apply the benefits from the REA process – from within CARE and BHRC as well as other agencies. This will ensure a broad dissemination of qualified persons experienced in the use of the tool. Future training sessions should, as a rule, be split into a theoretical and practical session, for enhanced appreciation of the REA tool. Recommendation 8. Revitalise or Abandon the Advisory Group. Should the REA project continue into a second phase, it is advisable that the role of the Advisory Group be revisited by CARE and the BHRC, in particular. Although it will add further demands to peoples’ time, if this group were to become more active in guiding and supporting implementation and application of the REA in various situations, or in assisting with contacts, it would assist the core team considerably and allow them to concentrate more on delivering the products. Much depends on whether the “management” considers it necessary to continue with a form of oversight body given that the subsequent phase, as planned, focuses mainly on roll-out through training. Recommendation 9: Improve the Visibility and Outreach of the REA Process. The current web site should be overhauled and made clearer, with easier access and a title that is easily remembered. Relevant documents should be clustered, e.g. Guidelines, Training Materials, Field Tests, Resources, etc., with one paragraph of text describing the contents of each cluster. If resources exist, a central e-centre could be established to handle enquiries about the REA process and to improve inter-agency communications, responding perhaps to simple enquiries itself and directing more complicated issues to the relevant experts. Consideration should also be given to developing a small REA newsletter which be primarily web-based. 5.5 SUGGESTED NEXT STEPS A large number of issues have been identified as “next steps” for this initiative. As one respondent stated “There is a long way to go in getting it [the tool] into the hands of others, but a high level of interest has been shown. The work, though, clearly starts here”. Many of the following have been suggested by more than one respondent to questionnaires or discussions. For convenience, they are listed as simple bullet points. Attention should, however, be given to these especially in the (re-)design of any Phase II proposal. Project Management • Identify an owner for this project for the purpose of focal responsibility and contact. • REA should become part of the institutional policy of the owner agency/agencies. • Integrate this REA into other kinds of assessment. • Disseminate this tool and training materials to partners. Governments, academia, donor and other groups should be engaged and prepared to help launch REAs in certain targeted 33 vulnerable regions. This would provide a good platform and, if the process proves itself positively, would facilitate and encourage institutionalisation. • Disseminate the REA tool to potential donors and advocate for their use of REA elements in proposal requirements Training • Identify and appoint a focal point for training. • Develop a training strategy with a double purpose of raising awareness among decision makers of the importance of the environment in disaster response, and train staff of NGOs and international organisations in their use. • Train trainers. • Translate training materials. 34 ANNEX I REA CHRONOLOGY This information was kindly provided by Charles Kelly, REA Lead Researcher, and is extracted from the (unseen) project Final Report. July 1996 – Presentation of Disaster and Environmental Change: The Impact of Population Displacement and Options for Mitigation, at the Pan Pacific Hazards 96 Conference, Vancouver, Canada. March 1999 – Presentation of Disasters and Environmental Impact: A Framework for Rapid Assessment and Planning by Response Personnel, at Green Cross Conference, London, United Kingdom. Mid 1999 to early 2001 – Discussions between C. Kelly, Debbie Williams and John Twigg of Benfield Hazard Research Centre, University College London and RedR on development of a methodology for rapid environmental impact assessment and training program. Early 2001 – Development of basic REA project proposal with input from Mario Pareja of CARE US and presentations to UNEP/OCHA, USAID/OFDA. June 2001 – Presentation on the REA to CARE Norge. August 2001 to January 2002 – Funding from UNEP/OCHA for development of the REA methodology. This work included discussions with Mario Pareja of CARE, and contacts with NGOs, Donors and I.O.s in the US and Europe. See Acknowledgements of the Guidelines for a list of organisations contacted. September 2001 – Presentations on the REA to environmental NGOs in Washington and to CARE International in Brussels. October 2001 – Field trip to Orissa India to discuss disaster-environment linkages, practical emergencies with environmental impacts in disasters and local needs and limits to assessment procedures. October 2001 – Presentation on Rapid Environmental Impact Assessment: Framework for Best Practice in Emergency Response, at Sharing Experiences on Environmental Management in Refugee Situations: A Practitioner=s Workshop, Geneva, 22-25 October 2001, hosted by UNHCR, Paper posted www.benfieldhrc.org under Disaster Management. December 2001 – Funding from CARE Norge for field testing and training on the REA. January 2002 – Completion of Guidelines for Rapid Environmental Impact Assessment in Disasters (version 1). February-March 2002 – Field test of the Guidelines in Afghanistan, hosted by CARE Afghanistan, followed by changes to the Guidelines document. June 2002 – Presentation of Assessing Environmental Impacts During Natural Disaster: the Development of a Rapid Environmental Assessment Methodology, The International Association 35 for Impact Assessment Meeting, The Hague (later published in the Journal of Environmental Assessment Policy and Management, Vol. 4, No. 4 ,December 2002.) August-September 2002 – Field test of Guidelines in Ethiopia, including community assessment, hosted by CARE Ethiopia, followed by changes to the Guidelines, and inclusion of section specifically on community assessment. January-February 2003 – Field test of the Guidelines in Indonesia hosted by CARE Indonesia. Field test included organisational level assessment and nine day community level assessment. (Kelly and Pareja participated in this field test.) February-March 2003 – Redrafting of the Guidelines to reflect input from field tests, giving equal weight to organisational and community assessment procedures and results. Redrafted document circulated for comment and provided to InterWorks as basis for their work on a training module. February-April 2003 – Development of a REA training module by InterWorks. April 2003 – Tests of REA training module in Oslo, Norway and Antigua, Guatemala. Participants in Oslo were largely not persons who were involved in field operations. One environmental impact assessment trainer and a disaster preparedness project manager from Madagascar also participated in the Oslo training. Training led by Paul Thompson (InterWorks) with Becky Myton of CARE Honduras attended as co-trainer. Participants in Antigua were drawn from each Central American country and included a mixture of NGO, I.O., and government personnel. This training was conducted in Spanish. Training led by Charles Dufresne of InterWorks with Mario Pareja and Becky Myton attended as co-trainers. June 2003 - February 2004 – Development of an eLearning module on the REA by InterWorks. June 2003 – Presentation on Gender, Disaster, and the Environment: Experiences from the Rapid Environmental Impact Assessment Project, at the International Emergency Management Society meeting, Provence, France. June 2003 – Presentation of Disasters Management and Environmental Impact Assessment: Gaps and Linkages at The International Association for Impact Assessment Meeting, Marakesh, Morocco. Mid-2003 – REA Guidelines used as the basis for a course on environmental impact assessment presented by Becky Myton in Honduras. September 2003 – Incorporation of Environment as cross-cutting issue in Sphere Standards and inclusion of the REA as a reference in the Shelter section of the Humanitarian Charter and Minimum Standards in Disaster Response Handbook., September 2003 – Presentation on the REA project made to I.O.s and NGOs based in Geneva. The presentation was hosted by UNEP/OCHA. Late 2003 – CARE Ethiopia begins field staff training on the REA, adapted to conditions in Ethiopia. 36 October 2003 – Presentation on the REA project to government, I.O. and NGOs based in Kobe Japan, hosted by Disaster Reduction and Human Renovation Institution. October 2003 – Presentation of Rapid Environmental Impact Assessment in Mega City Disasters: Issues and New Tools, at the International Symposium on New Technologies for Urban Safety of Mega Cities in Asia, Tokyo, Japan (paper posted to symposium web site and distributed in proceedings CD). November 2003 – Final presentation of REA training module at Bhubaneshwar Orissa) India in co-operation with Sphere India. Participants can be NGO, I.O. and government sectors. Training included 3 days of classroom instruction on the REA and 3 days of practical use, including a community assessment exercise. Training led by Paul Thompson of Inter-works with Samuel Tadesse of CARE Ethiopia attended as co-trainer and Jock Baker as observer. November 2003 – Project review and Phase II design discussions involving CARE US, CARE Ethiopia, InterWorks and Benfield Hazard Research Centre following the Orissa training. December 2003 – Half day training on the REA provided to LWF staff participating in a week￾long workshop on community level disaster preparedness, Konark, India. January 2004 – Development of a REA Quick Guide based on work originally done by InterWorks. 37 ANNEX II SAMPLE QUESTIONNAIRE USED IN THIS EVALUATION We may choose to lift some quotes directly from this response form to illustrate particular opinions or expressions. Such quotes will not be attributed to your name. However, if you prefer to not even have your comments reflected, please indicate this below. Thank you. Please return your completed questionnaire to: David Stone, CARE Consultant, either by fax (00 41 22 366 3818) or e-mail (davidstone@vtxnet.ch). 1. What has been your involvement in the development and trialling of the REA? 2. Have you had previous experience of REAs? Please describe. 3. What was your overall impression of the CARE REA as a tool? 4. What are your impressions of the REA Guidelines (Version 4.2)? Did/do you find them easy to use? Is there any aspect you would like to see changed? 5. What training have you been given in using the REA and/or Guideline? 6. What are your opinions on the training materials that accompany the REA Guideline? 7. Did you find the outcome(s) of the training or field experiences useful? Do they respond to your expectations/needs? 8. Do you believe that this REA process brings added value to relief and disaster assessments? If so, how? If not, what would you like to see changed? 9. What do you see as the next steps for this initiative? 10. Did you find the outcome useful? Does it respond to your expectations/needs? 11. Have you been able to apply some or all of the recommendations? If so, how? If not, why not (what were/are the main constraints)? 12. Any other comments you would like to make? THANK YOU! 38 ANNEX III TERMS OF REFERENCE Terms of Reference (ver. December 9, 2003) Evaluation of the Rapid Environmental Impact Assessment Project Introduction The Rapid Environmental Assessment is a collaborative effort started in 2001 involving CARE International, the Benfield Greig Hazard Research Centre, and InterWorks, with financial support from USAID’s Office of Foreign Disaster Assistance (OFDA), the UN Environmental Programme (UNEP), the UN Office for the Co-ordination of Humanitarian Affairs (OCHA), the Royal Norwegian Ministry of Foreign Affairs, and CARE International. The REA methodology equips emergency managers with a rapid means for identifying threats to the environment due either to the effects of the disaster event itself or unintended impacts from the resulting relief operation. Applicable to natural, technological or complex political disasters, the REA is a tool to identify, define, characterise and prioritise potential environmental impacts in disaster situations by means of a simple qualitative assessment process using brief concise descriptions, rating tables and check lists to identify and rank environmental issues and promote appropriate interventions. With an awareness of such environmental threats, humanitarian aid workers and disaster affected communities should be better able (1) to avoid environmental damage, (2) incorporate best practices into their programming to better protect the environment, and (3) improve the overall effectiveness of relief and recovery efforts. The REA methodology and associated training module was developed and refined through a range of assessments and trainings conducted in Asia, Africa and Central America. This consultancy is planned for to take place during three weeks in January 2004. The evaluation will be primarily desk-based, relying on document review and telephone interviews with key informants. Additional project-related documents are available at: www.benfieldhrc.org/DMU/DMUSetup/Projects/rea.htm Objectives The project has completed the initial development of the REA and related training materials and requires an evaluation to: 1. Document actual outcomes against performance measurement criteria stated in the relevant project and sub-project documents; 2. Assess the effectiveness of the REA process as a “best practice” tool in disaster management; 3. Consolidate and summarise perceptions of project participants and interested parties about the REA methodology and training materials, and, 4. Identify successes and improvements in the project implementation process. This evaluation is being funded by USAID/OFDA. The evaluation will need to measure and report on the performance of the project against the performance indicators established with OFDA. However, to the degree possible, evaluation should cover project activities from 39 inception (mid-2001) and the evaluation report will be shared with UNEP/OCHA and RNMFA at the review stage. In most cases, the OFDA-agreed indicators are applicable to the whole of the activities under the project. Evaluation Criteria The grant proposal to OFDA set out a number of performance indicators and measurements. These are set out below and form the basis for the evaluation of the project. III. Project Goal and Objectives B. Indicators The indicators to be used to measure project performance focus on process and output, and include: • Completion of the REA revision following the field test. • Publication of the REA, with hard and soft copies available. • Completion of one field test funded under this project (of three total). • Training materials produced: syllabus, facilitator and participant manuals. • Completion of two test training courses and one final training course. • Training of relief assistance cadres in REA procedures. • The number of priority issues identified in the test assessment which are addressed through changes to relief or rehabilitation activities within 30 days of completing the assessment. • The number of priority environmental issues identified in the assessment which are resolved within 30 days of completion of the assessment. The results of the REA will be used in relief operations assessment and planning during and after the three field tests and by relief cadres who participate in the training. This should result in a discernible improvement in how relief operations identify and deal with environmental issues and serve as an indicator of project success. VI. Monitoring and Performance Measurement B. Performance Measurement As indicated under in Section III B, the project indicators focus on process, outcome and impact items. Following the indicators mentioned in Section III B, the proposed performance measurements for the project are as follows: 1. Reporting: quarterly and final reports. 2. One REA revision. 3. Up to seven consultations of the Advisory Board. 4. One field test. 5. One set of REA training materials, including a course syllabus book, and manuals for facilitators and participants. The training materials will include a training module for use as a stand alone unit or as part of another training program, and self-study course work. 6. REA training development: Two test trainings and one final training provided: 20 relief cadres during each training session. 7. A 90 per cent pass rate for the relief cadre participating in the final training. 8. One web site with the REA training materials and other project-generated documents posted 40 and available to any user. 9. Number of disaster relief operations in which REA procedures are used for assessment and planning following the field testing and training activities. 10. At least 70% of the priority environmental issues identified will be resolved or addressed by changes to projects or activities, during the 30 day period after the assessment. In addition, those involved in the REA tests (including victims where possible) and in trainings will be questioned whether use of the REA has made disaster assessments and relief operations more effective and efficient than would be the case without the REA. This qualitative impact assessment will be done at the end of the REA field tests, among trainees at the end of the training and as follow-up to the trainings and as part of the final evaluation. The project expects that 60% of REA users to find that the REA has improved relief operations and the provision of assistance to disaster victims. The results of this survey will be documented and presented as a lessons learned study. In addition, the evaluation will: 1. Consider the effectiveness of the REA process as a best practice tool in disaster management. Specifically, are the REA Guidelines for Rapid Environmental Impact Assessment in Disasters and associated training materials useful to field personnel in defining environmental issues and linkages in disasters. 2. Consider the effectiveness of REA training materials based on feedback from trainees. 3. Document successes of the project and highlight how the implementation of the project could have been improved. The review of project implementation should include administrative arrangements and procedures. Evaluation Methods and Procedures The evaluation will be conducted using two methods: 1. The review of project documents available at the project web site, from CARE, Benfield Hazard Research Centre, InterWorks and from third parties. These documents include the OFDA project proposal, reports on the three REA field tests and three trainings, background information on the project and additional reports and feedback received from field test and training participants. 2. Phone, email or other contact with project staff, field test and training participants and other parties. CARE and BHRC will provide the evaluator with a list of contact names and phone and/or email addresses. It is not expected that formal quantitative surveys can be conducted as part of the evaluation with a significant part of the input coming from qualitative input based on questionnaires and interviews. The evaluator must be aware that many individuals involved in the REA project are actively involved in disaster relief and recovery operations, requiring an appropriate approach. Evaluator Profile & Selection Process Preference will be given to candidates with previous experience in evaluating post-disaster humanitarian assistance programmes, familiarity with the international NGO sector, and who are 41 able to demonstrate a knowledge of environmental issues faced during and after disasters. Selection of the evaluator through a consultative process involving project and sub-project managers. Outputs and Schedule The evaluator will provide to CARE a maximum of 20-30 page report (excluding annexes) that should include a concise Executive Summary of no more than 3 pages. The report should describe the background to the study, methodology of the evaluation, schedule of activities, results, summary of recommendations, and persons interviewed (in an annex). The use of tables and charts to summarise the evaluation results are encouraged, but elaborate graphics and formatting are not necessary. Annexes to the report should include relevant additional data and information collected during the evaluation that substantiates or expands on the evaluation results. Following distribution of the final report, a review will take place involving key internal and external stakeholders to identify specific actions to be taken in respect to: a) Finalisation of the Phase II REA proposal; b) Revision of the REA methodology and/or training modules; and c) Other relevant actions. A total of 15 person days are allocated to the evaluation. The evaluator should provide a draft report to CARE, sub-project managers (BHRC & InterWorks) and possibly other stakeholders for comment prior to the completion of the work time allocated for the evaluation. Comments will be provided within 10 working days of the draft submission and the evaluator will be responsible for making appropriate revisions prior to producing a final report. CARE is responsible for ensuring that the final report is of acceptable quality but otherwise editorial authority resides with the evaluator. CARE does however retain the right to attach a note, as an annex, of its own views. 42 ANNEX IV PEOPLE CONTACTED IN THIS EVALUATION The following people were contacted: Charles Dufresne, Paul Thompson, Charles Nelly, John Twigg, Jock Baker, Patricia Charlebois, Guillaume Aguettant, Debbie Williams, Mario Pareja, Marion Pratt, Harlan Hale, Sigrid Nagoda, Suparman Warman, Dereje Adugna, S. N. Mishra, Amar Vaid, Samuel Tedesse, Holly Solberg, Johan Kieft, Becky Myton, Paul Barrer, Rally Austin, Vivek Sharma, Rigoberto Giron, N. M. Prusty, Neville Pradhan, K.G. Mathaikutty, Alhaji Jeng, Campbell Day, Olav Myrholt, Helle Floisand, Jenny Myton, Knut Ragnar Johannessen, Moira Eknes, Paul Borsboom, Scott Solberg, Ginna Rakotoarimanana, Thale Kermit, Eirik￾Jarl.trondsen, Siddhant Das, Sibaprasad Mishra, Satyanarayan Jena, Rachna Singh, Prafulla K. Rath, Madhusmit Padhi, J. K. Mohanty, Inakhi Patra, Clark Efaw, C. Ashok Kumar, Binod C. Sabat, Bheda Anjana Rajesh, Steinar Sundvoll, Moira Eknes, Paul Ugarte, Wilmer Dan Teni Pop, Carlos R. Montes, Denis Pena Solano, Estaban Salavador Casado, Francisca Orellana, Henry Leonel Aldana, Juan Jose Sinay Garcia, Juan Manuel Giron, Luis Gonzalez, Magdelena Cortez, Maria Edna Vidaurre, Mario Flores, Miguel Omar Montoya, Oscar Juarez, Rene Molina, Roberto Peralta, Rohana Lalith Weragoda, Jennie Ownes, Julio Galvez Tan, Walter Knausenberger, Anshu Sharma, George Sammy, Franklin McDonald, Gaspard Bikwemu, Lousie Sperling, Ahuma Adodoadji, Howard Bell, Nancy Gelman, Alice Doyle, Graham Saunders, Colin Reynolds and Weston A. Fisher. Our thanks again to all those for participated in this evaluation. 43 ANNEX V MATERIALS CONSULTED General BHRC and CARE. No Date. Project Summary. 4pp CARE USA. January 2002. REA Phase II Project Proposal. Kelly, C. 2001. Rapid Environmental Impact Assessment: A Framework for Best Practice in Emergency Response. Paper presented at an international workshop on Practising and Promoting Sound Environmental Management in Refugee/Returnee operations, UNHCR, Geneva. Kelly, C. October 2003. Rapid Environmental Impact Assessment in Mega City Disasters: Issues and New Tools. Paper presented at the International Symposium on New Technologies for Urban Safety of Mega Cities in Asia, Tokyo, Japan. REA Manuals CARE. December 2003. Guidelines for Rapid Environmental Impact Assessment in Disasters. 109pp. CARE. December 2003. Quick Guide: Environmental Impact Assessment in Disasters. 40pp. CARE. December 2003. Rapid Environmental Impact Assessment in Disasters: Participant’s Handbook. 102pp. CARE. December 2003. Trainer’s Guide for the Rapid Environmental Impact Assessment in Disasters Workshop. 100pp. Training Workshop and Field Test Reports Thompson, P. April 2003. REA Training Workshop Report, Norway. 21pp. Dufresne, C. May 2003. REA Training Workshop Report, Guatemala. 19pp. Thompson, P. November 2003. REA Training Workshop Report, India. 16pp. Kelly, C. April 2002. Afghanistan Field Test Report. 89pp. Kelly, C. and Tadesse, S. September 2002. Ethiopia Field Test Report. 127pp. Kelly, C. and Pareja, M. February 2003. Indonesia Field Test Report. 73pp. PowerPoint presentations prepared by InterWorks.