Evaluation Report: The Minimum Economic Recovery Standards Program February 2015 This publication was produced for review by the United States Agency for International Development. It was prepared by Joan Hall, independent evaluation consultant, for the SEEP Network. DISCLAIMER: The author’s views expressed in the publication do not necessarily reflect the views of the United States Agency for International Development or the United States Government. Acronyms CaLP Cash Learning Project CoP Community of Practice CRS Catholic Relief Services DFID Department for International Development (UK) DRR Disaster Risk Reduction ECHO European Commission - Humanitarian Aid & Civil Protection EMMA Emergency Market Mapping Assessment FAO Food and Agriculture Organization GP General Practitioner (MERS training course) HFHI Habitat for Humanity International HH Household IFRC International Federation of the Red Cross ILO International Labour Organization INEE Minimum Standards for Education in Emergencies IRC International Rescue Committee LEGS Livestock Emergency Guidelines and Standards MERS Minimum Economic Recovery Standards MOU Memorandum of Understanding NGO Non-governmental organization OFDA Office of Foreign Disaster Assistance OCHA UN Office of Coordination of Humanitarian Affairs PRM State Department Office of Population, Refugees and Migration RFP Request for Proposal SC Steering Committee SOW Scope of Work ToR Terms of Reference UNDP United Nations Development Programme UK United Kingdom USAID United States Agency for International Development USAID/FFP USAID Food for Peace UNDP United Nations Development Fund UNHCR United Nations High Commission on Refugees UNICEF United Nations Children’s Fund WASH Water and Sanitation WFP World Food Programme Contents Executive Summary ....................................................................................................................................................1 Introduction ................................................................................................................................................................3 Purpose of the Evaluation ......................................................................................................................................4 Methodology ..........................................................................................................................................................4 Objectives of the MERS Program ...........................................................................................................................4 Current status of Activities – Planned vs Achieved ................................................................................................5 Findings .......................................................................................................................................................................7 Activity 1: Training and Capacity Building ..............................................................................................................7 Activity 2: Establishing a Steering Committee ..................................................................................................... 12 Activity 3: Outreach and Awareness Building ..................................................................................................... 14 Activity 4: Field Testing ........................................................................................................................................ 16 Activity 5: Establishing an Information Hub ........................................................................................................ 18 Summary of Survey Results ..................................................................................................................................... 19 Program Efficiency ................................................................................................................................................... 21 Program Management ............................................................................................................................................ 21 The Value Added from Being a SPHERE Companion Standard ............................................................................... 22 Conclusions .............................................................................................................................................................. 22 Annexes ................................................................................................................................................................... 23 Annex 1: Scope of Work ...................................................................................................................................... 23 Annex 2: Bibliography .......................................................................................................................................... 27 Annex 3: Survey Tool ........................................................................................................................................... 28 Annex 4: Interview Guide .................................................................................................................................... 30 Annex 5: List of Stakeholders Interviewed .......................................................................................................... 32 1 Executive Summary In late 2012, with support from the USAID’s Office of Foreign Disaster Assistance, SEEP launched a program to promote greater awareness and use of MERS. The program is called Sustainable Economic Recovery after Crisis: Promoting the use and adoption of Minimum Economic Recovery Standards. The program’s start date was June 1, 2012, with a planned end date of June 1, 2014. A modification of agreement was done through December 1, 2014, and a no-cost extension was provided by USAID/OFDA until Jan. 31, 2014. The project activities were: 1. Training and Capacity Building 2. Establishing a Steering Committee (SC) 3. Outreach and awareness raising 4. Field Testing In late 2014, SEEP commissioned an evaluation of this program. This document is the result of that evaluation. The output of the evaluation is meant to inform OFDA/USAID as well as MERS program management and other program stakeholders of the impact of the first phase of the MERS program and the next steps to take with regards to continuing the roll-out and promotion of the Standards. The methodology included:  Review of project documentation;  An on-line survey of participants in the MERS trainings; and  Key informant interviews with trainees/MERS graduates, Steering Committee members, and other stakeholders. Findings The evaluation finds that the activities have been substantially achieved. The training and capacity building activity has resulted in 44 certified Trainers in MERS and 114 people with General Practitioner certifications, across 5 regions and in 3 languages. The Steering Committee has six consistent and committed members, meeting every six months, representing a range of humanitarian assistance agencies and including technical and policy representatives. The Steering Committee has substantial buy￾in from its members, who have recently drafted a Strategic Plan for the continuance of MERS. SC members have also committed funding for MERS activities. The outreach and awareness raising activity has also been successful. SEEP has promoted MERS at its own conferences and via its website, while SC members have also promoted MERS in their own agencies and beyond. The field testing of MERS has been the least successful activity, due to a lack of budget. Still, the need for this persists and having lessons learned from field testing would improve the marketability of MERS. 2 The information hub is up and running, and contains MERS resources, including the Handbook, video testimonials, blogs, links, and so on. The Handbook, now a companion to the SPHERE Standards, has been downloaded over 2,500 times, far surpassing the target. In terms of the DAC evaluation criteria, stakeholder interviews and the on-line survey clearly indicate that MERS was relevant at the beginning of the project and continues to be relevant. As to effectiveness, the training, information hub, and outreach and awareness building activities have been very effective, as has the relationship with SPHERE and the other companion standards. The SC is gaining effectiveness and needs more time to achieve this fully. The field testing activity, since it did not occur, was not effective. SEEP managed the project efficiently and even generated some income from training, and stakeholders have been positive in their opinions about SEEP’s management. In terms of sustainability, this is a public good, and will need subsidies for some time. The impact of the project at the field level – during early economic recovery – has been limited due to insufficient time to reach a critical mass of MERS advocates and trained practitioners. This needs more time to achieve. Recommendations The highest priority recommendations are:  Continue to promote the training for MERS, and improve the training materials by adding more field evidence;  Continue building the strength of the Steering Committee – in terms of number of members, buy￾in to MERS, ability to expand the awareness and use of MERS, and funding for MERS activities;  Continue promoting the awareness of MERS, particularly to donors, so that they reference MERS in their policy and programming documents;  Source funds for field testing, and then consolidate this information for the information hub and to include in case studies in the training, and  Continue housing MERS within the SEEP Network;  Continue to build the relationship with SPHERE and LEGS, and look into opportunities to build a relationship with CaLP. Conclusions The program has been well-managed over the course of its 30 months by SEEP. The objectives have been achieved for the most part. The major constraints have been the limited number of staff on the project, no funding for field testing of the Standards, and the short time period of the project. These constraints have limited the impact of MERS. Awareness of MERS has increased over time, due to several of the activities. More people and institutions are trained now in MERS, and several donors are referencing MERS in their policy and programming documents. The relevance of MERS is clear, and there is still a need to continue outreach, awareness raising, and training. The engagement of the SC is growing, and this will be critical to seeing MERS applied more frequently and consistently in the field. MERS should be promoted further as a SPHERE companion standard, as this will increase its outreach. More funding is needed to support further extension of these activities, which have proved to be effective, except for the field testing, which was not funded. This also needs to be funded. 3 Introduction The SEEP Network hosts the Minimum Economic Recovery Standards (MERS) Initiative. The origins of the Minimum Economic Recovery Standards (“the Standards”) are rooted in past crises, such as the Indian Ocean Tsunami, the earthquake in Haiti, and complex humanitarian emergencies and protracted crisis in Afghanistan and the Horn of Africa. Several of SEEP’s members working in humanitarian emergency response identified the need for practitioners to share knowledge on the challenges and emerging practices of economic recovery in crisis environments. In 2007, with funding from USAID through the FIELD Support-LWA mechanism, SEEP convened a task force to develop the first draft of the economic recovery standards, and hosted a workshop in Washington D.C. to launch the consultative process. This process was made up of a broad consortium of practitioners from around 30 international humanitarian agencies. Each section of the MERS Handbook was developed by a practitioner-led working group supported by SEEP that brought together experts from economic development and humanitarian relief. In 2009 and 2010, SEEP held regional consultations on the Standards in East Africa, Latin America, the Middle East, Europe and South East Asia. The Standards were later field tested by Mercy Corps, ACDI VOCA, Catholic Relief Services, and pilot training courses were implemented in Indonesia and Jordan, with consultations in Ecuador, Kenya and the UK. In 2010, MERS’ original contributors as well as new members from academia, NGOs, research groups, donors and international organizations were reconvened. A second edition was published in 2010, which include input and feedback of more than 200 practitioners from 63 organizations. In 2011, the MERS was accepted as a companion to the Sphere Handbook. Sphere is the most recognized standard in humanitarian response. Alongside the other companion standards, Livestock Emergency Guidelines and Standards (LEGS), and Minimum Standards for Education in Emergencies (INEE), these minimum standards provide in depth technical guidance and harmonized standards to organizations working in crisis environments, beyond the basic life-saving standards included in Sphere. In late 2012, with support from the USAID’s Office of Foreign Disaster Assistance, SEEP launched a program to promote greater awareness and use of MERS. The program is called Sustainable Economic Recovery after Crisis: Promoting the use and adoption of Minimum Economic Recovery Standards. The program’s start date was June 1, 2012, with a planned end date of June 1, 2014. A modification of agreement was done through December 1, 2014, and a no-cost extension was provided by USAID/OFDA until Jan. 31, 2014. In late 2014, SEEP commissioned an evaluation of this program. This document is the result of that evaluation. The output of the evaluation is meant to inform OFDA/USAID as well as MERS program management and other program stakeholders of the impact of the first phase of the MERS program and the next steps to take with regards to continuing the roll-out and promotion of the Standards. 4 Purpose of the Evaluation The purpose of the evaluation is to evaluate if there has been an increase in the level of awareness of the Standards in the humanitarian sector, and if this increase in awareness has contributed to greater use in emergency program design and improved economic recovery. Please see Annex 1 for the Scope of Work of the evaluation. Methodology The methodology included:  A review of project documentation. The evaluator reviewed the original project proposals, all MERS publications, quarterly reports to USAID/OFDA on project progress, MERS training materials, MERS training evaluations, e-learning course data and other documents. For a complete list of documents reviewed, please Annex 2: Bibliography.  An on-line survey of participants in the MERS trainings was conducted. The number of respondents was 23, or 23% of the sample size of 100 (which represented the total number of people trained in the SEEP database.)1 Please see Annex 3 for the survey tool.  Held key informant interviews with trainees/MERS graduates, Steering Committee members, and other stakeholders. Fourteen key informants were interviewed. Please see Annex 4 for the interview guide and Annex 5 for a list of people interviewed.  A note was put out to the EMMA Community of Practice asking about the use of MERS, but no responses were forthcoming. Given the short period of the evaluation period and the fact that practitioners are always very busy, it was difficult to get a good response rate. However, this does not seem to affect the findings and conclusions of the evaluation. Many respondents were in agreement in their responses. Objectives of the MERS Program According to the SEEP Project Proposal to USAID (revised 2014), the purpose of the program is to “Increase awareness and adoption of the Minimum Economic Recovery Standards by practitioners/organizations involved in humanitarian situations.” According to its Monitoring and Evaluation Framework, the program has two major objectives:  To increase awareness and adoption of the MERS by practitioners/organizations involved in humanitarian situations; and, 1 However, some recipients did not receive the email with the survey link; there were approximately 10 error response messages. The email message may have also found its way into spam boxes which would lower the response rate. 5  To improve humanitarian response to crisis by increasing the institutionalization of the MERS in emergency response programming. The project activities were: 1. Training and Capacity Building 2. Establishing a Steering Committee (SC) 3. Outreach and awareness raising 4. Field Testing 5. Establishing an information hub Current status of Activities – Planned vs Achieved The table below illustrates the status of the project and compares the planned targets with the actual achievements: Table 1: Planned Vs Actual Achievements Activity Planned Actual Achieved? 1. Training and Capacity Building At least 60 Trainers certified; At least 8 certified trainers in 8 regions; At least 25 % of trainers are women; At least 23 organizations trained; Functional e-learning course available; 250 people trained on e-learning course 2 courses, each leading to a certificate (General Practitioner or Trainer of Trainers); Training materials in English, French, and Spanish exist; 1 training in French was done in Dakar, and one in Spanish in Guatemala (Jan 2015); 6 ToT trainings delivered; 158 practitioners trained (100 by SEEP); 30% female; 44 ToT and 114 GP trained; 54 agencies trained; Trainings in 5 regions (Bangkok, Dakar, Beirut, Nairobi, Washington DC, Manila and Guatemala); E-learning course piloted and launched; 10 people completed, 32 registered for, and 22 started the e-learning course, representing 34 countries; about half were not affiliated with the IFRC (data at Jan 19 2015) Substantially achieved 6 2. Steering Committee 1 SC with 10 members SC has met at least 4 times (at least once every six months and at least 1 time in person during LOP) A strategic plan exists 1 SC established 6 consistent members 4 meetings, every 6 months, once in person Strategic Plan produced SC members represent Catholic Relief Services, Mercy Corps, UNHCR, IFRC, Relief International, and the International Rescue Committee. Minutes exist for all meetings Substantially achieved 3. Outreach and Awareness Raising 4 conferences/events at which SEEP offers MERS training; 50 people trained 4 MOUs signed with organizations for the purpose of promoting MERS 2 conferences; 2 MOUs Partially Achieved 4. Field Testing 6 organizations trained in MERS for field testing 4 organizations trained in compiling lessons learned from field testing Lessons learned compiled in the form of case studies 6 NGOs identified 4 NGOs trained Limited field testing (no budget); lack of time on the part of practicing agencies 5. Information Hub Website has at least 50 Organizations as registered on the portal; the MERS document has been downloaded at least 500 times (by unique visitors) Handbook downloads: Total: 2,500 (registration was removed to improve download numbers) Achieved Overall, the evaluation finds that the expected results have been substantially achieved, with the exception of Activity 4: the field testing and compilation of lessons learned from the field experiences. 7 Findings The following sections contain the findings on each activity in the framework, evaluated from the DAC Evaluation Criteria perspective. Activity 1: Training and Capacity Building SEEP offers two different courses, a two-day course for General Practitioner, and a four-day course to become a Trainer of Trainers. Both lead to certificates. At the end of life of project, SEEP has trained 158 practitioners on the Standards. The training has reached a total of 54 leading humanitarian agencies working in over 27 countries. Regional trainings were delivered in Bangkok, Dakar, Beirut, Nairobi, Washington DC, Manila, and Guatemala City (5 regions of the 8 expected). The trainings have been replicated in some form by agencies, for example, Mercy Corps Ethiopia trained an additional 53 practitioners with a half-day training in 2013. An e-course, housed on the IFRC training site, has been launched, and 10 people have completed the course. Relevance of the Training Activity The evaluation concludes that both of the trainings are relevant. This is because:  The number of institutions that have trained their staff is large (54).  The institutions reached are the largest humanitarian institutions in the world (including UNDP, FAO, UNHCR, IFRC, Oxfam, Mercy Corps, Global Communities, Relief International, Catholic Relief Services, among others).  Approximately 30% of trainees were senior level/program or country directors, while 70% were technical livelihoods specialists or operations staff.  The feedback from participants in their evaluation forms about the relevance of the training to themselves and their institutions is positive; comments from participants included the usefulness of the training to their work in recovery, and also to their ability to train others. Examples: “We are working in economic recovery now and we can use the guide in planning, controlling, and even for donors.” (Beirut Training, 2013). Also, “I will apply this learning in proposal development and in a livelihoods project.” (Dadaab Training, 2013) and “I will try to mainstream the MERS standards in the tools used by UNDP in its function of coordination of early recovery, and I will train colleagues and partners.” (Washington, DC Training, 2014) DAC Evaluation Criteria: Relevance The extent to which the aid activity is suited to the priorities and policies of the target group, recipient and donor. In evaluating the relevance of a program or a project, it is useful to consider the following questions: • To what extent are the objectives of the program still valid? • Are the activities and outputs of the program consistent with the overall goal and the attainment of its objectives? • Are the activities and outputs of the program consistent with the intended impacts and effects? 8  The feedback from participants in individual interviews about the relevance of the training to their institutions is positive. One stakeholder indicated that the ToT training was quite useful as it taught her training techniques.  71% of respondents in the on-line survey had used MERS in designing programs, 28% in planning (DRR), 21% in sudden onset, 25% in slow onset, and 17% in other contexts, which indicates that MERS is relevant (multiple answers were possible).  Also from the online survey, on a scale of 1 – 5 to rank MERS relevance, with 5 being “very relevant”, 21% chose “5”, while 30% chose “4”, and an additional 30% chose “3”. This indicates that respondents feel that MERS is quite relevant to their work. There are a few caveats with regards to the relevance of the trainings. SEEP has noted in several reports that the demand for the training has been “much lower” than expected. Since the demand for the Standards themselves seems to be high, judging by the number of practitioners involved in contributing to them, and since this demand has been maintained for a long period of time (since 2007), it seems likely that there are other factors at work that are constraining the demand for the trainings. This could hypothetically be due to:  An overestimation of demand in the proposal,  A lack of adequate or advance promotion of the trainings,  A lack of adequate involvement in promotion of the training by Steering Committee,  An inadequate promotion of MERS as a companion to SPHERE,  A lack of understanding of the potential use of the MERS in contexts other than post-emergency,  The ongoing challenges of engaging busy staff,  Limited NGO budgets,  A launch of OXFAM’s market standards at the same time, creating confusion or competition,  The sporadic nature of emergencies and disasters, and/or  Other reasons. SEEP may be able to mitigate the first four of these bullet points in a subsequent phase. Also, the more people who are exposed to MERS, the more the demand for trainings should increase. Effectiveness of the Training Activity The effectiveness of the training activity is good. The reasons for this conclusion are:  The trainers are highly effective, experienced training professionals, who are also trainers of the Emergency Market Mapping Assessment program, among others. This opinion is corroborated in the MERS Training Participant Evaluations. DAC Evaluation Criteria: Effectiveness A measure of the extent to which an aid activity attains its objectives. In evaluating the effectiveness of a program or a project, it is useful to consider the following questions: • To what extent were the objectives achieved / are likely to be achieved? • What were the major factors influencing the achievement or non-achievement of the objectives? 9  The number of people trained (158) is good for a 30-month program.  The design and presentation of the training materials are both rated as very good by the majority of participants in all trainings. These training materials have undergone several revisions to improve them.  The fact that there is a general practitioner (GP) course offered is a good way to make practitioners and busy policy-makers and donors aware of the MERS without the obligation of becoming trainers; this is the same model followed by the EMMA initiative.  The training has led to the accreditation of 44 trained trainers in MERS.  The trainings have been replicated internally within relief organizations, such as Mercy Corps Ethiopia and ChildFund Kenya.  The indicators for this Activity have been substantially achieved (as seen in Table 1). There are a few caveats with regards to the effectiveness of the trainings. These caveats stem from observations of training participants, either via the on-line survey, the participant evaluation forms, or individual interviews. A common observation is the lack of examples of practical application of the MERS and lessons learned from the field. This is complicated by the conceptual nature of the Standards; they are not a tool but rather a set of standards. For examples, participants have asked for “more case studies and lessons learned”, “more practical examples”, “simulations”, “more field examples”, “more case studies”, and “more time for exchange of field scenarios among participants.” One individual interviewed stated that the Standards are hard to apply (given their conceptual nature), and that it would be useful to have more examples/case studies from the field. Impact of the Training Activity OFDA and SEEP are interested in the impact of the training on the humanitarian response field, in particular, whether the training leads to the use of MERS in humanitarian response/early economic recovery contexts. The survey shows positive impact from the training. Survey participants indicated that:  71% had used it in designing programs, 28% in planning (DRR), 21% in sudden onset, 25% in slow onset, and 17% in other contexts. (multiple answers were possible).  When asked “How comfortable do you feel using the tool?”, 38% indicated “very” and 48% indicated “comfortable”. DAC Evaluation Criteria: Impact The positive and negative changes produced by a development intervention, directly or indirectly, intended or unintended. The examination should be concerned with both intended and unintended results and must also include the positive and negative impact of external factors, such as changes in terms of trade and financial conditions. When evaluating the impact of a program or a project, it is useful to consider the following questions: • What has happened as a result of the program or project? • What real difference has the activity made to the beneficiaries? • How many people have been affected? 10  To the question “How well did the training prepare you to use MERS in a field context?”, 48% responded “very well”, and 48% responded “ok”. This indicates that some improvements could be made in the training content or presentation. In addition, trainees had used MERS in the following contexts (verbatim comments): Application during design of recovery programs It helped us in planning and designing our programs Shaped our economic recovery program; more holistic consideration of economic issues The targeting of beneficiaries Very useful in supporting DRR strategies Very useful in program design and monitoring and evaluation very useful for planning market-based programs I trained the other staff and shared the MERS handbook with them. We embrace MERS in all our programming work. I did roll out as part of DRR and response planning parts of MERS and later during recovery planning The Targeting of beneficiaries, importance of monitoring and assessment, importance of initial, study of markets, taking in to account the local context, etc. Stakeholders interviewed during the course of the evaluation had the following suggestions with respect to improving the training’s impact:  The training did not have enough practical examples from the field. This meant that trainees were not clear on how and when to use it, which made the financial and opportunity (time) costs of the training hard to justify.  There could be more clarity in the training on when to use MERS and when to use other SPHERE standards.  We are including local partners in our trainings in the regions, and we are developing MERS modules to include in our broader internal trainings.  We couldn’t really put the Standards into practice because there is no funding/support from donors, unlike Sphere training, which is supported by donors. From the stakeholder comments, it seems clear that the impact of the MERS training is constrained:  Lack of awareness of and commitment to the Standards by donors (lack of funding, not included in RfP requirements, etc.).  Lack of practical examples of field experiences. 11 Sustainability of the Training Activity The Handbook and other materials are provided free of charge. The e-course is free. Neither of these activities therefore contributes to financial sustainability, nor should they, since they are a public good. There have been fees for the ToT and GP trainings. Of the training participants, 92% were sponsored by their employers, which contributes to sustainability and also indicates relevance of the MERS to these agencies. A few consultants paid their own way, and a few participants were not charged for the course (e.g. Sphere focal points in Pakistan, Manila, Guatemala and hosting institutions in Kenya and Washington, DC, as well as to SEEP staff). Thirty per cent of trainees travelled to their training locale, another indication of the relevance of the training. A question was posed in the on-line survey to trainees regarding the value of the training to them and their agencies. Their responses were:  Regarding the cost of the training, 74% stated that it was worth the cost, while 22% stated that they didn’t know the cost. While this indicates that the trainees feel that they received value for money, it is not a clear indication that they would pay individually for the training. However, their institutions felt that it was relevant. Recommendations for Activity 1 1) Utilize the Steering Committee members to promote the Standards and the trainings. 2) Broaden the Steering Committee membership to other organizations, especially donors, so that the Standards and the trainings will be promoted internally in those new organizations. 3) Advertise the trainings more broadly and via more venues (communities of practice, such as CaLP and EMMA, conferences, etc.), and promote it as a SPHERE companion. 4) Strengthen the link between MERS and LEGS in the introduction of the training, to build on cross￾selling possibilities. 5) Review the LEGS training strategy to see if there are lessons that can be learned for more effective uptake of MERS trainings. DAC Evaluation Criteria: Sustainability Sustainability is concerned with measuring whether the benefits of an activity are likely to continue after donor funding has been withdrawn. Projects need to be environmentally as well as financially sustainable. When evaluating the sustainability of a program or a project, it is useful to consider the following questions: • To what extent did the benefits of a program or project continue after donor funding ceased? • What were the major factors which influenced the achievement or non-achievement of sustainability of the program or project? 12 6) The MERS training would benefit from lessons learned from field examples. These could be context￾and sector-specific to give trainees a broader understanding. This recommendation is also addressed under Activity 4: Field Testing. 7) MERS can be considered a public good and therefore should be provided free of charge, since the end result is to assist those affected by humanitarian, climate or political crises. The SC should fundraise for this purpose. Activity 2: Establishing a Steering Committee There is a terms of reference for the Steering Committee. At the end of the project, the Steering Committee has met 4 times. Two meetings were held via conference calls and two in-person steering committee meetings were held alongside the SEEP annual conference in 2013 and 2014. The Steering Committee includes 7 very active members, who are in the process of providing funding and in-kind contributions to ensure the continuity and sustainability of the initiative. There are minutes from each SC meeting. Steering Committee organizations have included CRS, Mercy Corps, UNHCR, IFRC, SPHERE Project/InterAction, IRC, and Relief International. The SC is thinking about the future of MERS - Steering Committee members approved a Concept Paper and a Strategic Plan, and are currently in the process of assisting SEEP to find additional funding for the next stage of the project. As well, SC members agreed in their last meeting to elect a Chair and to provide new names and organizations to expand the existing Steering Committee. Relevance of the Steering Committee The SC appears to have been relevant from the perspective of guiding and approving the MERS Initiative during the two-year project timeframe. The SC has reviewed program activities, identified industry needs and trends, promoted MERS in conferences and other public/practitioner venues, promoted MERS within their own organizations, and other activities. Effectiveness of the Steering Committee However, the SC has acted mostly as an advisory body and less so as a steering body. This is due to a combination of the following factors:  The MERS project was pre-defined for the 2 years of its life and needed oversight rather than “steering” during that time;  SC members are very busy, and sometimes absent (sometimes providing agency substitutes to attend meetings), which limits their involvement;  SC meetings are infrequent, which limits involvement;  The life of the project is short (30 months), which reduces the need for strategic planning; and  The ToR for the SC indicates that it is an advisory body. 13 These factors have limited the SC activities to primarily oversight, according to interviews of SC members. However, now at the end of the project, the SC members acknowledge the need to become more involved in the future of MERS. They have agreed to change the ToR of the SC to allow it to be more active, to fundraise and/or provide agency funding, to elect a Chair and Secretary, to network and liaise with other organizations to promote the Standards, and to broaden the membership of the SC. From SC meeting minutes and interviews with SC members, it seems they are clear on the need to become more involved in the continuity of MERS. Some comments from the SC member in interviews support these findings:  The SC needs to clarify its purpose, membership, and workplan (this is in process); The ToR need to be revisited;  The SC needs to help SEEP in identifying funding opportunities;  SC members should take on the responsibility to co-fund the MERS team and its activities;  SC members need to be more active in strategic planning, networking, and establishing partnerships;  SC membership needs to be broadened to include the UN agencies, DFID, ECHO, CALP, and others; and  In the past, the SC oversaw the product development of MERS; now the SC needs to go beyond that and promote the use and the institutionalization of MERS. The SC minutes from 4 Nov 2014 reflect these opinions:  Two SC members volunteered for the positions of Chair and Secretary;  The reference in the ToR for the SC to be “advisory, not steering” should be removed when revising the ToR;  SEEP needs to define the ToR for Steering Committee including roles and responsibilities for Chair and Secretary;  A log frame will be added to the concept note;  SEEP will follow up with each SC member on possible contributions towards filling funding gap; and  Steering committee members will send SEEP the names (nominations) of new members to invite to the Steering Committee. Impact of the Steering Committee The SC has finalized the development of the Standards and has raised awareness about MERS. SC members have spoken about MERS in conferences and provided some funding to the initiative. The SC has developed a strategic plan and a workplan. In terms of impact of the SC on the use of MERS in agency programming (the use of the Standards in post-disaster early economic recovery), this has been limited mostly to the agencies whose staff are on the SC. 14 Sustainability of the Steering Committee The work of the Steering Committee is completely voluntary and is therefore unsustainable. It is appropriate that this is voluntary, although it may impede members’ ability to participate fully, given that they are full-time staff at their institutions. Recommendations for Activity 2 1) The SC and SEEP should continue along the path already in process:  Broaden the SC membership, including both technical and policy levels;  Raise funds for the continuity of MERS;  Revise ToR and create roles (Chair and Secretary);  Develop tasks for individual SC members to include fundraising, internalizing MERS in their own agencies, promoting MERS in conferences and forums, and compiling field experiences. Activity 3: Outreach and Awareness Building By end of the project, SEEP had presented MERS at the following events:  World Conference of Humanitarian studies to an audience of over 50 individuals at the Panel Why Do Humanitarian Standards Matter?  SEEP Annual conference 2013  Markets and Crisis Event in New York City (April 2014)  SPHERE Board Meeting New York City (May 2014)  SEEP Annual Conference 2014 SEEP has also held meetings with key donor agencies to ensure MERS is included in donor guidelines. Currently MERS has been incorporated into the Livelihoods Strategy of the United Nations High Commission of Refugees (UNHCR), and the State Department Office of Population, Refugees and Migration (PRM) has also agreed to include MERS in its donor guidelines. It is referenced in the USAID OFDA Guidelines. Relevance, Effectiveness, and Impact of the Outreach and Awareness Building Activity Now that the second edition of the Standards has been published and included as a SPHERE companion standard, the importance of outreach and awareness building becomes more critical so that more agencies are applying the Standards and more donors are requesting or requiring them. Clearly the SC has a great role to play in this, as does SEEP’s own network. The SC has not been active in this role, but this seems to be changing (see Activity 2, above). SEEP has promoted MERS to its own network of members, via its website, and at its conferences. There is a MERS in Action webpage with video testimonials about MERS applications. The MERS Handbook appears on the SPHERE website. The earlier edition (2009) 15 appears on the Food Security Cluster website, and it appears on the USAID Microlinks website. The MERS e-course is on the IFRC learning site. The MERS Handbook does not appear on the Humanitarian Response website maintained by UN OCHA, nor on the UN Early Recovery Cluster list of tools and standards. SEEP maintains a database of people who have contributed to MERS, been trained in a MERS training, and/or are on or have been on the Steering Committee. This database is used for periodic updates on MERS. One constraint to the promotion of MERS is the limited project budget and the limited number of staff involved in the project (1 full-time SEEP staff member).2 Another constraint is that SEEP’s membership tends towards development practitioners rather than emergency response practitioners. Stakeholders interviewed mentioned some institutions that did not appear to be actively using MERS. Since this is an outsider perspective, this information should be taken with a grain of salt:  IRC  DFID  ECHO/EU  UNDP  OXFAM  UN clusters From the online survey, we find:  Nearly 46% of trainees had become aware of MERS between 1 – 3 years ago, while 33% had become aware of MERS within the last year.  46% had become aware of MERS through word of mouth or email communications, while 42% had become aware of it through conference or seminar presentations. This indicates that the MERS outreach and awareness raising is functional. In any case, it is certain that MERS could be more widespread internally in many agencies in addition to the ones listed above. A number of stakeholders commented that they were the only ones trained in their institutions in MERS, and/or that it was not widely used in their agencies. Importantly, it seems that many donors are less aware of the Standards (compared to practitioners), and do not refer to it in their RfPs or policy documents. Consequently, this is a disincentive for response agencies to apply the Standards. 2 The SEEP infrastructure was of course advantageous to the MERS project, even if the number of staff was limited. This created efficiencies. More needs to be done to articulate MERS to different audiences, especially in the area of cash transfers. How can MERS be used in different contexts? - Excerpt from stakeholder interview (not a direct quote) 16 Sustainability of the Outreach and Awareness Building Activity This activity does not generate a revenue stream and is therefore unsustainable. Funding will have to be acquired to continue this activity. It is appropriate for purposes of buy-in that some percentage of this funding come from agencies on the SC. Recommendations for Activity 3 1) In the next phase of MERS, it would be strategic to concentrate on raising donor awareness of MERS. 2) SC members can assist with outreach, and can ensure that their own agencies are applying MERS. 3) Practical examples of the use of MERS in programming in different contexts can provide incentives for agencies to adopt MERS. 4) The SC should ensure that funds are raised for this activity. Activity 4: Field Testing Relevance of the Field Testing Activity Field testing is very relevant to the roll-out of the Standards. Interviews with stakeholders attribute a certain lack of interest in using the Standards to the lack of evidence, sort of a chicken and egg dilemma. One stakeholder noted that it’s necessary to “build the case for stakeholders”, to showcase how it has been used to date, how it adds value to programming, how it influences donor behavior, and how it improves accountability in programming. Another stakeholder suggested that MERS needs a detail￾specific evidence base, with case studies and lessons learned. Effectiveness of the Field Testing Activity According to SEEP reports (Sept 2014 Quarterly Report), the following agencies have indicated interest in compiling lessons learned from the application of the Standards in programming: CRS, Mercy Corps, Relief International, UNDP, and UNHCR. However, field testing activities have been limited, and no case studies or lessons learned have been compiled to date. The original program design which did not allocate any funding to Field Testing, and saw it as a voluntary activity which rendered field testing unfeasible. At the same time, “Despite repeated discussions and email exchanges, it has been very challenging to get organizations to commit to field testing on a voluntary basis. The major issue at stake is the lack of time of field practitioners to document these lessons learned. Even in instances where programs have budget and personnel for knowledge sharing, sharing of stories and examples of good practice from the field prioritize internal sharing/or sharing on the organization’s own website, as opposed to sharing externally. SEEP anticipates that the best way to overcome this limitation is to have local learning events, or assigned resources for field testing.”3 3 SEEP Quarterly Report 2014. 17 One example of lessons learned from field experience comes from CRS and has been presented to the SEEP Network.4 The MERS presented the following challenges to conforming to the standards for the assessment team: - Market assessments focused more on input markets than output markets. Did not do in-depth assessment of markets’ ability to absorb increased vegetable production (Standard 1, Productive Assets) - Due to time and communication constraints, slightly different assessment tools and procedures were implemented. (Standard 3, Assessment and Analysis Standards) - Results of market assessment not written-up and disseminated. (Standard 5, Assessment and Analysis Standards) - The project facilitated short-term access to privately and/or community-owned lands for those without previous access. However, the project did not address longer-term land tenure issues. This has implications on sustainability of project-supported livelihoods. (Standard 1, Productive Assets Standards) - The project provided training on vegetable production but did not facilitate market linkages. (Standard 3, Productive Assets Standards) This is an example of lessons learned from field experience that has been and should be shared with other practitioners. Impact of the Field Testing Activity Since there has been no field testing by SEEP, there has been no impact. Sustainability of the Field Testing Activity The field testing activity requires subsidies and is therefore not sustainable. Recommendations for Activity 4 1) A simple way to begin to share experiences from the field would be to use the SEEP database. This would become sort of a mini Community of Practice. It would need to be moderated for some time, in order to stimulate discussion and debate. It could also request publications from agencies that have written up their experiences. These publications could be housed at the MERS in Action webpage. 2) Funds should be acquired and set aside for field testing with agencies (these could be SEEP funds used to match agency funds). Funds should also be acquired and set aside for consultants to compile lessons learned post-assessment. 4 How MERS Standards Influenced CRS’ Response. Dina Brick and Megan McGlinchy, CRS. 2014. 18 Activity 5: Establishing an Information Hub The MERS in Action website includes news, video testimonials, a blog roll, and a resource library. It also includes information and testimonials on the training program. The website is periodically updated with new content. Relevance, Effectiveness, and Impact of the Information Hub The relevance of the information hub is as a repository of free MERS publications, including evidence and training materials. It continues to be relevant as part of awareness raising about the Standards. The MERS Handbook is on the site, and it has been downloaded over 2,500 times, making it, according to SEEP, “the most downloaded item from the SEEP website.” The site serves as a one-stop-shop on MERS. Sustainability of the Information Hub The Hub does not generate a revenue stream, and therefore will need to be subsidized. Recommendations for Activity 5  Funds should be acquired and set aside for maintaining and expanding the Hub. 19 Summary of Survey Results The following section is a summary of the results of the on-line survey with MERS trainees. The complete survey summary report is found in Annex 6.  62% of respondents were INGO staff, while 4.2% were donor staff. This indicates that more donors need to be reached with training.  Slightly less than half had taken the GP training, and slightly more had taken the ToT training.  Nearly 71% of respondents work in development, 50% in humanitarian response, and 29% in training (multiple answers were possible).  100% of respondents were aware of MERS.  Nearly 46% had become aware of MERS between 1 – 3 years ago, while 33% had become aware of MERS within the last year. This indicates that the MERS outreach and awareness raising is functional.  46% had become aware of MERS through word of mouth or email communications, while 42% had become aware of it through conference or seminar presentations. This also indicates that the MERS outreach and awareness raising is functional.  17% of respondents indicated that they had not used MERS after the training. 71% had used it in designing programs, 28% in planning (DRR), 21% in sudden onset, 25% in slow onset, and 17% in other contexts, which were listed as “capacity building”, “referenced in policy and guidance publications” and “referenced in policy, guidance and assessment reports.” (multiple answers were possible).  37.5% indicated that MERS is used consistently in their organizations when applicable, while the majority (62.5%) said that it was not. This is an area for further outreach and awareness building.  As to other tools or standards used in addition to or instead of MERS in post-crisis economic recovery, the list included CBDRR, EBDRR, CMDRR, CaLP, EMMA, M4P, HAP, Sphere project, HEA, Post Disaster Needs Assessment (PDNA) Recovery Framework, and general market assessment tools. EMMA and SPHERE were mentioned most frequently.  On a scale of 1 – 5, with 5 being “very relevant”, 21% chose “5” for MERS relevance, while 30% chose “4”, and an additional 30% chose “3”. This indicates that respondents feel that MERS is quite relevant to their work.  When asked about the practicality of MERS on a scale of 1 – 5, with 5 being “very practical”, 17% chose “5”, 35% chose “4”, and 35% chose “3”.  When asked “How comfortable do you feel using the tool?”, 38% indicated “very” and 48% indicated “comfortable”.  When asked for information on which agencies were not using MERS, the respondents were not able to answer.  Regarding the cost of the training, 74% stated that it was worth the cost, while 22% stated that they didn’t know the cost.  To the question “How well did the training prepare you to use MERS in a field context?”, 48% responded “very well”, and 48% responded “ok”. This indicates that some improvements could be made in the training content or presentation. 20  When asked “What has been the most valuable aspect of MERS for you personally?”, most respondents indicated that it was the existence of the Standards that was valuable.  100% stated that they would recommend the Standards to a colleague. This shows that the trainings are a valuable mechanism for MERS outreach strategy.  87% indicated that they would like to participate in a CoP. When asked about the benefit of the training to their organizations, the respondents gave the following answers (the blue-shaded lines indicate an increase in awareness, while the grey-shaded lines indicate impact of MERS trainings on programming): Application during design of recovery programs Increased scope of training It gave me good ideas for training our own staff and to help me understand it better It helped us in planning and designing our programs It was not very applicable to the context of the Middle East where I received the training Raising awareness of the standard and how it should be integrated into recovery programming Shaped our economic recovery program; more holistic consideration of economic issues The Targeting of beneficiaries Very useful in supporting DRR strategies Very useful in program design and monitoring and evaluation. framework for common understanding of Economic recovery haven't had to use it yet to have that expertise on me and second to the organisation very useful for planning market-based programs UNHCR livelihood operational guidance documents make reference to MERS. The training help me draw the linkages and synergies of the tools. The training me to understand the expected minimum requiring for an effective recovery programme. MERS also draws linkages to the SPHERE tool used in emergency field operations I trained the other staff and shared the MERS handbook with them. We embrace MERS in all our programming work. This helped me as the head of programmes to have a better understanding of MERS and to be able to use the tool The training is definitely useful if the organisation would decide to go down the route of using MERS. It would have been of more benefit if we did more humanitarian and post-emergency work. But we are extremely development-focused. I did roll out as part of DRR and response planning parts of MERS and later during recovery planning The Targeting of beneficiaries, importance of monitoring and assessment, importance of initial, study of markets, taking in to account the local context, etc. When asked how they would improve MERS, a number of respondents mentioned field testing, some mentioned sharing experiences via a community of practice, lessons learned, and adapting to different contexts. 21 Program Efficiency One of the DAC criteria is Efficiency, which is best addressed from the perspective of the program level rather than the activity level. The ability to evaluate efficiency in this program is limited, because there are no alternatives to compare it to. Within the methodological limitations, it seems that the project has been implemented in an efficient manner. The activities were carried out within a 31-month time frame, which is efficient, especially since the e-learning course was added on as an activity later in the project. The funds were spent appropriately and seemingly in a timely manner, with an unobligated balance of $7,873 for the evaluation remaining at Jan 31, 2015. The program was supposed to earn $40,379 in income from training, and was able to earn $38,291, or 95% of expected. Program Management There are 2 questions under this section: 1) Has SEEP been a good project manager? 2) Should SEEP continue to house the MERS repository of information and lead the ongoing MERS roll-out? SEEP as project manager – By all accounts, SEEP has been a good manager of the project. The majority of the planned activities have been rolled out in a timely manner, and the objectives have also been achieved in the majority. The SC has been formed and coordinated well by SEEP in planning, providing information, taking minutes, distributing minutes, and communicating with members. Stakeholders interviewed were nearly uniformly positive in their feedback on SEEP’s management. SEEP has promoted MERS to its own network members and beyond. SEEP as host of MERS – The question arises as to whether housing MERS within SEEP is advantageous or not. On the one hand, SEEP has its membership to reach out to with MERS. SEEP also has the infrastructure to house MERS activities. On the other hand, are SEEP’s mandate, and its focus on development rather than humanitarian assistance, limitations to the expansion of MERS? Most stakeholders interviewed are of the opinion that SEEP is an appropriate place to house MERS for the next few years. SEEP is seen as a strong organization, capable of supporting the SC in carrying out the MERS strategic plan. Acting together, the SC and SEEP should be able to reach out beyond SEEP’s membership to a broader community. DAC Evaluation Criteria: Efficiency Efficiency measures the outputs -- qualitative and quantitative - - in relation to the inputs. It is an economic term which signifies that the aid uses the least costly resources possible in order to achieve the desired results. This generally requires comparing alternative approaches to achieving the same outputs, to see whether the most efficient process has been adopted. When evaluating the efficiency of a program or a project, it is useful to consider the following questions: • To what extent did the benefits of a program or project continue after donor funding ceased? • What were the major factors which influenced the achievement or non-achievement of sustainability of the program or project? 22 Recommendations for the Future of MERS 1) Continue to house MERS in SEEP for the next few years until MERS becomes a “household name” in early economic recovery practice. The Value Added from Being a SPHERE Companion Standard SPHERE has been involved in the development of MERS since the first working group happened. This was important, as it helped guide the creation of MERS in the format of a SPHERE companion standard. In the last 2 years of the MERS project, the relationship between SPHERE and MERS (and other companion standards) has gone from a bilateral relationship to an alliance. This has been very useful, since it allows better efficiencies – sharing of evaluation ToRs, joint presentations, and so on. The relationship between MERS and LEGS has also been valuable, since both are livelihoods-related. The formation of a formalized alliance of all the standards, now in process, will help increase awareness of the importance of standards as a concept to improve accountability and impact, and will help raise awareness of the individual standards as well. SPHERE representatives interviewed during the evaluation feel that MERS is very valuable, and that it needs further support and more recognition. One suggestion was to create more ties with CaLP. Conclusions The program has been well-managed over the course of its 30 months by SEEP. The objectives have been achieved for the most part. The major constraints have been the limited number of staff on the project, no funding for field testing of the Standards, and the short time period of the project. These constraints have limited the impact of MERS. Awareness of MERS has increased over time, due to several of the activities. More people and institutions are trained now in MERS, and several donors are referencing MERS in their policy and programming documents. The relevance of MERS is clear, and there is still a need to continue outreach, awareness raising, and training. The engagement of the SC is growing, and this will be critical to seeing MERS applied more frequently and consistently in the field. MERS should be promoted further as a SPHERE companion standard, as this will increase its outreach, and should look for opportunities to link with CaLP. More funding is needed to support further extension of these activities, which have proved to be effective, except for the field testing, which was not funded. This also needs to be funded. 23 Annexes Annex 1: Scope of Work Impact evaluation: Sustainable Economic Recovery after Crisis: Promoting the use and adoption of Minimum Economic Recovery Standards Introduction As part of the USAID/OFDA funded program “Sustainable Economic Recovery after Crisis: Promoting the use and adoption of Minimum Economic Recovery Standards”, the SEEP Network is carrying out an impact evaluation to determine the level of adoption of the standards by organizations and practitioners involved in humanitarian situations. The output of the evaluation will inform OFDA/USAID as well as MERS program management and other program stakeholders. Description of the Program Background The SEEP Network hosts the Minimum Economic Recovery Standards (MERS) Initiative. The origins of the Minimum Economic Recovery Standards (MERS) are rooted in past crises, such as the Indian Ocean Tsunami, the earthquake in Haiti, and complex humanitarian emergencies and protracted crisis in Afghanistan and the Horn of Africa. Several of SEEP’s members working in humanitarian emergency response identified the need for an open space for practitioner’s to share knowledge on the challenges and emerging practices of economic recovery in crisis environments. In 2007, with funding from USAID through the FIELD Support-LWA mechanism, SEEP convened a task force to develop the first draft of the economic recovery standards and hosted a workshop to launch the minimum economic recovery standards consultative process in Washington D.C. This process was made up of a broad consortium of practitioners from around 30 international humanitarian agencies. Each section of the MERS Handbook was developed by a practitioner-led working group supported by SEEP that brought together experts from economic development and humanitarian relief. In 2009 and 2010 SEEP held regional consultations on the Minimum Economic Recovery Standards in East Africa, Latin America, the Middle East, Europe and South East Asia. The Standards were later field tested by Mercy Corps, ACDI VOCA, Catholic Relief Services, and pilot training courses were implemented in Indonesia and Jordan, with consultations in Ecuador, Kenya and the UK. In 2010 MERS reconvened its original contributors as well as new members from academia, NGOs, research groups, donors and international organizations. A second edition was published in 2010 including the input and feedback of more than 200 practitioners from 63 organizations. 24 In 2011 the MERS was accepted as a companion to the Sphere Handbook. Sphere is the most recognized standard in humanitarian response. Alongside the other companion standards, Livestock Emergency Guidelines and Standards (LEGS), and Minimum Standards for Education in Emergencies (INEE), these minimum standards provide in depth technical guidance and harmonized standards to organizations working in crisis environments, beyond the basic life-saving standards included in Sphere. In late 2012, with support from the USAID’s Office of Foreign Disaster Assistance, SEEP launched a program to promote greater awareness and use of MERS. The program is called Sustainable Economic Recovery after Crisis: Promoting the use and adoption of Minimum Economic Recovery Standards. According to the Monitoring and Evaluation Framework the project is broken down into two major objectives: 1. Increase awareness and adoption of the Minimum Economic Recovery Standards by practitioners/organizations involved in humanitarian situations 2. Improve humanitarian response to crisis by increasing the institutionalization of the Minimum Economic Recovery Standards in emergency response programming. The project activities are: 1. Training and capacity Building 2. Establishing a Steering Committee 3. Outreach and awareness raising 4. Establishing an information hub Project Start Date: June 1, 2012 End Date: June 1, 2014 Current Status of Activities: Activity 1: Training and Capacity Building At the end of life of project, SEEP has trained 146 practitioners on the Minimum Economic Recovery Standards (MERS). SEEP offered two different courses each leading to a certificate. The two day course granted a certificate of General Practitioner, and a four day course including a rigorous accreditation process that awarded Trainer of Trainers certificates. SEEP certified a total of 44 Trainer of Trainers. The initial Monitoring and Evaluation framework aimed to certify 80 TOT Trainers through 4 TOT trainings with 20 participants, and provide direct trainings to 100 Practitioners through 4 direct trainings with 25 participants each. The actual turnout per training has been much lower than originally anticipated. Our trainers have gone on to train an additional 58 more practitioners indirectly. Mercy Corps Ethiopia trained an additional 53 practitioners but with a half day training in 2013. The training has reached a total of 50 leading humanitarian agencies working in over 20 countries. Regional trainings were delivered in Bangkok, Dakar, Beirut, Nairobi, Washington DC and Manila. 25 Activity 2: Establishing a Steering Committee At the end of the project, the Steering Committee has met four times. Two meetings were held via conference calls and two in person steering committee meetings were held alongside the SEEP annual conference in 2013 and 2014. The Steering Committee includes 7 very active members, who are in the process of providing funding and in kind contributions to ensure the continuity and sustainability of the initiative. Steering Committee members approved a Strategic Plan, and are currently in the process of assisting SEEP to find additional funding for the next stage of the project. Steering Committee organizations include Catholic Relief Services, Mercy Corps, United Nations High Commission of Refugees (UNHCR), International Federation of the Red Cross (IFRC), Relief International, and the International Rescue Committee. Steering committee members agreed in their last meeting to elect a Chair, and to provide new names and organizations to expand the existing Steering Committee. Minutes are available for all meetings. Activity 3: Outreach and awareness building At end of the project, SEEP presented MERS at the following events:  World Conference of Humanitarian studies to an audience of over 50 individuals at the Panel Why Do Humanitarian Standards Matter?  SEEP Annual conference 2013  Markets and Crisis Event in New York City (April 2014)  SPHERE Board Meeting New York City (May 2014)  SEEP Annual Conference 2014 SEEP has also held meetings with key donor agencies to ensure MERS is included in donor guidelines. Currently MERS has been incorporated into the Livelihoods Strategy of the United Nations High Commission of Refugees (UNHCR), and the State Department Office of Population, Refugees and Migration (PRM) has also agreed to include MERS in its donor guidelines. Activity 4: Field Testing At least 6 organizations have been identified to perform Field Testing. Activities are underway in Philippines, Kenya, Ethiopia and Myanmar. Field testing activities have been limited. The original program design which did not allocate any funding to Field Testing, and saw it as a voluntary activity which rendered field testing unfeasible. Activity 5: Establishing an information hub The MERS in action website is continually updated with new content and the MERS handbook has been downloaded over 2,500 times making it the most downloaded item from the SEEP website. The MERS in action website includes news, a blog roll, and a resource library. It also included information and testimonials on the training program. 26 The following project documents will be available for evaluation team:  Monitoring and Evaluation Framework  MERS Proposal  Steering Committee Meeting minutes  Quarterly Reports to USAID/OFDA  Results of training course evaluations Purpose The purpose of the evaluation is to have a clearer picture of the level of awareness of the Minimum Economic Recovery Standards in the humanitarian sector. Secondly the evaluation seeks to get feedback from practitioners that have been exposed to the MERS, about the adoption and use of the MERS by their organizations. The evaluation will serve as a key input for MERS program management and Steering Committee to inform future activities, and recommendations for improvement in terms of outreach and impact of MERS. The evaluation needs to identify which organizations have adopted and are using MERS. In the case of organizations that are not using MERS, the evaluation should explore why not. And in the case that they are using MERS the evaluation should gather feedback from end users of MERS in terms of its relevance, practicality and ease of application. Methodology We propose the following methodology: 1. Review of project documentation 2. Survey: design a survey targeted at Humanitarian Aid practitioners to evaluate their knowledge and understanding of MERS 3. Conduct key informant interviews with trainees/MERS graduates 4. Conduct interviews with key informants Deliverables  Initial work plan  Survey  Final study with impact evaluation results  Debriefing to MERS project Staff Level of Effort and Budget Total budget: $7800 Expected level of effort: 12 days @ $650 27 Annex 2: Bibliography SEEP MERS Evaluation Sustainable Economic Recovery after Crisis: Promoting and Using the Minimum Economic Recovery Standards: Proposal to USAID/OFDA (Revised version for Agreement Modification submitted by The SEEP Network) 01/28/2014. Quarterly reports to USAID, beginning April 30, 2013 through Sept 30, 2014. Concept Note MERS 2014. SEEP MERS Strategy 2014. MERS 2014 – 2020 Strategy Paper. 2014. MERS 2010 Handbook MERS Training Materials MERS Training Participant Evaluations MERS Steering Committee Minutes MERS Training Participant Certificates MERS M&E Framework (modified) 28 Annex 3: Survey Tool MERS Survey Questionnaire 1. Are you a: 1.1. Donor staff 1.2. INGO staff 1.3. Consultant (independent or firm) 1.4. Academic Other (please describe) 2. Is your field of expertise: 2.1. Humanitarian assistance 2.2. Economic recovery/markets 2.3. Training 2.4. Other (please describe) 3. Are you aware of the MERS? 4. When did you become aware of it? (within the last year, 1 – 3 years ago, 4 – 7 years ago) 5. How did you become aware of it: 5.1. Informal word of mouth 5.2. Conference or seminar presentation 5.3. Website or listserve 5.4. Print media 5.5. Other (please describe) 6. Where has MERS been useful to your organization (choose all that apply): 6.1. Has not been used 6.2. Designing programs 6.3. Planning (DRR) 6.4. Sudden onset crisis 6.5. Slow onset crisis 6.6. Other 7. Is MERS used consistently by your organization (in nearly every circumstance for which it was designed)? Yes/no 8. What other tools or standards are used by your organization in addition to or instead of MERS for post-crisis economic recovery? (please write in) 9. How relevant do you find MERS to be to your organization? (scale of 1 – 5) 10. How practical is MERS to your organization? (scale of 1 – 5) 11. How comfortable do you feel using the tool? (scale of 1 – 5) 12. Do you know any humanitarian assistance organizations that are NOT using MERS? If so, kindly list: 13. Have you personally received training in MERS? If “yes”, which training did you receive:  2-day General Practitioner course  4-day Trainer of Trainers course  Other course 14. Regarding the cost of the training, please choose one of the following answers: 29 14.1. I don’t know the cost of the training 14.2. The training was worth the cost 14.3. The training was not worth the cost 15. How well did the training prepare you to use MERS in a field context? (scale of 1 – 5) 16. What has been the most valuable aspect of MERS for you personally? 17. Would you recommend MERS to a colleague? 18. What would you recommend to improve MERS? 19. Would you be interested in participating in an online community of practice about MERS? 20. What was the relevance of the training to your organization (please write in a short answer) 21. Would you be willing to be contacted for a phone/Skype interview by the evaluation consultant? 30 Annex 4: Interview Guide SEEP MERS Evaluation – Interview Questionnaire Name of Person Interviewed: ___________________________ Position Title: __________________________________ Organization: ____________ Date: __________ Context 1. How does your institution handle early economic recovery and support to markets after a humanitarian crisis? 2. On a scale of 1 – 10, with 10 the best score, how would you rate your organization’s ability to support early economic recovery? Has this changed in the last 3 – 5 years, and if so, why? 3. What standards or benchmarks are you using to ensure the quality of programming? Is this issue of concern to your organization? Why or why not? 4. Are humanitarian response staff within your organization aware of the MERS? How did they become aware of it, in your opinion? Usefulness/Effectiveness/Satisfaction 1. Which departments/units use MERS? How frequently? How consistently is it used (i.e. how often is it used in circumstances where it could be useful, i.e. post-conflict, post-disaster)? 2. Is MERS being used for quality control or other purposes on designing programs? Where and when? Any constraints to this? 3. Is MERS being used for quality control when doing program assessment or analysis? Where and when? Any constraints to this? 4. Is MERS being used for quality control or other purposes in disaster risk planning? Where and when? Any constraints to this? 5. What are the most valuable aspects of the MERS, in your opinion? 6. What lessons learned can you share with regards to the use of MERS? 7. Overall, how satisfied are you with the MERS? What could be improved? 8. Are there any advantages to MERS compared to other standards? 9. Are there any constraints to the use of MERS? What are they? 10. Is your institution satisfied with the results when they use MERS? If not, why not? 11. Has your institution made any adjustment in the MERS, and if so, why? How effective have these adjustments been? 12. In working with national/local partner institutions, how is the knowledge of the MERS transmitted to those institutions? (probe: Are the local institutions aware of it? Are there constraints to their use of it, and if so, what are these?) 31 Industry-wide 1. How widespread is the concern about early economic recovery in the humanitarian disaster relief community? How has this concern changed over the last few years, if at all? If it has changed, what factors have helped change it? 2. How is the industry addressing the need for quality control and benchmarks in early economic recovery programming? What standards have been developed? What has been the result? 3. How do the other standards compare with MERS? 4. Which organizations are the proponents of MERS? 5. Which organizations, to your knowledge, don’t use MERS? Do you know why? 6. To what extent are donors aware of MERS? Do they find it useful? How are donors supporting the use of MERS? (probe: Are these donors supporting the use of MERS by mentioning it in their RFPs? Training their staff? Other?) 7. Are MERS-trained personnel readily available? Are they competent in MERS? 8. What has been the effect on humanitarian agency programming as a result of MERS? 9. What else needs to be done in order for MERS to become more accepted and more utilized? Steering Committee Members Only 1. How long have you been a member of the SC? 2. Is/was the scope of work of the SC clear? 3. How effective have the meetings of the SC been? What has contributed to its effectiveness (probe for: commitment of members, expertise of members, organizational skills of SEEP, right mix of organizations, punctual meetings, other) 4. Are there organizations that should be included in the SC that aren’t currently? Who are they? 5. What has the SC accomplished? Please give examples. (probe for: “expand the use and usefulness of MERS” responses) 6. What has the SC not been able to accomplish (that it should have or still needs to)? Please give examples. (probe: cluster chairs, donors, linkages, local partners, CoP, governments) 7. Are there sufficient mechanisms for promoting MERS and training people in MERS? (probe: e￾course, videos on youtube, pocket guide, modular online training, different languages, other) 8. Is there a viable strategic plan? 9. Is the SC likely to be ongoing for the foreseeable future? Do you have any recommendations on any aspect of MERS that you would like to mention? Thank you! 32 Annex 5: List of Stakeholders Interviewed SEEP MERS Evaluation SEEP (Implementing Agency) William Wallis – SEEP Network Steering Committee Members Tom Shaw, Catholic Relief Services Emma Delo, International Federation of the Red Cross Julien Schopp, Interaction / Sphere Board Joseph Mariampallai, Relief International Tracy Gerstle, Independent Consultant Donor Laura Meissner, OFDA Other Stakeholders Dina Brick, CRS Emergency Response Team Becky Steenberger, CRS Guatemala Tony Dines, Trainer Rod Snider, Red Cross Alexandra Guyetsky, FAO Aninia Nadig, SPHERE Cathy Watson, LEGS Project Contacted, no response or declined interview: Mike Albu, Practical Action Greg Matthews, IRC Jean-Martin Bauer, WFP Sharon Truelove, humanitarian consultant Anas Roudies, independent Margaret Herro, Pia Wanek, Global Communities Elizabeth Bellardo, InterAction Tregenna Myrabo, WVI Holly Christoferson, SAWSO Bethel Tsegaye, Mercy Corps Tiare Cross, Relief Int. Lloyd McCormick, Childfund Sybil Chidiac, CARE Aysha Twose, independent consultant Sarah Bailey, ODI UK Jesse Fripp, Shorebank Lauren Mitten, DAI Anita Van Brita, WWF Radha Rojkotia, IRC Tzvetomira Laub, INEE Sapenzie Ojiambo, ECHO Nairobi Melissa Scudo, Chemonics Jean-Francois Dubuisson, UNDP Devrig Velly, ECHO EU Susanna Sandstrom, WFP Minja Peuschel, Save the Children