American Red Cross Mid-Term Evaluation Institutional Capacity Building Grant (FY04-FY08) Cooperative Agreement No. AFP-A-00-03-00007-00 September 30, 2006 Draft October 5, 2006 Draft Revised October 18, 2006 Final Draft Evaluation Coordinators: Della E. McMillan, External Consultant Ange Tingbo, Senior Food Program Advisor, ARC Resource Persons: Pat McLaughlin Alice Willard Thomas Carmody Luke Greeves Tracy Reines Svafa Asgeirsdottir ARC ICB Mid-Term Evaluation. October 18, 2006. ii Executive Summary 1.0. Background The goal of the American Red Cross’s (ARC) Title II Institutional Capacity Building (ICB) grant (FY04-08) is to improve the agency’s institutional and technical capacity to design and manage food aid and nutritional interventions as an effective response to emergencies. The grant’s activities focus on achievement of two intermediate results (IRs): • IR1: ARC improves its institutional and technical capacity to design and manage food aid and nutrition interventions as an effective response to emergencies and • IR2: Red Cross/Crescent Movement partners develop the capacity to design and manage food aid and nutrition interventions as an effective response to emergencies. Specifically, this ARC ICB program was originally designed and funded to: (1) refine ARC’s existing systems and tools and develop new systems and tools for use in food programming as a response to emergencies, (2) conduct a pilot program in two countries for the development and testing of systems and tools for use in executing and expanding food programming, (3) disseminate the results of these pilots to Red Cross Movement partners and beyond, and (4) expand and scale-up food programming to ARC bilateral programs in other countries. A small sub-component that focused on promoting the importance and need for food in international emergency responses was also included. The total budget for the ICB program is $2,716,112, of which $1,800,000 has been awarded by USAID with a commitment from ARC of $916,112 (33 percent of the grant total) in matching funds. This report presents the findings of the grant’s mid-term evaluation. The task of assessing the this grant was complicated by the fact that the original M&E Plan was designed during the first year (FY04) before the program was adjusted to accommodate: • The fact that ARC did not have either of the two Title II-funded programs that it expected to have in FY04 (e.g. the funding for one of the Development Assistance Proposals [DAP] that ARC had applied for was withdrawn and ARC withdrew from the grant in the other), which (although funded separately) were planned to be “testing grounds” (simultaneous with the pilot country programs) for the new systems and tools being developed by this grant; and • The dramatic upsurge in activity and staff displacement that accompanied ARC’s active role in coordinating United States food assistance to three countries affected by the Tsunami in FY05. The net result of both the subsequent lack of Title II-funded programs and the massive aid effort for the Tsunami was a shift in the initial focus of the ICB from tools and partnerships that would benefit a Title II food programming office to a larger food audience that included the WFP and the United States Department of Agriculture (USDA), as well as the international Private Voluntary Organizations (PVOs) with active ARC ICB Mid-Term Evaluation. October 18, 2006. iii funding from USAID through Title II. This dramatic shift means that many of the indicators in the ARC ICB project tracking table are less relevant to the program as it is today than what was expected in FY04. For this reason the mid-term evaluation focuses on: • Determining accomplishments to date of the ICB grant based on the proposal plan, modifications to that plan, and any evidence (qualitative or quantitative) to date of project impact; • Determining factors that contributed to (or detracted from) development, utility, and viability of the current and proposed activities under the grant; and • Recommendations for enhancing grantee performance, including modifications to the proposed work plan and monitoring and evaluation system. Sections two and three describe (per IR1 and IR2, respectively) accomplishments to date and evidence of early impact on the quality of Red Cross and other CS food programming; the various factors that contributed to or detracted from development, utility, and viability of outputs; new opportunities and constraints; and priority recommendations for enhancing grantee impact. Section four reviews the project’s management, financial, training, and M&E systems, examines ways that these systems contributed to or detracted from execution of the project during its first half, and makes recommendations for how these could be strengthened. Section five presents the concluding observations, regroups the main recommendations of the analysis into five cross cutting areas1 (see Table 5.1) and makes the distinction between recommendations that require “quick action” that those that require more intensive long-term follow-up, and comments on the future of ARC food programming based on the mid-term findings. 2.0. IR1: ARC Improves its Institutional and Technical Capacity to Design and Manage Food Aid and Nutrition Interventions as an Effective Response to Emergencies 2.1. Output 1.1: Enhanced ARC staff technical knowledge base Sub-output 1.1.1: Improved tools for needs assessment, nutritional surveillance, and planning for emergency interventions. There is evidence that the ICB contributed to consolidation and revision of existing materials, as well as new tool development in several areas that dramatically increased ARC’s “institutional and technical capacity to design and manage food aid and nutrition interventions as an effective response to emergencies,”2 as well as longer term transition and recovery programs. At present, however, many of the tools developed are still in draft form and others are planned for initial development in FY07 and FY08. Access to the tools (through the web) or knowledge about how individual tools complement existing ARC, Federation, or FANTA-endorsed tools (through bibliographies) is lacking. Qualitative information (through interviews) shows that some of the ICB-supported formal training courses 1 Management and finance; strategic planning, monitoring and evaluation and reporting; tools development and distribution; ARC headquarters (HQ) capacity building; and pilot programs and regional disaster response teams (RDRTs) training. 2 Text of IR1 ARC ICB Mid-Term Evaluation. October 18, 2006. iv (primarily the FY06 IPP training), workshop attendance, and one-on-one technical assistance at HQ and in the field have had a powerful institutional impact. To date, however, this impact is difficult to monitor since the project has not developed a system for tracking how many people have been trained in different HQ departments (except for the IPP training),3 nor for measuring core capacity in principal operational units of the International Services Department (IDR and IP). Sub-output 1.1.2: Enhanced emergency response planning that integrates food and nutrition with water/sanitation and other public health interventions. Although the scale of the grant’s impact is greater than anticipated, there is still room for improvement in the resulting food/nutrition proposals. Given the weak understanding of basic food programming concepts and tools by most staff (many of which are newly hired)—both at headquarters and in National Societies—most senior staff who were interviewed during the mid-term feel that activities under this IR should return to the original focus of the grant: on developing core capacity of staff in the operational units of the International Support Department (e.g., International Programs, Tsunami Response Programs, and International Disaster Responses). Once the pilot country programs are launched in FY07, they will provide valuable information on how these lessons can be scaled up into actual programming. Sub-output 1.1.3: Standardized monitoring and evaluation systems to measure the nutritional impact of food in emergencies. Under the grant, ARC was expected to: • Organize a “Vulnerability Assessment and Risk Mitigation Conference” (ARC 2003: 13) (later changed to a “Practitioner’s Forum on Food Needs Assessments in Emergencies”), which was scheduled originally for FY04, then rescheduled for FY05, that would bring together representatives of other Title II Cooperating Sponsors, as well as experts from other agencies (e.g., Sphere Project, WFP, WHO, and UNICEF), to identify what they consider best practices and • Pilot test a number of core indicators in ARC’s own bilateral emergency programs. The idea was to eventually come up with an annotated toolkit. This is one of the few sub-outputs of the grant that has had very little activity. Many staff still feel it is very important. For this sub-output to move forward, however, it needs a wider institutional base that builds on ARC’s pre-existing knowledge and experience with emergency indicators, as well as input from other Title II PVO Cooperating Sponsors. Sub-output 1.1.4 (recommended for addition at mid-term): Stronger bridges and linked understandings between technical units in International Programs at ARC. Bridge￾building between technical units in ARC’s International Services Department is expected to be a major sub-output of the ICB grant. Indeed, one of the major under-documented institutional outcomes of the grant has been to foster this type of bridge-building. This critical role of the ICB in building bridges was highlighted as a major institutional impact of the ICB in almost every one of the HQ-based interviews. To date, however, it is an impact that remains hidden both to ARC administration and USAID by the fact that it is 3 The systems developed by the senior M&E advisor for the IPP trainings are a good model that could probably be “scaled up.” ARC ICB Mid-Term Evaluation. October 18, 2006. v not one of the original sub-outputs (hence the recommendation to add it), nor is it being tracked. 2.2. Output 1.2: Expanded collaboration between ARC and other PVOs [and institutions] in design and management of food and nutrition interventions in emergencies Since FY03 (when the ICB grant was designed), ARC has dramatically increased its interest in and willingness to engage in technical partnerships as a grantee, sub￾contractor, and even donor. This expanded partnership base includes collaboration with: • Individual Title II Cooperating Sponsors in Title II program design, most notably CRS in the production of the ARC/CRS monitoring and evaluation series; • The Wider Title II CS community through the Interaction Evaluation Interest Group (EIG), which ARC has hosted in its headquarters meeting rooms for the last two years; • The World Food Programme and various international PVOs for Tsunami relief on a scale that far exceeded anything anticipated in the proposal ($120 million in food programming through WFP alone); as well as • Major international research centers (e.g., Feinstein Hunger Center and the Academy for Educational Development [AED]) for applied research and trainings. These partnerships have expanded the impact and results of the grant well beyond the original target group, which was primarily countries where ARC expected to have Title II food security programs and pilot programs. The primary identified need is for better tracking of activities associated with these expanded partnerships and their “downstream” impact on the partners’ programming. 2.3. Output 1.3: Established [ARC domestic chapter advocacy network and] 4 support material for promoting the importance and need for food in international emergency responses [among ARC domestic chapters] ARC’s public website is clearly an example of “best practice” in building the US public’s awareness about the role of international PVOs and the US government (through WFP and international PVOs) in responding to international emergency responses. If anything, care should be given to ensure that USAID/FFP staff involved in public outreach and advocacy are aware of the website and associated tracking data. The chief limitation is that, to date, the website does not have a “place” for the more technical tools and reports being developed under the ICB. While technical papers are unlikely to appeal to the population to which the domestic chapter website is geared, they are important to certain audiences that may wish to have a better understanding of the types of technical assistance ARC provides to its international programs. The main recommendation is to create or find some sort of “cyber” home for these technical reports, either on ARC’s public website or elsewhere. 4 Text deleted at the request of USAID/FFP in FY04. ARC ICB Mid-Term Evaluation. October 18, 2006. vi 3.0. IR2: Red Cross/Crescent Movement Partners Develop the Capacity to Design and Manage Food Aid and Nutrition Interventions as an Effective Response to Emergencies 3.1. Output 2.1: Improved knowledge base in National Red Cross Societies to design and manage local food aid and nutrition interventions in emergencies and Output 2.2: Enhanced coordination between National Red Cross Societies and local institutions in food and nutrition during emergencies The genius of the original design of ARC’s proposal was its attempt to link new tools development, training, and pilot testing at the HQ level with improved programming in two pilot programs before scaling up. These pilot programs will enable ARC to improve technical knowledge that will benefit coordination and performance with the Operating National Societies (ONS). Although ARC’s Technical Assistance Unit has provided extensive case-by-case technical assistance to specific countries (see Annex I), they are two years behind execution of the pilot programs. Given the much shorter time (two years) that ARC now has to execute the pilot programs (compared to the four years planned in the proposal) and the current priority being placed on building National Society capacity in Africa, ARC has selected three African countries for pilot projects that are slated to start in FY07: Mozambique, Ethiopia, and Lesotho. Parallel to this activity the senior food program advisor is collaborating with the senior relief advisor (Tracy Reines) in the design of a series of trainings on food security for the southern Africa Regional Disaster Response Teams (RDRTs). The successful implementation of the pilot programs will require close tracking and supervision to ensure proper support and incorporation of “lessons learned” from the experience into HQ-level capacity building. 4.0. Management and Financial Systems 4.1. Management and finance The ICB’s management and implementation history can be classified into three broad periods. Each period was characterized by different management and staffing model, a different set of priorities in terms of existing and potential food security partners, and different Title II Cooperating Sponsor partnerships or partnership prospects. Probably the single most important weakness of the grant’s management that affected both the rate of activity, expenditure, and impact was staff turnover. A major strength was that ARC has internal systems for “tracking” these issues that are rare among international PVOs. ARC ICB Mid-Term Evaluation. October 18, 2006. vii In general the project has had extremely good financial oversight. The chief concerns raised by the ARC ISD program implementation officer are: • High rates of staff turnover, especially in the grant manager’s position; • The fact that the grant is under-spent; and • That there is no clear system for analysis and planning of ARC’s matching contribution to the grant. 4.2. Link between the ICB sponsored training and project management and planning The vast majority of formal and informal (workshop) training programs that were envisioned for FY04 and FY05 were never executed, which contributed to the grant being under-spent. Some of the funds slated for training were used to pay for HQ and field staff to attend training workshops (Annex III, Table 1). Most of the funds are still unspent. The lack of a clear system for documenting how many people at different levels (ONS and HQ staff in different technical units) benefited from the training and workshops (except for the IPP training) and what affect this had on core food programming capacity is a major constraint that makes it difficult to target capacity building programs. Another constraint is that many of the materials that were acquired during staff participation in the different workshops have been lost. 4.3. Link between the ICB monitoring and evaluation and project management Many of the current indicators in the project PITT are less relevant than they were when the ICB was designed due to major shifts in ICB program priorities and activities as discussed above. 5.0. Conclusions and Recommendations 5.1. Quick action recommendations Based on the mid-term analysis, the team identified a list of 32 recommendations for activities that could improve grantee performance during the next two years (Table 5.1). Almost three-quarters of the recommended actions (24 of the 32) are “quick action” recommendations that the project should be able to implement within four months, simultaneous with launching the pilot country programs, because they build on existing initiatives. The “quick action” items (indicated with a “Q” in Table 5.1) need to be resolved so that other activities can move forward at an accelerated rate include. 5.2. Five major categories of recommendations To facilitate follow-up, the 32 recommendations that are discussed in sections three and four of the report are re-grouped into five cross-cutting categories by topic in summary Table 5.1: management and finance; strategic planning, M&E, and reporting; tools development and distribution; ARC HQ capacity building and training; and pilot programs and southern Africa training. Correlating the 32 recommendations (rows in ARC ICB Mid-Term Evaluation. October 18, 2006. viii Table 5.1) with each of these cross-cutting categories (columns in Table 5.1) facilitates staff follow-up on certain issues—such as M&E—that need to be addressed for more than one output or sub-output (Table 5.1). Management and finance: • Ensure continuous staffing of the senior food program advisor position for pilot country programs and that the person filling this position has access to the necessary staff and consultancy support needed to conduct the job (finance and management recommendations 1-2); • Create a food programming committee and use the committee to oversee clear tracking and planning for project activities (finance and management recommendation 3); and • Revise the ICB budget so that it is in line with revised priorities and activities of the grant (finance and management recommendation 4). Strategic planning, M&E, and reporting: • Develop self-assessment tools needed to track food security capacity (on top of other core capacities being tracked) in the major ISD units (IDR, TRP, and IP) and in the ONS with whom the ISD units are working under the grant (Sub-output 1.1.2, recommendation 1; Output 2.1 and 2.2, recommendation 2); • Revise IR and sub-output statements and indicators where relevant (Sub-output 1.1.2, recommendations 3-4; Sub-output 1.1.4, recommendations 1-2; Output 1.2, recommendations 1-2; Output 1.3, recommendation 1; and Output 2.1 and 2.2, recommendation 3); • Consolidate ARC and IFRC’s existing experience with emergency indicators and identify other valuable experiences from within the community of Title II PVOs and pilot test some of these indicators in ICB-sponsored pilot country programs (Sub-output 1.1.3, recommendations 1-4); and • Create a web-based (internal or external) paper series and web page (Output 1.3, recommendation 2). Tools development and distribution: • Set priorities for tools development and completion (Sub-output 1.1.1, recommendation 4); • Assign clear responsibility for completing tools (Sub-output 1.1.1, recommendation 5); • Prepare bibliographies that situate tools within a broader internal and external context (Sub-output 1.1.1, recommendation 6); • Facilitate CS-wide distribution of appropriate completed tools (Sub-output 1.1.1, recommendation 7); and • Facilitate internal RC and external CS access to the ARC/CRS M&E module series (Sub-output 1.1.1, recommendation 8). ARC HQ capacity building and training: • Create a food security seminar series (Sub-output 1.1.1, recommendation 2); ARC ICB Mid-Term Evaluation. October 18, 2006. ix • Develop a pre and post-test for the seminar series and food security basics course (Sub-output 1.1.1, recommendation 3); • Strengthen existing food programming/security checklist (Sub-output 1.1.2, recommendation 1); • Participate actively in quarterly review of IP projects (Sub-output 1.1.2, recommendation 2); and • Develop flexible annual training plans and better systems to track who is trained, in what, and when (HQ staff) (Project systems for planning and tracking ICB￾related training recommendations 1-2). Pilot country programs and southern Africa training: • Design and execute pilot programs (3 planned) and RDRT training in southern Africa (Output 2.1 and 2.2, recommendation 1); • Create self-assessment capacity index that National RC Societies can use to identify and track their needs and progress (in collaboration with M&E activities above) (Output 2.1 and 2.2, recommendation 2); • Develop flexible annual training plans and better systems to track who is trained, in what, and when (ONS staff) (Output 2.1 and 2.2, recommendations 1-2). 5.3. Broader institutional impact and prospects for sustaining the existing and projected results Originally, the ICB grant was designed to develop the capacity of ARC within a given policy context that was influenced by the orientation of four major groups of actors: ARC itself, USAID/FFP, the International Federation of the Red Cross (IFRC), and the Red Cross/Red Crescent National Societies (Annex III, Table 2). A series of changes both within ARC and USAID/FFP—most notably the growing importance of building local capacity to identify and manage major risks, as well as to reduce food insecurity -- have brought the policy priorities of these major actors into closer alignment with the priorities of the ICB (Annex III, Table 2). While it would be handy to attribute these broader institutional changes to this grant’s activities this would not be correct. What we can say, however, is that the grant did contribute to these broader institutional changes. Some of the best indirect evidence of ARC’s commitment to the new development-relief paradigm being promoted by the grant is the long-term impact of the previous ISA (FY98-FY01) grant. This longer term perspective shows that ARC continued to support both the positions and the tools developed under the previous grant even during the two year period (FY02 and FY03) when ARC did not have a Title II capacity building grant.5 The current shift in ARC’s policy environment suggests that the principal tools and capacities being developed under the grant are part of deeper long-term changes that are likely to continue. In this revised context, the prospects for maintaining and updating the resulting tools and systems is probably greater than when the grant started, even with the shifts in Title II funding. This same shift in context enhances the chances that ARC could develop a highly flexible Title II food programming initiative that could reinforce some of the achievements in the pilot programs at some point in the near future. 5 Unlike most of the other Title II funded ISA grants, ARC’s ISA was for only three years. ARC ICB Mid-Term Evaluation. October 18, 2006. x Acknowledgements Tracy Hightower played a major role in backstopping the evaluation from start to finish. Marcella Baldwin was very helpful in preparing the final presentation and discussion. ARC ICB Mid-Term Evaluation. October 18, 2006. xi Table of Contents Page Executive Summary............................................................................................................. i Acknowledgements..............................................................................................................x Table of Contents............................................................................................................... xi List of Tables ................................................................................................................... xiv List of Boxes......................................................................................................................xv List of Acronyms and Abbreviations............................................................................... xvi 1.0. Introduction.................................................................................................................1 1.1. Institutional background and context precipitating grant ............................1 1.2. ARC strategy to address major constraints to effective emergency response........................................................................................................2 1.3. ARC’s current Title II ICB (Institutional Capacity Building) grant............4 1.4. Mid-term evaluation methodology ..............................................................6 2.0. Intermediate Result 1: ARC Improves its Institutional and Technical Capacity to Design and Manage Food Aid and Nutrition Interventions as an Effective Response to Emergencies............................................................................................................7 2.1. Output 1.1: Enhanced ARC staff technical knowledge base ......................8 2.1.1. Sub-output 1.1.1: Improved tools for needs assessment, nutritional surveillance, and planning for emergency interventions...............................................................................8 2.1.1.1. Accomplishments to date and evidence of existing or projected impact ...............................................8 2.1.1.2. Factors that have or may contribute to or detract from development, utility, and viability of outputs...................................................................9 2.1.1.3. Recommendations...............................................12 2.1.2. Sub-output 1.1.2: Enhanced emergency response planning that integrates food and nutrition with water/sanitation and other public health interventions.......................................................13 2.1.2.1. Accomplishments to date and evidence of existing or projected impact .............................................13 2.1.2.2. Factors that have or may contribute to or detract from development, utility, and viability of outputs.................................................................14 2.1.2.3. Recommendations...............................................15 2.1.3. Sub-output 1.1.3: Standardized monitoring and evaluation systems to measure the nutritional impact of food in emergencies..............................................................................16 2.1.3.1. Accomplishments to date and evidence of existing or projected impact .............................................16 2.1.3.2. Factors that have or may contribute to or detract from development, utility, and viability of ARC ICB Mid-Term Evaluation. October 18, 2006. xii outputs.................................................................17 2.1.3.3. Recommendations...............................................17 2.1.4. Sub-output 1.1.4 (recommended for addition at mid-term): Stronger bridges and linked understandings between technical units in International Programs at ARC...................................18 2.1.4.1. Accomplishments to date and evidence of existing or projected impact .............................................18 2.1.4.2. Factors that have or may contribute to or detract from development, utility, and viability of outputs.................................................................19 2.1.4.3. Recommendations...............................................19 2.2. Output 1.2: Expanded collaboration between ARC and other PVOs [and institutions] in the design and management of food and nutrition interventions in emergencies......................................................................20 2.2.1. Accomplishments to date and evidence of existing or projected impact.......................................................................................20 2.2.2. Factors that have or may contribute to or detract from development, utility, and viability of outputs ..........................23 2.2.3. Recommendations....................................................................24 2.3. Output 1.3: Established [ARC domestic chapter advocacy network and] 6 support material for promoting the importance and need for food in international emergency responses [recommend adding: among ARC domestic chapters] .....................................................................................25 2.3.1. Accomplishments to date and evidence of existing and projected impact.......................................................................25 2.3.2. Factors that have or may contribute to or detract from development, utility, and viability of outputs ..........................25 2.3.3. Recommendations....................................................................28 3.0. Intermediate Result 2: Red Cross/Crescent Movement Partners Develop the Capacity to Design and Manage Food Aid and Nutrition Interventions as an Effective Response to Emergencies..........................................................................28 3.1. Accomplishments to date and evidence of existing and projected impact.........................................................................................................28 3.2. Factors that have or may contribute to or detract from development, utility, and viability of outputs...................................................................29 3.3. Recommendations......................................................................................30 4.0. Management and Financial Systems.........................................................................31 4.1. Management and finance ...........................................................................31 4.1.1. Period one: September 3 – December 26, 2004......................31 4.1.2. Period two: December 27, 2004 – September 30, 2005 .........31 4.1.3. Period three: October 1, 2005 – present..................................33 4.1.4. Budget and financial systems...................................................34 4.1.5. Factors that may contribute to or detract from development, utility, and viability of outputs.................................................35 6 Text deleted at the request of USAID/FFP in FY04. To clarify the target audience the mid-term recommends adding “among ARC domestic chapters” to the phrasing of the output. ARC ICB Mid-Term Evaluation. October 18, 2006. xiii 4.2. Link between the ICB sponsored training and project management and planning......................................................................................................36 4.3. Link between the ICB monitoring and evaluation and project management ...............................................................................................36 4.4. Recommendations......................................................................................37 4.4.1. Finance and management.........................................................37 4.4.2. Project systems for planning and tracking ICB-related training .....................................................................................37 4.4.3. Project M&E systems ..............................................................38 5.0. Conclusions and Recommendations ..........................................................................38 5.1. Quick action recommendations..................................................................38 5.2. Five major categories of recommendations ..............................................38 5.3. Broader institutional impact and prospects for sustaining the existing and projected results .........................................................................................40 Annex I: List of ICB Project Activities, FY04-FY06.......................................................44 Annex II: ARC ICB Project PITT ....................................................................................63 Annex III: Supplemental Tables .......................................................................................70 Annex IV: References Consulted......................................................................................75 Annex V: Evaluation Schedule and List of Persons Interviewed.....................................78 Annex VI: Evaluation Scope of Work (SOW) .................................................................80 ARC ICB Mid-Term Evaluation. October 18, 2006. xiv List of Tables Table 1.1: Intermediate Results, Outputs, and Sub-Outputs Associated with the ARC ICB Grant Table 2.1: Tools Developed, Revised, Consolidated, or Scheduled for Development under the ARC ICB Table 2.2: Progress Toward the Execution of ARC’s Output of Establishing Support Material for Promoting the Importance and Need for Food in International Emergency Responses, FY04-FY06 Table 4.1: ICB Program Management Milestones Directly or Indirectly Linked to the Current Grant Table 5.1: Summary Mid-Term Recommendations for Enhancing Grantee Performance (Individual recommendations grouped by five major categories) ARC ICB Mid-Term Evaluation. October 18, 2006. xv List of Boxes Box 1.1: Clarification of Terms Related to the Red Cross Box 1.2: IFPRI Perspectives on Food Security—A Paradigm Shift Box 1.3: Major Constraints that Inhibit the Red Cross Movement’s Effective Response to Emergencies Box 2.1: Reflections on the Impact of a Joint IDR (International Disaster Response)- TAU (Technical Assistance Unit) Internal Evaluation of an Emergency Relief Program in Sri Lanka Facilitated by the ICB Box 2.2: ICB Series of Field Friendly M&E Modules Produced in Partnership between ARC and CRS and Distributed in FY06 Box 2.3: Lessons Learned from the Secondment of ARC Senior Associate for Food Programming to WFP Box 2.4: Results of Recent International Services Poll of US Adults in Nine Cities Concerning their Knowledge and Support of ARC International PVO Activities Box 3.1: Original Criteria for Countries Selected for ARC ICB Pilot Projects Box 4.1: The ARC ICB Project Budget and Financial Systems ARC ICB Mid-Term Evaluation. October 18, 2006. xvi List of Acronyms and Abbreviations ACDI/VOCA Agricultural Cooperative Development International-Volunteers in Overseas Cooperative Assistance AED Academy for Educational Development ARC American Red Cross CHF Community Habitat Finance International CRS Catholic Relief Services CS Cooperating Sponsor DAP Development Assistance Proposal EIG Evaluation Interest Group FANTA Food Aid and Nutrition Technical Assistance Project FFP Food for Peace (Office of) (Bureau for Democracy, Conflict and Humanitarian Assistance (DCHA) USAID FY Fiscal Year GL General Ledger HIV/AIDS Human Immunodeficiency Virus/Acquired Immune Deficiency Syndrome ICB Institutional Capacity Building Assessment (grant) IDR International Disaster Response Unit (of the International Services Department/ARC) IFRC International Federation of the Red Cross IP International Programs Unit (of the International Services Department/ARC) IPMS International Project Management System IPP Integrated Programming Plan IR Intermediate Results ISA Institutional Support Assistance grant ISD International Services Department (ARC) MCH Mother Child Health NHQ National Headquarters (ARC) ONS Operating National Societies PDA Personal Digital Assistance PITT Performance Indicator Tracking Table PVO Private Voluntary Organization RDRT Regional Disaster Response Team RFA Request for Assistance SAF Southern Africa SSC Shared Services Center (ARC) TA Technical Assistance TAPE Technical Assistance for Planning and Evaluation (TAPE) TAU Technical Assistance Unit (ISD/ARC) TRP Tsunami Recovery Program (ISD/ARC/ARC) UAH United Against Hunger UNICEF United Nations Children’s Fund ARC ICB Mid-Term Evaluation. October 18, 2006. xvii USAID United States Agency for International Development Wat/San Water and Sanitation WFP World Food Programme WHO World Health Organization WWF World Wildlife Fund ARC ICB Mid-Term Evaluation. October 18, 2006. Introduction. 1 1.0. Introduction 1.1. Institutional background and context precipitating grant For over 100 years the American Red Cross (ARC) has played a major role in coordinating United States assistance for domestic and international7 disasters among its other core programs. Two unique features of ARC that distinguish it from other Title II Cooperating Sponsor Private Voluntary Organizations (PVOs) are its commitments to: • Executing all of its assistance through a network of 185 international Red Cross, Red Crescent, or Red Crystal Operating National Societies, which are commonly referred to within the “Movement” as ONS (see Box 1.1 for a brief clarification of terms related to the Red Cross Movement) and • Ensuring that all of its international training and interventions conform to principles and the ongoing overarching program of the International Federation of the Red Cross (IFRC) based in Geneva. Box 1.1: Clarification of Terms Related to the Red Cross International Federation of Red Cross and Red Crescent Societies (IFRC): Also referred to as the Federation, IFRC is based in Geneva and oversees the international network of Operating National Societies (ONS). Operating National Society (ONS): Refers to the national PVO that is officially recognized by the IFRC as its representative in a country. The ONS in the United States is the American Red Cross. The Movement: Refers to the international Red Cross Movement—i.e., the IFRC and all its member Red Cross/Red Crescent/Red Crystal organizations. American Red Cross (ARC): Refers to the ONS of the Red Cross in the United States whose headquarters is in Washington, D.C. ARC’s role in coordinating US assistance for domestic and international disasters includes providing food assistance to populations affected by these catastrophes. To date, however, this food assistance has focused almost entirely on emergency feeding during the initial three months, the time limit ARC has traditionally used to classify an event as a “disaster,” (compared to the “transition” period, which follows the disaster period). Longer term planning was assumed irrelevant in the context of food assistance during the three month disaster period. 7 Core program areas for ARC include: (1) international public health; (2) disaster preparedness and response; (3) restoring family links; and (4) international humanitarian law dissemination. Lesotho Red Cross Society volunteers re-package seed for distribution. Photo by Jeff Weiss, ARC ARC ICB Mid-Term Evaluation. October 18, 2006. Introduction. 2 Hurricane Mitch in 1998 demonstrated the need for reconstruction and rehabilitation beyond the “come, do, go” (also referred to as the “truck and chuck” tactic—Box 1.2) premise of most ARC disaster relief efforts, as well as for the Movement as a whole (ARC 2003: 9). To address this, the agency adopted a new Strategic Plan. As part of this plan the agency announced its desire to increase its capacity to incorporate food into its emergency and development planning to help fulfill its global mission of reducing morbidity and mortality. Box 1.2: IFPRI Perspectives on Food Security—A Paradigm Shift “In spite of decades of dedicated effort and billions of dollars in aid, hunger still stalks the world. Why is hunger so entrenched? And what is the Movement doing about it?” “…After recognizing the limits of food aid, many humanitarian agencies are moving to a more holistic, longer-term approach of ensuring households can meet their basic economic needs. The aim is to strengthen people’s resilience to food insecurity by building on their existing healthy coping mechanisms and resources. The key parts of the new approach—a paradigm shift away from “truck and chuck” [added for emphasis]—include early warning systems, integrating food security into other programs and using agricultural aid and training to help people fend for themselves.” Source: Rosemarie North. 2006. Food Security—A New Paradigm. Red Cross Red Crescent. The Magazine of the International Red Cross and Red Crescent Movement. Issue 2. Geneva: IFRC. 1.2. ARC strategy to address major constraints to effective emergency response To achieve this mission of longer term transitional and development activities, ARC began building its capacity to address the longer term needs of the communities affected by disaster. In doing, so, however, the agency was confronted with a number of constraints (Box 1.3). Box 1.3: Major Constraints that Inhibit the Red Cross Movement’s Effective Response to Emergencies 1. Non-food responses are not always sufficient even when working in partnership with the WFP or other PVOs. 2. National Societies are easily overburdened because they lack management durability and surge capacity, especially when presented with multiple donors and multiple reporting requirements. 3. The nature of food aid programs has continued so that it is essential to incorporate short and longer term goals of a relief-through-development continuum and ensure a more integrated response. 4. National Societies lack capacity to plan for management of short and longer term responses to or manage multiple, disjointed efforts. 5. ARC and the Movement do not effectively leverage financial and human resources needed for the most efficient emergency response and impact. 6. Technical skills are often not available when and where needed to respond effectively. 7. There is an increasing need for improved communication between domestic chapters and international programming to ensure adequate surge capacity when needed. 8. There is a need to advocate for international responses including food aid. 9. ARC needs a mix of public and private funding to reduce reliance on any one source. Source: ARC. 2003. Project Proposal. Institutional Capacity Building Assistance Grant. Washington, DC: ARC. Pp: 9-10. ARC ICB Mid-Term Evaluation. October 18, 2006. Introduction. 3 To address these constraints, ARC developed an initiative within the International Services Department with the following four elements. • Technical Assistance for Planning and Evaluation (TAPE) unit: Housed within the International Services Department, the TAPE unit was created to specialize in the range of technical skills and the types of development programming (food assistance, primary health care, water/sanitation, organizational development, tracing [e.g., reuniting missing persons with their families], and international humanitarian law) the organization might pursue. • Integrated Planning Process (IPP): The IPP was developed as a tool8 for helping ARC improve technical design, implementation, alignment (e.g., with wider policy environments), and management of emergency and non-emergency programs. • Improved collaboration: Improvements were made regarding internal ARC communication (e.g., sharing technical expertise between food management staff and water management staff); sharing of technical expertise in the wider Red Cross Movement (e.g., lessons learned concerning logistics, nutrition, and health programming); and collaboration with other international PVOs that have expertise in transition and recovery programs that can strengthen ARC expertise on emergency response. • Stronger domestic ARC chapter support: Effort also concentrated on encouraging domestic chapter support for ARC’s international emergency response efforts, as well as support for international aid. Once the International Services Department had created a formal strategy to address their major constraints, the details of this strategy needed to be developed and refined. ARC’s first Title II capacity building grant—the Institutional Strengthening Assistance grant (FY99-FY01) was designed to build ARC’s international capacity in the first three of the four focal areas/elements listed above. The funds helped the TAPE unit build its capacity to design, manage, monitor and report on food programming.9 New tools were developed including the agency’s first Integrated Planning Process (IPP) training modules and ARC’s first commodity management toolkit. The grant also “seeded” the initial funding for two new positions--a food program administrator/advisor and senior M&E specialist with a food security background—for the TAPE unit. One of the best indirect indicators of ARC’s commitment to long-term institutional changes being targeted by the ISA grant is that many of the training activities and staff positions that were initially funded by the grant continued during the two year gap in funding between the ISA (FY99-FY01) and the ICB (FY04-FY08). 8 The planning process is based on helping staff better analyze four components: needs (elements that are missing or that present problems), interests (what people want to do and accomplish), resources (internal or external funding or non-monetary sources), and capacities (ability of ARC, Operating National Societies [ONS] and other partners to design and implement projects and programs). 9 John P. Mason. 2001. Final Evaluation. Institutional Support Assistance Grant. Washington, DC: ARC for USAID/FFP (June 29, 2001). ARC ICB Mid-Term Evaluation. October 18, 2006. Introduction. 4 1.3. ARC’s current Title II ICB (Institutional Capacity Building) grant ARC’s ICB grant was designed (ARC 2003:11) “to continue adjustments to the changing nature of ARC’s and the Movement’s use of emergency food aid, specifically” (ARC 2003: 11): • To improve ARC’s planning at all levels to better respond to unanticipated needs and to better accommodate the need for longer term interventions; • To monitor and evaluate ARC’s work to better adjust for program needs, derive lessons learned, and tell the ARC story to a range of audiences; and • To enhance ARC’s management from headquarters to National Societies to cope with a more diverse array of management needs, to expand operations when there is a need to surge (and to scale back when completed), and to better integrate ARC’s efforts between Operating National Societies and with other (non-Red Cross) partners, to ensure that skills are available when and where they are needed. The activities needed to achieve these goals were conceptualized under two intermediate results (IRs) broken down into specific outputs and sub-outputs (Table 1.1). The grant was designed and funded to (1) refine ARC’s existing systems and tools and develop new systems and tools for use in food programming as a response to emergencies, (2) conduct a pilot program in two countries for the development and testing of systems and tools for use in executing and expanding food programming, (3) disseminate the results of these pilots to Red Cross Movement partners and beyond, and (4) expand and scale-up food programming to ARC bilateral programs in other countries. A small sub-component that focused on promoting the importance and need for food in international emergency responses was also included (Table 1.1). The total budget for the ICB program is $2,716,112, of which $1,800,000 has been awarded by USAID with a commitment from ARC of $916,112 (33 percent of the grant total) in matching funds. The execution of the activities under the grant can be roughly subdivided into three different periods. Each of these periods was characterized by a different pattern of food programming within the agency, as well as different sets of food security partners. First period (September 2003-December 25, 2004): During the initial period, ARC prepared two additional Title II proposals (separate from the ICB proposal) with the expressed intent of building country programs in Tajikistan and Malawi. Each of these programs involved a bilateral collaboration between ARC and the relevant National Society, as well as a host of local and international PVO partners. The Title II funding for one grant (Tajikistan) was withdrawn; ARC withdrew from the other consortium when ARC’s funding was reduced. Although ARC did not receive any Title II grants in this year, it continued numerous activities to achieve the ICB IRs in conjunction with its other types of emergency and non-emergency food programming (Annex I). ARC ICB Mid-Term Evaluation. October 18, 2006. Introduction. 5 Table 1.1: Intermediate Results, Outputs, and Sub-Outputs Associated with the ARC ICB Grant IR1: ARC improves its institutional and technical capacity to design and manage food aid and nutrition interventions as an effective response to emergencies Output 1.1: Enhanced ARC staff technical knowledge base Sub-output 1.1.1: Improved tools for needs assessment, nutritional surveillance, and planning of emergency interventions Sub-output 1.1.2: Enhanced emergency response planning that integrates food and nutrition with water/sanitation and other public health interventions Sub-output 1.1.3: Standardized monitoring and evaluation systems to measure the nutritional impact of food in emergencies Output 1.2: Expanded collaboration between ARC and other PVOs in the design and management of food and nutrition interventions in emergencies Output 1.3: Established ARC domestic chapter advocacy network and support material for promoting the importance and need for food in international emergency responses IR2: Red Cross/Crescent Movement partners develop the capacity to design and manage food aid and nutrition interventions as an effective response to emergencies Output 2.1: Improved knowledge base in National Red Cross Societies to design and manage local food aid and nutrition interventions in emergencies Output 2.2: Enhanced coordination between National Red Cross Societies and local institutions in food and nutrition during emergencies Source: ARC 2003. Annex D. Second period (December 26, 2004- September 2005): Once the Tsunami hit, the focus of the ICB’s capacity building efforts shifted to the World Food Programme (WFP), which was now ARC’s principal partner for $120 million of food programming (20% of the TRP portfolio) to Indonesia and Sri Lanka, and to refining various tools developed under the ISA and during FY04 of the current program for use during the Tsunami relief effort. Since many of the key staff (including the ICB manager, Regis Chapman) were pulled off the grant for Tsunami relief, the rate and velocity of turning out new products, training new and existing staff, and revising older products that the grant had envisioned slowed down. Third period (October 2005-present): The dawn of a new fiscal year (October 1, 2005) was associated with the shift from short term emergency efforts to more long term Tsunami recovery programs. This period coincides with ARC attempting to absorb many of the lessons learned from the Tsunami both in terms of programming and philosophy. The same period coincided with a renewed commitment by ARC to addressing some of the slow onset disasters (such as HIV/AIDS and drought) that have a direct affect on World Food Programme food aid distribution in collaboration with ARC, Indonesia. Photo source: WFP ARC ICB Mid-Term Evaluation. October 18, 2006. Introduction. 6 local communities’ ability to identify and manage risk. This shift also coincided with the confirmation of a new IDR director.10 1.4. Mid-term evaluation methodology The principal objective of a Title II mid-term evaluation was to evaluate the grant’s progress in order to make recommendations for how to improve the program’s impact and results. The specific objectives of the mid-term evaluation were: • Determine the accomplishments and evidence of impact to date of the ICB grant based on the proposal plan and the modifications to that plan; • Analyze the principal factors that have contributed or seem likely to contribute to (or detract from) the utility and viability of proposed outputs under the current strategic plan for the ARC International Services Department (ISD) from document and data review and interviews with key staff; and • Provide recommendations for enhancing grantee performance, including modifications to the proposed work plans and monitoring and evaluation system. The task of assessing the grant was complicated by the fact that the original M&E Plan was designed during the first year (FY04) before the program was adjusted to accommodate: • The fact that ARC did not have either of the two Title II-funded programs that it expected to have in FY04 (e.g. the funding for one of the Development Assistance Proposals [DAP] that ARC had applied for was withdrawn and ARC withdrew from the grant in the other) which (although funded separately) were planned to be “testing grounds” (simultaneous with the pilot country programs) for the new systems and tools being developed by this grant; and • The dramatic upsurge in activity and staff displacement that accompanied ARC’s active role in coordinating food assistance to three countries affected by the Tsunami. The mid-term evaluation team used a variety of methods to determine the status of grant activities and accomplishments. These included: • Document review (project deliverables, ICB documents and documents related to various partnerships, including those with WFP); • In-depth interviews with key informants from principal operational units of the International Services Department (i.e., International Programs, Tsunami Response Programs, and International Disaster Response); • Email and phone interviews with overseas participants; and • Focus group discussions. A total of 23 persons were interviewed through these various methods between September 5 and October 3, 2006. 10 The unit had been without a director for six to nine months. ARC ICB Mid-Term Evaluation. October 18, 2006. IR1. 7 Della E. McMillan (external consultant11) served as team leader. Ange Tingo (ARC senior food program advisor) served as the internal evaluation manager and co-team leader. The team leaders were assisted by an evaluation committee comprised of representatives from three of the ISD (International Services Department) units: Patricia McLaughlin, Alice Willard, and Thomas Carmody from the Technical Assistance Unit (TAU), Tracy Reines from the International Disaster Response (IDR) unit, and Luke Reeves from the International Programs (IP) unit. Leah A.J. Cohen (external consultant12) was responsible for final editing and production. The remainder of the report is divided into five sections. • Sections two and three describe (per IR1 and IR2, respectively) accomplishments to date and evidence of early impact on the quality of Red Cross and other CS food programming; the various factors that contributed to or detracted from development, utility, and viability of outputs; new opportunities and constraints; and priority recommendations for enhancing grantee impact; • Section four reviews the project management and financial systems and examines ways that these systems contributed to or detracted from execution during the first half of the project and how these could be strengthened in the second half; and • Section five presents the concluding observations, regroups the main recommendations of the analysis into five cross cutting areas13 (see Table 5.1), and makes the distinction between recommendations that require “quick action” that those that require more intensive long-term follow-up, and comments on the future of ARC food programming based on the mid-term finding. 2.0. Intermediate Result 1: ARC Improves its Institutional and Technical Capacity to Design and Manage Food Aid and Nutrition Interventions as an Effective Response to Emergencies The activities under the grant that were designed to build capacity under IR1 focused on attaining three outputs: • Output 1.1: Enhanced ARC staff technical knowledge base; • Output 1.2: Expanded collaboration between ARC and other PVOs [and institutions] in design and management of food and nutrition interventions in emergencies; and • Output 1.3: Established [of ARC domestic chapter advocacy network and] 14 support materials for promoting the importance and need of food in international emergency responses [among ARC domestic chapters].15 11 Della E. McMillan, Ph.D. is an Associate Research Scientist in Anthropology at the University of Florida. 12 Based in Gainesville, Florida. 13 These five areas (which are listed as headers in the table) include: Management and finance; strategic planning, monitoring and evaluation and reporting; tools development and distribution; ARC headquarters (HQ) capacity building; and pilot programs and regional disaster response teams (RDRTs) training. 14 Text concerning the creation of an “ARC domestic chapter advocacy network” deleted at USAID request. 15 Text recommended for addition per mid-term evaluation. ARC ICB Mid-Term Evaluation. October 18, 2006. IR1. 8 2.1. Output 1.1: Enhanced ARC staff technical knowledge base ARC’s activities under Output 1.1 focused on achievement of three sub-outputs: • Sub-output 1.1.1: Improved tools for needs assessment, nutritional surveillance, and planning of emergency interventions; • Sub-output 1.1.2: Enhanced emergency response planning that integrates food and nutrition with water/sanitation and other public health interventions; and • Sub-output 1.1.3: Standardized monitoring and evaluation systems to measure the nutritional impact of food in emergencies. 2.1.1. Sub-output 1.1.1: Improved tools for needs assessment, nutritional surveillance, and planning for emergency interventions 2.1.1.1. Accomplishments to date and evidence of existing or projected impact The grant application identified five tools16 that ARC anticipated developing and using to build general capacity and the M&E system. The proposal’s emphasis on creating new tools or consolidating existing tools was very much in the spirit of the ICB Request for Assistance (RFA), which gave priority to grants that would develop tools that could be used by other Title II Cooperating Sponsors. Even with all the delays caused by the Tsunami, the Technical Assistance unit has overseen development, revision and/or consolidation of tools in five general areas (Table 2.1):17 • Food security basic concepts; • Integrated Planning Process (IPP); • Emergency/transition targeting of beneficiaries; • Food commodity management and distribution; and • M&E and reporting for food aid programs. In addition, programming tools to help projects deal with two emerging themes (the need for innovative partnership models to address slow onset disasters and the use of commodities for HIV/AIDS home based care) are slated 16 The tools included (ARC 2003: 12-13): (1) Training Module for Emergency Field Workers, (2) Emergency Handbook, Personal Digital Assistant Technology, (3) Humanitarian Logistics Software, (4) Vulnerability and Risk Mitigation Conference, (5) Monitoring and Evaluation Modules. 17 See Annex I of this document for a detailed analysis of the steps undertaken to develop each of these modules and tools by fiscal year. One of three pilot programs is planned in Lesotho in FY07 and FY08. This drip kit (a micro-irrigation system) is one of the current agricultural activities that will be incorporated into that pilot program. Drip kits were given out to HIV/AIDS affected households. Photo by Tim Cummings, ARC ARC ICB Mid-Term Evaluation. October 18, 2006. IR1. 9 for development in connection with the pilot country programs in FY07 and FY08 (Table 2.1).18 To date, the tools that have been most widely distribution and used under the ICB are (Annex I) the: • IPP modules that were developed under the ISA and revised under the ICB; • Food Commodity Toolkit (developed under the ISA), which was distributed very widely within the Movement, as well as to ARC bilateral programs under the ICB; • ARC/CRS training modules (five modules); and • “Pipeline” and “distribution” modules in the humanitarian logistics software programs. ARC has shared the M&E modules (all five modules), the IPP modules, and a PowerPoint presentation that summarizes ARC’s experience using Personal Digital Assistance (PDA) technologies for assessments and commodity management with other Title II Cooperating Sponsors through its membership in the Evaluation Interest Group (EIG) associated with Interaction (Table 2.1). Several new tools and studies19 that are slated for completion by the end of FY08 should be highly useful to the wider Title II CS community, as well as other USAID-related food partners, such as the World Food Programme (Table 2.1). Some of these tools are scheduled for development and pilot testing in conjunction with regional training workshops and pilot programs that ARC is scheduled to develop in southern Africa in FY07 and FY08 (Table 2.1, rows noted as FY07SAF). 2.1.1.2. Factors that have or may contribute to or detracted from development, utility, and viability of outputs The main factor that contributed to the successful development, consolidation, and training on the Integrated Planning Process (IPP), the ARC/CRS M&E module series,20 and the PDA summary presentation to the EIG—which are the three ICB tools that have been shared with the other Title II Cooperating Sponsors—is that their development and oversight were firmly tasked with the senior M&E specialist. 18 The pilot programs that were originally slated to begin in FY05 (after an initial selection process in FY04) were delayed, but have been rescheduled to begin (with the addition of a third pilot country) in FY07. 19 These include the AED/WFP/ARC case study of wider impacts of nutritional supplements on health programs which was slated to start in FY07, a paper summarizing lessons learned by ARC and IFRC about: (a) development and use of different models for humanitarian logistics software, (b) surge capacity indicators, and (c) emergency M&E indicators. 20 To date, the M&E module series has had the most extensive dissemination within ARC and to the other Title II Cooperating Sponsors. Three modules were released in version 1.0 in June 2006; two modules are still pending a final edit from CRS; one module is pending a draft review; one module is currently contracted and the draft is due at the end of October; two modules [internal ARC production] are under discussion. All three of the final modules have been disseminated throughout ARC and CRS field offices, as well as the respective headquarters. The modules have been posted on the EIG website and disseminated to the Federation for posting on FedNet (which the Federation manages). The completed modules were distributed on pen drives at both the EIG and AEA meetings. ARC ICB Mid-Term Evaluation. October 18, 2006. IR1. 10 Table 2.1: Tools Developed, Revised, Consolidated, or Scheduled for Development under the ARC ICB Focus Area and Tools Draft/Pilot Tested Individual General ARC General IFRC Other CS Public Website ONS Actively Using #1 Food Security Basics Food security fact sheet X Case studies X AED/WFP/ARC case study of impacts of nutritional supplements on health programs FYO7SAF FY08 Food security module FY07SAF FY07SAF #2 Integrated Planning Process* Revised basic module ISA & FY04 updates Updates X X X X Revised proposal guidance (Tsunami) X X X X FY07 X #3 Emergency/Transition Targeting of Beneficiaries Revision of existing IDR training materials FY06 X X FY07 FY07 Training module FY07SAF FY07SAF FY07 #4 Management and Distribution Commodity mgt. toolkit ISA X X X Food & Non Food Use of PDA technology X X X X X21 X22 Humanitarian Logistics Software 23 FY0724 --Pipeline X X X X X --Warehouse FY07 FY08 FY08 FY08 FY08 --Distribution X X X X X Monetization Guidance X X FY0725 FY07 n/a Training module FY07 FY07SAF -- -- -- #5 M&E & Reporting 21 The senior M&E advisor made several PowerPoint presentations about ARC’s experience using the PDAs at international meetings that were attended by other Title II Cooperating Sponsors, most notably the EIG roundtables. 22 Measles and baseline assessments. 23 Three types of software were developed. 24 Paper summarizing lessons learned from different pilot software packages to be developed. 25 Summary notes based on Viet Nam experience planned as reference document. ARC ICB Mid-Term Evaluation. October 18, 2006. IR1. 11 Focus Area and Tools Draft/Pilot Tested Individual General ARC General IFRC Other CS Public Website ONS Actively Using ARC/CRS modules X X X X X X Surge capacity indicators FY07-08 Training module FY07 FY07SAF -- -- -- Emergency M&E indicators FY07 FY08 FY08 FY08 FY08 FY09 #6 Slow Onset Disasters FY07SAF FY07SAF X -- FY0826 -- #7 HIV/AIDS Home Based Care FY07SAF FY07SAF X -- FY085 -- Source: Annex I and mid-term interviews. September 2006. CS: Title II PVO Cooperating Sponsor; ONS: Operating National Societies (e.g., local national societies in the International Federation of the Red Cross Movement); AED: Academy for Educational Development; PDA Personal Digital Assistants; M&E: Monitoring and Evaluation; FY: Fiscal Year; IDR: International Disaster Response unit (in the ARC International Services Department); FY07SAF to be developed in FY07 as part of the regional trainings or pilot programs in southern Africa; ISA: Initial draft prepared as part of the Title II funded Institutional Support Assistance grant (FY99-FY03). One of the key factors that detracted from consolidation and diffusion of the other tools (i.e. tools other than the M&E module series and the IPP) was that no one individual or position was consistently tasked with oversight. Additionally, the following factors hindered the utility of all the tools. • The original project PITT did not include an indicator that could track capacities being built with individual tools or tool sets. • There was limited access to the tools by field programs (who may not have access to materials that were distributed to M&E officers that attend EIG workshops) since there was no well advertised place for internal or external posting. • There was no comprehensive bibliography that presents the different modules and training materials for field staff and partners. ARC’s plans to launch a highly focused program of capacity building in four areas for the Red Cross Regional Disaster Response Teams (RDRT) and three pilot programs (Mozambique, Lesotho, and Ethiopia) in FY07 offer an unprecedented opportunity to develop a comprehensive system of “training modules” that the IFRC (International Federation of the Red Cross) and the Operating National Societies that belong to the IFRC can use to build staff capacity. Once the utility of these modules in building capacity of National Societies in the pilot countries is assessed, ARC can determine what, if any, revisions need to be made before disseminating the materials more widely. Currently, ARC has no plans to develop a more focused review process and system for tracking National Society and bilateral program “feedback” on the materials. This is a missed opportunity, as well as a potential risk. 26 Anticipate summarizing lessons learned from pilot programs and sharing with other bilateral programs to stimulate discussion about “best practice.” ARC ICB Mid-Term Evaluation. October 18, 2006. IR1. 12 2.1.1.3. Recommendations Recommendation #1: Develop self assessment tools needed to track capacity in the major ISD operational units (IDR, TRP, and IP). Replace the current impact indicators for IR1 with a new set of impact indicators that can better track capacity of different technical units (IDR, TRP, and IP) in the five core and two emerging areas.27 Recommendation #2: Create a food security (food programming) seminar series and associated modules. This series would regularly bring invited speakers (from other CSs and agencies, as well as from within the RC) for in-house training on topics linked to the five core and two emerging areas of technical expertise targeted by the ICB (including at least one, two-hour course on food security basics). In addition, the PowerPoint presentations from these invited speakers should be preserved and used to “seed” development of “canned” CD-based training modules that can be used at HQ and in the field. Recommendation #3: Develop a pre and post test for the food security (food programming) seminar series and “food security basics” course as a tool for monitoring comprehension of key issues. Recommendation #4: Set priorities for tools development and completion. Work with unit leaders of IP, IDR, and TAU (as part of the final debriefing and discussion of the mid-term evaluation), and the proposed food programming committee (see recommendation #3, management, section 4.4.1) to determine tools of primary focus. Recommendation #5: Assign clear responsibility for completing tools. Task individual TAU and IDR staff members with development, execution, and pilot dissemination of particular tools needed to execute the southern Africa Regional Disaster Response Team training, as well as other exercises (e.g., AED/WFP/ARC applied research) with ARC staff or ARC-hired consultants. Recommendation #6: Prepare bibliographies that situate tools within a broader internal and external context. In connection with current plans by ARC to build food security capacity of the southern Africa Regional Disaster Response Teams (RDRT) and to facilitate integrated programs in three pilot countries (discussed more in detail under IR2), the food security team within the Technical Assistance Unit (TAU) should develop a bibliography that lists all internal (and when relevant) and external references that staff might need to consider and where they can be accessed (through the web or through individual staff members). Recommendation #7: Facilitate CS-wide distribution of appropriate completed tools. When the ICB was designed it was anticipated that these technical documents would be distributed through the FAM website. Since FAM’s website is no longer being routinely 27 Core areas: Food security basic concepts; Integrated Planning Process (IPP); emergency/transition targeting of beneficiaries; food commodity management; and improved M&E for food aid programs. Emerging areas: HIV/AIDS home based care and slow onset disasters. ARC ICB Mid-Term Evaluation. October 18, 2006. IR1. 13 updated, this creates a challenge for ARC, as well as all the Title II Cooperating Sponsors and all Title II ICB grants. ARC appears to have two options, which are not mutually exclusive. - 7.a: Strengthen postings on Interaction website. Consider working with CRS and Interaction through the current ARC/CRS module series to develop a sub-area on their website focused on food programming. - 7.b: Consider creating a refereed28 ARC food programming technical paper series on the ARC public website. Consider developing a food programming technical paper series that could be posted on the public ARC website where critical documents and references to important internal (ARC, IFRC) documents related to emergency and non-emergency food programming can be posted for quick reference by ARC, IFRC, and National Society staff in the International Service Department’s “focus” countries, some of which will be directly implicated in the southern Africa RDRT training and pilot programs. Recommendation #8: Facilitate internal RC and external CS access to ARC/CRS M&E module series. Given the immediate relevance to ARC and IFRC programming worldwide of the ARC/CRS M&E module series, access to this should be increased through one or both options below.29 - 8.a: Post modules on a major website. Work with CRS to ensure that the ARC/CRS modules are posted on a website that is available to other Title II Cooperating Sponsors (whether or not they belong to EIG), once a system has been developed (through CRS or ARC) for tracking the number of “hits” to the documents so that downstream use can be monitored. - 8.b: Cross-reference the series on FANTA’s website. Consider asking FANTA to cross-reference the module series on its website (as they have with several other international PVO and university series). 2.1.2. Sub-output 1.1.2: Enhanced emergency response planning that integrates food and nutrition with water/sanitation and other public health interventions 2.1.2.1. Accomplishments to date and evidence of existing or projected impact By far the ultimate goal of the ICB is to improve the ability of ARC and National Societies to develop emergency programs that better integrate food and nutrition with water/sanitation and other public health interventions. The original indicators for this sub-output were designed to track ARC’s progress in introducing this new integrated approach into its emergency programming. 28 The term “refereed” is meant to signal the need for both internal and external review of technical papers before posting on the website. The term “technical paper series” is used to distinguish these finished “tools” from the presentations in the food security seminar series which is referred to in recommendations 1-3. 29 Although the series is posted on the EIG website (which can be accessed by EIG members through the Interaction website), this is not widely known and it is difficult to access for anyone outside the EIG working group. As a result the main avenue for distribution of the series to the other Title II Cooperating Sponsors has been through contacting the senior M&E advisors of both institutions. ARC ICB Mid-Term Evaluation. October 18, 2006. IR1. 14 There is qualitative evidence that: • A high percentage of the bilateral emergency programs developed by ARC in the last three years did more effectively integrate food and nutrition with water/sanitation and other public health interventions and • That many of the National Societies internalized some of the same lessons learned to build capacity in their own programs. It would be a mistake to assume that this ICB grant was the only, or even the principal, cause of these positive changes. It did contribute, however, and its contribution was substantial, if not fully documented. 2.1.2.2. Factors that have or may contribute to or detracted from development, utility, and viability of outputs Technical assistance by staff associated with the ICB grant contributed to the development of a more integrated approach in ARC’s transition and recovery programs. While staff’s awareness has been raised at all levels of the Red Cross to the critical importance of food security, the core understanding of the concepts and how to use some of the existing and proposed tools to improve food programming is still weak in comparison with other areas (such as water/sanitation and public health). All three International Service Department unit directors of IDR, IP, and TAU stated that this is not surprising given the inherent complexity of the topic and the fact that food security has not been a traditional focus of the Red Cross. This is a critical factor that detracts from the long-term sustainability of this new, more integrated approach. The two years remaining on the grant offer ARC an opportunity to return to its original focus on developing the core capacity of staff in the operational units (e.g., the International Programs, Tsunami Response Programs, and International Disaster Responses units) of the International Services Department. Once the pilot programs in Lesotho, Mozambique, and Ethiopia are launched, they would provide valuable lessons learned on how these programs could be scaled up. “…core understanding of the [food security] concepts and how to use some of the existing and proposed tools…is still weak in comparison with other areas (such as water/sanitation and public health).” Water and Sanitation Project, Indonesia. Photo source: The IDP Assistance Project: Meeting Needs in Tsunami-Affected Aceh (2006 Draft) ARC ICB Mid-Term Evaluation. October 18, 2006. IR1. 15 The principal risk associated with the new vision for activities under this sub-output (which is actually more of a return to the original vision that was outlined in the proposal) is that USAID/FFP and Red Cross administration may lose sight of the wider impact that the grant is having on “enhancing” emergency response, transition, and recovery planning through a large number of bilateral, IFRC, and National Societies programs. 2.1.2.3. Recommendations Recommendation #1: Strengthen the existing “food programming” and/or “food security” checklist form. The checklist (which was developed as part of the IPP Environmental and Organizational Analysis tool) is aimed at longer-term programs. This checklist is a resource that can be refined and used to facilitate the design of emergency, as well as longer term, programming. Recommendation #2: Participate actively in the quarterly reviews of transitional projects. This type of “lateral” technical assistance and review of projects in the International Programs unit should focus on helping the International Programs unit’s staff identify ways to better address food security (through direct incorporation into the project or more effective partnering with other actors) that would enhance program impact and efficacy. Several staff suggested that a good role model for this type of “lateral” capacity building is the active participation of the current senior advisor for water and sanitation (Wat/San). This advisor—who had considerable emergency experience, as well as longer￾term recovery experience—helped the staff involved in designing many of the recovery/development programs to better incorporate Wat/San components or to develop programs entirely focused on Wat/San. Most of these were in the recovery phase, post-emergency.30 Recommendation #3: Revise sub-output statement. Strengthen the capacity of the project to focus and track training impacts by adjusting the sub-output statement (from emergency response planning to emergency response, transition, and recovery planning). 30 The successful inclusion and expansion of the water and sanitation component to transitional and recovery projects was aided by an existing recognition by staff of the importance of addressing Wat/San issues in these projects. It may be more challenging to achieve the same results in food security since emergency response activities are more associated with emergency food aid rather than rebuilding food security systems. The water and sanitation advisor helped the staff involved in designing many of the recovery/development programs to better incorporate Wat/San components. Indonesia. Photo source: The IDP Assistance Project: Meeting Needs in Tsunami￾Affected Aceh (2006 Draft) ARC ICB Mid-Term Evaluation. October 18, 2006. IR1. 16 Recommendation #4: Revise indicators. Streamline indicators for this sub-output by focusing on original performance (monitoring) indicators that addressed capacity building, rather than how this capacity building affected all the emergency programs being executed.31 This streamlined set of indicators would include the following. - Retain the indicator “# of emergency proposals reviewed,” but consider rephrasing it to read “# of emergency proposals reviewed using the Food Programming Checklist” if the concept of the “checklist” is adopted. Staff should discuss whether a separate indicator should be developed to track the number of ongoing funded projects reviewed (as recommended above), as well as new proposals developed. - Retain the indicator “# of HQ people trained in integrated planning,” but break this information out by technical department (i.e., IDR, TAU, IP, and Tsunami Response Programs [TRP]). - Retain the indicator “% of trained people taking ex-post [post-training] test.” - Focus the targets and tracking of targets on HQ staff and consider breaking out this information by technical department (i.e., IDR, TAU, IP, and TRP). - Consider introducing the complementary indicators under IR2: o “Number of people on the southern Africa Regional Disaster Response Teams (RDRT) who have attended training seminars/workshops featuring the four core food security modules and o “Number of people associated with the pilot country programs who have attended training seminars/workshops on the four core food security modules. 2.1.3. Sub-output 1.1.3: Standardized monitoring and evaluation systems to measure the nutritional impact of food in emergencies 2.1.3.1. Accomplishments to date and evidence of existing or projected impact To date ARC’s emergency M&E and reporting systems have focused on the impacts of emergency programs (in general) on health and nutrition and there has been very little quantitative or qualitative analysis of the ways that including (or not including) a coordinated food response in emergencies can contribute to or detract from an emergency response. Under the current ICB grant, ARC was expected to: • Organize a “Vulnerability Assessment and Risk Mitigation Conference” (ARC 2003: 13) (scheduled originally for FY04 then FY05) that would bring together representatives of the other Title II Cooperating Sponsors, as well as experts from other agencies (e.g., Sphere Project, WFP, WHO, and UNICEF) to identify what 31 In the FY04 and FY05 reports only impact indicators that had been proposed by the ICB M&E plan and 2004 work plan were reported (i.e., “% food/nutrition emergency proposals funded; #food/nutrition emergency proposals developed; #emergency proposals developed”). These indicators, plus the capacity indicators described in the text, were in the original M&E Plan. The proposed changes, therefore, represent a streamlining that is based on the original vision of tracking capacity building and systems for project review, as well as the actual outputs in terms of proposal design. ARC ICB Mid-Term Evaluation. October 18, 2006. IR1. 17 they consider best practices for tracking the nutritional impact of food aid in emergencies and • Pilot test a number of core indicators in ARC’s own bilateral emergency programs. The conference, which was expected to catalyze activities under this sub-output was never held. As a result, this is one of the few sub-components of the grant that has had very little activity. Many staff within the Technical Assistance Unit still feel this sub￾component is very important even though ARC has enacted a new policy that no longer allows the IDR unit to be directly involved in a food component as part of emergency responses (ARC now aims to ensure that essential food components are executed through partnerships with other organizations such as WFP). Given that, the TAU decided that a practitioners’ forum on food assessments in emergencies (the sudden acute phase through recovery phase and chronic food insecurity are all allowed), would be a valuable substitute for the original forum. It was also felt that this type of conference would contribute to the creation of an annotated toolkit of food/nutritional emergency assessments. 2.1.3.2. Factors that have or may contribute to or detracted from development, utility, and viability of outputs The critical factor which detracted from the sound execution of activities under this sub￾output was insufficient staff. Especially damaging was the fact that the person filling the position of M&E specialist (who was to have a background in nutrition) was never hired. The combination of insufficient and insufficiently-trained staff and the pull of other priorities that emerged in FY05 that consumed the existing staff’s time and energy meant that this sub-component was delayed. ARC’s extensive experience in monitoring, evaluation, and reporting of emergencies is an opportunity that should be capitalized upon. The recent hiring (2005) of a new IDR director, who was the former head of the Sphere Project,32 creates an unprecedented opportunity for the Technical Assistance Unit M&E staff to move forward with this sub￾component in close collaboration with IDR. The principal risk that staff identified for this activity is that the review and pilot testing of the indicators might become come overly complicated and/or unfocussed. 2.1.3.3. Recommendations Recommendation #1: Determine whether indicators should focus on nutritional assessment or more general assessment. The original focus of this sub-component was 32 The Sphere Project was tasked with identifying minimum standards to be attained in disaster assistance in each of five key areas (water supply and sanitation, nutrition, food aid, shelter, and health services). The resulting handbook is the standard which most international organizations (including most Title II Cooperating Sponsors) use to identify indicators that they use to monitor emergency programs. ARC ICB Mid-Term Evaluation. October 18, 2006. IR1. 18 on nutritional assessment techniques. The different departments need to decide if this is still the principal concern or if they are looking at more general types of “core indicators” for all their projects. Recommendation #2: Consolidate information on ARC and IFRC’s existing experiences with emergency indicators. Once the decision has been made regarding focus, the Technical Assistance Unit needs to work with the International Disasters Response unit in consolidating “lessons learned” from emergency indicators currently being used by ARC and IFRC programs. One staff member recommended that the best way for achieving this would be through a series of small regional workshops . Recommendation #3: Identify other valuable experiences and indicators from Title II Cooperating Sponsors. Execute a small workshop to which outside experts (e.g., from CARE or CRS) will be invited to comment on the proposed ARC indicators. Recommendation #4: Identify a reduced number of “core indicators” that can be pilot tested. One staff member suggested that this was likely to be a larger list of 10 or so indicators. 2.1.4. Sub-output 1.1.4 (recommended for addition at mid-term): Stronger bridges and linked understandings between technical units in International Programs at ARC 2.1.4.1. Accomplishments to date and evidence of existing or projected impact Bridge building between the technical units in ARC’s International Services Department is expected to be a major sub-output of the ICB grant. Indeed, one of the major under￾documented institutional outcomes of the grant has been to foster this type of bridge building via training, workshops, “lateral” technical assistance of the senior food program advisor on projects in IDR and IP, joint TAU-IDR field missions (Box 2.1 and Annex I), and joint planning of the pilot country programs. This critical role of the ICB in building bridges was highlighted in almost every one of the HQ based interviews. Box 2.1: Reflections on the Impact of a Joint IDR (International Disaster Response)-TAU (Technical Assistance Unit) Internal Evaluation of an Emergency Relief Program in Sri Lanka Facilitated by the ICB This work—emergency response work—does not lend itself to reflection or revisiting both because of time constraints and because it is often such an intense experience that people don’t have the energy to look back. This experience [in Sri Lanka] let us go back and reassess what we did—beneficiary input, transition out of emergencies, etc. It gave us a chance to look deeply into these issues. Because I participated in the evaluation and am still working here, it has helped us to actually implement some of the recommendations. It gave us a chance to reflect and to put that reflection to good use in a real way. It confirmed things for us and it highlighted the need for quality in emergency response. Source: Interview, Tracy Reines, IDR/ARC, September, 2006. ARC ICB Mid-Term Evaluation. October 18, 2006. IR1. 19 2.1.4.2. Factors that have or may contribute to or detracted from development, utility, and viability of outputs Key factors that have contributed to building-bridging between the ARC International Service Department’s technical units were (Annex I): joint trainings (especially in IPP); joint supervision, planning, and evaluation planning; joint project proposal development and review; and “lateral” advocacy and technical assistance by the senior food program advisor for existing programs. Probably the single most important factor that detracted from the ICB’s attempts at bridge-building and joint programming was the dearth of food programming staff between January 2005 and the March 2006. A quick analysis of the “management milestones” (Table 4.1 in section four) of this report shows a quick resurgence of joint activity once a new senior food program advisor was confirmed in March 2006. ARC’s commitment to increasing the percentage of transition and recovery programs from about 10 percent of the International Programs unit’s portfolio to 40 percent over the next three years offers an unprecedented opportunity to test these bridges with new innovative programming and capacity building. Promising initiatives that are planned for FY07-FY08 include: • Allocating 12 percent of the time of a key IDR staff person to train Regional Disaster Response Teams (RDRT); • Joint collaboration between TAU and IP in the development of the SOW for the ICB-funded regional food program advisor to be based in Harare and for the pilot programs; • Joint collaboration between IDR unit and TAU in consolidation of materials to be used by the senior food program advisor and the senior advisor for relief for training the RDRT and ONS staff in the pilot countries (Mozambique, Ethiopia, Lesotho); and • Expressed willingness between IDR and TAU to collaborate on an internal review of ARC and IFRC emergency indicators that can be pilot tested as “core indicators” in projects in all three ISD operational units (IP, TRP, and IDR). 2.1.4.3. Recommendations To date, the ICB’s success in building core capacity within the technical units and more effective bridges is hard to document. Given the critical importance of these sub-outputs to ARC’s achievement of its broader goals, the following recommendations are made. Recommendation #1: Create sub-output. Create sub-output for output 1.1 focused on building “bridges” between the operational units affected by food programming. Recommendation #2: Create indicators. Using the same methodology being proposed for the capacity indicators discussed for sub-output 1.1.1, the food program advisor ARC ICB Mid-Term Evaluation. October 18, 2006. IR1. 20 should work with IDR, IP, and TAU staff to identify variables and indicators with which to track this bridge building.33 2.2. Output 1.2: Expanded collaboration between ARC and other PVOs [and institutions]34 in the design and management of food and nutrition interventions in emergencies 2.2.1. Accomplishments to date and evidence of existing or projected impact ARC’s comparative advantage lies in its extensive volunteer network, “surge capacity” in disasters, and “brand recognition.” While these advantages are well adapted to quick responses to disasters, they are less well adapted to more complex technical support. For this reason, ARC has anticipated the need for building its capacity to negotiate and manage complex partnerships with other international NGOs that have these skills. This increased capacity to manage partnerships was deemed a critical input into the attainment of IR1. When the ICB proposal was written, ARC anticipated having at least two Title II programs that would be executed with other partners, a long-term collaboration with CRS and a series of applied research partners. The original indicators for this output were designed to track these partnerships.35 Between FY03 (when the grant was designed) and the end of FY06, ARC has dramatically increased its interest in and willingness to engage in technical partnerships as both a grantee and sub-contractor and even as a donor (Annex I). While the indicators may have been useful when the project was designed (and had fewer partnerships) the current scale of partnerships has become so large that it is hard to track items like the “# of repeat collaborations.” In the current context, it is more useful to discuss broad categories of partnership, rather than the summary figures on all the partnerships. Collaboration with individual Title II Cooperating Sponsors in Title II program design: ARC was one of several international organizations and PVOs in the consortium (the 33 Two indicators that were proposed during interviews include: “# of tools/activities in work plan being co-executed by at least two technical units in the International Services Department” and “# of staff that have attended TAU-sponsored training programs in the four core areas (food security basics; food management; beneficiary targeting; and monitoring, evaluation, and reporting)” (broken out separately for the TRP unit, IP unit, IDR unit, and TAU). 34 Change recommended at mid-term to accommodate WFP as a partner. 35 Impact Indicator 1: # of repeat collaborations (any type); Impact Indicator 2: # of sources of internal funding for collaboration; Monitoring Indicator 1: # of organizations involved in collaborations; Monitoring Indicator 2: Types of collaborations (e.g., proposals, TA, training, evaluations, conferences, etc.) Monitoring Indicator 3: # of collaborations; Monitoring Indicator 4: # of lessons learned from the collaborations developed. Monitoring Indicator 5: % lessons learned disseminated outside of partner organizations. ARC ICB Mid-Term Evaluation. October 18, 2006. IR1. 21 Center for Disease Control [CDC], Counterpart International, Save the Children, and Mercy Corps) that designed the Title II program for Tajikistan, whose funding was withdrawn, and in the consortium (CRS, CARE, Africare, Save the Children, Salvation Army and World Vision) for Malawi, where ARC withdrew when total funding for the project was reduced. Even though ARC was forced to withdraw from the Title II consortium in Malawi, the training provided by ARC senior M&E advisor (with support from the ISA) played a key role in determining the final design of the consortium’s M&E system. Although ARC’s involvement in neither grant came to fruition, the fact that they were willing to even discuss the partnerships marks a major shift in ARC’s thinking about its transition and recovery programs. Collaboration with the wider Title II CS community through the Interaction Evaluation Interest Group (EIG): One of the most interesting new partnerships (e.g., that was not envisioned in the proposal) has come out of ARC’s willingness to host and help organize the Interaction Evaluation Interest Group (EIG). Since the FAM (Food Aid Management) grant terminated at the end of FY03, the EIG has taken over some of the functions once played by the FAM M&E working group. Especially important, the EIG has created one of the main forums for international PVOs, such as ARC, to share lessons learned from various ICB-supported applied research and capacity building efforts. These lessons learned are shared during their regular meetings (an average of six per year), as well as the annual EIG roundtable, which is organized annually just prior to the American Evaluation Association (AEA)’s annual meetings. During the first three years of the grant (FY04-06), ARC has (Annex I): • Organized regular EIG meetings at ARC headquarters office in Washington (an average of every other month starting in FY0536); • Presented papers that summarize lessons learned about the different tools developed under the ICB at the EIG annual roundtables that precede annual meetings of the American Evaluation Association (AEA); and • Supported one staff member’s participation in developing an Interaction policy paper on “Demonstrating NGO Effectiveness” (self-certification process), which was ultimately reviewed and adopted by the Interaction board. Many of these activities, as well as the travel costs associated with staff participating in the annual EIG roundtable discussions, were supported by the ICB. Collaboration with CRS in the Title II M&E module series, which have been distributed to all the Title II Cooperating Sponsors: Traditionally, ARC has followed the tendency of most relief organizations to conduct only the most cursory sorts of monitoring and evaluation of its emergency operations. While this trend has improved in recent years, in general, the “relief” world is far behind the “development” world in identifying ways to track local level impact of their activities. For this reason, ARC was very interested in partnering with another Title II Cooperating Sponsor that had significant experience in development M&E, as well as M&E for emergencies. The output of this partnership has 36 The number of meetings actually hosted, as well as the timing, can vary slightly since the group tends to organize its activities around other events such as the annual meetings of the American Evaluation Association and Interaction. Five EIG meetings were organized at ARC Headquarters in FY05 and five meetings in FY06 (Source EIG, Interaction project records). ARC ICB Mid-Term Evaluation. October 18, 2006. IR1. 22 been a series (a total of ten are planned) of field friendly M&E modules. Three of the modules have been distributed in FY06; the other modules are scheduled for finalization and distribution in FY07 and FY08 (Box 2.2). To date one or more of the modules have been used to build capacity in the design or evaluation of several of ARC’s emergency, transition, and recovery programs (Annex I) and a high percentage of CRS’s Title II programs, as well as some of the Title II programs of the NGOs ACDI/VOCA (Agricultural Cooperative Development International-Volunteers in Overseas Cooperative Assistance) and Africare. Most recently, FANTA has distributed the pre￾evaluation module to several Title II CSs. Collaboration with the World Food Program and other international PVOs for Tsunami relief: A second new partnership that was not envisioned in the proposal emerged when ARC became a donor charged with execution of various sub-components of the $570 million donated to ARC for Tsunami disaster and recovery relief. The food security component of this assistance ($120 million) was routed through the World Food Programme; other components were executed through signed contracts with the World Wildlife Fund, Mercy Corps, and Plan and Community Habitat Finance International (CHF). There is qualitative evidence (from group interviews) that the technical backstopping of ARC’s TAU helped enhance WFP’s emergency response planning. Especially important was the TAU’s technical assistance and use of specific tools37 and staff38 to build WFP’s capacity to manage the dramatic “surge” in food programming that accompanied Tsunami emergency and recovery programs (Box 2.3). ARC’s partnership with WFP presents it with an unprecedented opportunity to look critically at the different aspects of food programming. This is a luxury that most PVOs cannot afford and one that will add tremendous value both to the ICB grant and the wider international community. Box 2.2: ICB Series of Field Friendly M&E Modules Produced in Partnership between ARC and CRS and Distributed in FY06 Success and Learning Story Writing Packet (Draft completed September 2004) (Released in version 1.0 in June 2006) Planning for High Quality Delivery of Capacity Building Services (Draft completed FY05) (Released in version 1.0 in June 2006). Hiring M&E Field Staff (Draft completed FY05) (Released in version 1.0 in June 2006) Preparing for Evaluation (Draft completed FY06) (Released in version 1.0 in June 2006) Source: Senior M&E advisor. 37 These tools included the IPP module, the revised IPP proposal guidance [which was adapted to the Tsunami context], and the ARC/CRS M&E modules. 38 Especially important was the secondment of one of the NHQ (National Headquarters) senior associate for food programming in the old TAPE unit [now TAU] to WFP. ARC ICB Mid-Term Evaluation. October 18, 2006. IR1. 23 Box 2.3: Lessons Learned from the Secondment of ARC Senior Associate for Food Programming to WFP “There are vast opportunities for working in coordination and collaboration with WFP (and a couple of other UN agencies, such as the United Nations High Commission for Refugees (UNHCR), in which ARC and the Red Cross Movement’s strengths can complement the strengths of WFP. In particular, the Red Cross Movement brings the community-based programming and social mobilization, as has been the case in the Measles Initiative. Not all of the opportunities involve ARC as a donor. Overall, ARC needs to be much more strategic in terms of how it develops its’ relationship with WFP and other partners.” Specific recommendations (that are described more in-depth in the report) include the following. • Both organizations need to gain a better understanding of one another in order for this relationship to become a true partnership. • Both organizations need to pay more attention to the transition from relief to rehabilitation and development and the need for consistent and regular communications. • Both organizations need to pay more attention to transferring the skills and tools gained by the seconded delegate from working with WFP to ARC food programming and/or emergency response. Source: Regis Chapman. 2006. American Red Cross International Services. Delegate End of Mission Report. Indonesia. Washington DC: ARC. Collaboration with major international research centers for applied research and trainings: During FY04, ARC worked with the Feinstein Institute to organize a “Food University.” Two years later, the resulting lectures were probably the single most direct impact that staff in the IDR and IP units reported as building their personal capacity to work with food programming. A second major collaboration is being negotiated with the Academy for Educational Development (AED) to study the link between food programming and improved health. ARC’s collaboration with the Fritz Institute (in Arizona) was critical to the development and initial pilot testing of the Humanitarian Logistics Software. 2.2.2. Factors that have or may contribute to or detract from development, utility, and viability of outputs One factor that contributed to ARC’s successful record in increasing the scale and complexity of its partnerships is the care with which these partnerships were screened and pilot tested before scaling up or elimination. One of the principal constraints to new partnerships (such as the partnership with WFP) is the lack of a clear understanding of the roles of the different partners vis-à￾vis one another. This was especially true in the case of the WFP-ARC partnership in Indonesia, where the ARC delegate was The World Food Programme became a partner of ARC after the Tsunami distributing $120 million worth of food aid in affected areas. Photo by Rein Skullerud, WFP ARC ICB Mid-Term Evaluation. October 18, 2006. IR1. 24 more or less absorbed into the WFP operations.39 This was not a problem in the ARC/CRS partnership, where each partner stayed rooted in their own institution and communication between partners was easy and constant. The new pilot country programs offer an opportunity to explore new partnership models, methods, and ways of screening partners. The principal risk associated with ARC’s commitment to executing most of its food programming through partnerships is that the agency will not be able to maintain sufficient control over program quality and financial management. To minimize this risk, all food programming partnerships must necessarily include some level of capacity building. 2.2.3. Recommendations Recommendation #1: Revise output text to reflect expanded universe of partners. Include “and other institutions” in the text to address the significant partnership that ARC developed with USAID-funded WFP during the Tsunami disaster relief effort. Recommendation #2: Revise the performance indicators for this sub-output to reflect the expanded universe of partners. The six indicators in the current ICB tracking table reflect the grant’s original intent to focus primarily on building the capacity of local societies associated with the new Title II programs and pilot programs that ARC was planning in FY03. Far more relevant in the current context would be to track specific types of partners that are emerging. Four categories of partnership that were identified during the mid-term evaluation include: - Number of long-term food partners; - Number of long-term M&E partners (for emergency, transition and recovery programs); - Number of organizations benefiting from ARC facilitated EIG network; and - Number of collaborations with research centers and/or individual researchers for applied research and training. 39 Chapman 2006: 7. ARC ICB Mid-Term Evaluation. October 18, 2006. IR1. 25 2.3. Output 1.3: Established [ARC domestic chapter advocacy network and] 40 support material for promoting the importance and need for food in international emergency responses [among ARC domestic chapters]41 2.3.1. Accomplishments to date and evidence of existing or projected impact At the request of USAID,42 ARC eliminated the concept of developing an “advocacy network” (which might be construed as self-serving), but retained the concept of promotional materials. With the support of the ARC Chapter International Support (CIS) unit, ARC designed a campaign to disseminate educational and promotional materials about the importance and need for food assistance in international emergency responses. ARC’s motivation for supporting this output was both internal and external. The external motivation was to support Food for Peace’s new strategy to develop global leadership by capacity building and advocacy base in its countrywide network of domestic chapters. The internal motivation was to show ways that (ARC 2003: 18): “ARC is a vital part of the larger Movement and, putting a face to need, can give these chapters different fundraising avenues with individuals and corporations.” While no targets were set for the first year, ARC invested heavily in the development of information featuring Africa and ARC’s bilateral emergency and non-emergency food programming for its public web site. This information was reinforced by (Annex I): • Presentations and displays at annual meetings; • Volunteer training courses that include a general orientation to international services; and • Various regional meetings of ARC chapters for large catchment areas, including Des Moines, Los Angeles, and Chicago. 2.3.2. Factors that have or may contribute to or detract from development, utility, and viability of outputs The strong alignment between ARC’s internal goals for building chapter understanding of and willingness to support international emergency and non-emergency operations and the output 1.3 of the ICB is a major strength. This accounts for the tremendous over￾achievement of ARC’s goals (225%-659%) for the activities being tracked by ARC Communication and Marketing department for the ICB PITT (Table 2.2). Some of the best qualitative evidence of the impact of these activities is a threefold increase in the number of chapters contributing to emergency operations and a five fold increase in the number of chapters using some of the international case study material and “fact sheets” on the public access web site to build community awareness. 40 Text deleted at the request of USAID/FFP in FY04. To clarify the target audience the mid-term recommends adding “among ARC domestic chapters” to the phrasing of the output. 41 Text recommended for addition per mid-term evaluation. 42 USAID funds cannot be used in ways that appear to advocate for increased tax payer contributions to USAID. ARC ICB Mid-Term Evaluation. October 18, 2006. IR1. 26 Table 2.2: Progress Toward the Execution of ARC’s Output of Establishing Support Material for Promoting the Importance and Need for Food in International Emergency Responses, FY04-FY06 FY04 FY05 FY06 PITT Indicators Base line Target Ach. Ach. vs. Target Target Ach. Ach. vs. Target Target Ach. Ach. vs. Target Official Indicators # of chapters contributing to emergency operations (food and non-food) 307 -- 327 -- -- 824 -- NYA NYA NYA # of chapters using info. pieces for community outreach 155 -- 106 -- 125 824 659% NYA NYA NYA # of info. pieces about food programming -- -- 8 -- 8 30* 375% NYA NYA NYA Types of info. pieces about programming -- -- 4 -- 4 9 225% NYA NYA NYA Other Indicators Main webpage (Africa Food Crisis) (June￾September 2006) Webpage visits** -- -- -- -- -- X 3,766 Webpage views# -- -- 4,672 Individual Webpage visits (range) -- -- 139- 436 Individual Webpage views (range) 174- 583 Applied research/focus groups -- -- -- -- -- 1 Int’l Services Poll (9 cities) 1 100% *Estimated. NYA=Not Yet Analyzed. **A visit is when a single person comes and visits the webpage in a defined time frame. They may refresh the webpage several times, but it still only counts as one visit. #Webpage views show how many times a webpage was loaded in a web browser. This is a good indicator for showing repeat use by the same visitor. This figure show that approximately 10 percent of the webpage’s visitors go further than the main webpage of African Food Crisis. Source: ARC 2004 and 2005 annual reports and online services, Communications and Marketing Department. ARC. September 2006. One important factor that contributed to ARC’s successful record in achieving this output is the capacity of its Communication and Marketing department. One role of this department is to monitor chapter interest and the extent to which ARC messages are having their intended impact. A recent (June 2006) ARC International Services poll of 2,300 adults, 18 years and older, residing within the jurisdiction of nine chapters (New ARC ICB Mid-Term Evaluation. October 18, 2006. IR1. 27 York, Philadelphia, San Francisco, Los Angeles, Chicago, Minneapolis/St. Paul, Miami, Washington DC, and Dallas) found that (Box 2.4): • Sixty-two percent of respondents were aware of that ARC donations were used to help people in other countries, compared to 29, six, and five percent who were familiar with the international work of UNICEF, Save the Children, and CRS, respectively);43 • Eighty-three percent of respondents were willing to send food and clothing as international assistance; and • After emergency support for the victims of international disasters, the strongest level of support was for disease prevention in Africa and supporting Africans suffering from drought and food shortages. Although the poll highlights that a substantial percentage of the US public recognizes ARC and its activities, these impressions are general (as they normally are for the public and media). In order to successful familiarize and solicit participation by university and policy-based actors (and open up new avenues for funding and collaboration) ARC needs to complement these general impressions with some more rigorous technical papers that tell its story based on some of the ICB-supported research and the programs’ ongoing M&E systems. Participation by and collaboration with university and policy communities requires this type of extensive investment in compilation and dissemination of the technical aspects (both in terms of reported impacts and innovative tools and methods) of food security interventions. Box 2.4: Results of Recent International Services Poll of US Adults in Nine Cities Concerning their Knowledge and Support of ARC International PVO Activities • Type of Assistance Willing to Support: Food and clothing (83%), health and medical services (77%), assistance to disaster victims (73%), and children’s educations (50%) and religious activities (34%). • Awareness of ARC International Activities: Providing services for disaster victims (80%); disease prevention in Africa (40%), supporting Africans suffering from drought and food shortages (45%). • Stories Recalled about ARC Services Outside of the US: Tsunami relief (14%), AIDS efforts (9%), earthquake relief (5%), Iraq War (7%), disaster relief--general (5%), efforts in Africa (4%). • Likelihood to Donate to ARC for International Support (responses: “highly likely”): Disaster relief in other countries (32%), disease prevention in Africa (26%), supporting Africans suffering from drought and food shortages (26%). Source: ARC Communication and Marketing Department. Market Research and Planning. 2006. American Red Cross International Services Poll. July 5, 2006.2006. Pg. 3 43 Response to the question: “When you think of organizations that collect donations in the United States to help people in other countries, what organizations do you think of?” (ARC Communication and Marketing Department 2006: 3). ARC ICB Mid-Term Evaluation. October 18, 2006. IR2. 28 2.3.3. Recommendations ARC’s public website is clearly an example of “best practice” in building domestic awareness about the general role of international PVOs and the US food assistance in international emergencies. The functional link between the website and the development of speakers and programs for ARC domestic US chapters programs enhances its impact. Recommendation #1: Rework the phrasing of the output (to reflect the deletion of advocacy as an output). Recommendation #2: Create a web-based technical paper series and webpage. ARC might consider making a limited number of the existing (and projected) ICB technical papers available on their website. By creating a designated space (website) and official title for the technical paper series, members of the general public who are interested in the more technical aspects of ARC’s activities will have the resources to educate themselves. In addition, this series can serve as a somewhat separate resource for the other side (research, funding, and policy) of ARC’s activities. 3.0. Intermediate Result 2: Red Cross/Crescent Movement Partners Develop the Capacity to Design and Manage Food Aid and Nutrition Interventions as an Effective Response to Emergencies To achieve IR2, ARC envisioned two outputs: • Output 2.1: Improved knowledge base in National Red Cross Societies to design and manage local food aid and nutrition interventions in emergencies and • Output 2.2: Enhanced coordination between National Red Cross Societies and local institutions in food and nutrition during emergencies. 3.1. Accomplishments to date and evidence of existing or projected impact F04-FY06: The genius of the original design of the ARC ICB proposal was its attempt to link new tools development, training, and pilot testing at the HQ-level with improved programming in two pilot programs before scaling up (Box 3.1). ARC originally considered Ethiopia, Cambodia, and El Salvador because of their fit with ARC criteria for participation in the pilot (Box 3.1). Although the ARC Technical Assistance Unit has provided extensive case by case technical assistance to specific countries (see Annex I), they are three years behind in the execution of the pilot programs. FY07-FY08: Given the much shorter time (two years) that ARC now has to execute the pilot programs from start to end and the current priority being placed on building National Society capacity in Africa, ARC has selected three African countries (Mozambique, Ethiopia, and Lesotho) for pilot projects that are slated to start in FY07. Parallel to this activity the senior food program advisor, Ange Tingbo, is collaborating with the senior advisor for relief, Tracy Reines, of the International Disaster Response unit, in the design of a series of trainings on food security for the southern Africa ARC ICB Mid-Term Evaluation. October 18, 2006. IR2. 29 Regional Disaster Response Teams (RDRT). To facilitate the realization of these activities: • The TAU, IDR, and IP units are collaborating on the recruitment and hiring of a new regional food program advisor who will be based in either Zimbabwe or Lesotho; • The IDR unit and TAU collaborated on a joint design and needs assessment mission in October 2006; • The IDR unit and TAU will need to collaborate and backstop one another on consolidation of the training modules that they will be using for the southern Africa regional training and pilot country training programs; and • The TAU is working with the IP unit on integrating a food component into some of the other longer term projects of the Mozambican Red Cross, especially HIV/AIDS home based care and possibly maternal and child health activities. Box 3.1: Original Criteria for Countries Selected for ARC ICB Pilot Projects “ARC proposes to field test technical programs in two countries. All developed tools and training can be shared with other PVOs, US government staff, and other organizations. The selected countries for the two pilot projects must meet the following criteria: • Be disaster prone (likely to experience an emergency during LOP); • Be food insecure; • Have an ARC presence; • Demonstrate a fit within the USAID mission’s strategic framework and have mission support; • Offer an opportunity to work with local partners; • Have the support of the host country National Society; • Represent a strategic intersection of needs, interests, resources, and capabilities for both ARC programming and that of the host National Society; • Offer a setting that will allow us to increase our capacity to do nutritional interventions; and • Have the potential to expand programmatically into longer-term work.” Source: ARC 2003: 14. 3.2. Factors that have or may contribute to or detract from development, utility, and viability of outputs The principal reasons for the delays in the pilot programs were staff shortages and priority shifts that resulted from the massive “surge” in ARC relief due to the Tsunami in FY05. This constraint was addressed by the confirmation of Ange Tingbo as senior food program advisor in March 2006 and the recent decision by IDR to commit a percentage of the time of senior advisor for relief, Tracy Reines, for field-level training. Lesotho (one of the pilot project countries) Red Cross volunteers distributing seeds door to door. Photo by Jeff Weiss, ARC ARC ICB Mid-Term Evaluation. October 18, 2006. IR2. 30 Weak inter-unit collaboration for conceptualization and execution of the pilot programs was another factor that hindered progress during FY04-FY06. This is no longer the case given International Programs’ interest in building its African portfolio and the heavy focus of the IDR unit and TAU on building National Society capacity in Africa. The three case studies that will result from the country pilot programs offer a rare opportunity to have an impact in the field and to extrapolate valuable lessons learned that can be incorporated when the model is scaled up. The chief challenge for the pilot programs will be to balance the two priorities of having an impact in the field and extracting lessons learned regarding the tools and model used, so that results are achieved for both priorities. 3.3. Recommendations Recommendation #1: Design and execute country programs (three planned) and RDRT training in southern Africa. Recommendation #2: Create a self-assessment capacity index. This index should provide a grid that the senior food program advisor and the regional food program advisor (for southern Africa) can use to help the ONS in the pilot countries identify and track their needs and progress in the five “core” areas being targeted by the pilot programs (food security basic concepts; Integrated Planning Process [IPP]; emergency/transition targeting of beneficiaries; food commodity management; and improved M&E for food aid programs) and two “emerging” areas (slow onset disasters and HIV/AIDS home based care). Recommendation #3: Reorganize the indicators to reflect new project activities and priorities. - 3.a: Add two new indicators that track the impact of better partnerships as well as the number and content of the partnerships;44 - 3.b: Replace all but one of the original impact indicators for IR2 (which would probably require more than two years to show a measurable impact 45) except “Percent improvement on score of Social Mobilization Index”; and - 3.c: Track the number of RDRTs trained in southern Africa. 44 Sample indicators might include: Number of Pilot Countries that have developed new projects that include food programming as a result of their being perceived as more viable partners; and Partnership Capacity Index for the pilot countries (based on a capacity index similar to the one recommended in 3.a above). 45 These original indicators were (ARC 2003: Annex D): 1. Reduction in wasting among US food recipients; 2. Increase surge capacity 1. % of emergency proposals field initiated 2. % of emergency proposals using integrated approach 3. % of emergency proposals that require six month timeframe funded 3 3 3 3 3 3 15 # emergency proposals developed for >six month timeframe 6 5 5 5 5 5 26 # emergency proposals developed 10 5 5 5 5 5 30 Indicator Performance Tracking Table for ARC ICBA Grant Annual Report Format Year One Year Two Midterm Year Three Year Four Year Five End of Project Totals ARC ICB Mid-Term Evaluation. October 18, 2006. Annex II. 64 Baseline Planned Actual A/P Planned Actual A/P Planned Actual A/P Planned Actual A/P Planned Actual A/P Planned Actual A/P Planned Actual A/P 1.1.3 standardized monitoring and evaluation systems to measure the nutritional impact of food in emergencies # emergency M&E core indicators approved TBD** # emergency projects reporting to standard 10 10 15 15 15 15 70 1.2 expanded collaboration between the ARC and other PVOs in the design and management of food and nutrition interventions in emergencies # repeat collaborations (any type) 4 6 6 6 6 6 28 # sources internal funding for collaboration 2 3 3 3 3 3 14 # of organizations involved in collaborations 3 5 5 5 5 5 23 types of collaborations (eg., proposals, TA, trainings, evaluations, conference, etc.) 3 3 3 3 3 3 15 # of collaborations 10 10 10 10 10 10 50 % lessons learned from collaborations (such as case studies, reports, etc.) disseminated (eg, to Movement partners, FAM, USAID, etc.) TBD 1.3 established ARC domestic chapter advocacy network and support material for promoting the importance and need of food in international emergency responses # chapters with diaspora populations contributing to emergency operations (food and non-food) TBD** # chapters using info pieces for community outreach TBD** # of info pieces about food programming TBD** types of info pieces about food programming TBD** Intermediate Result 2: Red Cross/Crescent Movement partners develop the capacity to design and manage food aid and nutrition interventions as an effective response to emergencies increase in surge capacity index score (individually tracked for Movement partners) TBD** # people assisted TBD** amount of food distributed TBD** % trained volunteers retained for >1 emergency response TBD** % improvement on score of social mobilization index TBD** % response time towards ideal industry standard TBD** # of weeks to response time (start of program) TBD** planned/actual time distribution TBD** Annual Report Format Y ARC ICB Mid-Term Evaluation. October 18, 2006. Annex II. 65 ear One Year Two Midterm Year Three Year Four Year Five End of Project Totals Baseline Planned Actual A/P Planned Actual A/P Planned Actual A/P Planned Actual A/P Planned Actual A/P Planned Actual A/P Planned Actual A/P Output 2.1: Improved knowledge base in national Red Cross societies to design and manage local food aid and nutrition interventions in emergencies % trained members/volunteers retained > one year TBD** # people trained TBD** # courses offered TBD** % people taking >1 course TBD** Output 2.2: Enhanced coordination between national Red Cross societies and local institutions in food and nutrition during emergencies # MoUs established between pilot national societies and national/local governments and NGOs TBD** # repeat collaborations (any type) in-country TBD** # of collaborations TBD** ** measured only in two pilot countries if there is an emergency TBD = to be determined (new indicator, new application of old indicator) ARC ICB Mid-Term Evaluation. October 18, 2006. Annex II. 66 PITT Submitted with FY05 Annual Report to USAID/FFP Year One Year Two Baseline Plan Actual Plan Actual A/P LOP Strategic Objective: To develop the capacity of ARC and the International Red Cross Movement to protect and promote the nutritional well-being of people affected by emergencies Maintenance or reduction in wasting rates among US food recipients and/or weight for age as appropriate.1 n/a n/a n/a n/a 3 programs reaching 2,100,000 100% % of ARC emergency responses using food2 0 2 3 n/a 4 # ARC emergency responses 15 n/a 13 n/a 11 IDRU Intermediate Result 1: ARC improves its institutional and technical capacity to design and manage food aid and nutrition interventions as an effective response to emergencies Increase in surge capacity index score (aggregate for AmCross) TBD n/a n/a # people assisted 474,852 2,308,7203 2,783,572 Amount of food distributed TBD 210,000MT4 tsunami 9,544 MT Vietnam % of staff deployed 11% 10% 35%5 # of staff deployable 114 114 114 % response time towards ideal industry standard TBD n/a n/a n/a # of weeks to response time (start of program) TBD n/a n/a One day for tsunami, food within 1 The 3 programs refer to WFP general distributions in Indonesia, Sri Lanka and Maldives that were partially funded by ARC. The Vietnam school feeding program does not track wasting of under-fives. 2 The Indonesia, Sri Lanka and Maldives Tsunami response and Vietnam school-feeding. 3 2,100,000 tsunami and 208,720 Vietnam 4 60,000MT of 210,000MT was financed by ARC 5 This refers to deployment of 40 IDRU staff, over a previous number of trained staff since no training occurred in FY 05. However, trained “staff” does not actually equate to staff deployable since the Int’l Response team members are not staff (volunteers) and many have jobs so are only deployable with employers OK. We need to rethink the way we mange emergency rosters and this indicator may change after the mid-term ARC ICB Mid-Term Evaluation. October 18, 2006. Annex II. 67 Year One Year Two Baseline Plan Actual Plan Actual A/P LOP 5 Output 1.1: enhanced ARC staff technical knowledge base % staff with 80% or above scores on Ex Post Tests for IPP (six month/annual increments) TBD 0 % trained staff retained 69.50% 69.50% uncertain6 % increase on pre/post test scores TBD 0 # trained people retained 130 130 # people trained 187 15 0 20 37 # courses offered 2 1 0 3 0 1.1.1: improved tools for needs assessment, nutritional surveillance, and planning (e.g., food basket calculator) for emergency interventions # tools developed 2 2 2 1 0 # projects using tools in emergency programming 0 n/a 0 0 % tools disseminated (e.g., to Movement partners, FAM, USAID, etc.) 1 n/a 0 0 1.1.2: enhanced emergency response planning that integrates food and nutrition with water/sanitation and other public health interventions % food/nutrition emergency proposals funded* 3 3 4 4 # food/nutrition emergency proposals developed* 6 5 6 5 # emergency proposals developed 10 5 19 39 1.1.3: standardized monitoring and evaluation systems to measure the nutritional impact of food in emergencies # emergency M&E core indicators approved TBD 0 # emergency projects reporting to standard 10 10 0 6 We have not tracked this indicator properly with all the tsunami movement, and old staff in and out of deployment. What is certain is that with over 60 new staff in international programs, training in the Integrated Planning Process is a priority for 06! 7 Does not include on-the-job training in food/nutrition assessment but participation in more formal conferences and courses. ARC ICB Mid-Term Evaluation. October 18, 2006. Annex II. 68 Year One Year Two Baseline Plan Actual Plan Actual A/P LOP 1.2: expanded collaboration between the ARC and other PVOs in the design and management of food and nutrition interventions in emergencies # repeat collaborations (any type) 4 6 6 4 48 # sources internal funding for collaboration 2 3 3 2 3 # of organizations involved in collaborations 3 5 19 10 169 Types of collaborations (e.g., proposals, TA, trainings, evaluations, conference, etc.) 3 3 5 3 3 # of collaborations 10 10 9 11 16 % lessons learned from collaborations (such as case studies, reports, etc.) disseminated (e.g., to Movement partners, FAM, USAID, etc.) TBD 0 1 2 1.3 established ARC domestic chapter advocacy network and support material for promoting the importance and need of food in international emergency responses # chapters contributing to emergency operations (food and non-food) 307 327 n/a 824 # chapters using info pieces for community outreach 155 106 125 824 # of info pieces about food programming TBD 8 8 ~30 Types of info pieces about food programming TBD 4 4 9 Intermediate Result 2: Red Cross/Crescent Movement partners develop the capacity to design and manage food aid and nutrition interventions as an effective response to emergencies Increase in surge capacity index score (individually tracked for Movement partners) TBD** # people assisted TBD** Amount of food distributed TBD** % trained volunteers retained for >1 emergency response TBD** % improvement on score of social mobilization index TBD** % response time towards ideal industry standard TBD** 8 These are the m&e modules 9 In addition to the EIG participants, and the Use support participants, the ARC collaborated with WFP, FAO and WWF on food security and nutrition program design. ARC ICB Mid-Term Evaluation. October 18, 2006. Annex II. 69 Year One Year Two Baseline Plan Actual Plan Actual A/P LOP # of weeks to response time (start of program) TBD** Planned/actual time distribution TBD** Output 2.1: Improved knowledge base in national Red Cross societies to design and manage local food aid and nutrition interventions in emergencies % trained members/volunteers retained > one year TBD** # people trained TBD** # courses offered TBD** % people taking >1 course TBD** Output 2.2: Enhanced coordination between National Red Cross Societies and local institutions in food and nutrition during emergencies # MoUs established between pilot national societies and national/local governments and NGOs TBD** # repeat collaborations (any type) in-country TBD** # of collaborations TBD** ARC ICB Mid-Term Evaluation. October 18, 2006. Annex III. 70 Annex III: Supplemental Tables Table 1: Formal Trainings, Workshops, and Meetings Related to the Title II ICB Grant (FY04- FY06) Year Types of Training Themes Participation Location FY04 ICB-Supported Formal Training: Institutionalized Learning and Professional Development in Title II food aid programs by Tufts University’s International Feinstein Famine Center ƒ Introduction to Food Programming ƒ IFRC’s Strategy in Food and Nutrition ƒ Nutrition 101: Nutrition in Emergencies ƒ Port Discharge and Commodity Reception and Procedures ARC International services staff (incomplete records of staff who attended)80 ARC/HQ FY05 Workshop Staff Attended: Therapeutic Care ƒ Inter-Agency Community￾Based Therapeutic Care Two food programming staff ARC FY05 Informal Training: Needs Assessments in Emergencies ƒ Needs Assessments in Emergencies Two HQ staff ARC FY05 Workshop Staff Attended: Food Aid Conference ƒ USAID Rules and Regulations in Commodity Management ƒ Hearth/Positive Deviance Methodology One HQ programming staff Kansas City FY05 Partners Meeting: Evaluation Interest Group (EIG) ƒ Improving/Standardization Design, M&E Practices ƒ Rolling up Institutional Impact Indicators Eight PVOs ARC FY05 Routine Title II Meetings ƒ FFP Strategies and Regulations ƒ FANTA: Household food Insecurity Index Various PVOs FFP & AED FY05 Chapter Networking ƒ International Food Programming 80+ community, university, business and government officials from three states Des Moines FY05 Workshop Staff Attended: Commodities Management ƒ Commodity Management System Eight ARC and Viet Nam Red Cross Society staff Viet Nam 80 Through the website, some of the modules were promoted to more than 30,000 ARC Employees, Volunteers and nearly all the 879 domestic chapters. ARC ICB Mid-Term Evaluation. October 18, 2006. Annex III. 71 Year Types of Training Themes Participation Location FY05 Partners Meeting: American Evaluation Association ƒ Quality Control ƒ Institutional Indicators CARE, CRS, Winrock, Mercy Corps, USAID and other Int’l PVOs Toronto FY05 General Awareness ARC HQ (Capacity Building Basic Food Security Concepts): World Food Day and United Against Hunger ƒ General Awareness ARC/HQ staff, chapters network and general public ARC/HQ FY05 ICB Supported Formal Training: Special Course for the Relief Delegates ƒ Targeting, Monitoring and Distribution Persons deployed ARC/HQ by IDR &TAU FY05 ICB Supported Formal Training: HQ staff Capacity Building ƒ Needs Assessments ƒ Capacity Building in Food Programming in Large￾Scale Disasters ICB Manager (Regis Chapman) ARC/HQ by Tufts Consultant FY06 ICB Supported Formal Training: Integrated Planning Process ƒ Country Needs Assessments ƒ Strategic Planning ƒ Project Design & Proposal Development 60+ field staff Sri Lanka, Indonesia, Maldives, etc. FY06 Informal On-Site Training : Monetization ƒ Contract with Buyers HQ staff and one field staff HQ and Viet Nam FY06 Workshop Staff Attended: Food Aid Management Training ƒ USAID Food Commodity Management Principles ƒ MYAP & SYAP Design Two field staff and grant mgr Tanzania FY06 Workshop Staff Attended: Food Security Approach ƒ Risks & Vulnerability Reduction ƒ Relief and Development Continuum Grant mgr and 25+ field Red Cross staff Johannesburg Source: ARC Senior Food Program Advisor, ARC Senior M&E Advisor, ARC TA Unit Director based on project records. September 2006. ARC ICB Mid-Term Evaluation. October 18, 2006. Annex III. 72 Table 2: Wider Policy and Institutional Context of ARC ICB Grant at Time of Grant Design and Mid-Term (ARC, IFRC, National Societies, and USAID/FFP) When ICB was Designed Mid-Term ARC Food programming institutional context: ARC was considering having a stand-alone food sub-unit that would work with other IDR (International Disaster Relief) units to build capacity for and oversee food programming. Food programming institutional context: É ARC decided that all food programming must be conceptualized as part of a disaster response package (through their IDR office) and any parallel MCH programming through the national RC and any local partners working with them on this sub-component rather than a stand-alone sub-unit. É Decision made that IDR/ARC will not incorporate food into emergency response programs (e.g., those responding within two weeks to three-six months) in FY06 ISD: TAPE, IP, and IDR units involved in the grant ISD: Three operational departments dealing with emergency food programming: International Programs unit (approximately 50 staff), Tsunami Response Program unit (approximately 85 staff), and International Disaster Response unit (approximately six staff). Creation of Technical Assistance Unit (TAU): A technical resource for all three departments (approximately 16 staff). Role of ISD units in food programming: • IDR managing some food assistance in emergencies Role of ISD units in food programming: É International Disaster Relief (IDR) unit that oversees international disaster relief does not plan to incorporate food into its ARC response portfolio, but wants to understand food security basics since it is a priority for Movement partner. É International Programs (IP) (which focuses on development and recovery) is planning to use food in longer term programs. É International Programs committed to increasing portfolio for Africa from approximately 10 to 40 percent. Transition Programming: ARC food programming almost exclusively in connection with disaster/emergency programming (first three months), which was seen as the area where ARC had its greatest comparative advantage. Transition Programming: É Tsunami and African food crisis highlight importance of local societies facilitating the types of partnerships with other institutions that can address the root causes of slow onset disasters that make populations vulnerable. É Integrated food-water/sanitation-health is increasingly standard in ARC supported recovery phase interventions executed through partnerships since ARC’s comparative advantage lies in emergency response and volunteer network. ARC ICB Mid-Term Evaluation. October 18, 2006. Annex III. 73 When ICB was Designed Mid-Term Partnerships: No previous experience with large￾scale partnerships with WFP and limited experience in acting as a sub-contractor or sub-grantee on larger consortia efforts. Partnerships: Positive outcome of ARC routing food assistance through WFP sets the tone for future initiatives that could pave way for additional collaboration between WFP and ARC and between WFP and National Societies both for emergency, transition, and recovery programming. More and more long-term programming is executed with partners. Perception of Food Programming as High Risk: Food programming seen as risky. Perception of Food Programming as High Risk: É Food programming is still seen as risky (even though data shows that pro-active training and technical backstopping reduces risk). É Steady increase, however, in situations where use of food is needed to reduce vulnerability in Federation and ARC bilateral projects. Federation (RC/RC Movement) Federation Priorities: Primary focus on emergencies Federation Priorities: Federation decided to focus attention on improved disaster management at same time they are increasingly adopting food security/insecurity approach. Current issue of Federation magazine focuses on food security. National Societies Priorities of National Red Cross Societies: Strong volunteer networks through National Societies represent an effective mechanism for mobilizing relief during disasters. Priorities of National Red Cross Societies: É National Societies (183) are wanting more than short-term interventions, especially in areas where disasters are recurrent. É A growing number of National Societies, especially those in chronically food insecure countries in Africa that are prone to natural disasters, are incorporating food into their response to domestic disasters. USAID/FFP USAID/FFP Food Policy Paper (1995): Traditional focus of ARC on emergencies meant that there was no strong overlap between their comparative advantage and goals of USAID/FFP 1995 food policy paper. USAID/FFP Strategy Paper (2003 draft; 2005 final approved), which builds on 1995 Policy Paper: (+) Strong overlap between ARC priorities and support (through its collaboration with WFP and USDA, as well as Title II) for USAID in its 81 Need to add citation to this portion of the strategy paper. ARC ICB Mid-Term Evaluation. October 18, 2006. Annex III. 74 When ICB was Designed Mid-Term stated objective of becoming more of a global leader in reducing food insecurity (2005 USAID/FFP Strategy Paper) by influencing: É Design, implementation, and monitoring of quality for food programming that builds vulnerable people’s capacity to identify and manage risk and É Donor public (especially US donor public) perceptions about foreign assistance and food aid. (-) ARC’s two Title II proposals for FY05 were not funded. (-) Less overlap in priority countries between ARC and FFP (-) FFP continues to face a challenging funding environment and ARC is reluctant to link RC National Societies with donor funds that may be cut in mid-course. (+) New 2005 strategy paper is concerned with increasing USAID’s leadership role in influencing the wide food aid community (e.g., WFP, USDA)81 (+) Despite the risks, monetization and distribution of food from USAID Title II and USDA sources represents one of the accessible means that ARC has for helping National Societies build longer term program that addresses root causes of disasters and facilitates recovery. These sources are likely to become even more important once Tsunami funds are depleted. ARC ICB Mid-Term Evaluation. October 18, 2006. Annex V. 75 Annex IV References Consulted Official Guidance, Proposal, and Modifications of Assistance USAID/FFP. 2003. Guidance for the Title II Institutional Capacity Building (ICB) Program Monitoring and Evaluation Plan. ARC. 2003. Project Proposal. Institutional Capacity Building Assistance Grant. Washington, DC: ARC. ARC. 2003. Modification of Assistance. ARC ICB Grant. Award. Washington, DC: ARC. (August 23, 2004). ARC. 2003. Modification of Assistance. ARC ICB Grant. Award. Washington, DC: ARC. (September 18, 2004). Annual Work Plans and Reports ARC. 2004. FY04 Annual Work-Plan. Institutional Capacity Building Project. Washington, DC: ARC. ARC. 2004. FY2004 Annual Performance Report: Institutional Capacity Building Project. October 1, 2003-September 30, 2004. Washington, DC: ARC. ARC. 2005. FY2005 Annual Performance Report: Institutional Capacity Building Project. October 1, 2004-September 30, 2005. Washington, DC: ARC. ARC. 2006. FY07 Annual Work Plan. Institutional Capacity Building Project. Washington, DC: ARC. Internal Audits Asgeirsdottir, Svafa H. 2004. Interoffice Memorandum to Gerald Jones. Washington, DC: ARC. (November 1, 2004). World Food Programme Partnership World Food Programme (WFP). 2005. World Food Programme Funding Proposal to American Red Cross. Washington, DC: ARC. (14 January 2005). World Food Programme (WFP). 2005. Real-Time Evaluation of WFP’s Response to the Indian Ocean Tsunami—December 2004-June 2005. Washington, DC: WFP. (November 9, 2005; WFP/EB.2/2005/6-C/Rev.1). ARC ICB Mid-Term Evaluation. October 18, 2006. Annex V. 76 Pangaribuan, Rosnani and Luh Ade Ari Wiradnyani. N.d. Health and Nutritional Status Survey for Primary School Children: in Nanggroe Aceh Darussalam Province and Nias Island. Washington, DC: ARC. Chapman, Regis. 2006. Delegate End of Mission Report: Indonesia. Washington, DC: ARC. ICB Monitoring and Evaluation Documents ARC. 2004. Monitoring and Evaluation Plan: Institutional Capacity Building Project. Washington, DC: ARC. IPP (Integrated Planning Process) ARC. 2006. Integrated Planning Process: Environmental & Organizational Assessment (EOA) Guide. Version 3.0. Washington, DC: ARC. (January) (IP1). ARC. 2006. Integrated Planning Process: Regional Strategic Planning Guide. Washington, DC: ARC. (January 2006) (IP2). ARC. 2006. Integrated Planning Process: Country Annual Planning Guide: Country￾Level Implementation Guidance for Regional Strategic Plans. Washington, DC: ARC. (August 2005 revised in January 2006). (IP3) ARC. 2006. Integrated Planning Process: Project Design & Proposal Writing Guide. Washington, DC: ARC. (January 2006). (IP4) IPP Training Modules IRD Integrated Planning (January 2006) Integrated Planning Process: Regional Strategic Planning (January 2006) Integrated Planning Process: Country Annual Planning (January 2006) Integrated Planning Process: Project design Training (January 2006) ARC and CRS M&E Module Series Long, T. 2004. Success and Learning Story-Writing Packet. Version 1.0. ARC/CRS M&E Training Module Series. No. 1. Washington, DC: ARC and CRS. Norem, Rosalie and Constance M. McCorkle. 2005. Planning for High-Quality Delivery of Capacity-Building Services. Version 1.0. ARC/CRS Monitoring and Evaluation Series. ARC/CRS M&E Training Module Series. No. 2. Washington, DC: ARC and CRS. Norem, Rosalie. 2005. Hiring M&E Field Staff. Version 1.0. ARC/CRS M&E Training Module Series. ARC/CRS M&E Training Module Series. No. 3. ARC ICB Mid-Term Evaluation. October 18, 2006. Annex V. 77 Washington, DC: ARC and CRS. McMillan, Della E. and Alice Willard. 2006. Preparing for the Evaluation: Guidelines and Tools for Pre-evaluation Planning. Version 1.0. ARC/CRS M&E Training Module Series. No. 4. Washington, DC: ARC and CRS. McMillan, Della E., Guy Sharrock, and Alice Willard. 2006. Guidelines and Tools for the Preparation and Use of Performance Indicator Tracking Tables. Version 1.0. (Draft) ARC/CRS M&E Training Module Series. ARC/CRS M&E Training Module Series. No. 5. Washington, DC: ARC and CRS. ARC and IFRC Partnership—Logistics Software & Food Security Paradigm ARC. 2005. Annex 4. Secondment Agreement. Washington, DC: ARC. (November 1, 2005). Kim, Benjamin. 2006. Supply Chain Management—ARC Staff on Loan Secondment. International Federation of Red Cross and Red Crescent Societies Status Report. Reporting Period 8 July 2006-4 August 2006. North, Rosemarie. 2006. Food Security--A Paradigm Shift. Red Cross Red Crescent. The Magazine of the International Red Cross and Red Crescent Movement. Issue 2. Geneva: IFRC. Marketing and Communication Reports ARC Communication and Marketing Department. Market Research and Planning. 2006. American Red Cross International Services Poll. July 5, 2006. Title II ISA Project Mason, John P. 2001. Final Evaluation: Title II/FFP Institutional Support Assistance Grant (for the period covering October 1, 1998-June 30, 2001). Washington, DC: ARC. Commodity Management Smith, Mark J. and Nick Denton. 2004. The ARC Commodity Management Tool Kit. Washington, DC: ARC. Miscellaneous Sphere Project. 2004. Humanitarian Charter and Minimum Standards in Disaster Response. 2004 Edition. Geneva: Sphere Project (Distributed by OxFam). ARC ICB Mid-Term Evaluation. October 18, 2006. Annex V. 78 Annex V: Evaluation Schedule and List of Persons Interviewed Evaluation Schedule Dates Activity Outputs August 15 Pre-evaluation planning Identification of key pieces of quantitative data needed for the evaluation (Table) August 31 Pre-evaluation planning Outline for pre-evaluation document briefing book, the project activity briefing book (memo) September 5-September 11 NHQ interviewing and data collection -List of project activities -Revised set of indicators (to consider when assessing revised program) -Format for table documenting staff turnover -Management milestones (draft) -Policy context table (draft) NHQ draft September 12-15 Additional NHQ interviewing and basic data collection -Draft list of activities FY04- FY06 September 18-22 Preparation of draft document Draft report and executive summary, including provisional list of recommendations September 25-29 Revision of draft document (with Ange Tingbo) Draft document and executive summary completed October 6-8 Additional interviews and review Review and revision of draft October 17-18 Debriefing, discussion, and review of final document Final for submission to USAID ARC ICB Mid-Term Evaluation. October 18, 2006. Annex V. 79 List of Persons Interviewed or Included in Group Meetings Name Position Location Org. Regis Chapman Former field officer in Indonesia, former acting grant manager for ICBA; currently WFP staff in Thailand Thailand ARC Alice Willard Senior technical advisor, Monitoring & Evaluation Washington ARC Pat McLaughlin Director, Technical Assistance Unit Washington ARC Harold Tarver Director, Food for Development unit (used ARC/CRS pre-evaluation module) Washington Africare Jim Rugh Senior M&E advisor, CARE Atlanta CARE Guy Sharrock Senior M&E advisor, CRS Baltimore CRS Mark Smith Former technical advisor, Food Programs, former ICBA grant manager, currently director, Hurricane Recovery Program Washington ARC Luke Greeves Senior director, International Programs (former director of Technical Assistance Unit) Washington ARC Ange Tingbo Senior technical advisor, Food Programs Washington ARC Nan Buzard Director, International Disaster Response Washington ARC Svafa H. Asgeirsdottir Program implementation officer Washington ARC Ben Kim Chief technology officer seconded to the Federation Geneva IFRC Lisa Witte ICB grant officer FFP (not confirmed yet) Washington FFP Rachel Lucas Senior advisor, HIV/AIDS Washington ARC Michael Cohen Tsunami M&E advisor Washington ARC Scott Chaplowe Tsunami M&E advisor Washington ARC Tracy Hightower Program assistant Washington ARC Jean Koepsell Senior program advisor, Africa Washington WFP Leslie Stewart Program officer, Africa Washington ARC Alison Bain-Peachey Program assistant, International Programs Washington ARC Carol Miller International communications lead Washington ARC Carol Cernojevich Senior associate, International Communications Washington ARC Marcella Baldwin Administrative assistant Washington ARC ARC ICB Mid-Term Evaluation. October 18, 2006. Annex VI. 80 Annex VI: Evaluation Scope of Work (SOW) ICBA Mid-Term Review Draft SOW Background As with many five year grants, this one has undergone several changes of direction as the grantee has altered its stance on the purpose and mechanics of food programming. The mid-term review will take into account the initial proposed activities, as well as changes made more recently to respond to those organizational shifts. A. Original Proposal “The American Red Cross requests $1,879,094 in USAID funds to implement an Institutional Capacity Building Program over a five year period beginning in FY04. In addition to the funds requested from USAID, the American Red Cross (ARC) will commit an additional $1,096,766 in matching funds and request a further $265,500 in matching funds from the International Federation of Red Cross and Red Crescent Societies (IFRC). The total budget for program implementation is $3,241,359 over the five year period. The combined ARC - IFRC commitment of $1,362,266 represents a Red Cross commitment of 42% toward successful program implementation. The objective of the support from this proposed grant is to develop the capacity of the American Red Cross and the International Red Cross Movement to protect and promote the nutritional well-being of people affected by emergencies. There are two intermediate results that will be the focus of the efforts undertaken through this proposed effort: 1. Improved American Red Cross institutional and technical capacity to design and manage food aid and nutrition interventions as an effective response to emergencies. 2. Enhanced capacity of the Red Cross Movement partners to design and manage food aid and nutrition interventions as an effective response to emergencies. ARC ICB Mid-Term Evaluation. October 18, 2006. Annex VI. 81 The American Red Cross (ARC) has been making steady progress in realizing its vision of building its competence and experience to reduce food insecurity among the world’s vulnerable people. Using commodities and funds provided by USAID/Office of Food for Peace, Office of Foreign Disaster Assistance, USDA/Food for Progress, private sources and donations, ARC has provided food assistance to the most vulnerable all over the globe. Since 1999, ARC has helped more than 4.3 million people in 19 countries through food interventions. With this ICBA grant, ARC wants to improve our technical capacity to incorporate food into developmental relief planning as well as our longer-term development planning. As part of the Red Cross Movement, ARC operates both as an independent entity nationally and as a part of the International Federation of Red Cross and Red Crescent Societies, 178 other operating national societies, as well as the International Committee of the Red Cross. Together, these Movement partners cover the globe with unparalleled access to the most vulnerable. This proposal presents a plan to use this network to improve ARC’s technical capacity to program Title II food across three key initiatives: 1. Expanding ARC national headquarters capacity to program food in developmental relief situations, 2. Expanding ARC Red Cross Movement partners’ capacity to program food in developmental relief situations, and 3. Creating domestic advocacy for foreign assistance writ large and for food programming in particular. Each initiative provides ARC with intriguing possibilities for technical innovations and partnerships. These include: 1. Using food to leverage additional non-American resources and as a means of expanding the management capacity of Movement partners; 2. Using personal digital assistant tools for data collection, building on some recent successes during Measles programming; 3. Field testing Humanitarian Logistics Software (developed by the Federation); 4. Developing and adopting international standards of nutritional monitoring into developmental relief and sudden-onset emergency settings through existing peer networks within the Movement and the international PVO communities; and 5. Integrating food security, water security and health management into the development relief programming design. These efforts build upon current strengths in non-food inventory distribution, in existing project design and management, and on the access to both domestic and international networks. The first two of these initiatives were developed, in part, through the previous ISA grant, and how ARC plans to expand from that base forms the main thrust of this proposed ICBA effort. The ARC International Services Department through the Technical Assistance, Planning and Evaluation Unit and the International Disaster Response Unit will be able to improve the capacity of the partner organizations in the field to plan and implement emergency ARC ICB Mid-Term Evaluation. October 18, 2006. Annex VI. 82 responses through the training and improved tools described in this proposal, including when and how to best use food aid. This significantly supplements the ongoing work of the ARC Food Programming to integrate all ARC efforts to maximize the impact of our response. Adding food appropriately and effectively to non-food response resources will undoubtedly increase the number of people assisted and the nutritional effect on those assisted. For example, ARC can use its position within the international PVO community to provide a forum for the discussion of tools on vulnerability assessment and risk management. The ICBA will also allow us to more effectively and laterally coordinate the diverse operational units associated with international disaster response, ARC chapter international support, technical assistance, and international field support. This coordination is currently informal and ad-hoc, but has enormous potential to affect the domestic programs of ARC, as well as among the international programs of the Federation through its Better Programming Initiative. Both these coordination mechanisms are described in more detail in the body of the proposal itself. Finally, ARC specifically plans for enhanced partnerships with the 961 domestic ARC chapters, many of which are in areas that produce the Title II commodities. Providing the chapters with the faces of the most vulnerable and a frame of reference for international food programming is a unique facet of our organization. This information creates a powerful production to consumption causal linkage. Many of our volunteers are active, influential community members who have an impact on policy makers and legislators to advocate for overseas food programs to help the most vulnerable reduce their food insecurity.” (quotation from executive summary of grant application, 2003) B. Significant Modifications 1. Involvement with WFP (funded through Tsunami operations) “WFP is fairly clear in its primary interest in the Movement, they hope for competent local partners able to hit the ground running in a reasonably efficient manner and with the local knowledge that makes interventions more effective. Additionally they, like USAID, are interested in partnering with the Red Cross (and others) to amass an evidence base around the effectiveness of food in reducing food insecurity in the immediate post￾disaster phase and through targeted supplements. Given that USAID FFP provides a large part of WFP’s food, use of the ICB grant to improve the Movement’s ability to use Title II food provided through WFP, and report on that activity to enable USAID to leverage that effort, merits exploration. A strategy that focuses on building Movement capacity to effectively partner with WFP, (and where feasible, directly with FFP), is efficient in that there are more opportunities to apply knowledge acquired through ICB grant-funded training, than there are if capacity￾building is tied to occasions of successful MYAP and SYAP grant procurement. This strategy would give FFP increased reach, a way to leverage its contribution to WFP and to the ARC to reach many more beneficiaries, more effectively. ARC proposes leverage ARC ICB Mid-Term Evaluation. October 18, 2006. Annex VI. 83 of a limited but important ICB contribution (training in beneficiary targeting, distribution with dignity, monitoring and evaluation, and ops research) to enable a global network to use food resources (WFP food includes but not limited to Title II) more effectively and thus help position USAID as a global leader in reducing food insecurity. The above does not necessarily negate ARC’s original strategy to build Movement capacity to better use Title II food in pilot countries, rather we suggest that that strategy is best rolled out in designated FFP strategic management countries which have not yet been identified. In these countries a more direct relationship may be possible, and ARC will meet with relevant Movement partners when the countries are identified to explore potential interventions.” (email communication with USAID/FFP February 2006) 2. Collaboration between International Disaster Response Unit (IDRU) and Technical Assistance Unit (TAU) IDRU and TAU are working together to lay out strategies and define how ARC would use the ICB to build ONS’s capacity especially in Africa, in coordination with the Federation. The capacity strengthening is at two levels, both ARC and the ONS’s. a. Strengthen ARC capacity to understand the basic elements of food (in) security, especially as relates to what we need to be aware of in undertaking emergency needs assessments. How a food needs assessment might vary from relief items needs assessment and implications for Movement response. b. Strengthen ONS capacity in targeting, distribution, reporting in general management of relief supplies including food commodities. c. Strengthen ONS capacity to access, use and report on food use in longer-term programs, especially HIV/AIDS related. The idea is to work with the Movement to better define how we approach drought or disaster risk reduction programs. Mozambique and Ethiopia have been selected as pilot countries for an innovative approach with foods in both relief and longer-term interventions as per the ICB proposal. The two countries are on USAID list of food insecure countries. Lesotho was added as a third country for the pilot endeavor due to its chronic state of food insecurity and its high rate of HIV/AIDS prevalence. It’s a silent chronic emergency situation. The different types of capacity building trainings and technical assistance will be carried out in the context of the existing collaboration of the ONS’s and WFP or other Title II sponsored PVOs. 3. Additional changes include, but are not limited to: • Shift away from Title II MYAP development, paralleling decrease in available funding for those grants from FFP; • Expansion of support to Humanitarian Logistics Software IHLS); and ARC ICB Mid-Term Evaluation. October 18, 2006. Annex VI. 84 • Ongoing partnership with the Academy for Educational Development (AED) to conduct operations research on the use & effectiveness of food in programming. Details on these changes will be available from both written and interview sources, and the consultant would be expected to describe the changes and downstream effects from those changes on grant performance objectives. Objectives A. Determine the accomplishments to date of the ICBA grant based on the proposal plan and modifications to that plan. B. From document review and interviews with key staff and stakeholder, determine the utility and viability of proposed outputs under the current strategic plan for ISD. C. Assess the viability of completing proposed activities in the remaining timeframe. D. Provide recommendations for enhancing grantee performance, including modifications to proposed workplans. Methods The team will use a variety of methods to determine status. These will include: • Document review (project deliverables and project documents, ISD strategic planning documents) • Key informant interviews (long interview) • Email or phone interviews with stakeholders and overseas participants • (Potentially) focus group discussions with ISD staff on utility and possibilities of grant. Activities There are four key activities in this evaluation. The first is a thorough document review, anticipated to occur in Washington that will inform the development of standard interview protocols for the key informants and email interviews. The second is a series of key interviews (with grant personnel past and present, current management staff, current technical assistance staff). Many of these will occur in Washington and consultant will have assistance in creating a productive interview schedule with the key stakeholders. The third is a series of email interviews with more distant stakeholders (including CRS and the Federation). The fourth builds on the previous three data￾collection efforts to undertake an analysis of the utility, viability, and possibilities of the grant within the current operating environment of ISD. Schedule The evaluation will take place in the fourth quarter FY06 for a period of not more than three weeks. A preliminary schedule for 15 days is as follows. ARC ICB Mid-Term Evaluation. October 18, 2006. Annex VI. 85 Initial briefing & detailed outline 1 day Document review 2 days Key informant interviews (in-person and via email as needed) 4 days Draft preliminary report 5 days NHQ presentation & comments 1 day Final report writing 2 days Team Members The mid-term review will be conducted by an external consultant with experience in American Red Cross operations and the dynamics of the ICBA grant mechanism. Additional input and participation from NHQ staff (including TA staff) is anticipated, but specific staff assignments will vary depending on availability. Key Inputs from American Red Cross NHQ 1. All available project documents, preferably both hard and electronic copies 2. List of key people to interview and preliminary schedule of key interviews prior to evaluation start 3. Email, phone, computer access to conduct interviews Deliverables 1. Draft evaluation findings presented to ARC NHQ staff in brown bag/debriefing. 2. Mid-term review report within four weeks after return that addresses key issues in the SOW above. Appendices The Evaluation report will include the following key elements. • Executive summary of key findings • Methods section • Analysis of program • Overall statistics on project activities • Training and other materials developed • Conclusions and Recommendations • List of persons interviewed • Evaluation schedule • Evaluation SOW Sign-offs Luke Greeves, senior director, IRD