Evaluation Report Final Evaluation of the FY97-01 DAP Findings and Recommendations Catholic Relief Services/Ethiopia supported by a USAID Title II Program April, 2001 Prepared for CRS/Ethiopia by Principal Evaluators: Guy Sharrock Kari Egge Debra Brosnihan Tom Remington assisted by Amsalu G Selassie (CRS/ET) Messele Endalew (CRS/ET) and accompanied by Kebede Folle (USAID/ET) 24 May, 2001 DAP Final Evaluation Report ii Table of Contents Preface i Glossary iv Executive Summary 1 Main Report 9 Introduction and Background 9 Methodology 10 Findings and Results 11 Task 1: progress made in implementing the Mid￾Term Evaluation recommendations 11 Task 2: progress toward achieving the goal, objectives, and intermediate results of the DAP 25 Task 3: recommendations on specific strategies CRS/ET should incorporate into the next DAP proposal so that overall program quality and impact will improve 29 Overarching Themes for the Next DAP 76 Summary of Recommendations 81 References 88 Annexes A: Terms of Reference for the Final Evaluation B: Schedule 24 May, 2001 DAP Final Evaluation Report iii Preface It is not always easy to capture the essence of a project passionately supported by good people in a very short period of time, and it invariably requires considerable assistance. The Evaluation Team leader wishes to express his gratitude to our two CRS/ET team members listed below and to the staff of Catholic Relief Services, especially the Country Representative, Ms. Anne Bousquet, and Marc D’Silva, the Assistant Country Representative. Appreciation for their invaluable assistance also is due to the staff of the three implementing partners that the Team visited: Adigrat Diocesan Catholic Secretariat (ADCS), Hararghe Catholic Secretariat (HCS), and Meki Catholic Secretariat (MCS). Finally, we appreciated the interest in the evaluation shown by the USAID/ET Mission and the useful verbal contributions of its staff member, Kebede Folle, during our trips out from Addis Ababa. This has been a valuable opportunity for bringing together CRS/ET with its partners, both USAID/ET and the implementing counterparts. Guy Sharrock works at Imperial College at Wye, University of London. His contact address is g.sharrock@ic.ac.uk. Kari Egge is a Public Nutrition Technical Advisor with the CRS Emergency Response Team in Nairobi. She can be reached on kegge@crs-ert.or.ke. Debra Brosnihan is a Community Health Manager with CRS/Kenya. Her email address is debra@crsnairobi.org. Tom Remington is CRS’s East Africa Deputy Director for Programming and Agriculture based in Nairobi. He can be contacted at tremington@crsearo.org. Amsalu G Selassie is the Program Department Head at CRS/ET. His email address is crs@telecom.net.et. Messele Endalew works in the Program Department of CRS/ET and can be reached at the same email address. The evaluation was conducted in April 2001. Tom Remington joined the team for the first week, while Kari Egge and Debra Brosnihan arrived for the second two weeks of the evaluation. Guy Sharrock and Amsalu G Selassie were present for the whole three weeks of the evaluation. The Team was convened for a three￾week period in Addis Ababa, including draft report preparation, an initial presentation to CRS/ET staff and implementing partners, followed by final report writing and a final debriefing to CRS/ET and the USAID/ET Mission. The report was submitted at the end of the three-week period, prior to the departure of the evaluators. The Report is presented in two sections: Part I: Main Report Part II: Annexes 24 May, 2001 DAP Final Evaluation Report iv Glossary ADCS Adigrat Catholic Secretariat CBHC Community Based Health Care CHW Community Health Worker CPT Counterpart CRDA Christian Relief and Development Association CRS/ET Catholic Relief Services (in Ethiopia) CSB Corn Soya Blend CSTS Child Survival Technical Support Group DAP Development Activity Proposal DPPC Disaster Prevention and Preparedness EARO Ethiopian Agricultural Research Organization ECS Ethiopian Catholic Secretariat EMC Ethiopian Monetization Consortium EPI Expanded Program for Immunization FACS Food Assisted Child Survival FANTA Food and Nutrition Technical Assistance Project FAO Food and Agriculture Organization FEWS Famine Early Warning System FFW Food for Work FIVIMS Food Insecurity & Vulnerability Information & Mapping Systems, FAO GOE Government of Ethiopia GR General Relief HCS Hararghe Catholic Secretariat IARC International Agricultural Research Centre ICAVG International Vitamin A Consultative Group ICCIDD International Council for Control of Iodine Deficiency Disorders ICRISAT International Centre for Research in the Semi-Arid Tropics IIED International Institute for Environment and Development ILRI International Livestock Research Institute IWM Integrated Watershed Management IWMI International Water Management Institute JEOP Joint Emergency Operation Plan JRP Joint Relief Partnership KPC Knowledge Practice and Coverage LQAS Lot Quality Assurance Survey MCH Maternal and Child Health MFI Microfinance Initiative MOC Missionaries of Charity MOH Ministry of Health MTE Mid-Term Evaluation NACID Nazareth Children’s Center Integrated Development NRM Natural Resources Management 24 May, 2001 DAP Final Evaluation Report v OCF Other Child Feeding PA Peasant Association PAA Previously Approved Activity PAMM Program Against Micronutrient Malnutrition PLWHA People Living With HIV/AIDS REST Relief Society of Tigray SCF Save the Children Fund SPO Special Objective framework SPP Strategic Program Plan, 2001-2005 SSI Small Scale Irrigation STEP Skills Training and Education TBA Traditional Birth Attendant ToR Terms of Reference TT Tetanus Toxoid USAID/ET United States Agency for International Development (in Ethiopia) USLE Universal Soil Loss Equation VDC Village Development Committee VEWS Vulnerability Early Warning System VHC Village Health Committee WFP World Food Program WSC Women’s Savings and Credit WVI World Vision International 24 May, 2001 DAP Final Evaluation Report 1 Final Evaluation of the FY97-01 DAP Executive Summary CRS/ET is one of eight Cooperating Sponsors implementing Title II development programs in Ethiopia. CRS/ET has been supporting emergency relief and development projects in Ethiopia since 1958, although the DAP under review commenced in FY97 and was initially scheduled for completion in FY01, a one￾year extension has been granted. A Mid Term Evaluation was undertaken in FY99. The goal of the current DAP is to enhance food security of vulnerable populations, and there were initially five main projects, the three main being Agriculture and Natural Resource Management, Food-Assisted Child Survival/Community-Based Health Care and the Safety Net programs. Staff at CRS/ET and partner organizations have demonstrated dedication to their respective missions and worked diligently to deliver program resources to the targeted population. The Final Evaluation consisted of three tasks, as follows: · Task 1: concisely assess CRS/ET progress made in implementing the key recommendations of the various mid-term evaluations; · Task 2: concisely assess CRS/ET progress toward achieving the goal, objectives, and intermediate results of the DAP; · Task 3: use results and analysis from the first two tasks to provide guidance and recommendations on specific strategies CRS/ET should incorporate into the next DAP proposal so that overall program quality and impact will improve. The Evaluation Team was asked to place most emphasis on Task 3. Progress has been made on following-up the recommendations of the MTE, although the Evaluation Team has some important concerns regarding the implementation of the FACS/CBHC-related recommendations. In particular, the Evaluation Team has reservations about the quality of the data currently being collected,1 and the lack of clarity shown in some areas by government in training and remunerating Community Health Agents once the DAP is completed. At this point in time, project sustainability in this regard appears questionable. Indicators show improvements over baseline. Although significant progress has been made on some fronts, achieving project success across the range of indicators has been hampered by changes occurring outside of CRS/ET 1 The Evaluation Team recognize that due to GOE changes in food distribution, children cannot participate continuously thus making it difficult to produce the numbers originally envisaged since the underlying assumptions for data collection have changed (see section 3.2.2 ‘ Data Management). 24 May, 2001 DAP Final Evaluation Report 2 management control that have made it difficult to implement key aspects of the original program. Two factors in particular should be noted. Firstly, the change in regulations regarding the administration of credit that has prevented CRS/ET from offering seasonal credit to farmers for purchasing productivity-enhancing inputs, and from continuing with its planned women’s savings and credit scheme. Secondly, “the current monetization endeavor for Ethiopia is a very difficult one” (CBI, 2000). Expected income from monetization has been lower than expected in each of the years of DAP implementation. Uncertainties surrounding the monetization of food commodities have thus worked against project success. CRS/ET have responded positively by developing Business Plans for a CRS/ECS Microfinance Initiative and for a restructured Monetization Business Unit. CRS/ET management should be praised for addressing the problems that have arisen in order to proceed with DAP implementation. The Evaluation Team has found it difficult to assess the impact of the current DAP. At the time of the evaluation the final survey for the agricultural / natural resources management component had not yet been undertaken, and the KPC survey for the FACS/CBHC programs had yet to be completed. Agriculture / Natural Resources Management Programming The Evaluation Team endorses the Integrated Watershed Management strategy of CRS/ET and believes that this strategic planning approach will contribute to the overall goal of improving food security among targeted households. There is potential to improve agricultural productivity in the areas targeted by CRS/ET and its implementing partners, although any achievements will require high quality programming and, importantly, support. A key element of CRS/ET’s strategy is to adopt a more participatory approach to program planning and implementation. In the long-run this is expected to enhance project sustainability, although initially it will place greater demands upon the skills and resources of CRS/ET and its partners. Such an approach also has programmatic implications in terms of the amount of development activity that can be reasonably achieved in a given time period. Giving more weight to the preferences of community members will, not surprisingly, lead to more site￾specific programming and a move away from the ‘blanket’ approach to planning of the current DAP. If agricultural technologies are to have an impact it is important that they are targeted appropriately to different members within a community. Different technology ‘packages’ have differing resource requirements, and uptake is likely to vary between households. Partnership with ‘external’ partners, such as the EARO and ICRISAT in identifying and providing possible ‘off-the-shelf’ agricultural technologies suitable for resource-poor households will prove very beneficial in this regard, and support and guidance from the CRS East Africa Regional Office should be sought to further such institutional links. Additionally, for improving staff understandings of technology uptake participatory well-being ranking, which has been used previously by CRS/ET staff, should be piloted to see if useful insights into understanding farmers’ preferred technology choices are made possible. 24 May, 2001 DAP Final Evaluation Report 3 Integration is a key theme of the documents reviewed by the Evaluation Team. Developments in the SSI sector may provide a good opportunity to operationalize this concept. Road building activities, thorough market analysis, identification of suitable crop technologies, opportunities for accessing microfinance all need to be considered during the appraisal stage, prior to installing new SSI infrastructure. CRS/ET and counterpart staff are likely to need training support, and possibly technical assistance, to ensure that proper analysis can indeed be undertaken as proposed in the PEA Report (EPI, 2000). Given the likely market-orientation of SSI-related development, appropriate indicators will need to focus more on returns to producers, rather than on returns to land. The Lege Oda Mirga watershed appears to have achieved some successes during the current DAP. HCS staff should be commended for their commitment to ‘making it work’. It may prove to be a useful model for subsequent DAP interventions. Given its current prominence, it is recommended that the findings reported in the Case Study Report are verified by an ‘external’ agency to ensure that any conclusions drawn on the Lege Oda Mirga experience are based on rigorous analysis. Sustainability of social institutions (e.g. village-based committees) established during the current DAP is presently uncertain. CRS/ET and implementing partners appear to have mixed views regarding the likelihood that recently established committees will survive project phase-out. Monitoring will be required to enable CRS/ET and counterpart staff to learn from lessons gleaned from this more intangible aspect of DAP implementation. Health and Nutrition Programming Considerable progress has been made by CRS/ET in addressing recommendations raised during the MTE. A number of recommendations focused on the need to improve the health education component and IEC materials used. CRS has conducted an IEC assessment. Based on the findings from the assessment CRS and partners will undergo training in both appropriate development of IEC materials and health messages, which is endorsed by the Evaluation Team. A better understanding of mothers’ beliefs and attitudes towards health care practices will be critical to the development of effective health messages. Attitudes and beliefs vary from region to region, and therefore CRS will need to work closely with partners and communities to develop appropriate health messages. Alternative strategies to health education, such as nutritional counseling, mother support groups, involvement of men, should also be examined. Data management at all levels (community, counterpart and CRS) needs to be strengthened in order to improve the quality of health data. Registers need to be updated to include all pregnant women and children under two, irrespective of their participation in the food program. This will allow registers to collect population-based data, which can be then compared to baseline KPC surveys. CRS and counterparts should use supervisory checklists on all monitoring visits, with particular attention paid to the completeness of registers and progress 24 May, 2001 DAP Final Evaluation Report 4 reports. The use of supervisory checklists should improve the standardization and quality of monitoring visits. Equally important as the accuracy of the data, is understanding why it is being collected and using the information to make programming decisions. In the current FACS/CBHC program all children under two and pregnant and lactating women are targeted for health activities. This is the best targeting strategy for population based impacts and the team recommends that it should continue. Geographically, however, counterparts should try to saturate PAs with integrated programming rather than work in just one or two villages which is currently occurring in some areas. To achieve full integration with other development activities, target areas should also overlap with watersheds used for the agriculture project. One visible aspect of the FACS/CBHC program is the use of food. Despite its prevalence, there is some ambiguity in what the actual role of food is in the program. There are three types of food rations used in MCHN programs as articulated by FANTA: · Recuperative: to rehabilitate malnourished children, determined by the household caloric deficit, appropriate for areas where food availability is a problem · Preventive: to prevent children in vulnerable households from becoming malnourished, determined by the household caloric deficit, appropriate for areas where food availability is a problem. · Incentive: to provide motivation for participation and compensation for time spent attending activities, based on the opportunity cost of attending activities The FACS/CBHC program seems to use rations both as incentive, preventive and recuperative, but the composition of the ration does not correspond to any of the FANTA ration definitions. The rations currently used in the CBHC/FACS program provide slightly less than 1200 kcal per day. While this is sufficient for a young child, the reality that rations are shared amongst all household members is not acknowledged to serve as a recuperative or preventive ration that would require a near doubling of calories. Likewise, the ration is too large to meet the guidelines of an incentive ration. It is recommended that analysis be done to look at what are the goals of the program and what is the purpose of food after which a new appropriate sized ration can be developed. Included in the analysis of what is the role of food in the FACS/CBHC program should be an assessment of how food can actually address the causes of malnutrition in target areas. If it is found that food availability plays a small role in the causes of malnutrition, consideration should be given to modifying the food targeting in the FACS/CBHC program. Possible food targeting options include: · blanket feeding of all under twos and pregnant/lactating women for incentive purposes 24 May, 2001 DAP Final Evaluation Report 5 · blanket feeding of all under twos and pregnant/lactating women for recuperative/preventive purposes · targeted feeding of only malnourished children plus pregnant/lactating women · targeted feeding of only malnourished children · no feeding Government regulations and counterpart preferences for working with food in each region should also be considered, resulting in site specific CBHC/FACS packages. In addition to potential changes in ration size, the quality of rations and commodity included should be assessed. One issue to investigate is the appropriateness of wheat, a commodity that is not easily digested by young children and that is relatively low in protein. Another commodity such as lentils, which is more palatable for young children and culturally appropriate, could be substituted. Whatever commodities are included in the ration, though, they should be fortified with micronutrients, particularly vitamin A, iodine and iron. If fortification is not sufficient, CRS should work with the MOH and other local partners to ensure that sufficient micronutrient supplementation is provided to pregnant women and children. While many recommendations relate to future programming, there is still the consideration of what should happen in the last year of the current DAP. Counterparts are at different stages in implementation of the FACS program, which will have implications for the exit strategy and future programming. In some areas activities have been hampered by the uncertainty of CHA training. CRS/ET has adjusted by training CHWs on a shorter curriculum specific to project activities. The question is how to sustain CHWs once the program phases out, as well as how to sustain CHAs if the government is unable to pay their salaries. CRS has already begun thinking about alternatives with partners, and is encouraged to document lessons learned. Water and Sanitation Programming A key factor in achieving impact in health education depends on access to water and sanitation facilities. CRS has developed a potable water and sanitation strategy, which is endorsed by the Evaluation Team. CRS/ET is encouraged to consider expanding the strategy to include multiple uses for water beyond potable water and to carefully monitor water committees for sustainability. Based on lessons learned CRS/ET has shifted from the expensive VIP latrine construction, to a cheaper design produced with locally available materials, which will be built on a self-help basis. Sanitation and hygiene education will be integrated into health education. It is recommended that the same problem solving approaches and understanding of cultural practices be used to develop sanitation and hygiene messages, as done for health education. 24 May, 2001 DAP Final Evaluation Report 6 Use of Food Aid CRS/ET uses food in four ways: · Monetization · Food for Work (Agricultural activities) · FACS/CBHC (as an incentive and to some extent as a supplement/rehabilitation tool) · Safety Net Of these, monetization accounts for a third of allocated food, safety net just under half and FFW/FACS each using ten percent. The amount of food directly distributed throughout the DAP was highest in FY97, lowest in FY00 and basically the same for the other years, including requests for FY02. The continued levels of direct distribution in FY02 is of some concern as several areas should be phasing down in activities. The Team recommends that the use of large quantities in the last year of the DAP be assessed. Similarly, recommendations were made for analysis of the role of FFW in contribution to nutritional impact and in phase out strategies. Monitoring and Evaluation Much information is currently generated by CRS/ET and implementing partner staff. The Evaluation Team has concerns over the accuracy of some of what is collected, and the varying levels of use and perceived usefulness of the information generated. To enhance the usefulness, and thereby the use, of information it is proposed that CRS/ET organize in-house training workshops to promote the concept of ‘utilization-focused’ monitoring and evaluation. Linked to this, it is suggested that a pilot ‘information audit’ is undertaken to track information collection and use, initially on a pilot basis for one implementing partner. Once useful data have been identified, there is the related issue of collection. Many staff reported that they have little spare time for good quality data collection. This may be something that can be outsourced to local higher education institutions, although checks for quality control and cost will need to be established from the outset. One way of encouraging use of secondary sources of information is to institute a system that obliges staff to indicate what background material they have sourced prior to submitting a proposal. This might be print-based literature, if available, or simply a brief record of the outcomes of discussions held with ‘external’ agencies to discuss a particular topic. Similarly, the M&E activities of CRS/ET could benefit from greater networking with others doing similar work in Ethiopia. Such a network for M&E might already exist in Ethiopia (e.g. along the lines of the Kenya Evaluation Association) but if not, then CRS/ET could play a lead role in developing such a network. 24 May, 2001 DAP Final Evaluation Report 7 Expanding Partnerships with ‘External’ Partners The Evaluation Team views this as an important aspect of the next DAP. There are opportunities for different models of collaboration in most areas of CRS/ET activity. Nonetheless, while partnership is important there are also associated ‘transactions costs’, and CRS/ET needs to proceed judiciously. It is recommended that CRS/ET hold an in-house workshop, probably involving staff from the CRS Regional Office, to determine how as an institution it is possible to move forward in developing further beneficial collaborative relationships. Safety Net As mentioned under the discussion of food aid, the Safety Net program utilizes nearly half of CRS allocated Title II commodities. Though the program undoubtedly meets a great need in Ethiopian society, efforts to enhance the quality of the services provided should be made. Possible areas where CRS may want to work on with Safety Net partners include financial accounting systems, managerial systems, sanitation and hygiene, development of strategic visions and long-term planning. As with FACS, assessment of the appropriateness of the ration should be made to determine if a different amount or composition of commodities may increase impact. HIV/AIDS While there is still not much data on the prevalence of HIV in rural Ethiopia, all development and emergency programs need to address the issue in an integrated manner, as suggested in CRS/ET’s SPP. Agriculture and natural resources are critical components of the livelihoods of those affected by HIV/AIDS. Likewise the nutritional status of those infected and affected influences not only their welfare, but also their ability to further develop livelihoods in a manner that will help mitigate the impact of AIDS and prevent the spread of HIV. The nutrition component of FACS and the FFW component of AG/NRM can have both mitigative and preventive impact on targeted populations. CRS/ET should also consider developing an AIDS policy for the CRS/ET office in order to address such issues as awareness, stigmatization, risk management etc. Emergency Mitigation CRS not only has maintained emergency response capacity, but also is in the process of using that capacity in the large JEOP emergency response. The skills of the logistics and administrative staff of CRS are greatly lauded by the Evaluation Team. While a development program, many of the activities in the DAP have emergency mitigation and preparedness impacts. All of the natural resource management activities that reduce erosion and increase water retention help mitigate the effects of drought. Improved practices of the FACS program and accompanying ration help protect the most vulnerable sector of society – young children. By collaborating as much as possible with agencies producing 24 May, 2001 DAP Final Evaluation Report 8 early warning systems and also working in emergencies, CRS can stay prepared and be able to respond quickly to any disasters. Conclusions In summary, the evaluation points to the successes achieved under the current DAP in a sometimes difficult external environment. There is, however, potential for improvement in programming and support. Emerging from discussions during the course of the evaluation, the Evaluation Team identified a number of common themes that appeared to run through many of the topic-based recommendations; these can be summarized as: 1. The value of integration for realizing synergies from DAP activity 2. The value of site-specific programming 3. The need for meaningful community participation 4. The importance of ‘getting the basics right’ 5. Doing less, but doing it better These are not new issues for CRS/ET and counterpart staff. Indeed, in some instances there is evidence that some of this is happening already. The Evaluation Team would strongly endorse any action taken by CRS/ET and its partners to promote further movement in the broad direction of these five themes. 24 May, 2001 DAP Final Evaluation Report 9 Main Report Introduction/Background Ethiopia is a poor country, predominantly rural, and prone to pervasive food and nutrition insecurity. Around 50 per cent of the population live below the poverty line, while average per capita income is only $110 per annum. Most Ethiopians do not have access to basic health and education services contributing to a low life expectancy, high infant mortality and morbidity, widespread malnutrition and alarming rates of stunting among children (CRS/ET, 1999a). CRS/ET is one of eight Cooperating Sponsors implementing Title II development programs in Ethiopia. CRS/ET has a long history of working in emergency relief and development projects in Ethiopia dating back to 1958. The current Development Activity Proposal (DAP), which runs from 1997-2001 (plus a one year extension), aims to improve the food security of targeted communities. The elements of the program are: Goal: Enhanced Food Security of Vulnerable Populations There were originally three sub-goals, each with two strategic objectives as follows: Sub-Goal 1: Sustainable Increase in Food Availability Strategic Objective A: Sustainable increase in agricultural production Strategic Objective B: Enhanced sustainability of natural resources Sub-Goal 2: Improved Utilization of Food by Vulnerable Groups Strategic Objective A: Improved health status of young children and mothers Strategic Objective B: Develop sustainable community structures for promotion of the health of women and children Sub-Goal 3: Improved Access to Food for Vulnerable Groups Strategic Objective A: Improved incomes for vulnerable groups Strategic Objective B: Provision of a safety net for vulnerable groups After submission of the DAP. CRS/ET worked with the USAID/ET Mission and other Cooperating Sponsors to develop a new special objective framework (SPO) which had five major Intermediate Results: Intermediate Result 1: Increased Agricultural Production Intermediate Result 2: Increased Household Income Intermediate Result 3: Improved Health Status in target areas Intermediate Result 4: Natural Resource Base Maintained Intermediate Result 5: Emergency Response Capacity Maintained 24 May, 2001 DAP Final Evaluation Report 10 Methodology The evaluation followed a prescribed pattern established in the Terms of Reference. CRS/ET made it clear from the outset that: “we would like to use the final evaluation more as a planning tool to take the successes and lessons learned from this DAP and use it to refine programming strategies for the next DAP, which is proposed to commence in FY03.” The instructions go on to list three specific tasks of the final evaluation, as follows: Task 1: concisely assess CRS/ET progress made in implementing the key recommendations of the various mid-term evaluations; Task 2: concisely assess CRS/ET progress toward achieving the goal, objectives, and intermediate results of the DAP; Task 3: use results and analysis from the first two tasks to provide guidance and recommendations on specific strategies CRS/ET should incorporate into the next DAP proposal so that overall program quality and impact will improve. The ToR indicates a methodology for the Final Evaluation that is utilization￾focused and a learning process for CRS/ET staff and others involved. More specifically: “… CRS/ET wants to use this final evaluation as a means of working with its partners, both USAID and implementing counterparts … since a number of evaluations have already been conducted, the Evaluation Team will not be expected to collect much data. Rather, the emphasis should be on an extensive review and analysis of existing reports from a broader perspective.” The ToR indicate examples of documents to review, the need for visits to at least two implementing partners, and comprehensive discussions with primary stakeholders. The Team was formed by CRS/ET; individual consultants were contracted and met in Addis Ababa. Three weeks was allotted for the Final Evaluation, although team members were not all present for the whole three weeks. At an initial Evaluation Team meeting it was agreed that the team would include staff members from CRS/ET and the USAID/ET Mission, and involve regular debriefings to ensure a utilization focus to the evaluation. Team assignments were made, an initial report outline distributed (based on the ToR), and a preliminary identification of topics for deeper examination was made. A work plan was formulated to cover the three-week period, including three site visits (to ADCS, HCS, MCS), staff sector-based and plenary debriefing sessions, presenting findings, report preparation and finalization. 24 May, 2001 DAP Final Evaluation Report 11 Findings/Results Task 1: concisely assess CRS/ET progress made in implementing the key recommendations of the various mid-term evaluations Introduction The Mid-Term Evaluation recommendations are listed below together with a brief commentary indicating progress to date. This section ends with an overall evaluative comment on the progress achieved by CRS/ET. Findings MTE Recommendation # 1: CRS should develop a mechanism for monitoring the proper utilization of Title II FFW resources by the implementing counterparts to insure they are using FFW commodities in accordance with CRS’s norms and requirements. CRS/ET Progress: this recommendation arose from MTE concerns regarding FFW-supported activities undertaken by Nazareth Children’s Center Integrated Development (NACID). Since the MTE, CRS/ET no longer works with NACID, thereby lessening the concerns apparent at the time of the MTE. Furthermore, CRS/ET has adequate monitoring mechanisms in place involving systematic systems of reporting, frequent field visits, and regular accounts auditing. The Government of Ethiopia is currently revising the norms for FY02. MTE Recommendation # 2: Maintenance of cropland should be given equal emphasis with the construction of new bunds as a way of increasing the sustainability of this activity. CRS/ET Progress: the extent to which the maintenance of cropland bunds is emphasized varies between implementing partners. Although there is no quantitative reporting of maintenance activities, it is clear that in some cases the issue is being satisfactorily addressed (e.g. ADCS and HCS organize regular campaign days). Anecdotal evidence suggests that farmers are conscious of the value of maintaining the physical structures, particularly where they perceive the potential for private cropping benefits. Before commencing cropland bunding on private land, farmers negotiate a maintenance agreement with their VDC. Since maintenance work in communal areas does not qualify for FFW, the difficulties of maintaining communal soil conservation works remain in some of the places visited. MTE Recommendation # 3: To attain the maximum benefit from integrated interventions, farmers whose land has been bunded should be encouraged to use improved seeds and fertilizer. The combined effect of these interventions would produce enormous benefit and would encourage the farmers to continue using the introduced technology. 24 May, 2001 DAP Final Evaluation Report 12 CRS/ET Progress: whilst CRS/ET staff and implementing partners are very aware of the need for such “integrated interventions”, the increased use of seeds of improved varieties and inorganic fertilizer has been hampered by the restrictions imposed on project credit activities arising from Proclamation 40/1996 pertaining to microfinance institutions.2 The partners visited indicated that “improved seeds and fertilizer” are not the only means to improve crop productivity, and “lower-technology” options are available. Encouragingly, some implementing partners are actively promoting alternative output-enhancing interventions (e.g. manuring), although the gains from such interventions cannot match those envisaged in the original DAP. MTE Recommendation # 4: The technical capacity of counterpart staff needs to be upgraded through organizing practical training and experience-sharing visits within the country. Such training should also, whenever possible, include staff from line ministries. Inclusion of the counterparts in joint training program (whether government training programs or NGO training programs) would help strengthen relationships between counterparts and line ministries. CRS/ET Progress: ADCS and HCS each have, or are in the process of developing, staff hiring guidelines as one way to improve staff quality. However, both also indicated that they would benefit from additional training. Attempts have been made to share experiences between implementing partners (e.g. the Agricultural/Natural Resources Management Workshop, occasional exchange visits), and in a few instances government staff have attended training programs organized by implementing partners (e.g. in an HCS-organized agricultural animators training program). Closer training links between implementing partners and other NGOs has been less evident. Training is self-evidently not merely a classroom event, and there are instances where field visits have been organized to include line Ministry staff to raise and discuss issues arising from program activity (e.g. joint visits to the Lege Oda Mirga watershed). The value of collaborative training events is well recognized by the counterpart staff interviewed, although budgetary constraints were sometimes cited as a reason for not having done more in this regard. Inevitably, and ideally, more should be done with regard to identifying and pursuing collaborative training opportunities. MTE Recommendation # 5: The importance of using organic fertilizer in combination with chemical fertilizer is common knowledge today for generating sustained improvement in soil fertility. Given the likely depleted situation of soils, generally, in Ethiopia, the use of organic fertilizer should be encouraged in DAP operational areas, to the greatest possible extent. CRS/ET Progress: as was the case for Recommendation #3, CRS/ET and counterpart staff are well aware of the theoretical benefits to be derived 2 Proclamation 40/1996 on Licensing and Supervision of Micro Financing Institutions. 24 May, 2001 DAP Final Evaluation Report 13 from using organic fertilizer. However, the targets set for this aspect of program activity have not been met: ADCS Annual Reports (FY99 and FY00) indicate that no composting activities were undertaken as this is not an activity favoured by farmers in Eastern Tigray, whilst HCS staff argue that farmers have alternative uses for biomass, and that composting is not a priority. Instead, HCS staff have been actively promoting the use of manure among farmers in the DAP sites, and village groups have been organized to transport the material (usually located in large piles close to farmers’ homes) to farmers’ fields. Anecdotal evidence suggests that the farmers interviewed recognize the benefits to be gained from undertaking this effort. The issue of organic fertilizer was raised at the Agricultural / Natural Resources Management Workshop (1999a) and the following points were noted: · Organize training for the technical staff and farmers on the preparation and use of organic fertilizer (FY01); · Prepare a manual on organic fertilizer use (FY01). At the time of this evaluation these two action points had not been undertaken. MTE Recommendation # 6: Survey results should be analyzed, using both direct and proxy indicators, to determine the percent of households actually employing combinations of the different DAP agricultural sub-components (possibly on the same plots), what specific combinations, and with what results. CRS/ET Progress: the implementing partners all see this recommendation for specific cross-tabulation analysis as a CRS/ET-specific recommendation, and have not undertaken the recommended analysis. Tables presented to the Evaluation Team suggest that this analysis has not yet been undertaken by CRS/ET. It may be that this type of analysis is more appropriate to the final survey of program impact at the end of the DAP period. MTE Recommendation # 7: As a matter of some priority, CRS, using Birr resources, should commission a set of surveys to review in a thoroughgoing, participatory manner the experiences of participants with the credit they received under the program. The objective would be to determine whether there are factors seemingly associated with success and/or the lack of success by some poor women borrowers and savers which could be used by CRS, USAID or any other donor, NGO or unit of the Ethiopian Government in improving the design of future savings and credit programs for poor women. CRS/ET Progress: a report entitled “The Study of Micro-Finance Program” was written in August 2000 by Tesfaye Assefa in association with CRS/ET. MTE Recommendation # 8: CRS should even now be considering how it might participate in the post-2001 period in a women-focused micro-credit program which would bring to bear in an appropriate Ethiopian context their impressive 24 May, 2001 DAP Final Evaluation Report 14 experience in such programs in India, Indonesia, and other parts of the world. Giving further impetus to CRS’ long-term planning to renew involvement in women’s micro-credit in the future is the apparent up-coming reorganization of the cooperative sector under a cooperative commission with the mandate to license cooperatives to be, among other things, savings and credit cooperatives. CRS/ET Progress: a document entitled “CRS/ECS Microfinance Institution. The Business Plan, 2001-2005” has since been prepared and has been reviewed by the Evaluation Team in the course of its work. MTE Recommendation # 9: The correct growth curve should be identified. All incorrect CRS growth cards should be recalled and replaced with new cards with the growth curve and other information recorded correctly on the new card. CRS/ET Progress: Children have new cards. It is not clear whether all the information from the old cards was transferred. The cards seen had information correctly recorded for the new card period, but some evidence suggested that not all old information was not transferred. A more careful extensive study would have to be done to verify whether this recommendation was fully met. On some of the child registers, old information is still in the process of being transferred. MTE Recommendation # 10: Once reprinting of the CRS card has been accepted by the federal MOH, each child should have a single card which has complete immunization, Vitamin A, and growth monitoring information. In order for this to be achieved, directives have to be sent from the Ministry of Health to Regional Health Departments. CRS/ET Progress: In visits, two cards (MoH and CRS) were still being used. It appears this recommendation cannot be complied with, as despite discussions with the regional, zonal, and wereda government offices, MOH still wants their card to be used. CRS worked with the government to make sure that the CRS card complies with all government regulations and meets government approval. The Ministry also collects their cards after all immunizations have been done and CRS needs to have a card available for two years. MTE Recommendation # 11: CRS and its counterparts should seek out alternatives for CHWs. Community members could, perhaps, be trained for shorter periods to follow growth monitoring and provide instruction in basic health messages. CRS/ET Progress: This is still an issue with CRS. In some regions modified trainings have occurred to conduct short-term training for CHWs without certification, but to meet program needs. In other regions there is a stalemate and no CHAs or CHWs have been trained and only the animators are working in the field on the program. It is unclear what will be the role of government trained CHAs and how this interfaces with current animators. The number of CHAs/animators per PA in some counterparts appears to be 24 May, 2001 DAP Final Evaluation Report 15 insufficient to adequately cover all health education, home visit, food distribution and growth monitoring activities plus new programs such as home maintenance and latrines. Similar to this is the level of supervision, monitoring, training and technical support that the health coordinator from HCS can provide for the 23 animators in the program being the sole HCS staff member working in health including emergency duties. MTE Recommendation # 12: Growth monitoring must continue irrespective of the provision of food supplementation. CRS/ET Progress: It appears that only women and children participating in the food supplementation program are included on FACS/CBHC registers. Counterpart staff said that others are invited to come to health education sessions and some do attend, but there were no estimates of numbers nor is it possible to calculate non-food recipients’ participation as registers have not been kept up to date. MTE Recommendation # 13: More training in adult education methodologies is required to improve the health and nutrition education sub-component of the FACS/CBHC component of the DAP program. CRS/ET Progress: An IEC assessment was conducted and plans to improve the IEC materials have been made. Some counterparts have attended training of trainers for health education methodologies. On-site informal trainings have occurred between the IEC coordinator and counterparts. Designs will be regionally specific. MTE Recommendation # 14: Better health education materials are required. It is not cost effective for each NGO to produce its own printed materials, however, and low cost methodologies for imaging or imparting health information (e.g. felt boards) need to be developed in a cooperative manner. CRS/ET Progress: Homemade or locally made materials were not visible at visited health education centers; however, several government or UN commercially printed posters were attached to the walls. The appropriateness of these materials was questionable as many messages were contained in the text and not in the illustration so most of the women who are illiterate could not read it, plus non-local languages were used. Posters were not specific to messages being conveyed in health education sessions; rather they were whatever was available and being distributed for free. Animators expressed a need for having more visual aids, brochures and materials. These findings were confirmed in the IEC assessment and a training on IEC material development has already been planned by CRS. 24 May, 2001 DAP Final Evaluation Report 16 MTE Recommendation # 15: The effects of different models of health education within the DAP need to be identified, monitored, analyzed and compared in order to inform the design of future health education projects. CRS/ET Progress: The IEC assessment has identified different health education strategies, such as role plays, dramas, and mother to mother learning. A training is planned to design the most appropriate methodologies and IEC materials with partners and communities. MTE Recommendation # 16: Counterpart capability to engage in health education needs additional strengthening. CRS should employ a specific health education expert with international experience and links with other organizations and institutions to develop and train counterparts in adult education methodologies, develop material, and take part in evaluating which types of education are the most successful. CRS/ET Progress: a staff member has been hired by CRS to address IEC issues. This individual took the lead on the IEC assessment to identify gaps in the health education strategy. A training for partners has already been planned. MTE Recommendation # 17: It is unlikely that significant behavior changes and impact on health and nutrition conditions can be expected within a four to five year program. Since there has been delay in implementing most of the DAP FACS/CBHC activities, it will probably not be possible to complete all of the desired activities or to achieve the desired outcomes before the end of the DAP. The emphasis in the period through FY 2001 should be in continuing as many sub-components as possible. The results of careful monitoring and reporting should be used in a follow-up project that would continue to focus on changing traditional mindsets regarding household health and nutrition behavior and practice. CRS/ET Progress: In the next DAP CRS plans to continue with CBHC. In areas where CBHC has been implemented for less than five years, no phase out will occur. In the first years of the new DAP CBHC will be implemented for 2-3 years in old areas and then phase out considered. This is still a relatively new program and monitoring is happening to identify lessons learned and best practices. For example, IEC will be improved and water will be included in upcoming FACS/CBHC programs. MTE Recommendation # 18: The KPC surveys should be shared and fully discussed regarding specific program interventions with each counterpart. In addition, the survey should include a section on water availability and use. This information on water relates to many of the child health issues addressed. CRS/ET Progress: a final KPC has not been completed at this point, but water will be a component. 24 May, 2001 DAP Final Evaluation Report 17 MTE Recommendation # 19: CRS/ET should review the steps to developing effective caretaker behavior change strategies outlined in this recent technical report as a means of strengthening community level behavior change program interventions.3 CRS/ET Progress: see above comments on health education. MTE Recommendation # 20: Supervisory community assessment forms need to be completed and more attention devoted to qualitative reporting in order to determine what is succeeding, what changes are occurring, and what progress toward goals has been achieved. CRS/ET Progress: It appears supervisory and community assessment forms are not regularly being completed. Investigation needs to be done to see why these forms are not being used. Some of this information has been gathered in case studies or other surveys. CRS has discussed information collection and is in the process of identifying strategies to improve data collection with the CRS Monitoring & Evaluation Officer. MTE Recommendation # 21: The total amount of food allocation per village should be determined, but actual allocations should vary throughout the year, depending on household food availability, e.g. pre-harvest allocations should be larger and post-harvest smaller because of better household availability. CRS/ET Progress: this is not relevant to the health program as the ration serves as an incentive and is not tied to household food security status. This would be relevant to FFW programs. MTE Recommendation # 22: CRS should consider whether it wants to continue programs with counterparts when their development priorities appear to be different from those in the DAP. CRS should also work with other donors providing assistance to these counterparts to develop closer collaboration on common elements of institutional strengthening. CRS/ET Progress: CRS is looking at diversification particularly as a result of reduced monetization funding. CRS is making positive steps towards diversifying its donor base in order to reduce its dependence on monetization funding. In FY97 monetized funds accounted for 81% of CRS/ET’s total funding for the year; in FY01 the equivalent figure had fallen to 58%, and the projection for FY03 is 41%. A child survival grant is being pursued to obtain health funding plus DA funds have been requested for agriculture and other areas. MTE Recommendation # 23: Strong efforts should be made to keep animators on site for a minimum of two years (four years would be preferred) to help institutionalize new approaches in health education, growth monitoring, increased 3 Murray, J. et al. (1997) “Emphasis Behaviors in maternal and child health: focusing on caretaker behaviors to develop maternal and child health programs in communities” BASICS Technical Report, 1997. 24 May, 2001 DAP Final Evaluation Report 18 community acceptance of the major precepts of the FACS/CBHC approach, and to help insure its continuance after the food component of the programs has ended CRS/ET Progress: CRS is recommending that animators be hired on a yearly basis at which time their performance is reviewed and need in the community assessed. The feeling is that animators should not be permanent employees. They may stay for several years on a contractual basis, but not on a regular contract. In several counterparts animators have been around for long periods and the threat of working themselves out of a job hampers empowerment of communities. In addition the counterparts are constrained by having to keep employees who may not have the local language or other skills that are needed. Basically, CRS is not agreeing with this recommendation and has not implemented it. MTE Recommendation # 24: The sustainability of centrally-managed nurseries is doubtful. The justification for maintaining a central nursery is high demand for seedlings. However, when the project phases out of an area, there is no one to take over the responsibility of managing the nursery. Phase-out strategies need to be developed in advance. The option of transferring them to be managed by members of the community as a group, or of transferring them to the MOA should also be considered CRS/ET Progress: the implementing partners have different phase-out plans for the centrally-managed nurseries. ADCS currently make use of part of the Ministry of Agriculture’s (MOA) nursery which will revert to the MOA at the end of the project. HCS staff have initiated both private nurseries, and a central nursery where the land has been leased from a farmer; the intention is for the MOA to take over the site at the end of the DAP period. MTE Recommendation # 25: Livestock feed is critically short in most of the operational areas. Activities to improve livestock feed and related management should be initiated CRS/ET Progress: implementing partners have addressed this issue on two fronts. First, by including forage seedlings in the project nurseries; second, by seeking where possible to encourage communities to protect through closure communal areas inter alia for forage production. MTE Recommendation # 26: CRS – within the NGO context – should suggest to both USAID and the government that, at the appropriate time, a workshop or conference be held to establish both the short and long-term vision regarding NGO participation in future emergency-related food shortages in Ethiopia. CRS/ET Progress: at present CRS/ET is working with several other partners to respond to the drought in various parts of Ethiopia. This was sanctioned by the government. 24 May, 2001 DAP Final Evaluation Report 19 MTE Recommendation # 27: If CRS is informed by USAID (and/or by DPPC) that it should remain in a state of preparedness and would likely be called on to participate in emergency operations responding to drought or other fast- or slow￾onset disaster, CRS should develop or update contingency plans for all types of disasters for different regions of the countries in coordination with the JRP. CRS/ET Progress: no contingency plan as such exists in CRS/ET, but staff with emergency capacity are employed in various positions. MTE Recommendation # 28: Drought mitigation activities under the present DAP should be more explicitly and more aggressively pursued. Foremost among them should be efforts to assist households in semi-arid areas of Oromiya and Tigray to conserve more rainwater run-off for human use. Given what the Evaluation Team has heard and observed regarding the extent and severity of the water availability problem in the semi-arid and arid areas where there are DAP activities, it is clear that the lack of water for weeks and even months of every year drives out all other household livelihood security considerations. Pond construction, local irrigation infrastructure and other means of securing water for crops and livestock should also move up the priority lists of FFW-supported activities in many of the DAP areas. CRS/ET Progress: in Oromia, activities in rainwater catchment, spate irrigation and pond construction have been increased along with other water projects. MTE Recommendation # 29: CRS/ET and counterpart staff should investigate the feasibility of developing roof and rock rainwater catchments at the household level as a significant component of the project. Three locally-available commodities are needed: corrugated roofing to replace thatch, rain gutters/downspouts or cement, ground-level catchment channels and a cement or metal water storage container for each household. To the extent that these roof catchments might provide adequate water for maintaining modest sized household gardens, this would greatly help to make some highly nutritional food available to the household in times when the planting of rain-fed crops is delayed by a late start of either the short or long rains. CRS/ET Progress: HCS have constructed roofwater catchment facilities in eight locations throughout its development area. Such facilities have been constructed on public buildings, such as schools, mosques and churches. These institutions will be responsible for the care and maintenance of the installed systems. MTE Recommendation # 30: CRS/ET should seek clarification from USAID and the Government of Ethiopia about the long-term role envisioned for US-based NGOs in future handing of food and other commodities in emergency situations. In the meantime, the Team advises CRS/ET to continue in its efforts to maintain a high state of readiness in line with its responsibilities under IR 5. In this regard the following suggestions are made for CRS’ consideration: 24 May, 2001 DAP Final Evaluation Report 20 · Improved coordination with WFP, Save the Children/UK (SCF/UK) and with the FIVIMS4 officer in FAO regarding the measuring and tracking of levels of vulnerability to disasters in Ethiopia by geographic region is strongly suggested. WFP, working with FEWS, the SERA unit in DPPC, and Save the Children/UK have made great progress in many African countries in developing monitoring and tracking systems related to changes in the levels of vulnerability among poor, food deficit sub￾populations in these countries and has developed regularized reporting systems to provide information to interested users. · For CRS to use its own limited resources to, in any way, duplicate the gathering of information on household vulnerability to drought, or to increasing chronic or transitory food insecurity would seem not warranted, unless deemed necessary by the CRS Country Director. In this context, it may be possible to merge the present CRS VEWS work more formally with the early warning survey and reporting efforts of WFP, FAO, USAID SCF/UK and others, possibly eventually producing a combined quarterly report covering all the food insecure areas of Ethiopia and combining agro-climatology and vulnerability assessment reporting in a single quarterly bulletin which could even eventually be made available electronically over the Internet. CRS/ET Progress: as of October 2000, the VEWS program was discontinued by CRS. CRS now uses FEWS, FIVIMS and other data for project areas. MTE Recommendation # 31: CRS/ET should reconfigure its present E&M apparatus to enable it to design – and with outside professional help (e.g., from, say, the university social science faculty) – carry out participatory, community￾based surveys intended to identify the likely activities, processes, decisions, relationships and events which have influenced DAP achievements in places like Lege Oda Mirga. They should feature structured questionnaires and interviews intended to determine what has changed in the minds of the participant￾beneficiaries as a result of the project and what these beneficiaries see as the likely long-term benefit(s) to themselves, their families, their communities and their organizations resulting from the project. In addition, this research should seek out the factors in the surveyed project configuration that might suggest sustainability – or the lack of it – in the processes initiated by the project and in the attitudes of the beneficiaries, local government officials and the counterpart staff. In addition to informing itself and its counterparts about the results of these surveys and their implications, CRS/ET should consider creating a ‘newsletter’ type publication which could be used as a vehicle to publicize the results of this survey work, to describe projects like Lege Oda Mirga, and to discuss what it is attempting to accomplish in Ethiopia, presently and in the future. Such a (say, semi-annual) publication would not only provide a vehicle for the dissemination of 4 Food Insecurity and Vulnerability Information and Mapping Systems – an FAO-managed program operating in most of the world’s food insecure countries. 24 May, 2001 DAP Final Evaluation Report 21 this highly valuable survey work, it would also increase CRS’ visibility as a valid development ‘actor.’ It would, if the research were carefully accomplished and clearly written, reinforce the perception of CRS as a development ‘innovator.’ CRS/ET Progress: an in-depth case study was conducted in Lege Oda Mirga which has been discussed and distributed internally and to the USAID/ET Mission; a second study focusing on the impact of ADCS’s road construction activities was intended at Hayalom PA, Gulomakhada wereda in East Tigray. Unfortunately this had to be curtailed because of the Ethiopia-Eritrea conflict. Once comments have been received on the Lege Oda Mirga study, a final report will be prepared and circulated to key stakeholders. ‘Community Impact Briefs’ on the case study were developed to publicize the benefits of integrated watershed development programming. It is too early to evaluate the impact of these publications. MTE Recommendation # 32: CRS/ET, as part of the next PAA and for all results-type reporting thereafter, should discuss frankly and fully what evidence is has – anecdotal, narrative, trip report findings, discussions with project participants and from other sources – that adds the institutional development and organizational strengthening aspects to these reports. CRS/ET Progress: whilst CRS/ET staff trip reports may include institutional development and organizational strengthening issues as appropriate, the issues arising are not at present made explicit in the results-type reporting. MTE Recommendation # 33: In light of the monetization problem, CRS/ET should, as a first step, undertake a thoroughgoing review of all activities with the objective of ranking them in terms of importance to the overall objectives of the DAP, should continuation of constrained monetization proceeds require that the overall of effort be reduced. In that eventuality, the lowest ranking activities (either by functional component, or by counterpart) should be suspended or terminated early. CRS/ET Progress: due to the integrative nature of the present DAP activities, it has been difficult to reduce specific activity levels, rather entire counterparts have been phased out to compensate for constrained monetization proceeds. Counterparts have been requested to consider priorities for the DAP II. In light of the DAP-related problems encountered with monetization, CRS/ET commissioned a report which recommended the creation of a Monetization Business Unit (MBU). The MBU was established in October 2000 and has since prepared its first business plan. MTE Recommendation # 34: While one option might be to reduce, but continue, the efforts in all DAP areas and to propose to USAID that the present DAP be extended to, say 2003, in order that the original targets be retained, but achieved on an extended timetable, the Evaluation Team recommends that this not be done. USAID/Washington will be inundated with NGO requests for DAP extensions and is unlikely to be receptive to this approach. The Team believes it would be better for CRS/ET to close this DAP on schedule with a reduced set of 24 May, 2001 DAP Final Evaluation Report 22 well-documented successes, plus lessons learned from the less successful elements – all of which can be used to strengthen the design of follow-on activities. This would enable CRS/ET to start thinking early about the nature of the follow-on (and the Evaluation Team strongly believes there should be a follow on) and to undertake the analysis of the present components of interest which would then be used to build much stronger, well-documented arguments to support a DAP II. CRS/ET Progress: USAID made the decision to extend CRS’s DAP by one year which CRS will use to phase out of various geographic areas, counterparts and also to plan for the next DAP. MTE Recommendation # 35: CRS/ET should begin to identify a small number of potential new counterparts for inclusion in a follow-on DAP in the period beyond 2001. The strengthening of an ever-larger set of local NGO counterparts must be an essential element of successful, locally-based and extensive development in the 21st Century. CRS/ET Progress: four new partners (Sidama Development Corporation, Team Today and Tomorrow, Water Action, and Organization for Social Services for Aids) have been identified. MTE Recommendation # 36: CRS/ET should consult with other major NGOs in Ethiopia to find ways to achieve training ‘economies of scale’ and resultant reduced per-trainee costs by some form of ‘cross-training’ where one NGO provides training in, say, field surveys while another offers budgeting and management and a third has workshops in small enterprise development, etc. CRS/ET Progress: FANTA has been brought in for training all cooperating sponsors. However, in general this recommendation has not been actively pursued since CRS/ET’s specific training requirements do not necessarily match those of the other major NGOs in Ethiopia. MTE Recommendation # 37: Both CRS/ET and its counterparts must intensify efforts to include, as full and active partners – and at each level of project design, implementation, review, monitoring and evaluation – members of the local communities where the activities are operating, officers of the VDCs, the PAs (by whatever official name they are known), wereda, zonal, regional and national governmental bodies. There should be regular and frequent contact, and offers made to these officers to participate in DAP activities, reviews and planning processes.5 5 In defense of CRS and the counterparts, there have been numerous attempts to improve communication. There was the concept of the ‘local contract’ which was used in some areas in order to attempt to make all parties aware of their mutual obligations and of the ‘time limited’ nature of the DAP activities in their area. Unfortunately, in some of the DAP areas, it was apparently not enough. Some counterparts seem to have been better able or more willing to communicate – and cooperate – regularly with local government officers than others. 24 May, 2001 DAP Final Evaluation Report 23 CRS/ET Progress: implementing partners have to varying degrees involved local communities in planning and implementation. Full implementation of this recommendation has been constrained by the prescribed framework of the current DAP. It is expected that planning for the next DAP will involve a greater degree of community participation than was the case for the current DAP. MTE Recommendation # 38: CRS/ET and USAID should determine the feasibility of a “Year Zero” concept for future DAP activities in Ethiopia. This design modification could go a long way toward improving joint communication at all levels and hasten the time when all parties are willing and even anxious to work collaboratively to implement an activity that all have determined meets their joint objectives. CRS/ET Progress: effectively the one year extension of the current DAP has allowed for a period of planning for the next DAP. The utility of the extension period of reflection and planning should inform CRS and USAID regarding the feasibility of year zero for future DAPs. MTE Recommendation # 39: CRS/ET and its counterparts should initiate more “look, you can do it” trips as a means of increasing the effectiveness of their overall training efforts. CRS/ET Progress: the value of these kinds of trips is recognized by CRS/ET and its implementing partners. HCS has undertaken exchange visits internal to its development area (e.g. water caretaker training on user-fees). The degree to which this can usefully occur with the development area of an implementing partner depends upon the extent to which there is a diversity of experiences between weredas. Limited exchange visits have occurred between implementing partners (e.g. selected ADCS staff have visited HCS). CRS/ET’s ‘Community Impact Briefs’ support this type of exercise. MTE Recommendation # 40: Using the food security equation as a starting point, CRS staff should develop a methodology to quantify the level of household food security and food insecurity in Lege Oda Mirga as a means of developing a food security/insecurity measuring tool to be applied elsewhere. CRS/ET Progress: this recommendation has not been take up by CRS/ET as it has not been considered a priority issue. MTE Recommendation # 41: Building on the surveyed, evaluated experience of Lege Oda Mirga (and other areas of notable successes or failures) CRS should initiate planning in late 1999 and early 2000 for a new DAP built on the lessons learned from these components of the present DAP. CRS/ET Progress: delayed by a year due to the extension, this activity is now underway. A case study report has been written about the Lege Oda experience. 24 May, 2001 DAP Final Evaluation Report 24 MTE Recommendation # 42: CRS/ET should reconfigure its E&M unit to initiate and coordinate this effort. To date the unit has done a commendable job gathering information to inform tactical planning and operations efforts. What the reconfigured E&M unit would be asked to add to the work effort would be the capacity to inform long-term strategic planning in ways that would clarify the usually ill-defined relationship between what the project accomplishments at the output level and actual, sustainable progress by beneficiaries toward overall food security objectives. CRS/ET Progress: CRS/ET has been keen to see the M&E unit develop its role more as a facilitating unit, i.e. that it focuses its work in support of the regular M&E activities undertaken by CRS/ET project officers. It should be noted that the M&E unit currently comprises one staff member. MTE Recommendation # 43: The continued lack of consensus over the advantages and disadvantages of food aid in the Ethiopian context suggests the need for a national conference on the pluses and minuses of using food aid for development programs. This could be an attempt to try to get everybody reading from the same script – or at least talking the same language about food aid and food security in Ethiopia. CRS could take the early lead in suggesting such a conference and might be able to line up support from USAID, CARE, WFP, and participation by these organizations as well as other donors and NGOs and the involved government ministries, bureaus and offices. CRS/ET Progress: CRS/ET participates in food aid discussions, but does not consider a leading role to be appropriate at the current point in time. Evaluative Comment Progress towards implementing the MTE recommendations has generally been satisfactory. In some cases (e.g. recommendations # 40 and # 43) CRS/ET took the decision that these were not, at this stage, priority concerns. Other outstanding issues have been discussed with CRS/ET and counterpart staff and are dealt with under Task 3. 24 May, 2001 DAP Final Evaluation Report 25 Task 2: concisely assess CRS/ET progress toward achieving the goal, objectives, and intermediate results of the DAP Introduction Data for this section of the final report indicating planned and actual achievements are provided in the Results Reporting FY00 tables prepared by CRS/ET. An overall evaluative assessment of progress is provided at the end of this section. Although there were originally five Intermediate Results, assessment of IR2 (Increased Household Income) has not been undertaken given that CRS/ET agreed in FY98 to put on hold any expansion of microfinance activities planned under the DAP.6 The remaining four IRs, and their contributions toward achieving the goal and objectives of the DAP, are considered in turn: · Intermediate Result 1: Increased Agricultural Production · Intermediate Result 3: Improved Health Status in target areas · Intermediate Result 4: Natural Resource Base Maintained · Intermediate Result 5: Emergency Response Capacity Maintained Findings Intermediate Result 1: Increased Agricultural Production Progress towards achieving planned activities has been affected by well￾documented developments in the external environment (i.e. restrictions on issuing credit and budget cuts arising from monetization uncertainties). Critically, the outcome of these changes has been that: · DAP-funded credit activities were not implemented in FY99 or FY00; · the widespread introduction of productivity enhancing purchased inputs (improved varieties of seed and inorganic fertilizer) has fallen short of initial projections; · many of the small-scale source irrigation projects originally planned for the DAP were postponed due to budgetary constraints and in anticipation of the SSI Report (CRS, 1999). During the DAP period since the MTE only cropland bunding and, in some cases, organic fertilizers have been promoted extensively; the introduction of spate irrigation schemes have played a relatively minor role in DAP activities. Not surprisingly, the gains to be expected from this more limited range of interventions cannot match those envisaged in the original DAP. 6 This course of action was necessitated following the enactment of Proclamation 40/1996. In response to this regulation, CRS/ET commissioned an analysis of DAP microfinance activities which has recently been completed (Assefa, 2000), leading to the development of a business plan for a new CRS/ETCS microfinance institution (CRS/ET, 2000b). 24 May, 2001 DAP Final Evaluation Report 26 Proxy indicators for IR1 (cropland bunding, organic fertilizer use and irrigation) suggest improvements over the baseline estimates: the percentage of target crops in the target areas bunded, the estimated quantity of manure/compost, and the percentage of crops in the target areas irrigated were all up on baseline levels. These are important developments; they have been possible as they have not required farmers to purchase substantial DAP inputs beyond food. Although most of the original targets were not met this can be explained in part by the more time-consuming, though very laudable, attempts at a participatory approach to DAP implementation, and probably to a certain degree of over-optimism at the time of program preparation. It is difficult at this point in time to have confidence in appraising the impact indicators as the final results survey has yet to conducted; that said, it should be noted that there will be serious difficulties involved in attributing production increases in a five-year agricultural program of this nature (Casley and Kumar, 1987). For example, drought has affect agricultural production in the development areas of many of the implementing partners. Such issues concerning the use of crop yield as an indicator of DAP success should not be underestimated and should inform the selection of indicators for any subsequent DAP. The data collected by CRS/ET and its implementing partners appears to be of an acceptable quality with enumerators receiving reasonably adequate training and supervision. The method used by implementing partners, agreed for use by the Cooperating Sponsors, for selecting sample sizes for the annual results surveys (based on a percentage of population size, rather than required confidence limits and margins of error) should be reviewed (Scott, 1985). Intermediate Result 3: Improved Health Status in target areas All IR 3 indicators with data provided (KPC related indicators will be reported on when surveys are completed at the end of FY01) improved over baseline. There were many impressive impacts in key indicators such as reduction in moderate and severe malnutrition (e.g. 44% to 32% moderate malnutrition) full immunization rates (42% to 79%) and antenatal care visits (32% to 64.2%), mothers receiving 2 or more doses with TT (21% to 76.5%). While several targets were not met, the percentage change over baseline was significant. Targets that were not achieved such as moderate malnutrition may have been due to over ambitious percentage decreases and disruption in activities (e.g. HCS). It is difficult to comment on the progress towards the key impact indicators as follow-up KPCs have not yet been completed. KPCs that reflect true population based statistics may show quite different results from register based data. There is some concern about the representativeness of the data reported considering that the few CBHC sites visited had incomplete registers covering only children in the food program at present. The number of children registered has decreased since baseline further evidence that the registers are not 24 May, 2001 DAP Final Evaluation Report 27 recording all children under two in the communities. It is inappropriate to use the registers to represent population-based data that is reported in annual results indicators. In addition to the lack of representativeness, registers were also incomplete. Child registers had most information recorded, but some essential data such as the number of home visits made, Vitamin A coverage, and height upon exit were not complete. The maternal registers viewed had either been discontinued or were missing information such as antenatal care, TT rates and delivery status. While most indicators have shown impressive positive changes, there are still large gaps between present levels and final DAP targets. It is unlikely that follow￾up KPC data will show that most targets have been completely met. This perhaps is a reflection of the ambitious targets set by counterparts for each indicator. This should not be viewed as a failure of activities given that all trends were positive and that the operating environment was often problematic and non-conducive to rapid achievements. Intermediate Result 4: Natural Resource Base Maintained This IR focuses on reclaiming degraded areas. It is seen by CRS/ET and its implementing partners as contributing importantly to achieving successful outcomes in IR1. Hillside terracing is perhaps the ultimate symbol of the role of FFW in Ethiopia and of the courage of the Ethiopian people. Other related activities include gully reclamation, grass strip planting, microbasin construction, cut-off drain construction, and related tree nursery operations. Given that improvements on communal land demand comprehensive agreements with local communities, all of which takes substantial investments in time, the achievements of implementing partners as reflected in the improvements in indicators over baseline are encouraging. The potential benefits of hillside terracing include reduced soil erosion, increased rainfall infiltration, increased vegetative cover and fodder availability and, of significance in parts of the ADCS development area, increased pollen availability for bees. The principal impact indicator selected for IR4 focuses on the first of these benefits. Although the 1997-2000 target of 1,758ha for the area of communal land to be protected/reclaimed has not been achieved, this is not to deny the importance of the 1,544ha that has been protected. The impact of such natural resource management practices on reducing soil losses (estimated using the Universal Soil Loss Equation – USLE) and improving vegetation cover appears from the data collected to date to be positive. USLE data have been collected in 25 watershed sites. Visual evidence from the Lege Oda Mirga watershed, where there have been significant successes in protecting communal land, would support the reported USLE data. Nonetheless, there is current debate regarding the use of the USLE as a mechanism to measure actual soil losses, or whether it is more appropriate as a tool for projecting anticipated future changes. 24 May, 2001 DAP Final Evaluation Report 28 There appear to be marked differences between implementing partners in the degree to which protected/reclaimed areas remain so, and anecdotal evidence suggests that survival rates for tree species planted in communal plantations vary between 45-65%, although on private lands rates are higher. As agreed by the Cooperating Sponsors, IR4 indicators do not report these data as part of the quantitative reporting system. Although not included in the original indicator table, another impact of IR4 activities is increased groundwater recharge. HCS have collected data suggesting increased flow rates of selected springs and hand-dug wells of between 8 and 20%. These data are encouraging, and villagers’ comments would support these initial findings; nonetheless, CRS/ET and counterpart staff recognize that more careful assessment is required to validate the reported quantitative improvements. Efforts are now being made by CRS/ET and its implementing partners to collect such data on a more systematic basis, i.e. on a twice-yearly basis, complemented by the collection of rainfall data to provide the context for understanding any recorded changes in flow rates. HCS’s experience in Lege Oda Mirga suggests that long-run impact in maintaining the natural resource base is positively related to the quality of community participation which, in turn, is related to the level of trust implementing partners are able to engender among community members. Community members have been provided natural resource management training that has included the development of by-laws of associations through which communal natural resource management tasks will be administered. Intermediate Result 5: Emergency Response Capacity Maintained The fifth intermediate result of the USAID DAP Special Objective Framework is the maintenance of an emergency capacity. While the sole indicator for the IR is the presence of an updated emergency preparedness plan, there is no need to investigate the presence of a plan to maintain capacity as CRS/ET is presently proving its emergency capacity. During the past year CRS/ET has engaged in the Joint Emergency Operation Plan, an emergency response to recent drought, handling over 200,000 MT of food for CARE, SCF/US, WVI and JRP distribution. Evaluative Comment Progress towards achieving the indicators has seen in many cases improvements over baseline for many of the proxy indicators. As explained in the above text, there have been changes in the ‘external’ environment, i.e. outside of CRS/ET management control, which explain the poor performance on some of the indicators. 24 May, 2001 DAP Final Evaluation Report 29 Task 3: provide guidance and recommendations on specific strategies CRS/ET should incorporate into the next DAP proposal so that overall program quality and impact will improve Introduction As a result of the Team’s field observations and discussions with various stakeholders, the following topic areas and assigned responsibilities were identified as important for inclusion in this section of the report: 1. Agriculture/Natural Resources Management 2. Health and Nutrition 3. Water and Sanitation 4. Use of Food Aid 5. Monitoring and Evaluation 6. Expanding Partnerships with ‘External’ Partners 7. Safety Net 8. AIDS/HIV 9. Emergency Preparedness and Mitigation Strategies Guidance and Recommendations 3.1 Agriculture / Natural Resources Management / IWM 3.1.1 Introduction Rural Ethiopians face various categories of food insecurity, both short-term (acute, transitory, seasonal) but especially long-term (chronic). Whilst it is not possible to attribute all the blame for high rates of malnutrition and child mortality on household food insecurity, nevertheless a good deal of under-nutrition and ill￾health in Ethiopia can be explained by diets which are inadequate in terms of both quantity and diversity. 3.1.2 Integrated Watershed Management Strategy An Integrated Watershed Management (IWM) strategy is now a key conceptual framework for CRS/ET’s planning activities. Although the IWM framework was envisaged as a strategy to enhance the food security situations of target households under the current DAP it was not implemented as planned in all the operational areas of counterparts. CRS/ET and counterpart staff are now developing and consolidating their understandings of the IWM strategy in a series of papers, and discussions with CRS/ET and counterpart staff suggest that this strategy will be the core conceptual framework for all project planning activities under the next DAP. The Evaluation Team supports this strategic approach and it is in line with USAID/ET Mission strategy. The IWM strategy is appropriately seen as guiding “CRS/ET’s continuing move towards increased integration of sectors” (CRS/ET, 2001b). The key advantages of an IWM strategy are its potential to (a) encourage the integration of technical, 24 May, 2001 DAP Final Evaluation Report 30 economic, social and environmental perspectives, and (b) provide a useful way for CRS/ET and its implementing partners to operationalize considerations of inter- and intra-sectoral integration. Evidence of use and impact of this strategic approach is provided in the Lege Oda Mirga case study report. CRS/ET’s approach to IWM incorporates the following components: · Improved Crop Management · Improved Soil and Water Management · Improved Livestock / Feed Management · Alternative Land Use / Watershed Planning · Health and Nutrition Interventions · Water Resource Development · Household Income Diversification · Partnership Promotion From a broader perspective, CRS/ET’s IWM focus on environmentally sustainable improvements in the agricultural productivity of poor households in food insecure watersheds is in line with current pro-poor strategic thinking on rural development. Much intellectual effort by international (e.g. IFAD, 2001) and donor agencies, including USAID (USAID/ET, 2000), puts rural poverty reduction, and its related impacts – food security, nutrition and environmental protection – in pride of place. There is also increasing recognition that rural poverty-related issues are increasingly concentrated in lower potential areas – the kinds of vulnerable areas in which CRS/ET is proposing to work – that are hard to reach. CRS/ET’s strategic thrust is thus entirely appropriate. The livelihoods of resource poor farmers in Ethiopia are secured through the pursuit of a diverse range of activities. Activities within the agriculture/natural resources management component of an overall IWM strategy will constitute a significant strand in the complex web of household livelihood strategies. CRS/ET and its counterpart staff need to identify the potential to assist rural households in a variety of ways. This will include strengthening and diversifying their traditional cropping patterns, including those crops that can be grown in years of poor rainfall, with minimum external inputs, and still yield an ‘acceptable’, albeit low, level of output. 3.1.3 Guidance and recommendations on specific AG/NRM/IWM strategies Participation - establishing community trust and confidence One of the key lessons from HCS’s experiences in the Lege Oda Mirga watershed is the importance of establishing and maintaining the trust and, subsequently, participation of local government and PA officials, and other community members. Discussions with HCS staff indicated that this process initially took up to 12 months. Although seemingly long, this investment in time was later instrumental in ensuring successful implementation of project activities. Having undertaken the early confidence-building groundwork, staff were more easily able to discuss openly and freely with community members about sensitive 24 May, 2001 DAP Final Evaluation Report 31 issues such as the closure of communal land on reclaimed hillsides and how it would “disproportionately affect the poorest households” within the community (CRS/ET, 2000d). A more (at least compared to the planning approach adopted for the current DAP), participatory approach to planning is seen as a key element underpinning forthcoming planning activities for the next DAP. There is some concern among CRS/ET and counterpart staff about the low participatory skill-base and understandings7 existing within CRS/ET and implementing partners. Additionally, there are substantial programmatic considerations, i.e. whether there is already insufficient time to undertake participatory planning activities properly in time to meet the submission date for the next DAP. There are unlikely to be shortcuts to good quality participatory work. Recommendations: 1. Given the investments in staff time required to establish meaningful PA and village-level relationships CRS/ET and counterpart staff need to establish the implications for: · scheduling planning work prior to the next DAP submission; · establishing and agreeing upon the appropriate scale of operations of implementing partner activities for the next DAP period;8 · determining feasible outcomes and targets to be included in the next DAP proposal. CRS/ET and counterpart staff should learn from the HCS experience and be guided in setting realistic responses to the three points raised above. 2. Training, in some cases refresher, in participatory approaches is required for CRS/ET and counterpart staff given the importance attached to this conceptual approach to planning the next DAP. Site-specific programming - recognising diversity In the 1997-2001 DAP, agriculture productivity was to be increased by a combination of cropland bunding, improved nutrient management including access to fertiliser, use of improved technologies including access to seed of improved varieties, and increased access to irrigation. There was a degree of uniformity to the original agriculture/natural resources management DAP proposals that is not warranted by the diversity of the DAP sites. This was largely due to the planning approach adopted for planning the current DAP which involved only a limited level of participation of counterpart staff and community members. 7 For example, some CRS/ET and counterpart staff view the process of mobilising farmers for FFW and the work itself as being essentially ‘participatory’, while others anticipate a more pro￾active role in decision-making for farmers. 8 This recommendation ties in with the CRS/ET’s SPP concerning the “capacity of CRS/ET section to provide program quality support for a given intervention in the area” in its discussion on geographic targeting (CRS/ET, 2001b). 24 May, 2001 DAP Final Evaluation Report 32 Follow-up DAP planning activities are now underway, and CRS/ET are currently proposing to target ten geographic zones covering eight regions (CRS/ET, 2001b). Planning is expected to commence at field level, with implementing partners initiating a more participatory approach among their respective communities. These locations are agro-ecologically and socio-economically different from each other. The appropriateness and uptake of CRS/ET’s proposed cash crop component – sesame, chickpea, groundnut and sweet potato – will likely be different for each location. Such differences should inform and be reflected in the specific activities planned for each site and included in the final submission to USAID/ET Mission. Recommendations: 1. Agricultural / natural resources management proposals are expected to be initiated locally on the basis of discussions between counterpart staff and the communities within which they are planning to work. It is important that CRS/ET ensure that the diversity in agricultural and natural resource management systems across these zones is reflected in the next DAP proposal submitted to USAID/ET Mission. ‘Blanket’ agriculture / natural resources management proposals are to be avoided. 2. CRS/ET should provide counterpart staff with technical support and on￾the-job training to facilitate this process. Targeting - recognizing and working with intra-community differences Within the communities where the implementing partners are proposing to work, the access of individual households to rural livelihood assets (physical, natural, social, human, and financial) also differ between households in a community. Anecdotal evidence, for example, from the currently targeted areas suggests that land holding sizes vary from between 0.1 to 0.75 hectares, an almost eightfold difference in a key factor of production. In other words, some households have access to a greater set of resources than do others.9 Working with key informants in villages (e.g. PA officials once trust has been established) it is possible to rank households in a community using locally-derived criteria for ‘well-being’. Participatory well-being ranking has been used successfully by CARE in Zambia (Drinkwater and Rusinow, 1999) and in Tanzania (Temu and Due, 2000). Some CRS/ET and counterpart staff already have experience of the use of this technique in Ethiopia for relief food distribution purposes. Providing ranking work is undertaken with due care and sensitivity,10 it is a potentially powerful tool for programming: it enables project staff to disaggregate so-called ‘community’ problems into the more specific problems of the differently ranked groups within the community. If for no other reason, a gender sensitive 9 Raisin (2001) has recently proposed an approach to socio-economic differentiation for the “highland chronically insecure areas” of Amhara Region that is based on oxen ownership,9 and she suggests that “four basic household types can be identified.” 10 This aspect of participatory planning work is very important. 24 May, 2001 DAP Final Evaluation Report 33 strategy (CRS/ET, 2001b) demands disaggregation between men- and woman￾headed households. Agriculture / natural resources management technologies and support can then be more appropriately targeted whilst, at the same time, recognizing that “the household economy never stands alone from the wider community” (Raisin, 2001). Different groups may have different problems and/or different solutions to common problems; for instance, different socio-economic groups are likely to display different attitudes towards risk; in turn their ability and/or willingness to adopt new technologies promoted under the next DAP will vary. Understanding socio-economic differentiation within targeted communities permits more finely-tuned client-oriented agriculture / natural resources management interventions and analysis. As an example, the cash crop component of CRS/ET’s IMW strategy includes the promotion of groundnuts, sweet potato, pigeon pea, chick pea, sesame, and eucalyptus. What will be critical for CRS/ET and counterpart staff is identifying whether there are revealed preferences that differ between socio-economic groups within a community. Do women prefer one of these crops to another because it is more suitable for feeding to children, or because it allows her to generate her own source of income? Do farmers with limited access to financial resources express a preference for those crops that perform most satisfactorily at low levels of input use? It is thus vital that staff are conscious of the potential range of preferences of different groups of households so that investments in agriculture / natural resources management technologies and support can achieve significant returns. Recommendations: 1. With CRS/ET technical and financial support, selected implementing partners should pilot participatory well-being ranking, on an initially limited basis, during the planning period to contribute to activities proposed under the next DAP proposal. 2. If found to be useful for project planning purposes, the approach should be initiated elsewhere during program implementation, and in appropriate instances M&E data can be presented in a more disaggregated (by socio-economic category of household) manner. 3. It may be appropriate to seek external training assistance for the initial attempts at well-being ranking. Quality programming 1 – small scale irrigation planning For various reasons, SSI played a much smaller role than originally expected under the current DAP. This was for reasons largely beyond the control of program management. In addition, with careful inter- and intra-sectorally integrated planning within the IWM framework SSI proposals can be explicitly linked to other developments, including a consideration of output markets, planned under the next DAP. Indeed, SSI developments are most likely to be integrated with market-oriented 24 May, 2001 DAP Final Evaluation Report 34 interventions at the micro-level. Although the Evaluation Team was not able to see many SSI schemes developed under the current DAP, those that had been developed (e.g. pump irrigation schemes in Meki Catholic Secretariat, and at Kenchera near Dire Dawa) were explicitly linked to nearby output markets. This is likely to be an important consideration for CRS/ET and its implementing partners since recovering the costs of the initial SSI development is more likely to be achieved with the generated cash flows derived from marketed outputs. The potential for SSI development in the proposed development areas under the next DAP is limited: according to the PEA, it is the same agro-ecological and socio-economic circumstances which lead to food insecurity that also constrain an expansion of SSI (CRS, 1999). Significantly, it has been estimated that “the area with potential for irrigation in food insecure weredas is a maximum of 5 per cent of the total area” (CRS, 1999). Financial considerations will also preclude a large expansion of SSI under the next DAP, although up-to-date cost data are difficult to find (CRS, 1999). Nevertheless, it is possible to consider a small, but locally significant, and well-integrated role for SSI within the next DAP, but this must be underpinned by sound social, economic and institutional analysis, in addition to the technical appraisal that currently takes place.11 As the SSI Report bluntly states, SSI should only proceed with an “understanding of the percentage of program resources that SSI absorbs compared to the number of beneficiaries” (CRS, 1999). CRS/ET has a role here to support relevant capacity building within implementing partners. Implementing partners are likely to require assistance in undertaking such analyses. Concern has recently been raised about the “relatively weak” nature of “organization and participation in the water user committees or associations” (Messele, 2001). CRS/ET and its implementing partners could usefully develop their understandings of the functioning of such organizations during the planning period for the next DAP, and thereafter. Such knowledge would inform subsequent developments in this sector. Recommendations: 1. Implementing partners should identify the scope for integrating SSI development with market-oriented agriculture during the planning process. 2. CRS/ET should strengthen the capacity of implementing partners through training and other support to ensure that SSI developments planned under the next DAP are based upon sound technical, social, economic and institutional analyses. · SSI training needs of CRS/ET and counterpart staff need to be identified and met; 11 For example, the PEA Report was “unable to identify a single instance where [the concepts of cost/benefit] analysis have been realistically applied to the planning process associated with a given scheme” (CRS, 1999). Furthermore, the Report suggests a minimum of two years “advance effort” being required to ascertain the feasibility of SSI on a given site (CRS, 1999). 24 May, 2001 DAP Final Evaluation Report 35 · This may involve the development of links with external organizations for appropriate support (e.g. ADCS with REST in Mekele, or with Department of Agriculture Irrigation specialists). 3. CRS/ET should guide implementing partners in establishing and managing links with external organizations, e.g. with regard to preparing Terms of Reference for contracted work, and in overseeing the quality of their work. 4. CRS/ET and its counterpart staff should initiate a system for monitoring the ‘post-program’ performance of water user committees or associations established between 1997-2001, and use this knowledge to strengthen subsequent SSI program quality. 5. CRS/ET should strengthen its links with international networks related to SSI (e.g. IWMI, IIED). Quality programming 2 – farmer-led on-farm research for agricultural productivity improvements CRS/ET can help to steer agricultural activities implemented under the next DAP towards becoming more relevant to the needs of rural poor farmers. Agricultural productivity improvements planned under the next DAP should not rely on the introduction of technologies that require expensive associated inputs since seasonal finance is unlikely to be available to most targeted farmers under the next DAP.12 However, the proposed DAP provides an opportunity for selected CRS/ET implementing partners, in conjunction with external organizations, to undertake farmer (i.e. demand)-driven small-scale adaptive on-farm agricultural testing of potentially more appropriate lower-input technologies.13 Useful testing, possibly building on farmers’ own experimentation, will identify ways in which the marginal returns to small farmer labor can be increased without adding substantially to the costs, labor use and risk involved in producing crops. This would enable CRS/ET and counterpart staff to assess the preferences of farmers for the crop/livestock technologies they will be promoting under the next DAP. CRS has experience of this type of activity elsewhere in East Africa, and is planning to include this aspect of programming in the next DAP. The Evaluation Team supports this concept for the next DAP. The farmer-led work described above would provide a suitable vehicle for CRS/ET and its implementing partners to develop further its links with international and local organizations (e.g. ILRI, ICRISAT, IWMI, CIP, and the Ethiopian Agriculture Research Organization)14 in the cause of boosting poor 12 The Microfinance Initiative Business Plan will initially focus on urban areas. 13 An initial start could be made via farmer-led discussions regarding farmers’ preferred contents for an initial ‘Dream Pack’ (see Cromwell, 2001) of agricultural technologies that they could subsequently trial and evaluate. 14 For example: 24 May, 2001 DAP Final Evaluation Report 36 farmer productivity. HCS have already initiated links (inter alia with regard to seed multiplication) locally with Alemaya University. Such partnerships should be greatly encouraged under the next DAP via the identification of a specific collaborative venture. There are different models for promoting collaborative research (and other) ventures, but it is critical that any partnership within the agricultural/natural resources management program keeps in mind a focus on the priority needs and potential of different farmers. Recommendations: 1. CRS/ET and implementing partners should pursue further with the CRS East Africa Regional Office the possibility of incorporating farmer-led on-farm crops/varieties testing in the next DAP proposal. 2. CRS/ET should identify ways to build implementing partner capacity to facilitate farmer-led testing and evaluation. 3. Program quality will be enhanced in the next DAP proposal via greater partnership with technical organizations such as EARO and the IARCs in technology (and other) exchanges. CRS/ET has a key role in facilitating and supporting such developments, and the CRS East Africa Regional Office should be approached for assistance in this regard. 4. CRS/ET should use the DAP extension period to determine if there is an opportunity to include a collaborative venture under the follow-up DAP. Integration 1 – market orientation and market access For farmers in a position to benefit (e.g. because of their location near to a market), strategies that strengthen their ability to gain from profitable marketing activity are critical. It is important that implementing partners do not merely view private traders “as speculators who reap unjust profits at the expense of poor farm communities” (Remington and Kumssa, 2000). Poor farmers need technologies to increase output from their assets, and they also need markets to exchange that output freely and to best advantage. CRS has now accepted that, “the private sector plays a key role in ensuring that the [agricultural] system is efficient and sustainable” (Remington and Kumssa, 2000). The significant success of households in Kenchera, who now “earn income by marketing vegetables in Dire Dawa town, where the demand is continually high due to its · EARO has a Drylands Crop Research Program. They are likely to have locally-developed agricultural technologies to increase the productivity and profitability of small farmers; · ILRI has a long running program in integrated nutrient management based on manure and on improved forage production; · IWMI has expertise in multiple use of water with a focus on irrigation – also on environmental health, hydrologic monitoring, participatory planning and management and gender and irrigation; · ICRISAT has improved varieties of short duration sorghum, chickpea, and groundnut; · CIP has high Vitamin A sweet potatoes. 24 May, 2001 DAP Final Evaluation Report 37 proximity to the Djibouti market” (CRS/ET, 2000d), shows what is possible given the right circumstances, even in food insecure areas. HCS’s facilitating and guiding role in this instance was central. All things being equal, this kind of development is likely to be sustainable in the medium- to long-term. For poor farmers in more remote areas, however, market access problems are serious, and competition and information least adequate. In general, access to information is an overlooked area for the rural poor; more information would enable farmers to take better decisions on markets and services. With more integrated DAP planning, CRS/ET investment in rural roads is potentially very important in this regard, although the economic merits of undertaking such infrastructural development must be shown on a case-by-case basis. Established markets then provide the boundaries for additional DAP-funded interventions and farmer-led research to improve households’ returns to labor, not to land. Where DAP activity is more market-oriented, the focus should be on returns to agricultural producers. Recommendations: 1. Through its integrated planning process, CRS/ET should encourage its implementing partners to identify the current opportunities in each targeted area for supporting market-oriented agricultural activities in the next DAP proposal. 2. Given the lack of in-house agricultural marketing expertise, this will necessitate accessing appropriate technical assistance to appraise proposals put forward by the implementing partners. 3. Where DAP activity is more market oriented, indicators in the next DAP should focus more on returns to producers, and less on returns to land (i.e. yield in kg/ha). Integration 2 – seeking opportunities to benefit from the MFI: Microfinancial services are very important for small farm agriculture. As poor farmers in CRS/ET’s proposed areas experience population pressure on fragile environments, they have to decide whether to protect their livelihoods by intensifying, extensifying or exiting. The first option is the only one that can sustainably increase production and reduce food insecurity; whether or not it is taken depends crucially on access to, and the suitability of, available financial services. The current Business Plan (2001-2005) of the CRS/ECS Microfinance Initiative (MFI) indicates that, “given the infrastructure and high potential markets for microfinance services, more emphasis should be given to urban areas during the first plan period” (CRS/ET, 2000b). Thus the majority of poor farmers are unlikely to see benefits from the MFI planned for the next DAP, except those living close 24 May, 2001 DAP Final Evaluation Report 38 by to the proposed MFI’s urban localities.15 For the latter, market-oriented quality programming and support will be critical. For the former, agricultural credit is a complicated issue. The record of agricultural credit schemes in most of sub-Saharan Africa is dismal: conventional loan-only financial products usually increase risk for the rural poorest, and frustrate coping strategies that are based on the avoidance of risk. There is currently much discussion concerning the need to understand what design features can render microfinance a useful tool in protecting and enhancing the livelihoods of agriculture-dependent rural households. Debate is divided over the question whether it is possible for people who are malnourished to use production credit effectively. It now generally accepted that the demand for financial services varies by income, with the rural poor preferring to save and insure themselves16 but, as a rule, not borrow. This choice is sometimes, but not always, available to them and is a point to bear in mind if and when the CRS/ECS MFI does become more deeply located in CRS/ET’s targeted rural areas.17 Recent evidence now seems to suggest that with microfinance the degree of poverty reduction via the labor market is greater than the direct impact of poor people borrowing. Where active rural labor markets exist, microcredit to the non￾poor can reduce poverty by drawing very poor people into the labor market as employees of microfinance clients. Thus it is possible that the CRS/ECS MFI Business Plan, which is planning to focus initially on urban areas (e.g. Nazareth in year 1, Harar in year 3) can, if successful, generate off-farm labor opportunities for rural poor. This kind of impact will depend critically on the vibrancy of the rural labor market, not unrelated to the market orientation and access issues discussed earlier. Such considerations should inform integrated planning discussions under the next DAP; again, the issue of improving the mobility of rural farmers and the corresponding costs and benefits of road construction will be central. Recommendations: 1. The proposed CRS/ET Microfinance Business Plan will directly benefit poor farmers only if they live sufficiently close to the urban sites proposed in the MFI Business Plan. Market-oriented quality programming and support will be critical and agricultural marketing 15 That said, in West Hararghe and Dire Dawa recently, two finance associations has been established and legalised via approval from the Ministry of Justice. 16 There appears to be a big unmet demand for microinsurance with the emphasis of risk management systems of poor farmers. The World Bank is currently sketching a microinsurance scheme for inter alia Ethiopia. The scheme is based on rainfall rather than crop yield in order to avoid reporting problems. 17 Paradoxically it is sometimes NGOs (e.g. CARE in Zimbabwe, and the Gatsby Trust in Cameroon) which force people, who only wish to save, into borrowing also in order to be part of a financial services network. Note also CRS/ET’s statement that the “MFI will focus exclusively on the provision of financial services (first credit, then voluntary savings)…” in the Strategic Program Plan (CRS/ET, 2001b). 24 May, 2001 DAP Final Evaluation Report 39 input may be required at the planning/appraisal stage prior to the next DAP. · For farmers in more remote areas, the MFI Business Plan will likely limit the extent to which they are able to adopt improved crop and livestock technologies that necessitate purchased complementary inputs. · This will impact upon the choice of agricultural technologies to promote under the next DAP. Implementing partners should plan their agricultural programs for the next DAP on the assumption that farmers will not have access to purchased inputs. Lower input (i.e. less risky) technologies should be preferred and should be based on farmers’ own expressed preferences. · Opportunities to link with other institutions offering financial and agricultural services should be sought. 2. Evidence suggests that indirect benefits (via a more active rural labor market) are possible, and CRS/ET and its implementing partners should identify opportunities to improve rural labor mobility via the appropriate development of rural roads in its integrated planning discussions for the next DAP. 3. CRS/ET should monitor the progress of the finance associations recently established in West Hararghe and Dire Dawa. Land tenure issues Poor farmers spend substantial amounts of their income on food; they receive most of their calories from the production of staple food crops (e.g. sorghum, maize, wheat, barley), and earn income from growing them; hence the importance of private access to land resources for poor farmers. The experiences of the current DAP have shown, not surprisingly, that farmers prefer to invest in private, rather than communal, land resources. Older, more traditional models of communal land ownership do exist. All land at present, however, is owned by government, although in some regions there is increasing flexibility with regard to user rights (e.g. Tigray and Amhara). Whilst the issue of land ownership is critical for the long-term sustainability of DAP interventions, there is little that CRS/ET and its implementing partners can do at present except by keep a watching brief over land reform developments in regions such as Amhara and Tigray, and contributing whenever possible to discussion forums as appropriate. Monitoring sustainability According to Cromwell et al (2001), “‘sustainability’ means different things over different timeframes and to different stakeholders.” There are three strands: i) on￾farm technical sustainability issues; ii) the wider economic and environmental impact of current farming practices; iii) the sustainability of institutions. 24 May, 2001 DAP Final Evaluation Report 40 Farmers in the areas visited by the Evaluation Team suggested that their main concern is immediate household food security. The Lege Oda Mirga case study reported an average of almost 6 months of food crops over a 12-month period (CRS/ET, 2000d), substantially up from the baseline, but still indicating severe food shortages ranging from “moderate to severe.” Whilst HCS’s agricultural production achievements in this watershed are to be applauded, there is clearly no room for project complacency. Sustainability for such poor farmers is a season-by-season issue, and is all about producing food for the family using farming practices that are most likely to achieve this. HCS’s on-farm technical package (cropland bunding and manuring) may be ‘sustainable’, but is it sufficient? In times of stress there is the potential for conflict between farmers’ perceptions of sustainability and the longer-term view of economic and environmental sustainability. Interestingly, however, the Lege Oda Mirga data appears to indicate that farmers in the target watershed are now more conscious of the wider sustainability benefits of the ‘package’ of improved agricultural / natural resources management practices promoted by HCS. Given the critical importance of this issue for the CRS/ET and implementing partners’ model of sustainable development, it would be invaluable to have the findings presented in the case study report independently verified by non-CRS/ET and non-HCS staff. This is not to say that the data are ‘wrong’, more that longer-term sustainability is such a central plank to CRS/ET’s agriculture / natural resources management strategy that it merits rigorous external validation. It is too premature to assess the institutional sustainability of CRS/ET and HCS’s approach in the Lege Oda Mirga watershed. That said, at present things looks promising, but it will be vital for on-going institutional structure and processes monitoring work to continue once the intensive inputs of the current DAP activity are phased out. This will provide invaluable data for all subsequent DAP programming. Recommendations: 1. To address the issue of on-farm technical sustainability, recommendations 2 and 5 (see summary box below) are particularly applicable. 2. On-going longitudinal monitoring should be continued in the next DAP after phasing out from the current DAP sites, in order to confirm the institutional sustainability of current DAP activity. CRS/ET should consider local external input into this work. Option to consider if resources available: 3. It is suggested that given the prominence given to the Lege Oda Mirga model for sustainable development, CRS/ET and HCS commission an independent survey to validate the reported case study findings. 24 May, 2001 DAP Final Evaluation Report 41 3.2 Health and Nutrition 3.2.1 Health Education Most of the health and sanitation indicators rely heavily on health education in order to effect behavior change. The MTE revealed that the health education component was weak and that IEC materials used were largely inappropriate. In response to these findings CRS/ET conducted an IEC assessment, which is endorsed by the Evaluation Team. The IEC assessment revealed that 100% of FACS coordinators, 75% of animators, 72% of CHAs, and 100% of TBAs used lectures to conduct health education sessions, which lasted approximately 20 minutes per month. Often times the health education is located far from the village, forcing women to walk a long distance. Additionally animators have no job aides, and therefore must rely on what they remember from training. While 20 minutes is a short period of time, messages will have much greater impact if mothers are engaged in dramas or role plays, which allow them to identify their own problems and negotiate solutions to overcome barriers, rather than if the mothers are lectured at and have no opportunity to identify their problems. CRS/ET has recognized the need for training project staff on adult learning methodologies and development of appropriate IEC materials. One partner, HCS, has already taken the initiative to train staff and animators on such methodologies in collaboration with Alemaya University of Agriculture and Larense College of Agriculture, the Netherlands. Unfortunately the Evaluation Team was unable to observe any health education sessions in HCS to see how the new training was being used. CRS/ET has developed an IEC proposal, which focuses on the identification and design of appropriate IEC methodologies. Before developing new materials, CRS/ET should find out what materials and methodologies have been used by other NGOs and what lessons they have learned. This will avoid duplication of efforts. As discussed with CRS staff during the evaluation, equally important as the appropriateness of the IEC materials, is the message development. Messages will need to be tailored to the attitudes of women, which may vary from region to region. Therefore, different health education modules will need to be developed, depending on the attitudes and care practices of women in target areas. It may be useful to develop a framework for each health message, such as (information in the table is only an example, it does not reflect data from the evaluation): 24 May, 2001 DAP Final Evaluation Report 42 Health Message Current Practice Barrier to Optimal Practice Possible Solution Exclusively breastfeed from 0-6 months · Weaning foods introduced too early (2-3 months) · Mother does not breastfeed on demand · Mothers work away from home · Fear of “black eye” if a mother breastfeeds in public · Mother could carry child with her when she leaves the house · Cover the child with a blanket when breast￾feeding in public to avoid black eye. CRS has recognized that such training involves intensive follow up and support to trainees. Therefore they are planning to implement the training on a pilot basis close to Addis, so that the new activities can be more easily supported. While the new IEC strategy will most likely improve the quality of health education, CRS acknowledges it is it is questionable how much impact a 20 minute session once a month (irrespective of how well it is conducted) will have on behavior change. Social and cultural beliefs surrounding antenatal care, breastfeeding, infant feeding practices, hygiene and HIV/AIDS often involve deep-rooted beliefs/attitudes, which are not easy to change. Other strategies are therefore needed in order to impact on behavior change. Home visits are part of the DAP strategy, however animators reported that they generally make home visits to all children in the program and repeat the same messages given during health education. In the sites visited by the Evaluation Team, the topics during home visits predominantly focused on environmental hygiene/sanitation and home improvement. Given the high rates of underweight and stunting in the targeted areas, one option could be training animators on nutritional counseling. Home visits would be focused only on children who growth faltered in the previous growth monitoring session, which would reduce the number of visits an animator is required to make in a month, and would free up more time to spend with sick or at risk children. Nutritional counseling cards have been developed by BASICS/Manoff Group in Latin America, and have been adapted by CRS in Madagascar and Kenya. One side of the card pictorially depicts a health message, which the animator can discuss with the mother (therefore the mother does not have to be literate to understand the message). The reverse side of the card contains a job aid for the animator on how to conduct a home visit, key information on the message contained in card, as well as possible concerns or questions mothers might ask. 24 May, 2001 DAP Final Evaluation Report 43 During discussions with mothers in the field, many expressed difficulty in utilizing messages regarding family planning, antenatal care and HIV/AIDS. They stated that their husbands needed to hear these messages from the animators because women are unable to educate their husbands on these topics. Therefore in order for health messages to be adopted, animators need to develop strategies to target men. Animators could take advantage of opportunities when men gather together, such as drinking coffee, to discuss such topics. It should be noted that often it is more culturally appropriate for male animators to talk to men and female animators to talk to women about sensitive topics such as family planning and sexuality. A powerful strategy observed by the Evaluation Team was demonstration. Animators had been taught how to construct mud stoves and latrines, using locally available materials, and act as a change agent for the new innovation. This allowed other members of the community to see how the new innovation worked and the benefits that resulted, before adopting it in their own home. This component is in the initial stage of implementation in HCS, so it is too early to evaluate the effectiveness, however there was a great deal of enthusiasm around the demonstration households. Cooking demonstrations, using locally available foods, are also being planned. Recommendations: 1. The upcoming IEC training should focus not only on appropriateness of IEC materials but development of messages based on attitudes toward care practices in specific regions. It may be useful to develop a framework for each message based on input from CHAs and mothers. 2. Pilot a number of different strategies which are learner-centered and utilize problem solving approaches (nutritional counseling cards, mother to mother support groups, demonstration) to see what works best. 3. Network with other NGOs to see what health education materials and strategies are being used, so as to avoid duplication of efforts. 4. Develop strategies to target men, which take advantage of utilizing opportunities when men gather together in the community. 3.2.2 Data Management The community registers (both for pregnant women and children under two) are the source of all project information, apart from the KPC surveys. Therefore proper maintenance of the registers is critical to the validity and accuracy of project data. Currently, registers only include pregnant women and children receiving food rations. Registers need to be updated to include all pregnant women and children under two irrespective of their status in the food program. Registration should be continuous: every pregnant woman should be entered in the register as soon as she is identified, and her child transferred to the child register after delivery. If registers only include participants of the food program, it 24 May, 2001 DAP Final Evaluation Report 44 will not be possible to compare data in the registers to baseline (KPC) data, which is population based.18 The child and pregnant women’s registers seen by the Evaluation Team were for the most part incomplete. Progress reports received by CRS need to be more carefully analyzed to ensure data quality. For instance, it was observed that one partner is not reporting immunization coverage for three of its four catchment areas due to lack of vaccine availability. However rather than reporting these areas as zero, the three catchment areas were removed from the denominator, and reported only on the catchment areas that received immunizations, which results in an inflation of the actual figure. This was not an attempt to deceive, rather lack of understanding of the use of the indicator. Another partner is reporting on immunization figures for children less than 12 months and less than 24 months. This will make it impossible for CRS to consolidate the figures, as the correct indicator is 12-23 months (children less than 12 months may not be old enough to be fully immunized, and thus unnecessarily inflate the denominator). Information can be a powerful tool if understood and used correctly. For instance, in one village visited, EPI immunizations and tetanus toxoid had been discontinued due to short supply. Information in the registers can be used to advocate for action at the wereda and/or regional level. The community should be involved in deciding what information it wants to collect. If the community does not feel ownership of the information, and does not understand the value in collecting it, then it is not likely that community will continue to collect and monitor the information once the project phases out. This would be unfortunate as a great deal of energy and resources have been invested in the current MIS. The importance of collecting the information in the registers was not well understood at all levels (community, partner, CRS), and was not being used to make management/program decisions nor to prioritize monitoring visits. Information tends to flow up to the next level, but rarely flows back down in the form of feedback from supervisors. Likewise, information from the registers should be shared graphically with all mothers and the community so the community can use it as an advocacy and monitoring tool. For instance, growth monitoring has no value if it becomes simply an exercise in recording children’s weight. The value comes when communities use registers to track children’s growth, and to identify those who are faltering in order to arrest further deterioration through intensive counseling and care. Graphs can be displayed in 18 Participant data is perfectly acceptable when gathered correctly as a statistically significant representation of participants. The problem arises when participant data is compared to population based data. KPC baseline data is population based. To include both these numbers in a table reporting the same indicator over time is confusing and not recommended. The reason why many health indicators taken from the KPC were included as something to be reported annually is because the registers were intended to include all children under two in the community. In this case, they would represent the population and would serve as population￾based data while also being participant-based figures. Now that only those children receiving food are included in the register, the numbers are not representative of the population and should not be included in annual reports as population based numbers. 24 May, 2001 DAP Final Evaluation Report 45 the growth monitoring center. An example below depicts children 12-23 months who were fully immunized for the months of January to May: 0 10 20 30 40 50 Jan Feb Mar Apr May The animator can discuss with mothers why the percent decreased in March from February. If graphs are not well understood, the animator could use piles of stones to represent the number of children vaccinated, each pile representing a different month. The same could be done for the number of children underweight, or the number of women receiving full antenatal care. CRS and the community need to come up with creative ways that animators can share information with communities. The Evaluation Team observed that none of the registers were kept in the community, all were located either at the food distribution center or the health center and registers contained data from multiple villages. This limits the community’s access to the information and sense of ownership. A suggestion would be for the VHC to be responsible for keeping and maintaining village registers. CRS/ET reported that this is already happening in some areas, and this practice should be encouraged with all counterparts. Additionally, registers are too large and cumbersome. Animators and CHAs cannot easily transport them between health education and food distributions. Registers in the new DAP should be smaller and more portable, and each village should have its own register. Registers can be halved width wise by putting 12 months of GM records on one page and 12 months on the opposite open page. Registers can be quartered height wise to reflect potential program use for 2-5 years. This will reduce production costs in addition to becoming more usable for CBHC staff. Lastly, the community assessment forms were not being completed by any of the partners visited, as was recommended by the MTE. The form either needs to be adapted to make it more useful, or if it is not useful, the form needs to be discarded. It is important that only essential information be collected. In the new DAP, CRS should review its MIS with partners and communities to ensure that information collected is useful at all levels and reflects priorities. Recommendations: 1. LQAS (Lot Quality Assurance Survey) could be used to assess the completeness and accuracy of registers, since the Evaluation Team observed only a small number, which may not be representative of all communities/partners. 24 May, 2001 DAP Final Evaluation Report 46 2. Review MIS with partners and communities to ensure that information is useful and reflects priorities. Any forms not being used should either be adapted to be made more useful or discarded. 3. Training staff at all levels on uses of information for decision making and project management. Data should be used at all levels (animators, partners, CRS) to prioritize monitoring visits and make program/management decisions. 4. Reprint registers to make them more portable and easy to use by animators, and store them in a place that the community can access them. 3.2.3 Technical Support and Supervision CRS is not an operational organization, and therefore its role is to provide technical support to local counterparts to build their capacity to implement quality projects. The counterparts, in turn, pass on this technical support to the communities in which they are working. Training without supervision and support will have little impact on the capacity it was intended to build. Supervisory checklists have been developed for the FACS program for use by partners and CRS staff. These forms were not being used by CRS, or any of the partners visited, as recommended in the MTE. There may be a misunderstanding in the use of these checklists. There is a form for each FACS activity, such as growth monitoring, health education, home visits etc. The intention is not to use all of the forms on each monitoring visit. Rather, the checklists were to be used as modules. If training was recently conducted on health education, then focus the next monitoring visit on health education, using the appropriate checklist. This will reinforce recently acquired skills and help staff overcome difficulties in the utilization of new skills. Often times it seems that programmatic misunderstandings, such as the registering of pregnant women only in the food program (discussed above) or the proper use of forms, is a result of high turnover of staff and poor orientation of new staff. A thorough orientation of new staff is crucial to the proper implementation of activities according to the proposal design. It may be helpful if written guidelines for operationalizing the FACS program were given to all new staff. These guidelines should include (but not be limited to): · information on FACS health activities and principles of CBHC · use and maintenance of MIS data/impact indicators · supervision and reporting requirements · eligibility criteria for participation and ration size Even if a supervisor is using the checklists properly, supervision can be hampered by unrealistic supervisor to supervisee ratios. For instance, the HCS FACS coordinator is responsible for supervising 23 animators (who are geographically widespread) as well as all emergency health activities. While this 24 May, 2001 DAP Final Evaluation Report 47 is an extreme example, and CRS had already recommended hiring additional staff, it is important that CRS and partners maintain realistic supervision ratios in order to ensure that staff receive adequate support. Part and parcel of providing technical support is remaining abreast of new innovations and state-of-the-art programming. While it is recognized that there are many demands on the time of CRS health staff, it is crucial that they keep themselves and their partners aware of what is happening in their field from the perspective both of program quality and professional development. Community health is a rapidly changing field, particularly with the advent of HIV/AIDS. While accessing information on the Internet is not yet realistic in Ethiopia, there are a number of other options. In addition to the yearly health technical workshops and partner exchange visits CRS/ET is already doing, other possible suggestions include: participating in NGO nutrition forums, visiting successful projects both in Ethiopia (MICAH is doing vitamin A supplementation, PATH is social marketing ORS sachets, etc) and in the region (exchange visits to other CRS Title II programs in the East Africa region). There are a number of free publications from Linkages, CSTS (Child Survival Technical Support Group) and FANTA, whose purpose is to provide updates to program managers in the area of child survival on new studies, policies, and projects. The CRS East Africa Regional Office (EARO) is in the process of hiring a Health/HIV Regional Technical Advisor. Part of his/her job description should be to share lessons learned from other CRS programs in the region as well as pass along relevant information to country programs. CRS/ET should utilize this regional resource to its full potential. The IEC proposal mentioned establishing reference libraries on a pilot basis in order to provide partners with reference materials. Careful attention should be paid to the appropriateness of materials purchased for the library. In order to stay on top of what is happening in the public health field, current journals and newsletters may contain the most relevant information to partner CBHC staff. Recommendations: 1. Both CRS and partner staff should begin using the supervisory checklists on every monitoring visit, and adapt them if necessary. 2. Realistic supervisor to supervisee ratios should be maintained based on workload 3. CRS health staff need to keep themselves and their partners abreast of new innovations in the field of child survival, network with other NGOs working in Ethiopia, and visit successful CBHC projects in the region. 3.2.4 CBHC Targeting Upon visiting counterparts and conversations with CRS staff, it was stressed continuously that the impact of health programming depends on the implementation of other activities such as water and agriculture in target areas. To tell mothers that they should regularly bathe their child and use potable water in all cooking activities when there is no reliable source of water, is pointless. If crop production is low, there is no food to feed the children. 24 May, 2001 DAP Final Evaluation Report 48 It was also evident that working only in one village in a PA can create problems. In counterpart areas where the CBHC program was spread throughout the former center based target area including some, but not all villages, constant requests to be included from government officials and residents and even incidences of violence have resulted due to jealousies of not being included in the FACS/CBHC. Recommendations: 1. FACS/CBHC target areas should correspond to watershed management areas of the agriculture program. 2. Areas should be saturated rather than including only one or two villages in a PA. It is better to target all the villages in a PA than include one village in each of four or five Pas. This will make relations with PA government organizations simpler and impact will be greater. 3. In target areas, CBHC programs need to continue to target 100% of pregnant and lactating women and children under two. If possible, all women of reproductive age will be targeted for health education. If sessions can be designed at convenient times to include husbands, also, that would be effective for several subjects. 24 May, 2001 DAP Final Evaluation Report 49 3.2.5 Nutrient Content of CBHC Rations Based on a per person monthly ration Veg. Oil CSB Wheat Total % RDA average person 1 % RDA family of five % RDA <5s1 % RDA <2s1 % RDA Pregnant2 % RDA Lactating2 Quantity in 100g units 15 30 25 Energy (kcal) 13260 11271 8337.5 32868.5 53% 10% 85% 107% 48% 44% Protein (g) 0 516 295 811 59% 12% 106% 153% 45% 42% Lipids (g)3 1500 516 45 2061 147% 29% 239% 303% 136% 124% Calcium (mg) 0 24930 800 25730 191% 38% 214% 191% 71% 71% Iron (mg)4 0.3 524.7 107 632 96% 19% 234% 108% 70% 140% Thiamin (mg) 0 15.9 10 25.9 96% 19% 173% 216% 58% 54% Riboflavin (mg) 0 14.4 2.75 17.15 41% 8% 104% 88% 36% 32% Niacin (mg) 0 186.9 128 314.9 87% 17% 148% 198% 62% 52% Folate (mcg) 0 9000 975 9975 208% 42% 739% 899% 83% 119% Vitamin B-12 (mcg) 0 30 0 30 111% 22% 200% 364% 45% 38% Vitamin A (IU) 90000 78366 0 168366 340% 68% 436% 454% 211% 130% Vitamin D (IU) 0 5940 0 5940 130% 26% 50% 50% 50% 50% Iodine (mcg) 0 1707 0 1707 38% 8% 63% 71% 33% 28% 1RDAs from Annex 1, Management of Nutrition in Major Emergencies - WHO 2000 2RDAs from RDA Table, Annex IV Commodities Reference Guide, BHR/USAID 2000 3RDA calculated as 20% of energy as recommended by WHO/MSF and USAID, assuming children <2 are breastfeeding 4 Assuming low bio-availability 24 May, 2001 DAP Final Evaluation Report 50 The FACS/CBHC ration provides most of the energy, protein, fat and iron needed for a child under five. However, the ration provides less than half of calories and protein requirements for pregnant and lactating women. Micronutrient allowances are also low. The main issue is that the ration is more than likely shared between the whole household, at which time it provides only 10% of energy and 12% of protein needs. Analysis then must look at what are the goals of the program and what is the role of food in that program to determine how to make the ration more appropriate. There are three types of food rations used in MCHN programs as articulated by FANTA: · Recuperative: to rehabilitate malnourished children, determined by the household caloric deficit, appropriate for areas where food availability is a problem · Preventive: to prevent children in vulnerable households from becoming malnourished, determined by the household caloric deficit, appropriate for areas where food availability is a problem. · Incentive: to provide motivation for participation and compensation for time spent attending activities, based on the opportunity cost of attending activities In CRS documents the purpose of the FACS/CBHC ration is referred to as both an incentive and nutritional support. Although the size of ration does not differ by target recipient, the ration is intended to be both an incentive for mothers to come for health education, as supplementation to pregnant and lactating women and as a supplementation to children under 24 months. The ambiguity of the purpose of the ration is reflected in the ration size. Although calculations for an incentive ration size have not been made, it is assumed that the CRS FACS/CBHC ration is worth much more than the opportunity cost for attending activities. On the other hand, it is not a recuperative or preventive ration as it only meets 10% of an average household’s energy needs, under the estimated 20-30% daily caloric deficiency in Ethiopia. As it stands, the CRS FACS/CBHC ration is excessively large for an incentive and too small to address a household deficit. While an overly large incentive ration still serves as an incentive, it could be regarded as a waste of food. Meanwhile, a program using a ration that is less than the household food deficit will most likely not have the desired impact in lowering malnutrition rates and thus prove frustrating to both management and participants. A vague definition of the ration's purpose only leads to unattainable impacts and confusion. Before the next DAP, CRS is going to have to take a hard look at whether they want to deal with food for another five years in health programming. It will not be an easy decision as there are many arguments for and against the use of food and resistance will be met if ration sizes are increased. However, if food is used, the size and composition of the ration should be re-examined according to project goals and appropriate rations created. 24 May, 2001 Final Evaluation Report 51 FANTA recommendations for ration composition are fortified complementary foods for young children that are culturally acceptable and easily replaced with local varieties (Improving the Use of Food Rations in Title II Maternal/Child Health and Nutrition Programs, FANTA, 1999). The commodities included in the ration (wheat, CSB, and vegetable oil) do not all meet all these requirements. While CSB is fortified and appropriate for young children when made into a porridge, it is not easily replaceable, and wheat is not a particularly easy food to digest. It is also low in protein and iron. Recommendations: 1. The ration size should be adjusted according to the aim of the program. If it is determined that an incentive ration is more appropriate, research should be conducted to configure an appropriate ration size that meets economic opportunity costs of women participating in the program. Most likely this would mean a much smaller ration. If a preventive or recuperative ration is determined to be appropriate, the food deficit of households needs to be calculated and applied. As the average caloric consumption in Ethiopia is around 1700 kcal, thus leaving an average deficit of 400 kcal per person, a ration for an average household of five members would require 2000 kcal, a doubling of the present ration size. This would have to be determined by programming area where actual deficits may be much greater or less. 2. Alternative iron rich commodities that are more palatable and culturally appropriate such as lentils (which have twice the protein content per gram as wheat) should be considered. If one of the program targets is young children, wheat should be removed as a commodity as it is hard to digest. Lentils could be substituted for the wheat. 3. Food included in CRS provided food rations should be as fortified as possible with vitamin A, iron and other micronutrients. Vegetable oil should be fortified with vitamin A and investigation into fortification possibilities of other foods should be examined. Possible candidates for fortification include vitamin A, iron, iodine, B-complex vitamins, zinc, copper, and folate. Even if not all the energy needed by a pregnant woman per day is included in the ration, sufficient iron and other micronutrients should be provided to help meet normal diet deficiencies. 3.2.6 Food targeting If there were no negative repercussions from using food and local governments supported its use, current malnutrition levels in most areas in Ethiopia justify blanket feeding of all under twos, or even all under fives, plus pregnant and lactating women. However, there are many potential difficulties encountered using food in Ethiopia including: 1) creating dependency, 2) supporting negative behaviors, 3) lack of sustainability, 4) accusations of increasing birth rates, and 5) restrictions from government bodies. In addition the expected impact on child 24 May, 2001 Final Evaluation Report 52 malnutrition often does not result due to the severity of household food insecurity and resulting pressures to share or sell the food. While arguments abound for not working with food, however, there are times where food distribution is appropriate. It is questionable whether depending only on behavior change to nutritional status is wise for children who are already malnourished. The long term repercussions of malnutrition on intellectual capacity and lifetime health of children are significant enough that food interventions are justified in the short term until health education, water and agriculture programs provide sufficient improvements to stabilize nutritional status. In addition, improved practices may prevent nutritionally nourished children from becoming malnourished, but they may not be sufficient to rehabilitate malnourished children into a normal status. CRS must acknowledge the difficulties and concerns of working with food and work with partners to identify the best methods to target and distribute commodities. Due to the differences among regions regarding regulations and nutritional needs, CRS needs to be flexible in its FACS/CBHC package as the current blanket approach is not suitable for all areas. How food is targeted in FACS/CBHC programs should be decided at the counterpart level with input from local government (with CRS assistance) considering causes of malnutrition, local problems with food distribution, counterpart experience with food and government regulations. CRS can play an advocacy role using data of local nutritional status and examples of where food has worked without creating dependency or other problems. Before deciding targeting methods to distribute food in CBHC/FACS, assessments should be conducted to determine why children are malnourished and how much food availability factors into nutritional status in each area. If food availability is not a problem in the community and causes of malnutrition are related to feeding practices, no food or limited targeted food would be most appropriate. If food is not readily available in a community and despite the best care practices children will be malnourished, then more food is appropriate. Possible options for food targeting include: · Blanket distribution to all under 2s and all pregnant/lactating women. In areas where causes of malnutrition are greatly influenced by food availability, a preventive ration accompanies health education. Open to all children under two and pregnant/lactating women during the project period for multiple enrollments per household. Addresses food availability cause of malnutrition in addition to sub-optimal behaviors. · Blanket distribution to all under 2s and all pregnant/lactating women for one round of health education. In areas where causes of malnutrition are care practice based use an incentive ration. Ration may stop after one round of health education. Goal is to change sub-optimal child care practices to improve child nutritional status. One ration per household either for the mother or for the child used. 24 May, 2001 Final Evaluation Report 53 · Targeted distribution to malnourished children under 2 and pregnant/lactating women. A recuperative/preventive ration is used to add food to the household diet so malnourished children are rehabilitated and pregnant/lactating women have sufficient increased energy to meet metabolic requirements resulting in positive birth outcomes. Pregnant and lactating women should get rations from the second trimester to six months after birth. Malnourished children should receive rations until over 85% median weight for age or above –2 z-score for two or three weighings (in the green section of the growth monitoring card). Any targeting of malnourished children must be accompanied by intensive education to mothers to help address sub-optimal care practices. Depending on occurrences of purposeful starvation of children to obtain rations, the exit criteria for children can be modified. Possible adjustments include setting maximum amounts of time a child can be in the program (e.g. six months) and only permitting entrance one time in a child’s life. Another possible modification is distribution until the child has been above 85% weight for age for two months with a maximum time of 8-12 months with removal of the child from the program if there is not weight gain each month without a reason (such as illness). · Targeted distribution to just malnourished children. Same requirements as above just no distribution to pregnant or lactating women. · No food for anyone, only health education. The CRS ration is given to women from pregnancy until her child is 24 months old and to children from 4-24 months. Therefore, households receive two rations if they have a child between 4-24 months. As the health education is intended to convey information on improved care practices that will assumedly lead to better nutritional status, a household can only participate in the program for one “round” (a pregnancy, six months of lactation and the time from when a child is 4-24 months old). Depending on when the pregnancy is confirmed, a household may receive rations for 25-30 months. It is unclear why the child is allocated a ration if the ration is meant to be an incentive for the mother to attend health education sessions. Likewise, if the purpose of the ration is supplementation for the child, it is unclear why the mother is allocated a ration as it would be assumed that a supplementation for the child would be sufficient incentive to participate in the program. Either way, there does not seem to be a rationale for distributing two rations per mother– child dyad for the time period when the child is 4-24 months old. The issue of when to include pregnant and lactating women is contentious. On one hand some people may say that supplementation to pregnant women encourages birth rates (which is doubtful). Besides education to prepare for their new baby, the purpose of including pregnant and lactating women in a food assisted program is to supplement the normal diet to compensate for the 500- 700 extra calories needed per day. Extra food will decrease the incidence of low birth weight babies and limit the wasting of mothers while lactating. While education provided to pregnant mothers will encourage appropriate infant and 24 May, 2001 Final Evaluation Report 54 young child feeding practices at birth, an opportunity to have significant impact on births is missed by not providing food supplementation to all pregnant and lactating women irregardless of their previous program experience. As the MOH recommendation in Ethiopia is to exclusively breastfeed until 6 months, which is also the message sent by WHO, UNICEF and most international bodies, CRS should not send any mixed messages by giving rations at 4 months. FACS/CBHC should provide rations for children beginning at 6 months and supplement the mother only with food until the child is 6 months old. The message should be continuously stressed in food programming that breastfeeding is always best and should continue along with complementary feeding as long as possible, certainly until the child is over two years old. It is important to reiterate that no matter what food targeting mechanisms are applied health education and integrated programs need to continue in all FACS/CBHC target areas. Recommendations: 1. CRS should be flexible in its FACS/CBHC package, allowing for regional differences in nutritional needs and government regulations. 2. Before deciding how to use food in CBHC/FACS, assessments should be conducted to determine why children are malnourished and how much food availability factors into nutritional status. 3. Food targeting options should be chosen according to the causes of malnutrition in the target area, counterpart preferences and government regulations. 4. If possible, pregnant and lactating women should receive preventive rations. 5. Only one ration should be given at any time to a household, i.e. either the mother gets it or the child. 6. Rations for children should start at six, not four months to avoid sending mixed messages to breastfeeding mothers. 3.2.7 Micronutrient deficiencies and supplementation PROFILES analysis shows that Vitamin A deficiency in the country is accountable for 17% of child deaths and that 800,000 newborns a year will suffer from intellectual disability from iodine deficiency. While CRS is promoting consumption of more balanced diets and production of vitamin rich foods, the provision of sufficient quantities of vitamin A and iodine rich foods may be difficult to obtain, especially in the short term (Nutrition in Ethiopia. Time to Act: A PROFILES Analysis of the Nutrition Situation in Ethiopia and its implications for the Human and Economic Development of the Country). Recommended dietary allowances of micronutrients increase greatly for pregnant and lactating women. Iron requirements, for example, double from 15mg a day in women of reproductive age to 30mg a day in pregnant women. In a population 24 May, 2001 Final Evaluation Report 55 that is normally micronutrient deficient, it is likely that pregnant and lactating women are particularly deficient. Micronutrient deficiencies impair women’s health, pregnancy outcomes, growth potential, development and health of breastfed infants (LINKAGES The Case for Promoting Multiple Vitamin/Mineral Supplements for Women of Reproductive Age in Developing Countries). Studies suggest that micronutrient supplementation improves birth weight, reduces prematurity, fetal death, premature rupture of membranes and pre-eclampsia. Studies in Nepal showed decrease in maternal mortality by 50% by women who received vitamin A for at least three months before and during pregnancy. Folate reduces neural tube defects, found to be a cause of up to 10% of infant mortality in some countries. Micronutrient supplementation of pregnant and lactating women also improves the quality of breastmilk and infant micronutrient status. Interactions between micronutrients such as Vitamin A is needed for iron transport, provide arguments for multiple micronutrient supplementation. Priorities for micronutrient supplementation would be: · Pregnant women · Breastfeeding women and young children · Adolescents and women of reproductive age recently married · Adolescents and women of reproductive age Recommendations: 1. CRS needs to assure that every pregnant mother and child under two is not micronutirent deficient. Potential programs to accomplish this include: 2. Health education to help mothers identify micronutrient rich foods and eat the most balanced diet as possible to maintain adequate nutritional stores while pregnant and in young children. 3. Work with the MOH to make sure that within 6 weeks of giving birth every mother receives a single dose of 200,000 IUs vitamin A and every child receives a vitamin A capsule every six months 4. As iodine was identified by PROFILES as a prominent concern in Ethiopia, and due to cutoff of supplies from Eritrea, CRS should assess the use of iodized salt in target areas and if low usage, work with ICCIDD or similar agencies to educate households and bring in iodization equipment. Kiwanis International has taken on iodine deficiency as an international campaign and provides funds for eradication. 5. Advocate for iodinization of local salt or if not possible distribution of iodine oil capsules to pregnant and lactating women and children 6. Integrate agriculture programs with CBHC to promote micronutrient rich foods 24 May, 2001 Final Evaluation Report 56 7. Obtain fortified commodities for food distribution (for all food programming, including FFW and emergencies) 8. Work with agencies that specialize in micronutrient supplementation such as Helen Keller International, LINKAGES, ICCIDD, MICAH, PAMM, and IVACG to obtain information on best practices in reduction of micronutrient deficiencies and serve as networks for collaboration. 3.2.8 Sustainability and Exit Strategies In the DAP, CRS/ET defines sustainability through the increased capacity of the partner to carry out activities after phase out and the increased capacity of communities to implement and carry out integrated activities of Ag/NRM with FACS and microfinance. Sustainability of the FACS program is defined through the movement to community based programming and ensured by beneficiary contributions, which should continue to pay for CHAs and to finance health services. Linkages with MOH for the provision of immunizations and clinical services are also part of the sustainability plan. There are several concerns for the sustainability of the FACS activities. First is the training of CHAs. In some areas, the training of CHAs has not taken place, and it is not clear when it will take place and how the government will support the salaries of the newly trained CHAs. In HCS, the government is no longer training CHAs, but Community Reproductive Health Agents. It is unclear whether this is simply a change in title or a significant shift in curriculum. Even if the CHAs do get trained in the next year, there is no guarantee they will carry out the same activities as the animators. Since they are government employees, it will be difficult to hold them accountable to the activities outlined in the DAP. As suggested in the MTE, CRS will need to consider alternatives to CHAs in areas where training is not taking place. It is not clear what will be the role of the VHC once project activities are phased out. They could be trained to take on some of the responsibilities of the animators. CRS has trained many animators and CHWs due to the uncertainty of the government training. However the stipend of the animators is paid out of contributions from the food program. It is questionable as to whether mothers will continue to pay for animators/CHA services once food is withdrawn. Unlike water projects, where participants pay a fee and in return receive a tangible product such as water, it is much more difficult to convince participants to pay for preventive services, like growth monitoring and health education, which are less tangible. Alternatives to remuneration will need to be sought. The transition from center based programming to community based programming has begun, however is not yet complete. Due to the expense and logistics of moving food, most areas have shifted to satellite centers, where several villages come to receive food. There is concern that mothers are still spending significant amounts of time on project activities. In Harare, for instance, mothers spend one day in health education, one day on growth monitoring, and one day to receive food, and for most women, they have to walk to 3 different sites. While it is 24 May, 2001 Final Evaluation Report 57 recognized that it may not be possible to move the food to every community, efforts should continue to be made to move the health education and growth monitoring to every village. The transition from center to community based is not just a question of physically moving food and health education closer to the community. CBHC is also about a shift in the conceptual framework of the program, from mothers being simply recipients of food and Facts For Life health messages, to mothers being actively involved in monitoring the health status of the children in their community. CBHC is about mothers identifying health problems within their communities and working together to find solutions. As with the physical transition from center to community, the conceptual shift is not yet complete. The upcoming IEC training will be one opportunity to further the shift. It is clear that different partners are at different stages of project cycle, and are implementing activities in very different operating environments. In some areas CHAs have been successfully trained and integrated into the CBHC program, in other areas training has not yet begun and activities have slowed. In some areas visited, basic elements of the program, such as filling in the registers, was not being correctly implemented. CRS and partners need to decide with communities which elements of the project they would like to see continue, and decide what can be done in the last year of the DAP to move in that direction. Therefore phase out plans will need to be carefully considered in each project area according to the circumstances. In summary, sustainability of the project will come through improving the capacity of CHWs to deliver quality services to the community, such as health education. If the community sees value in the health education they will continue to participate in the sessions. Currently, this is not happening in the projects visited. Participant numbers are dropping, and women who have graduated from the food program are not returning to participate in the health education or growth monitoring. The IEC proposal was designed to address the issues of the quality and appropriateness of the health sessions, which will hopefully result in better services that the community will buy into. Barry Riley’s paper gave some examples of case studies, which he considered successful. There is no recipe for sustainability, and it varies according to circumstance such as government support, charismatic/dynamic community members, availability of medicines/vaccines, etc. This is true for CRS/ET counterpart projects: in some areas there is strong government support, in other areas CHWs have not been trained at all. In some areas there are restrictions on how food can be used, in others there is a lot of flexibility. CRS will need to assess each counterpart separately and decide what is the most appropriate strategy. Recommendation: 1. Site specific phase out plans will need to be considered for each implementing partner 24 May, 2001 Final Evaluation Report 58 2. CRS will need to work with counterparts and communities to come up with incentives for CHWs after the phase-out of the food program.19 3.3 Water and Sanitation CRS/ET has developed a potable water and sanitation strategy, which is endorsed by the Evaluation Team. It has a conceptually strong framework, however the team would like to offer a few comments for consideration when operationalizing the strategy. As with any guidelines, CRS will need to provide a great deal of support to partners to ensure that guidelines are incorporated into project design. CRS may want to consider expanding its strategy beyond potable water, to include water for multiple uses. Potable water implies adhering to certain standards of water quality, which has high cost implications. Potability at the water point does not guarantee that it will be potable at the point of consumption, as distance provides opportunities for contamination. Additionally water is used for multiple purposes at the household level, such as personal hygiene, washing clothes, household sanitation etc, which do not necessitate potable water quality. It would be useful to be able to weigh the cost of potable water interventions against the number of people benefiting from the intervention. The Evaluation Team observed examples of communities maximizing the benefits from capped springs in HCS, by constructing wash basins nearby and channeling run off to vegetable gardens. Given the extensive needs assessments outlined in the strategy, it should be possible to estimate the cost per beneficiaries served, which could justify the cost of potable water structures. There is some question as to whether or not the water committees are the best form of management for the water and sanitation strategy. The strategy talks of reactivating old water committees. CRS should look into why these committees need to be revitalized and if there are traditional systems in place that might be disrupted by the introduction of new structures. The strategy provides detailed guidelines on how the committees are to be formed and how they should function, which appears quite top-down. The strategy also suggested that two women should sit on each committee. While representation of women is important on the committees, since they have the most to benefit by reducing distance to water points, it should be noted that sitting on the committee will not guarantee participation. An enabling environment needs to be created so that women are able to go beyond a mere representation role to one of active participation. Allowing flexibility in committee design may improve the sense of community ownership and thus sustainability of the committees. Low level 19 The health unit needs to carefully think about the issue of incentives. Incentives do not necessarily imply cash, they can also be in-kind. It is not realistic to expect volunteers to carry out all the project activities without any compensation. The community is best placed to decide what kinds of incentives are most appropriate. Some examples might be to have the community work on the land of volunteers to compensate them for time spent doing health education and home visits. Another example might be to provide CHWs with a shirt or cap that would identify them as a CHW upon successful completion of their training. 24 May, 2001 Final Evaluation Report 59 monitoring should continue after phase out in order to evaluate the performance of the committees after project phase-out. An integral part of the water strategy is the linkage with sanitation and health, which has been very weak in the current DAP. The draft strategy states the three main components of the sanitation component involve protection of water points, containment of human waste and improved personal hygiene. All three will be addressed through health education, although the first two will require investment in infrastructure. CRS should be careful not to assume that knowledge is lacking. As mentioned above under health education, the project needs to understand why a community has poor hygiene practices; there may be a very good reason for “sticking to traditions, cultures and languages.” This is a key principle of WHO’s PHAST (Participatory Hygiene and Sanitation Transformation) methodology. It takes the community through a process of identifying problems and then finding solutions. Only after prioritizing sanitation, are facilities introduced. This methodology or similar problem-solving approaches should be used when developing IEC messages. The sanitation strategy shifts from the expensive VIP latrine design to a much cheaper design produced with locally available materials. The new design is much more cost-effective, which allows it to be built on a self-help basis. Communities seemed genuinely enthusiastic about the new design when observed in the field. While the new design is much cheaper and easier to construct, education will still need to be a key component in order to ensure they are properly and consistently used in the community. Recommendations: 1. Explore expanding focus of the water strategy beyond potable water to include multiple uses of water. 2. Continue low level monitoring of water committees after phase out in order to evaluate sustainability. 3. Hygiene and sanitation education should include learner-centered and problem solving approached to understand the community’s beliefs and attitudes towards hygiene practices. 24 May, 2001 Final Evaluation Report 60 3.4 Use of Food Aid Allocation of Food CRS/ET uses food in four ways: · Monetization · Food for Work · FACS/CBHC (as an incentive and to some extent as a supplement/rehabilitation tool) · Safety Net When monetization is excluded, use of food in the safety net program encompasses 70% of Title II food being distributed in FY02. On average, Safety Net clients receive over twice the annual food allocation (.193 tons per person) compared to participants in agriculture/NRM (.097 tons per person) or FACS/CBHC (.084 tons). This is not surprising as safety net clients are often dependent on Title II food for the majority of their daily intake. Allocation of Title II Food FY02 FACS/CBHC 10% Safety Net 47% Ag/NRM Monetization 10% 33% Metric Tons of Direct Food Distribution versus Monetization FY97-01 0 2,000 4,000 6,000 8,000 10,000 12,000 14,000 16,000 18,000 FY 1997 FY 1998 FY 1999 FY 2000 FY 2001 FY 2002 MT of Title II Food MT Direct Distribution MT Monetization 24 May, 2001 Final Evaluation Report 61 The amount of food directly distributed in the DAP program was highest in the first year of the DAP, FY97, and hit a low in FY00, moving back up to FY98 and FY99 levels for FY01. Monetization requests steadily increased until FY02, but this is confounded by the increase of wheat for monetization. There is no decrease in direct distribution for the final year. Food for Work The main mechanism used in the agriculture sector to distribute food is food for work. In recent research on the impact of Title II FFW projects in Ethiopia, it was found that FFW does not have a demonstrable impact on nutritional status of children under five in the target area. This implies that if programs are going to be evaluated according to nutritional status improvements of children under five, FFW may not be the appropriate activity to achieve these goals. FFW can have an impact on household food security status by creating beneficial assets. This is reiterated in multiple studies that state that FFW projects are only successful if activities are adequately technically and supported with sufficient materials. “Make work” schemes create few or no assets that do little to change the food security status of targeted households. FFW rations should consist of commodities worth slightly less than the daily wage on the market to serve as a self-selection activity that does not upset market economies. However, with the paucity of employment opportunities in Ethiopia, ration sizes would need to be reduced to minimal amounts that would have no chance of improving household food security. In order to have an impact on household food security, household food deficit amounts would have to be calculated and ration sizes adjusted accordingly. However, FFW rations in Ethiopia are standardised by the DPPC, with little flexibility in changing commodities or energy content, so the specific food security needs of a community cannot be addressed through altered FFW rations. While FFW may not have an impact on the nutritional status of under fives, it does serve as a household income source. The Cooperating Sponsor food security baseline study of 1997 showed that the average household participating in CS FFW projects participated for over 40 days of rations during the year, or eight weeks of work. This provides supplementary food for one-sixth of the year. What this implies is that if an area is suddenly cut off from FFW rations without provision of alternative food resources, the impact could be quite negative. Recommendations: 1. CRS/ET should look at why in the final year of the DAP project implementation levels involving direct food distribution are slightly greater than the previous year. Especially for CBHC, the number of eligible beneficiaries in each target area should be decreasing substantially as each woman/child pair is only eligible for one “round” of food. The impact on communities should be examined. 24 May, 2001 Final Evaluation Report 62 2. FFW phase out plans should be reviewed with counterparts to ensure that phase out is being done in a manner that will not result in negative impacts on the food security situation. 3. Program level impact indicators should be reviewed with the Mission to verify that FFW activities are not expected to have an impact on the nutritional status of children under five. 24 May, 2001 Final Evaluation Report 63 3.5 Monitoring and Evaluation Introduction It is clear that CRS/ET and the implementing partners that were visited during the course of the evaluation produce much DAP activity-related information in a range of reporting formats. For example, the Evaluation Team was shown strategy papers, annual results reports, baseline data, Mid-Term Evaluation Report, Workshop Reports, a case study report, other related studies, and ‘Community Impact Briefs’, and various items of ‘grey literature’. This work is commendable. Equally commendable is CRS/ET’s strategy to ensure that all technical staff perceive Monitoring and Evaluation activities as their own responsibility within their subject area. In this case, the M&E Unit (staffed by one person) is seen as providing a technical advisory service to the technical sections. There is little doubt that CRS/ET has complicated information requirements given the wide range of programs that it administers, and stakeholders who have an interest in its activities. It has not been possible to do anything more than a cursory review of the existing Management Information System (MIS) for DAP activities in the time available. Nonetheless, in general the DAP data collection reporting system is well understood by CRS/ET and counterpart staff, and it appears to function reasonably well through well-established reporting systems. Following the compilation of the annual results surveys, some feedback is provided to the implementing partners by the CRS/ET M&E office and comments sought (although see the weaknesses of the current M&E system reported by CRS/ET staff below). Use of information As to the question ‘why’ the data are being collected and reported, there seems to be less ground for satisfaction, although only a few partners were visited and their use of M&E information varied enormously. Observations obtained during visits to the three implementing partners suggested in some instances there was evidence of: · superficial problem analysis; · planning processes that were not comprehensive; · a perception that monitoring reports were something that were completed for an external higher-level funding authority; · limited in-depth studies, although the Lege Oda Mirga case study is clearly an exception (and, to be fair, another was planned);20 · concerns have been expressed elsewhere in this report about the quality of some of the reported health information. 20 See the recommendation in section 3.1 above suggesting an independent verification of the results reported in the Lege Oda Mirga case study. 24 May, 2001 Final Evaluation Report 64 In the draft Guidelines for M&E currently in preparation the following weaknesses of the current CRS/ET system have been identified:21 · “ … inadequate skills for data analysis and management at counterpart level: counterparts often lack data analysis skills so collected data ends up unanalyzed and unused; · lack of feedback: the flow of CRS/ET’s information system is one way – bottom-up. Feedback is rarely given on submitted reports. When feedback is given, it is usually in response to specific critical issues, and dependent on an individual rather than the system. The information system would be more effective if there was a systematic feedback mechanism on the quality of reporting, lessons learnt and project performance; · emphasis on quantitative data: as it is easier to compare and summarize quantitative rather than qualitative information. However, over￾emphasizing quantitative information means little information is gathered about the qualitative effects and impacts on CRS projects on people’s lives; · donor driven: many staff tend to see M&E as something that is necessary to please donors, rather than as important for the project and their own work. Donor schedules and demands can mean that M&E work may be rushed and not undertaken carefully” (CRS/ET, 2000e). It is timely for CRS/ET and its counterpart staff to tackle some of the concerns raised above as to how information can best be used during the next DAP, and what implications that has for current DAP planning activities. Guidance and recommendations on Monitoring and Evaluation Utilization-focused MIS CRS/ET is preparing Guidelines of M&E following the June 2000 Program Quality Retreat. The draft Guidelines seen by the Evaluation Team look promising, and once completed will be widely circulated among CRS/ET and implementing partner staff. To improve the likelihood that information us used requires a user-led basis (or “utilization-focus” (Patton, 1997)) for all CRS/ET and implementing partner staff MIS design work. Patton’s statement that an “evaluator’s primary function is to maximize intended uses by intended users of evaluation data” should be interpreted more generally to CRS/ET’s M&E strategy. 21 The list is abridged from the original (CRS, 2000e). 24 May, 2001 Final Evaluation Report 65 Recommendation: 1. CRS/ET and implementing partner staff should be introduced to the concept of utilization-focused monitoring and evaluation through a series of in-house training workshops prior to and during the next DAP. Undertaking an information audit Given the concerns raised earlier by the Evaluation Team and participants at the June 2000 Retreat, it may be appropriate for CRS/ET to consider, on a pilot basis, ‘mapping out’ current practice and then considering whether it does or does not make the best use of information collected for each organization. This would be a good way for CRS/ET and its counterpart staff to develop their thinking about these issues. This is known as an information audit (e.g. see Powell, 1999) and should be managed by the M&E unit. An information audit is equivalent to a case study, but specifically aimed at tracking information through the system. Data or information can be followed through the implementing partners and CRS/ET so that specific questions of value and benefit may emerge: are all the data being collected actually used? Are there data that would be useful for management purposes that are not being collected? Is the output of M&E actually being used by project officers as they plan new activities? Recommendation: CRS/ET and its implementing partners should undertake a pilot information audit during the planning period for the next DAP to determine if current information use is optimal, and to identify opportunities for quality improvements prior to the commencement of the next DAP. This should be piloted with one implementing partner to determine if the audit is perceived to be useful for improving program quality. Using external partners in M&E work Implementing partner staff are required to work very closely with rural communities in order to establish the necessary levels of trust and confidence for subsequent DAP activity. With the best will in the world this makes it very difficult for those same staff to then be required to operate ‘objectively’ in the case of, say, collecting program impact data, either quantitative or qualitative. CRS/ET M&E unit recognize this issue and in the Lege Oda Mirga Case Study brought in HCS staff from other sites. Another reason for considering the use of external partners arises from the comments made at the June 2000 Retreat that M&E work often has to be rushed because of other competing demands on staff time. Providing external partners do exist, freeing up staff time will allow them to undertake other duties. Local universities (staff and students) could usefully be approached to see if there is interest in undertaking this kind of work. HCS already has links with Alemaya University. There is an issue of quality control and cost, so CRS/ET and its implementing partners would need to proceed cautiously, but where 24 May, 2001 Final Evaluation Report 66 appropriate, and with appropriate safeguards in place, such an association might add independent weight to the impact results reporting of DAP activities. Recommendations: 1. CRS/ET and its implementing partners should approach local higher education institutions with a view to identifying a possible role for staff/students in undertaking DAP M&E data collection and analysis activities. 2. Such a partnership could be tested on a pilot basis initially to determine its value for improving the quality of M&E data. Requiring evidence of the use of information CRS/ET is keen to improve its programming quality. One way to do this is to encourage the use of secondary information in project submissions. CRS/ET could start to encourage the use of information from secondary sources by requiring that implementing partners submitting a proposal for new activity must list the material which they have consulted about similar work carried out in the past. This has three key advantages: · offers whoever is evaluating the proposal some idea of how thoroughly it has been considered; · implementing partners are obliged to make links with ‘external’ organizations; · offers CRS/ET during appraisal a clear idea of the knowledge-base upon which the proposal was based. There is a role, though potentially time-consuming, here for CRS/ET and the East Africa Regional Office to supply up-to-date information to implementing partners, and their role in this process is something that these offices need to consider. Recommendations: 1. CRS/ET should encourage/require implementing partners to source background material prior to submitting DAP-related proposals. This could be reference materials, if available, or evidence of discussions with local stakeholder organizations. 2. CRS/ET and the CRS East Africa Regional Office need to determine their respective roles in accessing and providing relevant information to in￾house and counterpart staff. M&E networking Although the Evaluation Team has some concerns about the validity of the Lege Oda Mirga case study findings, the benefits of circulating findings such as those reported, with appropriate caveats, is likely to outweigh the costs, in terms of lost opportunities to develop inter-NGO M&E networks, of not doing so. It is important that CRS/ET and its implementing partners seek to improve networking among 24 May, 2001 Final Evaluation Report 67 NGOs in Ethiopia in order to keep up to date about evaluation methods and approaches. To date, M&E discussion between the Cooperating Sponsors seems to have been driven very much by the requirements of the FFP Office. It would be beneficial to use this already-established network to promote discussion, analysis and support on wider M&E issues. Additionally, the African Evaluation Association is currently headquartered in Nairobi. It might be beneficial if efforts could be made to enquire from the African Evaluation Association if an Ethiopian Evaluation Association exists and, if so, for CRS/ET’s M&E officer to further his professional development by establishing contact with both organizations. Recommendations: 1. CRS/ET should seek to initiate and develop an inter-NGO network for strengthening the quality of M&E work undertaken. One way to start this process is to circulate reports that discuss the methodologies and findings of surveys undertaken by CRS/ET and implementing partner staff. 2. CRS/ET should, through its Regional Office, enquire about the existence of an Ethiopian Evaluation Association from the office of the African Evaluation Association. For professional development purposes, CRS/ET’s M&E officer should be encouraged to join. 24 May, 2001 Final Evaluation Report 68 3.6 Expanding Partnerships with ‘External’ Partners Introduction The Evaluation Team believe that CRS/ET should seek to expand its working contacts with ‘external’ partners. This will, if successful, be an important feature of the next DAP. There are many different types of partners – government, Cooperating Sponsors, university and other research-based organizations, and so on. This section briefly looks at the current status of CRS/ET and its implementing partners’ relations with external partners, with a view to developing a recommendation for developing this aspect of DAP quality programming and support. Current status Counterparts believe that they have good working relationships with government partners. They plan together, monitor activities, share technical expertise, and so on. In some areas (e.g. HCS), however, the relationship that exists is influenced by external circumstances; for example, when there is a food shortage in an area, food programs are endorsed by government partners; when there is less shortage of food, government officials tend to prefer other programs. Clearly, flexibility to change program direction is essential for maintaining good relations. There exists some level of collaboration with research and academic institutions, e.g. HCS’s link with Alemaya University (in training and in programming veterinary services, seed multiplication/adoption trials); CRS with Jimma University (in FACS/CBHC training programs), and the Ethiopian Agriculture Research Organization (in developing a watershed program). Further development of these partnerships would be beneficial to CRS/ET and implementing partner programs, and opportunities for specific opportunities should be sought under the next DAP (e.g. see the recommendations listed under section 3.1 ‘Agriculture / Natural Resource Management). The level of effort made to expand relationships with related institutions needs to be improved in ADCS, e.g. ADCS staff are keen to undertake a range of programs, including SSI development, and should consider working more closely with REST, a large NGO based in Mekele. The Regional Commission for Sustainable Agriculture and Environmental Rehabilitation is another potential partner for ADCS’s agricultural and environmental activities. CRS/ET has a role to play in helping ADCS develop such linkages. Since CRS/ET and its implementing partners have only limited capacity in certain areas, the gap should be filled by expanding partnerships (e.g. via contracting) with institutions specialized in specific subject areas. At present, however, there are no adequate guidelines to steer implementing partners in this regard: when and how should such commercial partnerships be established? Should such arrangements be organized through CRS/ET, or directly by the implementing partners with the agency in question? 24 May, 2001 Final Evaluation Report 69 Some partnerships may start with personal relationships established between the staff of CRS/ET, implementing partners and external institutions. Such partnerships do not last long, or tend to end by the time staff from either party leave the organizations. Can such partnerships be institutionalized? Recommendation: CRS/ET should organize a 1-day in-house workshop to address the issue of expanding partner relations under the next DAP. The agenda items would include such topics as identifying: · the range and type of partners in Ethiopia that would be of direct interest to CRS/ET; · the advantages/disadvantages of working with partners, and the need to manage partnerships; · the ‘models’ of collaboration between CRS/ET and/or implementing partners that would be feasible under the next DAP; · the availability of any policy guidance on this matter from CRS’s East Africa Regional Office; · a role that CRS/ET would play, if any, in working with counterpart staff seeking to develop external partnerships; · the need or otherwise for developing CRS/ET guidelines on this issue for circulation to all staff and implementing partners; · the need to host a similar workshop for all implementing partners once CRS/ET has developed guidelines regarding partnerships. 24 May, 2001 Final Evaluation Report 70 3.7 Safety Net Currently, CRS is programming nearly 10,000 MT of food per year for over 50,000 safety net beneficiaries. Safety net issues cannot be ignored as it will compose at least 42% of the projected entire CRS/ET portfolio 2001-2005 and 30% of cash and 60% of direct food distribution of the DAP II. Safety Net programming is not an area CRS has concentrated on during the past DAP. Strategic objectives for safety net portion of portfolio elaborated in the SPP are minimal, only including one programming specific action recommendation. A symposium to look at Institutional Development for selected Safety Net Institutions was held in 1998 that successfully brought private business and safety net organizations together to build awareness and examine financial support opportunities. Although viewed as having great potential little impact seems to have resulted due to lack of follow-up. Since the special project in 1997-98, little has been done to look at transition options for institutions or program improvement options. As the Safety Net portfolio consists of such a large percentage of CRS direct food distribution and monetization funds, more critical analysis should be paid to how these institutions function and how they can be improved. CRS believes in a rights based approach where it is not sufficient to be satisfied that any services provided to the most vulnerable is good enough. Services should be appreciated and lauded and then examined to see how the situation can be improved. The Safety Net Program is an avenue that CRS can use to help address the increasingly devastating effect of HIV/AIDS in Ethiopia. Regular Safety Net institutions are already seeing a rapid increase of HIV/AIDS related clients. Food helps booster the compromised immune system of people with HIV/AIDS and helps improve the quality of life and the amount of time an infected person is economically active. Food to families living with HIV/AIDS provides compensation for the loss or debilitation of an economically active member of the household. Food also helps prevent harmful coping mechanisms such as asset sales, reduction in meals provided to children and other family members, and resorting to income generating activities such as prostitution and crime to meet the economic needs of the household including high care costs for the positive family member(s). The safety net rations are composed of: Safety Net Program General Relief/STEP Other Child Feeding Missionaries of Charity Commodity Dry Wet Wet Dry Wheat 10kg 4kg 6kg 0 CSB 2kg 2kg 0 2kg Oil 1kg 1kg 1.5kg 1.5kg Bulgur wheat 0 0 12kg 12kg 24 May, 2001 Final Evaluation Report 71 Rice 0 0 2kg 2kg Kcal 1648 988 2760 2353 Protein (g) 53 12 98.47 61.27 Fat (g) 43.33 39.73 59.87 60.27 Vitamin A (IU) 2513 2513 5557.5 7299 Iron (mg) 22.99 16.19 24.93 29.26 These rations vary greatly in energy and nutrient content. OCF and GR/STEP rations provide around 50% (children 4-10) and between 57-72% (adult male/female) of the required kcal daily respectively. On the other hand MOC dry rations provide 81% of kcal needs for adult men and 107% of needs for adult women. MOC wet rations similarly provide 95% and 125% of energy needs. For the children in the MOC programs, well over 100% of energy needs are met per ration. Except for OCF rations, protein content of rations is sufficient. Iron provisions in all the rations are substantial, as are the vitamin A provisions in the MOC rations. As MOC is supplementing rations with other foods, it is unclear why so much food is provided per beneficiary. We recommend that CRS/ET consider recommendations 1-4 below as of a higher priority than those thereafter; recommendations 5-10 should be viewed as options to consider if time and funds permit. Recommendations: 1. The rations of safety net programs should be examined in relation to program goals and additional supplementation. If GR/STEP rations serve an incentive purpose, then they should be adjusted to reflect opportunity costs. Likewise, the percentage of the diet OCF rations are intended to provide for clients should be researched. If the OCF ration is the only food source provided to clients each day, then the ration should be increased. The MOC ration should be examined more closely with the MOC sisters to see how exactly the food is being distributed. If more people are actually consuming the food than recorded, then the number of beneficiaries should be adjusted to reflect the true number and an appropriate ration given for the larger number of people. 2. As outlined in the SPP, safety net institutions should be encouraged or mandated to develop a strategic vision with a corresponding five-year plan. Other goals and objectives to be explored could include: expansion of the donor base, institutional sustainability, improved management capacity, improved fundraising strategy/materials, detailed targeting criteria, higher quality of services provided to clients, and plans to ensure continued support of operations from government authorities. 24 May, 2001 Final Evaluation Report 72 3. Funding should be pursued to fill in gaps not met by DAP resources especially transport costs for food. Support from the domestic constituency in the US could be accessed by sponsorship programs from US parishes. 4. Agencies that support PLWHA would be recommended for any Safety Net program expansion. Options to consider if resources available: 5. Criteria for continued support to safety net institutions need to be established. Criteria should include standards in financial management, accountability, staff to client ratios, sanitation and hygiene conditions, quality of services provided, integration into other DAP objectives, recognition by government structures. 6. Lessons learned in the Institutional Development project in 1997-98 should be applied to look at phasing out assistance to safety net institutions, support to develop business plans and how to improve handicraft enterprises should be pursued 7. CRS could instigate a coalition/network of agencies that providing similar services to serve as an advocacy group and for joint fundraising. Standards can be developed for certification of quality services and accountability to assure outside donors. Local sponsorship and joint fundraising campaigns can be explored. 8. Enhanced program quality or the quality of services provided in safety net programs should be pursued. Using a rights based approach, CRS should support minimum standards in services provided with the aim of enhancing the quality of life of safety net clients. Areas of intervention may include: nutrition, sanitation and hygiene, education, health care, disease prevention and living conditions. 9. If not previously covered, training should be provided to safety net staff in how to safely handle HIV positive clients that also can be incorporated into health education messages for clients and their families. 10.Methods to use LIFE rations should be explored such as for providing incentives for TB positive patients to appear for treatment at outpatient clinics, family support to households taking in HIV orphans, institutions caring for PLWHAs, or support to home base care programs. If rations are to be used for HIV positive clients, more calories should be provided than in the GR/STEP program as energy needs are much higher with HIV than in normal adults than provided in the GR ration. 24 May, 2001 Final Evaluation Report 73 3.8 HIV/AIDS HIV/AIDS was not a component of the original DAP, however the increasing rates at the national level have made it an issue that can not be ignored. AIDS is one of CRS’ SPP health objectives, which aims to “Increase community-based care and support for those affected and infected by HIV/AIDS by 2005.” The DAP concept paper includes activities related to HIV/AIDS, such as training voluntary counselors and care-givers aimed at helping PLWAs to cope with their affliction, provide home-based care for PLWAs and AIDS orphans, and increased emphasis on IEC strategies to promote behavior change. The first two are projects in and of themselves, and it would be ambitious to include them as components of the DAP program. After meeting with the CRS health unit, it was discovered that the first two activities are part of CRS’ SPP, and were not meant to be included in the DAP. Health education on AIDS is already happening in some of the FACS areas, and will be incorporated into the design of the upcoming IEC training. It is suggested that CRS include questions on AIDS in the next KPC surveys in order to better understand mothers’ knowledge and attitudes, and better design the IEC strategy. From what is understood of the relationship between HIV and nutrition, nutrition could serve as both a preventative and mitigative intervention for HIV. Poor nutritional status is known to increase susceptibility to HIV infection and reduce the body’s ability to fight off opportunistic infections. In HIV positive individuals poor nutritional status can lead to increased HIV replication and speed up disease progression. HIV increases the body’s nutrient requirements and reduces the efficacy of nutrient absorption and utilization. HIV infected individuals are also most likely to suffer appetite loss, resulting in reduced dietary intake, at a time when nutrition needs are higher (Haddad and Gillespie, 2001). Pregnant and lactating women are particularly vulnerable due to the increased nutrient and energy requirements on their body during pregnancy and lactation. The interaction between HIV and nutrition is particularly relevant in Ethiopia, where such a large percent of their population is malnourished. MOH has not yet developed a policy on HIV and breastfeeding. CRS/ET should support the MOH policy, when developed. In the mean time, LINKAGES recommends that if the mother is not aware of her HIV status, she should be advised to exclusively breastfeed her child, as the risks from not breastfeeding outweigh the risk of mother to child transmission (LINKAGES). Nutrition relevant actions aimed at preventing or mitigating HIV are no different from what CRS/ET is already doing in its DAP program, however efforts could be intensified through nutritional counseling. Good nutrition, however, depends on food security, access to health and environmental services and care (Haddad 2001). Supporting livelihood options which are sustainable, can help reduce pressure to engage in livelihoods that involve risky behavior or prolonged periods of non-residence. The agriculture/NRM programs CRS is implementing can have both a mitigative and preventative impact on HIV by utilizing FFW for asset preservation and assessing and supporting community coping mechanisms. 24 May, 2001 Final Evaluation Report 74 AIDS is not just a health problem, and therefore needs to be addressed through integrated programming design. AIDS awareness and prevention should not just be targeted at project beneficiaries, but CRS and partner staff as well, as they have already begun to be affected by it in their own work environment. CRS/ET should consider developing an office policy on AIDS, which aims, for example, at increasing awareness, addressing stigmatization, managing the illness and job security, risk management, access to voluntary testing and counseling services etc. Recommendations: 1. Include questions on HIV/AIDS in the next KPCs, and use data to shape IEC strategy to be incorporated into FACS health education. 2. Increase emphasis on nutrition interventions in FACS program, particularly regarding nutrition of pregnant and lactating women, exclusive breastfeeding, and access to water and sanitation facilities. 3. Continued emphasis on agriculture/NRM, to support community coping mechanisms and to preserve assets through utilization of FFW. 4. Development of an AIDS policy for CRS/ET staff. 24 May, 2001 Final Evaluation Report 75 3.9 Emergency Mitigation and Preparedness Strategies Ethiopia is a country in a constant state of chronic emergency, slipping into full￾fledged drought on average every three years. Faced with very few acute onset natural disasters, the emergency situations CRS/Ethiopia must deal with are drought and civil conflicts. As little can be done to prevent drought and arguably civil conflicts, energies must be placed into mitigation and preparedness efforts. While not specifically classified as emergency oriented, many of the activities in the DAP have positive mitigative effects and help communities prepare for disaster. For example, the construction of bunds, terraces, livestock ponds and catchment systems all help retain rainfall to mitigate effects of a drought. Likewise, closures, savings programs, and roads allow households to diversify income sources and provide access to markets, so when drought destroys agricultural income, there are economic alternatives. Supplementary feeding programs serve as a community level safety net, providing food to the most vulnerable. Food for work activities also provides a safety net for vulnerable households. Established committees such as the village development and water committees provide communities with experienced leaders and groups that can quickly be called upon to be representatives in emergencies. The information known about target communities allows for quick and appropriate emergency responses. Existing relationships and programs in communities permits emergency responses to be integrated into development strategies. By continuously working with Title II food, CRS and its counterparts have established warehouses, contacts with trucking firms, and logistical knowledge to handle numerous tons of emergency food. This capacity was utilized in the recent JEOP response where CRS was responsible for the movement of over 200,000 MT of emergency food. The network of counterparts across the country allows CRS to quickly respond to emergencies around the country. Relationships with organizations such as USAID, ECS, Caritas Internationalis, CRDA and JRP, permit CRS to access many donor and distribution networks for emergencies. Recommendations: 1. CRS should continue to strengthen relationships with national and international organizations that respond to or fund emergencies in Ethiopia. 2. CRS should establish contacts with agencies collecting early warning information such as SCF/UK, FEWS, FAO and DPPC, so any information on the deterioration of food security status in target areas is quickly disseminated to partners and incorporated into program response. 24 May, 2001 Final Evaluation Report 76 Overarching Themes for the Next DAP Emerging from the above discussion, the Evaluation Team has identified a number of overarching areas that should inform planning activities for the next DAP, as follows: 1. Integration 2. Site-specific programming 3. Meaningful community participation 4. Getting the basics right 5. Doing less, but doing it better These themes are not sector specific, and they are certainly not new themes to CRS/ET and implementing partner staff. By (re-)stating them here the Evaluation Team is, in part, acknowledging and lending support to program quality and support improvements that have already taken place within the CRS/ET and implementing partner offices that were visited. There is another reason for stating these five themes, and that is to re￾emphasize their importance and, importantly, to draw attention to the fact that their acceptance will have resource and, thus, targeting (i.e. programmatic) implications that will need careful consideration during the planning of the next DAP. Inter- and Intra-Sector Integration We acknowledge that CRS/ET and its partners have invested much intellectual effort in developing a strategy for integration (CRS/ET, 2001b). The IWM strategy explicitly recognizes the benefits of planning with integration in mind. Indeed, CRS/ET and its implementing partners appear well aware of the potentially beneficial synergies arising from integrating project activities and their understandings, and acceptance of the concept, of integrated planning, is reflected in the documentation that was reviewed by the Evaluation Team. Noteworthy progress has been made in the current DAP to put into practice the concept of integrated planning and activity (e.g. Lege Oda Mirga). Nevertheless, there were sufficient examples obtained during our field visits, where a lack of integration was hampering real development progress,22 to lead us to believe that emphasizing integration as a core theme for the next DAP would not be wasted effort. In order to operationalize the notion of integrated planning, CRS/ET should make it clear to its implementing partners that submissions for the next DAP will be appraised with inter- and intra-sector integration as one of the criteria against which projects would be assessed and approved. Initially, there would be an 22 One example will suffice: health education messages that stress washing hands where water is scarce. 24 May, 2001 Final Evaluation Report 77 important supporting role for CRS/ET in providing advice to its implementing partners on programming for integrated activity. Over time CRS/ET would be able to develop a portfolio of examples of how activities can be considered, and planned for, from an ‘integrating’ perspective. Examples of integrated activities might be: · integrated planning – mapping the targeted areas for roads, water, health, agricultural marketing potential to make sure the greatest benefit is reaped from the investments made by CRS/ET and its implementing partners; · recognising that changes in health knowledge, leading to health improvements via changes in behaviour, can have important and sustainable impacts on food security through increased agricultural labor availability; · identifying overlap in the audiences for agriculture and health activities, thereby helping to operationalize part of CRS/ET’s strategic approach to gender issues (CRS/ET, 2001b); · integrating sanitation improvements with potable water developments to ensure that improvements at the water point are extended through to the place of drinking; · linking road construction activities to SSI developments so that producers can more easily market their agricultural produce and/or indirectly benefit (e.g. via increased employment opportunities) from MFI activity in urban areas. Site-specific programming Although programmatically more complicated, both administratively and in terms of the likely technical support requirements, the Evaluation Team believe that the ‘bottom-up’ approach to planning the next DAP will appropriately lead to more site-specific programming. It hardly needs saying that there are significant agro￾ecological and socio-economics differences in the different locations that CRS/ET is planning to support under the next DAP. ‘Blanket’ recommendations are unlikely to work, either in the agriculture / natural resource management context, or in the health sector. The FACS program cannot be implemented across all implementing sites as if communities are somehow homogeneous. Due to different cultural beliefs health education messages should differ by location. There is already evidence from the current DAP where, for example, the blanket recommendation of organic composting has been rejected by farmers in a number of locations. Whilst composting may be best practice from a theoretical agronomic perspective, there are a range of constraints across the current DAP sites all working in opposition to general acceptance of this ‘technology’. For example, there are differences in the physical quantity of biomass available; differing uses of any biomass that is available; and local concerns about the 24 May, 2001 Final Evaluation Report 78 danger of composting pits with children and livestock living in close proximity. Such variation will need to be reflected in plans submitted under the next DAP. Nonetheless, it is recognized by the Evaluation Team that site-specific programming is more complex in terms of the demands that it places upon CRS/ET and counterpart staff. In response, we consider such an approach to programming is an ideal that CRS/ET and its implementing partners should be striving towards under the next DAP and beyond. Meaningful community participation Throughout the documentation reviewed by the Evaluation Team, the issue of participation by local communities is stressed. Yet, according to the ADCS when asked what had been important ‘failings’ of the current DAP the responses included the limited (or “theoretical”) progress achieved in meaningful community participation. To be fair, concerns about meaningful participation are not just an ADCS phenomenon! There seems to be a gap between participation as mere rhetoric in present development policy debate, on the one hand, and participation as operational practice on the other. According to Farrington (1998), “participation is becoming a devalued term.” He suggests that partly in response to the desire of donors to identify their development programs as participatory, “much of the rhetoric, and occasionally the form, of participation are deployed without the substance.” Experience suggests that meaningful participation generally enhances project performance (USAID, 1996). There is a difficulty of operationally defining participation since definitions vary a great deal. For example, according to interviewed CRS/ET staff, there is a perception among some implementing partner staff that FFW projects are participatory as the community participates in the directed activities. It was not possible for the Evaluation Team to observe a wide range of counterpart staff-community interactions. Nevertheless, it is the Team’s belief that ‘participation’ in the current DAP is sometimes interpreted to mean a range of stakeholder roles, few of which actually involve an active and influential hand in shaping management decisions that affect the lives of the community. As with site-specific programming, meaningful participation has programmatic implications for CRS/ET and its implementing partners. Time is required for building up trust between stakeholders in a community, and it is important to recognize that changes in CRS/ET and counterpart staff can make the whole process vulnerable (e.g., Mosse et al, 1998). The challenge for CRS/ET under the next DAP will be to manage the process involved in moving towards more meaningful participatory DAP programming. Getting the basics right Whilst the above themes suggest a greater degree of sophistication on the part of CRS/ET and its implementing partners, the Evaluation Team is keen to stress 24 May, 2001 Final Evaluation Report 79 that it is some of the ‘basic’ operations that also must be managed well. For example, the Team has earlier in this Report raised a number of concerns regarding the quality of data collected under the FACS/CBHC program that will make it very difficult to undertake an assessment of the impact of the current DAP program. It is more difficult, consequently, to improve program quality in the next DAP. This is but one example; there are others. ‘Getting the basics right’ also involves CRS/ET and counterpart staff improving their understandings of the resource limitations facing men and women in different rural communities. Such understandings will allow them to respond with appropriate project interventions. For example, slow uptake of specific health education messages may not simply be that community members do not understand the content, the response being that they need to be reiterated; rather that the messages themselves may be inappropriate, the response then being that the messages should be revisited and altered accordingly. Getting the basics right requires good program support from CRS/ET staff, i.e. regular and careful supervision from CRS/ET staff, careful analysis of reported results, and considered responses. This aspect of program implementation is essential for improving program quality under the next DAP. There are, again, programmatic implications. Doing less, but doing it better All of the above four points suggest greater demands upon staff resources. There is little doubt that site-specific programming, meaningful participation, and getting the basics right will place additional burdens upon existing staff. This issue is how this can best be managed? One consideration will be geographic targeting. CRS/ET has been working with local counterparts (mainly the Church and Church affiliates) for many years in drought/famine affected areas. Through relief work, CRS/ET and its implementing partners have strengthened their ability to undertake different programs in the northern, central and eastern parts of Ethiopia. These areas are appropriate for continued DAP support since they include food insecure weredas and lessening this problem is a priority for both organizations. As a major objective of CRS is to build the capacity of local partners, the presence of such partners is a key determinant of geographic targeting. Once partners are identified, a range of ‘objective’ criteria are used to select sites, e.g. those indicated in the SPP.23 It should be recognized that other (political, cultural) factors influence CRS/ET’s process of geographical targeting; local (wereda, Zonal) government demands that CRS/ET and its partners work in the more remote and vulnerable areas of Ethiopia. Twenty per cent of HCS’s sites in East Hararghe were selected in response to such pressures. 23 The Safety Net program does not involve the geographic targeting process to determine areas of operation. 24 May, 2001 Final Evaluation Report 80 ADCS is keen to expand its area of coverage under the next DAP, both in area coverage and complexity (i.e. components). How should CRS/ET appraise this desire to grow? Bearing in mind the notion of ‘doing less, but doing it better’, the expansion plans of partners need to be assessed with regard to the capacity and resources available to provide a minimum level of well-supported integrated activities in the targeted area. Given a solid foundation such as this, further investments in the community then become increasingly attractive. 24 May, 2001 Final Evaluation Report 81 Summary of Recommendations 3.1 Agriculture / Natural Resources Management Recommendations 1 1. Given the investments in staff time required to establish meaningful PA and village￾level relationships CRS/ET and counterpart staff need to establish the implications for: · scheduling planning work prior to the next DAP submission; · establishing and agreeing upon the appropriate scale of operations of implementing partner activities for the next DAP period; · determining realistic outcomes and targets to be included in the next DAP proposal. CRS/ET and counterpart staff should learn from the HCS experience and be guided in setting realistic responses to the three points raised above. 2. Training, in some cases refresher, in participatory approaches is required for CRS/ET and counterpart staff given the importance attached to this conceptual approach to planning the next DAP. 2 1. Agricultural / natural resources management proposals are expected to be initiated locally on the basis of discussions between counterpart staff and the communities within which they are planning to work. It is important that CRS/ET ensure that the diversity in agricultural and natural resource management systems across these zones is reflected in the next DAP proposal submitted to USAID/ET Mission. ‘Blanket’ agriculture / natural resources management proposals are to be avoided. 2. CRS/ET should provide counterpart staff with technical support and on-the-job training to facilitate this process. 3 1. With CRS/ET technical and financial support, selected implementing partners should pilot participatory well-being ranking, on an initially limited basis, during the planning period to contribute to activities proposed under the next DAP proposal. 2. If useful for project planning purposes, the approach should be initiated elsewhere during program implementation, and in appropriate instances M&E data presented in a more disaggregated (by socio-economic category of household) manner. 3. It may be appropriate to seek external training assistance for the initial attempts at ranking. 4 1. Implementing partners should identify the scope for integrating SSI development with market-oriented agriculture during the planning process. 2. CRS/ET should strengthen the capacity of implementing partners through training and other support to ensure that SSI developments planned under the next DAP are based upon sound technical, social, economic and institutional analyses. · SSI training needs of CRS/ET and counterpart staff need to be identified and met; · This may involve the development of links with external organizations for appropriate support (e.g. ADCS with REST in Mekele, or with Department of Agriculture Irrigation specialists). 3. CRS/ET should guide implementing partners in establishing and managing links with external organizations, e.g. with regard to preparing Terms of Reference for contracted work, and in overseeing the quality of their work. 4. CRS/ET and its counterpart staff should initiate a system for monitoring the ‘post- 24 May, 2001 Final Evaluation Report 82 program’ performance of water user committees or associations established between 1997-2001, and use this knowledge to strengthen subsequent SSI program quality. 5. CRS/ET should strengthen its links with international networks related to SSI (e.g. IWMI, IIED). 5 1. CRS/ET and implementing partners should pursue further with the CRS East Africa Regional Office the possibility of incorporating farmer-led on-farm crops/varieties testing in the next DAP proposal. 2. CRS/ET should identify ways to build implementing partner capacity to facilitate farmer￾led testing and evaluation. 3. Program quality will be enhanced in the next DAP proposal via greater partnership with technical organizations such as EARO and the IARCs in technology (and other) exchanges. CRS/ET has a key role in facilitating and supporting such developments, and the CRS East Africa Regional Office should be approached for assistance in this regard. 4. CRS/ET should use the DAP extension period to determine if there is an opportunity to include a collaborative venture under the follow-up DAP. 6 1. Through its integrated planning process, CRS/ET should encourage its implementing partners to identify the current opportunities in each targeted area for supporting market-oriented agricultural activities in the next DAP proposal. 2. Given the lack of in-house agricultural marketing expertise, this will necessitate accessing appropriate technical assistance to appraise proposals put forward by the implementing partners. 3. Where DAP activity is more marketed oriented, indicators in the next DAP should focus more on returns to producers, and less on returns to land (i.e. yield in kg/ha). 7 1. The proposed CRS/ET Microfinance Business Plan will directly benefit poor farmers only if they live sufficiently close to the urban sites proposed in the MFI Business Plan. Market-oriented quality programming and support will be critical and agricultural marketing input may be required at the planning/appraisal stage prior to the next DAP. · For farmers in more remote areas, the MFI Business Plan will likely limit the extent to which they are able to adopt improved crop and livestock technologies that necessitate purchased complementary inputs. · This will impact upon the choice of agricultural technologies to promote under the next DAP. Implementing partners should plan their agricultural programs for the next DAP on the assumption that farmers will not have access to purchased inputs. Lower input (i.e. less risky) technologies should be preferred and should be based on farmers’ own expressed preferences. · Opportunities to link with other institutions offering financial and agricultural services should be sought. 2. Evidence suggests that indirect benefits (via a more active rural labor market) are possible, and CRS/ET and its implementing partners should identify opportunities to improve rural labor mobility via the appropriate development of rural roads in its integrated planning discussions for the next DAP. 3. CRS/ET should monitor the progress of the finance associations recently established in West Hararghe and Dire Dawa. 8 1. To address the issue of on-farm technical sustainability, recommendations 2 and 5 (see summary box below) are particularly applicable. 2. On-going longitudinal monitoring should be continued in the next DAP after phasing out 24 May, 2001 Final Evaluation Report 83 from the current DAP sites, in order to confirm the institutional sustainability of current DAP activity. CRS/ET should consider local external input into this work. Option to consider if resources available: 3. It is suggested that given the prominence given to the Lege Oda Mirga model for sustainable development, CRS/ET and HCS commission an independent survey to validate the reported case study findings. 3.2 Health and Nutrition Recommendations 1 Health Education 1. The upcoming IEC training should focus not only on appropriateness of IEC materials but development of messages based on attitudes toward care practices in specific regions. It may be useful to develop a framework for each message based on input from CHAs and mothers. 2. Pilot a number of different strategies which are learner-centered and utilize problem solving approaches (nutritional counseling cards, mother to mother support groups, demonstration) to see what works best. 3. Network with other NGOs to see what health education materials and strategies are being used, so as to avoid duplication of efforts. 4. Develop strategies to target men, which take advantage of utilizing opportunities when men gather together in the community. 2 Data Management 1. LQAS (Lot Quality Assurance Survey) could be used to assess the completeness and accuracy of registers, since the Evaluation Team observed only a small number, which may not be representative of all communities/partners. 2. Review MIS with partners and communities to ensure that information is useful and reflects priorities. Any forms not being used should be either adapted to be made more useful or discarded. 3. Training staff at all levels on uses of information for decision making and project management. Data should be used at all levels (animators, partners, CRS) to prioritize monitoring visits and make program/management decisions. 4. Reprint registers to make them more portable and easy to use by animators, and store them in a place that the community can access them. 3 Technical Support and Supervision 1. Both CRS and partner staff should begin using the supervisory checklists on every monitoring visit, and adapt them if necessary. 2. Realistic supervisor to supervisee ratios should be maintained based on workload 3. CRS health staff need to keep themselves and their partners abreast of new innovations in the field of child survival, network with other NGOs working in Ethiopia, and visit successful CBHC projects in the region. 4 CBHC Targeting 1. FACS/CBHC target areas should correspond to watershed management areas of the agriculture program. 2. Areas should be saturated rather than including only one or two villages in a PA. It is better to target all the villages in a PA than include one village in each of four or five Pas. This will make relations with PA government organizations simpler and impact will 24 May, 2001 Final Evaluation Report 84 be greater. 3. In target areas, CBHC programs need to continue to target 100% of pregnant and lactating women and children under two. If possible, all women of reproductive age will be targeted for health education. If sessions can be designed at convenient times to include husbands, also, that would be effective for several subjects. 5 Nutrient Content of CBHC Rations 1. The ration size should be adjusted according to the aim of the program. If it is determined that an incentive ration is more appropriate, research should be conducted to configure an appropriate ration size that meets economic opportunity costs of women participating in the program. Most likely this would mean a much smaller ration. If a preventive or recuperative ration is determined to be appropriate, the food deficit of households needs to be calculated and applied. As the average caloric consumption in Ethiopia is around 1700 kcal, thus leaving an average deficit of 400 kcal per person, a ration for an average household of five members would require 2000 kcal, a doubling of the present ration size. This would have to be determined by programming area where actual deficits may be much greater or less. 2. Alternative iron rich commodities that are more palatable and culturally appropriate such as lentils (which have twice the protein content per gram as wheat) should be considered. If one of the program targets is young children, wheat should be removed as a commodity as it is hard to digest. Lentils could be substituted for the wheat. 3. Food included in CRS provided food rations should be as fortified as possible with vitamin A, iron and other micronutrients. Vegetable oil should be fortified with vitamin A and investigation into fortification possibilities of other foods should be examined. Possible candidates for fortification include vitamin A, iron, iodine, B-complex vitamins, zinc, copper, and folate. Even if not all the energy needed by a pregnant woman per day is included in the ration, sufficient iron and other micronutrients should be provided to help meet normal diet deficiencies. 6 Food Targeting 1. CRS should be flexible in its FACS/CBHC package, allowing for regional differences in nutritional needs and government regulations. 2. Before deciding how to use food in CBHC/FACS, assessments should be conducted to determine why children are malnourished and how much food availability factors into nutritional status. 3. Food targeting options should be chosen according to the causes of malnutrition in the target area, counterpart preferences and government regulations. 4. If possible, pregnant and lactating women should receive preventive rations. 5. Only one ration should be given at any time to a household, i.e. either the mother gets it or the child. 6. Rations for children should start at six, not four months to avoid sending mixed messages to breastfeeding mothers. 7 Micronutrient Deficiencies and Supplementation 1. CRS needs to assure that every pregnant mother and child under two is not micronutirent deficient. Potential programs to accomplish this include: 2. Health education to help mothers identify micronutrient rich foods and eat the most balanced diet as possible to maintain adequate nutritional stores while pregnant and in young children. 3. Work with the MOH to make sure that within 6 weeks of giving birth every mother 24 May, 2001 Final Evaluation Report 85 receives a single dose of 200,000 IUs vitamin A and every child receives a vitamin A capsule every six months 4. As iodine was identified by PROFILES as a prominent concern in Ethiopia, and due to cutoff of supplies from Eritrea, CRS should assess the use of iodized salt in target areas and if low usage, work with ICCIDD or similar agencies to educate households and bring in iodization equipment. Kiwanis International has taken on iodine deficiency as an international campaign and provides funds for eradication. 5. Advocate for iodinization of local salt or if not possible distribution of iodine oil capsules to pregnant and lactating women and children 6. Integrate agriculture programs with CBHC to promote micronutrient rich foods 7. Obtain fortified commodities for food distribution (for all food programming, including FFW and emergencies) 8. Work with agencies that specialize in micronutrient supplementation such as Helen Keller International, LINKAGES, ICCIDD, MICAH, PAMM, and IVACG to obtain information on best practices in reduction of micronutrient deficiencies and serve as networks for collaboration. 8 Sustainability and Exit Strategies 1. Site specific phase out plans will need to be considered for each implementing partner. 2. CRS will need to work with counterparts and communities to come up with incentives for CHWs after the phase-out of the food program 3.3 Water and Sanitation Recommendations 1 1. Explore expanding focus of the water strategy beyond potable water to include multiple uses of water. 2. Continue low level monitoring of water committees after phase out in order to evaluate sustainability. 3. Hygiene and sanitation education should include learner-centered and problem solving approached to understand the community’s beliefs and attitudes towards hygiene practices. 3.4 Use of Food Aid 1. CRS/ET should look at why in the final year of the DAP project implementation levels involving direct food distribution are slightly greater than the previous year. Especially for CBHC, the number of eligible beneficiaries in each target area should be decreasing substantially as each woman/child dyad is only eligible for one “round” of food. The impact on communities should be examined. 2. FFW phase out plans should be reviewed with counterparts to ensure that phase out is being done in a manner that will not result in negative impacts on the food security situation. 3. Program level impact indicators should be reviewed with the Mission to verify that FFW activities are not expected to have an impact on the nutritional status of children under five. 3.5 Monitoring and Evaluation Recommendations 1 1. CRS/ET and implementing partner staff should be introduced to the concept of utilization-focused monitoring and evaluation through a series of in-house training workshops prior to and during the next DAP 2 1. CRS/ET and its implementing partners should undertake a pilot information audit during the planning period for the next DAP to determine if current information use is 24 May, 2001 Final Evaluation Report 86 during the planning period for the next DAP to determine if current information use is optimal, and to identify opportunities for quality improvements prior to the commencement of the next DAP. This should be piloted with one implementing partner to determine if the audit is perceived to be useful for improving program quality. 3 1. CRS/ET and its implementing partners should approach local higher education institutions with a view to identifying a possible role for staff/students in undertaking DAP M&E data collection and analysis activities. 2. Such a partnership could be tested on a pilot basis initially to determine its value for improving the quality of M&E data 4 1. CRS/ET should consider encouraging implementing partners to source background material prior to submitting DAP-related proposals. This could be reference materials, if available, or evidence of discussions with local stakeholder organizations. 2. CRS/ET and the CRS East Africa Regional Office need to determine their respective roles in accessing and providing relevant information to in-house and counterpart staff 5 1. CRS/ET should seek to initiate and develop an inter-NGO network for strengthening the quality of M&E work undertaken. One way to start this process is to circulate reports which discuss the methodologies and findings of surveys undertaken by CRS/ET and implementing partner staff. 2. CRS/ET should, through its Regional Office, enquire about the existence of an Ethiopian Evaluation Association from the office of the African Evaluation Association. For professional development purposes, CRS/ET’s M&E officer should be encouraged to join. 3.6 Expanding Partnerships with ‘External’ Partners 1 1. CRS/ET should organize a 1-day in-house workshop to address the issue of expanding partner relations under the next DAP. 3.7 Safety Net 1. The rations of safety net programs should be examined in relation to program goals and additional supplementation. If GR/STEP rations serve an incentive purpose, then they should be adjusted to reflect opportunity costs. Likewise, the percentage of the diet OCF rations are intended to provide for clients should be researched. If the OCF ration is the only food source provided to clients each day, then the ration should be increased. The MOC ration should be examined more closely with the MOC sisters to see how exactly the food is being distributed. If more people are actually consuming the food than recorded, then the number of beneficiaries should be adjusted to reflect the true number and an appropriate ration given for the larger number of people. 2. As outlined in the SPP, safety net institutions should be encouraged or mandated to develop a strategic vision with a corresponding five-year plan. Other goals and objectives to be explored could include: expansion of the donor base, institutional sustainability, improved management capacity, improved fundraising strategy/materials, detailed targeting criteria, higher quality of services provided to clients, and plans to ensure continued support of operations from government authorities. 3. Funding should be pursued to fill in gaps not met by DAP resources especially transport costs for food. Support from the domestic constituency in the US could be accessed by sponsorship programs from US parishes. 4. Agencies that support PLWHA would be recommended for any Safety Net program expansion. 24 May, 2001 Final Evaluation Report 87 5. Options to consider if resources available: 6. Criteria for continued support to safety net institutions need to be established. Criteria should include standards in financial management, accountability, staff to client ratios, sanitation and hygiene conditions, quality of services provided, integration into other DAP objectives, recognition by government structures. 7. Lessons learned in the Institutional Development project in 1997-98 should be applied to look at phasing out assistance to safety net institutions, support to develop business plans and how to improve handicraft enterprises should be pursued 8. CRS could instigate a coalition/network of agencies that providing similar services to serve as an advocacy group and for joint fundraising. Standards can be developed for certification of quality services and accountability to assure outside donors. Local sponsorship and joint fundraising campaigns can be explored. 9. Enhanced program quality or the quality of services provided in safety net programs should be pursued. Using a rights based approach, CRS should support minimum standards in services provided with the aim of enhancing the quality of life of safety net clients. Areas of intervention may include: nutrition, sanitation and hygiene, education, health care, disease prevention and living conditions. 10. If not previously covered, training should be provided to safety net staff in how to safely handle HIV positive clients that also can be incorporated into health education messages for clients and their families. 11. Methods to use LIFE rations should be explored such as for providing incentives for TB positive patients to appear for treatment at outpatient clinics, family support to households taking in HIV orphans, institutions caring for PLWHAs, or support to home base care programs. If rations are to be used for HIV positive clients, more calories should be provided than in the GR/STEP program as energy needs are much higher with HIV than in normal adults than provided in the GR ration. 3.8 AIDS/HIV Recommendations 1 1. Include questions on HIV/AIDS in the next KPCs, and use data to shape IEC strategy to be incorporated into FACS health education 2. Increase emphasis on nutrition interventions in FACS program, particularly regarding nutrition of pregnant and lactating women, exclusive breastfeeding, and access to water and sanitation facilities 3. Continued emphasis on agriculture/NRM, to support community coping mechanisms and to preserve assets through utilization of FFW 4. Development of an AIDS policy for CRS/ET staff 3.9 Emergency Preparedness and Mitigation Recommendations 1. CRS should continue to strengthen relationships with national and international organizations that respond to or fund emergencies in Ethiopia. 2. CRS should establish contacts with agencies collecting early warning information such as SCF/UK, FEWS, FAO and DPPC, so any information on the deterioration of food security status in target areas is quickly disseminated to partners and incorporated into program response 24 May, 2001 Final Evaluation Report 88 References General: Boelee, E., H. Laamrani and W. van der Hoek. (2000). ‘Multiple use of irrigation water in dry regions of Africa and South Asia’. Communication texts, Volume 1, Session 1B – 51-58, International Conference “Water and Health – Ouaga 2000. Health and nutritional impacts of water development projects in Africa.” Ouagadougou, Burkina Faso, 21-24 November 2000. Abstract in Cahiers Agricultures, 9(5): 434. Casley, D.J. and K. Kumar. (1987). Project Monitoring and Evaluation in Agriculture. John Hopkins University Press. Ch.8. Cromwell, E., P. Kambewa, R. Mwanza and R. Chirwa. (2001). Impact Assessment Using Participatory Approaches: ‘Starter Pack’ and Sustainable Agriculture in Malawi . Network Paper No. 112, AgREN Network, Overseas Development Institute, London. January. Drinkwater, M. and T. Rustinow. (1999). CARE’s Livelihoods Approach. Presentation to DFID Natural Resource Advisers Conference, 1999. (www.livelihoods.org/info/nrac/care.pdf) Farrington, J. (1998). Organizational Roles in Farmer Participatory Research and Extension: Lessons from the Last Decade. Natural Resource Perspectives, No. 27, Overseas Development Institute, London. January. Haddad, L. and S. Gillespie. (2001). Effective food and nutrition policy responses to HIV/AIDS: What we know and what we need to know. IFPRI Draft FCN Discussion Paper. March. IFAD. (2001). Rural Poverty Report 2001: The challenge of ending poverty. Rome. February. LINKAGES Project. (n.d.). Frequently Asked Questions on Breastfeeding and HIV/AIDS. www.linkagesproject.org. Mosley, P. (2001). Agricultural Microfinance in Africa. Seminar at Imperial College at Wye, University of London. March. Mosse, D., J. Farrington and A. Rew. (eds.). (1998). Development as Process: Concepts and Methods for Working with Complexity. Routledge, London. Patton, M.Q. (1997). Utilization-focused Evaluation. Sage Publications, London. Powell, M. (1999). Information Management for Development Organizations. Oxfam Publications, Oxford. Scott, C. (1985). Sampling for Monitoring and Evaluation. World Bank, Washington, D.C. Temu, A. and J. Due. (2000). Participatory appraisal approaches versus sample survey data collection: a case of smallholder farmers well-being ranking in Njombe district, Tanzania. Journal of African Economies, Vol. 9 (1). pp. 44-62. 24 May, 2001 Final Evaluation Report 89 USAID. (1996). Conducting a Participatory Evaluation. Performance Monitoring and Evaluation Tips, No. 1. Centre for Development Information and Evaluation. Washington D.C. 24 May, 2001 Final Evaluation Report 90 Project Related Documents: Abera, Y., L.E. Thompson and D. Kabat. (2001). Business Plan. Monetization Business Unit, Ethiopian Monetization Consortium. Addis Ababa. March. ADCS. (Various). Annual and Quarterly Progress Reports. Assefa, T. (2000). The Study of Micro-Finance Program. CRS/ET, Addis Ababa. CBI. (2000). Report and Recommendations to Ethiopian Monetization Consortium. Worthington, Ohio, USA. September. CRS. (1999). Programmatic Environmental Assessment of USAID-Funded Small-Scale Irrigation Carried Out By Cooperating Sponsors In Ethiopia. September. CRS/ET. (2001a). Potable Water and Sanitation Strategy and Guidelines. Draft prepared for internal review and comments. March. CRS/ET. (2001b). Strategic Program Plan, 2001-2005. Addis Ababa. CRS/ET. (2000a). Concept Paper to USAID for a DAP II (FY02-06). October. CRS/ET. (2000b). CRS/ECS Microfinance Institution. The Business Plan, 2001- 2005. September. CRS/ET. (2000c). Working Paper on Agriculture/Natural Resource Management. Agriculture/Natural Resource Management Working Group. August. CRS/ET. (2000d). Lege Oda Mirga Case Study Report. Addis Ababa. December. CRS/ET. (2000e). Monitoring and Evaluation Guidelines. Draft for discussion. Addis Ababa. November. CRS/ET. (1999a). Proceedings of a Workshop on Agriculture/Natural Resources Management. Nazareth, 22-29 November. CRS/ET. (1999b). FY1999 Annual Results Report to USAID. Addis Ababa. CRS/ET. (1999c). Mid-Term Evaluation. CRS/Ethiopia. 1997-2001 DAP. Final Report. Addis Ababa. June. EPI. (2000). Water and Food-Aid in Environmentally Sustainable Development. An Environmental Study of Potable Water and Sanitation Activities Within the Title II Program in Ethiopia. Report prepared for USAID Mission to Ethiopia. Contract No. PCE-1-00-96-00002-00. March. Kumssa, G. (n.d.). Developing an Integrated Watershed Management Strategy. Paper prepared for the EARO/Nairobi, Ag. TC Workshop. CRS/ET, Addis Ababa. Raisin, J. (2001). Beyond the Merry Go Round. The Relief-Development Continuum. Prospects for Linking USAID Relief and Development Resources in Amhara National Region State. USAID/ET Mission, Addis Ababa. Draft. April. 24 May, 2001 Final Evaluation Report 91 Remington, T. and G. Kumssa. (2000). Watershed Management: The cornerstone of a Catholic Relief Services Agriculture Strategy in East Africa. Paper presented at the Joint Vertisol Project Symposium, 28 November – 1 December. Ethiopian Management Institute, Debrezeit, Ethiopia. USAID/ET. (2000). Integrated Strategic Plan. Addis Ababa. 24 May, 2001 Final Evaluation Report 92 Annex A CATHOLIC RELIEF SERVICES / ETHIOPIA PROGRAM FINAL EVALUATION OF THE FY97-01 DAP TERMS OF REFERENCE 1. BACKGROUND Catholic Relief Services/Ethiopia (CRS/ET) is one of eight Cooperating Sponsors implementing Title II development programs in Ethiopia. CRS/ET has been supporting emergency relief and development projects in Ethiopia since 1958. The current Development Activity Proposal (DAP), which runs from FY1997-2001, aims to improve the food security of targeted communities by addressing the following problems: · Low agricultural production; · Pervasive land degradation and soil erosion; · Poor health conditions resulting from insufficient investment in rural health facilities and complicated by over population; · Low household income which constrains access to food and health care; · Lack of access to food by the chronically poor, including the terminally ill, orphans, mentally/physically handicapped, and persons living with AIDS. The goal of CRS/ET’s DAP is Enhanced Food Security of Vulnerable Populations. There were originally three sub-goals, each with two strategic objectives: Sub-goal 1: Sustainable Increase in Food Availability · Sustainable increase in agricultural production · Enhanced Sustainability of Natural Resources Sub-goal 2: Improved Utilization of Food by Vulnerable Groups · Improved health, status of young children and mothers · Develop sustainable community structures for promotion of the health of women and children Sub-goal 3: Improved Access to Food for Vulnerable Groups · Improved incomes for vulnerable groups · Provision of a safety net for vulnerable groups After submission of the DAP, CRS/ET worked with the USAID Mission and other Cooperating Sponsors to develop a new special objective framework (SPO), which had five major intermediate results: IR 1. Increased Agricultural production IR 2. Increased Household Income IR 3. Improved Health Status in target areas IR 4. Natural Resource Base Maintained IR 5. Emergency Response Capacity Maintained CRS/ET has been working to achieve these intermediate results through five main projects: · Agriculture and Natural Resource Management (AG/NRM) · Food-Assisted Child Survival / Community-Based Health Care (FACS/CBHC), · Women’s Savings and Credit (WSC) · Safety Net programs, including Other Child Feeding (OCF), General Relief (GR), Skills Training and Education Program (STEP), and Missionaries of Charity (MOC) 24 May, 2001 Final Evaluation Report 93 · Vulnerability Early Warning System (VEWS) The DAP has been implemented in Tigray, Oromia, Dire Dawa, Harari, SNNPR, Gambella, Somali and Addis Ababa. The major counterparts implementing the DAP include Adigrat Catholic Secretariat (ADCS) in Tigray, Hararghe Catholic Secretariat (HCS) in Dire Dawa, Harari and East Hararghe, Oromia; Nazerath Children’s Integrated Development (NACID), Meki Catholic Secretariat (MCS) and Wonji Catholic Church (WCC) in East Shewa, Oromia; Archdiocesan Catholic Secretariat (ACS) in Gurage, SNNPR, and Cheshire Foundation Ethiopia (CFE) and Integrated Holistic Approach (IHA) in Addis Ababa. The MOC, OCF, GR and STEP projects consist of 37 centers spread across 9 regions throughout the country. The original DAP budget for FY97-01 included 97,853 MT of food commodities for distribution, a monetization budget equivalent to US$12,879,490 for programming support costs (including secondary transport), and US$1,456,379 from 202(e). II. PURPOSE OF THE FINAL EVALUATION CRS/ET conducted a mid-term evaluation (MTE) in Year 3 (FY99) of the five year DAP. The MTE was conducted in May 1999, with the final report received by August 1999. A copy of this report will be provided to the final Evaluation Team for their reference. The MTE succeeded in providing a very thorough assessment and analysis of the DAP, including: · Strengths and weaknesses of programming strategies and M&E indicators · Problems associated with monetization efforts · Problems of securing government cooperation and approvals · Implications of Proclamation 40/1996 for DAP microfinance activities · Logistical problems associated with the closure of Assab port, truck availability, and delays in clearing DAP-funded vehicles · Results of Years 1 and 2, including successes and lessons learned · Strengths and weaknesses of implementing partners CRS/ET believes that the MTE has provided a comprehensive analysis and review of the DAP, and many of the conclusions and recommendations are still providing useful and relevant guidance to senior management in FY01. In addition, a number of other evaluations have been conducted on DAP-related activities in the past several years, including a Programmatic Environmental Assessment (PEA) of Small-Scale Irrigation activities, a USAID-sponsored evaluative study on potable water activities, an evaluative study on the Women’s Savings and Credit (WSC) project, and an evaluation of the Ethiopia Monetization Consortium (EMC). CRS/ET’s preference is that the final evaluation not duplicate what these detailed evaluations have already accomplished. Rather, we would like to use the final evaluation more as a planning tool to take the successes and lessons learned from this DAP and use it to refine programming strategies for the next DAP, which is proposed to commence in FY03. Thus, the consultant team leader for the final evaluation should be prepared to conduct a utilization-focused evaluation (UFE) which will provide CRS/ET senior management with a foundation and proposed structure from which the new DAP proposal can be developed. This UFE would also need to include an update of achievements and impacts which focuses on FY00 and the first half of FY01. III. SPECIFIC TASKS The main tasks of the final evaluation will be to: 1) concisely assess CRS/ET progress made in implementing the key recommendations of the various mid-term evaluations; 2) concisely assess CRS/ET progress toward achieving the goal, objectives, and intermediate results of the DAP; and 3) use results and analysis from the first two tasks to provide guidance and recommendations on specific strategies CRS/ET should incorporate into the next DAP proposal so that overall program quality and impact will improve. Task #1: Concisely assess CRS/ET progress made in implementing the key recommendations of the various mid-term evaluations and annual results reports. 24 May, 2001 Final Evaluation Report 94 Accomplishing this task will first require the Evaluation Team to obtain copies of the formal MTE DAP evaluation, as well as PEA for small-scale irrigation, the potable water study, the WSC study, the EMC evaluation, the FY99 Results Report, and the FY00 Results Report. These reports should then be analyzed and discussed with USAID/ET, CRS/ET and counterpart staff. CRS/ET staff have already undertaken a number of initiatives to address the recommendations presented in these reports. These include the development of a new five-year strategic plan, potable water and sanitation strategy, integrated watershed management strategy, microfinance business plan, and EMC business plan, among others. The team will need to obtain copies of these new documents as part of their assessment. Task #2: Concisely assess CRS/ET progress toward achieving the goal, objectives, and intermediate results of the DAP. Accomplishing this task will require the Evaluation Team to obtain copies of the annual results reports. These reports should then be analyzed and discussed with USAID/ET, CRS/ET and counterpart staff, with a view to evaluating project outputs and outcomes vis-à-vis annual targets and baselines, and in the context of the historical operating environment. The Evaluation Team is also expected to review the financial and commodity inputs, with the aim of developing a cost￾benefit analysis of DAP activities. In other words, CRS/ET management wants to know whether project results have justified its costs. It is important that the Evaluation Team consider both quantitative, as well as qualitative, results. Task #3: Use results and analysis from the first two tasks to provide guidance and recommendations on specific strategies CRS/ET should incorporate into the next DAP proposal so that overall program quality and impact will improve. CRS/ET management considers this final task to be the most important as it is the one which will be most directly used by program staff to improve program quality in the coming years. As such, the Evaluation Team should ideally limit its investment of time and energy on the first two tasks to that which can directly link into this task. A review of the new strategic plan, potable water and sanitation strategy, integrated watershed management strategy, microfinance business plan, and EMC business plan should provide the Evaluation Team with an indication of the direction the program is moving. The strategies and activities presented in these documents should be analyzed for their appropriateness, feasibility, sustainability, and potential for positive impact on food security and nutrition at the rural household level. The Evaluation Team should also familiarize itself with the USAID/ET Mission’s new integrated strategic plan (ISP), copies of which will be made available. It is important that the strategies CRS/ET adopts for the new DAP fit into the framework of the Mission’s ISP. IV. METHODOLOGY The final evaluation should be a learning process which is utilization-focused. [A comprehensive description of UFE has been developed and published by Michael Quinn Patton.] In this instance, CRS/ET wants to use this final evaluation as a means of working with its partners, both USAID and implementing counterparts, to assess current programming strategies and results with a view to improving them in the future. Since a number of evaluations have already been conducted, the Evaluation Team will not be expected to collect much data. Rather, the emphasis should be on an extensive review and analysis of existing reports from a broader perspective. The Results Reports for FY99 and FY00, as well as the case study on Lege Oda Mirga and the community impact briefs from East Hararghe zone, should provide most of the updated information needed to evaluate post-MTE activities of the DAP. This review should be accompanied by extensive discussions with CRS/ET staff, counterpart staff, USAID/ET staff, and Government officials. Field visits to at least two counterparts should be conducted during the course of the evaluation. The Team Leader will provide overall leadership and coordination to the Evaluation Team. Following consultations with other team members and CRS/ET management, s/he will be 24 May, 2001 Final Evaluation Report 95 responsible for preparing a detailed work plan for review and approval by CRS/ET before the start of the evaluation. S/he will facilitate all processes needed to complete all three tasks, and will be responsible for writing and presenting the final report at a meeting attended by both CRS/ET and USAID management. Two Sectoral Specialists, one for agriculture/natural resource management and one for health/emergency mitigation, will be responsible for taking the lead in their respective areas. They will need to work closely with the Team Leader to ensure that their analysis is focused on accomplishing the tasks. CRS/ET will provide support staff to participate in the final evaluation when requested and as appropriate. These staff will primarily include the Asst. Country Representative for Program Quality, the Head of Program Dept., and the Head of Program Quality Unit. In addition, a point person will be assigned from the Administration Dept. to coordinate logistical and administrative issues for the team. CRS/ET will be responsible for all in￾country transportation needed to conduct the evaluation. V. TIMEFRAME While evaluation dates have yet to be confirmed, it is proposed that the evaluation be limited to 2 days for preparation, 14 days for implementation, and 4 days for final report writing/presentation/revision, making a total of 20 days. This period should take place preferably during the April 1-21st, 2001 period, or alternately during the May 10th -31st period. VI. REPORT PREPARATION A draft report will be prepared and submitted on the day of the final presentation. At that point, USAID and CRS/ET management will have the opportunity to provi de any feedback for incorporation into the final report, which is due no later than one week after the completion of the evaluation. VII. OBLIGATIONS OF CRS/ET CRS/ET agrees to the following obligations for the final evaluation: 1. Meet the Evaluation Team and provide initial briefing 2. Provide in-country transport facilities associated with the evaluation 3. Pay consultancy fees as per the terms of the contract 4. Provide documents and information as necessary 5. Include evaluation findings in the next CSR4 report. VIII. COMPOSITION OF THE EVALUATION TEAM The core Evaluation Team is proposed to comprise the following professionals: 1. One Team Leader (expatriate) with extensive food security and evaluation experience 2. One Ag/NRM expert (expatriate) with extensive field experience in integrated watershed management, particularly in East Africa 3. One Health/Emergency Mitigation specialist with extensive field experience in community-based health care and nutrition programming, particularly in Ethiopia Detailed terms of reference for each core team member are provided in the appendix. In addition to the core Evaluation Team, a variety of CRS/ET staff and other stakeholders will participate in the process to ensure that local experiences and attitudes are fully taken into account. 24 May, 2001 Final Evaluation Report 96 APPENDIX Roles and Responsibilities of Team Leader: · Provide overall coordination for the evaluation, team processes and outputs · Facilitate review of all issues with CRS/ET senior management in a pre-evaluation briefing meeting · Refine specific TOR for each team member if necessary and as appropriate · Facilitate review of lessons learned and next-steps with CRS/ET senior management in a final debrief meeting · Review reports generated to date by the program (Mid Term Evaluation, SSI PEA, Potable Water study, WSC Evaluation, EMC Evaluation, Lege Oda Case Study, FY99 Results Report, FY00 Results Report, draft Potable Water and Sanitation Strategy, draft Integrated Watershed Management Strategy, MFI business plan, EMC business plan) · Assess validity of data for discussions with sector specialists and CRS staff · Summarize key elements of the data for the final report · Deal with strategic issues that cut across sectors, most notably: the use of food aid; the level of intra- and inter-sectoral integration; the appropriateness of geographic targeting; the impact of the DAP on nutrition; and the appropriateness, quality and accuracy of the M&E system · Evaluate the MFI business plan and EMC business plan with a view to providing recommendations on how they can be improved. · Meet with key CRS, USAID, and CPT officials on the above issues · Write final report with input from sectoral specialists. AG/NRM Sector Specialist · Review reports generated to date by the program (Mid Term Evaluation, SSI PEA, Potable Water study, WSC Evaluation, EMC Evaluation, Lege Oda Case Study, FY99 Results Report, FY00 Results Report, draft Potable Water and Sanitation Strategy, draft Integrated Watershed Management Strategy, MFI business plan, EMC business plan) · Assess validity of data for discussion with CRS, USAID, and CPT staff · Coordinate analysis of the AG/NRM sector through document review, field visits, and stakeholder discussions with a view to developing a list of lessons learned and recommendations · Evaluate the progress CRS/ET has made over the life of the DAP in adopting a genuine integrated watershed management approach to improving food security. Focus specifically on small-scale irrigation, impact on income/markets, and impact on nutrition. Document the successes and where the gaps/problems are, with recommendations on how to fill the gaps and remedy the problems. · Evaluate the CRS/ET draft integrated watershed management strategy with a view to providing recommendations on how it can be improved. · Evaluate the CRS/ET draft potable water and sanitation strategy with a view to providing recommendations on how it can be improved. · Provide guidance on the best way CRS/ET can improve the cash crop component of its draft integrated watershed management strategy, with specific emphasis on sesame, chickpea, groundnut, and sweet potato. · Summarize methods used, key data, lessons learned and recommendations (action plan) for final written report. Health/Emergency Mitigation Sector Specialist · Review reports generated to date by the program (Mid Term Evaluation, SSI PEA, Potable Water study, WSC Evaluation, EMC Evaluation, Lege Oda Case Study, FY99 Results Report, FY00 Results Report, draft Potable Water and Sanitation Strategy, draft Integrated Watershed Management Strategy, MFI business plan, EMC business plan) 24 May, 2001 Final Evaluation Report 97 · Assess validity of data for discussion with CRS, USAID, and CPT staff · Coordinate analysis of the HE/Emergency Mitigation sectors through document review, field visits, and stakeholder discussions with a view to developing a list of lessons learned and recommendations · Evaluate the progress CRS/ET has made over the life of the DAP in adopting a genuine community-based approach to improving health. Focus specifically on sustainability aspects (such as TBA/CHA remuneration, linkage of food aid with health education, water management systems, etc) and impact on nutrition. Document the successes and where the gaps/problems are, with recommendations on how to fill the gaps and remedy the problems. · Evaluate how well DAP projects (AG,HE,WSC,NRM) are designed to promote disaster mitigation. Document the successes and where the gaps/problems are, with recommendations on how to fill the gaps and remedy the problems. · Evaluate the CRS/ET draft potable water and sanitation strategy with a view to providing recommendations on how it can be improved. · Evaluate the CRS/ET M&E system with a view to providing recommendations on how it can be improved. · Summarize methods used, key data, lessons learned and recommendations (action plan) for final written report. 24 May, 2001 Final Evaluation Report 98 Annex B CATHOLIC RELIEF SERVICES / ETHIOPIA PROGRAM FINAL EVALUATION OF THE FY97-01 DAP SCHEDULE Date Activities 1 April Arrival of Guy Sharrock (Team Leader) and Tom Remington (CRS/ETARO-DRD) 2 April 10.30a.m.: Briefing by CRS/ET management 3 April Travel to Adigrat / Adigrat Catholic Secretariat 6 April 1.30p.m.: Return to Addis Ababa Arrival of Kari Egge (CRS/EARO) and Debra Brosnihan (CRS/Kenya) 6.30p.m.: Dinner at CR’s house 7 April Team meetings and document review 8 April Departure of Tom Remington 9 April 9a.m.: Meeting with USAID/ET Mission 1.30p.m.: Travel to Dire Dawa / HCS 10 April Travel to Gorogutu and Meta field sites 11 April Travel to Harar and Dire Dawa field sites 12 April p.m.: Return to Addis Ababa 13-16 April Team discussions and report writing 10.30a.m.: brunch at ACR’s house 17 April a.m.: Travel to East Shewa / Meki Catholic Secretariat 18 April p.m.: Initial presentation to CRS/ET and implementing partners 19 April a.m. Continue discussions with CRS/ET and implementing partners p.m. Report writing 20 April a.m.: Final debriefing with CRS/ET management and USAID/ET 21 April Finalise report 22 April Submit draft report Departure from Addis Ababa