Catholic Relief Services/Rwanda Development Assistance Program (DAP) Food Security Program CRS/RW Project Number 676-00003 Transfer Authorization Award No. FFP-A-00-00-00086-00 Final Evaluation REPORT 11 January 2005 FY 2001 - FY 2005 DAP Final Evaluation Report 11 January 2005 2 Acknowledgement The team of external evaluators would like to gratefully acknowledge the support provided by the staff of CRS Rwanda. In particular, we would like to thank the following persons who provided extensive support to the evaluation team and ensured that the conclusions reached were grounded in reality. Emilienne Kayitasirwa, Project Coordinator Safety Net/LIFE Andre Gihanza, Project Coordinator AIDS Relief Juvenal Rubayiza, Project Coordinator, OVC Jean-Baptiste Hategikimana, Head of Capacity Building and Marais Development Justin Nsengimana, Deputy DAP Manager, Caritas Kibuye Noelle Nkuranziza, Head of Infrastructure & Irrigation Bethany Cole, CRS Fellow We would also like to thank Jean Nyemba, DAP Manager, and Kossi Kpogo, Assistant Country Representative, and Sean Gallagher, Country Representative for their prompt and complete support in implementing the evaluation. Finally, innumerable other staff of CRS, representatives of the DAP partners, and a multitude of DAP participants and beneficiaries provided useful insights, guidance, or logistical support for the evaluation. Our gratitude goes out to them as well. Sincerely, Michael D. DeVries, TANGO International Professor Abel Dushimimana, Consultant Dr. Marshall D. Ashley, Agricultural Consultant Serge Rwamasirabo, Rural Action FY 2001 - FY 2005 DAP Final Evaluation Report 11 January 2005 3 The FY 2001-2005 Development Assistance Program (DAP) FINAL EVALUATION TABLE OF CONTENTS Page I. EXECUTIVE SUMMARY 1 II. BACKGROUND 3 III. PROGRAM PROCESS ASSESSMENT 4 A. Program Management 4 1. Overview of the Management of the DAP 2. Implementation of MTE Recommendations Related to Program Management 3. Observations Relative to Program Management Vision and Leadership Financial Management Human Resources Communication Operating Agreements Program Support 4. Major Lessons Learned - Management B. Natural Resources Component 8 1. Overview of Implementation Strategies for IR 1.1 (Lowland Productivity) 2. Implementation of MTE Recommendations Related to IR 1.1 3. Observations Relative to Implementation Processes for IR 1.1 4. Major Lessons Learned Related to Implementation Processes for IR 1.1. 5. Overview of Implementation Strategies for IR 1.2 (Hillside Erosion) 6. Implementation of MTE Recommendations Related to IR 1.2 7. Observations Relative to Implementation Processes for IR 1.2. 8. Major Lessons Learned Related to Implementation Processes for IR 1.2. 9. Overview of Implementation Strategies for IR 1.3 (Livestock) 10. Implementation of MTE Recommendations Related to IR 1.3 11. Observations Relative to Implementation Processes for IR 1.3 12. Major Lessons Learned Related to Implementation Processes for IR 1.3 13. Overview of Implementation Strategies for IR 1.4 (Productive Technologies) 14. Implementation of MTE Recommendations Related to IR 1.4 15. Observations Relative to Implementation Processes for IR 1.4 16. Major Lessons Learned Related to Implementation Processes for IR 1.4 17. Overview of Implementation Strategies for IR 1.5 (Associations) 18. Implementation of MTE Recommendations Related to IR 1.5 19. Observations Relative to Implementation Processes for IR 1.5 20. Major Lessons Learned Related to Implementation Processes for IR 1.5 21. Overview of Implementation Strategies for IR 1.6 (Market Access) 22. Implementation of MTE Recommendations Related to IR 1.6 23. Observations Relative to Implementation Processes for IR 1.6 24. Major Lessons Learned Related to Implementation Processes for IR 1.6 C. Safety Net Component 23 1. Overview of the Safety Net Component 2. Implementation of MTE Recommendations Related to the Safety Net Component 3. Observations Relative to the Safety Net Component Safety Net Institutions Capacity Building FY 2001 - FY 2005 DAP Final Evaluation Report 11 January 2005 4 Income-Generating Activity(IGA) Grants Commodities Targeting Communications 4. Lessons Learned - Safety Net Component D. LIFE Component 26 1. Overview of the LIFE Component 2. Implementation of MTE Recommendations Related to the LIFE Component 3. Observations Relative to the LIFE Component Commodities Capacity Building Other Observations 4. Lessons Learned - LIFE Component E. Program Integration 29 1. Overview of Program Integration 2. Implementation of MTE Recommendations Related to Integration 3. Observations Relative to Integration Integration Between the Natural Resource and Safety Net/LIFE Components of the DAP Integration Between the Natural Resource Components in Butare and Kibuye Integration of the DAP with Wider Development Strategies 4. Lessons Learned - Integration F. Partnership 32 1. Overview of Partnerships in the DAP 2. Implementation of MTE Recommendations Related to Partnerships 3. Observations Relative to Partnerships Caritas Kibuye LIFE Partners (ANSP+, CARE. Caritas Rwanda and World Relief) ACDI/VOCA Government Partners ISAR 4. Lessons Learned - Partnerships G. Commodity Management 33 1. Overview of Commodity Management 2. Implementation of MTE Recommendations Related to Commodity Management 3. Observations Relative to Commodity Management Ration Composition Clearing and Forwarding in Mombassa. Commodity Losses Regulation 11 Compliance 4. Lessons Learned - Commodity Management H. Monitoring and Evaluation 35 1. Overview of Monitoring and Evaluation 2. Implementation of MTE Recommendations Related to Monitoring and Evaluation 3. Observations Relative to Monitoring and Evaluation Commodity Monitoring Monitoring for Reporting Purposes Monitoring for Programmatic Decision-Making. Evaluation 4. Lessons Learned - Monitoring and Evaluation I. Environmental Monitoring and Impact Mitigation 37 1. Overview of Environmental Monitoring and Impact Mitigation 2. Observations Relative to Environmental Monitoring and Impact Mitigation 3. Lessons Learned - Environmental Monitoring and Impact Mitigation FY 2001 - FY 2005 DAP Final Evaluation Report 11 January 2005 5 IV. PROGRAM IMPACT ASSESSMENT 39 A. Natural Resource Component 39 1. Outputs Produced Relative to IR 1.1 2. Quality of Outputs Produced Under IR 1.1 3. Outcomes: Scale of Impact on Food Security (and Livelihood Security) from Outputs Produced Under IR 1.1 4. Who has Benefited from Outputs Produced Under IR 1.1? 5. Outputs Produced Relative to IR 1.2 6. Quality of Outputs Produced Under IR 1.2 7. Outcomes: Scale of Impact on Food Security (and Livelihood Security) from Outputs Produced Under IR 1.2 8. Who has Benefited from Outputs Produced Under IR 1.2? 9. Outputs Produced Relative to IR 1.3 10. Quality of Outputs Produced Under IR 1.3 11. Outcomes: Scale of Impact on Food Security (and Livelihood Security) from Outputs Produced Under IR 1.3 12. Who has Benefited from Outputs Produced Under IR 1.3? 13. Outputs Produced Relative to IR 1.4 14. Quality of Outputs Produced Under IR 1.4 15. Outcomes: Scale of Impact on Food Security (and Livelihood Security) from Outputs Produced Under IR 1.4 16. Who has Benefited from Outputs Produced Under IR 1.4? 17. Outputs Produced Relative to IR 1.5 18. Quality of Outputs Produced Under IR 1.5 19. Outcomes: Scale of Impact on Food Security (and Livelihood Security) from Outputs Produced Under IR 1.5 20. Who has Benefited from Outputs Produced Under IR 1.5? 21. Outputs Produced Relative to IR 1.6 22. Quality of Outputs Produced Under IR 1.6 23. Outcomes: Scale of Impact on Food Security (and Livelihood Security) from Outputs Produced Under IR 1.6 24. Who has Benefited from Outputs Produced Under IR 1.6? B. Safety Net Component 48 1. Outputs Produced 2. Quality of Outputs 3. Outcomes: Scale of Impact on Food Security (and Livelihood Security) 4. Who has Benefited? C. LIFE Component 51 1. Outputs Produced 2. Quality of Outputs 3. Outcomes: Scale of Impact on Food Security (and Livelihood Security) 4.Who has Benefited? V. RECOMMENDATIONS FOR THE DESIGN OF NEXT DAP 52 A. Key Principles to Guide the Development of the DAP 52 Use of Food Being More Participant Driven Livelihoods Programming Targeting Remember - Do No Harm Exit Strategies Monitoring and Evaluation Participants and Beneficiaries B. Building on the Foundation 54 FY 2001 - FY 2005 DAP Final Evaluation Report 11 January 2005 6 Natural Resources In Butare Natural Resources in Kibuye Safety Net LIFE Integration VI. CONCLUSION 56 ANNEXES 58 Annex A: Final Evaluation Scope of Work Annex B: Evaluation Methodology Annex C: Detailed Analysis of the CRS Response to MTE Recommendations Annex D: LOA Expenditures Summary Annex E: Safety Net IGA Grants Annex F: Safety Net Evaluation Survey Summaries Annex G: LIFE Evaluation Survey Annex H: Commodity Detailed Summaries LIST OF TABLES AND FIGURES Executive Summary Table 1. Assessment of Achievement by IR 2 Executive Summary Table 2. Assessment of Impact Against Program 2 Strategic Objectives Table 1. Key Dates for the CRS DAP (Food Assistance Program) 3 Figure 1. Implementation Phases for the FY 2000-2005 DAP 5 Table 2. LOA Cash Expenditure Summary (US$) 6 Table 3. Strategic Objectives and Intermediate Results for the Safety Net Component 22 Table 4. Strategic Objectives and Intermediate Results for the LIFE Component 26 Table 5. Projected LOA Commodity Summary (MT) 32 Table 6. LOA Commodity Loss Summary through FY 04 Shipments Received (MT) 33 Table 7. Summary of the Number of Food Recipients in the Safety Net Component 48 Table 8. Disaggregation of Food Recipients in Safety Net Institutions 50 Table 9. Summary Assessment of Achievement by IR 55 Table 10. Summary Assessment of Impact Against Program Strategic Objectives 56 FY 2001 - FY 2005 DAP Final Evaluation Report 11 January 2005 7 LIST OF ACRONYMS AER Annual Estimate of Requirements AIDS Acquired Immuno-Difficiency Syndrome ANSP+ Association Nationale de Soutien aux Personnes Victime de VIH/SIDA CAHW Community Animal Health Worker CEW Community Extension Worker C & F Cost and Freight CDC Community Development Committee CIP International Potato Center CRS Catholic Relief Services CS Cooperating Sponsor CSB Corn Soy Blend DAP Development Assistance Program DIP Detailed Implementation Plan FFP Food for Peace FFW Food for Work FY Fiscal Year GOR Government of Rwanda HIV Human Immuno-Difficiency Virus ICRAF International Council on Research in Agroforestry ICRISAT International Crops Research Institute for the Semi-Arid Tropics IEE Initial Environmental Examination IFP Institutional Feeding Program IGA Income Generating Activity IITA International Institute of Tropical Agriculture ILRI International Livestock Research Institute IR Intermediate Result IRRI International Rice Research Institute ISAR Institute des Sciences Agronomique du Rwanda ISP Integrated Strategic Plan IWMI International Water Management Institute LIFE Leadership Initiative for Fighting Epidemics LOA Life of Activity M&E Monitoring and Evaluation MINAGRI Ministry of Agriculture MINALOC Ministry of Local Governance and Social Affairs MINITERRE Ministry of Lands, Reinstallation and Environmental Restoration MOU Memorandum of Understanding MT Metric Ton MTE Mid-Term Evaluation NICRA Negotiated Indirect Cost Rate Agreement NGO Non-Governmental Organization NRM Natural Resource Management OVC Orphans and Vulnerable Children PASAR Projet de Securité Alimentaire au Rwanda PHAST Participatory Hygiene and Sanitation Transformation FY 2001 - FY 2005 DAP Final Evaluation Report 11 January 2005 8 PL 480 Public Law 480 PLWHA People Living With HIV/AIDS PRA Participatory Rural Appraisal REO Regional Environmental Officer SMC Source Management Committee SN Safety Net SO Strategic Objective TANGO Technical Assistance to NGOs TAP Transition Assistance Program USAID United States Agency for International Development USG United States Government WFP World Food Program FY 2001 - FY 2005 DAP Final Evaluation Report 11 January 2005 9 The FY 2001-2005 Development Assistance Program (DAP) FINAL EVALUATION I. EXECUTIVE SUMMARY CRS commissioned a final evaluation of its FY 2001-2005 Development Assistance Program (DAP) in December of 2004, seven months before the completion of the program. This report summarizes the findings of the evaluation which investigated the outputs produced by the program, the response of CRS to recommendations from a Mid-Term Evaluation, the outcomes resulting from the outputs produced, who actually benefited from the program, the quality of the processes used to produce the outputs, and major lessons learned from implementation of the program. The evaluation was conducted by a four person team over the period December 1 through 21, 2004 The FY 2001-2005 DAP has three components, a natural resource component implemented in two locations, Butare and Kibuye, a safety net component implemented with safety net institutions across the country and an HIV/AIDS component called the Leadership Initiative for Fighting Epidemics (LIFE) that was implemented through four sub-recipient partners, covering most of the country. The total value of the program was initially projected to be around US$ 33.7 million of which two-thirds of the value was to be invested in commodities to be used for direct distribution in the Safety Net and LIFE components, for food for work (FFW) in the natural resource components and for monetization. By the end of the program in July 2005, it is projected that 96.7% of the proposed food resources will have been distributed and 84% of the proposed cash budget spent. Over its life, the DAP proceeded through four implementation phases. The first phase, or start-up phase, lasted through the end of calendar year 2001. The second phase described as the initial implementation phase of the program began in January of 2002 and lasted until the time of the Mid-term Evaluation (MTE) which was conducted in the middle of 2003. The third phase involved six months of negotiations in the latter half of 2003 between CRS and USAID to decide how to proceed with the program following the MTE. The final and current phase of the program began in January 2004 to present and represents a second implementation phase. Over these phases the Safety Net and LIFE components scaled up fairly quickly to full implementation with distributions of food. The natural resource components implemented directly by CRS in Butare and by Caritas Kibuye in Kibuye both got off to slow starts. The program in Kibuye continue to lag even up to the time of the MTE. The MTE was highly critical of the program and made over sixty recommendations for mid-course corrections. One of the recommendations made by the MTE was to recruit a DAP Manager. This was done in August of 2003 and for the period from then until the end of the year, CRS and USAID reached agreement on how to proceed with improving the implementation of the program. From January 2004 to present, the DAP has been able achieve significant impact in all components. The tables on the following page summarize the findings relative to performance against the eight Intermediate Results (IRs) and how achievements on IRs have produced impact at the strategic objective level. In the judgment of the final evaluation team, the investments made through the DAP in the Safety Net and LIFE components have generated satisfactory returns over the life of the program. The investments made in the natural resource components in FY 2001 - FY 2005 DAP Final Evaluation Report 11 January 2005 10 Butare and Kibuye, however, have not generated satisfactory returns over the life of the activity, mainly because relatively little was achieved over the first three and a half years of Executive Summary Table 1. Assessment of Achievement by IR Intermediate Result Summary Assessment of Achievement Natural Resources Component IR1.1: Increased lowland productivity for 15,000 families in targeted areas by 2005. The targets were not achieved on this IR relative to the number of beneficiaries but a good foundation has been established and benefits will continue to accrue IR1.2: Reduced erosion from steep lands for 28,500 families in targeted areas by 2005. The targets were not achieved on this IR relative to the number of beneficiaries but a good foundation has been established and benefits will continue to accrue IR1.3: Selected farms in targeted areas restocked with 13,500 goats and swine by 2005 Very little was achieved on this IR. IR1.4: Productivity enhancing technologies introduced to 15,000 families by 2005. Good demonstrations have been established by the program, but the program is not reaching 15,000 families IR15: 75 farmers’ associations established or strengthened in targeted areas by 2005. Most associations have been established and are functioning IR16: Improved market access for 43,500 families in targeted areas by 2005. Roads and bridges have been repaired, but the ownership and maintenance of the infrastructure is unclear and the number of small-holder farm families benefiting is far below the target. Safety Net IR 2.1: Increased food security for 7,000 institutionalized orphans, street children, elderly, and handicapped persons; and 4,400 vulnerable households affected by HIV/AIDS by 2005. This has been fully achieved through food distributions made over the full life of the program. IR 2.2: Strengthened capacity for self-sufficiency among partner institutions caring for unaccompanied children, orphans, HIV/AIDS victims, and other vulnerable people by 2005. Some progress has been made in capacity building but safety net institutions are not self-sufficient. LIFE IR 2.1: Increased food security for 7,000 institutionalized orphans, street children, elderly, and handicapped persons; and 4,400 vulnerable households affected by AIDS by 2005. This has been fully achieved through food distributions made over the full life of the program. IR 2.2: Strengthened capacity for self-sufficiency among partner institutions caring for unaccompanied children, orphans, HIV/AIDS victims, and other vulnerable people by 2005. Some progress has been made in capacity building, especially through complementary programs implemented by sub-recipient partners. Executive Summary Table 2. Assessment of Impact Against Program Strategic Objectives Strategic Objective Summary Assessment of Impact SO1: To increase the ability of 43,500 smallholder families in targeted areas to ensure their food security by 2005. During the program’s life, not much impact had been achieved until CY 2004, but benefits will accrue after the program ends. SO2: To ensure increased food security for 29,000 of Rwanda’s most vulnerable people each year 2001-2005. There has been significant impact on short term food insecurity, but sustained food security has not been achieved. FY 2001 - FY 2005 DAP Final Evaluation Report 11 January 2005 11 the program. The last year of implementation has yielded significant impact, but these returns do not sufficiently offset the cost of five years of implementation. The main factors accounting for the relatively poor performance in the early phases of the DAP were the absence of strong leadership in the program and weak monitoring and evaluation (M&E) systems that did not enable the program to evolve and respond to experience and changes in the operating context. II. BACKGROUND The Title II Development Assistance Program (DAP), the Food Security Program1 , being implemented by CRS officially began implementation on August 1, 2000, with an expected completion date of July 31, 2005. The program is composed of three components, (1) a natural resource component implemented in two parts of the country, Butare and Kibuye, (2) a component called the Leadership Initiative for Fighting Epidemics (LIFE) working with associations of people living with HIV/AIDS (PLWHA) and (3) a safety net component working directly with orphanages, centers for street children, centers for the disabled, and homes for the aged across the country. The goal of the program is to increase food security in Rwanda for small-holder farm families and for Rwanda's most vulnerable. The total program cost at approval was estimated to reach $33,733,235, including $22,868,500 for commodities (C&F), $7,760,300 for inland transport, $1,998,395 for 202e, and cost sharing from CRS of $1,106,040. The cost sharing was later adjusted to $563,050. The five year program was approved by the Ministry of Finance on June 8, 2000. Table 1 lists the key dates in the life of the program Table 1. Key Dates for the CRS DAP (Food Assistance Program) Date Submission/Event 19 April 2000 DAP proposal submitted to USAID Rwanda 8 June 2000 Program approved by the Ministry of Finance 10 July 2000 Transfer Authorization (TA) received by CRS 1 August 2000 Official start date as per the TA 1 October 2000 Start date as per the Project Document being used by CRS Program Management 3 October 2000 Transfer Authorization signed by CRS 1 January 2001 Actual Implementation commences 3 January 2002 Detailed Implementation Plan (DIP) Revised 14 May to 15 June 2003 Mid-Term Evaluation Implemented By 1 August 2003 Final Mid-Term Evaluation Report Received by CRS 1 August 2003 DAP Manager Position Filled Late August 2003 CRS Response to the Mid-term Evaluation Report submitted to USAID with a request for a one year no-cost extension and shift in commodities from FFW to LIFE 16-24 October 2003 Field Visit by USAID 13 November 2003 Letter to CRS with recommendations from field trip and rejection of no￾cost extension request. 2 December 2003 Revised Targets Implementation Plan submitted to USAID Late December 2003 Revised Targets and Budget Approved by USAID 4 February 2004 Cooperative Agreement revised with approvals for FY 2004 and 2005 commodities and 202e 1-20 December 2004 Final Evaluation implemented 10 January 2004 Final Evaluation report to be received by CRS 31 July 2005 Completion Date for the DAP 1 While the DAP has been given a name, the universal convention with CRS Rwanda and its partners is to refer to the program as the DAP. FY 2001 - FY 2005 DAP Final Evaluation Report 11 January 2005 12 This report summarizes the findings of a final evaluation conducted for the program over the period December 1 through 20, 2004. A four person team of external evaluators was commissioned to review the program to assess impact and identify lessons learned to inform the design of another DAP proposal to be submitted to Food for Peace (FFP) for FY 2006. Annex A contains the Scope of Work for the evaluation. The evaluation team focused the investigation on (1) the processes used in implementing the program, (2) the implementation of Mid-Term Evaluation (MTE) recommendations, (3) the outputs produced by the program viewed both quantitatively and qualitatively, (4) the outcomes resulting from the outputs, (5) the beneficiaries of program interventions and (6) key lessons learned from the program. Some key features of the methodology used in the evaluation were: · The choice of work sites and institutions to visit by the evaluation team was made randomly. · Information was gathered using a number of different techniques, including in-depth interviews with key persons (sometimes multiple interviews), key informant interviews, focus group discussions, observation, and document review. · An iterative process in which information was gathered and analyzed over at least two cycles was used. · A rapid quantitative survey was conducted in the safety net and LIFE components to be used to triangulate with qualitative information. Annex B contains documentation describing the methodology used in the evaluation, including descriptions of methodologies, work schedules, tools used, and persons interviewed. In the next section, the report documents the observations of the evaluation team relative to the project processes used, including those for managing the program, for implementing each of the programmatic components, for integrating the program internally and externally, for commodity management, for monitoring and evaluation, and for environmental monitoring and impact mitigation. Section IV of the report summarizes the impact of the program and Section V proposes recommendations for the design of the next DAP. In the conclusion provided in the last section of the report, the evaluation team summarizes the findings. III. PROGRAM PROCESS ASSESSMENT A. Program Management 1. Overview of the Management of the DAP Two components of the DAP evolved from previous activities implemented by CRS. The natural resource component of the program was built upon a foundation established by a Transition Assistance Program (TAP), and the safety net component followed on an Institutional Feeding Program (IFP). The LIFE component evolved out of a separate program design process. The natural resource component of the program targets different beneficiaries (small-holder farm families) than the safety net and LIFE components (the most vulnerable). At the beginning of the program, separate management structures were established for the program. The management of the Safety Net/LIFE component became the responsibility of a full time Safety Net Manager, later to become a Safety Net Project Coordinator. This position has been filled by the same individual from the beginning of the program. Responsibility for the natural resource component initially rested with a part time Agriculture FY 2001 - FY 2005 DAP Final Evaluation Report 11 January 2005 13 Team Leader (40%) who supervised a DAP Agricultural Coordinator assigned full-time to the program and who resided in Butare. Overall management of the DAP when it began implementation rested with the Assistant Country Representative who was allocated 40% to the program. The life of the DAP up to the present can be described as having passed through four phases. Figure 1 provides a schematic to illustrate the phases. The first phase, or start-up of the program, comprised the first year of implementation from January through December of 2001. During this period, the safety net component began disbursing food fairly early in the period since many of safety net centers had already been receiving support through the IFP. The LIFE component also began implementation fairly early in the period since three of the sub-recipient partners, CARE, Caritas Rwanda2 and World Relief, already had programs on the ground. The fourth partner, ANSP+ (Association Nationale de Soutien aux Personnes Victime de VIH/SIDA), began implementation the following year. The natural resource component, however, got off to a slow start in both Butare where CRS was delivering services directly and in Kibuye where implementation of the program was being done in collaboration with Caritas Kibuye. The second phase of the program, consisting of the second year (calendar year 2002) up to the time of the mid-term evaluation held in the middle of 2003 can be described as the first implementation phase. In the safety net and LIFE components, the program continued to deliver food, and capacity building activities with participant organizations expanded. In the natural resource component, CRS realized that progress during the start-up phase of the program was not satisfactory. The Detailed Implementation Plan (DIP) was revised and CRS made a push to get progress back on track. The Mid-Term Evaluation (MTE) was conducted from mid-May to mid-June and represents a major turning point in the program. The MTE was highly critical of the program and made nearly sixty recommendations for mid-course corrections. One of the major recommendations from the MTE was to recruit a DAP Manager and to clarify the management structure of the program. In August 2003, shortly after the receipt of the MTE report, CRS put in place a new management structure which included a DAP Manager. The six-month period after the MTE until the end of calendar year 2003 represents the third phase of the program which can be described as a negotiation phase in which, under the leadership of the new DAP Manager, changes in the program based on the MTE evaluations were discussed and agreed upon with USAID. During this phase, program activities in the field were reduced back to basically maintenance activities, i.e., continuing food distributions in the safety net/LIFE components and maintaining project relations and capacities in the agricultural component while discussions were held with the donor. The final phase of the program comprised of calendar year 2004 can be described as the second implementation phase. During this phase significant progress was made in all components. Given the number of recommendations made by the MTE, it would have been nearly impossible to completely address them all. Nevertheless, the DAP made significant progress in addressing many ofthem and made at least some progress in addressing nearly all of them. Figure 1. Implementation Phases for the FY 2000-2005 DAP SSSSSSSSSSSS111111111111111111NNNNNN222222222222……. 1 Jan 2001 1 Jan 2002 1 July 2003 1 Jan 2004 20 Dec 2004 31 July 2005 2 The DAP has collaboration with Caritas at two levels. The LIFE activities are being implemented with Caritas Rwanda based in Kigali. The agricultural activities in Kibuye are being undertaken with Caritas Kibuye. FY 2001 - FY 2005 DAP Final Evaluation Report 11 January 2005 14 Start-Up 1st Implementation Phase Negotiation 2nd Implementation Phase 2. Implementation of MTE Recommendations Related to Program Management Relative specifically to management of the DAP, the MTE made one recommendation at the program level and a number of recommendations relative to management in each of the components of the program. Annex C contains tables which summarize CRS's response to each of the MTE recommendations and an assessment by the evaluation team as to the quality of progress on each recommendation. On the critical program level recommendation which was to recruit a DAP Manager and restructure the senior staffing of the program, CRS did promptly recruit a DAP Manager and define a new management structure for the program, and the impact of this on the quality of the program has been substantial. Other important positions in the new management structure, however, namely the Agri-Business Coordinator and the M&E Coordinator, have been either plagued with a high turnover rate (the former) or have been affected by indecision (the latter, explained in more depth in Section H). 3. Observations Relative to Program Management Program management covers a range of activities, including vision and leadership, financial management, human resource management, communications, operating agreements, and program support. The key observations that follow touch on these dimensions of management. Vision and Leadership. The DAP in its first two phases (start-up and the first implementation phase) was affected by lack of vision and leadership especially in the natural resource component, but even for the program as a whole. No one within CRS Rwanda during this period was delegated full-time with responsibility for overall vision and leadership for the DAP. At the very senior level in CRS Rwanda, the organization had difficulty placing a long-term, full-time Country Representative in Kigali. As a result, direction and leadership at the very top was less effective than had the organization a full-time country representative as it does now3 . After the MTE, a DAP Manager fully responsible for the entire program was recruited and implementation improved substantially in the last phase of the program as a result of the vision and leadership. Financial Management. Table 2 provides a projected summary of the Life of Activity (LOA) cash expenditure for the DAP. By the end of the program, it is expected that 84% of the approved budget will have been spent. The slow start in the first two phases of the program accounts for the under-expenditure. Apart from the projected under-expenditure, financial management in the program has been effective, although budget management in the past was highly centralized, also contributing perhaps to the under-expenditure. CRS has recently introduced a more decentralized budget preparation and management process that will enable managers in the field to more effectively manage budgets. Human Resources. The DAP Program has been affected by relatively high staff turnover. A number of positions are presently vacant and it has taken an inordinate amount of time to fill some positions. The following critical staff vacancies in the third and fourth phases of the project after the MTE have affected program implementation. o The M&E Coordinator position, vacant from October 2003 to present. 3 Recent efforts by the current Country Representative to implement a Strategic Planning Process (SPP) and to decentralize the budgeting process have been instrumental in empowering staff and providing a vision for program activities. FY 2001 - FY 2005 DAP Final Evaluation Report 11 January 2005 15 o The Agri-Business Coordinator position, vacant from June to November 2004, o The two technical staff positions in the Safety Net/LIFE component, vacant from April to September. o The Livestock Technician position w Table 2. LOA Cash Expenditure Summary (US$) Cost Center Monetization Proceeds 202e CRS Funds Total Safety Net 300,206 12,062 61,577 373,845 CRS 392,546 72 38,511 431,129 CARE 249,741 --- --- 249,741 World Relief 137,278 --- --- 137,278 Caritas Rwanda 96,616 --- --- 96,616 LIFE ANSP+ --- --- --- --- CRS 1,162,402 215,152 45,255 1,422,809 Agriculture Caritas Kibuye 359,3484 49,887 10,329 419,564 Administration 1,957,057 1,155,228 340,946 3,453,230 NICRA 919,156 218,751 0 1,137,907 ACDI (Monetization Costs) 70,339 0 0 70,339 TOTAL Expenses & Projected Expenses for LOA 5,644,689 1,651,151 496,619 7,792,459 TOTAL Approved Budget 6,838,977 1,893,217 541,795 9,273,989 Percent of Approved Budget Expected to be Spent 82.5% 87.2% 91.7% 84.0% Difficulties in recruiting and retaining staff seem to be a function of a relatively small pool of talent from which to recruit for highly technical or management positions and a compensation and benefit package that may not be competitive in the market5 . It was noted that the program has had a budget for staff development that has been consistently under-spent. While training should always be focused on enabling staff to perform their functions more effectively, it is also perceived to be a valuable benefit. A second observation relative to human resources explained in more detail below in Section C, pertains to the availability of human resources relative to the commitments made. In the case of the Safety Net/LIFE component, there were not enough human resources allocated in the program to achieve the proposed outputs for capacity building. Communications. Communications are critical at four levels, (i) within the program, (ii) with implementing partners, (iii) with program participants and (iv) with external contacts, particularly the donor. In general, communications within the program have been effective, and they improved substantially with the arrival of the DAP Manager in 2003. While there was some confusion in the latter half of 2003 as senior staff filling the new program structure defined their roles more clearly, staff in the program now generally feel that communications are good, although some feel that explanations of the rationale behind decisions made is not always communicated. 4 This includes CRS costs in Kibuye with one staff member working in the Caritas Kibuye office. 5 CRS has recognized the problem of competitiveness and is in the process of developing and implementing a comparative compensation and benefits survey. FY 2001 - FY 2005 DAP Final Evaluation Report 11 January 2005 16 With implementing partners, it appears that for those partners who have offices in Kigali, i.e., the LIFE partners, communications are good. Program staff are prompt in communicating with partners and are participatory in addressing issues. For partners located out of Kigali, however, i.e., Caritas Kibuye, communications and information sharing are felt to be mostly one-way. Communications are sent to CRS but the flow of feedback the other direction is much less. With program participants, communications at the field level between field-based staff and participants mainly in the natural resource component are perceived to be effective. In the safety net component in which CRS works directly with program participants, however, the program has not been as responsive and timely in sharing information and communicating with safety net centers, particular on matters that require formal, written communications. Some safety net institutions, for example, have submitted proposals for Income-Generating Activities (IGAs) as long ago as last year and are still unclear as to the status of the proposals. Communications with the donor also appear to be good. CRS has been reasonably prompt in responding to the findings from the MTE. The initial response to the MTE recommendations was made within a month of receiving the MTE report, even with the transformation of the program staff structure; and the revised DIP was submitted approximately three weeks after a response to the initial draft was received from USAID. Operating Agreements. The DAP got off to a slow start early in its life, especially in Kibuye in which implementation of the program was to be undertaken in collaboration with Caritas Kibuye. The absence of clear leadership for the DAP within in CRS and staff changes in Caritas Kibuye were major contributing factors to the delay. An operating agreement clearly specifying expectations, roles and responsibilities would have enabled the program to still make progress during this period. However, the first full operating agreement between Caritas Kibuye and CRS was drafted in November, 2003, reviewed and negotiated by Caritas Kibuye and CRS from December through April, 2004, and signed in May. Program Support. Program support, i.e., logistics, procurement, vehicle control and inventory management, did not emerge as a major constraint to program implementation and was effective in providing support to implementation. There are always ways to improve systems, however, and it was observed that centralized decision-making on administrative matters has resulted in relatively small details having to go to high levels. 4. Major Lessons Learned- Management A number of major lessons have emerged from the experience of the DAP relative to management of the program. ¸ An individual must be clearly delegated with full responsible for leadership, vision and overall management of a program. ¸ Continuity of staff is critical to being able to implement a program effectively. Persistent vacancies in staff positions can cripple a program ¸ It is critical to have clarity on expectations, roles and responsibilities with implementing partners to enable program activities to proceed through periods in which leadership or program management are in transition. B. Natural Resources Component The natural resource component of the DAP covered a range of interventions under six intermediate results in two locations, Butare and Kibuye, using different delivery channels. In Butare, CRS delivered natural resource program interventions directly. While in Kibuye, FY 2001 - FY 2005 DAP Final Evaluation Report 11 January 2005 17 program interventions were delivered through Caritas Kibuye. The following sections discuss implementation strategies, the implementation of MTE recommendations, evaluation team observations and lessons learned for each Intermediate Result. 1. Overview of Implementation Strategies for IR 1.1 Intermediate Result 1.1 Increased lowland productivity for 15,000 families in targeted areas by 2005. Butare. DAP interventions in Butare targeted watershed and marshland management in lowland marais areas in Maraba and Kiruhura which are among the poorest districts in the country. The package of interventions in Butare focused primarily on building water management infrastructure through food-for-work (FFW), facilitating equitable access to the arable land created through the infrastructure, and rehabilitating or constructing potable water and livestock water stations in the marais. CRS worked in partnership with the Agricultural Research Institute (ISAR) in Butare and private construction contractors as well as with technicians from the provincial and district government departments. In order to ensure the effective management and maintenance of the water management infrastructure built by the program, the DAP organized marais management committees composed of program beneficiaries. These committees were led by committee chairmen and supervised by local authorities at the sector level with the district mayor given responsibility for final resolution of disputes, mostly around disposition and ownership of parcels in the marais. In implementation FFW activities, CRS hired technicians and supervised the work while participants provided labor primarily for construction of drains, canals and bunds. Potable water management committees were formed by the program to take responsibility for management of the potable water and livestock watering points. The committees were charged with mobilizing resources from participants to undertake maintenance and repair activities. DAP staff tested water quality twice per year, and district water agents visited the sites regularly. Kibuye. Because of the late start of implementation in Kibuye, interventions in marais areas were not undertaken. The focus of the DAP under IR 1.1 was to increase productivity through rehabilitation of hillside irrigation systems. The rehabilitation was implemented through existing associations of farmers in the targeted hillsides. Sources of water were rehabilitated to supply water to these systems, and sweet potato cuttings were provided to increase farm production and livelihood security. 2. Implementation of MTE Recommendations Related to IR 1.1 Butare. The MTE made eight recommendations relative to activities under IR 1.1. Of these, three recommendations were completely accomplished (the hiring of a French-speaking irrigation engineer, verification of equitable plot distribution, and monitoring of water management committees). While the remaining recommendations were not completely fulfilled, significant progress was made on each. Annex C contains a table which summarizes CRS's response to each of the MTE recommendations in Butare and an assessment by the evaluation team as to the quality of progress on each recommendation. Kibuye. Of the eight recommendations made by the MTE under IR 1.1, three were applicable to activities being implemented in Kibuye and were included in the Caritas Kibuye workplan, including hiring a French speaking engineer be to manage the infrastructure FY 2001 - FY 2005 DAP Final Evaluation Report 11 January 2005 18 construction/rehabilitation, reducing the irrigation area to be rehabilitated to 50 hectares, and reviewing the use of FFW. The first and third recommendations were fully achieved. The second was partially achieved. Annex C contains a table which summarizes CRS's response to each of the MTE recommendations in Kibuye and an assessment by the evaluation team as to the quality of progress on each recommendation. 3. Observations Relative to Implementation Processes for IR 1.1 Butare. Following are the key observations made by the evaluation team relative to implementation processes related to IR 1.1 in Butare. * In implementing activities under IR 1.1 in Butare, it appears that it took an inordinate amount of time for the program to decide on priorities relative to lowland management. There is still work to be completed in blocks 1, 2, 3, 4, 5, and 6. The population is not informed when this is going to be done, and beneficiaries are impatient and ready to provide all the manpower required to complete the work. * While work was underway, technical responsibility rested with CRS. Because of a lack of experience and expertise within CRS, however, the program had difficulty monitoring effectively the technical quality of the work done by private contractors. * In terms of ownership and management of lowland areas, while there may have been problems initially, at the time of the evaluation, ownership of plots seems to have been done equitably, respected decision-making mechanisms to resolve disputes were in place, and local authorities were appropriately engaged. * Without any continuous CRS interventions, there are farmers who are continuing to grow rice in the wetlands that have been improved. * The training received by participants from the infrastructure team has been effective and farmers are capable of using and maintaining the water management systems. * Relative to the potable water points established, participants generally know the value of potable water and respect the installation that has been achieved by the program with an adequate level of ownership in the installation. The training on water point management provided by the program is highly appreciated by participants. * A diffusion effect has occurred between blocks. Owners of block 5 parcels are ready to provide free labor to finish all of the infrastructure development because they see that people in block 6 are earning income each season. * At the end of the program, some program participants expressed keen interest in working closely with CRS to complete the rest of the work. It also appears that if for some reason the work does not get completed, their interest will turn to significant disappointment. Kibuye. Following are the key observations made by the evaluation team relative to implementation processes related to IR 1.1 in Kibuye. * The distribution of the “preferred” sweet potato varieties being tested represents an effective farmer-driven process. * Communication between CRS and Caritas Kibuye was not as effective as it should have been over the first three and a half years of the DAP. Caritas Kibuye was not fully aware of the entire program strategy and, as a result, did not include some activities in their plans that were critical to achieving the planned outputs. 4. Major Lessons Learned Related to Implementation Processes for IR 1.1. FY 2001 - FY 2005 DAP Final Evaluation Report 11 January 2005 19 Butare. Following are the major lessons learned emerging from implementation of activities under IR 1.1 in Butare. ¸ The approach of wide-scale leveling of a marais beginning in the middle of the drainage is technically inappropriate. Development of water control systems best begins where the quantity of water is manageable, i.e., at the head of drainages in a watershed. Once the water control structures are in place at the head of the drainages, they can then be extended down through the drainage. ¸ The prototype for livestock troughs introduced at the beginning of DAP was inappropriate. People who raised livestock had to dig a large hole and use buckets to draw water from the canal to fill the hole. This system was rejected by the beneficiaries in favor of collecting potable water from the mountains and bringing it through gravity flow pipes to troughs. Kibuye. Following are the major lessons learned emerging from implementation of activities under IR 1.1 in Kibuye. ¸ A rotating loan system providing seed or reproductive material for “farmer preferred” crops can be an effective means of increasing crop productivity. However, in order to be sustainable, management of the loan system should not reside with a project. ¸ A cross-cutting lesson learned between Butare and Kibuye is that lessons learned in either Butare or Kibuye need to be communicated to the other program area in a timely way so that they can inform activities. 5. Overview of Implementation Strategies for IR 1.2 Intermediate Result IR1.2 Reduced erosion from steep lands for 28,500 families in targeted areas by 2005. Butare. In Butare, FFW was used initially to support development of trenches and planting of trees. The work was done by the beneficiaries and 5,450 hectares of land were protected, representing the entire watershed. The approach used was to terrace from the top of the hills. The last terraces near the lowland were large and deep to reduce the speed of water and soil from the hill. These last terraces also protected irrigation canals. In order to reinforce the terraces, trees and shrubs were introduced such as calleandra leucena to be used also as green fertilizer, sticks for climbing beans or animal fodder. CRS contracted participant associations to establish nursery trees and plants. These nurseries supplied seedlings not only to CRS, but also to other organizations and people who raise livestock in the zone. Following the MTE, the DAP discontinued use of FFW for hillside tree planting. Kibuye. In Kibuye, the strategy for IR 1.2 was to reduce erosion through tree and grass plantings and the construction of contour ditches on hillsides whose erosion would adversely affect the planned marais irrigation works. Farmer participation in the targeted watersheds was used in selecting the species and number of each to be planted. Associations whose members live on these targeted areas established and managed nurseries and provided tree seedlings. Outplanting was be done by those demanding trees and grasses on the hillsides. All associations selected for participation in the DAP in Kibuye were pre-existing. The selection was based on the following criteria: · The association was registered with the district · The association had an established bank account · Preference was given to associations that had a positive credit history. FY 2001 - FY 2005 DAP Final Evaluation Report 11 January 2005 20 6. Implementation of MTE Recommendations Related to IR 1.2 Butare. Of the five recommendations made by the MTE relative to IR 1.2, the DAP has completely achieved two of the recommendations (discontinuing FFW on hillsides and coordination between natural resource and infrastructure teams). A good start has been made on two other recommendations (focusing on soil and water conservation and focusing on localized interventions). Progress has been made on the last recommendation (farmers evaluating technologies), but the program in the field seems to be focused more on the topic mentioned in the recommendation (soil and water conservation) rather than on the process of engaging farmers in participatory monitoring and evaluation as applied to participatory technology development. Annex C contains a table which summarizes CRS's response to the MTE recommendations and an assessment by the evaluation team as to the quality of progress on each recommendation. Kibuye. In Kibuye, all five of the recommendations from the MTE were addressed to varying degrees. As in Butare, there was some confusion as to how to fully engage farmers in participatory technology development, so relatively little progress was made on the recommendation for enabling farmers to evaluate technologies. Farmers have been involved in choosing species, but enabling them to assess performance themselves through simple comparison trials has not been done. On other MTE recommendations, those related to using FFW and coordination between natural resource and infrastructure teams have been completed. Some progress has been made on the remaining two recommendations (soil and water conservation and localized interventions). A table is provided in Annex C indicating the CRS response to the MTE recommendation and the assessment of the evaluation team on progress made toward implementing the recommendation. 7. Observations Relative to Implementation Processes for IR 1.2. Butare. Following are the key observations made by the evaluation team relative to implementation processes related to IR 1.2 in Butare. * Management of the areas developed by the program at the time of the final evaluation was the responsibility of farmer associations. * Because the majority of the land is located on the hillsides in the program area, it was a good investment to protect this land in order to increase yields and improve food security in the zone. FFW was also an appropriate resource in an area where food supply is a big problem. However, since the infrastructure is now complete, the focus needs to shift to promoting more innovative use of the land. This will require staff with other kinds of knowledge and skills. * Since terracing was done by the farmers, they acquired some capacity to maintain those terraces * Terraces were rehabilitated and constructed. In both cases, owners of those structures had to maintain them, and CRS provided them with training. The associations formed by the program on the hillsides also followed-up on miantenance. It would be good to introduce high value crops on the hillsides so that farmers can see more clearly the relationship between soil protection and income from crop production. Kibuye. Following are the key observations made by the evaluation team relative to implementation processes related to IR 1.2 in Kibuye. FY 2001 - FY 2005 DAP Final Evaluation Report 11 January 2005 21 * Associations are capable of quality nursery production of tree seedlings * The assignment of the program’s irrigation engineer has resulted in an understanding by both natural resource management (NRM) and infrastructure staff of their mutual objectives and how the NRM team can support the marais work * DAP agricultural staff and other staff are trained sufficiently well in basic soil and water conservation and nursery management * The amount of work on ditching has proceeded very slowly because of the farmers’ unwillingness to give their time and labor for doing this without food for work or other compensation. 8. Major Lessons Learned Related to Implementation Processes for IR 1.2. Butare. Following are the major lessons learned emerging from implementation of activities under IR 1.2 in Butare. ¸ There is need to complement the soil conservation measures with more innovative farming systems on the protected land, so that farmers can see the benefits of soil conservation measures more clearly. ¸ It would have been useful to have worked with participating farmers at the start of the program to develop land management plans with them to cultivate a stronger sense of ownership in the soil conservation activities. Kibuye. Following are the major lessons learned emerging from implementation of activities under IR 1.2 in Kibuye. ¸ The system of association selection criteria in Kibuye avoided the political/social problems of forming “new” associations that just want project resources only (payment for seedlings produced, tools such as picks and shovels) for doing project work on the hillsides. Those associations which already have organizational skills and resources (e.g., bank accounts) will likely succeed more often than those starting from scratch, regardless of motivation. However, this system doesn’t assure that the disadvantaged (e.g., single parent headed households, orphans, the destitute) will be included. ¸ Soil conservation improvements on hillsides should be completed before water control infrastructure in lowlands so that erosion or excessive, rapid runoff from the hillsides will not damage or delay the construction of infrastructure. ¸ FFW is not necessary to promote the construction of erosion control structures on hillsides, except perhaps where the construction is highly labor intensive and perceived to be a community benefit rather than a direct benefit to the persons providing the labor. FFW is not needed for the planting of trees or grasses for soil conservation conservation purposes on hillsides. Farmers see this as a personal benefit to the lands they farm. 9. Overview of Implementation Strategies for IR 1.3 Intermediate Result IR1.3 Selected farms in targeted areas restocked with 13,500 goats and swine by 2005. Butare. Prior to the MTE, a decision was made by the DAP to suspend restocking activities and to focus on improving the existing livestock population. The MTE recommended that the program investigate other ways to facilitate livestock restocking rather than direct distribution of animals. In Butare, the DAP identified participants to serve as Community Animal Health Workers (CAHW), and provided training to them with the support of District Veterinarians FY 2001 - FY 2005 DAP Final Evaluation Report 11 January 2005 22 and ISAR on animal health care for both large and small livestock. The program moved toward facilitating forage development, and the CAHWs conducted surveys in their communities of people who raise livestock in order to determine forage needs. Given the GoR strategy to encourage penned livestock production (as opposed to free range), the program also provided training to CAHWs on pen construction. As originally proposed, medical kits and bicycles were to be provided to CAHWs after they had completed training. Because of resource constraints6 , the distribution of kits was delayed. They are expected to be distributed in the next few months. Kibuye. The decision to suspend restocking activities under this IR occurred before significant activities began in Kibuye. As a result, no activities were planned under IR 1.3 to take place in Kibuye, other than to provide advice on the rehabilitation of sheds for the collection of manure. 10. Implementation of MTE Recommendations Related to IR 1.3 Butare. The MTE made four recommendations relative to this IR. Of these, the program is beginning to make progress on three of them (kits/bicycles conditional disbursement, CAHW association development and expanded functions of the CAHW), however, with only seven months left in the program, the impact of these activities will be limited. On the other recommendation (participant managed restocking), the program proposed an internal credit mechanism that was rejected by USAID because the time remaining in the program was insufficient to ensure sufficient progress. The DAP did not explore other options. Annex C contains more detailed information on these recommendations. Kibuye. No activities were implemented in Kibuye relative to IR 1.3 apart from providing advice on the construction or rehabilitation of sheds for manure collection. 11. Observations Relative to Implementation Processes for IR 1.3 Butare. Following are the key observations made by the evaluation team relative to implementation processes related to IR 1.3 in Butare. * The recruitment of CAHWs was well done and the technical training provided to them was effective. * For a basic intervention in animal health care, the CAHW approach seems to have potential. However, because of the range of animal diseases present in the program area, more technical training may be required * CAHWs have not been trained in animal delivery (birthing) methods. * CAHWS have grouped themselves into an association and have begun to save money to buy drugs once the CRS stock is exhausted. * CRS has begun to integrate the animal treatment work of CAHWs in more livestock development activities, for example, with animal feeding. Kibuye. Thirty shelters were identified for rehabilitation, but no work to rehabilitate them was completed. Some of this may be attributed to the farmers not having the funding to buy materials for the rehabilitation. 12. Major Lessons Learned Related to Implementation Processes for IR 1.3 6 Apparently a former CRS staff member and a local government official colluded to pilfer resources from the program that were to be used to procure kits and bicycles for CAHWs. FY 2001 - FY 2005 DAP Final Evaluation Report 11 January 2005 23 Butare. Following are the major lessons learned emerging from implementation of activities under IR 1.3 in Butare. ¸ The first stock of drugs should not be disseminated without cost. A strategy to cover drug replacement costs should be introduced from the beginning. ¸ People who raise livestock need more than just basic animal health care to preserve and expand their livestock assets. Animal breeding and feeding, for example, are also important dimensions of animal husbandry. ¸ The spontaneous formation of an association of CAHWs is beneficial for facilitating social capital relationships between CAHWs. An association also has greater potential to be able to access support from finance institutions and other sources of support and to provide a conduit for useful information from these sources to CAHWs. ¸ In the design of the DAP initially, There was to be an implicit connection between animal restocking and soil and water conservation activities on the hillsides. Households with more animals are likely to have greater interest in using uplands for fodder species. A lesson from the DAP is that people are less likely to plant fodder grasses along their anti￾erosion bunds if they have no animals to feed. Kibuye. The major lesson learned from implementing activities under IR 1.3 in Kibuye were that training alone is not enough to stimulate change. 13. Overview of Implementation Strategies for IR 1.4 Intermediate Result IR1.4 Productivity enhancing technologies introduced to 15,000 families by 2005. Butare. In Butare, the DAP works with farmer participants to identify varietal needs and then brings seeds from ISAR for adaptability trials. As varieties are tested on marais or hillside plots managed by farmers, other farmers are invited to observe the growth of the plant through harvest. The program has established a system of seed multiplication for the best performing varieties by contracting farmers to produce seed, some of which is returned to CRS for onward distribution to other farmers and some of the seed is kept by the seed producers. The DAP has introduced three high yielding varieties of rice from ISAR. The trials are being done in farmers' fields in collaboration with ISAR . For Irish potato and maize, associations were formed to produce seed for sale to farmers in the marais and on hillside. Uphill associations are producing species such as pennisetum and trypsacum as forage with the objective of improving animal feeding and for erosion control. Species such as calleandra and leucena have been introduced to control soil erosion and to produce green fertilizer. Kibuye. In Kibuye, activities under IR 1.4 are being implemented using several different strategies. These include involving farmer associations in crop variety trials, both in the marais and the hillsides, training of Community Extension Workers (CEW) to do training of other farmers, training farmer associations to set-up and manage nurseries growing agroforestry and fruit tree species, training of agriculture program staff in how to set-up crop variety trials and in technology transfer methods, and developing linkages with ISAR for technical training. 14. Implementation of MTE Recommendations Related to IR 1.4 Butare. The MTE made five recommendations related to IR 1.4. The DAP has made progress in addressing all of these to some degree in Butare. The recommendation to conduct FY 2001 - FY 2005 DAP Final Evaluation Report 11 January 2005 24 a process to reach consensus on a technology transfer approach has been completed with staff in Butare. Progress has also been made on making a transition to more participatory approaches. The relationship with ISAR has been strengthened, and ISAR provides strong support when requested by the DAP. However, the influence of the DAP on the research agenda of ISAR, ensuring that research results are highly appropriate for smallholders, is still relatively limited. There has also been contact made with other USAID-funded activities, but this has not yet resulted in significant changes or benefits to the DAP. A systematic method for working with farmers to evaluate cropping patterns in the marais has not been done. Annex C contains more detail on the response of the DAP to the MTE recommendations. Kibuye. In Kibuye, relative less progress was made on the five recommendations made byt the MTE for IR 1.4. A systematic review of the various approaches used for technology evaluation and transfer was not undertaken. Activities in Kibuye still indicate a range of different approaches with little comparative assessment of the quality of the different approaches being used. Less progress has been made on the recommendation to make a transition to more participatory approaches. While evaluation of marais cropping patterns by the DAP program staff has been undertaken, farmer evaluation of cropping patterns and trials has not yet taken place on large scale. Only the sweet potato trials have been evaluated by farmers and advanced to the multiplication stage. Farmer evaluation of other trials is planned before the end of the program, however. The relationship with ISAR is stronger, as in Butare, but still not fully systematized. Annex C contains more detail on the response in Kibuye to the MTE recommendations. 15. Observations Relative to Implementation Processes for IR 1.4 Butare. Following are the key observations made by the evaluation team relative to implementation processes related to IR 1.4 in Butare. * The MTE was highly critical of the technology introduction activities of the DAP, reporting that (1) neither a paysan animateur nor model farmer approach was being used effectively, (2) inputs were provided late, (3) the approach was very top-down, (4) trials were poorly designed, (5) the collaboration with ISAR was too unstructured, (6) there was little contact with other USAID-funded activities and (7) the focus was too much on hillside crops at the expense of lowland crops, especially rice. At the time of the final evaluation, the program had been able to address all of these to varying degrees. * There is high demand for improved seeds on the part of farmers, which represents an opportunity for development of private sector suppliers of seed (rather than the program providing seed). * In addition to the fact that the DAP assisted farmers to exploit a market opportunity for a profitable crop, the associations were also given motivation by the district which offered to buy seeds of Irish potato. Since there is always a market for Irish potatoes in Butare, producers will continue to benefit. * The associations for maize production produce more for their own consumption, unlike the associations for Irish potato and rice, which sell significant amounts in the market. * All inputs were provided by the program on time through the associations of producers, and community participation was a key factor to introducing and disseminating new technologies. * The associations of rice growers unfortunately do not presently market their produce collectively, although they are thinking about doing so. At the moment, there are separate associations for those who produce seeds and for those who produce for market. FY 2001 - FY 2005 DAP Final Evaluation Report 11 January 2005 25 Kibuye. Following are the key observations made by the evaluation team relative to implementation processes related to IR 1.4 in Kibuye. * The integration of the agricultural staff and the technical transfer staff in doing technology transfer to farmers worked well because the knowledge gained by crossing skills is stimulating to the staff. * Unfortunately, the Kibuye program has not yet applied a farmer field school approach to training talented farmers to train other farmers in technology transfer. * Massive diffusion of technologies introduced by the program have not yet occurred, although the technologies were relatively recently introduced only in calendar year 2004. 16. Major Lessons Learned Related to Implementation Processes for IR 1.4 Butare. Following are the major lessons learned emerging from implementation of activities under IR 1.4 in Butare. ¸ Working with the associations is an effective approach for timely service delivery, economy of scale and facilitation of diffusion of new technologies. ¸ New technology needs to be provided as a package and not as just seed, including only fertilization, post harvest technology, marketing, and cultivation practices. ¸ Unless a new idea or technology has immediate and obvious benefit, technology transfer and adoption can be a long term process of selection, adaption and adoption. A community participatory approach in which a wider segment of the community is engaged in processes can facilitate the process. Kibuye. Following are the major lessons learned emerging from implementation of activities under IR 1.4 in Kibuye. ¸ Having well trained trainers is essential for program success ¸ Variety trials if organized well and placed properly can serve to effectively diffuse project activities to farmers not already participating in the project ¸ “Model” farmers are an effective way of getting community participation in DAP activities. By definition, this person is a farmer, a community member, and an innovator that understands farmer concerns and farming systems in his/her community and who will be “people oriented” by nature. ¸ Sometimes it takes time for diffusion of innovation. For example, one can’t promote a crop variety when farmers haven’t observed its performance. Trials and demonstrations need to be located in highly visible areas, and other activities can be undertaken to facilitate exposure of more farmers to the new ideas and technologies, including agricultural fairs and field days. 17. Overview of Implementation Strategies for IR 1.5 Intermediate Result IR1.5 Farmers’ associations established or strengthened in targeted areas by 2005. Butare. Four types of farmer's associations were targeted for establishment and strengthening in the DAP. These included groups of farmers formed around farmer field schools with trained Community Extension Workers, groups of farmers who had been allocated land in newly developed marais lowlands, small committees to oversee management and FY 2001 - FY 2005 DAP Final Evaluation Report 11 January 2005 26 maintenance of water points developed in the marais, and management committees to oversee the management and maintenance of each newly developed marais. In Butare, some associations already existed, including those who worked with WFP, other emergency organizations and other NGOs especially in lowlands. These include associations who produced vegetables, associations of livestock owners, associations of coffee growers in Maraba district, associations of sorghum production and marketing, and fruit growers associations. On the hillsides, there were associations of vegetable producers, Irish potato and maize growers. CRS strengthened these existing associations, mainly in marketing. Centre IWACU is specialized in the cooperation movement and has provided training for most of the associations who need accounting and financial skills. In the DAP, the center has provided training to the rice growers associations. Kibuye. The delayed implementation of activities in Kibuye and the initial emphasis on hillside treatments has seen organizational development focused on three of the above groups, farmer associations, water management associatiosn (for irrigation in Kibuye) and a marais management association. A total of 26 hillside farmer associations involved in soil and water conservation and marais associations involved with the crop variety trials were trained in organizational and financial management. One hillside irrigation water committee and one marais management committee also received this training. 18. Implementation of MTE Recommendations Related to IR 1.5 Butare. Of the six recommendations made by the MTE relative to IR 1.5, five are relevant for Butare. Three of the recommendations were fully achieved (reduction of targets on farmer associations, the hiring of a local consultant on hygiene education and water borne diseases and investigation of materials from CORD). Some progress was made on needs assessments and problem-solving with water committees, and the DAP has made a commitment to implement a more participatory approach to health and sanitation using the PHAST (Participatory Health and Sanitation Transformation) approach in the remaining life of the program. No progress was made on the fifth recommendation for developing a rational strategy for organizational strengthening of associations. Annex C contains more detail on the response of the program to the MTE recommendations. Kibuye. All of the six MTE recommendations associated with IR 1.5 were relevant for Kibuye. Two of the recommendations (reduction of association targets and reduction of management committees) were fully implemented. On the other recommendations, even though a consultant was hired for hygiene and sanitation education, the consultant did not work in Kibuye. Similarly, the results of the investigation into the appropriateness of the materials from CORD were not communicated to Caritas Kibuye, and the PHAST initiative that resulted in part from this investigation is also not known to the Kibuye component of the program. No water source work has been done in Kibuye, apart from capacity building with one irrigation water committee, so there hasn't been need for an infrastructure team to visit rehabilitated water sources in Kibuye. As in Butare, no progress was made on the fifth recommendation for developing a rational strategy for organizational strengthening of associations. 19. Observations Relative to Implementation Processes for IR 1.5 Butare. Following are the key observations made by the evaluation team relative to implementation processes related to IR 1.5 in Butare. FY 2001 - FY 2005 DAP Final Evaluation Report 11 January 2005 27 * There appears to be some spontaneous formation of farmers associations with farmers not participating in the program who have observed benefits accruing to the associations participating in the DAP. * Following a study tour facilitated by the program to Ruhengeri, producers associations around Irish potato were trained by the program. * An association is an appropriate structure for service deliver when there is need to access a large number of people who implement the same activity. Because the people elect their representative, one can assume that there is some democracy in those associations. However the power is not equally shared, people who are elected have more power than ordinary members. The president of the association very often uses his or her influence for decision making during the meetings. He or she is the one who talks to local authorities and who can influence decisions for his or her benefit, decision which are not always in the best interest of the association or other association members. This is something which needs monitoring by and arbitration from local authorities to make sure that there is no unfair influence. * There is strong ownership of the infrastructure in the marais because of the existence of the associations and the way in which tasks have been organized. * Users are ready to pay fees to maintain their plots. This commitment represents a form of solidarity group control. * There persists a problem of a lack of water in some areas of the watershed during the dry season which has implications on the appropriate cropping patterns. * People have a good sense for working in committees. * Members are proud of their responsibilities in the potable water committees and rice, Irish potato, maize growers committees of the respective associations. * The farmers associations are still young and there is a need for a plan to know when the DAP is going to leave so that paysans animateurs will continue to work by themselves and under which conditions. Kibuye. Following are the key observations made by the evaluation team relative to implementation processes related to IR 1.5 in Kibuye. * All DAP natural resource management staff were trained in association and cooperative development through a week long workshop in Kigali. * Program wide training on conducting crop variety trials, soil and water conservation techniques, and nursery management was provided initially to the agricultural, technology transfer, and monitoring staff . However, all of the Caritas staff stated that they felt the need for continuing training and the need to periodically meet with their counterparts in Butare to compare and share their experiences and needs for further training. * All associations have asked for and received some training on agro-business/marketing, and all are aware of the potential value of marketing their produce cooperatively. * There have been communication difficulties between CRS and Caritas Kibuye in terms of the strategy of the program. Early on, it was not clear to Caritas Kibuye, for example, what they could, or could not, do in the DAP. As a result, some commitments beyond the scope of the DAP were made with communities and local authorities. The agreement signed in 2004 has clarified the program including the roles and responsibilities of each partner. FY 2001 - FY 2005 DAP Final Evaluation Report 11 January 2005 28 * Some activities that were intended for implementation in both Butare and Kibuye, have not been well coordinated. Joint meetings, for example, between the Butare and Kibuye staff to assure that training, monitoring and evaluation methods are done consistently program wide have not been systematically organized. * There is a need for continuing training of DAP Kibuye staff. Several staff expressed that they would like to advance their knowledge so as to improve their effectiveness in both technology transfer and association organization. All of the staff have an elementary/secondary level of training in agronomy or other specialty (A2). However, only two have a two year training at university level and only 1 a four year degree * The training of associations in general has been effective. All committee officers and members interviewed stated that they felt that the officers managed the association to the benefit of all members and that the members shared equally in the association’s benefits. * The criteria used for selection of associations in Kibuye did not include provision for guaranteeing representation of the disadvantaged. District officials were consulted to find existing associations that had proven potential (e.g., established bank accounts, participation in credit schemes). Given these criteria, it is likely that association members had above average resources for their communities. 20. Major Lessons Learned Related to Implementation Processes for IR 1.5 Butare. Following are the major lessons learned emerging from implementation of activities under IR 1.5 in Butare. ¸ The greater the direct benefits that members of an association see from their participation, the greater the likelihood that the association will become self-sustaining. ¸ Democracy is very fragile, executive committees of different associations have more power and they may use their power to exploit members. ¸ Committees are essential for easy communication inside and outside the associations. ¸ A committee network is a useful entry point for monitoring activities and facilitating implementation. Kibuye. Following are the major lessons learned emerging from implementation of activities under IR 1.5 in Kibuye. ¸ Associations provide distinct advantages in shared labor for such activities as crop variety trials and tree nursery management. ¸ Associations provide a collective advantage to those not having sufficient land or financial resources for the individual members to support themselves. ¸ Since associations are all registered and regulated by District officials, associations have leverage when requesting repeated use of the same lands for farming. 21. Overview of Implementation Strategies for IR 1.6 Intermediate Result IR1.6 Improved market access for 43,500 families in targeted areas by 2005. Butare. Activities under IR 1.6 concentrated on using FFW to rehabilitate or construct feeder roads and bridges. In Butare, district authorities were consulted by the program to identify the priority roads and bridges to be rehabilitated or constructed. Manpower for FFW was recruited under the supervision of Community Development Committees (CDCs). For major FY 2001 - FY 2005 DAP Final Evaluation Report 11 January 2005 29 bridge work, tenders were prepared by the program in collaboration with the District and CDC to select an engineering company. Technical supervision was the responsibility of the program with the support of the district road engineer. Once work was completed, a handover ceremony was held to transfer responsibility for the road or bridge to the district. Kibuye. In Kibuye, a similar process was used. The strategy was to provide access to markets and egress for agricultural products through road rehabilitation and construction or repair of bridges and water traverses (culverts) using FFW to pay the unskilled labor selected from communities adjacent to the roads. The provincial government planning representative and key members of Kibuye’s districts were consulted to find which roads would benefit most from such work. The communities were to be motivated to sustain road maintenance so that the value of the initial rehabilitation would not be diminished. 22. Implementation of MTE Recommendations Related to IR 1.6 Butare. The MTE made four recommendations relative to IR 1.6. In Butare, one recommendation was fully achieved (to institute a transfer of responsibility ceremony). On a second recommendation (creative ways to promote road maintenance), the program has made some progress using the umuganda7 . This is not likely to be the long-term solution to the road maintenance issue, however. The recommendation to establish a camber on road surfaces was rejected by CRS as infeasible from a resource perspective. No progress was made in Butare toward identifying the impact of road improvements on market access. Annex C contains more detailed information on the response to the MTE recommendations. Kibuye. Similar progress was made in Kibuye on the four recommendations. One has been achieved (transfer ceremonies). One has been partially achieved (using umuganda for road maintenance). One has been rejected as infeasible (camber on roads), and no progress has been made on the other (impact indicators). See Annex C for more detail. 23. Observations Relative to Implementation Processes for IR 1.6 Butare. Following are the key observations made by the evaluation team relative to implementation processes related to IR 1.6 in Butare. * The road network of 109 kilometers has been completed, and 10 bridges have been repaired or rehabilitated. The feeder roads allow access to the watersheds of the two main marais, Runukangoma and Rusuli, which in the future will facilitate marketing of production from hillsides and lowlands. * In some cases, the roads that have been rehabilitated may not be producing the intended impact on market access because vehicles are not yet common on the roads. To get to market on foot, many community members choose footpaths that are shorter in length than the roads. * Since community members may not be realizing significant economic benefit from roads, they do not have much sense of ownership or feel much responsibility for keeping roads maintained. For many, roads are viewed as a government responsibility, not a community responsibility. * Roads and bridges are just one part of the solution to facilitating better market access. Marketing associations and basic marketing infrastructure such as storage will likely be needed in the near future. 7 Umuganda is the system of involuntary community service in which Rwandan citizens are required to allocate one half day per month for national community service and a half-day per week for district level community service. FY 2001 - FY 2005 DAP Final Evaluation Report 11 January 2005 30 * CRS has used a participatory approach to choosing the priority roads and bridges for rehabilitation and construction. However there was no clear policy for maintaining the structures from the beginning. Kibuye. Following are the key observations made by the evaluation team relative to implementation processes related to IR 1.5 in Kibuye. * A “feasible, creative” way to do road rehabilitation has not yet been identified although the use of the umuganda offers a partial way of providing road maintenance. There is evidence, however, that it is mostly the poor and least influential members of the community who are doing most of the work in the umuganda, which raises a rights and justice issue. The program should not find itself in the position of promoting a practice that is inequitable to its target group. * The FFW deliveries, while FFW work was used (3 months) were not delivered in a timely manner, and this left some in the local communities with a bad feeling about working with the DAP. * Other projects in the program area where the DAP targeted road work were also giving FFW. This significantly reduced the number of workers when the DAP had inconsistent food deliveries, and the potential workforce left the DAP to work on those projects giving more reliable FFW. To offset this, the DAP signed agreements with communities so that the commitment of food was confirmed and the communities agreed to carry out FFW activities and be paid when the food arrived. * The contractors building bridges have done a very satisfactory job. The new bridges appear well built, using good materials and building techniques. * The provincial roads official sees FFW as the primary way to rehabilitate secondary, farm-market roads. Since local communities do not see road rehabilitation as one of their priorities, government officials need to be made more aware of this and encouraged to seek alternative ways of funding this work. * At present, FFW appears to be used as a means of defraying food expenses and providing families with alternative ways to use their income from other sources to improve their household assets and livelihood. 24. Major Lessons Learned Related to Implementation Processes for IR 1.6 Butare. Following are the major lessons learned emerging from implementation of activities under IR 1.6 in Butare. ¸ Before rehabilitation of roads or bridges begins, responsibility for long term maintenance must be defined and agreed upon. Without such agreement, rehabilitation or construction should not occur. ¸ Roads are an important feature of rural infrastructure which require large investments, however communities must take some responsibility for road maintenance either through taxes or directly. Kibuye. Following are the major lessons learned emerging from implementation of activities under IR 1.6 in Kibuye. ¸ FFW must be delivered in a timely manner or this will disrupt planned work ¸ The Provincial government apparently is unable to support secondary road rehabilitation, thus some other source of maintaining this infrastructure must be found. Either FFW or some form of user tax appear as the most viable alternatives to rehabilitating these roads. FY 2001 - FY 2005 DAP Final Evaluation Report 11 January 2005 31 ¸ Long term maintenance of the infrastructure (roads and bridges) has been correctly identified as a priority. However, local communities do not see this as their priority. C. Safety Net Component 1. Overview of the Safety Net Component The safety net component of the DAP under the 2nd Strategic Objective (SO) contributes to Intermediate Results (IR) 2.1 and 2.2. The SO and IRs are shown in Table 3 below. The safety net target is the 7,000 institutionalized orphans, street children and handicapped persons mentioned in IR 2.1. The DAP provided a wet ration composed of corn meal, corn-soy blend (CSB), pinto beans and vegetable oil designed to meet between 50% (for children in foster homes) to 70% (for children in institutions) of a normal daily ration. Over the life of the activity, the safety net component served orphanages, street children centers, institutions for the disabled, homes for the elderly and a small number of children institutionalized with their mothers in prisons. Food deliveries to safety net institutions were made approximately every two months and end use checkers monitored the management of commodities with visits generally sometime after every delivery. The amount of commodities for each institution was based on an approved list of beneficiaries updated annually. Table 3. Strategic Objectives and Intermediate Results for the Safety Net Component Strategic Objective 2: To ensure increased food security for 28,000 of Rwanda's most vulnerable people in each year from 2001 through 2005 Intermediate Result 2.1: Increased food security for 7,000 institutionalized orphans, street children, and handicapped persons, and 4,400 vulnerable households affected by AIDS by 2005. Intermediate Result 2.2: Strengthened capacity for self￾sufficiency among partner institutions caring for unaccompanied children, orphans, AIDS victims, and other vulnerable people by 2005 To achieve results under IR 2.2, the program implemented activities to support capacity building of the various institutions. These involved, for the most part, centralized trainings of around five days, with cross-visits to see best practices. Staff in the program also conducted field visits and provided on-site technical advice. One set of activities under the capacity building included providing guidance for identifying potentially viable income-generating activities and developing funding proposals to obtain resources from the program for these. The DAP provided grants to those proposals for activities likely to succeed. Over the life of the program, activities in the safety net component were implemented by a Safety Net/LIFE Manager or Coordinator and one technical staff position (a second position in the unit supported the LIFE component). Up until April of 2004, the technical staff position was focused on technical support for all aspects of the program, but especially on capacity building around income-generating activities. In April, the person holding the position was transferred to another CRS program just starting up, and the DAP took the opportunity to restructure the staffing for the Safety Net/LIFE component. The new structure included a full time nutrition specialist and two half-time technical specialists in capacity building agriculture supporting both the safety net and LIFE components. FY 2001 - FY 2005 DAP Final Evaluation Report 11 January 2005 32 2. Implementation of MTE Recommendations Related to the Safety Net Component Relative to the safety net component of the DAP, the MTE made twelve recommendations. Annex C contains tables which summarize CRS's response to each of the MTE recommendations and an assessment by the evaluation team as to the quality of progress on each recommendation. Of the twelve recommendations, significant progress was made on two of the recommendations, to continue commodity delivery and to expand the number of street children centers. Four new centers were added for FY 05. On one other important recommendation, building capacity for growth monitoring, a good start was established with a training workshop, although follow-up since the workshop has been limited. The position of nutrition specialist for the program was filled in September 2004. On all other recommendations, some progress was made, although the progress was limited. 3. Observations Relative to the Safety Net Program Following are key observations relative to the safety net component. They have been clustered relative to observations around the safety net institutions, capacity building, commodities, targeting, and communications. Safety Net Institutions. There is much variation between the safety net centers in terms of the resources they have available to support their programs. Some have land, some don't. Some have access to reliable sources of funds; others are highly fragile in terms of resources. Similarly, there is a lot of variation in the management quality of the centers. Some have innovative, high quality managers. Others have managers with limited management experience or vision. Finally, most centers have connections directly (infected beneficiaries) or indirectly (beneficiary status a result of HIV/AIDS) with HIV/AIDS. The numbers of children requiring safety net assistance is increasing, mainly as a result of HIV/AIDS. Capacity Building. A significant amount of capacity building has occurred with safety net institutions participating in the DAP. This was mainly achieved through the release of resources as a result of the food provided by the program. Safety net institutional resources, including both staff time and financial resources, that would have been used for obtaining food were released to pursue other activities or investments because the DAP provided food. The capacity building that resulted was not monitored by the program, so it is difficult to determine its scale and scope. All institutions surveyed as part of the evaluation reported either increased services for beneficiaries or at least improved staff morale from not having to devote extensive time and resources to pursuing food. Annex F contains information from a rapid survey conducted with a sample of safety net institutions. The Safety Net component of the DAP was short-staffed relative to the commitments made to capacity-building. Capacity building needs to build upon the existing capacities and resources of a safety net center. In order to understand what capacities and resources are available to an institution, to establish credibility and working relationships with the institution, and to develop participatory capacity building plans with an institution requires time, more than one visit per month. To really achieve significant capacity building toward self-sustainability, the program would have needed additional field staff to work with safety net centers. The commodity elements of the safety net component took priority over capacity building elements, especially in the allocations of staff time. Program staff had to devote time to solving problems related to commodity management at the expense of time for capacity building activities. Strengthening social capital is an area of capacity building in which the safety net component made contributions through the annual partner meetings. These meetings facilitated the FY 2001 - FY 2005 DAP Final Evaluation Report 11 January 2005 33 development of relationships between safety net institutions, just by the act of bringing them together. These social capital relationships could have been strengthened even further, however, through strategic cross-visits between centers on technical matters. However, there simply were not enough staff in the component to undertake this additional task. The introduction to growth monitoring systems has been generally well-received by participants, although there seems to be fairly wide variation on how much growth monitoring has been undertaken after the training. Participants value this as a systematic way for identifying food recipients who may require supplemental feeding, rather than relying on subjective observations. The program does not have monitoring tools in place to assess how effective the training has been and to identify centers requiring or interested in receiving additional support to establish good growth promotion activities. Income-Generating Activity (IGA) Grants. A total of fourteen IGA grants were made over the life of the program. The program did not have monitoring systems in place for tracking the performance of IGAs8 to determine whether additional financial or technical training would be required to ensure that the investments were sustained. As a result, there is wide variation in the performance of the IGAs. Some were sustained, a few actually grew and others failed. The transition to an IGA support program in which resources are provided by the program through pseudo-revolving loan funds9 is an interesting strategy to build ownership and commitment to an IGA. The policies and procedures for this activity have not yet been documented, so there is a little confusion as to its full purpose and nature. As a tool for mobilizing capital and generating ownership in the IGA, the approach certainly has merit. As a tool for building the capacities of centers to manage credit, however, it is likely to be ineffective10. To build credit management capacities, the credit experience must be realistic. The highly subsidized interest rate and loose terms, including no reference to collateral, do not reflect the reality of obtaining capital from a formal lending institution. Commodities. The food pipeline has been fairly reliable, although there has been one break in July and August of 2004. There have also been minor breaks of a few days for specific centers related to reporting. On rations, most centers were happy with the commodities provided, but those with children under two years of age expressed interest in having commodities more suitable for infants. The commodity ration otherwise appears suitable, although the same ration over the full five year implementation period represents a lost opportunity to measure increases in the capacity to be self-sustaining. Since safety net institutions more or less had a guaranteed supply of food for the life of the DAP, they were not as highly motivated to develop alternative sources of food as if the supply of food were less guaranteed. 8 The best information on the number and scale of IGA grants was obtained from the finance department. 9 The six safety net centers who have received funds under the new terms have been told that they have received a loan. In fact, CRS does not intend to retain any payments made, but rather to keep them in the account that has been opened for the IGA. At the end of the repayment period, a decision will be made with the safety net center to either reinvest the funds in the current IGA or invest them in another IGA to be implemented by the safety net institution. 10 While loans are one way to generate ownership in an IGA, there are other ways that could be considered such as two or three stage conditional grants. A loan mechanism arguably builds the strongest sense of ownership. However, to build credit management capacities, linkages to banks or other micro-finance institutions through guarantee programs, for example, are likely to be more effective than the pseudo-revolving loan fund being tested by CRS. FY 2001 - FY 2005 DAP Final Evaluation Report 11 January 2005 34 Targeting. Regarding the geographic scope of the program, it is difficult to justify any changes in geographic targeting on purely the need for support. Children in institutions are highly vulnerable, and there appears to be no significant geographic clustering of safety net institutions in Rwanda apart from normal urban clustering. However, if the safety net program becomes integrated into a wider livelihoods programming strategy, then the resources may not be sufficient to maintain the current geographic scope. Prioritization would then be a function of the design of the program, i.e., the range of activities, potential partners, roles of partners and resources available. Relative to recipient targeting, for the most part, those who receive the food are entitled to receive the food. The situation in which their may be some slippage in targeting is with those institutions who are not housing children but rather negotiating space for them in foster homes. The children of some foster families, some of whom may not be food insecure or highly vulnerable, appear to be benefiting from food distributions made by the program. Communications. Communications between the program and safety net institutions was not always timely or consistent, particularly for written communications. Some centers have still not been officially informed on the status of their IGA proposals, for example, and improved communications is mentioned occasionally as an area in which the participating institutions would like to see improvements in the next phase of the program. The volume of communications coming to the program is large relative to the number of staff, however, making it difficult to be able to respond completely and in a timely fashion. 4. Lessons Learned- Safety Net Component The following major lessons have emerged from the experience of the DAP relative to the safety net component. ¸ Institutions seem to learn effectively from other institutions, perhaps even more effectively then learning directly from CRS staff. This was seen mainly in areas of commodity management, but the principle can be applied more broadly. ¸ Directors of some street children centers and orphanages are reluctant to fully support reunification and reintegration of their beneficiaries because it could mean that the institutions will close and they may no longer have jobs. ¸ Facilitating the integration of handicapped children and street children into the formal school system has had additional positive benefits in terms of raising the self-esteem and confidence of the children. ¸ Because street children have had to deal with some harsher conditions in life (i.e., have had to become more mature at an earlier age), when they receive opportunities for eating regularly and attending school, they occasionally excel in their performance both academically and socially. ¸ If a commitment is made to achieve a certain level of performance, adequate human resources need to be made available to achieve the targets. D. LIFE Component 1. Overview of the LIFE Component The Leadership Initiative for Fighting Epidemics (LIFE) is a program started by US President Bill Clinton to help reduce the socio-economic impact of HIV/AIDS in twelve African countries most affected by the disease. In the DAP as part of Strategic Objective 2, LIFE focuses on making contributions to the same two Intermediate Results as the safety net FY 2001 - FY 2005 DAP Final Evaluation Report 11 January 2005 35 component and targets people who are living with HIV, AIDS orphans and child headed households. The strategic objective and intermediate results for the LIFE component are shown in Table 4. The target of 4,400 households mentioned in IR 2.1 is the LIFE target. LIFE is currently working in nine of Rwanda’s twelve provinces. Table 4. Strategic Objectives and Intermediate Results for the LIFE Component Strategic Objective 2: To ensure increased food security for 28,000 of Rwanda's most vulnerable people in each year from 2001 through 2005 Intermediate Result 2.1: Increased food security for 7,000 institutionalized orphans, street children, and handicapped persons, and 4,400 vulnerable households affected by AIDS by 2005. Intermediate Result 2.2: Strengthened capacity for self￾sufficiency among partner institutions caring for unaccompanied children, orphans, AIDS victims, and other vulnerable people by 2005 The strategy underlying activities in the LIFE component is to (1) improve the well being of people living with HIV/AIDS, (2) reduce stigma and discrimination against people living with HIV/AIDS, their families and the people who care for them and (3) contribute to the food security of orphans and households affected by HIV/AIDS. The primary intervention in the LIFE component is food, and the program works through four sub-grantees, Caritas Rwanda, World Relief, ANSP+ and CARE International who work directly with 66 associations, health centers and churches. Commodities are disbursed to sub-recipients every two months and distributed to beneficiaries every month. In addition to food assistance, the DAP through the four partners also implements activities for HIV/AIDS prevention and provides psychosocial support to people living with HIV/AIDS. The program advocates for home and hospital visits as well as on issues related to the status of AIDS orphans and people living with HIV/AIDS. To increase institutional capacity of organizations and partner associations, the program encouraged these associations to develop income generating activities so that they may become more self-sufficient. Unlike in the safety net component, however, grants of program resources were not made available for income-generating activities. 2. Implementation of MTE Recommendations Related to the LIFE Component There were a total of thirteen recommendations made by the MTE for the LIFE component. Of these recommendations, ten have been addressed in some form or another by the DAP and the remaining three are under consideration or were not addressed. Those recommendations addressed included: · Investigation of CARE’s commodity distribution (completed) · Improved reporting from LIFE centers (significant progress) · Updated recipient lists so there will be no duplication of food distribution with other CRS and WFP programs (completed) · Institutional capacity building for partner organizations (some progress) · Appointment of a DAP Manager (completed) · Clarification on the roles and responsibilities among CRS departments (nearly completed) · Becoming better organized (some progress) FY 2001 - FY 2005 DAP Final Evaluation Report 11 January 2005 36 · Evaluation of the flow of information between CRS and partner organizations (some progress) · Advocate for more nutritional support to households (some progress) · Integration of LIFE with other HIV/AIDS programs (some progress) One on-going recommendation is to look for opportunities for synergistic programming with the natural resource components of the DAP in the area of income generation for HIV￾affected households. Little progress was made on this recommendation. Two recommendations were not addressed. It was recommended that LIFE look into reaching more households by increasing food provided by USAID/Rwanda or diverting food from the other components of the DAP. USAID rejected a proposal made by the DAP to divert FFW resources to LIFE in favor of the program focusing on implementation of the other recommendations made by the MTE. It was also suggested by the MTE that a study be conducted to understand why so few men were participating in the program. This study was not done. Annex C contains a table providing more detail on the response of the DAP to the MTE recommendations and a assessment on the part of the final evaluation team as to the quality of the progress made on each recommendation. 3. Observations Relative to the LIFE Component Following are key observations relative to the LIFE component. Commodities. Since the start of the program, there has been no major loss of food at the association level. When small quantities of food have been lost, the program has replaced it so that associations or members are not penalized. The dates for distribution are respected when there is food available. The two or three breaks that have occurred in the food pipeline have happened in Mombassa or in transport from Mombassa to Kampala and Kigali (see section G below). The commodity logistics for the program are sufficiently developed and permit transportation of food in good condition. The beneficiaries appreciate the food and find the rations relatively sufficient. Following the MTE, the coordination of activities and food management has improved. Finally, there is active participation by beneficiaries in the management of food, for example, in the offloading (they are paid 100 to 400 RFr per day for their work), distribution and ensuring proper use of the food. On the negative side, some participant associations lack adequate institutional capacity such as volunteer management, low staff education, and poor health of members. Families of different sizes receive the same amounts of food, which penalizes families with more than six members. End use inspections have tended to be policing in nature, making it difficult at times to also provide advice for improving commodity management. The availability of food has been instrumental in increasing the number of members in participating associations. The food recipient lists are fixed each year so that the same households receive a ration for an entire year. Given the relative scarcity of food in light of the increasing numbers of potential beneficiaries and need for food, this might not be the most effective way to use food to meet the most critical food security needs. Food might have greater impact in terms of reducing malnutrition if it were made available to those households with the highest levels of malnutrition. A household that has received regular food for a period of six months, for example, likely has better nutritional status than another household that has not. Capacity Building. The capacity building activities have had marginal impact. The LIFE component does not really have a strategy for promoting income-generating activities which will support the sustainability of participating associations. FY 2001 - FY 2005 DAP Final Evaluation Report 11 January 2005 37 Exit Strategy. The LIFE component does not have a clear exit strategy for deciding when assistance is no longer necessary for a participating household or associations. While some households will always require assistance, others may actually be graduated from the program to independence. Other Observations. The program is responding to true food insecurity and is consistent with the strategies of local and national authorities. Program resources, however, are insufficient in relation to population need especially since the number of households seeking assistance is increasing. The LIFE partners all have very interesting activities to support their program activities associated with the DAP and all are aware of the complexity of the existing problems for the target groups. The DAP does not have a mechanism in place, however, for sharing ideas and lessons between the partners in order to develop synergies and enhance each others' programs. The monitoring and evaluation systems for the DAP do not have systems in place for monitoring the benefits of better nutrition from food provided by the program. The LIFE component seems to have limited engagement of local authorities, whether for monitoring nutritional status or for ensuring that LIFE activities reinforce national strategies. In fact, LIFE is highly compatible with national strategies. This has been more by coincidence, however, then through engagement of national authorities and an understanding of national strategies. 4. Lessons Learned- LIFE Component The following major lessons have emerged from the experience of the DAP relative to the LIFE component. ¸ Management of PLWHAs demands particular attention since they are easily discouraged or depressed ¸ Working with partners can be very valuable in terms of learning new ideas and best practices from them. The DAP experienced this with commodity management, specifically, but the lesson can be applied much broader to other aspects of programming. ¸ Commodity distribution motivated persons to join associations and expanded the fight against HIV/AIDS by improving the solidarity and the spirit of association between persons living with HIV. E. Program Integration 1. Overview of Program Integration The natural resource and Safety Net/LIFE components of the DAP evolved from different directions (the TAP and IFP) and therefore they do not lend themselves easily to integration in the usual sense. They have operated and continue to be implemented as independent components. There are opportunities, however, to provide mutual support and sharing of lessons and experiences between the components, especially with agricultural technical capacity. There has been so far, however, limited exchange of information between the agriculture and Safety Net/LIFE components. Within the agricultural component, the Butare and Kibuye programs have the same target groups but utilize different program delivery strategies. In Butare, CRS implements activities directly while in Kibuye, DAP natural resource activities are implemented through Caritas Kibuye. Nevertheless, the program has made efforts to establish mechanisms for sharing of FY 2001 - FY 2005 DAP Final Evaluation Report 11 January 2005 38 information and experiences between the two sites. The Irrigation Engineer is responsible for supporting both sites and the Agricultural Coordinator before the position was vacated was also responsible for both sites. Within the agricultural component, the DAP has teams devoted to infrastructure development, natural resource and technology transfer. Up until the time of the MTE, these teams worked relatively independently of each other. Since the MTE, however, there has been more interaction and coordination between the teams. With regard to the integration of the DAP with other programs managed by CRS, the Safety Net/LIFE components have a natural affinity because of targeting with the OVC/ART/ABY program, and CRS has had discussions about ways to cultivate collaboration and integration with this program. The other major program within the CRS Rwanda portfolio is the Peace and Justice Program. There has not yet been much interaction or practical collaboration between the DAP and the Peace and Justice Program. Relative to other programs, being implemented in the same sectors as the DAP but by other organizations, there have been contacts made with various USAID-funded activities, but little direct collaboration. 2. Implementation of MTE Recommendations Related to Integration Relative to integration, the MTE made six recommendations. Annex C contains tables which summarize CRS's response to each of the MTE recommendations and an assessment by the evaluation team as to the quality of progress on each recommendation. Of the six recommendations, two refer to cultivating better integration between the the natural resource and Safety Net/LIFE components of the DAP. Little significant progress was made on either of these recommendations. A third recommendation encouraged closer interaction between different teams within the agricultural component, and significant progress has been made in implementing this recommendation. The last three recommendations focused on integration of the DAP with other programs outside of CRS, mainly with other USAID-funded activities, WFP and utilizing the partnerships developed in the LIFE Program to expand HIV/AIDS programming. The focus of the program since the MTE has been internal in order to facilitate the major shoft in strategy represented by the range of recommendations made by the MTE. As a result, there has not been much progress made on the external integration opportunities that were identified. 3. Observations Relative to Integration Following are key observations relative to integration in the DAP. Integration Between the Natural Resource and Safety Net/LIFE Components of the DAP. Because the Safety Net/LIFE component and the natural resource components have very different target groups, i.e., the most vulnerable and small farm families, respectively, it is unrealistic to expect much integration between the two components. The set of interventions that have evolved in the natural resource component around marais or hillside agriculture and animal husbandry are, for the most part, unsuitable for residents of safety net institutions, households of people living with HIV/AIDS or the institutions and associations that support them. These people and institutions generally have a different set of assets than do small￾holder farm families in terms of access to land, labor, and capital, including social capital networks. That is not to say that natural resource interventions are not important for the target groups and institutions in the Safety Net/LIFE component. Agriculture and animal husbandry are important livelihood activities for generating income for institutions and associations serving FY 2001 - FY 2005 DAP Final Evaluation Report 11 January 2005 39 street children, orphans, the disabled, elderly and households affected by HIV/AIDS. Agriculture and animal husbandry can also become important livelihood activities to enable beneficiaries to graduate from safety net institutions. However, the package of interventions most appropriate for them are not those that have been developed in the natural resource component of the DAP, apart from some specific varietal tests or animal husbandry techniques. The set of interventions most appropriate for the target groups in the safety net/LIFE component could include, for example, facilitating access to land, urban/peri-urban agriculture, or small-scale agro-processing. For the institutions and associations serving these target groups, appropriate natural resource interventions could include, for example, production of high value crops on small parcels, both large and small animal husbandry, and agro-processing. Because the assets and management capabilities of the institutions and associations vary widely, appropriate income-generating activities around natural resource management should be tailored to fit each institution, so a significant intervention with each organization will be to build capacities for market identification and analysis and comparative advantage analysis. In summary, integration of the DAP natural resource component with the Safety Net/LIFE component would be less effective than integrating a natural resource strategy tailored to fit the assets of the safety net/LIFE target groups and organizations that serve them. Integration Between the Natural Resource Components in Butare and Kibuye. Since the MTE, there has been better interaction between the DAP natural resource components in Butare and Kibuye, mainly through the efforts of the irrigation engineer. For the most part, however, program staff in each location work independently. The DAP does not have a mechanism in place, such as, for example, regular meetings between staff in each location to share ideas and experiences. Integration of the DAP with Wider Development Strategies. Currently the DAP, in the natural resource components of the program, is making significant contributions to two Intermediate Results under Strategic Objective 7 of USAID Rwanda's Integrated Strategic Plan (ISP). SO7 focuses on expanded economic opportunities in rural areas. The three intermediate results targeted under SO 7 are IR 7.1 expanded adoption of improved agriculture and business practices, IR 7.2 rural finance more accessible, and IR 7.3 rural infrastructure improved. The DAP is directly contributing to IRs 7.1 and 7.3. Through the Safety Net/LIFE components, the DAP is also making a significant contribution to IR 6.4 of the USADI ISP, improved community-level response to health issues, under the Health Strategic Objective. The safety net/LIFE components are building the capacities of community groups to promote positive health behaviors. The institutions working with the DAP are demonstrating expanded capacities for being able to respond to the needs of the most vulnerable populations. In terms of GoR development strategies and other key donors in Rwanda, the DAP has sought to be consistent with the strategies of the GoR when the strategies are well announced. For example, animal restocking activities were suspended following the government's announcement of a strategy to control the outbreak of a major animal disease. CRS Rwanda has undertaken a Strategic Planning Process (SPP) in which government strategies will be taken into account more rigorously as CRS crafts a strategic plan for the next planning period. This strategic plan will be a useful tool for ensuring that programs are embedded in wider development strategies. FY 2001 - FY 2005 DAP Final Evaluation Report 11 January 2005 40 4. Lessons Learned- Integration The major lesson that has emerged from the experience of the DAP relative to integration is that integration is best achieved in the design phase of a program. It is difficult to merge separately established projects or activities into an integrated program after they have begun implementation. F. Partnership 1. Overview of Partnerships in the DAP CRS has a number of different types of partners in the DAP. Implementing partners include Caritas Kibuye for the natural resource component in Kibuye, Caritas Rwanda in the LIFE component, CARE International also in the LIFE component, World Relief in the LIFE component and a local partner, ANSP+ in the LIFE component. The DAP relies on ACDI/VOCA for generating monetization proceeds. The major technical partner in the DAP is the Agricultural Research Institute (ISAR). Government partners include the Ministry of Local Government (MINALOC), the Ministry of Agriculture (MINAGRI), the Ministry of Land (MINITERRE), the Ministry of Health (MINISANTE) and the Ministry of Gender (MIGEPROF). The DAP works most closely with district and provincial representatives of these government departments. 2. Implementation of MTE Recommendations Related to Partnership The MTE did not make any specific recommendations related to partnerships. 3. Observations Relative to Partnerships Following are the key observations made relative to the partnerships in the DAP. Caritas Kibuye. While there have been significant bumps in the road over the course of the evolution of the relationship between CRS and Caritas Kibuye in the DAP, the relationship is currently at its strongest. LIFE Partners (ANSP+, CARE. Caritas Rwanda and World Relief). Relationships with the LIFE partners are functional and effective, although the nature of the relationship is more contractual than a mutually collaborative relationship. The DAP perceives its role as to determine policies and procedures and then is fairly rigid in enforcing them with the partners. This has occasionally resulted in some frustration on the part of some partners who are testing different approaches. In general, DAP program staff are generally unaware of the programs of the LIFE partners and how the support provided through the DAP fits into the integrated strategies of each of the partners. In addition to the food provided by the program, the DAP also provided a moderate amount of funding to three of the four LIFE partners to cover some of the operating costs associated with managing and delivering the food. The one partner that does not receive this support, ANSP+, is also the partner with the greatest need for such support. ACDI/VOCA. The relationship with ACDI/VOCA for monetization is functional and effective. Government Partners. At the central level with the various government ministries which have a stake in the program, the DAP has good relations with the Ministry of Local Government. DAP staff have paid courtesy calls on senior ministry staff in other ministries and entertained occasional field visits of ministry senior staff. Despite these efforts, however, most other ministries indicated that they may have heard about the DAP but knew FY 2001 - FY 2005 DAP Final Evaluation Report 11 January 2005 41 little about the strategy of the DAP, notably the Ministry of Agriculture (MINAGRI) and the Ministry of Health (MINISANTE). At the field level, especially in Butare, local government partners are generally well￾informed about the program and fulfill important roles in supporting program activities. ISAR. The relationship with ISAR is presently at its strongest. Although regular mechanisms for cultivating a more collaborative relationship are not in place, the DAP has developed a strong working relationship with ISAR for training and technology development. 4. Lessons Learned Relative to Partnerships Following are the major lessons learned relative to partnerships in the DAP. ¸ When significant physical distance exists between partners, special effort has to be made to develop, cultivate and maintain the relationship. In implementing the DAP, the relationship with Caritas Kibuye, for example, suffered because of weak communications early in the life of the DAP. ¸ There are terrific advantages to be gained through mutually collaborative relationships in which strategies, ideas and best practices can be shared between partners, raising the qualities of all programs. G. Commodity Management 1. Overview of Commodity Management Over the life of the DAP, the program received a variety of commodities for use in distribution, FFW and monetization activities. In the safety net/LIFE component, the ration consisted of maize meal, pinto beans or green peas (FY 2001 only), corn-soy blend, and vegetable oil. The FFW ration was composed of maize, pinto beans or green peas (FY 2001 only), and vegetable oil. Vegetable oil was the only commodity monetized. The program is expected to disburse 33,510.3 MT of food representing 96.7% of the total proposed LOA food. The allocations of food by program component are 15.4% for the Safety Net component, 37.8% for LIFE, 25.0% for FFW within the natural resource component and 21.8% for monetization. Table 5 summarizes the actual and projected commodity allocations over the life of the activity by fiscal year. Table 5. Projected LOA Commodity Summary (MT) Safety Net LIFE Natural Resource FFW Monetization TOTAL FY 01 1,050.2 2,693.6 3,435.9 1,587.7 8,767.3 FY 02 700.9 1,664.3 1,904.9 1,550.0 5,820.1 FY 03 1,106.2 2,766.5 2,287.8 1,543.6 7,704.2 FY 04 (Projected) 1,356.0 3,244.0 624.0 1,575.0 6,799.0 FY 05 (Projected) 940.0 2,313.0 117.0 1,050.0 4,420.0 LOA Projected Disbursements 5,153.0 12,681.4 8,369.6 7,306.3 33,510.3 Proposed LOA Quantities 5,118.0 12,639.0 9,551.0 7,335.0 34,643.0 Percentage Achieved 100.7% 100.3% 87.6% 99.6% 96.7% FY 2001 - FY 2005 DAP Final Evaluation Report 11 January 2005 42 Following the MTE, FFW activities were reduced, and 2,880 MTs of food in the pipeline for FFW was transferred to World Vision in February, 2004. 2. Implementation of MTE Recommendations Related to Commodity Management The MTE made two recommendations related to commodities, to reduce the use of FFW in the natural resource component, particularly for hillside soil conservation activities, and to transfer this food to the LIFE component to meet the rising demand for food with the growing membership in LIFE associations. The program fully implemented the first recommendation. The second recommendation was not implemented upon instructions from USAID. 3. Observations Relative to Commodity Management Key observations related to commodity management are provided below related to the ration composition, clearing and forwarding, commodity losses, and Regulation 11 compliance. Ration Composition. In general, the ration for distribution in the safety net/LIFE components was appropriate in terms of size and composition. The only flaw that emerged was that the ration was not suitable for infants. However, those institutions serving infants used their own resources to obtain appropriate food. Early in the life of the DAP, a decision was made to replace green peas with pinto beans, mainly for the LIFE component, since the latter are a more appropriate digestible food for persons affected by HIV/AIDS. In the FFW activities in the natural resource component, the value of the ration exceeded the minimum wage at certain times of the year. The ration was not based on a minimum wage criterion, but rather on a nutritional criterion. Clearing and Forwarding in Mombassa. For the first two years of the DAP, the transport and logistics company contracted by the DAP in Mombassa had moved the food to Kigali without a CRS presence in Mombassa. Problems began occurring in FY 2003, and CRS sent several warning letters to the supplier before finally sending one of its logistics staff to Mombassa. Investigation revealed that the contractor had been focusing more on seeking new contracts with other organizations and letting responsibilities on the contract with CRS slide. Commodity Losses. Table 6 below summarizes commodity losses over the life of the program. A detailed summary of commodity losses by commodity and fiscal year is included in Annex H. The major loss occurred in FY 2003 with a call forward of yellow corn that was shipped improperly. Apart from this loss, the DAP has otherwise exceeded industry standards for commodity management. Following the loss, CRS conducted a major audit of its commodity management systems and revised its pipeline management procedures in Mombassa in order to have tighter control over food resources. Table 6. LOA Commodity Loss Summary through FY 04 Shipments Received (MT) Fiscal Year Amount Purchased Ocean Losses Inland Losses Amount Received in Kigali Percent Lost FY 01 8,916.9 32.1 117.5 8,767.3 1.7% FY 02 5,858.1 19.7 18.3 5,820.1 0.6% FY 03 8,938.9 1,184.5 50.2 7,704.2 13.8% FY 04 4,980.4 28.7 27.2 4,924.6 1.1% TOTAL through FY 04 28,694.3 1,265.0 213.2 27,216.2 5.2% FY 2001 - FY 2005 DAP Final Evaluation Report 11 January 2005 43 Regulation 11 Compliance. At the time of the final evaluation, the commodity management systems in place from call forward to end use in the DAP were capable of effectively tracking commodities and meeting reporting requirements. Improvements were made in the systems following the internal commodity audit conducted by CRS following the FY 2003 losses. On the issue of the best structure for managing regional commodity pipelines, a more rigorous and comprehensive analysis is required, ideally engaging all CSs operating in the region. The options that are under consideration are (1) for CRS to manage multiple pipelines to CRS programs in different countries in the region versus (2) a consolidated pipeline for each country in the region in which commodities for all CSs operating in a country are consolidated under single management. Certainly the former is more costly. The additional cost needs to be weighed against the value of reduced commodity losses, as well as other economic costs related to organizational reputations and timely programming. 4. Lessons Learned Related to Commodity Management The major lessons emerging from the experience of the DAP relative to commodity management are described below. ¸ In identifying private contractors for clearing and transport of commodities, a competitive process must be used. The process should be replicated periodically to ensure that contractors do not become complacent in providing services. ¸ Call forwards must clearly specify shipping instructions for commodities to ensure that they are shipped properly. H. Monitoring and Evaluation 1. Overview of Monitoring and Evaluation Monitoring. Monitoring of implementation begins with development of implementation plans. CRS has revised the Detailed Implementation Plan for the DAP at least three times over the life of the program. The first revision occurred early in 2002 when it became apparent that the slow start in the agricultural component was delaying achievement of targets. The second revision occurred immediately after the MTE, and the final revision occurred after the decision from USAID was received on the proposed revised DIP and request for a no-cost extension. In each of the components, weekly team meetings are held to share progress and make plans for the coming week. Shortly after the arrival of the DAP Manager in August 2003, he implemented a monthly reporting system that eventually evolved into quarterly reporting. The main purpose of the reporting was to gather information to be able to keep the donor informed on changes being made and progress in the DAP. In the Safety Net components, monthly reports are received from safety net centers which list beneficiaries, food distributed, general subjective information on health and nutritional status, and other sources of resources. The reports are kept in the commodity section and are used primarily for monitoring and reporting on commodities. Evaluation. Three baseline surveys were implemented in support of the DAP. In February of 2000, during the design phase of the DAP, CRS carried out an organizational assessment FY 2001 - FY 2005 DAP Final Evaluation Report 11 January 2005 44 of its existing partner institutions in the Institutional Feeding Program11 . An agricultural baseline survey and a baseline study for the HIV/AIDS Project were conducted in September 2001. A Mid-Term Evaluation was conducted in May and June of 2003. The evaluation was highly critical and listed around 60 recommendations (including those listed in the Executive summary and those in the main text). 2. Implementation of MTE Recommendations Related to Monitoring and Evaluation Relative to monitoring and evaluation, the MTE made basically one recommendation at the program level and a number of recommendations relative to monitoring and evaluation in each of the components of the program. Annex C contains a table which summarize CRS's response to each of the MTE recommendations and an assessment by the evaluation team as to the quality of progress on each recommendation. On the critical program level recommendation which was to modify the indicator tracking table, CRS has revised indicators and updated the tracking table. The CSR4 reporting on FY 2004 results that was submitted on November 1, 2004 has the updated tracking table. On the other recommendations, for those related to the natural resource component, the DAP did recruit an M&E specialist who filled the position for two months in August and September of 2003, but the position has been vacant since. On the other recommendations oriented around identifying indicators and processes for assessing the effectiveness of CAHW, technology development, and road rehabilitation, some limited progress has been made. The DAP is making an effort to monitor environmental variables more closely. In the safety net component, the program has introduced a growth monitoring system (see section C above). On the two other recommendations made by the MTE (more follow-up on monthly reports and identification of organizational capacity indicators) little progress has been made. In the LIFE component, progress was made on both of the recommendations made by the MTE relative to M&E (reporting from LIFE centers and investigation of CARE's commodity distribution). 3. Observations Relative to Monitoring and Evaluation Monitoring in the DAP can be disaggregated into three focus areas, monitoring commodities, monitoring for reporting purposes and monitoring for programmatic decision making. Commodity Monitoring. In general, the systems for monitoring the procurement, receipt, transport and distribution of commodities is good. Monitoring for Reporting Purposes. The gathering of information for reporting purposes is also done effectively. Reports are generally submitted on time with sufficient information. Monitoring for Programmatic Decision-Making. Systems for obtaining information to be used for decision-making, however, are not very well developed. Information is gathered through periodic reports, but the information is rarely analyzed. In other cases, such as for example, monitoring the effectiveness of training activities or other investments made in capacity building, systems for obtaining information are not in place. A successful program never appears the same near the end of its life as it did at its start. M&E systems are used to modify strategies and implementation plans to be able to adjust to 11 The DAP proposal makes reference to the survey (p. 34), however, a copy of the survey summary or report could not be located. FY 2001 - FY 2005 DAP Final Evaluation Report 11 January 2005 45 contextual changes as well as to use experience to enhance the impact of the program. The DAP does indeed appear differently nears its end, mostly, however, because of the MTE and not because of the program's M&E tools. The lack of effective monitoring systems to facilitate programmatic decision-making seems to be mostly a function of skills related to using M&E information for decision-making. Evaluation. The baselines that were implemented for the DAP did not focus on information pertaining to impact indicators for the program, since impact indicators were not defined. They were descriptive in nature and more useful as assessment tools for guiding program design. Without a functional baseline against which to assess impact, impact has to be assessed with other more qualititative methods that can determine whether impact has occurred but are less effective at measuring the scale of impact. The MTE provided a large number of useful recommendations that have been instrumental in guiding CRS toward a more effective and efficient program. That said, however, the presentation of the MTE appears to have been somewhat counterproductive. A formative evaluation can be viewed in some sense as similar to a mid-year performance review for a staff member. The purpose of this review is to provide guidance to the individual on how to improve performance in a way that does not demoralize or demotivate her or him. In the case of the MTE, the overemphasis on corrective recommendations and virtual absence of recognition for the parts of the program that were functioning well resulted in a period of a few months when morale plummeted. As part of the restructuring after the MTE, the position of M&E Coordinator was established for the DAP. The position was filled for two months in August and September, 2003, but remains vacant up to now. Since the MTE, the DAP has made efforts to define performance indicators that are measurable and relevant, and the indicator tracking table has been updated. While there are some indicators intended to measure impact, for example, changes in yields of agricultural crops, the program has not defined impact indicators for capturing the impact of the program on food security at the strategic objective level. 4. Lessons Learned- Monitoring and Evalation The following major lessons have emerged from the experience of the DAP relative to monitoring and evaluation. ¸ If a program does not use M&E information for decision-making, it will not be able to respond effectively to changes in the operating context or use experience to improve the effectiveness or efficiency of its approaches. Monitoring is not just obtaining information on progress to be able to report to donors. Programs must evolve and program staff at all levels must have information available to them to make decisions to improve the efficiency or effectiveness of programs, i.e., modifying activities, discontinuing activities that are not effective, or expanding activities that achieve impact. If a program does not have useful M&E systems, the quality of the program is seriously threatened. I. Environmental Monitoring and Impact Mitigation 1. Overview of Environmental Monitoring and Impact Mitigation Environmental issues for the DAP program covered a wide range of concerns. As implementation began, other unforeseen environmental issues and concerns also arose, especially as marais development began. These issues and concerns were: · Use of pesticides or chemical fertilizers FY 2001 - FY 2005 DAP Final Evaluation Report 11 January 2005 46 · Increased disease incidence from disease bearing insects and water borne pathogens following increased, prolonged water flows following marais treatments · Contamination of water sources during the rainy season and from animal usage · Improper irrigation design or construction for the marais resulting in erosion damage to the irrigation structures · Soil fertility decline from soil disturbance during marais treatments and year round cropping of rice · Excessive hillside erosion as a result of poor water delivery design to hillside irrigation canals · Excessive erosion as a result of improper road rehabilitation and subsequent maintenance · Deforestation resulting from improved road access · Impact on rare or endangered species Many of these issues were recognized by CRS in the IEE and by the MTE team. The DAP formulated responses to each with means for monitoring and mitigating environmental impact. 2. Observations Relative to Environmental Monitoring and Impact Mitigation Generally speaking, the present DAP staff now realize from experience that most of the environmental issues and concerns first described in the IEE are real. DAP staff working in marais and hillside irrigation have incorporated training and technology transfer to DAP farmers on how to mitigate environmental problems associated with these activities. While the staff know how to correct irrigation system design problems, treat erosion, replace chemical fertilizers with organic fertilizers, they are not familiar with or trained how to mitigate some of the other potential problems, such as the proper use of pesticides, integrated pest management, water quality testing, or how to assess and mitigate human and animal health risks. The DAP has had problems with marais irrigation system design and construction and excessive erosion from improperly implemented hillside irrigation. The DAP staff have recognized these problems and have undertaken work to mitigate damages resulting from them. Water flows and the hydrology of the rehabilitated marais of the surrounding watersheds in Butare are also now being monitored to improve future work so that these problems will not occur again. Further, the DAP has successfully promoted the use of cut and carry feeding to reduce erosion potential from animal grazing and manure as a desirable alternative to chemical fertilizer use. Not all of the issues or recommended activities are being addressed, however. Water quality testing of some sources was done for a short time in Butare, but has not been monitored consistently and not done at all in Kibuye. Other issues have been recognized, but limited effort has been made to monitor or mitigate them. A study of available reports indicates that none of the commonly used pesticides for the crops grown in the DAP target areas are on any banned list (WHO, EPA). There are also no known rare endangered plants or animals living in the two program areas. 3. Lessons Learned The following major lessons have emerged from the experience of the DAP relative to monitoring and evaluation. ¸ Program field staff may not be familiar with environmental monitoring and mitigation methods, and training must be made available to them early in a program to ensure effective environmental monitoring. FY 2001 - FY 2005 DAP Final Evaluation Report 11 January 2005 47 ¸ Field staff need to coordinate environmental monitoring and mitigation efforts with District and Provincial officials to promote the sustainability of these activities. IV. PROGRAM IMPACT ASSESSMENT A. Natural Resource Component The following sections describe the outputs produced in terms of quantity and quality, the outcomes that resulted and the people who benefited from the natural resource activities of the DAP by each Intermediate Result in each location, Butare and Kibuye. 1. Outputs Produced Relative to IR 1.1 Intermediate Result 1.1 Increased lowland productivity for 15,000 families in targeted areas by 2005. Butare. The following outputs have been produced in the Natural Resource component under IR 1.1 in Butare over the life of the program. · 150 hectares of lowland have been brought under new irrigation infrastructure, although in most cases the infrastructure is not completely finished. · 2,739 families have access to new arable land in the lowland · Since June 2001, a total 8,369 MT of food has been disbursed through FFW12 · 33 potable water points have been installed Kibuye. The following outputs have been produced in the Natural Resource component under IR 1.1 in Kibuye over the life of the program. · 330 m of hillside irrigation canals have been rehabilitated where landslips have taken place disrupting water supply. This has renewed irrigation to 10 hectares. · Four hundred eighty-four farmers have planted the “preferred” sweet potatoes and have started harvesting them recently. 115,118 kg of cuttings have been repaid by farmers. The harvest is not complete and the payback is presently 45% of that distributed. 2. Quality of Outputs Produced Under IR 1.1 Butare. The marais was divided into six blocks for planning and implementation of infrastructure development. Different progress was made in the blocks in terms of the level of infrastructure completed. In Blocks 1 and 2, irrigation and drainage canals have not been completed. In Blocks 3 and 4, the structures for irrigation and drainage have been completed, but dams for water catchment are still under construction. Block 5 needs the main irrigation canal (canal de ceinture) to be completed. The DAP team reinforced the canal on the hillside to control siltage and water from the Kabakanka River, and according to the users, there was a significant and positive effect. Block 6 has most of the infrastructure completed, however there is a need to improve the main canal and to control the Gasuma River which may damage the infrastructure. The distribution of plots in the marais was properly done in general. However, there are disputes which arise and are resolved by local authorities on the sector level. A committee for land distribution was formed by the program to implement plot distribution with the support of local authorities. 12 8,369 MTs is the total for both Butare and Kibuye for all FFW under IRs 1.1, 1.2 and 1.6. FY 2001 - FY 2005 DAP Final Evaluation Report 11 January 2005 48 The potable water systems established by the program were well done in efficient ways with materials found locally. The sites are well located in densely populated areas. There was full participation of the users. This can be seen with the free labor provided by three elected water committee members who work each Friday on water point management issues. The control of the work is done by the community. Potable water users provide labor to construct or to rehabilitate those infrastructures. Water has been tested to see if it was potable Kibuye. The repairs in one irrigation system in the field were inspected, and the work was very well done. The system was being actively used and when we interviewed the association doing minor cleaning, etc of it, they said they were committed to maintaining the system and that they would now gain at least one more crop per year. The sweet potato distribution was done on a rotating credit scheme and has apparently worked quite successfully. Caritas Kibuye had the farmers receiving the cuttings sign a contract detailing what would be done with the harvest. 3. Outcomes: Scale of Impact on Food Security (and Livelihood Security) from Outputs Produced Under IR 1.1 Butare. Rice producers are getting approximately 300 kilograms of paddy rice per season from a plot of five ares. This represents an average value of 30,000 RwF. From the upper areas of the marais in which water is insufficient for rice, farmers produce Irish potatoes and maize. During the last season, they produced 30.6 MT of Irish potatoes and 33 MT of Maize in blocks 3, 4, and 5. It is clear that participating farm families have been able to increase their incomes. As a result, in addition to food security benefits, they also have easier access to health care. Parents are able to pay schools fees of their children, and rice producers are improving their houses and buying animals. Kibuye. In our interviews with farmers it became clear that a large portion of harvests are consumed by the farm families. Having better and longer growing conditions from the irrigation will substantially increase the food security of these families, as well of those of many others from the surplus produce that they will sell. 4. Who has Benefited from Outputs Produced Under IR 1.1? Butare. The beneficiaries from activities under IR 1.1 are the 2,151 families who produce rice in the marais and 588 families who produce in the bras de marais (upper areas of the marais with less water). These beneficiaries were people who had plots in the marais before the infrastructure was developed, small landholders with land on the margins of the marais and vulnerable groups selected by the community. For the most part these are small-holder farm families. In addition to these beneficiaries, others in the community benefit from the potable water points established by the program. Preference is not given to low income, marginalized or food insecure households for access to water. Everyone has access. Relative to gender, since IR 1.1 targets farm families, men, women and children are all benefiting from program activities. Kibuye. The DAP does not have systems in place for accurately determining whether participants fall under a definition of small-holder farm families. It can be inferred that certainly some of the beneficiaries fall within this category, however, the beneficiaries also include significant numbers of other households who lie outside of a definition of small￾holder farm family. FY 2001 - FY 2005 DAP Final Evaluation Report 11 January 2005 49 A significant number of the beneficiaries are women, since over half of the membership of the associations participating in the program are women. 5. Outputs Produced Relative to IR 1.2 Intermediate Result IR1.2 Reduced erosion from steep lands for 28,500 families in targeted areas by 2005. Butare. The following outputs were produced by the DAP under IR 1.2 in Butare. · 5,450 hectares of land were brought under improved soil conservation measures on hillsides · FFW distributions were made in support of hillside infrastructure development (see footnote 12). · Five participant associations were contracted by the DAP to produce forage plants and three associations to produce maize and Irish potatoes seed to be redistributed. Kibuye. In Kibuye, the following outputs were produced under IR 1.2. · 400 hectares of hillside have been treated to minimize erosion with tree and grass planting and the digging of contour ditches · 420 members of 14 associations have been trained in the managerial and technical skills to operate agroforestry-fruit tree growing nurseries. The associations have grown 361,016 seedlings (291,016 agroforestry seedlings for fodder or fuelwood production and 70,000 prun du japon fruit trees). · DAP has purchased French Cameroon grass cuttings (the preferred grass of farmers in Kibuye) and distributed on demand to farmers for planting along the berg of their contour trenches, providing more money into the communities in the target areas. · Grass plantings have been established on contour ditch berns that will provide not only erosion protection, but will supply animal feed, preferably cut and carry. 6. Quality of Outputs Produced Under IR 1.2 Butare. The watershed has been completely protected with discontiguous trenches. A variety of plants and trees have been established including agro-forestry species, animal fodder species, and species of trees to protect trenches. Some of these, including leucena calleandra and grevillea, also improve soil fertility. Labor for bulidng soil conservation strauctures was provided by local participants with technical supervision of CRS. Kibuye. The members of the associations formed by the DAP under IR 1.2 have been trained to set-up and operate the nurseries and to produce high quality seedlings. The nursery managers and farmers are well trained and motivated to plant the trees and grass properly. All of this in turn indicates that the DAP agricultural and technology transfer staff are sufficiently trained and doing a good job of passing their sills and knowledge on to the farmers. The species grown were those requested by the association members and others on the watersheds surrounding the nurseries. They have also been planted in sufficient numbers to effect erosion control (in combination with the contour ditches) on the farmers’ lands. This participation from the bottom up will enhance the opportunity for sustainability. The contour ditches will not only help stabilize the hillsides, they will provide moisture retention for more productive vegetable and other crop growth. This will provide increased crop yields and this in combination with increased animal production from feed provided by FY 2001 - FY 2005 DAP Final Evaluation Report 11 January 2005 50 the grasses grown on the ditch will produce even further food or income for livelihood security. 7. Outcomes: Scale of Impact on Food Security (and Livelihood Security) from Outputs Produced Under IR 1.2 Butare. The increase in land productivity brought about by the program will result in more food production and income for around 6,000 households. In addition, many other households received seeds, seedlings or cuttings through the program, including for agro￾forestry species such as calleandra leucene, grevillea, papayer, maracuja, taurilla cedrella Irish potato, sweet potato, improved cassava, maize and vegetables. The higher yields expected from these improved varieties will improve food security Kibuye. The fruit trees planted with the support of the DAP will not only serve as soil stabilizers, but will enhance the income and nutrition of those families on whose lands they have been planted. The were approximately 450 planters in the first planting and this will likely translate to increased food and income for some 900 planters by the end of program (approx. 4,500 people). The prun japon has already proved a high income earner in Kibuye from plantings in other projects and will enhance food and livelihood security. 8. Who has Benefited from Outputs Produced Under IR 1.2? Butare. Activities under IR 1.2 targeted all of the population on a hillside. Because the district selected for the program area is among the poorest, the beneficiaries are expected to include a disproportionately higher number of small-holders. Monitoring systems in the program, however, cannot provide information on how many of the actual beneficiaries fall within the target group for the program13 . Relative to the participation of women, approximately 55% of the participants are female and 45% are male. Kibuye. Well over 400 families have benefited directly from the program to this time. The families downstream in the watersheds will also benefit from less flooding potential and less deposit of gravel and rocks on their lands. Information is not available from the program to determine what proportion of these beneficiaries actually fall within a definition of small￾holder farm family. 9. Outputs Produced Relative to IR 1.3 Intermediate Result IR1.3 Selected farms in targeted areas restocked with 13,500 goats and swine by 2005. Butare. Outputs produced in Butare from activities implemented under IR 1.3 include the following: · 20 community animal health workers (CAHW) recruited and trained · 17.09 hectares of trypsacum, 14.8 hectares of pennisetum, 2.60 hectares of mucuna and 1.05 hectares of desmodium were planted for multiplication and dissemination · An association of CAHWs was formed Kibuye. In Kibuye, the only activities undertaken under IR 1.3 were to provide training on animal shelter construction or rehabilitation, but no shelters were constructed or rehabilitated following the training. 13 The DAP has not developed a practical definition for identifying who is a "small-holder". FY 2001 - FY 2005 DAP Final Evaluation Report 11 January 2005 51 10. Quality of Outputs Produced Under IR 1.3 Butare. The recruitment of the CAHWS was properly done. They received training on basic animal health care, accounting and financial management . The CAHWs have good working relationships with the district veterinarian. CAHWs have grouped themselves into an association and they have started collecting cash in order to open an account in the local bank. Kibuye. There appears to be some discrepancy between what Caritas Kibuye and CRS expected in Kibuye for activities under this IR. Caritas interpreted their responsibility to mean advice only or support following farmer requests, while CRS assumed that some financial support for construction/rehabilitation would be provided by the program. Accordingly, nothing has been done other than to identify shelters needing construction/tehabilitation and some limited advice provided on what is needed for rehabilitation in response to specific farmer requests. 11. Outcomes: Scale of Impact on Food Security (and Livelihood Security) from Outputs Produced Under IR 1.3 Butare. Because the CAHWs have not yet received the kits and medications that the program had planned to make available, the impact of activities under IR 1.3 have been limited to the benefits that have accrued to the CAHWs themselves and their immediate contacts. 12. Who has Benefited from Outputs Produced Under IR 1.3? Butare. Traditionally, households who own animals, especially large ruminants, are not generally considered the poorest in the community. 13. Outputs Produced Relative to IR 1.4 Intermediate Result IR1.4 Productivity enhancing technologies introduced to 15,000 families by 2005. Butare. Outputs produced in Butare as a result of activities implemented under IR 1.4 include the following: · There are one paysan animateur, three members of an association committee, and two field schools that have been formed on the hillside. · There are three paysan animateurs and three field schools in the marais · There are 28 other paysan animateurs in the program area who serve as extensionists for other farmer associations formed by the program · The farmer associations have produced 33 MT of maize seed of which CRS bought 2.5 MT for distribution in an exhibition show. The district bought 7.5 MT for multiplication for the population, and the rest was used by the association members themselves or sold to other farmers outside of the program area. · The farmer associations produced 30 MT of Irish potato seed which was sold. Kibuye. Outputs produced in Kibuye as a result of activities implemented under IR 1.4 include the following: · Twenty farmer associations received technical training to plant and tend variety trials · Six trials were established in the marais and fourteen on the hillsides, including two rice trials (both in the marais), ten maize trials (three in marais, seven on hillsides), eight Irish potato trials (one in marais, seven on hillsides) FY 2001 - FY 2005 DAP Final Evaluation Report 11 January 2005 52 · Seven sweet potato multiplication demonstrations were established by trained organizations (two in the marais, five on hillsides) · Two sweet potato varieties have been accepted as superior to the others tested for performance and taste and 78,311 cuttings collected from multiplication trials already harvested · Two model farmers received managerial and technology transfer training · ISAR will participate in the monitoring and evaluation of the field trials that will be harvested over the next few months · Fourteen farmer associations have been trained to manage nurseries growing agroforestry and fruit tree species · One hillside irrigation committee has been trained in source water management for irrigating hillside crops · Farmer evaluation of cropping patterns for the marais that are programmed for reclamation has been completed (identifying crops and cropping sequences by season) 14. Quality of Outputs Produced Under IR 1.4 Butare. In Butare, the yield of maize was 3.5 MT/ha, 17.6 MT/ hectare for Irish potato, 16.5 MT /ha for sweet potato and 5 MT/ha to 6 MT of rice, 10 MT/ha of cassava. For comparative purposes, the yields for maize from non-improved seed average 1.2 MT/hectare, 5 MT/hectare for Irish potato, 3 MT for sweet potato, 1 MT/ hectare for rice, 5 MT/ hectare ofor cassava. Kibuye. In Kibuye, the agriculture staff received training in technology transfer concepts and methods from ISAR. These staff live and work in the same sectors as the association members with whom they are working and they meet with the association members at least 2 times per month. Technology transfer is being done by all of the NRM staff, and both the agricultural staff and the technology transfer staff find this cross responsibility makes their jobs more interesting. The farmer associations receive organizational and technical training at least one day per month. 15. Outcomes: Scale of Impact on Food Security (and Livelihood Security) from Outputs Produced Under IR 1.4 Butare. In Butare, 1,200 families received and planted improved bean seeds. 1,000 families received and planted improved maize seeds. 251 associations received and planted improved Irish potato seeds. 2,500 families received and planted improved sweet potato seeds. 2151 families received and planted improved rice seeds. As a result, the higher yields bring more production and more food is available in the region and for participating families. Kibuye.. Several hundred farmers will benefit from knowing which varieties of several food crops are most productive, provide produce of better taste or are preferred by farmers for some other reason. Several hundred other farm families will benefit from the sweet potato multiplication trials either as food, income from market sales or as a source for further reproduction. The improved water delivery and irrigation system maintenance provided by the water committees will result in higher crop yields that will likely increase annual harvest by several percent. All of this will have a very beneficial impact on food and livelihood security. Since most of these outcomes are part of a chain of expanding production through use of more productive varieties and multiplication, the scale of impact will be very significant in the target areas, even if it can not be exactly quantified at this time. FY 2001 - FY 2005 DAP Final Evaluation Report 11 January 2005 53 16. Who has Benefited from Outputs Produced Under IR 1.4? Butare. Beneficiaries from activities under IR 1.4 include those families who had access to improved seeds in targeted program areas or outside the program area. During the distribution of plots in the lowlands, the majority of new owners were female or widows, girls who headed households and other vulnerable groups mainly landless people. On the hillside, one of the criteria to select associations which had to produce improved seeds was to be a member of a vulnerable groups, defined as those who are food insecure. In the lowlands, the proportion of participants was 44% female and 56% of male. On the hillside, 52% of the association membership was female and 48% male. Kibuye. The participating farmers, consumers of their produce, and farmers adopting varieties proven superior performers will all benefit from outputs produced under IR 1.4. The farmers participating in the agro-forestry and fruit tree production activities will also have a chain of benefits similar to that mentioned for the trials. Those other farmers not participating in the program who have adapted and adopted varieties introduced by the program will have these same benefits. Because of the selection process used for identifying "successful" associations for participation, most participants are likely not be from among the poorest farmers. They are also not likely to be the most affluent. Anecdotal evidence suggests that the most affluent farmers do not want to belong to associations for various reasons. More than 50% of the participants in activities under IR 1.4 were women. 17. Outputs Produced Relative to IR 1.5 Intermediate Result IR1.5 Farmers’ associations established or strengthened in targeted areas by 2005. Butare. Outputs produced in Butare from activities implemented under IR 1.5 include the following: · There are one hundred and five associations formed in the marais, and each association has a management committee of three members. At the level of each block there is a management committee of three people. All the committees are operational when it comes to distributing water in different plots or when it comes to reducing the speed of the water in the main canal or to clean canals · The one hundred and five associations of rice producers in the marais were grouped into one federation · Twenty CAHWs were grouped in one association. · Five farmer associations were grouped together for forage plant production and producing plants to protect trenches · Thirty-two water management associations were formed · Three associations were formed to produce agro-forestry species · Forty-three farmer associations were formed for maize and Irish potato seed production Kibuye. Outputs produced in Kibuye from activities implemented under IR 1.5 include the following: · Twenty-five associations having 460 members have been trained in organizational and financial management FY 2001 - FY 2005 DAP Final Evaluation Report 11 January 2005 54 · One organization is presently rehabilitating two storage depots for stocking maize and potatos · One marais committee has received initial management training 18. Quality of Outputs Produced Under IR 1.5 Butare. In Butare, some associations existed before the DAP began implementation and had received support related to emergency response programs from different organizations. Others have formed spontaneously from observing the experience of some of these associations. In the marais, all associations were newly formed with DAP support. On the hillside, four associations decided to form one big association for seed production, especially for species which fix soil, and for marketing, to construct a storage facility. Management committees were selected by the community and most of the time members were selected from among the vulnerable groups. All of the committee members have been elected, there are regular meetings and Centre IWACU has been contracted by CRS to train members of committee. Because rice is a major source of cash income, all the committees are strongly engaged in the work, and rice producers monitor the committees to make sure the work is fairly done. Trainings and study tours were organized by DAP staff for capacity building, for example, for seed multiplication, water management for rice production, canal cleaning, animal health care, accounting and financial management. Kibuye. All Kibuye staff have received a brief, initial training in their area of responsibilities relating to association organization. Association interest in DAP activities is maintained in part by having at least one meeting per month with DAP staff 19. Outcomes: Scale of Impact on Food Security (and Livelihood Security) from Outputs Produced Under IR 1.5 Butare. The work with various associations has resulted in easier access to services and dissemination of new ideas and technologies for participants. The association structures also facilitate effective resolution of disputes and conflict. With well-defined leadership and governance within associations, members tend to take more community responsibility. Water committees, for example, have been effective in mobilizing members to clean and maintain water points. There is a stronger sense of ownership in the management and maintenance of structures, more so with the water points and irrigation infrastructure than with roads, however. Most important, perhaps, people are conscious that they have a larger voice to defend their interests through being members of an association. All of these have impact on food and livelihood security. Kibuye. All twenty-five associations are trained and functioning well. The exact impact of this on food and livelihood security can only be guessed at this time as no baseline or harvest data are yet available. In all cases, the association members interviewed, officers and members at large, indicated that the associations were functioning in a democratic manner, with all having an equal voice in directing their leadership. Although some internal dissatisfaction must surely exist in any organization, none of any serious nature was detected in the DAP associations. This may reflect that the associations in Kibuye were chosen from those already functioning successfully, thus probably already having resolved internal organizational differences between members. 20. Who has Benefitedfrom Outputs Produced Under IR 1.5? Butare. In Butare, a range of beneficiaries are served by the putputs produced under IR 1.5, including the members of the different associations as well as others who benefit from FY 2001 - FY 2005 DAP Final Evaluation Report 11 January 2005 55 services provided by the from associations. Many ofthese are vulnerable groups who can not work in umuganda. Many, however, are also not food insecure or members of the small￾holder farm family target group for the DAP. In the lowland associations, membership was approximately 44% female and 56% of male. In the hillside associations, 52% of the membership was female and 48% male. Kibuye. In Kibuye, all of the association members have benefited directly from outputs produced under IR 1.5, and many more will benefit because the trials and multiplication plantings have performed well, in part as a result of good organization. However, as a result of the criteria used to select associations for participation (see Section B.19), some small￾holder farm families may have been excluded from participation. Selection of associations having bank accounts, for example, suggests that their members have more than average resources. In any case, at least 50%, and perhaps as many as 60%, of the participants in the DAP are women. 21. Outputs Produced Relative to IR 1.6 Intermediate Result IR1.6 Improved market access for 43,500 families in targeted areas by 2005. Butare. Outputs produced in Butare from activities implemented under IR 1.6 include the following: · 109 km of roads have been rehabilitated. · 10 bridges have been repaired and built · FFW food distributed to local community workers (see footnote 12 above) Kibuye. Outputs produced in Kibuye from activities implemented under IR 1.6 include the following: · 17 km of road to key markets rehabilitated · 3 bridges built to contracted standards · 8.6 km of roadside ditches dug or cleaned · 9.1 MT of food distributed to local community workers 22. Quality of Outputs Produced Under IR 1.6 Butare. For construction of bridges, a professional construction company was hired through a competitive process. For the moment, the roads and bridges are in good condition even though the rains have been heavy. This is a positive indicator that the construction, including drainage, has been done well. There is a on-going need for minor maintenance, and the umuganda is providing the required labor. Kibuye. In Kibuye, the bridges were built by contractors selected through competitive bidding . The work was done to accepted, professional standards using skilled labor (masons, etc.), and the roads that have been rehabilitated are still readily passable, several months after their rehabilitation (including a rainy season). This compared with pre-DAP evidence that indicated that the DAP work was successfully done. 23. Outcomes: Scale of Impact on Food Security (and Livelihood Security) from Outputs Produced Under IR 1.6 Butare. As a result of the improved transport infrastructure, people have easier access to their production sites, easier communication with district authorities and sector staff, easier FY 2001 - FY 2005 DAP Final Evaluation Report 11 January 2005 56 transportation of produce to the market, easier access to agricultural inputs, health facilities, and schools. All of these have resulted in improved food and livelihood security. The numbers of person affected, however, and the scale of benefits cannot be accurately estimated with the information available. Kibuye. In Kibuye, the short term impact of the food deliveries for FFW on livelihood security was positive and significant as evidenced by workers shifting their labor to other programs having FFW in the communities around the targeted roads. There was no baseline data on road traffic and no data on this has been collected since the roadworks were started. Therefore, it is not possible to make a reliable estimate of the impact or changes in the amount of goods using these roads. 24. Who has Benefited from Outputs Produced Under IR 1.6? Butare. Roads and bridges benefit whole communities including small-holder farm families. Given the information available, it is not possible to say now whether the particular roads and bridges rehabilitated in the DAP have produced disproportionate benefits for small-holders. Kibuye. There have been several beneficiaries from the activities for this IR, including: o The recipients of the FFW and their families o The contractors, their skilled laborers (and their families) who have received cash payments for contracted work completed on the bridges o The skilled and other labor paid by the contractor o Although limited, the truckers, merchants and others using the roads that have been rehabilitated Only the first group clearly falls within the definition of small-holder farm families. B. Safety Net 1. Outputs Produced The following outputs have been produced in the Safety Net component over the life of the program. · 3,929.7 MT14 of food has been distributed through FY 04. · Between 6,292 to 7,069 beneficiaries have been fed per year beginning April 2001 of which 55% were children in orphanages, 27% were street children, 14% were in handicapped centers, 4% were elderly in homes for the aged, and a small percentage were children living in prisons with their mothers. Table 7 summarizes the food recipients per year over the life of the DAP. · A total of 58 Safety Net institutions received food through the program, with 56 still participating. · The following training programs were implemented over the life of the program: o Entrepreneurial skills and income-generating activities conducted from February through May of 2002 o Two food management trainings done in 2001 and 2004 o April/May, 2004, training on nutritional assessment tool with nutrition education · Capacity for commodity management built in safety net institutions through on site monitoring and provision of technical advice for commodity management by end use 14 This figure is the sum of amounts recorded in the commodity management system for FY 2001, FY 2002 and FY 2003, plus the FY 2004 figure reported in the latest CSR4. FY 2001 - FY 2005 DAP Final Evaluation Report 11 January 2005 57 checkers through visits approximately once every two months to each safety net institution · Social capital established between safety net institutions through three annual partnership meetings which were held usually in March of each year for all safety net institutions · Fourteen grants for Income-Generating Activities (IGAs) have been made to safety net centers, totaling 17,713,310 RwF (approximately $34,627). Annex E contains a list of the recipients of the grants. Table 7. Summary of the Number of Food Recipients in the Safety Net Component Fiscal Year Distribution Period Number of Recipients Explanation of Changes FY 01 April 2001 through September 2001 7,011 to 7,069 One orphanage accepted a large number of repatriated unaccompanied children FY 02 Entire Year 7,069 to 6,,982 Internal fluctuation FY 03 Entire Year 6,982 to 6,292 IRC began implementing a reunification and foster home program FY 04 Entire Year 6,292 to 6,999 Two centers closed (one for administrative reasons the other because of success with the IRC program) and 6 new centers added (four for street children and two handicapped centers) FY 05 October 2004 through July 2005 6,999 to 6,844 Internal fluctuation 2. Quality of Outputs After almost five years of commodity distribution and support from CRS staff, safety net institutions demonstrate strong capacity for managing commodities. The warehouses observed were all well-kept with up to date stock cards. The Safety Net institutions who provided quantitative information for the sample survey carried out as part of the final evaluation were able to complete the information to the level of detail requested. Annex F contains the summary tables for the survey which obtained information from 17 of the 58 safety net centers. The capacity is good, although the sample survey did note discrepancies between the food received by each institution and the amount recorded in the commodity records kept by CRS, indicating additional capacity building may be necessary. In general, however, the DAP has established strong capacity in safety net institutions for effective commodity management. Those training programs for commodity management were consistently cited by participants as useful and effective. Relative to other capacity building with safety net institutions, the IGA training was effective in enabling institutions to prepare proposals. The amount of resources made available, however, to safety net institutions for IGAs over the life of the activity was far below the target of $50,000 per year. Additional proposals were received over the three grant cycles that were implemented, however, CRS was not convinced by the quality of the proposals to make funding available, and the DAP did not have additional staff who could follow up with the safety net institutions submitting the proposals to provide additional technical support to strengthen the proposals. The nutrition education and growth monitoring training was also cited by a number of institutions as a useful capacity building event, however, because of fairly limited follow-up, the momentum established by the training may have been lost. Some institutions with health care or nutrition staff have started using the growth monitoring tools to varying degrees, FY 2001 - FY 2005 DAP Final Evaluation Report 11 January 2005 58 others have adapted the tools while still others are waiting for the program to provide more technical assistance. In the annual partnership meetings in which all safety net institutions met to discuss the program, the DAP was able to cultivate relations between institutions as well as to address concerns and issues. These meetings were useful and effective. 3. Outcomes: Scale of Impact on Food Security (and Livelihood Security) Without question, there is visible impact on the food security of participants, who are almost by definition highly food insecure. The M&E systems for the DAP are unable to pick up the scale of impact15 , but anecdotal information confirms that impact has occurred. Visitors to the safety net institutions comment on children's health. Children show more energy and enthusiasm for school and their studies. Mothers from neighboring households bring extremely malnourished children to the institutions seeking support, and so on. The DAP has contributed substantially to its second strategic objective through activities implemented under Intermediate Result 2.1 in the safety net component. For Intermediate Result 2.2, however, the outcomes are less obvious. Some capacity building of safety net institutions has roccurred, mainly from the release of other resources resulting from the food provided by CRS. Resources used in the past to procure food have been used to purchase condiments or other commodities, for investments in income generating or food production activities, or for other needs such as school fees. This also applies to the human resources that were used to procure food. Staff of the institutions no longer have to run behind obtaining resources for food and can focus their attention on other matters. Some centers have demonstrated visible capacity improvements, e.g., expanded services, expanded IGAs, increased physical capacity, etc. Other centers have more or less breathed a sigh of relief that they no longer have to struggle to find food for their beneficiaries. Capacity built by CRS through strategic capacity building activities, however, is limited to commodity management systems and the impact of IGA grants. On the commodity side, the capacity building has been highly successful. A random sample of safety net institutions was visited during the evaluation, and all had effective systems in place for keeping stock, monitoring disbursements and reporting. Information on food disbursements was always readily at hand. The impact of this expanded commodity management capacity in terms of contributing to the strategic objective is tied to the provision of DAP food, however, and does not contribute much to the ability of a safety net institution to be able to become self-reliant. Relative to capacity built through IGA grants, some of the IGAs supported by the DAP have become established, others have either failed or declined. The activities supported by the grants were designed to be used for training, production of food for consumption, and/or income-generation to support the center or buy other food commodities to complement the CRS food. There is only minimal evidence that the IGAs are really being used for training. Beneficiaries provide labor for the IGAs but rarely assume roles as managers or apprentice managers. A few participants do replicate IGAs when they leave the institution, but it is relatively few. Some impact has been achieved toward enabling a few safety net institutions to move toward self-reliance, however, the number of institutions involved and the scale of activities are too small to say that there has been much impact on the strategic objective. 15 There are some very interesting opportunities to assess the impact of DAP activities with indicators such as changes in mortality rates (in homes for the elderly), morbidity rates, health care costs, school performance, education national exam performance, nutritional status comparisons when children go on holiday and return, growth monitoring, and so on. As mentioned, however, impact indicators were not defined by the program. FY 2001 - FY 2005 DAP Final Evaluation Report 11 January 2005 59 4. Who has Benefited? Generally beneficiaries were well targeted by the safety net program, especially for those recipients who are housed within the safety net institutions. Some of the food provided by CRS is distributed to others in the safety net institutions who may not be food insecure, such as cooks or laborers who are preparing the food. Since these people are necessary to achieve the desired food security impact of the targeted beneficiaries, this sort of slippage on targeting is relatively harmless. Where there appears to have been slippage on targeting that would reduce the impact of DAP activities on food security was in those centers that work with foster children, i.e., street children or orphans that have been placed with foster families. When targeted recipients come to receive their meals, they often bring other children from the families that are supporting them. Some of these other children are likely to also be food insecure, although it is equally possible that many are not. Table 8 provides a breakdown of the age and gender of DAP program food recipients in the safety net component by type of institution. The most noticeable pattern emerging from the data is that girls and women are underrepresented in the street children's centers. Table 8. Disaggregation of Food Recipients in Safety Net Institutions Type of Institution Percentage Male Over Age 18 Percentage Female Over Age 18 Percentage Male Under Age 18 Percentage Female Under Age 18 Orphanages 12.4% 13.3% 42.6% 31.7% Street Children's Centers 16.4% 2.4% 58.7% 22.5% Homes for the Elderly 25.5% 46.2% 16.0% 12.3% Homes for the Disabled 44.9% 46.5% 4.9% 3.8% Total for All Institutions 16.2% 14.0% 43.2% 26.6% C. LIFE 1. Outputs Produced The following outputs have been produced in the LIFE component over the life of the program. · 8,155.5 MT16 of food has been distributed through FY 04. · Approximately 4,400 households have been supplied with food per year beginning in April, 2001. · A total of 66 Associations received food through the program. · These associations received training on the following topics over the life of the program: o Commodity management o Nutritional surveillance o The Fight against HIV/AIDS · Expanded commodity management capacity through on site monitoring and provision of technical advice by end use checkers through visits approximately once every two months. 2. Quality of Outputs The project has helped create a sense of solidarity among PLWHAs, who now do things together like home and hospital visits. The project has also fostered a stronger commitment 16 This figure is the sum of amounts recorded in the commodity management system for FY 2001, FY 2002 and FY 2003, plus the FY 2004 figure reported in the latest CSR4. FY 2001 - FY 2005 DAP Final Evaluation Report 11 January 2005 60 of the beneficiaries to the fight against the HIV/AIDS and has improved the lives of beneficiaries by increasing their productivity and household income. 3. Outcomes: Scale of Impact on Food Security (and Livelihood Security) Because of the food disbursed through the DAP, beneficiaries have seen an increase in their health status and ability to work and create revenue for themselves. They are also able to participate actively in educating others about HIV/AIDS and the behavioral change required to reduce infection. While the scale of improved health status cannot be measured in the absence of a baseline, many stakeholders, including implementing partners, CBOs, local authorities and particularly the beneficiaries themselves remark that the improved health status of people receiving food has decreased morbidity, mortality and hospitalizations. The program has also reduced the stigma and discrimination associated with infection from HIV/AIDS, which has facilitated the socio-economic integration of PLWHAs and orphans. There is more community solidarity resuting in more home visits, visits to the sick in the hospital and people implementing income-generating activities together. During the implementation of the evaluation, it was observed that there were significant numbers of PLWHAs who had not received food through the program. Since they generally are in an earlier stage in the disease, their health status is still robust and they have not yet shown appreciable signs of AIDS. These people do, however, have a higher incidence of malnutrition in their families especially young infants and older women. Annex G contains a summary of information obtained through a rapid survey of a sample of LIFE associations. Of 55 households surveyed totaling 278 members, 20% have less than four members per family, 60%, have between four and six members and 20% have more than six members per family. The amount of food distributed in the LIFE component is calculated on the needs of a family of five which means that 20% of the beneficiaries are not getting enough food and 20% are getting more food than they need. Of the 278 members, 9% are children under six years old. 4. Who has Benefited? The number of 4,400 households was attained and continues to grow following sensitizing by association members. The members of the association are practically all women of lower socio-economic status and were extremely vulnerable prior to receiving food. V. RECOMMENDATIONS FOR THE DESIGN OF NEXT DAP A. Key Principles to Guide the Development of the DAP Following are some suggestions regarding principles to bear in mind as CRS continues to develop a new DAP. Use of Food. The use of food for distribution is highly appropriate for the targeted beneficiaries in the safety net and LIFE programs, for the most part. Food may not be the best intervention, however, for safety net centers that support foster children. While the food does provide an incentive for children to participate in institution-supported programs, mainly basic education or vocational training, since they are placed in foster homes, a more relevant intervention would be activities to expand the livelihoods of the foster families so that they can afford to feed the children themselves. FFW is customarily used to meet the food security needs of the highly food insecure while at the same time to build community assets. It is generally ineffective in areas where food in security is not significant or when used to build household assets, and not community assets. FY 2001 - FY 2005 DAP Final Evaluation Report 11 January 2005 61 Being More Participant Driven. The resources and opportunities available to communities, groups within communities, or institutions working in communities are unique to each community. No one knows better the resources and opportunities available to them than the people themselves. To capitalize on this uniqueness, communities must be allowed to modify interventions brought to them by outsiders to fit more closely with their circumstances. Participatory processes can be used to facilitate program stakeholders in communities to analyze their constraints and opportunities and develop their own implementing plans for achieving the results envisioned by the program, letting them "drive" the process. Livelihoods Programming. The agriculture activities appropriate for a small-holder farm family are very different then for a safety net institution, a young adult about to graduate from a safety net institution or a household who has someone infected by HIV. A livelihoods programming approach takes into account these differences by first clearly defining a target group, analyzing the livelihood strategies of this group, and identifying appropriate interventions for building assets within the livelihood strategies of the selected target group. In many cases, a wide range of possible interventions has been identified through a livelihoods analysis, often more interventions than are possible for a single program given the resources available. In cases such as these, new programs need to look at other existing programs and linkages with these through partnerships. A single program cannot usually do it alone. Linkages and partnerships are critical to effective livelihoods programming. Targeting. Effective targeting is critical at all stages of program design and implementation. In the design phase, a clearly defined target group facilitates identification of interventions that are likely to have the greatest impact for that target group. During the implementation phase, a program must maintain a strong focus on the groups upon whom the program intends to have impact in order to ensure that program benefits do indeed go to these groups. Remember - Do No Harm. In designing and implementing development or developmental relief interventions, it is critical to understand not only how benefits accrue to targeted beneficiaries, but also how program activities affect other groups, either positively or negatively. A Do No Harm analysis facilitates identification of other groups that will be affected by an intervention and enables programmers to consider these affects in designing interventions or modifying them during implementation. Exit Strategies. An exit strategy is critical to ensuring that development or developmental relief interventions achieve sustainable impact. An exit strategy, at a minimum, defines a sustainability vision for a program in terms of what the program wants to see sustained after it has ended. This can include the impact that needs to be sustained, the services that need to be sustained or the institutions that need to be sustained. Once the sustainability vision has been defined, the obstacles and opportunities to achieving the vision are identified and the program defines activities to remove obstacles or capitalize on opportunities. Monitoring and Evaluation. Given the impact that weak M&E systems have had on the current DAP, it is paramount that in the next DAP, effective M&E systems be put in place. This involves building the capacities of staff at all levels to value and use M&E information for decision-making. Once this has been completed, then appropriate tools should be designed that will provide relevant information in cost-effective ways. An additional dimension of M&E that will be important in the next DAP is to engage participants in designing and implementing participatory M&E systems. In this process, participants define performance indicators, decide what information and processes are required to measure the indicators, analyze information and make decisions based on the analysis. FY 2001 - FY 2005 DAP Final Evaluation Report 11 January 2005 62 Participants and Beneficiaries. A wide range of people participate in development programs. Not all of these, however, represent those persons upon whom the program has been designed to have beneficial impact. In order to distinguish between those upon whom programs are targeting impact and those with whom it is necessary to work to achieve the impact, it can be useful to refer to the former as participants and the latter as beneficiaries. B. Building on the Foundation Established in the Current DAP The following sections summarize the views of the evaluation team on the most effective strategies for building on the investments made in the current DAP to further enhance impact on food security. Natural Resources in Butare. Infrastructure construction activities in Butare will be virtually complete by the end of the DAP. Unfortunately, capacity building of the various groups responsible for managing the infrastructure has lagged. In the next DAP, the primary focus should be on group capacity-building to enable the "soft" side to catch up with the "hard". Specifically, the next DAP may want to consider: * Discontinuing FFW because the community infrastructure work will be completed * Expanding capacity building activities for the various groups established by the current DAP using a farmer driven approach. * Expanding the Agro-Enterprise Development pilot activities with the specific intent to enable the program to become more market-driven. * Using a livelihoods programming approach to identify complementary interventions that will improve the livelihood security of targeted beneficiaries and communities. * Exploring opportunities to work in collaboration with one or more local implementing partners to establish effective communications, understand their broader development strategies, and build their capacities to replicate the strategy of the program consistent with their goals and objectives. * Defining an exit strategy in which CRS will be able to step away at the end of the next DAP with confidence that the investments in agriculture will continue to generate benefits for the food insecure. Natural Resources in Kibuye. In Kibuye, the progress on infrastructure development as proposed in the current DAP has been slower. The program has really only in the last year been able to achieve significant progress, and it is the feeling of the evaluation team that a period of additional time is necessary to proceed far enough with establishing the basic infrastructure. As in Butare, the "soft" side of building group ownership and management capacities may be lagging, so significant time and resources should be dedicated in the next phase to building up this side of the strategy. Specifically, the next DAP may want to consider: * Fully engaging Caritas Kibuye in the design of the next DAP so that the strategy that emerges is highly consistent with the goals, objectives and vision for both CRS and Caritas Kibuye. * Continuing the road maintenance and rehabilitation activities with limited food for work. * Understanding the nature of the current groups in the program, all of whom were formed before the current DAP for various reasons, to confirm that their goals, objectives and values are compatible with those of the DAP. FY 2001 - FY 2005 DAP Final Evaluation Report 11 January 2005 63 * Expanding capacity building activities with the various groups who have been identified and new groups formed where necessary using a farmer driven approach. * Expanding the Agro-Enterprise Development pilot activities with the specific intent to enable the program to become more market-driven. * Using a livelihoods programming approach to identify complementary interventions that will improve the livelihood security of targeted beneficiaries and communities. * Defining an exit strategy in which CRS will be able to step away at the end of the next DAP with confidence that the investments in agriculture will continue to generate benefits for the food insecure. Safety Net. Food is an appropriate resource for safety net programming. In most nations, however, it is the function of government to manage the social safety net. Two factors in Rwanda, however, justify additional support in the near term for the safety net, the effects of the 1994 genocide and the rising number of orphans from the effects of HIV/AIDS. The current DAP has succeeded in building capacities to manage the food. The attention now should be to complement that with capacity building for safety net institutions and the individuals that reside in them to become self-reliant. Specifically, the next DAP may want to consider: * Continuing with the supply of food to safety net institutions in which beneficiaries reside, but gradually reducing the amount of food provided by the program in order to cultivate more self-reliance. * Expanding substantially capacity building activities for safety net institutions in participant-driven ways that capitalize on the resources and opportunities unique to each institution in order to enable them to become more self-reliant. * Supporting the expansion of education and vocational training opportunities for beneficiaries of participating safety net institutions. * Working more closely with the Government of Rwanda and other NGOs to sustainably reduce the safety net caseload through reunification or reintegration. * Defining an exit strategy in which (1) the needs of the safety net caseload will be adequately met by the safety net institutions with the support of government and other stakeholders and (2) children will be able to graduate from the safety net to sustainable livelihoods when they become adults. LIFE. In the near term, the number of people living with HIV/AIDS is likely to grow and they and their households are highly vulnerable to food insecurity. The need for support will likely continue to grow especially since membership in PLWHA associations is increasing and the time period over which food assistance may be required is getting longer. A coherent strategy for solving the problem needs to be developed. Specifically, the next DAP may want to consider: * Continuing with the supply of food to PLWHA associations through the current partners, but gradually reducing the amount of food provided by the program. * Expanding substantially capacity building activities for both associations and households in participant-driven ways to enable associations to meet more of the needs of their constituents and households to become self-reliant. * Building the organizational capacity of ANSP+ to more effectively deliver services and advocate for its constituents. FY 2001 - FY 2005 DAP Final Evaluation Report 11 January 2005 64 * Designing a program that builds on and utilizes the skills and expertise of the other partners, i.e., ANSP+, CARE, Caritas Rwanda, and World Relief, in an integrated strategy * Understanding the strategies of and working more closely with the relatively large number of NGOs who have become engaged in HIV/AIDS programming. * Defining an exit strategy in which (1) the needs of PLWHA will be adequately met by the their associations with the support of government and other stakeholders and (2) households of PLWHA will be able to achieve a sustainable livelihood. Integration. Because they have different target beneficiaries, the natural resource components of the current DAP and the Safety Net/LIFE components do not lend themselves easily to integration with each other in a coherent program strategy (see Section E.3). However, it may be possible to implement both sets of interventions in a community in which multiple types of beneficiaries may be targeted. In other words, the program could be structured to implement an appropriate set of interventions for small-holder farm families along side an appropriate set of interventions for the highly vulnerable. An analysis should be conducted to compare the impact of this approach, which, given the resources available, would work on a smaller geographic scale, as opposed to the approach in which an appropriate set of interventions for either one target group or the other is implemented more widely. While the components of the current program do not integrate logically, the Safety Net/LIFE component would benefit from and complement the strategies of the OVC/ART/ABY program that CRS Rwanda is beginning to implement. These program activities all have the same or similar target groups. Also, as mentioned in Section E.3, there is significant potential for including appropriate natural resource interventions with safety net/LIFE institutions to enable them to become more self-reliant, with young adults graduating from safety net institions to enable them to attain sustainable livelihoods, and with households supporting people living with HIV/AIDS to enable them to attain sustainable livelihoods. VI. CONCLUSION Tables 9 and 10 on the following page summarize the findings of the final evaluation of the FY 2001 to FY 2005 DAP relative to performance against each Intermediate Result and how achievements on Intermediate Results have produced impact at the Strategic Objective level. Bottomline. In the judgment of the final evaluation team, the investments made through the DAP in the Safety Net and LIFE components have generated satisfactory returns over the life of the program. The investments made in the natural resource components in Butare and Kibuye, however, have not generated satisfactory returns over the life of the activity, mainly because relatively little was achieved over the first three and a half years of the program. The last year of implementation has yielded significant impact, but these returns do not sufficiently offset the cost of five years of implementation. The main factors accounting for the relatively poor performance in the early phases of the DAP were the absence of strong leadership in the program and weak M&E systems that did not enable the program to evolve and respond to changes in the operating context and experience. FY 2001 - FY 2005 DAP Final Evaluation Report 11 January 2005 65 Table 9. Summary Assessment of Achievement by IR Intermediate Result Summary Assessment of Achievement Agriculture Component IR1.1: Increased lowland productivity for 15,000 families in targeted areas by 2005. The targets were not achieved on this IR relative to the number of beneficiaries but a good foundation has been established and benefits will continue to accrue IR1.2: Reduced erosion from steep lands for 28,500 families in targeted areas by 2005. The targets were not achieved on this IR relative to the number of beneficiaries but a good foundation has been established and benefits will continue to accrue IR1.3: Selected farms in targeted areas restocked with 13,500 goats and swine by 2005 Very little was achieved on this IR. IR1.4: Productivity enhancing technologies introduced to 15,000 families by 2005. Good demonstrations have been established by the program, but the program is not reaching 15,000 families IR15: 75 farmers’ associations established or strengthened in targeted areas by 2005. Most associations have been established and are functioning IR16: Improved market access for 43,500 families in targeted areas by 2005. Roads and bridges have been repaired, but the ownership and maintenance of the infrastructure is unclear and the number of small-holder farm families benefiting is far below the target. Safety Net IR 2.1: Increased food security for 7,000 institutionalized orphans, street children, elderly, and handicapped persons; and 4,400 vulnerable households affected by HIV/AIDS by 2005. This has been fully achieved through food distributions made by the program over the full life of the program. IR 2.2: Strengthened capacity for self-sufficiency among partner institutions caring for unaccompanied children, orphans, HIV/AIDS victims, and other vulnerable people by 2005. Some progress has been made in capacity building but safety net institutions are not self￾sufficient. LIFE IR 2.1: Increased food security for 7,000 institutionalized orphans, street children, elderly, and handicapped persons; and 4,400 vulnerable households affected by AIDS by 2005. This has been fully achieved through food distributions made by the program over the full life of the program. IR 2.2: Strengthened capacity for self-sufficiency among partner institutions caring for unaccompanied children, orphans, HIV/AIDS victims, and other vulnerable people by 2005. Some progress has been made in capacity building, especially through complementary programs implemented by sub-recipient partners. Table 10. Summary Assessment of Impact Against Program Strategic Objectives Strategic Objective Summary Assessment of Impact SO1: To increase the ability of 43,500 smallholder families in targeted areas to ensure their food security by 2005. During the program’s life, not much impact had been achieved until CY 2004, but benefits will accrue after the program ends. SO2: To ensure increased food security for 29,000 of Rwanda’s most vulnerable people each year 2001-2005. There has been significant impact on short term food insecurity, but sustained food security has not yet been achieved. FY 2001 - FY 2005 DAP Final Evaluation Report 11 January 2005 66 ANNEXES Annex A: Final Evaluation Scope of Work Annex B: Evaluation Methodology Annex C: Detailed Analysis of the CRS Response to MTE Recommendations Annex D: LOA Expenditures Summary Annex E: Safety Net IGA Grants Annex F: Safety Net Evaluation Survey Summaries Annex G: LIFE Evaluation Survey Summary Annex H: Commodity Detailed Summaries