The  Consortium  for  Food  Security  in  Mali: Catholic  Relief  Services  – United  States  Conference  of  Catholic  Bishops Helen  Keller  International Save  the  Children  International Implementing  Partners: Caritas,  Tassaght Report  of  the   Final  Qualitative  Evaluation  of  the  Consortium  for   Food  Security  in  Mali  (CFSM) “Nema”  Program P.L.  480  Title  II  Multi-­‐Year  Assistance  Program FFP-­‐A-­‐00-­‐08-­‐00068-­‐00 December  2013 Recommended Citation: Della E. McMillan and Sidibe Sidikiba. 2013. Final Qualitative Evaluation of the Consortium for Food Security in Mali (CFSM) “Nema” Program, P.L. 480 Title II Multi-Year Assistance Program, FFP-A-00-08-00068-00. Bamako, Mali: CRS (December). Table of Contents Executive Summary ..................................................................................................................... i Acronyms .................................................................................................................................... vii Chapter 1 Context and Methodology ................................................................................... 1 1.0. Context .............................................................................................................................. 1 1.1. Global Context 1.2. Early Evolution of the Program 1.3. Evolution of the Program During and After the Conflict 1.4. Identified Needs for the Final Evaluation 2.0. Evaluation Goal, Global Objectives, and Expected Results ........................................ 2 2.1. Evaluation Goal and Global Objectives 2.2. Specific Objectives 2.3. Key Questions 3.0. Methodology ..................................................................................................................... 5 3.1. SOW Expectations for Data Collection Methodologies 3.2. Global Organization of the Qualitative Survey and Key Informant Interviews 3.3. Steps Involved in the Design and Analysis of the Qualitative Survey and Report 4.0. Organization of the Report ........................................................................................... 11 Chapter 2 SO1: Livelihood Strategies More Profitable and Resilient ............................ 13 1.0. Nema SO1 Strategy/Theory of Change ........................................................................ 13 2.0. Evolution of the Nema Strategy and Activities and Early Evidence of Impact ....... 16 2.1. Evolution of Activities and Early Evidence of Results 2.2. Early Evidence of Results and Effects 3.0. Current Level of SO1 Activities Supported by Nema ................................................ 23 3.1. Agro-Enterprise Groups 3.2. Savings and Internal Lending Communities 3.3. Farmer Field Schools 3.4. Food for Work 3.5. Safety Nets 4.0. Global Impact of the SO1 Activities on Household Food Security and Vulnerability ................................................................................................................... 34 4.1. On Average Household Food Security and Resilience 4.2. On Vulnerable Households 4.3. Effectiveness of the SO1 Monitoring and Evaluation Systems 5.0. Lessons Learned and Recommendations ………………............................................. 41 Chapter 3 SO2: Children Under 5 Years Less Vulnerable to Illness and Malnutrition .. 46 1.0. Nema SO2 Strategy/Theory of Change ........................................................................ 46 2.0. Evolution of Program Activities for IR 2.1. Caregivers of Children Under 5 Years and Pregnant Women are Applying Improved Nutrition and Feeding Practices ... 48 2.1. PECIMA: Screening and Care of Acute Malnutrition at the CSCOM Level 2.2. PD/Hearth Strategy Evolution 2.3. Promoting Improved Nutrition and Feeding Practices Through Community Radio in 130 Villages 3.0. Evolution of Program Activities for IR 2.2. Caregivers of Children Under 5 are Applying Improved Hygiene and Sanitation Practices ............................................... 59 3.1. Sanitation 3.2. Drinking Water 4.0. Current Level of SO2 Activities in the Nema Villages for IR 1.1 and IR 1.2 ........... 60 4.1. Activities of the Nema-Created Health Committees and Nema-Trained Community Health Volunteers 4.2. Nutrition and Sanitation Practices of the Mothers Who Participated in the PD/Hearth Activities 4.3 Community-Based Activities Offered by the Local Offices of the Ministry of Health 5.0. Lessons Learned and Recommendations ..................................................................... 65 5.1. Key Issues 5.2. Key Factors Which Reduced the Impact of the Nema Health Strategy in Nutrition in 130 Target Villages 5.3. Lessons Learned and Recommendations Chapter 4 SO3: Targeted Communities Manage Shocks More Effectively ................... 78 1.0. SO3 Strategy/Theory of Change ................................................................................... 79 2.0. Evolution of the Nema SO3 Strategy and Activities ................................................... 82 2.1. IR 3.1. Community Early Warning and Response Systems are in Place 2.2. IR 3.2. Community Safety Nets are in Place 2.3. Mid-Term Evaluation Assessment and Recommendations 3.0. Early Evidence of Impact on the Establishment of Sustainable GAP/RUs in the Target Villages ............................................................................................................... 84 3.1. Results During Nema—Efficient Preparation of a Relief Plan in Response to Data from the GAP/RU 3.2. Results During Nema—Partial Execution of a Relief Program in the Douentza Villages 3.3. Results Post-Nema—Nema-Trained Safety Net Committees Helped Facilitate the Allocation of Other Humanitarian Aid During the Crisis 4.0. Current Level of the SO3 Activities Supported by Nema .......................................... 86 4.1. Food for Work 4.2. Community-Based Early Warning and Response Systems 4.3. Community Safety Nets 5.0. Lessons Learned and Recommendations ..................................................................... 95 Chapter 5 Cross-Cutting Activities: Literacy, Good Governance, and Local Organizational Capacity ................................................................................... 98 1.0. Nema Strategy /Theory of Change for Cross-Cutting Activities: Literacy, Good Governance, and Capacity Building ............................................................................ 99 1.1. Literacy 1.2. Local Capacity Building and Good Governance 2.0. Evolution of the Nema Strategies for Literacy, Organizational Capacity Building, Good Governance, and Early Evidence of Impact .................................................... 101 2.1. Early Evolution of the Program 2.2. Mid-Term Evaluation Findings and Recommendations 2.3. Activities After the Mid-Term 3.0. Current Level of Activity of the Nema-Supported Literacy Activities ................... 105 3.1. Current Level of Activity 3.2. Summary Assessment of Strengths, Areas that Need Strengthening, and Opportunities in the Current Context: Literacy Training 4.0. Current Level of Activity of the Nema-Supported CBOs ........................................ 108 4.1. Current Level of Functioning 4.2. Summary Assessment of Strengths, Areas that Need Strengthening, and Opportunities in the Current Context: Local Organizational Capacity and Good Governance 5.0. Lessons Learned and Recommendations ................................................................... 111 Chapter 6 Conclusions ....................................................................................................... 113 1.0. Main Lessons Learned from the Program Experience ............................................ 113 1.1. Global Overview of Results 1.2. Capacity for a Rapid Start Up in the Case of a Recuperation Effort 2.0. Lessons Learned and Recommendations ................................................................... 116 2.1. Strategic Objective 1: Livelihood Strategies More Profitable and Resilient 2.2. Strategic Objective 2: Children Under 5 Years Less Vulnerable to Illness and Malnutrition 2.3. Strategic Objective 3: Targeted Communities Manage Shocks More Effectively 2.4. Cross-Cutting Objectives: Basic Literacy, Good Governance and Capacity Building 2.5. Monitoring & Evaluation Annexes Annex I. Terms of Reference for Final Evaluation of Years 1-4 of MYAP Nema Annex II. Mali CFSM MYAP Final Indicator Performance Tracking Table Annex III. Field Survey Interview Guides Annex IV. Qualitative Data from the Focus Groups Annex V. Key Informant Interviews Annex VI. Photo Essay Nema Final Qualitative Evaluation List of Tables Table 1.1. Nema Project Logical Framework Table 1.2. Key Questions Asked in the SOW for the Nema Final Evaluation Table 1.3. The Four Virtual Working Groups Involved in the Nema Qualitative Final Evaluation Table 1.4. Steps Involved in the Design, Execution, and Reporting of the Nema Final Qualitative Survey Table 1.5. Retroactive Qualitative Classification of Nema Villages Used to Determine the Sample Frame for the Final Qualitative Survey Table 1.6. Number of individuals interviewed Nema Final Qualitative Survey Table 2.1. Targets and Achievements for the AEGs and Number of Nema Communities that Benefitted from the Activity, Years 1-4 Table 2.2. Targets and Achievements for the FFSs and Number of Nema Communities that Benefitted from the Activity, Years 1-4 Table 2.3. Targets and Achievements for the SILCs and Number of Nema Communities that Benefitted from the Activity, Years 1-4 Table 2.4. Number of Villages and Beneficiaries that Benefited from the Major Categories of Nema SO1 Activities, Years 1-4 Table 2.5. Retroactive Participatory Classification of Nema Program Villages by Level of Performance and Food Insecurity When the Program Closed, March 2012 Table 2.6. Functionality of the AEG Interviewed During the Nema Program Final Evaluation Survey, November 2013 Table 2.7. Functionality of the SILC Groups Interviewed During the Nema Program Final Evaluation Survey, November 2013 Table 2.8. SILC Activities During and After the Nema Program Table 2.9. SILC Financial Resources During and After the Nema Program Table 2.10. Current Functionality of Nema-Created Infrastructure Visited During the Final Evaluation Table 2.11. Evidence of Post-Nema Food Distributions in the Nema Villages, September 2012-present Table 2.12. Evolution of the 2012 Food Crisis and Nema and Non-Nema Responses and Impacts Appendix I, Chapter 2: SILC Tool Used for Tracking Organizational Capacity Table 3.1. Targets vs. Achievements for the Main Nema SO2 Activities, Years 1-4 Table 3.2. PD/Hearth Implementation Plan for Nema Program, Years 2-3 Table 3.3. Hearth/DP Activities in Nema Program, Years 2-3 Table 3.4. Number of Groups and People Interviewed about the Current Functionality of the Nema-Funded Activities in the 130 Nema Villages in Douentza and Bourem, November 2013 Table 3.5. Current Functional Status of the Formal and Informal Groups and Community Health Volunteers Trained by the Nema Program in Years 1-4, November 2013 Table 3.6. Level of Malnutrition in Douentza and Bourem Areas Using Indicative Data of CSCOMs’ Staff Qualitative Survey, 2011-2013 Table 3.7. IR 2.1. Caregivers of Children Under 5 and Pregnant Women are Applying Improved Nutrition and Feeding Practices Table 3.8. MI 2.1.6. Percentage of Beneficiary Caregivers Who Practice Behaviors Shown to Be Successful to Rehabilitate Underweight Children Table 3.9. Nema Monitoring Indicators Based on Data Collected by the Annual Mothers Survey Appendix I, Chapter 3: Evolution of SO2 Activities Table 4.1. Impact and Monitoring Indicators Used to Track the Progress of the Nema Program’s Intermediate Results in Its IPTT Table 4.2 Major SO3 Achievements During Nema, Years 1-4 Table 4.3. Evidence of Post-Nema Food Distributions in the Nema Villages, September 2012-present Table 4.4. Number of Infrastructures Created, Direct Beneficiaries, and Current Level of Functioning of the FFW-Funded Infrastructure Created by the Nema Program, November 2013 Table 4.5. Functionality of the GAP/RU and Safety Net Committees Interviewed During the Nema Program Final Evaluation Survey, November 2013 Table 4.6. Evolution of the Nema Safety Net and GAP/RU Activities, March 2012- November 2013 Table 5.1. Number of Students from Different Nema-Supported Structures that Enrolled in Literacy Training and the Number Who Were Active Participants, Years 1-4 Table 5.2. Types of Training and Incorporation of the Local Committees and Community￾Based Experts into the Program’s Internal and Official Tracking Systems, Years 1-4 Table 5.3. Indicators Used in the Nema IPTT to Track the Evolution of the Local Community Organizations’ Capacity in Key Areas Needed to Execute and Sustain Their Activities, Years 1-4 Table 5.4. Average Number of SILC and AEG Members Who Attended the Nema￾Sponsored Literacy Training Who Can Read, Write, and Conduct Simple Calculations, March 2013 Table 5.5. Functionality of the Nema-Facilitated CBOs Interviewed During the Nema Final Evaluation by SO, November 2013 Table 6.1. Qualitative Assessment of Current Security Levels in the Former Nema Villages in Bourem and Douentza, November 2013 Table 6.2. Qualitative Assessment of Government Extension Worker Coverage in the Former Nema Villages in Bourem and Douentza, November 2013 Table 6.3. Qualitative Assessment of the Major Outside Development Partners Currently Working in the Former Nema Villages in Bourem and Douentza, November 2013 Table 6.4. Qualitative Assessment of the Direct and Indirect Impact of the 2012 Occupation on the Former Nema Villages in Bourem and Douentza, November 2013 Table 6.5. Corrective Actions Recommended for Increasing the Sustainability of the Nema Program’s Accomplishments and Monitoring or Strengthening the Benefits Initiated by the Program List of Figures Figure A. Mali CFSM (Nema) MYAP—Before March 2012 and After (March 2012- present) Figure 1.1. Mali CFSM (Nema) MYAP—Before Suspending Its Activities in Douentza and Bourem in March 2012 and Today (March 2012-present) Figure 2.1. Nema Strategy for SO1 (Years 1-4) Figure 2.2. Nema Results and Effects in the Douentza Villages: Average Household Food Security (MAHFP) Before and During the Nema Program Figure 2.3. Nema Results and Effects in the Douentza Villages: Average Household Food Security (MAHFP) Before, During, and After the Program Figure 2.4. Nema Results and Effects in the Douentza Villages: Household Vulnerability Levels Before, During, and After the Program Figure 2.5. Nema Results and Effects in the Bourem Villages: Household Vulnerability Levels Before, During, and After the Program Figure 3.1. SO2: Theory of Change Strategies, Nema Program 2013 Figure 3.2. Evolution of the Number of Malnourished Children Identified by Nema, Years 1- 4 Figure 3.3. Evolution of the Number of Malnourished Rehabilitated by Nema, Years 1-4 Figure 3.4. Number of Children Rehabilitated in the Nema DP/Hearth Activities in 27 Nema Villages, Years 2-4 Figure 4.1. Nema SO3 Strategy/Theory of Change Figure 5.1. Nema Strategy for Literacy, Good Governance, and Local Capacity Building List of Text Boxes Text Box 2.1. The INTSORMIL Project in the Nema Village of Oualo Text Box 2.2. Sample Responses to Questions Asked to the SILC Focus Groups Text Box 3.1. Case Study on the Implementation of Voluntary Contributions to Support PD/Hearth Programs in the Villages of Boumbam (Douentza) and Moudankan (Bourem) Text Box 3.2. Testimonials from Six Different Community Health Volunteers on What They Learned During Nema, and Whether or Not They Continue Their Activities Text Box 3.3. Testimony of PD/Hearth Participant Fanta Boucary Maiga During the Focus Group Held with the PD/Hearth Group in Densa Village Text Box 3.4. Assessment of the Impact of Nema Activities by Partners Linked to Health Facilities in Douentza and Bourem, November 2013 Text Box 3.5. Main Suggestions or Recommendations Made by the Technical Partners and Beneficiaries in the Improvement of Program Activities Text Box 4.1. Major Conclusions of the Mid-Term Evaluation Concerning SO3 Text Box 4.2. Beneficiary Responses to Questions During the FFW Focus Groups Text Box 4.3. Community Leader Observations About the Implementation of the Nema Safety Net Programs in the Cercle of Douentza Text Box 5.1. Comments Made About the Role of Literacy Training During the AEG Focus Groups Nema Final Qualitative Evaluation. Executive Summary. December 30, 2013. Draft. i Executive Summary1 Since August 2008, a consortium comprised of Catholic Relief Services (CRS), Save the Children International (SCI), and Helen Keller International (HKI) has executed the Consortium for Food Security In Mali (CFSM) led by CRS. The original goal was—and continues to be—“to reduce the food security prevalence in vulnerable populations” through three strategic objectives related to improved livelihood strategies, improved health and nutritional status, and increased capacity to manage shocks. Two transversal themes—functional literacy and good governance— support all three strategic objectives (SOs). The five-year program (2008-2013), known as the “Nema”2 program, or the Multi-Year Assistance Program (MYAP), is funded by the United States Agency for International Development (USAID) through Food for Peace (FFP). The original zone of intervention was 130 of the most vulnerable villages in two very food insecure cercles3 —Douentza and Bourem (Figure A). The estimated number of people benefitting from the program in both cercles combined was 124,859 (See Figure A, showing the location of the two cercles). To carry it its activities, CRS worked through local partner, Caritas/Mali in the cercle of Douentza and SCI worked through the Malian NGO Tassaght in Bourem. In March 2012, the Consortium was forced to suspend its activities in the Douentza and Bourem region due to the rebel occupation of the area by Islamic groups, and redeployed many of the same activities and staff to: • A highly effective emergency internally displaced person (IDP) program benefitting 39,830 IDPs in the regional capital of Mopti (April-September 2012); and • A quick-start Title II initiative for three vulnerable cercles further south (Figure A)— using a slightly amended version of the original mode—that is now entering its second year (2012-present). Both of these follow-on programs have benefited from the considerable effort invested in refining the initial Nema strategy in the Douentza and Bourem regions during the first four years of the program (August 2009-March 2012). 1 Della E. McMillan and Sidibe Sidikiba. 2013. Final Qualitative Evaluation of the Consortium for Food Security in Mali (CFSM) “Nema” Program, P.L. 480 Title II Multi-Year Assistance Program, FFP-A-00-08-00068-00. Bamako, Mali: CRS (December). 2 Locally the CFSM program is known as the “Nema” program. In Bambara, the term “Nema” means “at ease” or “comfortable.” The term refers to the program’s intention to reduce household food insecurity and render them more comfortable by improving their agriculture and livelihood systems. 3 Mali is divided administratively into eight regions. Each region is further divided into districts (cercles), and these districts are made up of communes. Nema Final Qualitative Evaluation. Executive Summary. December 30, 2013. Draft. ii Figure A. Mali CFSM (Nema) MYAP—Before March 2012 and After (March 2012-present) Although these follow-on activities are a critical part of the Nema story—past (i.e. the IDPs) and present (the ongoing activities in the new MYAP cercles) 4 —the official objective of this qualitative final evaluation of the Nema program was to conduct a post-implementation assessment of the longer-term impacts of the program in the war-affected communes of Douentza and Bourem, where the program started. The significance of this evaluation is at two levels: • Given the gradual stabilization of northern Mali since January 2013, there is a huge interest in determining what types of strategies should be used to re-engage with the local communities in these vulnerable areas, which are typical of the southern frontier of northern Mali; and • Since Nema’s activities were suspended in March 2012, this evaluation was expected to provide both USAID and its major partners (SCI, HKI, CRS, and Caritas) with some valuable lessons about what types of Title II initiatives are most likely to help build local communities’ abilities to manage major shocks. This final qualitative survey and evaluation process was expected to: 4 Each of these follow-on programs has—in the eyes of the evaluation team—important impacts on the Nema program that deserve an independent assessment and appreciation in and of themselves since they include a large number of examples of best practice, some of which are discussed the text of this report and its annexes. Especially important, the Nema story includes many valuable lessons learned about the critical importance of donors and partners being flexible and adapting program models to new constraints and opportunities. Nema Final Qualitative Evaluation. Executive Summary. December 30, 2013. Draft. iii • Assess the current level of activity and functioning of the program’s supported activities almost two years after Nema officially suspended its activities in Douentza and Bourem; and • Use a participatory process engaging local communities, government officials (Ministries of Health, Social Action, Agriculture, Water, and the Governor’s office), execution partners (Caritas), as well as a high proportion of the original staff who designed and executed the program (from SCI, HKI, and CRS) in the identification of what types of follow-on support might be needed to strengthen these activities and sustain them in the current context. The evaluation was led by a two-person team with high levels of input from the current and previous Nema program staff that worked in the Douentza and Bourem regions over a five-week period in November and December 2013. The evaluation was based on three types of data: • The Nema program’s internal M&E data from Years 1-4 of the program; • The qualitative survey of a representative sample of 21 of the 130 Nema villages in which the evaluation team conducted individual and group interviews to assess the current functioning levels of the activities created by the Nema program, taken over a three-week period in November 2013; and • Key informant interviews that the team conducted in the region and local cercles that enabled the team to better understand the direct impact of the war and current trends on donor and government re-engagement in the communities since January 2013. This analysis provides clear evidence that in March 2012 Nema was well on its way to achieving most of its targets in the original proposal and that many of these activities were having a very positive impact on household food security, vulnerable groups, and resilience by the time the program was suspended in March 2012. There is also ample evidence from the beneficiary and partner interviews that many of the program’s activities helped the local people better manage the shocks that started descending upon them in June 2011. Specific achievements include: • Building the capacity of vulnerable people in agro-enterprise development through the development of agro-enterprise groups (AEGs) and savings and internal lending communities (SILCs), the majority of which are still functional (based on the results of the sample survey), albeit at a low level; • Introducing a wide variety of new crop technologies that helped local people increase their food security and reduce their vulnerability to drought; • Building 143 new infrastructure works using Food for Work (FFW)—the majority of which is still functional—that made local people less vulnerable to drought, flooding, and other shocks; • Supporting the new Centre de Santé Communautaire (CSCOM, or community health center) model for identifying and rehabilitating malnourished children in collaboration with the CSCOMs that complied with the national Management of Acute Malnutrition (MAM) strategy in 130 of the 130 villages (100%), and pilot testing a new model for community-based PD/Hearth model for identifying and rehabilitating malnourished children in 27 of the 130 villages; Nema Final Qualitative Evaluation. Executive Summary. December 30, 2013. Draft. iv • Encouraging improved hygiene and sanitation practices through a concerted effort to promote household use of latrines (most of which appear to be in use) that have continued to help improve local sanitation practices; • Constructing 15 water points—14 of which are still functional—in water-starved villages where successive water resource development programs had previously failed; • Strengthening the community-based systems for emergency early warning and response by developing: (a) a community-based system of early warning and response (Groupements d’Alerte Precoce et Reponse d’Urgence or GAP/RU); and (b) a safety net committee that was capable of working with outside agencies to coordinate a series of post-Nema food aid responses during the crisis in 130 of the 130 villages; and • Creating a system of community-based literacy trainings that helped build the capacity of all of the community-based organizations (CBOs) being supported by the program in 130 of the 130 villages, as well as the specific capacity of the AEGs and SILCs that were a special target of these programs. Although the net impact of these activities was very positive, the survey identified a number of corrective actions that are needed to sustain and strengthen the benefits that were initiated by Nema during its first four years of operation in Douentza and Bourem. Strategic Objective 1: Livelihood Strategies More Profitable and Resilient The principal recommendations for strengthening and sustaining the Nema program’s SO1 activities are: • Re-engaging in the villages where the Nema program was most active as the security situation permits, which is approximately 50% of the villages in Douentza and Bourem that benefitted from the highest concentration of Nema program innovations in Years 1-4; and • Extending the basic program to most vulnerable Douentza and Bourem villages that benefitted from fewer Nema program interventions in Years 1-4. These villages—which represent about 50% of the Douentza and Bourem villages—tend to be isolated agro￾pastoral villages. Re-engaging the Nema program in these villages will require: (a) intensifying FFW to overcome some of the major constraints that these villages face in terms of isolation; (b) developing some sort of profitable agro-enterprise; and (c) helping to facilitate the development of the AEGs and SILCs as was done in the more accessible villages during Years 1-4. Strategic Objective 2: Children Under 5 Years Less Vulnerable to Illness and Malnutrition Given the limited sustainability of the original program model for rehabilitating malnourished children, the evaluation team and health-working group recommend: • Completely rethinking the program’s support for the integrated MAM (PECIMA, or Prise en Charge Intégrée de la Malnutrition Aiguë) model that involves: (a) strengthening the local communities’ support, involvement, and ownership of the CSCOM-supported growth monitoring and rehabilitation activities; and (b) making these activities less dependent on the program’s food distribution for success; Nema Final Qualitative Evaluation. Executive Summary. December 30, 2013. Draft. v • Based on the successful pilot testing of the PD/Hearth model in just 27 of the 130 Nema villages during the initial phase: - Extending the PD/Hearth model for rehabilitating moderately malnourished children using local products (i.e. not using food aid) to each and every Nema village; and - Improving the effectiveness and sustainability of the program by revamping certain elements of the program to make it better adapted to a non-emergency context such as: (a) shifting the focus from a large number of volunteer mothers (VMs) to training just two mama lumieres per village; (b) helping the mothers of the malnourished children to form ‘care’ or ‘mothers’ groups’ that will help the vulnerable mothers of the children treated by the PD/Hearth programs develop the income-generating activities (IGAs) that they need to sustain improved health practices; and (c) strengthening the Nema systems for tracking the children’s progress both during and after treatment; • Diversifying the program’s efforts to address the target villages’ chronic water access problems; and • Anticipating a major scale up in all of the villages of the new Mali government’s community-led total sanitation (CLTS, also known as Assainissement Total Piloté par la Communauté [ATPC]) model, which was pilot tested in five Nema villages at Douentza during Years 1-4. Strategic Objective 3: Targeted Communities Manage Shocks More Effectively Although the safety nets programs were much appreciated by the local people, there is little likelihood that the 130 communities will have the means to develop improved safety nets from community resources that are more extensive than the traditional methods they have always used. It is therefore critical for future programs to examine a new string of activities that will reduce the total number of vulnerable people who will need to rely on either these traditional safety nets or outside food assistance. These activities should include: (a) developing a sub-group of FFW activities that targets both ‘vulnerable’ individuals and groups to develop activities that they can manage with limited labor and resources; and (b) providing them with appropriate technical assistance (through the SO1 activities) to make these enterprises profitable. To address this issue, the evaluators working in close collaboration with the SO3 working group developed a list of recommendations that focused on: • Anticipating the creation of food security committees in all of the Nema villages to coordinate the different Nema and non-Nema community structures that affect food security, with a special accent on better integrating the functions of the GAP/RU with the safety net committees; • Anticipating the involvement of all the key partners (Village Food Security Committees, Red Cross/Mali, Systèmes d’Alerte Précoce [SAPs])—who showed their capacity to manage major crises during the recent 2012-2013 crisis—in the conception and execution of any future community-based early warning and emergency response system; Nema Final Qualitative Evaluation. Executive Summary. December 30, 2013. Draft. vi • Anticipating a new series of FFW and development (SO1) activities that strengthens the food security of the most vulnerable households and reduces their dependence on community-based safety nets which depend heavily on outside food assistance; • Intensifying FFW activities in the most vulnerable villages in order to build their food security and income and reduce community-level vulnerability to risk and shocks; and • Strengthening the capacity of the Nema monitoring and evaluation (M&E) system to monitor the activities of the Village Food Security Committees and other key community-based organizations. Cross-Cutting, Literacy, Good Governance, and Local Capacity Building One major output of the Nema program was to build the capacity of the local groups to manage these activities and sustain them. The final evaluation survey shows that the vast majority of the SILCs and AEGs that the program helped create are still considered functional. The same survey results show the critical importance of the SCI-sponsored literacy and good governance training in building this capacity. The pre-existing capacity of the groups—as well as other program￾trained community specialists like the Farmer Field School (FFS), Training of Trainers (TOTs), and SILC Animateurs Villageois (AVs)—is a major capital that future programs can build on when planning their strategies for re-engagement in the zone. Two of the key recommendations for strengthening the impact and sustainability of these activities involve: • Strengthening the linkages between literacy training and the actual activities (financial, planning, etc.) of the CBOs; and • Strengthening the capacity of the program managers and administration to monitor the core capacity of the CBOs created by the program in order to better target training and technical support and plan the program’s exit strategy. Monitoring and Evaluation A fourth set of recommendations suggests various ways that CRS’s existing systems for Title II M&E, which were pilot tested on this program and are already being refined in the new area, can be adjusted to track these new activities and new strategies. Especially important, this analysis suggests various ways that the existing M&E systems put in place under Nema can be adjusted to better track the program’s considerable accomplishments in three areas: • Reducing the percentage of Nema villages classified as chronically food insecure; • Reducing the percentage of households classified as chronically food insecure; and • Developing better systems for monitoring the core capacity of the local organizations charged with sustaining these activities as a basis for the development of successful exit strategies. Nema Final Qualitative Evaluation. Acronyms. December 30, 2013. Draft. vii Acronyms ADC Community Development Agents ADESAH Association for the Endogenous Development of the Sahel AE Addunction Eau (water points) AEG Agro-Enterprise Group AES Adduction D'eau Sommaire AFAR Appuix aux Systemes Ruraux de Production ALCOP Aluminum Company of Pakistan ATPC Assainissement Total Piloté par la Communauté (Community-Led Total Sanitation) AV Animateurs Villageois (for the SILCs) BCC Behavior Change Communication CAP Centre d’Animation Paysanne (Center for Rural Extension) CBO Community-Based Organization CDA Community Development Agent (Agent de Développement Communautaire – ADC) CFA West African CFA Franc CFSM Consortium for Food Security in Mali CHV Community Health Volunteer (relais) CLTS Community-Led Total Sanitation CMAM Community Management of Acute Malnutrition CRS Catholic Relief Services CSREF Centre de Santé de Reference (District-Level Health Center) CSCOM Centre de Santé Communautaire (Commune-Level Health Center) DDS Dietary Diversity Scores DNFLA Direction Nationale des Langues et de l'Alphabetisation (National Directorate for Languages and Literacy) EI Environmental Indicator ENA Essential Nutrition Actions EWG Early Warning Group FCFA CFA Franc FFP Food for Peace FFS Farmer Field School FFT Food for Training FFW Food for Work GAE Groupement Agro-Enterprise GAP/RU Groupements d’Alerte Precoce et Reponse d’Urgence (Early Warning Group/Emergency Response) GOM Government of Mali GRAT Groupe de Recherche et d 'Applications Techniques HH Household HKI Helen Keller International ICRISAT International Crops Research Institute for the Semi-Arid Tropics IDP Internally Displaced Person IEC Information, Education, and Communication Nema Final Qualitative Evaluation. Acronyms. December 30, 2013. Draft. viii IER Institut d’Economie Rurale (Institute of Rural Economy) IESA Initiative Eau et Sécurité Alimentaire pour l’Afrique IFRC International Federation of the Red Cross IGA Income-Generating Activity II Impact Indicator ILRI International Livestock Research Institute IMF Institution of Micro-Finance INTSORMIL Sorghum and Millet Research Support Program IPTT Indicator Performance Tracking Table IR Intermediate Result IYCF Infant and Young Child Feeding LOA Life of Activity MAM Management of Acute Malnutrition M&E Monitoring and Evaluation M&EAL Monitoring, Evaluation, Accountability, Learning MI Monitoring Indicator MOH Ministry of Health MOU Memorandum of Understanding MSF Médecins sans Frontières MUAC Mid-Upper Arm Circumference MYAP Multi-Year Assistance Program NGO Non-Governmental Organization OR Operations Research OXFAM Oxfam International PECIMA Prise en Charge Intégrée de la Malnutrition Aiguë (Integrated Management of Acute Malnutrition) PD Positive Deviance PDI-MUZELM Projet de Developpement Integre, Mission Evangelique Lutherienne au Mali PM Program Manager PRA Participatory Rural Analyses PREP Pipeline and Resource Estimate Proposal PSI Population Services International PSP Private Service Providers SAM Severe Acute Malnutrition SAP Système d’Alerte Précoce (Early Warning System) SCI Save the Children International SILC Savings and Internal Lending Communities SMILER Simple Measurement of Indicators for Learning and Evaluation-Based Reporting SO Strategic Objective SOW Scope of Work SUN Scaling Up Nutrition SYAP Single-Year Assistance Plan TOT Training of Trainers UAVES Union pour un Avenir Ecologique et Solidaire. Nema Final Qualitative Evaluation. Acronyms. December 30, 2013. Draft. ix USG U.S. Government USAID United States Agency for International Development VAD Visites a Domicile (Home Visits) VM Volunteer Mother WASH Water, Sanitation, and Hygiene WFP World Food Programme Nema Final Qualitative Evaluation. Chapter 1. Context and Methodology. December 30, 2013. Draft. 1 Chapter 1 Context and Methodology 1.0. Context 1.1. Global Context The Nema program (a.k.a. Consortium for Food Security in Mali [CFSM]) is a Multi-Year Assistance Program (MYAP) implemented through a consortium comprised of Catholic Relief Services (CRS), Helen Keller International (HKI), and Save the Children International (SCI) in the regions of Mopti and Gao through two implementing partners, Caritas/Mali and the Malian NGO Tassaght. Through integrated interventions from 2008 through 2012, the Nema program intended to improve the livelihoods of vulnerable households and strengthen the community capacity for resilience to shocks. The program intervened in three priority domains identified by the Consortium to fight against food security through increasing capacity in: (a) agro-enterprise development; (b) nutrition, hygiene, and sanitations; and (c) disaster-risk reduction (Table 1.1). In addition to these three core domains, the program included two additional cross-cutting interventions—literacy and savings and internal lending communities (SILCs)—designed to strengthen household resilience, especially for women, during future crises. The Nema program targeted 124,858 participants in 130 villages in the cercles of Bourem in the Gao Region and Douentza in the Mopti Region (Figure 1.1). Table 1.1. Nema Project Logical Framework Framework for the MYAP GOAL: Vulnerable rural households in the regions of Mopti and Gao have reduced their food insecurity Strategic Objective 1: Livelihood strategies are more profitable and resilient • Intermediate Result 1.1. Household agricultural production is increased • Intermediate Result 1.2. Targeted household revenues increase Strategic Objective 2: Children under 5 years less vulnerable to illness and malnutrition • Intermediate Result 2.1. Caregivers of children under five and pregnant women are applying improved nutrition and feeding practices • Intermediate Result 2.2. Caregivers of children under five are applying improved hygiene and sanitation practices Strategic Objective 3: Targeted communities manage shocks more effectively • Intermediate Result 3.1. Community early warning and response systems are in place • Intermediate Result 3.2. Community safety nets are in place Transversal Activities: Functional literacy and training in governance Source: Annex I. Nema Final Qualitative Evaluation SOW. 1.2. Early Evolution of the Program At the beginning of Nema, the Consortium carried out a baseline study (September –October 2008), which permitted the determination of reference data and fixed targets for performance indicators for the duration of the program. Nema Final Qualitative Evaluation. Chapter 1. Context and Methodology. December 30, 2013. Draft. 2 In May 2011, the program executed a mid-term evaluation to determine the evolution of the program towards its original objectives.5 Based on the results of this evaluation, the program made a number of recommendations, including: • Adding a sub-component that focused on community-based rehabilitation of moderately malnourished children, which would permit moderately malnourished children to be rehabilitated in the community rather than the hospital; • Adding a second community-led total sanitation (CLTS) component to improve hygiene and nutrition; and • Reducing the number of water points that the program was going to build and/or rehabilitate from 60 to 49. 1.3. Evolution of the Program During and After the Conflict In March 2012, the program was obligated to cease activities due to the rebel occupation of the area by Islamic groups (Figure 1.1). Currently, the original program area is nearly liberated: 11 of the 15 communes in Douentza are free of rebel movements, and access roads are free and controlled by the armed forces. Until recently, the remaining four communes and the cercle of Bourem have been less secure. 1.4. Identified Needs for the Final Evaluation Although the original program never completed its full program cycle, it did execute a critical mass of activities in the area. There is also a great deal of qualitative evidence that the program’s activities built the resilience of the local population to weather the rebel occupation. 2.0. Evaluation Goal, Global Objectives, and Expected Results 2.1. Evaluation Goal and Global Objectives The goal of the current evaluation is to examine the results that exist in the field since the occupation and consequent liberation currently underway by the Malian and French Armies and their allies. The global objectives6 of the evaluation are to evaluate: • The results, implementation process, and sustainability of the MYAP program in terms of food security, nutrition, and vulnerable groups in the target populations; and • The current functionality of the systems and structures put in place (including the program’s community-based early warning systems) in the areas where the program has stopped operating with the intention of informing a possible rapid start up. 5 Kathy Tilford. 2011. Report of the Mid-Term Evaluation of the Nema Program in Mali. Bamako: The Consortium for Food Security in Mali (CFSM) (Catholic Relief Services, Helen Keller International, Save the Children International), June. 6 The seven specific objectives for achieving these two global objectives are outlined on page 3 of the SOW. Nema Final Qualitative Evaluation. Chapter 1. Context and Methodology. December 30, 2013. Draft. 3 Figure 1.1. Mali CFSM (Nema) MYAP—Before Suspending Its Activities in Douentza and Bourem in March 2012 and Today (March 2012-present) 2.2. Specific Objectives The specific objectives of the evaluation as outlined in the approved scope of work (SOW) were to: • Describe whether the program has attained its goals, strategic objectives (SOs), and intermediate results (IRs), and how this was accomplished; • Identify all other results of the program, intended and unintended; • Describe how the activities implemented from 2008 to 2012 (SILCs; agro-enterprise; nutrition and water, sanitation, and hygiene [WASH]; and disaster risk reduction) continue to be applied in the previously occupied program zones by the remaining program participants; • Provide an update on the implementation of recommendations made during the mid-term evaluation; • Draw principle lessons learned (positive and/or negative) during the four years of MYAP implementation; • Analyze the capacity for a rapid start-up of agro-enterprise groups, SILCs, and early warning committees, among other health systems; and • Analyze the efficiency and sustainability of agro-enterprise groups, SILCs, Farming Field Schools (FFSs), and early warning committees. 2.3. Key Questions In addition to these specific objectives, the evaluation SOW listed 30 key questions to be Nema Final Qualitative Evaluation. Chapter 1. Context and Methodology. December 30, 2013. Draft. 4 answered during the course of the evaluation. These questions included (Table 1.2): • 10 intervention-specific questions related to the Nema program’s major SOs; • 17 cross-cutting questions that were expected to inform the key partners of the program (CRS, HKI, SCI, Caritas) and its stakeholders about the Nema program’s major results, the efficiency of the process, and the sustainability of the program activities; and • Three cross-cutting questions related to gender and environmental impacts. Table 1.2. Key Questions Asked in the SOW for the Nema Final Evaluation Intervention-Specific Questions by Strategic Objective Strategic Objective 1 (SO1) 1. Are agro-enterprise groups the most appropriate strategy/relevant livelihood approach for strengthening household food security given the marginal land conditions, distance from markets etc.? 2. How effectively was the Farmer Field School (FFS) approach carried out? Was it implemented as expected? If not, what was the problem? What can we learn from this? 3. Do the agro-enterprise and SILC groups continue to operate and have they remained financially solvent after the end of the program? Strategic Objective 2 (SO2) 4. What are the lessons learned from the shift from health facility delivery to community-based health services in this MYAP context? Was this a good change? In what ways? How might it be improved further? 5. Are communities applying the techniques introduced for improved nutrition of infants in the context of the crisis in the north? If yes, what are the factors that encourage this use? 6. What was the Behavior Change Communication (BCC) strategy and was it relevant and efficient to positively affect nutrition and hygiene behavior change and outcomes? Describe and support your response with evidence. Strategic Objective 3 (SO3) 7. Are village-level early warning systems integrated into the regional (SAP- Systeme d’Alerte Precoce) early warning system? How? And what are the lessons learned? 8. State of infrastructures completed: Observe and evaluate the functionality of the public works and infrastructures created by the program. Cross-Cutting: Literacy and Capacity Building 9. Has the program improved the capacity of targeted community organizations? If yes, how so? 10. Has the program improved the capacity of state services and other partners in the implementation of food security programs? In what way? Has the program improved program participants’ literacy? If yes, how? Cross-Cutting Questions Related to Results, Efficiency, Sustainability, and Resilience Results and Effects: 1. Have the program activities (SILCs, AEGs, literacy, PD/Hearth, FFSs etc.) and their related strategies affected program participant households’ capacity to respond to shocks and natural disasters that affect their food security? If so, how? 2. According to participants interviewed, have the program activities improved production and agricultural yields? Household (HH) Income? Food security? Dietary diversity? Nutritional state of children in targeted households? If yes, how? If not, why? 3. Are there other unexpected but important effects in the targeted communities as a result of the program activities? 4. What are the points of view of partners, leaders, and program participants about implementation of the program and its results? Effectiveness of the Implementation Process: 5. What was the implementation process for achieving main program objectives (improvement of food security, reduced malnutrition, and increase in incomes)? Was it appropriate? Efficient? How might it be improved? 6. Assess the effectiveness of the program’s theory of change. Have the outputs contributed to the realization of the program’s intermediate results and strategic objectives? Are there components that were more effective than others for achieving the intermediate results and objectives of the program? 7. Is the Monitoring, Evaluation, Accountability, and Learning (M&EAL) system appropriate for measuring Nema Final Qualitative Evaluation. Chapter 1. Context and Methodology. December 30, 2013. Draft. 5 the objectives and the program indicators? 8. How effective was the technical assistance provided throughout the program? 9. How effectively did the program use selection criteria to target vulnerable households? 10. What aspects of the program were particularly ineffective? Sustainability: 11. What mechanisms/arrangements have been put in place to ensure the sustainability of the program’s results (in all domains including early warning systems and community safety net programs)? Evaluate whether these are likely to be sufficient. 12. What are the arrangements for a sustainable management of public works and infrastructures? 13. Has the technical assistance provided via program activities translated into the adoption of best practices? Do program participants continue to apply the techniques they learned? 14. Are there other benefits participants continue to have/experience after the end of the program? If so, what are they? Resilience: 15. What are the implemented activities that positively impact the target population’s resilience? What are the results of the conflict for the affected households and communities? 16. How have they managed the shock? What coping mechanisms are they using? (Both program-introduced and otherwise.) Why? Do they perceive themselves as vulnerable? 17. What is their perception of the result of the shock? Do they believe their situation has improved? Gotten worse? Not changed? Cross-Cutting Questions Related to Gender and Environment and Literacy 1. What have been the effects of the program on the lives of women program participants and their households? 2. How has the program affected the gender relationships within targeted households? 3. What effect has the program had on the capacity of households to manage the negative effects of their activities on their environment? Source: Annex I. Nema Qualitative Final Evaluation SOW. 3.0. Methodology 3.1. SOW Expectations for Data Collection Methodologies Given the fact that the Nema program suspended its activities almost two years ago, and the high levels of insecurity that still persist in some communities, the Consortium elected to conduct a qualitative final survey instead the normal type of quantitative survey most Title II food security programs conduct in their final program. This final qualitative survey and evaluation process was expected to: • Assess the current level of activity and functioning of the program’s supported activities almost two years after Nema officially shut down its activities in Douentza and Bourem; and • Use a participatory process to engage local communities, local government officials (Ministry of Health, Social Action, Agriculture, Water Resources), execution partners (Caritas, Tassaght), as well as a high proportion of the original staff of SCI, HKI, and CRS, who designed and executed the program, in the identification of what types of follow-on support might be needed to strengthen these activities and sustain them in the current context. The approved SOW7 anticipated that the evaluation would be based on two types of data: 7 The approved SOW is in Annex I. Nema Final Qualitative Evaluation. Chapter 1. Context and Methodology. December 30, 2013. Draft. 6 • Qualitative Survey: A survey of a representative sample of villages that were accessible in the low security zone; and • Key Informant Interviews: A series of open-ended interviews with key informants, including current and former implementation staff associated with the Consortium’s major implement partner in Douentza (Caritas); current and former staff associated with HKI, SCI, and CRS; and key government partners in Mopti (the regional capital for Douentza and current Caritas base), Gao (the regional capital for Bourem and current SCI base), as well as with some of the cercle- and commune-level government officials who were familiar with the program and the villages in the two areas. The same SOW anticipated that the evaluation would be led by a team of external consultants8 who would work with a core team of Consortium staff to execute the methodology under the direct supervision of the CRS monitoring and evaluation (M&E) department and the Nema program director and deputy director. 3.2. Global Organization of the Qualitative Survey and Key Informant Interviews 3.2.1. Qualitative Survey: The qualitative survey was designed and executed by a core survey team that represented all of the major partners in the Consortium under the direction of Dr. Sidikiba Sidibe and Deputy M&E Specialist Isack Dolo, with high levels of involvement from many of the current and former staff associated with the program. The same survey included key informant interviews with 13 staff members associated with the area commune-level health centers (Centre de Santé Communautaire or CSCOMs). 3.2.2. Key Informant Interviews at Mopti: Based in Mopti during the field portion of the qualitative survey, the second co-team leader, Dr. Della McMillan, conducted a series of key informant interviews in Sevare and Mopti. These interviews had two purposes: (a) to talk to key government partners about their appreciation of Nema’s impact both during and after the program; and (b) to gather as much secondary information as possible about the current status and level of government involve in the villages. One very important activity was series of participatory meetings organized by SCI for Bourem and Caritas for Mopti. The output of these meetings was a second participatory classification of the 130 villages in terms of various categories such as: (a) war-related impacts; (b) current levels of security for development 8 Co-Team Leader, Dr. Della McMillan has over 35 years of experience in African rural development, most of it in francophone West Africa. She is a livelihoods specialist with an extensive background in gender and capacity buildingShe has worked on the design and evaluation of Title II food security programs in Mali, Niger, Burkina Faso, Chad, Guinea, Ethiopia, and Uganda. She has participated in 15 evaluations, 12 times as team leader. Co-Team Leader, Dr. Sidibe Sidekiba (based in Conakry, Guinea) has over 15 years of experience in the design and execution of the health, sanitation, and nutrition components of Title II food security programs. As the senior health advisor of Africare’s Title II program in Guinea, he pilot tested one of the Hearth model programs for the rehabilitation of moderately malnourished children in francophone West Africa. As a MYAP coordinator in Rwanda, Dr. Sidibe pilot tested a series of highly innovative programs for HIV/AIDS-affected households. Dr. Sidibe has a University Diploma from the London School of Economics in health statistics, a Masters degree in epidemiology and a Masters in public health from the University of Bordeaux. Nema Final Qualitative Evaluation. Chapter 1. Context and Methodology. December 30, 2013. Draft. 7 workers; (c) current level of Ministry of Agriculture staff working in the area; (d) post-Nema sources and timing of aid in the communities; and (e) the number of registered cooperatives in each community. Although the 130 villages’ profile data was highly qualitative—and often based on telephone interviews and the SCI, Caritas, and Ministry of Agriculture staff’s familiarity with the zone, the interviews provided some ‘global’ data that helped situation the qualitative interviews within a wider context. 3.2.3. Participatory Elaboration of the Recommendations with Four Virtual Working Groups: Both types of data—the qualitative survey and the qualitative profile of the 130 Nema villages— have informed a core list of recommendation which the members of the strategic objective (SO) working groups helped formulate during an initial debrief session on November 29, 2013, and through email communication and document sharing with the other members of the virtual working groups9 that have grown up around this exercise for agro-enterprise development, nutrition/sanitation, early warning and response systems, and basic literacy, good governance, and capacity building (Table 1.3). Table 1.3. The Four Virtual Working Groups Involved in the Nema Qualitative Final Evaluation Group Individuals Involved and their Organizations SO1 • Chery Traore (CRS); • Abdoussalam Tiemogo Maiga (SCI/Gao); • Pierre Togo (Caritas/Mali); • Ahmadou Tadina (Agricultural Sector Chief, Douentza); and • Fanta Kone (CRS) SO2 • Dr. Fatou Koite (HKI); • Kerri Agee (CRS); • Dr. Aboubacar Halidou (HKI/Gao); • Abdoussalam Tiemogo Maiga (Field Office Manager SCI, Gao); and • Abdel Kader Sidibe (former CRS PD/Hearth specialist at Douentza) SO3 • Adama Sangare (CRS); • Pierre Togo (Caritas/Mali); • Abdoussalam Tiemogo Maiga (SCI/Gao); and • Ahmadou Tadina (Agricultural Sector Chief, Douentza) Cross Cutting • Abdoussalam Tiemogo Maiga (SCI/Gao); • Pierre Togo (Caritas/Mali); and • Isack Dolo (CRS) Source: Nema Final Mid-Term Evaluation. November 2013. Methodology: The final evaluation working groups were informal working groups that formed around the development and revision of the interview forms and the interview results. 3.3. Steps Involved in the Design and Analysis of the Qualitative Survey and Report 3.2.1. Step One: Initial Debriefing and Review of the SOW and Program (November 5, 2013): A 9 The consultants’ original proposal anticipated the development of four working groups that would provide the sounding board for the research results. Due to the geographic dispersion of many of the original and current Nema staff members, it was impossible to assemble all of the key staff at one time. In the course of executing the qualitative survey and key informant interviews, four working groups did assemble and provided the principle sounding board for the review of the focus group guides, research results, draft chapters, and recommendations. Nema Final Qualitative Evaluation. Chapter 1. Context and Methodology. December 30, 2013. Draft. 8 team meeting of major stakeholders from the different Bamako-based partners (SCI, HKI, and CRS) was organized, during which Nema Deputy Coordinator Adama Sangare provided an overview of the program and the preliminary work plan (Table 1.4). Table 1.4. Steps Involved in the Design, Execution, and Reporting of the Nema Final Qualitative Survey Steps Leadership Teams Participants Step One: Initial debriefing (Bamako) Adama Sangare (Nema Deputy Coordinator) and Isack Dolo (Charge de Suivi-Evaluation/CRS) All the Bamako-based stakeholder groups Step Two: Sampling (Bamako) Dr. Sidibe, Dr. McMillan, Isack Dolo, and Boureima Sacko Current and former staff affiliated with the different SO activities Step Three: Initial review and revision of the draft discussion guides (Bamako) Dr. Sidibe, Isack Dolo, and Boureima Sacko Step Four: Enumerator training and final revision of the forms (Severe) Dr. Sidibe (Consultant), Isack Dolo and Boreima Sacko (Suivi￾evaluation/CRS), Dr. Fatou Koite HKI, and Abdoussalam Tiemogo Maiga (SCI/Gao) Enumerators Step Five: Field interviews and initial data entry and analysis (Douentza and Bourem) Douentza: Dr. Sidibe (Consultant), Fatou Koite (HKI), and Isack Dolo (CRS) 15 Communities Bourem: Boureima Sacko (CRS), Adoulssalam Tiemogo Maiga (SCI), and Aboubacar Halidou (HKI) 6 Communities Step Six: Initial analysis and write up of the survey results (Severe) Dr. Sidibe and Dr. McMillan, Isack Dolo, and Boureima Sacko, with extensive backup from Pierre Togo (Caritas/Mali) Step Seven: Initial debriefing of the survey results and results of the key informant interviews (San) Dr. Sidibe and Dr. McMillan CRS and Caritas staff associated with the survey Step Eight: Preparation of the draft chapters and revised list of recommendations Dr. Sidibe and Dr. McMillan, with backup analytical support from Isack Dolo and Boureima Sacko Step Nine: Stakeholder debriefings Dr. Sidibe and Dr. McMillan -Representatives of all the Consortium partners (HKI, SCI, CRS, and Caritas) -USAID/Mali Title II manager Step Ten: Final revision and editing (December 7-23, 2013) Dr. Sidibe, Dr. McMillan, and Lynn Hurtak The Final Qualitative Evaluation working groups as reviewers (Table 1.3) Source:Nema Final Mid-term Evaluation. November 2013. 3.2.2. Step Two: Sampling (November 6-7, 2013): A four-part strategy was adopted to insure a representative sample of the villages for inclusion in the qualitative survey. • Preparation of a List of Program Activities by Community: Based on the end-of-program records, Nema M&E Specialist Boureima Sacko prepared a list of all program activities by SO. • Qualitative Ranking of Activities by Level of Success: Based on this initial list, the consultants facilitated a participatory ranking of the activities based on their level of success in the last year of the program. To facilitate the group exercise, these activities Nema Final Qualitative Evaluation. Chapter 1. Context and Methodology. December 30, 2013. Draft. 9 were color coded, with green meaning ‘highly successful,’ yellow meaning ‘under execution,’ and red meaning ‘just started or not working very well.’ The villages where the activity never started were left blank. • Qualitative Ranking of the Villages by their Performance and Perceived Level of Vulnerability at the End of the Program: 10 Based on this M&E-department facilitated exercise, the full team of sector specialist was reconvened to conduct a participatory grouping of the villages in terms of their end-of-program status (Table 1.5). • Selection of Sample Villages from the Clusters: Based on this initial clustering, the senior M&E specialists identified a sample of 21 villages from the different clusters—15 in Douentza and six in Bourem—from each category of communities that were considered secure enough for field visits (Table 1.5).11 Table 1.5. Retroactive Qualitative Classification of Nema Villages Used to Determine the Sample Frame for the Final Qualitative Survey Retroactive Qualitative Clustering of the Villages at the Time Nema Closed (March 2012) Bourem Douentza # Villages in Category at the End of the Project (% Total Villages) # Villages in Survey Sample (% of Total Sample) # Villages in Category at the End of the Project (% Total Villages) # Villages in Survey Sample (% of Total Sample) Performing Well/Less Vulnerable: The majority of the communities in this category benefited from the complete package of Nema interventions. By the end of the program, these were considered to be the most food secure Nema villages 3 (10%) 1 (17%) 24 (24%) 6 (40%) Average Performance/Average Vulnerability: The villages in this category were in the early stages of adopting the full package of innovations proposed by Nema 21 (70%) 4 (66%) 46 (46%) 7 (47%) Poor Performance/Very Vulnerable: Because of their extreme isolation, the villages in this category had only the bare minimum of Nema interventions. Many of the villages in this category were agro-pastoral and/or often very isolated from roads and markets. Some villages also had extreme water problems, very low levels of literacy, and social problems. Most of the villages were still considered highly food insecure when the program closed 6 (20%) 1 (17%) 30 (30%) 2 (13%) Total 30 Nema villages 6 sample villages 100 Nema villages 15 sample villages Methodology: Participatory classification of Nema villages at the time the program closed based on working group 10 A similar system of classifying villages based on performance was used during the qualitative mid-term evaluation of the Nema program (Tilford 2011:10). 8 For security reasons, four of the original sample villages had to be changed to four other villages in the same category during the first week of the fieldwork. Nema Final Qualitative Evaluation. Chapter 1. Context and Methodology. December 30, 2013. Draft. 10 discussions in November 2013 survey planning exercise. 3.2.3. Step Three: Revision of the Draft Interview Guides and Enumerator Recruitment (November 6-8, 2013): Parallel to the sampling process, Dr. Sidibe and Isack Dolo, the CRS M&E specialist charged with overseeing the evaluation exercise, facilitated a detailed review of the data-collection tools the Consortium had developed and submitted to Food For Peace (FFP) in conjunction with the evaluation SOW. During the same time period, CRS M&E Specialist Boureima Sacko facilitated the recruitment of 15 experienced enumerators that had worked on previous CRS and non-CRS food security research studies in the Bourem and Douentza regions. 3.2.4. Step Four: Enumerator Training and Final Revision of the Forms (November 11-12, 2013): A two-day workshop was organized at the Caritas office in Sevare, during which the core survey team (led by Dr. Sidibe and Isack Dolo) facilitated an in-depth review of the semi￾structured discussion guide and conducted an in-class simulated pilot test of the forms. During this same time period, the CRS M&E Specialists Boureima Sacko and Isack Dolo prepared a series of pre-coded data-entry tools (masques) to facilitate real-time data entry and analysis. 3.2.5. Step Five: Field Interviews and Initial Data Entry and Analysis (November 14-23, 2013): The field portion of the exercise was led by two inter-partner teams over an eight-10-day period in Bourem and Douentza. The teams interviewed a total of 1124 people (482 [43%] of them women) in 117 focus group discussions (38 at Bourem, and 79 at Douentza) (Table 1.6). To facilitate real-time data analysis and write up, the team leaders facilitated an initial debriefing and data entry session at the end of each field day. Table 1.6. Number of Individuals Interviewed in the Nema Final Qualitative Survey Community-Based Organizations Bourem Douentza # of Groups Interviewed # of Persons Interviewed # of Groups Interviewed # of Persons Interviewed M F T M F T AEG 3 9 23 32 11 115 94 209 SLIC 3 3 92 95 12 100 121 221 CHV (Relais) 5 20 8 28 9 31 17 48 FFW 5 31 14 45 8 68 0 68 PD/Hearth 4 0 34 34 8 2 60 62 GAP-RU 6 24 3 27 9 20 1 21 Leader 6 49 1 50 14 152 6 158 Radio 0 0 0 0 1 1 0 1 Macon 2 2 0 2 2 2 0 2 Health staff 3 1 2 3 4 5 0 5 Water Point Management Committee 1 0 4 4 1 7 2 9 Total 38 139 181 320 79 503 301 804 Source: Nema Final Qualitative Evaluation. November 2013. 3.2.6. Step Six: Initial Analysis and Write up of the Survey Results (November 25-28, 2013): The CRS M&E specialists working in close collaboration with Dr. Sidibe completed an initial analysis and write up of the qualitative survey data at the Caritas field office over a three-day period. Nema Final Qualitative Evaluation. Chapter 1. Context and Methodology. December 30, 2013. Draft. 11 3.2.7. Step Seven: Initial Debriefing of the Survey and Results of the Key Informant Interviews (November 29, 2013): An initial debriefing of the results was conducted at the CRS office in San, during which the consultants facilitated a group discussion of the data that generated an initial list of global recommendations for each strategic objective and a list of specific sub￾recommendations. 3.2.8. Step Eight: Preparation of the Draft Chapters and Revised List of Recommendations (December 2-5): During the final week of the exercise, the team developed a list of recommendations and fine-tuned them with input from different staff associated with the different program stakeholders—HKI, SCI, CRS, and Caritas. 3.2.9. Step Nine: Stakeholder Debriefings (December 6, 2013): A final debriefing with representatives of major partners HKI, SCI, CRS, and Caritas was held on Friday, December 6, 2013, followed by a brief debriefing of the principal conclusions with USAID/Mali Title II Manager John Mullenax. 3.2.9. Step Ten: Final Revision and Editing (December 7-23, 2013): Based on the input from the stakeholders, the evaluation consultants developed an English-draft version of the draft chapters, many of which were in French. Each chapter was reviewed by the members of the technical working groups (Table 1.3). Final editing was conducted by the consultants in collaboration with Editor Lynn Hurtak. 4.0. Organization of the Report Chapters 2-5 provide a brief overview of the results of the final qualitative survey and working group discussions for the program’s three strategic objectives (Chapters 2-4) and the cross￾cutting activities related to literacy, good governance, and capacity building (Chapter 5). Each chapter follows the same basic format that examines: • Strategy: The original strategy or “theory of change” that the activities were designed to address; • Evolution of Activities (Years 1-4): The evolution of the program’s activities related to that particular SO or transversal activities, early evidence of results, and some of the key factors that affected the efficiency of these processes during Years 1-4 of the Nema program; • Current Level of Activities (November 2013): - The current level of activity of the Nema-supported activities and any evidence that they have affected household resilience; and - Some of the key factors that affected these outcomes—i.e. the results and effects of the different activities, the effectiveness of the implementation process, the sustainability of the results, and the activities’ impact on the local people’s resilience; and • Lesson Learned and Recommendations: The major lessons that can be learned from the program experience, as well as recommended actions for: 12 12 CRS/HKI/SCI. 2013. SOW. Pp. 6-7. Nema Final Qualitative Evaluation. Chapter 1. Context and Methodology. December 30, 2013. Draft. 12 - Increasing the sustainability of the program accomplishments; and - Monitoring or strengthening the benefits initiated by the program. Chapter 6 summarizes the major conclusions and lessons learned from the chapters. Nema Final Qualitative Evaluation. Chapter 2. SO1. December 30, 2013. Draft. 13 Chapter 2 SO1: Livelihood Strategies More Profitable and Resilient13 This chapter provides a brief overview of the results of the Final Qualitative Survey and Evaluation for the Nema program’s Strategic Objective 1 (SO1) activities. The chapter is in five sections: Section 1.0. SO1 Strategy/Theory of Change: Describes the intervention strategy and key actors involved in executing the strategy’s two major intermediate results (IRs). Section 2.0. Evolution of the Strategy for SO1 IR 1.1 (Years 1-4): Describes the evolution of the implementation strategy for IR 1.1 during the first four years. Section 3.0. Evolution of the Strategy for SO1 IR 1.2 (Years 1-4): Describes the evolution of the implementation strategy for IR 1.2 during the first four years.14 Section 4.0. Current Level of the SO1 Activities in the Nema Villages for IR 1.1 and IR 1.2: Describes: • The early evidence about how these activities have affected household food security in the target communities; and • The effectiveness of the Nema monitoring and evaluation (M&E) system for measuring the objective, the intermediate results, and the principal impact and monitoring indicators. Section 5.0. Lessons Learned and Recommendations: Presents a list of recommendations that the evaluation team—working in close collaboration with the key implementation, execution partners have identified for: • Corrective actions recommended for increasing the sustainability of the program impacts; and • Strengthening the benefits initiated by the program. 1.0. SO1 Strategy/Theory of Change Despite the vulnerability of the north, the Mopti region has historically had a large, well￾developed network of cooperatives. The Ministry of Agriculture estimates that there are about 60 registered cooperatives in the 100 Nema villages in Douentza.15 The original intent of the program strategy for agro-enterprise groups (AEGs, or GAEs [Groupements Agro-Enterprise]) was to help the vulnerable households in the villages served by the USAID Food for Peace 13 This chapter was prepared with the assistance of an informal Nema SO1 working group that developed around the final evaluation team’s field and key informant interviews. This working group included: Abdoussalaam Tiemogo Maiga (SCI/Gao); Pierre Togo (Caritas/Mali); Ahmadou Tadina (Agricultural Sector Chief, Douentza); Fanta Kone (CRS); and Chery Traore (CRS). The working group was supported in its analysis of previous and current activities by Boureima Sako and Isack Dolo (M&E department CRS), who co-facilitated the field portion of the exercise with Dr. Sidikiba Sidibe (Consultant, Nema) and Salam Maiga (SCI/Gao) in Bourem. 14 I.e. The results and effects, the effectiveness of the implementation process, the sustainability of the results, and the activities impact on the local people’s resilience. 15 The team was not able to get this information for Bourem. Nema Final Qualitative Evaluation. Chapter 2. SO1. December 30, 2013. Draft. 14 (FFP)-funded one-year emergency Single-Year Assistance Program (SYAP) to: • Build their capacity to engage in commercial agriculture; and • Eventually transition into the societes cooperatives (registered cooperative societies) needed to sustain these activities through linkages to local micro-finance institutions, input, and output markets. A second set of activities, the savings and internal lending communities (SILCs), were designed to strengthen the capacity of vulnerable individuals who did not have the financial resources and/or labor to participate in the more high-risk agro-enterprises being supported by the Nema￾supported AEGs and local cooperatives (Figure 2.1). It was anticipated the SILC training programs, combined with their internal saving, would build the vulnerable groups’ financial capacity to engage in more profitable agro-enterprises and help a certain position of the members to transition to more permanent institutions (i.e. cooperatives or established women’s groups) once they were ready. A third set of activities helped the program organize farmer field schools (FFSs) as a tool for facilitating farmer access to information about more productive, drought-tolerant crop and livestock production technologies being promoted by the local representatives of the Ministry of Agriculture and Institute of Rural Economy (IER). The initial intent of the program was to train a network of trainers to conduct the first round of trials under the direct supervision of the Consortium and its key local partners (Caritas/Mali, Tassaght, Ministry of Agriculture, IER), and that these trainers would then conduct a second round of trials in other villages and/or sections of the villages (many of which are large). It was anticipated that the Nema-trained trainers would provide the villages with a point of contact between the local producers and the Ministry of Agriculture and IER. This idea of strengthening the Douentza and Bourem farmers’ access to IER, the leading agricultural research institution in Mali, was highly innovative in 2006-2008 when the Nema program was designed. The original concept was that CRS and the executing partners and the Ministry of Agriculture would provide the ‘feet’ for testing new technologies that IER had identified as profitable and that these same institutions would feed the trial data back to IER for analysis. This collaboration was expected to strengthen IER’s presence in the zone and to help the farmers and Ministry of Agriculture officials to ‘draw down’ the top leading technologies being tested in Mali for their respective agro-ecological zones. Nema Final Qualitative Evaluation. Chapter 2. SO1. December 30, 2013. Draft. 15 Figure 2.1. Nema Strategy for SO1, Years 1-4 A separate set of Food for Work (FFW) activities (administered under SO3) were designed to create various agro-sylvo-pastoral investments that the communities needed to sustain certain agro-enterprise activities (Figure 2.1). A third set of safety nets (filets de securite) also administered under SO3 was designed to build the food security of the most vulnerable households who were unable to benefit from the program supported FFW activities or the mainstream agricultural activities supported by the cooperatives with assistance from IER and the Ministry of Agriculture. The anticipated impact of this model, which was tracked through the program Indicator Performance Tracking Table (IPTT), was to make the community-based livelihood strategies more profitable and resilient through the achievement of two IRs (Figure 2.1): • IR 1.1. Household agricultural production increased; and • IR 1.2. Targeted household revenues increased. The principle mechanism for executing this strategy was the Nema Agents de Developpement Communautaire (ADCs), who were recruited and trained by the two executing partners: Tassaght at Bourem and Caritas at Douentza. Each ADC received an initial baseline training, which was followed by more specialized training in the technical areas they were supposed to backstop during the first year of the program. During the first and second year, the ADCs were specialized with certain ones focused on SO1, SO2, and SO3 activities. In order to accelerate the program’s achievement of its major outputs, the mid-term evaluation recommended converting this system of specialized ADCs to a system of multi-purpose (polyvalente) ADCs in June 2012 (end of the third year) and all agents were retrained and redeployed as multi-purpose ADC agents in the fourth year (till March 2012). Although the conversion of the health and nutrition Nema Final Qualitative Evaluation. Chapter 2. SO1. December 30, 2013. Draft. 16 ADCs had a negative influence on the efficacy of the program’s SO2 health and nutrition activities, it helped accelerate the achievement of the program targets for creating AEGs and SILCs. Within the Consortium, CRS was the lead for SO1 and provided technical assistance to the two local NGOs (Tassaght and Caritas) that were responsible for field implementation for the development and monitoring of the AEGs, SILCs, and FFSs. The principal technical partners for the FFS component were IER and CRS. The same ADCs that were responsible for the execution of the SO1 activities were responsible for overseeing the committees that were responsible for managing the FFW and filet de security activities. 2.0. Evolution of the Nema Strategy and Activities and Early Evidence of Impact 2.1. Evolution of Activities and Early Evidence of Results 2.1.1. IR 1.1. Household Agricultural Production is Increased 2.1.1.1. AEGs: The principal activity for this IR was to reinforce the institutional and technical capacities of 75 agro-enterprise groups over the life of the program. The AEGs are groups of 20-25 community members who work together and contribute to a shared production and marketing plan with technical and financial support from the program. Training covers organization development, basic business skills, and understanding market requirements. With guidance from the SO1 team and its partners, AEG members undertake market opportunity identification exercises and conduct market chain analyses to better identify opportunities that complements each group’s resources, strengths, and potential. 16 This training is supposed to enable the AEGs to make more rational decisions on what to produce, when to produce, how to produce, and at what price to sell to realize maximum profits. Once groups have had sufficient hands-on experience through the training exercises, they develop a business plan and submit it to the program for financing. In Year 2 alone, the program provided 20 investment grants worth almost $100,000; by the end of Year 3, all but five of the 75 AEG business plans were financed. The mid-term evaluation considered this an impressive achievement since the “AEGs must satisfy a number of stringent criteria before they qualify for financing.”17 Given the many benefits of AEGs, one of the key questions in the mid-term evaluation was, “Is it possible to have an AEG in each of the 130 villages in the Nema Program?” Based on interviews, the mid-term evaluations concluded: “The answer is NO, as certain conditions need to be met before an AEG can be successful. These conditions include social cohesion within the group, a shared vision, the availability of markets, a certain level of experience among group members, and the 16 K. Tilford. 2011. Nema Mid-Term Evaluation. Pg. 14. 17 K. Tilford. 2011. Nema Mid-Term Evaluation. Pg. 14. Nema Final Qualitative Evaluation. Chapter 2. SO1. December 30, 2013. Draft. 17 availability of resources.”18 This decision to stay with the original AEG model meant that although the Nema program achieved or surpassed all of its original targets, the activity benefited only 38% of the villages in Douentza; and only 30% of the Bourem villages (Table 2.1). Had the Nema program continued as planned, this highly unequal distribution of one of the program’s principal activity sets would have no doubt had a very negative impact on the achievement of the principal impact indicators which measured the program’s population-based impact on household food security, dietary diversity, and household revenues (see Nema IPTT, Annex I). Table 2.1. Targets and Achievements for the AEGs and Number of Nema Communities that Benefitted from the Activity, Years 1-4 Activity Target (LOA) Achieved (2012) Achievement vs. Target Nema Communities Covered by the Activities Units # # % % Targets vs. Achievements for the Major Indicators AEGs established 75 90 120% AEGs that have developed business plans 75 73 97% AEGs whose business plans have been funded 75 73 97% # of Nema Communities Where AEGs Were Created Bourem N/A* 9/30 N/A 30% Douentza N/A* 38/100 N/A 38% Source: Disaggregated analysis of the Nema program tracking data by Isack Dolo and Boureima Sacko during the Qualitative Final Evaluation. Acronym: LOA: Life of Activity *Neither the MYAP proposal nor the IPTT set geographical targets for this activity. 2.1.1.2. Farmer Field Schools (FFSs): To reinforce the technical capacities of the AEGs, the MYAP organized FFSs. 19 These peer school were designed in close collaboration with Mali’s IER as a way of strengthening farmers’ access to improved crop technologies and better production techniques. Although the memorandum of understanding (MOU) for the activity was signed with IER, most of the day-to-day coordination of the FFSs was conducted by the program-trained training of trainers (TOT) FFS trainers (formateurs) in partnership with the extension agents of the Ministry of Agriculture. Each FFS started with a participatory diagnostic exercise to select the themes for the school. For agricultural production, contrasting demonstration plots were planted to facilitate a comparison of traditional methods and inputs with improved techniques and inputs. The participants analyzed the results and drew their conclusions. Following the FFS, the participants were encouraged to set up replication plots to disseminate what they had learned. Based on a recommendation made during a USAID/FFP supervision mission, the SO1 activity was significantly expanded during the third year of the program. 20 This 18 K. Tilford. 2011. Nema Mid-Term Evaluation. Pg. 14. 19 K. Tilford. 2011. Nema Mid-Term Evaluation. Pg. 15. 20 Evolution of the Targets for the FFSs in Years 2-4 Nema Final Qualitative Evaluation. Chapter 2. SO1. December 30, 2013. Draft. 18 shift in strategy enabled the program to overachieve the original proposal target of 1,500 target farmers by over 350%, and to train almost double the original number of TOTs they had planned to train (Table 2.2). The same shift in strategy meant a much larger percentage of the target communities were affected by the FFSs than by the AEGs—about 50% of the Nema communities at Bourem and 90% of the Nema communities at Douentza (Table 2.2). Based on the farmers’ enthusiastic endorsement of the FFSs, the program expected that the FFSs would, after a slow start, increase the famers’ adoption of the new agro-enterprise technologies being promoted by the FFSs, which was being tracked by Nema Monitoring Indicators 1.1.1 and 1.1.2 (Annex I). Unfortunately, the program was forced to suspend its activities before these trails had a major impact on technology adoption, which was measured by monitoring indicators 1.1.1 and 1.1.2 (Annex I). Table 2.2. Targets and Achievements for the FFSs and Number of Nema Communities that Benefitted from the Activity, Years 1-4 Activity Target (LOA) Achieved (2012) Achievement vs. Target Nema Communities Covered by the Activities Units # # % % Targets vs. Achievements for the Major Indicators Total number of persons participating in the FFS 1,500^ 5,251 350% FFS TOT trainers 15 29 193% FFS participants who have replicated the technologies 1,500 5,276 352% # of Nema Communities Where FFSs Were Created Bourem N/A* 15/30 N/A 50% Douentza N/A* 9/100 N/A 90% Source: Nema IPTT (Annex I) and disaggregated analysis of the Nema program tracking data by Isack Dolo and Boureima Sacko during the Qualitative Final Evaluation. *Neither the MYAP proposal nor the IPTT set geographical targets for this activity. ** Rainy season trials never executed. ^ Original target number. Original LOA target was revised upward in Year 3. 2.1.2. IR 1.2. Targeted Household Revenues Increased: The MYAP strategy for achieving this IR focused on the establishment of 585 SILCs. The SILC approach was based on a simple premise: Participants should use their own resources to set up internal loan funds that they manage themselves. They can also set up a social insurance fund for the emergency needs of their members. This particular model was based on a successful approach to SILC that CRS pilot tested in other West African countries and other areas of Mali.21 Year Revised Target Achievement % Target achieved Year 2 175 117 67% Year 3 1,000 4,766 477% Year 4 4,950 368 Crop year not completed 21 K. Tilford. 2011. Nema Mid-Term Evaluation. Pg. 16. Nema Final Qualitative Evaluation. Chapter 2. SO1. December 30, 2013. Draft. 19 The SILC component of the MYAP was set up as a strategy for building the livelihood assets of vulnerable women who often did not have the capacity or social access to the main AEG and cooperative groups. It was expected that:22 • The women would use the SILC loans to develop and expand small-scale commercial activities, thereby increasing their household revenue; and • Once trained, the SILCs would be able to function independently with no outside support. The initial MYAP SILCs were established by community development agents (CDAs, also known as Agents de Développement Communautaire or ADCs), who also identified community members interested in being trained as field agents (animateur villageois or AVs). Once trained, the AVs contract with interested community members to set up additional SILCs on a fee-for￾service basis. The mid-term evaluation considered the SILCs as: “One of the success stories of the MYAP for a number of reasons. First of all, the members help to ensure short- and long-term household economic resilience to periodic food security shocks. For example, households [with members in the SILCs] are better able to pay basic fees and transportation costs to the nearest health centers, purchase varied and more nutritious food, and pay for school-related expenses for their children.”23 When the program ended, the SILCs had already exceeded the Nema program’s life of activity targets for the total number of first generation (i.e. CDA-created) SILCs and AVs to be trained, and was well on its way to achieving the 585 SILC program-end target. Unfortunately, the program’s SILC activities suffered from the same concentration of activities in the more accessible villages, which meant that only 57% (17) of the Bourem villages had SILCs and only 58% (58) of the Douentza villages (Table 2.3). 22 K. Tilford. 2011. Nema Mid-Term Evaluation. Pg. 16. 23 K. Tilford. 2011. Nema Mid-Term Evaluation. Pg. 17. Nema Final Qualitative Evaluation. Chapter 2. SO1. December 30, 2013. Draft. 20 Table 2.3. Targets and Achievements for the SILCs and Number of Nema Communities that Benefitted from the Activity, Years 1-4 Activity Target (LOA) Achieved (2012) Achievement vs. Target Nema Communities Covered by the Activities Units # # % % Targets vs. Achievements for the Major Indicators SILCs (1st generation) 60 80 SILC village leaders (AVs), trained and available to start new groups 60 86 SILCs (2nd generation) 525 146 # of Nema Communities Where SILCs Were Created Bourem N/A* 17 57% Douentza N/A 58 58% Source: Source: Nema IPTT (Annex I) and disaggregated analysis of the Nema program tracking data by Isack Dolo and Boureima Sacko during the Qualitative Final Evaluation. *Neither the MYAP proposal nor the IPTT set geographical targets for this activity. 2.1.3. Cross-Cutting Activities: Three critical cross-cutting activities that affected the profitability of the other SO1 activities and/or the three principal impact indicators for SO124 were (Table 2.4): • FFW: The program’s FFW activities (under SO3), which provided short-term in-kind employment for 58,919 persons that built 143 agro-sylvo-pastoral, roads, and rural infrastructures in 100% of the Bourem villages and 78% of the Douentza villages; • Safety Net: The program’s safety net activities (under SO3) provided short-term food assistance (three months) for the poorest five percent of the population, which included a total of 27,013 beneficiaries of the first four years; and • Gender Integration: The program’s strong focus on gender insured that women were very strongly involved in almost all the SO1 activities and cross-cutting FFW and safety net activities. Table 2.4. Number of Villages and Beneficiaries that Benefited from the Major Categories of Nema SO1 Activities, Years 1-4 Activity # Villages (%) Direct Beneficiaries Bourem Douentza Bourem Douentza Male Female Male Female AEG 9 (30%) 38 (38%) 285 191 928 692 SILC 17 (57%) 58 (50%) 160 1,016 299 2,933 FFS 15 (50%) 90 (90% 893 590 2290 993 SO1: FFW 30 (100%) 78 (78%) 10,808 11,249 18,062 18,800 SO3: Safety Nets 30 (100%) 100 (100%) 5673 5905 7563 7872 Source: Disaggregated analysis of the Nema program tracking data by Isack Dolo and Boureima Sacko during the Qualitative Final Evaluation. November 2013. 24 These were: Impact Indicator 1.1. # months of adequate household food provisioning; Impact Indicator 1.2. Average HHDD score (Household dietary diversity score); and Impact Indicator 1.3. % of households wo report increase in financial resources. Nema Final Qualitative Evaluation. Chapter 2. SO1. December 30, 2013. Draft. 21 2.2. Early Evidence of Results and Effects 2.2.1. Average Household Food Security: It was anticipated that the SO1 activities, combined with the crosscutting FFW and safety net activities, would increase household food security using the standard Title II FFP endorsed measure of the Months of Adequate Household Food Provisioning (MAHFP). The baseline measure of the MAHFP was 3.8 months for all of the villages—2.8 for the Bourem villages and 5.8 for the Douentza villages. Since the program did not re-measure this indicator during either the mid-term or before closing, the final evaluation survey conducted a retroactive measure using an early Participatory Rural Analyses (PRA) version of the MAHFP. Using this qualitative measure of the MAHFP to measure household food insecurity before, during, and after the program, the survey showed that household food security did increase during the program (Figure 2.2): • From 2.0 months to 2.6 months in the Bourem villages; • From 5.6 months to 7.4 months in the Douentza villages; and • From 3.8 months to 5.0 months for both areas.. This substantial increase in food security was attributed to the combination of the FFW and safety nets, with a slight increase in crop productivity due to the early introduction of the improved varieties and crop production technologies. Figure 2.2. Nema Results and Effects in the Douentza Villages: Average Household Food Security (MAHFP) Before and During the Nema Program Methodology and Source: Based on food security calendar Participatory Rural Analyses (PRAs) during the community focus groups conducted as part the Nema Final Qualitative Survey, November 2013. 2.0 5.6 3.8 2.6 7.4 5.0 0.0 1.0 2.0 3.0 4.0 5.0 6.0 7.0 8.0 Bourem (n=6 villages) Douentza (n=14 villages) All (n=20 villages) Before NEMA During NEMA Nema Final Qualitative Evaluation. Chapter 2. SO1. December 30, 2013. Draft. 22 2.2.2. Percentage of Households Classified as Less Vulnerable: Although average food security appears to have increased in most of the program villages, there is clear evidence from the qualitative final retroactive survey that this successful impact was more pronounced in some communities than in others (Table 2.5). When the Nema program started, all 130 of the program villages would have been classified as extremely food insecure, or having ‘average’ food security levels. By the end of the program: • In general, the most successful villages—i.e. the 10-24% of Nema program villages that staff classified retroactively during the final qualitative evaluation as being less food insecure—were those that had the highest concentration of Nema program interventions. In most cases, these were also the most accessible villages and villages which had had anterior investments by other donor-funded programs (Table 2.5); • The least successful villages—i.e. the 20-30% villages that the Nema classified as still being chronically food insecure at the end of the program—were the more isolated villages that had never benefitted from the Nema AEG and SILC activities and that had had—because of their isolation—lower levels of FFW investment. These were typically the most food insecure villages; and • The middle category of villages, which were still considered food insecure, were the villages where Nema program activities were just starting when the program ended in March 2012. Table 2.5. Retroactive Participatory Classification of Nema Program Villages by Level of Performance and Food Insecurity When the Program Closed, March 2012 Retroactive Qualitative Clustering of the Villages at the Time Nema Closed Bourem Douentza # Villages in This Category at the End of the Program (% Total) # Villages in This Category at the End of the Program (% Total) Performing Well/Less Vulnerable: The majority of the communities in this category benefited from the complete package of Nema interventions. By the end of the program, these were considered to be the most food-secure Nema villages 3 (10%) 24 (24%) Average Performance/Average Vulnerability: The villages in this category were in the early stages of adopting the full package of innovations proposed by Nema 21 (70%) 46 (46%) Poor Performance/Very Vulnerable: Because of their extreme isolation, the villages in this category had only the bare minimum of Nema interventions. Many of the villages in this category were agro-pastoral and/or often very isolated from roads and markets. Some of the villages also had extreme water problems, very low levels of literacy, and social problems. Most of the villages in this category were still considered highly food insecure when the program closed 6 (20%) 30 (30%) Source and Methodology: Participatory classification conducted by the Nema final evaluation working groups based on program data concerning the status of the program interventions and the communities’ food security levels at the time the program closed. This table was developed to facilitate a representative sample of villages for the study. Nema Final Qualitative Evaluation. Chapter 2. SO1. December 30, 2013. Draft. 23 3.0. Current Level of SO1 Activities Supported by Nema 3.1. Agro-Enterprise Groups 3.1.1. Current Level of Functioning of the AEGs Interviewed: The majority of the groups that were interviewed in the final qualitative survey were considered ‘functional’ (Table 2.5). The researchers identified a functional group as one that continued to work on its business plan (plan d’affaire), organized meetings, kept minutes on the meetings, and where the amount of money the group had in the bank was known to the AEG members (Tables 2.6 and 2.7). The two groups that were assessed as ‘non-functional’ were groups that were no longer able to account for the money they received and/or had divided the group’s operating capital (fonds de caisse) between the members and no longer conducted meetings. Table 2.6. Functionality of the AEGs Interviewed During the Nema Program Final Evaluation Survey, November 2013 On-Going Activities Bourem Douentza # of Groups Interviewed 3 11 # of Groups Classified as Functional 3 9 % of Total 100% 82% % of the AEGs that continue to apply the production techniques that they learned during Nema 67% 100% Average Number of Persons Per Group that Continue to Apply New Production Technologies Learned During Nema Male 2 10 Female 16 15 Average Number of Persons Per Group that Attended the Nema-Sponsored Literacy Training Sessions Male 18 17 Female 7 14 Average Number of Persons Per Group that Know How to Read, Write, and Conduct Basic Calculations Male 9 6 Female 6 6 Source: AEG focus groups, Nema Final Evaluation Survey. November 2013. 3.1.2. Summary Assessment of Strengths, Areas that Need Strengthening, and Opportunities in the Current Context: 25 • Organizational Capacity Building and Systems for Monitoring this Capacity: The Nema AEG model made consistent use of a grille (capacity index) that enabled Nema Agricultural Specialist Chery Traore to monitor the core capacities of the groups in order to better target technical assistance and training. • Gender Roles and Relations: The Nema targets for women’s participation in the AEG resulted in a much higher representation of women in this activity than was anticipated, almost equivalent to the participation of men (Table 2.3). • Complementary FFW Assistance: The AEGs received a high level of technical assistance and many AEG communities were able to request and receive FFW-funded infrastructure development (irrigated perimeters, etc.) that helped them be successful. 25 The results and effects, the effectiveness of the implementation process, the sustainability of the results, and the activities impact on the local people’s resilience. Nema Final Qualitative Evaluation. Chapter 2. SO1. December 30, 2013. Draft. 24 • Insufficient Time for Developing the Appropriate Level of Financial Management Before the Program Ended: In March 2013, the vast majority of the AEGs had been operating for less than two years, which meant that the program systems needed to manage the AEGs’ operating capital received after their initial training (ranging from 50,000 CFA to 450,000 CFA) were just starting to develop. • Emerging Commercial Opportunities and Cooperative Structures in the Two Areas: Faced with the weak capacity of their groups to mobilize the resources they needed to market the crops once the Nema program ended, many of the AEG members encountered during the final evaluation survey appear to have joined—as the program expected— some of the established cooperative groups in their local communities (i.e. the registered cooperatives). Two of the 60 registered cooperatives in the program intervention area are linked to international partners and have supported a new low-cost pilot initiative to promote the community-based cooperatives’ ability to support intensive commercial millet and nebie (cowpea or black eyed peas) production. During the final evaluation key informant interviews, the Ministry of Agriculture sector chief identified about 40 of the 60communities with registered cooperatives as having good potential to scale up this new package. This new package builds on more than 20 years of collaborative Ministry of Agriculture/IER/Sorghum and Millet Research Support Program (INTSORMIL)/International Crops Research Institute for the Semi-Arid Tropics (ICRISAT) and World Bank support in the Mopti region. The same intensive package is benefitting from a new initiative to support cooperative-managed marketing to support new technologies that the McKnight Foundation has funded at Oualo since 2009 (Text Box 2.1). Many of the Nema AEG-trained farmers have joined these cooperatives and/or formed sub-groups and/or contacted the Caritas staff for assistance with marketing, as well as accessing seeds and fertilizer that they need to sustain their interventions. One simple indicator of this persistent interest is the number of calls that Caritas staff receive to assist the representatives of the Douentza AEG groups with input and marketing issues. Nema Final Qualitative Evaluation. Chapter 2. SO1. December 30, 2013. Draft. 25 Text Box 2.1. The INTSORMIL Project in the Nema Village of Oualo Il comporte les phases suivantes : 1. Phase de Production qui consiste : Transfert de Technologies : qui permet d’améliorer la production et dégager un surplus de production à travers : -Un Appui en intrants agricoles à la société coopérative (semence améliorée, engrais minérale et fongicide) et les techniques culturales fournis par l’agent technique. Le comité de gestion redistribue aux membres de la société coopérative. Chaque membre reçoit par exemple pour 1ha 50kg d’urée, 50kg DAP ,8-10kg de semence et 1sachet de fongicide. -Un Appui technique est fourni par l’agent du service agricole qui les forme pour l’application correcte des techniques culturales (Préparation du sol-dose et application de l’engrais- densité de semis￾démariage et entretiens culturaux…..etc.) -Placement de carrés de rendement : qui permet de déterminer et renseigne sur le niveau de rendement et la production à l’hectare 2. Phase de remboursement Chaque membre de la coopérative rembourse la valeur des semence-engrais et fongicide en nature au prix à la récolte auprès de la coopérative. Le Président, le secrétaire à la production et le secrétaire administratif sont chargés de la récupération et de stocker au magasin. 3. Phase de stockage Les productions sont mises dans des sacs PICS et stocker au magasin sur des palettes pour éviter la détérioration précoce des grains. 4. Conditionnement du Produit : un magasin approprié qui respecte les normes est fourni par la coopérative pour stocker les productions afin de conserver et d’avoir un produit de qualité. 5. Phase de commercialisation(ou vente) : La vente est réalisée pendant la période où le prix est très élevé sur le marché (généralement les mois d’avril￾mai-juin-juillet).Le secrétaire à la commercialisation et le trésorier sont chargés de la vente. La coopérative pourra réaliser environ 150% des recettes en aval et accroitre considérablement ses revenus. 6. Phase renforcement des capacités organisationnelles de la coopérative. -L’animation/Sensibilisation sur les rôles et les responsabilités des membres de la coopérative et des membres du comité de gestion et du comité de contrôle pour le remboursement intégral, l’achat et l’approvisionnement correct des membres en intrants. -L’ouverture de compte bancaire par la coopérative: l’argent issu de la vente des céréales est versé dans le compte de la coopérative sous la direction du président, du trésorier et un membre de contrôle. La coopérative peut aussi contracter un prêt pour ses membres -La tenue du bilan de la campagne et compte rendu à tous les membres. -Mise en relation de la coopérative avec les acheteurs potentiels à qui un produit de qualité est présenté. Source: Chef du Secteur, Ministere d’Agriculture, Douentza. Novembre 2013. 3.2. Savings and Internal Lending Communities 3.2.1. Current Level of Functioning of the SILCs Interviewed: • 13 of the 15 groups that were interviewed during the qualitative final evaluation are considered functional in that they still consider themselves to be a group and they still have their core funding (Tables 2.7 and 2.8). • The principal reasons given for low lending level of the groups were varied and generally related to the high level of insecurity in the zone. Nema Final Qualitative Evaluation. Chapter 2. SO1. December 30, 2013. Draft. 26 • Despite these problems, all 15 SILCs interviewed reported that they were still getting together regularly, still contributing small amounts to the group savings program, and still contributing to the social fund; even 10 of the 12 groups still had their bank accounts (Table 2.9). • In general, however, the SILCs were not circulating the funds (through loans) at the same level they had during Nema’s support, which is completely understandable given the high levels of insecurity in the zone (Table 2.9) • A key theme that emerged during the SILC focus groups was the impact that each group’s membership in the SILC had on their ability to manage the crisis (Text Box 2.2). 3.2.2. Summary Assessment of Strengths, Areas that Need Strengthening, and Opportunities in the Current Context: 26 • Organizational Capacity Building and Systems for Monitoring this Capacity: All of the SILCs interviewed expressed their deep appreciation of the training and technical assistance they received during the program. The Nema program’s capacity to provide appropriate technical assistance and training was helped by its consistent use of a standard CRS capacity-monitoring tool used in Burkina, Mali, and Niger. Table 2.7. Functionality of the SILC Groups Interviewed During the Nema Program Final Evaluation Survey, November 2013 Number of Groups Considered Functional Average Number of People Who Continue to Apply the Training Received During Nema Average Number of People Per Group that Continue to Apply the Training Learned During Nema Average Number of Persons that Know How to Read, Write, and Conduct Basic Calculations in the SILCs # % Male Female Male Female Bourem (n=3) 3 100% 0 7 22 2 7 Douentza (n=12) 10 83% 1 4 11 1 5 Source: SILC focus groups, Nema Final Qualitative Evaluation Survey, November 2013. Table 2.8. SILC Activities During and After the Nema Program Activity Bourem Douentza During After During After Organize meetings 3 1 12 Group savings 3 1 12 9 Give out credit 3 3 11 10 Group contributions to a social fund 3 2 12 10 Compte la Caisse 0 0 4 3 Purchase and storage of paddy rice 1 Source: Sidibe Sidikiba, Isack Dolo, and Boureima Sacko. Nema Final Qualitative Evaluation Survey, November 2013. 26 The results and effects, the effectiveness of the implementation process, the sustainability of the results, and the activities impact on the local people’s resilience. Nema Final Qualitative Evaluation. Chapter 2. SO1. December 30, 2013. Draft. 27 Table 2.9. SILC Financial Resources During and After the Nema Program Activities Douentza (n=10) Bourem (n=3) Average During the Program (FCFA) Current Average Post￾Program (FCFA) Average During the Program (FCFA) Current Average Post￾Program (FCFA) Current savings 373,819 151,654 825,000 466,000 Current credits 389,022 320,318 38,500 304,500 Social fund 77,702 16,736 50,000 101,866 Fines 3,322 2,597 3,750 5,000 Interest 42,107 26,000 1,666 Total Available 885,972 517,305 917,250 879,032 Source: Sidibe Sidikiba, Isack Dolo, and Boureima Sacko. Nema Final Qualitative Evaluation Survey, November 2013. Text Box 2.2. Sample Responses to Questions Asked to the SILC Focus Groups Depuis que les activités SILC ont commencé, quels changements/différences constatez-vous dans vos vies? • Village Ngono : Autonomie financière, Epargne –Formation d’un groupement • Village Fombory. Renforcement du climat social (25 membres composés de 2 hommes et 23 femmes). Le SILC est fonctionnel et le groupement disposait des épargnes et des caisses mais actuellement les activités de SILC ont été arrêtées en Mais 2013 après le partage de l’argent entre les membres. Nous avons appris beaucoup de choses pendant le SILC. • Village de Kiro : le groupement est fonctionne et création a entrainé une cohésion sociale entre les membres a travers la solidarité et l’entraide entre les membres du groupement. Notre revenu a augmenté nous permettant de subvenir a certains besoins primaires de nos familles sans demander aux maris. Nous utilisons également des légumes provenant du maraichage dans nos sauces. Quelles dispositions avez-vous prises pour continuer les activités de SILC après le retrait du projet ? • Gono, Petaka : Aucune disposition n’a été prise parce quelle attendait le retour du projet pour reprendre les activités SILC . • Village Fombory : Réplication du SILC dans le village. SILC a été mis en place par une animatrice villageoise. Ce groupement est un groupement SILC de deuxième génération. Au total deux groupement ont été mis en place par le projet et deux autres ont été mis en par le relais communautaire. • Village Kiro : Le SILC (travail de bilan Koro) continue chez nous même après le retrait du projet. Notre dernière réunion est très récente et elle date du 14 Novembre 2013. Cela est consigné dans notre cahier qu’on vous a montré. • Ibissa : le SILC est fonctionnel, la dernière réunion remonte le 11 novembre 2013 Quels sont les perspectives majeures du groupement depuis le retrait du projet Village Gono, Petaka • Apprendre la teinture ; • Apprendre la couture et vendre les habits dans le village. • Aider le village sur le plan de la Sante Village Kiro : • Continuer les activités SILC parce que nous avons compris les avantages à travers l’augmentation de nos revenus. Quelles suggestions ou recommandations faites-vous dans le cadre du suivi et renforcement des groupements du SILC ? • Appuyer pour les AGR (activités d’embouche, Machine a coudre, Moulin), Apprentissage • Ajouter les thèmes de l’embouche sur la formation des membres de SILC (Fambory) • Appuyer a la construction d’un CSCOM dans le village pour alléger la pénibilité chez les femmes. • Reprendre les formations sur la gestion des activités SILC en faveur de notre groupement. Ce qui permettra de renforcer d’avantage nos compétences dans la gestion des activités SILC • Eviter la période des travaux champêtres pour la mise en place des groupements SILC Nema Final Qualitative Evaluation. Chapter 2. SO1. December 30, 2013. Draft. 28 Village Dalla : • Halimatou Dicko, une femme âgée de 40 ans, mère de 6 enfants. Cette habitante du village de Dallah situé dans le cercle de Douentza est une ménagère qui a suivi des cours d’Alphabétisation grâce a l’intervention du projet Nema dans leur village. Halimatou est la présidente du groupe SILC « Waalde Jam» qui signifie Association pour la Paix. Selon elle, avant la formation de leur groupement, elle était une mère de famille analphabète qui élevait ses enfants avec beaucoup de difficultés financières pour joindre les deux bouts. Elle a déclaré « J’étais obligée de demander de l’argent à mon mari très souvent et cela entrainait parfois des mésententes et des épisodes de conflits conjugaux. Si mon mari ne donnait pas d’argent, j’étais obligée de sillonner tout le village à la recherche crédits ou prêts » sans satisfaction. Selon, elle le SILC a créé la solidarité entre les femmes du village. Elle était devenue autonome faisant face à ses besoins élémentaires et ceux de ses enfants sans demander très souvent à son Mari. Au sein du groupement SILC, elle a également bénéficié des formations sur l’hygiène et l’alimentation des enfants. Elle déclare, « en plus des formations reçues, j’ai également suivi des cours d’alphabétisation, c’est ce qui a surtout ouvert mon esprit ». elle a maintenant un esprit ouvert, elle appris a épargner et est devenue plus autonome pour résoudre ses problèmes quotidiens. Pour elle, grâce au cours d’alphabétisation fonctionnelle, elle arrive à lire et écrire son nom ainsi que le nom de tous les membres du groupement. Pour montrer la preuve a l’équipe, après le focus group, elle a écrit le nom de toutes les femmes ayant participé aux discussions de groupe. En fin, Mme Halimatou, aspire à augmenter ces connaissances et entreprendre d’autres activités génératrices de revenu. Elle sollicite au près de Projet, de reprendre les cours d’Alphabétisation fonctionnelle pour les femmes de leur village. Village Kiro • Je m’appelle Noumoudjou Maiga dite Gako, j’ai 33 ans et mon numéro de téléphone est de 70585592. J’ai pris un crédit de 10000 FCFA dans notre caisse SILC pour faire de l’embouche en payant un bouc il y a 6 mois de cela que je compte revendre dans un mois pour rembourser mon crédit avec un intérêt de 5 pourcent Source: SILC Focus Groups, Nema Final Evaluation Survey, November 2013. • Gender Roles and Relations: The tight linkage between the SILCs and the AEGs supported by the program seems to have two unintended but very important consequences for women. First, it seems to have strengthened the community-level understanding and willingness to support the women developing this new set of semi￾autonomous income generating activities (IGAs). Second, it seems to have protected their access (in most cases) to the land needed to engage in these activities. This seems to have helped the SILC beneficiaries who got involved in agricultural AEGs to avoid the common problem that many such initiatives have experienced throughout Mali and other areas of the Sahel, which is that a powerful male leader or men’s group will often reclaim land allocated to women’s groups (for irrigation, demonstration plots) once these activities are shown to be profitable and/or when the donor funding ends. • Limited Scale Up Since the Program Ended: The SILC model used by Nema helped develop of a strong network of trainers. Although many of these trainers were actively engaged in replicating SILCs in other villages during the program, they have completely stopped replicating since the program ended. Given the security situation and the limited connection between these groups and various other area actors (like the Ministry for the Promotion of Women and the INSORTMIL/Purdue/McKnight Initiative), this is understandable. One critical challenge for the next phase will be to facilitate the existing base of AVs into private service providers (PSPs) to help strengthen the linkages between the SILCs and the existing base of micro-finance services within the region; and to help to help transform certain SILCs into AEGs, and ultimately into cooperatives. • Limited Number of Villages Where SILCs Were Developed: To date, only 57% of the Nema villages in Bourem and 58% of the Nema villages in Douentza have SILCs (Tables 2.3 and 2.4 above). The ones that have SILCs often have several due to the huge demand Nema Final Qualitative Evaluation. Chapter 2. SO1. December 30, 2013. Draft. 29 for them and the easy access these communities have to the Nema-trained AVs. A major challenge for the next phase will be to develop various networks of SILCs in all of the Nema-assisted communities. 3.3. Farmer Field Schools 3.3.1. Current Level of Functioning of the FFS-Supported Activities: The FFS demonstration plots covered the majority of the Douentza villages (90%), but only 50% of the Bourem villages (Tables 2.2. and 2.4 above). When villages did receive the FFS, they were much appreciated by the local people. Every single AEG and SILC focus group that was interviewed listed the program’s efforts to promote new technologies through the FFS as one of the principal program impacts. Sixty-seven % of the functional AEGs at Bourem reported that they continue to use the new production technologies that they learned during Nema; the figure was 100% for the functional AEGs interviewed in Douentza (Table 2.6 above). Although the focus group discussion for the SILCs did not ask any specific questions about their interest in the types of technology being promoted at the FFSs, it is clear from the open-ended discussions that many of the members benefitted from these activities and appreciated them. The chief reason the AEG focus groups stated for no longer practicing the new technologies were: • Limited access to the seeds and other inputs—fertilizer (DAP27 and Urea28), pesticides for intensive cowpea (nebie) production, and PIC29 sacks for nebie conservation; and • Limited access to markets because of high levels of insecurity along the major roads and more limited market activity in general. Despite farmers’ complaints during the focus groups about the current difficulty they are having continuing the new farming practices, there is ample evidence from the key person interviews with the Ministry of Agriculture, Caritas, and SCI staff that: • Some (if not all) of the AEG and SILC groups in Bourem have maintained their pre￾existing levels of crop activity (especially irrigated gardening by women’s groups); and • Several of the AEGs in Douentza had successful harvests this year for both intensive nebie and millet. One of the best indirect qualitative indicators showing that some of the AEGs and SILCs continue to practice some of the improved intensive technologies that Nema promoted is the number of calls30 from AEG representatives to the Consortium’s national NGO partner Caritas asking for assistance with marketing, purchasing sacks, and seeds. Another indirect indicator of success is the growing number of commercial seed producers in the former Nema villages both during the occupation and today. 31 There is also a great deal of anecdotal evidence from the 27 DAP is 18% nitrogen-46% phospherous-0% potassium. 28 Urea is 44-46% nitrogen-0% phosphorous-0% potassium. 29 PIC (Purdue Insect Control) sacks are triple plastic sacks that keep the air out of stored nebie, which kills any insects that might come in from the field. 30 Caritas maintains a master lists of calls to their main offices from AEG and SILC representatives. The frequent calls from AEGs and SILCs to the current and former Nema extension agents have not been tracked. 31 Six of the Nema facilitated AEGs were reported to be moving into commercial seed production in the Douentza. Nema Final Qualitative Evaluation. Chapter 2. SO1. December 30, 2013. Draft. 30 qualitative survey, as well as the interviews with various Ministry of Agriculture staff in Bourem and Douentza, that at least some of the Nema-trained farmers and groups have joined pre￾existing cooperatives and are getting seed and market access through these group; this observation was confirmed during the focus group discussions. 3.3.2. Summary: Assessment of Strengths, Areas that Need Strengthening, and Opportunities in the Current Context: 32 • Organizational Capacity Building and Systems for Monitoring This Capacity: One of the major institutional impacts of the FFS component of Nema was the training of 29 FFS TOT trainers (Table 2.2). Based on interviews with SCI and Caritas staff, it appears that almost all of the TOTs are living in the villages and that they continue to be important go-to people for agricultural production and marketing information. One of the major institutional challenges during the next phase of the program will be to: (a) strengthen the direct connections between the FFS TOTs and the main private-sector and public-sector technology providers (i.e. IER and the Ministry of Agriculture); and (b) help each of the TOT trainers that are interested to become private-sector seed producers. Given the high demand for seed production, the profitability of private-sector seed production in both Bourem and Douentza, and IER’s willingness to backstop seed certification, this is a key activity that will help strengthen Nema’s current and programed impact on private-sector seed production. • Gender Roles and Relations: Nema’s record for gender inclusion in the FFSs is very good (30% of the direct beneficiaries for Douentza and 40% for Bourem were women [Table 2.4]). The modes by which this gender inclusion occurred (i.e. high involvement of the AEGs and SILCs in selecting the trainers, as well as the sites and communication channels used to publicize the trials) should probably be studied in order to identify a number of best practices that could be scaled up through future Title II and non-Title II programs in Mali. • Limited Access of Most Farmers to Complementary Inputs: The principal reason cited by most farmers for not practicing the new technologies that were pilot tested at the FFSs was the farmers’ limited access to improved inputs. One output of the Consortium’s collaboration with IER has been a series of publications showing that in the absence of complementary inputs (fertilizer, pesticides, PIC storage sacks for nebie), the farmers are unlikely to realize the full-yield potential of the improved seed, which makes it difficult to justify the cash costs of purchasing new seed or the additional labor needed to increase their use of organic fertilizer or the recommended package of crop-production techniques. 33 One major challenge for the next phase of any interventions in the Nema Caritas was able to verify the production figures for two of these groups. Villages Seed Production (kg) 2011-2012 2012-2013 2013-2014 Tabi (Millet) 3,000 4,600 4,500 Boubam (Milletl) 1,800 2,000 0 32 The results and effects, the effectiveness of the implementation process, the sustainability of the results, and the activities impact on the local people’s resilience. 33 Aliou Traore, Moussa Keita, Fatoumata Ba, Seydou Coulibaly, Seydou Sanogo. 2012. Rapport de Recherche CRS/CRRA de MOPTI. Campagne 2010-2011. Mopti: Centre Régional de Recherche Agronomique de Mopti. Décembre 2012. Nema Final Qualitative Evaluation. Chapter 2. SO1. December 30, 2013. Draft. 31 villages will be to strengthen sustainable access to improved inputs through cooperative and expanding the current base of private-sector seed suppliers. • Collaborating Partners: During Years 1-4, the Nema program maintained tight linkages with IER for the identification and supervision of the demonstration plots. This collaboration helped the program access a wide range of promising technologies. The same collaboration model can open new doors to potential partners like the Ministry of Agriculture Partnership with INTSORTMIL/Purdue and Initiative Eau et Sécurité Alimentaire pour l’Afrique (IESA) that are trying to strengthen the local cooperatives’ access to the input and output markets that they need to sustain productivity increases. 3.4. Food for Work 3.4.1. Current Level of Functioning of the FFW-Supported Agro-Sylvo-Pastoral Infrastructure: A total of 30 of the 143 agro-sylvo-pastoral infrastructures created by the program during Years 1-4 to help local communities build agro-enterprise development and better manage risk were visited during the final qualitative survey (Table 2.10). This survey showed that 85% of the completed infrastructures were still functional, though the level of maintenance was lower than desired in most cases. Only one of the infrastructures visited—a fully operational perimetre marachiere (irrigated vegetable garden) in the village of Oualo—was completely non-functional (locked up) for more than one year. The reasons for the non-functioning of this infrastructure are not fully understood, but appear to be related to internal problems within the AEG that was in charge of it. Table 2.10. Current Functionality of Nema-Created Infrastructure Visited During the Final Evaluation* Infrastructure Bourem Douentza Total Assessed as Functional (%) Drinking water infrastructure 1 1 2 50.0 Agro-sylvo-pastoral and gardening infrastructure 6 14 20 85.7 Sanitation infrastructure 6 6 83.3 Other infrastructures (bridges, rural roads) 0 2 2 100 Total Visited 7 23 30 N/A Source: Nema Final Evaluation Survey, November 2013. *n=Number of infrastructures visited during the final qualitative survey and evaluation. 3.4.2. Summary Assessment of Strengths, Areas that Need Strengthening, and Opportunities in the Current Context: 34 • Sustainability and Resilience: The FFW component of the SO1 Nema strategy helped the program address some of the major impediments to more profitable agro-enterprise development in the majority of the Nema villages. The same activities have helped certain communities’ capacity to manage routine risk like sand and water erosion and to Moussa Keita. 2012. Rapport de champagne pour la conduite des Champs Ecoles Paysans sur l’échalote et la tomate dans les villages de Adia, Debere, Mougui, Koiraberi et Koundiou (Cercle de Douentza). Mopti: Centre Regional de Recherche Agronomique de Mopti. Decembre 2012. 34 The results and effects, the effectiveness of the implementation process, the sustainability of the results, and the activities impact on the local people’s resilience. Nema Final Qualitative Evaluation. Chapter 2. SO1. December 30, 2013. Draft. 32 protect them from periodic flooding. To date, however, the average number of FFW per village in the Bourem region has been less than in Douentza. • Vulnerability: The successful implementation of these FFW infrastructure programs, combined with solid AEGs and SILCs (i.e. the full Nema technical package), seems to be one of the principal reasons that three of the Bourem villages (10% of the total) and 24 of the Douentza villages (24% of the total) were assessed by staff as ‘relatively food secure/not very vulnerable’ after four years (Table 2.5 above).35 One of the principal recommendations from the Caritas, SCI, and CRS staff is for future programs to intensify their use of FFW to develop the types of agro-sylvo-pastoral infrastructure that the more isolated villages will need to reduce the high levels of food insecurity and vulnerability to routine risk and major crises. 3.5. Safety Nets 3.5.1. Current Level of Functioning of the Safety Net Committees in the Nema Villages: The Nema-sponsored safety net programs helped build the capacity of the local communities to better target emergency food assistance and safety nets. Prior to the program, any food aid was simply distributed to the village chief, which meant that it seldom reached the poorest groups. One of the most important consequences of this community-level capacity building seems to have been to help attract and manage the two major regional-level humanitarian relief programs that the team was able to document (through the key informant interviews with the Ministry of Agriculture, Caritas, SCI, and the Red Cross/Mali) for Douentza and Bourem (Table 2.11), as well as some of the more community-specific smaller, short-term food assistance programs aimed at helping internally displaced persons (IDPs). A major challenge for the next phase will be to better understand how this training helped better engage and manage this assistance. The short-term impact of the successful implementation of these post-Nema safety net programs appears to have been: • Improving the aggregate food security (MAHFP) and dietary diversity scores (DDS) for about 5% of the most vulnerable households in each community; and • Helping the community avoid some of the conflicts that had accompanied food assistance in these villages during earlier crises like the 2004 cricket infestation. Another unintended consequence of this assistance that was observed in some of the villages during the final survey was strengthening the commitment of the community leaders (the chief and counselors) to reflect on: (a) better targeting the traditional tithing practices in the villages to very vulnerable groups; and (b) better ways to supplement this traditional support by connecting it to some of the local NGOs that foreign migrant communities in Bamako and France have developed in order to help their home communities.36 35 Based on the end-of-program classification of aggregate vulnerability status (see Table 2.4) and staff assessments during the baseline, almost all of the Nema villages were in the very vulnerable or highly vulnerable category in 2009, which was the end of the one year USAID FFP one-year emergency program. 36 One of the communities interviewed during the final evaluation survey has already identified ways of using funds coming from expatriate resortissants to continue strengthening its support through the Nema-created safety-net groups to help the most vulnerable members of the community. Nema Final Qualitative Evaluation. Chapter 2. SO1. December 30, 2013. Draft. 33 Table 2.11. Evidence of Post-Nema Food Distributions in the Nema Villages, September 2012-present Humanitarian Agency Distributing Food Aid Percentage of the Population Targeted by the Distribution (i.e. % of Most Vulnerable Households) # of Villages % of Nema Villages Covered by Distribution Douentza (100 Villages Nema) January-February 2013 IFRC 13% (during 2 months) 100 100% March-October 2013 WFP 33% (during 6 months) 24 IFRC 31% (during 6 months) 76 Bourem (30 Villages Nema) OXFAM/WFP (July-Dec 2013) 30-40% (during 6 months) 27 90% IFRC (Sept 2013) 100% (during 1 month) 30 100% Vivres ADESAH (Sept-Nov 2012; Sept-Nov 2013) 50% (during 3 months) 3 10% Source: Key informant interviews, Ministry of Agriculture staff, Nema Final Evaluation, November 2013. Acronyms: IFRC: International Federation of the Red Cross; WFP: World Food Programme 3.5.2. Summary Assessment of Strengths, Areas that Need Strengthening, and Opportunities in the Current Context: 37 • Capacity Building: Based on this qualitative assessment, it appears that the training given to the safety net committees (comites de filet de securite) was far more effective in catalyzing a safety net emergency response after the Nema closing than any information transmitted “up the food chain” to the cercle-level early warning systems (SAPs or systemes d’alerte precoce). In sum, the foods aid response after the program ended is best described as being “supply driven” not “demand driven” –i.e. villages that were well organized and well trained were more likely to benefit from the specific IDP and general food allocations than those that were not. • Slow Emergency Response: Although the community’s safety net committees were able to manage an emergency response, the first distributions did not occur for eight months after Nema closed (January 2013) (Table 2.11). One of the major challenges of any future program should be to anticipate ways that the local communities can accelerate the speed needed to mobilize a major response to a major crisis. • Lack of Pre-Negotiated Connection to the Humanitarian Donors Capable of Mobilizing a Major Emergency Response to a Major Civil-War Related Crisis: Although this is an area that the evaluation team was not able to study in detail, it appears that none of the responses generated to the crisis were the result of a re-negotiated memorandum of understanding between either the SAPs or the villages. Instead, they were responses launched by major donors (International Federation of the Red Cross [IFRC] and the World Food Programme [WFP]) to an emergency situation, and the fact that the villages were organized helped them benefit more quickly and efficiently. A major challenge in the next phase will be to identify better ways that the local communities can work more 37 The results and effects, the effectiveness of the implementation process, the sustainability of the results, and the activities impact on the local people’s resilience. Nema Final Qualitative Evaluation. Chapter 2. SO1. December 30, 2013. Draft. 34 effectively with the principal humanitarian donors to pre-negotiate some sort of pre-crisis memorandum of understanding just in case a major crisis does occur. 4.0. Global Impact of the SO1 Activities on Household Food Security and Vulnerability 4.1. On Average Household Food Security and Resilience The evaluate team used a qualitative “food security calendar” to measure the impact of Nema on the MAHFP. This calendar, also referred to in the literature as the MAHFP qualitative tool, was used extensively by Title II NGOs prior to the new FANTA requirement that programs use the quantitative measure that the Nema program used during its baseline survey (Impact Indicator 1.1). The fact that the baseline measure from the qualitative MAHFP (3.8 months) is almost exactly as the same from the baseline Nema quantitative survey (4.11 months) suggests that the tools are very complementary. This analysis showed (Figure 2.3) a major increase in the average MAHFP of the households in comparison with the baseline. This substantial increase at Douentza seems to be related to the large amount of FFW and safety net food that was distributed in the villages that were studied during the final survey, as well as a slight increase in technology adoption and the use of improved inputs that was Nema-facilitated. The much lower increase in household food security in the Bourem villages seems to be related to their chronic food insecurity, the much lower rate of technology uptake by the communities (from the FFSs, AEGs, and SILCs), as well as the lower amounts of FFW implemented during the program (Figure 2.3). The same analysis showed that the average MAHFP is currently at about the same level as at the Nema baseline. This is not surprising given the multiple shocks and crises that the Nema communities have endured over the last two years (Table 2.12). It is clear that this figure would have been far lower without the small amounts of food assistance that came in post-Nema and the beneficiaries’ use of some of the improved crop technologies and income earned from IGAs to purchase food. Nema Final Qualitative Evaluation. Chapter 2. SO1. December 30, 2013. Draft. 35 Figure 2.3. Nema Results and Effects in the Douentza Villages: Average Household Food Security (MAHFP) Before, During, and After the Program Methodology and Source: Based on food security calendar Participatory Rural Analyses (PRAs) during the community focus groups conducted as part the Nema Final Qualitative Survey, November 2013. 2.0 5.6 3.8 2.6 7.4 5.0 2.0 4.4 3.2 0.0 1.0 2.0 3.0 4.0 5.0 6.0 7.0 8.0 Bourem (n=6 villages) Douentza (n=14 villages) All (n=20 villages) Before NEMA During NEMA Today(Ater NEMA) Nema Final Qualitative Evaluation. Chapter 2. SO1. December 30, 2013. Draft. 36 Table 2.12. Evolution of the 2012 Food Crisis and Nema and Non-Nema Responses and Impacts Risks/Shocks Impact Nema Communities Douentza Bourem Bad harvest MAHFP very low in August 2011 Yes Yes Nema safety net program (104/130 of the most vulnerable villages for 2 months) in January 2012 Short-term increase of MAHFP in 104 Nema villages for 3 months Yes Yes Nema prepared an emergency response program for 104/130 most vulnerable Nema villages that was only partially executed (see Chapter 4) MAHFP very low March 2013 to January 2013 (9 months) Yes Yes Emergency humanitarian assistance that targets the most vulnerable households (15-30%) starts to arrive in September 2012 (See Table 2.10) Slight increase in MAHFP, but only for the most vulnerable households Yes Yes Good rainfall but weak access to inputs (seeds and fertilizer) Despite good rainfall, most farmers do not produce much (low MAHFP and low revenues) Yes Yes Many regional activities are closed or diminished in activity Yes Yes Sharia law in some areas makes it difficult to organize group meetings (for the AEGs, SILCs, and cooperatives) Yes Yes Certain irrigated perimeters and gardens non-functioning due to poor maintenance Yes Yes Sharia law forbids smoking and tobacco production 17 riverside villages in Bourem with limited access to fields suitable for sorghum, millet, or nebie lost their principal source of revenue for purchasing food. Very negative impact on household food security n/a 17/30 Bourem commun ities IDPs return Increased demands on the limited household food stocks Yes Yes Source: Final Nema Qualitative Survey, November 2013. 4.2. On Vulnerable Households Using the same MAHFP data profile, the survey team was able to develop a qualitative profile of the local people’s assessment of Nema’s impact on vulnerable households. 4.2.1. Douentza: This analysis shows (Figure 2.4): • The Nema program decreased the number of households classified as highly food insecurity based on the average MAHFP from 50% of the households in the community to 31% by the end of the program; but • The current level of vulnerable households is approximately the same as at the baseline (53% vs. 50% at baseline). The fact that the number of vulnerable households did not increase dramatically suggests that the villages’ capacity to manage the shock made them more resilient than in previous crises (such as the 2004 cricket crisis, which required many years of sequential CRS-facilitated emergency initiatives in Douentza to get back to the level they had been before the crisis). Nema Final Qualitative Evaluation. Chapter 2. SO1. December 30, 2013. Draft. 37 Figure 2.4. Nema Results and Effects in the Douentza Villages: Household Vulnerability Levels Before, During, and After the Program Methodology and Source: Based on food security calendar PRAs during the community focus groups conducted as part the Nema Final Qualitative Survey, November 2013. 4.2.2. Bourem: In contrast to Douentza, the percentage of households classified as highly vulnerable did not show any major change since the program started (Figure 2.5). 30% 39% 29% 50% 31% 53% 0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100% Before NEMA During NEMA Today (After NEMA) "Very Vulnerable (high food insecurity)" "Average Vulnerability" "Less Vulnerable (More food secure)" Nema Final Qualitative Evaluation. Chapter 2. SO1. December 30, 2013. Draft. 38 Figure 2.5. Nema Results and Effects in the Bourem Villages: Household Vulnerability Levels Before, During, and After the Program Methodology and Source: Based on food security calendar PRAs during the community focus groups conducted as part the Nema Final Qualitative Survey, November 2013. 4.3. Effectiveness of the SO1 Monitoring and Evaluation Systems 4.3.1. Strengths of the System in Reporting SO1 Results and Effects: The program developed rigorous systems for tracking activities, outputs, and impact. From the beginning the program used the Simple Measurement of Indicators for Learning and Evaluation-Based Reporting (SMILER) system to identify which types of data needed to be tracked to measure the official IPTT indicators as well as the various internal monitoring indicators used for program planning and assessment. One innovative ‘best practice’ in the first year was the team’s decision to integrate all of these different monitoring systems into a single data base. Once the data base was created, it acted as a standard ‘masque’ that all of the partners could use to collect certain types of standard information in ways that would feed directly into the main tracking indicators. Once the base was created, every one of the senior partner staff members and M&E staff were trained in how to use. This integrated data base made it possible for the evaluate team to monitor long￾term trend data on a number of key internal and official indicators. It also facilitated real-time analyses of the field data for planning and reporting, as well as the transmission of some of the data being collected by the ADCs and community groups (SILCs, AEGs, GAP/RU, safety net committees) to the CRS M&E agent charged with maintaining the central data system. 13% 13% 13% 40% 40% 40% 47% 47% 47% 0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100% Before NEMA During NEMA Today (After NEMA) "Very Vulnerable (high food insecurity)" "Average Vulnerability" "Less Vulnerable (More food secure)" Nema Final Qualitative Evaluation. Chapter 2. SO1. December 30, 2013. Draft. 39 Another major strength of the program is that all of the M&E officers are well trained and imbued with a strong sense of ethics and responsibility. Their unwillingness to be ‘flexible with’ (i.e. fudge) the data was mentioned by several persons during the key informant interviews as both a strength and a source of frustration. 4.3.2. Areas That Could be Strengthened in Order to Better Track Program Results and Effects: Although the M&E systems were rigorous, certain elements of the IPTT tracking system made it hard to appreciate the program’s significant achievements in a number of areas that are critical to resilience, food security, and sustainability. 4.3.2.1. Limited Disaggregation of the Tracking Data by Intervention Zone: One of the unintended consequences of presenting an aggregate analysis of the main monitoring indicators in the IPTT for the two intervention areas was to hide some very important differences between the level of activity and impact in the two areas for certain groups of activities. The program’s tendency to aggregate all of the planning and reporting data for both zones had several downstream consequences, which included: • Downplaying certain problems that the partners needed to discuss, like the fact that the technology package being promoted was far more successful at Douentza than in the more arid agro-pastoral communities at Bourem; • Decreasing the utility of the IPTT as a partner planning and coordination tool; and • Making it somewhat time consuming (but not impossible) to go back and disaggregate Year 2-4 data retroactively. Two very useful lessons and recommendations that the Nema team is already using in its new area of intervention is for future programs: • Anticipate the need to disaggregate all of the data for the principal internal and donor￾driven indicators from the start; and • Train all execution partners (M&E officers and program managers) in the methodologies needed to conduct disaggregated IPTT analyses for their zone of information. This same type of disaggregated analysis of the indicators should make it easier to build the local partners’ (both NGO and government agencies) capacity to collect and analyze certain types of data and to provide M&E backup to the partners if this data cannot be collected through normal channels due to a shift in security or other constraint. This is a lesson learned that the Consortium has already set in motion in the new zone. 4.3.2.2. Limited Tracking and Reporting of Program Impacts on Community and Household Level Vulnerability and Resilience: Three of the unintended consequences of not tracking the program’s impact on vulnerable groups and vulnerable households (except for safety nets) were: • Masking the Nema program’s considerable impact on vulnerable villages, vulnerable groups, and resilience; • Making it difficult to target appropriate SO1 assistance (FFSs, AEGs, SILCs, FFW) to vulnerable villages that were less suitable for the main package of innovations being promoted; and • Making it difficult to design special FFW programs that could work with individual households to build “agro-sylvo-pastoral infrastructure” that would reduce their Nema Final Qualitative Evaluation. Chapter 2. SO1. December 30, 2013. Draft. 40 vulnerability since there was no systematic way (other than the safety net system) to track the impact of these initiatives. Several Title II programs in West and East Africa have used this type of individual FFW to reduce vulnerable groups’ dependence on food assistance. Three options for strengthening the program’s capacity to track vulnerability in ways that build on the existing tracking systems might include: • Adding a new indicator to the SO1 Impact Indicators that measures percentage of households classified as chronically food insecure. This indicator could be calculated from the same information used to calculate the average MAHFP indicator the Consortium tracked during Years 1-4. • Adding a new indicator measuring the percentage of villages classified as chronically food insecure to the tracking table, and developing targets for reducing the number of villages classified as chronically food insecure and/or vulnerable. This indicator could be calculated several ways. One option would be to develop an index that would be calculated from the existing program data. Since the program is used to this, it would be an internal M&E exercise that would require very little additional labor that could be pilot tested in the process of developing the exit plans for the new area where Nema is currently working. Another option would be to consider using the tool that SCI has developed for measuring community-level vulnerability in some of its other programs. • Adding a qualitative measurement of the MAHFP using the ‘food security calendar’ methodology used during the final evaluation. Based on the experience of several other Title II programs in Africa, various local community groups could be trained to use and report on as part of the incipient community-based M&E system that is already under development. This qualitative indicator could even be included in the IPTT as a backup impact indicator that could be used to track program impact, as was done in the final evaluation, should the program not be able to conduct a quantitative final survey. 4.3.2.3. Extensive Tracking but Limited Reporting of Programmatic Impacts on Local Community Capacity: A major strength of the initial and current phase of the Nema program— which is rare in Title II programming—was the Consortium’s commitment to the consistent use of three internal tools for tracking this capacity: one for the SILCs (which is used by CRS throughout the sub-region), one for the AEGs and one for the GAP/RUs (under SO3). Staff stated that the tools helped them track capacity and better target assistance. To date, however, the only one of these indicators being tracked in the IPTT is the one for the GAP/RUs (under SO3). The downstream consequences of reporting on this data internally rather than as part of the official M&E system has been: • Making it difficult to track the significant impact of the Nema program on local community capacity during the first four years of the program; and • Making it difficult to identify and discuss the key challenges these groups were likely to face and how to address them in an exit plan. One of the key challenges for the next phase will be to build on these initial tools to develop SO1-specific tools for capacity building for the SILCs and AEGs that can be more easily tracked in the IPTT and reports. This type of tracking is critical to the development of effective exit plans and long-term sustainability. Two options for addressing this include: Nema Final Qualitative Evaluation. Chapter 2. SO1. December 30, 2013. Draft. 41 • Develop a more consistent system for tracking and reporting on local organizational capacity in the main program IPTT that builds on the existing AEG and SILC tools that are already being used to track capacity; and • Move in the direction of turning the tool into self-assessment tools (that builds on existing tools being used by the AEG and SILC officers to track their programs) that the Consortium can use to build local communities’ capacity to develop realistic exit and sustainability strategies. Given the fact that the Consortium has already been tracking the capacity of the SILCs and AEGs since the beginning of the program, it is probably wise to build these activities on the existing tools rather than develop a new one. 5.0. Lessons Learned and Recommendations Based on the assessment of the data from the final evaluation survey, the consultants and field staff who participated in the initial debriefing at San on November 30, 2013 identified six major recommendations for the next phase, as well as a list of specific recommendations. The consultants synthesized this list with their initial analyses and presented a revised list to the working group members on December 3, 2013 for their review. Based on this participatory review, the team reached agreement on the following seven recommendations. Recommendation 1.1. Consolidate the investments made by the Nema program in building agricultural production and increasing revenues in the communities where the SO1 activities were most active during the initial phase (Years 1-4). Specific Recommendations: 1. AEGs: Assess the current level of functioning activities of each AEG, including the profitability of the original activities it has engaged in, and: - Build the core institutional capacity they need to become registered cooperatives and/or to join and established cooperatives; - Identify any technical assistance they might need (such as increased access to improved food-processing technology) to further sustain their existing agro￾economic enterprises or any new ones they have developed; and - Strengthen the program’s collaboration with the local technical services and donor-funded initiatives (e.g. INSORMIL/Purdue and IESA) that support the new intensive technology package in order to facilitate their support for the scale up of the new technologies pilot tested during the FFSs for the Nema target villages. 2. SILCs: Conduct a similar sort of institutional review of the SILCs in the initial pre￾planning phase of any sort of follow-up programming. 3. Track the impact of this activity on the aggregate vulnerability of the communities as a basis for planning a program exit strategy (see Recommendation 1.7). Recommendation 1.2. Extend the total package of SO1 innovations into the Nema communities where the program was less active in the first phase in ways that build on the capacities of the local agro-ecological capacity. Nema Final Qualitative Evaluation. Chapter 2. SO1. December 30, 2013. Draft. 42 AEG: Since the initial AEG methodology focused on reinforcing existing initiatives, many of the most vulnerable villages were not eligible for participation. These are often vulnerable villages without ready access to markets and/or those focused on agro-pastoral enterprises. These are villages that were often under-served by other programs with few if any pre-existing groupements or cooperative activities to build on. Specific Recommendations: 1. Target underserved, vulnerable villages with an intensified package of FFW activities to build their capacity for the agro-enterprises that they can develop (especially livestock, commercial forage production, and irrigated gardening). 2. Encourage commercial fodder production. 3. Develop a new group of safety net programs that focuses on the use of individualized FFW that vulnerable households can execute to help them develop small micro￾enterprises (like poultry production, stall feeding, manure pits, and kitchen gardens). 4. Build the core capacity of the team to support commercially oriented, sustainable livestock activities by recruiting an senior animal scientist (zoo-technician) who can provide appropriate technical assistance and intensified linkages to the emerging livestock initiatives being developed by the Mali Ministry of Livestock and international partners like the International Livestock Research Initiatives (ILRI), as well as some of the new livestock initiatives being supported by USAID in Burkina Faso and Niger. 5. To insure appropriate conclusion of women and vulnerable groups, a progressive scale up of the SILCs should follow the progressive scale up of the AEGs in the most vulnerable communities. 6. Track the impact of this activity on the aggregate vulnerability of the communities as a basis for planning a program exit strategy (see Recommendation 1.7). Recommendation 1.3. Strengthen the level of integration and joint planning between SO1 and SO2. During the first four years, these collaborative activities focused on: • The promotion of various high nutrient crops (like the orange fleshy sweet potato); and • Training and equipping a small group of SILC members (created under SO1) in the basic principles of health and nutrition that the program was promoting under SO2. To date, there has been no direct linkage between the PD/Hearth component of the program and the SILCs, AEGs, or FFSs as anticipated in the proposal. Given the critical importance of this linkage, the SO2 team has identified a number of concrete recommendations for how these linkages could be developed and strengthened in ways that would help sustain any future PD/Hearth initiatives. Specific Recommendations (cross-cutting with SO2): 1. Encourage the mothers of children graduating from the SO2-supported PD/Hearth to organize themselves into mothers groups that are SILCs. 2. Provide the mothers’ group with technical assistance and support to help them increase their crop production and revenues. Nema Final Qualitative Evaluation. Chapter 2. SO1. December 30, 2013. Draft. 43 3. Given the difficulty that all of the PD/Hearths have had in providing the basic ingredients and fuel wood needed to support the PD/Hearth in Bourem, link future FFW investment in the development of irrigated market gardens (culture maraichages)—one of the most potentially profitable value chains in Bourem—to the group’s commitment to: (a) providing a certain percentage of their harvest to the PD/Hearth in that community; and (b) planting a haie vive (around the plot) that the women’s group can maintain as a source of fuel wood for the future PD/Hearth. 4. Track this increased capacity through a new self-assessment tool and report on it regularly to strengthen the preparation of any new program’s exit strategy (see Recommendation 1.7). Recommendation 1.4. Transform and certify the FFS trainers and help them become commercial seed producers as a way to strengthen their role as community-based extension agents for both the Nema-founded extension groups as well as for other pre-existing community groups. Specific Recommendations: 1. Retrain current and train new TOTs in basic skills, as well as the skills needed to become certified seed producers through the IER program designed to promote this. 2. Provide technical assistance and monitoring to facilitate the TOTs becoming seed private-service providers (PSPs). 3. Track this increased capacity through a new self-assessment tool and report on it regularly to strengthen the preparation of the exit strategy (see Recommendation 1.7). Recommendation 1.5. Strengthen the organizational capacity of the local service providers that are essential to the successful achievement of the activities. Specific Recommendations (cross-cutting with literacy and capacity building): 1. Both categories of AEG and SILC capacity building (listed under Recommendations 1 and 2) would benefit from literacy training programs that build the capacity of the members with basic reading and writing skills (see Tables 2.5 and 2.5) to manage the basic AEG and SILC planning and financial tools. Specific Recommendations (SILC): 2. Certify the SILC AVs in PSP. 3. Finalize the process of training the AV in PSP and help them to create a network (reseau). 4. Better connect the SILC groups with the existing base of institutions of micro-finance (IMFs). 5. Facilitate the transformation of certain groups (as they develop) into AEGs and ultimately, when they are ready, into cooperatives. 6. Track this increased capacity through a new self-assessment tool and report on it regularly to strengthen the preparation of the exit strategy (see Recommendation 1.7). Recommendation 1.6. Require all future AEGs to be organized into cooperatives before co￾financing their business plans based on measured levels of performance. Nema Final Qualitative Evaluation. Chapter 2. SO1. December 30, 2013. Draft. 44 Specific Recommendations: 1. Strengthen the organizational capacity of the AEGs before financing their business plans. 2. To insure fiscal responsibility and allow them to approach local micro-enterprise institutions, require them to be registered cooperatives before co-financing their business plans. 3. Anticipate the need to provide the cautionnement (collateral) for the AEGs initial IMF loans. 4. Track this increased capacity through a new self-assessment tool and report on it regularly to strengthen the preparation of the exit strategy (see Recommendation 1.7). Recommendation 1.7. (cross-cutting with SO3, capacity building, and M&E) Develop new indicators based on existing tracking tools that will enable the Consortium to better monitor its impacts on community-level capacity building and vulnerable groups. Specific Recommendations: 1. Anticipate the need to disaggregate all of the data for the principal internal and donor driven indicators from the start. 2. Train all execution partners (M&E officers and program managers) in the methodologies needed to conduct disaggregated IPTT analyses for their zone of information. 3. Consider adding a new indicator to the SO1 impact indicators that measures the percentage of households classified as chronically food insecure. 4. Consider adding a new indicator measuring percentage of villages classified as chronically food insecure to the tracking table and developing targets for reducing the number of villages classified as chronically food insecure and/or vulnerable. 5. Consider adding a qualitative measurement of the MAHFP using the ‘food security calendar’ methodology used during the final evaluation. 6. Develop a more consistent system for tracking and reporting local organizational capacity building in the IPTT and reports that builds on the existing AEG and SILC tools (see Annex I, Chapter 2) that are already being used to track capacity. 7. Move in the direction of turning the tool into a self-assessment tool that builds on existing tools being used by the AEG and SILC officers to track their programs that the Consortium can use to build local communities’ capacity to develop realistic exit and sustainability strategies. Nema Final Qualitative Evaluation. Chapter 2. SO1. December 30, 2013. Draft. 45 Appendix I, Chapter 2: SILC Tool Used for Tracking Organizational Capacity BAREM - FICHE D'EVALUATION DE CHANGEMENT DE PHASE DES GROUPEMENTS SILC PHASE: SUPERVISION INTENSIVE Barème sur la fiche d'évaluation SILC Module 1: Groupes, Leadership et Elections 5 4 3 2 1 0 pourcentage du nombre des membres qui ont participés vivement dans cette activité 100 80 60 40 20 0 MINIMUM: 80% Auto-sélection 100 80 60 40 20 0 MINIMUM: 80% Rôle d'assemblée générale 100 80 60 40 20 0 MINIMUM: 80% Rôle des leaders du groupe 100 80 60 40 20 0 MINIMUM: 100% Préparations pour les élections 100 80 60 40 20 0 Module 2: Développement du Fond du Groupement toutes les procédures ont été respectées concepts et procédures d'épargne OUI -- -- -- -- NO N toutes les procédures ont été respectées concepts et procédures de crédit OUI -- -- -- -- NO N toutes les procédures ont été respectées Procédures de remboursement OUI -- -- -- -- NO N toutes les procédures ont été respectées Procédures de partage des dividendes OUI -- -- -- -- NO N Module 3: Développement des Règlements Intérieurs Rôle de chaque membre du comité et durée de son mandat Gouvernance de groupe OUI -- -- -- -- NO N Le montant à cotiser est-il bien précisé? Politique de fond social OUI -- -- -- -- NO N Le montant à cotiser est-il bien précisé? Politique de crédit OUI -- -- -- -- NO N Le montant à cotiser est-il bien précisé? Politique d'épargne OUI -- -- -- -- NO N Module 4: Tenue des Documents de Suivi Les registres sont-ils à jour? Tenu de registre: Epargnes OUI -- -- -- -- NO N Les registres sont-ils à jour? Tenu de registre: Crédit OUI -- -- -- -- NO N Les registres sont-ils à jour? Politique de fond social OUI -- -- -- -- NO N Module 5: Procédures de Réunion Pourcentages des procédures d'épargne suivies Maitrise de la procédure d'épargne 100 80 60 40 20 0 Taux des arriérés Maitrise de la procédure de crédit 0 1 2 3 4 >5 Pourcentages des procédures de fond social suivi Maitrise de la procédure de fonds social 100 80 60 40 20 0 Pourcentage de données saisies sans erreur Maitrise de la procédure de tenu des registres 100 80 60 40 20 0 Module 6: Réunion d'épargne et de crédit Pourcentage des membres qui sont présents au début et fin de la réunion 100% des membres sont présents à la séance 100 80 60 40 20 0 Pourcentage des membres qui sont a l'heure Tous les membres arrivent a l'heure 100 80 60 40 20 0 Pourcentage des procédures d'épargnes respectées les procédures d'épargnes sont suivies 0 1 2 3 4 >5 Nombre des membres de comité présents et actifs Le comité de gestion joue t- il bien son rôle? 5 4 3 2 1 0 les soldes des fonds sont correctement rappeler par le groupe Les membres du groupement sont-ils capables de rappeler exactement la situation actuelle des comptes OUI -- -- -- -- NO N Oui ou non Le secrétaire fait point financier du groupement a la fin de la séance OUI -- -- -- -- NO N Module 7: Le partage des dividendes Oui ou non Les dividendes sont partagés entre tous les membres OUI -- -- -- -- NO N Nema Final Qualitative Evaluation. Chapter 3. SO2. December 30, 2013. Draft. 46 Chapter 3 SO2: Children Under 5 Years Less Vulnerable to Illness and Malnutrition38 This chapter provides a brief overview of the results of the Final Qualitative Survey and Evaluation Program for the Nema program’s Strategic Objective 2 (SO2) activities. The chapter is in five sections: Section 1.0. SO2 Strategy/Theory of Change: Describes the intervention strategy and key actors involved in executing the strategy’s two major intermediate results (IRs). Section 2.0. Evolution of the Strategy for SO2 IR 1.1: Describes the evolution of the implementation strategy for IR 1.1 during the first four years. Section 3.0. Evolution of the Strategy for SO2 IR 1.2: Describes the evolution of the implementation strategy for IR 1.2 during the first four years. Section 4.0. Current Level of the SO2 Activities in the Nema Villages for IR 1.1 and IR 1.2: Describes the current level of activity in the Nema villages based on both the results of the qualitative survey (November 2013) and the key information interviews with the commune-level health centers (Centre de Santé Communautaire or CSCOMs) in the municipalities covered. Section 5.0. Lessons Learned and Recommendations: Presents a list of recommendations from the evaluation team and key implementation and executive partners designed to: • Increase the sustainability of the program accomplishments; and • Strengthen the benefits initiated by the program. 1.0. Nema SO2 Strategy/Theory of Change The aim of Nema’s second strategy was to improve food utilization by addressing health issues. The strategy was expected to achieve two strategic objectives (SOs): • IR 2.1. Caregivers of children under 5 and pregnant women are applying improved nutrition and feeding practices; and • IR 2.2. Caregivers of children under 5 are applying improved hygiene and sanitation practices. The sub-strategy for the achievement of IR 2.1 anticipated the use of a two-pronged approach 38 This chapter was prepared with the assistance of an informal Nema SO2 working group that developed around the final evaluation team’s field and key informant interviews. This working group was co-facilitated by Dr. Fatou Koite (HKI) and Sidibe Sidikiba (Consultant) and included the direct input of Dr. Bouba Halidou (HKI/GAO), Abdoussalam Tiemogo Maiga, (SCI/Gao), Abdel Kadere Sidibe (former CRS Hearth specialist at Douentza), and Kerri Agee (CRS Health Specialist). The working group was supported in its analysis of previous and current activities by Boureima Sako and Isack Dolo (M&E Department CRS), who co-facilitated the field portion of the exercise with Dr. Sidikiba Sidibe (Consultant, Nema), Dr. Fatou Koite (HKI), and Dr. Bouba Halidou (HKI). Dr. Della McMillan (consultant) provided analytical support. Nema Final Qualitative Evaluation. Chapter 3. SO2. December 30, 2013. Draft. 47 that targeted:39 • Behavior change communication focused on Essential Nutrition Actions (ENA);40 and • Rehabilitation of malnourished children through two specific types of interventions for rehabilitating malnourished children: - Complementary feeding for severe and moderate cases of acute malnutrition without complications using complementary take-home rations at the CSCOMs for children under 5 who are acutely malnourished, in addition to one weekly meal and materials for cooking demonstration; and - Scaling up the positive deviance/Hearth (PD/Hearth) model for community-based rehabilitation of moderately malnourished children using local food that CRS pilot tested in Douentza under the a one-year USAID Title II-funded emergency single-year assistance plan (SYAP). The proposal anticipated that IR 2.2 would be achieved by teaching caregivers of children under 5 improved hygiene and sanitation practices; promoting innovative sanitation infrastructures and hygiene messages; training communities in integrated water management; and increasing potable water points. The main partner in the implementation of this component of the Nema program was is the Ministry of Health through the CSCOMs. 41 Each CSCOM covers a number of villages, including those in the Nema program. Douentza and Bourem CSCOMs were heavily involved in the implementation of the component operations. The monitoring of the nutrition activities was completed by five trained community health volunteers (CHVs, or relais) in the different areas of intervention. A total of five CHVs in each village, who had a certain level of basic literacy, were selected and trained in nutrition, basic hygiene, and sanitation. Thus, to ensure proper integration with government structures and policies, the conceptualization and implementation of the activities was made in close collaboration with the most relevant government structures to ensure compliance with the National Protocol for the Treatment of Acute Malnutrition. 42 These activities were expected to decrease malnutrition levels by promoting sustainable behavior changes and strengthening the capacity of the local communities to sustain access to basic services and results with support from the CSCOMs and district-level health centers (Centre de 39 CRS, HKI, SCI. 2008. PL480 Title II MYAP. CFSM. Nema Program. Pp. 15-16. 40 “ENA consists of optimal breastfeeding practices, improved complementary feeding, appropriate nutritional care of sick and malnourished children, use of iodized salt, use of Vitamin A and improved nutrition for pregnant and lactating women, and the control of anemia” (CRS, HKI, SCI 2008: 15). 41 The 29 CSCOMs are led by two CSREFs (17 at Douentza and 12 at Gao) for Management of Acute Malnutrition (MAM) activities, PD/Hearth and radio broadcasts; the Directorate of Hygiene and Sanitation within the Ministry of Environment for sanitation component; the Regional Directorate of Water for the water component; and the Ministry of Education for the literacy component related to health programs and sanitation. 42 Ministère de la Santé du Mali. Protocole de Prise en Charge Intégrée de la Malnutrition Aiguë au Mali. Secrétariat Général Direction Nationale de la Santé/Division Nutrition Version révisée en 2011. The 29 CSCOMs are led by two CSREFs (17 at Douentza and 12 at Gao) for MAM activities, PD/Hearth and radio broadcasts; the Directorate of Hygiene and Sanitation within the Ministry of Environment for sanitation component; the Regional Directorate of Water for the water component; and the Ministry of Education for the literacy component related to health programs and sanitation. Nema Final Qualitative Evaluation. Chapter 3. SO2. December 30, 2013. Draft. 48 Santé de Reference or CSREFs) (Figure 3.1). Figure 3.1. SO2: Theory of Change Strategies, Nema Program 2013 2.0. Evolution of Program Activities for IR 2.1. Caregivers of Children Under 5 Years and Pregnant Women are Applying Improved Nutrition and Feeding Practices This section provides a brief overview of the principal activities and results of the Nema program’s SO2 activities. Since none of the impact indicators were measured before the program shut down in March 2012, this analysis is based largely on the results of the program’s internal tracking data (Table 3.1). The initial program document identified three main groups of activities to achieve IR 2.1: • Screening and care of acute malnutrition at CSCOMs in 130 villages; • The organization of community-based programs to rehabilitate moderately malnourished children using the PD/Hearth model in 45 villages; and • The promotion of improved nutrition and feeding practices by community radio in 130 villages. Nema Final Qualitative Evaluation. Chapter 3. SO2. December 30, 2013. Draft. 49 Table 3.1. Targets vs. Achievements for the Main Nema SO2 Activities, Years 1-4 Activity LOA Target Achieved % of Target Achieved Units # # % Training of health center staff at the district/municipality level on ENAs and CMAM 805 790 98% Monthly screening for malnutrition in the 130 villages using MUAC, first food ration, and reference to CSCOM Monthly 6,087 children admitted (3,004 rehabilitated) Training of SILC Groups in ENAs 200 66 33% Implementation of PD/Hearth sessions of 12 weeks in 45 villages 45 27 60% Radio broadcasts on nutrition/hygiene in local languages in the program areas 45 18 40% Initiation of the CLTS approach in 45 PD/Hearth villages 16 4 25% Construction of 90 latrines, 2 in each PD/Hearth village 90 88 98% Number of water points completed 49 15 31% Number of children affected by nutrition programs supported by USG (USAID) 20,909 20,330 97.2% Number of people trained in child health and nutrition through USG￾supported programs in the field of health (USAID) 6,932 8,326 120% Number of people in target areas with access to safe drinking water with the help of USG (USAID) 19,600 9,600 49% Source: Nema program M&E records. 2013. Acronyms: ENAs: Essential Nutrition Actions; CMAM: Community Management of Acute Malnutrition; MUAC: Mid-Upper Arm Circumference; SILC: Savings and Internal Lending Communities; CLTS: Community-Led Total Sanitation; USG: U.S. Government; LOA: Life of Activity. 2.1. PECIMA: Screening and Care of Acute Malnutrition at the CSCOM Level 2.1.1. Basic Strategy: The main objective of this component was to identify malnourished children and their rehabilitation. This approach was based on the use of Mid-Upper Arm Circumference (MUAC) tapes to measure children in accordance with the national policy of the Ministry of Health in order refer them to the CSCOMs for support. To achieve this goal, the Nema lead for health—Helen Keller International (HKI): • Organized a series of cascade trainings on essential nutrition actions (ENAs), 1 hygiene, and care of malnutrition for key players involved in the implementation of the activities; and • Distributed food rations and drugs such as albendazole, iron, folic acid, and vitamin A to encourage the rehabilitation of moderately malnourished children at the community level with their families’ involvement. The food support was very important for the most vulnerable families in the program area. In all areas covered by the Nema program, active detection of moderate acute malnutrition (MAM) cases was provided by trained CHVs under the direct supervision of the program community development agents (CDAs), who are leaders of the two implementing agencies (Caritas/Mali at Douentza, and Tassaght at Bourem). Nema Final Qualitative Evaluation. Chapter 3. SO2. December 30, 2013. Draft. 50 2.1.2. Evolution of the PECIMA Strategy (Years 1-4) 2.1.2.1. Program Year 1 (August 1, 2008 - July 31, 2009): During the first year, Nema’s SO2 activities focused on the design and execution of thee baseline survey, the recruitment and training of the program staff (CDAs), the training of health workers from the Ministry of Health (e.g. doctors, CSCOM station chiefs, and midwives) (Table 3.1), and the distribution of MYAP food for the treatment of moderately malnourished children who were identified at 25 CSCOM health centers. 2.1.2.2. Program Year 2 (August 1, 2009 - July 31, 2010) CSCOM-Based Activities: The program continued to emphasize medical staff training for health workers (midwives, nurses, nursing assistants, and community volunteers) during the second year. Altogether, 650 CHVs were trained (Table 3.1). As defined by Mali's national health and nutrition policy, the CHV is a goodwill volunteer chosen by the community to conduct health activities. The Nema program training focused on three activities: (a) the ENAs to prevent malnutrition; (b) hygiene and sanitation; and (c) the national strategy for the care of malnutrition. The CHVs were also trained on how to complete certain management tools, most notably the MAM registry and forms needed to distribute food rations.43 Each CHV was provided with basic equipment, including a MUAC tape for active screening of malnourished children and one registry per village, to record the results of the MUAC screenings. Once the program identified a child as malnourished, the program issued the mother a ration card for the child and a ‘protective’ ration for herself and other children in the household. Despite the incentive of food rations at the CSCOMs, the number of malnourished children being identified and rehabilitated remained very low compared to the target, which was based on the CSCOMs’ estimate of the number of children in the 130 villages (Figure 3.2).44 43 The training lasted four days and was held at the CSCOMs. 44 The second-year target for admitting acutely malnourished children to the CSCOMs for rehabilitation was 35%; only 28% were admitted (Mali CFSM MYAP final IPTT-Updated October 29, 2012). Nema Final Qualitative Evaluation. Chapter 3. SO2. December 30, 2013. Draft. 51 Figure 3.2. Evolution of the Number of Malnourished Children Identified by Nema, Years 1-4 Source: Nema program M&E data. 2013. Adama Sangare. PowerPoint presentation. November 2013. 2.1.2.3. Program Year 3 (August 1, 2010 - July 31, 2011): To increase the number of malnourished children being identified and referred to the CSCOMs, Nema decided to pilot test a new outreach strategy (stratégie avancée) for community-based (as opposed to CSCOM-based) screening and food aid distribution. This outreach strategy was supposed to complement (not replace) the CSCOM-based screening and food aid distributions. 45 The initial scale-up of this new stratégie avancée for MAM (Management of Acute Malnutrition) during Year 3 focused on the: • Use of the CHVs to identify malnourished children instead of waiting for mothers to take them to CSCOMs for screening; • Referring the children diagnosed as being severely malnourished to the CSCOMs for medical and nutritional intervention; • Provision of rations to the children brought to the CSCOMs for rehabilitation; and • Community-based distribution of food assistance at the village level—under the supervision of CSCOM-based government health workers and Nema CDAs—to facilitate the rehabilitation of children suffering from MAM. The short-term impact of this new stratégie avancée was a sharp increase in the number of malnourished children identified in the villages and referred for treatment to the CSCOMs in Douentza and Bourem (Figure 3.1). This increase was due to the fact that, in contrast to the old strategy, the stratégie avancée encouraged the CHVs to seek the children out and offer them 45 This innovative initiative was first discussed in October 2010 with all partners involved in the implementation of the program and a four-day training session was organized for CSCOM managers of Hombori village, ADC, and community health workers on distribution and support approach at the community level. The new approach was then tested in the village of Kiri (10 km from the Hombori CSCOM) before being extended in other areas of intervention. Nema Final Qualitative Evaluation. Chapter 3. SO2. December 30, 2013. Draft. 52 rations to encourage the mothers to send them to the CSCOMs for rehabilitation. Although the new stratégie avancée increased the percentage of children identified as malnourished in Year 3(Figure 3.2), it did not increase the number of children who were taken to the CSCOMs for rehabilitation (Figure 3.3). According to the mothers of those children, the predominant reason for not going to the CSCOMs was that they were unable to both attend the aforementioned program and maintain their household and farm work. Figure 3.3. Evolution of the Number of Malnourished Rehabilitated by Nema, Years 1-4 Source: Nema program M&E data. 2013. Adama Sangare. PowerPoint presentation. November 2013. To deal with these problems, the Nema team—in collaboration with the USAID-Food for Peace (FFP) (Dakar)] 46—proposed a new MAM approach to rehabilitation that targeted food-aid support to the malnourished children and their families until the child was considered completely cured, either in the village or in a CSCOM. This new initiative increased the number of children identified/detected, as well as the number of children who were monitored to full recovery because it reduced the number of families that abandoned or interrupted the rehabilitation process. To facilitate community support, HKI organized a series of public awareness sessions that were co-facilitated by the Nema CDA teams and CSCOMs. This training, in conjunction with the more active involvement of the CHVs, increased the involvement of the community in both the screening and care of the malnourished children. The short-term impact of the new strategy was a dramatic increase in the number of children admitted to CSCOMs for treatment of acute malnutrition from 28%47 in 2010 to 68%48 in 46 Dr. Fatou Koite (HKI), personal communication, November 2013. 47 Based on the number of children estimated with CSCOM figures (Mali CFSM MYAP final IPTT-Updated October 29, 2012 0.00% 5.00% 10.00% 15.00% 20.00% 25.00% 30.00% 35.00% 40.00% 45.00% 50.00% Baseline An  1 An  2 An  3 An  4 Target Achieved Nema Final Qualitative Evaluation. Chapter 3. SO2. December 30, 2013. Draft. 53 2011(Figure 3.2). Also, the number of acutely malnourished children who were rehabilitated jumped from 39% in 2010 to 42% in 2011 (Figure 3.3). In fact, the actual change in the percentage of rehabilitated children was probably even higher since the figures in the Indicator Performance Tracking Table (IPTT) reflected only the official figures that were registered by the CSCOM officials in their official registry. Although the CHVs—both individually and through the CDAs—transferred this information to the CSCOM staff, the CSCOM staff was often overwhelmed by the sheer volume of reporting data and failed to note these figures coming in from the villages. For all these reasons, the official figures—which do show a dramatic increase—under-estimate the full impact of the shift in strategy. Four factors that affected the transmission of information about the community-level impacts of the stratégie avancée were:49 • The disincentive of some CHVs to conduct the additional work without any form of cash or in-kind payment; • That the Nema program had only seven nutrition CDAs (facilitators of the two implementing agencies: Caritas and Tassaght) covering 130 villages, none of whom were directly supervised by HKI; • Insufficient quarterly and bi-annual coordination meetings between the officials involved in supervising the Nema CDAs; and • The lack of harmonized oversight and reporting between the different implementing partners. 2.1.2.4. Short-Term Impact of the Lessons Learned and Recommendations from the Mid-Term Evaluation: The mid-term evaluation made 16 recommendations50 for strengthening the Nema SO2 strategy;51 all but one of these recommendations was adopted. One cross-cutting recommendation from the mid-term—which had a direct impact on the execution and supervision of the MAM activities—was the decision to convert all of the specialized CDA positions (including those for nutrition) into multi-purpose CDA positions. The consequence of this conversion was to make one CDA responsible for the coordination of all the Nema activities in the villages that he or she covered, including agriculture, nutrition, food for work, and literacy. Since most of the multi-purpose CDAs had only undergone basic training in general agriculture, this approach required additional training of these CDA in nutrition in Year 4. Unfortunately, this conversion of the CDA nutritional specialists into multi-purpose CDAs had a very negative impact on the effective monitoring of the program’s nutrition activities. This was due to the unwieldy volume of activities the new multi-purpose CAS were expected to monitor, as well as the diversity of the target groups who were assisted on-site. This problem was exacerbated by the fact that two or three activities could be programmed in the same village at the same time with the same versatile CDA. Thus, the conversion of nutrition CDAs into the more versatile multi￾purpose CDAs has made it difficult to supervise many of the nutrition activities (like the PD/Hearth) that required high levels of supervision to be effective. Without the support of a 48 Based on official Nema M&E data (Mali CFSM MYAP final IPTT-Updated October 29, 2012 (Annex II). 49 Key informant interviews, Nema final qualitative evaluation interviews. November, 2013. 50 Six general recommendations; six related to the program’s nutritional activities; two for hygiene; and four for radio (Tilford 2011: 30-31). 51 K. Tilford. 2011. Nema Mid-Term Evaluation. Pp. 19-30. Nema Final Qualitative Evaluation. Chapter 3. SO2. December 30, 2013. Draft. 54 CDA focused on nutrition, community volunteers had difficulty filling out forms and logbooks at both of the CSCOM and village levels. A second impact of the mid-term was the program’s development of a more comprehensive behavior strategy.52 Based on this recommendation, the team conducted a formative research study to inform the development of this strategy and implemented a series of training programs for all of the Nema and CSCOM staff. 53 Unfortunately, the program was forced to close before the impact of the new training could be felt on the ground. 2.2. PD/Hearth Strategy Evolution 2.2.1. The Original Strategy: In addition to MAM rehabilitation activities (see Section 2.1 above), the program document proposed the implementation of the PD/Hearth methodology for rehabilitating moderately malnourished children using local products. The Nema proposal anticipated creating the PD/Hearth program in 45 of the 130 Nema villages based on a number of selection criteria. This criteria included “assessing the communities to assure that PD/Hearth can be fully integrated with other activities” as well as the availability of local food, availability of drinking water points, and proximity to the local health facilities’ CSCOM.54 The goal of the PD/Hearth activity was to help parents better understand the link between diet, nutritional status, and overall health by showing them how the children could gain weight and regain their health status through improved feeding practices. The PD/Hearth activities targeted 6-36-month-old children with a weight deficit identified through weighing (using weight/age index). The program consisted of three 12-day sessions during which the mothers of the malnourished children were exposed to improved feeding, hygiene, and education practices by a group of volunteer mothers (VMs) (Table 3.2). Once a village was selected for a PD/Hearth program, the program selected 10-1455 VMs, who were charged with sharing feeding practices, recipes that use locally available foods, and promoting positive behaviors such as frequent breastfeeding (four to five times daily) and complementary feeding of children after six months of age (fish based and other enriched porridges). The mothers were also charged with the task of taking children to the CSCOMs and with the organization of the PD/Hearth activities under the direction of the Nema ADCs. 56 52 K. Tilford. 2011. Nema Mid-Term Evaluation. Pp. 20-21. 53 At the beginning of Year 4 (FY11) (September 2011), HKI conducted a survey on "boundaries to behavior change'' on a sample village. The main finding of this survey was the need to integrate the negotiation of behavior change in the training of participants, with a particular emphasis on interpersonal communication strategy, including home visits. The survey indicated that women have some knowledge of nutrition but the practice remains a problem. So the new approach will aim to encourage beneficiaries to apply improved practices learned and not just the dissemination of messages. To implement this approach, HKI initiated a new series of courses for three target groups (health workers, relays, CDA). Radio messages have been revised to better reflect this new BCC approach. 54 CRS, HKI, SCI. 200. MYAP Proposal. Nema Program. Pg. 17. 55 The program anticipated choosing 10 “mamans volontaires” (VMs) by village based on the following criteria: (a) being an exemplary mother (i.e. having a child in good nutritional health); (b) being available; (c) agreeing to do the work as on a voluntary basis; and (d) being motivated to help malnourished children. The program anticipated the recruitment of 10 volunteers in order to be able to organize multiple sessions at the same time (May, June, July). 56 CRS/HKI/SCI. 2008. PL480 Title MYAP. CFSM. Page 17. Nema Final Qualitative Evaluation. Chapter 3. SO2. December 30, 2013. Draft. 55 Table 3.2. PD/Hearth Implementation Plan for Nema Program, Years 2-3 Session Activities Role of Key Individuals Supervision of Activities 1 (12 days) -Initial entry weighing of the children -Preparation of cereals -Culinary demonstrations in the preparation of porridge (toh porridge, etc.) 10 VMs (maximum) selected from the community to support 10 moms with 10 children MAM Nutrition CDA (Years 2 and 3) CRS nutrition supervisor (at Douentza), and SCI (at Bourem) 2 (12 days) Mothers return with children in individual homes to apply the knowledge received in group session, under CDA and VMs’ supervision 10 VMs make daily home visits to mothers of malnourished children under the CDA supervision Nutrition CDA and VMs 3 (12 days) - The group (VMs, mothers, and children) return to PD/Hearth to discuss shortcomings, improve knowledge and see together what worked and what did not work -Back weighing ("exit") of child growth and data reporting to CSREF and the M&E (Nema) CDAs, VMs, and nutrition supervisor for the area Nutrition CDA (Years 2 and 3) CRS nutrition supervisor (at Douentza) SCI (at Bourem), and HKI nutrition advisor Source: Abdoussalam Tiemogo Maiga, Field Office Manager SCI, Gao. November 2013 2.2.2. Evolution of the Strategy (Years 1-4): During the second and third year of the program, Nema (Table 3.3): • Trained 650 CHVs, 278 VMs, 189 committee members in 27 villages; • Constructed 29 hangars to facilitate this training at the relevant CSCOMs for the 27 villages: • Organized 60 sessions, and • Admitted 589 children to the PD/Hearth sessions. Table 3.3. PD/Hearth Activities in Nema Program,* Years 2-3 Year # Volunteers (Relay) # Volunteer Mothers # Committee Members # Sheds Constructed # of Villages Hosting Foyer # Foyer Sessions Organized # of Malnourished Children Douentza 500 190 131 17 19 45 439 Bourem 150 88 58 12 8 15 141 Total 650 278 189 29 27 60 580 Source: Nema program M&E data, November 2013. * Number of CHVs, VMs, and committee members trained in improved nutrition and hygiene practices; CSCOM hangars constructed for cooking demonstrations; villages hosting DP/Hearth; number of Hearth sessions organized; and number of children aged 6-36 months who participated. Nema’s PD/Hearth approach was greatly appreciated during the mid-term evaluation as a mechanism to get the community to focus on the impact of nutrition on the health and wellbeing of children.57 To cope with the low number of mothers available to care for malnourished children, the mid-term evaluation recommended helping local communities to mobilize cereals 57 K. Tilford. 2011. Nema Mid-Term Evaluation. Pp. 26-27. Nema Final Qualitative Evaluation. Chapter 3. SO2. December 30, 2013. Draft. 56 for PD/Hearths right after the harvest season when they were more available and less expensive to buy (November, December, and January). This change in strategy was quickly understood by the community and built into the PD/Hearth sessions, which were scheduled to start in Year 4, but never actually implemented due to the program ending. Given the limited analysis of the PD/Hearth data during the program, Dr. Sidibe, the lead consultant for nutrition and survey analysis, conducted an independent review of the data. His new data from the analysis of 60 PD/Hearth sessions organized in a total of 27 villages— eight villages from Bourem and 19 villages from Douentza (Figure 3.4)—shows a gradual improvement in the nutritional status of children, indicated by weight gain (weight/age anthropometric measures), through six weeks of a combination of PD/Hearth and home-visit sessions (Figure 3.4). In only six weeks, over 57% (332 of 580 children) were completely rehabilitated through the PD/Hearth approach using only the readily available local foods. Of the 580 children who participated in the PD/Hearth, 452 gained at least 300 grams of weight. Moreover, only 75 of the participating children lost weight, and only 52 remained stationary. These data suggest that mothers were able to apply the knowledge and improved nutrition and feeding practices learned during the PD/Hearth sessions, resulting in a significant improvement in the nutritional status of some malnourished children who participated in the PD/Hearth sessions. In several visited villages the evaluation team found that many children who participated in the PD/Hearth kept their improved nutritional status despite the crisis. They also found that mothers applied the same practices to their children born after the program withdrew from their villages. Nema Final Qualitative Evaluation. Chapter 3. SO2. December 30, 2013. Draft. 57 Figure 3.4. Number of Children Rehabilitated in the Nema PD/Hearth Activities in 27 Nema Villages, Years 2-4* Source: Nema PD/Hearth program records. Methodology: Reanalysis of the existing program M&E data using excel. * This re-analysis of the program data indicates that after six weeks, out of 580 children admitted, 453 children gained weight, 332 children were completely rehabilitated (had good nutritional status), and 121 gained weight but were not completely rehabilitated. The implementation of PD/Hearth activities was easier at Douentza than Bourem. Since most of the critical PD/Hearth foods (e.g. peanuts, cereals, and cowpeas) were grown by both men and women at Douentza, the women could access the ingredients without purchasing them. In contrast, it was virtually impossible for the women at Bourem to access to either these foods without purchasing them. Even firewood—which was critical to the preparation of the PD/Hearth recipes—was not available in most of the Bourem communities unless it was bought. To cope with these difficulties, some of the health committees worked with community leaders to collect some of the crucial ingredients for the PD/Hearth programs during harvest (e.g. peanuts, beans, sorrel leaves, and baobab leaves). Under the supervision of the health committee, VMs kept these voluntary contributions. The surplus was often sold and the money used to purchase products not available at the village level, such as sugar, oil, and fish (Text Box 3.1). Baseline  (n=580),   580   6  Weeks  after   (n=580),  127   121   332   0%   20%   40%   60%   80%   100%   Baseline   (n=580)   6  Weeks   after   (n=580)   Malnourished     Gained  weight  but   not  completely   rehabilitated   Completely   rehabilitated     Nema Final Qualitative Evaluation. Chapter 3. SO2. December 30, 2013. Draft. 58 Text Box 3.1. Case Study on the Implementation of Voluntary Contributions to Support PD/Hearth Programs in the Villages of Boumbam (Douentza) and Moudankan (Bourem) The villages of Boumbam (Douentza) and Moudankan (Bourem) were leading examples of mobilizing local resources to support PD/Hearth. As part of the implementation of PD/Hearth and the promotion of local products in the prevention of child malnutrition, the local people mobilized and stored a lot of local cereals, groundnuts, cowpea, and monkey bread to support the PD/Hearth Program. Source: Abdoussalam Tiemogo Maiga, Field Office Manager SCI, Gao. November 12, 2013. 2.3. Promoting Improved Nutrition and Feeding Practices Through Community Radio in 130 Villages 2.3.1. The Original Strategy: The third part of the IR 2.1 activities anticipated creating a series of nutrition and sanitation programs to be broadcast through nine rural radio stations in Nema’s intervention area. It was expected that these radio broadcasts would cover all of the Bourem villages and approximately three-fourths of the Douentza villages. 2.3.2. Evolution Strategy (Years 2-4): During the second year of the program, 15 radio hosts58 were trained. In addition, 18 radio messages, written in no fewer than seven of the most prevalent local languages, were developed on key themes of nutrition, hygiene, and sanitation and then recorded for a daily broadcast. Unfortunately, some of the records suffered from poor audio quality; however, this neither prevent the dissemination of key messages for two months in some areas of intervention, nor did it significantly degrade the quality of the messages being broadcast. By mid-term, only a small number of women’s listening groups had been established in villages in both districts. The mid-term evaluation concluded that: “[While there] is anecdotal evidence that the messages are being heard in some MYAP villages… how often and by whom [the messages are heard] is not clear. It is also unclear to what extent, if any, CDAs are supporting the listening groups. This support is critical, especially for new listening groups.”59 One major impact of the mid-term was that it highlighted the need for a more effective strategy for listening groups. To address this issue, the consortium selected five members from each of the savings and internal lending communities (SILCs) that existed in 58% of the Douentza villages and 57% of the Bourem villages to function as listening groups. During the Year 4, the Nema program trained 80 listening groups (713 people) to listen to the radio broadcasts on ENAs and share key messages with other SILC members during weekly meetings. Altogether, 713 people were trained on the new approach during Year 4 of the program. After training each of the listening groups received a certain amount of basic equipment (radios, batteries, ENA leaflets on nutrition and sanitation). 58 The training sessions organized by HKI for the radio journalists lasted two days and focused on essential nutrition actions and good dietary, drinking water, and sanitation practices. 59 K. Tilford. 2011. Nema Mid-Term Evaluation. Pg. 28. Nema Final Qualitative Evaluation. Chapter 3. SO2. December 30, 2013. Draft. 59 Despite training those involved, the impact of this new approach was not noticeable due to: (a) the abrupt withdrawal of program activities following the rebel incursion; (b) the interruption of radio messages transmissions; and (c) the lack of group monitoring by the CDAs to ensure radio broadcasts were received and shared with other members of the community. 3.0. Evolution of Program Activities for IR 2.2. Caregivers of Children Under 5 are Applying Improved Hygiene and Sanitation Practices The second IR of the SO2 strategy had two main themes: • Sanitation: Behavioral changes to hygiene practices; and • Drinking Water: Improving access and quality of drinking water. 3.1. Sanitation 3.1.1. Basic Strategy: The proposal anticipated that the program would build public awareness about latrines. The initial strategy focused on the construction of model platform latrines in the villages selected for the PD/Hearth program. It was anticipated that the model latrines, in combination with the dissemination of the radio and CDA messages that promoted the use of latrines, would accelerate the adoption of improved latrines. To facilitate the construction of latrines, the program trained and equipped 13 masons.60 The model latrines were constructed of imported materials (2 iron bars and 3 cement bags) provided by the Nema program, for two latrines in each PD/Hearth village. 3.1.2. Evolution of the Strategy (Years 1-3): A total of 10 model latrines were built in the PD/Hearth villages in the Year 3 and 14 in Year 4 for a total of 24. 3.1.3. Mid-Term Recommendations and Follow-Up: Given the program’s weak record on encouraging latrine use, the mid-term evaluation recommended that the program put more emphasis on this issue in the second half of the program. Based on this recommendation, the MYAP team put in place a technical team to carry the community-led total sanitation (CLTS, also known as the assainissement total piloté par la communauté or ATPC) activities in the intervention villages in collaboration with the Mali government state sanitation services. Four pilot villages were chosen to host the first awareness workshops on the CLTS approach. After an initial training workshop, each community was asked to commit to building latrines. This approach led to the construction of 142 latrines during the last two months of Year 3 and 190 latrines in Year 4 totaling 332 latrines built by the communities without any financial support from the program. This approach has contributed significantly to the improvement of hygiene practices in the beneficiary communities where it was pilot tested. Unfortunately, this good approach initiated by the program was only introduced in four of the 130 communities61 during Year 3 of the program and was interrupted by the rebel incursion in the program intervention areas. 60 The training lasted four days. 61 Although other communities were received the CLTS training, they never started the activities. Nema Final Qualitative Evaluation. Chapter 3. SO2. December 30, 2013. Draft. 60 3.2. Drinking Water 3.2.1. Strategy: The initial Nema needs assessments identified clean drinking water as one of the critical constraints to improving the nutritional status of children under 5 years. This challenge was confirmed by the Nema baseline survey, which showed that only 37% of villages (33% in Douentza and 41% in Bourem) had sufficient access to drinking water. Based on this analysis, Nema planned to construct 60 water points. 3.2.2. Evolution of the Drinking Water Strategy: Given the high cost of wells (especially the large diameter wells that were needed), the program planned to fund this component from the sale of MYAP food. Unfortunately, due to a shift in Mali government policy that reduced the capacity of the Nema program to generate cash through the monetization (e.g. sale) of MYAP commodities, the program was forced to reduce the number of water points from 60 to 49, out of which only 15 began construction (8 were completed; 7 remain unfinished though they have water). Out of those 15 wells completed, only one (in the village of Agamore) failed to produce water. This successful record of well construction, which was a notable accomplishment for villages a with history of multiple well failures, is attributed to the program’s strong collaboration with the state water service and strong supervision by the SO3 field team’s water specialists. Each of the completed wells is managed by a water management committee, whose members were trained in basic literacy and principles of infrastructure management. 4.0. Current Level of SO2 Activities in the Nema Villages for IR 1.1 and IR 1.2 One important objective of the final qualitative evaluation was to describe how the activities completed from 2008 to 2012 (i.e., SILCs, water resource development and management, feeding and hygiene practices, resilience to shocks) are currently being implemented by the beneficiaries after the withdrawal of the program in the areas occupied during the crisis (Table 1.2, Chapter 1).62 This analysis is based on team interviews with 55 of the 110 focus-group discussions that the final evaluation team conducted. In the course of these 55 discussions, the evaluation team interviewed 514 individuals (340 women, 174 men) (Table 3.4). 62 (1) What are the lessons learned from the shift from health-facility delivery to community-based health services in this MYAP context? Was this a good change? In what ways? How might it be improved further? (2) Are communities applying the techniques introduced for improved nutrition of infants in the context of the crisis in the north? If yes, what are the factors that encourage this use? (3) What was the Behavior Change Communication (BCC) strategy and was it relevant and efficient to positively affect nutrition and hygiene behavior change and outcomes? Describe and support your response with evidence. (Note: See Table 1.2, Chapter 1). Nema Final Qualitative Evaluation. Chapter 3. SO2. December 30, 2013. Draft. 61 Table 3.4. Number of Groups and People Interviewed about the Current Functionality of the Nema-Funded Activities in the 130 Nema Villages in Douentza and Bourem, November 2013 Community-Based Structures Interviewed During the Final Evaluation’s Focus Groups Bourem N=6 Villages Douentza N=15 Villages # of Focus Groups # of People # of Focus Groups # of People M F T M F T SILCs 3 3 92 95 12 100 121 221 CHVs 5 20 8 28 9 31 17 48 PD/Hearth 4 0 34 34 8 2 60 62 Radio 0 0 0 0 1 1 0 1 Mason 2 2 0 2 2 2 0 2 Health Staff 3 1 2 3 4 5 0 5 Water Point Management Committees 1 0 4 4 1 7 2 9 Total 18 26 140 166 37 148 200 348 Source: Team MYAP Nema, November 2013. See Table 1.6, Chapter 1 for a complete list of people interviewed. 4.1. Activities of the Nema-Created Health Committees and Nema-Trained Community Health Volunteers Based on the focus group discussions conducted during the final qualitative survey,63 the evaluation concluded that: • Most of the committees set up by the program to support the program’s nutrition, hygiene, and sanitation activities are not functional (e.g. Health committees, Adduction D'eau Sommaire [AES] management committee, etc.) (Table 3.5). • In most of the villages visited, the Ministry of Health’s CHVs that Nema had trained (650 community-based volunteers, 500 in Douentza and 150 in Bourem) were no longer actively supporting any sort of health and nutrition activities (Table 3.5; Text Box 3.3), although a few them did appear to participate in CSCOM outreach activities for vaccination; and • Almost all the Nema community-based nutrition activities (growth monitoring, replication of PD/Hearth activities, and household nutrition demonstration) stopped soon after the program withdrawal. The only functional CHV was found in a Bourem village, where he conducts screening of malnourished children in partnership with the NGO Médecins sans Frontières (MSF). Several groups of beneficiaries that met during the focus groups (Table 3.4) attributed these setbacks to the lack of Nema food to care for malnourished children: • "Since the withdrawal of the program, we did not conduct any activity because we were afraid to get together during the crisis"; and others asserted • "We did not conduct any activity due to lack of motivation and food since the withdrawal of the program from our community." 63 This includes focus groups with: (a) mothers of children who were rehabilitated through the CSCOM-managed PECIMA program, (b) CSCOM personnel, (c) leaders, and (d) CHVs. Nema Final Qualitative Evaluation. Chapter 3. SO2. December 30, 2013. Draft. 62 Table 3.5. Current Functional Status of the Formal and Informal Groups and Community Health Volunteers Trained by the Nema Program in Years 1-4, November 2013 Community Base Organizations Supported by the Project (# = Need for Official Recognition) Bourem Operating Level Proposed by Nema and CSCOM Douentza Operating Level Proposed by Nema and CSCOM # of Groups Met 1 2 3 4 # of Groups Met 1 2 3 4 Health Committees 4 4 9 9 Mothers Health Group (mothers who participated in rehabilitation programs)** 4 4 9 9 Water Management Committees 1 1 1 1 Committee Hygiene/Sanitation (CLTS Committees) 3 3 CHV (Relais) (nutrition activities) 5 1 4 15 15 Source: Team MYAP Nema. November 2013. * 1 = high, 2 = medium, 3 = low, 4 = non-functional **Organizing mothers into groups was not an objective of this program. Since this is a normal activity of most PD/Hearth programs, it is assessed here. Text Box 3.2. Testimonials from Six Different Community Health Volunteers on What They Learned During Nema, and Whether or Not They Continue Their Activities Oualo Relais (CHV). "My name is Souley Maba, Walo village Relais (CHV). Before the Nema program, I had no idea about nutrition and recognition of malnutrition in children. Thanks to the training received and activities with Nema program, I learned how to recognize malnutrition in a child even if he is not sick. I also learned how to screen for malnutrition at the community level and how to advise women to prepare recipes rich in vitamins for children. I would like to see again the Nema program in my village to continue with nutrition activities that had helped many children and mothers of our village. The nutritional and health status of children was much improved. With the abrupt withdrawal of Nema program and the crisis, all activities were stopped, although some women continued to apply improved feeding practices learned in their household. There are still many women in our village who do not know these practices, which is why we beseech the rapid and immediate resumption of activities with the Nema program.” Village CHV from Ngono, Fombory, Petaka, Dalla, Kiro: Community relais (CHVs) met say they currently do not conduct nutrition activities since the withdrawal of the program and noted an increase in cases of malnutrition, especially after the crisis. All relais were unanimous: "The children are often sick and emaciated. Some mothers continue to receive plumpy nut food from the CSCOM, but the vast majority remain in the village due to the distance causing an increase in children diseases.” Source: CHV Focus Groups, Nema Qualitative Final Evaluation. November 2013 4.2. Nutrition and Sanitation Practices of the Mothers Who Participated in the PD/Hearth Activities The results of the final evaluation focus group sessions with the mothers of malnourished children (Table 3.4) suggest that some mothers continue to apply improved feeding and hygiene practices learned especially during the Nema program despite the complete cessation of growth monitoring of children by the CHVs. This was especially true in the villages benefitting from the Nema-sponsored PD/Hearth activities. During the focus groups, the women testified that (Text Box 3.3: Nema Final Qualitative Evaluation. Chapter 3. SO2. December 30, 2013. Draft. 63 • “We continue the practices learned during the program and we also organized demonstrations sessions with porridge recipes fortified with beans;" • “We continue to apply the practices in households (Oualo Village);” and • “We are drying and storing millet powder to increase its availability in the household which facilitates the preparation of fortified porridge (Hombori village).” Text Box 3.3. Testimony of PD/Hearth Participant Fanta Boucary Maiga During the Focus Group Held with the PD/Hearth Group in Densa Village Fanta Boucary Maiga is a 32-year-old mother of eight children, all living in Densa village. Her fifth child, Halimatou, participated in PD/Hearth activities at age one. "My daughter Halimatou got sick several times when I was pregnant with her little sister. I became pregnant very early and abruptly weaned Halimatou. She was very thin and very weak. When the [CHV] came into the village, he weighed Halimatou and informed us that she was malnourished (her weight was very low compared to her level). The child received food. Halimatou began to eat and gain weight after several days. After Hearth, I continued with the same practices with the help of another mom who visited us. My child was saved by Hearth. We were also educated on the spacing of children birth. Currently Halimatou is five, in good health, and now has two sisters acquired without much problem thanks to Hearth. I thank God and the program brought to educate us.” Source: PD/Hearth Focus Group, Nema Final Qualitative Survey, November 2013 4.3 Community-Based Activities Offered by the Local Offices of the Ministry of Health In addition to the qualitative data collection, the evaluation team conducted in-depth interviews with eight health workers at three CSCOMs in Bourem and three CSCOMs in Douentza, the focal point of nutrition and the chief medical officer (hospital director) in Douentza. During the first four years, the Nema program trained large numbers of CSCOM and CSREF staff in essential nutrition actions, hygiene, and care of acute malnutrition at the CSCOMs and community level. Program records show that the program facilitated the training of 30 CSCOM station heads (12 in Bourem and 18 in Douentza), five CSREF agents, 18 nursing assistants, 18 midwives, 500 CHVs (5 per village) at Douentza, and 150 CHVs plus 27 midwives at Bourem. Some of the key joint activities with the CSCOMs included:64 • The distribution of food for moderately malnourished children and their households at the village level; • Referring severe malnutrition cases to the CSCOM for treatment; • Joint supervision of nutrition activities in the field and continuing education of all participants (CHVs, health workers, and leaders); • Joint supervision to improve the quality of care in the community-based and CSCOM￾based diagnostic and treatment activities; • Joint development of key messages on nutrition, hygiene, and training of local media workers to disseminate messages of improved nutrition practices and health hygiene; and • Construction of 17 hangars in Douentza CSCOMs and 12 at Bourem for nutritional demonstration and dissemination of messages on improved nutrition practices. 64 Based on interviews conducted with the CSCOM facilities staff during the Nema Final Qualitative Evaluation (November 2013). Nema Final Qualitative Evaluation. Chapter 3. SO2. December 30, 2013. Draft. 64 Unfortunately, almost all of the government staff members trained by Nema are no longer at their original posts. Many of the new staff, as well as the few who are still at their posts, expressed their appreciation for the training and made suggestions for how to improve the care of malnourished children in the former Nema villages (Text Box 3.4). During the qualitative survey, the field team collected information on the status of malnutrition at the CSCOM level in the former Nema program intervention area. These data showed (Table 3.6): • A sudden decrease in the number of children screened at the CSCOM level during the crisis which is probably due to the closure of many of these structures during the war and the lack of food for the care of malnourished children screened in CSCOMs that remained open (2012); and • A slight recovery recorded in 2013 compared to the Nema program period (2011) was attributed to the lack of food for the care of children at the village and CSCOM level. Table 3.6. Level of Malnutrition in Douentza and Bourem Areas Using Indicative Data of CSCOMs’ Staff Qualitative Survey, 2011-2013 Type of Malnutrition 2011 2012 2013 Douentza Bourem Douentza Bourem Douentza Bourem Moderate Acute Malnutrition (MAM) 547 _ 55 603 1689 662 Severe Acute Malnutrition (SAM) 2886 _ 545 60 1017 221 Source and Methodology: Bourem data are from three CSCOMs in the villages surveyed. Douentza data come from nutrition focal points at CSREF (Reference Centre) for the 100 Nema villages. Nema Final Qualitative Evaluation. Chapter 3. SO2. December 30, 2013. Draft. 65 Text Box 3.4. Assessment of the Impact of Nema Activities by Partners Linked to Health Facilities in Douentza and Bourem, November 2013 Positive impacts cited by the people interviewed during the focus groups were: • Increased active detection of acute malnutrition cases and care of moderate cases at the community level by community volunteers and program facilitators; • Increased CSCOM visits leading to an increase in malnutrition care at CSCOM centers through the distribution of food to moderately malnourished children and distribution of protection ration. This strategy led to a strong motivation of mothers with children for early case detection and immediate care at the community level; • Care of 100-200 malnourished children per month (CSCOMs Bourem); • Reduction of severe acute malnutrition to refer to CSCOMs and CSREFs following the early and wide treatment of moderate cases; • Considerable improvement of the nutritional status of children under 5 years through food distribution sessions and demonstrations at the CSCOMs and community level; • Increased primary curative care at CSCOMs; • Strengthening of CSCOM and CSREF staff capacity for screening and nutritional care of malnutrition (“Before the program, I did not have sufficient training in nutrition, especially for the care of acute malnutrition. With the series of trainings organized by our partners and regular supervision, we were able to significantly improve the care of moderate acute malnutrition cases, resulting in a decrease in severe cases referred to CSREF”); and • Motivation of CHVs through the training received and the distribution of food at the community level. The same interviews identified a number of areas that the staff felt could be strengthened. This included: • The lack of a regular meeting through a monitoring system of activities and quarterly data to discuss all issues identified in the implementation of program activities and propose solutions in accordance with the national nutrition policy; • Non-coverage of all villages in the health district Douentza. Out of 262 villages, the program only covered 100 for a rate of 100/262. With 500 CHVs trained, the program could have considered two CHVs per village instead of five; and • The fact that the program stopped just when the participants involved in the care and active screening had begun to master the approach and have an impact on health indicators. Some of the current challenges that the CSCOMs face in dealing with the resurgence in the number of malnourished children are listed below: • “With the increasing number of cases of severe acute malnutrition seen in health facilities, inadequate screening of malnutrition at the community level, lack of staff, and especially the departure of MSF, with whom we have established a free access system, the major challenge for health facilities remains to maintain the level of care of severe malnutrition cases and the implementation of prevention activities for early detection and care of moderate malnutrition. This challenge cannot be met without the support of development partners for the resumption of the fight against malnutrition in the Circle of Douentza.” • “Currently, there is a slowing or complete cessation of the implementation of nutrition activities (screening, nutrition demonstrations, and dissemination of messages) by community volunteers who no longer work because of the crisis. After the crisis, the continuation of activities still faces [CHVs] demotivation, lack of food to distribute to screened cases, and the absence of close supervision. The major challenge remains the resumption of these activities on the field.” Source: CSCOM Key Informant Interviews, Nema Final Evaluation, November 2013. 5.0. Lessons Learned and Recommendations One of the strengths in the implementation of SO2 activities was the flexibility of the partners involved in the management of Nema program. They did acknowledge some shortcomings in the Nema Final Qualitative Evaluation. Chapter 3. SO2. December 30, 2013. Draft. 66 program document and agreed to adopt approaches best suited to realities in the field—in accordance with the Ministry of Health policy—to better achieve the objectives. This section provides a brief overview of the main findings and lessons learned from the final qualitative evaluation field surveys and key informant interviews. 5.1. Key Issues 5.1.1. Effectiveness 5.1.1.1. Care of MAM Cases: The initial MAM approach was keenly geared towards the CSCOM health facilities; however, it was very well reoriented during Years 3 and 4 of the program to a more community-based approached for treating children suffering from MAM; referring the more Severe Acute Malnutrition (SAM, also known as Malnutrition aigue severe or MAS) cases to the CSCOMs for treatment; and distributing food. Specifically, the Nema program helped pilot test a new stratégie avancée program for nutrition that was consistent with the national malnutrition strategy.65 The approach allowed Nema program agents and CSCOM staff to conduct joint monitoring and supervision activities in health areas and villages. Two key aspects of the activities that had a negative impact on program implementation were: (a) the fact that Nema program budget (during Years 1-4) did not enable the principal consortium partner that was responsible for the design and execution of the nutrition activities to supervise and direct the activities on the ground; and (b) the weak motivation of the unpaid CHVs to execute the Nema nutrition activities given the other work they were expected to do for the Ministry of Health and their own crop and livestock production activities. This first recommendation, which was incorporated into the proposal for Nema’s new areas of intervention in the south (2012-present), has helped the program very effectively achieve the new targets in a short period of time. Two key challenges for the next generation of the Nema program will be to: (a) provide some way of compensating the CHVs; and (b) train 80% of the SILCs created by the Nema extension agents (120 out of 140 groups) in ENAs and sanitation. 5.1.1.2. PD/Hearth: The PD/Hearth approach was very good, but it was implemented in only 27 of the 130 Nema villages. The next phase of the program should extend the PD/Hearth methodology to every one of the villages, taking into account the weight/height index with a particular emphasis on extended monitoring (i.e. continuing to monitor the children at progressive intervals of two months, six months, then one year after their release from PD/Hearth). This improved tracking system—standard in non-emergency PD/Hearth programs— will help the program to better monitor the impact of the PD/Hearth activities on the children’s nutritional status and the application of improved practices learned by mothers during PD/Hearth. 5.1.1.3. Behavior Change Communication: Unfortunately, the introduction of the new “behavior changes communication (BCC) through interpersonal communication and home visits” approach that was recommended by the mid-term evaluation did not start until the beginning of the 65 Direction Nationale de la santé, Division Nutrition. 2011. Protocole: Prise en charge intégrale de la malnutrition aigue. Bamako: Direction National de la santé, division nutrition. . Nema Final Qualitative Evaluation. Chapter 3. SO2. December 30, 2013. Draft. 67 program’s fourth year, so it is impossible to assess its impact over longer period of time in the original Nema villages. 5.1.1.4. Training Modules: During the first four years, the Nema program developed a series of PowerPoint modules tailored to SO2 topics, including topics on HIV/AIDS and the national strategy for malnutrition care (e.g. appropriate community-based methodologies for MAM, BCC, etc.). These modules represent a program asset that is being exploited in the new MYAP area of intervention. Another pedagogical impact of the program was the creation of a series of health, nutrition, and sanitation lessons that were incorporated into the literacy training manuals that were used in all 130 Nema villages. 5.1.1.5. Sanitation: Despite the importance of the "theory of change" granted to sanitation at the beginning of the program, the program document did not propose a good model for implementation of outreach activities for the use of latrines. The integration of the Mali government’s innovative CLTS approach, in collaboration with the Ministry of the Environment, has improved the impact of the use of latrines by the beneficiary communities. Nema’s funds helped hire a series of consultants in areas where the local Ministry of Environment staff in Douentza did not have sufficient experience. Future programs need to extend the CLTS initiative to all program villages from the start. 5.1.1.6. Drinking Water: The Nema program budget attempted to fund the well development activities as part under the most vulnerable part of the financing (i.e. monetization), resulting in this component of the program never executing fully. Consequently, there are still villages— even CSCOMs—with inadequate drinking water. This is a major critical constraint to achieving IR 2.2, since, in the absence of clean drinking water, one cannot even talk about proper hygiene. 5.1.2. Sustainability: A major strength of Nema’s SO2 activities was its effective involvement of administrative authorities (city mayors, prefects) and state technical services (e.g. Ministries of Health, Environment, and Water) in the implementation, monitoring, and evaluation of the program’s nutrition, water resource development, and sanitation activities. The close collaboration was instrumental in: (a) mobilizing the local beneficiary communities’ adherence in activities implementation; (b) insured the successful integration of these activities into the national sector strategies for nutrition, rural water resource development, and sanitation; (c) the construction of 29 cooking demonstration hangars at the CSCOM level; and (d) the adoption of improved practices in health and nutrition by mothers who took advantage of nutrition activities. 5.1.3. Other Unexpected Impacts: Two other impacts of the program, which were in a sense unexpected (i.e. not anticipated in the program document), include: • Training the SILC groups in health and nutrition; and • Including a wide range of training lessons on nutrition and hygiene themes in the books used for literacy training in all 130 Nema villages. 5.1.4. Monitoring and Evaluation 5.1.4.1. Global M&E Strategy for SO2 Nema Final Qualitative Evaluation. Chapter 3. SO2. December 30, 2013. Draft. 68 5.1.4.1.1. Impact Indicators: The Nema program was unable to conduct a quantitative survey to measure the impact indicators that were assessed at the beginning of the program before its abrupt discontinuation in March 2012. It is still possible, however, to observe some of the nutritional impacts of the program through some of the program’s internal qualitative data (from the Nema final qualitative evaluation) on the adoption of improved feeding practices learned by some mothers during the PD/Hearth sessions. 5.1.4.1.2. Monitoring Indicators: Unfortunately, monitoring indicators in the IPTT poorly tracked many of the program’s main SO2 achievements. This issue was raised during the mid￾term evaluation but was never acted on due to the termination of the program—only eight months after the mid-term.66 The disconnect between the actual activities on the ground and those being tracked was due to the fact that the first list of indicators was adopted from the initial strategy, a strategy that was based on all of the activities being undertaken at the level of the CSCOMs. As a result, the methodology used to track many of the Nema program’s SO2 indicators was based on the CSCOMs’ official tracking data. When the Nema SO2 strategy changed in Year 3, a lot of data on community-level activities was no longer reported to the CSCOMs. This gave the impression that the program had less impact in SO2 component implementation. The conversion of the nutritional CDAs into multi-purpose CDAs in Year 4 made reporting even more difficult. This lesson learned was taken into account in the new intervention area by recruiting six additional officers as nutrition support staff directly attached to HKI—in addition to the HKI nutrition supervisor. The HKI-supervised staff was responsible for both execution and insuring that the CSCOM registry figures are correct. One of the main impacts of Nema’s technical assistance during the program’s first year was to make the Nema program teams—as well as implementing agents (CRS, SCI, Caritas, and Tassaght) and local Ministry of Health teams—more effective in nutrition through training and capacity building. Unfortunately, this component of the program (unlike the SO1 and SO3 components) did not have indicators that tracked local capacity building. 66 K. Tilford. 2011. Nema Mid-Term Evaluation. Pp. 43-45. Nema Final Qualitative Evaluation. Chapter 3. SO2. December 30, 2013. Draft. 69 5.1.4.2. Monitoring Indicators in the IPTT and Other Nema Program Quantitative Indicators Table 3.7. IR 2.1. Caregivers of Children Under 5 and Pregnant Women are Applying Improved Nutrition and Feeding Practices Baseline Year 2 (FY 10) Year 3 (FY 11) Year 4 (FY 12) Target Achieved Target Met Target Achieved Target Met Target Achieved Target Met MI 2.1.2. % of children admitted to CSCOM for treatment of acute malnutrition 27% (22.67, 31.14) 35% 28% 47% 40% 68% 170% 45% 23% 51% MI 2.1.3. % of children 0-59 months with severe acute malnutrition who are rehabilitated [1] at the CSCOM 10% (6.86, 12.18) 30% 24% 48% 40% 11% 28% 50% 3% 6% MI 2.1.4. % of children with moderate acute malnutrition who are rehabilitated at CSCOMs 21% (17.89, 23.12) 50% 39% 50.6% 40% 42% 105% 50% 6% 12% Source: Nema IPTT, Annex II. Three indicators from the CSCOM registry (Indicators 2.1.2, 2.1.3, and 2.1.4) show a gradual change in the number of children ages 0-5 followed through growth-monitoring activities at the CSCOMs and also the percentage of children rehabilitated at the CSCOMs (Table 3.7). Despite this positive trend, these figures are lower than expected for the following reasons: • From Year 3 of the program, the number of children rehabilitated at village level has increased significantly due to the implementation of the stratégie avancée for MAMs case rehabilitation. For example, the number of children affected by programs (MI 2.1.9: Number of children affected by the nutrition program) increased from 749 in Year 2 to 10,991 in Year 3, and even to 8,590 at the end of the third quarter of Year 4. • Unfortunately, these figures under-estimate the actual number of children and their rehabilitation because a lot of the data on the children who were identified and rehabilitated in the communities was never transmitted to the CSCOM level. Once the CDA positions that focused on nutrition were converted to generalist positions, it became even harder to monitor the program’s community-based nutrition activities. 5.1.4.3. PD/Hearth Monitoring Indicators (IPTT) Table 3.8. MI 2.1.6. Percentage of Beneficiary Caregivers Who Practice Behaviors Shown to Be Successful to Rehabilitate Underweight Children Base Year 2 (FY 10) Year 3 (FY 11) Year 4 (FY 12) Goal Achieved Goal Achieved Goal Achieved Goal Achieved Goal Achieved Goal Achieved NA 10% 1.8% 18% 15% 1.3% 9% 20% ND Source: Nema IPTT, Annex II. Unfortunately, the IPTT indicator which should follow the community-based rehabilitation of the children under both the advanced strategy and the PD/Hearth model (MI 2.1.6, Table 3.8) was measured through an annual survey that the Nema M&E staff conducted on a sample of all mothers affected by the program (i.e. mothers involved in the SILC listening groups, PD/Hearth, MAM, health committees, etc.). Because of this, the indicator only measures the mothers Nema Final Qualitative Evaluation. Chapter 3. SO2. December 30, 2013. Draft. 70 covered in this annual survey who also participated in PD/Hearth activities, which is why the achievement is so low. 5.5.1.4. Monitoring Indicators from the Annual Mothers Survey Conducted by the Nema M&E Staff Table 3.9. Nema Monitoring Indicators Based on Data Collected by the Annual Mothers Survey Baseline Year 1 (FY 09) Year 2 (FY 10) Year 3 (FY 11) Year 4 (FY 12) Targe t Achieved Target Met Target Achieved Targe t Met Targe t Achieved Target Met Targe t Achieved Target Met MI 2.1.6. % of beneficiary caregivers who practice behaviors shown to be successful to rehabilitate underweight children NA -- -- -- 10% 1.8% 18% 15% 1.3% 9% 20% ND MI 2.1.7. % of beneficiary mothers who had at least one post-partum checkup 8% (6.19, 9.81) -- -- -- 25% 19% 76% 30% 38% 127% 35% ND MI 2.1.8. % of beneficiary pregnant women who attend at least three prenatal visits 33% (31.48, 34.52) -- -- -- 75% 75% 100% 78% 79% 101% 81% ND MI 2.1.9. # of children reached by USG-supported nutrition programs (USAID) 0 0 0 -- 10313 749 8.1% 6120 10991 180% 6720 8590 128% EI 2.2.7. % of targeted caregivers who report that they sweep/clean their house daily 75% 69% 92% 75% 77% 103% 75% ND Source: Nema IPTT, Annex II. The indicator data from the annual mother survey done by the Nema M&E unit in November of each year showed a clear trend in all indicators related to key behaviors changes (hand washing with soap, prenatal care—Indicators 2.1.7, 2.1.8, 2.1.9, and 2.2.7) except behaviors linked to transporting drinking water (2.2.3) and use of latrines (2.2.6). The poor performance of these IPTT monitoring indicators was due to the facts that: • The program could not fund the original number of water points it needed to address the most pressing water issues in the Nema villages; and • The new model of platform latrines was found only in the four villages that benefited from the CLTS-model latrines and in some the PD/Hearth villages. 5.2. Key Factors Which Reduced the Impact of the Nema Health Strategy in Nutrition in 130 Target Villages 5.2.1. The Nema “Theory Of Change” was Treatment-Focused, Not Prevention-Focused: It is vital to remember that the paradigm shift in development-oriented nutrition—that is to say, from treatment focused to prevention focused—occurred largely after the Nema program was developed in 2006-2007. The Lancet series on chronic malnutrition, Scaling Up Nutrition (SUN), and 1000 Days all gained momentum after this program was designed,67 approved, and implementation had started. In looking at this program through today's prevention lens, it appears the Nema “theory of change” was very much off-base, although, at the time, it was appropriate. 5.2.2. Poor Integration of Resource Persons at the Community Level: The core program 67 See http://www.bread.org/event/gathering-2013/international-meeting/pdf/june-10-summary-report.pdf. Nema Final Qualitative Evaluation. Chapter 3. SO2. December 30, 2013. Draft. 71 document intended to target other resource persons in addition to mothers, such as grandmothers, fathers, and community leaders. Unfortunately, the communication strategy developed by MYAP did not focus on targeting these resource persons in training and communication for behavior change in the context of improved nutrition and feeding practices at the household level. Future programs need to consider the influence of resource persons in decision making for nutrition of children under 5 years. 5.2.3. Inadequate Focus on Pregnant and Lactating Women: The Nema program proposal anticipated that the communication strategy for behavior change would focus on the ENAs needed to encourage households and the beneficiary community at large to find better strategies for appropriate infant and young children feeding, as well as nutrition during pregnancy and lactation. During its implementation, the emphasis was switched to child nutrition and recovery of malnourished children at the community and CSCOM levels. As a result, there was no focus on feeding women during pregnancy, which is a crucial step to preparing future mothers for the adoption of improved feeding practices during pregnancy and breastfeeding. 5.2.4. Not Taking Into Account the Weight/Height Index in Monitoring PD/Hearth Children: The Nema PD/Hearth program used only weight/age to determine children’s eligibility for and progress from the PD/Hearth activities. This is not the measurement that is recommended by the Ministry of Health’s National Malnutrition Policy. This policy especially recommended the use of weight/height index and MUAC for acute malnutrition screening. In addition to the weight/age (underweight) index, future PD/Hearth programs should include the weight/height index in the children selection criteria, as recommended by the Ministry of Health. During the interview with key informants, it was found that often some children identified as severely malnourished and referred to the health center by the CHVs were not cared for, as the weight/height criteria (for acute malnutrition detection) excluded these children as severely malnourished. This was a source of frustration among some mothers. 5.2.5. Inadequate Support from the Nema CDAs: The low coverage of nutrition activities in the first two years of the program has been exacerbated by the conversion of the nutrition CDA positions into generalist CDA positions in July 2011 (end of Year 3). To facilitate sustainability, the PD/Hearth approach requires close monitoring during implementation to ensure ownership of activities by the beneficiaries communities. 5.2.6. Lack of Exit Strategy: With the lack of implementation of an exit strategy in different areas of program intervention, the abrupt cessation of the Nema program’s SO2 activities had a very negative impact on sustainability and local ownership of most activities (PD/Hearth, AEG, SILC, PECIMA) both by the people and the CSCOMs. 5.2.7. Lack of Development of Communication Materials for Behavior Change on ENAs and Feeding of Pregnant and Lactating Women: Many of the groups that were supposed to continue the nutrition activities were left without any training materials, such as picture boxes, flyers on ENAs, or the program’s training materials on improved nutritional practices for pregnant and lactating women. 5.3. Lessons Learned and Recommendations Nema Final Qualitative Evaluation. Chapter 3. SO2. December 30, 2013. Draft. 72 5.3.1. Recommendation 2.1. Rethink the Nema program’s support for the national government’s PECIMA strategy through the CSCOMs. 5.3.1.1. Summary Observations: 1. Involvement of Health Facilities and Local Authorities: The involvement of administrative authorities (city mayors, prefects) and technical services of the state (health, environment, and water) in the implementation M&E of Nema’s SO2 activities during Years 1-4 contributed to the mobilization and adhesion of the beneficiary communities. Conversely, the sudden withdrawal of the program due to the crisis had a negative impact on ownership and the continuation of nutrition activities at the community level. Various key informants, with whom the evaluation team met at the local health facilities (CSCOM, CSREF), and mayors confirmed that not one of the Nema-trained CHVs that are the official representative of the CSCOMs in the Nema villages is currently promoting community-based nutrition programs. 2. Limited Post-Nema Involvement of the CHVs in Nutrition: The program document emphasized the role of CHVs in monitoring SO2 activities. The CHVs are community￾level volunteers that are recognized by the Ministry of Health and made available to all stakeholders (NGOs and the government). The Nema program trained 650 CHVs on the following subjects: (a) exclusive breastfeeding; (b) nutritional supplements; (c) malnutrition screening; (d) cooking demonstrations; and (e) the design and execution of PD/Hearth activities. Unfortunately, the volume of work and low motivation of CHVs did not yield the expected results. Future programs need to determine if and how the CHVs’ role could be supplemented by the role of the community health worker (CHW), whose salaries must be paid for by the community. Some health districts in Mali are pilot testing CHWs as an intermediary between the CHVs and CSCOMs. At present, this new policy is not yet applied in the areas of Bourem and Douentza. The fact that the CHVs were non-functional following the departure of the program should not be a surprise given the fact that the number of CHVs was insufficient from the start of the program. To encourage sustainability, the Nema program tried to build its activities on the existing system of CHVs. Future programs need to help the health structures recruit and retain the most qualified CHVs. Equipping these individuals with proper materials to do their jobs and getting the communities to provide them with more in-kind and financial support will allow motivated volunteers to continue to spread the message after the departure of the program. 3. PECIMA’s Heavy Reliance on Food Assistance to Get Mothers to Bring Their Children in for Rehabilitations: The PECIMA component (SO2, IR 2.1) was implemented as an emergency program, with activities that focused on active screening and care of malnutrition that were strongly dependent on the distribution of food to malnourished children including a protective ration for the malnourished children’s families. Thus, the program did not focus on promoting monitoring (monthly weighing and cooking demonstrations activities) of children at the community level that could be accompanied by the distribution of food. The principal reasons that most CHVs and staff gave for not continuing the nutrition activities after the program closed was the absence of food aid: • "We have no more food to give to children who are detected, this is why we cannot continue with the activities, for moms were motivated by the presence of Nema Final Qualitative Evaluation. Chapter 3. SO2. December 30, 2013. Draft. 73 food"; and • “Douentza is a food insecure area, so you need food, and the program must return to continue the activities of active screening and treatment of malnourished children at the community level. “ 4. Weak But Poorly Documented Impact of the IEC Radio Programs: To promote communication activities for behavior change, the Nema program (through HKI) trained radio hosts in ENA, hygiene, and the national strategy for care of malnutrition. The same initiative developed a series of messages about improved nutritional practices and the care of acute malnutrition that were translated into local languages for broadcasting by the local radio stations. To date, no one has evaluated the impact of these broadcast messages, but it appears that their greatest impact was in sites where the taped messages were distributed to the SILC listening groups. 5.3.1.2. Key Recommendations: 1. Retrain the CHVs and provide them with training materials (IEC support picture boxes, posters, pamphlets, etc.) on the prevention and community-level treatment of acute malnutrition. 2. Build public awareness about the need to support the CHVs though the development of income generating activities (IGAs) through the AEGs and SILCs, community support for their farming activities, or setting up a system of in-kind compensation through the mobilization of community resources. 3. Strengthen the baseline training of the health personnel (at the CSCOM and CSREF levels) in the identification and treatment of moderately malnourished children, and organize regular retraining sessions. 4. Provide additional training on nutrition for the SILC and AEG members to help them better train the mothers and fathers in the monitoring and community-based rehabilitation of moderately malnourished children. 5. Anticipate the need for a progressive transfer of knowledge and responsibility to the CHVs and make sure they have the tools and equipment they need to conduct growth monitoring in compliance with the national norms for fighting malnutrition. 6. Integrate the promotion of micro-nutrient rich foods into the routine activities of the CHVs, and put them in contact with other community-based structures or women contact persons in order to promote appropriate health messages concerning dietary diversity and the promotion of local foods that are rich in micro nutrients (vitamin A and iron). 7. Introduce food technology activities to process local food (production of weaning foods, local foods enrichment). 8. Strengthen the frequency of in-home training visits by the CHVs and volunteer mothers in order to create the enabling environment required for behavior change. 5.3.2. Recommendation 2.2. Improve the execution of the PD/Hearth-model programs in the program villages. 5.3.2.1. Summary Observations: 1. PD/Hearth Better Suited to an Emergency Context: The Nema model for PD/Hearth focused heavily on cooking demonstration and the promotion of local products that could provide high-quality weaning foods. The heavy involvement of the volunteer mothers in Nema Final Qualitative Evaluation. Chapter 3. SO2. December 30, 2013. Draft. 74 the organization of the cooking demonstrations limited the amount of time the mothers had for apprenticeship. This had the unintended consequence of limiting the mothers’ ownership of the improved practices being promoted. Other problems were caused by the fact that many of the women who volunteered to be model mothers were unable to continue the PD/Hearth activities after the program closed. Though the focus was on replication of DP/Foyer approach by participating moms in their respective families/communities, this replication did not occur once the CDAs completed their conversion to generalists. The best strategy would have been that program staff be based in the villages to enhance the transfer of skills to CHVs and help establish health committees to support nutrition activities in the village. This strategy would have enabled a better understanding of target groups, their difficulties, and the M&E of field activities. 2. Low Coverage of Neediest Nema Villages by PD/Hearth: Even with its acknowledged weak points (see #1 above), the Nema PD/Hearth approach was very good, but its implementation only covered 21% of the intervention villages (27/130). In addition to the crisis, which shut down the Nema program and most CSCOMs for many months, the criteria (availability of local products, drinking water, a nearby health center) used by the program to build a village FARN approach limited its extension to the neediest villages. Future programs need to take this into account and consider steps to extend PD/Hearth coverage to all villages, with particular emphasis on developing PD/Hearth programs in the villages that are identified as most vulnerable in food security and nutrition. 3. No Long-Term Monitoring of the Children After They Graduated from the PD/Hearth Program: Two weak points of the Nema approach for PD/Hearth were: (a) not taking into account the weight/height index; and (b) insufficient monitoring of children in the longer term (i.e., two months, six months and one year after the children leave PD/Hearth). This type of post-PD/Hearth growth monitoring is needed for two reasons. First, it enables the program to measure the impact of PD/Hearth activities on the nutritional status of children and the application of improved practices learned by mothers during PD/Hearth. Second, it provides a mechanism for the program to support the mothers’ continuing the improved practices learned during PD/Hearth. The focus group discussions show that many of these mothers were able to continue these practices, even for other children born later in their households. 4. Lack of Integration of SO2 Activities with SO1 Activities: The MYAP proposal anticipated a strong link between the program’s SO1 and SO2 activities through training PD/Hearth mothers in gardening techniques and support for the development of gardening techniques that would increase household access to micronutrient-rich vegetables. Unfortunately, this linkage was not well developed until the third year of the program. 5. Weak Monitoring of PD/Hearth Activities by the Nema CDAs: Given the volume of activities the CDAs were responsible for, they were not very effective in monitoring women who participated in the approach. At the start of the implementation of PD/Hearth activities, the sessions often coincided with the preparation of farm activities and the lean seasons (June-July). This frequently led to difficulties in obtaining local foods and the effective participation of mothers with malnourished children. Based on lessons learned from program Year 2, the implementation period of PD/Hearth activities was taken back to the months of December-April in program Year 3. During the Year 4 (2012) of the program, mobilization and identification activities of malnourished children were made, Nema Final Qualitative Evaluation. Chapter 3. SO2. December 30, 2013. Draft. 75 but because of the crisis, PD/Hearth was not undertaken. 5.3.2.2. Key Recommendations: 1. Given the observed impact of the PD/Hearth model programs on mothers’ health practices both during and after the program, future Nema interventions should implement a revised PD/Hearth model in every village where the program intervenes that includes revised model for tracking the children’s progress both during and after treatment.68 2. Encourage the development of IGAs for the vulnerable mothers of the children participating in the PD/Hearths through the development of irrigated gardening activities and organizing them into care groups. 3. Involve community volunteers in early PD/Hearth activities, and strengthen the CSCOM’s supervision of the CHVs (at least twice per quarter) to ensure sustainability of the achievements in the field, especially in new areas of program intervention. 4. Introduce food technology activities for local food processing (production of weaning foods, local foods enrichment). 5. Transform the earlier Nema "voluntary mothers" system into the system of "Mama Lumieres" used by most non-emergency PD/Hearth programs. 5.3.3. Recommendation 2.3. Improve access to clean drinking water and strengthen hygiene and sanitation practices by requiring all villages to support the Mali government’s regional and cercle-level CLTS initiative. 5.3.3.1. Summary Observations: Since clean drinking water is still a pressing constraint in about half the Nema villages due to budget constraints that restricted the program’s ability to create wells, a third set of recommendations focuses developing a wider network of support for creating potable water drinking points. Given the successful record of the government-sponsored CLTS program in adjacent areas and the successful pilot testing of this program in five Nema villages during the start-up phase, the health team is encouraging future programs to support the government’s CLTS initiative to promote community sanitation programs. 5.3.3.2. Key Recommendations: 1. Since drinking water is still one of the principal constraints to improving the nutritional status of the most vulnerable population in the villages, the consortium partners might think of exploring other opportunities for funding drinking water infrastructure through other organizations or private-donor funds. These funds, as well as any future funds, need to give priority to water resource development in the most vulnerable villages. 2. Future programs should anticipate introducing the government’s new CLTS initiative, which was pilot tested in five of the Nema villages, in every one of the Nema communities. 5.3.4. Recommendations Made by the Partners and Beneficiaries: During interviews with the technical partners of state services, some recommendations were made to improve the care of malnutrition (Text Box 3.5). Beneficiaries also made recommendations during focus groups. 68 This includes using weight/height index and monitoring the children’s progress two months, six months, and one year after the children leave PD/Hearth. Nema Final Qualitative Evaluation. Chapter 3. SO2. December 30, 2013. Draft. 76 Text Box 3.5. Main Suggestions or Recommendations Made by the Technical Partners and Beneficiaries in the Improvement of Program Activities Les principales suggestions ou recommandations formulées par les partenaires techniques dans le cadre de l’amélioration des activités du projet • « Nous sollicitons le retour du projet et la reprise des différentes activités dans 100 villages qui étaient couverts en mettant l’accent sur la formation de deux relais par village ». • Reprendre le dépistage actif des cas de malnutrition modéré au niveau communautaire avec la distribution des vivres aux enfants malnourris • Reprendre les activités de renforcement des capacités des acteurs impliqués dans la prise en charge de la malnutrition aigue à travers l’organisation des formations de recyclage pour les anciens (tous les relais communautaires et le personnel de santé) et la formation des nouveaux agents de Santé en nutrition (Actions essentielles en Nutrition) et sur le protocole national de prise en charge de nutrition) • En collaboration avec les structures sanitaires, mettre en place un système de monitorage régulier des activités de nutrition développées sur le terrain. • Mettre en place un système de motivation des relais communautaires pour assurer la durabilité de leurs activités sur le terrain Les principales suggestions ou recommandations formulées par les bénéficiaires dans le cadre de l’amélioration des activités du projet • La reprise du projet Nema pour la poursuite des activités d’assistance en faveur de nos enfants malnourris. • Reprise des activités par le projet Nema et renforcer la formation des mères sur la nutrition des enfants <> • Nous sollicitons le retour du projet pour reprendre les foyers et poursuivre la sensibilisation des mères dans le village en faisant des témoignages pour amener les autres mères a adopté les bonnes pratiques. Source: Key informant and focus group discussion. Nema Final Evaluation, November 2013 Nema Final Qualitative Evaluation. Chapter 3. SO2. December 30, 2013. Draft. 77 Appendix I, Chapter 3: Evolution of SO2 Activities Year 1 Year 2 Year 3 Year 4 A. Rehabilitation of moderate acute malnourished children Training of CSREFs/ CSCOMs 35 105 28 105 CSCOM equipment 28 29 29 Creating hangars for CSCOMs 17 12 CHV training 670 645 CHV equipment (Shakir, one register per village) 650 630 SILC training (in listening groups and distribution of ENA equipment) 20 40 80 Number of people following literacy program (four months): 90% of women with an emphasis on sanitation and hygiene themes (sanitation, nutrition, management of drinking water) Training of radio hosts (training and messages—the design, translation, and dissemination of messages in local languages) 15 15 Creating radio messages 18 35 Screening for PECIMA (children) 0 575 12196 5094 Number of CSCOMs that received food to support the rehabilitation of malnourished children 25 28 29 29 B. PD/Hearth Preparation of villages for PD/Hearth activities 23 18 Training of volunteer mothers and health committees (# trained) 74 433 139 Number of PD/Hearth sessions (achieved/planned) 11/30 49/52 18/36 Supervision (number of trips) by the adviser on health /nutrition (HKI) 2 4 ¼ Programs to encourage the volunteer mothers 74 433 0 Training of health workers on PD/Hearth 58 C. Latrines/CLTS Training of masons 13 Providing the trained masons with equipment 10 Construction of model platform latrines 14 10/26 Initial training (aka “triggering”) of CLTS community-based programs (Number of villages) 4 16 Certification of CLTS villages (out of number trained) 2/20 Latrines of CLTS (traditional latrines constructed by the community members after the initial CLTS public-awareness building) 142 190 Water Points AES (temporary water supply) 3 Wells 3 7 4 Wells drilled 6 11 Water supply committees trained 17 13 Source: Nema M&E data, November 2013 Nema Final Qualitative Evaluation. Chapter 4. SO3. December 30, 2013. Draft. 78 Chapter 4 SO3: Targeted Communities Manage Shocks More Effectively69 This chapter provides a brief overview of the results of the Final Qualitative Survey and Evaluation for the Nema program’s Strategic Objective 3 (SO3) activities. The chapter is divided into five sections. Section 1.0. SO3 Strategy/Theory of Change: The intervention strategy and key actors involved in executing specific components of the strategy to achieve two intermediate results (IRs): • IR 3.1. Community early warning and response systems are in place; and • IR 3.2. Community safety nets are in place. Section 2.0. Evolution of the Nema SO3 Strategy and Activities (Years 1-4): The evolution of the implementation strategy for the three principal components of the program: the Early Warning Group/Emergency Response (Groupements d’Alerte Precoce et Reponse d’Urgence or GAP/RU), Food for Work (FFW), and safety net activities. Section 3.0. Early Evidence of Effects and Results of Nema’s SO3 Activities on the Establishment of Sustainable Community Early Warning and Response Systems (Years 1- 4): Some of the ways that the SO3 activities helped the local people manage a series of crises that started in June 2011 and continue to the present time. 70 Section 4.0. Current Level of the SO3 Activities Supported by Nema (November 2013): • 4.1. The current level of activity in the villages based on the results of the qualitative survey (November 2013) for the principal categories of activities including: - FFW-created infrastructure; - Community-based early warning systems; and - Community-based safety nets; and • 4.2. A summary assessment of some of the key factors that affected these outcomes. 71 Section 5.0. Lessons Learned and Recommendations: A list of recommendations that the evaluation team—working in close collaboration with the key implementation and execution partners—have identified for: 69 This chapter was prepared with the assistance of an informal Nema SO3 working group that developed around the final evaluation team’s field and key informant interviews. This working group included: Adama Sangare (CRS); Pierre Togo (Caritas/Mali); Abdoussalam Tiemogo Maiga (SCI/Gao); and Ahmadou Tadina (Agricultural Sector Chief, Douentza). The working group was supported in its analysis of previous and current activities by Boureima Sacko and Isack Dolo (M&E Department CRS), who co-facilitated the field portion of the final qualitative survey exercise with Dr. Sidikiba Sidibe (Consultant, Nema) and Abdoussalam Tiemogo Maiga (SCI/Gao) in Bourem. Della McMillan (Consultant, Nema) provided analytical and editorial support to the team’s activities. 70 See key questions on resilience (see Table 1.2 in Chapter 1). 71 I.e. The results and effects, the effectiveness of the implementation process, the sustainability of the results, and the activities impact on the local people’s resilience. Nema Final Qualitative Evaluation. Chapter 4. SO3. December 30, 2013. Draft. 79 • Corrective actions recommended for increasing the sustainability of the program impacts; and • Strengthening the benefits initiated by the program. 1.0. SO3 Strategy/Theory of Change Since 2007, the USAID has encouraged all of the Title II food security programs to include a set of activities designed to build the capacity of local communities to better manage risks and shocks. To address this issue, most Title II programs designed since 2007 have included: • GAP/RU: The creation of some variant of a community-based early warning and response system; • FFW: A system for planning technically and socially acceptable programs trading work for food; and • Safety Nets: A culturally appropriate system for safety nets for vulnerable groups that usually includes some combination of: - Direct distribution to help vulnerable groups complement their short-term food needs; and - Individual FFW to help the vulnerable groups develop infrastructure and activities that will help them develop income-generating activities (IGAs) that they can manage with their limited labor; as well as • Scale-Up Plan: A plan for scaling this combined package up or down depending on the trigger indicators for assistance. Given the Douentza and the Bourem region’s long history of drought and emergency response— in particular the very recent devastating drought that followed the 2004-2005 locus epidemics in both areas—the Nema program focused on the design of an integrated GAP/RU that could be scaled up or down depending on need. The original plan, which was outlined in the proposal, anticipated the creation of GAP/RU in each of the 130 villages. The role of the GAP/RU was to collect information that they could use to help the communities anticipate crises and develop appropriate internal responses, and to transfer this information to the commune-level early warning systems (Système d’Alerte Précoce or SAPs), whose role was to monitor food security risks and coordinate any larger-scale emergency response that might be needed to a major crisis. This section of the Nema proposal is very well designed and included a very well-thought-out plan for: • The development of the GAP/RUs and their relationship to the SAPs; • An extensive program to development FFW activities that would help the local communities develop more profitable agro-enterprise activities and reduce routine risk by infrastructure development that reduced erosion, flooding, and other period risks; • The creation of a Safety Net Committee comprised of village leaders whose role would be to: - Identify Vulnerable Households and Rations to be Distributed: Develop a prioritized list of vulnerable groups that would receive a daily ration for three Nema Final Qualitative Evaluation. Chapter 4. SO3. December 30, 2013. Draft. 80 months during the soudure (hungry period before the harvests).72 In contrast to many earlier programs, the Nema program provided a donation for the family members of vulnerable persons as well as for their rehabilitation; and - Coordinate Distribution: Discreetly manage the appropriate ration size to the vulnerable individuals and their families in ways that minimized any shame or social conflict related to the safety net. Although the short-term focus of the safety net program was on relieving the severe food constraints that many of these households faced, the program hoped that the training the committees received would help these community leaders strengthen the traditional systems for helping vulnerable groups once the program ended. The same baseline document included a well-thought-out plan for how these activities SO3 activities—as well as the Prise en Charge Intégrée de la Malnutrition Aiguë (PECIMA) component of SO2—could be scaled up or decreased if a major risk hit that affected more than 25% of the population. These SO3 activities (FFW, GAP/RU, safety nets) were expected to have a host of short-term and longer-term impacts. In the short-term, they were expected to facilitate the adoption of more crop technologies and IGAs by improving basic infrastructure and reducing the drain on household investments and labor created by looking after very vulnerable groups. The impacts of these activities on household food security, income growth, and vulnerability were measured by the SO1 Indicator Performance Tracking Table (IPTT) indicators (see Chapter 2). The same SO3 activities (FFW, GAP/RU, safety nets) were expected to increase the long-term sustainability of these impacts under the program by setting up a sustainable system of community early warning and response systems and community-based safety nets. To insure appropriate tracking, the IPTT included a series of indicators that tracked the creation of these two institutions—the GAP/RU (Monitoring Indicators 3.1.1 and 3.1.2) and Safety Net Committees (Comite de Filet de Securite) (Monitoring Indicators 3.2.1 and 3.2.2). 72 Distribution made in June to cover July, August, and September. Nema Final Qualitative Evaluation. Chapter 4. SO3. December 30, 2013. Draft. 81 Figure 4.1. Nema SO3 Strategy/Theory of Change Table 4.1. Impact and Monitoring Indicators Used to Track the Progress of the Nema Program’s Intermediate Results in Its IPTT Indicator Year 2 Year 3 Year 4 IR 3.1. Community early warning and response systems are in place GAP/RU II 3.1. % of Title II-assisted communities with disaster early warning systems in place (FFP) 30% 55% 0 MI 3.1.1. % of HOH (sic) in targeted communities who can cite at least two concrete strategies the community is using to improve their resiliency to future shocks ND ND ND MI 3.1.2. # of community groups that are actively tracking trigger indicators 26 67 0 FFW II 3.2. % of Title II-assisted communities with improved physical infrastructure to mitigate the impact of shocks 32.3% 82% ND MI 1.1.3. # of beneficiary farmers accessing improved agro-sylvo-pastoral infrastructure (created with FFW) 35,210 66,746 0 MI 1.1.4. # of improved infrastructures completed (disaggregated by type of infrastructure: km of road, number of bridges, hectares irrigated, etc.) 42 143 0 MI 1.1.9. # of mitigation actions to prevent or reduce natural resource degradation implemented 11 18 0 MI 3.1.3.-3.1.9. Environmental indicators IR 3.2. Community safety nets are in place MI 3.2.1. Total # of assisted communities with safety nets in place to address the needs of the most vulnerable members (FFP) 96 130 130 MI 3.2.2. Total # of communities who strengthen safety nets over the life of the activity, as shown by the reported increase in the diversity of shocks the safety net is capable of responding to (FFP) 42 130 130 MI 1.1.7. # of vulnerable households benefitting directly from USG assistance (safety nets) 3,438 3,859 3,640 Source: Nema program IPTT, Annex I. Acronyms: II: Impact Indicator; MI: Monitoring Indicator; FFP: Food for Peace; USG: U.S. Government; ND: No data. Nema Final Qualitative Evaluation. Chapter 4. SO3. December 30, 2013. Draft. 82 2.0. Evolution of the Nema SO3 Strategy and Activities 2.1. IR 3.1. Community Early Warning and Response Systems are in Place 2.1.1. GAP/RU: The process of setting up and training the GAP/RU started in the second year. Each village was expected to nominate a committee of between seven and 12 persons per group, including at least one woman, who was usually a community health volunteer (CHV or relais) or the village representative for women. Once the GAP/RU committee was identified, members received a three-day training. In Douentza, this initial training was conducted by a sub-contract to the Ministry of Agriculture in the first years. Unfortunately, since the concept was very new and extension agents doing the training had no experience in GAP/RU, this initial training was very weak and had to be redone in the second year with strong input from Caritas/Mali and extensive input from the senior SO3 advisor for the Consortium. Tassaght was responsible for the initial selection and training of the GAP/RU at Bourem. Despite the fact that they were not involved in either the initial selection or training of the GAP/RU groups, the Bourem administrators supported their participation in their development and participation in the cercle-level meetings. Thus most of the Bourem GAP/RU groups were up and running by the second year. Faced with the regional SAP’s unwillingness to let the GAP/RU committee members attend their meetings at Douentza, the SO3 Consortium advisor from CRS and the Caritas program manager (PM), who managed all of the program activities at Bourem, were forced to spend a great deal of time advocating on behalf of their inclusion. The groups were finally allowed to attend the commune-level SAP meetings in the third year. At the time of the mid-term evaluation, 79 of the 130 GAP/RUs were considered ‘functional’ in terms of (a) organizing monthly meetings; (b) collecting information and putting it in the notebooks; (c) participating in the commune-level meetings of the SAP; (d) collecting regular rainfall data; (e) sharing the information gathered on the ‘trigger indicators’ with their communities; and (f) assisting the community in the event of a shock.73 One strength of the Nema monitoring and evaluation (M&E) system was its consistent collection and analysis of the data needed to monitor the major monitoring indicators (MIs) that were designed to track the efficiency and results of the GAP/RUs as they developed (Tables 4.1 and 4.2).74 2.1.2. FFW: A total of 143 infrastructures (114 agro-sylvo-pastoral infrastructures and 29 roads and bridges) were built with FFW (Table 4.2). Each FFW program was managed by a FFW management committee. Once the activity was terminated, the village was expected to create a separate committee to maintain the infrastructure. 73 K. Tilford. Nema Mid-Term Evaluation. Page 31. 74 One group of internal indicators was collected by the CDAs and transmitted directly to the SO3 specialist and the executing partner PMs on all 130 the GAP/RUs. Each year, the Consortium’s M&E specialist conducted an independent audit of these numbers on about half the GAP/RUs during his annual field visits. Nema Final Qualitative Evaluation. Chapter 4. SO3. December 30, 2013. Draft. 83 Table 4.2. Major SO3 Achievements During Nema, Years 1-4 Activities LOA Target Achieved (Year 4) GAP/RU linked to the regional-level early warning system (System d’Alerte Precoce or SAP) 130 84 Bourem 30 28 Douentza 100 56 Community infrastructure in place to increase production, avoid shocks, and reduce erosion through FFW 260 143 Number of vulnerable individuals targeted for safety net programs using specific criteria 14,362 8,726 Bourem 4,309 2,618 Douentza 10,053 6,108 Source: Nema program data, November 2013. Acronym: LOA: Life of Activity. 2.2. IR 3.2. Community Safety Nets are in Place In the past, the principal conduit of food assistance in the program intervention areas was the chief, who would usually distribute the food to a group of households that would be co￾determined with his counselors. One objective of the CRS safety-net program was to try to strengthen this traditional system by training village leaders to use a simple but very rigorous system for identifying the most vulnerable groups and determining an appropriate ration size to the most vulnerable 2-3% of the local community. Each safety net group was trained by the Consortium’s three SO3 community development agents (CDAs, also known as Agents de Développement Communautaire or ADCs).75 The fact that the first non-emergency food distribution followed one month after the second training helped hone some of the training messages and gave a great sense of importance to this group. A total of two ‘routine’ emergency safety net distributions were executed during the soudure in Years 2 and 3 of the program. The first emergency safety net (which was the third safety net executed by the Nema program) was designed in October 2011 and executed in January-March 2012 to provide safety nets for vulnerable households in 104 of the Nema villages that were most seriously affected by the 2011 drought (see Section 3.1. below).76 2.3. Mid-Term Evaluation Assessment and Recommendations The mid-term evaluation was not optimistic about the long-term sustainability of the GAP/RU system, but made a list of recommendations for strengthening the groups’ functioning. The mid￾term was very positive, however, about the technical quality and prospects for sustaining the infrastructure created with the program’s FFW activities and the way the safety net program was targeting vulnerable groups (Text Box 4.1). 75 The program had two SO3 CDAs for the 100 villages at Douentza and one SO3 CDA for the 30 Bourem villages. 76 Nema. 2001. Rapport Situationnel/Douentza/Bourem/Mali. 2 Novembre 2011. Nema Final Qualitative Evaluation. Chapter 4. SO3. December 30, 2013. Draft. 84 Text Box 4.1. Major Conclusions of the Mid-Term Evaluation Concerning SO3 GAP/RU: “The interviews with EWG (Early Warning Group) members during the field survey showed that the EWGs do understand how the information they collect is used at a higher level. The members are especially interested in tracking rainfall…..Fortunately, no major shocks have occurred recently in the MYAP communities, but the absence of shocks makes it difficult to assess whether the EWGs [early warning groups] and the communities as a whole have actually improved their ability to identify and respond to shocks. ….The main concern with the EWS is whether they will continue to carry out their functions once the MYAP ends.” FFW: “Most of the programs are directly related to improving livelihoods…Other programs protect the communities from flooding or from the encroachment of sand dunes. ….This activity is one of the most successful and appreciated by the beneficiaries for two reasons. First, the food is provided during the ‘hungry season.’….Second, the type of infrastructure to be built is chosen by the community, and almost all programs benefit the entire community in one way or another. The exception is the irrigated perimeters where the number of plots might be limited.” Safety Nets: “The major issue with this activity is sustainability once the FFP rations are no longer available. Given how poor most people are, there is a certain reluctance to donate part of one’s harvests to neighbors and others who are casual acquaintance…For this reason it is highly unlikely that all 130 villages will set up their own system before the program ends.” Source: K. Tilford, 2011. Nema Mid-Term Evaluation. Pp. 32-33. 3.0. Early Evidence of Impact on the Establishment of Sustainable GAP/RUs in the Target Villages The true results and effects of an effective early warning and response system are: • How it reacts when an actual emergency occurs; and • How it performs when the program that created it is no longer there to assist it. To date, there are very few of the Title II-funded programs that have been tested to the degree that Nema was tested starting in June 2011, at the very end of year three just after the mid-term evaluation. For all this reasons, USAID/FFP is very interested in assessing how the early warning system that was set up in Years 1-3 actually functioned when a real crisis hit. 3.1. Results During Nema —Efficient Preparation of a Relief Plan in Response to Data from the GAP/RU The first evidence of a major drought started filtering through their GAP/RU reports to Caritas and SCI in June and July 2011. Based on this information, the Consortium’s SO3 specialist made a series of field visits to verify the information, including a full field visit in October 2011 in which he and the SCI, Caritas, and Tassaght staff identified a list of the 104 (80%) most vulnerable villages. With input from some of the better-organized GAP/RU committees, the consortium prepared a “bump-up” plan that was submitted to USAID that basically followed the “scale-up” plan that had been outlined in the original Nema proposal.77 Specifically, the “bump up” anticipated: 77 Nema. 2001. Rapport Situationnel/Douentz/Bourem/Mali. 2 Novembre 2011. Nema Final Qualitative Evaluation. Chapter 4. SO3. December 30, 2013. Draft. 85 • Doubling the amount of FFW in all villages—not just the most vulnerable ones—during the soudure months in two periods (January–March and April-June); • Doubling the amount of food aid for the regular PECIMA program (which was scheduled to start in 2012) for up to 10,800 children instead of 5,400; and • Doubling the amount of food budgeted to the direct distribution safety nets. The fact that the program could prepare such a solid, well-researched proposal in only two months is a vivid indicator of the degree to which the Nema training had built the capacity of some of the better organized GAP/RUs (that were able to help), the Safety Net Committees, and the Consortium’s local executing partners, Caritas and Tassaght. It is a testament to the quality of the proposal that the first shipment of supplementary food assistance and cash arrived within two months (January 2012). Another output was the team’s effective mobilization of an emergency safety net program in January-March 2012 in the hopes of stabilizing the male work force until the emergency FFW program could be geared up for all 130 (100%) of the villages. The speed and effectiveness of the response was even more important given the fact that it overlapped with an even larger crisis that was clearly starting to emerge in January 2012.78 3.2. Results During Nema—Partial Execution of a Relief Program in the Douentza Villages Once the Mali government fell in March 2012, most government services in Douentza and Bourem shut down. This, in combination with the progressive military occupation of both cercles, forced both local executing partners (Tassaght and Caritas) to shut down their local offices. The Caritas office in Douentza was moved back to Mopti in March. These events— combined with agency (CRS, SCI, and HKI) and US government security measures for the north—meant that Nema’s carefully orchestrated emergency relief plan (plan d’urgence) for the 2011 drought was only partially executed. That is to say, Nema was only able to execute the very first part of the relief program—i.e. the first three-month safety net (January-March) and an initial increase in the amount of food given to the community health centers (Centres de Santé Communautaire or CSCOMs) for treating malnourished children. 3.3. Results Post-Nema—Nema-Trained Safety Net Committees Helped Facilitate the Allocation of Other Humanitarian Aid During the Crisis The full impact of the Nema training of the community-based Nema Safety Net Committees became apparent when the first non-Nema food assistance to the area started filtering in, which was not until January 2013 (Table 4.3). 78 The first real attacks in northern Mali started in January 2012 as the arms from the demobilized Libyan troops started moving into the far north. These arms ignited age-old animosities that quickly swirled into series of rebellions and occupations in the north. Faced with growing opposition to the way the government was managing this second emerging crisis, the military staged a coup d’etat in Bamako March 22nd, which accelerated the speed with various armed groups, started moving south in hopes of occupying Mopti, which is the capital of the region that the cercles of Bourem and Douentza report to. Nema Final Qualitative Evaluation. Chapter 4. SO3. December 30, 2013. Draft. 86 Table 4.3. Evidence of Post-Nema Food Distributions in the Nema Villages, September 2012-present Humanitarian Agency Distributing Food Aid % of Population Targeted by Distribution (% of Most Vulnerable Households) Number of Villages % of Nema Villages Covered by Distribution Douentza (100 Nema Villages) January-February 2013 IFRC 13% (during 2 months) 100 100% March-October 2013 WFP 33% (during 6 months) 24 IFRC 31% (during 6 months) 76 Bourem (30 Nema Villages) OXFAM/WFP (July-Dec 2013) 30-40% (during 6 months) 27 90% IFRC (Sept 2013) 100% (during 1 month) 30 100% ADESAH (Sept-Nov 2012; Sept￾Nov 2013) 50% (during 3 months) 3 10% Source: Key informant interviews, Ministry of Agriculture staff, Nema Final Evaluation, November 2013. Acronyms: IFRC: International Federation of the Red Cross; WFP: World Food Program; OXFAM: Oxfam International; ADESAH: Association for the Endogenous Development of the Sahel Based on feedback from the focus group and key informant discussions during the evaluation, the training received under Nema helped these Safety Net Committees: (a) develop a draft list79 of vulnerable households, which complied with the humanitarian donors (i.e. 13-20% of the population); (b) make sure that the ration complied with donor requirements; and (c) (very important) ensure that the donor got the records that the rations were received by the appropriate category of beneficiary. There is anecdotal evidence that this increased village-level capacity helped some of the Food Security Committees attract smaller food-aid allotments as they came in over and above the main distributions, such as restricted allotments for specific groups like internally displaced persons (IDPs). 4.0. Current Level of the SO3 Activities Supported by Nema 4.1. Food for Work 4.1.1. Current Level of Functioning of the FFW-Supported Activities Designed to Reduce Environmental Risk and Promote Agro-Sylvo-Pastoral Enterprise Development: Out of the seven non-FFW-facilitated infrastructures visited by the final evaluation survey team, 100% were still functional. Out of the 26 FFW facilitated infrastructures visited, 78% were functional (Table 4.4). In general, however, the level of maintenance is sub-standard and most maintenance committees do not appear to function. 79 It appears that this draft list was often just a revised list of the original list of the vulnerable groups used to target the most vulnerable households in Years 2-3 of Nema. Nema Final Qualitative Evaluation. Chapter 4. SO3. December 30, 2013. Draft. 87 Table 4.4. Number of Infrastructures Created, Direct Beneficiaries, and Current Level of Functioning of the FFW-Funded Infrastructure Created by the Nema Program, November 2013 Activity/Indicator During the Program (by Year) After Nema (November 2013) 2 3 4 Total # Visited During Final Survey # Visited That Were Assessed as Functional # of persons benefitting from FFW Men 28,870 Women 30,049 Non-FFW infrastructure created 7/7 (100%) Sanitation and hygiene infrastructure Incomplete information 5 5/5 Drinking water infrastructure 15 2 2/2 FFW infrastructure created # FFW-facilitated infrastructures created 65 78 143 26 (18% of created) 19/26 (78%) FFW-facilitated infrastructure for agro-sylvo-pastoral and gardening activities 49 65 114/143 21 (15/21) FFW-facilitated other (bridges, rural roads) 16 13 29/143 4 (4/4) Source: Field visits to observe infrastructure, Nema Final Evaluation Survey, November 2013. 4.1.2. Summary Assessment of Strengths, Areas that Need Strengthening, and Opportunities in the Current Context: 80 • Beneficiary, Mid-Term, and Partner Assessments: In general, the villagers interviewed during the final evaluation survey expressed their deep appreciation for the FFW—both as a food supplement and as a source of investment in infrastructure that benefitted (in most cases) the entire village (Text Box 4.2). Almost all of the key informants interviewed during the final evaluation concurred with the conclusion expressed in the mid-term evaluation that this was “one of the most successful program interventions.”81 • All Infrastructure Activities Integrated into the Government Commune Plans: A major strength was that all activities integrated into the previously created commune development plans. If any infrastructure was not identified before the program, Nema made sure it was added in order to strengthen the prospects for long-term maintenance. • Delays in Materials: The chief complaint about the FFW was substantial delays in the provision of some of the materials. This point was also raised in the mid-term evaluation.82 • Gender Roles and Relations: The Consortium’s insistence that all programs respect the target for gender has resulted in a high percentage of the direct beneficiaries being women. • Impact on Vulnerable Households: All of the focus groups underscored the critical importance of FFW on household food security (Text Box 4.2). One challenge for the 80 I.e. The results and effects, the effectiveness of the implementation process, the sustainability of the results, and the activities impact on the local people’s resilience. 81 K. Tilford. 2011. Nema Mid-Term Evaluation, Page 32. 82 K. Tilford. 2011. Nema Mid-Term Evaluation. Page 32. Nema Final Qualitative Evaluation. Chapter 4. SO3. December 30, 2013. Draft. 88 next phase of the program should be to create a new sub-category of FFW that would create new opportunities for vulnerable women (as both individuals and groups) to use FFW to develop small IGAs that would reduce their long-term dependency on both the traditional and donor supported safety net programs. Text Box 4.2. Beneficiary Responses to Questions During the FFW Focus Groups Quels sont les avantages tirés des activités FFW: • Dans notre village (Oualo), nous avons eu des avantages doubles (Double gains) dans le cadre de vivre contre le travail, nous avons bénéficié des ouvrages (Construction du Radie, Périmètre Maraicher et deux puits) et des vivres. La construction des ouvrages nous a permis de réduire les difficultés pour joindre le village pendant la saison hivernage. L’enseignant du village affirme que la réalisation du radier a eu un impact sa consommation en Carburant. Cette consommation est passée de 1 litre avant la construction des radiers à 0,5 litres après la construction. Sans le radiers, ils parcourraient 6 km pour atteindre le village mais après la construction, la distance a été réduite à moitie (3 km seulement à parcourir). La clôture du périmètre maraicher a permis de diminuer la déforestation, la coupure des arbres et également la protection du périmètre contre la divagation des animaux. • Village Fombory : Le vivre pour travail a permis de réduire l’impact de la période de soudure dans certains ménages qui ont bénéficié des vivres. Les vivres sont venus trouver que certaines famille n’avaient pas a manger mais ces famille ont bénéficie des vivres pendant le travail et cela leur a permis de cultiver les champs et éviter l’humiliation vis a vis des autres ménages et de leur semblables. • Village Kiro : Les activités de Vivres contre le travail nous a apporté beaucoup d’avantage. En plus des vivres que nous avons bénéficié, le FFW nous a permis d’aménager nos périmètres rizicoles. Cette distribution des vivres nous a permis pendant les période difficiles (période de soudure) de couvrir nos besoins alimentaires et de faire des travaux physiques dans nos champs. En plus de ces avantages, l’activité aura permis de retenir les jeunes dans le village en freinage l’exode rural et d’avoir des connaissances en matière de nouvelles techniques pour la riziculture. Il y a eu aussi un renforcement de la solidarité entre les membres de la communauté pour entreprendre ensemble des activités d’intérêt commun. • Village Dalla : C’était l’une des meilleures stratégies du projet du projet Nema, car toutes les activités ont été menées pour les communautés avec la participation active. La distribution des vivres a surtout créé un engouement populaire et une forte motivation de la communautaire pour la réalisation des infrastructures communautaire. • Les vivres ont contribué à la satisfaction de leur besoin. Les vivres sont venus de la période de soudure • Réduction de l’exode rurale (nous avons constaté la réduction de l’exode rurale des jeunes dans le village • La construction de la digue a permis de sécuriser le village contre les inondations pendant les saisons pluvieuses et augmenter les surfaces cultivables a travers la déviation de l’eau par la déviation • Village Bore : l’aménagement des points de retenus d’eau dans la marre a permis d’augmenter la disponibilité d’eau dans la mare (cours d’eau) pendant les 12 mois de l’année. Nous produisons les pépinières de riz pour les transplanter pendant la période culturale. Il y a également une disponibilité d’eau pour les femmes pour faire les lessives et pour entreprendre les activités maraîchères pendant la saison sèche. Les caïmans sont préservés et restent dans la retenue d’eau pendant toute l’année. Avant la construction des retenues d’eau, les caïmans n’étaient pas protéger, et beaucoup d’entre eux mouraient pendant la saison sèche (Avril et Mai). Actuellement, les avantages de la retenue d’eau ont dépassé nos attentes. Les vivres contre le travail, nous a également permis de bénéficier les vivres et des matériels de travail. Fombory, Dalla , Kiro: • la période de distribution des vivres coïncident a la période des vivres • Double bénéficie, le village a bénéficié de l’infrastructure et rations alimentaires • Engouement de la communauté au tour des FFW • Les vivres sont venus pendant la période de soudure Quelles suggestions et recommandations faites-vous pour renforcer les acquis du projet: • Village Kiro : La difficulté majeure était que nos femmes ne savaient pas comment préparer certains aliments qui n’étaient pas dans nos habitudes alimentaires (comme le petit pois et la semoule). Les vivres ont été également livrés seulement après les activités. Les matériels sont également arrivés en retard dans le village. Ces matériels n’étaient pas résistant, de bonne qualité et n’étaient pas adapté à nos sols argileux : • Village Bore : L’Herbe plante en bordure de la mare n’a pas reçu à cause de la crise Nema Final Qualitative Evaluation. Chapter 4. SO3. December 30, 2013. Draft. 89 Villages Fombory, Dalla • Augmenter la quantité du matériel distribué • Les ouvriers locaux doivent être accompagnés par les œuvres qualifiés pour assurer leur formation • Les abreuvoirs sont très profonds et ne respectent pas les normes techniques Village Dalla • Augmenter la quantité de la ration alimentaire et Ajouter l’Huile • Reprendre la suite des travaux pour finaliser la digue • Retour immédiat du projet Nema et les aider a augmenter la superficie du périmètre marcher • Augmenter la longueur du Radier et augmenter la quantité des vivres a distribuer • Village Kirio : Augmenter la profondeur de la retenue d’eau et revoir la composition des aliments en fonction des habitudes alimentaires de la population. Leçons apprises/ Témoignage/ proverbes. Témoignage sur le Food for Work : Focus avec les Sages du Villages Dallah Selon les sages, les activités du FFW ont eu un impact significatif sur le niveau de sécurité alimentaire de leur village. Les travaux de FFW ont Coïncidée au début de la période de soudure dans le village. « Dans notre village, le vivre contre le travail a permis à certains ménagé de protéger les semences pour la production agricole, le début de FFW a trouvé que certains ménages avaient déjà commencé à consommer une partie de leur semence ». Parlant de l’impact des digues, malgré que l’inachèvement de l’infrastructure, les sages étaient tous unanimes pour affirmer que la réalisation des digues a créé de la quiétude dans le village en réduisant la peur constante qui était liée à des inondations répétées des habitations dont les ménages étaient victimes. Pour les sages, chaque année le village était victime d’inondation avec la destruction des habitations et des greniers. « Grace à Dieu et l’intervention du projet, le phénomène d’inondation répétée a été maitrisée et cette digue joue un grand rôle dans le village, nous sollicitons au près du projet, pour la reprise des travaux afin d’achever cet ouvrage si important pour le village. Source: FFW Focus Groups, Nema Final Qualitative Evaluation. November, 2013. 4.2. Community-Based Early Warning and Response Systems 4.2.1. Current Level of Functioning of the Community-Based Early Warning and Response Systems: • Limited Functioning of the GAP/RU Committees as Intended in the Proposal: Only eight (38%) of the 21 GAP/RU committees contacted during the final evaluation survey are considered ‘functional’ in the sense that they still collect and report rainfall data in the local community; only one of the 21 still transmits data to the SAP (Table 4.5). Most committees are referred within the village as the meteo (rainfall gauge), which reflects the villagers’ opinion that their principal role is for data collection, not response. • Limited Functioning of the Safety Net Committees as Intended in the Proposal but New Role in Orchestrating Emergency Response: Although only 33% of the Safety Net Committees were considered functional in terms of the original purpose that they were supposed to perform (Table 4.5), they have become the principal point of contact for most of humanitarian food assistance coming in (Table 4.4 above). Nema Final Qualitative Evaluation. Chapter 4. SO3. December 30, 2013. Draft. 90 Table 4.5. Functionality of the GAP/RU and Safety Net Committees Interviewed During the Nema Program Final Evaluation Survey, November 2013 Current Level of Functioning Bourem (n*=6 interviewed) Douentza (n=15 interviewed) Total # % of total interviewed # % of total interviewed GAP/RU Committees 1. Strong 0 0 2. Average 0 0 3. Weak 1 17% 7 47% 8 functional but weak (38%) 4. Non-Functional 5 83% 8 53% 13 non-functional (62%) Safety Net Committees (Filets de Securite) 1. Strong 2. Average 3. Weak 6 100% 1 7% 7 (33% functional but weak) 4. Non-Functional (Traditional System Only) 14 93% 14 (67%) non￾functional Source and Methodology: Nema Final Evaluation Survey based on the responses given during the community focus groups, November 2013 *= Number of groups interviewed 4.2.2. Summary Assessment of Strengths, Areas that Need Strengthening, and Opportunities in the Current Context: • Very Good Training and Supervision of the Safety Net Committees during Nema: The Nema program provided very good baseline training and persistent, very careful, close supervision of the Safety Net Committees. This type of in-depth technical assistance was needed to insure that they were able to do their job. • All of the Nema Villages Benefitted: All of the communities benefited from both types of support that currently provide the nucleus around which the current de-facto Food Security Committees are coagulating—the Safety Net Committees and the GAP/RUs. • Limited Involvement of the SAPs in the Initial Set Up of the GAP/RU Committees Reduced the Committees’ Effectiveness in Douentza: One of the chief weaknesses associated was the weak implementation of the SAPs in the initial set up of the GAP/RU committees in Douentza. This reduced their support from and contribution to the SAPs. • Unintended Impacts—Managing a Major Crisis: In the final analysis, the Nema GAP/RU system did what it was supposed to do—it monitored emerging risks and facilitated the preparation of an emergency response, which was successfully submitted to USAID for funding in October/November 2012. This type of community-based system was never designed to identify or manage a major crisis with Title II or another type of donor support. However, the Nema support did help these communities manage the food assistance that they needed to handle the major crisis when it arrived. • Limited Integration of the GAP/RU and Safety Net Programs with Other Emergency Response Systems in the Target Areas: This assistance could have been more efficient and quicker had the community-based systems been better linked to some of the other emergency systems that the government and the Red Cross/Mali had set up in Douentza and Bourem. This would have had two benefits: It would have better-integrated the training and response systems with the pre-existing SAP and Red Cross/Mali system of Nema Final Qualitative Evaluation. Chapter 4. SO3. December 30, 2013. Draft. 91 volunteers that were critical players in the crisis response, and would have enabled the Consortium to pre-negotiate certain types of memorandums of understanding (MOUs) for collaboration, which might have gotten the food to the local people more efficiently and quickly once a major disaster struck. 4.3. Community Safety Nets 4.3.1. Current Level of Functioning: The final evaluation survey team conducted 20 interviews with representatives of the Safety Net Committees as part of the community-leader focus groups. These interviews confirmed that: (a) most of the committees still include their original members; (b) they continue to oversee certain traditional mechanisms for helping vulnerable groups; and (c) hey are the main point of point of contact for distributing food aid if and when it comes to the villages. They also confirm the communities’ deep appreciation for the safety nets (Text Box 4.3). The interviews also confirmed the suspicion that was voiced in the mid-term evaluation that: “It is very unlikely that all 130 villages will set up their own system before the program ends.” 83 To date, only one of the committees interviewed has been able to do this through a link up to one of the associations des resortissants (expatriate clubs) that supports certain village charities (Table 4.6). Table 4.6. Evolution of the Nema Safety Net and GAP/RU Activities, March 2012-November 2013 Activity/Indicator After Nema Actual (November 2013) Year 2 Year 3 Year 4 (January) MI 3.2.1.Total # of assisted communities with safety nets in place to address needs of the most vulnerable members 96/130 villages 130/130 villages 130/130 villages All but one of the villages visited in the final survey have reverted to traditional safety nets with one exception that is mobilizing expatriate support MI 1.1.7. # of vulnerable households benefitting from USG assistance via safety nets 3, 438 3,859 3,640 0 II 3.1.% of Title II assisted communites with disaster early warning systems in place (FFP). 84 n/a 30% 65%* Only one of the GAP/RU in the village of Boumbam functions fully—i.e. relays the data it collects to the SAP. The others continue to collect the data, which they convey to their fellow villagers through local radio or verbal communication. Source: Nema program data and the results of the Nema Final Qualitative Survey, November 2013. *The figure in the IPTT (Annex I) is 52%. The correct figure according to staff is 84/13o or 65%. 83 K. Tilford. 2011. Nema Mid-Term Evaluation. Pg. 33. 84 “Functional groups consist of those who meet regularly, collect monthly data, submit monthly reports to SAP, and who intervene in the event of a shock.” (Annex I, Nema IPTT, Page 10). Nema Final Qualitative Evaluation. Chapter 4. SO3. December 30, 2013. Draft. 92 Text Box 4.3. Community Leader Observations About the Implementation of the Nema Safety Net Programs in the Cercle of Douentza Pouvez-vous décrire comment la sélection des bénéficiaires du filet de sécurité a été faite? Fambory, Ngono, Ouala , Kiro, Bore: • Les vulnérables sont choisis par le chef de village et ses conseiller • Réunion des conseils du village. Distribution des cartes aux bénéficiaires (Personnes vulnérables identifiées). • Village Oualo Réunion communautaire a travers une assemblée organisée par les leaders pour l’identification des personnes vulnérables dans le village (Ouala) en se basant sur les critères de vulnérabilité définie. Quels sont les critères de ciblages des bénéficiaires Nema (Fambory, Ngono, Ouala , Kiro): • Femmes veuves • Personnes âgées sans enfants • Personnes âgées abandonnées par leurs enfants dans le village. • Pauvres (N’ont pas à manger) • Handicapé • Victimes d’inondations Apres le retrait du projet la communauté a continué à utiliser les mêmes critères pour sélectionner les personnes vulnérables dans le village (Avec HCR et Islamique Relief) (Gono) Que pensez-vous de l’impact du filet de sécurité (distribution gratuite des vivres) mené dans le cadre du programme Nema dans votre village (Impact sur la vie des plus vulnérables)? Fambory, Ngono, Ouala , Kiro, Bore: • Maintenir les personnes vulnérables dans le village • permet aux personnes vulnérables d’économiser de l’argent pour faire face a d’autres problèmes à la scolarité de leur enfants, santé, fourniture scolaire pour les enfants • Ces personnes étaient une charge pour la communauté leur appui a diminué la charge de la communauté • Quiétude dans la communauté et l’esprit de solidarité envers les personnes vulnérables et renforcement de la cohésion sociale dans le village • Soutient des personnes vulnérables pendant la période de soudure. Appui des vulnérable sans aucune autre condition. • D’éviter de s’endetter, ou quémander et garder leur dignité humaine (Village Gono) et n’avait plus de souci pour la recherche des vivres. • (Oualo village) : Allègement de la charge sociale au niveau communautaire pour la prise en charge des personnes vulnérables • Diminuer les charges de la communauté envers les personnes vulnérables (Fambory) et la diminution de l’exode des bras valides Quelles ont été les faiblesses/ difficultés dans la mise en œuvre du filet de sécurité dans le cadre du Nema ? (noter toutes les faiblesses évoquées): • La crise est venue arrêtée le processus • Présence d’insectes dans la farine de Semoule • Village Kiro : Nous avons souvent rencontrée des difficultés dans la mise œuvre du filet de sécurité alimentaire pour les vulnérables. Si par exemple on écrit 25 personnes comme bénéficiaires, à l ‘arrivée des vivres, on reçoit seulement pour 20 personnes lors de la distribution. Les 5 autres personnes vulnérables identifiées ne bénéficiaient pas de ces vivres. Cela créait souvent des frustrations parmi ces personnes identifiées pour recevoir des vivres et ils retournent chez eux sans aucune ration alimentaire Au niveau communautaire, quelles sont les stratégies mise en place pour soutenir les plus vulnérables à l’absence du projet? (noter toutes les stratégies soulignées) Fambory, Ngono, Ouala , Kiro, Bore: • Avec le retrait brusque du projet Nema, comme dans la tradition, la communauté fait des gestes de solidarité envers ces personnes vulnérables Octroi des zakat et des zakats/aumônes (NGono) • Appui a travers les actions de solidarité traditionnelle (Aumône, • Village (Oualo) : La communauté continue avec le système traditionnel d’appui aux vulnérables a travers des Nema Final Qualitative Evaluation. Chapter 4. SO3. December 30, 2013. Draft. 93 aumônes, Zakat et des dons volontaires. • Soutien traditionnel (Fambory) : solidarité familiale • A Kiro, les vulnérables sont soutenus par toute la communauté à travers les actions de solidarité en les aidant à cultiver leur champ de la préparation des travaux jusqu’à la récolte des champs, en les appuyant pour les cérémonies de baptêmes et de mariage Quelles suggestions ou recommandations avez-vous dans le cadre de la réduction de l’insécurité alimentaire au niveau de groupes les plus vulnérables? Fambory, Ngono, Ouala , Kiro, Bore: • Les personnes vulnérables constituent une charge, leur soutient reste indispensable pour améliorer leur situation de sécurité alimentaire • Reprendre les activités au village et Augmenter la quantité des vivres devant être distribués aux personnes vulnérables (Fambory, Ngono ; Kiro) • Le projet devrait mettre un accent sur les AGR en faveur des groupes vulnérables • Faciliter la mise en place d’une structure sanitaire dans le village pour augmenter l’accessibilité des femmes et des personnes vulnérables aux services de sante • Mettre en place des centre Alpha pour la formation des groupes cibles (Groupement, association) en Alphabétisation • Appui en intrants pour se prendre en charge après le retrait du projet (Fambory) • Formation des payants en technique d’aménagement des périmètres rizicoles et pour augmenter la profondeur des retenues d’eau pour la riziculture. (Village Kiro) Leçons apprises/ Témoignage/ proverbes.(Note : prêter attention aux récits pertinents par rapports aux réussites, histoires de changements) Le témoignage d’un leader : L’appui à travers le filet de sécurité aux vulnérables nous rappelle de l’appui de parents pendant la grande sècheresse survenue vers 1945 pendant la période coloniale. A cette époque, nous parents ont reçu des aides a travers la distribution des vivres. C’est qui avait sauvé tout le village de la faim. L’appui du Nema en faveur des personnes vulnérables ressemble à cette situation. Nous pouvons dire que l’appui du Nema a sauvé tout le village surtout les plus personnes vulnérables pendant les périodes difficiles. La charge de la population a été aussi allégée surtout pendant ces périodes difficiles. Nous sollicitons que Le Projet Nema reprenne ces activités dans notre village pour achever les travaux et les bonnes initiatives que nous avons entreprises ensemble. (Village Oualo) Source: Community focus groups, Nema Final Qualitative Evaluation, November 2013. 4.3.2. Summary Assessment of Strengths, Areas that Need Strengthening, and Opportunities in the Current Context: 85 4.3.2.1. General: • Culturally Appropriate Model Increased Efficiency and Sustainability During and After Nema: The Nema safety net model incorporated all of the same people that were involved in the execution of the traditional system, as well as one or two health volunteers (relais) as the official representative of the Ministry of Health. This has helped build group cohesion, acceptance, and sustainability both during and after the program. • De Facto Committee Structures Needs a Formal Identify and Technical Support: The fact that the de facto committee is still an informal structure that does not benefit from the steps of direct types of communication that were envisioned for the GAP/RU in program proposal and initial set up has a number of downstream consequences. It both limits the 85 I.e. The results and effects, the effectiveness of the implementation process, the sustainability of the results, and the activities impact on the local people’s resilience. Nema Final Qualitative Evaluation. Chapter 4. SO3. December 30, 2013. Draft. 94 degree to which the committee members can interact with the other relevant actors involved in early warning and response (most notably the Red Cross/Mali volunteers), and their interaction with the SAPs. • Recent Creation of Community-Level Food Security Committees as a Legal Structure within the SAPs: The national SAP committee has recently recommended that each community create a village-level Food Security Committee, and that this committee be the principal contact point between the community and the cercle and commune-level SAPs. One of the main recommendations coming out of the San debriefing workshop on November 29, 2013 was that the next phase of the program anticipates the need to have this type of Food Security Committee be created to oversee all of the SO3 activities and have specific sub-committees to do specific functions (like the GAP/RU, the Safety Net Committees, the FFW committee, and infrastructure management). This same structure could provide an umbrella for coordination with the other Nema committees both during and after program phase out. 4.3.2.2. Effectiveness of the Monitoring, Evaluation, Accountability, Learning (M&EAL) System: • Strong System for Tracking the Capacity of the Safety Net and GAP/RU Committees: A major strength of the SO3 M&E system was the creation and use of a simple system for tracking the capacity of the two most important intended organizational outputs: the Safety Net Committees and the GAP/RU. This system made it possible for the SO3 to identify problems—like the non-communication between the groups and the cercle-level SAPs—and to better target technical assistance and capacity. • Areas that Could Be Strengthened in Order to Better Track Program Results and Effects: Although the M&E systems were rigorous, certain elements of the IPTT tracking system made it hard to appreciate the program’s significant achievements in a number of areas that are critical to resilience, food security, and sustainability. • Limited Disaggregation of the Tracking Data by Intervention Zone: The fact that most of the SO3 reporting data was aggregated made it harder to track certain site specific issues like the much weaker capacity of the GAP/RU in Douentza than in Bourem. • Limited Capacity of Community-Level Capacity: The SO3 institutional capacity indicators were very useful, but the fact that they focused on specific committees—not the entire community early warning and response system—made it harder to: - Track the “bigger picture” of how the community was responding to risk and various options for emergency response (like aid from other donors like WFP, IFRC, and Red Cross/Mali both during and after Nema); and - Determine when and what types of technical assistance might be needed to strengthen the emergency response when the major crisis hit. Had a more general tracking system been in place that could track the aggregate capacity of the de facto Food Security Committee that was calling the shots in most villages, it would have been much easier for the Consortium members to continue to backstop these committees and collaborate with the donors that were coordinating the emergency response. In the absence of this type of system, none of the major implementing partners (HKI, SCI, CRS) or local execution partners like Caritas had any concrete information on what type of emergency response was going on or who was doing it until the final qualitative evaluation in November Nema Final Qualitative Evaluation. Chapter 4. SO3. December 30, 2013. Draft. 95 2013. 5.0. Lessons Learned and Recommendations Based on the assessment of the data from the final evaluation survey, the consultants and field staff who participated in the initial debriefing at San on November 30, 2013 identified six major recommendations for the next phase, as well as a list of specific recommendations. The consultants synthesized this list with their initial analyses and presented a revised list to the working group members on December 4, 2013 for their review. Recommendation 3.1. Anticipate the creation of Food Security Committees in all of the Nema villages to coordinate the different Nema and non-Nema community structures that affect food security, with a special accent on the activities started under SO3. Specific Recommendations: 1. Transform the existing level of collaboration between the different Nema-created structures that are functioning as an informal food security into a series of Village Food Security Committees (Comite villageois de sécurité alimentaire). 2. Conduct a participatory needs assessment in each of the former Nema program villages to determine the level of functioning of the SO3 structures put in place and how the community managed the crisis. 3. In conjunction with this participatory needs assessment, identify all of the key humanitarian assistance partners (including the local Red Cross volunteers and brigades) and their activities in the Nema villages during the crisis. 4. Create a self-assessment tool that the Village Food Security Committees can use to self￾assess the core capacities that they will need to manage routine risk and major shocks. 5. Based on this baseline self-assessment, help each committee develop a strategic plan for building their capacity in collaboration with the program and other partners such as Red Cross/Mali. 6. Based on these needs assessments, work with local aid and SAP partners to develop joint training programs that will strengthen the Village Food Security Committees and sub￾committees (e.g. the Safety Net Committee and the GAP/RU) and build strong regional networks that can backstop these committees during a crisis. 7. Require each Village Food Security Committee to update its self-assessment index (which includes an assessment of their level of collaboration with other aid partners and SAP structures outside the community) and try to insure that some of this information is tracked in indicators in the IPTT. Recommendation 3.2. Anticipate the involvement of all the key partners (Village Food Security Committees, Red Cross/Mali, and SAP) in the conception and execution of any future community-based early warning and emergency response system. Specific Recommendations: 1. Insure that the regional and local SAPs and major humanitarian organizations working in the area are involved in the initial conception of any future early warning and response system (i.e. in the MYAP design), as well as its implementation. Nema Final Qualitative Evaluation. Chapter 4. SO3. December 30, 2013. Draft. 96 2. Anticipate signing a MOU with the aid partners that might be needed in a major emergency as part of any future MYAP designs to facilitate quick response and more realistic emergency preparedness for worst-case scenarios. 3. Each year, conduct a participatory review of all MOU and processes needed to activate them with the Village Food Security Committees and update the agreements if the context has changed. 4. Continue to support the GAP/RU sharing the early warning information that they collect on the local community radio stations. 5. Anticipate the necessity of linking the GAP/RU data-collection efforts to the larger data needs of the SAPs and other humanitarian partners. Recommendation 3.3. Anticipate activities that strengthen the food security of the most vulnerable households that reduce their dependence on community based safety nets. Specific Recommendations: 1. Strengthen current safety net programs with activities that develop IGAs for the most vulnerable households. 2. Help vulnerable groups to organize themselves into groups to facilitate the supervision and support of activities that will strengthen their livelihoods and reduce the case loads of future safety-net programs. 3. Anticipate a sub-category of FFW activities for vulnerable groups (as both individuals and groups) that will support the development of IGAs. 4. Develop appropriate M&E systems for tracking the impact of these activities on vulnerable groups. Recommendation 3.4. Strengthen the linkages between SO3 and SO2 activities. Specific Recommendations: 1. Intensify the existing efforts of Nema to use FFW to expand the development of fruit trees and gardens managed by women. 2. Anticipate FFW activities that can help the savings and internal lending communities (SILC) groups develop irrigated vegetable gardens that increase their revenue and household dietary diversity. Recommendation 3.5. Intensify FFW activities in the most vulnerable villages in order to build their food security and income and reduce community-level vulnerability to risk and shocks. Specific Recommendations: 1. Identify FFW activities needed to jumpstart agro-enterprise development in the most vulnerable villages. 2. Develop appropriate M&E systems for tracking the impact of these activities on household and village-level vulnerability. Recommendation 3.6. Strengthen the capacity of the Nema M&E system to monitor the activities of the Village Food Security Committees and other key community-level management structures. Nema Final Qualitative Evaluation. Chapter 4. SO3. December 30, 2013. Draft. 97 Specific Recommendations: See Recommendation 3.1, sub-recommendations 1-7 above. Nema Final Qualitative Evaluation. Chapter 5. Cross Cutting. December 30, 2013. Draft. 98 Chapter 5 Cross-Cutting Activities: Literacy, Good Governance, and Local Organizational Capacity86 This chapter provides a brief overview of the results of the Final Qualitative Survey and Evaluation for the Nema program’s transversal activities related to basic literacy and good governance. Section 1.0. Nema Strategy/Theory of Change for Cross-Cutting Activities: Literacy, Good Governance, and Capacity Building: Describes the Nema program’s literacy and good governance activities that were expected to “enhance beneficiaries’ human and financial capital.”87 Section 2.0. Evolution of the Nema Strategies for Literacy, Organizational Capacity Building, Good Governance, and Early Evidence of Impact: Describes the evolution of the implementation strategy for literacy and good governance activities. Section 3.0. Current Level of Activity of the Nema-Supported Literacy Activities: Describes: • 3.1. The current level of literacy training in the centers, as well as some of the longer￾term impacts of this training on the community groups that benefited from it based on the results of the qualitative survey (November 2013); and • 3.2. A summary assessment of some of the key factors that affected these outcomes. 88 Section 4.0. Current Level of Activity of the Nema-Supported Community Based Organisations (CBOs): Provides: • 4.1. A brief overview of the current level of functioning of the community-based organizations (CBOs) that were targeted by Nema’s literacy and good governance training; and • 4.2. A summary assessment of some of the key factors that affected these outcomes. Section 5.0. Lessons Learned and Recommendations: Presents a list of recommendations that the evaluation team—working in close collaboration with the key implementation and execution partners—have identified for: • Corrective actions recommended for increasing the sustainability of the program impacts; and • Strengthening the benefits initiated by the program. 86 This chapter was prepared with the assistance of an informal Nema literacy and capacity working group that developed around the final evaluation team’s field and key informant interviews. This working group included: Abdoussalam Tiemogo Maiga (SCI/Gao), Pierre Togo (Caritas), and Isack Dolo (CRS). The working group was supported in its analysis of previous and current activities by Boureima Sako and Isack Dolo (M&E Department CRS), who co-facilitated the field portion of the exercise with Dr. Sidikiba Sidibe (Consultant, Nema), and Abdoussalam Tiemogo Maiga (SCI/Gao) in Bourem. 87 CRS 2008: 5. 88 I.e. The results and effects, the effectiveness of the implementation process, the sustainability of the results, and the activities impact on the local people’s resilence. Nema Final Qualitative Evaluation. Chapter 5. Cross Cutting. December 30, 2013. Draft. 99 In addition to the three interconnected technical areas addressed under Strategic Objective 1 (SO1), the Nema program included two cross-cutting activities—basic literacy and governance—that were meant to support all three SOs.89 The underlying premise was that training beneficiaries in these themes would contribute to the smooth functioning and sustainability of the various groups associated with the Multi-Year Assistance Program (MYAP): agro-enterprise groups (AEGs), savings and internal lending communities (SILCs), PD/Hearth participants, early warning groups (GAP/RUs), Safety Net Committees, etc. (Figure 5.1).90 Figure 5.1. Nema Strategy for Literacy, Good Governance, and Local Capacity Building 1.0. Nema Strategy/Theory of Change for Cross-Cutting Activities: Literacy, Good Governance, and Capacity Building 1.1. Literacy The original proposal anticipated that the MYAP would: 89 The original proposal (CRS 2008: 5) outlined two cross-cutting interventions. The first was “thematic literacy courses related to agro-enterprise, nutrition, and hygiene.” The second was SILCs as a means of improving resiliency to economic shocks, based on a methodology developed by CRS and successfully implrlemented in Mali and numerous other countries in Africa” (CRS 2008: 5). By the time of the mid-term evaluation, the SILCs had been incorporated into SO1 and were evaluated as part of SO1 and the issue of “good governance” was added to literacy as a cross cutting theme (Tilford 2011: 34-36). 90 K. Tilford. 2011. Nema Mid-Term Evaluation. Pg. 34. Nema Final Qualitative Evaluation. Chapter 5. Cross Cutting. December 30, 2013. Draft. 100 “Build on materials and approaches developed by other NGOs to dispense thematic literacy courses related to agro-enterprise, nutrition, and hygiene.”91 The approach that was adopted was the Samogoya, a new method that had just been pilot tested by the Direction Nationale des Langues et de l'Alphabetisation (National Directorate for Languages and Literacy or DNFLA) in Mali that is now the standard approach used in all of the Mali government’s literacy training centers. This approach emphasized: (a) the presentation of images in the students’ handbooks to facilitate a debate between the instructor and the student on key themes, followed by (b) the identification of key words that linked the image to their every day lives. The theory under girding the approach was that the trainees (auditeurs) would be more receptive to the training if it were linked to something concrete they could relate to. The goal of this activity was to establish a literacy center with at least two trained trainers (formateurs) in each of the 130 MYAP villages who could continue to provide training once the program ended. A literacy center referred not to a physical structure but to the gathering of a group of people who came together to learn to read, write, and perform simple calculations. The actual classes could be held under a tree, in a schoolroom, in the shade of a building, or under a hangar. To ensure an appropriate level of community commitment to the development of the training center and the trainees, all of the community development agents (ADCs) were trained to oversee a participatory process that required high levels community engagement in the choice of the trainers, the literacy training site, and the trainees.92 The community was responsible for providing a literacy center, identifying the trainers, and providing certain in-kind support to the trainers. The program trained the teachers and provided basic equipment for the classroom, as well as the training manuals and supervision and student review needed to certify literacy levels. To insure appropriate training, the program invested heavily in the selection and training of two trainers per village, 93 some of whom had been trained as literacy workers by earlier programs. Once the literacy trainers were chosen and trained, the literacy supervisors worked with each community to identify a training calendar and recruit various members of the different community groups being established. It was anticipated that each session would last three months and train 15 men and 15 women. Given the high rates of illiteracy and the need for literacy skills within the various MYAP￾sponsored groups, this activity was deemed essential to insuring the long-term sustainability of a number of the MYAP activities.94 Although literacy was a major focus of the program, the Indicator Performance Tracking Table (IPTT) did not include an indicator that tracked the evolution of these activities, nor their impact.95 91 CRS 2008: 5. 92 SCI. 2011. Présentation Approche Sanmogoya Nema / MYAP Août 2011. Douentza du 3 AU 6 Aout 2011 93 One of the first tasks was to evalute the level of the local existing trainers then to recruit new trainers—if needed—to insure two qualified trainers per village. All existing and new trainers received a basic training in the Samogoya methodology. Once this was done, the program conducted a public forum in each village to outline the training program in the presence of the ADCs, trainers, and communities. 9494 K. Tillford. 2011. Nema Mid-Term Evaluation. Pg. 34. 95 The Mid-Term Evaluation recommended adding four indicators for functional literacy to the IPTT (Tilford 2011: 36). Nema Final Qualitative Evaluation. Chapter 5. Cross Cutting. December 30, 2013. Draft. 101 Save the Children International (SCI) was the technical lead and assigned a full-time staff person to insure that the literacy activities were executed according to plan through four sub-contracts to: (a) Caritas/Mali and Tassaght, and the Ministry of Education for the actual training; (b) Center for Rural Extension (Centre d’Animation Paysanne or CAP) for testing and assessment; and (c) DNFLA for the production of the instruction manuals. SCI was also the lead on the good governance training, which started at the end of the second year (June 2010). 1.2. Local Capacity Building and Good Governance Although the program proposal did not include an explicit strategy for local capacity building, the Nema program included an implicit one based on building the capacity of the local groups charged with the execution and maintenance of the activities the program was creating through its intensive investment in community-level training programs. Most training programs were conducted by the ADCs, who then trained the community-based trainers (the Animateurs Villageois [AVs] for the SILCs, the Training of Trainers [TOT] Farmer Field School [FFS] trainers for the FFSs, and the community health volunteers [CHVs or relais]). Most of the GAP/RUs, Safety Net Committees, AEGs, and water management committees were trained directly by the ADCs. Most of these core training programs—including the ones for the SILCs and AEGs—included some basic training on good governance and financial management of group-managed activities. The implicit assumption of the program proposal was that this training—in combination with the literacy training—would “enhance beneficiaries’ human and financial capital…[and have]….greater capacity to resist and recover from shocks and to capitalize on existing or initiate new livelihood opportunities.” 96 The evolution of some of the key capacities being developed was tracked as part of the IPTT. 2.0. Evolution of the Nema Strategies for Literacy, Organizational Capacity Building, Good Governance, and Early Evidence of Impact 2.1. Early Evolution of the Program 2.1.1. Literacy Training: By the time of the mid-term (June 2011), 122 of the 130 Nema communities had created literacy centers. Although some centers were considered to be functioning at less than optimal capacity, the mid-term evaluation concluded that the activity was very much appreciated by the beneficiaries. Based on interviews with members of literacy centers in each of the 16 villages, the mid-term evaluation concluded that the literacy programs were making a major contribution to the Nema program. They noted that the health messages were especially appreciated by the beneficiaries, and that 15 of the 16 groups noted the usefulness of the literacy programs for their group activities: keeping minutes of meetings, filling out forms, and preparing plans and reports. The two principle challenges at mid-term were the low salaries for the trainers and the high dropout rate in some centers. Just prior to the mid-term, the salaries were increased from 15,000 CFA/month to 25,000 CFA a month. Some of the staff recommended using Food for Training (FFT) as an incentive for attendance during the mid-term, 96 CRS. 2008. PL480 Title II MYAP Proposal. Consortium for Food Security in Mali (CFSM) Nema Program. Pg. 5. Nema Final Qualitative Evaluation. Chapter 5. Cross Cutting. December 30, 2013. Draft. 102 but this was rejected during the second mid-term evaluation workshop.97 2.1.2. Good Governance Training of Trainers: In June 2010, SCI led a five-year TOTs workshop in governance for approximately 25 MYAP staff. Topics included leadership, community organization, supportive supervision, gender, synergy, and self-governance. At the end of the training, the participants received a comprehensive manual. Between July 2010 and May 2011 (time of the mid-term evaluation), the MYAP staff trained all of the AEGs and SILCs on the good governance themes in the SCI manual. This training supplemented the modules in the CRS manuals that were used for baseline training and retraining of the AEG and SILC groups. The CRS manuals used for training both groups included modules on governance-related themes such as leadership roles, the democratic process for conducting meetings and making decisions, conflict management, and the importance of keeping written records of meetings. By the end of the mid-term evaluation, 53% of the AEGs had been trained in governance (Environmental Indicator 1.1.11). 2.2. Mid-Term Evaluation Findings and Recommendations Based on their field interviews, the mid-term evaluation included a list of recommendations for trying to improve the motivation of the community-based literacy trainers (formateurs) and reduce student drop out. The evaluation also recommended that the program:98 • Determine the essential elements of governance that all MYAP-sponsored groups [e.g. AEGs, SILCs, water management associations, GAP/RUs, Safety Net Committees] be trained in, referring first to the training already conducted for AEGs and SILCs; and • Develop a one- to two-day training module incorporating the essential elements, and ensure that the other MYAP-supported groups also benefit from governance training. 2.3. Activities After the Mid-Term 2.3.1. Literacy Training: The final evaluation of the centers by the CAP, which was the official government agency charged with auditing the quality and impact of literacy training, concluded in August 201199 that Nema had: • Created centers in all 130 of the Nema villages, and 121 centers were open; and • Enrolled 3177 students (out of 3900 anticipated) from a broad cross-section of all the groups targeted for the training (including AEGs, SILCs, emergency preparedness [GAP/RUs], and PD/Hearth volunteer mothers) (Table 5.1). However, only: • 2608 (67%) of the 3077 students were active participants; and, out of that number, 97 This was rejected for two reasons. First, many of the participants made a distinction between activites in which food distribution is important (such as FFW projects and the safety nets) and activities where the obvious benefits of participation should provide sufficient motifvation, e.g. the advantages of being able to read, write, and do basic caluclations. The second reason given was that the workshop participants felt it was unlikely that Food for Peace (FFP) managers would agree to the use of food for this activity. 98 K. Tilford. 2011. Nema Mid-Term Evaluation. Pg. 36. 99 Carte des Auditeurs. Alpha 2. August 2011. Nema Final Qualitative Evaluation. Chapter 5. Cross Cutting. December 30, 2013. Draft. 103 • 472 of the students at Bourem100 were evaluated (out of 880 who signed up to be tested), of whom 215 were actually certified as being neo-alphabetes (newly literate) (Table 5.1). Although 42% of the active students were women (1090 out of 2608), only 30% (18/60 total) of the students trained in Bourem and 15% (19/136) of the students at Douentza that were assessed as being neo-alphabetes at Bourem were women. The final evaluation of the Nema literacy training by CAP gave several reasons for the low rattendance and certification rates.101 Table 5.1. Number of Students from Different Nema-Supported Structures that Enrolled in Literacy Training and the Number Who Were Active Participants, Years 1-4 Structures Number of Students Enrolled Number Who Completed Training Male Female Total Male Female Total AEGs 341 203 544 289 156 445 SILCs 130 453 583 14 379 393 Emergency Response 313 45 358 238 37 275 Health 223 70 293 182 67 249 Volunteers 838 561 1399 795 451 1246 Total 1845 1332 3177 1518 1090 2608 Source: CAP. 2011. Carte de Auditeurs. PowerPoint provided by Abdoussalam Tiemogo Maiga (SCI/Gao). 2.3.2. Local Governance Training: After the mid-term, the program was plunged into a series of trainings that were needed to convert the Nema program’s ADC subject specialists (who focused on nutrition, GAP/RU, and agriculture) into ADC generalists (who would cover all community￾level activities). This new training seems to have included a much greater emphasis on good governance, which was integrated into the community-level training of the local groups— especially the AEGs and SILCs. By the time the program closed, 97% of the AEGs had been trained using the new good governance module (Environmental Indicator 1.1.11). 2.3.3. Other Pertinent and Overlapping Trainings: By the time Nema ended, most of the local committees had experienced a wide variety of technical trainings, as well as basic literacy and training in good governance (Table 5.2). The program had also invested heavily in training and technical assistance to develop three types of TOTs—the AVs for the SILCs, the FFS TOTs, and the CHVs. The program was also starting to use some of the FFS TOTs and CHVs to collect data for the official IPTT indicators, as well as the internal tracking systems (Table 5.2). 100 Based on the auditor’s report, which was given by the SCI representative on the evaluation team, it appears that the final evalution of the literacy centers at Douentza was never completed. 101 Two of the main reasons cited were: (1) the last months of most literacy sessions overlapped with the start of the growing season; and (2) the high rates of food insecurity for the students, which made it hard for them to take the time away from their routine crop and IGAs. Carte d’auditeurs. Alpha 2. 2011. PowerPoint Presentation. Nema Final Qualitative Evaluation. Chapter 5. Cross Cutting. December 30, 2013. Draft. 104 Table 5.2. Types of Training and Incorporation of the Local Committees and Community-Based Experts into the Program’s Internal and Official Tracking Systems, Years 1-4 Technical Training Training in Good Governance Training in Financial Management Literacy Trainings Collection & Reporting of Nema M&E Data SO1 AEGs X X# X X IPTT* SILCs X X# X X Internal FFS TOTs X X Internal SO2 Health comitteees X X X X Internal*** Volunteer mothers (for the PD/Hearth activities) X X X Internal*** Water point management committees X X X X Internal*** Sanitation and hygiene commitees (community￾led total sanitation or CLTS) X X X X Internal*** Community health volunteers (Rélais) X X X X IPTT** SO3 GAP/RUs X X X X Internal*** Safety Net Committees X X X X Internal*** Infrastructure management committees (and FFW management committees) X X X X Internal*** Source: Equipe MYAP Nema. November 2013 # Special modules developed by SCI in June 2010, as well as the core AEG and SILC training modules. *Monitor yields on the FFS replication fields. **Monitor malnourished children and their rehabilitation. ***Internal program tracking data only. 2.3.4. Early Evidence of Effects (Based on the IPTT): A major strength of the Nema program was its investment in a series of internal tracking tools to track the capacity of the SILCs, AEGs and GAP/RUs. Some of the information from these tools was reported in the IPTT and used to track the evolution of some of the most critical local committees being supported, including the SILCs, AEGs, water-point management associations, GAP/RUs, and Safety Net Committees (Table 5.3) The SILC tool is a regional tool used in all the CRS-sponsored SILC programs in the sub-region; the AEG tool is borrowed from a sub-regional tool. Staff argued that these tools were very useful and helped them better target its technical assistance and training. The same tools helped track certain key capacities in the IPTT like good governance training for the AEGs (Environmental Indicator 1.1.11). Given the critical importance of local capacity building, the program created capacity assessment tools that were used to track the capacity of the SILCs, AEGs, and GAP/RUs. The evolution of some of some of the most important core capacities and trainings were tracked in the program IPTT (Table 5.3). Although the committees themselves tracked some of these indicators, any official data reported in the IPTT was verified through an Nema Final Qualitative Evaluation. Chapter 5. Cross Cutting. December 30, 2013. Draft. 105 independent audit by the Nema M&E officer based on a sample of program households that was conducted annually. Table 5.3. Indicators Used in the Nema IPTT to Track the Evolution of the Local Community Organizations’ Capacity in Key Areas Needed to Execute and Sustain Their Activities, Years 1-4 Year 1 Year 2 Year 3 Year 4 & % of Target AEG EI 1.1.11. # of AEGs trained in governance 0 9 29 73 (97%) MI 1.2.6. % Title II-assisted producers who are members of a functional agro-enterprise group 0 19% 44% ND SILC MI 1.2.1. % of SILC members who have increased their financial assets 0 50% 87% 36% MI 1.2.2. % increase in value of net worth of the SILC groups combined 0 22.1% 38% 53% Water User Groups MI. 2.2.9. % water management committees who properly maintain completed water points as indicated by lack of stagnant water and cleanliness of site 0 0 33% ND GAP/RU II. 3.1. % of Title II-assisted communities with disaster early warning systems in place 0 30% 52% 64%102 MI 3.1.2. # of community groups that are actively tracking trigger indicators 0 26 67 ND Safety Net Committees MI 3.2.1. Total # of assisted communities with safety nets in place to address the needs of the most vulnerable members 0 96 130 130 MI 3.2.2. Total # of communities who strengthen safety nets over the life of the activity, as shown by the reported increase in the diversity of shocks the safety net is capable of responding to 0 42 130 130 Source: Nema program IPTT, Annex I. Acronyms: EI: Environmental Indicator; MI: Monitoring Indicator; II: Impact Indicator. 3.0. Current Level of Activity of the Nema-Supported Literacy Activities 3.1. Current Level of Activity During the 15 focus group interviews with the SILCs and the 14 focus group interviews with the AEGs, the evaluation team asked a series of questions about the level of involvement and benefits of literacy training. These interviews suggest that most of the Nema-trained literacy trainers (formateurs) are still living in the village, though not one of them has organized any literacy training since the Nema program ended. The same interviews reflected some of the CAP data, which showed that only between 30-50% of the SILC and AEG members trained in literacy during the program were considered able to read, write, and conduct basic calculations (Table 5.4). Although the ratio of persons trained to persons able to read, write, and perform basic calculations was low (Table 5.4), most of the persons who attended the focus groups were vocal 102 This figure was not reported in the final IPTT, which states “ND,” but is the result of the Nema program’s internal tracking data (Source: Adama Sangare, November 2013). Nema Final Qualitative Evaluation. Chapter 5. Cross Cutting. December 30, 2013. Draft. 106 in their appreciation of the training and the role that the literacy training played in helping them manage their groups. One of the most frequent demands for follow-up assistance in the focus groups with the SILCs and AEGs was for literacy training (Text Box 5.1). Table. 5.4. Average Number of SILC and AEG Members Who Attended the Nema-Sponsored Literacy Training Who Can Read, Write, and Conduct Simple Calculations, March 2013 Structure # of Groups Interviewed Average # of Members Who Attended the Literacy Training Average # of Members Who Can Read, Write, and Calculate Percentage of Those Trained Who Can Read, Write, and Calculate AEGs Bourem 3 31 15 48% Douentza 11 31 12 39% SILCs Bourem 3 22 11 50% Douentza 12 9 6 67% Methodology and Source: AEG and SILC focus groups, Nema Final Evaluation, November 2013. Text Box 5.1. Comments Made About the Role of Literacy Training During the AEG Focus Groups Quelles suggestions ou recommandations faites-vous dans le cadre du suivi et renforcement des vos activités d’agro entreprises Villages Fombory, Bore: • Le Reprise immédiate des activités du projet pour nous assister dans le financement de nos chiffres d’affaires. • Formation en Alpha et Domaine agricole • Village Fombory, Ibissa: Le retour rapide du projet Nema pour la continuation des activités initiées avec le groupement, cela va permettre de bien comprendre l’approche, de garantir l’appropriation et la vulgarisation des bonnes pratiques dans le village • Village Kiro: Nous sollicitons l’aide en matériel agricole (Charrue, bœufs de labour, ânes et charrette) et intrants agricole (augmentation en semences améliorées) Impact de la Formation des membres en Alphabétisation pendant le projet Nema • Village Fombory: Dans notre groupement, 30 personnes (25 hommes et 5 femmes) ont participé à la session de formation en Alphabétisation. Nous avons appris a lire, écrire et a calculer. Parmi les membres 10 personnes (9 hommes et 1 femme) savent a la fois lire, écrire et faire des petits calculs. Cette formation nous a permis de faire une bonne tenue de nos documents de gestion (remplissage des outils de gestion de l’association : PV des réunions, le mouvement de caisse, l’enregistrement des noms). • Village Bore, Ibissa: l’alphabétisation nous a permis de tenir les outils de gestion (tenue des PV pendant les réunion), de faire des correspondance. Les dettes et les recettes des activités commerciales étaient notées dans notre registre. Recommandations pour alphabétisation • Village Fombory: Le retour immédiat du projet Nema pour la poursuite de l’alphabétisation a fin d’augmenter le nombre de personnes qui peuvent lire, écrire et faire des petits calculs. • Village Ibissa: Renforcer les capacités des membres en lecture, calcul et en écrire. Augmenter le nombre d’auditeurs et appuyer a la construction d’un centre d’Alphabétisation des groupements dans le village. Source: AEG Focus Groups, Nema Final Evaluation. November 2013. 3.2. Summary Assessment of Strengths, Areas that Need Strengthening, and Opportunities in the Current Context: Literacy Training 3.2.1. Results and Effects: • People were trained, and some did learn to read, write, and perform basic calculations: In sum, the literacy programs during Nema did train a large number of people in the local CBOs to read and write but not to the level that was desired by the program, which is not Nema Final Qualitative Evaluation. Chapter 5. Cross Cutting. December 30, 2013. Draft. 107 surprising given that the program stopped before the third year of training could be executed. It is possible that this might have been greater had the literacy trainings focused more concretely on the actual tools and activities of the CBOs that it was designed to help. In any case, the strong impact of these programs during the first four years has left an important human capital that can be built on in future programs. • Literacy activities were active in all 130 villages and most trainers are still in the villages: The literacy component of the program was one of the few Nema components that touched every single village (Table 5.4). The final evaluation study showed that most of the centers are still recognized and most of the trainers are in the villages. • Positive impact of the literacy training manuals on nutrition and sanitation messages: One unintended consequence of the training seems to have been to provide a major component of the program’s Information, Education, and Communication (IEC) strategy for the Nema sanitation and nutrition activities. In contrast to the SILCs, which covered only 50% of the Nema villages, the literacy programs covered all 130. This impact was mentioned during the mid-term and in several of the focus group discussions. 3.2.2. Effectiveness of the Implementation Process and Gender Integration: • Strong linkages with the key government partners involved in literacy training (CAP and DNFLA): Another strength of the program was its tight linkage with the government services that support and monitor literacy programs. This apparently helped increase quality and supervision and set the tone for sustainability. • Strong implication of the communities in the initial choice of the literacy trainers: Several key informants felt that this recruitment process helped build local ownership and support for the trainers that earlier training programs had not had. • Weak motivation of the teachers: One of the key problems that contributed to the lower￾than-expected results of the literacy training in most centers was the weak motivation of the teachers and the students (auditeurs). Although the training manuals touched on Nema program links, there was very little direct linkage between the training and the actual tools the committee members were being asked to use. • Monitoring, Evaluation, Accountability, and Learning (M&EAL) tracking system tracked student attendance and success, not the impact of the training on the local groups: Although CAP’s technical supervision of Nema’s community-based literacy program activities was very thorugh, this technical assistance focused on assessing the technical quality of the training and student success—not the impact of the training on the local groups that it was designed to address. • Effective targeting of women as participants, but lower success rate: The literacy activities’ gender targets helped insure that a high percentage of the trainees were women. Although women participated in the literacy training, their certification rates were lower: i.e. 42% of the active students were women (1090 out of 2608), but only 30% (18/60 total) of the students trained in Bourem and 15% (19/136) of the students at Douentza that were assessed as being neo-alphabetes were women. 3.2.3. Sustainability: Strong collaboration with government agencies increases the chances that the Mali government agencies charged with oversight of literacy training will be willing to support the Nema-founded training centers and trainers in future programs: Certain elements of the program—that were put in place to guarantee its sustainability—are very weak and need to Nema Final Qualitative Evaluation. Chapter 5. Cross Cutting. December 30, 2013. Draft. 108 be strengthened, which would enhance the chances that this type of literacy program will be sustained—e.g. the weak collaboration between the program and the two cercle-level CAP offices and the program’s emphasis on TOT. For this government-endorsed literacy training program to have greater impact, it needs to be better oriented to the actual skills and tools that the trainees will need to manage group activities. This type of thematic literacy training program is being pilot tested by several NGOs in both Bourem and Douentza. 3.2.4. Resilience: Indirect impact of literacy on resilience: Based on the responses during the focus groups, there is clear evidence that in the beneficiaries felt that the literacy training helped them better manage the multi-dimensional shocks that started descending upon them after June 2011 (see qualitative responses to the survey in Annex I). 4.0. Current Level of Activity of the Nema-Supported CBOs 4.1. Current Level of Functioning Based on the focus group discussions, the evaluation team developed a matrix with which they assessed the degree of functionality of the groups they encountered (Table 5.5; See also Tables 2.5 and 2.6 in Chapter 2). Although many of the groups were still functional, they were considered weak. It is not a surprise that some of the strongest, most-functional groups were the AEGs and SILCs, whose training emphasized their becoming autonomous from the start. Most groups emphasized the critical importance of the literacy training in helping them manage the tools and funds needed to execute their core functions, and requested that the program re-start some of the literacy training to help them better manage these tools (Text Box 5.1). Table 5.5. Functionality of the Nema-Facilitated CBOs Interviewed During the Nema Final Evaluation by SO, November 2013 Nema SOs and CBOs Level of Functioning* Bourem Douentza # Groups Interviewed Level # Groups Interviewed Level 1 2 3 4 1 2 3 4 SO1 AEGs 3 2 1 15 1 4 3 7 SILCs 3 3 12 2 4 2 4 FFW management committees 5 5 10 10 SO2 Health committees 4 4 9 9 Water point management committees 1 1 1 1 Hygiene and sanitation committees (CLTSs) 3 3 CHVs 5 1 4 15 15 SO3 GAP/RU 6 1 5 15 7 8 Safety Net Committees 6 6 15 1 14 Literacy groups 4 4 15 2 13 Source: Nema Final Evaluation Survey, November 2013. * 1=strong, 2=average, 3=weak, 4= non-functional One of the chief weaknesses across all of the local organizations was the weak capacity to Nema Final Qualitative Evaluation. Chapter 5. Cross Cutting. December 30, 2013. Draft. 109 manage budgets and accounting. This was especially true for the AEGs, which received large budgets from the programs before they were really ready to manage them. The general consensus from many focus groups and MYAP staff was that the program ended before the groups were sufficiently trained in financial management. 4.2. Summary Assessment of Strengths, Areas that Need Strengthening, and Opportunities in the Current Context: Local Organizational Capacity and Good Governance 4.2.1. Results and Effects: • Major impact on local organizational capacity of the groups Nema supported:103 There is strong quantitative evidence—from the program’s own reporting data and the key informant and beneficiary data—that the Nema training and technical assistance helped build the capacity of the local organizations they supported in several key areas, including: - Strategic planning; - Basic principles of organization, including government registration processes; - Financial management and access to micro-finance institutions; and - Marketing. • Many of the same trainings helped build the capacity of certain state institutions—like the Ministry of Agriculture extension service and Institute of Rural Economy (Institut d’Economie Rurale or IER) for agriculture, the Ministry of Health and the Regional Directorate for Sanitation (for SO2), as well as the regional- and commune-level SAP (early warning systems)—to improve their activities in the local communities. The fact that almost all of the Nema activities—without exception—supported the current national strategies and policies for these sectors was very much appreciated by all of the persons interviewed during the key informant interviews (health, water resources, IER, Ministry of Agriculture, and the Governor’s office). • Major local impact on two local NGOs: The program helped strengthen the capacity of two local NGOs, Tassaght and Caritas. The trainings helped facilitate the training of local experts in both institutions who continue to be in contact with some of the local leaders. The same collaboration helped both institutions identify a number of best practices in management, financial management, administration, and human resources that these NGOs are continuing to adopt (2013). Especially important for Caritas was the addition of two new intervention themes (nutrition, and community-led total sanitation [CLTS] for sanitation) that they had never supported before. 4.2.2. Effectiveness of the Implementation Process and M&E Systems: • Training programs were well designed and included government partners, which increased efficiency:104 Three elements of the Nema approach that seem to have facilited its trainings having such an important impact in such a short period of time (3.75 years) include: - The strong implication of state structures in the conception and execution of the trainings; 103 Cross-cutting questions from the SOW related to effects and results. 104 Cross-cutting questions from the SOW related to efficiency. Nema Final Qualitative Evaluation. Chapter 5. Cross Cutting. December 30, 2013. Draft. 110 - The rigorous supervision of the training by the executing agencies (Caritas, Tassghet) and partners (SCI, HKI, CRS) in collaboration with the relevant government agencies, which also implicated these state agencies in enforcing the Nema program’s gender targets; and - The Nema training budget was well designed and anticipated the need for baseline training and retraining. • Most and least efficient trainings: Some of the most effective activities appear to be the community-based technical training programs; the least effective were the CSCOM training programs, since most of the staff was transferred to other positions during the crisis. • Strong M&E for tracking the capacity of some groups, but limited analysis of the link between these capacities and literacy and Nema’s eventual exit strategy: The Nema program was committed to tracking the capacity of the four main groups the program worked with—the SILCs, AEGs, GAP/RUs, and Safety Net Committees. This information helped the program better orient its technical assistance. Unfortunately, there was very little real-time, annual analysis of the role that the literacy training was—or was not—having in relation to these programs. There was also very limited reporting on this enhanced capacity in the major program documents. Given the critical importance of this capacity in sustaining these groups’ activities during the crisis, future programs need to anticipate the need for: - More in-depth analysis of the comparative capacity needs of all of the groups, not just the AEGs, SILCs, GAP/RUs, and Safety Net Committees; - The linkages between the program support for literacy and this capacity; and - The link between these groups’ capacity and the preparation of the programs’ exit strategy from the zone. • Limited capacity building of groups that target vulnerable groups: The Nema program document did not anticipate the development of any groups that target the most vulnerable groups in each village. The program did, however, promote the use of SILCs as a mechanism for building the livelihoods of women (many of whom were probably classified as vulnerable). Two challenges for future interventions in Douentza and Bourem will be to: (a) better monitor the impact of participating in the SILCs on the socioeconomic status of the beneficiary populations; and (b) develop a sub-set of cross￾cutting activities (SO1, SO3) that target vulnerable groups through individual and group activities. 4.2.3. Sustainability: • The large number of committees created by the program has made it hard to determine which ones are sustainable: The program’s theory of change envisioned the local committees as a mechanism for building stakeholder input and understanding of the program. Unfortunately, the program pulled out before most of them were sustainable and/or linked to the larger community-level structures that are needed to improve the inter-community coordination between groups. One of the key challenges for the next phase will be to move in the direction of a gradual consolidation of these groups into SO￾level coordinating committees that are linked to some sort of community-level structure like the new Food Security Committee structure described in Chapter 4 (see Recommendation #3.1, Section 5, Chapter 4). This type of flexible structure—which is Nema Final Qualitative Evaluation. Chapter 5. Cross Cutting. December 30, 2013. Draft. 111 recognized by the Mali government—will make it easier for future programs to insure that the local groups are aware of and in communication with some of the most relevant government (hydraulic, sanitation, health sanitation) and local NGO programs (Caritas, Tassaghet, Norvege, Near East Foundation, Red Cross, etc.) that they need to sustain their activities in normal times as well as crises. • Most Nema-created groups are still informal—i.e. not recognized by the government: The strongest organizations in the current context are those related to income-generating activities (IGAs)—the SILCs and AEGs—but, unfortunately, the program closed before most of these groups could file for their official legal recognition from the government as cooperatives (for the AEGs and SILCs) or as networks (reseau). This type of legal recognition is critical to their sustainability and impact since it affects their ability to access micro-finance and a host of other government supports, as well as to benefit from other donor programs. 4.2.4. Resilience: There is clear evidence from the participants and key informant interviews that the capacities the groups developed under Nema helped them better manage the crisis by providing internal access to savings and emergency funds (the SILCs), critical inputs and markets for the crops they produced (AEGs), and identifying and managing the food aid that other donors (PAM, Red Cross) provided to the region during the crisis. 5.0. Lessons Learned and Recommendations Based on the assessment on the data from the final evaluation survey, the consultants and field staff who participated in the initial debriefing at San on November 30th 2013 identified a number of major challenges for future literacy and capacity-building for future programs. The consultants synthesized this list with their initial analyses and presented a revised list to the working group members on December 7, 2013 during their final debriefing. Based on the feedback from staff during this initial debriefing, the evaluation participants reached consensus on two major recommendations for future programs. Recommendation 4.1. Strengthen the linkages between literacy training and the actual activities (financial, planning, etc.) of the CBOs. Specific Recommendations: 1. Facilitate a participatory baseline diagnostic (with input from the members) of each group’s literacy needs as a basis for planning a training program and appropriate training materials. 2. Strengthen the existing systems set up under Nema for working with CAP to monitor the quality of the literacy programs. 3. Build on the existing base of Nema capacity monitoring matrices/tools to develop self￾assessment tools that the program can use to help CBOs self-assess their capacity in key technical areas, as well as financial management, strategic planning, and literacy. 4. Facilitate each group conducting an annual update of its capacity index as part of its strategic plan for the coming year. 5. Ensure that the results of the community-based self-assessments of the CBOs’ local capacity are reported in the annual reports and (if possible) in the IPTT. Nema Final Qualitative Evaluation. Chapter 5. Cross Cutting. December 30, 2013. Draft. 112 Recommendation 4.2. Strengthen the capacity of the program managers and administration to monitor the core capacity of the CBOs created by the program in order to better target training and technical support and plan the program’s exit strategy. Specific Recommendations: 1. Anticipate the need to transform certain informal structures and community organizations into formal structures that are recognized by the government, and build this information into the training programs. 2. (Cross–cutting with SO2) Given the critical importance of the village food security committees in emergency and planning with the SAPs, anticipate the development of these groups in all of the future interventions as an umbrella organization that can help coordinate with the other CBOs. 3. Encourage the establishment of a network (reseau) linking the current and any new AEGs and SILCs established under future programs to ensure their access to improved inputs and markets. 4. Strengthen staff training in conflict management to minimize the impact of community￾level conflicts on program activities. 5. Anticipate the key government, NGO, and private-sector linkages that each CBO will need, and monitor the evolution of these linkages as part of the annual self-assessment process to help facilitate an appropriate exit strategy. Nema Final Qualitative Evaluation. Chapter 6. Conclusions. December 30, 2013. Draft. 113 Chapter 6 Conclusions This chapter provides a brief overview of the principal conclusions of the Nema final evaluation that are described in much greater detail in Chapters 1-4 for the program’s principal strategic objectives (SOs) and intermediate results (IRs). Each chapter assesses the program’s impact for the specific SOs based both during and after the program using three data sets: • The Nema program’s internal monitoring and evaluation (M&E) data from Years 1-4 of the program; • The qualitative survey in 21 villages that was conducted over a three-week period in November 2013; and • The key informant interviews that the team conducted in the region and local cercles that enabled the team to better understand the direct impact of the war and current trends on donor and government re-engagement in the communities since January 2013. 1.0. Main Lessons Learned from the Program Experience 1.1. Global Overview of Results This analysis provides clear evidence that in March 2012 Nema was well on its way to achieving most of the original targets it had committed to in the original proposal, and that many of these activities were having a very positive impact on household food security, vulnerable groups, and resilience by the time the program shut down at the end of its fourth year in March 2012. There is also ample evidence from the beneficiary and partner interviews that many of the program’s activities helped the local people better manage the shocks that started descending up on them in June 2011. Specific achievements include: • Helping build the capacity of vulnerable people in agro-enterprise development through the development of agro-enterprise groups (AEGs) and savings and internal lending communities (SILCs), many of which still exist and seem to be functioning at a low level; • Helping introduce a wide variety of new crop technologies that helped local people increase their food security and reduce their vulnerability to drought; • Building a wide range of agro-sylvo-pastoral infrastructure (143 separate structures) that made local people less vulnerable to drought, flooding and other shocks; • Pilot testing a new community health center (Centre de Santé Communautaire or CSCOM) and community-based (PD/Hearth) model for identifying and rehabilitating malnourished children; • Encouraging improved hygiene and sanitation practices through a concerted effort to promote household use of latrines (most of which appear to be in use) that have continued to help improve local sanitation practices; • Constructing 15 water points (14 of which are still functional) in water-starved villages where successive water resource development programs had failed time after time; • Strengthening the community-based systems for emergency early warning and response by developing a: (a) community-based system of early warning and response (Groupements d’Alerte Precoce et Reponse d’Urgence or GAP/RU); and (b) Safety Net Nema Final Qualitative Evaluation. Chapter 6. Conclusions. December 30, 2013. Draft. 114 Committee that was capable of working with outside agencies to coordinate a series of post-Nema food aid responses during the crisis; and • Creating a system of community-based literacy training that helped build the capacity of all the community-based organizations (CBOs) being supported by the program. 1.2. Capacity for a Rapid Start Up in the Case of a Recuperation Effort Based on key informant interviews, the Ministry of Agriculture extension agents estimated that about 40% of the villages are safe enough for routine government services to start back up (Level 3 and 4, Table 6.1). As of November 2013, about 47% of the Douentza villages were estimated to have “average” or “good” extension coverage, and about 26% of the Bourem villages (Table 6.2). Only 29% of the Douentza villages and 17% of the Bourem villages were considered too dangerous for agents to work there (Table 6.2). There are also a growing number of international development donors and local non-governmental organizations (NGOs) moving back in especially in the Bourem region (Table 6.3). Future programs should probably consider a progressive re-engagement in the area: • Level I Villages: One category of re-engagement should probably follow the progressive re-instatement of ‘normal’ government services that is already very much underway in the some of the less isolated villages, which were also those that were less directly affected by the Islamist occupation and internally displaced persons (IDP) movements in 2012 and 2013 (e.g. Level 1, 2, and 3 villages in Table 6.4). These villages are typical of the villages that the evaluation team visited during the Final Qualitative Evaluation Survey. • Level II Villages: The program will probably need to adopt a different approach—one that works in close collaboration with various emergency relief programs that are working with the IDPs—if it chooses to re-engage with the Nema communities that were most directly affected by the war and post-war insecurity that still exists but is getting better and better every day (the Level 5 villages in Table 6.4). Nema Final Qualitative Evaluation. Chapter 6. Conclusions. December 30, 2013. Draft. 115 Table 6.1. Qualitative Assessment of Current Security Levels in the Former Nema Villages in Bourem and Douentza, November 2013105 Qualitative Assessment of Security Levels Bourem Douentza # % # % 0 Very limited security with high levels of risk to vehicles and motorcycles from roving bandits. No permanent security force in those areas; some areas border the Burkina frontier, which is a known hideout area 6 20% 33 33% 1 Very unstable area. No permanent security in these areas. High risk of roadside robberies, which are difficult to monitor 0 0% 26 26% 2 Limited stability. No permanent security forces in the area. Residual banditry 11 37% 1 1% 3 Somewhat secure area with limited banditry. Security agents are on the ground. Villages benefit from routine army patrols because they are located near the main highways where army patrols are focused 13 43% 20 20% 4 Stable areas with limited residual banditry and permanent security forces on the ground 0 0% 20 20% 5 Secure area with permanent security forces and only limited banditry 0 0% 0 0% Total Number of Nema Villages 30 100% 100 100% Table 6.2. Qualitative Assessment of Government Extension Worker Coverage in the Former Nema Villages in Bourem and Douentza, November 2013 Qualitative Assessment of Extension Levels Bourem Douentza # % # % 0 No government extension agents posted or serving the villages 5 17% 17 17% 1 In theory the agents cover these villages, but in reality their contact with the villages is minimal 0 0% 12 12% 2 Extension agents are assigned to the village and support certain village activities remotely 17 57% 24 24% 3 Average level of extension services with some limited support from outside partners (IFRC, GRAT, Oxfam, SCI)* 8 26% 34 34% 4 Good level of extension services supported by the presence of outside partners 0 0% 13 13% 5 Extension services are sufficient to support any existing or emerging unions and federations that producers need to sustain their commercial activities 0 0% 0 0% Total Number of Nema Villages 30 100% 100 100% Source: Same as Table 6.1. *Level at which the Ministry of Agriculture feels comfortable posting an agent. Acronyms: IFRC: International Federation of the Red Cross; GRAT: Groupe de Recherche et d 'Applications Techniques; SCI: Save the Children International. 105 Methodology and Sources: These four tables (Tables 6.1-6.4) are based on information gathered during key informant group meetings co-chaired by Abdou Salam Tiemogo Maiga, (SCI/Gao); Amadou Tadina (Chef du Secteur Agricole, Ministry of Agriculture, Douentza); and Pierre Togo (Caritas/Mali). Theses individuals used their connections with community leaders and the Ministry of Agriculture field agents in the ground to verify certain types of data on all 130 Nema villages. Other key participants in the group meetings included Aboubacar Halidou (HKI); Sidi Maiga (Chef du Secteur Agricole Bourem, by telephone); and Emmanuel Goita (Caritas/Mali). Nema Final Qualitative Evaluation. Chapter 6. Conclusions. December 30, 2013. Draft. 116 Table 6.3. Qualitative Assessment of the Major Outside Development Partners Currently Working in the Former Nema Villages in Bourem and Douentza, November 2013 Partners Number of Nema Villages Where They are Working Bourem (n=30) Douentza (n=100) # % Total # % Total OXFAM 25 83% IFRC 5 17% 6 6% GRAT 12 40% UAVES 5 17% SCI 6 20% AFAR 4 13% ALCOP 10 10% PDI-MUZELEM 3 3% IESA 1 1% Purdue/INTSORMIL /McKnight 1 1% Source: Same as Table 6.1. Acronyms: OXFAM: Oxfam International; IESA: Initiative Eau et Securité Alimentaire; ALCOP: Aluminum Company of Pakistan; PDI-MUZELM: Projet de Developpement Integre, Mission Evangelique Lutherienne au Mali; UAVES: Union pour un Avenir Ecologique et Solidaire; AFAR: Appuix aux Systemes Ruraux de Production. Table 6.4. Qualitative Assessment of the Direct and Indirect Impact of the 2012 Occupation on the Former Nema Villages in Bourem and Douentza, November 2013 Qualitative Assessment of Impact Bourem Douentza # % # % 0 Very little affected 0 0% 0 0% 1 Very little affected. Little direct contact with the occupying forces but limited access to markets 3 10% 8 8% 2 Limited impact but occasional contact with the occupying forces (estimated direct contact once every 15 days) 3 10% 16 16% 3 Somewhat affected. More regular direct contact but not very affected by displacements from the villages 1 3% 16 16% 4 Affected. Regular contact with the Jihadists but no permanent military bases in the villages. Important population displacement from the villages during the occupation. 6 20% 9 9% 5 Directly affected. Regular direct contact and military presence in the villages (i.e. brigades, application of sharia law, military bases, sharia sentencing and courts, involuntary recruitment of children). Larger scale displacement from certain villages 17 57% 51 51% Total Number of Nema Villages 30 100% 100 100% Source: Same as Table 6.1. 2.0. Lessons Learned and Recommendations Although the net impact of these activities was very positive, the survey identified a number of corrective actions that are needed to sustain and strengthen the benefits that were initiated by Nema during its first four years of operation in Douentza and Bourem. 2.1. Strategic Objective 1: Livelihood Strategies More Profitable and Resilient Nema Final Qualitative Evaluation. Chapter 6. Conclusions. December 30, 2013. Draft. 117 2.1.1. Summary Observations: Based on the results of the food security calendars that were conducted during the final qualitative evaluation, the project did increase agricultural production and revenues for the households of the AEG and SILC groups that were the principal target of this activity. There is also a great deal of evidence that the program’s wide-ranging farmer field school (FFS) activities increased farmer’s access to new higher-yielding technologies. The short￾term impact of this integrated approach was a substantial increase in the average Months of Adequate Household Food Provisioning (MAHFP) during the program. Although the results of the MAHFP analysis that was conducted during the final qualitative field survey suggest that the current MAHFP levels in the survey villages are about the same level as at the program’s baseline, this is not surprising given the successive shocks that the study villages have experienced since June 2011. This analysis highlights the critical importance of strengthening various program activities that will enable the Nema program to have a more broad-based impact on whole communities, rather than an impact that is based on just building the capacity of the AEGs and the SILCs, which were created in fewer than 50% of the program villages.106 Although the program’s successful development of a new model of FFSs in 90% of the Douentza villages and 50% of the Bourem villages did enable it to have a broader population-based impact, this was insufficient in and of itself to guarantee the farmers’ access to all of the inputs they would need to sustain these broader impacts over time. Future programs need to capitalize on the core capacity of the AEGs and SILCs that were developed during the first phase to strengthen the local communities access to the improved seeds and inputs that they will need to have a broader population-based impact on yields and household revenue. This includes building more sustainable sources of income for the Training of Trainers (TOTs), FFS trainers, and SILC trainers (known as AVs or Animateurs Villageois) that will enable them to continue to be community-based change agents and contact persons for the government agricultural research and extension services and regional micro-finance institutions that the farmers need to sustain higher yields and incomes. A critical next step for any future plan must address these issues at two levels: • The first level involves reactivating and strengthening the existing AEG, SILC, and FFS activities in the areas where the program was already active; and • A second level involves extending the core program into the more marginal villages that did not generally benefit from either AEG or SILC activities during the first phase of the program. Due to the brusque cut off of the program—and the fact that many of these villages were not considered eligible for AEG—this represents about 50% of the program villages. Many of these villages are agro-pastoralist in vocation and were not particularly suited to the core program that was promoted by the FFSs. In contrast to the first level of villages where the program was most active in the first four years, this second group of villages are likely to need extensive FFW investment to create new opportunities for 106 Only 30% of the Nema villages in Bourem have AEGs, and only 38% of the Douentza villages (Table 2.4, Chapter 2); SILCs were created in only 57% of the Bourem villages and only 58% of the Douentza villages (Table 2.4, Chapter 2). The FFSs were organized in 50% of the Bourem Villages and 90% of the Douentza Villages (Table 2.4, Chapter 2). Nema Final Qualitative Evaluation. Chapter 6. Conclusions. December 30, 2013. Draft. 118 commercial livestock and irrigated gardening for local groups, which could then be organized (once these new activities are created) into AEGs and SILCs. Both categories of communities—those that received the full Nema package during the original program and those that did not—would likely benefit from: • An expansion of the Nema program’s efforts to strengthen the linkages between its SO1 and SO3 activities and the promotion of fruit trees and micronutrient-dense tuber and food crops; • A focused program to help the community-based volunteers and CBOs (AEGs and SILCs) that are charged with executing these programs by helping the volunteers to develop income generating activities (IGAs) that are related to their extension activities and by building the capacity of the SILC and AEG groups so that they can sustain these activities once the program ends; and • Extending certain types of assistance and capacity building (such as training) to some of the pre-existing groups—like registered cooperatives--that may be already active in the communities. 2.1.2. Key Recommendations (see Table 6.5 at the end of this chapter for the specific recommendations for implementing the key recommendations): • Recommendation 1.1. Consolidate the achievement of Nema in the first phase in the communities where the SO1 activities were concentrated during the initial Phase (Years 1-4). • Recommendation 1.2. Extend the total package of SO1 innovations into the Nema communities where the program was less active in the first phase in ways that build on the capacities of the local agro-ecological capacity. • Recommendation 1.3. Strengthen integration between the Nema program’s livelihood and safety net activities and the project’s efforts to promote improved nutrition and health in all villages. • Recommendation 1.4. Transform and certify the FFS Trainers (TOT) and help them become commercial seed producers as a way to strengthen their role as community-based extension agents that serve the AEGs and SILCs, as well as other groups within the community such as pre-existing cooperatives. • Recommendation 1.5. Strengthen the organizational capacity of the local service providers that are essential to the successful achievement of the activities. • Recommendation 1.6. Require all future AEGs to be organized into cooperatives before co-financing their business plans based on measured levels of performance. • Recommendation 1.7. (Cross cutting with SO3, capacity building, and M&E) Develop new indicators based on existing tracking tools that will enable the Consortium to better monitor its impacts on community-level capacity building and vulnerable groups. 2.2. Strategic Objective 2: Children Under 5 Years Less Vulnerable to Illness and Malnutrition 2.2.1. Summary Observations: One direct impact of the Nema program’s support for the CSCOM-based programs to identify and rehabilitate malnourished children was a sharp increase in the number of children identified and treated. Not surprisingly, there was a sharp precipitous Nema Final Qualitative Evaluation. Chapter 6. Conclusions. December 30, 2013. Draft. 119 decrease in the number of children identified and treated once the project quit providing food assistance to the families of the children being rehabilitated. In 2012, the same clinic records that were analyzed during the final survey are currently (November 2013) showing a sharp increase in the number of children being identified as malnourished, but very few being rehabilitated either at the CSCOM or in the villages since most mothers argued that they could no longer afford either the expense or time that this type of rehabilitation would involve. The chief reason given for the very limited sustainability of these gains was that most of the Nema health activities were implemented as if they were emergency programs, with high levels of food rations being distributed to the malnourished children including a ‘protective’ ration for the families. Although certain mothers of malnourished children who were treated by the program continue to model some of the good nutritional and sanitation practices that they learned during the program, but this impact varied enormously between activities: • The mothers who reported the most active follow-up and continued practice were those that had participated in the Nema program’s limited community-based rehabilitation programs for moderately malnourished children following the PD/Hearth model activities in 27 of the 130 Nema villages; and • The survey showed almost no impact on current practices among mothers who had only participated in CSCOM-based rehabilitation programs for moderately and severely malnourished that were supported with the Nema food distribution that were organized in 130 of the 130 Nema villages. 2.2.1.1. PECIMA- The Identification and Treatment of Children Suffering from Moderate Acute Malnutrition (MAM) and Severe Acute Malnutrition (SAM) at the Community Health Centers (CSCOM): Based on this analysis, the health team concluded that the Nema program needs to completely rethink its support to the national Prise en Charge Intégrée de la Malnutrition Aiguë (PECIMA) program through the CSCOMs that it supported through training and food assistance for the families of the children who were identified as malnourished. A critical first step involves strengthening the local communities’ support, involvement, and ownership of the CSCOM-supported growth monitoring and rehabilitation activities. Key recommendations include trying to increase the willingness and ability of the government’s officially recognized community health volunteers (CHVs or relais) to execute the program through: • Improved training; and • Improved compensation of their efforts by making them part of Nema-supported AEGs or SILCs or compensating them through other types of collective labor or in-kind payments. 2.2.1.2. PD/Hearth: Given the successful record of Nema’s pilot PD/Hearth program, the team is recommending that future programs support the development of the community-based PD/Hearth model for rehabilitating moderately malnourished children using local products (i.e. not using food aid). The same recommendation includes a list of recommendations (based on successful PD/Hearth-model programs in other parts of Mali and West Africa) for developing: Nema Final Qualitative Evaluation. Chapter 6. Conclusions. December 30, 2013. Draft. 120 • A simpler, more stream-lined model of PD/Hearth that should be easier for the project and the CSCOMs to backstop and monitor the correct execution of using a ‘model mother’ approach; as well as • A strong program of post-PD/Hearth program support to the vulnerable mothers of the rehabilitated children to help them develop the type of IGAs needed to sustain the positive impacts of the PD/Hearth program on their children. 2.2.1.3. Water/Sanitation: Since clean drinking water is still a pressing constraint in about half the Nema villages due to budget constraints, which restricted the project’s ability to create wells, a third set of recommendations focuses developing a wider network of support for developing potable water-drinking points. Given the successful record of the government-sponsored community-led total sanitation (CLTS, also known as Assainissement Total Piloté par la Communauté or ATPC) program in adjacent areas and the successful pilot testing of this program in five Nema villages during the start-up phase, the health team is encouraging future programs to support the government’s CLTS initiative to promote community sanitation programs. 2.2.2. Key Recommendations: • Recommendation 2.1. Rethink the Nema program’s support for the national government’s PECIMA strategy through the CSCOMs. • Recommendation 2.2. Improve the execution of the PD/Hearth model programs in the program villages. • Recommendation 2.3. Improve access to clean drinking water and strengthen hygiene and sanitation practices by requiring all villages to support the Mali government’s regional and cercle-level CLTS initiative. 2.3. Strategic Objective 3: Targeted Communities Manage Shocks More Effectively 2.3.1. Summary Observations 2.3.1.1. Early Warning and Response Systems: The true results and effects of an effective early warning and response system are: • How it reacts when an actual emergency occurs; and • How it performs when the program that created it is no longer there to assist it. To date, there are very few of the Title II-funded programs that have been tested to the degree that Nema was tested starting in June 2011, at the very end of year three just after the mid-term evaluation. For all these reasons, USAID/FFP is very interested in assessing how the early warning system that Nema set up in Years 1-3 actually functioned when a real crisis hit. One major impact of the GAP/RU community-based early warning and response committees— that the Nema program created in all 130 of the Nema villages—was to provide the regional early warning systems (SAPs or Système d’Alerte Précoce) in both Bourem and Douentza with some of the first data on the emerging food security crisis in in June 2011. Working through some of the most functional Nema-trained GAP/RU and Safety Net Committees, the Consortium was able to very quickly and efficiently: Nema Final Qualitative Evaluation. Chapter 6. Conclusions. December 30, 2013. Draft. 121 • Develop a follow-up proposal for USAID that got emergency food and cash to the program by January 2013; and • Orchestrate the first phase of food security nets and enhanced PECIMA activities of this well-planned emergency response. Another major impact of the program that emerged during the execution of the final qualitative evaluation was the critical role played by the Nema-trained Safety Net Committees in orchestrating the food assistance that moved into the Nema villages six to 10 months after the Nema program was forced to close because of the military occupation. In addition to better targeting the post-Nema food assistance coming in, the trained committees seem to have helped the former Nema communities attract some additional special allocations of food assistance that supplemented the main distributions from the World Food Programme (WFP) and International Federation of the Red Cross (IFRC) (see Table 4.3, Chapter 4). This experience highlights the critical importance of integrating the early warning and response functions of the GAP/RU into a new structure—the Village Food Security Committees—that the Mali government recently proposed as part its SAPs. Although this new function focuses primarily on emergency preparedness, the same structure could provide a framework for improved coordination between all of the different sub-committees being created by the project. 2.3.1.2. FFW as a Cross-Cutting Tool for Increasing Agricultural Production, Raising Yields, and Reducing Risk: The program’s commitment to the creative use of FFW and similar safety￾net programs helped to minimize the risk associated with vulnerable households adopting the new practices and technologies needed to build stronger livelihoods. In the course of execution, the FFW activities built 143 types of basic infrastructure, most of which (based on the data gathered during the survey) is still considered functional. In general, the villagers interviewed during the final evaluation survey expressed their deep appreciation for the FFW—both as a food supplement and as a source of investment in infrastructure that benefitted—in most cases—the entire village. This experience highlights the critical role that FFW-generated infrastructure can play in supporting stronger livelihood systems and reducing the aggregate vulnerability of certain highly food-insecure communities. 2.3.1.3. Safety Nets as a Tool for Helping Vulnerable People Better Manage Risk: Although the safety nets program were much appreciated by the local people, there is little likelihood that the 130 communities will have the means to develop improved safety nets from community resources that are more extensive than the traditional methods they have always used. It is therefore critical for future programs to examine a new string of activities that will reduce the total number of vulnerable people who will need to rely of either these traditional safety nets or outside food assistance to survive. Sample programs might include providing short-term food assistance, technical support, and inputs to vulnerable people that helps them develop profitable IGAs that they can manage with limited labor, like intensive poultry production, stall feeding, or kitchen gardens. 2.3.2. Key Recommendations: • Recommendation 3.1. (Cross cutting with SO1, SO2, and SO4) Anticipate the creation of Food Security Committees in all of the Nema villages to coordinate the different Nema Nema Final Qualitative Evaluation. Chapter 6. Conclusions. December 30, 2013. Draft. 122 and non-Nema community structures that affect food security with a special accent on the activities started under SO3. • Recommendation 3.2. Anticipate the involvement of all the key partners (Village Food Security Committee, Red Cross/Mali, SAP) in the conception and execution of any future community-based early warning and emergency response system. • Recommendation 3.3. (Cross cutting with SO1) Anticipate activities that strengthen the food security of the most vulnerable households that reduce their dependence on community-based safety nets. • Recommendation 3.4. (Cross cutting with SO2) Strengthen the linkages between SO3 and SO2 activities. • Recommendation 3.5. (Cross cutting with SO1) Intensify FFW activities in the most vulnerable villages in order to build their food security and income and reduce community-level vulnerability to risk and shocks. • Recommendation 3.6. (Cross cutting with M&EAL) Strengthen the capacity of the Nema M&E system to monitor the activities of the Village Food Security Committees and other key CBOs. 2.4. Cross-Cutting Objectives: Basic Literacy, Good Governance and Capacity Building 2.4.1. Summary Observations: One major impact of the Nema program was to build the capacity of local CBOs around the activities that they created. Given the high illiteracy rates in the area, the project proposal anticipated the need for literacy training to equip these groups with the basic skills they would need to manage group enterprises. Although the two issues—literacy training and core capacity—overlap, they were evaluated separately with a set of interconnected recommendations. 2.4.1.1. Literacy: The program created literacy centers and developed trainers in all 130 of the Nema villages. Although the short-term output of this exercise was an apparent increase in the number of literate people involved in the different CBOs, the rate of drop out was high and many of the groups still have very few members who understand the most important tools that they need to execute and sustain their interventions. Future programs can build on this existing base of literacy centers and trainers in order to: (a) continue to offer basic literacy training; and (b) develop a more focused set of “stage two” trainings that emphasize the actual tools needed to run a specific group enterprise. Given the local peoples’ appreciation of the literacy training, this is a clear priority for the next phase, and one that needs to be linked to a cross-cutting program of capacity building. 2.4.1.2. Capacity Building: Although the program increased the capacity of local groups, many of these groups are still quite fragile. A clear priority for the next phase will be to: (a) strengthen the capacity of the existing groups; and (b) develop new groups in the villages that were underserved during the first phase. The ultimate goal of this capacity building is to build the organizations’ capacity to manage the activities without the project. To achieve this objective— which the final evaluation has shown is critical to having a sustainable impact on resilience—the program will need to strengthen its existing systems for monitoring CBO capacity. 2.4.2. Key Recommendations: Nema Final Qualitative Evaluation. Chapter 6. Conclusions. December 30, 2013. Draft. 123 • Recommendation 4.1. Strengthen the linkages between literacy training and the actual activities (financial, planning, etc.) of the CBOs. • Recommendation 4.2. Strengthen the capacity of project managers and the administration to monitor the core capacity of the CBOs created by the project in order to better target training and technical support and plan the project’s exit strategy. To achieve this recommendation—which the final evaluation has shown is critical to having a sustainable impact on resilience—the program will need to strengthen its existing systems for monitoring CBO capacity. Specific recommendations for doing this would be to: - Transform the existing tools the Nema program has used to track capacity in the AEGs and SILCs into self-assessment tools that the program could use to track the evolution of the key capacities they would need to sustain their activities once the program closed; and - Train local groups to use this tool (with support from the Nema agents and partners) as a basis for developing strategic plans and training activities, including those that are designed to cross-link with the new generation of more focused literacy programs. • Once these tools are in place, they will provide a useful basis for both Nema and the local partners to use to plan the next generation of the program’s exit strategy as well as progressive engagement with a wider circle of NGO and government partners in the region that the villages will need to sustain these activities over time. 2.5. Monitoring & Evaluation 2.5.1. Summary Observations: The final evaluation qualitative survey was completed with speed and efficiency by the CRS M&E specialists due to their ability to enter and analyze the qualitative surveys using a series of CRS data-base management systems that were pre-adapted to the forms. This data entry analysis facilitated the formulation of certain simple but very powerful analyses that are presented in the report, a brief summary of which will be presented here. The Nema M&E systems’ sophisticated data base summarized all the tracking data for the program from Years 1-4. The original model was the CRS Simple Measurement of Indicators for Learning and Evaluation-Based Reporting (SMILER) system. This system exists because the initial Nema M&E team took the unusual step of developing a single data bank in the first year of the program that facilitated data entry. This data base management system was soon recognized as an example of best practice and generalized to all the CRS programs in Mali. The fact that it was pilot tested on Nema is important because it facilitated a disaggregated analysis of the program data by region and by gender that would have been impossible to do retroactively. Three of the key challenges for the next phase will be for Nema to build on this existing base of solid M&E systems to strengthen the capacity of Nema project to track: • The percentage of Nema villages classified as chronically food insecure; • The percentage of households classified as chronically food insecure; and • The core capacity of the local organizations charged with sustaining these activities as a basis for the development of successful exit strategies. Nema Final Qualitative Evaluation. Chapter 6. Conclusions. December 30, 2013. Draft. 124 3.5.2. Key Recommendations: Given the critical importance of these recommendations, they have been mainstreamed into the recommendations for the main SOs. Table 6. 5. Corrective Actions Recommended for Increasing the Sustainability of the Nema Program’s Accomplishments and Monitoring or Strengthening the Benefits Initiated by the Program107 Recommendations Specific Recommendations SO1: Livelihood strategies more profitable and resilient Recommendation 1.1. Consolidate the investments made by the Nema program in building agricultural production and increasing revenues in the communities where the SO1 activities were most active during the initial phase (Years 1-4). 4. AEGs: Assess the current level of functioning activities of each AEG, including the profitability of the original activities it has engaged in, and: - Build the core institutional capacity they need to become registered cooperatives and/or to join and established cooperatives; - Identify any technical assistance they might need (such as increased access to improved food-processing technology) to further sustain their existing agro￾economic enterprises or any new ones they have developed; and - Strengthen the program’s collaboration with the local technical services and donor-funded initiatives (e.g. INSORMIL/Purdue and IESA) that support the new intensive technology package in order to facilitate their support for the scale up of the new technologies pilot tested during the FFSs for the Nema target villages. 5. SILCs: Conduct a similar sort of institutional review of the SILCs in the initial pre-planning phase of any sort of follow-up programming. 6. Track the impact of this activity on the aggregate vulnerability of the communities as a basis for planning a program exit strategy (see Recommendation 1.7). Recommendation 1.2. Extend the total package of SO1 innovations into the Nema communities where the program was less active in the first phase in ways that build on the capacities of the local agro-ecological capacity. 1. Target underserved, vulnerable villages with an intensified package of FFW activities to build their capacity for the agro-enterprises that they can develop (especially livestock, commercial forage production, and irrigated gardening). 2. Encourage commercial fodder production. 3. Develop a new group of safety net programs that focuses on the use of individualized FFW that vulnerable households can execute to help them develop small micro￾enterprises (like poultry production, stall feeding, manure pits, and kitchen gardens). 4. Build the core capacity of the team to support commercially oriented, sustainable livestock activities by recruiting an senior animal scientist (zoo-technician) who can provide appropriate technical assistance and intensified linkages to the emerging livestock initiatives being developed by the Mali Ministry of Livestock and international partners like the International Livestock Research Initiatives (ILRI), as well as some of the new livestock initiatives being supported by USAID in Burkina Faso and Niger. 5. To insure appropriate conclusion of women and vulnerable 107 CRS/HKI/Save the Children. 2013. SOW. Nema Final Qualitative Evaluation. Lessons Learned and Recommendations. Key Questions. Nema Final Qualitative Evaluation. Chapter 6. Conclusions. December 30, 2013. Draft. 125 Recommendations Specific Recommendations groups, a progressive scale up of the SILCs should follow the progressive scale up of the AEGs in the most vulnerable communities. 6. Track the impact of this activity on the aggregate vulnerability of the communities as a basis for planning a program exit strategy (see Recommendation 1.7). Recommendation 1.3. Strengthen the level of integration and joint planning between SO1 and SO2. Cross-cutting with SO2: 1. Encourage the mothers of children graduating from the SO2-supported PD/Hearth to organize themselves into mothers groups that are SILCs. 2. Provide the mothers’ group with technical assistance and support to help them increase their crop production and revenues. 3. Given the difficulty that all of the PD/Hearths have had in providing the basic ingredients and fuel wood needed to support the PD/Hearth in Bourem, link future FFW investment in the development of irrigated market gardens (culture maraichages)—one of the most potentially profitable value chains in Bourem—to the group’s commitment to: (a) providing a certain percentage of their harvest to the PD/Hearth in that community; and (b) planting a haie vive (around the plot) that the women’s group can maintain as a source of fuel wood for the future PD/Hearth. 4. Track this increased capacity through a new self￾assessment tool and report on it regularly to strengthen the preparation of any new program’s exit strategy (see Recommendation 1.7). Recommendation 1.4. Transform and certify the FFS trainers and help them become commercial seed producers as a way to strengthen their role as community-based extension agents for both the Nema-founded extension groups as well as for other pre-existing community groups. 1. Retrain current and train new TOTs in basic skills, as well as the skills needed to become certified seed producers through the IER program designed to promote this. 2. Provide technical assistance and monitoring to facilitate the TOTs becoming seed private-service providers (PSPs). 3. Track this increased capacity through a new self￾assessment tool and report on it regularly to strengthen the preparation of the exit strategy (see Recommendation 1.7). Recommendation 1.5. Strengthen the organizational capacity of the local service providers that are essential to the successful achievement of the activities. Cross-cutting with literacy and capacity building: 1. Both categories of AEG and SILC capacity building (listed under Recommendations 1 and 2) would benefit from literacy training programs that build the capacity of the members with basic reading and writing skills (see Tables 2.5 and 2.5) to manage the basic AEG and SILC planning and financial tools. SILC: 2. Certify the SILC AVs in PSP. 3. Finalize the process of training the AV in PSP and help them to create a network (reseau). 4. Better connect the SILC groups with the existing base of institutions of micro-finance (IMFs). 5. Facilitate the transformation of certain groups (as they develop) into AEGs and ultimately, when they are ready, into cooperatives. 6. Track this increased capacity through a new self￾assessment tool and report on it regularly to strengthen the preparation of the exit strategy (see Recommendation 1.7). Nema Final Qualitative Evaluation. Chapter 6. Conclusions. December 30, 2013. Draft. 126 Recommendations Specific Recommendations Recommendation 1.6. Require all future AEGs to be organized into cooperatives before co￾financing their business plans based on measured levels of performance. 1. Strengthen the organizational capacity of the AEGs before financing their business plans. 2. To insure fiscal responsibility and allow them to approach local micro-enterprise institutions, require them to be registered cooperatives before co-financing their business plans. 3. Anticipate the need to provide the cautionnement (collateral) for the AEGs initial IMF loans. 4. Track this increased capacity through a new self￾assessment tool and report on it regularly to strengthen the preparation of the exit strategy (see Recommendation 1.7). Recommendation 1.7. (cross-cutting with SO3, capacity building, and M&E) Develop new indicators based on existing tracking tools that will enable the Consortium to better monitor its impacts on community-level capacity building and vulnerable groups. 1. Anticipate the need to disaggregate all of the data for the principal internal and donor driven indicators from the start. 2. Train all execution partners (M&E officers and program managers) in the methodologies needed to conduct disaggregated IPTT analyses for their zone of information. 3. Consider adding a new indicator to the SO1 impact indicators that measures the percentage of households classified as chronically food insecure. 4. Consider adding a new indicator measuring percentage of villages classified as chronically food insecure to the tracking table and developing targets for reducing the number of villages classified as chronically food insecure and/or vulnerable. 5. Consider adding a qualitative measurement of the MAHFP using the ‘food security calendar’ methodology used during the final evaluation. 6. Develop a more consistent system for tracking and reporting local organizational capacity building in the IPTT and reports that builds on the existing AEG and SILC tools (see Annex I, Chapter 2) that are already being used to track capacity. 7. Move in the direction of turning the tool into a self￾assessment tool that builds on existing tools being used by the AEG and SILC officers to track their programs that the Consortium can use to build local communities’ capacity to develop realistic exit and sustainability strategies. SO2: Children under 5 years less vulnerable to illness and malnutrition Recommendation 2.1. Rethink the Nema program’s support for the national government’s PECIMA strategy through the CSCOMs. 9. Retrain the CHVs and provide them with training materials (IEC support picture boxes, posters, pamphlets, etc.) on the prevention and community-level treatment of acute malnutrition. 10.Build public awareness about the need to support the CHVs though the development of income generating activities (IGAs) through the AEGs and SILCs, community support for their farming activities, or setting up a system of in-kind compensation through the mobilization of community resources. 11.Strengthen the baseline training of the health personnel (at the CSCOM and CSREF levels) in the identification and treatment of moderately malnourished children, and organize regular retraining sessions. 12.Provide additional training on nutrition for the SILC and AEG members to help them better train the mothers and fathers in the monitoring and community-based Nema Final Qualitative Evaluation. Chapter 6. Conclusions. December 30, 2013. Draft. 127 Recommendations Specific Recommendations rehabilitation of moderately malnourished children. 13.Anticipate the need for a progressive transfer of knowledge and responsibility to the CHVs and make sure they have the tools and equipment they need to conduct growth monitoring in compliance with the national norms for fighting malnutrition. 14.Integrate the promotion of micro-nutrient rich foods into the routine activities of the CHVs, and put them in contact with other community-based structures or women contact persons in order to promote appropriate health messages concerning dietary diversity and the promotion of local foods that are rich in micro nutrients (vitamin A and iron). 15.Introduce food technology activities to process local food (production of weaning foods, local foods enrichment). 16.Strengthen the frequency of in-home training visits by the CHVs and volunteer mothers in order to create the enabling environment required for behavior change. Recommendation 2.2. Improve the execution of the PD/Hearth-model programs in the program villages. 1. Given the observed impact of the PD/Hearth model programs on mothers’ health practices both during and after the program, future Nema interventions should implement a revised PD/Hearth model in every village where the program intervenes that includes revised model for tracking the children’s progress both during and after treatment.108 2. Encourage the development of IGAs for the vulnerable mothers of the children participating in the PD/Hearths through the development of irrigated gardening activities and organizing them into care groups. 3. Involve community volunteers in early PD/Hearth activities, and strengthen the CSCOM’s supervision of the CHVs (at least twice per quarter) to ensure sustainability of the achievements in the field, especially in new areas of program intervention. 4. Introduce food technology activities for local food processing (production of weaning foods, local foods enrichment). 5. Transform the earlier Nema "voluntary mothers" system into the system of "Mama Lumieres" used by most non￾emergency PD/Hearth programs. Recommendation 2.3. Improve access to clean drinking water and strengthen hygiene and sanitation practices by requiring all villages to support the Mali government’s regional and cercle-level CLTS initiative. 1. Since drinking water is still one of the principal constraints to improving the nutritional status of the most vulnerable population in the villages, the consortium partners might think of exploring other opportunities for funding drinking water infrastructure through other organizations or private￾donor funds. These funds, as well as any future funds, need to give priority to water resource development in the most vulnerable villages. 2. Future programs should anticipate introducing the government’s new CLTS initiative, which was pilot tested in five of the Nema villages, in every one of the Nema communities. Recommendation made by the partners and beneficiaries: 108 This includes using weight/height index and monitoring the children’s progress two months, six months, and one year after the children leave PD/Hearth. Nema Final Qualitative Evaluation. Chapter 6. Conclusions. December 30, 2013. Draft. 128 Recommendations Specific Recommendations 3. During interviews with the technical partners of state services, some recommendations were made to improve the care of malnutrition (Text Box 3.5). Beneficiaries also made recommendations during focus groups. SO3: Targeted communities manage shocks more effectively Recommendation 3.1. Anticipate the creation of Food Security Committees in all of the Nema villages to coordinate the different Nema and non￾Nema community structures that affect food security, with a special accent on the activities started under SO3. 1. Transform the existing level of collaboration between the different Nema-created structures that are functioning as an informal food security into a series of Village Food Security Committees (Comite villageois de sécurité alimentaire). 2. Conduct a participatory needs assessment in each of the former Nema program villages to determine the level of functioning of the SO3 structures put in place and how the community managed the crisis. 3. In conjunction with this participatory needs assessment, identify all of the key humanitarian assistance partners (including the local Red Cross volunteers and brigades) and their activities in the Nema villages during the crisis. 4. Create a self-assessment tool that the Village Food Security Committees can use to self-assess the core capacities that they will need to manage routine risk and major shocks. 5. Based on this baseline self-assessment, help each committee develop a strategic plan for building their capacity in collaboration with the program and other partners such as Red Cross/Mali. 6. Based on these needs assessments, work with local aid and SAP partners to develop joint training programs that will strengthen the Village Food Security Committees and sub￾committees (e.g. the Safety Net Committee and the GAP/RU) and build strong regional networks that can backstop these committees during a crisis. 7. Require each Village Food Security Committee to update its self-assessment index (which includes an assessment of their level of collaboration with other aid partners and SAP structures outside the community) and try to insure that some of this information is tracked in indicators in the IPTT. Recommendation 3.2. Anticipate the involvement of all the key partners (Village Food Security Committees, Red Cross/Mali, and SAP) in the conception and execution of any future community-based early warning and emergency response system. 1. Insure that the regional and local SAPs and major humanitarian organizations working in the area are involved in the initial conception of any future early warning and response system (i.e. in the MYAP design), as well as its implementation. 2. Anticipate signing a MOU with the aid partners that might be needed in a major emergency as part of any future MYAP designs to facilitate quick response and more realistic emergency preparedness for worst-case scenarios. 3. Each year, conduct a participatory review of all MOU and processes needed to activate them with the Village Food Security Committees and update the agreements if the context has changed. 4. Continue to support the GAP/RU sharing the early warning information that they collect on the local community radio stations. 5. Anticipate the necessity of linking the GAP/RU data￾collection efforts to the larger data needs of the SAPs and Nema Final Qualitative Evaluation. Chapter 6. Conclusions. December 30, 2013. Draft. 129 Recommendations Specific Recommendations other humanitarian partners. Recommendation 3.3. Anticipate activities that strengthen the food security of the most vulnerable households that reduce their dependence on community based safety nets. 1. Strengthen current safety net programs with activities that develop IGAs for the most vulnerable households. 2. Help vulnerable groups to organize themselves into groups to facilitate the supervision and support of activities that will strengthen their livelihoods and reduce the case loads of future safety-net programs. 3. Anticipate a sub-category of FFW activities for vulnerable groups (as both individuals and groups) that will support the development of IGAs. 4. Develop appropriate M&E systems for tracking the impact of these activities on vulnerable groups. Recommendation 3.4. Strengthen the linkages between SO3 and SO2 activities. 3. Intensify the existing efforts of Nema to use FFW to expand the development of fruit trees and gardens managed by women. 4. Anticipate FFW activities that can help the savings and internal lending communities (SILC) groups develop irrigated vegetable gardens that increase their revenue and household dietary diversity. Recommendation 3.5. Intensify FFW activities in the most vulnerable villages in order to build their food security and income and reduce community￾level vulnerability to risk and shocks. 1. Identify FFW activities needed to jumpstart agro-enterprise development in the most vulnerable villages. 2. Develop appropriate M&E systems for tracking the impact of these activities on household and village-level vulnerability. Recommendation 3.6. Strengthen the capacity of the Nema M&E system to monitor the activities of the Village Food Security Committees and other key community-level management structures. See Recommendation 3.1, sub-recommendations 1-7 above. Cross-Cutting Literacy/Capacity Building Recommendation 4.1. Strengthen the linkages between literacy training and the actual activities (financial, planning, etc.) of the CBOs. 6. Facilitate a participatory baseline diagnostic (with input from the members) of each group’s literacy needs as a basis for planning a training program and appropriate training materials. 7. Strengthen the existing systems set up under Nema for working with CAP to monitor the quality of the literacy programs. 8. Build on the existing base of Nema capacity monitoring matrices/tools to develop self-assessment tools that the program can use to help CBOs self-assess their capacity in key technical areas, as well as financial management, strategic planning, and literacy. 9. Facilitate each group conducting an annual update of its capacity index as part of its strategic plan for the coming year. 10.Ensure that the results of the community-based self￾assessments of the CBOs’ local capacity are reported in the annual reports and (if possible) in the IPTT. Recommendation 4.2. Strengthen the capacity of the program managers and administration to monitor the core capacity of the CBOs created by the program in order to better target training and technical support and plan the program’s exit strategy. 6. Anticipate the need to transform certain informal structures and community organizations into formal structures that are recognized by the government, and build this information into the training programs. 7. (Cross–cutting with SO2) Given the critical importance of the village food security committees in emergency and Nema Final Qualitative Evaluation. Chapter 6. Conclusions. December 30, 2013. Draft. 130 Recommendations Specific Recommendations planning with the SAPs, anticipate the development of these groups in all of the future interventions as an umbrella organization that can help coordinate with the other CBOs. 8. Encourage the establishment of a network (reseau) linking the current and any new AEGs and SILCs established under future programs to ensure their access to improved inputs and markets. 9. Strengthen staff training in conflict management to minimize the impact of community-level conflicts on program activities. 10.Anticipate the key government, NGO, and private-sector linkages that each CBO will need, and monitor the evolution of these linkages as part of the annual self￾assessment process to help facilitate an appropriate exit strategy. Source: Chapters 1-5.