Food and Nutritional Security Program PROSAN- Rayuwa A Title II program of USAID Final Evaluation Report October 2011 i i PROSAN- Rayuwa Final Evaluation Report TANGO International October 2011 Acknowledgements This evaluation report incorporates a tremendous amount of effort by a large number of people, each of whom played a critical role to the success of the mission. The consultants would like to express our heartfelt gratitude to the staff of CRS, CARE, HKI in Niger, to USAID and all of its partners, and to beneficiaries for their full cooperation in the conduct of the evaluation. Everyone made him or herself available and was open to a full discussion of program events and how they have unfolded over the course of the program’s implementation. Finally, our greatest debt of gratitude is to the people of the districts surveyed who so warmly offered their unmeasured hospitality in their communities and homes. Despite the interruptions that we caused in their daily routines, they shared their experiences, and became our partners in research. We hope that this effort may someday make a difference in their very difficult lives. To all these people, we reiterate our sincere thanks and acknowledge their respective contributions to this evaluation. ii ii PROSAN- Rayuwa Final Evaluation Report TANGO International October 2011 ACRONYMS AND ABBREVIATIONS ASC Community Health Agent AOP Annual Operation Plan CSI Integrated Health Center CVD Village Development Committee CCC Communication for Behavior Change COSAN Health Committee COGES Management Committee CSCOM Health Community Center CRS Catholic Relief Services CS Cooperating Sponsors (Africare; CRS, HKI, CARE) DAP Development Activity Plan DIP Detailed Implementation Plan ECD District Support Team FFW Food for Work FFP Food for Peace GAA Food Support Group GSA Breastfeeding Support Group HKI Helen Keller International IB Bamako Initiative IDA Health II project funded by the World Bank IEB Immediate and Exclusive Breastfeeding IEC Information, Education and Communication INRAN Niger National Agronomic Research Institute ISA Food Security Initiative iii iii PROSAN- Rayuwa Final Evaluation Report TANGO International October 2011 MYAP Multi-Year Assistance Program ONG Non-Governmental Organization PEV Extended Vaccination Program PN Project Number SCVM Households Livelihood Security SIS Health Information System S&N Health and Nutrition SCAP-RU Community Early Warning System and Responses to Emergencies SRO Oral Rehydration Drink SSS Salted and Sweet Drink USAID United States Agency for International Development iv iv PROSAN- Rayuwa Final Evaluation Report TANGO International October 2011 Contents Acknowledgements .......................................................................................................... i ACRONYMS AND ABBREVIATIONS .........................................................................ii Index of Tables................................................................................................................ v Index of Figures ............................................................................................................. vi EXECUTIVE SUMMARY ........................................................................................... vii I. INTRODUCTION ................................................................................................... 1 A. Background ......................................................................................................... 1 B. Objectives of (the) MYAP-PROSAN ................................................................... 2 C. Critical Issues in the Operating Environment of MYAP ....................................... 3 D. Objectives of this evaluation ................................................................................ 4 E. Evaluation Methodology ...................................................................................... 5 II. PROSAN Activities and Achievements ................................................................... 9 A. SO1 Activities and Achievements ...................................................................... 10 B. SO2 Activities and Achievements ...................................................................... 21 C. SO3 Activities and Achievements ...................................................................... 40 D. Program Impacts ................................................................................................ 42 III. PROSAN PROCESS ASSESSMENT ................................................................ 51 A. Overview of management systems ..................................................................... 51 B. Monitoring and Evaluation................................................................................. 51 C. Capacity development, partnership and sustainability ........................................ 53 D. Resource Management ...................................................................................... 54 IV. Conclusions and Recommendations ................................................................... 55 Appendix A: IPTT Indicators ........................................................................................ 62 v v PROSAN- Rayuwa Final Evaluation Report TANGO International October 2011 Index of Tables Table 1: Number of households interviewed, by region ................................................... 8 Table 2: Household awareness of and participation in CVDs, by region .......................... 9 Table 3: Adoption of agricultural practices, by region ................................................... 11 Table 4: Adoption of agricultural practices, by SO1 participation category .................... 12 Table 5: Percentage of HH reporting “extremely satisfied” with specific adopted agricultural practices, by SO1 participation category ..................................................... 15 Table 6: Production of millet, sorghum, and cowpea, by region ..................................... 16 Table 7: Production of millet, sorghum, and cowpea, by SO1 participation category ..... 16 Table 8: HH engaged in agroprocessing and group marketing, by region ....................... 18 Table 9: HH engaged in agroprocessing and group marketing by SO1 participation category ........................................................................................................................ 19 Table 10: Household ownership of livestock, by region ................................................. 20 Table 11: Household ownership of livestock, by SO1 participation category ................. 20 Table 12: Adoption of improved livestock practices, by SO1 participation category ...... 21 Table 13: Anthropometric measures of children, by region ............................................ 21 Table 14: IYCF practices, by region .............................................................................. 31 Table 15: IYCF practices, by SO2 participation category.............................................. 33 Table 16: Percentage of children 12-24 months who have received vaccinations, by region ............................................................................................................................ 34 Table 17: Vaccinations, by SO2 participation category .................................................. 34 Table 18: Diarrhea incidence and treatment, by region .................................................. 35 Table 19: Diarrhea counseling received, by region ........................................................ 36 Table 20: Diarrhea incidence and treatment, by SO2 participation category ................... 37 Table 21: Diarrhea counseling received, by SO2 participation category ........................ 38 Table 22: Mean HDDS by survey round and type of support provided .......................... 43 Table 23: Mean MAHFP by survey round, and type of support provided ....................... 44 Table 24: Household production of food crops (millet, sorghum, cowpea), kg/HH, by survey round and district ............................................................................................... 44 Table 25: Anthropometric indicators of U5 children, by region ..................................... 45 Table 26: Percentage of women 15 – 49 with BMI < 18.5, by region ............................. 46 Table 27: Program outcome and impact indicators, by survey round ............................. 47 vi vi PROSAN- Rayuwa Final Evaluation Report TANGO International October 2011 Index of Figures Figure 1: Map of Intervention Zone ................................................................................. 3 Figure 2: Adoption of agricultural practices by SO1 participation group ........................ 13 Figure 3: Adoption of irrigation, by SO1 participation group (% of HH) ....................... 13 Figure 4: Source of developed irrigation, by region ....................................................... 14 Figure 5: Household production of millet, sorghum, and cowpea, by SO1 participation category (median kg/HH) .............................................................................................. 17 Figure 6: Percentage of HH engaged in group marketing, by SO1 participation category ...................................................................................................................................... 18 Figure 7: Percentage of HH engaged in agro-processing activities, by SO1 participation category ........................................................................................................................ 19 Figure 8: Percentage of children aged 6 to 59 months with stunted growth (Height for age <-2SD), by survey round and region .............................................................................. 23 Figure 9: Percentage of underweight children aged 0 to 59 months (Weight for age <-2 SD), by survey round and region ................................................................................... 23 Figure 10: Comparison of wasting in children aged 0 to 59 months (Weight for height <-2 SD), baseline and endline surveys ................................................................................. 24 Figure 11: Percentage of women with BMI < 18.5 kg/m2 by region and survey round ... 24 Figure 12: Comparison of women receiving any antenatal visits, baseline and final surveys .......................................................................................................................... 26 Figure 13: Comparison of mothers with any ANC to mothers with 4 or more ANC visits, by SO2 participation category ....................................................................................... 27 Figure 14: Mean number of antenatal visits for women of children under 24 months, by SO2 participation category ............................................................................................ 27 Figure 15: Comparison of iron supplementation for mothers of U2 children, baseline and final surveys .................................................................................................................. 28 Figure 16: Comparison of women who received vitamin A post partum, baseline and final surveys .......................................................................................................................... 28 Figure 17: Mothers who received vitamin A post partum, by SO2 participation category ...................................................................................................................................... 29 Figure 18: Eexclusive breastfeeding (0-5 months), by survey round and region ............. 31 Figure 19: Infants breastfed within the first hour after birth, by survey round and region ...................................................................................................................................... 32 Figure 20: Household production of major food crops (millet, sorghum, cowpea), kg/HH, by survey round and district........................................................................................... 45 Figure 21: Adoption of recommended practices, by survey round .................................. 47 Figure 22: Household production of staple food crops (millet, sorghum cowpea), by survey round.................................................................................................................. 48 Figure 23: Mean HDDS and MAHFP, by survey round ................................................. 48 Figure 24: Comparison of stunted and underweight children, baseline and final survey . 50 Figure 25: GIS data of households and villages of surveyed sites................................... 53 vii vii PROSAN- Rayuwa Final Evaluation Report TANGO International October 2011 EXECUTIVE SUMMARY MYAP and PROSAN In response to high levels of food insecurity in target areas in Niger, CRS and its partners, CARE, and HKI initiated a five-year USAID Multi-Year Assistance Program (MYAP). Also called PROSAN, the “Programme de Sécurité Alimentaire et Nutritionnelle” was begun in 2007 and completed in 2012. The program consists of three major activities (below) implemented by Catholic Relief Services (CRS/Niger) in partnership with Cooperative for Assistance and Relief Everywhere, Inc. (CARE) and Helen Keller International (HKI) in three geographical areas of Niger: Dosso, Tahoua, and Zinder. The central goal is to reduce food and livelihood insecurity by 2011 for vulnerable households in three districts. The program seeks to enhance economic opportunities for vulnerable groups through increased crop production and income generating activities, and to improve the quality of life through better health and nutrition. Food Security in Niger Niger is an agricultural country where 85 percent of the population is rural and engaged in agriculture as a primary source of income. Ranked 167 out of 169 countries in the Human Development Index, Niger is among the world’s poorest countries, with an average annual income of $357 in 2010; a high infant, child, and maternal mortality rates; and a life expectancy of less than 46 years. Only 46 percent of Nigeriens have access to clean water, and just 12 percent have adequate sanitation. The food security status of households in Niger is determined by the interplay of natural and human resources, the viability of livelihood systems and survival strategies, the politics of resource allocation and use, and the impact of development intervention. Niger is plagued by natural calamities such as persistent drought and climate change including excess rainfall and flooding, this, along with ravages caused by the world economic crisis. The unpredictability of natural disasters, coupled with a profound lack of assets and revenue-generating activities, makes it difficult for households to mobilize sufficient resources to recover from losses while meeting current household needs. Poor roads, or the absence of roads and other physical infrastructure, exacerbate each of the above-mentioned sources of vulnerability of this primarily farming population. These factors have contributed to the deteriorating food conditions and may have negatively impacted certain project results during particular periods and in particular areas, influencing, for example, variable rates of child wasting by region and declining rates of food production and food access from the midline to final evaluation. The latter would be coupled with the more obvious fact that the final evaluation took place during the “hungry season”. Chronic malnutrition (stunting) rates would not have been significantly impacted by the timing of the evaluations since those rates are less influenced by seasonal changes than by long term factors such as poverty, poor access to safe drinking water and sanitation infrastructure, limited access to micronutrient rich foods, limited availability viii viii PROSAN- Rayuwa Final Evaluation Report TANGO International October 2011 and access to family planning services, etc. Yet the analysis of the evolution of the nutritional status of children under five in terms of wasting shows overall improvement since the baseline, another aspect of project success in spite of numerous obstacles. Key Objectives The components of PROSAN build upon past achievements realized in the first Development Assistance Program (DAP-ISAN), which was implemented in 2002 to improve the livelihoods of poor households. DAP’s implementation strategy was based upon participatory community development. Building upon the successes of DAP, PROSAN consists of three strategic objectives to address key behavioral and systemic constraints related to food availability, access and utilization. The objectives and expected results were as follows: 1) To protect and enhance livelihoods of vulnerable households through increased agro-pastoral production and improved agro-enterprises by 2012. 2) To enhance human capabilities through improved health and nutritional practices in vulnerable in targeted communities. 3) To protect and enhance the resiliency of target communities through improved abilities to identify and respond to recurrent shocks by 2012. The Evaluation The final evaluation presented in the Final Report combines qualitative and quantitative data aimed at gaining a comprehensive understanding of how PROSAN has been implemented and how its implementation has succeeded in bringing about the desired changes in the lives and livelihoods of household and institutional participants. The final evaluation aims to: (i) assess the extent to which CRS and its partners have accomplished the stated goals and objectives of the five-year program, as amended; (ii) assess the effectiveness of the technical approach by reviewing program activities that have been successful and those that have not, including reasons why (as much as is feasible); (iii) obtain answers to key questions that may contribute to better applying lessons learned, best practices, sustainability, and recommendations for future programming; and (iv) document and summarize the overarching lessons learned from the program to a wider audience including the consortium, partner organizations, donors, Government of Niger and other stakeholders. The evaluation is designed to capture the maximum amount of information in a limited time frame. It is based on a qualitative and quantitative field study, in addition to monitoring of indicators collected by PROSAN field staff during the course of the program. The methodology deploys a set of complementary research activities, survey instruments, and techniques for gathering and analyzing information. The field evaluation centered on three main groups involved in program activities: PROSAN and its partners’ administrators, coordinators and supervisors at national and ix ix PROSAN- Rayuwa Final Evaluation Report TANGO International October 2011 local levels, and field agents and program beneficiaries at the village level. The fieldwork activities applied a range of tools including key informant interviews, focus group discussions, large group discussions, community meetings, observation of program activities and outcomes, and several participatory research techniques. The evaluation team received a pre-field training and participated in designing the focus group discussion outline and randomly selecting the field sites. Villages selected for qualitative assessment were selected by the evaluation team based on variations between agro-ecological zones, livelihood systems, and levels of vulnerability. The selection reflected the most common types of activities in each SO with a mix of both activities completed early in the program life as well as activities, which were started later. The team also looked for opportunities to be able to see innovative activities, even if not widely replicated in the program, to be able to consider them for promising practices. The objectives in the scope of work were distilled into a set of critical questions and interview topic outlines. The focus group outline covered the following general topics: (i) MYAP objectives and program activities, (ii) level of community participation, (iii) community-level impacts, (iv) partnership, capacity building and sustainability, and (v) suggestions for improvement. Positive Impacts and Results CRS and its partners in Niger have made considerable progress in meeting the majority of its objectives. Throughout the evaluation, it has become evident that PROSAN has had numerous positive impacts on the lives and livelihoods of participants. Focus groups reported positive changes in living conditions since the beginning of the program, for example, household access to nutritional diets was increased, building upon household level agricultural production supported by SO1. Increased group participation was another positive impact. In terms of the specific activities and subcommittees, the activity with the greatest participation was Cash for Work (CFW), in which almost one-third of all surveyed households participated. Participation in this activity was higher in Zinder than in Tahoua and Dosso. One-quarter of all households participated in health/nutrition nutrition subcommittees. Approximately 20 percent of all interviewed households were engaged in cereal banks and habannaye groups, and over 15 percent were engaged in literacy groups. The groups with the lowest levels of participation were in agriculture (13 percent) and agro-processing (7 percent). Overall one of the key lessons learned in the program is that people have increased their trust and willingness to engage in group activities and productive community-driven initiatives. There is a growing sense of ownership and pride about group activities such as habbanaye, vegetable gardening, and Early Warning Groups. Participants feel that they are gaining the tools to take charge of their own development activities. The program has provided opportunities to increase good governance and management, as well as transparency at the village level by forming village level farmer farmers groups with elected positions. This is especially important at a time when Niger is undergoing political and administrative change and decentralization, when local communities will x x PROSAN- Rayuwa Final Evaluation Report TANGO International October 2011 have increasing opportunities to get involved in governance. Another positive aspect is that communities are learning how to mobilize resources to manage their own projects and extend the program activities in other domains. Technically, beneficiaries have been exposed to many new techniques in agriculture, nutrition, hygiene and management that will have lasting effects in improving the quality of life beyond the life of the program. The status of women is progressively changing as a result of the high level of women’s participation in various groups. Women’s visibility and leadership has increased through their engagement with the program and their participation in all project activities. This is a success story that CRS and its partner should build on for future programming in Niger. The first strategic objective of PROSAN is to achieve sustainable improvements in the availability of food for poor households in selected districts. It focuses on creating changes that will lead to increased production of food crops in villages throughout the program area. The focus is on smallholder households—providing technical assistance and training to promote sustainable farming practices, more productive and diversified farming systems, and increased agro-enterprise for producers of target area households. These activities aim to increase sustainable yields, thus contributing to improvements in the availability of food for poor rural households, both presently and in the future. Intermediate results under this SO are intended to restore basic production and increase productivity with expanded marketing capacities for participants. Food crop production has increased as a result of program initiatives. Technical assistance and training was given to smallholder households to promote sustainable farming practices, more productive and more diversified farming systems, and improved agro-enterprise activities. Agricultural potential was enhanced through the rehabilitation of small-scale irrigation and drainage systems, as well as soil and water conservation infrastructure utilizing food-for-work (FFW) programs. Irrigation is the most transformative agricultural investment option within the three regions, providing farmers with extremely productive and profitable new agricultural options, and PROSAN has been a major force in providing farmers with access to irrigation within the program area. Small- scale irrigation is the key to success in the future, decreasing long vulnerability. A high proportion of households that received SO1 training from the program grew greater quantities of each of the three staple food crops than those who did not receive training. Importantly, the production per household of each of the three crops is significantly higher for households that received training than those that did not, indicating that the adoption of recommended practices has had a positive impact on household production of major food crops. In addition, the production per household of training participants in “old” villages is higher (measured by the median) than households in the “new” villages. This result supports the argument that over time farmers adopt the agricultural practices that provide them with the greatest benefits (measured by household production of food crops). Adoption rates of all types of agro-processing and group marketing are significantly higher for households that have received SO1 training. PROSAN Strategic Objective 2 is xi xi PROSAN- Rayuwa Final Evaluation Report TANGO International October 2011 to protect and enhance the health and nutritional status of vulnerable household members, represented particularly by women of reproductive age and children under five, by increasing access to quality primary health care and improving household health and nutrition practices and sanitary and nutritional coverage in the three targeted regions of Tahoua, Dosso and Zinder. The results of the quantitative assessment of the nutritional status of children 6 to 59 months showed an overall rate of chronic malnutrition and chronic lack of growth at 37.4% for the entire area covered by the program. Several intermediate results of PROSAN show an increase in positive behavior for women, however, favorable to the good health and nutrition of vulnerable groups of the population. We saw, for example, increased visits to antenatal clinics, and the percentage of women who received iron supplementation now exceeds 80%. The rate of exclusive breastfeeding of infants under six months is high, at nearly 80% in all three areas of intervention program. Feeding practices for infants and young children are better in areas that received ISAN / PROSAN than compared with the areas that received only PROSAN, or had no support from the two programs. These results are very important in terms of long-term impacts and sustainability, as they reflect a positive cumulative impact of training over time. Immunization coverage of children against major killer diseases (tuberculosis, polio, whooping cough, measles and yellow fever) is very good—over 80% in all regions and regardless of the type of vaccine. However, the percentage of children from 1 to 2 years that were fully vaccinated is quite low, not even reaching 50%. The program did not succeed as hoped in the management of diarrhea, which remains an area of critical vulnerability. The analysis of the evolution of the nutritional status of children under five years in terms of wasting in the intervention area PROSAN shows overall improvement of the latter compared with the results of the baseline survey. This could reflect a positive impact of PROSAN upon the nutritional status of children. While it is difficult to attribute this positive effect exclusively to PROSAN, it can be said that the program has contributed to improving this situation in its area of intervention. To improve access of vulnerable populations in their area of operation, PROSAN officials have implemented a number of strategies including the renovation or construction of health infrastructure and facilitation of training for appropriate management and training of community workers on the actions needed to prevent malnutrition and infection (essential nutrition actions). The results of impact indicators and health practices, and nutrition at the household level, have shown some improvement even if it is slow, due to chronic malnutrition which has deeper causes, some of which have not yet been adequately taken into account (hygiene and sanitation, improving living conditions of women, etc.). Infant and young feeding practices, for example, have improved. The resiliency of target communities was increased through improved abilities to identify and respond to recurrent shocks. The multiple shocks associated with recent droughts and xii xii PROSAN- Rayuwa Final Evaluation Report TANGO International October 2011 floods are eroding the asset base of households to the point where recovery could be compromised. The results of impact indicators for health and nutrition practices at the household level have shown some improvement even if it is slow. Chronic malnutrition has deeper causes, some of which have not yet been adequately addressed (access to hygiene and sanitation infrastructure, improving living conditions of women, etc.) On the other hand, infant and young child feeding practices have improved. 1 PROSAN- Rayuwa Final Evaluation Report TANGO International October 2011 I. INTRODUCTION A. Background Niger is an agricultural country where 85 percent of the population is rural and engaged in agriculture as a primary source of income. Ranked 167 out of 169 countries in the Human Development Index, Niger is among the world’s poorest countries, with an average annual income of $357 in 2010; a high infant, child, and maternal mortality rates; and a life expectancy of less than 46 years. Only 46 percent of Nigeriens have access to clean water, and just 12 percent have adequate sanitation. In addition to the negative impact on agricultural production caused by the world economic crisis, persistent droughts have affected production throughout the country, especially in districts covered by the program. Overall the past decade has been characterized by considerable climatic change with regard to rainfall, with greater seasonal rainfall variation than in the past. These shifts in rainfall patterns have disrupted the agricultural production calendar, further leading to an increase in crop failures. The food security status of households in Niger is determined by the interplay of natural and human resources, the viability of livelihood systems and survival strategies, the politics of resource allocation and use, and the impact of development intervention. Households, the ultimate unit of analysis for measuring impact, make critical decisions about the allocation of productive resources. Thus their demographic characteristics, resource endowments, revenue-generating activities, and strategies of production, trade, and consumption constitute a set of critical livelihood and food security indicators. In addition, the food security status of households in Niger is determined by the interplay of natural and human resources, the viability of livelihood systems and survival strategies, the politics of resource allocation and use, and the impact of development intervention. A most significant factor is the program area’s vulnerability to natural calamities. These calamities wreak havoc on household assets, including livestock, crops, and pasture. Chief among the recent calamities are drought, the bane of rain-fed agriculture, and floods. The unpredictability of natural disasters in the PROSAN zone, coupled with a profound lack of assets and revenue-generating activities, makes it difficult for households to mobilize sufficient resources to recover from losses while meeting current household needs. Farming remains the predominant occupation throughout the area of intervention, and is the most widely practiced livelihood system. Rural households are chronically vulnerable to livelihood system failure and food shortages. Households must endure prolonged drought and other natural disasters. Crop harvests from small, dispersed landholdings of varying quality are highly seasonal and, although petty trade almost universally accompanies farming, the rhythm of market activity fluctuates with seasonal production. Access to nonagricultural employment for rural households is limited by the geographical distribution of jobs and their small number. The shortcomings of public and private education place even limited jobs beyond the reach of most rural dwellers. Only 14 2 PROSAN- Rayuwa Final Evaluation Report TANGO International October 2011 percent of Niger’s people are functionally literate; constituting a significant factor in the poverty that is so prevalent. Indeed, educational opportunities are limited for many boys and girls. Of the rural population attending school, only 10 percent complete primary grades, while secondary school and college attendance falls well below. Likewise, craftwork, a potentially important source of full-time and dead-season employment that requires supplementary training rather than basic education, is handicapped by undercapitalized production techniques and weak market links. Poor roads, or the absence of roads and other physical infrastructure, exacerbate each of the above-mentioned sources of vulnerability. Residents of most rural localities walk or move their goods on the backs of animals, rather than utilizing motorized vehicles. Even pedestrian and animal traffic is hampered during the rainy season, when heavy downpours, cresting rivers, and flooding make roads, paths, and trails impassable. Consequently, transportation costs for people and goods are high, and in many cases, spoilage consumes agricultural products before products can reach the market. B. Objectives of MYAP-PROSAN In response to the extremely critical situation described above, the “Programme de Sécurité Alimentaire et Nutritionnelle” (PROSAN) was begun in 2007 and completed in 2012. The program consists of three major activities (below) implemented by Catholic Relief Services (CRS/Niger) in partnership with Cooperative for Assistance and Relief Everywhere, Inc. (CARE) and Helen Keller International (HKI) in three geographical areas of Niger: Dosso, Tahoua, and Zinder. The components of the program build upon past achievements realized in the first Development Assistance Program (DAP-ISAN), which was implemented in 2002 to improve the livelihoods of poor households. DAP’s implementation strategy was based upon participatory community development. Building upon the successes of DAP, PROSAN consists of three strategic objectives to address key behavioral and systemic constraints related to food availability, access and utilization. The objectives and expected results are as follows: 1) To protect and enhance livelihoods of vulnerable households through increased agro-pastoral production and improved agro-enterprises by 2012. This objective focuses on creating changes that will lead to increased production of food crops. The focus will be placed on smallholder households—providing technical assistance and training to promote sustainable farming practices, more productive and more diversified farming systems, and improved agro-enterprise activities. Agricultural potential will be enhanced through the rehabilitation of small-scale irrigation and drainage systems, as well as soil and water conservation infrastructure utilizing food-for-work (FFW) programs. 2) To enhance human capabilities through improved health and nutritional practices in vulnerable in targeted communities. This objective focuses on improving household access to a nutritious diet. In doing so, it builds on household￾level agricultural production supported by SO1. 3 PROSAN- Rayuwa Final Evaluation Report TANGO International October 2011 3) To protect and enhance the resiliency of target communities through improved abilities to identify and respond to recurrent shocks by 2012. PROSAN is implemented in three districts of the three regions of Niger (Figure 1): Figure 1: Map of Intervention Zone C. Critical Issues in the Operating Environment of MYAP Two features of the operating environment affected the implementation of PROSAN: 1. The first aspect of the operating environment that affected program implementation was the late start-up in 2008. The Government of Niger’s efforts to end Food For Work activities in 2007 created major problems in the planning of PROSAN activities, as these were heavily based on the Food for Work model, in which these development activities were funded by monetization. (Food surplus is sold in the market of developing countries and then used to fund development activities.) At numerous meetings it became clear that the government of Niger has informed the NGOs that it is no longer committed to Food for Work as a tool for development. The government’s new position led to major delays in the implementation of the program, which was partially based on Food for Work as a development strategy. Thus CRS and its partners had to change their implementing strategies and restructure activities to adapt to the new context. These factors may have reduced the effectiveness of PROSAN efforts in various sectors during particular periods and in particular geographic areas. 4 PROSAN- Rayuwa Final Evaluation Report TANGO International October 2011 The program chose to adopt cash-for-work (CFW) activities to implement project initiatives, while program personnel experienced major delays and frustration in the implementation of PROSAN, including, for example, disruptions in the process of set-up and staffing of offices. It was only in October 2008 that the program started effectively, leaving only 3 years before the scheduled completion of the overall activities. For most of 2007-2008 there were numerous disruptions to program activities, as the partnership with the various stakeholders in the region was taking place in the context of a new program design and an extremely volatile political environment. 2. The second critical factor to consider in this review process is the impact of droughts and floods on the livelihood of program beneficiaries. These environmental shocks have severely hindered agricultural activities in the program areas and have likely affected health indicators, below. Many households in the program area lost most of their productive assets. This situation caused a major slowdown in the execution and outputs of planned activities. Approximately 50 percent of the households that THE MYAP had begun working with reported total destruction of crops. Access to the beneficiaries in the affected areas also limited most of the planned activities for the following months. Many beneficiaries lost the inputs provided by the program due to persistent drought, making it more difficult to successfully promote some of the program’s activities. D. Objectives of this evaluation This final evaluation combines qualitative and quantitative data aimed at gaining a comprehensive understanding of how PROSAN has been implemented and how its implementation has succeeded in bringing about the desired changes in the lives and livelihoods of household and institutional participants. The final evaluation aims to: (i) assess the extent to which CRS and its partners have accomplished the stated goals and objectives of the five-year program, as amended; (ii) assess the effectiveness of the technical approach by reviewing program activities that have been successful and those that have not, including reasons why (as much as is feasible); (iii) obtain answers to key questions that may contribute to better applying lessons learned, best practices, sustainability, and recommendations for future programming; and (iv) document and summarize the overarching lessons learned from the program to a wider audience including the consortium, partner organizations, donors, Government of Niger and other stakeholders. Three external consultants were commissioned over the period July 8 through August 4, 2011 in order to conduct a final evaluation for the program. PROSAN staff from CRS, CARE, and HKI provided support for the evaluation. The evaluation reviewed the program to assess progress and formulate recommendations. 5 PROSAN- Rayuwa Final Evaluation Report TANGO International October 2011 E. Evaluation Methodology The evaluation is designed to capture the maximum amount of information in a limited time frame. It is based on a qualitative and quantitative field study, in addition to monitoring of indicators collected by PROSAN field staff during the course of the program. The methodology deploys a set of complementary research activities, survey instruments, and techniques for gathering and analyzing information. The study began with a review of related program documents in order to frame the relevant issues. A non-formal presentation by each component coordinator and other staff provided crucial background information. The team reviewed several key documents compiled by program staff, including:  the MYAP proposal  the baseline study  annual results reports  MYAP program design document  trimester and annual reports  audit reports  other relevant reports in the targeted districts. The field evaluation centered on three main groups involved in program activities: PROSAN and its partners’ administrators, coordinators and supervisors at national and local levels, and field agents and program beneficiaries at the village level. The fieldwork activities applied a range of tools including key informant interviews, focus group discussions, large group discussions, community meetings, observation of program activities and outcomes, and several participatory research techniques. The evaluation team received a pre-field training and participated in designing the focus group discussion outline and randomly selecting the field sites. This process included rural household survey—for example, health and nutrition questionnaires, that provided data on key indicators tracked by the program. Villages selected for qualitative assessment were selected by the evaluation team based on variations between agro-ecological zones, livelihood systems, and levels of vulnerability. The selection reflected the most common types of activities in each SO with a mix of both activities completed early in the program life as well as activities, which were started later. The team also looked for opportunities to be able to see innovative activities, even if not widely replicated in the program, to be able to consider them for promising practices. The objectives in the scope of work were distilled into a set of critical questions and interview topic outlines. The focus group outline covered the following general topics: (i) MYAP objectives and program activities, (ii) level of community participation, (iii) community-level impacts, (iv) partnership, capacity building and sustainability, and (v) suggestions for improvement. 6 PROSAN- Rayuwa Final Evaluation Report TANGO International October 2011 The quantitative household survey is designed to obtain information on households within the PROSAN operational area with respect to their overall food security conditions, and for program participants, information about their perceptions of program interventions, and adoption of behaviors promoted by the program. In addition, the household survey is designed to obtain information about outcome and impact level indicators in the program IPTT, to compare with results from the baseline and mid-term surveys. As recommended by FFP, the household survey is a population-based survey in which the sample frame for the survey includes all households residing within the PROSAN intervention area. The quantitative household survey sample size was computed using the following formula: n = D [(Zα + Zβ) 2 * (P1 (1 - P1) + P2 (1 - P2)) /(P2 - P1) 2 ] KEY: n = required minimum sample size per survey round or comparison group D = design effect P1 = the estimated level of an indicator measured as a proportion at the time of the first survey or for the control area P2 = the expected level of the indicator either at some future date or for the program area such that the quantity (P2 - P1) is the size of the magnitude of change it is desired to be able to detect Zα = the Z-score corresponding to the degree of confidence with which it is desired to be able to conclude that an observed change of size (P2 - P1) would not have occurred by chance (α - the level of statistical significance), and Zβ = the z-score corresponding to the degree of confidence with which it is desired to be certain of detecting a change of size (P2 - P1) if one actually occurred (β-level of statistical power). The indicator used as the basis of sampling is stunting rates for children under five years of age. The level of stunting of U5 children observed in the program intervention area was 39.9% in the baseline survey, so P1 is set at 0.39. The sample size calculation has been set to be able to detect a reduction of 6 percentage points (15%) in the stunting rate. Applying the following parameter values, the equation will yield the minimum required sample size per stratum: D = 2.0 P1 = .399 P2 = .339 Zα = 1.645 (α=0.95) Zβ = 0.84 (β=0.80) 7 PROSAN- Rayuwa Final Evaluation Report TANGO International October 2011 Using these values, the estimated value of n, or the minimum required sample size, is 1,600. Because not all households have U5 children, the minimum sample size of households needs to be adjusted to ensure that a sufficient number of children (1,600) are measured. However, in the baseline survey, the total of children under 5 surveyed from the sampled households was greater than the number of households sampled (2,987 measured from 1,841 households), so a sample of 1,600 households within the program area is expected to provide at least 1,600 U5 children. A two-stage cluster design was employed for the baseline survey, with the first stage involving selection of clusters (villages) and the second stage being the selection of households. The sampling frame consisted of all villages within the PROSAN operational area and since this is a population-based survey, all households residing in these villages were eligible for interviewing. Villages were selected using probability proportional to size (PPS) from the list of all villages in the program’s operational area. The number of villages (clusters) was chosen to maximize the number of total villages to be consistent with the logistically feasible minimum feasible number of households to be interviewed within each village. Overall, a total of 54 villages were selected, with a target of 30 households to be interviewed per village, to give a target total sample of 1,620 HH to be surveyed. The selection of villages was stratified along two dimensions. First, the sample was stratified by region, with 18 villages selected in each of the three regions of Tahoua, Dosso, and Zinder. The second dimension of stratification was designed to capture and compare information about i) “old” villages that are currently in PROSAN and also participated in the previous DAP (the ISAN program), and ii) “new” villages which are in PROSAN but were not covered under ISAN. The purpose of this stratification was to be able to assess the cumulative impacts over time of support provided by Title II programs. Within each region, 9 “old” (PROSAN + ISAN) villages and 9 “new” (PROSAN) villages were selected. The sample was weighted to account for these stratifications in the selection of villages. Note that in Zinder only 16 separate villages are reported, because two large villages were selected twice in the PPS process. In these two villages the target number of households to be interviewed was 60 rather than 30. Tables of results from the quantitative household survey are broken down by region, and by categories of household participation in PROSAN, and in the previous Title II program, ISAN. In each of these tables, statistical tests were conducted to detect a difference in the means (or proportions) of the indicators across the different subgroups reported in the tables. If the statistical tests indicate a difference across the subgroups with at least a 90 percent level of confidence (significance level of the test less than 0.10) a star is placed next to the appropriate value in the tables. The use of a 90 percent level of confidence as a cutoff value is a widely used standard in cross-section surveys, which typically have very high levels of variability in all measured variables, making it very difficult to measure differences across subgroups with a higher level of confidence. 8 PROSAN- Rayuwa Final Evaluation Report TANGO International October 2011 Table 1 provides information about the actual number of households interviewed by region, and the computed numbers, based on the sampling weights: Table 1: Number of households interviewed, by region Region Actual # HH interviewed Weighted # HH interviewed Tahoua 550 903 Dosso 489 324 Zinder 572 400 Total 1,611 1,627 Within selected villages, households were then selected to be interviewed using the random walk process described in the FANTA sampling guide. Interviews were conducted with both the household head and the mothers of children under five years old present in the interviews, to ensure that accurate information was obtained in the interview. If the household members with the necessary information were not available at the time of first contact with the household, the enumerators would arrange to return to the household at a later time when they could meet with the appropriate household members. Key Definitions The household was the primary unit of analysis used in data collection and analysis. The definition of household utilized is “a group of people who are related, who live together, produce together and eat together most of the time.” In the context of Niger, villages may be scattered or broken into smaller clusters as households migrate, and village boundaries are sometimes difficult to define. For the purposes of this program, a village is defined as “a collection of households in a shared physical space in which households share basic infrastructure and engage in similar livelihood activities.” Field staff met with leaders of communities in order to better define the boundaries of specific nomadic communities. Community leaders and members enthusiastically welcomed the PROSAN evaluation team in all field sites visited. The communities made every effort to facilitate their work by hosting field agents and agreeing to participate in community meetings and focus groups at a time when most people are busy preparing for the planting season. The strong rapport and level of trust established between PROSAN and its partners’ agents and community members was evident during the study. The analysis and interpretation of the results was largely under the control of external evaluators, although much assistance was provided by PROSAN staff during the process. 9 PROSAN- Rayuwa Final Evaluation Report TANGO International October 2011 II. PROSAN Activities and Achievements The PROSAN program operates in communities through Village Development Committees (Comite Villagois de Developement, CVD), which have been established by the program. These committees are composed of several subcommittees that are responsible for oversight and management of specific program activities within the villages. As shown in Table 2, village residents are very aware of the CVDs. Overall, over 80 percent of all surveyed households were aware of CVDs within their communities. In addition, participation in activities managed by CVDs was very high, with over half of all interviewed households indicating that they participated in activities supported by the CVDs. Participation in CVD activities was significantly higher in Zinder, at 60 percent of all interviewed households, compared with 50 percent in the other two regions. In terms of the specific activities and subcommittees, the activity with the greatest participation was Cash for Work (CFW), in which almost one-third of all surveyed households participated. Once again the participation in this activity was higher in Zinder than in Tahoua and Dosso. One-quarter of all households participated in health/nutrition nutrition subcommittees. Approximately 20 percent of all interviewed households were engaged in cereal banks and habannaye groups, and over 15 percent were engaged in literacy groups. The groups with the lowest levels of participation were in agriculture (13 percent) and agroprocessing (7 percent). Table 2: Household awareness of and participation in CVDs, by region Tahoua Dosso Zinder Total % HH aware of CVD 82.7 72.3a 85.9b 81.4 Participation in CVD committees Participation in CVD 49.3 53.1 59.7a,b 52.6 Participation in subcommittees: CFW 31.5 28.9 35.1b 31.9 Health/nutrition 23.5 28.7a 22.1b 24.2 Cereal banks 26.2 11.4a 14.1a 20.3 Habannaye 29.1 4.2a 11.4a,b 19.8 Literacy 11.4 26.6a 19.8a,b 16.5 Agriculture/livestock 10.9 21.1a 12.8b 13.4 Agroprocessing 6.8 10.7a 5.8b 7.3 n 903 324 400 1,627 a different from Tahoua at 0.10 significance level b different from Dosso at 0.10 significance level 10 PROSAN- Rayuwa Final Evaluation Report TANGO International October 2011 A. SO1 Activities and Achievements The first strategic objective of PROSAN is to achieve sustainable improvements in the availability of food for poor households in selected districts. It focuses on creating changes that will lead to increased production of food crops in villages throughout the program area. The focus is on smallholder households—providing technical assistance and training to promote sustainable farming practices, more productive and diversified farming systems, and increased agro-enterprise for producers of target area households. These activities aim to increase sustainable yields, thus contributing to improvements in the availability of food for poor rural households, both presently and in the future. Intermediate results under this SO are intended to restore basic production and increase productivity with expanded marketing capacities for participants. However, activities under this SO1 are largely dependent on external factors such as the climatic conditions that were unfavorable during most of the program’s duration. In addition, the decision by the GoN in 2007 to ban Food for Work for non-emergency programming delayed the start of the activities and reduced the level of resources that were planned for this activity. Although focus group discussion revealed some level of satisfaction about activities accomplished under SO1, many challenges were also mentioned. Beneficiaries noted: • Many field agents started working late in the program (end of 2008 and beginning 2009) and their workloads were too high. • Few demonstrations were in place before the 2009 cropping season, due to a lack of budget for equipment and supplies, delays in purchasing, and so forth. • Farmers in the new villages only began implementing and evaluating sustainable agricultural practices on a large scale in their fields in the 2009 cropping season. Table 3 provides information about adoption of agricultural practices promoted by PROSAN. Several practices are very widely adopted, with almost 90 percent of all surveyed households reporting the use of organic fertilizer and natural regeneration techniques. Three-quarters of all households reported implementing soil improvement techniques, particularly zai/tassa. Approximately 50 percent of all households reported use of mineral fertilizers, improved seeds, and chemical pesticides. Natural pesticides and fallowing were adopted relatively less, by one-quarter or fewer of households. Some important patterns of variation across the three regions are evident: use of organic fertilizer and compost was significantly lower in Zinder than in the other regions, while use of fallow was higher. Irrigation is also significantly less in Zinder than the other regions. Note that the adoption of irrigation is constrained by physical limitations – farmers can only adopt irrigation if readily exploitable water sources are available near to farmers’ fields. For further detail on LOA targets, see revised IPTT Table in Appendix A. 11 PROSAN- Rayuwa Final Evaluation Report TANGO International October 2011 Table 3: Adoption of agricultural practices, by region Tahoua Dosso Zinder Total Soil improvement techniques 74.3 75.3 76.0 74.9 zai/tassa 63.2 57.3a 27.4a,b 53.2 demilunes 38.2 30.7 a 38.8b 36.8 Plant trees 10.7 13.5 9.5b 11.0 Mulching (paillage) 20.4 21.5 39.5 a,b 25.4 Organic fertilizer 93.5 97.6 a 64.6 a,b 87.2 Fallow 18.0 18.6 28.1 a,b 20.6 Natural regeneration techniques 82.7 94.4 a 93.0 a 87.5 Compost 35.7 40.0 16.2 a,b 31.7 Mineral fertilizer 48.8 68.2 a 22.7 a,b 46.3 Improved seed 48.9 33.5 a 58.6 a,b 48.2 Irrigation 30.5 32.5 13.8 a,b 26.8 Chemical pesticides 49.2 59.2 a 47.1b 50.7 Natural pesticides 28.3 22.2 a 22.0 a 25.5 Improved cropping practices (appropriate planting densities, crop rotation, appropriate weeding, etc.) 40.8 20.5 a 66.0 a,b 42.9 n 903 324 400 1,627 a different from Tahoua at 0.10 significance level b different from Dosso at 0.10 significance level Table 4 provides information about adoption of agricultural practices broken down into three categories of participation in training in SO1 interventions. The first category is households that did not receive any training from the program in SO1 activities (agricultural production, NRM, agroprocessing, and agricultural marketing). The second category is households that received SO1 training, and who reside in “new” program villages, participating only in PROSAN. The third category is households that received SO1 training under PROSAN, but live in villages that were also previously supported under the ISAN DAP. The results in Table 4, and also shown graphically in Figure 2, show a very distinct pattern for the adoption of agricultural practices. Households that received SO1 training have significantly higher adoption rates of all types of recommended practices than households that did not receive training, suggesting that the training has had a significantly positive impact on household adoption of these However the other very distinct pattern is that with only one exception—natural regeneration techniques, adoption rates were lower (or at the most, no higher) in the “old” villages than in the “new” villages. In other words, the rate of adoption of individual techniques was generally lower in villages that received support for a longer time. This somewhat surprising result may be explained by the fact that farmers in the “new” villages are more likely to try out all the new techniques being promoted by the program, 12 PROSAN- Rayuwa Final Evaluation Report TANGO International October 2011 whereas farmers in the “old” villages are more likely to have selectively adopted the techniques that provide them with the greatest benefits. This explanation is supported by the results on agricultural production across the three SO1 participation categories. presented in subsequent pages. Table 4 and Figure 2 also demonstrate that some practices recommended by the program (organic fertilizer, natural regeneration techniques, improved cropping practices, improved seed, and mineral fertilizer) are quite widely adopted by farmers that did not participate in program trainings. These are activities which farmers clearly see advantages to adopting. However, even for these popular practices the adoption rate is higher for households that participated in training. Table 4: Adoption of agricultural practices, by SO1 participation category No training Training (PROSAN) Training (PROSAN+ISAN) Total Soil improvement techniques 54.3 86.1a 88.0a 74.9 zai/tassa 35.3 67.8a 59.9a,b 53.2 demilunes 16.8 50.2 a 47.3a 36.8 Plant trees 2.6 16.9a 15.0a 11.0 Mulching (paillage) 16.2 29.8a 31.7a 25.4 Organic fertilizer 78.9 96.6a 87.8a,b 87.2 Fallow 20.5 17.7 23.5 20.6 Natural regeneration techniques 78.4 90.7 a 95.1a,b 87.5 Compost 17.0 47.5 a 33.6a,b 31.7 Mineral fertilizer 32.6 60.3 a 48.7a,b 46.3 Improved seed 38.6 57.8 a 50.1a,b 48.2 Irrigation 17.1 35.2 a 29.9a,b 26.8 Chemical pesticides 33.0 63.4 a 59.0a 50.7 Natural pesticides 10.9 37.4 a 31.1a,b 25.5 Improved cropping practices ( appropriate planting densities, crop rotation, appropriate weeding, etc.) 35.8 51.9 a 42.7a,b 42.9 n 604 500 524 1,627 a different from households with no training at 0.10 significance level b different from households with training in PROSAN areas at 0.10 significance level 13 PROSAN- Rayuwa Final Evaluation Report TANGO International October 2011 Figure 2: Adoption of agricultural practices by SO1 participation group Adoption of irrigation is presented separately in Figure 3 because adoption of irrigation is constrained by farmers’ access to available water resources. As can be seen in the figure, a higher proportion of households that received SO1 training have adopted irrigation than those who have not received training. The sources of the investment in the farmers’ irrigation systems are presented in Figure 4. The most important sources are private investment by the farmer, and PROSAN, which is by far the largest source of irrigation investment in Dosso and Zinder. In Tahoua, the highest proportion of farmers installed their irrigation systems privately, but even in this region around one-third of all farmers with irrigation benefitted from PROSAN irrigation investments. Irrigation is the most transformative agricultural investment option within the three regions, providing farmers with extremely productive and profitable new agricultural options, and PROSAN has been a major force in providing farmers with access to irrigation within the program area. Figure 3: Adoption of irrigation, by SO1 participation group (% of HH) 14 PROSAN- Rayuwa Final Evaluation Report TANGO International October 2011 Figure 4: Source of developed irrigation, by region *Percentage calculated only for households who have developed irrigation. Household members were asked about their level of satisfaction for each type of recommended agricultural practice they adopted. Possible responses included: “very satisfied”, “somewhat satisfied”, and “not satisfied”. Table 5 reports the percentage of households that reported being “very satisfied” with each specific type of agricultural practice. Overall the reported level of satisfaction was very high for all practices. Interestingly, the level of satisfaction was generally no higher for households that received training from PROSAN than for those that did not receive training directly from PROSAN. That is, the training provided by PROSAN has a great impact on adoption rates (see Table 4), but the training has not greatly changed farmers’ perceptions of the benefits of these practices, among farmers who have adopted the practices. These results suggest that the real benefit of the SO1 training has been to encourage new practices, and farmers who have adopted these practices without direct training from PROSAN, either through learning from PROSAN farmers, or from training provided by other organizations, have the same perceptions of the benefits of these new practices. One result that is somewhat surprising in Table 5 is the relatively low proportion of households reporting being extremely satisfied with irrigation, compared with many of the other practices. Given the critical importance of water for agriculture in the program areas, and the highly productive options that access to irrigation affords to farmers, it might be expected that satisfaction with access to irrigation should be very high. The relatively low rates of response of “extremely satisfied” may reflect the fact that irrigation systems do not completely eliminate farmers’ exposure to production risks: the irrigation systems can still go dry at critical points in the off-season production cycle, the systems experience breakdowns, and conflicts among water users may disrupt supplies to individuals. These limitations may explain the relatively low proportion of farmers who reported being extremely satisfied with irrigation. 15 PROSAN- Rayuwa Final Evaluation Report TANGO International October 2011 Table 6 and Table 7 report on household production of the three major food crops grown by farmers in the program area: millet, sorghum, and cowpea. As can be seen in Table 6, almost all households grew millet and cowpea, and almost two-thirds of households grew sorghum. Sorghum is less widely grown in Dosso compared with the other two regions, whereas cowpea is somewhat more prevalent there than in Tahoua or Zinder. Mean and median production per household are reported in these tables. Mean values are reported to compare with other published information on crop production figures. However, the production quantities are typically very skewed, with a small number of farmers who grow very large quantities compared with their neighbors. For these skewed distributions, the sample median is a better measure of the “average” production of most farmers, because this measure is not greatly affected by a few extreme values. Production of millet Table 5: Percentage of HH reporting “extremely satisfied” with specific adopted agricultural practices, by SO1 participation category No training Training (PROSAN) Training (PROSAN+ISAN) Total Soil improvement techniques 90.2 89.1 86.8 88.5 n 327 430 461 1,219 Organic fertilizer 89.5 92.1 88.8 90.2 n 476 483 460 1,419 Fallow 66.7 83.7a 86.2b 78.3 n 123 89 123 335 Natural regeneration techniques 87.7 88.0 86.8 87.5 n 473 453 498 1,425 Compost 88.6 83.2 81.3 83.6 n 103 238 176 516 Mineral fertilizer 89.1 85.2 84.2 85.9 n 197 301 255 753 Improved seed 85.6 79.4a 79.5b 81.3 n 233 289 262 784 Irrigation 68.5 73.6 63.7c 68.8 n 103 176 157 436 Chemical pesticides 81.8 81.4 83.2 82.2 n 199 317 309 825 Natural pesticides 77.9 70.7 57.6b,c 66.7 n 66 187 163 416 Improved cropping practices 88.9 80.0a 80.1b 82.8 n 216 259 224 699 a different from Tahoua at 0.10 significance level b different from Dosso at 0.10 significance level 16 PROSAN- Rayuwa Final Evaluation Report TANGO International October 2011 and cowpea per household is higher in Dosso than in the other two regions, whereas production of sorghum is much lower in Dosso. Table 7: Production of millet, sorghum, and cowpea, by SO1 participation category No training Training (PROSAN) Training (PROSAN+ISAN) Total Millet % HH 93.0 98.0a 99.1a 96.5 Mean production 749.0 1,031.9a 964.2a 908.4 Median production 550.0 687.5 811.3a,b 687.5 Sorghum % HH 57.7 74.1a 59.4b 63.3 Mean production 189.1 225.0 219.8a 211.3 Median production 100.0 137.5a 137.5a 125.0 Cowpea % HH 84.8 94.3a 95.9a 91.3 Mean production 132.3 190.8a 235.0a,b 185.6 Median production 90.0 135.0a 184.0 a,b 92.0 n 604 500 524 1,627 a different from households with no training at 0.10 significance level b different from households with training in PROSAN areas at 0.10 significance level Data in Figure 5 show that a higher proportion of households that received SO1 training from the program grew greater quantities of each of the three staple food crops than those who did not receive training. Importantly, the production per household of each of the three crops is significantly higher for households that received training than those that did not, indicating that the adoption of recommended practices has had a positive impact on Table 6: Production of millet, sorghum, and cowpea, by region Tahoua Dosso Zinder Total Millet % HH 95.4 98.2a 97.7a 96.5 Mean production 914.2 1,142.9a 704.5a,b 908.4 Median production 687.5 962.5a 500.0b,c 687.5 Sorghum % HH 67.5 49.1a 65.3b 63.3 Mean production 211.5 69.9a 297.0a,b 211.3 Median production 137.5 41.3a 250.0a,b 125.0 Cowpea % HH 90.2 96.8a 89.4b 91.3 Mean production 153.5 256.5a 196.6a,b 185.6 Median production 90.0 184.0a 115.0 a,b 92.0 n 903 324 400 1,627 a different from Tahoua at 0.10 significance level b different from Dosso at 0.10 significance level 17 PROSAN- Rayuwa Final Evaluation Report TANGO International October 2011 household production of major food crops. In addition, the production per household of training participants in “old” villages is higher (measured by the median) than households in the “new” villages. This result supports the argument outlined above, that over time farmers adopt the agricultural practices that provide them with the greatest benefits (measured by household production of food crops). The adoption rate for particular practices is somewhat lower in “old” villages than in “new” ones, but agricultural production is higher among trained farmers in “old” villages. Figure 5: Household production of millet, sorghum, and cowpea, by SO1 participation category (median kg/HH) Agricultural Processing and Marketing Under SO1, PROSAN is also supporting the development of agroprocessing and group marketing of crops. PROSAN has launched its agricultural processing techniques only during the last two years. These include the following: Under ISAN, the Cooperating Sponsor promoted an improved method of processing cowpea into a locally known and consumed form of couscous called, béroua. A new label was designed to try to gain market share, but the product spoiled quickly and had a short shelf life. CRS started promoting in 2009 different solar drying technologies for vegetables, okra in particular. According to the focus group discussion, the experience is not conclusive yet. In most cases agricultural processing is done on an individual basis. There were few cases of community-level processing and marketing systems. In some cases, however, women come together informally to help each other, for example, to extract peanut oil or dry okra. An exception is vegetable producers, which, as group units supported by CRS and by CARE in the Dosso region, seem to operating cohesively, with more than 100 members in each group. 18 PROSAN- Rayuwa Final Evaluation Report TANGO International October 2011 Table 8 and Table 9 provide results for the number of households adopting these practices, by region and by participation in SO1 training respectively. Overall, about 40 percent of all surveyed households reported being engaged in some type of agro￾processing, and slightly over 20 percent were engaged in group marketing. As can be seen in Table 8, adoption of agro-processing activities is lower in Zinder than in Tahoua or Dosso, while the making of oil is significantly more widespread in Dosso than in the other two regions. Group marketing is relatively less prevalent in Dosso. Adoption rates of all types of agro￾processing and group marketing are significantly higher for households that have received SO1 training (Table 9; Figure 6; Figure 7). However, as in the case of agricultural production practices, the rates of adoption are lower in the “old” villages than in the “new” villages. These results suggest that over time a significant number of households have given up on these practices. These results should not be taken as a negative assessment of the training. It is important to understand that not all households are expected to adopt agro-processing activities. Some households will specialize in these Table 8: HH engaged in agroprocessing and group marketing, by region Tahoua Dosso Zinder Total % HH engaged in agro￾processing 41.3 50.0a 32.8a,b 40.9 Milling (cereals) 27.2 35.1a 18.8 a,b 26.7 Drying (cereals) 27.0 18.2a 15.8a 22.5 Oil pressing 14.3 28.7a 14.5b 17.2 Smoking 14.3 5.4a 9.8 a,b 11.4 Paste (peanut) 3.6 15.0a 2.3b 5.5 Grinding (cereals) 4.0 14.9a 1.2a,b 5.5 Brewing (cereals) 5.9 0.9a 1.9a 3.9 Frying 2.7 0.9a 0.3a 1.8 % HH engaged in group marketing 25.2 13.0a 21.0 21.7 n 903 324 400 1,627 a different from Tahoua at 0.10 significance level b different from Dosso at 0.10 significance level Figure 6: Percentage of HH engaged in group marketing, by SO1 participation category 19 PROSAN- Rayuwa Final Evaluation Report TANGO International October 2011 activities, based on their particular interests, capacities, and access to credit or savings. The program strategy is to provide training in these activities to many households, with the expectation that households will try them on their own, but also with the understanding that many households will ultimately decide that they do not want to continue in these activities. Table 9: HH engaged in agroprocessing and group marketing by SO1 participation category No training Training (PROSAN) Training (PROSAN+ISAN) Total % HH engaged in agro-processing 18.2 60.8a 48.1a,b 40.9 Milling (cereals) 8.6 44.6a 30.5 a,b 26.7 Drying (cereals) 9.8 37.5a 22.8 a,b 22.5 Oil pressing 4.1 25.9a 24.1a 17.2 Smoking 1.2 23.2a 11.9 a,b 11.4 Paste (peanut) 1.4 7.3a 8.6a 5.5 Grinding (cereals) 0.1 7.0a 10.3 a,b 5.5 Brewing (cereals) 1.2 8.9a 2.3b 3.9 Frying 2.0 2.4 0.9b 1.8 % HH engaged in group marketing 13.8 32.3a 20.8 a,b 21.7 n 604 500 524 1,627 a different from households with no training at 0.10 significance level b different from households with training in PROSAN areas at 0.10 significance level Figure 7: Percentage of HH engaged in agro-processing activities, by SO1 participation category 20 PROSAN- Rayuwa Final Evaluation Report TANGO International October 2011 Livestock ownership and practice Information about livestock ownership is provided in Table 10 and Table 11. Overall, over 80 percent of all surveyed households reported having some livestock, with the highest percent (90 percent) in Dosso. On average, households had about 2.5 sheep, two goats, one cow, and three poultry. Households who received SO1 training were more likely to have animals, and more animals, than those who did not receive training. Table 12 provides information about adoption of livestock practices by categories of participation in SO1 training. Not surprisingly, adoption of recommended practices is higher for participants than non-participants. As in the case of agriculture, the rate of adoption is lower in “old” communities than in “new” communities. Also interesting to Table 10: Household ownership of livestock, by region Tahoua Dosso Zinder Total % HH with livestock 81.0 89.5a 82.3 83.0 Average number of animals per HH: sheep 2.1 2.1 1.9 2.0 goats 2.4 3.4a 2.6b 2.6 cattle 1.2 1.5a 1.0 1.2 camels 1.1 0.0 0.0 0.6 donkeys 0.7 0.2a 0.3a,b 0.5 poultry 3.3 5.4a 2.1a,b 3.5 horses 0.0 0.0 0.1a,b 0.1 other 0.1 0.1 0.0b 0.1 n 903 324 400 1,627 a different from Tahoua at 0.10 significance level b different from Dosso at 0.10 significance level Table 11: Household ownership of livestock, by SO1 participation category No training Training (PROSAN) Training (PROSAN+ISAN) Total % HH with livestock 71.6 91.4a 88.2a,b 83.0 Average number of animals per HH: sheep 1.3 2.6a 2.3a 2.0 goats 1.8 3.2a 3.0a 2.6 cattle 0.9 1.5a 1.2b 1.2 camels 0.0 0.0 1.7 0.6 donkeys 0.4 0.7a 0.4b 0.5 poultry 2.1 4.3a 4.3a 3.5 horses 0.0 0.1 0.1 0.1 other 0.0 0.0 0.2 0.1 n 604 500 524 1,627 a different from households with no training at 0.10 significance level b different from households with training in PROSAN areas at 0.10 significance level 21 PROSAN- Rayuwa Final Evaluation Report TANGO International October 2011 note in this table is the high rate of adoption of recommended practices by households that have not received SO1 training, suggesting that these are activities that are perceived to be beneficial by many households, whether or not they have received training directly by the program. Table 12: Adoption of improved livestock practices, by SO1 participation category No training Training (PROSAN) Training (PROSAN+ISAN) Total Improved feeds 58.0 80.5a 70.7a,b 70.0 Deworming 47.3 76.6a 68.0a,b 64.3 Vaccinations 50.2 79.3a 72.6a,b 67.7 Fodder storage 46.8 71.7a 60.3a,b 59.8 n 432 457 462 1,351 a different from households with no training at 0.10 significance level b different from households with training in PROSAN areas at 0.10 significance level B. SO2 Activities and Achievements PROSAN Strategic Objective 2 is to protect and enhance the health and nutritional status of vulnerable household members, represented particularly by women of reproductive age and children under five, by increasing access to quality primary health care and improving household health and nutrition practices and sanitary and nutritional coverage in the three targeted regions of Tahoua, Dosso and Zinder. The main approaches used by PROSAN to reach this objective include (i) training health workers on Integrated Management of Child Illnesses (IMCI) focusing on the treatment of common childhood illnesses, (ii) training of community workers on Essential Nutrition Actions (ENA) and in Community IMCI, (iii) facilitating the construction or renovation and equipment of health infrastructures, (iv) facilitating community outreach antenatal visits and immunization sessions, (v) developing community-based distribution of micronutrients (iron/folic acid, vitamin A), (vi) implementation of Information, Education and Communication (IEC) sessions by community health workers to sensitize community members and especially women on best preventive health and nutrition behaviors, (vii) implementation of Positive Deviance/Hearth, and (viii) screening of malnutrition cases and referral to health facilities for adequate management. Three impact indicators were selected to measure the impact of PROSAN activities on health and nutritional status of women and children in the targeted program areas. • Impact indicator 2.1. a. Percentage of children aged 6 to 59 months with stunted growth (Height for age <-2 SD) by vulnerability status and gender. • Impact indicator 2.1.b. Percentage of underweight children aged 0 to 59 months (weight for age < - 2SD) by vulnerability status and gender. • Impact indicator 2.2. Percentage of women with a body mass index below 18.5 kg/m2. Table 13: Anthropometric measures of children, by region 22 PROSAN- Rayuwa Final Evaluation Report TANGO International October 2011 Tahoua Dosso Zinder Total % children N % children N % children N % children N Stunting (HAZ < 2sd) Female 27.7 303 34.1a 293 52.8a,b 159 35.5 755 Male 31.2 295 32.5 271 64.7a,b 170 39.4 736 Both 29.4 598 33.3 564 59.0a,b 329 37.4 1,491 Underweight (WAZ < 2sd) Female 21.0 281 29.7a 195 35.9a 142 27.2 618 Male 29.3 287 34.2 199 48.1a,b 154 35.3 640 Both 25.2 568 32.0a 394 42.2a,b 296 31.3 1,258 Wasting (WHZ < 2sd) Female 12.5 273 9.3 183 4.5a,b 132 9.7 588 Male 22.6 283 15.2a 178 13.0a 146 18.1 607 Both 17.6 556 12.2a 361 9.0a 278 14.0 1,195 a different from Tahoua at 0.10 significance level b different from Dosso at 0.10 significance level The results of the quantitative assessment of the nutritional status of children 6 to 59 months show an overall rate of chronic malnutrition (stunting) at 37.4% for the entire area covered by the program (Table 13). Tahoua has the lowest percentage of stunted children (29%) compared to other regions. Unlike the other two regions, the stunting rate has increased in Zinder. The disturbingly high percentage of stunted children in Zinder (59%) is comparable to that found in the last survey in June 2011 (SMART Niger 2011), suggesting that it is indeed accurate. The evaluation team is unable to comment on why stunting has steadily increased in this region. Further research is required to better understand the trend in Zinder. 23 PROSAN- Rayuwa Final Evaluation Report TANGO International October 2011 Within the entire program area, 31.3% of children 0-59 months are underweight. The comparison with baseline results shows a statistically significant improvement in the overall intervention area (from 41% to 31.3%), and the final rate is even lower than the revised target of 33%. This improvement has been in all regions, but especially in Dosso. Figure 10: Comparison of wasting in children aged 0 to 59 months (Weight for height <-2 SD), baseline and endline surveys Figure 8: Percentage of children aged 6 to 59 months with stunted growth (Height for age <-2SD), by survey round and region Figure 9: Percentage of underweight children aged 0 to 59 months (Weight for age <-2 SD), by survey round and region 24 PROSAN- Rayuwa Final Evaluation Report TANGO International October 2011 shows that across the sample, acute malnutrition (wasting) for children under 5 years old at the time of the final survey was 22% lower than baseline levels (4 percentage point reduction). In Zinder, wasting levels were more than halved (from baseline 21% to 9% at the final survey); while in Tahoua, the percentage of children suffering from acute malnutrition increased by five percentage points. Figure 10: Comparison of wasting in children aged 0 to 59 months (Weight for height <-2 SD), baseline and endline surveys The Zinder region’s chronic malnutrition, and that of Tahoua for acute malnutrition, are exceptions to positive evolution of the nutritional status of children in program operational areas. In the Zinder region, this could be explained by the very poor, difficult conditions of access to safe drinking water exacerbated by insecurity. The increase in wasting observed in Tahoua could be related to the time of year the data was collected; final survey data was collected in Tahoua during the “hungry season” (“soudure”)— baseline data was not. Figure 11: Percentage of women with BMI < 18.5 kg/m2 by region and survey round 25 PROSAN- Rayuwa Final Evaluation Report TANGO International October 2011 The third impact indicator selected by PROSAN consists of the percentage of women of childbearing age with a chronic energy deficiency characterized by a body mass index (BMI) below 18.5 kg/m2. The evolution of this indicator shows deterioration in the nutritional status of women of childbearing age in the entire area covered by the program, and in each individual region (Figure 11). Group discussions with women of childbearing age in the qualitative survey highlighted some factors that may be involved and require deepening. These include an overload of domestic work that leaves women little time for proper care of their health and nutrition of their children, problems of poverty that affect women more than men (for example, lack of land, income-generating activities), closely-spaced pregnancies, heightened maternal health risk due to remote health facilities without adequate capacity to provide pre- and post-natal care; poor hygiene and sanitation (lack of drinking water, latrines), and a high vulnerability to infectious diseases (malaria, diarrhea, etc.). 26 PROSAN- Rayuwa Final Evaluation Report TANGO International October 2011 B. SO2 Intermediate results. IR 2.1. Targeted households have improved access to primary health care. To improve access of vulnerable populations in their area of operation, PROSAN officials have implemented a number of strategies including the renovation or construction of health infrastructure and facilitation of training for appropriate management and training of community workers on the actions needed to prevent malnutrition and infection (essential nutrition actions). The results of impact indicators for health practices and nutrition at the household level have shown some slow improvement to chronic malnutrition, although some of the deeper causes of chronic malnutrition have not yet been adequately taken into account (hygiene and sanitation, improving living conditions of women, etc.). Several intermediate results of PROSAN indicate positive health and nutrition behavior change for women. The percentage of mothers with children under two years old (U2) who visited antenatal clinics at any time during their last pregnancy has increased throughout the intervention area (from 60 to 91%) and in all three regions (Figure 12). The percentage of mothers who attended four or more ANC visits is significantly higher for women who participated in PROSAN and PROSAN/ISAN compared with mothers who received no training from either program. A statistical difference between program participants and non￾participants is also present for mothers of U2 children who have attended an ANC visit at least once. Both indicators suggest the programs have positively influenced mothers’ ANC attendance. Overall, 91% In Dosso region, 99% of mothers of U2 children reported at least one ANC visit. The mean number of prenatal visits is statistically higher in areas covered by PROSAN (4 visits) compared with women who have also benefited from ISAN (3.7 visits) and those who did not participate in any training provided by the two programs (Figure 14). Figure 12: Comparison of women receiving any antenatal visits, baseline and final surveys 27 PROSAN- Rayuwa Final Evaluation Report TANGO International October 2011 Figure 13: Comparison of mothers with any ANC to mothers with 4 or more ANC visits, by SO2 participation category Figure 14: Mean number of antenatal visits for women of children under 24 months, by SO2 participation category 28 PROSAN- Rayuwa Final Evaluation Report TANGO International October 2011 The percentage of mothers of U2 children who received iron supplementation during their last pregnancy has dramatically increased compared with baseline data, and now exceeds the program’s 80% target in all three regions of operation (Figure 15). This level is highest in Zinder (95 %) followed respectively by Tahoua (91%) and Dosso (86%). The percentage point increase for all regions since the baseline survey is large; data from Tahoua shows the greatest increase. Figure 15: Comparison of iron supplementation for mothers of U2 children, baseline and final surveys The percentage of mothers of U2 children who received vitamin A capsules within eight weeks after delivery also increased considerably, especially in Zinder where the rate is 80%, corresponding to what is recommended for an impact on the public health of micronutrient interventions (Figure 16). Figure 16: Comparison of women who received vitamin A post partum, baseline and final surveys 29 PROSAN- Rayuwa Final Evaluation Report TANGO International October 2011 More than three-fourths of mothers of children under two years old who were familiar with Vitamin A capsules, reported receiving the capsules (Figure 17). Data show that a statistically significant higher percentage of women (80%) who participated in both PROSAN and ISAN programs reported receiving Vitamin A capsules compared with mothers who did not receive training from either program. However, at 70% the level of consumption of vitamin A is not negligible for women who did not receive training from either program. The high levels of health-promoting behaviors among women participating in both PROSAN and ISAN suggests the cumulative effects of reinforcing health and nutrition messages over time to increase the positive impact on behavior changes. IR 2.2 Targeted households have improved health and nutrition behaviors: Infant and young child feeding practices Figure 17: Percentage of mothers who received vitamin A post partum, by SO2 participation category Data only calculated for U2 mothers who are familiar with the Vitamin A capsule (71 % of all mothers of children U2. 30 PROSAN- Rayuwa Final Evaluation Report TANGO International October 2011 The rate of exclusive breastfeeding of infants under six months is high, at nearly 80% in the overall sample (Table 14). It is particularly and significantly higher in Dosso (96%) compared with the other two regions (80% Zinder and 72% Tahoua). This rate is much higher than the national level and than that found in most countries of the region. Similarly, the percentage of infants who were breastfed within the first hours after birth is close to 80% across the entire program area; the percentage is significantly higher in Zinder (78%) compared with the other two regions. 31 PROSAN- Rayuwa Final Evaluation Report TANGO International October 2011 While no statistical comparisons were carried out, Figure 18 shows that the percentage of mothers who reported that they practiced exclusive breastfeeding for children six months and younger almost doubled between the baseline and final surveys (41% versus 79%). The percentage of mothers who reported breastfeeding their infant within the first hour of birth increased similarly (Figure 19). These behaviors are often faced with very strong social taboos and the resulting changes suggest effective communication on the part of PROSAN and other partners. Figure 18: Exclusive breastfeeding (0-5 months), by survey round and region Table 14: IYCF practices, by region Tahoua Dosso Zinder Total % of children aged 0-5 months exclusively breastfed 72.4 95.5a 79.3b 78.6 n 127 44 58 229 % of children 0-24 months breastfed within first hour 61.1 61.3 77.9a,b 65.1 n 460 173 199 832 % of children 6-24 months consuming minimum number of meals 70.8 82.4a 70.7b 73.4 n 298 125 123 546 a different from Tahoua at 0.10 significance level b different from Dosso at 0.10 significance level 32 PROSAN- Rayuwa Final Evaluation Report TANGO International October 2011 Figure 19: Infants breastfed within the first hour after birth, by survey round and region Data in Table 15 show that feeding practices for infants and young children are better for households who received support from PROSAN and ISAN compared with households who only participated in PROSAN activities, or who had no support from the two programs. The results of this table are very important in terms of long-term impacts and sustainability. The results suggest positive cumulative impact of training over time. Adoption rates are higher in “old” communities. These results suggest that communities continue to adopt best practices over time, and that continued training in communities has had a cumulative positive impact. It is also very important to note the high level of reported adoption of all practices, even among households who received no training. These results likely reflect that the training provided by PROSAN, as well as other government and non-government programs, has increased households’ awareness of appropriate child feeding practices. It should also be emphasized that these are self￾reporting results, and the prevalence of actual practices of appropriate child feeding behaviors are likely to be substantially lower than these figures. 33 PROSAN- Rayuwa Final Evaluation Report TANGO International October 2011 Table 15: IYCF practices, by SO2 participation category No training SO2 Training (PROSAN) SO2 Training (PROSAN+ISAN) Total % of children aged 0-5 months exclusively breastfed 80.2 67.2 86.8b 78.6 n 86 67 76 229 % of children 0-24 months breastfed within first hour 61.5 63.0 72.8a,b 65.1 n 377 219 235 832 % of children 6-24 months consuming minimum number of meals 68.6 75.2a 79.7a 73.3 n 258 141 148 547 a different from HHs without any training at 0.10 significance level b different from new MYAP areas at 0.10 significance level Immunization coverage Immunization coverage of children against major diseases that contribute to infant and child mortality (tuberculosis, polio, whooping cough, measles and yellow fever) is high— over 80% in all regions for all types of vaccine ( 34 PROSAN- Rayuwa Final Evaluation Report TANGO International October 2011 Table 16: Percentage of children 12-24 months who have received vaccinations, by region ). The percentage of children 12-24 months old who have received DTCoq and BCG vaccinations is significantly higher for household who participated in program activities versus households who did not (Table 17). Vaccination rates for polio, rougeole, and yellow fever do not appear to improve relative to length of program participation or presence. The percentage of children 1 to 2 years old that were fully immunized is low (less than 50%). Results from the qualitative findings indicate that this is usually due to the fact that women tend to forget to return after some time to finalize the immunization cycle. Communication should be focused on sensitizing mothers to complete the program. 35 PROSAN- Rayuwa Final Evaluation Report TANGO International October 2011 Table 16: Percentage of children 12-24 months who have received vaccinations, by region Tahoua Dosso Zinder Total BCG 95.9 96.2 83.5a,b 93.0 n 241 78 97 416 Polio 98.8 100.0 100.0 99.3 n 241 79 102 422 DTCoq 88.6 98.6a 88.5b 90.6 n 201 73 87 361 Rougeole 87.7 90.1 87.5 88.1 n 211 71 88 370 Yellow fever 82.7 85.5 78.8 82.3 n 202 69 85 356 % of children aged 12-24 months full vaccinated (filters out children who said DNK to any of the vaccinations) 40.8 48.5 30.9 40.2 n 157 66 68 291 a different from Tahoua at 0.10 significance level b different from Dosso at 0.10 significance level Table 17: Vaccinations, by SO2 participation category No training SO2 Training (PROSAN) SO2 Training (PROSAN+ISAN) Total BCG 90.5 97.2a 94.4 93.0 n 199 109 108 416 Polio 98.5 100.0 100.0 99.3 n 204 110 107 422 DTCoq 84.4 93.6a 98.0a 90.6 n 167 94 101 361 Rougeole 86.0 91.8 88.9 88.1 n 171 98 99 370 Yellow fever 81.2 84.2 82.3 82.3 n 165 95 96 356 % of children aged 12-24 months fully- vaccinatedc 32.8 45.1 47.6a 40.2 n 125 82 84 291 a different from HHs without any training at 0.10 significance level b different from new MYAP areas at 0.10 significance level c filters out children who said DNK to any of the vaccinations Diarrhea incidence and treatment The incidence of diarrhea is still very strong throughout the program area where 45% of children experienced an episode during the two weeks preceding the survey. The situation has not improved since this figure has increased from 43% at the time of the baseline survey. The incidence of diarrhea is linked to poor hygiene and the unfavorable sanitation environment for the vast majority of people living in program areas for those 36 PROSAN- Rayuwa Final Evaluation Report TANGO International October 2011 who are chronically vulnerable—women and children under five years of age. The incidence of diarrhea is significantly lower for children in households that participated in PROSAN and ISAN than for children in households that did not participate in either program (Table 18). This could be due to the long-term promotion of diarrhea prevention in those sites. Table 18: Diarrhea incidence and treatment, by region Tahoua Dosso Zinder Total % of children experiencing diarrhea in previous two weeks 54.4 36.4a 32.7a 45.4 n 456 173 199 828 % of children experiencing diarrhea that continued to breastfeed 98.1 95.1 100.0 97.9 n 216 61 60 337 Amount of breastmilk fed to affected child More than normal 29.4 31.0 27.1 29.3 Same as normal 37.9 39.7 44.1 39.3 Less than normal 27.0 27.6 28.8 27.4 DNK 5.7 1.7 0.0 4.0 n 211 58 59 328 % of children experiencing diarrhea that continued to drink water 97.1 93.9 98.1 96.7 n 205 49 52 306 Amount of water fed to affected child More than normal 51.8 46.7 46.2 50.0 Same as normal 29.1 33.3 40.4 31.8 Less than normal 17.6 20.0 11.5 16.9 DNK 1.5 0.0 1.9 1.4 n 199 45 52 296 % of children experiencing diarrhea that continued receiving food 93.7 89.8 80.8 90.8 n 205 49 52 306 Frequency of feeding More than normal 19.9 15.6 8.0 17.2 Same as normal 42.3 33.3 24.0 37.8 Less than normal 34.7 46.7 56.0 40.2 DNK 3.1 4.4 12.0 4.8 n 196 45 50 291 a different from Tahoua at 0.10 significance level b different from Dosso at 0.10 significance level Among mothers with children experiencing an episode of diarrhea, 69% do not decrease the amount of breastmilk offered to the child, 82% increase or offer the same amount of water to the ill child, and 55% offer the same or an increased amount of food to the ill child. With respect to treatment for diarrhea, across the program area the vast majority of mothers (83%) have utilized the assistance of a health worker (Table 19). Just over half of women (54%) received oral rehydration salts (ORS). Very few (12.5%) used the sweet and saline (SES). 37 PROSAN- Rayuwa Final Evaluation Report TANGO International October 2011 Table 19: Diarrhea counseling received, by region Tahoua Dosso Zinder Total % of children experiencing diarrhea for whom treatment was sought 73.8 76.2 81.5 75.5 n 248 63 65 376 Source of treatment Health worker 82.3 90.6 79.5 83.2 First Aid Trainers 1.0 1.9 2.0 1.3 Matrons 3.8 0.8 2.0 3.0 Village facilitator 0.0 0.0 2.6 0.5 Pharmacy 1.3 0.0 2.0 1.2 Marabouts/Bokas 0.0 0.8 0.5 0.2 Neighbors/parents 11.6 11.0 14.9 12.1 Other 4.8 0.0 4.1 3.9 n 185 48 54 287 Treatment received None 16.3 11.6 10.0 14.5 ESS 13.1 11.3 11.6 12.5 ORS packet 54.9 58.9 44.8 53.8 Ruwan dawo 1.0 0.0 0.5 0.7 Millet based solution 0.0 0.6 0.0 0.1 Concoction/infusion 2.3 4.8 2.2 2.7 Medicine 38.8 52.0 37.3 40.7 Zinc tablets 4.7 8.1 26.8 9.1 Other 4.0 0.0 7.3 3.9 DNK 2.9 2.1 0.5 2.3 248 62 65 341 Two-fifths of children (41%) received medical treatment. It was not clear if they received antibiotics or if the diarrhea is a simple diarrhea. However, health workers tend to use antibiotics excessively to treat diarrhea. During visits to health centers (CSI) this practice was observed, emphasizing the need for district staff to address this issue during IMCI post-training and supervision. The incidence of diarrhea is significantly lower for children whose households participated in PROSAN/ISAN activities than for children of households newer to the program (PROSAN only) and for children of households who did not participate in either program ( 38 PROSAN- Rayuwa Final Evaluation Report TANGO International October 2011 Table 20: Diarrhea incidence and treatment, by SO2 participation category ). 39 PROSAN- Rayuwa Final Evaluation Report TANGO International October 2011 Table 20: Diarrhea incidence and treatment, by SO2 participation category No training SO2 Training (PROSAN) SO2 Training (PROSAN+ISAN) Total % of children experiencing diarrhea in previous two weeks 50.1 49.1 34.5a,b 45.4 n 375 216 235 828 % of children experiencing diarrhea who continued to breastfeed 99.4 96.7 96.2 97.9 n 169 90 79 337 Amount of breastmilk fed to affected child More than normal 27.2 28.4 35.1 29.3 Same as normal 45.6 29.5 37.8 39.3 Less than normal 23.1 36.4 25.7 27.4 DNK 4.1 5.7 1.4 4.0 n 169 88 74 328 % of children experiencing diarrhea that continued receiving water 96.0 100.0 95.2 96.7 n 150 92 63 306 Amount of water fed to affected child More than normal 44.4 57.0 53.3 50.0 Same as normal 40.3 17.2 31.7 31.8 Less than normal 15.3 22.6 11.7 16.9 DNK 0.0 3.2 3.3 1.4 n 144 93 60 296 % of children experiencing diarrhea that continued receiving food 87.3 95.7a 92.1 90.8 n 150 92 63 306 Frequency of feeding More than normal 20.6 14.8 13.1 17.2 Same as normal 41.8 31.8 36.1 37.8 Less than normal 30.5 53.4 44.3 40.2 DNK 7.1 0.0 6.6 4.8 n 141 88 61 291 a different from HHs without any training at 0.10 significance level b different from new MYAP areas at 0.10 significance level 40 PROSAN- Rayuwa Final Evaluation Report TANGO International October 2011 Table 21: Diarrhea counseling received, by SO2 participation category No training SO2 Training (PROSAN) SO2 Training (PROSAN+ISAN) Total % of children experiencing diarrhea for whom treatment was sought 76.6 72.6 77.8 75.5 n 188 106 81 376 Source of treatment Health and sanitation training 83.6 89.1 75.1 83.2 First Aid Trainers 0.6 0.0 4.6 1.3 Matrons 3.2 3.6 1.7 3.0 Village facilitators 0.0 0.4 1.7 0.5 Pharmacies 1.6 0.0 1.7 1.2 Marabouts/Bokas 0.0 0.9 0.0 0.2 Neighbors/parents 11.8 6.1 19.9 12.1 Other 3.7 6.2 1.4 3.9 n 148 77 63 287 Treatment received None 11.8 17.3 17.0 14.5 ESS 11.9 13.5 12.6 12.5 ORS packet 56.1 52.0 50.9 53.8 Ruwan dawo 0.2 2.3 0.0 0.7 Millet based solution 0.0 0.4 0.0 0.1 Concoction/infusion 1.6 3.0 4.7 2.7 Medicine 41.8 35.7 44.7 40.7 Zinc tablets 8.7 9.8 9.0 9.1 Other 3.7 2.3 6.3 3.9 DNK 0.4 6.8 1.1 2.3 188 106 81 341 Health Infrastructure and Capacity Strengthening PROSAN has used Integrated Management of Childhood Illness (IMCI) an approach to improving child health that focuses on the well-being of the whole child. IMCI aims to reduce death, illness and disability, and to promote improved growth and development among children under five years of age. IMCI includes both preventive and curative elements that are implemented by families and communities as well as by health facilities. PROSAN management have implemented a number of IMCI strategies including the renovation or construction of health infrastructure; facilitation of training for appropriate management; and training of community workers on the actions needed to prevent malnutrition and infection (essential nutrition actions). In most health facilities visited during this evaluation, health workers, and CSI in particular, said they had been trained in IMCI, primarily through the support of other 41 PROSAN- Rayuwa Final Evaluation Report TANGO International October 2011 partners such as UNICEF, and NGOs such as Médecins World, etc. The problems they face include out of stock drugs such as anti-malarial drugs and packets for the management of diarrhea, or inadequate supervision by the health district. The officers of health huts, including a significant number of nurses, also stressed similar problems: limited equipment, limited means of transport for themselves and for the evacuation of patients, shortages of medicines, difficult living conditions (drinking water, housing, motorcycle, etc.), and failure of supervision. In some villages community-based distribution of micronutrients was not performed due to social discord between the inhabitants of nearby villages, or sometimes due to out of stock of micronutrients. IMCI requires a number of conditions to achieve the desired results. All of PROSAN’s strategies are relevant and contribute to improving child health. Health services should be better organized and better supervised at the health district level. In the same vein, it appears that the remuneration of CSI for outreach activities may be prejudicial to the project’s sustainability. The evaluators suggest better communication for roles and responsibilities by both the Ministry of Health at the national and local levels and by the project, PROSAN. For the betterment of the program, the Ministry of Health should be providing these outreach services, with an emphasis on the prolonged sustainability of the program beyond the life of the project. 42 PROSAN- Rayuwa Final Evaluation Report TANGO International October 2011 C. SO3 Activities and Achievements Community resilience is achieved when community members and households are educated regarding natural hazards and shocks; are empowered to mitigate risk; and have social networks to fall back upon. The goal is to protect the community’s resilience through improved capacity to identify and respond to recurrent and future shocks. By doing so, communities and households will be more resilient to shocks that threaten their livelihoods. PROSAN sought to decrease community and household vulnerabilities to natural shocks through increased community preparedness and improved organizational response capacity. The exposure to natural disasters such as droughts, floods, and disease outbreaks is one of the key factors contributing to household susceptibility to food and livelihood insecurity. In order to accomplish this objective, PROSAN has identified three intermediate results: IR 3.1: Target communities have operational emergency early warning systems. IR 3.2: Community assets are protected during shocks. IR 3.3: Local early warning and crisis management structures have demonstrably enhanced their administrative, managerial, leadership and accountability skills. The approach used under SO3 is premised on the principle that by focusing implementation activities at the village level to increase institutional capacity in disaster preparedness, community engagement in disaster preparedness and awareness would increase. Through preparedness planning and awareness activities, communities and households would more likely take measures that would save lives. Capacity of the communities to anticipate and manage crises through SCAP-RU Community-based SCAP-RUs have been established in all the program areas. In the Konni areas, the effectiveness of these village-level institutions is due in large part to the field support provided by agents of ISCV, the local NGO contracted by CARE to implement PROSAN's community level activities. In Tanout/Mirriah and Dogongoutchi/Loga, due to delays in the recruitment and training of new field workers for new villages, SCAP-Rus started later. Selection of members Members are usually selected during a general assembly of the village from a list of volunteers. Approximately 70% of SCAP-RU members are also members of other committees. SCAP-RU members are accepted by communities and enjoy some popular and technical legitimacy. However, the most vulnerable members of the community are rarely represented. Operationalization of the SCAP-RU 43 PROSAN- Rayuwa Final Evaluation Report TANGO International October 2011 • Most members know their roles and responsibilities. • There exists little feedback from the communes and the technical services. This discourages participation by the members of the SCAP-RU. • There is a lack of standardized forms for collecting data. PROSAN created oversight village committees to supervise and coordinate all activities undertaken in the village. In Konni/ Illéla they are called ‘CUSA’ and in the CRS departments of Tanout/Mirriah and Doutchi/Loga they are called Village Development Committees (VDC). CUSA and VDC are the driving forces behind the execution of PROSAN activities. The presidents of the VDC or CUSA tend to be members of the SCAP-RU. Village Chiefs and their roles in the various activities It is important to note that the position of village chief is the only legal institution in the village recognized by the State. Legal documents confer upon chiefs the authority and responsibility for managing civil conflicts and managing village infrastructure. According to the key informant interviews, many committee members tend to be relatives or friends of the village chiefs. SCAP-RU collects information on food security, health and nutrition, agriculture and livestock production, environment and social relations, and different levels of alert identified by the community. By providing micro-level data, they reinforce the national early warning system, which compensates for the lack of government extension agents who would normally be responsible for collecting crop production information at the village level. Observatories for Monitoring Vulnerability (OMV) These are made up of representatives of the technical services (agriculture, livestock and health), the village chiefs, and representatives of the SCAP-RU. They meet twice a year and direct the meetings. In principle, communes were supposed to summarize SCAP-RU reports to be integrated into the National Early Warning System. In reality most communes lack the human, technical, and financial resources to do this job effectively. Nonetheless, SCAP-RU reports have helped improve decision-making in some of the communes. Some municipal counselors have used the reports to plan development activities. Protection of community assets during shocks Village-based response mechanisms have been developed in 80% of the sites surveyed by the qualitative team. Some committee members and other villagers were cultivating a communal field, the harvest to be used for the community contribution to the FARN for 44 PROSAN- Rayuwa Final Evaluation Report TANGO International October 2011 feeding malnourished children in the village. In another case, the village cereal bank is being restocked with contributions from heads of households. Administrative, managerial, leadership and accountability skills of SCAP-RU There exists little evidence that these skills have improved. However, the alerts and information provided by the SCAP-RU help the community meet and discuss issues in order to gain consensus on issues at stake. SCAP-RU members conduct evaluations using agreed upon indicators, including heath, social interactions in villages, purchase prices of the principal agricultural product in the area, and the state of natural resources (pasture availability, environment management). Analysis of these data leads to discussions within committees as well as in communities. Focus group discussions and key-informant interviews reveal a real willingness to deal with the risks and effects of natural disasters. The lessons learned from past disasters and especially the increased awareness gained from program communication campaigns, from travel, and from increased access to mass media, represent real promise that communities are willing to take more responsibility for their own development. Mismanagement and theft are reported to be major problems in the surveyed communities. There is a need to develop more activities for greater transparency, accountability, and good governance. Technical agents from PROSAN are perceived in many villages as top-level managers to whom they should obey. Greater efforts can be made in this area to increase village self￾sufficiency and empowerment. D. Program Impacts Program impacts on household food security status are measure by the following indicators: • Household diet diversity score (HDDS) • Months of adequate household food provisioning (MAHFP) • Household production of major food crops (millet, sorghum, cowpea) • % U5 children stunted (HAZ < -2 sd) • % U5 children underweight (WAZ < -2sd) The HDDS is a proxy indicator of household food access. It is computed by adding up the total number of different food categories consumed by the household in the 24 hours prior to the interview, out of a total of 12 food categories. Table 22 shows the values of the HDDS from the three rounds of household survey, and by type of support provided by the program. The first panel of the table shows the HDDS values by survey round. After exhibiting a large increase from the baseline to the mid-term survey, the HDDS value in the final survey is lower than that of the baseline. For interpretation of these results it is important to note that the final household survey was conducted during the hungry when all households have lower HDDS values than at other times of the year. The difference between the final survey and the earlier rounds may be due to the different seasons in which the information was collected, or to any of a number of exogenous 45 PROSAN- Rayuwa Final Evaluation Report TANGO International October 2011 variables constituting the declining food situation, including, for example, natural disasters (droughts and floods), lack of infrastructure and lack of government support for the Food for Work program. The second panel of the table shows the HDDS values for households that were not supported by the program (received no training and did not participate in CFW), those that received training only, those that received CFW only, and those that received CFW and training. The households that received any type of program support had HDDS statistically higher than households that received no support, but the magnitude of differences are quite small in. The last panel of the table examines the impact of different types of training provided by PROSAN which are intended to influence household food security, namely trainings in SO1 and SO2 program areas. Only households that receive training in both SO1 and SO2 exhibited significantly higher HDDS than those that received no training. Table 22: Mean HDDS by survey round and type of support provided HDDS (mean) n Baseline 3.8 Mid-term 5.2 Final (total sample) 2.9 1,627 Type of support provided by PROSAN No support 2.6 265 Training only 3.0a 455 CFW only 2.9a 178 Training+CFW 3.0a 729 Type of training provided by PROSAN No training 2.8 482 SO1 only 2.8 297 SO2 only 3.0 121 SO1+SO2 3.1b 727 a different from no support at 0.10 significance level b different from no training at 0.10 significance level Table 23 provides information about the MAHFP indicator by survey round and by types of support provided to the households by PROSAN. This indicator has shown a strong increasing trend over the survey rounds, increasing by over 60 percent from the baseline (5.5) to the final value (9.0). However, households that received program support had somewhat lower values of MAHFP than those that did not receive support, although these differences are very small in magnitude. 46 PROSAN- Rayuwa Final Evaluation Report TANGO International October 2011 Table 23: Mean MAHFP by survey round, and type of support provided MAHFP (mean) n Baseline 5.5 Mid-term 8.4 Final (total sample) 9.0 1,627 Type of support provided by PROSAN No support 9.5 265 Training only 9.0a 455 CFW only 8.9a 178 Training+CFW 8.8a 729 Type of training provided by PROSAN No training 9.2 482 SO1 only 9.2 297 SO2 only 9.0 121 SO1+SO2 8.7b 727 a different from no support at 0.10 significance level b different from no training at 0.10 significance level Another indicator of household food security which is extremely important in the context of rural Niger is the total household production of major food crops—millet, sorghum, and cowpea. Household production of these crops per year are reported in Table 24 and shown graphically in Figure 20. After exhibiting significant increases in production from the baseline to the mid-term survey rounds, the level of household production of these three crops is actually lower in the final round than in the baseline. It is important to note that these figures are averages for all households in the survey, both program participants and non-participants. However, as shown above, production levels of these three crops are significantly higher for households that received SO1 support compared with those that did not receive support. Table 24: Household production of food crops (millet, sorghum, cowpea), kg/HH, by survey round and district Tahoua Dosso Zinder Total Baseline 1,393 1,704 682 1,400 Mid-term 1,472 2,446 1,129 1,670 FInal 1,181 1,463 1,058 1,207 47 PROSAN- Rayuwa Final Evaluation Report TANGO International October 2011 Figure 20: Household production of major food crops (millet, sorghum, cowpea), kg/HH, by survey round and district Table 25: Anthropometric indicators of U5 children, by region Tahoua Dosso Zinder Total % children N % children N % children N % children N Stunting (HAZ < -2sd) Female 27.7 303 34.1a 293 52.8a,b 159 35.5 755 Male 31.2 295 32.5 271 64.7a,b 170 39.4 736 Both 29.4 598 33.3 564 59.0a,b 329 37.4 1,491 Underweight (WAZ < -2sd) Female 21.0 281 29.7a 195 35.9a 142 27.2 618 Male 29.3 287 34.2 199 48.1a,b 154 35.3 640 Both 25.2 568 32.0a 394 42.2a,b 296 31.3 1,258 Wasting (WHZ < -2sd) Female 12.5 273 9.3 183 4.5a,b 132 9.7 588 Male 22.6 283 15.2a 178 13.0a 146 18.1 607 Both 17.6 556 12.2a 361 9.0a 278 14.0 1,195 a different from Tahoua at 0.10 significance level b different from Dosso at 0.10 significance level 48 PROSAN- Rayuwa Final Evaluation Report TANGO International October 2011 Table 26: Percentage of women 15 – 49 with BMI < 18.5, by region Tahoua Dosso Zinder Total % Women 15-49 w BMI < 18.5 20.8 12.8a 19.6b 17.7 n 452 360 199 1,011 a different from Tahoua at 0.10 significance level b different from Dosso at 0.10 significance level Finally, Table 27 shows trends in the key program outcome and impact indicators over the three rounds of household surveys. The first panel of the table shows outcome indicators related to household adoption of practices promoted by the program. Adoption of natural resource management (NRM) practices and sustainable agricultural production practices shows a large increase from the baseline to the final survey. At the time of the final survey, essentially all farmers had reported adopting at least one recommended activity in these two areas. Adoption of agro-processing activities supported by the program shows a substantial increase from the baseline to the mid-term, but then a slight decrease from the mid-term to the final survey round. Adoption of recommended agricultural marketing practices shows a large increase, almost doubling from the time of the baseline to the mid-term, but then falls back to the baseline level by the time of the final survey. Adoption of recommended child feeding practices is summarized by the percentage of children 0-6 months exclusively breastfed. This indicator jumped from 26 percent of all children 0-6 months in the mid-term, to almost 80 percent by the time of the final survey round. This high figure reflects a dramatic increase in mothers’ increased awareness of appropriate child feeding practices, but because it is based on self-reporting information, it may not correspond to changes in actual behaviors. The second panel of Table 27 shows changes in program impact indicators. Total production of the three major food crops is slightly lower in the final survey round than in the baseline, after an increase in the mid-term round. The HDDS is also lower in the final round compared with both the baseline and mid-term, but this difference may be explained by the fact that the final survey was conducted during the hungry season, when households generally have less diverse diets than at other times in the year. And, despite the project’s best efforts, droughts and lack of infrastructure and lack of government support for Food for Work severely affected production during various periods and in particular areas, as it has affected household food access, above. The number of months of adequate household food provisioning (MAHFP) has increased substantially over the life of the program, by 36% from the baseline to the mid-term, and an additional 7 percent from the mid-term to the final survey. Finally, the anthropometric measurements of children (stunting and underweight rates) improved slightly from the baseline to the final survey rounds. 49 PROSAN- Rayuwa Final Evaluation Report TANGO International October 2011 In summary, the survey results show that the program has supported significant changes in behaviors of individuals and households over the life of the program. In particular, the adoption of natural resource management and improved agricultural production practices recommended by the program has increased significantly. The fact that households that received SO1 training had significantly higher production than households who did not receive training suggests that the adoption of these practices has had positive impact on farming households. This is true even though the reported household production of major food crops for all surveyed households was lower in the final survey than in the baseline. The nutritional status of children, which reflects the combined impacts of all program interventions directed toward household food security as well as many other non-program factors, has improved by from 6 percent in terms of long-term impact (stunting rate) to almost 30 percent in shorter term effects (underweight rate). For more detail on LOA target values for Table 27, see Appendix A. Table 27: Program outcome and impact indicators, by survey round Baseline Mid-term Final Outcome indicators % HH Adopting of at least on improved practice: NRM 75.2 80.1 97.0 Sustainable agriculture 74.5 76.0 98.0 Agro-processing 9.5 48.0 40.9 Marketing 23.1 42.1 21.7 Exclusive breastfeedinga n.a 26.0 78.6 Impact Indicators Major food crop production (kg/HH) 1,400 1,670 1,207 HDDS (score) 3.8 5.2 2.9 MAHFP (months) 5.5 8.4 9.0 Stunting (% HAZ < -2 sd) 39.9 37.4 Underweight (% WAZ < -2 sd) 44.7 31.3 aChildren 0-6 months who received only breastmilk (or ORS) in 24 hours prior to survey. Figure 21: Adoption of recommended practices, by survey round 50 PROSAN- Rayuwa Final Evaluation Report TANGO International October 2011 Figure 22: Household production of staple food crops (millet, sorghum cowpea), by survey round Figure 23: Mean HDDS and MAHFP, by survey round 51 PROSAN- Rayuwa Final Evaluation Report TANGO International October 2011 52 PROSAN- Rayuwa Final Evaluation Report TANGO International October 2011 Figure 24: Comparison of stunted and underweight children, baseline and final survey 53 PROSAN- Rayuwa Final Evaluation Report TANGO International October 2011 III. PROSAN PROCESS ASSESSMENT A. Overview of management systems The ban on Food for Work in 2008 that delayed implementation of PROSAN created challenges for the overall management of PROSAN at its earlier stage: revision and re￾approval of program activities, staffing turnovers, and travel restrictions in some areas. Despite these constraints, the evaluation team acknowledges the programming results that have been demonstrated during the implementation of the program. There are, however, specific issues that came up over and over during the evaluation process and that need to be addressed: • Under present conditions the staff is overworked, especially at the lowest level of services providers (field agents). Difficulties in identifying qualified individuals to fill vacant positions and rapid turn-over has resulted in overtaxed staff with insufficient time to remedy perceived problems and implement new approaches. • Some aspects of program management seem to constrain operations. A case in point is the monthly financial reporting and accountability requirement. This measure helps to keep track of financial flows within THE MYAP but could also be limiting attainment of program targets. • An aspect of program management as reported by service providers are the numerous changes being made to reporting requirements. This leaves staff in suspense about what could be coming next. A lot of staff’s management time is expended trying to make adjustments to the changes. • CRS and its partners will need to pay attention to competitive staff remuneration if it is to avoid a high turnover. Excessive turnover will potentially undermine the ability to consolidate achievements and aim for excellence. This will not be possible if human resources and effort must be devoted to getting new staff up to speed. B. Monitoring and Evaluation Monitoring is considered to be a collection of specific indicators in order to inform management and key stakeholders about the achievement of objectives. Monitoring should be used to adjust certain aspects of the program if this is considered necessary. Evaluation is an analysis of an activity in order to inform program managers and stakeholders about the achievement of objectives and other results and the relevance, effectiveness, and efficiency of the activity. There is strong indication that the program is offering relevant services and that many of the stated objectives are met. 54 PROSAN- Rayuwa Final Evaluation Report TANGO International October 2011 CRS and its partners utilized primary and secondary data collection methods in monitoring key indicators. Primary data collection methods included information gathered by the on-going activities of the Monitoring Unit. Partners are required to provide CRS quarterly and yearly reports that include information such as production levels, the number of training participants, and performance of activities. However, the observed monitoring system focuses more on input and process indicators. The current monitoring and evaluation system is complex, and it has generated a large volume of data. However, the analysis of the data and the integration of this analysis into program management have not been maximized and should be expanded. A GIS component should be added to the system in order to revamp data collection, analysis, and reporting systems. Such changes will increase efficiently and effectiveness of program monitoring. There is an urgent need to improve the capacity of the monitoring system for close attention to the provision of management information, both to support the implementation of the program’s goals and to feed back into the design of new initiatives in changing environments. The Evaluation team found the M&E database hard to manage, making updating of information cumbersome. Objectives and indicators should have been discussed at great length in consultation with beneficiaries and stakeholders, so that objectives and targets are jointly "owned." Many partners felt that some of the indicators were “parachuted.” There are exogenous indicators that cover factors outside the control of the program but which severely affect the outcome of the program’s current leading indicators. Concerns to monitor both the program and its wider environment call for a data collection capacity outside the program, placing an additional burden on the program's M&E effort. A recent example of droughts in the program area demonstrates the importance of monitoring risk indicators. Given the greater transparency now expected of the development community, CRS and its partners need to strengthen the current monitoring system. The quality of record keeping in the field should be standardized for better use of the data, and merits careful attention. M&E supervisors should examine existing recordkeeping and the reporting procedures used to assess the capacity to generate the data that will be needed. At the same time, they should explain how and why the indicators will be useful to field, intermediate, and senior levels of program management. The inclusion of simple household level data linked with key break variables such as the village or parish should significantly improve the scope for analysis. One approach is to structure reporting from the field so that aggregates or summaries are made at intermediate stages. In this way, field staff can see how averages or totals for specific villages, or districts, enable comparisons to be drawn and fieldwork improved. The household data available at the village level should be integrated into the current monitoring system. The use of geographic information could improve the efficiency of the Monitoring and Evaluation Unit. A Geographic Information System (GIS) is a set of computer programs used to input, store, analyze and output geographically referenced data. In programs 55 PROSAN- Rayuwa Final Evaluation Report TANGO International October 2011 related to livelihood and food security, GIS has become an indispensable tool for adequately collecting analyzing multiple parameters involved in monitoring key indicators. A good GIS should: boost productivity, improve coordination, create base to make better decisions, and enhance service thereby making better use of MYAP resources. We recommend that CRS and its partners develop an integrated information system enhanced by the use of a GIS as a useful tool in program monitoring and evaluation. Figure 25: GIS data of households and villages of surveyed sites Some of the advantages in using GIS in program monitoring and evaluation include:  GIS allows a user to tell a visual story: “a picture is worth a thousand words”  GIS can help to eliminate ambiguities in the baseline—baseline data can be easily integrated in a MYAP district  GIS illustrates linkages in related systems which allows for a more comprehensive approach to monitoring and evaluation by overlaying related data sets that are available which can link direct and indirect program impact  GIS makes data standardized and more accessible to people with non￾technical backgrounds  ‘Before and after’ changes due to program activities are clearly visible, can be animated, and are a more convincing representation. C. Capacity development, partnership and sustainability Capacity development is widely acknowledged to be important, but is often poorly defined. It entails: 56 PROSAN- Rayuwa Final Evaluation Report TANGO International October 2011  upgrading skills in monitoring and evaluation, which includes: program analysis, design of indicators and reporting systems, socioeconomic data collection, and information management;  improving procedures to create functional systems that seek out and use information for decisions; and  strengthening organizations to develop skilled staff in appropriate positions, accountable for their actions. One of the central goals of PROSAN is to increase capacity for beneficiary communities to continue and expand program activities after the eventual departure of CRS and its partners. The main elements to ensure sustainability over time are the small farmholder groups and community actors who are involved in extending program activities on a volunteer basis. Most communities believe that their groups will continue to function at the end of the program. In addition, neighboring villages and small farmholder associations have begun collaborating with each other; this is an area that could be greatly expanded in the next MYAP. The evaluation team has compiled a set of priorities and recommendations to improve program sustainability. To be sustainable, the program must address the issue of droughts and explore ways to make water available for both agricultural and domestic use. The program should promote the development of water harvesting techniques and small-scale irrigation schemes and occasional micro-dam construction in order to increase agricultural productivity by reducing rainfall dependence in the drought prone food deficit regions. D. Resource Management Resource management issues were recently thoroughly reviewed in an audit of MYAPS in Niger by the Office of the Inspector General. In this report, the auditors found that from March through June of FY 2010 some commodities were not available to beneficiaries. For example, the auditors found that between December 2009 and March 2010 over 100 people who had participated in trainings had not received food their entitled food distributions. PROSAN consortium reported that lengthy delays in the call￾forward process created periodic shortages in commodity availability, and that the program was continually playing “catch-up” with food distributions. This has led to breakdowns in distributions to beneficiaries. The auditors’ response to this analysis of the problem was that the consortium must adequately incorporate these delays into their planning process. They noted that cooperating sponsors are not required to place their call forward requests in November, but rather, they are encouraged by Food for Peace to place their requests as early as possible. The auditors recommended that Food for Peace work with cooperating sponsors to develop and implement plans so that commodities are timely available, and to develop a plan to provide food to participants in the distribution backlog. The consortium is currently working with Food for Peace on these plans. 57 PROSAN- Rayuwa Final Evaluation Report TANGO International October 2011 IV. Conclusions and Recommendations In response to high levels of food insecurity in target areas in Niger, CRS and its partners, CARE, and HKI initiated a five-year USAID Multi-Year Assistance Program (MYAP). The central goal is to reduce food and livelihood insecurity by 2011 for vulnerable households in three districts. The strategic objectives are: 1. To protect and enhance livelihoods of vulnerable households through increased agro-pastoral production and improved agro-enterprises; 2. To enhance human capabilities through improved health and nutritional practices in vulnerable in targeted communities; 3. To protect and enhance the resiliency of target communities through improved abilities to identify and respond to recurrent shocks. The program seeks to enhance economic opportunities for vulnerable groups through increased crop production and income generating activities, and to improve the quality of life through better health and nutrition. This evaluation is a study aimed at gaining an understanding of how PROSAN has been implemented, how its implementation track resulted in changes that the program attempted to bring. CRS and its partners in Niger have made considerable progress in meeting the majority of its objectives. Throughout the evaluation, it has become evident that PROSAN has had numerous positive impacts on the lives and livelihoods of participants, despite challenges in terms of droughts and floods, an unstable political climate that directly affected the Food for Work program, and poor infrastructure and variable access to project beneficiaries. Focus groups reported, overall, positive changes in living conditions since the beginning of the program. The program has successfully increased community awareness regarding the importance of early warning systems, improved farming practices and the need to improve nutrition practices. One of the key lessons learned in the program is that people have increased their trust and willingness to engage in group activities and productive community-driven initiatives. There is a growing sense of ownership and pride about group activities such as habbanaye, vegetable gardening, and Early Warning Groups. Participants feel that they are gaining the tools to take charge of their own development activities. The program has provided opportunities to increase good governance and management, as well as transparency at the village level by forming village level farmer farmers groups with elected positions. This is especially important at a time when Niger is undergoing political and administrative change and decentralization, when local communities will have increasing opportunities to get involved in governance. Another positive aspect is that communities are learning how to mobilize resources to manage their own projects and extend the program activities in other domains. 58 PROSAN- Rayuwa Final Evaluation Report TANGO International October 2011 Beneficiaries have been exposed to many new techniques in agriculture, nutrition, hygiene and management that will have lasting effects in improving the quality of life beyond the life of the program. Food crop production has increased as a result of program initiatives. Technical assistance and training was given to smallholder households to promote sustainable farming practices, more productive and more diversified farming systems, and improved agro-enterprise activities. Agricultural potential was enhanced through the rehabilitation of small-scale irrigation and drainage systems, as well as soil and water conservation infrastructure utilizing food-for-work (FFW) programs. Irrigation is the most transformative agricultural investment option within the three regions, providing farmers with extremely productive and profitable new agricultural options, and PROSAN has been a major force in providing farmers with access to irrigation within the program area. Small- scale irrigation is the key to success in the future, decreasing long vulnerability. A high proportion of households that received SO1 training from the program grew greater quantities of each of the three staple food crops than those who did not receive training. Importantly, the production per household of each of the three crops is significantly higher for households that received training than those that did not, indicating that the adoption of recommended practices has had a positive impact on household production of major food crops. In addition, the production per household of training participants in “old” villages is higher (measured by the median) than households in the “new” villages. This result supports the argument that over time farmers adopt the agricultural practices that provide them with the greatest benefits (measured by household production of food crops). Adoption rates of all types of agro-processing and group marketing are significantly higher for households that have received SO1 training. PROSAN SO2 is to protect and enhance the health and nutritional status of vulnerable household members, represented particularly by women of reproductive age and children under five, by increasing access to quality primary health care and improving household health and nutrition practices and sanitary and nutritional coverage in the three targeted regions of Tahoua, Dosso and Zinder. The results of the quantitative assessment of the nutritional status of children 6 to 59 months showed an overall rate of chronic malnutrition and chronic lack of growth at 37.4% for the entire program area. Several intermediate results of PROSAN show an increase in positive health and nutrition behavior for women, however, favorable to the good health and nutrition of vulnerable groups of the population. For example, survey data show increased visits to antenatal clinics and an increase in the percentage of women who received iron supplementation, which now exceeds 80%. The rate of exclusive breastfeeding of infants under six months is high, at nearly 80% in all three program areas. Feeding practices for infants and young children are better in areas that received ISAN / PROSAN compared with the areas that received only PROSAN support, or had no 59 PROSAN- Rayuwa Final Evaluation Report TANGO International October 2011 support from either program. These results suggest positive cumulative impact of training over time. Immunization coverage of children against major killer diseases (tuberculosis, polio, whooping cough, measles and yellow fever) is very good—over 80% in all regions and regardless of the type of vaccine. However, the percentage of children from one to two years old, who are fully vaccinated is quite low, not even reaching 50%. The program did not succeed as hoped in the management of diarrhea, which remains an area of critical vulnerability. The analysis of the evolution of the nutritional status of children under five years in terms of wasting in the intervention area PROSAN shows overall improvement of the latter compared with the results of the baseline survey. This could reflect a positive impact of PROSAN upon the nutritional status of children. While it is difficult to attribute this positive effect exclusively to PROSAN, it can be said that the program has contributed to improving this situation in its area of intervention. To improve access of vulnerable populations in their area of operation, PROSAN officials have implemented a number of strategies including the renovation or construction of health infrastructure and facilitation of training for appropriate management and training of community workers on the actions needed to prevent malnutrition and infection (essential nutrition actions). The results of impact indicators and health practices, and nutrition at the household level, have shown some improvement even if it is slow, due to chronic malnutrition which has deeper causes, some of which have not yet been adequately into account (hygiene and sanitation, improving living conditions of women, etc.). The status of women is progressively changing as a result of the high level of women’s participation in various groups. Women’s visibility and leadership has increased through their engagement with the program and their participation in all program activities. This is a success story that CRS and its partners should build on for future programming in Niger. Recommendations The following recommendations emerged from a combination of focus group discussions, key informant interviews, and the follow-up plenary session with field agents, project managers, and stakeholders. During the field assessment, beneficiaries were asked to contribute their recommendations to improve the program. The recommendations are presented by programmatic sectors. They are designed to enhance the impact of future MYAP programs. A. Priority Recommendations 1. Focus more on excellence and impact at the household level. It is clear that the program has completed scaling up both geographically and programmatically. Everyone in the surveyed districts recognizes that the focus now needs to shift more on quality and excellence. But the question remains what does quality mean 60 PROSAN- Rayuwa Final Evaluation Report TANGO International October 2011 and how should the program begin to enhance the quality. The monitoring unit should focus on that direction. Quality indicators should be identified and efficient information gathering tools developed to obtain information about the quality (in terms of impacts). Review the format of Monthly, Quarterly, and Annual to hold more discussion on the quality of work completed, not only on the quantity. A "top-down" personality has evolved in the Program. This has been useful in getting the program to where it is now, fully scaled-up on a solid foundation. Given the need to address the quality of the program, it is time now to give more attention to finding best practices and capitalizing on lessons learned. Reward staff that practice excellence learning behavior. Staff are recognized and rewarded for having completed planned work, and the work has been focused on getting outputs produced. Now that it is time to improve the quality of the program, staff that are making progress in improving quality with new ideas and best practices should be recognized and rewarded. 2. Future MYAP should prioritize measures that address availability and access to water—the number one constraint mentioned by most households. More emphasis needs to be placed on water for consumption and small-scale irrigation. In addition water harvesting techniques should be promoted. 3. The adult literacy program is a real success story: there is a lot of enthusiasm for it and it increases the likelihood that program activities will be sustainable. In addition, it opens new doors for employment in development projects and NGOs (controller of CFW, secretary of a committee, etc), and to political parties. Any future MYAP should include this component in its activities. 4. Monitoring and Evaluation • There is an urgent need to improve the capacity of the current Monitoring and Evaluation Unit for close attention to the provision of management information, both to support the implementation of programs goals and to feed back into the design of new initiatives in changing environments. The current system must be expanded to monitor quality of outputs, adding GIS tools. • Baseline, midterm, and final evaluations should take place during the same time of year and should involve consistent methodologies, It has been recommended that the same organization perform all three evaluations. • The use of geographic information could improve the efficiency of the Monitoring and Evaluation Unit. GIS has become an indispensable tool for adequately collecting analyzing multiple parameters involved in monitoring key indicators. A good GIS should: boost productivity, improve coordination, create base to make better decisions, and enhance service thereby making better use of MYAP resources. We recommend 61 PROSAN- Rayuwa Final Evaluation Report TANGO International October 2011 that CRS and its partners develop an integrated information system enhanced by the use of a GIS as a useful tool in program monitoring and evaluation. GPS units should be used to geo-reference program data and show distribution of activities on a map of the program area. This will help managers to map out all the indicators and increase transparency and accountability at all levels and attract future funders with solid data and an innovative monitoring technology. 5. The program should go beyond simple message delivery to ensure that participants internalize messages and apply them into their own households. The Program is doing a good job in developing behavioral change strategies, targeting specific messages and utilizing multiple communication channels. However, implementation of the strategies across all strategic objectives is being hampered somewhat by the methodologies being used at the frontlines to transfer information. These tend to focus on top-down, lecture methodologies, ensuring that the message is stated. More rapid and complete behavioral change can be achieved by using methodologies that are more interactive and reflective, ensuring that the messages are understood and internalized. Program leadership should work toward developing sustainability strategies to uphold the impact of the program. 6. Integrate SO3 with other PROSAN SOs: SO3 operates with considerable independence from other SOs. There is a strong need to horizontally integrate disaster preparedness and management in all of the sector activities being carried out by PROSAN in its Title II programming. By not integrating Disaster Preparedness with other strategic objectives, PROSAN is missing opportunities for a synergistic effect. To build resilient communities, a comprehensive approach is needed that combines risk reduction activities (infrastructure, appropriate seeds, livestock protection activities, etc.), early warning and disaster response, and livelihood recovery. Field agents could be trained to respond to disaster-related health threats as a means of integrating disaster preparedness with the rest of the program. The program may consider reducing the geographical coverage to implement a more comprehensive approach since resources will need to be more concentrated, while still promoting early warning and emergency response in the wider area. 7. Commodity Management/Monetization: CRS and its partners should consider for increased management efficiency, and greater programming/distribution activity integration and means of beneficiary verification should consider employing “smartcard, cell phone and biometrics” technologies to reduce human inputting error, increase accountability and account for services provided to each beneficiary. B. Other Recommendations 62 PROSAN- Rayuwa Final Evaluation Report TANGO International October 2011 8. Expand groups in targeted districts to include more female-headed households. The evaluation team found a growing number of female-headed households that appear to be the most vulnerable. 9. Livestock provide important protein to household diets, a source of household income, and a critical means of transport. As such, animals constitute a critical component of the overall livelihood security strategy. Income earned off-farm—to the extent possible—is sometimes invested in the household herd as a form of savings as well as a means of coping with unforeseen and unpredictable difficulties. The farmers’ view is that a mitigation enterprise such as improved poultry and small ruminants can effectively help with the droughts and other emergency situations. 10. In order to further reduce chronic malnutrition (stunting) rates, it is recommended that the project introduce an agriculture component that focuses less on staple foods and micronutrient-rich foods. 11. Develop and strengthen capacities in conflict resolution: At minimum, follow-up programs should ensure that everyone involved in the program receives appropriate training in conflict resolution. It is recommended that communities be categorized in terms of the risk of violent conflict, and that appropriate protection measures are taken given these categorizations. 12. Implement transparent bookkeeping methods for the non-literate: Whatever model communities choose for group activities, a follow-up program must place greater emphasis on the use of bookkeeping methods that are equally transparent to non￾literate and literate/numerate persons. 13. CRS and its partners need to develop clear exit strategies related to disengagement from programming in the program areas. Exit strategies are essential to reduce the potential for dependency and prevent undermining development structures. It is important to convey issues related to time frames to communities, the government, and the USAID mission. The exit strategy should include a phase down, phase over and phase out analysis. Phase down, which involves a gradual reduction of program inputs, is the preliminary stage to both phase over and phase out. Phase over refers to the transfer of responsibility for activities aimed at accomplishing program goals to another entity. 63 PROSAN- Rayuwa Final Evaluation Report TANGO International October 2011 64 PROSAN- Rayuwa Final Evaluation Report TANGO International October 2011 Appendix A: IPTT Indicators Baseline FY07 Target FY08 Target FY08 Achieved FY09 Target FY09 Achieved FY10 Target FY10 Achieved FY11 Target FY12 targets LOA Target 5.5 NT NT 8.4 6.3 NC NT 5.52 7 7 7 3.8 NT NT 5.2 4.5 NC NT NC 5.2 5.2 5.2 9.50% NT 10% 48.50% 10.80% NC 11.30% 60% 75% 75% 75% 23.10% NT 24,2% 42.10% 26.30% NC 27.50% 57% 75.00% 75.00% 75.00% 37.70% NT 39% 46.30% 41.80% NC 44.70% 98.20% 100.00% 100.00% 100.00% 0 NT 8,558 981 2,347 1,296 2,450 1,116 4,950 672 9,015 75.20% NT 77.30% 80.10% 80.60% NC 84% 86% 84.60% 84.60% 84.60% 74.50% NT 75.50% 76% 79.80% NC 83% 88% 88.00% 88.00% 88.00% 0 NT 3,500 2,477 8,300 6,361 6700 5,999 5,503 2160 22,500 0 NT 240 279 70 NC 30 76 90 0 376 ND NT 110 42 40 NC 30 NC 91 54 100 40% NT NT 39% 38% NC NT 33.51% 37% NT 37% 48.80% NT NT 41% 46.70% NC NT 34.30% 44.90% NT 44.90% 30.90% NT NT 37% 29.50% NC NT 32.70% 28.40% NT 28.40% 41% NT NT 35% 39% NC NT 38.02% 38% NT 38% 52.00% NT NT 34% 49.70% NC NT 36.10% 47.80% NT 47.80% 31.00% NT NT 36% 29.60% NC NT 39.90% 28.50% NT 28.50% 0 NT 10,954 3,511 4,500 28,380 5,000 26,939 29,500 15,000 103,330 0 NT 5275 1004 3167 2856 1450 1549 4262 1221 10892 53.40% NT 54.80% 67% 57.50% NC 61.40% 92% 75.00% 75.00% 75.00% ND NT ND 26% NT NC NT 37.54 40% 40% 40% 0 NT 26,000 2,062 50,390 23,442 44,051 8,382 14,181 3,920 22,444 30 240 240 240 240 240 240 240 240 240 240 ND NT ND 35% NT NC NT 100% 100% 100% 100% Last IPTT revision May 19, 2011 NT = No Target NC = Not collected FY08 Achieved represents Mid Term Evaluation Data Indicators Goal: Reduce food insecurity of rural Nigerien HHs from the most vulnerable communities in the regions of Dosso, Tahoua and Zinder by 2012. Strategic Objective n°1 : protect and enhance livelihoods improving conditions for increased agriculture production and improved agro-enterprise Impact indicator 1.1: Average number of months of adequate household food provision (IF FFP) Consortium Impact indicator 1.2: Household Dietary Diversity Score (HDDS): (IF FFP) Consortium IR 1.1 : Households and communities have adopted agricultural processing and value added practices promoted by the project Monitoring indicator 1.1.1: : Percentage of households that have adopted at least one of the processing practices promoted. Consortium Monitoring indicator 1.1.2: Percentage of households that have adopted at least one of the marketing practices promoted. Consortium Monitoring indicator 1.1.3: Percentage of communities that established a system for processing and marketing of agro pastoral products Consortium IR 1.2: HHs and communities have adopted natural resource management and agro-pastoral production practices promoted by the project. Monitoring indicator 1.2.1: Number of individuals who have received MYAP supported short term agricultural sector productivity training (MI USAID) Consortium Monitoring indicator 1.2.2: Percentage of HHs who adopt at least one NRM practice promoted by the project Consortium Monitoring indicator 1.2.3: Percentage of beneficiaries (farmers) using a project-defined minimum number (1) of sustainable agriculture technologies (IS FFP). Consortium Monitoring indicator 1.2.4: Number of additional hectares under improved technologies or management practices as a result of MYAP assistance (MI USAID) Consortium IR 1.3: Local POs have demonstrably enhanced their administrative, managerial, leadership and accountability skills. Monitoring indicator1.3.1: Number of producers organizations, water users associations, trade and business associations, and community-based organizations (CBOs) receiving MYAP assistance (MI USAID) Consortium Monitoring indicator 1.3.4: Number of assisted communities with improved community capacity as a result of project assistance (IS FFP) Consortium Strategic Objective n°2 : protect and enhance human capabilities by improving household health and nutrition status, especially that of children under five and women of childbearing age Impact indicator 2.1a: Percentage of children 6-59 months who are stunted (i.e., <-2 HAZ), disaggregated by gender. (IF FFP) Consortium Boys Impact indicator 2.1b: Percentage of children 0-59 months who are underweight (<-2 z-score weight-for-age), disaggregated by gender (IF FFP) Girls Consortium Boys Girls Monitoring indicator 2.1.2 : Number of antenatal care (ANC) visits by skilled providers from MYAP assisted facilities (MI USAID) Consortium IR 2.1: Households within target population have improved access to primary health care. Monitoring indicator 2.2.1 : Number of people trained in child health and nutrition through MYAP supported health programs (MI USAID) Consortium IR 2.2: Health workers use new knowledge and skills to improve implementation of IMCI within target locations. Monitoring indicator 2.3.3: Number of children reached by USG-supported nutrition programs (MI USAID) IR 2.3 : HH health and nutrition practices are improved Monitoring indicator 2.3.1: % Households who have adopted at least one program- recommended health or nutrition practice Consortium Consortium Strategic Objective n°3 : protect and enhance communities’ resiliency by improving their abilities to identify and respond to crises and shocks Monitoring Indicator: 2.3.2 % of children exclusively breast fed during the last 24 hours preceding the survey Consortium IR 3.1: Target communities have operational emergency early warning systems Monitoring Indicator 3.1.1 : Number of assisted communities with disaster early warning and response systems in place (IS FFP) Consortium IR 3.2: Community assets are protected during shocks Monitoring Indicator3.2.2 : Number of assisted communities with improved physical infrastructure to mitigate the impact of shocks in place (IS FFP) Consortium