SALOHI Multi-Year Assisstance Program (MYAP) Final Evaluation Report 27 May 2014 Prepared by: Vonifanja Andianaonitsoa, Bernard Crenn, Zoelimalala Ramanase, Kathy Tilford (Team Leader) and Ange Tingbo This assessment was made possible by the support of the American People through the United States Agency for International Development (USAID), Office of Food for Peace (FFP). The contents of this report are the sole responsibility of the SALOHI program and do not necessarily reflect the views of USAID or the United States Government. CRS/MG SALOHI MYAP FE/USAID-FFP-2-00002 i Acknowledgements The Qualitative Assessment team pays tribute to the many individuals who made our time in Madagascar productive and enjoyable, enabling us to complete our work with a minimum of problems. First and foremost, we thank the many beneficiaries who met with us in the field. We appreciated their time, candor and many suggestions. We also thank the local stakeholders who took part in the key informant interviews, providing another perspective on SALOHI’s effectiveness and impact. Other individuals and groups that made a difference are:  We would like to thank USAID/FFP staff in Washington and Pretoria, particularly the M&E team, for their technical assistance before, during and after the exercise;  USAID Madagascar for their support, their time and the thought-provoking questions they asked and suggestions they made during the debriefing  SALOHI implementing partners including ODDIT, FITEA, BDEM and CARITAS  SALOHI staff that set up the visits to the 14 fokontany for the Qualitative Study: Without exception, all the individuals and groups we needed to see each day were ready and waiting for us – a model of superb organization and coordination. We appreciated the field staff who accompanied us – field agents, coordinators, chefs de projets, and of course the drivers who put in long hours.  Members of the PCU tirelessly answered questions, produced data and provided documents throughout the assessment: Heritiana Delphin, Faly Rabetaliana, Noro Hasina Ratsimbazafy, Jules Bosco Bezaka, Avo Ratoarijaona, Mbinina Randimbiarison, and Martial Andrianasolo.  A special debt of gratitude is owed to Jennifer Peterson, the SALOHI Chief of Party, who ensured that the team had the best of everything to facilitate their work.  CRS/Madagascar provided sterling logistical support throughout. We reserve a special Misaotra Tompoko to Voahary Razafinandriamandroso, Logistical Officer, and all the CRS/Tana drivers.  Ricko Ratovoson and Aina Randriamialy who drove us around Madagascar for three weeks. Their excellent driving skills and their good humor day in and day out made every day enjoyable.  Landy Raharijaona provided superb administrative support on a daily basis, including the organization of workshops and meetings.  Dina Rakotoanadahy ensured that all our travel and lodging arrangements were made in a timely fashion – and in nice places.  Meeting the IT needs of six people is no easy task but Charles Randrianirina (Tom) met the challenge successfully, responding to each and every plea for help.  Daniel Andriatsimba (Dany) provided beautiful maps upon request.  Rob Green was our go-to guy for random requests: we especially appreciated his compiling the information-packed binders and for giving us a sneak preview of the Activity Sheets, which we used in this report.  Nick Reed Krase kindly shared drafts of the lists of Best Practices and Lessons Learned that he is compiling.  Two members of the Qualitative Study team deserve special recognition: Soloarisoa Ranoromalala (also a member of the PCU) served as our liaison with the SALOHI team, especially those in the field. She did all our scheduling and solved each and every problem that arose so that the field work progressed without a hitch. Zoely Ramanase spent two weekends working with a printer to make sure we had the documents we needed for a workshop and for field work. CRS/MG SALOHI MYAP FE/USAID-FFP-2-00002 i Table of Contents Acknowledgements ...................................................................................................................................... i Table of Contents ......................................................................................................................................... i Abbreviations and Acronyms ....................................................................................................................... v A. Executive Summary ......................................................................................................................... vii 1. Overview of the Assessment Purpose and Objectives ............................ Error! Bookmark not defined. 2. Key Findings ......................................................................................................................................... vii 3. Conclusions .......................................................................................................................................... viii 4. Key Recommendations for Future Programming .................................................................................... x B. Introduction ............................................................................................................................................ 1 1. Overview of SALOHI .............................................................................................................................. 1 2. Map of Target Zones ................................................................................................................................ 2 C. Assessment Methodology ...................................................................................................................... 3 1. Final Quantitative Survey (FQS) .......................................................................................................... 4 1.1 Sampling Design ....................................................................................................................... 4 1.2 Staffing and Training ................................................................................................................ 4 1.3 Data Collection Period and Processes ...................................................................................... 5 1.4 Data Entry and Analysis ........................................................................................................... 5 1.5 Data Limitations ....................................................................................................................... 5 2. Qualitative Study .................................................................................................................................. 6 2.1 Staffing and Training ................................................................................................................ 6 2.2 Description of Process: Team, Sites, Tools .............................................................................. 6 2.3 Qualitative data limitations ....................................................................................................... 7 3. Data Dissemination............................................................................................................................... 8 D. Findings and Analysis............................................................................................................................ 8 1. Demographics and Household Characteristics ..................................................................................... 8 2. SO1 - Health and Nutrition ................................................................................................................. 11 2.1. Brief Description of Interventions .......................................................................................... 11 2.2. Overall Results ....................................................................................................................... 13 2.3. Maternal and Child Nutrition (IR1) ........................................................................................ 15 2.4. C-IMCI: Disease Prevention Practices (IR2).......................................................................... 19 2.5. Conclusion .............................................................................................................................. 23 2.6. SO1-specific Recommendations ............................................................................................. 24 3. SO2 – Livelihoods .............................................................................................................................. 25 3.1. Brief Description of Interventions .......................................................................................... 25 3.2. Overall Results ....................................................................................................................... 26 3.3. Farm Productivity (IR 1) ........................................................................................................ 28 CRS/MG SALOHI MYAP FE/USAID-FFP-2-00002 ii 3.4. Agribusiness (IR 2) ................................................................................................................. 31 3.5. Village Savings and Loan Associations (IR 3) ....................................................................... 32 3.6. Key Findings........................................................................................................................... 33 3.7. Significant external factors affecting project outcomes ......................................................... 34 3.8. Conclusions ............................................................................................................................ 34 3.9. SO2 Specific Recommendations ............................................................................................ 35 4. SO3 - Disaster Risk Reduction ........................................................................................................... 36 4.1 Achievements ......................................................................................................................... 36 4.2 Community Resilience: The Community Vulnerability Index (CVI) .................................... 38 4.3 Emergency Preparedness (IR 3.1) .......................................................................................... 39 4.4 Resource Management (IR 3.2) .............................................................................................. 40 4.5 Construction/rehabilitation of Canals and Dams (IR 3.3) ...................................................... 41 4.6 Social Protection for Urban Households ................................................................................ 42 4.7 Conclusions ............................................................................................................................ 43 4.8 SO3-Specific Recommendations for Future Programs ........................................................... 43 E. Cross-cutting Themes .......................................................................................................................... 44 1. Gender ................................................................................................................................................. 44 1.1. The Gender Dimension in the SALOHI Program .................................................................. 44 1.2. Relevance of the Gender Dimension ...................................................................................... 44 1.3. Impact of Gender Dimension in SALOHI .............................................................................. 47 1.4. Sustainability of the Gender Dimension and its Impact ......................................................... 49 1.5. Conclusion and Key Gender Recommendations .................................................................... 51 2. Environment ....................................................................................................................................... 51 2.1 Brief Description of the Environment Interventions .............................................................. 51 2.2 Evaluation and Analysis of Environmental Activities ............................................................ 52 2.3 Conclusion .............................................................................................................................. 54 2.4 Recommendations .................................................................................................................. 54 3. Good Governance ............................................................................................................................... 54 4. Partnership and Sustainability ............................................................................................................ 56 F. Program Strengths and Weaknesses .................................................................................................. 61 G. Recommendations................................................................................................................................ 63 H. Bibliography of Key Documents Consulted ...................................................................................... 65 CRS/MG SALOHI MYAP FE/USAID-FFP-2-00002 iii Appendices Appendix 1: SALOHI Results Framework Page 1 Appendix 2: SALOHI Final IPTT, updated Page 2 Appendix 3: Scope of Work Page 10 Appendix 4.A: Final Quantitative Survey Team Page 28 Appendix 4.B: Qualitative Study Team Page 29 Appendix 5: Map of Assessment Sites Page 30 Appendix 5.A: Final Quantitative Survey Sites and selection process Page 31 Appendix 5.B: Qualitative Study Sites Page 36 Appendix 6: Training Program – Final Quantitative Survey Page 27 Appendix 7: Interviewer Guide – Final Quantitative Survey Page 45 Appendix 8.A: Final Quantitative Survey Questionnaires and Other Tools Page 67 Appendix 8.B: Qualitative Study Guides and Other Tools Page 116 Appendix 9: SALOHI Performance Management Plan Page 157 Appendix 10: Performance Indicator Reference Sheets Page 191 CRS/MG SALOHI MYAP FE/USAID-FFP-2-00002 iv List of Tables Table 1: Composition of children under five years in households by geographic zone ................ 8 Table 2: Average age of the head of household by zone and by NGO (years) .............................. 9 Table 3: Heads of household who can read, by zone and by NGO (%) ......................................... 9 Table 4: Marital status of heads of households .............................................................................. 9 Table 5: Women who reported being pregnant or lactating, by zone and by NGO (%) .............. 10 Table 6: Income-generating activities reported by heads of households (%) .............................. 10 Table 7: Use of revenue reported by head of household (%) ....................................................... 10 Table 8: Ownership of assets reported by head of household (%) ............................................... 11 Table 9: SO1 Key indicators and achievements ........................................................................... 13 Table 10: Effect of RANO and SantéNet2 projects ..................................................................... 22 Table 11: SO2 key indicators and achievements ......................................................................... 26 Table 12: SO3 Achievements ....................................................................................................... 36 Table 13: Gender ratios by zone .................................................................................................. 50 Table 14: SALOHI Environmental Indicator Status, Target vs. Achieved LOA, 2013............... 52 List of Figures Figure 1: Map of target zones ........................................................................................................ 2 Figure 2: Stunting and Underweight values, Baseline vs. Final Evaluation ................................ 14 Figure 3: Practice of Immediate Breastfeeding (LOA Target = 81%) ......................................... 15 Figure 4: Practice of Exclusive Breastfeeding (LOA Target = 76%) .......................................... 15 Figure 5: Increase in GMP participation (LOA Target = 75%) ... Error! Bookmark not defined. Figure 6: Changes in Hygiene Practices ...................................................................................... 20 Figure 7: Changes in months of HH food provisioning: +1.3 months (FQS, 2013) .................... 26 Figure 8: Changes in HH food diversity score ............................. Error! Bookmark not defined. Figure 9: Changes in HH coping strategies (FQS 2013) .............................................................. 28 Figure 10: Yield increase for the 5 target crops in kg/ha ............................................................. 29 Figure 11: Use of at least two improved agricultural techniques (FQS 2013) ............................. 30 Figure 12: Agribusiness participation (FQS 2013) ...................... Error! Bookmark not defined. Figure 13: VSLA participation (FQS 2013) ................................. Error! Bookmark not defined. Figure 14: VSLA benefits (FQS 2013) ........................................................................................ 33 Figure 15: Benefits of SALOHI infrastructure ............................................................................ 37 Figure 16: % HH using various coping strategies during food insecurity ................................... 38 Figure 17: HH awareness, maintenance and use of SALOHI infrastructure ............................... 40 Figure 18: Percent of Female Headed Households in SALOHI zones, 2013 .............................. 45 Figure 19: % male headed HH participation in health and nutrition activities ............................ 46 Figure 20: % male headed HH by type of participation in health/nutrition activities ................. 46 Figure 21: % male headed HH by type of participation by NGO ................................................ 46 Figure 22: Go Green ..................................................................................................................... 52 Figure 23: Proportion of HH participating in environmental activities ....................................... 53 Figure 24: Environmental changes in communities according to final qualitative evaluation survey respondents ....................................................................................................................... 53 CRS/MG SALOHI MYAP FE/USAID-FFP-2-00002 v Abbreviations and Acronyms AB Agribusiness ADRA Adventist Development and Relief Agency AROPA Projet d'Appui au Renforcement des Organisations Professionnelles et aux services Agricoles ASC Agricultural Service Center (Ministry of Agriculture) AT Agent de Terrain AUE Water Users Group (Association des Usagers de l'Eau) AUP Road Users Group (Association des Usagers de Piste) (S)BCC (Social and) Behavior Change Communication BDEM Bureau de Développement de l'ECAR Mananjary BDR SALOHI overall database CARE Cooperative for Assistance and Relief Everywhere CBO Community Based Organization CHV Community Health Volunteer CI Confidence Interval C-IMCI Community level Integrated Management of Childhood Illnesses CPS Social Promotion Centers – Centre de Promotion Sociale CRS Catholic Relief Services CSB Local Health Center – Centre de Santé de Base CSI Coping Strategy Index CVI Community Vulnerability Index DAP Development Assistance Program (USAID) Dina Local group or community bylaws or regulations DPMP Disaster Preparedness and Mitigation Plan DRDR Direction Régionale du Développement Rural DREL Direction Régionale de l’Elevage (Ministry of Livestock) DRR(M)(C) Disaster Risk Reduction (Management) (Committee) EBF Exclusive Breastfeeding ENA Essential Nutrition Actions EWS Early Warning System FAO Food and Agriculture Organization FFA Food For Assets FFS Farmer Field School FFT Food for Training FFP Food For Peace FGD Focus Group Discussion FHH/MHH Female/Male Headed Household FITEA Fampivoarana Iombonana amin'ny Tsara Entina ho an'ny Ankohonana FL Farmer Leader (Paysan Modèle) fokontany Administrative division (sub-divisions of a commune). SALOHI works in 592 fokontany FQS Final Quantitative Survey GOM Government of Madagascar GMP Growth Monitoring and Promotion GRET Groupe de Recherches et d'Echanges Technologiques GTZ Gesellschaft für Technische Zusammenarbeit (German development agency) CRS/MG SALOHI MYAP FE/USAID-FFP-2-00002 vi Ha or HA Hectare HDDS Household Dietary Diversity Score (from FANTA) HH Household(s) IBF Immediate Breastfeeding IEC Information Education Communication IGA Income Generating Activity IMA Infrastructure Management Association IPM Integrated Pest Management IPTT Indicator Performance Tracking Table IR Intermediate Result IYCF Infant And Young Child Feeding KII Key Informant Interviews LOA Life Of Activity LOL Land O’Lakes MCH Maternal and Child Health MCHN Mother and Child Health and Nutrition MOH Ministry of Health mos. Months MTE Midterm Evaluation MYAP Multi-Year Assistance Program NCHS National Center for Health Statistics NGO Non-Governmental Organization NHM Nde'ho Maitso (Go Green) (C)NRM Natural Resource Management (Committee) ONN National Nutrition Office PD/Hearth Positive Deviance/ Hearth(FARN) PIC Pole Integré de Croissance (European Union program) PLW Pregnant and Lactating Women PROSPERER Programme of Support for Rural Microenterprise Poles and Regional Economies SAMBAIKA Support group for pregnant and lactating women SE1 Southeast 1 SE2 Southeast 2 SO Strategic Objective SOLDIS Solar Distillation SRI System of Rice Intensification Tavy Slash and burn VA Village Agent (for VSLA support) VSLA Village Savings and Loan Association WASH Water, Sanitation and Hygiene WFP World Food Program WG Working Group WHO World Health Organization CRS/MG SALOHI MYAP FE/USAID-FFP-2-00002 vii A. Executive Summary In order to respond to chronic and transitory food insecurity in vulnerable areas of Madagascar, Catholic Relief Services – United States Conference of Catholic Bishops (CRS) received funding from USAID's Office of Food for Peace (FFP). CRS formed a partnership with three international NGOs: the Adventist Development and Relief Agency (ADRA), the Cooperative for Assistance and Relief Everywhere, Inc. (CARE), and Land O'Lakes International Development Division (LOL) to implement the Strengthening and Accessing Livelihood Opportunities for Household Impact (SALOHI) Program. CRS works through four local implementing partners: BDEM, CARITAS, ODDIT and FITEA. The goal of this five-year food aid program (July 2009 – June 2014) is to reduce food insecurity and vulnerability through three integrated Strategic Objectives: improve health and nutritional status of children under five; improve livelihoods of food-insecure households; and increase community resilience to shocks. Four cross-cutting themes reinforce these three objectives: gender, environment, good governance and sustainability. SALOHI currently reaches approximately 630,000 people located in five geographical zones and seven regions, with each zone having one or more NGOs implementing activities: South (CARE/CRS), Southeast 1 (SE1; ADRA, BDEM, FITEA), Southeast 2 (SE2; LOL), Center (ADRA), and East (CARE/ODDIT and CARITAS/Fenerive Est). Among beneficiaries, female-headed households (FHH), farmers with small landholdings (less than 1.5 ha), and remote communities are particularly at risk. 1. Key Findings1  92.5% of women with children under five years of age reported that their children participated in growth monitoring and promotion sessions, 34% participated in SALOHI PD hearth groups (for malnourished children), 57.5% in SALOHI pregnant and lactating support groups, and 99% of households with children under five received visits from Community Health Volunteers (CHVs).  Among children 0-59 months of age, stunting declined from 47% to 41% and underweight decreased from 29% to 20% (slightly less than program targets of 39% stunting and 19% underweight).  Immediate and exclusive breastfeeding increased from 71% to 81% (meeting the program target) and 56% to 75% (falling slightly short of the final evaluation target of 76%).  The percentage of women who reported practicing three or more promoted food hygiene behaviors (44.4% - 66%) and hand washing with soap after at least two critical events (21% to 38%) increased from 2009 - 2013, falling slightly short of program targets (69% and 46% respectively).  The percentage of women who reported that their child slept under a mosquito net increased from 61% to 81% from 2009 to 2013 (SALOHI baseline and final).  The percentage of mothers who reported that their children were sick in the two weeks prior to the survey decreased from 44% to 31% between the baseline and final evaluation.  48% of household heads and 33% of mothers of children under five surveyed participated in SALOHI FFS groups; 29% of mothers of children under five and 38% 1 All of the key findings reported below are statistically significant at an alpha level of 5%; p-value or probability is < 0.05. CRS/MG SALOHI MYAP FE/USAID-FFP-2-00002 viii of HH heads participated in SALOHI VSLA, and 7 - 10% (women and men interviewed) participated in SALOHI agribusiness groups.  The percentage of farmers who reported using two more improved agricultural technologies during the last growing season increased to 79% in 2013 (surpassing the program target of 50%).  The yields of key crops increased (80% for rice, 38% for cassava, and 20% for maize) between the baseline survey and the final evaluation.  Over the life of the program, the average number of months of food provisioning increased from 7.7 months at baseline to 9.1 months at the final evaluation, surpassing the LOA target of 8.7 months.  The SALOHI program constructed 900 km of roads and irrigated 9,800 ha of land (99 irrigation systems rehabilitated or constructed). An additional 4,295 ha of land was reforested, contributing to improved agricultural production and community resilience.  80% of all households surveyed participated in FFA activities, and the average household received 100 kg of rice or sorghum, and 5 kg of refined vegetable oil.  The Coping Strategy Index (CSI) decreased from 24.9 to 12.6, a 50% reduction (the final target was 12.5), and the community vulnerability index increased from 4 to 10, exceeding the program target of 82 .  According to focus group discussions and key informant interviews, as well as the results from the final quantitative survey, the effects and impacts of the SALOHI program appear likely to be sustained over time (according to 75% or more of beneficiaries interviewed). 2. Conclusions Overall, SALOHI has had a highly visible and measurable impact in the program area, meeting or exceeding almost all targets for major indicators, including impact indicators. Success factors contributing to these improvements in child health include a tightly-focused health and nutrition program that involved the entire community; the promotion of a relatively small number of key nutrition and hygiene practices that required few resources and little risk to adopt; and effective investment in training community health volunteers to reach as many women and children as possible. Without increasing the budget, SALOHI reached an additional 138,000 beneficiaries over the original target, demonstrating efficiency and careful management of resources. Improving household food provisioning was a major goal of the program. The combination of the Farmer Field School (FFS) approach with the recruitment and training of Farmer Leaders proved effective in convincing farmers to adopt new practices and in increasing yields. Especially noteworthy are spill-over effects: everyone interviewed in the field during the qualitative exercise knew about the agricultural techniques promoted by the program, even if they had not participated directly in SALOHI activities. Based on comments made during FGD from VSLA participants, another highly appreciated livelihood intervention is the creation of Village Savings and Loan Associations (VSLA), an intervention that helps people 2 The Coping Strategy Index (CSI) was developed by CARE and WFP to measure the frequency of which households use asset depleting strategies during periods of food insecurity (home.wfp.org/stellent/.../wfp211058.pdf). The Community Vulnerability Index is a composit index developed by the SALOHI team to measure community vulnerability to shocks, and includes the number of months the community is accessible by emergency vehicles, the existence of a disaster risk reduction (DRR) committee and a disaster prevention and mitigation plan (DPMP), DRR simulation exercises, and the proportion of households in the community with access to irrigation. CRS/MG SALOHI MYAP FE/USAID-FFP-2-00002 ix meet pressing financial needs but also allows them to save money in order to invest in productive activities. Noteworthy progress was also made helping households find better ways to cope with shocks that affect their food security. Helping communities to 1) better manage their natural resources such as forests, land and water and to 2) prepare for disasters such as cyclones and flooding was the third major focus of the program, with particular attention paid to disaster risk reduction (DRR) and infrastructure development. Based on comments during FGD, SALOHI beneficiaries especially appreciated infrastructure development, partly because every household had the opportunity to be fully involved in selecting these activities and working on them. As for women’s participation, half of FFS group members were women and over half of VSLA group members were women. However, according to the qualitative data, some women wanted to participate in the various SALOHI activities but were not able to due primarily to three factors: a lack of time (due to women’s domestic and commercial activities; 46% of non-participants), lack of awareness (22% of non-participants) and limited access to land (2.6% of non-participants). Although the gender component of the program was not as strong as it could have been (based on staff interviews and level of engagement in and awareness of gender issues), there were positive outcomes: men are now more involved in their children’s health; women make up more than a third of Farmer Leaders and almost half of agents who help set up Village Savings and Loan Associations; just over half of the agribusiness participants are women; and women and men alike state that women have a greater voice in household decisions and are more inclined to speak up and be heard at community meetings. Based on FGD and KII, document reviews, and participant observation, the strengths of the program include: an excellent design coupled with good implementation; improved social cohesion as a result of group work within all three Strategic Objectives; strong endorsement and active support from community leaders; excellent management by the Program Coordination Unit; better awareness of the importance of gender equity and the need for women to be more involved in decision-making at the household and community levels; a special emphasis on capacity building of community volunteers and management structures as preparation for sustainability; widespread dissemination of lessons learned and best practices both internally and externally; and proactive leadership with a proven track record in identifying obstacles and resolving problems3 . The final evaluation team noted that this is one of the first projects they have seen with clear graduation criteria for community based groups who are being actively prepared for program handover. SALOHI activities are highly relevant to the needs of the 638,000 beneficiaries and the focus on building the local capacity of individuals and of groups will help to ensure that the impacts the effects of these activities continue after the program ends. By starting with an excellent program design and implementing it through dedicated community volunteers and community committees, SALOHI has achieved most of its objectives and met or exceeded almost all major targets. Results are evident: healthier children, increased agricultural production, greater social cohesion, improved management of natural resources, increased access to markets and essential services, and communities that are better prepared to cope with disasters. The spill-over effect, especially for the livelihoods component, is evident and the program has certainly benefited many more people than just direct program participants. 3 Examples of proactive leadership include corrections made following the midterm evaluation, and partnership changes made by Land O’Lakes and CRS when required. CRS/MG SALOHI MYAP FE/USAID-FFP-2-00002 x 3. Key Recommendations for Future Programming For an even greater impact in future programs, consider the following recommendations: 1. Develop and implement a strong gender component, including gender questions in the baseline survey; an in-depth gender analysis; a focused gender communication strategy; at least one full time focal point to ensure gender mainstreaming; and clear accountability so that all staff are responsible for gender results. 2. Include interventions for female heads of households, especially those with little or no access to land. This could include providing alternative livelihood activities such as poultry raising, bee-keeping and other productive options that do not require much land. 3. Start activities in at least 80% of target communities in the first year; avoid implementation in successive waves as communities reached last are unlikely to have the time to consolidate achievements and benefit fully from the program’s interventions.4 4. Ensure that NGO field staff receive training in general development approaches such as adult learning, group dynamics, quality assurance, and post-project sustainability and replicability. 5. Include a WASH component in programs where disease prevention is a goal. 6. Incorporate simple appropriate technologies to solve common problems. Examples include: treadle pumps for vegetable farming near bodies of water; simple drinking water solutions; methods to promote hand washing such as Tippy Taps; and chicken coops made from local materials. 7. The dissemination of knowledge across Strategic Objectives is essential for maximum impact and better integration. Develop a simple plan along the lines of an integrated training curriculum so that everyone knows which messages to include for each activity. 4 Working in communities for a longer period of time allows staff to (1) cover more than 1 – 2 growing seasons, (2) preare more groups to graduate to ensure effects and impacts are sustained (3) allows for a full/ longer cycle for VSLA and for VSLA replication. Field staff where communities started later expressed these concerns during key informant interviews. CRS/MG SALOHI MYAP FE/USAID-FFP-2-00002 1 B. Introduction 1. Overview of SALOHI The most vulnerable populations in Madagascar include rural farmers living in eastern coastal areas ravaged by cyclones and floods, farmers living along protected forest corridors in inaccessible areas in the east-central part of the country, and pastoralists and farmers in the drought stricken south. Among these groups, female-headed households (FHH) are particularly at risk. In order to respond to continued chronic and transitory food insecurity in these areas, Catholic Relief Services – United States Conference of Catholic Bishops (CRS) formed a partnership with three other international NGOs: the Adventist Development and Relief Agency (ADRA), the Cooperative for Assistance and Relief Everywhere, Inc. (CARE), and Land O'Lakes International Development Division (LOL) to implement the Strengthening and Accessing Livelihood Opportunities for Household Impact (SALOHI) Program. CRS works through four local implementing partners: BDEM, CARITAS, ODDIT and FITEA. The principal goal of this five-year food aid program (July 2009 – June 2014) is to reduce food insecurity and vulnerability through three closely-integrated Strategic Objectives (SO): - SO1: Improve health and nutritional status of children under five - SO2: Improve livelihoods of food-insecure households - SO3: Increase community resilience to food security shocks through disaster risk reduction Four cross-cutting themes reinforce these strategic objectives: gender, environment, good governance and sustainability. Approximately 37% of program resources are dedicated to support agriculture production and income generation activities, 36% to support health, nutrition and water sanitation activities, and 21% to support disaster management, mitigation and governance. The remaining six percent of program resources are dedicated to social protection centers and extremely vulnerable households in urban areas. The program Results Framework is included in Appendix 1. The SALOHI program originally targeted seven regions, 21 districts, 120 communes and 544 fokontany (an administrative unit composed of one or more communities) for a total of 492,000 beneficiaries. However, the program was able to exceed the original beneficiary target by 130,000 and currently reaches approximately 630,000 people in 112 communes and 592 fokontany, without additional financial resources. Beneficiaries are located in five geographical zones (see map on following page), with each geographical zone having one or more NGOs implementing activities: South (CARE and CRS), Southeast 1 or SE1 (ADRA and CRS/BDEM and CRS/FITEA), Southeast 2 or SE2 (Land O’Lakes)5 , Center (ADRA), and East (CARE/CRS/ODDIT and CRS/CARITAS). These geographical areas correspond to priority areas identified by the Government of Madagascar (GOM) and the World Food Program (WFP) for high levels of vulnerability to shocks, and food insecurity. 5 In the SALOHI baseline report, data is presented for four regions (Center, East, South, South East). Data from the Final Quantitative Survey [FQS] is disaggregated into five geographic regions (Center, East, South and SE1 and SE2). CRS/MG SALOHI MYAP FE/USAID-FFP-2-00002 2 2. Map of Target Zones Figure 1: Map of SALOHI target zones, by Region, District and NGO CRS/MG SALOHI MYAP FE/USAID-FFP-2-00002 3 C. Assessment Methodology As detailed in the Scope of Work in Appendix 3, the overall purpose of this final assessment was to determine the impact of the SALOHI program in target zones at the beneficiary, household and community levels and to gather sufficient data and information to: 1. Assess whether the MYAP outcomes and impacts were achieved in line with the stated goal, objectives and intermediate results; 2. Describe current knowledge, attitudes and practices in SALOHI target communities with respect to program interventions; 3. Determine current adoption levels for practices that SALOHI staff have promoted over the life of the program (health and nutrition practices, crop production practices, marketing practices, and disaster prevention and mitigation strategies); 4. Document best practices, lessons learned and innovations 5. Collect quantitative data to measure changes in key project indicators since 2009, comparing indicator values at program end date against targets and baseline values; 6. Collect qualitative data to help staff identify opportunities and environmental contexts that contributed to (or detracted from) program implementation and results; 7. Assess challenges to the project and the impact of these challenges on project performance; 8. Assess the effectiveness of program strategies, approaches and tools; 9. Determine potential program sustainability; 10. Provide recommendations for the current project and for future programs; and 11. Share information on project impacts with target communities. The principal sources of data for this assessment came from the final quantitative survey (FQS) conducted in September 2013, the qualitative study carried out in October-November 2013 using focus group discussions (FGDs) and key informant interviews (KII), and the baseline survey conducted in September/October 2009 (which was both qualitative and quantitative). The assessment team used additional sources of information to validate findings and for a more in-depth picture of the program: an extensive document review, including reports, studies and periodic monitoring and evaluation summaries; the SALOHI relational data base; in-depth discussions with local stakeholders and with SALOHI and other NGO staff; and informal observations in the field during the qualitative study. By triangulating these sources, the assessment team was able to determine current knowledge, attitudes, practices, resources and capacities that affect household food security and to obtain updated information on key program elements including food availability, food access, food utilization and community resiliency. To facilitate the evaluation of program impacts, data was disaggregated by NGO, zone, and gender where possible and appropriate. The team carrying out the qualitative study also looked at the benefits and impacts from synergies resulting from collaboration with other development programs, including USAID-funded projects, in target zones (comparing results in zones where SALOHI staff collaborated with other development partners, to results where those development partners don’t exist). CRS/MG SALOHI MYAP FE/USAID-FFP-2-00002 4 1. Final Quantitative Survey (FQS) The FQS was conducted in September 2013 by a professional survey service (PROESSECAL) using the same survey tools and sampling methodology as the 2009 baseline survey, with some additional questions to evaluate behavior changes and impacts over time. This survey included a random sample of all households in SALOHI zones (beneficiary and non-beneficiary). The purpose of this population-based household survey was to evaluate household food security, resilience, and the health and nutrition status of children under five in communities targeted by the SALOHI program. The FQS included beneficiaries and non-beneficiaries. A Type I/Simple pre-post comparison (“Adequacy”) design for the study which involves collecting data before and after program implementation (pre and post test), was used to document levels of change in key indicators over the life of the program. 1.1 Sampling Design Calculation and sampling of respondents was the same as for the baseline survey. A two-stage stratified cluster design was used in the SALOHI Final Evaluation; the primary sampling units were local communities (fokontany) and households (HHs) were secondary sampling units. Fokontany were selected using PPS (Probability Proportional to Size). Because each community and NGO target zone differs in terms of overall population, a weighted analysis was used to account for differences in population levels. Both fokontany and HH were randomly sampled. A full list of all fokontany and all households in each fokontany were used to randomly select fokontany and households. A total of 124 fokontany were randomly selected – 120 target fokontany and four replacement fokontany (in case the original fokontany selected was not reachable or had a cultural event which precluded their participation in the survey). A total of 30 households (HH) were randomly selected from household lists provided by each of the fokontany sampled – 24 primary households, and six replacement households. Please see Appendix 3 and Appendix 5 for more detailed descriptions of sampling protocols, including a description of how the evaluation took into account households with children under five, and households with women 15 – 49 years of age. To allow each NGO to evaluate the impact of their individual activities and to measure impact in their program zones, the sampling protocol provided results that are representative for each NGO. Although not statistically representative, the results for each geographic zone are also presented along with data disaggregated by male- and female-headed HHs (MHH and FHH). Appendix 4.A includes a map of FQS sites. FQS survey tools were developed so that the structure and contents matched the baseline survey as closely as possible. Tools included a Head of Household Questionnaire; a Caretaker Questionnaire (administered to all women 15-49 years of age in each randomly selected HH, with a special section for mothers and caretakers of children 0-59 months); and an Anthropometric Questionnaire, to weigh and measure children under five. Appendix 8.A includes all final evaluation tools. 1.2 Staffing and Training Household surveys were administered by PROESSECAL, a local consulting company with experience leading large evaluations; experienced community health volunteers collected anthropometric data. Appendix 4A lists the members of FQS teams. Prior to field work, the SALOHI Monitoring and Evaluation team and the local consulting firm organized a two-week training program (Appendix 6) for field enumerators, that included a review of the SALOHI program’s goals and objectives, a review of program activities and outputs, a review of key food security concepts, a review of data collection tools, role plays, and field practice CRS/MG SALOHI MYAP FE/USAID-FFP-2-00002 5 to test data collection tools and iPad touch data entry protocols. Appendix 7 contains the complete Interviewer Guide developed for this survey. 1.3 Data Collection Period and Processes In consultation with the SALOHI team, the local consulting firm developed a data collection plan that took into account the number of teams, the number of households reachable per day, and the subdivision of survey zones according to accessibility. Data was collected during the month of September, two weeks earlier than the 2009 baseline but still during the lean season. Please see Appendix 3 for the detailed quantitative data collection plan. A number of measures were put in place to ensure data quality: - Only experienced and highly recommended data collectors who had demonstrated health and/or agriculture experience and prior experience conducting large scale surveys were hired. - They were well-trained and had practice using the data collection tools in the field. - Data collection forms were designed to be simple and as concise as possible; where feasible, they included pre-populated responses. - Field supervisors used a standard checklist to ensure completeness and accuracy of data collected and to oversee the integrity of the interview process during field work. 1.4 Data Entry and Analysis Data collection was automated using iPad touch and iPad technology, and iFormBuilder. CAPSULE, a local firm, produced requested data tables using SPSS version 19. ENA and WHO ANTRHO were used to evaluate anthropometric data. Anthropometric data is presented using both NCHS and WHO growth standards, to facilitate comparisons over time and between countries. It should be noted that iform builder provided segmented data forms, which were reconstituted into a consolidated data set for each questionnaire (three data sets – one for head of household, one for reproductive aged women and women with children under five, and one with anthropometric data). The data cleaning and consolidation process is described more thoroughly in Annex 11. 1.5 Data Limitations Like all surveys, there are limits to how the data in this report should and can be used. First, the data was not collected as part of a randomized control trial. There was no attempt to collect information from communities where the program was not implemented. There is no “control” population (just a “before” and “after” population). The approaches used by the SALOHI program have been subjected to years of research on station and on farm (for agriculture), were piloted in past DAPs, and the efficacy of the health and nutrition approaches used has been widely studied and published, most recently in the Lancet (http://thousanddays.cmail2.com/t/r-l-bljdrjt-jhilpikjy-e/). However, attempts have been made to control for major external variables, including weather, economic factors (changes in household livelihoods and food prices), demographic changes and economic changes which could have contributed to improvements (or decreases) in household food security, in addition to program interventions. In general, 2008 was an excellent year in terms of rainfall and agricultural production, and 2013 was an “ok” year, with pockets of food insecurity. The economic situation in Madagascar has deteriorated since the coup in 2009, and inflation was particularly high in 2013 (FAO 2013; FEWSNET 2013). The food security situation generally deteriorated in Madagascar from 2009 to 2014. Security has worsened, especially in the South. CRS/MG SALOHI MYAP FE/USAID-FFP-2-00002 6 Other development programs also operated in the SALOHI zones. Beneficiaries were asked to what extent other programs contributed to the adoption of promoted behaviors, and data was analyzed to determine any significant differences in SALOHI zones where other development actors were present, to measure possible effects from program synergies on improvements in local food security. The Malagasy language varies considerably by region and ethnic group, so some questions may not have been correctly understood by beneficiaries, or their responses may not have been correctly interpreted by data collectors. However, attempts were made to translate the questionnaire into local dialects, and the questionnaire was translated from English to Malagasy and back to English, and then into French for sharing with other development partners in Madagascar. Questions were also triangulated and pre coded responses used where possible to reduce data collection errors. All responses were based on beneficiary recall, and thus subject to errors of recollection and interpretation. Finally, data collected cannot be extrapolated to non SALOHI communities, communes, districts or regions in Madagascar. The data only pertains to areas where the SALOHI program intervened. Data was not collected from non SALOHI communities in the same communes where the SALOHI program intervened, or from communes or districts in SALOHI regions, where the program did not intervene. 2.Qualitative Study 2.1 Staffing and Training A six-person team composed of five external evaluators (three international and two national) and the SALOHI Livelihoods Coordinator conducted a qualitative study in October-November 2013. (For a list of the team members, see Appendix 4.B.). The purpose of this exercise was to facilitate quantitative data analysis and the interpretation of final results, to provide in-depth answers to evaluation questions, and to develop final evaluation recommendations. 2.2 Description of Process: Team, Sites, Tools Over a four-week period the team visited 14 SALOHI fokontany that had also participated in the FQS. Selection criteria included: 1) communities in each NGO zone (ADRA, CARE, CRS, and Land O’Lakes), 2) communities in each geographic zone (central highlands, east, southeast 1 and 2, south), 3) accessibility by road during the evaluation period, and 4) varying levels of performance (strong communities, weak communities and “average” performing communities). The levels of performance were determined by SALOHI staff in each zone and verified by PCU members using data from the program M&E system. An effort was made to include sites where other health and nutrition projects had also been active (in particular RANO HP from 2010-2013 and SantéNet2 from 2008-2013). The following list identifies fokontany visited, the implementing NGO and whether RANO HP and/or SantéNet2 operated in the same community: Center Zone (ADRA) 1. Ivolo Firaisana (ADRA): Good performance; SantéNet2 2. Ankarinarivo (ADRA): Weak to average performance, depending on the SO; SantéNet2 South Zone (CARE and CRS) 3. Tsimanakiaraky (CRS): Good performance; SantéNet2 4. Marofoty (CRS): Weak to average performance, depending on the SO; SantéNet2 5. Helibondro (CARE): Good performance; SantéNet2 and RANO HP CRS/MG SALOHI MYAP FE/USAID-FFP-2-00002 7 6. Berano Ville (CARE): Weak to average performance, depending on the SO; SantéNet2 and RANO HP Eastern Zone (CARE and CRS/FITEA) 7. Sahabe (CARE): Good performance; SantéNet2 and RANO HP 8. Ambodivandrika (CARE) Weak to average performance, depending on the SO; SantéNet2 and RANO HP Southeast 1 Zone (CRS/FITEA and BDEM, ADRA) 9. Ambodifandramanana (CRS): Average performance ; SantéNet2 10. Ankarimalaza (CRS): Good performance; SantéNet2 Southeast 2 Zone (Land O’Lakes) 11. Beanana (LOL): Good performance; SantéNet2 and RANO HP 12. Mideboka (LOL): Average performance; SantéNet2 and RANO HP 13. Anosivelo (LOL): Average performance 14. Manambotra Center (LOL): Good performance; RANO HP At each site the evaluation team used focus group discussion (FGD) guides and key informant interview (KII) tools for each SO and for cross-cutting themes. Additional guides were used for meetings with non-participants, for visual observations, and for discussions with village leaders and authorities. SO1 FGD groups included women participating in SO1 activities; CHVs, head nurses at health centers, and model mothers provided information for the KII. For SO2 the FGD included approximately six men and six women participating in SO2 activities and KII were directed toward Farmer Leaders and the Village Agents responsible for Village Savings and Loan Associations. The SO2 team also observed FFS sites and other agricultural and agroforestry activity sites. For SO3 the focus group included an average of six men and six women participating in SO3 activities. Key informants were primarily committee members who manage community infrastructures. The SO3 team also visited infrastructure including roads, canals and dams at each of the 14 sites. Questions for the cross-cutting themes of Gender and Environment were incorporated into the FGDs for the three SOs. (See Appendix 8.B. for qualitative tools used and the detailed data collection process.) The quantity of data gathered at the 14 sites did not require coding. All members of the team visited the same fokontany each day and spent the evening comparing findings for the community, checking facts and compiling a list of questions for follow-up with either a document search or an interview with a SALOHI staff person. Data collected in the field was triangulated with other sources, as the team: - conducted extensive individual interviews with SALOHI staff and local stakeholders; - carried out a thorough document review; - analyzed reports for common trends; and - visited two Social Protection Centers and interviewed staff and SALOHI participants. 2.3 Qualitative data limitations Although sites surveyed met selection criteria and were varied in terms of performance and geographical context, the team was only able to visit 14 out of a possible 592 fokontany. It is therefore not possible to generalize qualitative findings too broadly. Second, the sites were selected by SALOHI field staff in conjunction with one Qualitative Study team member and there may be some inherent bias in the selection process. Third, the quality of the translators at each site varied widely and some information may not have been captured or poorly translated. CRS/MG SALOHI MYAP FE/USAID-FFP-2-00002 8 3. Data Dissemination The SALOHI team has a number of plans for sharing the results of this final assessment including a national dissemination workshop for SALOHI partners, external stakeholders and SALOHI staff; seven regional workshops for stakeholders and beneficiaries, to be held in Malagasy; “town hall” meetings at the local level; press releases; and a summary booklet to be published in English, French and Malagasy. At the international level results will be shared through the DEC and TOPS websites and during international workshops. 6 D. Findings and Analysis 1. Demographics and Household Characteristics The purpose of this section is to summarize the socio-economic characteristics of the population in target zones, to facilitate comparison to results in other countries or programs. Data is presented primarily by NGO, by zone and where relevant (where important differences have been identified), by gender. Household size: The average household size is 5.6 persons, with little variation among implementing partner zones. The South has the highest number of persons per household (6.6), while the East has the fewest (5.1). These results are similar to those identified during the SALOHI baseline (5.7 people; 2009), and larger than the average household size reported in the last Demographic Health Survey (4.8 people per rural HH; 2008/2009). Children under five years: Overall, children under five years of age represented 18.3% of the population in SALOHI target zones. This is similar to the percentage of children under five reported in the last DHS (16% in rural zones; 2008/2009), but it is less than the percentage reported in the SALOHI baseline survey (24%; 2009). SALOHI households tend to be larger, less literate and more vulnerable than those in the general population. Table 1: Composition of children under five years in SALOHI households by geographic zone (FQS 2013) Final evaluation Age group 0-5 mos. 6-23 mos. 24 - 59 mos. 0 – 59 months Center 1.2% 4.7% 9.2% 15.1 South 1.9% 8.2% 12.2% 22.3 SE 1 1.9% 6.7% 10.8% 19.4 SE2 1.7% 6.4% 10.5% 18.6 East 1.5% 5.1% 6.7% 13.3 TOTAL 1.7% 6.4% 10.2% 18.3 6 Unfortunately due to time constraints (the final report has to be submitted to FFP within 90 days of data collection), input from beneficiaries and partners cannot be used to revise the final report. CRS/MG SALOHI MYAP FE/USAID-FFP-2-00002 9 Gender of head of household: The percentage of female-headed households in SALOHI target zones is 23%. This is similar to the percentage of FHH identified in the SALOHI baseline (19%) and the last DHS (22%). Age of head of household: The average age of the head of household in SALOHI zones is 43 years. This is slightly older than the average age of household heads in the baseline survey (38 years of age). Table 2: Average age of the head of household by zone and by NGO (years) (FQS 2013) Zone/NGO Center East South SE1 SE2 ADRA CARE CRS LOL TOTAL Years 46.6 43.1 43.4 38.5 45.6 42.0 42.9 42.1 45.6 42.9 Literacy of the head of household: The literacy rate of the head of household is 62.3% (based on respondent’s self-identification). However, there are some important differences in literacy rates in different geographic zones, and in NGO zones. In Land O’Lakes (LOL) zones, 45% of household heads are literate, compared to 74% in ADRA zones. In the South and Southeast2 (Atsimo Atsinanana) only .45 and 48% of household heads can read, compared to 88% in the Central highlands, and 73% in the East. These findings are similar to those reported in the SALOHI baseline survey (61% of Household Heads were literate), and in the last DHS (73% rural literacy overall). Table 3: Heads of household who can read, by zone and by NGO (%)(FQS 2013) Zone/NGO Center East South SE1 SE2 ADRA CARE CRS LOL TOTAL Percent 88.4 73.2 48.2 60.4 45.0 73.7 58.1 60.9 45.1 62.3 Marital status of the heads of household: The final evaluation found that 73% of household heads are married, 10% were divorced, and 10% widowed. Table 4: Marital status of heads of households (SALOHI Baseline 2009; FQS 2013) Baseline Final Evaluation Legally married 14.8% 12.2% Traditionally married 65.1% 61.1% Divorced 5.8% 10.3% Widowed 4.4% 10.0% Never married 9.1% 6.4% Other 0.8% - Pregnant and lactating women (PLW): 44.8% of reproductive aged women surveyed reported being pregnant or lactating during the final evaluation survey. This varied from a low of 26% in the East, to a high of 56% in the South and 52% in the Southeast1. CRS/MG SALOHI MYAP FE/USAID-FFP-2-00002 10 Table 5: Women who reported being pregnant or lactating, by zone and by NGO (%)(FQS 2013) Zone/ NGO Center East South SE1 SE2 ADRA CARE CRS LOL TOTAL Final 37 26 56 52 46 47 42 43 46 45% Household income-generating activities: Agriculture and animal husbandry are the most reported form of income generation in households. Table 6: Income-generating activities reported by heads of households (%)(SALOHI Baseline 2009; FQS 2013) Baseline Survey Final evaluation Agriculture 93.7 94.1 Animal Husbandry 68.9 75.0 Fishing 8.7 7.1 Mining 5.8 4.3 Artisanal production 18.2 23.4 Commerce 16.9 21.1 Day labor 17.9 9.4 Occasional salary 25.7 27.8 Permanent salary 5.2 6.3 Household use of revenue: Food, schooling, and clothes are the top three uses of revenue in SALOHI communities. Fifty-five percent of households use income for school purposes and 76% use income for clothing. Forty-seven percent of HH use revenue for social and cultural purposes and 24% of households use revenue for investment purposes. Table 7: Use of revenue reported by head of household (%)(SALOHI Baseline Survey 2009; FQS 2013) Baseline Survey Final evaluation Food 97.6 98.6 Schooling 47.7 54.7 Clothing 67.9 76.3 Housing 4.6 3.5 Social/cultural 35.7 47.4 Investment 30.0 23.6 Savings 9.7 8.0 Other 22.5 9.9 Household ownership of assets: Asset ownership in SALOHI zones did not change significantly between baseline and final evaluations, except in two key areas: 49% of households reported owning a radio in the final evaluation (vs. 39.7% during the baseline) and 40% of CRS/MG SALOHI MYAP FE/USAID-FFP-2-00002 11 households reported owning cattle in the final evaluation (vs. 32% in the baseline evaluation). Both of these changes are statistically significant. Table 8: Ownership of assets reported by head of household (%)(SALOHI Baseline 2009; FQS 2013) Baseline Survey Final evaluation Land 92.2 92.4 Housing 94.4 98.2 Television 2.3 3.1 Radio 39.7 49.4 Sewing machine 9.6 7.0 Bicycle 11.7 14.5 Motorcycle 0.3 0.7 Ox cart 8.5 8.6 Plow 9.9 9.1 Canoe 6.5 5.1 Fish net 6.3 4.5 Cattle 32.3 40.1 Pigs 22.0 20.8 Sheep 11.7 13.4 Poultry 71.0 71.5 Average number of assets/ HH 4.2 4.3 2. SO1 - Health and Nutrition 2.1. Brief Description of Interventions The objective of SO1, improving the nutritional status and overall health of women and young children, is at the heart of the SALOHI program. One of the most striking characteristics of SO1 activities is the community approach: Everyone is encouraged to participate, not just parents of young children. The idea that the entire community has a vested interest in and responsibility for healthy children is a common thread in SO1 interventions. Mothers, fathers, community leaders, volunteers, staff at the nearby CSB – each group has its part to play in creating and maintaining a healthy environment for children as well as for pregnant and lactating women (PLW). The combined efforts of these groups mutually reinforced SALOHI’s messages and help to creative a supportive environment for behavior change. SO1 focuses on two strategies proven to be effective in improving nutritional outcomes: adopting positive nutrition practices and preventing diseases that undermine nutrition. The strategies, each with its set of interventions/activities, reinforce each other. Certain activities such as behavior change communication (BCC) and the contribution of community volunteers are common to both strategies. The principal interventions for these strategies are: CRS/MG SALOHI MYAP FE/USAID-FFP-2-00002 12 ENA: Essential Nutrition Actions are an internationally-accepted set of nutrition actions that have proven effective in promoting optimal health and nutrition for women and young children. SALOHI emphasizes: breastfeeding (immediate breastfeeding within one hour of birth and exclusive breastfeeding for children 0-5 months); complementary feeding for children 6-23 months with continued breastfeeding; appropriate nutritional care for sick or severely malnourished children; and prevention of micronutrient deficiencies and anemia. The Integrated Management of Childhood Illnesses at the community level (C-IMCI): Preventing childhood diseases is a major SO1 objective. A comprehensive C-IMCI approach would include three components: Improving the case management skills of health workers; improving the health system to deliver IMCI services; and improving household and community practices. SALOHI focused on the third component with a particular emphasis on two key sets of behaviors and practices: personal hygiene, especially hand washing at critical moments, and food hygiene. Given the link between diarrheal disease and nutritional status, this focus was appropriate. Behavior Change Communication (BCC): BCC is an essential component of the entire SALOHI program, cutting across all three SOs. Promoting BCC is the responsibility of several groups: SALOHI field agents; the community volunteers for each SO and committee members, especially for SO3 structures; and local leaders and authorities in SALOHI communities. Training and support of Community Health Volunteers (CHVs): Essential to the successful results of SO1 to date are the CHVs, dedicated men and women who volunteer their time and talents to improving the health of their communities, especially children’s health. SALOHI implements its activities through new and existing CHVs, who are elected or designated by their community. Some CHVs were already in place prior to SALOHI, working through the local health center (CSB) or with other health and nutrition partners such as the National Nutrition Office (ONN). Others were elected more recently since one of SALOHI’s goals was to expand the reach of health and nutrition actions by increasing the number of CHVs to an average of eight per fokontany. Growth Monitoring and Promotion (GMP): The GMP sessions, held on a monthly basis, are designed to help families monitor their children’s health so that timely interventions can be initiated if a child is faltering. Monthly sessions include the promotion of behaviors, especially ENA, which can give children under five the best start in life: breastfeeding, young child feeding, hygiene practices, and childhood disease management and prevention. PD/Hearth: The PD/Hearth approach is used to rehabilitate moderately malnourished children, usually those identified during GMP sessions, and to help prevent future malnutrition. PD/Hearth begins with a positive deviance inquiry to identify the dietary practices of women in the community with well-nourished children. From these positive deviant mothers Model Mothers are chosen to work with CHVs to promote these good practices to caretakers of moderately malnourished children. Support groups for pregnant and lactating women (SAMBAIKA): These support groups, composed of PLW and model mothers, use participatory capacity building techniques to promote healthy pregnancies, reduce low birth weights, and ensure that children have an excellent start in life. CRS/MG SALOHI MYAP FE/USAID-FFP-2-00002 13 2.2. Overall Results Table 9: SO1 Key indicators and achievements INDICATORS BASELINE STUDY FINAL EVALUATION Target Value Confidence Interval7 Value Confidence Interval8 SO 1: IMPROVING THE NUTRITIONAL AND HEALTH STATUS OF CHILDREN 0-59 MONTHS IN 96,000 Impact indicators Percentage of stunted (HAZ<-2) children 0-59 months / NCHS Standards T : 41% M : 44% F : 39% [95% CI: 39.4% – 42.6%] T : 36% M : 37% F : 36% [95% CI: 33.8% – 39.0%]*** -8 % points 33% Percentage of stunted (HAZ<-2) children 0-59 months / WHO Standards T : 47% M : 51% F : 44% [95% CI: 45% – 50%] T : 41% M : 43% F : 40% [95% CI: 39% – 44%]*** 39% Percentage of underweight (WAZ<-2) children 0-59 months / NCHS Standards T : 34.5% M : 36% F : 33% [95% CI: 31.8% – 37.2%] T : 26% M : 26% F : 26% [95% CI: 23.7% – 28%]*** -10% points 24.5% Percentage of underweight (WAZ<-2) children 0-59 months / WHO Standards T : 29% M : 31% F : 27% [95% CI: 26% – 32%] T : 20% M : 22% F : 19% [95% CI: 18% – 22%]*** 19% Outcome indicators Percentage of children 0-5 months of age who were exclusively breastfed 56% MHH: 54% FHH: 62% [95% CI: 48% – 63%] 75% MHH: 76% FHH: 74% [95% CI: 69% – 79%]*** + 20% points 76% Percentage of children 0-5 months of age breastfed within one hour of birth 71% MHH:70,2% FHH:72,7% [95% CI: 65% – 77%] 81% MHH:82% FHH:80% [95% CI: 78% – 83%]*** +10% points 81% Percentage of caregivers demonstrating/reporting proper personal hygiene behaviors (hand washing at critical moments) 20% MHH:20,8% FHH: 20% [95% CI: 19% – 22%] 38% MHH:41% FHH:31% [95% CI: 36% – 40%]*** +25% points 45% Percentage of caregivers demonstrating/reporting proper food hygiene behaviors (preparation of food, cooking, food storage) 44,% MHH:45% FHH: 41% [95% CI: 42% – 47%] 66% MHH:66% FHH:63% [95% CI: 65% – 68%]*** +25% points 70% ***observed change is statistically significant (P<=0.0001) The two impact indicators for children 0-59 months of age [stunting (height/age < - 2 SD) and underweight (weight/age < - 2 SD)] improved between 2009 and 2013. Figure 2 shows the decrease in these malnutrition rates, with a comparison to LOA targets (using both NCHS and WHO growth standards). 7 Confidence Interval is only applicable for the total value. 8 Idem CRS/MG SALOHI MYAP FE/USAID-FFP-2-00002 14 Figure 2: Stunting and Underweight values in SALOHI zones, 2009 – 2013 (Baseline vs. Final Evaluation)9 Although stunting declined slightly less than the LOA target, this decrease still represents an important achievement (a decrease of 13%). Given that there are few nutrition programs in SALOHI zones10 and that local health centers rarely have the staff or resources for preventive and curative nutrition services, it is likely that the combination of SALOHI activities, including SO2 agricultural interventions, FFA activities, increased community resilience, and MCH rations, have positively affected children’s nutritional status. Data from other surveys (DHS 2008/09, CFSVA 2010 and ENSOMD 2012/13) do not show similar national level improvements over the same time period. The CFSVA 2010 estimated stunting at 48.7% nationally, and underweight at 27.9% of children under five (using WHO growth standards; similar to SALOHI baseline values of 47% and 29%), whereas the most recent national level survey (ENSOMD 2012/13) estimated stunting at 47.3% nationally (no change), and underweight at 32.4% (a deterioration from 2010). Another important indicator that shows a steep decline since the start of the SALOHI program is the “% of children sick in the two weeks prior to the survey”. This rate declined from 44% at baseline to 31% in the FQS. The decline in malnutrition rates and the 14% decline in the incidence of illness are matched by corresponding improvements in all SO indicators, where most targets were met or surpassed. Interventions that contributed to these results are organized around two IRs: 1) Maternal and Child Nutrition and 2) Integrated Management of Childhood Illnesses at the Community level (C-IMCI), focusing on disease prevention; each is discussed in detail below. Note that some aspects of SO1 9 NCHS standards are included as the Government of Madagascar continues to use them. Detailed information on the differences between NCHS standards and WHO growth monitoring standards can be found on the internet (www.childinfo.org/files/Comparison_implications.pdf and http://www.cdc.gov/mmwr/preview/mmwrhtml/mm5821a4.html). 10 SECCALINE promotes growth monitoring in SALOHI zones, and GRET collaborates with SALOHI communities in the South. 0% 5% 10% 15% 20% 25% 30% 35% 40% 45% 50% NCHS WHO NCHS WHO Stunting Underweight 41% 47% 35% 29% 33% 39% 25% 19% 36% 41% 26% 20% Baseline Target Final CRS/MG SALOHI MYAP FE/USAID-FFP-2-00002 15 such as the work of CHVs, the integration of health messages into SO2 and SO3, and the emphasis on involving fathers and community leaders across both IRs. 2.3. Maternal and Child Nutrition (IR1) (96,000 households adopt recommended maternal and child nutrition practices.) During the Qualitative Study Focus Group Discussions (FGDs) and Key Informant Interviews (KIIs), women routinely discussed the importance of immediate breastfeeding (IBF), exclusive breastfeeding (EBF), and an adequate and diverse diet for young children starting at six months of age. They recited recipes, knew which foods were high in micronutrients such as Vitamin A and iron, and described the positive physical changes they had seen in their children as a result of the SALOHI program. Using Immediate and Exclusive Breastfeeding (IBF and EBF) rates as examples, the Qualitative Study team found that the FQS data corroborated what they heard in the field about the adoption of maternal and child nutrition practices. Even though the IBF baseline rate was already relatively high at 71%, Figure 3 shows that almost every zone and each NGO made progress, with the exception of ADRA zones and the SE1 (Mananjary, Nosy Varika, where ADRA works, along with BDEM and FITEA). The South East 2 (LOL zones, Manakara and Farafangana) showed the largest increase in practicing IBF. Figure 3: Immediate Breastfeeding in SALOHI zones, 2009 – 2013 (Target 81%) Improvements in the practice of exclusive breastfeeding (EBF) were quite dramatic in some areas, especially in the South where it increased from 26% in the baseline to 75% in the FQS, and in the East where it increased 34%. Overall, the East had the highest rates of exclusive breastfeeding (89%) in the final evaluation (whereas it was the South East during the baseline survey, when SE1 and SE2 were combined in one geographic zone). Improvements were made across all project except the SE2 zone (Manakara and Farafangana, LOL zones). Although the Qualitative Study team 0.0% 10.0% 20.0% 30.0% 40.0% 50.0% 60.0% 70.0% 80.0% 90.0% 100.0% Baseline Final CRS/MG SALOHI MYAP FE/USAID-FFP-2-00002 16 conducted inquiries during field work with SALOHI staff, beneficiaries and health personnel, there was no clear explanation for this poor performance, especially as knowledge levels were high among the women interviewed. It should be noted that the number of samples for exclusive breastfeeding is quite small (292 children under six months of age), so differences between NGO and geographic zones are not statistically valid, and are presented for illustrative purposes only. Figure 4: Practice of Exclusive Breastfeeding in SALOHI zones (LOA Target = 76%) The principal SALOHI interventions that could have contributed to these positive changes in maternal and child nutrition practices include GMP, PD/Hearth and SAMBAIKA. The strengths and weaknesses as well as the overall impact of each intervention are summarized below. GMP: Prior to the implementation of the SALOHI program, the baseline survey showed that only 55% of children under two had participated in GMP sessions. The SO1 team made GMP a high priority, seeing it as 1) a cost-effective way to help parents monitor their children’s health, 2) an ideal vehicle for the dissemination of health and nutrition messages, and 3) a means to identify malnourished children for PD Hearth programs and to recruit PLW for SAMBAIKA groups. As Figure 4 shows, the SALOHI team succeeded: the LOA target of 75% was exceeded by 17.5 percentage points. 64% 48% 53% 59% 58% 55% 26% 70% 59% 56% 79% 78% 63% 52% 79% 89% 76% 64% 52% 69% 0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100% ADRA CARE CRS LOL CENTER EAST SOUTH SE1 SE2 SALOHI Baseline Final CRS/MG SALOHI MYAP FE/USAID-FFP-2-00002 17 Figure 5: Increase in GMP participation in SALOHI zones, 2009 – 2013 (Target = 75%) One of the more interesting SO1 results is not only the increase in GMP participation, but the way communities embraced this activity, as reported during FGD. It is, for example, the activity that fathers are most likely to be involved in, bringing the child to monthly session when the mother cannot. It is also the SO1 activity the Qualitative Study respondents mentioned most often as one they will continue after the project ends because it is relatively easy to do, requires no large investment of resources other than time, and provides visible results as parents can immediately see their child’s weight gain or loss. Other reasons that help explain the success of this activity are the fact that it is participatory, provides an opportunity for social interaction and allows fathers to be directly involved in children’s health. Women whose children participated in GMP said they received advice on hand washing with soap (67%), complimentary feeding (66%), food conservation and processing (54%), conservation and treatment of drinking water (46%) and care of the sick child (40%), as well malaria prevention (31%) and foods rich in iron and vitamin A (24%). Most of them reported they had applied at least one of the practices they had learned at home (60%). Promising practices: The main factor contributing to the success of this activity was decentralization of GMP sessions to the hamlet level (parents and children didn’t have to travel so far to participate). Although the national standard is to have at least two CHVs per Fokontany, the SALOHI program trained up to 10 CHVs per Fokontany, to facilitate access to essential services in remote locations, and in widely dispersed communities. SALOHI also trained both men and women CHVs, to facilitate communication with both parents, and to reduce individual CHV workload. SAMBAIKA: These support groups for pregnant and lactating women are designed to improve women’s health and to give children a healthy start in life, even before birth. As with GMP, the effort paid off: in the FQS 58% of mothers of children under 2 had participated in SAMBAIKA, exceeding the target of 50%. According to the FQS, the topics they recalled most were infant and young child feeding (IYCF) (92% of respondents), hygiene (75%) and key C-IMCI practices (44%). The main reason for not participating in SAMBAIKA groups (given by those who were not members) was lack of time (57%). 51% 52% 52% 52% 70% 55% 95% 93% 89% 91% 94% 93% 0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100% CENTER SE1 EAST Increase in GMP Participation BLS FQS CRS/MG SALOHI MYAP FE/USAID-FFP-2-00002 18 Although there was no difference in the number of prenatal consultations for women who participated in SAMBAIKA compared to those who didn’t, SAMBAIKA participants had much higher levels of knowledge about micronutrients: 62% of SAMBAIKA participants could cite foods rich in Vitamin A compared to 38% for non-participants and 66% knew about foods rich in iron compared to 37% of those who did not attend sessions (all of these differences are statistically significant at p=.001). Promising practices: Among SAMBAIKA best practices, three are promising practices that could enhance sustainability: 1. Using traditional birth attendants (matrones) in SAMBAIKA encourages participation as they are trusted and usually the first outside the immediate family to know about a new pregnancy. 2. Combining a SAMBAIKA session with the prenatal consultation at the CSB or with GMP sessions facilitates participation. 3. Working with the CSB to conduct prenatal consultations at the community level ensures that more women receive the minimum number of consultations during pregnancy. PD/Hearth: As noted earlier, nutrition services for rehabilitating malnourished children are limited in the SALOHI program area and in Madagascar as a whole. PD/Hearth helped fill the gap, providing an effective way to rehabilitate children at the community level where the whole community could play a role, especially caretakers, CHVs and Model Mothers. Although some children attended the 12-day sessions more than once, the intervention overall seems to have been effective, with 93% of children maintaining their health status according to caretakers interviewed during the FQS. The PD/Hearth sessions were also an excellent vehicle for promoting behavior change. When asked what they had learned during these sessions, mothers and caretakers especially cited IYCF (92%) and hygiene practices (75%). 34% of children under five included in the survey participated in PD Hearth sessions, and 78% of them participated in all 12 days of the program. The main reason for not completing all 12 days of the program was lack of time of the mother (38%), followed by lack of money (25%) and lack of ingredients (13%). Participation in PD Hearth (PDH) (SALOHI FQS 2013) % of mothers/ caretakers of children under 5 Participated in PD hearth sessions 34% Participated in all 12 days of PD Hearth sessions 78% of those who participated in PDH Reasons for not completing all 12 days Lack of time 38% of those who participated in PDH Lack of money 25% of those who participated in PDH Stopped because the nutritional status of child improved 16% Lack of ingredients 13% of those who participated in PDH Type of advice received during PDH sessions Infant and young child feeding practices 90% Hygiene 75% IMCI/ Disease prevention 41% Care of the sick child 39% Care and affection for intellectual development of the child 30% Followed advice received 84% of those who participated in PDH CRS/MG SALOHI MYAP FE/USAID-FFP-2-00002 19 Reasons for not following advice Lack of money 74% of those who did not follow advice Lack of time 43% of those who did not follow advice Percentage of mothers who said their child’s nutritional status improved following participation in PDH sessions 93% Percent of mothers who said their child’s nutritional status has been maintained at a satisfactory level since the PDH 93% One of the ongoing issues with the PD/Hearth intervention is women maintaining that their HHs did not always produce the necessary ingredients suggested in the recipes or have the money to buy them. Following recommendations from the midterm evaluation, positive deviant inquiries (PDI) were conducted with model mothers, and were documented and included in field staff reports. However, even with this step to determine how women with varying levels of resources were able to feed their young children well, findings were not always widely disseminated to mothers participating in PD/Hearth. A second concern is that in some cases, NGO staff, CHVs and community members provided ingredients to women who did not have food for their PD Hearth session. (Food aid was not used to support PD Hearth sessions). In spite of these concerns, it is likely that PD/Hearth contributed to the reduction in malnutrition rates since because there are few options for recuperating malnourished children in project zones. In any case, learning about portion size (91%), active feeding (30%) and hygiene practices (75%) during PD/Hearth sessions is valuable in and of itself. Promising practices: One of the best practices that increased participation in PD/Hearth was organizing sessions at the hamlet level, which improved both targeting and reach and likely contributed to the reduction in malnutrition rates since more children living in more isolated sites were reached (based on focus group discussions and key informant discussions with CHVs, model mothers and health staff). Recommendations: The current IYCF approach emphasizes prevention rather than a focus on rehabilitation, and the trend is toward a more inclusive intervention, such as combining PD/Hearth with Care Groups. In line with USAID/FFP’s emphasis on prevention, in future programs NGOs could promote PD/Hearth sessions as learning sessions for all mothers with children under two, holding sessions once or twice a month for a year. Another option is to organize PD/Hearth sessions for five days in a row for all mothers, followed by daily home visits to malnourished children for the next seven days to help mothers prepare food and rehabilitate malnourished children at home. Observations concerning adoption of maternal and child nutrition practices: The FQS did not show a statistically significant correlation between participation in GMP, PDH, SAMBAIKA or HH visits and immediate breastfeeding. However, there are statistical correlations between exclusive breastfeeding and participation in GMP (p=.01), PDH (p=0.05), SAMBAIKA groups (p=.002), and household visits (p=.002). During the Qualitative Study it was clear that women and a large proportion of men knew key ENA messages. 2.4. C-IMCI: Disease Prevention Practices (IR2) (96,000 households adopt recommended disease prevention practices.) Two of the key messages for disease prevention are hand washing with soap (proper personal hygiene) and food hygiene. Figure 5 below shows the extent to which these practices were adopted, with the more dramatic improvement in food hygiene practices which went from 44% at CRS/MG SALOHI MYAP FE/USAID-FFP-2-00002 20 baseline to 66% at the FQS, slightly less than the LOA target of 69%. Improvements in hand washing with soap are also important (17 percentage points, or an 81% increase), especially since it is typically more difficult to effect change for this practice. Figure 6: Changes in Hygiene Practices in SALOHI zones (2009 – 2013) Recommended personal hygiene behavior in the SALOHI program includes hand washing with soap at critical moments, such as: before cooking food; before eating; after using the toilet; before giving food to children; and after washing children who have used the toilet. A minimum of three of these criteria is required to meet the standard. Recommended food hygiene behaviors include: using potable water or treated drinking water; using clean, appropriately stored cooking utensils; adequate storage of leftovers; and reheating food before eating it. Adopting a minimum of three, including using potable or treated drinking water, is required to meet the standard of “proper food hygiene practices”. From Figure 5 above, it is obvious that hand washing presented more challenges for adoption than food hygiene. This is not surprising as food hygiene actions are relatively easy to perform, with the main challenge being clean drinking water. But regular hand washing at critical moments in any context is not an easy behavior to promote. However, the percentage of respondents who reported never washing their hands with soap decreased from 68% at baseline to 19% at final evaluation. There is no statistical correlation between participation in growth monitoring or PDH and the adoption of promoted food hygiene behaviors. However, there is a correlation between participation in SAMBAIKA groups or receiving home visits and adoption of promoted food hygiene behaviors (p=.001). There is also a correlation between participating in all four health activities and the adoption of promoted food hygiene behaviors (p=.05). There is a statistical correlation between the adoption of promoted hand washing behaviors and participation in growth monitoring and promotion (p=.000), PDH (p=001), SAMBAIKA (p=.001), and household visits (p=.000). In addition to hand washing and food hygiene, participation in SALOHI activities may have had a positive impact on other disease prevention behaviors, for example, the percentage of mothers 0% 10% 20% 30% 40% 50% 60% 70% Personal hygiene practices (handwashing) Food hygiene practices 21% 44% 46% 69% 38% 66% Baseline Target Final Evaluation CRS/MG SALOHI MYAP FE/USAID-FFP-2-00002 21 of children under five who reported that their child slept under a mosquito net the night before the survey rose from 61% at baseline (SALOHI Baseline Survey 2009) to 81% (SALOHI Final Evaluation 2013). Success Factors for Both Intermediate Results: CHVs: Critical to the success of both disease prevention interventions and the adoption of ENAs are CHVs. In addition to supporting Model Mothers and helping to organize SAMBAIKA groups, GMP and PD/Hearth sessions, CHVs routinely carry out home visits to households with young children and pregnant women. These visits allow for systematic follow-up (when a child’s growth is faltering for example, or 6 – 12 weeks after PD hearth sessions) and often include fathers and other family members. Proud CHVs in uniform in Ankarinarivo Imady To effectively reach more households, SALOHI set a goal of identifying and training an average of eight CHVs per fokontany rather than the usual two. SALOHI devoted considerable effort to training and supporting the additional CHVs, who were instrumental in the implementation of GMP and household visits and the achievements described in the preceding sections. During the FQS, 99% of SALOHI beneficiaries reported receiving a home visit from a CHV, with 63% reporting a monthly visit and 21% a weekly visit. The ongoing involvement of CHVs in community health promotion after the SALOHI program ends is important to continue the promotion of behaviors that have led to improved health outcomes, but motivation can be a factor in whether CHVs continue to provide community health services. Those CHVs who are formally attached to a health center and/or to another project such as ONN have an advantage as they may be more likely to have the necessary resources, support and motivation to continue. During the Qualitative Study, the team encountered CHVs who were not formally attached to a CSB or a project, yet they had worked for years on a purely voluntary basis; this is promising for continued CHV activity post-SALOHI. Another promising finding from the Qualitative Study is that all the CHVs interviewed stated that they planned to continue their activities after the project ends, with the majority of them citing as motivation their desire to contribute to the development of their community. As one CHV stated: “SALOHI does not pay me yet I do the work. Why wouldn’t I continue since the project ending does not change anything for me?” Involvement of men and local leaders: As part of its gender approach, SALOHI made a concerted effort to involve fathers more in the health concerns of women and children. This effort was partially successful although there was no statistically significant correlation between a father being involved and a child’s nutritional status. Women interviewed during the Qualitative Study did express their appreciation for their husbands’ involvement and stated that those fathers now participating are likely to continue their support. For a detailed discussion of fathers’ involvement and the impact on children’s health, see the Gender section under E. Cross￾cutting Themes. CRS/MG SALOHI MYAP FE/USAID-FFP-2-00002 22 As for support from local leaders and authorities, it was clear during the Qualitative Study that at a minimum the tacit support of leaders – elected and traditional – is essential for a project to succeed. In the 14 fokontany visited, it was clear that local leaders knew all SALOHI activities, appreciated the positive changes they had observed, and often were directly involved themselves as problem-solvers, as animators and as participants. Impact of other health and nutrition programs: In certain SALOHI zones, other health and nutrition projects have likely contributed to increases in knowledge and adoption of health and nutrition behaviors. These include 1) two water projects, RANO-HP (CRS and CARE are among the implementing partners, with CRS as consortium lead; 23 communes overlapping) and Ranon’Ala (CRS is again the consortium lead, with one commune overlapping) and 2) ONN’s SEECALINE program, which is present throughout much of Madagascar. In addition, SanteNet2 targeted 85% of SALOHI communes (CRS and CARE were implementing partners). When FQS respondents were asked what health and nutrition practices had been adopted with the help of these other projects, the most frequent response was GMP (61% of respondents). This is not surprising since the SEECALINE program’s principal activity is growth monitoring and the program has activities throughout the country. Participants mentioned this program more often than the others: 64% of them knew about SEECALINE compared to 20% for RANO-HP11 . Other practices adopted with the help of other projects are very much in line with SALOHI’s emphasis on prevention of disease through improved hygiene: drinking water hygiene (cited by 54%); hand washing with soap and water (41%); prevention of malaria and diarrhea (39%); and food hygiene (33%). The following table shows that in communities with SALOHI activities and one of these projects, there was often a statistically significant positive effect on the adoption of good food and personal hygiene behaviors but a mixed effect on stunting and underweight rates: Table 10: Adoption of Hygiene behaviors and malnutrition levels in SALOHI zones which overlap with other USAID funded projects with hygiene components (FQS 2013) Indicator Only SALOHI With RANO Project With SantéNet2 Project Acceptable food hygiene (p value = .000) 63% 69% 85% Acceptable personal hygiene (p value = .015) 36% 41% 42% Percentage of children who were not stunted (p value = .018) 64% 52% 63% Percentage of children who were not underweight (p value = .000) 76% 58% 84% Weaknesses That Apply to Both Intermediate Results Encouraging CHVs: As with many programs that rely on volunteers, finding ways to encourage CHVs to provide essential prevention services through some compensation mechanism is a 11 SALOHI and RANO HP overlapped in 20% of SALOHI communes. SALOHI and SantéNet2 overalapped in 78% of SALOHI communes. RANO N’ALA overlapped in 10% of SALOHI communes. CRS/MG SALOHI MYAP FE/USAID-FFP-2-00002 23 recurrent issue. In Madagascar, this compensation can be in the form of regular tangible support including cash or in-kind contributions from the project or the community. CHVs very much appreciated the hats, bags, cloth wrappers (pagnes) and T-shirts provided by SALOHI, items that identified them as CHVs. Recognition ceremonies were also appreciated. However, these gestures are short-term and in the long run, only the community can provide this kind of ongoing encouragement and recognition. Of the CHVs interviewed in 14 fokontany visited, only one received regular in-kind contributions from her community in the form of rice. Having tools and materials provided by SALOHI is a source of pride for the CHVs, gives them credibility, and provides encouragement. But the distribution of materials has been uneven. In some cases 6-8 CHVs share one set of communication materials, which means that in many situations, CHVs do not have visual aids for group meetings and home visits. Quality of SBCC Strategy: Although there is a Communication Strategy, which covers both external and internal communications as well as behavior change communication, each NGO approached behavior change communication from its own organizational perspective; a consistent strategy was not in evidence in any of the zones visited. There was relatively little variety in the communication methods and channels selected (pagnes, radio, posters, fairs and flip charts/ pagi volts). 2.5. Conclusion Relevance: The evaluation team determined that the SO1 approach and actions are highly relevant and responsive to the needs of the SALOHI communities. First and foremost, improving children’s health responds to one of the most pressing concerns of mothers who do not hesitate to devote the necessary time and resources to ensure that their children are healthy when they see immediate results. Second, the lack of community-level health services coupled with the sub￾optimal functioning and shortage of staff at many CSBs translate into a compelling need for prevention and for community-based solutions to common health problems, including the treatment of moderate malnutrition in children. A third way that SO1 demonstrates responsiveness to beneficiaries’ needs is the fact that most of the health, nutrition and hygiene actions promoted by SALOHI require few resources to implement and more importantly, adoption of new practices is relatively low-risk. And finally, the relevance of the SALOHI approach for health and nutrition is evident in that it closely follows government policies and protocols and is complementary to other health initiatives in the project area. Impact: As discussed above, the FQS data showed excellent results for almost all the principal SO1 indicators, results that were validated by the findings from the Qualitative Study. There are a number of factors that have contributed to these results. First, the SO1 strategy is based on a community approach, the goal being to have everyone take responsibility for the well-being of the community’s children: mothers, fathers, CHVs, community leaders, and local authorities. During the qualitative study, it was remarkable how many of the community leaders cited first “Our children are healthier” when asked what changes they had observed. Second, the SO1 team focused on a relatively small number of messages with the emphasis on ENA, hand washing and food hygiene. The constant repetition of these messages in every SO1 activity and across all three SOs encouraged adoption and reinforced retention. The fact that many communities heard the same messages through other projects was an added benefit. Third, the significant increase in the number of CHVs combined with the dedication of the Model Mothers was also a determining factor that likely contributed to SO1 achievements as CRS/MG SALOHI MYAP FE/USAID-FFP-2-00002 24 these dedicated volunteers facilitated home visits and provided new adopters with the personal attention and encouragement that was crucial to helping them maintain new behaviors. Additional success factors include: - Mothers and caregivers can see tangible results: weight gain and less diarrhea. - The actions promoted require relatively few external resources. - NGOs were willing to take drastic steps when progress was slow, e.g., annual output targets not being met. LOL, for example, decided to hire its own health team when a local NGO contractor did not produce desired results after the first year of implementation. - Certain CSB staff and MOH regional health directors actively supported SALOHI efforts. - The provision of MCH rations was an added incentive to participate in SALOHI activities. Sustainability: The SALOHI Sustainability Strategy document defines sustainability as follows: “Sustainability involves making the effects and impacts of program activities last. It is not the activities that should necessarily continue, but the EFFECTS of the activities that should continue (adoption of behavior change, or provision of services, for example). Sustainability is a process of transformation and/or behavior change and the institutionalization of these changes. “ It does not necessarily mean that all the project activities will continue post-project. Examples of sustainability for SO1 include: - CHVs have the capacity and motivation to lead GMP activities, to continue to carry out home visits and to serve as resources for their communities. - Parents of young children continue the good health and nutrition practices they adopted during SALOHI and promote these same practices with their neighbors. - Community leaders are aware of the importance of good HH and community hygiene and sanitation practices and continue to promote these practices. Based on this definition, the FQS, including those discussed above, and the Qualitative Study results both indicate that much of what SO1 has accomplished is likely to be sustained post￾project. During the Qualitative Study, for example, the majority of respondents in the SO1 FGDs noted that they are likely to continue the following promoted behaviors after the SALOHI program ends: community based growth monitoring and promotion; improved breastfeeding practices, good food hygiene practices; regular prenatal consultations; improved IYCF; and improved personal hygiene practices. 2.6. SO1-specific Recommendations  Focus on time saving approaches to facilitate more mothers’ participation (38% - 57% of mothers who didn’t participate in SALOHI activities cited lack of time as the main constraint to participation in SALOHI activities).  For PD/Hearth, ensure that the strategies used by PD mothers for feeding their children are widely disseminated (25% of mothers whose children didn’t complete all 12 days of their PD Hearth session cited lack of money to buy resources for PD/Hearth as a constraint to program completion). Consider combining PD/Hearth activities with another approach such as Care Groups or SAMBAIKA groups to reach more caretakers and highlight prevention.  Institutionalize community support/reward systems for CHVs.  Ensure that all CHVs have adequate materials for conducting communication activities (minimum: one set of visual aids per volunteer). CRS/MG SALOHI MYAP FE/USAID-FFP-2-00002 25  Develop a more comprehensive, varied Social and Behavior Change Strategy, including finding ways to involve other population groups such as older children (who are often caretakers), older women (e.g., the Grandmother Project), and schools.  Where resources permit, include a WASH component for disease prevention. Where a full WASH component is not an option, promote simple hygiene and sanitation measures such as Tippy Taps to encourage hand washing (www.tippytaps.org). 3.SO2 – Livelihoods 3.1. Brief Description of Interventions SO2 interventions include a combination of increasing agricultural production through Farmer Field Schools (FFS) and Farmer Leaders (FLs) (IR2.1); increasing access to credit and savings through Village Savings and Loan Associations (VSLAs) and Village Agents (VA) (IR2.2); and linking farmers to markets through agribusiness (AB) activities (IR2.3), with an aim to integrate the three IRs for maximum impact. Ideally, the process starts with the near-simultaneous creation of FFS and VSLA groups (same membership in both) and after one year or so, if the group is dynamic and motivated, it moves into AB activities. FFS groups graduate after about two crop cycles, VSLAs graduate after their first cycle (one year), and AB groups graduate after their first successful business transaction (at most two years). After the first year of FFS and VSLA, the most dynamic and motivated members, usually their presidents or committee members, are asked if they would be willing to become community volunteers as Farmer Leaders (FLs) and Village Agents (VAs) to help support existing and new groups and to spread their knowledge throughout their communities. The role of the FL is to disseminate knowledge and promote the application of new agricultural and livestock techniques, with or without FFS groups, while the role of the VA is to support current VSLA groups and also to create and support new ones. These community volunteers do not receive anything from their communities nor from the project. All FFS, VSLAs and ABs are managed by elected committees. Farmer Field Schools (FFS) are farmer-led groups promoting participatory problem solving and analysis and the adoption of one of 16 improved agricultural practices through shared experiences and innovation. Groups of 10-30 farmers meet regularly to set their own learning agenda and use their own fields as a hands-on laboratory for learning, with technical support from SALOHI staff and external partners. Village Savings and Loan Associations (VSLA) are self-capitalized groups of 20 members who use and manage their pooled money for savings, loans and a grant fund for personal emergencies. Groups determine their own rules such as how much and how often to deposit savings, how to access savings, how to obtain loans, and how to set interest rates. Groups set specific time periods for a savings cycle (generally 10 - 12 months) after which they disperse the total fund (savings plus interest from loans) in proportion to the number of shares each member has bought. Agribusiness (AB) activities were implemented in three different ways by SALOHI partners, with each NGO using variations of the SALOHI AB strategy. CRS used the simplest approach, encouraging local marketing through FFS groups; this AB strategy reached the most farmers but did not usually include the full spectrum of AB training. CARE and LOL chose an intermediate approach with informal producer groups and FFS unions carrying out buying and selling CRS/MG SALOHI MYAP FE/USAID-FFP-2-00002 26 operations. ADRA supported more formal cooperatives, allowing for the most thorough training in and implementation of AB but reaching fewer farmers overall. 3.2. Overall Results Table 11: SO2 key indicators and achievements Indicator Baseline CI FQS CI Average number of months of adequate household food provisioning*** 7.7 7.5 – 7.9 9.1 8.9-9.1*** Average Household Dietary Diversity Score* 4.8 4.4 – 5.1 4.5 4.3-4.7* Coping Strategy Index (CSI)*** 25 24.3 – 25.3 12.6 11.7-13.5*** Yield: Rice (Kg/Ha)*** 560 548 – 572 1,030 960 – 1103*** Yield: Maize (Kg/Ha)** 344 327 - 360 413 361 – 466** Yield: Cassava (Kg/Ha)*** 1,347 1325 - 1369 1,855 1758 – 1953*** Yield: Beans (Kg/Ha)* 360 350 - 370 416 360 – 471* Yield: Sorghum (Kg/Ha)* 150 134 - 164 163 146 – 193* ***Observed change is statistically significant (P < 0.0001) **Observed change is statistically significant (P < 0.05) *Means/average not significantly different (P>0.05) Changes in HH Food Provisioning: The findings of the Qualitative Study (FGD/KII) support the results of the Final Quantitative Survey (FQS): the average increase of HH food provisioning is 1.3 months (+17%). In FGDs, a number of respondents noted larger increases in HH food provisioning (2 – 4 months) and in some cases the complete elimination of hungry periods. Figure 7: Changes in months of HH food provisioning: +1.3 months (FQS, 2013)  The smallest increase (+6%) and the lowest overall level of food provisioning is in the South. 7.7 8.7 9.1 9.59 9.30 7.37 9.22 9.74 0 2 4 6 8 10 12 Baseline FQS CRS/MG SALOHI MYAP FE/USAID-FFP-2-00002 27  MHHs have an average 9.14 months of food provisioning compared to 8.62 months for FHHs.  81% of respondents participated in FFA and 68% received MCH rations. USAID food rations could contribute 5 - 30% of an annual HH food budget. Using a simplified analysis: FFA HHs receive on average 115 kgs of cereal rations (20% of the HH’s annual food budget) and, on average, MCH HHs receive two to four months of rations for one adult or one child (16% - 33% of the annual food budget for one individual). However, only 20% of SALOHI target communities received food aid in any given year, and few communities received food aid for more than one year.  Another potential confounding variable is rainfall/weather patterns at the time of the baseline and the final evaluation. According to FAO, Madagascar produced 4,540,435 MT of paddy rice in 2009 (generally considered a good year), and 4,000,000 MT in 2012/2013 (http://faostat.fao.org/site/339/default.aspx). According to the joint FAO/WFP crop assessment in October 2013, rice production in 2013 was 21% below the 2012 average, and the rice production deficit was estimated at 240,000 MT. Nationally, maize production decreased 15% over 2012, and cassava production decreased 1 % (www.fao.org docrep 018 aq115e aq115e.pdf ).  70% of farmers reported a bad harvest during the baseline survey, compared to 50% during the final evaluation. This difference could potentially be explained by the use of new technologies by SALOHI farmers.  Two additional potential confounding variables affecting this indicator are road and market access and crop pests. Road access generally improved in SALOHI zones between the baseline and the final evaluation. As the SALOHI program rehabilitated 900 km of rural feeder roads, this is not surprising. Pest attacks were mentioned as a factor in low crop production by 28% of farmers during the baseline survey, and 29% of farmers during the final evaluation. Figure 8: Changes in HH food diversity score Changes in the HDDS: The FQS points to a small but statistically insignificant decrease in the household dietary diversity score, from 4.8 to 4.55. The LOA target was 6.8 food groups (+2) out of a total possible 12. This is a drop of 0.25 or a 5% decrease. 4.8 6.8 4.6 0 1 2 3 4 5 6 7 8 HFDS baseline HFDS FQS CRS/MG SALOHI MYAP FE/USAID-FFP-2-00002 28 The only increase in HDDS (not statistically significant), from 2.2 to 2.8 (+27%), was in the South (which had the lowest HDDS in the baseline survey). MHHs have an average HDDS of 4.55 compared to 4.18 for FHHs. Thus, as a proxy for food access, HHs have not improved their food access during the life of the project. However, as an indication of HH food diversity, the findings from the Qualitative Study do not support the results from the FQS. FGD participants indicated people have increased their dietary diversity, especially HHs that have been involved with SO1 nutrition activities. However, some of the changes noted concerned increased quantity of food rather than the diversity of food categories. In addition, increased food diversity was usually for limited periods during the year (the vegetable growing season, immediately after harvests, or during special occasions). Confounding variables for this indicator include market access, market food availability, and seasonality. Changes in HH Resilience: The CSI is a composite non-parametric index based on HH coping strategies associated with food management12. The FQS points to a large increase in HH resiliency, as reflected in a decrease of the CSI from 24.9 to 12.6 (out of a worst score of 56), or a 50% reduction; this is very close to the final target of 12.5. The biggest decreases have been in the East (-66%) and SE1 (-61%). MHHs have an average CSI of 12.3 compared to 13.6 for FHHs (FHH being slightly less able to cope with shocks). Figure 9: Changes in HH coping strategies (FQS 2013 and SALOHI Baseline 2009) Results from the Qualitative Study support this quantitative result. 3.3. Farm Productivity (IR 1) Results: - 47% of respondents (30% of women) are or have been FFS group members (FQS) - 5% of respondents participated in livestock FFS (FQS) - 62,383 farmers participated (51% women) in 4,188 FFS groups (BDR) - 4,531 Farmer Leaders (34% women; BDR) - 25% of FFS members are from the same HH (BDR) 12 The CSI used is as described in: http: pdf.usaid.gov pdf docs pnads3 0 .pdf and the same weights were used for the baseline and the FQS. 24.9 12.5 12.6 29.7 9.9 0 5 10 15 20 25 30 35 Baseline FQS CRS/MG SALOHI MYAP FE/USAID-FFP-2-00002 29 - 71% of FFS members are members of FFS groups only, whereas 18% are members of FFS groups and VSLA, and 10% are members of FFS groups, VSLA and Agribusiness groups (BDR) - Average FFS group size: 15 (BDR) Ninety-four percent of respondents practice agriculture and 78% also have livestock, of which: 93% have poultry, 51% cattle, 28% pigs and 17% goats. Cattle and goats are primarily managed by men and women are the primary managers of vegetable production and chicken production; all other tasks are shared. Agricultural production techniques: The use of all promoted agricultural techniques increased, especially for rice and cassava (65% of farmers) and in particular the practice of planting in rows, using improved seed, using basket composting (especially for cassava production), and the use of organic fertilizer (+40%). The smallest changes were for crop rotation and weeding (<+15%), due to already high practice rates (>80%), and for mulching and direct planting (<+18%) both with the lowest practice rates (<20%). Promoted agricultural production techniques were effective for two reasons: First, they were technically appropriate (no or low cost, readily usable and not exceptionally labor intensive) and second, they were widely disseminated. After the FFS groups demonstrated that techniques were effective, they spread to other community members through a spill-over effect via through demonstration activities, and through FFS members and FLs. The spill￾over effect reached between 40% and 120% more farmers than the original FFS farmers, depending on the technique. This figure shows average increases in yields. During FGDs, reported yield increases of 100- 300% were not uncommon, especially for rice and cassava. The FGD findings agree with the findings in the FQS in that both production and productivity improved as a result of using new production techniques. Figure 10: Yield increase for the 5 target crops in kg/ha Although these figures represent significant increases over baseline values, they are still much lower than national level production figures (average cassava yields in Madagascar are 6.7 – 9.4 MT/HA, rice yields average 2.9 MT/ HA and maize yields average 1.3 – 1.5 MT/HA, from 2009 – 2012; http://faostat3.fao.org/faostat-gateway/go/to/download/Q/QC/E). FGD respondents often commented on the impact of using SRI on rice production, and basket composting for cassava productivity, despite added labor. SRI is a method developed in Madagascar using a combination of five techniques which were promoted by the SALOHI program (http://www.srimadagascar.org/). CRS/MG SALOHI MYAP FE/USAID-FFP-2-00002 30 There is a statistical correlation between participation in FFS groups and the adoption of two or more agricultural techniques (p=.000) and there is a statistical correlation between the adoption of two or more agricultural techniques and the yield of rice and maize (p=.001; but not cassava, beans or sorghum). Food availability was improved (as evidenced by the increase in the average number of months of HH food security) through: increased productivity and production; linkages with SO3 for access (roads) and water management (dams and canals); linkages with SO1 for improved food utilization, especially for children but also for adults’ nutrition; better linkages to markets; and decreasing animal mortality through training in animal health and improved access to veterinary services (in the South only). During FGDs, it was found that food insecurity for FFS participants was reduced for the following reasons: increased production (kgs); increased productivity (kgs/ha); increased efficiency (yield/ seed used, yield/time to maturity or yield/effort); new or more vegetable production; and the adoption of other crops such as sweet potato, yam and vegetables. 48% of HH surveyed participated in FFS groups, including 33% of mothers and caretakers of children under five. However, some HHs did not participate in FFS due to lack of time (56% of non-participants) and lack of awareness (23% of non-participants). Some HHs, especially FHHs, did not have time to attend FFS because they are day laborers, are in the trading business, or cannot leave home. Where land access was an issue (2.6% of non FFS participants), some women did learn new techniques indirectly through other farmers or farmer leaders. Between the baseline and the FQS there was a 9% overall increase in the use of at least two techniques (70% to 79%). The most pronounced increases were in ADRA communities (88%) and in the SE1 zone (93%). The overall average for all SO2 SALOHI participants is 92% (from routine data collection from FFS group members) vs. 79% in the general population. Figure 11: Use of at least two improved agricultural techniques (FQS 2013) ADRA, CARE, CRS, LoL and area statistics are end-line data. 70% 79% 80% 76% 88% 72% 84% 72% 85% 75% 76% 93% 72% 0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100% CRS/MG SALOHI MYAP FE/USAID-FFP-2-00002 31 The combination of the FFS approach with farmer leaders: FGD results showed that the FFS+FL approach was effective in improving the dissemination of technical knowledge and thus indirectly in improving the food security status of households in the target zones. The FQS indicates an overall increase of 5% in the use of crop-specific agricultural techniques for respondents who received support from FFS farmers or FLs (+19% for vegetables, +12% for corn and +8% for beans). Note: The FQS did not make a distinction between FLs and FFS members. 3.4. Agribusiness (IR 2) Results: - 19,868 farmers reached (52% women) in 847 producer groups - 22% of producer group members are from the same HH - 9.4% of HHs actively participate in AB activities (cooperatives) - Average group size: 23 - Fewer FHHs participate in AB than MHHs (7.3% vs. 10%). Figure 12: Agribusiness participation (FQS 2013) Participation in agribusiness groups (AB): Participation in agribusiness groups also improved the food security status of participating households, but AB groups were organized differently by each NGO. The FQS indicates that AB participants increased the number of months of adequate food provisioning by four weeks (13%) as compared to five and half weeks (18% or 1.36 months ) for all SALOHI respondents (+45% for the months of October-January and -25% for February-March). For the impact of AB participation on the HDDS, the FQS data was inconclusive. It is important to note that AB activities take at least two years to start showing results (after the FFS and AB training phase). Only 10% of the population participated in AB in SALOHI zones and certain conditions had to be met to successfully promote the approach with a given group: - Sufficient experience and commitment working together as a group (such as in FFS and VSLA) - Access to markets by vehicle (sometimes with an SO3 linkage) - Having a reliable surplus to sell and having control over crop production 3.6%5.0% 22.0% 6.9% 9.4%7.5% 5.3% 14.9% 10.8% 6.9% 10.0% 7.3% 0% 5% 10% 15% 20% 25% CRS/MG SALOHI MYAP FE/USAID-FFP-2-00002 32 - A value chain is already in place in the region. In the case of SALOHI, existing value chains for corn, rice, beans, vegetables and groundnuts were used though SALOHI did not exploit the honey value chain in the SE. 3.5. Village Savings and Loan Associations (IR 3) Results: - 34,562 HH reached (59% women) in 1,810 VSLAs - 1,163 VAs trained (45% women) - By Year 4, nearly 100% of savings used in revolving credit (the SALOHI maximum is 300%) - 39% of all HHs participated in VSLAs (FQS) - Average VSLA group size: 19 Figure 13: VSLA participation (FQS 2013) Note that (1) the low number of VSLAs in ADRA zones and in the Center is a result of the delayed promotion of VSLA groups by ADRA, which only introduced VSLAs after the midterm evaluation; and (2) the high number of households who participated in VSLA in CARE zones, and in the South. Fifty percent of VSLAs were added in Year 4. Participation in VSLA groups did not directly improve the food security status of participating households. Although 39% of FQS respondents belong to a VSLA, FQS results show no statistically significant impact of VSLA on the number of months of food provisioning, HH dietary diversity, or HH CSI. Fifty-one percent of VSLA members used loans, which were used primarily for trading (22%), food (17%), health and schooling (11%), and for agricultural production (10%). 8% 52% 45% 41% 32% 14% 41% 57% 27% 43% 0% 10% 20% 30% 40% 50% 60% 70% CRS/MG SALOHI MYAP FE/USAID-FFP-2-00002 33 Figure 14: VSLA benefits (FQS 2013) This figure shows that nearly all the benefits of VSLA have a positive effect on HH resiliency. The direct effect on food security includes reducing the need to sell food. Indirect VSLA benefits gleaned from the FGDs were: - An appreciation for better discipline and accountability - In the more fragile South, the use of the Solidarity Fund was repeatedly mentioned as a positive effect. - A reinforced sense of social cohesion. 3.6. Key Findings 1. The South had the least increase in NMHHFS (+0.4) in and the lowest number of months in food provisioning (7.7); it has the lowest HDDS (2.8) and recorded the second smallest decrease in CSI (-7.4). 2. The Center had the second lowest increase in food provisioning (+0.8) and the second lowest decrease in CSI (-6.4). 3. Results in the East, SE1 and SE2 are very similar in most indicators and closest to average SALOHI values, except for the large decrease in CSI in the East (19.8% lower than the final SALOHI average). 4. FHHs have lower results (5 - 8%) than SALOHI averages, with the biggest difference being in household resiliency (CSI). 5. Targets for number of moths of adequate food provisioning and HH resiliency were reached or surpassed, while the HDDS remained static. 6. According to FGDs, community volunteers such as FLs and VAs have been crucial for leadership, sustainability, motivation, learning and coaching, knowledge dissemination, and linkages to authorities, leaders and partners. 7. According to FGDs, SO2 participants benefited from partnerships with other programs (NGO or international) and local agencies of the Ministry of Agriculture, especially for SALOHI complementary activities (marketing, inputs and trainings) and for obtaining appropriate complementary resources (chickens, seeds, tools, and didactic materials). 8. In each community, early adopters (usually the more literate and older people according to the 2011 Barrier Analysis) benefitted the most from being the first to join FFS groups and VSLA, and demonstrated the most change in knowledge and practices. Early majority adopters (Year 64% 52% 47% 32% 18% 17% 0% 10% 20% 30% 40% 50% 60% 70% Emergency cash needs No other debts Financial benefits Social advantages No food sales Schooling CRS/MG SALOHI MYAP FE/USAID-FFP-2-00002 34 2) were also able to benefit after observing the success of early adopters. The late majority adopters of agricultural techniques (third wave) were able to participate through FLs but much less so in VSLAs. 3.7. Significant external factors affecting project outcomes  Agriculture in project zones is mainly rain-fed, thus prone to drought which would reduce yields regardless of techniques used (except irrigation); this could discourage farmers from adopting new practices. The project did promote drought resistant crops (sorghum, orange fleshed sweet potatoes, and yams) and techniques (mulching, irrigation, conservation farming, and keyhole gardens).  The quality of support from traditional authorities was instrumental in the degree of take-up among communities. Where traditional authorities actively participated in motivating people and in joining FFS, VSLAs and ABs, farmers were also much more willing to participate in SALOHI activities and to find creative solutions to their production problems.  The quality of support from administrative authorities was less influential on outcomes than that from traditional leaders, but it nonetheless helped with: a) finding land for women to use, b) linking with external resources, c) negotiating administrative conflicts, and d) giving credibility to both individuals (FLs, VAs) and groups (VSLAs, ABs).  Agriculture activities pre-suppose access to sufficient crop land, but there is a small portion (3%) of the population without access to enough land to meet their food needs, making it almost impossible for them to put into practice what is promoted in FFS groups. This is especially the case for women in general and for FHHs in particular. The SALOHI team negotiated access to land for FHHs who participated in FFS groups, especially in the South (for dry season gardens) and the Southeast (for both rain fed and gardening crops).  Other confounding variables for SO2, which could be further explored if information is available, include: road access (especially for selling); market access (buying and selling); off-farm income opportunities; “free” labor availability (family and community); and social networks. 3.8. Conclusions Relevance: Considering the level of chronic food insecurity in SALOHI project zones, the three IRs are very relevant to the needs of smallholder farmers. The most relevant, in that it applies to all farmers, is that of increasing productivity through no or low-cost improved techniques using the participatory approach of combining FFS and Farmers Leaders. The second most relevant, in that it can apply to most farmers, is very sustainable and can be self-replicable, is the promotion of VSLAs to enable families to have low-cost access to credit and savings, which also helps build social capital. Agribusiness is very relevant only to a minority of better-off farmers who have the necessary pre-requisites (access to land, assets and capital) to practice advanced marketing techniques. The integration of these activities with other SO’s is also very relevant. Impact: There are three types of impacts: what can be measured (trough quantitative indicators), what can only be appreciated (through discussion and observation) and, ideally, the combination of both. In this case, in both the FQS and the there is strong agreement on the high impact of improved food production techniques on yields and months of food provisioning, and resiliency (decreasing the CSI). It is difficult to assess the impact on dietary diversity or consumption using data from the final evaluation. According to the FQS and focus group meetings, agribusiness activities had a positive impact on livelihoods, but only 10% of SALOHI households participated in these activities. VSLAs had a positive impact on livelihoods and the quality of life of members, but CRS/MG SALOHI MYAP FE/USAID-FFP-2-00002 35 it is not measured in the FQS. The project has had a difficult to measure but positive impact on community empowerment, through its participatory approaches. Sustainability: Similarly, there are three levels of sustainability of outcomes and (not necessarily activities): what is perceived by the participants (based on activities and their perception of the future), what is perceived by the evaluators (based on desired outcomes and experience) and the combination of both. There is strong agreement of high sustainability for VSLAs (high perceived value, reasonable cost of participation, provision of longer term support services). There is also strong agreement of medium sustainability for the practice of the promoted techniques, most likely through Farmer Leaders rather than through continuing FFS. Only the well-established AB groups will be sustainable (about half of them). SALOHI is to be commended for having a well-developed sustainability strategy from the beginning, though it has been uneven in its application. General conclusions: SO2 reached all IR quantitative targets and one of two impact targets. SALOHI activities have had a positive impact on the food security of target HHs, as demonstrated by food provisioning and CSI indicators. While the HDDS indicator did not increase, FGDs indicate that HH dietary diversity has improved. The spread of new agriculture and livestock/small animal husbandry knowledge has been thorough and appreciated and is largely sustainable, as promoted technologies are simple to apply, do not require continued external support, and have had enough of a critical mass effect to be self-sustaining. VSLAs have been very successful and are sustainable. VSLAs are good for improving resilience to small shocks but the amount of money available to a HH either as credit or savings is not sufficient to have a significant impact on food provisioning alone. AB activities have been successful but only 10% of the population participated in this activity (which was the original target, due to the extremely vulnerable target population). There is no simple explanation for the large drop in CSI. One hypothesis is that the combination of many separate actions yields a bigger impact than the sum of each action individually, because: 1) the combination of activities gives people choices and flexibility to respond to shocks; 2) it prevents a vicious circle of negative compounding actions (such as selling productive assets, contracting exorbitant debts or not being healthy to work); and 3) it decreases negative social consequences (borrowing from neighbors, being suspected of theft, etc.). But a counterfactual and more detailed analysis is needed to prove this hypothesis. 3.9. SO2 Specific Recommendations  To better assist more vulnerable HHs, low land access HHs and FHHs, FFS groups should include more than just agricultural techniques. Consider poultry and other small livestock if appropriate, fruit and spice trees, bee-keeping, and fish farming (in rice paddies).  All FFS trainings should also include basic marketing (as most farmers sell some crops), improved post-harvest practices (as it is common for smallholder farmers to lose up to 30% of their crops), and nutrition information, especially when combined with MCH groups (as micronutrient deficiencies are very common).  Worldwide, VSLAs have shown themselves be highly appreciated by their members (especially women), very sustainable and contributing to social and economic stability and development. Thus VSLAs should be used as the engine of community development from the beginning of the program, particularly by: - Linking all SO groups and community volunteers to VSLAs. - Using a shorter cycle time in the first-cycle if the share-out can take place at a more relevant time (important national holidays, fêtes, cultural events, and school fees). CRS/MG SALOHI MYAP FE/USAID-FFP-2-00002 36  To better understand changes in food provisioning due to rations and other activities, use a method to quantify the percentage of USAID food rations in the HH annual food budget. 4. SO3 - Disaster Risk Reduction SALOHI target communities in the east and south coasts of Madagascar are characterized by drought (90% of HHs in the South), floods (75% of HHs in SE2) and cyclones (48% of HHs in SE2). During the Qualitative Study respondents also reported that other natural shocks plague the communities from time to time, including: strong winds, locusts, pests, hail, fire and other weather-related phenomena. Furthermore, SALOHI communities living in the central highlands are inaccessible by vehicle during most of the rainy season, and insecurity is a problem in several communes. In response to these shocks, SALOHI SO3 interventions are implemented to prepare people to mitigate the impacts of natural disasters in three phases: 1) Preparation/Prevention, 2) Response, and 3) Rehabilitation. The most vulnerable beneficiaries are either smallholder farmers with limited access to land (less than 1.5 ha), pastoralists or female-headed households. The majority of SALOHI beneficiaries live precariously and are chronically at risk of a financial or food crisis. A number of the SO3 themes and interventions are discussed in Section E, under the cross￾cutting theme Environment. Topics in that section include the impact of SALOHI activities on the environment, improvements in natural resources management, and the benefits of infrastructure built or rehabilitated. 4.1 Achievements Table 12: SO3 Achievements (FQS 2013) Indicators Baseline Target LOA Achieved LOA Achievement (% of target) Community Resilience – CVI (Community Vulnerability Index) 4 8 10 CI: ND P: .000 125% Percentage of communities accessible by car year-round 42% N/A 65% N/A Number of fokontany with a DPMP N/A 592 592 100% % of communes with an EWS 44% 76% 94% 123% Kilometers of transportation infrastructure constructed or repaired N/A 691 899 129% % of water systems constructed which are functional N/A 90% 97% 108% Ha of irrigated land created or rehabilitated by FFA activities (99 irrigation systems built or rehabilitated) N/A 4,000 9,811 245% CRS/MG SALOHI MYAP FE/USAID-FFP-2-00002 37 Indicators Baseline Target LOA Achieved LOA Achievement (% of target) Number of IMAs trained in infrastructure construction and management N/A 159 277 174% Number of organizations and structures supported by the SALOHI program that are trained in good governance principles (participation, transparency, accountability N/A 2,260 3,195 141% Percentage of organizations and local community groups which diffuse information relative to their activities and results at the community level N/A 60% 56% 93% Figure 15: Benefits of SALOHI infrastructure (FQS 2013) 86% of HH surveyed reported that their community had benefited from SALOHI infrastructure activities, of which 70% were roads, 49% irrigation, 31% irrigation and 31% community sanitation interventions. Infrastructure built or rehabilitated by the SALOHI program has provided many benefits. FQS respondents especially appreciated the fact that infrastructure improved access into and out of their communities (60%), but the positive impact on food 0.0% 10.0% 20.0% 30.0% 40.0% 50.0% 60.0% 70.0% 33.6% 60.5% 23.9% 10.0% 16.6% 16.6% 16.8% 10.6% 5.6% CRS/MG SALOHI MYAP FE/USAID-FFP-2-00002 38 security (improved access to food, protection of lands, increase in cropped areas, and improved production) are also ranked as benefits of SALOHI infrastructure. Figure 16 below shows that there has been a dramatic drop in the use of every kind of coping strategy between the baseline and final evaluation, ranging from 50 to 95%, illustrating an important decrease in HH vulnerability to shocks. Figure 16: % HHs using various coping strategies during periods of food insecurity (SALOHI baseline 2009; FQS 2013) 4.2 Community Resilience: The Community Vulnerability Index (CVI) In the SALOHI program, community resiliency was measured using a composite index called the Community Vulnerability Index, based on four SALOHI-specific criteria: - Community Accessibility (number of months per year when the community is accessible by vehicle) - Percentage of households with access to irrigated land - Existence of a DRR committee and a DPMP - Existence of annual simulation/sensitization/ mobilization exercises by DRR committees Scoring ranges from a low of 4 for very vulnerable, to 5 - 8 for average vulnerability and over 8 for least vulnerable. The CVI score at baseline was 4 and the LOA target was 8.The final average score was 10 (between 9 and 11 in each zone), and suggests that communities are now much less vulnerable and are more resilient in the face of natural disasters and shocks, a measurable achievement that exceeded the project target. 0 10 20 30 40 50 60 70 80 90 100 % HH using various coping strategies during periods of food insecurity Baseline FQS CRS/MG SALOHI MYAP FE/USAID-FFP-2-00002 39 4.3 Emergency Preparedness (IR 3.1) (592 communities are prepared to respond to shocks.) Results: Since SALOHI started SO3 activities, there has been no loss of life and very few damaged houses following cyclones and floods in the communities visited during the Qualitative Study. According to beneficiaries interviewed, communities and beneficiaries are now better prepared to face natural disasters as a result of: - Better sensitization of the community about disaster preparation - Identification of safe shelters and other places for better safekeeping of personal items - Establishment of a reliable warning communication system (with megaphones, flags and FM radios provided by SALOHI) - Movement of animals to higher grounds - Containers for storage of drinkable water (provided by SALOHI) - Availability of village volunteers to secure people and property and for crisis management - Strengthening of houses (walls and roofs) - Construction of community granaries to save part of their harvests (very few) - Annual simulation exercises Disaster Preparedness and Monitoring Plan (DPMP): All 592 communities have a DPMP, but according to the FQS, only 29% (40% for CRS) of community members know about its existence, a finding confirmed during the Qualitative Study. The DPMP includes the identification of risks such as: types of natural disasters; protection measures for people and their assets, agriculture, livestock and infrastructure; and the construction/rehabilitation, maintenance and repair of roads, dams and canals. Between 79 and 96% of FQS respondents who knew about the DPMP participated in DPMP activities led by the Disaster Risk Reduction (DRR) Committee. Approximately 0% of all DPMP’s have been implemented Disaster Risk Reduction Committees: All communities have a trained DRR committee. In the FQS 59% of HHs know of the DRR committee in their communities and are aware of DRR committee activities, including: community mobilization and sensitization (mentioned by 71% of respondents); alerting the community during disasters (3 % ); and organizing community activities to prepare and recover from disasters and shocks (31%). A high percentage of HHs (97%) are satisfied with their local DRR committee and 88% indicate that DRR activities will continue after SALOHI. Between 72% and 88% of HHs participate in annual disaster prevention and response simulation exercises. A DRR Committee is in place in every community, serving as the lead force for risk prevention and mitigation. The ratio of men to women in DRR committees is 70/30 but slowly improving; communities elect members to their positions. In addition to producing and updating the DPMP, sensitizing communities, running annual simulations and taking leadership roles in case of an emergency, committees also learned how to request additional resources if needed after a shock. There are several examples of DPMPs in SALOHI communities affected by shocks. One example in the community of Ifasina (Mahanoro District) included the construction of a local food storage unit, and cyclone resistant housing. When this community was threatened by Cyclone Giovanna in 2012, the DRR committee implemented evacuation protocols and CRS/MG SALOHI MYAP FE/USAID-FFP-2-00002 40 communication campaigns, and as a result, impacts from the cyclone were limited and the community was able to recover relatively quickly from the event. Food Security Monitoring System (SSSA): Although 84% of communes have an SSSA, most are weak. The SSSA was originally planned in conjunction with WFP and the GOM, but this collaboration was not effective due to USAID restrictions on government collaboration, and contracting difficulties with WFP. The Food Security Monitoring System is limited in scope and not well understood by DRR committee members at the fokontany level (the SSAA activity operates at the commune level). The SSSA activity was started in Year 3 and involved the collection of food prices in markets, water prices, rainfall, labor and nutrition data. The purpose of the SSSA was to help program staff identify when it was necessary to adapt program activities from development to emergency response, and vice versa. It was also used to monitor environmental changes which could require programmatic responses. Most of the obstacles to successfully implementing this activity were not within SALOHI’s control, since the SSSA is part of a much larger national endeavor. 4.4 Resource Management (IR 3.2) (592 communities improve management of land, water and roads.) Title II commodities (rice, sorghum and oil) were used as Food for Assets (FFA) for the completion of roads, dams, canals and reforestation efforts. Men and women participated equally in FFA activities and received the same food rations although women were usually assigned less physically-demanding work. Figure 17: HH awareness, maintenance and use of SALOHI infrastructure Not surprisingly, roads have the highest amount of awareness and use. Reforestation ranks lowest in awareness and use but this is to be expected as the trees are still young and not yet exploited. (Note: Not all communities received roads, irrigation or reforestation systems.) 71% 50% 31% 31% 85% 87% 82% 91% 98% 74% 63% 96% 0% 20% 40% 60% 80% 100% 120% Roads Water systems Reforestation Community Sanitation HH awareness, maintenance and use of SALOHI infrastructures Mentioned Maintained Used CRS/MG SALOHI MYAP FE/USAID-FFP-2-00002 41 4.5 Construction/rehabilitation of Canals and Dams (IR 3.3) Results: Of the 99 irrigation systems rehabilitated or constructed by SALOHI, 97% are functional. Using FFA rations, SALOHI worked with communities to irrigate 9,811 ha of land, resulting in the drainage of flood water, the irrigation of rice paddies, and increased production of vegetables and other crops. Canal construction by community members (both men and women) in Ankarinarivolmady Road Rehabilitation: To date, 898 km of rural feeder roads have been constructed or rehabilitated by the SALOHI program, meeting the project target. Communities have better access to basic facilities such as health centers, schools, administrative offices and markets and are not cut off after disasters. Merchants can more easily reach communities with their vehicles to buy agricultural produce, putting the beneficiaries in a better negotiating position. This is a clear demonstration of the integration of SO2 activities and SO3 activities, where the increased production as a result of FFS activities is now sold for better prices due to better accessibility. Infrastructure Management Associations (IMA): At the end of Year 4, 277 IMAs had been formed and trained. The construction or rehabilitation of canals, dams and roads by SALOHI led to the creation of several types of management associations: Road Users Associations, Water Users Associations, Natural Resource Management (NRM) and Infrastructure Management Associations. During the FQS 76% of HHs stated they know of IMAs and 32% are familiar with NRM associations. The roles and responsibilities of these associations are to: - Mobilize resources necessary for infrastructure maintenance (regular and as-needed) as per the maintenance plans. - Mobilize community members to participate in maintenance work and supervise the actual work. - Monitor and evaluate the implementation of the infrastructure maintenance plan. - Monitor water distribution in rice paddies to prevent conflicts from unauthorized water use. - Develop and maintain partnerships with public and private entities for technical assistance and support. CRS/MG SALOHI MYAP FE/USAID-FFP-2-00002 42 Some of the committees have initiated mechanisms to raise funds for the maintenance of infrastructure through toll fees for commercial vehicles using roads or through the local bylaws (dina), issuing substantial fines for those who violate canal and dam regulations. Proceeds are used as incentives for maintenance work. Toll fees on a rehabilitated road Although these are promising developments, there are some issues to be resolved: - In communities with poor leadership, canals are littered with debris or obstructed. - There is a general lack of transparency in the toll collection systems. In spite of the dina, some communities fail to fine violators. - The end of FFA distribution in some communities reduced motivation and may have contributed to poor maintenance of infrastructures at a time when committees are still struggling to raise funds through tolls or the dina. - In many communities there is low accountability of associations/committees to their communities. Committee members do not report systematically to the communities and do not record their plans and implementation performance. 4.6 Social Protection for Urban Households (2,500 extremely food insecure families in urban areas access critical support from service providers.) Results: Of the 3,125 beneficiaries of this urban initiative, 99% are women in very precarious conditions (widows or single women with no income but caring for a family of four to six children) who have significantly improved their situation. This component, implemented by CRS alone with its church partners, has had remarkable success reaching the most vulnerable households in urban settings. Most of the participants are female heads of HHs and before entering the program, many relied on begging to feed their families. As a result of the program, beneficiaries are equipped with skills to dramatically improve their situation and that of their dependent children, resulting in better livelihood security on all fronts. To achieve this, SALOHI provides assistance to 15 Social Protection Centers through FITEA/ CODFIDES, a local association. Unlike other SALOHI activities carried out in rural settings, social protection activities are implemented in urban centers: Toamasina, Antananarivo and Fianarantsoa. Beneficiaries participate in Social Protection Center activities in groups of 40 to 50 for a period of 10 months, receiving training in income-generating activities such as cooking, cake baking, sewing, market gardening, and raising small animals. During this period, participants receive Title II food rations for their family. Food distributions are carried out by the beneficiaries themselves, which is good for capacity building. In addition to learning a trade, they also receive training in basic life skills, working in groups, hygiene and sanitation, HIV/AIDS, household management, and family and business budgeting. After their business plan is approved, graduates receive equipment and tools to start their own small businesses. CRS/MG SALOHI MYAP FE/USAID-FFP-2-00002 43 4.7 Conclusions Strengths - Communities have now received comprehensive training and practiced simulations to better respond to natural disasters. - Communities have DRR and NRM committees with women’s participation. - FFA work for reforestation, road and canal construction was well-executed under the supervision of field technicians, paying attention to the environment. - Stronger community social cohesion is a result of the extensive group work. - For those with new roads, there is now year-round connectivity to basic facilities and services and good complementarity with 1) SO2 activities for crop marketing and 2) SO1 access to health care. - Social Protection activities give new life skills to women beneficiaries (99%) in very precarious living conditions who usually have a large family to care for. - New and previous beneficiaries are enrolled in VSLA activities as soon as they start their own small trade activity. Weaknesses - Few community members know the content of the DPMP plan. - Communication between associations/committees and their communities is generally weak. - There is a lack of community meeting places. - Where there is no reliable system for the provision of material and financial resources, sustainability is questionable for the maintenance and repair of infrastructure. - There are too few partnerships between infrastructure management associations and other institutions. - Social Protection activities are not implemented by all SALOHI members although the need is widely-felt. - There is a lack of specialized trainers at the Social Protection Centers. CRS’s partners in this endeavor rely on volunteers. - The SSSA is not well-understood and is not effective. Sustainability: Activities of the Centre de Protection Sociale need two main things to happen for a long-term sustainability: effective partnerships and steady funding (both national and international). There is need to have other NGOs (not only CRS and its diocesan partners) involved in activities implementation. Advocacy for funding (to run the centers, provide the graduation kits, pay salary or incentives to specialized trainers, etc.) is to be brought to another level. As a national effort, radio or TV interviews of the program should continue and be more structured and tailored to specific audiences. Financial and material support will come from there as well. Communication in all its aspects will be an efficient contributor. 4.8 SO3-Specific Recommendations for Future Programs  Design an overall development master plan and include the DPMP and other interventions for vulnerable groups such as women, school children, the elderly, the disabled and the youth. CRS/MG SALOHI MYAP FE/USAID-FFP-2-00002 44  Coach and train communities to keep records of their meetings; infrastructure maintenance schedules; and major events. Illiteracy should not be seen as a limiting factor. Secretaries are elected with basic reading and writing skills.  Help communities and communes put in place strong, transparent mechanisms for the management of toll fees and fines as they are crucial for the maintenance of infrastructures. E. Cross-cutting Themes 1. Gender 1.1. The Gender Dimension in the SALOHI Program This section of the report briefly presents background information regarding gender issues in SALOHI zones; discusses the relevance of gender for each SO; describes the impact and potential for sustainability of gender; and summarizes key recommendations. In the original proposal, SALOHI proposed to « promote gender equality and ensure equal access to program resources and their control ». Although baseline data was disaggregated by gender, it contained few gender-specific questions. In addition, the SALOHI team had neither the time nor the technical capacity to perform a gender analysis prior to implementation. In 2010-11, the SALOHI team, in collaboration with a local consultant, started to promote gender analysis within the program. The objective was to ensure gender considerations, instill a gender “reflex” at all levels in SALOHI activities and to define specific activities to ensure gender mainstreaming. Based on the gender disparities identified, the program’s strategic and conceptual frameworks were revised. For SO1 the gender disparity noted most frequently is the lack of special consideration for PLW due to their status, and particularly their nutritional requirements. The second disparity is the lack of interest on the part of men in the health of their children and wives. For SO2, it was noted that women generally have limited access to productive resources and thus to the opportunities offered by development projects: trainings, membership in cooperatives and farmer groups, and participation in decision making. And for SO3, it was noted that women are weakly represented in community structures and little involved in decision making due to socio-cultural constraints. 1.2. Relevance of the Gender Dimension For the Qualitative Study, the relevance of gender considerations can be measured by 1) whether the program’s activities respond to the gender disparities identified above, 2) feedback from beneficiaries, and 3) the level of acceptance of gender equity in communities. In terms of program responsiveness, this expectation has been met for SO1: Communication activities targeting men, especially fathers of young children, raised their awareness and resulted in their increased participation in health and nutrition activities. Specific actions for women and especially for FHH in SO2 have also been relevant, based on their poor access to productive resources, including land, which constitutes an important gender disparity affecting them socio￾economically. By promoting women in FFS, VSLA and other activities, SALOHI has started a process of more equitable access to resources and has contributed to a reduction in gender disparity. For SO3, in response to the low representation of women in local committees (30%), SALOHI has implemented awareness raising activities for greater participation of women in decision making processes. CRS/MG SALOHI MYAP FE/USAID-FFP-2-00002 45 A common characteristic of the SALOHI intervention zones is the high rate of FHH. In the following graph from the FQS, these rates are significantly higher than the national rate of 19.4% (Enquête Périodique auprès des Ménages, 2010) except in the SE1 zone where it is lower (18.9%). Figure 18: Percent of Female Headed Households in SALOHI zones (FQS 2013) Focusing on the situation of FHH is an imperative in any poverty reduction program as the rate of FHH can be at the same time an indicator of poverty and an indication of gender disparities. It can be said that the higher the rate of FHH in a zone, the higher its poverty rate and its gender disparities. The final evaluation has also revealed another type of FHH, “FHH by circumstance”, especially in the Center zone where almost all the women were neither divorced nor separated nor widowed but who were the de facto head of household and performed all productive activities and family duties while their husbands sought income generating employment outside the community. In conclusion, the high rate of FHH in the SALOHI intervention zones justified the focus on this category of beneficiaries and the design and implementation of positive activities in their favor, especially those designed to improve their productive assets and their income. Relevance of gender considerations for SO1: The majority of women participating in the Qualitative Study’s focus groups gave very positive feedback on efforts to encourage men and fathers to participate in SO1 activities. FGD male respondents acknowledged that they had not used to take care of children, bring them to weighing sessions or accompany them and their mothers to health facilities. Since SALOHI, they now do it more easily when they are able to because they want to help their wives and because they love their children. 00% 05% 10% 15% 20% 25% 30% CENTER EAST SOUTH SOUTH EAST 1 SOUTH EAST 2 ZONE 26% 20% 25% 19% 24% Percent of Female Headed Households in SALOHI zones, 2013 CRS/MG SALOHI MYAP FE/USAID-FFP-2-00002 46 Figure 19: % male headed HH participation in health and nutrition activities (FQS 2013) Figure 20: % male headed HH by type of participation in health/nutrition activities (FQS 2013) Figure 21: % male headed HH by type of participation by NGO (FQS 2013) Of the men who participate in health and nutrition activities, 71% do it to help their wives, suggesting that these men think positively of the need for mutual assistance and of the gender complementarity in the household, opening the door for gradual and more equitable sharing of workloads. A few community and traditional leaders expressed satisfaction with the empowerment of men and fathers for the health and nutrition of children. According to them, the 00% 10% 20% 30% 40% 50% 60% 70% Center East South South East 1 South East 2 ZONE 64% 52% 61% 55% 50% 0% 10% 20% 30% 40% 50% 60% 70% 80% 71% 19% 5% 4% 00% 10% 20% 30% 40% 50% 60% 70% 80% ADRA CARE CRS LOL NGO Helps / Supports mother Does some activities Takes lead on certain activities Takes lead on all activities CRS/MG SALOHI MYAP FE/USAID-FFP-2-00002 47 involvement of fathers and support from traditional authorities have led to positive results for children’s health. Level of acceptance of gender equity in SO2 and SO3: The acceptance by communities of the concept of gender equity is evidence of the relevance of the SO2 and SO3 gender-specific activities. Limited access to land for women is a glaring gender disparity in SO2 and is a limiting factor for women to participate in FFS, VSLA and AB. To remedy this, some field teams took the issue to local authorities and traditional leaders to ask them to provide practical solutions so that women can benefit from program’s activities as well as men. Thus some mayors, traditional leaders, church leaders and even wealthier private citizens loaned plots of land to women so that they could implement SALOHI SO2 agriculture activities. It is hoped that this type of behavior will have a positive influence on communities, which will progressively internalize the concept of gender equity. When family members or even simple community members agreed to loan plots to FHH, it helped to strengthen family solidarity and social cohesion. This action is encouraging for the promotion of gender equity. Finally, SALOHI promoted greater participation of women in SO3in the implementation and management of community activities and in decision-making, but progress has been relatively slow. Overall, women's participation in local committees is roughly 30%, including NRM. Despite these mixed results, the inclusion of gender in the activities of SO3 remains relevant – women transitioned from their usual invisibility into becoming members of community decision￾making bodies. Factors constraining the gender dimension: Limitations include the lack of activities in the Gender Strategy, including the lack of a gender communication strategy. This created confusion among communities, both in understanding the concept of gender and in seeing how gender equity translates into concrete actions. At the community level there is fairly widespread ignorance of the gender concept, leading to a misunderstanding of the relationship between the activities implemented by SALOHI and SALOHI’s commitment to promoting gender equity. At the level of the SALOHI team, the gender approach is often limited to encouraging the participation of men and women in program activities. This results in very weak communications on gender and does not help people understand gender issues. This lack of involvement of the team in the promotion of gender equity was confirmed when many of the communities visited said they did not receive any sensitization on gender. It should also be noted also that support to the emergence of women leaders at the community level has not been sufficiently developed by the SALOHI team. The above-mentioned results and the following additional data are explicit: women constitute only 34% of Farmer Leaders, 45% of VSLA agents, and 31% of DRR Committees. Women's participation in decision-making is thus limited. 1.3. Impact of Gender Dimension in SALOHI Both the FQS and the Qualitative Study demonstrated that there has been noticeable progress in changing attitudes and behaviors where gender equity is concerned. The following assessment analyzes the changes that have occurred as a result of the integration of the gender dimension into each SO. Positive Changes in SO1: The first positive change noted by both the FGS and the Qualitative Study is a greater involvement of fathers in child care. Fathers are now involved in weighing sessions, consultations at the health center, reminding mothers of vaccination dates, and even CRS/MG SALOHI MYAP FE/USAID-FFP-2-00002 48 following the child’s growth curve in the health book, all activities that fathers were previously not involved with. A positive change was also noted in a more equitable sharing of family meals. Following sensitization on the importance of improving the nutrition of the mother and children, the meal is now taken together in some families, each receiving the same portion of food. A final positive change in men’s practices occurred when fathers were able to participate in awareness sessions or training in health and nutrition - 33% of health/nutrition trainees in 2013 were men. The FQS respondents gave the following comments on the positive impact of men’s involvement in the activities of health and nutrition: 1) child’s health is better; 2) mother has more time for other household tasks, and 3) facilitates mother’s participation in health nutrition activities if the husband is supportive. In summary, the impact due to the integration of the gender dimension in SO1 is entirely positive and fully consistent with the objective of SO1, which is improving the health of children. Positive Changes in SO2: To incorporate the gender dimension, SO2 activities were oriented toward greater women's participation to fulfill SALOHI’s commitment "to ensure equal access to program resources and their control". The result is that women did have good access to program resources as they were the major beneficiaries of the main activities of SO2.According to women respondents in the FQS, most women participating in SO2 noted several positive impacts; the main ones are listed below in order of importance: Activities Impact n°1 Impact n°2 Impact n°3 FFS Good harvests Knowledge of new techniques Capacity building VSLA Efficient solution in case of emergency No need for other credit Financial benefits Agribusiness More income More Production Shorter lean period The Qualitative Study identified other positive impacts, the one most appreciated by women being the opportunity to save with the VSLAs. SO2 activities have provided significant and tangible benefits for the participating women. However, the results were uneven, mitigating somewhat this otherwise positive assessment. The main issue is the reasons why some women don’t participate in SO2 activities. For FFS, non￾participants cited the following reasons for not participating (in order of priority): I have no information on FFS; my agricultural production is already good; and someone close to me won’t allow me to participate. Reasons given for not participating in VSLA: I don’t have the money to make my contribution; I don’t have time to go to the meetings; and the Village Agents and project staff don’t provide regular monitoring and support. The reasons evoked here demonstrate that SALOHI could have promoted more participation by improving its communication strategy, promoting the benefits of FFS and VSLA. While women's access to program resources and their benefits was reached to a large degree, the same cannot be said for control of these resources and benefits. By probing and cross-checking information, the Qualitative Study team learned from both men and women that women were CRS/MG SALOHI MYAP FE/USAID-FFP-2-00002 49 only the "custodians of the money" and men were the "final decision makers", especially when it came to large expenditures. These nuances have been mentioned in order to draw attention to the lessons to be learned and not to negate the significant positive impacts from SO2 activities. Positive Changes in SO3: Just as in SO2, SO3 activities have focused on encouraging women to engage in activities, to join local structures, and to participate in in the decision making process of these structures. In terms of the positive impact of the inclusion of gender in SO3, it was clear that women were highly motivated to participate in FFA, community hygiene initiatives, and reforestation activities. In addition to the benefits they bring to the entire community, FFA activities provided a significant revenue opportunity for women in general and for female-headed households in particular. Indeed, the majority of FHH in the SALOHI intervention areas are agricultural laborers. Participation in FFA helped them earn enough to provide for their families for several days. FFA activities also brought positive attention to the concept of gender equity and its application in daily life. For example, when recruiting for jobs in FFA SALOHI did not discriminate. The gender ratio was favorable to women (60 % women and 40 % men on average). The food ration was strictly equal for men and women whereas the traditional agricultural payment practice gives an advantage to men (2500 Ariary per day) compared to women (2000 Ariary per day).Finally, women were allocated less strenuous FFA work in order to protect their health. Of all the positive impacts of the program, “evidence-based awareness building” in favor of gender equity appears to be the most important. This impact is noteworthy in terms of the progress made via the practical implementation of the gender approach and noteworthy because observable positive changes in gender relations are gradually happening; it is a notable behavior change worth maintaining and reinforcing. Finally, it has been said that women are still under-represented in the various local committees. However, since the implementation of SO3 activities, some people state that people listen more now when women express their opinions. The biggest positive change is that men have realized that they cannot exclude women from the management of community affairs. A man participating in a focus group in Helibondro - Behara expressed it in his new vision of community life: "It is not an all-community meeting if the women are not there.” The conclusion is that the positive impacts of gender are noticeable, especially if we take into account the socio￾cultural context of these areas. 1.4. Sustainability of the Gender Dimension and its Impact The SALOHI Program does not provide for any specific measures to ensure the sustainability of the gender approach. However, in SO1, women felt that men will continue to participate in health care for their children because their motivation for doing this is genuine, they have been trained, and there is no need to mobilize resources for this. One of the achievements that women will also maintain as a result of the promotion of gender equity is the equal sharing of family meals. In addition, they will continue to give quality food to their children. A woman in a FGD in Marofoty – Ambanisarika summed it up this way: “We’ve already seen the light and there is no turning back into darkness.” Concerning SO2 activities, women are particularly motivated to continue practicing new agricultural techniques they have acquired with FFS. VSLA activities will also be continued, CRS/MG SALOHI MYAP FE/USAID-FFP-2-00002 50 probably because of the financial benefits they bring (cash available when needed and savings). For SO3, the most important factor to sustain the gains in women's representation at the community level is the recognition of women's capabilities by some traditional leaders (ampanjaka). Their support will be decisive for the integration of more women into community decision-making structures. Another important point: Some women believe that the promotion of gender equity will only change existing gender relations if the State intervenes and "forces" men to respect women's rights. Such is the case for women's access to land, which cannot be institutionalized because men are opposed to it. According to these respondents, the duty of the State is to help men to recognize women's right to have access to land and to enforce compliance. Gender Sensitivity: Rural Madagascar is a male-dominated society and community members continue to see certain types of activities as “male-oriented”. Tradition has limited women to some roles like: taking care of children, raise poultry instead of cattle, working in the kitchen instead of maintaining roads or canals, not owning lands, etc. Thanks to SALOHI, these things are slowly changing. Men in the communities, elders, local authorities (mayors), traditional chiefs, etc., with whom the evaluation team met are all open to the idea of including women in more development activities in their communities. We saw during focus group discussions women who could talk freely before their male community members, women in FFS groups and FFA activities and men taking their children to PD/Hearth-sessions. Things are gradually changing and it will take NGO, communities and governments efforts to keep the momentum. The table below shows the trend and indicates that in some FFS, women outnumber their male companions. Table 13: Gender ratios by zone REGION Comm unes FKTY FFS groups FFS HH % HH FFS Number FFS members Wom en % Men TOTA L AMORON'I MANIA 26 157 999 12,059 48% 6,564 47% 7,416 13,980 VATOVAVY FITOVINANY 38 180 1,291 15,186 28% 8,554 47% 9,719 18,273 ATSIMO ATSINANAN A 6 57 365 5,012 39% 4,000 59% 2,800 6,800 ATSINANAN A 9 40 238 2,802 31% 1,952 50% 1,985 3,937 ANALANJIRO FO 11 33 392 2,932 38% 2,234 60% 1,497 3,731 ANOSY 6 45 284 4,308 46% 2,724 54% 2,286 5,010 ANDROY 16 80 460 4,452 36% 3,286 49% 3,369 6,655 TOTAL 112 592 4,029 46,751 36% 29,314 29,07 2 58,386 CRS/MG SALOHI MYAP FE/USAID-FFP-2-00002 51 1.5. Conclusion and Key Gender Recommendations To sum up, the inclusion of gender in the SALOHI program had a positive impact on the quality of its implementation. The relevance of the program was reinforced by the consideration of the problems and needs of men and women, especially FHH, and a willingness to provide appropriate responses to it. This conclusion is drawn mainly from the results of the Qualitative Study and a generalization to all SALOHI intervention areas is not appropriate. However, there is strong evidence that the movement toward greater gender equity may be slow but a project such as SALOHI can be a motivating factor. Key recommendations include:  Establish a community mechanism (e.g., an informal group) to ensure the preservation of gains in gender equity and to promote additional changes.  For program design: ensure that gender analysis is an integral part of program design and included in all surveys. This will facilitate the development of a complete gender strategy that can be mainstreamed for program staff.  Hold project staff accountable for gender results.  Do not hide behind the shield of: “It may not be culturally acceptable to promote gender equity”. Development entails changes in awareness and in behavior.  A project should have a specific communication strategy for gender, with explicit materials detailing why the project promotes gender equity, what the benefits are and how each individual can contribute.  Include a women’s empowerment process in the project, including 1) support for helping women to organize themselves, create groups, and strengthen their cohesion and solidarity and 2) a component for women’s productive activities to enable them to have access to land and opportunities to increase their revenue – and control over the revenue. 2. Environment 2.1 Brief Description of the Environment Interventions Environmental activities within SALOHI are cross-cutting, including all three SOs. Activities are based on the “Go Green Strategy” developed by the SALOHI program. “Go Green” is based on anchoring one or several environmental actions in the community through small scale activities with repetitive implementation and evaluation until it becomes an “environmental reflex”. The activities, defined and managed by the community, include: 1. Environment and SO1: Better personal hygiene, including hand washing; clean house and yard; clean kitchen utensils; using clean water for household consumption; garbage pits; composting kitchen waste; latrines; and use of a sustainable fuel wood source. 2. Environment and SO2: Promoting environmentally-friendly farming techniques such as using compost, agroforestry, integrated pest management (IPM), soil and water conservation, irrigation, reforestation, and the prohibition of slash and burn practices. 3. Environment and SO3: Protection against soil erosion and infrastructure damage from water and wind; reforestation; the application of dina for reforestation and management of infrastructure; and reinforcing traditions that protect natural resources. CRS/MG SALOHI MYAP FE/USAID-FFP-2-00002 52 2.2 Evaluation and Analysis of Environmental Activities All 592 fokontany were evaluated by program staff based on how well communities performed on environmental practices promoted within the 3 SOs, with green being a strong rating, yellow being average and red being a weak or low rating. The maximum score is 25 points. As Figure 21 shows, the overall result was: 27% of fokontany were “green”, 9% yellow and 24% red. The two best zones were the SE2 (Land O’Lakes zones) with the highest percentage of “green” fokontany at 44% and no fokontany in the red group, and the East (CARE and CARITAS) with only 5% red fokontany. Figure 22: Go Green By the end of June 2013, SALOHI had already met or exceeded the LOA targets for three IPTT indicators that are directly related to environmental activities: Table 14: SALOHI Environmental Indicator Status, Target vs. Achieved LOA, 2013 Indicators Target LOA Achieved LOA Success rate HA of land reforested or protected 1300 4,295 330% % of roads constructed using improved environmental practices 90% 100% 111% % of water systems constructed which are protected from erosion 90% 97% 108% Although targets were met, the FQS showed that at most, only 18% of survey respondents participated in one or more SALOHI environmental activities. 0% 20% 40% 60% 80% 100% Go Green Contest RED/WEAK YELLOW/MEDIUM GREEN/ GOOD CRS/MG SALOHI MYAP FE/USAID-FFP-2-00002 53 Figure 23: Proportion of HH participating in environmental activities (FQS 2013) However, as the figure below illustrates, environmental activities have had a noticeable impact on communities, according to survey respondents. They especially noted a decrease in bush fires, improvements in forest cover, and better soil fertility - all essential elements for better agricultural production and improved food security. Figure 24: Environmental changes in communities (FQS 2013) Utilization of infrastructures: The percentage of HH using infrastructure rehabilitated or developed by SALOHI is fairly consistent across all zones: 58% for roads, 24% for village sanitation, 16% for irrigation systems, and 7% for reforestation. Also showing little variation across zones were the benefits respondents cited: improved access to the community (cited by 60%); improved access to food (22%); improvements to health (18%); improvement of access to basic services (10%); and increase in cultivated land (10%). 18% 3% 1% 7% 4% 5% Reforestation Erosion prevention Sand dune stabilization Environmental sensitization Environmental training Fight against bush fire Proportion of HH participating in environmental activities 56% 44% 27% 15% 13% 13% 9% Decrease in bush fire Improvement of existing forest Improvement of soil fertility Land slide prevention Erosion prevention Flooding prevention Improvement of water resources Environmental changes in communities CRS/MG SALOHI MYAP FE/USAID-FFP-2-00002 54 2.3 Conclusion SALOHI environmental activities have been highly relevant because they helped beneficiaries to 1) manage external shocks better, 2) maintain infrastructure in good condition, 3) improve agricultural productivity, and 4) improve community cleanliness for a better living environment. The direct impacts of environmental activities have primarily been 4,294 hectares of reforested land, a decrease in bush fires (especially in the SE1/SE2), the reduced siltation of rice paddies, and the mastery of technical elements for setting up and managing plant nurseries (especially in SE1) and reforestation (especially in the South and SE1/SE2). One of the indirect impacts of environmental activities has been a strong sense of ownership of infrastructure as a result of the investment of labor during FFA activities. Other indirect impacts include: better understanding of the relationship between climate change, shocks and the environment; increased household income through better access to markets due to well￾maintained roads (collectors come directly to the village with vehicles); a perceived decrease in epidemics after storms; and improved health through better environmental sanitation, including a reduction in the practice of open-air defecation. In terms of sustainability, two examples illustrate why the impact of environmental actions will likely continue. First, reforestation efforts are very much appreciated by the beneficiaries; they now have a better understanding of the importance of forest cover and have a keen desire to pass on better natural resources to the next generation as part of their heritage. In one case, reforestation constitutes a source of income for the community to ensure the maintenance of their public infrastructures. Second, sustainability of community infrastructure is highly likely as beneficiaries are motivated to maintain canals for increased agricultural production and to maintain roads, which make communities less remote, improve access to markets and basic services, and increase community pride. 2.4 Recommendations It is clear from the final evaluation that the best point of entry for environmental actions is through increasing HH and community resilience to shocks. For this reason, the following steps are recommended:  Conduct a context analysis in relation to each shock the beneficiaries face.  For each shock, help communities to develop environmentally-friendly solutions.  Develop an Action Plan that includes awareness raising, activity monitoring and communication tools. For effective communication on environmental issues, it will be important to:  Build on local environmental knowledge to enhance community environmental knowledge.  Use a communication approach geared to a largely illiterate population.  Use practical exercises but reinforce them with appropriate visual aids.  Promote exchange visits to encourage a spirit of innovation. 3. Good Governance Initial plan: In the proposal design, SALOHI anticipated to strengthen the capacity of community committees to work on their food security needs. The plan was also to work on the capacity of local authorities to collaborate and cooperate with communities by responding to their food security needs and complementing their efforts. However, due to USAID restrictions on working with the CRS/MG SALOHI MYAP FE/USAID-FFP-2-00002 55 Government of Madagascar, capacity building activities with local authorities were suspended and the SALOHI team dropped commune level activities. The IR was reformulated to reflect the focus on the governance capacity of community-based organizations and strengthened partnerships, rather than commune level governance. In the updated DIP for FY12 ARR, IR 3.4 was reformulated as follows: “Activity cancelled. During the midterm evaluation review and as a result of the ongoing political crisis in Madagascar and USG regulations which prohibit USG support to locally elected officials, it was decided that the program will not strengthen the capacity of communal authorities or support the adoption of local development plans by commune authorities, but will focus on strengthening the capacity of community groups and community based organizations to influence decisions which affect their food security”. SALOHI themes of Good Governance: SALOHI selected three main governance principles for the promotion of community governance: 1) equitable participation of men and women in decision making (addressed above in the gender section), 2) transparency in terms of beneficiary selection, group membership and resources management, and 3) accountability of community committees and associations to all community members. The governance strategy included clarifying the required actions, roles and responsibilities of each member of community associations (DRR, AUP, AUE, VSL, PL, GRN, etc.). The objective of governance activities was to improve participation, transparency and accountability in all SALOHI groups and associations. Good Governance and the three Strategic Objectives: Like environment and gender equity, Good Governance is a cross-cutting objective that cuts across all three strategic objectives. This includes improving processes in the formation of groups in SO1, SO2 and SO3 to ensure transparency, democratic processes, a good gender balance, and women’s participation in decision￾making. Achievement - Elected members: Evaluation team members found that members of the different committees and associations (DRR, IMA/GRN, AUP, AUE, PL, FFS, etc.) have been elected and not appointed by community members or a political body. This is the first indication of transparency. Men and women are both represented even though the ratio is approximately 70/30 in favor of men for SO3. Workers for FFA activities are not randomly chosen but both men and women are selected using vulnerability criteria established by project staff in discussion with local communities. Some communities have even put in place a “Comité de Ciblage” and brought transparency and participation to another level where decisions are unanimous. Food rations are equally distributed even though women are given easier tasks. Transparency in food distribution is so effective that the evaluation team didn’t hear of any disputes resulting from cheating or mismanagement of food commodities at the community level. Selection of AVs, CHVs and PL’s are conducted in a transparent manner (SO1, SO2). Leaders for FFS, VSLA, AB (SO2) are also democratically elected. These practices have been well-integrated into all committees and association processes. All communities who participated in FFA activities have put in place Infrastructure Management Associations (IMA) which are also responsible for the management of natural resources to protect those infrastructures. Roads and canals are rehabilitated or constructed taking into account environmental protection measures. Trees and grass are planted on road-sides and canal banks to protect against erosion. Community members are aware of environmental issues to monitor during infrastructure maintenance work. Some dinas have provisions to fine violators swimming or defecating in canals. This reflects local responsibility and ownership. As beneficiaries in more than one community said: “We are doing all these activities for CRS/MG SALOHI MYAP FE/USAID-FFP-2-00002 56 our children, the next generation”. Combined with transparency, this type of thinking has brought more social cohesion in communities since the implementation of SALOHI. Challenges: Some aspects of Good Governance did not work very well and have remained a challenge for all three SOs: accountability and the communication of progress on ongoing activities or information on new interventions on the part of the various committees to the wider audience is not systematic. Committee members consider their daily informal interaction as enough. As one DRR member pointed out: “we are in the same village and see each other every day”. The situation is worsened by the fact that there is very little in writing. It is recommended to ensure that SALOHI groups and community based structures (i.e. FFS, VSLA, PD/Hearth leaders, PLs, DRR, AUP, AUE, IMA, etc.) set a timetable for meeting in their individual committees, produce minutes from each meeting, and set up planning for future activities with the involvement of community members. This lack of accountability results in a very low percentage of the population who know the contents of their community’s DPMP. The other challenge is the presence of the same members in multiple committees. Wearing many hats is potentially a recipe for biased decisions and sometimes it may result in a lack of dedication to a given sector or assignment among many others. When U.S. policy towards the Government of Madagascar (GOM) changed following the coup d’état in 2009, it was no longer allowable to provide any financial compensation or technical assistance to GOM employees or institutions. Nonetheless, the evaluation team found that mayors and local officials in communities visited (Anasivelo, Manambotra, Beanana/Ambila, Mideboka/Marofariny, Ambodifandramanana, etc.) were very appreciative of SALOHI interventions, and they were well informed about program activities and benefits. However, partnerships formed differed from one community to another. Therefore, commune-level support to community committees and associations is not systematic and needs to be improved with or without political restrictions. There are avenues to explore like the inclusion of commune authorities in meetings with community members (in SALOHI communities) where issues on good governance will be discussed. No per diem for local authorities is required. The separation of SALOHI activities from local authorities may not yield good results as community development plans (DPMP) need to be approved by and to receive support from local authorities. By participating informally in SALOHI meetings, local authorities will also learn about democratic processes. More importantly, strengthening partnerships between communities at the field level and elected authorities at the commune level is instrumental to put into place mechanisms for the management of toll fees collected by road users associations (AUP) for the maintenance of rehabilitated roads. It can become a sustainability issue if not addressed. 4. Partnership and Sustainability One of the key themes investigated during this evaluation was sustainability, specifically looking at to what extent the impacts and behavior change promoted by the SALOHI program will continue after the project ends. As defined in the SALOHI Sustainability and Exit Strategy (Version 4, November 2012), “Sustainability involves making the effects and impacts of program activities last. It is not the activities that should necessarily continue, but the EFFECTS of the activities that should continue (adoption of behavior change, or provision of services, for example).” CRS/MG SALOHI MYAP FE/USAID-FFP-2-00002 57 Preparing for Sustainability The SALOHI team has done an excellent job preparing for sustainability: A sustainability strategy was developed with the project proposal and revised by SALOHI staff annually. In 2011 the annual revision process also included input from 1) beneficiaries and local stakeholders, 2) the SALOHI Advisory Board, and 3) the country directors of the four implementing partners. Monitoring of the strategy is conducted on a quarterly and annual basis, with regular updates to USAID via Annual Results Reports (ARR) and PREP submissions. The strategy itself is exceptionally detailed, with concrete actions and a timeline to follow. It is more a manual than a strategy, providing clear guidelines to field staff and managers and numerous examples to inspire implementers. The strategy was rolled out to 100 SALOHI team members at the SALOHI annual workshop in Fort Dauphin in 2011 and to SALOHI field agents in 2012. Sustainability “cheat sheets” were developed to simplify the process for field agents, and a separate monitoring system developed in 2012, to evaluate community based structures and promote “graduation” and withdrawal. To gather further information on what helps to ensure sustainability, SALOHI conducted a post￾project sustainability assessment of previous food-assisted programs in Madagascar to determine whether the impacts and effects had continued and if so, what the key factors were that helped ensure sustainability. The results of this assessment were widely shared among SALOHI staff to help them understand factors that facilitated sustainability in previous projects, which could be applied to SALOHI. Another proactive step that SALOHI staff took was to organize a sustainability workshop so that the entire implementation team would have a common understanding of sustainability in the SALOHI context and how to achieve it. Finally, SALOHI paid particular attention to the individuals and groups who will be responsible for helping beneficiaries maintain and even increase the positive effects of their participation in the SALOHI program. Significant time and resources were devoted to training people in every SALOHI community, training that was very much appreciated by those who participated. As stated in the Sustainability and Exit Strategy, the goal of this investment in training and coaching is to “transfer capacity, responsibility and power”. To assess whether individuals and groups are ready to assume responsibility, thus enhancing the potential for sustainability, SALOHI developed a very useful and user-friendly Guide pour la Graduation des Structures Relais au Niveau Communautaire, complete with tools for gauging the readiness and capability of each type of volunteer and community structure. Regular progress reports were made on what percentage of groups was ready for graduation in each project zone. Assessing the Potential for Sustainability The tools for both the FQS and the Qualitative Study included questions about the potential and necessary conditions for sustainability for each SO. Since effective partnerships are crucial to the long-term sustainability of many of project impacts and effects, the assessment also examined the quality of partnerships that SALOHI has engaged in. These include administrative partnerships (with authorities and leaders for example); technical partners such as other projects, NGOs (both international and national) and government services; and commercial partners, especially for SO2 activities in agriculture and VSLAs. Below is a summary of findings by SO. The sustainability potential for Gender is discussed in the preceding Gender section SO1: Beneficiaries and project staff are confident that a majority of the SALOHI participants will continue to practice many of the new behaviors they adopted as a result of SALOHI; these include nutrition and hygiene behaviors especially. Beneficiaries are also confident that many fathers will CRS/MG SALOHI MYAP FE/USAID-FFP-2-00002 58 continue to play a role in their children’s health after SALOHI because they are convinced that it makes a difference to their children’s health. And although it is unlikely that all CHVs will continue to be active post-project, there is a corps of dedicated volunteers who will remain active, organizing GMP and perhaps PD/Hearth-type sessions, providing advice, and organizing community IEC sessions. The FQS showed some promising results in terms of the sustainability of the impact of SALOHI on the health and nutrition of women and children. For example, over 90% of the women interviewed gave a resounding vote of confidence to CHVs, stating that the volunteers in their communities have both the competencies and the motivation required to help maintain the impact of SO1 activities. In terms of rating their own competencies, the majority (65%) of mothers said they have the competencies but not necessarily the resources (only 34% said they have the necessary resources to maintain the positive effects.) Almost 80% of the respondents believe that local authorities are sufficiently motivated to provide support and encouragement to the CHVs. These are all positive indications for sustainability. In terms of establishing effective partnerships to reinforce the potential for SO1 sustainability, the logical partner for SO1 is the Ministry of Health, specifically the CSBs and health teams at the district and regional levels. Unfortunately, the USG injunction against working with Malagasy government entities severely handicapped these partnerships. In short, other than other projects such as SALOHI that might be active in some areas, there are few viable permanent partnerships at this time to reinforce sustainability. SALOHI staff do participate in regional nutrition and health cluster meetings, and in some regions and districts there are health projects funded by USAID (Mikolo, PMI, etc.), or the EU (AINA), UN activities (especially UNICEF, WFP and UNFPA), ONN activities, or NGOs (GRET, RTM, MSF, etc.) with whom SALOHI CHVs can receive additional support after program withdrawal. SO2: The FQS revealed that the proportion of the population committed to continue SO2 activities/practices is 63% for the use of agricultural techniques (of whom only 40% actually participated in FFS groups; the remainder are secondary adapters); 45% with VSLA (only 39% of the population actually participated in VSLAs); and 11% with AB (9.4% participated in AB). These figures demonstrate that even people who did not participate directly in activities have a strong desire to use agricultural techniques promoted by the program and to participate in VSLAs beyond what was accomplished during the life of the project. However, beyond sustainability, a project also needs to take into account the post-project environment not only for the sustainability of impact and practices, but also for the replicability of activities and practices for those who were not direct project beneficiaries; these are usually people living in or close to project areas but sometimes outside of target zones. FFS are principally a method to encourage farmers to innovate and adapt improved agricultural techniques and they need not be sustainable in and of themselves, unless they also provide other socio-economic benefits such as FFS group sales. What is more important is that farmers in a community not only continue to apply what they have learned in these groups, but that they keep experimenting with different techniques (especially in response to weather and climate change) and keep learning from each other. For SALOHI, the continuation of acquired practices is very sustainable according to FGD and KII participants, as the practices introduced and developed have clear advantages and do not require CRS/MG SALOHI MYAP FE/USAID-FFP-2-00002 59 external inputs (save seed, which can be acquired locally). FFS groups themselves will continue only if they benefit from a clear socio-economic advantage. For example, some groups cultivate beans together which they sell in order to buy and store rice which they then distribute to FFS members during the lean season. Other FFS groups sell a group crop and use the money partly as individual dividends and partly for further group activities such as purchasing seeds and other inputs, tools, and labor. VSLAs have been sustainable the world over for many years, and so they are with SALOHI, as indicated by the graduation rate and from the FGDs. They are also very self-replicable with the right approach. The lack of attention to replicability of VSLAs is a weakness found in SALOHI NGOs, particularly ADRA and LOL. There are three reasons to promote VSLA replicability: 1. There is a proven international model for implementing VSLAs and the SALOHI NGOs used this rigorous but flexible model with minor adaptations. 2. After a few months, VSLA members see the many benefits of the VSLA, starting with social cohesion, access to the solidarity fund for emergencies, access to credit (for both productive and consumptive activities), and access to savings after a full cycle (usually ten months to one year). 3. In the international model, the NGO provides only training and technical support as all the financial and material resources needed should be provided or paid for by the members themselves. This is a weakness within SALOHI that affects VSLA replicability because:  SALOHI groups were given VSLA kits instead of putting in place a system of reimbursement to a VSLA support association or paying forward (buying a kit for a new group).  During the life of the project, all VSLA village agents (VA) need to demonstrate their capacity to start up and support new groups and they should receive an incentive from the new groups for their services. There has been some effort to facilitate replicability by (1) establishing VSLA networks to access start-up kits and other technical support and (2) encouraging VAs to start new groups. But this effort is not systematic. Ensuring sustainability is hardest for the AB activity, as ABs need consistent group work, good communications, and prompt adaptability to external factors. For SALOHI, the sustainable ABs will only be the ones active and already well-developed at the end of 2013, about 50% according to SALOHI. Those groups that have only received training to date will not be able to grow into functional ABs without sustained external support. As for partnerships and their role in sustainability, SO2 activities were particularly strengthened through partnerships, especially for agriculture and livestock activities; this is likely to have a positive effect on sustainability. The partnerships were primarily of four types: 1. Agriculture Service Centers, primarily for market information (purchase and sale); 2. Other national/regional projects, such as AROPA, PIC, PROSPERER and NGOs such as GRET (for seeds) and GTZ, primarily for inputs, but also for training; CRS/MG SALOHI MYAP FE/USAID-FFP-2-00002 60 3. Commercial actors (such as buyers for ABs) and microfinance institutions or telecommunication companies (mobile money) to provide cash security for some VSLAs (instead of keeping cash in a box); 4. Unions and federations of VSLA for sustainability and growth, promoted by SALOHI for VSLAs and AB groups. Of all the SO2 activities, the AB are the groups that have most needed and used partnerships, for inputs, marketing, finance and business support. There was good collaboration between SALOHI and local government services such as the ASC, DRDR and DREL, though collaboration was constrained by lack of resources (GOM) and the USAID injunction on funding government entities. This injunction had negative repercussions as some government entities (such as the ASC) have as part of their mandate promoting participation in national initiatives and international projects and facilitating linkages between buyers and suppliers. SO3: There is good potential for sustainability of the impact and effects of SO3 activities. The main condition created by SALOHI is an increased social cohesion due to the construction and maintenance of infrastructures and less conflicts due to good management of irrigation canals. The conclusion is based on the following facts: 1. All communities have a trained DRR committee and several types of management associations: Road Users Associations, Water Users Associations, Natural Resource Management (NRM) and Infrastructure Management Associations (IMA) with elected and not designated leaders. 2. There is a local responsibility and ownership. As beneficiaries said in more than one community: “We are doing all these activities for our children, the next generation”. 3. Of FGD respondents, 88% do not think that maintaining infrastructures will pose a problem. 4. There are established AUPs and AUEs (community associations) whose main role is to organize the maintenance of the infrastructures. 5. Some communities are collecting toll fees on rehabilitated roads from merchants and all communities have dina to generate funds. In both cases, the money will be used for maintenance work. 6. The main challenge in the management of those funds is risks of lack of transparency. The associations or committees in charge the funds need to be coached on how to maintain a simple ledger and how funds have to be disbursed in a very transparent way. It’s advisable to involve or inform local authorities. 7. Associations have already developed schedules for the maintenance of irrigation systems and roads. 8. There a better sensitization of the community about disaster preparation i.e. Containers for storage of drinkable water (provided by SALOHI), movement of animals to higher grounds, reinforcement of house structures (sides, roofs) etc. 9. Pre-identification of safe shelters and other places for better safekeeping of personal items 10. Existence of a reliable warning communication system (with megaphones, flags and FM radios provided by SALOHI) 11. Availability and commitment of village volunteers to secure people and property and for crisis management 12. Construction of community granaries to save part of their harvests (very few) 13. Annual simulation exercises CRS/MG SALOHI MYAP FE/USAID-FFP-2-00002 61 14. More community participation in developing the DPMPs will be an asset as well as expanding the DPMPs to development activities (not emergencies oriented only). Conclusion SALOHI has devoted considerable attention to sustainability and to using the final year of the project as an exit year to consolidate achievements and to prepare individuals and volunteers to assume responsibility to ensure that the effects of SALOHI interventions continue. The only concern at this time is that some activities requiring a minimum of two years for full development (like agribusiness) may have started late, and the full cycle of training and practice may not be completed before the project ends. This is especially true in areas where NGOs chose to integrate communities into the program in successive waves, with the result that the last wave of communities may not receive the full benefit of SALOHI activities due to time constraints and to the fact that the number of field agents has been reduced in Year 5, to support program withdrawal. Nevertheless, there is strong potential for sustainability and other than starting some activities sooner, it is difficult to see how SALOHI could have done more to pave the way. As this assessment took place several months before the end of the project, the evaluators cannot make a determination at this time about whether the sustainability strategy will be successful. A strong recommendation is to carry out a post-project sustainability study after one year, to further inform future sustainability strategies. F. Program Strengths and Weaknesses A number of the strengths and weaknesses of the SALOHI program have been discussed in the SO sections and under the cross-cutting themes. The following lists summarize additional strong points and weak points that apply to the overall program and to specific interventions. Strengths: - The design of the program coupled with good implementation facilitated close integration among the SOs and with the cross-cutting themes of gender and environment. - Ongoing research, studies and analysis (e.g., the 2011 barrier analysis study) were widely shared among the SALOHI team, improving implementation and fostering innovation. - Strong, proactive project leadership from the Chief of Party and the current Deputy Chief of Party ensured that project implementation is on track, problems are identified in a timely manner and plans developed to resolve them. - Based on the SALOHI team’s response to the midterm evaluation, it appears that the team on the whole is receptive to feedback and suggestions for improvement. There was a sustained effort to respond to the numerous recommendations from the midterm evaluation and to hold everyone accountable for timely course corrections. - Investments were made in staff development, resulting in a corps of better-trained Malagasy development workers. - Good performances on the individual, community and NGO levels are routinely recognized and rewarded. - According to the beneficiaries, participating in SALOHI has fostered greater social cohesion, especially as people become more used to working in groups for their own benefit and for the development of the community. CRS/MG SALOHI MYAP FE/USAID-FFP-2-00002 62 - Beneficiary capacity building is a priority for individuals and groups, an approach mentioned by a number of beneficiaries during the FQS and the Qualitative Study. - Sufficient attention has been paid to sustainability: The strategy was developed before implementation and refined several times; it includes “graduation” criteria for community volunteers and groups. - SALOHI activities are relevant and responsive to beneficiaries’ needs. - The project is exceptionally well-documented. - SALOHI has demonstrated efficiency and a good use of resources, reaching 138,000 more beneficiaries without an increase in budget. - There is good targeting of (1) geographic areas, including more remote areas not usually reached by development activities, and (2) beneficiaries, with a particular focus on reaching women. There is no apparent exclusion or discrimination. - SALOHI has developed many useful tools and guides to aid staff in program implementation - materials that could be easily adapted to other projects. - SO2: The integration of FFS/VSLA/AB, especially FFS/VSLA, was a key success factor and an improvement from the midterm findings. This was particularly the case for FFS that were started in 2010 as they had the time to fully develop AB activities, particularly with CRS. - SO2: There was a high direct participation in agricultural activities (FFS) and a significant number of indirect agricultural beneficiaries (spill-over effect). During the Qualitative Study, everyone interviewed knew about the new techniques whether they were SALOHI beneficiaries or not. - An important integration between SO2 and SO1 has been the combined work of CHVs and FLs to help SO1 women’s groups to grow vegetables. - SO3: There is increased social cohesion due to the construction and maintenance of infrastructure and less conflict due to good management of irrigation canals. - Environment: Valuing and using local knowledge (e.g., use of local materials for nursery bags and the promotion of local species) promotes ownership of reforestation efforts. - Gender: Promoting the participation of men in health and nutrition activities not only brings benefits to children but also helps to break down gender barriers. - Holding contests is an excellent way to motivate beneficiaries, community volunteers and project staff. - SO1: SALOHI made a concerted effort to increase the number of CHVs, facilitating a wider dissemination of messages and increasing support to mothers and caretakers. - SO1: Building on lessons learned and best practices of other projects was an efficient and effective way to jump start SO1 activities. Weaknesses: - Community volunteers are central to both implementation and sustainability yet there is no systematic effort to develop mechanisms for their own communities to reward them on a regular basis (some individual communes have great initiatives, but there is nothing systematic across all communes). - The sustainability strategy was well-designed but is not being systematically or completely implemented in some communities, especially for VSLAs and the recruitment and mentoring of VAs. - In some areas there were insufficient staff at the beginning of the project and at least one NGO let field staff go early in Year 5. There is concern about quality implementation when there are too few staff. CRS/MG SALOHI MYAP FE/USAID-FFP-2-00002 63 - The gender theme was not made explicit and insufficient attention was paid to the gender strategy by implementing partners. Staff were not held accountable for ensuring progress on the gender front. - SO2: Efforts to address the particular needs of FHH were not systematic, particularly for livelihood activities. There was insufficient attention to issues such as access to land and providing alternative livelihood activities such as poultry raising, bee-keeping, and fruit/spice trees for these women. - Across all three SOs there was at times a lack of accountability on the part of committee members, e.g., failure to report back to the community. This applied also to a number of beneficiaries who went on exchange visits. - Generally speaking, the SBCC strategy does not seem to be well-developed or creative; implementation is inconsistent and didactic materials are inadequate in some sectors. - SO3: Community participation in developing the DPMP was not evident. In addition, the DPMPs were oriented toward emergencies. - SO3: The Food Security Monitoring System was not well-implemented. G. Recommendations Certain recommendations specific to an SO or a cross-cutting theme are already included in previous sections of this report. The following are additional suggestions intended for (1) the SALOHI team implementing the final year of the project, (2) local stakeholders, and (3) those designing new programs. For the SALOHI team and local stakeholders (before the project ends): 1. Hold a workshop with all the implementers and reflect on “other” lessons to be learned from unanswered questions and recommendations from the final evaluation, such as: field staffing (numbers, organization and management, qualifications, responsibilities and length and period of service); the pros and cons of using non-specialist field staff and paying community volunteers (ADRA); the pros and cons of the three different AB approaches; and the use of food such as FFT for trainees at the Social protection Centers. 2. SO2: Ensure that FLs and VAs have official recognition and certificates of competence as requested by them to give them more credibility and opportunities to assist others. 3. SO3: Help communities and communes put in place mechanisms for the transparent management of toll fees and fines; this is crucial for infrastructure maintenance. 4. Gender: o Organize communication campaigns to share or reinforce what the program has done for the promotion of gender equity and the underlying reasons for including a gender dimension in SALOHI. o Explain why SALOHI promoted the participation of men in SO1 activities: existence of gender disparities, men's participation has a positive impact on the state of health and nutrition of children. o Give concrete positive examples from SO1 and cite examples from deserving communities to encourage emulation. Do the same for SO2 and SO3. o Help interested communities to establish a community mechanism to ensure the preservation of the gains in gender equity and their continuity. It is not a matter of putting in place an additional structure but to seek the involvement of people of good will who are committed to gender equity and who are interested in being mobilizers for the promotion of gender equity. Influential people such as traditional leaders, CRS/MG SALOHI MYAP FE/USAID-FFP-2-00002 64 local authorities, community leaders and Sefom-biavy (traditional women leaders) are to be encouraged to take this role. Similarly, if there are associations of local women, their representatives should be invited to participate. In practice, this small team will animate exchanges and discussions within the community. For new program design: 1. During the design phase, focus not only on the sustainability of project activities and outcomes, but also on post-project replicability of activities such as VSLAs and agricultural and marketing experimentation and learning. 2. Work with schools for all SOs and some cross-cutting themes such as environment and gender. 3. Ensure that NGO field staff receive training on general development approaches such as adult learning, group dynamics, quality assurance, and post-project sustainability and replicability. Well-trained field staff can, in turn, lead FLs to understand and promote experimentation and adaptation of the “new techniques” to local conditions. For example, SRI with rice is the combination of several independent techniques for a cumulative effect. Even if some farmers cannot apply all the techniques, especially where control over water resources is a constraint, they can be encouraged to apply some of them, such as planting in rows, regular spacing, and weeding. 4. For improved efficiency and efficacy, human resources should be based on pre-determined ratios of facilitators to participants. For example, FLs and VAs can only assist 5-6 groups each and community volunteers should not have more than three roles (preferably inter-SO). NGO field staff should cover only a reasonable number of communities and groups and avoid being too stretched geographically. 5. If planning to use the final year as an exit year to improve sustainability, ensure that is not used as a “catch up” year to reach quantitative targets. This will require careful attention to the implementation schedule, building in the usual project delays and allowing adequate time for late joiners to complete activities, especially for VAs and VSLAs with a 12-month cycle. 6. Start activities in at least 80% of communities in the first year. Avoid second waves for activities which require a longer time to implement (i.e., more than one year such as for VSLAs/VAs and more formal ABs). 7. Perform a progress evaluation 12 to 18 months after the start of activities so as to maximize the time to make improvements before the end of the project. A MTE conducted 30 months after start-up leaves only 24 months to make course corrections before program draw down. 8. For key documents with findings and recommendations such as the 2011 Barrier Analysis and Annual Survey, provide an equivalent reply document addressing the original findings and recommendations. Ensure systematic follow-up as was done so well for the midterm evaluation (MTE). 9. Minimize food distributions that undermine motivation and sustainability such as Food for Training. If such food distributions are implemented, ensure that they are associated in return with some form of contribution from the recipients: an activity, some behavior change, a knowledge test, etc. 10. Start with the integration of all SOs from the beginning with respect to project participants, especially MCHN women’s groups, as one of the key objectives of the project is to improve the nutritional status of young children and PLW. CRS/MG SALOHI MYAP FE/USAID-FFP-2-00002 65 11. Use pre-and post- testing for all significant trainings for staff and beneficiaries. This provides instant feedback on both the participants and the facilitators’ performance, contributes to quality and accountability, and ensures that all take the trainings seriously. 12. Ensure economic and social sustainability and replicability of groups and community volunteers. The recipients who benefit from community volunteer support should be strongly encouraged to provide the volunteers with small rewards for their services such as food if they had to travel far, money from the VSLA in the case of VAs, or part of harvests in the case of FLs and CHVs. All VSLAs should “pay” for their kit and put in place a system of making kits easily accessible to new groups (either through a VSLA network or through each VSLA.) All FLs should know where and how to get seed/seedlings/cuttings for the crops promoted. Any group (such as farmers) receiving “free” seeds and tools should know how to maintain and replace their stock of free gifts and not to expect other gifts. 13. While the AB component of SO2 is worthwhile, it only applies to a minority of the population and its title is bigger than the reality. What might be preferable is to include “marketing” as part of both FFS and VSLA modules (this would include basic agricultural accounting, crop planning and quality management) and to develop “AB” fully only if a value chain already exists and if the community context is conducive. 14. Include simple appropriate technologies to solve common problems. Examples include: treadle pumps for vegetable farming near bodies of water; drinking water solutions (a common issue raised in FGDs) such as IDE’s Ceramic Water Purifier; methods to promote hand washing such as Tippy Taps; ways to weigh crops using a standard water container (20 liters) and a lever and fulcrum if scales are not available; and chicken coops made from local materials (as was promoted in SALOHI). 15. The dissemination of knowledge across the three SOs was not systematic and seemed to have occurred ad hoc through those community volunteers who were active in more than one SO. A simple plan, perhaps even a “curriculum” of what messages to reinforce during each type of activity, would be helpful for ensuring that key SALOHI messages reach all beneficiaries (as included in SALOHI SO strategies). H. Bibliography of Key Documents Consulted 1. SALOHI Final Proposal submitted to USAID, April 2009 2. SALOHI ARR: FY09, FY10, FY11, FY12 3. SALOHI DIPs for 2009-2013 4. SALOHI Baseline Survey Results Report (October – December 2009) 5. SALOHI Annual Survey (July/August 2011) 6. SALOHI Midterm Evaluation Report (2012) 7. SALOHI implementation strategies (SO1, SO2, SO3, Governance, Partnership, Communication, Environment, Gender) 8. SALOHI Year 2, Year 3, Year 4 PREP 9. SALOHI Activity Briefing Papers 10. SALOHI operational research (2010 – 2013) 11. Guide pour la Graduation des Structures de Relais au Niveau Communautaire 12. A Study on Local Determinants of Malnutrition; SALOHI Project in Ambositra (ADRA) 13. Manuel de Mise en Oeuvre de Réflexe Environnemental: Stratégie Nde’ho Maitso pour le Programme SALOHI (Ramanase Zoelimalala, Environmental Consultant) 14. Analyse Socio-économique des Barrières à l’Adoption du Système de Riziculture Intensive (Rakotoniaiana Andriantoky, 2013) CRS/MG SALOHI MYAP FE/USAID-FFP-2-00002 66