Sustainable Nutrition and Agriculture Promotion (SNAP) Program A Multi-Year Assistance Program (MYAP) implemented by ACDI/VOCA, IMC and OICI in Sierra Leone FINAL EVALUATION REPORT March 2017 This publication was produced at the request of the United States Agency for International Development. It was prepared independently by International Solutions Group (www.theisg.com) under contract to ACDI/VOCA. SNAP Final Evaluation Report March 2017 International Solutions Group i Acknowledgement The members of the Evaluation Team would like to gratefully acknowledge the support provided by the staff of the SNAP Program. Special thanks go to the staff who went out of their way to provide logistical support to enable the team to implement an intensive schedule with a large team of consultants and translators. We would also like to thank the technical and implementing staff from the partners who were available for supporting the field work, participating in various events during the Evaluation, and providing insights while answering our multitudinous questions. Many program participants and beneficiaries also provided useful responses to our questions and participated in stimulating discussions. Our gratitude goes out to them as well. Hopefully we've listened well, our observations are grounded in reality, our assessment is accurate, and the lessons learned that we have identified will be useful for future programs. Sincerely, PHOTO CREDITS: Cover photos: and . All other photographs used in this report were taken by the Evaluation Team members during the Evaluation. DISCLAIMER The authors' views expressed in this publication do not necessarily reflect the views of the United States Agency for International Development or the United States Government SNAP Final Evaluation Report March 2017 International Solutions Group ii Contents I. EXECUTIVE SUMMARY ...............................................................................................................................6 II. BACKGROUND INFORMATION................................................................................................................10 A. Overview of the SNAP Program..........................................................................................................10 B. Contextual Factors Affecting Program Implementation or Impact ....................................................12 C. Evaluation Methodology.....................................................................................................................13 III. EVALUATION FINDINGS: PROGRAM THEORY OF CHANGE AND TARGETING ........................................18 A. Theory of Change................................................................................................................................18 B. SNAP Program Targeting.....................................................................................................................18 IV. EVALUATION FINDINGS: OUTPUTS PRODUCED UNDER EACH INTERMEDIATE RESULT........................21 A. Outputs Produced Under IR 1.1: Improved access to quality nutrition and health services for women and children aged 0-59 months.................................................................................................22 B. Outputs Produced Under IR 1.2: Improved health and nutrition practices at the household level ..28 C. Outputs Produced Under IR 2.1: Increased agricultural productivity at the household level ...........33 D. Outputs Produced Under IR 2.2: Increased access to financial and marketing resources.................42 E. Outputs Produced Under IR 2.3: Improved Community Assets .........................................................50 V. EVALUATION FINDINGS: PROGRAM OUTCOMES AND IMPACT ASSESSMENT.......................................53 A. Impact under SO 1: Reduced chronic malnutrition among children under five.................................53 B. Impact under SO 2: Enhanced livelihoods for vulnerable people, especially women and youth ......55 C. Collective Impact at the Development Objective-Level .....................................................................59 D. Cross-Cutting Theme: Resilience ........................................................................................................61 E. Cross-Cutting Theme: Youth Productivity...........................................................................................62 F. Cross-Cutting Theme: Gender Equity..................................................................................................62 G. Cross-Cutting Theme: Environment ...................................................................................................64 H. Cross-Cutting Theme: Governance.....................................................................................................65 VI. EVALUATION FINDINGS: PROGRAM IMPLEMENTATION SYSTEMS .......................................................66 A. Program Management........................................................................................................................66 B. Partnerships........................................................................................................................................67 C. Knowledge Management....................................................................................................................68 D. Integration & Complementarity .........................................................................................................71 E. Resource Management.......................................................................................................................72 V ..................................................................................................................................................................77 II. SPECIAL TOPICS .......................................................................................................................................77 A. SNAP Program Extensions ..................................................................................................................77 B. Impact of Ebola ...................................................................................................................................77 SNAP Final Evaluation Report March 2017 International Solutions Group iii C. Geographic Targeting Strategy ...........................................................................................................77 D. Equitability of Benefits Within Communities.....................................................................................78 E. Food-Related Dependency..................................................................................................................78 F. Opportunities for Synergies Between Food for Peace and Feed the Future Programming ...............78 G. Cost Effectiveness of Large Programs ................................................................................................79 VIII. CONCLUDING REMARKS & RECOMMENDATIONS...............................................................................80 Annexes.......................................................................................................................................................83 Annex A: Evaluation Scope of Work ...........................................................................................................83 Annex B Evaluation Operational Plan .........................................................................................................83 Annex C Endline Survey Report ..................................................................................................................83 Annex D: Supplemental Annual Results Survey on Agricultural Technology Adoption..............................83 Annex E: Summary of Persons Interviewed for Qualitative Data Collection..............................................83 Annex F: Final SNAP IPTT ............................................................................................................................83 Tables and Figures Table 1. SNAP Development Objective, Strategic Objectives and Intermediate Results Figure A. Life of the SNAP Program with Key Turning Points Table 2. Qualitative Information Gathering Respondents Table 3. Disaggregation of Selected Villages by Intensity of Intervention & Accessibility Table 4. PHU Staff and Community Members Trained by SNAP Table 5. IPTT Indicators for IR 1.1 Table 6. WASH Investments in SNAP Table 7. IPTT Indicators for IR 1.2 Table 8. IPTT Indicators for the FFS/FFS Cluster Component Table 9. IPTT Indicators Related to Technology & Knowledge Transfer Under IR 2.1 Table 10. IPTT Indicators Under IR 2.1 for Crop Yields and Sales Table 11. IPTT Indicators Under IR 2.1 Related to Cluster Business Relationships Table 12. IPTT Indicators for the VSLA Component Table 13. IPTT Indicators for the Literacy/Numeracy & Vocational Skills Training Component Table 14. Vocational Skills Training Participants Table 15. IPTT Indicators for IR 2.3 Community Assets Table 16. Performance Indicators for SO1 Table 17. Performance Indicators for Diet Diversity & Food Provisioning Ability Table 18. Performance Indicators for Household Revenue Table 18a. Household crop revenue and overall revenue weightings and comparisons Table 19. Performance Indicators for Non-Farm Income Table 20. Number of Direct Beneficiaries for Different Components of SO2 Table 21. Degree of Program Impact on Different Types of SNAP Households Table 22. SNAP Environmental Indicators in the IPTT Table 23. Financial Resource Summary Table 24. Commodity Summary for the SNAP Program Figure B. Timeline for SNAP Food Distributions Table 25. Monetization Cost Recovery Table 26. Commodity Loss Summary SNAP Final Evaluation Report March 2017 International Solutions Group iv ACRONYMS, INITIALISMS & ABBREVIATIONS ABC Agri-Business Center ANC Antenatal Care ARR Annual Results Report BCC Behavioral Change Communications BEmONC Basic Emergency Obstetric and Neonatal Care BSF Bio Sand Filters CF Community Facilitator CG Care Group CLTS Community-led Total Sanitation CMAM Community Management of Acute Malnutrition CoP Chief of Party CSB Corn Soy Blend CU Care Unit CU2 Children Under Two Years Old CY Calendar Year DHMT District Health Management Team DO Development Objective DOM District Office Manager DMO District Medical Officer EA Enumeration Area EBF Exclusive Breast Feeding ECD Early Childhood Development EVD Ebola Virus Disease FaaFB Farm as a Family Business FANTA-2 Food and Nutrition Technical Assistance II FFP Food for Peace FFS Farmer Field School FGD Focus Group Discussion FLEW or FEW Front-Line Extension Worker (MAFFS) FP Family Planning FY Fiscal Year GMP Growth Monitoring and Promotion GOSL Government of Sierra Leone HP Health Promoter IGA Income Generating Activity IMAM Integrated Management of Acute Malnutrition IMC International Medical Corps IMNCI Integrated Management for Newborn Childhood Illnesses IPTT Indicator Performance Tracking Table IR Intermediate Result ISG International Solutions Group IY Implementation Year IYCF Infant and Young Child Feeding KII Key Informant Interview LM Lead Mother SNAP Final Evaluation Report March 2017 International Solutions Group v LOA Life of Activity or Life of Award MAFFS Ministry of Agriculture, Forestry and Food Security MAM Moderate Acute Malnutrition MAP Men as Partners MCG Mother Care Group M&E Monitoring & Evaluation MTE Mid-Term Evaluation MYAP Multi-Year Assistance Program MOHS Ministry of Health and Sanitation MOWR Ministry of Water Resources MUAC Mid-Upper Arm Circumference NBI Nestbuilders International NGO Non-Government Program OICI Opportunities Industrialization Centers International PBF Performance Based Financing PHU Peripheral Health Unit PLW Pregnant and Lactating Women PM2A Preventing Malnutrition in Children under 2 Approach PMP Performance Monitoring Plan PNC Postnatal Care PREP Pipeline and Resource Estimate Proposal SAM Severe Acute Malnutrition SNAP Sustainable Nutrition and Agricultural Program SO Strategic Objective SOE State of Emergency VSLA Village Savings and Loan Association WASH Water, Sanitation and Hygiene 6 SNAP Final Evaluation Report March 2017 International Solutions Group I. EXECUTIVE SUMMARY ACDI/VOCA commissioned a Final Evaluation of the USAID-funded Title-II Multi-Year Assistance Program (MYAP) entitled Sustainable Nutrition and Agriculture Promotion (SNAP). The program wasimplemented by a consortium of three partners1 under the leadership of ACDI/VOCA in eighteen chiefdoms in the four districts of Bombali, Kailahun, Koinadugu and Tonkolili. With a Development Objective (DO) to reduce food insecurity and increase resilience among vulnerable populations, the program had five Intermediate Results (IRs) under two Strategic Objectives (SOs), to (1) reduce chronic malnutrition among children under five and (2) enhance livelihoods for vulnerable people, especially women and youth. The total program cost to Food for Peace was for a period of six and a half years, including a one-year cost extension during the Ebola Virus Disease (EVD) outbreak and a six month no-cost extension, extending the program from 10 June 2010 through 31 December 2016. In addition to funding from the US government, the program also included over of Gifts-in-Kind contributions provided through International Medical Corps. This Final Evaluation comprises three major elements: 1. A population-based2 quantitative Endline Household Survey (building on a 2010/2011 Baseline Survey) undertaken by a research firm, Nestbuilders International, over the period August 15-27, 2016; 2. A qualitative research component conducted by a team of four external consultants from research firm International Solutions Group (ISG), who conducted field research in Sierra Leone from October 9 through November 5, 2016, in Sierra Leone; and 3. Analysis and synthesis of all available quantitative and qualitative data by the ISG team to present evidence and conclusions as to the overall performance of the SNAP program via this report, prepared between November 2016 and March 2017. In addition to analyzing this quantitative and qualitative information, the Evaluation Team reviewed existing secondary sources of information, including other quantitative data, and used qualitative survey methods to obtain information directly from program participants, implementing staff, leadership of consortium partners and representatives of government stakeholders to understand program outputs, immediate and likely sustained impact generated by the program and lessons learned from the SNAP experience relevant for future programming. Over 650 individuals provided their views on the program in the interviews and discussions conducted during the qualitative portion of the evaluation research. The SNAP Program encountered some significant challenges during implementation, most notably obligations to curtail program activities from mid-2014 to November 2015 due to the outbreak of Ebola Virus Disease (EVD) in the country and wider region, but also administrative constraints such as high staff turnover in senior positions at the start of the program. However, the Final Evaluation concludes that, since the arrival of the current Chief-of-Party in early 2013, the program made good progress on most proposed activities. Over its life, the SNAP Program reached an estimated 55,200 households, exceeding the target of 54,601 for LOA. The ultimate achievements of the program in outcome and impact terms were, in aggregate, positive, with the SNAP endline survey recording achievements either approaching or surpassing targets in most key performance indicators in children’s nutrition, household diet, agricultural production, and household income. 1ACDI/VOCA, International Medical Corps and Opportunities Industrialization Centers International 2 i.e. the sample was drawn from the entire population of the areas where the program was conducted, rather than selecting program participants only – a more time/resource intensive approach. 7 SNAP Final Evaluation Report March 2017 International Solutions Group There is strong evidence to indicate that the SNAP Program achieved substantial impact on:  Children who would have become malnourished or suffer hygiene-related illnesses;  Household members who benefited from increased crop production and greater household income from crop sales;  Members of VSLAs who were making good investments with loans and share-outs, thus improving household financial stability and status;  FFS cluster members who were in clusters that have strong marketing relationships, and;  Artisans trained by the program who gained new income-generating opportunities. The Evaluation Team sought to triangulate findings of the quantitative endline survey with qualitative and anecdotal data solicited directly from program participants and other program stakeholders (such as government partners) in order to verify and provide context to the quantitative data. In general, the performance across these quantitative metrics was indeed supported by the qualitative findings, supporting the conclusion that SNAP was directly responsible for the demonstrated positive changes in peoples’ lives. Table ES1 compares endline estimates to baseline estimates for FFP Key Performance Indicators for the SNAP Program. Table ES1. SNAP Performance for FFP Indicators FFP Performance Indicators Baseline Estimate LOA Target Endline Estimate % of Target Achieved Prevalence of underweight children under five years of age (IM2) 20.3% 14.3% 14.8% 92% Prevalence of stunted children under five years of age (IM3) 38.4% 31.4% 32.3% 87% Percent of acutely malnourished children 6-59 months (OC1) 9.6% 7.2% 6.5% 129% Percent of children 0-5 months of age exclusively breastfed (OC9) 39.5% 75% 75.6% 101% Percent of caregivers demonstrating proper food hygiene behaviors (OC10) 78.5% 76% 56.1% 74% Percent of caregivers demonstrating proper personal hygiene behaviors (OC11) 87.4% 69% 90.5% 130% Average Household Dietary Diversity Score (HDDS) 6.74 8.5 5.44 Negative Number of Months of Adequate Household Food Provisioning (MAHP) 7.74 9.5 9.67 110% Percent of farmers who have adopted at least three agricultural technologies (OC17) --- 75% 25.5% 34% Percent of farmers who have adopted at least one agricultural technology (OC17.1) --- No target3 30% --- Analysis of the findings of the quantitative and qualitative research provides evidence to support a number of outcome/impact-related conclusions. Sections IV and V of the main report also include comparisons for other performance indicators including F indicators for USAID Sierra Leone, other USAID Sierra Leone indicators, performance indicators defined by the program itself and indicators specified by the GoSL, as described in the program's IPTT/PMP. Child Malnutrition - Underweight, Stunting and Wasting. The Endline Household Survey indicated significant change on all three anthropometric indicators, achieving a 5.5 percentage point reduction in underweight (92% of the LOA target of 6 percentage points), a 6.1 percentage point reduction in stunting (87% of the LOA target of 7 percentage points) and a 3 This indicator was introduced by FFP subsequent to the commencement of the program, hence no target was set. The prior indicator (OC 17) was the previous iteration. 8 SNAP Final Evaluation Report March 2017 International Solutions Group 3.1 percentage point reduction in wasting (129% of the LOA target of 2.4 percentage points). Further, additional analysis of Endline Survey data disaggregated the household data by participants and non-participants. While of limited statistical robustness (due to the overall population-based nature of the survey), they are illustrative in providing evidence of change among both groups, but with program participants consistently showing slightly better changes. Qualitative research among participating mothers supports this evidence, and the conclusion that improvements among communities may, at least in part, be attributed to SNAP activities. The evidence indicates that the program successfully contributed to reducing the prevalence of acute malnutrition among children under five. This was achieved through hygiene promotion messages, mainly around handwashing; training on acute malnutrition self-screening using Mid-Upper Arm Circumference (MUAC) tapes, and increasing access to health facilities. For approximately 32% of the mother-child pairs (an estimated 15,000 children) participating in the program who completed the full 30 months of the Preventing Malnutrition in Under Two Approach (PM2A: receiving food rations, BCC messages and access to health facilities), the program likely had strong impact on reducing the potential for chronic child malnutrition. Exclusive Breastfeeding. Qualitative research indicated that exclusive breast feeding was one of the messages best understood and adopted by SNAP participants. The estimates from the Endline Household survey support this, showing that over 75% of respondents with infants were practicing exclusive breast-feeding (102% of the LOA target), a significant increase from the 39.5% found in the baseline. Proper Personal Hygiene Behavior. One of the other messages that participants reported to the Evaluation Team as best understood and adopted was handwashing, with the messages reinforced by the national campaigns during the EVD outbreak which brought greater focus on handwashing behavior. The program introduced and provided support for the use of handwashing stations, which reinforced the hygiene messages. However, the Evaluation Team observed very few of these facilities still functioning in villages (the project provided construction/maintenance materials which households were not replacing when they became damaged or worn out). Despite this, the quantitative measurement of this indicator (via the Endline Survey) summarized responses on handwashing behavior before eating, before preparing food, and before feeding/cleaning babies, and found that 90.5% of total respondents were practicing at least two of these three behaviors (88.4% non-program beneficiaries; 97.5% program beneficiaries). This combined measure was 131% of the LOA target of 69%. Diet Diversity. Both quantitative and qualitative data indicated little overall change in diet diversity. Community members did not appear to the Evaluation Team to sufficiently understand messages on the benefits of diet diversity provided through mother-care groups. While there was solid quantitative data that the program contributed to increased agriculture production, qualitative research found that this was mostly with respect to staple crops and marketable cash crops. The Endline Household Survey estimate of diet diversity indicated a decline in diet diversity from the baseline (an aggregate score of 6.74 food groups at the baseline versus 5.44 food groups at the endline). However, the validity of the comparison may be confounded by the baseline data having been collected at a calendar period when food availability and incomes are higher, while the endline data was collected at the peak of the lean season, when these indicators are typically lower. Household Food Provisioning Ability. Survey data indicated that household food provisioning capacity increased from 7.74 months at the baseline to 9.67 months at the endline (110% of target). The research suggests that this was due to crop production activities undertaken by Farmer Field School (FFS) Clusters, complemented by increased access to 9 SNAP Final Evaluation Report March 2017 International Solutions Group capital through Village Savings and Loan Associations (VSLAs). Qualitative research further suggested that these benefits were passed to others in the community who indirectly benefited from seed or capital made available via Cluster groups or VSLAs, respectively. Adoption of Agriculture Technologies. The Endline Survey did not capture adoption of technologies by farmers due to survey methodology errors. Results showed that 25.5% of respondents reported adopting at least three new practices or technologies and 30.0% reported adopting at least one new practice or technology. The qualitative research indicated that the categorization of ‘technology adoption’ by this survey underestimated the true rates, which were likely much higher, with the most often reported new ideas or technologies adopted being plant spacing and seed/seedling density. Considering the primary impacts described above, the limited program penetration achieved in the first two and half years, the activity cutbacks from poor results of monetization, and the impact of the EVD crisis, the Evaluation Team assesses the investment in SNAP as a better than average, and worthwhile, MYAP investment. Best Practices and Recommendations for Future/Related Programming • The implementation of the baseline and endline quantitative household surveys at different seasonal times resulted in discrepancies due to seasonal variations, particularly with respect to food availability and nutritional data. Future such surveys should be carefully planned to take place at the same time of year to optimize comparability between metrics. • Investment in infrastructure-related elements of the program, which require ongoing investment in infrastructure for maintenance, should be very carefully considered during future programming, and not implemented unless concrete plans and commitments to their sustainability are in place. • Involving men from the outset in Behavior Change Communication activities produced excellent results, and should be replicated in programs that have a focus on health and nutrition behavior change. • Using a farmer field school approach to test new ideas and approaches, and then clustering two to three of these groups for increasing production continues learning processes introduced by the FFS while also enhancing benefits for members through increased production and economies of scale. • The VSLA approach used by SNAP was often cited by interviewees as the most important intervention introduced by SNAP, will likely contribute to sustained benefits and should be prioritized in future programs. • Given reasonable risk of crises, monitoring of trigger indicators and a framework for emergency response facilitates better preparation and more timely response. • To be better able to monitor gender changes that may be occurring because of program activities, program monitoring systems should have indicators at all levels to capture these changes. Further, priority should be given within implementing strategies to the recruitment and hire of female staff, particularly in leadership roles. • Although distribution of food likely had positive nutrition-related outcomes, a substantial BCC strategy focused on food diversity and hygiene, with consumption of local foods bolstering local agricultural production, may have had larger and more sustainable results. • There are good opportunities for synergies between Food for Peace and Feed the Future programming. Value chain development at different levels (household/community to central level) should be the focus of these investments. • Evaluation findings indicate that the most substantial outcomes of the program were where the greatest integration/overlap occurred across Strategic Objectives. Integration across objectives, activities targets and locations should be a central pillar of future programs. 10 SNAP Final Evaluation Report March 2017 International Solutions Group II. BACKGROUND INFORMATION A. Overview of the SNAP Program On May 11, 2010, ACDI/VOCA submitted a final revised proposal for the Sustainable Nutrition and Agriculture Promotion Multi-Year Assistance Program with an approved start date of June 10, 2010 and an expected completion date of June 9, 2015. The program proposed to use 64,350 MT of commodities, over 75% of which was to be monetized, with a total proposed/expected cash budget of , including nearly in monetization proceeds. The program was designed to have impact on over 405,000 direct beneficiaries with a strategy oriented around seven Intermediate Results under two Strategic Objectives. By the time ACDI/VOCA submitted the first Annual Results Report on November 11, 2011 (for FY 2010 and FY 2011), however, the program strategy had been revised to the current structure with five Intermediate Results under two Strategic Objectives as shown in Table 1. Consistent with the original design, ACDI/VOCA implemented SNAP in 18 chiefdoms in the four districts of Tonkolili, Koinadugu, Bombali and Kailahun. As well as providing overall leadership, ACDI/VOCA was responsible for implementing most activities under SO2 as well as commodity management and overall program monitoring and evaluation. Consortium partners International Medical Corps (IMC) and Opportunities Industrialization Centers International (OICI) were responsible for implementing activities under SO1 and youth training under SO2, respectively. Table 1. SNAP Development Objective, Strategic Objectives and Intermediate Results DEVELOPMENT OBJECTIVE: Reduce food insecurity and increase resiliency among vulnerable rural populations in Bombali, Kailahun, Koinadugu and Tonkolili Districts. SO1: Reduce chronic malnutrition among children under five SO2: Enhance livelihoods for vulnerable people, especially women and youth IR 1.1 Improved access to and use of quality nutrition and health services for women and children 0-59 months IR 2.1 Increased agricultural productivity at the household level IR 1.2 Improved health and nutrition practices at the household level IR 2.2 Increased access to financial and marketing resources IR 2.3 Improved community assets Program progress from start-up in June 2010 to January 2013 was intermittent. During this period, the program had three different Chiefs of Party (CoP) and several other senior positions, including the team leader for SO1 and the M&E coordinator, were vacant for extended periods. Commodity monetization was also problematic due to the Government of Sierra Leone raising concerns about the proposed monetization. Once these concerns were alleviated, poor returns 11 SNAP Final Evaluation Report March 2017 International Solutions Group on monetization were realized due to poor commodity prices and high losses (see section VI.E.2 for more details). These challenges affected program vision and leadership, implementation decision-making, and program financing. Three significant turning points occurred over the course of the SNAP Program. Two of these were positive, accelerating implementation, and the third was negative. The first of these, the arrival of a new CoP in January 2013, provided a consistent vision for the program and more rapid decision-making, enabling the program to accelerate implementation. A Theory of Change was developed and several key programmatic decisions were made, including the introduction of new crops, restructuring of the youth training component, addressing bottlenecks in the commodity pipeline, and expanding local technical partnerships. A second significant turning point occurred early in FY 2014 with the decision to suspend monetization and replace the resources with 202e USD resources. As explained further in section VI.E.2, the returns on monetization were very low, and the switch to cash resources ultimately resulted in more cash for the program and released monetization-focused human resources to other tasks. At this point, SNAP began smooth implementation of the vision brought by the new CoP. The third significant milestone commenced on May 23, 2014, with the first case of Ebola Virus Disease (EVD), diagnosed in Kenema. As cases began mounting, the GoSL declared a State of Emergency on July 31, 2014. From this point until the lifting of the State of Emergency on November 7, 2015, normal SNAP program activities were suspended, although as will be discussed throughout the report, some community-based activities continued without program support. It was clear at this point that, because of the effect of the EVD outbreak, SNAP was unlikely to fully meet targets. With the FY 2015/16 Pipeline and Resource Estimate Proposal (PREP), ACDI/VOCA requested a costed extension of SNAP for one year from June 10, 2015 through June 9, 2016 to be able to shift food resources, as per the 10% allowance for emergency response as per FFP policies, to support households affected by EVD as well as to have more time to achieve targets, presuming a short duration of the crisis. However, the EVD State of Emergency extended well into this 6th year of implementation so a further six month no-cost extension was approved for SNAP, extending the end date for the program to December 31, 2016. The figure below illustrates the life of the SNAP Program. Figure A. Life of the SNAP Program with Key Turning Points June 2010-Dec 2012 Jan 2013-July 2014 Aug 2014-Nov 2015 Nov 2015-Dec 2017 Current CoP Arrives Monetization Suspended EVD Cost Extension Three CoPs, Other Key Staff Vacancies & Problematic Monetization No Cost 12 SNAP Final Evaluation Report March 2017 International Solutions Group B. Contextual Factors Affecting Program Implementation or Impact The SNAP Final Evaluation identified three major contextual factors that affected either program implementation or impact. The first of these, as noted, was the outbreak of EVD that affected implementation from July 2014-November 2015. Marketing activities were severely constrained during this time, and productivity declined as farmers were unable to travel outside of the village. As per IMC monthly community screening, cases of acute malnutrition slightly rose. SNAP PM2A food distributions were discontinued, with food already in pipeline rechanneled to EVD-affected households. All of this resulted in a major slowdown in the program's ability to implement activities and achieve impact. A second major contextual feature that affected program implementation and impact was excessive expectations by villages and other stakeholders, including program staff and government partners, that NGOs should be distributing materials to people. In 2002 following the end of the civil war in Sierra Leone, the need to rebuild the country led agencies, including NGOs, to freely distribute rehabilitative materials, including food, agriculture inputs, other livelihoods tools and housing materials. While necessary at the time, it cultivated a strong sense that "things" are valued more than "ideas" from NGOs, and NGOs are viewed as contractors, coming to villages to have work done and paying people to do the work for them. During the information-gathering phase of the Evaluation, the evaluators found that community members were frequently more likely to respond to the evaluator's questions with responses intended to elicit new support than with responses describing the achievements of the program4 . In this context, it was challenging for a program to only promote ideas, no matter how useful, and to build ownership to enhance potential sustainability of the impact achieved. A third contextual factor affecting program implementation was road infrastructure. The SNAP Program selected many remote and inaccessible chiefdoms, especially in Koinadugu, Kailahun and parts of Bombali Districts, particularly during the rainy season. This constrains economic development as markets are small and opportunities are limited by transport cost. It costs, for example, twice as much to go by public transport from Neya Chiefdom in Koinadugu District to Kabala, the Koinadugu District capital, as it does to go from Kabala to Freetown. The quality of basic services is also constrained since it is challenging to post experienced staff in remote chiefdoms. The difficult roads made the delivery of food a major challenge for SNAP5 . Approximately 15% to 17% of villages participating in the SNAP Program can only be reached by foot at any time during the year, with approximately 17% to 19% estimated by the Evaluation Team as reachable by motorcycle but not by car. During the rainy season, reaching those villages that have roads can be extremely challenging even for cars. 4 When evaluators received a response noting ongoing need, they would suggest a visit to a farm or house to verify this. If the respondent provided excuses for why this could not be done, e.g. "it's too far", "the people are gone to market", "the leader is not available", etc., then doubt was cast on the veracity of the response. 5 In an interview during the Evaluation, one of the SNAP District Operations Managers, the senior position in a district overseeing all implementation, responded to the question on the major achievement that he was most proud of…..being able to deliver food to all distribution points. Lessons Learned - Trigger Indicators SNAP's experience with the EVD crisis suggests that programs in Sierra Leone (and elsewhere) will occasionally face crises. Monitoring of good trigger indicators associated with (at least) a framework for emergency response facilitates preparation and timely response. The original design of SNAP included trigger indicators, but these were not incorporated into Monitoring and Evaluation systems. While none of these indicators would have identified an emerging EVD crisis, the concept is sound, and attention should be given to identifying good indicators and setting up cost efficient mechanisms for monitoring them. 13 SNAP Final Evaluation Report March 2017 International Solutions Group C. Evaluation Methodology 1. Evaluation Methodology Overview. The Final Evaluation of the SNAP Program was a summative performance evaluation intended to assess the performance of the program toward achieving its DO, two SOs, and five IRs. The Evaluation examined the outputs produced under each IR, assessed the impact achieved, and identified good practices and lessons learned relevant for future programming. This Final Evaluation comprises three major elements: 1. A population-based6 quantitative Endline Household Survey (building on a 2010/2011 Baseline Survey) undertaken by a Sierra Leonean research firm, Nestbuilders International, over the period August 15-27, 2016; 2. A qualitative research component conducted by a team of four external consultants from US-based research firm International Solutions Group (ISG), who conducted field research in Sierra Leone from October 9 through November 5, 2016, in Sierra Leone; and 3. Analysis and synthesis of all available quantitative and qualitative data by the ISG team to present evidence and conclusions as to the overall performance of the SNAP program via this report, prepared between November 2016 and March 2017. The Endline Survey was conducted prior to the qualitative exercise so that the endline estimates on key performance indicators would be available to the qualitative team during the planning and data collection. The qualitative information gathering was used to clarify and enhance the interpretation of the quantitative estimates. The following sections provide an overview of the two data collection exercises. Annex A contains the Scope of Work for the Final Evaluation. The Evaluation Plan that was implemented for the qualitative information gathering is provided in Annex B, and Annex C contains the detailed Final Report on the Endline Survey. 2. Endline Household Survey. The specific purpose of the SNAP Final Evaluation Endline Survey was to generate endline estimates for the program's four impact indicators and fifteen outcome indicators listed in the Indicator Performance Tracking Table (IPTT) for the program. These estimates are described in Section V of this report in the analysis on outcomes and impact generated by the program. The Evaluation Endline Survey used FANTA Sampling Guidelines7 to calculate a target sample size for the endline capable of detecting the targeted seven percent reduction in the child stunting indicator over the project life. This is consistent with the sampling strategy used in the Baseline Survey, and a target sample size of 1,750 households was set for the Endline Survey. The Endline Survey also used the Enumeration Areas (EAs) defined by the Sierra Leone General Population and Housing Census as a sample frame and used the identical 108 EAs chosen in the sample for the Baseline Survey. Within a selected EA, enumerators used a random-walk method to sample dwellings with a starting point for the walk identified from an EA boundary map prior to the start of fieldwork. When a dwelling or compound contained multiple households, the enumerator used a Kish grid with a random number generator to select one of the households. At the individual level, enumerators targeted four respondent groups for different modules in the questionnaire, including the (1) head of household or responsible adult, (2) primary caregiver or mother of children under five years of age8 , (3) household primary caregiver and (4) children under the age of five. For the agriculture module, enumerators 6 i.e. the sample was drawn from the entire population of the areas where the program was conducted, rather than selecting program participants only – a more time/resource intensive approach. 7 Magnani, Robert. 1999. Sampling Guide. Washington, D.C.: FHI 360/FANTA. 8 The terms “children under 5” and “children aged 0-59 months” are used interchangeably in this report. 14 SNAP Final Evaluation Report March 2017 International Solutions Group interviewed the household head or an informed family member. For the child and maternal health questions, enumerators selected the mother or caregiver of youngest child under the age of five in the household. If there was no child under the age of five in the household, this module was skipped. For household nutrition and hygiene questions, enumerators interviewed the primary caregiver. Lastly, for the anthropometric measurements, enumerators included all children under the age of five. The Endline Survey instrument was based on that of the Baseline Survey, but adapted to reflect changes that had been made by USAID to a limited number of indicators9 since the SNAP Baseline Survey was conducted. Survey questions for these revised indicators were largely adapted using questions from the FFP Standard Indicators Handbook. 10 The final endline questionnaire was entered onto a Computer-Assisted Personal Interviewing mask using Android phones for mobile data collection. A team of twenty-three enumerators supported by six supervisors, a Team Leader and a Research Manager undertook data collection over the period from August 15 to 27, 2016. ACDI/VOCA secured cooperation and participation of Ministry of Health and Sanitation (MOHS) staff, trained on accurate measurement11, proper use of the hand-held device for data collection and field procedures (sampling, building rapport, recording progression of work, supervision, etc.) took anthropometric measurements for children under five. Interviews were completed for a total of 1,816 households in 109 communities from all 18 chiefdoms targeted by the program. SurveyCTO software was used for data summarization and the dataset that was generated in CSV format was converted into an SPSS (Version 21) database for data analysis. Epi Info/ENA Software was used for the anthropometric data analysis using WHO 2006 child growth standards. Endline estimates were generated for the program as a whole as well as for each of the four districts. The district level sample sizes, ranging between 450 and 460 households, however, were too small for valid comparison with baseline estimates, so the Final Evaluation analysis focuses on program-level baseline-endline comparisons, factoring in the final iteration of program targets for each indicator (when presenting target achievement in the indicator tables below, the calculation method is (Endline-Baseline)/(LOA Target-Baseline) *100). 3. Qualitative Information Gathering. A four-person team of technical experts12, including both expatriate and Sierra Leonean technical specialists, implemented the qualitative information gathering with support from ten Sierra Leonean local language translators. The Evaluation Team gathered information over the period from October 9 through October 28 using key informant interviews or focus group discussions with stakeholders in Freetown, stakeholders in each of the four District capitals, and with program participants from a sample of 32 villages. Table 2 summarizes the number of persons who contributed qualitative information for the Evaluation. Detailed lists of the persons interviewed are provided in Annex D. 9 An annex to the Endline Survey Report provided in Annex C describes these indicator changes. 10 USAID. (2011). Food for Peace Standard Indicators Handbook (Baseline-Final Indicators). Available at http://pdf.usaid.gov/pdf_docs/pnadz580.pdf. 11 The USAID 'Anthropometric Indicators Measurement Guide' was used to guide this portion of the training. (See: Cogill, Bruce. 2003. Anthropometric Indicators Measurement Guide. Washington, DC: Food and Nutrition Technical Assistance (FANTA) Project, FHI 360). 12 (DME Specialist & Evaluation Team Leader), (MCHN/Reproductive Health Specialist), (Agriculture Specialist) and (Public Health and Nutrition Specialist). The detailed responsibilities for each team member can be found in the Evaluation Operational Plan in Annex B. 15 SNAP Final Evaluation Report March 2017 International Solutions Group Table 2. Qualitative Information Gathering Respondents Type of Respondent Number of Interviews of FGDs Persons in Each Location Total Freetown Persons District HQ Community SNAP Staff - ACDI/VOCA 17 7 24 31 Former SNAP Staff – IMC 6 2 15 17 Former SNAP Staff – OICI 2 1 1 --- 2 MOHS Representatives 15 1 8 --- 9 MAFFS Representatives 8 5 14 3 22 Representatives from Technical or Business Partners 8 2 14 3 19 District Council Representatives 4 --- 11 --- 11 Paramount Chiefs, Section Chiefs, Town Chiefs and their Advisers 12 --- --- 18 18 Staff of Peripheral Health Units 7 --- --- 11 11 Lead Mothers 16 --- --- 38 38 Care Units 14 --- --- 102 102 Men as Partners Members 7 --- --- 20 20 WASH Committee Members 3 --- --- 18 18 FFS Cluster Members 12 --- --- 155 155 Community Facilitators 5 --- --- 5 5 VSLA Members 5 --- --- 40 40 Private Service Providers 3 --- --- 3 3 Literacy/Numeracy and Vocational Training Participants 5 --- --- 49 49 Drying Floor Committee Members 3 --- --- 13 13 Non-Participants 10 --- --- 68 68 TOTAL 162 18 87 546 651 The sampling strategy for selecting communities for key informant interviews and focus group discussions was done strategically to ensure that the Evaluation Team would be able to see at least some examples of each of the major outputs produced by the program. The Evaluation selected villages using two criteria, (a) the intensity of intervention and (b) accessibility. For the intensity of intervention, communities that participated in the SNAP Program were stratified into those in which interventions from only SO1 (Around 60%), only SO2 (around 7%), or both SOs (around 33%) were implemented. For accessibility, in the absence of an existing SNAP village database with accessibility information, the selection of villages was done in collaboration with SNAP District Operations Managers for each district to get a mix of accessible and remote villages. The Evaluation Team Leader randomly selected a village from a master list representing an intensity stratum and then queried the DOMs on accessibility, randomly selecting alternatives when necessary to get a balance of accessible and inaccessible communities. Because of the time available for the Evaluation, the Evaluation Team eliminated from consideration villages for which it took more than thirty minutes to reach by either motorbike or on foot. Table 3 classifies the villages in the sample by each criterion. Table 3. Disaggregation of Selected Villages by Intensity of Intervention and Accessibility Intervention Intensity Accessibility Reasonably Accessible Somewhat Difficult to Reach Only SO1 (13) 8 5 Only SO2 (4) 2 2 Both SOs (10) 7 3 16 SNAP Final Evaluation Report March 2017 International Solutions Group The Evaluation Team held focus group discussions or key informant interviews with a range of different types of participants as shown in Table 2. The Evaluation Team used topical outlines, provided in the Evaluation Plan in Annex B, to guide information gathering. Many of the interviews and discussions were recorded on portable recorders for review later during information analysis. 4. Combined Data Analysis. The Evaluation Team analyzed both the quantitative information generated by the Endline Survey as well as the information obtained in the qualitative exercise. During qualitative interviews and discussions, evaluators probed and explored topics in more depth with respondents to ensure clear understanding. A second level of analysis occurred when evaluators cross-referenced responses from interviews and discussions with existing data bases and the Endline Survey results to verify findings. A third level of analyses occurred immediately before a Verification Workshop when the Evaluation Team shared with each other the presentations planned by each for the Verification Workshop. This allowed team members to present relevant information for other members of the team to consider in the analysis. A fourth level of analysis occurred during a Verification Workshop in which the Evaluation Team presented preliminary observations and findings to SNAP Program implementation staff. If implementation staff questioned specific findings presented by the Evaluation Team, further information gathering and analysis was undertaken to obtain clarity. Finally, in preparing this report, Evaluation Team members were able to take more time to review recordings and information sources to further refine evaluation findings. Throughout the analysis process, the Evaluation Team focused on the fundamental questions underlying the Evaluation. These were: • Relevance: How important are the interventions in the SNAP Program for the targeted impact groups and to what extent do they address their needs and interests? • Effectiveness: To what extent was SNAP effective in achieving its strategic objectives and intermediate results? • Efficiency: How were activities delivered to produce the desired results, relative to timeliness and the project implementation plans? • Sustainability: What organizations, services or relationships are required to sustain the impact being achieved by the SNAP project and how were the threats to these addressed? 5. Evaluation Limitations. Several limitations emerged during implementation of the Endline Survey and the qualitative information gathering as described below. Timing of Baseline and Endline Data Collection. Baseline data collection occurred in March of 2011 while Endline data collection occurred in August of 2016. These are very different periods in the year relative to farm production, sales, access to food, and degree of food insecurity, so all program performance indicators that are influenced by these factors will be affected, as discussed in Section V. March, for example, is in the middle of the dry season, only a few months after harvest so food is available and accessible. August is in the middle of the rainy season, just before the main harvest and represents the peak of the hunger season when food is less available and accessible. As noted in the sections which follow, this has implications for analyzing the comparison between endline and baseline estimates on specific indicators, most notably those related to household diet and food availability. Influence of Expectations in the Sierra Leone Context. Both the Endline Survey and qualitative information gathering encountered challenges with responses from respondents that seemed to reflect more of a desire to solicit future support than a recollection of actual experience. 17 SNAP Final Evaluation Report March 2017 International Solutions Group Respondents tended to under-report on some topics, such as production and sales, to give the impression that the situation was worse than reality to obtain additional support. Respondents also reported (at times) that the functioning of groups and the learning that they obtained were better than reality to give the impression that they were effective partners with whom to work in the future. The Evaluation Team used triangulation from other data sources and direct observation to identify potentially misleading responses. For example, evaluators would request to see the farm of an FFS Cluster that they indicated was producing. Similarly, a group indicating learning of a specific skill/behavior, such as using a MUAC tape, were asked to demonstrate said behavior. Time Limitations in Sierra Leone for Qualitative Information Gathering and Analysis. The extent and depth of the qualitative evaluation research in Sierra Leone was constrained by the resources available. The scale (four districts) and scope (two SOs with multiple interventions) of the SNAP Program necessitated that the Evaluation Team focus on conducting interviews and focus group discussions with very little time for analysis, and even so, was unable to visit all eighteen chiefdoms that participated in the program. Any notable missing data points were solicited via the Evaluation Verification Workshop at the conclusion of the research phase, although such information could not be directly verified. Much of the refined analysis underlying the findings in the report was done by the two international consultants on the Evaluation Team after leaving Sierra Leone, with remote support from contacts in Sierra Leone. Findings from the qualitative research are supported throughout with data from the Endline Survey and ACDI/VOCA’s interim and final reporting on SNAP. Other Limitations. Over the life of the program, changes were made on some performance indicators effectively eliminating valid comparisons between a baseline (calculated for one indicator definition) and the endline (calculated using a different indicator definition). Annex 4 of the Endline Household Survey Report included in Annex C of this report lists the six indicators affected. The SNAP Program was implemented in eighteen chiefdoms containing ten different local languages13 . The Evaluation Team sought to use independent translators in these languages for interviews and discussions and retained an independent translation firm, Dalan Consultants, on short notice to provide ten translators for a complex schedule. For the most part, the translation services worked well. In a few cases, however, the translator provided was not fluent in the dialect being spoken or the language spoken in a selected village was different than that specified by the program for which a translator was recruited. In these cases, interviews were conducted in Krio, and not all participants in a focus group were fluent in Krio. At the time of the qualitative information gathering, the SNAP Program was in it close-out phase. Staff for consortium partners IMC and OICI were no longer engaged by the program. To its credit, ACDI/VOCA was able to identify and provide compensation to a number of staff from these partners to participate at key points in the Evaluation, including the field work and Verification Workshop. The Evaluation Team also notes that the rotation of staff within government stakeholders in the program areas often meant that interviews during the Evaluation were held with persons who had little firsthand knowledge about the program. Identification and sourcing of previous incumbents was not feasible with the available resources. 13 Temne, Limba, Kuranko, Julanko, Mandingo, Fula, Loko, Mende, Kono and Kissi 18 SNAP Final Evaluation Report March 2017 International Solutions Group III. EVALUATION FINDINGS: PROGRAM THEORY OF CHANGE AND TARGETING A. Theory of Change The original design for the SNAP Program was not based on a Theory of Change (ToC). For the first few years of program implementation, the Results Framework developed at the design stage was used to guide strategic thinking for the program. In early CY 2013, a new Chief of Party arrived and, as part of reviewing and revising strategic plans, facilitated a process for developing a ToC encompassing the SNAP Program. The ToC that was developed is provided in Annex F. While this ToC does not conform to today's typical usage14 , it was a useful exercise for guiding changes in the program strategy. In ideal use, the ToC not only justifies the strategy of a program, i.e., selection of appropriate interventions, it is also used to build a common vision for a program among implementation staff. It should be possible to see in the ToC, for example, how a program should be implemented to achieve various benchmarks toward achieving a vision for change. The SNAP ToC is too complex for this function, and for its life, SNAP used the Results Framework for orienting and guiding implementation staff. B. SNAP Program Targeting 1. Geographic Targeting. Positively, SNAP targeted some of the more remote chiefdoms in the four districts15, based on a contextual analysis that was conducted during program design. Within the targeted chiefdoms, the program planned to implement capacity building for all government Peripheral Health Units (PHUs) existing in the chiefdom, so SNAP could directly benefit any village that falls within a PHU catchment area. The GoSL assigned all villages in a chiefdom to a PHU catchment area, but some villages are prohibitively long distances from the PHU to which they were assigned. Except for emergencies, people rarely go to these PHUs because of the distance. To remedy this, the GoSL is establishing new PHUs when it can. In principle, then, all villages in the eighteen targeted chiefdoms were reached in some way by SNAP, ranging from those villages in which full implementation of both SOs was undertaken to other villages in which residents did not receive direct SNAP benefits, but benefitted indirectly by receiving services from a PHU that had been strengthened by the program. As described under Evaluation Methodology, the Evaluation Team requested a list of villages showing the major SO1 and/or SO2 activities implemented in each. Since the two components of the program started with separate databases oriented towards food distributions and PHUs and all other activities over the life of the project, it was difficult at this late stage to determine exactly how many villages had participated in activities for SO1 only, for SO2 only, or for both SOs. The SO1 database, managed by IMC, was oriented more around PHUs and their catchments and the SO2 database, managed by ACDI/VOCA was oriented around villages. Matching locations and location names between the databases was a challenge, even to the extent that ‘official’ data from the GoSL may have multiple spellings for the same location and well as multiple communities with the same names. 14 In general terms, a ToC, in the way that it used today, describes the process for achieving a desired change with a specific impact group. It specifies the impact group, describes the type of change that is being sought, and defines pathways (sequences of steps for moving from the present to the desired change) within domains of change (a thematic area). 15 SNAP Chiefdoms include Gbanti Kamaranka, Sanda Loko, and Sella Lemba Chiefdoms in Bombali District; Kissi Tongi, Mandu, Peje West, Penguia, and Yawei Chiefdoms in Kailahun District; Diang, Folosaba Dembelia, Kasunko, Mongo, Neini, Neya, and Wara Bafodia Chiefdoms in Koinadugu District; and Kafe Simira, Kalansongoia and Yoni Chiefdoms in Tonkolili District. 19 SNAP Final Evaluation Report March 2017 International Solutions Group Based on information from the SNAP M&E unit provided for the Evaluation, however, approximately 60% of the villages saw only SO1 activities, approximately 7% of the villages saw only SO2 activities implemented, and approximately 33% of the villages saw both SOs implemented. The relative small percentage of villages in which only SO2 activities were implemented resulted from a delay at the beginning of the program in selecting SO1 communities, so SO2 moved ahead with forming FFSs in villages that they selected in order not to lose an agricultural season. 2. Participant Targeting. The SNAP Program oriented the beneficiary targeting strategy around mother-child units composed of a mother with a child under the age of five years grouped under a care unit and constrained by the presence of health promoters in the targeted community. The Program defined as presence of a pregnant woman in the household, with no other vulnerability distinction (such as on an economic/income basis). The direct participants for SO2 were to be drawn from communities that had been selected for SO1. These participants were to be self￾selected and could include members from households participating in SO1, but also youth (age 15-35) participating in local government registered groups, pre-existing farmer groups, producer associations, marketing associations or agri-businesses. ACDI/VOCA designated the following targets for SNAP. SO1: 50,064 mother-child units with an additional 350,448 family members SO2: 45,375 individuals from communities in which SO1 was implemented - 31,000 participants in FFSs - 10,000 participants in conditional literacy, numeracy and vocational training - 5,000 youth for agribusiness and entrepreneurship training - 144 nascent agribusinesses for training and/or in-kind grants - 70 input suppliers for capacity building - 170 VSLAs with approximately 4,250 people The total unique beneficiary target for both strategic objectives was 405,049. The Evaluation discusses performance against these targets in Section V. Consistent with these numbers, the program's Indicator Performance Tracking Table (IPTT) had a Life of Award (LOA) target of 54,601 households for the program as whole with 50,064 pregnant and lactating women and 400,512 other household members in SO1 and 45,377 individuals for SO2. Evaluation Observations. In the first year of program implementation, all pregnant or lactating women in a targeted village were selected for participation. It was recognized after start-up, however, that the most effective intervention associated with PM2A begins with targeting pregnant women, so the targeting strategy shifted to selecting only pregnant women and previously selected lactating women were deselected from the program. Not all households in a selected SO1 village participated in SO1 activities over the life of the program since the household must have had a pregnant woman at least once over the life of the program. In villages in which both SOs were implemented, the Evaluation found extensive overlap in terms of households having members participating in the two SOs. 20 SNAP Final Evaluation Report March 2017 International Solutions Group Under SO2, the training participants for literacy/numeracy and vocational skills training for the first two years of the program were not drawn from SO1 communities as intended. Participants, exclusively youth, were drawn near the town-based training centers. This changed in 2013 when the strategy was modified to move the training out from the towns, closer to the rural areas where the program was working. Since there were fewer eligible and interested youth in these areas, the groups began including others outside of this group. Relative to others in targeted communities who were not direct beneficiaries, but nevertheless benefitted from program activities, communities are cohesive and the Evaluation Team noted spill-over of benefits in many villages. Some of the seed provided by SNAP for FFS Cluster production was being disseminated to other households by the clusters through seed loans. The Evaluation Team also noted cases of loans from VSLAs being taken for relatives by VSLA members. The health and nutrition messages extended by the program were diffusing in at least some villages, depending on the activeness of the Health Promoter (HP) and/or Lead Mothers (LMs). Finally, food distributions provided by the program were shared beyond the intended beneficiaries (discussed in more depth on page 21). This diluted the demonstrable nutritional impact from the food, but program participants still perceived it as a benefit. Relative to exclusion of qualified households, the Evaluation Team did not find any significant systematic exclusion by the program. The only types of exclusion identified in qualitative interviews were of families who did not receive food because they had lost registration cards, arrived too late after a distribution was completed, their name did not appear on the registration list, or their alternates were not recognized or had exceeded the alternative eligibility period. Several participants in VSLAs withdrew after a VSLA cycle because they did not have sufficient cash to purchase shares, and other households withdrew their labor from FFS Clusters to work more on their own farms. LESSONS LEARNED – TARGETING It takes time to place health promoters, identify PM2A households and effectively cluster women from these households into care units. Requiring other program components to wait until this is done before identifying their own participants is an inefficient use of time. Future programs should develop targeting strategies that allow components to begin in a timely manner in ways that facilitate integration of components later. 21 SNAP Final Evaluation Report March 2017 International Solutions Group IV. EVALUATION FINDINGS: OUTPUTS PRODUCED UNDER EACH INTERMEDIATE RESULT As part of the analysis of SNAP program performance over its lifetime, this section of the evaluation report describes the specific outputs that resulted from the input of the program resources. As per the program logic (articulated in the Theory of Change, discussed above) these outputs are the precursors to the outcomes and impacts that the program ultimately aimed to achieve. Assessment of these outputs was part of the scope of the Evaluation research, to determine the extent to which the program met proposed output targets and to assess elements of the program that were integrated to maximize efficiency and contribute towards program effectiveness. The examination of the level of achievement of the program outputs includes some discussion of the contribution to intermediate and longer term impacts among the program population. This is followed in the subsequent section by an examination of changes in key variables noted among the program beneficiary population by the quantitative and qualitative research. The Evaluation Team assesses the evidence that such changes can be attributed to the program outputs, i.e. that they can be considered likely outcomes and impacts of SNAP, as well as any evidence that these changes will be sustained beyond program completion. The chart below presents a summary of achievement of 39 output indicator targets. Overall, the program has performed well, with two-thirds of indicator values (where measurable) within 75% of proposed targets, and only seven achieving 50% of targets or less. The following sections provide a more detailed analysis of data. Eight indicators could not be successfully compared across baseline and target values due to either missing values or baseline values being higher than either endline or targets due to either differences in measurement methodology between baseline and endline, or (approved) target changes over the course of SNAP. The Evaluators note the specific instances of this in each table below. 13 7 4 4 3 8 0 2 4 6 8 10 12 14 >100% of Target 76-100% of Target 51-75% of Target 26-50% of Target 0-25% of Target n/a SNAP Output Indicator Aggregate Performance 22 SNAP Final Evaluation Report March 2017 International Solutions Group A. Outputs Produced Under IR 1.1: Improved access to quality nutrition and health services for women and children aged 0-59 months The SNAP Program produced three sets of outputs under IR 1.1, including capacity building of Peripheral Health Units (PHUs) and Health Promoters; Water, Sanitation and Hygiene (WASH) capacity building; and strengthening of acute malnutrition screening services. 1. PHU Capacity Building and Health Promoters. The SNAP Program was implemented directly with PHUs building capacities of PHU staff through targeted training, providing medical equipment and drugs to detect and treat common children illnesses including infant acute malnutrition, and increasing access to the PHU for pregnant women and their children. ACDI/VOCA identified training topics during the design of SNAP and trainings were carried out by MOHS trainers in collaboration with IMC staff and District Health Management Teams (DHMTs). These trainings included, but were not limited to, Integrated Management for Newborn Child Illnesses (INMNCI), Integrated Management of Acute Malnutrition (IMAM) including the community-based component, Infant and Young Child Feeding (IYCF), Basic Emergency Obstetric Neonatal Care (BEmONC) and medical waste management. Local district health authorities were involved in the implementation of SNAP from the beginning of the program, and joint supervision and monitoring were conducted at the PHU and community-levels. This collaboration extended to the provision of a Gift-in-Kind package of drugs, medical equipment and supplies to each PHU. To implement behavior change and communication (BCC) activities and other sensitization activities, SNAP took advantage of the Community Health Workers (CHWs) already present at village level and attached to the PHUs. During SNAP implementation, those CHWs became Health Promoters (HPs) for the program and were selected by their communities for their commitment and engagement in health and nutrition issues at the community-level. Starting first as volunteers, HPs became employees of IMC in 2012. HPs were trained on the MCG approach before being sent into their communities to train and supervise LMs on a bi-weekly basis on four training modules, including Exclusive Breastfeeding (EBF), complementary feeding, Family Planning (FP) and hygiene promotion. HPs were also responsible for transmitting the names of new pregnant women to SNAP district managers to be enrolled in the PM2A food distributions. Achievements. The SNAP Program provided support to 123 PHUs and 43 Health Promoters. Table 4 provides information on the number of PHU staff and community members trained by SNAP. Table 4. PHU Staff and Community Members Trained by SNAP Topic CHWs/HPs PHU Staff DHMT Integrated Management of Newborn and Child Illness 235 231 0 Community Management of Acute Malnutrition 71 65 0 Infant and Young Child Feeding 0 107 0 Family Planning 0 22 0 Mother Care Group Approach (4 modules) 43 0 0 Essential Nutrition Actions 43 0 Growth Monitoring 0 123 0 Reproductive Health 0 236 0 Basic Emergency Obstetric & Newborn Care 0 120 0 Medical Waste Management 0 100 12 EPI 281 100 0 TOTAL 673 1104 12 23 SNAP Final Evaluation Report March 2017 International Solutions Group Table 5 provides information on IPTT Indicators for IR 1.1 related to PHU activities. Major Observations. The PHU capacity building by SNAP resulted in increasing access for Antenatal Care (ANC) and skilled birth attendance as well as intensified access to FP and growth monitoring. While SNAP did not explicitly train Skilled Birth Attendants, emergency obstetric & newborn care were part of the training support, and the program trained Health Promoters in communities, who in turn promoted assisted child births. Family planning services played a major role in spacing pregnancies and enhancing gender-balance between man and woman and an entire BCC module was dedicated to FP to reinforce the link between community and PHU. Despite that there were no indicators to capture this data, findings from the qualitative research indicate knowledge of family planning increased and modern FP methods were adopted. Further, the Evaluation Team analyzed endline data to disaggregate results from program participants and non-participants. While not sufficiently rigorous to be statistically valid (in that a formal control group was not established and thus the results are liable to confounding by other variables), the results suggest that performance improvements were higher across these key indicators among those who participated in, or benefited from, SNAP interventions. Table 5. IPTT Indicators for IR 1.1 Indicator (source of indicator – FFP/USAID or A/V) Baseline LOA Target Endline (95% Confidence Interval) Percent of Target Achieved16 Overall Benef/Non￾benefic Percent of children 0-59 months regularly attending growth monitoring (OC2) (A/V) 94.0% 76.0% 23.6% (21.1%-26.1%) 31%/20.7% Can’t be calculated (baseline > target) Percent of mothers of children age 0-23 months who had four or more antenatal visits when they were pregnant with the youngest child17 (OC3) (A/V) 53.2% 82.0% 76.9% (74.1%-79.8%) 77.4%/76.7% 82% Percent of births with a skilled birth attendant (SBA) in USG assisted programs (OC4) (F/SL￾USAID) 52.3% 78.8% 91.3% (89.9%-92.8%) 95.4%/89.7% 147% Percent of mothers participating in postpartum/newborn visit within 2 days of birth in USG programs (OC5) (F/SL-USAID) 34.2% 94.2% 75.5% (73.2%-77.7%) 83.9%/72.3% 69% Percent of mothers receiving 3 postnatal visits (OC6) (A/V) 38.6% 90.0% 76.3% (74.1%-78.5%) 82.7%/73.8% 73% Percent of newborns (0-3 months) receiving essential newborn care through USG supported programs18 (OP5.1) (F-USAID) Not Estimated 87.0% 69.6% (61.1%-78.2%) 80%/66.7% Not available (no baseline) Percent of newborns (0-1 month) receiving essential newborn care through USG supported programs (OP5.1b) (F-USAID) 55.7% No target 50.0% (31.5%-68.5%) 66.7%/44.4% Not available (not target) Percent of child diarrhea cases treated in hospital/clinic/PHU (OC7) (SL-USAID) 21.1% 85.0% 82.6% (74.5%-90.6%) 90%/78.2% 96% The quantitative endline results of growth monitoring at PHU level and postnatal/newborn care, however, suggest limited integration of these activities in service delivery utilization. The 16 The formula for determining % of Target achieved is (Endline-Baseline)/(LOA Target-Baseline) *100 17 The baseline measured mother of children 0-59 months 18 Differences between baseline and endline (see Endline Report in Annex C) 24 SNAP Final Evaluation Report March 2017 International Solutions Group impacts of the EVD crisis on health systems were particularly evident in the growth monitoring of children, although SNAP reported a stricter measurement of the indicator (relying on evidence of growth monitoring and neonate care rather than self-reporting). In general, the number of people using PHU services increased because of SNAP, and PHU staff could manage the increased workload, with the assistance of community volunteers such as WASH committees.19 Qualitative research highlighted strong evidence of good collaboration between SNAP and local health authorities. Most of the MOHS representatives interviewed had substantive knowledge of the program, noting frequent consultations, supervision and frank cooperation. Local health authorities interviewed were aware of the SNAP close-out strategy and relevant stakeholders reported good participation in meetings with communities and beneficiaries. The collaboration between MOHS and SNAP went beyond strict implementation of the program. Financial and logistical support was provided by SNAP to the DHMT during mass campaigns for immunization, deworming and other health-related activities in the chiefdoms. Over the life of the SNAP Program, 43 HPs were identified and trained by the SO1 district supervisors on the four core training modules. SNAP conducted the training over a ten-month (non-continuous) period. Regrettably, due to low levels of female literacy in the communities, and family obligations, only two female HPs were among the 43 who completed the training. One of these was a former LM. HPs helped with the validation of Pregnant and Lactating Women (PLW) previously identified by LMs for the PM2A food distributions. HPs met with LMs each month to train them and to verify transmission of messages at the household level. The Evaluation found that in some villages the HPs were still meeting with LMs to teach and supervise them in their household meetings. Immediate LOA Impact. Despite some inconsistency between the quantitative and qualitative data, overall the Evaluation concludes that the SNAP Program and the training provided to the PHU health staff resulted in an increase in patient loads in PHUs mostly for ANC, FP and growth monitoring. The GoSL instituted a national policy making it illegal to give birth outside of a health facility and this, along with the messages promoted by SNAP, resulted in reports in program villages of reductions of maternal and child morbidity-mortality due to an increase in births attended by a skilled birth attendant. Mothers interviewed by the Evaluation Team noted that “no one dies giving birth here anymore”. SNAP promotion of family planning as part of BCC messages and the capacity building of PHUs, improved service delivery, demonstrating that addressing both the supply and demand side of health services works well. Since the HPs are part of the government structure and formally linked with the PHUs, they will probably continue to be present in the community. The main challenge for the sustainability of the impact achieved under this IR remains the “voluntary” aspect of the work of HPs, and the concern is whether they will have the self-confidence and the motivation to continue to coach the communities in health and nutrition issues. Long-term Sustained Impact. It is probable that women will continue to visit the PHUs for the foreseeable future. The DHMTs were also aware of their responsibilities for continuing to support PHUs and HPs as part of the sustainability plan that was negotiated with them by SNAP during the exit phase of the program. 19 The Evaluation Team, as part of the qualitative field research, observed that not all PHUs received the GIK package because the distribution of the packages was per PHU needs, and some PHUs were well-stocked. 25 SNAP Final Evaluation Report March 2017 International Solutions Group 2. WASH. As per other support under SO1, WASH activities were also focused at the PHU level with the construction of Ventilated Improved Pit (VIP) latrines, incinerators and handwashing stations. SNAP also distributed Bio Sand Filters (BSFs) for safe water access and provided some rehabilitation support to PHUs (painting, roof repair and crack repair). For both latrines and incinerator construction, SNAP followed national directives. At the community-level, handwashing stations called Tippy-Taps were introduced to PM2A beneficiaries through LMs. WASH committees, composed of volunteer community members nominated by their communities and trained by SNAP staff, followed MOHS and Ministry of Water Resources (MOWR) policies on promoting community sanitation and hygiene. Achievements. Table 6 summarizes the investments made by the SNAP Program in WASH activities. In addition, SNAP created and revitalized 123 WASH Committees to support hygiene promotion messages and activities at the PHU and in the community. Table 6. WASH Investments in SNAP (Baseline = 0) Indicator (source of indicator – FFP/USAID or A/V) LOA Target Actual Functional PHU Rehabilitation (OP 11) (A/V) 0 6 Yes VIP Latrines/shower blocks construction or rehabilitation (OP 11) (A/V) 0 23 Yes Bio Sand Filters Distribution (OP 11) (A/V) 98 98 (100%) Partially Incinerators constructed (n/a) 0 20 Yes Handwashing stations constructed (OP 11) (A/V) 0 21 Partially Major Observations. As indicated above, SNAP distributed BSFs to 98 PHUs. SNAP chose these PHUs at the beginning of the program with lack of proximity to a safe water point as the sole criterion. However, the Evaluation Team noted during the qualitative field research that all the PHUs visited had (at that point) access to a permanent protected water source (borehole or hand pump wells). The bio-sand filters inspected were rarely used, and most filters had been removed from the PHUs by program staff. Bio sand filtration is not common in Sierra Leone due to high maintenance needs (every 3 months the sand needs to be washed and changed). However, it remains a good choice in situations where water sources are unsafe or during the dry season when the water in the wells is not available. Findings from the qualitative research confirmed that people were aware of the effectiveness of boiling water to make it safe for drinking, but interviewees reported this to be rarely practiced. Water filtration with a fine cloth is more common when safe water is not available. In promoting the Tippy-Taps as a community-adapted method for handwashing, SNAP provided jerry-cans and ropes for the facility construction. While the Evaluation Team rarely observed tippy-taps functioning in the villages, they did observe community members coping with the lack of handwashing stations via using the additional equipment, buckets of water and soap in some of the latrines visited. This mechanism was directly linked to the hygiene promotion messages received. Unused Bio-Sand Filters at a SNAP Office 26 SNAP Final Evaluation Report March 2017 International Solutions Group The Evaluation found that WASH committees reviewed were consistently composed of five men and two women (the composition of the WASH committee as established by DHTM which SNAP replicated). These committees were active in maintaining the PHU facilities and sensitizing communities on a bi-monthly basis on hygiene, mainly handwashing. SNAP provided training to the WASH committee members and a kit of cleaning materials (cutlass, hoes, shovel, and rake) which was distributed at the end of the training. A savings box was in place to buy spare parts in case the hand pump needed maintenance. WASH Committee members mobilized communities to improve feeder roads and connect hard-to-reach villages. The MOWR, DHMTs and District Councils worked in partnership with SNAP, and the Evaluation found no evidence of duplication or overlap with WASH committees implemented by other NGOs. During the EVD outbreak, hygiene messages were diffused via megaphone by WASH committees, with the primary messages focused on personal contact and handwashing. Immediate LOA Impact. Exposure to proper hygiene and sanitation messages focused on handwashing significantly reduced incidence of water-borne disease. Since PHUs already had access to safe water, impact of the bio-sand filters is negligible. Long-term Sustained Impact. Handwashing and personal hygiene behaviors are likely to continue as result of the sensitization, but the presence and engagement of WASH Committees after the program ends is much less assured. Without ongoing reinforcement of health and hygiene messages, sustainability beyond the medium-term is not assured. 3. Acute Malnutrition Screening. SNAP provided training and material for screening and referrals of Moderate and Severe Acute Malnutrition cases (MAM/SAM) for both PHU staff and community health workers using MUAC tapes. Achievements. The Evaluation Team could not determine how many MUAC tapes were distributed nor the number of people trained in their use. The Evaluation did determine that they were widely available and that men were able to use them and understand the readings. SNAP kept records on community screening using MUAC, but this information was not made available to the Evaluation. The program also supported PHUs for growth monitoring and nutritional surveillance. Major Observations. CMAM screening campaigns were organized in collaboration with PHUs, and SNAP worked with the DHMTs to disseminate CMAM on-the-job trainings/supervision to service providers. During the Ebola Virus Disease (EVD) outbreak and following the “no￾touch” national policy, LMs and community mothers (both SNAP food and non-food beneficiaries) were taught to measure their own children with MUAC tapes for self-screening. A MAM/SAM pictorial was distributed to help the identification of acute malnutrition signs. Mainly due to the self-assessment and behavior messages, during the Ebola epidemic only mild increases of malnutrition cases were reported in SNAP intervention’s areas during the EVD outbreak. The uses of MUAC by mothers is also resulting in reduced PHU staff workload for monitoring acute malnutrition. LESSONS LEARNED – WASH Communities should be involved in the planning and assessment of appropriate safe water solutions. Introduction of innovations as bio-sand filter should be carefully motivated and tested before implementation. Tippy-tap construction is inexpensive, adaptable to local material and sustainable but should be implemented without material donation to avoid dependency. 27 SNAP Final Evaluation Report March 2017 International Solutions Group Immediate LOA Impact. The Evaluation Team saw evidence of a reduction in the number of acute malnutrition cases in villages visited. Respondents also reported that children identified with acute malnutrition using the MUAC were referred promptly before the onset of the most severe symptoms. This impact is clearly linked to the self-assessment check-up, and the education given to the mothers to recognize malnutrition. Long-term Sustained Impact. The long-term sustainability of the community screening practice will depend on the knowledge retention of the health providers and the mothers, which in turn will require resources and reinforcement. MUAC tapes are easy to use, however, and readily available when they need to be replaced. 28 SNAP Final Evaluation Report March 2017 International Solutions Group B. Outputs Produced Under IR 1.2: Improved health and nutrition practices at the household level The SNAP Program produced three sets of outputs under IR 1.2, including implementing BCC strategies through a MCG approach; distributing PM2A food rations; and building capacities of mother care groups to undertake kitchen gardens. 1. Behavioral Change Communications Through a Mother Care Group Approach. SNAP adopted the MCG approach model to promote behavior change messages to PM2A households via LMs. This approach was implemented from the beginning of SNAP in 2010 and imitates the Mother-to-Mother (MtM) model used in Sierra Leone by the MOHS. LMs were chosen by the PM2A mothers for their motivational skills and their engagement at the community level. They were members of the community and also food beneficiaries. LMs received training from HPs on four modules that included EBF, complementary feeding, FP and hygiene, and they met with the HPs once a month. Each LM was working with 7 to 15 households targeted for PM2A food rations with whom they met twice a month to pass on and discuss the lessons they learned. These meetings were held mainly at the mother’s household but groups also gathered in public spaces. Each LM received a set of pictorials flip-charts to display the BCC messages and a “recognition kit” composed of t-shirts, lappas (a two meter piece of cloth used as a wrap), an ID card, and a cooking set after graduation. In 2014, following the Mid-Term Evaluation (MTE) and recommendations from the SNAP Gender Strategy, a Men as Partners (MAP) approach was introduced in SNAP. This model paralleled the MCGs, but focused on men/husbands. A Lead Father (LF) was chosen from a group who received a four-day training on EBF, complementary nutrition, FP and hygiene promotion. The LF also received a set of gender-oriented pictorial flip-charts is to facilitate passing on the messages to other men. The MAPs were trained by both the SNAP gender specialist and the SO1 district supervisors. As for USAID demand and to demonstrate that positive behavior change could be achieved through BCC alone, in 2013 SNAP started to track new beneficiaries, ultimately called non￾food beneficiaries, to monitor those people that joined BCC groups (MCGs and MAPs) without the benefit of food. Achievements. Over the life of the SNAP Program as per data received from IMC, 3368 LMs were trained and 1569 of them were graduated. At the time of the Evaluation, these LMs were included in 180 active Mother Care Groups (92% of the LOA target). In addition, 41 Men as Partners Groups were created by SNAP comprised of 615 lead fathers and 8,610 men organized into husband schools. 29 SNAP Final Evaluation Report March 2017 International Solutions Group Table 7 provides information on the IPTT indicators for IR 1.2. Table 7. IPTT Indicators for IR 1.2 Indicator (source of indicator – FFP/USAID or A/V) Baseline LOA Target Endline (95% CI) % of Target Achieved (end-base)/(target-base) Overall Benef/Non￾benefic Percent of children 0-5 months of age exclusively breastfed (OC9) (FFP) 39.5% 75% 75.6% (69.9%-81.2%) 69%/78.2% 101% Percent of caregivers demonstrating proper food hygiene behaviors (OC10) (FFP) 78.5% 76% 56.1% (53.5%-58.7%) 69.4%/50.4% Can’t be calculated (baseline > target) Percent of caregivers demonstrating proper personal hygiene behaviors (OC11) (FFP) 87.4% 69% 90.5% (89.0%-92.0%) 95.7%/88.4% Can’t be calculated (baseline > target) Major Observations. As proposed by SNAP, the MCG approach was launched at the beginning of the program with the selection of LMs to help the identification of pregnant women eligible to be enrolled in the food distributions. Most LMs received training on the four-modules and had in place monitoring tools to check their home visits. However, the most recently identified LMs that joined the program at the end received just two or three of the four training modules, and some of them did not receive the recognition kit nor the flipcharts as the production of the flipcharts by the program had already stopped. By and large, the MOHS adopted the counseling materials and posters developed by SNAP in a harmonized version of the MtM/MCG manuals. These materials had been fully tested and carefully developed in other countries by IMC, and the adoption of the materials by the MOHS will facilitate wider replication of the impact of SNAP. The Evaluation qualitative research found that some LMs were focused on working with households that had benefitted from SNAP food distributions, and were not paying enough attention to other households in the village that did not have a pregnant woman, but who could still benefit from the SNAP messages. The Evaluation Team also noted other LMs who were visiting non-food beneficiaries. In most cases, LMs were still visiting the PM2A households even though food distributions had ended in 2014. Generally, LMs interviewed expressed a perception of increased status amongst their peers in the community, with the qualitative observations of their interaction with community members supporting the conclusion that they are recognized as important members of their communities. EBF, handwashing, and FP messages were the messages best understood by SNAP beneficiaries that were interviewed as part of the Evaluation research. Stakeholders noted that the trainings also increased women's empowerment, giving them more confidence to speak during public gatherings. The Evaluation qualitative research found, however, that beneficiaries in isolated communities did not practice the BCC messages with the same intensity and many appeared to have forgotten most of the teachings received by LMs. LESSONS LEARNED – MEN AS PARTNERS and BCC ADOPTION Creation and implementation of Men as Partner approach should be done at the same time of Mother Care Groups to reinforce the BCC messages and drive an inclusive change in behaviors. Continuing to spread health and nutrition actions at household and community level with the involvement of HPs, LMs and LFs will avoid to forget the behaviors acquired. Harmonized MCG & MtM Manuals 30 SNAP Final Evaluation Report March 2017 International Solutions Group Further, while additional (informal) analysis of quantitative survey responses by program participation did suggest better performance among participants with respect to personal and food hygiene, fewer program participants were exclusively breastfeeding children vs. non￾participants (69% vs.78%). These figures are within the indicator confidence intervals (69- 81%), however, so are interpreted with caution. The MAPs approach appeared to the Evaluation Team to have been effective despite having started late in the life of the program. However, the qualitative research indicated that the MCGs in isolated communities was not very effective, however, with men in remote villages less active in practicing FP or gender messages. IMC targeted a number of pregnant women to sign up as non-food beneficiaries but unfortunately this initiative was disrupted by the EVD outbreak. When the crisis abated and care group activities resumed, many of these women began to participate in the BCC activities. A total of 13,874 non-food beneficiaries including new mothers and men as partners signed-on in 2015/2016 under SO1. These participants participated only in BCC messaging activities. Immediate LOA Impact. Despite entry criteria confusion at the beginning of the program (selecting both pregnant and lactating women – see Section III B.2: Participant Targeting for more details) and the short implementation period for the MAPs approach, most BCC messages promoted by SNAP were broadly adopted. The awareness and practicing of exclusive breastfeeding is likely to have contributed significantly to demonstrated increases in child weight, as reported by the beneficiaries, and better bonding between mother and child. New mothers at the time of research were still joining the Care Units (CUs) without the incentive of the food rations because they see the improved health status of the children who participated in the program. Both the sensitization through the LMs and the integration of the different components of the Title II program resulted in husbands and fathers contributing to increasing the awareness of men about the importance of supporting their wives and children, particularly with household chores and accompanying their spouses to the PHU for ANC. Men interviewed as part of the qualitative research articulated a growing understanding that women need to reduce their physical workload during pregnancy and eat a variety of food. Long-term Sustained Impact. The qualitative and anecdotal evidence suggests that many SNAP LMs will likely continue to visit their households because of the continuation of the Mother￾to-Mother approach as per the MOHS health and nutrition community-based strategy, a good example of how the SNAP program activities integrate into longer-term initiatives to sustain gains. 2. Food Rations. Food ration distributions for PM2A beneficiaries started in mid-2011 and were planned to end in November 2014. Due to EVD outbreak, however, SNAP suspended distributions in July 2014, with commodities in the pipeline re-directed to quarantine and post￾quarantine households covered by the SNAP Plus (SNAP +) Program20. Per the policy of FFP, programs are allowed up to 10% of development commodities to be diverted with approval to emergency uses. Commodity management and reporting on these commodities remained under SNAP. In the SNAP PM2 Approach, the entry criterion for receiving food rations was pregnant women from the fourth month of pregnancy, with exit set at 24 months of child age. To be considered “active” and eligible to receive food, the mother should participate in the bi-weekly visits with 20 SNAP + was a one year emergency program funded by FFP Emergency Funding working relatively independently from the SNAP Program. 31 SNAP Final Evaluation Report March 2017 International Solutions Group LM and attend ANC or other health services at a PHU, such as growth monitoring. In order to receive rations, the pregnant woman must have received an ANC card from the PHU. Food distributions took place once a month at a PHU, following a fixed-day calendar. If a PLW was unable to attend the distribution, a designated alternate could collect the rations for a maximum of two consecutive months. After that period, if the beneficiary had not appeared, she was withdrawn from the food program. SNAP put in place a system for tracing absentees using the HPs and the LMs to verify ineligibility. The SNAP food ration for PLW and children under two (CU2) was comprised of Corn-Soy Blend (CSB), vegetable oil and lentils. A protection family ration was also distributed during the lean season (June to September only), composed of bulgur, vegetable oil and lentils. At the time of the design, the SNAP Program proposed to distribute 16,050 Metric Tons of food to 50,064 mother-child pairs and their families. Achievements. ACDI/VOCA, in its annual reports, noted a total of 8,150.61 Metric Tons of food distributed as PM2A rations to 32,245 households over the life of the SNAP Program. Major Observations. Entry criteria to enroll participants in the program were not entirely clear at the beginning of SNAP, and this resulted in the registration of both pregnant and lactating women in the first year. This situation was corrected, however, before the start of food distributions with deregistration of the lactating mothers. PM2A mothers interviewed expressed satisfaction with the food ration composition and attributed the wellbeing of their children to this. The Evaluation found no evidence of sales of USAID commodities in local markets, nor, did the team identify evidence of the rations being an incentive to women to become pregnant. There was considerable qualitative and anecdotal evidence, however, that the food rations, including those designated exclusively for the PLW and CU2, were shared within the household, with the extended family, and also with neighbors. While sharing of food is a normal practice in Sierra Leonean families (and SNAP did include a supplemental ration to account for immediate family members sharing food) the Evaluation concludes that designers did not anticipate the scale of the (widespread) sharing of rations with extended family and neighbors. Immediate LOA Impact. Focus group discussion respondents consistently reported that the food distributions had improved the nutritional status of children, attributed mainly to the consumption of CSB. Respondents also reported an increase in use of PHU services, mainly for ANC visits, due to the conditional delivery of the commodities through the PHUs. Participants had clearly become more accustomed to and more comfortable using PHU services because of the food distributions21 . Long-term Sustained Impact. The findings of the qualitative research suggest that the nutritional benefit from food distributions was less than anticipated, a finding supported by the evidence noted above that food rations were widely shared beyond immediate family members, thus diminishing the long-term impact from the food on chronic malnutrition. The Evaluation concludes that the BCC activities and better access to PHU services had greater impact on preventing child malnutrition than the food distributions. The biggest benefit from the rations was the increased use of PHU services by participants as a result of the conditional food rations being delivered through PHUs. The sustainability of these relationships appears likely unless the quality of services in PHUs declines because of staff turnover or insufficient materials. The 21 The Evaluation considered making the use of food as incentive in this way a good practice (Section VII) because of its effectiveness. This was rejected, however, because the use of food in this way was not very efficient. 32 SNAP Final Evaluation Report March 2017 International Solutions Group SNAP food distributions were discontinued in 2014, however, and the final evaluation found many participants still using PHU services, underscoring the likelihood of ongoing facility use. 3. Kitchen Gardens. In addition to the health and nutrition messages, the concept of kitchen gardens to promote diet diversity and production of nutrient-rich crops was taught to LMs. SNAP also created a cookbook with Sierra Leonean recipes, and food demonstrations were held as a complementary activity to gather mothers together and teach them how to use local food produced in kitchen gardens. Achievements. A total of 45 women's vegetable groups with 1,350 members (1269 women and 81 men), mostly from PM2A Care Units, were identified, trained and supported in Koinadugu and Kailahun Districts. These groups received nine different types of seed (carrot, okra, onion, pepper, tomato, cowpea, crain crain, greens and sweet potato) and assorted tools to prepare group vegetable plots with technical support from SO2 staff. In FY 2014 and FY 2015, the 45 groups harvested a total of 4,892 kilograms of production from the 9 crops. An estimated 61% of the production was consumed by the household, and 39% was sold in the market, generating SLL 1.5 million in revenue. Major Observations. Support for women's vegetable groups started in the second half of 2014 and continued until September 2015. Mainly due to the EVD outbreak, the vegetable groups dissolved, and by March 2016 only nine out of the original forty-five groups were still functioning as per the program's assessment following the second round of seed distributions. To eliminate the need for continuing seed distributions, the program reminded groups that additional seed was not forthcoming from the program and that they should plan to access capital from VSLAs to purchase seed in the future. The Evaluation found that the messages on kitchen gardens and diet diversity were not well adopted or retained, even though cooking demonstrations were held with LMs and a cookbook was tested and distributed in 2015. Cultivation of non-traditional vegetables like carrots and beans was not well-understood and not adopted. Some groups at the time of research were still continuing, however, with pepper and okra, crops that are well known. Immediate LOA Impact. Since food diversification messages were the least adopted and the establishment of kitchen gardens was started late in the program and was focused in just two districts within SNAP implementation area, the immediate impact of the kitchen gardens component was minimal. However, some of the training provided was, at the time of research, still being used by vegetable group members who were able to access individual loans or share￾outs from the VSLA to purchase seed or hire labor to prepare gardens or farms. Long-term Sustained Impact. Since there was little immediate impact, there is also not likely to be long-term sustained impact from the kitchen gardens promotion activities. If households were not producing kitchen gardens with diverse, nutrient-rich crops by the time of the Evaluation, they are unlikely to decide to do so in the future because of the SNAP intervention. 33 SNAP Final Evaluation Report March 2017 International Solutions Group C. Outputs Produced Under IR 2.1: Increased agricultural productivity at the household level Four major outputs were produced under IR 2.1 in the SNAP program. These were the creation of Farmer Field Schools (FFS) and FFS Clusters, capacity building of Community Facilitators, the transfer of new technology and knowledge to farmer participants, and cultivation of business linkages between FFS Clusters and various input suppliers and buyers. 1. FFS & FFS Clusters. SNAP proposed to establish new FFS or work with existing FFS's and to build upon the approach used by the PAGE Project to create rural private sector enterprises by clustering producer groups, i.e., the FFSs, and linking them to local, regional and international end markets. The FFS, as used in SNAP, is a long duration "learning by doing" technology and knowledge transfer methodology that enables farmers to actively observe the effects of new practices through comparison trials with traditional practices. Farmers participating in the SNAP FFS were organized into groups of 25 to 30 persons from households participating in SO1, targeting both men and women, in many cases, from the same household. The FFSs were provided with training over the course of a full cropping season from seed procurement or provision through harvest on Agro-Ecosystem Analysis (AESA) and guidance on Good Agricultural Practices (GAP). In addition, the SNAP Program provided training to FFS on Farming as a Family Business (FaaFB) and messages on the essential nutrition and essential hygiene actions being promoted under SO1. After a year of FFS implementation, SNAP formed FFS Clusters composed of two or three FFS located in proximity to each other. The purpose of the FFS Cluster was to facilitate learning between FFS while also establishing group farms for producing food and income. Depending on the crops selected by a cluster, the program provided seed to get the production established. The program also facilitated formal registration of clusters as farmer-based organizations (FBOs) within the GoSL strategy for promoting agricultural development. In the last year of the program life, cross-visits were facilitated between FFS Clusters so that they could share experience and learn from each other. Achievements. Over the life of the SNAP Program, a total of 955 FFS were formed with approximately 28,650 members of which 54% were women. These 955 FFSs were formed into 302 FFS Clusters. Table 8 below summarizes information on IPTT indicators for FFS & FFS Clusters. Table 8. IPTT Indicators for the FFS/FFS Cluster Component Indicator (source of indicator – FFP/USAID or A/V) LOA Target LOA Actual/ Endline (95% CI) Percent of Target Achieved (Baseline = 0) Overall Benef/Non￾benefic Percent of farmers reporting use of association benefits (OC16) (A/V) 75% 14.9% (12.7%-17.2%) 29.1%/8.4% 20% Number of FFS groups graduating into producer or marketing associations (OC21) (A/V) 624 906 n/a 145% # of individuals who have received USG supported short term agricultural sector productivity training (OP20) (F/SL-USAID) 27,710 32,430 n/a 117% The first indicator shows a surprisingly small percentage of respondents in the Endline Survey who said that they used association benefits. The survey was population-based so the sample includes a significant number of respondents who were not project participants and therefore not members of a FFS Cluster, thus skewing results. Disaggregation of respondents to the Endline Survey indicates that 29.1% of FFS Cluster members indicated using association 34 SNAP Final Evaluation Report March 2017 International Solutions Group benefits, with 8.4% of non-members noting the same. Therefore, while the overall utilization rates were low, they are significantly higher for FFS members. Further, the qualitative interviews with FFS Clusters indicated that most were focused on production on a group farm. The harvest was usually stored until a suitable market is found, but a respondent in the Endline Survey may not consider this using association benefits, since he or she did not place his/her own harvest with the group harvest. None of the FFS Clusters interviewed were purchasing bulk inputs. SNAP defined the second indicator as those FFSs formed into FFS clusters, which are defined as producer or marketing associations. A total of 955 FFSs were formed by the program, and 906 of these were clustered into 302 FFS Clusters. For the third indicator, the number reported as actual was the sum of the persons trained in each fiscal year and likely includes duplication since many persons participated in more than one training. A better estimate of the actual number of persons trained by the SNAP program would be the sum of FFS members, estimated at 28,650 plus the 193 Community Facilitators who were trained. Major Observations. The approach used by SNAP with the introduction of an experiential learning process with comparison trials in the FFS, followed by support for FFS Clusters for agricultural production was an effective mechanism for building farmer technical capacities that can then be applied in significant scale production. The approach also engaged a significant number of women, with many of these in leadership positions. The qualitative interviews verified that a very high proportion of FFS participants were from the same households participating in SO1, so there were synergies between the nutrition education benefits from SO1 and the food production and income benefits from increased production through FFS Clusters. The FFS concept was new in most villages, and all FFSs were newly formed by the program. SNAP did not work with any pre-existing FFSs formed by other projects. In a few cases, the program discontinued forming a new FFS in a location because participants expected to receive “Food for Work” as part of their engagement with SNAP. For the most part, membership in an FFS Cluster was restricted to members of the FFSs that were grouped to form the cluster. The qualitative evaluation found a few cases, however, of FFS Clusters expanding membership to include other farmers, who were not members of an FFS. The Evaluation also found a few cases of FFS members withdrawing from an FFS Cluster, mainly so that they could work on their own farms with the labor that they would have had to dedicate to the FFS Cluster group farm. The qualitative evaluation found that almost all clusters were still producing on cluster farms, generating income and food for members. Most reported that harvested production was used in three ways. One part was kept for seed for the following season, one part was kept for sale, and the third part was divided among the members for them to use as they chose, either for consumption or sale. Meetings were held shortly before harvest with all members, or their representatives, to make decisions on how to use the harvest. The Evaluation Team identified a few highly successful FFS Clusters that were expanding their activities in significant ways, e.g. purchasing a small truck in Koinadugu, cassava processing in Tonkolili and Kailahun, and purchasing rice mills in Kailahun. 35 SNAP Final Evaluation Report March 2017 International Solutions Group Immediate LOA Impact. The immediate impact of the FFS/FFS Cluster was a function of the food supply and income benefits obtained from improved production on own farms and the production from the FFS Cluster group farm. In addition, the FFS Cluster provides a platform for accessing additional support from government or other projects. Long-term Sustained Impact. The long-term sustainability of the FFS Cluster depends primarily on the quality of the leadership of the cluster and whether members will continue to receive enough benefits to ensure their on-going participation. The Evaluation found that, for the most part, FFS Cluster leaders were dynamic and committed to their roles in leading cluster activities. The Evaluation also found, however, variation in the quality of production on group farms, ranging from a few groups that did not have a farm, to groups that had farms but the quality of the production was low, to other farms which were showing good quality production. The reasons given for not having a farm presently were that it was the off-season for the crops that had been chosen, but many of these groups could not discuss specific plans for the coming season. The reason most often given for low production was that it was difficult to mobilize members to work on the farm. The farms showing good production at the time of research are likely to continue to function. It remains to be seen whether the first group will continue to function, and the second group will likely become dysfunctional soon. 2. Community Facilitators. To be able to work with a significant number of FFS, the SNAP program identified and built the capacities of community-based training facilitators called Community Facilitators (CFs). Each CF trained by the program was expected to work directly with three to five FFS's, setting up comparison trials in demonstration plots and passing on the ideas that they had received in their training, and they continued to work with FFS Clusters after they had been formed. CFs received a small monthly stipend for their time. Achievements. The SNAP Program trained 193 community facilitators. The IPTT did not have any specific performance indicators related to CFs. Major Observations. The Evaluation Team found that the Community Facilitators interviewed were well-motivated individuals engaged not only in CF-related work but also in other livelihoods activities, such as their own farming or other agri-business activities. Those interviewed, for example, were producing maize seed, testing integrated IVS rice/fish production, working with their own CBO or involved in produce marketing. Most were high school graduates or attending school. They reside in FFS communities and were members of FFS Clusters, enjoying the same benefits as other members. Many were still motivated to serve as CFs, although the main benefits that they had received while SNAP was being implemented, i.e., the small stipend, access to improved seed and access to new ideas, were all either suspended or reduced with the closure of the program. FFS Cluster Farm Showing Good Production FFS Cluster Farm Showing Poor Production 36 SNAP Final Evaluation Report March 2017 International Solutions Group The program exposed CFs to different sources of new ideas and technologies through the trainings provided by experts from the Sierra Leone Agricultural Research Institute (SLARI), the Sierra Leone Indigenous Business Association (SLIBA) and other organizations, but the program did not explicitly link them to these sources so that they could continue to access new ideas and technologies from these sources post-SNAP. The program also made both the MAFFs extension staff in each district aware of the existence of the CFs and CFs aware of District MAFS staff and the Frontline Extension Workers (FEWs) of MAFFS who were nominally responsible for their communities. In neither case, however, could either the CF or the MAFFS staff specifically state names and phone numbers of the other in interviews during the Evaluation, indicating a superficial relationship at present. MAFFS staff at both the national and district levels clearly recognized that the CFs provide an opportunity to further extend their extension structures, but they also said that MAFFS did not have the resources to compensate CFs nor was it easy to be recruited through the government staff recruitment systems, even if recommendations were made to hire specific CFs to fill MAFFS vacancies. MAFFS staff also expressed an interest, however, in otherwise trying to find a way to work with CFs. Immediate LOA Impact. As conduits for extension messages, CFs served a critical function during the life of the SNAP Program in supporting FFSs to test new ideas and technologies and in supporting FFS Clusters with production advice and marketing strategies. They also benefited from their roles with recognition as experts, as well as with tangible personal benefits from the new ideas and technologies introduced by the program. Long-term Sustained Impact. In the absence of new ideas or technologies after the program ended, most CFs interviewed expressed commitment to devoting more of their time to their own livelihoods pursuits rather than repeating the same extension messages that were promoted by SNAP. In many cases, this may involve them staying engaged with FFS Clusters to benefit much as other members benefit. In many other cases, however, CFs may no longer be identified as CFs but will continue their education or engage in other occupations. 3. Technology and Knowledge Transfer. Through the FFS, farmers were exposed to concepts of Agro-Ecosystem Analysis for analyzing the farming agro-ecosystem and Good Agriculture Practices (GAP) including plant spacing, seed density, planting time, more frequent weeding, proper harvest time, nursery preparation and management, thinning after germination, and new crop varieties. The GAP were focused on nine types of crops over the life of the program, including inland valley swamp (IVS) or lowland rice, ground nut, maize, cassava, sweet potato, pigeon pea, sesame, cowpea, and sorghum. The program also promoted, with the support of SLIBA, messages on proper post-harvest handling and storage of crops. SNAP provided seeds and tools to FFSs and start-up seed to FFS Clusters groups. SNAP also proposed to build the capacities of existing local input suppliers, youth groups or nascent agro-enterprises formed by vocational skills training graduates, FFS clusters interested in becoming input suppliers, or women's associations to improve their competitiveness and efficiency in making inputs available. Lessons Learned - Community Facilitators The SNAP Program built capacities of community-based semi-volunteers to serve as local extension agents. The challenge to sustaining these people in this role was to ensure sufficient tangible or intangible benefits to motivate them to continue, such as recognition of their efforts. A suggestion by senior staff at MAFFS in Freetown was to find ways for CFs to provide services for a fee much as para-vets provide vaccines and vaccination services. Examples include linking CFs to input suppliers to sell inputs on commission, building capacities of CFs to serve as market brokers or building CF capacity to conduct internet searches on problems or new technologies. 37 SNAP Final Evaluation Report March 2017 International Solutions Group Achievements. Over the life of the SNAP Program, 32,439 participants (not unique beneficiaries, i.e. some received more than one training) were exposed to new technologies and ideas either directly or through CFs. A total of 102.612 MT of seed (ground nuts, rice, sesame, sorghum, pigeon pea, cow pea, maize) and 918 bundles of cassava and sweet potato cuttings were also distributed. Eighteen agri-businesses, including government-supported Agri￾Business Centers (ABCs) and private firms were supported with grants and training, and fifteen FFS Clusters were trained as input suppliers. Table 9 provides information on IPTT indicators related to technology and knowledge transfer. Table 9. IPTT Indicators Related to Technology and Knowledge Transfer Under IR 2.122 Indicator (source of indicator – FFP/USAID or A/V) LOA Target LOA Actual/ Endline (95% CI) % of Target Achieved (Baseline = 0) Overall Benef/Non￾benefic Percent of farmers who have adopted at least three agricultural technologies (OC17) (FFP) 75% 25.5% (22.7%-28.2%) 45.2%/16.4% 34% Percent of farmers who have adopted at least one agricultural technology (OC17.1) (FFP) No target 30% Not available 90%23 Number of farmers, processors, traders, input suppliers, and others who have adopted at least three new technologies or management practices (OC18) (SL-USAID) 29,38324 6,152 Not available 21% Number of farmers, processors, traders, input suppliers, and others who have adopted at least one new technology or management practice (OC18.1) (SL-USAID) No target 7,099 Not available n/a Number individuals reporting consumption of micronutrient rich crops as a result of FFS training (OC22.1) (A/V) 308,200 98,438 Not applicable 32% Number of new technologies or management practices made available for transfer as a result of USG assistance (OP21) (SL-USAID) 6 7 Not applicable 117% Number of demonstration plots established (OP22) (A/V) 1,240 955 Not applicable 77% The endline estimate on adoption of agricultural technologies (OC17 – see row 1 of the above table) is deemed by the Evaluation Team to be unreliable due to how the interview questions were structured. Respondents were first asked whether they had engaged in any agriculture activities in the past five years, and only respondents who indicated yes to this question were asked the more detailed questions on adoption. Only 52.3% responded yes to this question which was inconsistent with another question in the survey in which 91.6% of the households surveyed reported growing a crop in the previous year. The qualitative interviews triangulate with the latter number as being more realistic. Nearly all respondents in focus group discussions reported having produced a crop in the last year, and many respondents noted that seed-spacing and density training was adopted. Therefore, the Evaluation Team feels the percentage who engaged in agriculture activities in the past five years should be this high or higher. Nonetheless, despite this measurement discrepancy, considerably more program participants reported adoption of technologies than non-participants (45.2% vs. 16.4%) as reported by the 22 Subsequent to the completion of this report, revised data was presented by the contracted end-line/quantitative survey company in response to data quality concerns. This data is included in the annexes to this report and suggests that anywhere from 70%-90% of farmers made sales on each crop they grew in the last 12 months. 23 This data is from the revised agricultural data analysis, see Annex D. 24 ACDI/VOCA reports that it over-estimated this target during project design by including all SO2 beneficiaries including non-production beneficiary VSLA members 38 SNAP Final Evaluation Report March 2017 International Solutions Group Endline Survey. The revised agricultural data (presented in Annex D) indicates adoption rates of at least one new technology of 90% The IPTT also had performance indicators related to yields and household crop sales as shown in Table 10. The endline estimates for yields are from the Endline Survey (see Annex C). The estimates for sales are taken from the IPTT, but the source of the information is unknown. The numbers are inconsistent with the estimates generated in the Annual Survey for FY 2015, the last annual survey undertaken by the program. Table 10. IPTT Indicators Under IR 2.1 for Crop Yields and Sales Indicator (source of indicator – FFP/USAID or A/V) Crop Baseline LOA Target LOA Actual/ Endline (95% CI) Percent of Target Achieved (end-base)/(target￾base) Overall Benef/Non-benefic Average Yield, Kilograms per Hectare (OC19.1) (A/V) Ground Nuts 310 418 710 (669-751) 764.39/681.25 370% Lowland Rice 610 771 748 (705-790) 819.93/710.52 86% Cassava 2102 445.5 761 (697-824) 799.81/737.87 Can’t be calculated (baseline > target) Sweet Potato 3581 1,003 581 (482-679) 664.62/528.42 Can’t be calculated (baseline > target) Average Value of Crop Sales (OC20.1) in SLL (A/V) Ground Nuts 215,587 377,277 396,271 SLL 1,090,639.52/ 732,268.65 (overall average) 112% Lowland Rice 31,170 48,314 408,600 2202% Cassava 39,170 60,714 251,054 983% Sweet Potato 11,650 18,058 145,314 2086% Major Observations. Yield estimates as shown in Table 10 should be viewed with caution since production units used by farmers vary from district to district and area estimates are subjective. Nevertheless, qualitative interviews verified that yields increased, especially for ground nuts and cassava, mainly because of the seed/cutting spacing promoted by the program. The Evaluation reports also heard positive reports on yields and quality of the sweet potato varieties promoted by the program. The limited change in yields of lowland rice demonstrates some inconsistency with participant positivity during qualitative research about the Nerica L-19 variety. The Evaluation Team consistently observed poor water management in lowland rice, however, which certainly limits yields of most improved varieties. Importantly, further informal analysis of data by program participation, which not statistically robust (as no formal control group was established) indicates consistently higher yields among SNAP participant households, supporting the hypothesis that SNAP participants outperformed more general improvements in yields. As indicated in Table 9, adoption rates of the new ideas and technologies promoted by the program were much lower than LOA targets, and in focus group discussions held during the Evaluation, participants confirmed that the adoption of new ideas and technologies that they learned from SNAP were relatively limited. The most consistently cited new ideas were plant spacing (often described in terms of planting in lines, but when queried on the real benefit, plant spacing surfaced), seed density, and the Nerica L-19 lowland rice variety. Interviewees occasionally noted other ideas, but clearly these were the most important new ideas and technologies brought by the program. One of the biggest benefits of the program was the provision of high quality seed and other planting materials. The qualitative interviews found that the Nerica L-19 and sweet potato 39 SNAP Final Evaluation Report March 2017 International Solutions Group varieties, in particular, were being disseminated to others either freely by SNAP participants or via seed loans made by FFS Clusters to non-members. The FFS approach typically uses a so-called season-long approach to training FFS facilitators on all stages in the cropping cycle before they work over a season with an FFS. This practice was not utilized in SNAP. The training was conducted over a shorter period, and facilitation skills were augmented by on-the-job training with support provided to CFs by SNAP technical staff once the FFSs got underway. Immediate LOA Impact. Increases in yields and crop sales indicate more food and income for program participants. While adoption of large numbers of new ideas and technologies was not seen from the qualitative and quantitative data, those few that were taken up were seen to be having the desired impact. Long-term Sustained Impact. Good ideas are resilient, and the ideasthat the qualitative research indicate were being adopted by participants, especially plant spacing and seed/seedling density, are simple concepts that have a strong likelihood of being continued and passed on to others and future generations. 4. FFS Cluster Business Relationships. The SNAP Program proposed four sets of activities to facilitate access to markets for SNAP participants. These were to (1) form groups and make market information available to them while integrating them into existing marketing systems, (2) link FFS groups to established producer and marketing associations supported by PAGE, (3) facilitate access for farmer groups to WFP's Purchase for Progress program, and (4) create a cadre of local mid-level traders or middlemen. Achievements. SNAP built the capacities of the 302 FFS Clusters formed by the program to be able to identify market opportunities and develop business plans to capitalize on these. The development of a business plan was part of the requirement for FFS/Clusters to become registered as Farmer-Based Organizations, and most FFS/Clusters had become registered as FBOs. Over the life of the program, but especially in the last year, SNAP facilitated the development of business relationships between FFS Clusters and the following buyers and processors. ➢ 7 FFS Clusters in Kailahun with Moamaleh Marketing Association (rice), Ihitiama Marketing Association (rice), and the Rural Initiative Development (rice) ➢ 41 FFS Clusters in Koinadugu with Sabatie ABC (pigeon peas, ground nuts), Karakontor ABC (pigeon peas), Koinadugu Women's Vegetable Farmers Cooperative (rice, ground nuts), and Sorbeh ABC (pigeon peas) ➢ 3 FFS Clusters in Tonkolili with Bonka Serie ➢ 2 FFS Clusters in Bombali with Big Things Poultry Farm (maize) ➢ Tadaranie for rice, groundnuts and sesame ➢ Sanda Integrated for cassava cuttings ➢ Bennimix for sesame ➢ Other individual buyers based in district markets The program supported the engagement of FFS Clusters in the WFP P4P Program, building on work that was initiated in the PAGE Project. SNAP facilitated business relationships between these PAGE-supported buyers (Sabatie ABC and the Koinadugu Women's Vegetable Farmers Cooperative in the list above) and FFS Clusters producing rice, pigeon peas and ground nuts. Beginning in FY 2013, SNAP expanded partnerships with SLARI (training on maize, pigeon pea, sesame and sorghum; seed quality and certification for input suppliers), the International 40 SNAP Final Evaluation Report March 2017 International Solutions Group Institute for Tropical Agriculture (cassava processing), Pa Jah Poultry Farms (maize processing), and Vicsons Business Development Solutions (cluster capacity building) The program also trained fifteen FFS Clusters to become suppliers of seed to other FFS Clusters as well as other farmers. The SNAP Program also collaborated with CGIAR/IITA on value added processing and product development for cassava. Table 11 provides information on the IPTT indicators under IR 2.1 related to business development. Table 11. IPTT Indicators Under IR 2.1 Related to Cluster Business Relationships Indicator (source of indicator – FFP/USAID or A/V) LOA Target LOA Actual/ Endline (95% CI) Percent of Target Achieved (Baseline = 0) Overall Benef/Non￾benefic Number of agri-businesses supported (OP23.1) (A/V) 27 18 Not applicable 67% Number of input suppliers with increased capacities (OP24.1) (A/V) 15 15 Not applicable 100% Major Observations. While all FFS Clusters were provided with agribusiness and record keeping training leading to the development of annual business plans, the Evaluation found that the FFS clusters were not really using the plans nor developing subsequent plans. Almost all FFS Clusters are, nevertheless, selling produce locally or to established buyers, but, again, not as part of a business plan. The SNAP Program decided not to work with mid-level traders as planned for a couple of reasons. The program found it difficult to cultivate links between them and farmer groups, and there was an impression that the prices offered by these traders to farmers were unfair and non￾negotiable. The P4P relationship worked very well in Koinadugu with the Sabatie ABC and the Koinadugu Women's Vegetable Farmers’ Cooperative to which the program facilitated business relations for selected FFS Clusters. The relationships did not develop in Kailahun because the FFS Clusters did not want to wait for payment for their rice until after the ABC had been paid by WFP. In general, FFS Clusters interviewed were risk-averse and preferred no-risk options, e.g. selling locally at a lower price rather than holding onto their product and seeking out better prices. The final evaluation found little evidence that those (15) FFS Clusters that had been trained to become input suppliers were planning to continue to produce and sell seed. All of those interviewed (the Evaluation Team interviewed two Clusters that had been trained to provide inputs, but also obtained information from at least one other organization, Sanda Integrated, that had been buying seed inputs from clusters, which supported the analysis) were reverting back to crop production, since they felt it was less risky. They were certain to be able to sell production, but less certain that they would be able to sell seed. Lessons Learned - Monitoring the Strength of Business Relationships Although functional business relationships are key to the sustainability of FFS Clusters, the SNAP Program's M&E system did not have any indicators to monitor these relationships other than sales data. While knowing that FFS Clusters are making sales and monitoring trends in sales volume is useful, additional indicators for identifying to whom the sales are made and whether there are repeat sales to the same buyers/processors is also helpful for monitoring the strength of business relations. 41 SNAP Final Evaluation Report March 2017 International Solutions Group Immediate LOA Impact. An immediate impact of the cultivation of business relations was the income generated from sales, and most FFS Clusters were finding ways to sell their product either locally or in nearby towns. FFS Clusters were using this income in various ways, including dividing it up among members, capitalizing self-managed loan funds, and in a few cases, making larger investments in rice mills or other processing equipment. Successful sales also encourage continuation of groups. Long-term Sustained Impact. The sustainability of the impact achieved by SNAP depends on whether existing sales relationships could be maintained and new relationships developed after project end. If a cluster sold twice to the same buyer, then the relationship can be considered robust, and the strongest such relationships were with those FFS Clusters selling production in the P4P Program. FFS Clusters that had not developed these relationships expressed less enthusiasm during interviews and are deemed to be more at risk of dissolving unless the relationships can be developed. The Evaluation notes that the EVD outbreak occurred at a critical time just after formation of the first FFS Clusters. It was at this point that the program intended to facilitate functioning business relations between FFS Clusters and buyers, but lost the opportunity to do so because of the State of Emergency. With the program concluded and no more facilitation by SNAP staff, sustained impact will depend on FFS Clusters finding their markets. The Evaluation found that FFS Clusters were able to find at least local buyers for their production, but there was little evidence that they were able to find the established buyers/processors like those listed above without program assistance. Other programs were engaging a few FFS Clusters, however, such as the Feed the Future Integrated Rice Fish Pilot Project in Tonkolili District. 42 SNAP Final Evaluation Report March 2017 International Solutions Group D. Outputs Produced Under IR 2.2: Increased access to financial and marketing resources Three major outputs were produced under IR 2.2 in the SNAP program. These were the creation of Village Savings and Loan Associations (VSLAs) to mobilize capital, capacity building of community-based resource persons, called Private Service Providers (PSPs), to provide technical support for the start-up and operations of VSLAs, and training of women and youth on functional literacy/numeracy and vocational skills. 1. Village Savings & Loan Associations (VSLA). In the original proposal, ACDI/VOCA expected the VSLA component in SNAP to begin implementation in the first year following the selection of a local partner to implement the component. An organizational capacity assessment of the first partner identified for this role resulted in the decision not to engage them. After considering other alternatives, ACDI/VOCA decided to take responsibility for the VSLA component. Even after this decision, staff recruitment constraints resulted in further delays in implementation. The first VSLAs were not formed until late in FY 2012. The design of the VSLA approach in SNAP was based on a review of best practices in the industry. The VSLA is a member-managed group, holding weekly meetings to purchase shares, make social fund payments, and make loan payments. Loans from the savings fund are granted to members, and social fund grants and interest-free loans are granted to members for emergencies. A nine to twelve-month savings cycle was conducted, at the end of which shares and returns from interest income on loans are/were shared out to members. SNAP provided training and a metal cash box, calculator, and ledger for start-up. Participants are expected to use locally available copy books for passbooks. All VSLAs were also trained in basic business and financial management, business planning, record keeping, marketing and credit policies. Achievements. Table 12 shows the achievement of the program against the indicators for this component in the IPTT. The SNAP Program significantly exceeded its target for establishing VSLAs, which includes the original 170 VSLAs targeted, plus 89 VSLAs established in the last year by the PSPs. In addition, the Evaluation found a significant number of other VSLAs that had been established outside of these numbers. A few PSPs reported, for example, that they had established a third VSLA; and, most important, the Evaluation found spontaneous, or spin-off, VSLAs founded by current VSLA members. Based on information obtained from both interviews and the SNAP M&E unit, the Evaluation Team estimated that there were at least 55 additional spontaneous VSLAs functioning at the time of research (from 145 formed, reported by ACDI/VOCA). Table 12. IPTT Indicators for the VSLA Component Indicator (source of indicator – FFP/USAID or A/V) LOA Target LOA Actual/ Endline (95% CI) % of Target Achieved (Baseline = 0) Overall Benef/Non￾benefic Percent of VSLA members meeting their savings commitment (OC23.1) (A/V) 95% 96% Not applicable 101% Number of VSLAs assisted (OP25.1) (A/V) 170 259 Not applicable 152% Number of VSLAs members trained (OP26.1) (A/V) 6,735 6,739 Not applicable 100% Number of women in leadership positions in VSLAs (OP27.1) (A/V) 234 807 Not applicable 345% Major Observations. The Evaluation found that the average VSLA size to be consistent at 30 members with predominantly women (around 80%). While the total membership in the 170 VSLAs directly supported by the program was approximately 5,000, many VSLAs had 43 SNAP Final Evaluation Report March 2017 International Solutions Group multiple members from the same household, so the total number of households benefitting from these VSLAs was likely 3,500 to 4,000. Member retention was close to 100% between cycles, with most replacements coming when a member passes away or moves to another village, and approximately 60% of VSLA members were from households participating in SO1 and SO2 FFS Cluster activities. All VSLAs had completed at least two savings cycles and many had completed three or more cycles. The Evaluation found that almost all members received a loan within a cycle, a very positive finding, even if loan sizes were small. The uses of loans, social funds and share-outs was found to be consistent with the intended uses, i.e., productive activities at least in part for loans, addressing household problems with social funds, and making major investments in household livelihoods activities or assets using the share-out. The Evaluation also noted evidence that current VSLA members were involved in multiple VSLAs, also a good thing as long members would therefore have cash with which to purchase shares. A key metric for assessing VSLA financial performance was the average value of share-outs per VSLA. This represents the culmination of savings behavior, lending practices and repayment over a cycle. By FY 2014, VSLAs had started sharing out, and the average value of the share-out per VSLA was $1,023. By FY 2015, this amount had increased to $1,298 per VSLA; and, by FY 2016, it had increased to $1,764 per VSLA, a very positive trend indicating a successful VSLA program. In terms of ownership of the VSLA, the Evaluation found enough variation in individual VSLA policies and procedures, while keeping core business consistent, to demonstrate that VSLAs were making their own decisions, a strong indicator of ownership in the "SNAP box" (as it was called locally). Some members did express concern about accurate calculation of the share-out after SNAP conclusion. Immediate LOA Impact. Participants in qualitative interviews were emphatic that they were realizing immediate positive impact from the “SNAP Box”. They reported using loans and share-outs for school fees, medical costs and a range of livelihoods investments related mainly to farming and petty trade, and, as mentioned, nearly all members received a loan in a cycle. The social fund (“problem money”) was used for medical emergencies and school fees. In addition to these benefits, the availability of cash through the VSLA reduced tension within the household as well as between households. It was no longer necessary to approach family and neighbors as often for financial assistance. A significant benefit, often cited, was that households with emergencies no longer had to approach a moneylender. The availability of the “SNAP box” also encouraged people to save more than they normally would, and the pressure of attending a weekly meeting motivated members to find at least enough money to purchase one share when they attended the meeting. VSLA members appreciated the opportunity to put their savings in a secure place. The first share out of a VSLA was noted by members interviewed by the Evaluation Team as an exciting time and afforded a chance to really see the benefit of the VSLA. It was reported in two instances that a member exclaimed that “I have never in my life held this much money at one time!” upon receiving her share-out. Some groups noted anecdotally that the VSLA was building social capital within villages, e.g. men and women working together, interacting socially regularly, and cultivating interest in what was happening to others, although this was not mentioned consistently. 44 SNAP Final Evaluation Report March 2017 International Solutions Group Long-Term Sustained Impact. The main threats to the sustainability of a VSLA are the risk of (1) VSLA leadership coopting the VSLA for other purposes, (2) poor VSLA funds management including calculating share-outs, (3) VSLAs taking on too much debt before management capacity has become capable, and (4) theft or robbery of funds. The first threat is minimal. The management structures in place were well-tested and qualitative data indicate that the program performed well in following best practice. The Evaluation did observe, however, two secretaries taking advantage of their position to obtain exceptional benefits from the VSLA. The secretary position can be very powerful if the person in the position is the only one who can accurately keep records and calculate the share-out. In the two cases interviewed, the secretaries were obtaining interest-free social fund loans for a productive activity, in one case, and personal school fees, in the other case. Calculating share-outs was the greatest funds management challenge, but for most VSLAs, the secretary was able to do the calculations. In the few cases, in which the secretary was not able to calculate the share-out, the solutions presented by the group were to (a) ask the SNAP VSLA officer to do the calculations (this VSLA will likely fail at some point), (b) ask the secretary of another VSLA to do the calculations, (3) ask a local teacher to do the calculations (qualitative research indicates that many VSLA secretaries were indeed teachers), or (4) look for a PSP to assist them. For the third threat, some of the VSLAs had already opened bank accounts at the urging of the project staff but had not yet tried to access loans. When asked why, the most common response was that "we are uncomfortable having to manage a loan from a bank"; and to its credit, SNAP did not force VSLAs to take external loans. In a few exceptional cases, VSLAs themselves had combined their resources with other groups to make a major investment. The most notable example was in Koromasilaia Village (a Mandingo village) in Folosaba Dembelia Chiefdom in Koinadugu in which assets from six groups, including two SNAP VSLAs, were invested by the SNAP FFS Cluster in a small truck being used effectively for year-round transport of vegetable production and other goods between Koinadugu to Freetown. The fourth threat was a legitimate and growing risk as it became more and more important how much money accrues in the box shortly before share-out. VSLAs required further coaching on security precautions. The Evaluation found several instances of the VSLA box being with insufficient precautions against theft. Given that most VSLAs had, at the time of research, completed three share-outs, and the practice, using a good model, was well-understood by all members queried by the Evaluation Team, many VSLAs had the tools and capacities for medium to long-term survival. For the PSP￾supported VSLAs, none of these had yet Lessons Learned - VSLA Phasing While the VSLA component started later than planned, this actually appeared to benefit the SNAP Program. Agricultural productivity activities under SO2, which started implementation earlier, increased household cash income. When VSLAs were formed later, households had some cash to purchase shares and make the VSLA process function. Then, VSLA loans and share-outs were used to further leverage farm investments through purchase of seed, labor and livestock. The lesson learned from the SNAP experience is that it is easier to establish VSLAs if the activity is sequenced after other interventions to increase income start to have impact. Aluminum Pot-Making Enterprise Established with a VSLA Loan 45 SNAP Final Evaluation Report March 2017 International Solutions Group to compete one cycle, so their sustainability was much more questionable, although many are likely to be sustained, since the model being used represents good practice. It is worth noting that significant impact has been achieved for many VSLA members and their families. These include households that established new and significant income-generating activities as a result of VSLA loans and/or share-outs and children who were enrolled in school because parents can pay school fees. 2. PSPs. In FY 2015, the program observed the emergence of spontaneous VSLAs that had been formed as spin-offs by members of SNAP VSLAs. This was a very positive development, implying that participants appreciated the VSLA methodology to such an extent that they sought to replicate it themselves. It also raised concerns, however, with the technical quality of the new VSLAs. The program identified the PSP model developed by Catholic Relief Services as a potential solution to ensuring that these new VSLAs had sufficient technical support and obtained approval from USAD/FFP to implement the model in the last year of the program. The PSP model was oriented around training community-based agents to provide services to start-up and operating VSLAs for which the VSLAs provide some sort of compensation, either cash or in-kind. SNAP selected 45 current VSLA secretaries and trained them to provide VSLA services as a PSP. Each PSP was provided with two VSLA start-up kits (safe box, calculator, and ledger) to give to new VSLAs on the condition that they be replaced at some point by the VSLA, especially after the first share-out. SNAP staff also facilitated dialogue between PSPs and new groups to try to reach agreement on compensation for the PSP, including provision of a small monthly transport allowance and a commitment to replace the start-up kit. Achievements. At the time of the Evaluation, 45 PSPs had been trained by the program, and at least 89 new VSLAs had been established, although none had yet completed a savings cycle. Major Observations. In focus group discussions, the Evaluation found that participants were excited about being able to have a "box", but they were concerned about the loss of the technical support provided by SNAP and were also unhappy with the idea that they were being asked to pay for services that had been provided to other VSLAs free of charge. The fact that the groups had not yet completed a savings cycle and realized the benefits of the VSLA had not yet offset these concerns. The Evaluation found variation in the size of PSP-formed VSLAs. While the SNAP VSLAs were consistently sized at approximately 30 members, two PSPs reported having VSLAs as large as 45 members. When queried on why so large, the response was typically that that was how many people were interested, and there didn't seem to be recognition that the amount of work for the secretary had increased and the potential intra￾group tensions or risk of weak cohesion was greater. Most PSPs are capable of providing services envisioned for which they would be compensated. A small number were noted by the Evaluation Team, however, to be unable to calculate a share￾out. The Evaluation found little consistency in actual or promised compensation for PSPs by the VSLAs. Again, not having completed a cycle makes it difficult to assess how much Lessons Learned - PSPs While not really applicable in the case of SNAP since the PSP concept was only introduced late in the program, future programs should seek ways to begin implementation of VSLA activities that will support the establishment of PSPs later. For example, the free distribution of VSLA start-up kits initially caused problems for SNAP PSPs who were providing start-up kits to be replaced by VSLAs. Future programs may want to put practices in place at the beginning in which VSLAs are expected to replace start-up kits at some point in the future. 46 SNAP Final Evaluation Report March 2017 International Solutions Group compensation will be realized. The Evaluation did find, however that a few VSLAs had made partial payments of transport allowances; and, while no real commitments were made to replace start-up kits, some VSLAs said that “we will give him something when we share-out”. The expectation was high that, because SNAP VSLAs got free start-up kits, new PSP VSLAs should also get a free start-up kit. The Evaluation Team noted an example of a PSP who was paid SLL100,000 (approximately USD$18) to calculate the share-out for an existing SNAP VSLA, exactly the kind of arrangement that will support a PSP. Immediate LOA Impact. The PSP activity began late in the life of SNAP and none of the VSLAs established by PSPs had completed a savings cycle by the time of research, so the immediate impact for VSLA participants was minimal. Some PSPs benefitted from limited transport allowance payments and a few PSPs had begun offering services more widely, especially for share-out calculations. Long-Term Sustained Impact. A small number of PSPs, i.e., those most enthusiastic about the VSLA concept and with good technical skills, were likely to become firmly established as PSPs receiving adequate compensation for their services. Most of the PSPs, however, will struggle with trying to find adequate compensation, and a few, i.e., those providing poor quality services, will fail in the near future25 . 3. Literacy/Numeracy and Vocational Skills Training. Literacy/numeracy and vocational skills training began implementation in FY 2011 using existing training facilities in Kabala, Makeni and Kailahun towns. The original SNAP proposal suggested that vocational training centers would be established or built by the program, but there was no need to establish these facilities since facilities were readily available in major towns. Initially, separate participants were identified for literacy/numeracy training and vocational training. Later, these were sequenced so that functional literacy/numeracy was provided first, e.g. to learn numbers, and then vocational skills training was provided, e.g. to be able to use a tape measure. Also in the first year, participants were not drawn from SO1 households. In FY 2013, the literacy/numeracy trainings were restructured, moving the training sites from district towns further into targeted communities to facilitate integration with SO1 and SO2. Thereafter, training participants were selected mainly from households participating in other components of the program. The training period for the literacy/numeracy training ranged from three to six months over the life of the program. Vocational skills training was usually three months, five days a week, averaging four to six hours a day. Trainers were recruited locally for literacy/numeracy and for some vocational skills. For more technically demanding vocational skills such as gara tie￾dyeing, trainers were recruited from district towns. After graduation, vocational training participants living in proximity to one another were organized into groups of five and provided with business start-up kits including equipment and materials relevant to their new skills. Achievements. Table 13 shows achievements of SNAP relative to the performance indicators defined in the program's IPTT. For both types of training, the program fell well-short of the targeted number of participants. Key informants submitted several reasons for the low performance: Delays occurred at the start-up of activities because beneficiaries expected to be 25 For example, one PSP interviewed (a town chief) who did not know how to calculate a share-out is unlikely to continue with services for long. 47 SNAP Final Evaluation Report March 2017 International Solutions Group paid, or at least fed, while they were participating in training. When the strategy subsequently shifted to sequencing the training, participants had to complete literacy training before commencing vocational skills training, meaning that a single participant had to be available for up to nine months of training. In 2014, it was also agreed with USAID to stop vocational skills trainings and focus on numeracy/literacy, but subsequent to the outbreak of EVD and resumption of classes, teachers were more focused on enabling regular students to catch up from the lost year, further delaying the resumption of adult literacy/numeracy classes. Table 13. IPTT Indicators for the Literacy/Numeracy & Vocational Skills Training Component Indicator (source of indicator – FFP/USAID or A/V) LOA Target LOA Actual/ Endline (95% CI) % of Target Achieved (Baseline = 0) Overall Benef/Non￾benefic Number of youth receiving vocational training (OP28) (A/V) 4300 2,122 Not applicable 49% Number of individuals trained in literacy and numeracy (OP29) (A/V) 6000 3,197 Not applicable 53% From 2011 through 2015, a total of 4,467 participants were enrolled in literacy/numeracy trainings, with approximately two-thirds of these being women. Not all enrollees were able to complete the training, however, and the total number of participants completing the training was, as indicated, 3,197. From 2011 through 2014, SNAP conducted 43 vocational skills training events. Table 14 shows the number of participants in each of the different types of vocational skills training that was provided. Major Observations. In only three to six months of literacy training, it is reasonable to expect participants to at least be able to read and write out the alphabet, spell and write their names, develop a signature, and read and write out numbers from 1 to 100. The Evaluation found numerous participants able to write their names, alphabet and numbers, but also found several participants, including many older women, who were not yet able to do so. They responded that they could not regularly attend sessions, and they also did not get enough attention from the teacher. When the training shifted from the district towns to chiefdom headquarters, the pool of teachers available was reduced, making it more difficult to find teachers with experience in adult literacy. Table 14. Vocational Skills Training Participants Type of Training LOA Target LOA Actual Men Women Total Number of Trainings Soap Making --- 99 592 691 12 Tailoring --- 93 247 340 8 Gara/Tie Dye --- 11 317 328 7 Masonry --- 258 148 406 6 Carpentry --- 197 0 197 5 Hairdressing --- 1 124 125 3 Auto Mechanics --- 56 2 58 1 Home Science --- 0 50 50 1 All Vocational Skills 4,300 715 1,480 2,195 43 48 SNAP Final Evaluation Report March 2017 International Solutions Group SNAP targeted youth and women for literacy/numeracy trainings, and many women participated. The broad definition of youth was used, i.e., up to age 35, and most participants, judging from qualitative interviews, were between the ages of 30 and 35. Training classes in some areas were also reported to be very large, presumably due to the difficulty in finding qualified teachers. The large groups included participants from all vocational training groups, making it difficult to tailor the training to functional literacy/numeracy related specifically to the vocational skills for which participants were to be trained. Due to time constraints, the Evaluation was unable to interview trainees from across the different types of vocational skills training that had been provided by the program. With those observed, the soap making and masonry training were most effective at enabling participants to acquire skills for viable income-generating activities. These both involve relatively low levels of inputs and technologies, and both soap and masonry are in constant demand at the village level. Soap is a constant consumption item for households, and people are building or renovating houses when they have money, so masons are in demand. The gara/tie dye training was challenging. The imported ink is difficult to obtain outside major towns, and the program also did not provide training on natural dyes as an alternative. In addition, there was extremely weak demand for the final product which competed with used clothing widely available in local weekly markets. The Evaluation found very little, usually relatively poor quality, gara/tie dye product held in stock by trained participants. The teaching of carpentry skills was useful, although the training in at least some areas was around building a table (a luxury item) rather than building roof frames (in greater demand). In some areas, the carpentry training also included construction of pallets used subsequently in SNAP food warehouses. Measuring, cutting and joining skills, however, were useful regardless of the product produced. In general, the market demand for products or services was not taken into account in deciding how many persons should be trained. In two locations, for example, fifty participants were trained in gara/tie dye for a market that might be able to provide sales opportunities for maybe ten producers. The group approach used by SNAP with small groups of five persons working together using a set of tools provided by the program was still functioning well for those groups that were producing products or selling services. All of the soap making groups interviewed, for example, were still meeting weekly (on average) to produce soap, some of them more than two years after receiving training. For carpenter and mason groups, the group leader was known to be the key contact person, and he spreads job opportunities to members and facilitates access to the set of tools. Lessons Learned - Literacy & Numeracy Training In moving the literacy/numeracy training from district towns to chiefdom headquarters, the SNAP Program encountered a trade-off. The targeting became better in terms of being able to enroll participants who were engaged in other SNAP activities and it also gave an opportunity for adult basic education to persons that have fewer such opportunities. The negative side, however, was the difficulty in finding trained teachers who were available for an extended period in a more remote area. The key lesson is to move the training to rural areas, but more extensive adult literacy pedagogical training may be necessary for the trainers who are available. 49 SNAP Final Evaluation Report March 2017 International Solutions Group Immediate LOA Impact. While the literacy and numeracy training was too short to have profound impact, numerous respondents reported that being able to write or sign their names has made them proud and increased their dignity. They were anxious to demonstrate these new abilities when asked to do so. Some even indicated that that they felt more confident about taking loans because they no longer had to sign with a thumbprint. For the major trainings provided, i.e., soap-making, tailoring, gara tie-dyeing, masonry and carpentry, the soap making vocational skills increased income for almost 700 households. The amount of income was small per household (approximately SLL50,000/USD$9 per person per week), however, it was continuous and less dependent on seasonal variability, although demand for soap likely goes down during the hungry season. The masonry and carpentry skills provided some income for most trainees; and in some cases, a member of the household has acquired a whole new occupation, especially some masons. The gara/tie dye training was unsuccessful except for a very small number of trainees with some entrepreneurial skills who will continue making product on demand. The household income impact, however, is very small. The tailoring skills are somewhat more useful for trainees, depending on demand which is also affected by the availability of used clothing. Long-Term Sustained Impact. Since the trainings were completed over two years ago, the evaluators consider that, barring a major disaster requiring the sale of assets such as tools, most of the impact being seen at the time of research among trainees will likely be sustained. Soap makers will continue to make soap and replace equipment as it wears out. Carpenters and masons will continue to use their skills, and improve them, and carpentry and masonry tools are readily available when they need to be replaced. Substantial impact was achieved for a few participants, mainly masons and carpenters, whose livelihoods were re-oriented based on the training received. 50 SNAP Final Evaluation Report March 2017 International Solutions Group E. Outputs Produced Under IR 2.3: Improved Community Assets Under IR 2.3, the SNAP Program originally proposed to jointly address the problem of youth unemployment and the need for construction or repair of community infrastructure, such as roads, bridges, hospitals, houses, clinics, shops and warehouses by implementing a "Labor for Learning" approach. In this approach, the program intended to provide literacy/numeracy and vocational skills training for three months on the condition that they provide ten days of labor to address the community infrastructure problem. The approach proved to be infeasible once implementation began. Discussions with youth indicated that they were unwilling to provide any free labor and expected either cash for work or food for work to participate in community infrastructure construction or rehabilitation. The approach was abandoned and community infrastructure work was postponed because monetization proceeds were not providing sufficient funding. In FY2014, the community assets component was revived, with a plan to pilot drying floors and grain stores as a package in a few locations then to roll it out to other FFS Clusters. This plan was changed in 2015 after negotiations with WFP to provide Food For Work (FFW) failed, and the plans were again revised to complete the four pilot stores/drying floors to which SNAP had already made commitments with FFS Clusters and communities26 and to construct 285 drying floors. Stores and drying floors were expected to be managed by FFS Clusters, and the locations identified for drying floors were based on assessments of the performance of FFS Clusters. Achievements. At the time of the Evaluation, 110 drying floors had been completed and 60 drying floors were still under construction. Four community-based grain stores were also under construction, but near completion, and brushing and minor maintenance had been completed on 64 kilometers of feeder roads, 16 kilometers total in each district. Table 15 provides information on the IPTT indicators for IR 2.3. Table 15. IPTT Indicators for IR 2.3: Community Assets Indicator (source of indicator – FFP/USAID or A/V) LOA Target LOA Actual (95% CI) % of Target Achieved (Baseline = 0) Overall Benef/Non￾benefic Percent of farmers using improved storage (OC24) (A/V) 45% 19% 31.8%/13.2% 42% Number of community-assets projects completed (OP31.1) (A/V) 782 106 Not applicable 14% Major Observations. As the table indicates, only 42% of the program target for use of improved storage was achieved, overall. However, this result may be skewed by the inclusion of non￾program participants in the Endline Survey. A disaggregation of respondents, while of limited statistical validity, indicates that program participant use of improved storage, at 31.8%, is considerably higher than non-participants (13.2%), suggesting a potential program target achievement of 71%. The design of the drying floors was highly appreciated27 because of the elevation above ground-level minimizing exposure to contamination. The design, however, was half the size of the designs being promoted by MAFFS and the National Commission for Social Action 26 The locations are Yainkassa Village, Sande Loko Chiefdom, Bombali District; Ropothka Village, Yoni Chiefdom, Tonkolili District; Musaia Town, Folosaba Dembeya Chiefdom, Koinadugu District; and Gbangiyema Town, Yawei Chiefdom, Kailahun District 27 The design was reported to the Evaluation Team to have been developed by a Peace Corps Volunteer 51 SNAP Final Evaluation Report March 2017 International Solutions Group (NaCSA). Nonetheless, MAFFS staff expressed appreciation for the design and said that they would advocate for use of the design by other initiatives. What was missing, in their view (a view shared by the Evaluation Team), was a mechanism for generating revenue from the drying floor, i.e., user fees in-kind or cash, to be able to maintain the floor and generate funds to build additional floors. The NaCSA design has this feature, and although it was not part of the SNAP design, the Evaluation found that a few FFS Clusters were discussing the idea of user fees. A few completed drying floors were, at the time of research, already showing some structural defects, e.g. cracks appearing and the packed soil beneath the floor receding, leaving hollows. Since the drying floor construction occurred at the end of the rainy season, it is very difficult to eliminate soil moisture and fully pack the dirt under the floor. The program also had only one construction engineer per district to monitor construction and ensure quality. The construction of drying floors was taking place as quickly as possible to complete the work before the program ended in December 2016. Drying floor construction committees were formed to facilitate meeting community commitments for providing local materials, including sand, gravel, bush sticks and boulders. FFS Clusters were expected to be responsible for managing the drying floors once constructed, but very little capacity building for planning, operations, maintenance and repair had been done with them by the program. Given the time shortage, the focus of the program was primarily on the constructional elements. The Evaluation noted a relatively high proportion of unused drying floors. The research was conducted during the off-season for main crops, although some rice, pepper and ground nuts were observed on mats or old drying floors (sometimes near a newly constructed but empty SNAP drying floor). In focus group discussions, the main reason given for the drying floor not being used was that "it has not been officially turned over to us". From the perspective of communities, the drying floors belong to SNAP, which is not surprising since almost all construction costs including purchased materials, tools and payments for labor were paid by the program and little capacity building was done with FFS Clusters for managing the asset. ACDI/VOCA reported that it had undertaken ‘dedication’ ceremonies to formally hand over responsibilities and ownership for drying floors to communities by end 2016. The road maintenance supported by the program included cash-for-work and tools to clear brush from roadsides, fill in potholes, and clear side drains and culverts. This is a recurring requirement for Sierra Leone roads after each rainy season. The SNAP intervention was a one￾off to facilitate a linkage between District Councils and WFP FFW, but the FFW never materialized. Immediate LOA Impact. Since the work on community assets had been completed shortly prior to the evaluation, there had been very little immediate impact from the drying floors and storage facilities. The work on feeder roads had some limited impact, however, on creating better access to markets and basic services, although the amount of road rehabilitated was relatively small. Long-Term Sustained Impact. When completed, and assuming they are used, the drying floors may indeed improve food and product quality via reduced contamination. The improved food quality may thus reduce incidence of foodborne illness. The improved quality could also, in principle, result in a somewhat higher price in the market, although this is unlikely since the product will not be visibly different from other competing products in the market. 52 SNAP Final Evaluation Report March 2017 International Solutions Group Completed and properly-utilized storage facilities will contribute to reductions in post-harvest storage losses and FFS Clusters will have the opportunity to store product while they seek good prices. This should result in a slight increase in sales income for the FFS Clusters. Since only four storage facilities were constructed, however, the number of people benefitting was relatively small, probably less than 500. Long-term impact of the road maintenance work done by the program is very doubtful since such maintenance is a recurring requirement. While ACDI/VOCA reports a range of capacity￾building activities with District Councils for designing proposals for cash-for-work/food-for￾work that may meet this need, the Evaluation Team saw no such proposals being prepared by District Councils. 53 SNAP Final Evaluation Report March 2017 International Solutions Group V. EVALUATION FINDINGS: PROGRAM OUTCOMES AND IMPACT ASSESSMENT This section of the Evaluation Report analyzes changes noted among the program beneficiary population by the quantitative and qualitative research, notably the key outcome/impact related performance indicators reported on by SNAP and/or measured at program baseline and endline via the quantitative survey. These, and other, changes are further analyzed in the context of qualitative data that was gathered directly by the Evaluation Team to triangulate evidence that such changes can be (or not) attributed to the program outputs, i.e. that they should or should not be considered likely outcomes and impacts of SNAP, as well as any evidence that these changes will be sustained beyond program completion. This analysis also covers evidence of the extent to which key cross-cutting themes were prioritized by SNAP implementers within the program. A. Impact under SO 1: Reduced chronic malnutrition among children under five Endline estimates on the prevalence of underweight, stunting and wasting in Children Under 5 (CU5) as specified by the three impact indicators in the IPTT for SO1 are provided in Table 16. Table 16. Performance Indicators for SO1 Indicator Baseline LOA Target Endline (95% CI) % of Target Achieved (end-base)/(target-base) Overall Benef/Non￾benefic Prevalence of underweight children under five years of age (IM2) (FFP) 20.3% 14.3% 14.8% (13.1%-16.7%) 12.9%/15.6% 92% Prevalence of stunted children under 5 years of age (IM3) (FFP) 38.4% 31.4% 32.3% (30.0%-34.7%) 30%/33.3% 87% percent of acutely malnourished children 6-59 months (OC1) (FFP) 9.6% 7.2% 6.5% (5.2%-8.0%) 6.5%.6.4% 129% Both qualitative and quantitative evidence indicate improvements in the nutritional status of CU5 and decrease in acute malnutrition. Evidence suggests this was due to hygiene practices, especially handwashing, MUAC self-assessment, and increased access to health facilities. For Children Under 2 (CU2), there likely28 was impact on reducing the potential for chronic malnutrition in approximately 32% of the mother-child units who participated in the program and received the complete package of interventions (food rations, BCC messages, access to health services at PHU and proper WASH) for the full 30 months as for the PM2 approach. For the rest of the mother-child units that received part of the full package of intervention and did not receive food rations due to the EVD epidemic, the impact was likely to have been less. SO1 potential impact was also diluted by (a) Family habits of sharing food, including the rations specifically for PLW and CU2, (b) nutrient rich food diversification and utilization messages not being readily adopted, and (c) the EVD outbreak that disrupted reinforcement of BCC messages. Knowledge and practice on exclusive breastfeeding and handwashing messages for most participants likely will continue to be practiced after program end. PHU access may also continue in the communities where the engagement of HPs, LMs and LFs was high, and the distance from health facilities relatively moderate. The long-term impact of expanded 28 On the basis of current understanding of the PM2A approach, which is currently the subject of FANTA-implemented clustered, randomized trials in two FFP programs in Burundi and Guatemala. See http://www.fantaproject.org/research/impact-cost-effectiveness-pm2a 54 SNAP Final Evaluation Report March 2017 International Solutions Group knowledge on nutritional diversity during pregnancy and complementary feeding after six months is more difficult to predict as the Evaluation Team noted incomplete knowledge acquired by participants. The Final IPTT (see Annex E) reports that the SNAP Program had impact on 45,371 beneficiaries under SO1. While this is slightly less than the target of 50,064 for the LOA, the evidence (including the overall better performance of program participants vs. non-participants suggests that the program achieved impact on behavioral change that can potentially be sustained after the program ends. LESSONS LEARNED Impact on Preventing Child Malnutrition Indirect benefits in chronic malnutrition prevention should be attributed to children that remained in the program for less the full 30 month PM2A period because these mothers received at least the BCC messages. 55 SNAP Final Evaluation Report March 2017 International Solutions Group B. Impact under SO 2: Enhanced livelihoods for vulnerable people, especially women and youth The qualitative information gathering and analysis found that productive activities with FFS Clusters had impact on increasing income and food availability. In addition, the VSLA activity increased income from non-farm activities as well as further leveraged the gains made in agricultural production through the use of loans and share-outs for farm production activities. The vocational skills trainings resulted in small increases in income for those trained in some skills, mainly carpentry, masonry and soap-making. The impact of the community assets development in the SNAP Program, however, was limited at the time of the Evaluation research, as they had not (yet) been used, little community capacity-building on their use had taken place, and the numbers of such assets were small compared to the underlying need. The drying floors were the asset most likely to generate value for communities, should the issues around ownership and utilization (discussed in Section E above) be addressed. Table 17 compares the endline and baseline estimates on key indicators related to household diet diversity and food provisioning ability, showing negative impact on the first and positive impact on the second. Table 17. Performance Indicators for Diet Diversity & Food Provisioning Ability Indicator (source of indicator – FFP/USAID or A/V) Baseline LOA Target Endline (95% CI) % of Target Achieved (end-base)/(target-base) Overall Benef/Non￾benefic Average Household Dietary Diversity Score (HDDS) (IM4) (F/SL-USAID) 6.74 8.5 5.44 (5.34-5.55) 5.61/5.4 Negative Number of Months of Adequate Household Food Provisioning (MAP) (IM6) (F/SL￾USAID) 7.74 9.5 9.67 (9.60-9.74) 9.54/9.69 110% As noted in the Evaluation Limitations above, the baseline data was collected in the middle of the dry season (March) when food is more readily available and households still have income from crop sales, while the endline data was collected in the middle of the lean season (August) when both food and cash are more scarce, so it is not surprising to see the endline estimate lower than the baseline estimate. Most of the increased production observed is for crops that were already being produced, so in terms of food production, little increase in diversity was observed from the data. As reported in the previous section, SNAP diet diversity promotion activities were also relatively ineffective, so households were not much motivated to seek out and obtain more diversity in their diets. SNAP did promote new, nutrient-rich crop production by obliging FFS Clusters to choose two crops for receiving seed from the program, one a staple and the other a nutritious crop. All FFS Clusters interviewed in qualitative discussions indicated that they were producing and/or planning to produce crops that are marketable. There was not much demand yet for the nutritious crops so their attention was on the staple crops that can be readily sold. The Evaluation concludes that the program had little or no effect on household diet diversity. The Evaluation also believes, however, that SNAP had positive impact on food provisioning capacity as supported by the quantitative estimates. More food was available from agricultural production, and higher incomes from farm production and non-farm income generation were being used to obtain food at critical times. Table 18 compares the endline and baseline estimates for revenue indicators from the program's IPTT. The Evaluation Team notes that targets inconsistent with the baseline appear several times in the IPTT as the FFP regional M&E specialist worked with the program in 2012 (post 56 SNAP Final Evaluation Report March 2017 International Solutions Group baseline) to revise targets in the IPTT. This is discussed further below. In addition, several factors preclude valid comparisons between the two estimates on each indicator, as follows: Table 18. Performance Indicators for Household Revenue (Amounts in SLL) Indicator (source of indicator – FFP/USAID or A/V) Baseline LOA Target Endline (95% CI) % of Target Achieved (end-base)/(target-base) Overall Benef/Non￾benefic Average Rural Household Revenue (OC14) (A/V) 3,128,690 1,035,913 1,768,051 (1,569,315- 1,966,786) 1,847,413/ 1,732,156 Can’t be calculated (baseline > target) Average Household Revenue from Crops (OC 14a) (A/V) 2,391,025 --- 899,228 (811,726- 986,730) 1,138,309/ 768,118 No target Average Household Revenue from Livestock (OC 14b) (A/V) 71,229 --- 491,224 (285,101-697- 347) 634,407/ 390,667 No Target Average Household Revenue from Non-Farm Activities (OC 14c) (A/V) 639,933 --- 1,519,977 (1,264,698- 1,795,256) 1,027,396/ 1,745,128 No Target Note: Exchange rate at the time of the baseline was SLL 3855/US$ and at the endline SLL 6000/US$ First, the average household size at the baseline was 10.8 and at the endline the household size had declined to 6.4. At the time of the baseline, it was not unusual to find large households. It is more interesting that household size fell significantly by the time of the endline. Unfortunately, the qualitative evaluators did not notice the reported difference in household size until the final endline data was received, after the field work was completed, so could not obtain information to explain why household size declined. Household income is proportional to household labor capacity, so comparing income for an average household of 10.8 persons to income for an average household of 6.4 persons is not a valid comparison. Second, the estimates for household revenue were calculated differently for the baseline and endline. The baseline household revenue estimate includes a crop component that was based on total crop production valued at an average price for each crop. This represents the total value of crop production, not crop sales29 . However, the crop component of the endline estimate was based on average crop sales, a different method which, ceteris paribus, will likely produce different results. The baseline estimate of crop sales was SLL 437,124. Using this metric rather than the value of crop production, the total household revenue estimate becomes SLL 1,148,286 (excluding fish sales - not included in the endline estimate). If data is further weighted per average household size, i.e. reducing the endline revenue estimate down to a household of 6.4, then the average household revenue estimate becomes SLL 680,466, which makes a more valid comparison to the overall rural household revenue endline estimate of SLL 1,768,051. The evaluators also note that comparisons between program participants and non-participants, albeit not statistically rigorous, indicate better financial performance across all indicators other than revenue from non-farm activities (OC 14c). 29 See page 37 of the SNAP Baseline Report, Volume 1 57 SNAP Final Evaluation Report March 2017 International Solutions Group Table 18a: Household crop revenue and overall revenue weightings and comparisons Average HH size HH crop sales revenue estimate Weighted sales revenue (Adjusting for different HH size) PPP adjusted overall HH revenue in SLL/USD Baseline 10.8 437,124 437,124 (no adjustment) SLL 680,466/USD $177 Endline 6.4 1,148,286 680,466 SLL 1,738,051/ USD $295 However, for better comparison, the estimates should be further adjusted for purchasing power parity. Using the extant exchange rates, the adjusted baseline estimate is comparable to US$ 177 (@3855) and the endline estimate comparable to US$ 295 (@6000). This is a valid comparison with the quantitative data available, and it is consistent with the findings of the qualitative investigations which indicated that incomes had increased as a result of the SNAP Program. Further, the total household revenue estimate for the baseline was calculated from the sum of estimates for each of the different components. This is acceptable since the estimates of sales for each component are the average for all households, not just those households reporting sales. In the endline, a specific question was asked on total sales, and the estimates for each component are based on the number of households reporting sales, so the total household revenue is not the sum of the different components. This does not have implications for total household revenue, however, it does have implications for comparing estimates for a specific cluster of sales. Table 19 compares endline and baseline estimates for IPTT indicators related to non-farm income, showing average household revenue from non-farm income increasing, but the percentage of households declining. The comparison as shown on revenue is of limited validity since the baseline estimate is the average for all households in the sample while the endline estimate is the average only for households reporting non-farm income. If the former30 is adjusted to represent an average for households reporting non-farm income, then the comparison becomes SLL 1,009,358 to SLL 1,519,977. This estimate could be further adjusted by household size. In any case, however, it is clear that while there was no significant increase in the percentage of households with non-farm income, there was a significant income increase for those households with non-farm income. Table 19. Performance Indicators for Non-Farm Income Indicator (source of indicator – FFP/USAID or A/V) Baseline LOA Target Endline (95% CI) % of Target Achieved (end-base)/(target-base) Overall Benef/Non￾benefic Average Household Revenue from Non-Farm (OC 14c) (A/V) 639,933 --- 1,519,977 (1,264,698- 1,795,256) 1,027,396/ 1,745,128 No target Percent of Households with Non￾Farm Income (OC 15) (A/V) 63.4% 52% 53.7% (51.5% - 56.0%) 54.5%/54% Can’t be calculated (baseline > target) The results on these indicators are consistent with qualitative findings. The VSLA and vocational skills interventions produced most of the impact on non-farm income. VSLA respondents reported significant increases in income from petty trade and other activities financed from loans and share-outs. The VSLA and vocational skills participants, however, 30 SLL 639,933 times 1515 HH (N) = SLL 969,498,495, divided by 961 HH (63.4% of 1515 with non-farm income) = SLL 1,009,358. 58 SNAP Final Evaluation Report March 2017 International Solutions Group were relatively few, and their numbers were offset by the relatively larger number of respondents in the endline sample who were only engaged in FFS activities or from non￾participating households. Table 20 shows the number of direct beneficiaries for various interventions under SO2 to provide an idea on the scale of the impact achieved by the SNAP Program. The number of beneficiaries in the FFS Cluster intervention was on target, while the number of direct beneficiaries from VSLA activities significantly exceeded targets. All other achievements are below target. Table 20. Number of Direct Beneficiaries for Different Components of SO2 Type of Beneficiary LOA Target LOA Achievement % of Target Achieved (Baseline = 0) Members of FFS Clusters 31,000 28,650 (955 FFSs) 92% Members of SNAP VSLAs 4,250 (170 VSLAs) 5,100 (170 VSLAs) 120% Members of PSP VSLAs 2,700 (90 VSLAs) >2,700 (89 VSLAs) 100% Members of Spontaneous VSLAs - 4,350 (145 VSLAs) No Target Literacy & Numeracy Trainees 6,000 3,197 53% Vocational Skills Trainees 4,300 2,122 49% Input Suppliers 27 18 67% Nascent Agri-businesses 15 15 100% Total Beneficiaries 45,375 43,419 95.7% 59 SNAP Final Evaluation Report March 2017 International Solutions Group C. Collective Impact at the Development Objective-Level The SNAP Program had only one IPTT indicator at the Development Objective level, i.e. total number of rural households benefiting directly from USG interventions, and with a reported achievement of 55,211 households, the program exceeded its target (54,601). However, there is wide variation on the degree of impact achieved among these households. The greatest impact of the SNAP program has been on households that participated in all three of the major activities of the project, i.e. had a member in a mother care group, had a member in a FFS cluster, and had one or more members in a VSLA. The FFS Cluster membership increased income and food availability through increased agricultural production. VSLA members also increased income and further leveraged agricultural production as additional investments were made from loans and share-outs. The mother care group used the increased income and behavioral change to reduce child malnutrition. This occurred in around 25% of the villages, and in these villages it is estimated that around 3,100 households will have members who participated in the three major activities. For these households, the impact on food security and resilience has been substantial. The program was not designed initially to have a greater degree of integration across SOs than what was put in place. This is a good achievement, but a more effective program would have recognized this impact early on and worked with FFP to make adjustments to the strategy to achieve more integration31 . At the other extreme, SNAP achieved little impact on food security and resilience in remote villages and with only SO1 activities. In these villages, no increase in agricultural production was observed; nor any increase in access to capital via VSLAs; and no one was trained in new vocational skills. The only evidence of program outcome was from health and nutrition BCC training and better access to health services, via stronger PHUs relationships. Without increased income and food production, however, nutritional status and resilience changes were minimal. In these villages, life was little changed from when SNAP started. As these villages were harder to reach, they received less follow-up attention from health promoters and program staff. Analysis by the Evaluation Team of SNAP data32 , indicates approximately 20% of SNAP villages (11,000 households) in this category. However, villages within this category are significantly smaller on average than more accessible villages (a consistent pattern in West Africa). The Evaluation Team therefore estimates the number of households for which SNAP likely had little impact as closer to 5,000. Further, ACDI/VOCA noted a specific request by USAID after the mid-term evaluation and monitoring visit to continue registering SO1 beneficiaries that voluntarily joined in the absence of food incentives. Hence, potentially up to 13,000 SO1 beneficiaries were "non-food" beneficiaries joining after food distributions (and new VLSA/FFS activities) had ceased – the emphasis was on strengthening existing groups. Between these two extremes, three other types of households benefitted directly from SNAP. These were households in (a) accessible villages but in which only SO1 activities were implemented, (b) villages with only SO2 activities and no SO1 activities, and (c) villages in which both SOs were implemented but households did not have members participating in all three program activities (MCG, FFS Cluster and VSLA). The first represents a significant number of households, estimated at 40% of villages, larger villages on average, but of limited impact. Positive nutritional outcomes resulted from BCC training and better access to health services, but the Evaluation identified little evidence of impact on food availability/access. 31 The new Refine and Implement model used in current DFAP design and implementation would likely have facilitated this process. 32 Approximately 60% of villages had SO1 only, and 16% of villages are accessible only by foot and 18% are accessible only by motorcycle. 60 SNAP Final Evaluation Report March 2017 International Solutions Group For the second group, there is evidence of greater impact, but for a smaller number of households, roughly 7%. These households are likely to have increased availability and access to food via increased agricultural production and non-farm income, as well as increased resilience via improved farm production assets, financial assets and social capital. The Evaluation noted some changes in food utilization that interviewees attributed to health and nutrition messages passed through FFSs and VSLAs. Impact on the third group, approximately 15,100 households, was dependent on the types of activities in which household members were engaged. If a household member was part of a MCG, then there will have been impact on child malnutrition. If a household member participated in an FFS Cluster, then there will have been impact on food production and income; and if a household member participated in a VSLA then there would have been impact on farm production and income. In all cases, the degree of likely impact for households in this category is less than for households with members who participated in all three major activities. Table 21 summarizes the Evaluation's assessment on the degree of impact for different types of households benefitting directly from SNAP interventions33 . Table 21. Degree of Program Impact on Different Types of SNAP Households Type of Direct Beneficiary Households Estimated # of Households Degree of Impact Households with members participating in MCG, FFS Cluster and VSLA. 3,100 Increased farm production, increased income, and reduced child malnutrition. Households in accessible villages in which only SO1 activities implemented 28,000 Reduced child malnutrition, but otherwise minimal impact on food production or income Households in villages in which only SO2 activities were implemented 3,800 Increased food production and income but only marginal impact on child malnutrition Households in villages in which both SOs were implemented but members did not participate in all three major activities. 15,100 Depending on with which activities household members were engaged, increased farm production, increased income, and/or reduced child malnutrition. Households in inaccessible villages participating only in SO1 5,000 Limited impact on child malnutrition and no other impact. Total Households 55,000 Indirect Beneficiaries. The qualitative research found a wide range of coverage in participating villages in terms of the percentage of households who had participated in some way or another in SNAP Program activities. In smaller villages, almost all households had been reached by at least some SNAP activities. In larger villages, SNAP engaged with only a percentage of village households, but interviews with town chiefs in some of these villages indicated that almost all households in the village had been reached directly or indirectly by SNAP. Life-Changing Impact. Evaluations look for life-changing impact on program beneficiaries, i.e., households or individuals for whom the program has had such substantial impact that their lives have changed significantly. The Evaluation found four types of beneficiaries for whom such changes occurred. This includes (a) approximately 15,000 children completing the full PM2A cycle who may have become chronically malnourished without program intervention, (b) members of VSLAs who have used loans and share-outs to establish new or significantly expand income-generation activities, (c) members of those FFS Clusters who developed strong marketing relationships with established buyers/processors, and (d) a few of those masons and carpenters trained by the program who had new occupations. 33 Key Assumptions: For Intensity of Impact: SO1 only = 60%, SO2 only = 15% and Both SOs = 25% (page 11), For Accessibility: Accessible, by car = 66%, Inaccessible, by foot or motorcycle, = 34% (page 9), VSLA households = 4000 with 60% also participating in SO1 and SO2 FFS Clusters (page 40), and inaccessible villages are on average half the size of accessible villages. 61 SNAP Final Evaluation Report March 2017 International Solutions Group D. Cross-Cutting Theme: Resilience The SNAP Program specified resilience as a cross-cutting theme and identified four approaches to implement it. These were to: (1) Enable rural families to plan for major annual expenditures and the lean season so they have sufficient food and/or cash at key times, (2) Improve access to inputs, enhance skills, improve community infrastructure, increase crop and dietary diversity, and build a better understanding of nutrition, (3) Improve access to quality health services to create a healthier generation of children more able to withstand future shocks, and (4) Share project data with communities and partners as well as national bodies tracking vulnerability. Observations. The concept of resilience as a cross-cutting theme was described in the original proposal, but the program did not implement a plan to operationalize the concept. There was no training of staff, for example, on the concept of resilience. The original proposal included trigger indicators, which would be part of the anticipatory phase of resilience programming, but these indicators were not incorporated into the program's M&E systems. The program also did not take advantage of the opportunity to assess resilience capacities with the EVD outbreak and recovery, which would have been done if resilience were truly a cross-cutting theme. With respect to the activities that the project proposed to undertake, households already implement plans commensurate with their resources for cyclical stress events that they face, e.g. major expenditures such as school fees and agricultural inputs, and the demands of the lean season. Planning capacity is less of a challenge as is possessing sufficient assets to cope with these cyclical stresses. The second and third activities were indeed part of the program strategy, and the program demonstrably shared information on an ad hoc basis with stakeholders, especially during the EVD outbreak. SNAP, for example, participated with WFP and MAFFS in conducting the nationwide Rapid Food Security Assessment in 2015. While SNAP had limited explicit engagement on resilience, the concept was an integral part of the development objective for the program, and thus SNAP impacted resilience implicitly. In focus group discussions and interviews, the Evaluation Team noted numerous examples of how participants coped with the EVD crisis via capacities and resources developed as a result of SNAP. VSLAs, for example, used savings funds to procure goods in the market to generate revenue, even though the VSLA was not allowed to meet during the SOE. Loans were also given to members during the SOE without meetings, and although repayment had to be extended, most loans were repaid. After the SoE, VSLA loans were used to restart agricultural production and non-farm income￾generating activities, such as soap making. Other smaller events can also be used to illustrate resilience capacities. A wild fire in Bumbukoro in Neya Chiefdom of Koinadugu District, for example, passed through town destroying forty houses. SNAP participants were the first to be able to recover from this event (see picture). 62 SNAP Final Evaluation Report March 2017 International Solutions Group E. Cross-Cutting Theme: Youth Productivity To address the cross-cutting theme of youth productivity, SNAP proposed to assist youth (defined from age 15 to 35) to gain skills for income generation, including basic literacy and numeracy, vocational skills, rural business development skills, and farming skills. It was also suggested that youth could become engaged in managing storage facilities, managing processing operations and microenterprises, and running trade and transport companies for inter-village trading. Observations. As with resilience, youth productivity was identified as a cross-cutting theme in the original proposal, but the program did not rigorously develop approaches that targeted youth productivity. Youth were specifically targeted in the literacy/numeracy and vocational skills training component, and the Evaluation observed a significant number of youth participating in this component. For other activities under SO2, ACDI/VOCA selected participants in collaboration with SNAP communities, and participants self-selected after the criterion of coming from a household participating in SO1 was met. The program's M&E systems did not disaggregate data by age so as to be able to determine how many youth were actually participating in the program, as would normally have been done in a program in which youth productivity was defined as a cross-cutting theme, so specific quantitative data in this regard is unavailable. F. Cross-Cutting Theme: Gender Equity SNAP promised to promote gender equity across all program activities, as follows: To comply with the cross-cutting theme of gender, SNAP employed a gender specialist from 2010 to 2015. Under the guidance of this specialist, the program undertook an in-depth gender analysis to better understand the specific issues relevant to changing gender roles and responsibilities in program areas. This analysis led to a Streamlined Gender Action Plan that tailored gender interventions, including the MAP model. ACDI/VOCA organized specific gender trainings twice a year for SNAP staff, and a three-day national gender training was organized and carried out by SNAP held in Freetown. This training saw the involvement of all the Sierra Leonean ministries and the gender-specific discussions were introduced in the agenda of each ministry. From focus group discussions conducted during the Evaluation, especially with FFS Clusters and participants in VSLAs, the Evaluation Team estimates that 60% of female participants had become more empowered and were represented in leadership positions. Gender equity concepts were introduced in the FFS, VSLA, MCG, LMs and LFs trainings, but vocational trainings implemented by OICI 34 SNAP program proposal, 2010 LESSONS LEARNED Presenting Vocational Skills Opportunities The SNAP Program did not specifically target types of vocational skills training to men or to women. The program did learn, however, that if two options were presented for vocational skills training to a group of men and women, then men gravitated toward the one that was most associated with men and women did the same for the training associated with women. If only one type of training was offered, then both men and women expressed interest in participating, even though the activity was more associated with one of the sexes. 63 SNAP Final Evaluation Report March 2017 International Solutions Group were strongly led by gender role perceptions of participants, i.e., “masonry is for men; making soap is for women”. The Evaluation Team also found that behavioral change messages addressed to PLW and CU2 beneficiaries at the household level by the LMs first and through the Lead Fathers later, directly influenced the men/husband of the same household. Men in these households were more aware of, informed and involved in nutrition, hygiene and health care than before SNAP. Creation of Lead Fathers changed behaviors of men and husbands at the household and community levels, and respondents in qualitative interviews said that there was much less conflict between husbands and wives. A substantial majority (95%) of the SNAP health promoters were men, because (as qualitative research indicated) in Sierra Leone, especially in rural areas, women are not interested in taking a voluntary job that includes traveling long distances from home. This was not a problem for SNAP, however, since the health promoters who were recruited did not have problems relating to LMs and speaking to women about pregnancy-related or women￾related issues. Nevertheless, it is still better for programs to identify, select and employ female community staff, when possible. To be better able to monitor gender changes that may be occurring because of program activities, program monitoring systems should have indicators at all levels to capture these changes. The SNAP Program had only two gender-sensitive indicators in its IPTT: both output indicators. Gender impact indicators should measure differences in how men and women benefit from or were affected by the program, and if the concepts of gender equality and gender equity were well integrated. Relative to impact on gender within the organizations who implemented SNAP, the Evaluation observed that the percentage of female staff working on the program in leadership positions across the three organizations (ACDI/VOCA, IMC and OICI) was poorly represented. Although gender sensitivity in SNAP changed when two women came on board in director positions at IMC and ACDI/VOCA, NGOs, in principle, should give priority to females during employment for all positions, including leadership, in programs that have significant women's participation and gender equity as a cross-cutting theme. 64 SNAP Final Evaluation Report March 2017 International Solutions Group G. Cross-Cutting Theme: Environment The SNAP program proposed two types of activities under the cross-cutting theme of environment. These were (1) to ensure environmental monitoring and mitigation activities, where necessary, and (2) to promote the prevention of environmental damage in all training. Indicators on environmental compliance were added to the IPTT following an environmental mitigation workshop held in 2012; and in 2013, SNAP staff attended an Environmental Best Practices training organized by USAID. The Evaluation Team notes that the Initial Environmental Evaluation (IEE) for the SNAP Program was conducted in June 2012, nearly two years after program start-up. As part of the process of implementing the community assets component of the program (IR 2.3), SNAP contacted Global Initiative, a local consulting firm, to conduct an Environmental Impact Assessment in FY 2015 on the proposed community assets rehabilitation activities for drying floors, storage facilities and feeder roads. Observations. SNAP focused on building capacities of FFS members to reduce slash and burn cultivation and trained FFS clusters to: (a) Select fallow land and not forests to plant short duration crops, (b) Set up small group slash and burn experiments at FFS demonstration plots from which to protect soil biodiversity which helps to improve soil fertility and use of mulching, (c) Use farm tools in ways that prevent soil compacting, and (d) Reduce use of agrochemicals on farms. Notably, the concept of integrated pest management for crops did not emerge in interviews and focus group discussions with participants during the Evaluation. FFS participants indicated that what they learned from the FFS demonstration plots about environmental protection will work on small plots but extending the practice to large farms in an atmosphere of labor shortage would be difficult, i.e. they understood the concept but were not planning to implement it. The SNAP IPTT had six environmental indicators, with performance against these indicators shown in Table 22. Generally, activities associated with this element of the IPTT were not addressed to the same degree as the rest of the IPTT. That said, the program complied with local requirements for obtaining Environmental Impact Assessment Licenses from the Sierra Leone Environmental Impact Agency for of drying floor and store construction. Table 22. SNAP Environmental Indicators in the IPTT Indicator (source of indicator – FFP/USAID or A/V) LOA Target Endline % of LOA Target Achieved (Baseline = 0) # of vocational training instructors, field assistants and beneficiaries sensitized on proper waste management (OP 32) (A/V) 3,000 954 32% # of FFS groups trained on the environmental effects of slash and burn (OP 33) (A/V) 310 310 100% # of health professionals trained on pharmaceutical system management (OP 34) (A/V) 124 100 81% # of bio-sand filters equipped with covered buckets (OP 35) (A/V) 98 98 100% % of fully maintained health facility bio-sand filters (OP 36) (A/V) 100% Not measured --- % of health facilities with new or rehabilitated WASH infrastructure (latrines, water points, or handwashing stations) demonstrating proper maintenance (OP 37) (A/V) 100% Not measured by SNAP --- 65 SNAP Final Evaluation Report March 2017 International Solutions Group H. Cross-Cutting Theme: Governance To address governance as a cross-cutting theme, the SNAP Program proposed to train community groupsin democratic governance principles, including transparency, accountability of leadership and advocacy. In addition, the program intended to work with government ministries at the national and regional level to (a) to ensure that the program did not set up parallel systems, (b) to establish working groups for each technical area and hold semi-annual meetings, and (c) to share work plans with the government ministries and include them in program events. Observations. The FFS Training manual, for example, mentions governance three times but does not have any specific training module for a FFS on governance. However, the agri￾business training provided to the FFS Clusters did include a module on transparency and governance. The VSLA Field Officer Manual had extensive information and training on good governance practices. The qualitative interviews did find some striking differences between FFS Clusters and VSLAs in their understanding and application of accountability, transparency, representation and participation, the key good governance practices. As for the other proposed activities under the governance cross-cutting themes, working groups were never established by the program, nor were semi-annual meetings regularized. The program did an exceptional job, however, of engaging local government authorities, keeping them informed of program activities and including them in program activities, where appropriate (see Section VI-D for more information on the program's efforts toward integration and complementarity). 66 SNAP Final Evaluation Report March 2017 International Solutions Group VI. EVALUATION FINDINGS: PROGRAM IMPLEMENTATION SYSTEMS This section of the Evaluation Report seeks to determine the extent to which SNAP implementers designed and operationalized systems that contributed to program efficiency, and hence effectiveness. These systems, while not part of the intervention logic, are essential elements of good program management. The specific areas assessed by the Evaluation are: personnel management, partnerships, knowledge/data management, integration/synergies and resource management. A. Program Management Program management refers to the vision, leadership, decision-making, problem-solving and planning systems used by a program. In the SNAP Program, management systems were overseen by an ACDI/VOCA-appointed Chief of Party. Implementation for SO1 was led by a Team Leader assigned to IMC, and programmatic implementation for SO2 was led by a Program Director and SO2 Team Leader for ACDI/VOCA. In 2014, ACDI/VOCA created the position of Deputy Chief of Party to oversee day-to-day operations for SO2, supervising District Operations Managers (DOMs). At the District-level, the DOMs for SO2 and SO1 Team Leaders with IMC managed field implementation, sharing and coordinating plans. The SNAP Program did not have a Steering or Advisory Committee. For operational planning, the program conducted Annual Retreats beginning in IY3 to review program progress. During the Annual Retreat, IMC developed and shared SO1 plans, and ACDI/VOCA developed and shared SO2 plans. The CoP merged these plans during preparation of the annual Pipeline and Resource Estimate Proposal. For decision-making and problem-solving, ACDI/VOCA and IMC held weekly meetings in Freetown, and ACDI/VOCA, IMC and OICI held monthly meetings in the districts. SNAP was required to comply with multiple planning periods in preparing operational and financial plans. The IPTT and PREP for USAID/FFP were prepared on an Implementation Year basis, which for SNAP ran from June to May. USAID Reports, including the ARR, CSR and RSR, reported on a USG fiscal year from October through September. Major Observations. The SNAP Program encountered high turnover in leadership positions for the first three years of the program, especially at the CoP level (three CoPs in the first three years) as well as with the SO1 Team leader position and the M&E manager. As a result, program progress was intermittent, as CoPs left and were replaced. The SO1 component was also delayed, further affecting the targeting of SO2 which was dependent on SO1, and the development of effective and efficient M&E systems was delayed. In early 2013, the current CoP arrived and SNAP gained momentum. Changes to overall strategy made it more effective. Commodity pipelines were expanded to ensure more streamlined food delivery and decision-making across the program became more timely. Some components of the program that were understaffed at the beginning of the program were strengthened in 2013 and 2014. VSLA staffing was expanded, for example, and agribusiness positions were added to the program to facilitate the cultivation of business relationships between FFS Clusters and buyers/processors. Staffing for the community infrastructure component was also expanded. At the district-level (Makeni for Bombali and Tonkolili, Kabala and Kailahun), one office for SNAP was created, with all partners housed there, facilitating planning and coordination. All in all, the Evaluation found a general feeling of satisfaction across the program with current management at the consortium level. 67 SNAP Final Evaluation Report March 2017 International Solutions Group B. Partnerships The SNAP Program was implemented by a consortium led by ACDI/VOCA with IMC responsible for SO1 and OICI responsible for the literacy/numeracy and vocational skills training under SO2. ACDI/VOCA was responsible, as lead, for overall management of the program, program-level M&E, commodity management and all other parts of SO2. The Evaluation Team noted strong joint representation by the consortium partners (CPs). Both in the field and at the central level, representatives from the CPs introduced themselves as SNAP, rather than the name of the NGO with whom they worked. This meant that “the program carried only one flag” and it eliminated any confusion among local authorities and other external stakeholders as to who was being represented. Partners held regular (weekly, monthly) coordination meetings and logistical coordination in the field allowed better use of the resources by the CPs. Major strategic decisions for the program were made in-country and this optimized assets and vision. Key interviews with ACDI/VOCA managers pointed out that having technically specialized partners such as IMC and OICI in the consortium was much more useful than recruiting new in-house technical expertise for a program. The close-out phase of the program was organized by all CPs well before the planned end of SNAP in June 2015, and stakeholders saw the face of SNAP and not individual partners in close-out meetings. Some stakeholders expressed confusion about the close-out date, however, resulting from the two program extensions approved by USAID/FFP moving the close-out date back. Partnership agreements and letters of attestation from GOSL ministries were signed at the beginning of the program and renewed every year specifying roles and responsibilities for SNAP and partners. Communications between ACDI/VOCA and partners in Sierra Leone were reported to be cordial and productive. LESSONS LEARNED Partnerships Having a common office at the district level in which all CPs were housed cultivated good relationships and facilitated the sharing of information and logistics.. Having a common project identity increased recognition for all partners for good work and assigned joint responsibilities for solving problems. Typical SNAP Sign Board 68 SNAP Final Evaluation Report March 2017 International Solutions Group C. Knowledge Management Knowledge management refers to (a) how good practices, lessons learned and other types of knowledge are brought into a program to improve effectiveness or efficiency, (b) how M&E systems generate information that is used for decision-making and reporting, and (c) how good practices, lessons learned and other knowledge generated by the program is captured and disseminated to other development practitioners. 1. Bringing New Approaches and Other Knowledge into the Program. New approaches are brought into a program through staff recruited who bring their ideas and experience, through cross-visits to or meetings with other projects, through training events organized by consortium members or others, or through internet searches. Observations. Over its life, the SNAP Program implemented the following that brought in new ideas and approaches.  Collaborated with the SLARI on vegetable cultivation, seed viability testing and input handling techniques, and cultivation of nutritious, high-value crops including pigeon peas, maize, sorghum and sesame.  Contracted Vicsons Business Development Services to provide agri-business training using training modules developed by the Agribusiness and Marketing Specialist hired from the PAGE Project for a 3-month period.  Attended a one-day conference on obstetric fistulas organized by Health Poverty Action and the Haikal Foundation, sponsored by the United Nations Population Fund to discuss fistulas in relation to maternal mortality and morbidity.  Met with Helen Keller International (HKI) to understand preliminary results of a pilot study conducted by HKI to show that the integration of vitamin A supplements, family planning, IYCF and growth monitoring will improve coverage of essential health services.  Obtained VSLA guidelines from several organizations implementing a VSLA approach to develop implementation guidelines and a manual for SNAP.  Obtained business and financial management training modules from the FAO/GAFSP support to the Smallholder Commercialization Programme  Contracted SLIBA for Post-Harvest Loss Management Training 2. Monitoring and Evaluation. The M&E function in SNAP was managed by an M&E Manager who oversees a team composed of a database manager and four data entry staff based in Freetown. In addition, each district office had an M&E officer who coordinates data collection in the field. During large surveys such as the Annual Survey on SO2 Indicators, additional data entry clerks were hired on an ad-hoc basis. This team was primarily responsible for data collection and management associated with SO2. Under SO1, an IMC M&E Coordinator manages M&E and separate M&E officers for SO1 in each district. Information from this separate M&E system was provided to the ACDI/VOCA M&E Manager to update the IPTT. In May and June of 2012, a FFP Data Quality Assessment specialist visited the program to review data quality as well as assist the program to accelerate the development of the M&E systems. With the assistance of the FFP regional M&E specialist, SNAP annual targets and baseline values in the IPTT were revised and indicators related to environmental compliance from the Initial Environmental Evaluation (IEE) conducted in June, 2012 were added to the IPTT. The SNAP Program began implementing Annual Surveys to obtain information on SO2 performance indicators from the IPTT beginning in FY 2012 through FY 2015. 69 SNAP Final Evaluation Report March 2017 International Solutions Group Quarterly reports and Annual Results Reports were prepared for submission to FFP and were shared with government stakeholders. The SNAP Program proposed to implement operational research on Early Childhood Development (ECD) to determine the impact of mother-child interaction, nutrition and feeding practices on child growth and cognitive development to inform the creation of a training module on ECD to be used with the MCGs. IMC had implemented similar research in Uganda and Ethiopia and had planned a pilot phase starting in 2011 in three SNAP Chiefdoms (Sella Limba, Sanda Loko and Gbanti Kamaranka) in Bombali District with the distribution of ECD￾focused training material to a few HPs and some LMs. This pilot continued in 2012 in the same chiefdoms and a rollout phase to the other three districts was planned pending analysis of the pilot. For reasons that were not made clear to the Evaluation Team, the ECD research was never completed nor extended to the other districts. Major Observations. The M&E unit for SNAP experienced high staff turnover. This was particularly critical at the beginning of the program with the need to set up efficient and effective data collection systems at the start. The SNAP Program recognized early the need for a centralized program participant database. ACDI/VOCA hosted meetings in January 2012, to synchronize the beneficiary registration lists across the three consortium partners. A year later, SNAP launched a Household Survey to identify SNAP households, link each beneficiary in the targeted household to the SNAP program components to populate the programs integrated database. The survey data was never fully analyzed, however, and the program did not have an integrated household database. Even up to October 2015, following a training conducted by ACDI/VOCA’s Regional Systems and Technology Director, the program continued to work on integrating the various program databases that existed. In addition to the regular monitoring implemented through the program's M&E systems, the SNAP program conducted several ad-hoc M&E studies to generate information to improve performance, including the following.  In FY 2015, a Financial Viability Assessment (FVA) on all 170 VSLAs to identify strengths and weaknesses to inform on-going training plans.  In FY 16, a study of five ABCs and the twenty-six SNAP FFS Cluster groups located near them to identify strategies for profitably linking SNAP FFS Clusters with ABCs.  In FY 16, a field audit with 45 vegetable groups to assess their sustainability with continued vegetable cultivation without further SNAP seed support. Right up into FY 2016, M&E tools were still being revised by the program. Within the past year, for example, the M&E team re-designed data collection and entry sheets to enhance data collection and increase quality for entry in the projects databases for SO1 and SO2. Training reporting forms were in an easy to use format with drop down menus (e-copy) or multiple data field tick boxes on paper to avoid ambiguities in writing training titles and related information. Similarly, attendance sheets were streamlined to cut out writing and excessive use of paper. These changes increased data quality, but came very late in the program life. Lessons Learned - Integrated Participants' Database The SNAP Program reaffirmed the experience of many multi-sectoral programs regarding establishing a program database that indicates which participants were engaged in different program activities. SNAP recognized the value in having such a database, but began working on merging different databases set up by the program after they had already been established. A clear lesson learned is that it is much more difficult to merge databases after they have been created than to create a unified database at the very beginning of a program. 70 SNAP Final Evaluation Report March 2017 International Solutions Group 3. Capturing and Disseminating Knowledge Generated by the Project. The SNAP program did not have a dedicated staff position for the function of capturing and disseminating knowledge generated by the program. The function was performed as monitoring reports were prepared and through participation in various events and meetings Major Observations. Success stories were submitted with Annual Results Reports. Program staff also participated in various sectoral inter-agency meetings at the national level, including food security working group meetings coordinated by FAO and health sector working group meetings at the MOHS. SNAP was also well-represented in district-level coordination meetings organized by the GoSL. A number of training manuals were developed by the program describing the approaches being used by the program. These include: o Various Mother Care Group Training Manuals for different Modules o Farmer Field School/Farming as a Family Business Training Manual o Vegetable Production Training Module for Agriculture Field Agents and Community Facilitators o Postharvest Loss Management Training Module o Village Savings and Loans Association (VSLA) Field Officers’ Training Manual o VSLA Private Service Provider (PSP) Training Manual o Financial Literacy Training Manual 71 SNAP Final Evaluation Report March 2017 International Solutions Group D. Integration & Complementarity The final evaluation examined the program systems for facilitating integration across activities within the program as well as how the program complemented programs of other stakeholders to identify good practices and lessons learned. 1. Integration Across Sectors and Activities Within the Program. SNAP, as in many Title II programs, was designed as an integrated food security program and planned to implement as many activities as possible from different SOs in the same household, although ultimately this was achieved in only around 33% of the villages in which the program was implemented. The Evaluation Team found that even if activities in the two different SOs were implemented by different partners, there was some integration across sectors. In the first year and a half of implementation, SO1 fell behind because of staff turnover and there was confusion around the selection criteria for food recipients which affected the targeting for SO2, and the program was on the verge of losing integration between SO1 and SO2. ACDI/VOCA made efforts later, however, to integrate SNAP activities across SOs and encourage cross-fertilization among field staff. All SNAP staff were trained, for example, in gender equity; FFS received health and nutrition messages; LMs were trained in kitchen gardens by SO2 staff; and staff from all consortium partners were motivated to meet on monthly-basis. Staff interviewed during the final evaluation reported their appreciation for these efforts, which resulted in a better overall coordination and greater synergies across SOs and in project management/administration, for example, in logistics and budget resources usage. Integration between SOs, especially starting from 2013, improved to contribute to smoother program implementation. 2. Complementarity with Agendas and Programs of Other Stakeholders. SNAP committed to full integration and alignment with relevant national and donor program strategies and with strategic plans at district and central-levels. Evidence from the interviews with representatives of government stakeholders at different levels found that Ministry officials at all levels were fully aware of the aims and activities of SNAP, receiving updated program implementation information on monthly basis. Efforts to integrate with national policies/strategies were made (i.e. Mother-to-Mother MOHS model harmonized with SNAP MCGs training modules) and even the SNAP quarterly plans were incorporated in the MAFFS and MOHS strategies at the district level. SNAP also implemented joint monitoring and supervision with government stakeholders, and provided support for national health campaigns and surveys (e.g. MOHS provided households in the survey with soap purchased by SNAP from vocational training graduates). The Close-Out/Exit strategy was clearly presented by consortium partners to stakeholders in 2014 with roles and responsibilities clearly outlined. During the interviews, some stakeholders expressed a desire to be able to see the strategy before they were required to implement it. The general qualitative/anecdotal impression gathered via the Evaluation research was that these actors were either not in place during the close-out phase or were not aware of it at the time. Over the life of the SNAP, interaction with NGO stakeholders and local authorities promoted knowledge-sharing and successfully ensured no duplication of interventions. 72 SNAP Final Evaluation Report March 2017 International Solutions Group E. Resource Management 73 SNAP Final Evaluation Report March 2017 International Solutions Group 74 SNAP Final Evaluation Report March 2017 International Solutions Group Figure B. Timeline for SNAP Food Distributions March 2011 started PLW enrollment June 2011 started food distribution for PLW February 2012 started food distribution for CU2 October 2013 stopped pregnant women enrollment July 2014 stopped food distribution due to Ebola 75 SNAP Final Evaluation Report March 2017 International Solutions Group 3. Human Resources. Each consortium partner manages its own human resources for the program. The CoP and Deputy CoP positions were managed by ACDI/VOCA. SO1 staff were managed by IMC, and SO2 staff were managed by ACDI/VOCA, apart from those staff who were engaged in literacy/numeracy and vocational skills training implemented by OICI. Major Observations. The SNAP Program faced serious staffing challenges, especially at senior levels early in the program. In its first three years, for example, the program had three CoPs and turnover at the SO1 Team Leader and M&E Manager positions. At the field level, the M&E unit had high turnover over the life of SNAP, with challenges in recruiting VSLA officers and field officer positions. The SNAP Program chose to work in some very remote locations and the primary reason given to explain the high turnover/recruitment difficulties was that specialized staff were able to find other opportunities for easier work at the same or better compensation levels. Food distribution staffing was increased in 2013 in order to improve the timeliness of delivery, especially to more remote locations. The SNAP Program had a Gender Officer position responsible for building staff capacities and mainstreaming gender. This position was vacated during the EVD crisis, and the decision was made not to replace her because almost all gender activities/trainings were finished and new beneficiaries were not going to be added to the program. However, regional gender specialists helped cover gender areas after EVD. 76 SNAP Final Evaluation Report March 2017 International Solutions Group During the EVD crisis, staff still continued minimal administrative work. Expatriate staff were evacuated from July to October 2015, but continued to coordinate activities remotely. When the crisis first emerged, staff in-country were provided with transport to and from the office, to minimize exposure to EVD. During the peak of the crisis, staff were paid administrative leave for seven months until the program received a costed extension. During this period, when it appeared that SNAP may be closing, a number of staff left the program to join other projects. Since food distributions were suspended when the EVD occurred, food distribution staff were released to fill other vacant positions in the district offices after food distribution stopped. No SNAP staff were personally affected by EVD. The program provided job-related training to staff, as well as on-line training in key topics for staff with access to the internet. In 2010, a few SNAP staff were discovered to be engaged in fraudulent activities related to program commodities. Five staff, including the M&E Manager, M&E Coordinator, the employment contracts of two Warehouse Officers and the Training Coordinator were terminated as a result of the investigations. SNAP systems and controls were adequate to monitor and detect this issue and it was appropriately reported to FFP. 77 SNAP Final Evaluation Report March 2017 International Solutions Group VII. SPECIAL TOPICS In the development of the terms of reference for the SNAP Final Evaluation, USAID/FFP Sierra Leone noted several topics of interest relative to program implementation strategies and contextual factors (notably the impact of the EVD outbreak in 2014) that may have a bearing on the design/implementation of future, similar, programs. Specific information is provided on these here. A. SNAP Program Extensions The SNAP Program received a costed extension for one year and then a no-cost for six months because of the EVD crisis. The question was raised as to whether this was a good decision or whether the program should have been closed out when the EVD crisis occurred. This evaluation concludes that the extensions to the program were critical to leveraging and sustaining the benefits achieved to that point. The additional time facilitated the recovery of the various types of participants in the program, allowed the program to continue implementing the behavioral change models under SO1, and solidified both the FFS Cluster and VSLA organizations. B. Impact of Ebola The EVD crisis certainly severely disrupted program activities, as has been described; and for some, such as the construction of community infrastructure and the formation of women's vegetable groups, this occurred at a particularly critical developmental juncture. The EVD crisis also had the effect, however, of pushing community participants to depend more on themselves to maintain their livelihoods, and it certainly reinforced the personal hygiene messages, e.g. handwashing, promoted by SNAP. Household productivity declined but some groups, such as the VSLAs, actually became more cohesive. C. Geographic Targeting Strategy The SNAP Program targeted eighteen chiefdoms in four districts and in each chiefdom touched nearly all villages in some way. The selection was based on a rigorous analysis to identify the most food insecure chiefdoms in the five districts (including Kenema) specified for targeting in the call for proposals37. In interviews with stakeholders during the Evaluation, the full coverage in a selected chiefdom was highlighted a number of times by both Paramount Chiefs and district-level authorities as a sound strategy. The Paramount Chiefs cited some disappointment at the differing intensity of intervention between villages but appreciated that an attempt at full coverage was better than partial coverage. This was reaffirmed at the district level by stakeholders, with their main interest in having more chiefdoms in the district engaged. A more important consideration was the program strategy itself. The greatest impact was on households benefiting from the three main interventions: MCGs, FFS Clusters and VSLAs. With the available resources, it would not have been possible to reach more households with this integrated package in eighteen chiefdoms. The Evaluation concludes that a better approach would have been to select a smaller number of chiefdoms and implement the more integrated approach. It would also be advisable to retain the same number of districts, and even consider adding Kenema next time, with more local stakeholders exposed to approaches. 37 Fiscal Year 2010: Title II Proposal Guidance and Program Policies, Supplementary Fiscal year 2010 Title II Multi-Year Assistance Program, Country Specific Guidance: Sierra Leone. 78 SNAP Final Evaluation Report March 2017 International Solutions Group D. Equitability of Benefits Within Communities The final evaluation did not find any major social class or political tensions existing to any great extent in the villages visited during the Evaluation, which are in some of the more remote chiefdoms in Sierra Leone. Villages were surprisingly homogenous, and the main sources of tension following the war, i.e., tension from disenfranchised youth or ex-combatants, had largely dissipated, as youth have either been integrated into communities or have left to seek work elsewhere. The only significant area of marginalization that the Evaluation noted was in villages of a minority ethnic group who receiving less attention from paramount chiefs and other local authorities. E. Food-Related Dependency The distribution of food in the SNAP Program did not create any significant food-dependency on the part of program participants, although had the EVD outbreak not forced attention elsewhere, this could have changed. The amount of food per household was relatively small, in any case. A more significant consideration is that, because it was widely shared with extended family and friends as is customary in Sierra Leone, distributed food in SNAP had limited impact on reducing child malnutrition, while consuming substantial logistical and financial resources. The greatest impact of the SNAP food was accustoming participants to the services of PHUs, and there are much cheaper, more efficient, ways to provide this incentive. The Evaluation concludes from the available evidence that the SNAP Program could indeed have achieved similar impact without the use of distributed food. A substantial BCC strategy focused on food diversity and hygiene, with the consumption of local foods bolstering local agricultural production, could have had even larger and more sustainable results. Future Title II programs, especially in contexts like Sierra Leone which have substantial food production potential, should consider limited use of imported commodities, potentially focused on the start-up phase to encourage beneficiary enrollment. F. Opportunities for Synergies Between Food for Peace and Feed the Future Programming At the time of research, the Feed the Future Program was just starting up in Sierra Leone, and the Evaluation Team was queried as to how best to interface Food for Peace (FFP) and Feed the Future (FTF) programming. A key consideration is the different targeting strategies of FFP and FTFL: FFP generally targets the most food insecure households and geographic areas, while FTF generally targets geographic areas that have good agricultural development potential and participants who have good potential for expanding roles in value chain development. FFP is a poverty-reduction investment by USAID while FTF is an agricultural development investment. To have the greatest impact on their objectives, the two programs may not target the same populations or geographic areas. However, the Evaluation Team identified opportunities for effective overlap and interface between the two investments. FFP programs are focused primarily on household and community-level capacity building, increasing productivity and building links to markets. This productivity should be set in the context of one or more value chains extending from input supply through production and marketing along various channels including local consumption, national consumption and exportation. Value chain development should be the focus of FTF investments, to expand channels and open up new opportunities for engagement of targeted impact groups in the value chains. The SNAP Program, for example, saw good progress on increasing farm productivity and forming groups, i.e., the FFS Clusters, to serve as producers, collectors and contact points for buyers and processors. Fitting these groups into expanding value chains, i.e., analyzing value 79 SNAP Final Evaluation Report March 2017 International Solutions Group chains to identify bottlenecks and leverage points and then addressing these through appropriate interventions, should be the next step toward reducing food insecurity and facilitating agricultural development. G. Cost Effectiveness of Large Programs At , the SNAP Program was perceived by all stakeholders interviewed (and especially the Government of Sierra Leone) to be an expensive program in Sierra Leone. This is particularly true in a context in which material investments, such as drying floors and other construction, are perceived by national stakeholders as higher-value development outcomes than capacities or ideas. Much of the investment by SNAP was in capacities or ideas, e.g. essential nutrition actions, essential hygiene actions, simple cultivation techniques such as appropriate plant spacing and seed density, and the VSLAs. The knowledge gained from these ideas is potentially long-lasting and replicable as it is passed on to others and future generations. This is the true impact achieved by programs like SNAP that are human capacity building in nature. 80 SNAP Final Evaluation Report March 2017 International Solutions Group VIII. CONCLUDING REMARKS & RECOMMENDATIONS The total cost of the SNAP Program to USAID/FFP, including the value of the commodities obtained and shipped to Sierra Leone for monetization and distribution, was for 6.5 years. This section discusses how this investment compares to the sustained impact generated by the program and the efficiency in use of resources. Four fundamental questions underlie the final evaluation of the program: 1. How relevant were program intervention for the targeted impact groups? The Evaluation concludes that all elements of the SNAP Program strategy were relevant, except the Learning for Labor idea which was dropped from the strategy at an early stage. 2. How effective were SNAP interventions in achieving impact? As described above, the strongest positive program-related outcomes were observed where the greatest integration occurred across SOs. The program could have been more effective, had there been more such integration. 3. How efficient was the program in using resources to produce the desired results? For the first two and a half years of the program, both commodities and human resource were not used as efficiently as expected. the high staff turnover resulted in the program moving intermittently both strategically and operationally. When monetization was discontinued and staff positions filled at senior levels, the program began to use resources more effectively. 4. How sustainable is the impact that has been achieved? Sections describing Outputs Under Intermediate Results (Section IV) and Outcomes and Impact (Section V) describe findings on the sustainability of the impact achieved. The Evaluators note that this evaluation focused on program performance at the conclusion of SNAP, and thus was limited in its scope to assess the sustainability of medium to long-term program impact. The SNAP Program worked in a total of around 2,000 villages, reaching 55,200 households comprising over 350,000 persons. While the program had positive impacts on these 55,000 households, more substantial impact was achieved by the SNAP Program on an estimated 3,200 VSLAs. households who had members participating in all of the three major program interventions, i.e., a MCG, a FFS Cluster, and VSLA. In addition, the program strongly benefited children who would have become chronically malnourished without program intervention, members of VSLAs, some FFS cluster members, and masons and carpenters trained by the program and had new occupations. Offsetting this positive impact, the high staff turnover at the start of the program resulted in slow start-up for the first two and half years of implementation. the EVD crisis brought the program to a virtual standstill just as it was ready to gain traction. Considering the full life of the program, including the slow start-up, the impact given the total investment, and the other considerations affecting implementation, this evaluation concludes that the SNAP Program was a better than average, and hence worthwhile, investment. 81 SNAP Final Evaluation Report March 2017 International Solutions Group Key Best Practices & Recommendations: • The implementation of the baseline and endline quantitative household surveys at different seasonal times resulted in discrepancies due to seasonal variations, particularly with respect to food availability and nutritional data. Future such surveys should be carefully planned to take place at the same time of year to optimize comparability between metrics. • Further discrepancies between program metrics resulted from misalignment of program targets with baseline figures and a revision of the monitoring framework. Such mid￾program changes, while sometimes unavoidable, should be very carefully planned so as not to lead to issues in drawing conclusions on program effectiveness, as with this evaluation. • The infrastructure-related elements of the program were among the least impactful and sustainable elements of the program, particularly given the need for ongoing investment in infrastructure for maintenance (such commitments, while solicited from the GoSL, were not realized). Investment in these areas should be very carefully considered during future programming, and not implemented unless concrete plans and commitments to their sustainability are in place. • Involving men in SO1 Behavior Change Communication activities was a strong approach to sensitizing and increasing men's participation and support for women's and children's health. The practice of using men to inform and sensitize men/husbands produced excellent results, and should be replicated in programs that have a focus on health and nutrition behavior change. However, the Evaluation Team would stress the importance to set-up and implement the father care groups at the same time as the mother care groups to facilitate gender equity. • Establishment of small village-based groups of 25 to 30 members to use a farmer field school approach to test new ideas and approaches, and then clustering two to three of these groups for increasing production was a strategy that the Evaluators recommend or future consideration in similar programming. Such an approach continues learning processes introduced by the FFS while also enhancing benefits for members through increased production and economies of scale. • The VSLA approach used by SNAP was often cited by interviewees as the most important intervention introduced by SNAP, further evidenced by observed replication of the model without program intervention at the time of research As the VSLA methodology will likely contribute to sustained benefits the Evaluation Team recommend that it be included in future programs. • Given the reasonable risk of crises in program countries, monitoring of trigger indicators and a framework for emergency response (originally part of SNAP design, but not incorporated into M&E Systems) facilitates better preparation and more timely response. The Evaluators recommend that future programming revive and implement this process. • Sequential activities (placing health promoters, identifying participant households, clustering beneficiaries) required other program components to wait before identifying their own participants. This could be done more efficiently in future programs by development of targeting strategies that allow components to begin in a timely manner in ways that facilitate integration of components later. • The tracking of functional business relationships, key to the sustainability of FFS Clusters, are important metrics to help identify to whom the sales are made and whether 82 SNAP Final Evaluation Report March 2017 International Solutions Group there are repeat sales to the same buyers/processors. Future programming should incorporate such indicators. • The SNAP Program had only two gender-sensitive indicators in its IPTT: both output indicators. To be better able to monitor gender changes that may be occurring because of program activities, program monitoring systems should have indicators at all levels to capture these changes. Further, priority should be given within implementing strategies to the recruitment and hire of female staff, particularly in leadership roles. • While there is good evidence to indicate that distribution of food did indeed have positive nutrition-related outcomes, the SNAP Program could have achieved similar impact without the use of distributed food. A substantial BCC strategy focused on food diversity and hygiene, with the consumption of local foods bolstering local agricultural production, could have had even larger and more sustainable results. Future Title II programs, especially in contexts like Sierra Leone which have substantial food production potential, should consider limited use of imported commodities, potentially focused on the start-up phase to encourage beneficiary enrollment. • The Evaluation noted good opportunities for synergies between Food for Peace and Feed the Future programming. Value chain development at different levels (household/community to central level) should be the focus of these investments, to expand channels and open up new opportunities for engagement of targeted impact groups in the value chains. • The evidence gathered by the Evaluation indicates that the most substantial outcomes of the program were where the greatest integration/overlap occurred across Strategic Objectives. The program could have been more efficient and effective had there been more such integration, and as such, the Evaluation recommends that integration across objectives, activities and locations be a central pillar of future programming. 83 Annexes Annex A: Evaluation Scope of Work Annex B Evaluation Operational Plan Annex C Endline Survey Report Annex D: Supplemental Annual Results Survey on Agricultural Technology Adoption Annex E: Summary of Persons Interviewed for Qualitative Data Collection Annex F: Final SNAP IPTT SCOPE OF WORK Sierra Leone Sustainable Nutrition and Agriculture Promotion Project (SNAP) Final Project Performance Evaluation Agreement period: June 10, 2010 – December 31, 2016 Cooperative Agreement No.: AID‐FFP‐A‐10‐00014 1 Introduction ACDI/VOCA is seeking expressions of interest from external consultants to design and carry out a final end–of‐project summative performance evaluation for the five‐year Sierra Leone Sustainable Nutrition and Agriculture Promotion project (SNAP), implemented in four districts of Sierra Leone. The evaluation will cover the five‐year period, including the project extension due to the Ebola outbreak from June 2010 to December 2016. The successful bidder will conduct a final evaluation of ACDI/VOCA’s SNAP project in Sierra Leone that consists of two components to determine to what extent the food security status of the targeted population has changed and how the project has contributed to the resiliency of the targeted communities. Part One will be a review of the quantitative end‐line survey consisting of a population‐based, survey conducted during March‐  May 2016. This end‐line survey measures change and assesses the results achieved by the SNAP program by statistically comparing the end‐line survey results in SNAP operational areas against the results of the baseline survey that was conducted during the same period in 2011. Final year (FY15) results for indicators not covered in the end‐line survey will be reviewed in the FY15 Indicator Performance Tracking Table. Part Two will be a qualitative evaluation analyzing changes in awareness, perception, knowledge, and behavior of target beneficiaries. A series of focus groups, collection of most significant change stories, and key informant interviews will be conducted to gather information on respondents’ attitudes and practices related to agriculture, market, health, nutrition, and gender. The respondents will be sampled from beneficiaries in the SNAP intervention area, including key stakeholders from District and local government authorities, NGO and CBO representatives. 1.1 Description of Program ACDI/VOCA and partners, the International Medical Corps (IMC) and Opportunities Industrialization Centers (OIC) are implementing a USAID Food for Peace (FFP) funded project, Sustainable Nutrition and Agricultural Promotion (SNAP). The overarching goal of the SNAP project is to reduce food insecurity and increase resiliency among the most food insecure and vulnerable rural populations in targeted communities in 18 Chiefdoms in the Districts of Kailahun, Bombali, Tonkolili and Koinadugu. SNAP is 1 addressing the root causes of malnutrition, food insecurity, and vulnerability to shocks through improved health, hygiene, and nutrition practices as well as improved livelihood opportunities. To achieve the project goal the SNAP MYAP is organized in two Strategic Objectives (SO). SO1 aims at reducing chronic malnutrition among children under five, while SO2 aims at enhancing livelihoods among women and youth in the project communities. In addition, the project addresses five crosscutting themes: resiliency to shocks, productive youth, gender equity, environmental stewardship, and good governance. SNAP provides the targeted population with the necessary tools to improve their food security situation by addressing the key factors influencing utilization, availability, and access to nutritious and diverse foods. SNAP facilitates linkages among donor, government, and community activities to deepen impact and sustainability. SNAP’s sustainability strategy strengthens the capacity of communities as well as public and private stakeholders. 1.2 Areas of Operation SNAP is implemented in 18 chiefdoms, spanning 4 districts:  Kailahun District (Kissi Tongi, Kpeje West, Mandu, Penguia, Yawei)  Bombali District (Sella Limba, Sanda Loko, Gbanti Kamaranka)  Tonkolili (Kafe Simira, Kalasongia, Yoni)  Koinadugu (Diang, Folosaba Dembelia, Kassunko, Mango, Neya, Neini, Wara Wara Bafodia) With the outbreak of Ebola, the fear of the disease, coupled with the restrictions under the State of Emergency within the country, and the timing during the peak agricultural season impacted the economy, 2 disrupted farming activities and markets ‐ particularly in remote areas ‐ and has led to a decline in nationwide economic activities with the potential to impact a wide variety of socioeconomic factors. SNAP activities suffered the same reality and was extended by one year to address the Ebola outbreak and its impact. Figure 1: SNAP Results Framework IMPACT Project Goal: Reduce food insecurity and increase resiliency among vulnerable rural populations in Bombali, Kailahun, Koinadugu, and Tonkolili. OUTCOME Objective 1: Reduce chronic malnutrition among children under five IR 1.1: Improved access IR 1.2: Improved health to and use of quality and nutrition practices nutrition and health at the household level. services for women and children 0‐59 months. Objective 2: Enhance livelihoods for vulnerable people, especially women and youth IR 2.1: IR 2.2: IR 2.3: Improved Increased Increased community agricultural access to assets. productivity at financial and the household marketing level resources INPUT/ OUTPUT Activities Activities Activities Activities Activities  Train DHMT staff on  Train households in  Farming as a  Create,  Infrastructur pre and post‐natal key family nutrition family expand and e repairs care services for behaviors business train village including pregnant & lactating  Expand caregivers’  training savings and markets and  women cooking trainings Expand FFS loans  feeder roads Train MOHS district through lessons on training to  associations Expand health staff and health preparation of include more Provide community  promoters on IMCI nutrient rich and women, training in crop storage Improve PHU hygiene diverse foods particularly literacy and facilities and sanitation  Distribute nutritious  mothers numeracy for  Engage youth  conditions foods to vulnerable Promote women and in labor for Train community groups and diverse crops  youth learning members in detection household members for added Provide program of SAM and referral to  Train households on nutritional training for (vouchers)  treatment facilities vegetable growing for  value youth in Support growth increased diversity of Provide vocational monitoring within foods training in skills, PHUs through training  Form mother care crop agribusiness  and GIK groups preservation and Educate target  Train mother care and entrepreneur communities on groups  processing  ship proper health seeking  Provide education Conduct value Train input behaviors messages on the chain analysis suppliers in essential hygiene on crops for management actions to mother family food practices 3 2 Purpose of the Final Evaluation The purpose of this final evaluation is to assess the relevance, efficiency, effectiveness, and sustainability of the project and provide an assessment of the results achieved, reasons for levels of achievement or non‐achievement, and lessons learned from the SNAP project. The evaluation will assess both the intended and the un‐intended outcomes of the project, and will document the performance results of the project, and share learning and challenges. The final evaluation will employ both quantitative and qualitative methodologies to fulfill the following specific objectives:  Evaluate the theory of change of the SNAP project, through the quantitative measurement of endline outcomes and impacts as identified in the Indicator Performance Tracking Table (IPTT), utilizing the definitions of the indicators described in the Performance Indicator Reference Sheets, and then creating plausible links between outputs and outcomes/impacts.  Evaluate the results of cross‐sectoral integration across the activities, SOs, and implementing partners through comparison of households participating in multiple activities and through comparison of results between districts.  To identify constraints or/and difficulties, opportunities as well as successes encountered in project implementation (“critical analysis,” which involves determining the strengths and weaknesses of the project).  Identify project strategies and interventions that are demonstrating elements of sustainability and ascertain whether targeted groups and households will continue to practice/display the acquired behavior changes and experiences beyond the life of the project.  Highlight and document salient lessons learned to aid future programming of Title II interventions in Sierra Leone and in similar contexts.  Assess the effectiveness of project coordination within the partners and identify best practice for future programing. Findings and results of the population‐based impact indicators measured through the end‐line survey: Table 1: Indicators to be considered during the evaluation No. Impact (IM) Indicators Disaggregation IM2 % of underweight children 0‐59 months of age (weight‐ District, Age, Sex for‐age z‐score (WAS) <‐2SD) IM3 % of stunted children 6‐59 months of age (height‐for‐age District, Age, Sex z‐score (HAZ) <‐2SD IM4 Average Household Dietary Diversity Score District IM6 Average # of months of adequate household food District provisioning 4 Indicators measuring change among SNAP Program Beneficiaries as measured in the Indicator Performance Tracking Table (IPTT), specifically: No. Outcome (OC) and Impact (IM) Indicators Disaggregation SO1: Reduced Chronic malnutrition among children under 5 IR 1.1 Improved access to quality nutrition and health services for women and children aged 0‐59 months OC2 % of children 0‐59 months regularly attending growth Chiefdom, Age, Sex monitoring OC3 % of pregnant women completing 4 antenatal visits District, Age OC4 # of deliveries with a skilled birth attendant in USG assisted Chiefdom programs OC5 # of mothers participating in postpartum / newborn visits Chiefdom within 2 days of birth in USG programs (F) OC6 % of mothers with children (6‐23 months) receiving 3 District, Age postnatal visits OC7 # of cases of child diarrhea treated with ORS or Zinc in USG Chiefdom, Age, Sex programs IR 1.2 Improved health and nutrition practices at the household level OC9 % of children 0‐5 months of age exclusively breastfed Chiefdom, Sex OC10 % of caregivers demonstrating proper food hygiene Chiefdom, Sex of the head of behavior HH, Age OC11 % of caregivers demonstrating proper personal hygiene Chiefdom, Sex of the head of behavior HH, Age SO2: Enhanced livelihoods for vulnerable people, especially women and youth OC14 Average rural household revenue Source of revenue (farm vs non‐farm job) Chiefdom, Sex of the head of household OC15 % of participants with non‐farm income Type of job, Chiefdom, Sex, Age IR 2.1 Increased agricultural productivity at the household level OC16 % of farmers reporting use of association benefits Chiefdom, Sex OC17 % of farmers who have adopted at least three agricultural District, Sex, Age technologies OC18 # of farmers, processors, traders, input suppliers and others Sex who have adopted three or more new technologies or management practices 5 No. Outcome (OC) and Impact (IM) Indicators Disaggregation OC19 Average yield per hectare (Lowland rice, groundnut, Crop, District cassava, sweet potato) OC20 Average value of crop sales (Lowland rice, groundnut, Crop, District, Sex of the cassava, sweet potato) head of household OC21 # of FFS groups graduating into producer or marketing Chiefdom associations OC22 # of individuals reporting consumption of micronutrient rich Chiefdom, Sex of the crops as a result of FFS training respondent IR 2.2 Increased access to financial and marketing services OC23 Average VSLA savings (Archived) Chiefdom OC23 % of VSLA members meeting their commitment savings Chiefdom, Sex OC24 % of farmers using improved storage Sex, District 3 Key Evaluation Questions Overall the evaluation will attempt to respond to the questions below. The Evaluation Team Leader will determine the key questions to be addressed under each SO, IR and Sub IR. Criteria Main Evaluation Questions for both SO1 and Suggested Sub‐Questions SO2 Specific for each SO Relevance   To what extent was SNAP effective in Under SO 1 achieving its strategic objectives and  What behavior change intermediate results? Were the planned promoted by the program activities appropriate for the food has had the greatest security problems identified in the impact on affecting program results framework? change in nutritional   How effective were the partnerships status of pregnant and developed with GoSL at national, district, lactating women (PLW) and Chiefdom level as well as other with children under the development actors in delivering age of two? intended results aligned with country  What were the most development objectives? Was there a positive intervention(s) well‐developed exit strategy effectively that impacted on the communicated? behavior change   How did the stakeholders perceive the promoted by the project activities and understand program? anticipated outcomes? Under SO 2  What technologies are mostly adopted by farmers? What factors had the most positive 6 Criteria Main Evaluation Questions for both SO1 and Suggested Sub‐Questions SO2 Specific for each SO influence on the adoption rate?   How well did SNAP integrate gender aspects in its SO2 programming to favor women’s access to agriculture resources and benefits? Effectiveness   What are the factors that Under SO 1 hindered/assisted the effective  How effective was integration of activities? conditional food aid in   Were activities sufficient and delivered promoting behavior timely to produce the desired results? If change? How has the not, how could they have done discontinuation of food aid otherwise? impacted on participation   How effective was the project in care groups and support management structure and the to pregnant women in the management of the project. What was communities? Did the the relationship between partners and project maintain minimum how did this impact on achieving standards beneficiaries program objectives? meet expected standards?   Did the project have a clearly defined If not, what were the comprehensive M&E plan? To what issues? extent was the M&E system in place Under SO 2 effective and functional?  Was the farmer field   Evaluate the status of the sustainability school approach using or exit strategy implemented by the community facilitators project. Provide concrete effective in promoting recommendations to strengthen these improved agricultural strategies in the remaining life of the technologies to individual project.  farmers? Was provision of agricultural inputs & market linkages effective in promoting farmers to diversify and grow more nutritious crops? Potential Impact   Are there any intended and/or SO1 & SO2: unintended, positive and negative, long  Are there systematic term effects of SNAP? evidences of improved   How did the external conditions, practices or behavior particularly the outbreak of Ebola Virus changes that can be linked Disease affect (positively and negatively) to SNAP approaches for the project framework? example Farmer Field 7 Criteria Main Evaluation Questions for both SO1 and Suggested Sub‐Questions SO2 Specific for each SO   Were there promising practices, Schools, Cluster Groups, approaches, methods, tools used by the Care group models and project that have the potential to VSLAs?  How can the improve effectiveness of food security approaches be more and nutrition projects in the country, in effective? the region, and beyond the region that are worth highlighting?   What are the “institutional legacies” that the project will leave behind? Sustainability   Is there evidence suggesting that the Under SO1 approaches promoted by the project are  As part of its efforts to practiced by the participants? In what ensure sustainability of ways have the beneficiary target groups the Care Group model and individuals adopted the practices SNAP has trained and and technologies promoted by the worked with lead mothers project? Are there outside factors that – to what extent will these influenced the success rate? What are actors be able to carry on key obstacles to continuing approaches with activities after the after the program ends? project ends?   Do beneficiaries believe their households Under SO2 are more resilient as a result of the  Is there evidence that program? More importantly, will they suggests that cluster remain so after the program ends? If not, groups will continue how do they believe they will continue to working together as well be supported? as expand their agricultural activities after SNAP ends?   Will Community Facilitators continue to be a source of knowledge for farmers in their communities? Lessons Learned   What can we learn from SNAP that will inform future FFP Title Programming in the region? Cross‐cutting   Have the project services been  Do women and men Themes environmentally compliant and or benefit equally from the facilitated environmental sustainability? program? To what extent   How effective is the project at reaching has women’s’ participation women? To what extent did SNAP in different activities succeed in the promotion of more improved their leadership equitable access to resources for and decision power? women? 8 4 Evaluation Methodology The evaluation study will focus on changes to behavior at the household level (technology uptake and social behavior change). The study will use both primarily qualitative methodologies including both a review of the progress made in terms of the performance indicators, and use other existing data to triangulate the results. The qualitative data will be collected using focus group discussion, key informant interviews, most significant changes stories, and secondary data review. Focus group discussion and the key informant interviews include major stake holders, partners, and beneficiaries. For gender sensitivity, the evaluation will ensure that there are some gender disaggregated interviews, focus groups, and meetings. Additional sources of information for the evaluation will include review of project data and recommendations made in the mid‐term evaluation. The qualitative review will focus on gathering relevant qualitative data that will facilitate a deeper understanding of processes and approaches, perceptions, and behaviors and other factors that have contributed to achievement and/or non‐ achievement of objectives. The outcome of the quantitative end‐line survey, and significant outcome indicators as reported in the IPTT that have to be considered in this evaluation are identified in Table 1 above. In addition to identified indicators the consultant should refer to the results framework as well as review each indicator in the IPTT and PMP to understand project objectives and to include any relevant related indicators that should be addressed in the evaluation. The Evaluation Team Leader will propose a detailed methodology and evaluation work plan to carry out the framework suggested by this Scope of Work. The proposed evaluation design should be pre‐post design, adequacy design, and will mirror the geographical disaggregation used at baseline. Components to be taken into consideration: 1. Definitions: Definitions of indicators, means of verification, and means of measurement should follow SNAP Performance Indicator Reference Sheets (PIRS). 2. Evaluation Plan: An evaluation plan must be prepared that explains what methodologies will be used and a clear description of how beneficiaries will be selected for participation in the evaluation to gather qualitative data to answer the main evaluation questions for SO1 and SO2 from key program beneficiaries such as lead mothers, farmers, VSLA members, vocational trainees, and participants in numeracy/literacy trainings. The evaluation plan must be approved by USAID before implementation. Methodologies anticipated to answer evaluation questions include but are not limited to:  Focus group discussions  Key informant interviews  Observation for substantiation of adoption of behavior change promoted by the program.  Most significant change stories that demonstrate impact on individuals and communities.  Success stories that emerge from focus groups, interviews, and observations. 9 3. Approval of Tools and Methodologies: The tools and methodologies should be finalized in consultation with SNAP M&E Director, and should be forwarded to USAID/FFP for their concurrence. 4. Field Procedure Manual: It is expected that the contractor will develop a field guide explaining methods and techniques for identification and selection of Group Discussions / Key Informant Interviews, and how the number of Group Discussions and group composition was determined. The evaluation will also include the following:  Review of SNAP Project Documents: First, the external consult team should conduct a review of SNAP project documents to understand the nature of project, objectives, focus, and requirements. The documents to be reviewed include the approved SNAP proposal, quarterly, and annual project result reports, the IPTT and performance monitoring plan, routine commodity and post distribution monitoring (PDM) survey reports, annual survey reports, and other relevant project‐related documents.  Review Midterm Evaluation Report and Baseline: The evaluation team should review the Midterm Evaluation (MTE) of the project and baseline results to assess progress against objectives and draw sampling methodology.  Focus Groups Discussion and Key Informant Interviews Protocols: In each of the selected community, FGDs, collection of most significant change stories, and key informant interviews with women, men, youth, and county/district sector officials and community leaders will be conducted.  Partners Interviews: To complement key informant interview at filed level qualitative inquiry, a sample of partner interviews, including ACDI/VOCA and project implementing partners at Monrovia level will be conducted in order to get primary information from partners. 5 Deliverables The full process of final evaluation will commence on September 1, 2016 by beginning the development of the tools and the work plan for the final evaluation. The consultant firm will prepare: 1) an evaluation work plan (including evaluation methods and tools etc.); and 2) an evaluation report in accordance with standards identified in the Evaluation Scope of Work. Below are the main deliverables of the final evaluation:  An inception report (prior to any field work) including a detailed evaluation methodology, evaluation question matrix, proposed data collection tools and analysis approach, and evaluation work plan (with corresponding timeline). The evaluation plan will include the methodology and tools and the logistics, staffing, and field work plans.  A presentation to the Sierra Leone SNAP staff initial findings (using PowerPoint) for discussion and feedback prior to development of the draft report. The evaluation will also be prepared to present to the host government, USAID / FFP and/or others as requested by ACDI/VOCA prior to leaving Sierra Leone. 10  Evaluation first draft report to ACDI/VOCA for internal review  Evaluation final draft report due to ACDI/VOCA 6 Evaluation Report and Components The report content and structure must comply with current USAID’s Evaluation policy (http://transition.usaid.gov/evaluation/USAIDEvaluationPolicy.pdf), USAID’s Marking and Branding Guidance, and FFP guidance and relevant guidelines. It should follow any applicable final evaluation report guidelines from USAID’s Office of Food for Peace. A summary of USAID’s Evaluation Policy may be found in the Policy’s Annex 1. This includes the requirement that modifications to the scope of work, whether in technical requirements, evaluation questions, evaluation team composition, methodology or timeline need to be agreed upon in writing by the technical officer (in this case, the AOR). 6.1 Report Format The format for the evaluation survey report is as follows:  Cover page, Table of Contents, List of Acronyms.  Executive Summary should be a clear and concise stand‐alone document that states the most salient findings, conclusions, and recommendations of the evaluation study and gives readers the essential contents of the performance evaluation report in two or three pages. The Executive Summary helps readers to build a mental framework for organizing and understanding the detailed information within the report.  Introduction should include purpose, audience, and synopsis of task.  Project background should describe SNAP goal, theory of change, targeting, geographic scope, history and key timeline.  Methodology should describe sampling design, study methods, data collection techniques, constraints and limitations of the study process and rigor, and issues in carrying out the study.  Tabular summary of results should present quantitative evaluation survey and qualitative results in table form for all the indicators by strategic objective and district.  Findings should present findings on all of the key indicators by strategic objectives and evaluation criteria.  Conclusions and Recommendations should provide additional analysis of the data and results, drawing out programmatic and organizational recommendations for future project design. Recommendations must be relevant to project and context.  Annexes should document the study methods, scope of work, schedules, interview lists and tables and be succinct, pertinent, and readable. These annexes must include: i) references; ii) 11 lists of meetings, interviews, focus group discussions (dates, numbers, types); iii) data collection instruments; iv) data tables showing progress against the IPTT indicators (by strata and for the entire project);  Bibliography The final report should be submitted both in hard copy and electronic formats to the donor and lead implementing partner (ACDI/VOCA). The evaluation must also include the notes and transcriptions of all meetings and interviews. 7 Evaluation Team Composition and Qualification The team will be multidisciplinary to ensure that all the key components of the project are evaluated by an expert. To minimize costs and maximize the potential for local institutional capacity building, the evaluation should use a combination of international and/or local experts both of whom will be external to ACDI/VOCA. SNAP will provide all required logistical support to fulfill the required. The team will be composed of: 1) A Team Leader with extensive experience in evaluation of Title II USAID/Food for Peace programs coupled with significant knowledge of food security, nutrition, agriculture, and education. S/he, in coordination with the SNAP team, will be responsible for planning and organizing the overall evaluation. The team leader will lead the qualitative survey team and may also undertake one of the following roles. 2) A Health/Nutrition Specialist with extensive experience in final evaluation and good knowledge of gender. 3) An Agricultural/Livelihood/Value Chain Specialist with rich experience in evaluating Title II USAID programs and deep understanding in environmental compliance and disaster risk management. Table 2: Desired Profile of the Evaluation Team Position Required Qualifications Responsibilities Team Leader   Advanced degree in a relevant academic  Ensue timeliness and quality field of all products/activities   Demonstrated experience leading at least required to complete this three multi‐sectoral, program evaluation performance evaluations, preferably of  Develop and modify the data USAID Title II funded programs in collection tools developing countries, (questionnaire; focus group   At least 10 years’ experience in guides, interview protocol, conducting final evaluations for complex MSC, etc.) , as appropriate, in and multi‐sectoral programs including consultation with the food security/health and nutrition Nutritionist and Agriculture Specialist 12 Position Required Qualifications Responsibilities program being implemented by non‐   Ensure that translations of governmental and private sector actors. forms/documentation are   A solid understanding of the importance accurate of markets to support income generation  Submit a complete standard and stimulate uptake of technologies report using analyzed data   Demonstrated experience leading teams. with outlines and fully   Strong critical analysis and report‐writing written text in a timely skills required. manner   Excellent verbal and written  Submit to SNAP all the communication in English required; documents related to the   Previous experience working in rural study (electronic versions of Sierra Leone or West Africa region will be the collected data, fieldwork preferred. logs, etc.)   • Experience working with USAID FFP assessment tools is preferred. Health/Nutrition   Advanced degree in a relevant academic   Support the development of Specialist field MCHN focus groups guides   At least 10 years’ experience in  and tools Mother/Child Health & Nutrition, Participates in the including experience conducting organization and animation evaluations of MCHN and/or Preventing of FG on the knowledge and Malnutrition under 2 years of Age(PM2A) practices SMI programs  Analyze MCHN performance   Expertise in promoting behavior change data and indicators evolution in maternal/child health;   Experience in collecting and analyzing anthropometric data.   Experience in qualitative evaluation methods and participatory approaches   Previous experience working in rural west Africa Agriculture   Advanced degree in agriculture or related  Provide support in Specialist academic field developing and modification   More than 10 years’ experience in of the data collection tools (; agriculture/food security and value chain focus group guides, interview programs, including experience protocol, etc.), conducting program performance  Support team leader to evaluations, preferably of USAID Title II analyze evaluation data, and funded programs in developing countries, the factors which have   Expertise in promoting value chain influenced the data quality activities in rural areas;  and report writing Analyze food security and livelihood data 13 Position Required Qualifications Responsibilities   •Experience in livelihood for food aid   Provided technical analysis program implementation (specifically for on agricultural value chains USAID/FFP Title II) performance   Experience in collecting and analyzing  Support farmers focus group performance evaluation data. discussions   Previous experience working in rural areas of Sierra Leone or West Africa 8 Timeframe The final evaluation will be commenced in September 1, 2016 and the final evaluation report should be made available by November 30, 2016. Upon returning to Freetown, the evaluator will prepare a presentation of preliminary findings to be presented to the project’s senior management team and selected project partners before departing the country. Summary of schedule for evaluation process is described in the table below. Table 3: Timeline Activity/Tasks Payment Due date 1) An inception report (prior to any field work) that includes a detailed methodology to carry out the study, a summary of findings from the literature review of key program documents, key questions for interviews for each IR and Sub IR, data collection tools and instruments, a data analysis plan, and detailed implementation plan that includes, training of data collector, a logistics and data 20 Percent of total contract Due September 30, 2016 analysis plan. award 2) Plan for incorporating previously collected quantitative data and methodology for cross‐linking with qualitative analysis to be performed. The methodology should include a conceptual framework outlining the evaluation questions under each IR and Sub IR. The inception report will be delivered electronically and by a presentation to the SNAP team. 14 3) Evaluation first draft report to ACDI/VOCA for internal. 4) Evaluation second draft report to ACDI/VOCA for USAID. 5) Final draft of PowerPoint 30 Percent of total contract presentation of key final evaluation Due November 11, 2016 findings to be presented to USAID, award the host government, USAID/FFP and or others requested by ACDI/VOCA, submitted to and reviewed by ACDI/VOCA and submitted to USAID for review. 6) Final PowerPoint presentation and 50 Percent of total contract key findings presented to USAID and Due November 30, 2016 ACDI/VOCA. award 7) Acceptance of evaluation final report by ACDI/VOCA. 15 ACDI/VOCA Sierra Leone Sustainable Nutrition and Agricultural Promotion (SNAP) Project Final Evaluation Plan (Final Draft) 14 October 2016 SNAP Evaluation Plan (PRE-FIELDWORK DRAFT) 7 October 2016 2 INTRODUCTION ACDI/VOCA has been implementing a Multi-Year Assistance Program (MYAP) entitled the Sustainable Nutrition and Agricultural Promotion (SNAP) Project in Sierra Leone funded by USAID's Office of Food for Peace (FFP). The project is being implemented in 18 chiefdoms in the four districts of Tonkolili, Koinadugu, Bombali and Kailahun by ACDI/VOCA, Opportunities Industrialization Centers International (OICI), and International Medical Corps (IMC). The Project Goal is to reduce food insecurity and increase resiliency among vulnerable populations in Bombali, Kailahun, Koinadugu and Tonkolili through achievement of five Intermediate Results (IRs) under two Strategic Objectives (SOs) as shown in Table 11. The project was designed to have impact on over 405,000 direct beneficiaries with a life span, initially, of five years from June 2010 to June 2015, later extended to December 2016. Table 1. SNAP Goal, Strategic Objectives and Intermediate Results DEVELOPMENT OBJECTIVE: Reduce food insecurity and increase resiliency among vulnerable rural populations in Bombali, Kailahun, Koinadugu and Tonkolili. SO1: Reduce chronic malnutrition among children under five SO2: Enhance livelihoods for vulnerable people, especially women and youth IR 1.1 Improved access to and use of quality nutrition and health services for women and children 0-59 months IR 2.1 Increased agricultural productivity at the household level IR 1.2 Improved health and nutrition practices at the household level IR 2.2 Increased access to financial and marketing resources IR 2.3 Improved community assets This document describes the plan for implementing the qualitative portion of the final evaluation, including specification of the evaluation purpose, a description of the information to be obtained, an overview of the evaluation schedule with details on key steps in the process, a description of the sampling methodology, the evaluation team composition and assignment of responsibilities to team members, and a description of the planned logistics for the evaluation.                                                              1 The original project proposal described seven IRs under the same two SOs. SNAP Evaluation Plan (PRE-FIELDWORK DRAFT) 7 October 2016 3 A. EVALUATION PURPOSE The Final Evaluation of the SNAP Project is a summative evaluation intended to assess the performance of the project toward achieving the project goal, strategic objectives (SOs), and intermediate results (IRs). The evaluation will describe the outputs that have been produced under each IR, explain the outcomes and impact that has been achieved as a result, and will identify good practices and lessons learned relevant for future programming. In addition to focusing on IRs and SOs, the evaluation will also analyze the performance of the project on five cross-cutting themes, including resiliency to shocks, productive youth, gender equity, environmental stewardship and good governance. The good practices and lessons learned to be identified by the evaluation will include not only those associated with project interventions but also good practices and lessons learned associated with project implementation systems, including project management, partnership, knowledge management, project integration and complementarity with other programs, and resource management, including financial, commodity, human and material resources. The evaluation process will assess the relevance of project interventions, the efficiency of the project in delivering interventions, the effectiveness in terms of achieving both intended and unintended impact, and the likely sustainability of the impact that has been achieved. The findings of the evaluation will be documented in a report following the format provided in Annex A. B. INFORMATION TO BE OBTAINED The evaluation will investigate the following features of the SNAP Project. 1. Project History & Operating Context. The evaluation will document the history of the project, particularly how it has evolved since inception, how it has adapted to a number of significant challenges including the emergence of Ebola virus disease (EVD) and other critical features of the operating environment that have affected, positively or negatively, project implementation. Information for understanding the history and contextual factors affecting implementation will be obtained from project monitoring reports as well as through interviews with project implementation staff and representatives of implementing partner leadership. 2. Project Outputs. The SNAP Project currently has five Intermediate Results (IRs). The evaluation team will summarize the outputs produced under each of these IRs, compare this performance with life-of-project targets on outputs, and assess the effectiveness and efficiency of the processes used to produce the outputs. Lessons learned will be extracted on what has worked well and what may not have worked well in producing project outputs. Those activities and outputs that have been identified as producing SNAP Evaluation Plan (PRE-FIELDWORK DRAFT) 7 October 2016 4 significant impact will be highlighted as good practices. Information on project outputs will be obtained from quarterly monitoring reports, project databases, other ad hoc project reports, reviews of key secondary data, interviews/focus group discussions with project participants, and interviews with project implementation staff. The following outputs under each Intermediate Result will be the primary focus of the evaluation. Discussions will be held with the project implementation team in Sierra Leone early in the evaluation process to ensure that these cover all major outputs being produced by the project, and additional outputs may be added following these discussions. Under SO 1 Maternal and Child Health and Nutrition (targeting 50,000 mother￾child units, 400,512 direct beneficiaries)  IR 1.1: Improved access to and use of of quality nutrition and health services  Training of and material support for District Health Management Teams (DHMT) on prenatal and postnatal care services for pregnant and lactating women  Training of Ministry of Health and Sanitation (MOHS) district health staff and health promoters on Integrated Management of Childhood Illnesses (IMCI), Reproductive Health and Family Planning.  Capacity building of Water Sanitation and Hygiene (WASH) Committees and improvement of Peripheral Health Unit (PHU) hygiene and sanitation conditions with bio-sand filters, hand washing stations, incinerators and latrines construction/rehabilitation  Training of and material support/drugs for community members in detection and referral of Severe Acute Malnutrition (SAM) children under five  Support health services quality improvement within PHUs through training and Gifts In-Kind (GIK)  Health services rehabilitation and equipment  Education of target communities for proper health seeking behaviors  Community sensitization on EVD prevention  IR 1.2: Improved health and nutrition practices at the household level  Formation of mother-care groups  Training of lead mothers  Formalization of men as partners to compliment and support mother care groups among male cohorts  Creation of a food recipe manual  Cooking demonstrations  Training caregivers on preparation of nutrient rich and diverse foods  Distribution of nutritious food to PLW and CU2 part of a PM2A strategy  Training of lead mothers on vegetable gardening for increased food diversity (considered part of IR 2.1 in Annual Results Reports) SNAP Evaluation Plan (PRE-FIELDWORK DRAFT) 7 October 2016 5  Other Activities Proposed or Initiated that were Suspended or Not Undertaken Under SO1  Early childhood development research Under SO 2 Enhanced livelihoods for vulnerable people, especially women and youth. (45,375 direct beneficiaries)  IR 2.1: Increased agricultural productivity at the household level  Formation of Farmer Field Schools (FFS) and FFS clusters  Training of FFS Clusters on business planning and management  Training on farming as a family business (FaaFB)  Promotion of crop diversification for added nutritional value  Training on Good Agricultural Practices (GAP), integrated pest management, and crop preservation and processing  Distribution of post-harvest kits (bags, tarps and scales) to FFS clusters  Training of input supplier groups in basic business management practices & handling  IR 2.2: Increased access to financial and marketing services  Completion of value chains analyses on crops for family food  Training of Farmer Based Organizations on business development and marketing skills  Cultivation of linkages between FFS clusters and Agricultural Business Centers and private sector buyers  Creation, expansion & training of Village Savings and Loan Associations (VSLA)  Training of women and youth on literacy and numeracy  Training for youth on vocational skills, agri-business and entrepreneurship  Training for input supplier groups on management practices  Provision of seeds and tools procured from input supply groups  IR 2.3: Improved community assets  Repair of infrastructure, including drying floors, markets and feeder roads  Expansion of community crop storage facilities  Other SO2 Activities Proposed or Initiated that were Suspended/Not Undertaken  Value Chain Stakeholder Workshops  Engagement of youth through vouchers in labor for learning  Capacity building of village-based traders and mid-level traders/middlemen  Development of linkages to LLCs and Chambers of Agriculture established by the Promoting Agriculture, Agriculture and Governance (PAGE) Project  Family planning and reproductive health messaging to FFS and VSLA groups  Training and grants for agri-business development  Linkages of VSLAs to MFIs  Establish food processing facilities (fixed rice mills, cassava graters, and mobile rice mills) SNAP Evaluation Plan (PRE-FIELDWORK DRAFT) 7 October 2016 6  Test technologies for drying fruits and vegetables 3. Project Targeting. The evaluation team will obtain information on the impact groups that have been affected by the SNAP Project. These include not only targeted beneficiaries, but others who may have been affected positively or negatively by project interventions. The evaluation will also examine the targeting processes for selecting communities and individuals for participation. Project participants will benefit to differing degrees depending on whether they are participating in a single intervention or multiple interventions, and the evaluation team will obtain information to be able to describe what percentage of participants are benefitting substantially from multiple interventions. Information on targeting will be obtained from analysis of the project's databases, interviews/focus group discussions with project participants, and interviews with project implementation staff. 4. Project Outcomes and Impact. The Final Evaluation Report will document the outcomes and impact that have been achieved by the SNAP Project. Qualitative information on outcomes and impact will be obtained through key informant interviews and focus group discussions with project participants and implementers. This information will be collated with information from a quantitative endline survey being implemented separately. For each of the two Strategic Objectives, the collective outcomes of the IRs implemented under the SO will be assessed. Then, the evaluation team will assess the collective impact of the two SOs on the Final Goal for the project. The evaluation will investigate the expected impact as per the project document, but will also be observant for new areas of unexpected impact, both positive and negative, that have occurred as a result of project interventions. The evaluation team will assess the likely sustainability of the impact that has been observed as well as the potential replicability of the project's interventions in achieving impact with other populations. The focus of the sustainability analysis will be on the institutions, services and relationships that have been established or strengthened by the project that are necessary to sustain the impact achieved by the project. Lessons learned associated with project logic or other dimensions of the impact that has been identified will be also extracted. Information on outcomes and impact will be obtained through reviews of project monitoring reports, reviews of key secondary data including Annual Results Surveys, interviews/focus group discussions with project participants, interviews with project implementation staff and correlation with findings from the quantitative Final Evaluation survey. 5. Clarification of Preliminary Quantitative Survey Findings. Analysis of the data collected in the quantitative survey has highlighted a number of findings that are either (a) significantly higher or lower than expected or (b) are inconsistent. The qualitative SNAP Evaluation Plan (PRE-FIELDWORK DRAFT) 7 October 2016 7 data collection will be used to obtain additional information to verify the first and clarify the second. Listed below are some of the major quantitative findings that will be further investigated during the qualitative interviews and focus group discussions. Additional topics are likely to be identified as the evaluation team studies the endline survey report.  The diet diversity score calculated from the endline data is smaller at the endline than at the baseline, but the two data sets were collected at different times, and the endline data was collected in the middle of the lean season. The qualitative survey will obtain information to determine how seasonality may have affected this as well as estimates of other indicators.  Inconsistencies were reported between the question in the endline survey that asked whether the household was producing from a set of crops (response very high) and whether the household had a farm in the last five years (relatively low). The qualitative information gathering will hold discussions around farming in the last few years to identify whether there are reasons to explain why respondents would say that they have not had farms.  There are also some inconsistencies being reported in the endline survey between production/yields and crop sales. The qualitative FGDs will be used to identify crop sales patterns and area measurement units (used to estimate yields) to help explain this.  The endline survey measures benefits of being a group member in terms of access to inputs, production and markets. There is no indicator that captures the social capital benefits, so the qualitative information gathering will give added attention to identifying these. The results of these investigations will be reported in the previous section describing outcomes and impact when information from both the quantitative survey and the qualitative information gathering are integrated to describe outcomes and impact. 6. Project Theory of Change (ToC) and Logic Structure. The evaluation team will examine the theory of change which underlies the strategy of the SNAP Project and the Results Framework that is being used to guide implementation of the project. Using evidence from field investigations, the evaluation will determine the relevance of the project's interventions and identify strengths and weaknesses in the ToC to inform the design of future projects. 6. Cross-Cutting Themes. The SNAP Project has committed itself to having positive impact on five cross-cutting themes. These include increasing resilience to shocks, facilitating increased productivity of youth, promoting gender equity, maintaining environmental stewardship and promoting good governance. SNAP Evaluation Plan (PRE-FIELDWORK DRAFT) 7 October 2016 8  Resilience - The evaluation will investigate how resilience capacities at both the household and community levels have changed as a result of project activities. This will involve examining how assets have changed, including not just physical, financial and natural assets but also human, social capital and political capital assets. The evaluation will assess how well community and household assets have enabled households to anticipate, absorb and adapt to livelihoods shocks. The experience of actual livelihood shocks and stress that have occurred over the life of the project, such as the outbreak of EVD, will be analyzed for evidence of increased resilience capacities. Information will be obtained from interviews/focus group discussions with project participants and interviews with project implementation staff.  Productivity of Youth - As part of the targeting investigations, the extent of the participation of youth in various interventions will be identified, and the impact on their productivity as a result will be analyzed. Specific focus group discussions will be organized with groups of these targeted youth to discuss changes in their productivity, specifically in ways that will enable them to engage in sustainable livelihoods.  Gender Equity - The evaluation team will obtain information on the impact of the project on women's empowerment, women's workload, men and women's' roles in the household and community, and other changes in gender norms through interviews/focus group discussions disaggregated by sex with project participants and interviews with project implementation staff.  Environmental Stewardship - The systems used by SNAP to monitor environmental impact will be analyzed and good practices and lessons learned will be identified. o Good Governance - The evaluation will examine the extent to which various groups that have been organized or strengthened by the project are practicing good governance principles, including representation, transparency, participation and accountability. In addition, the project will examine how well project implementers themselves are practicing principles of good governance consistent with current concepts of social accountability. Information to assess governance practices will be obtained from review of project implementation reports, observance in the field, interviews/focus group discussions with representatives of various groups and interviews with project implementation staff. 7. Implementation Systems. The systems used to implement the project will be investigated to assess their effectiveness in facilitating the achievement of impact or efficiency in using project resources. The specific systems that will be examined include project management, partnership relations, knowledge management including SNAP Evaluation Plan (PRE-FIELDWORK DRAFT) 7 October 2016 9 monitoring and evaluation and how information has been used for adaptive management, project integration internally across components and well as integration externally with relevant programs being implemented by other organizations, and resource management, including systems for managing financial, commodity, and human resources. Across all implementation systems, the evaluation team will identify good practices and lessons learned. Information on implementation systems will be obtained from monitoring reports and key informant interviews/focus group discussions with project implementation staff at different levels from management and technical staff through front-line implementation staff, directors and other key staff from each of the partner organizations, and senior staff from the ACDI/VOCA country office. C. EVALUATION KEY QUESTIONS Table 2 summarizes the fundamental evaluation questions that will be addressed by the SNAP Final Evaluation relative to the key criteria of relevance, effectiveness, efficiency and sustainability. These questions are further refined in the specific topical outlines that have been developed for the evaluation that are included in Annex C. SNAP Evaluation Plan (PRE-FIELDWORK DRAFT) 7 October 2016 10 Table 2. Fundamental Evaluation Questions for the SNAP Final Evaluation Fundamental Evaluation Questions Sub-Questions For SO1 Sub-Questions For SO1 Relevance How important are the interventions in the SNAP Project for the targeted impact groups and to what extent do they address their needs and interests? How do the planned activities compare to the food security problems identified in the program results framework? To what extent do the SNAP Project interventions correspond to the most recent objectives of the GoSL and USAID? What behavior change promoted by the program has had the greatest impact on affecting change in nutritional status of pregnant and lactating women (PLW) with children under two? What were the most positive intervention(s) that impacted on the behavior change promoted by the program? What technologies are mostly adopted by farmers? What factors had the most positive influence on the adoption rate? How well did SNAP integrate gender aspects in its SO2 programming to favor women’s access to agriculture resources and benefits? Effectiveness To what extent was SNAP effective in achieving its strategic objectives and intermediate results? How has the strategy of the SNAP Project adapted to contextual factors in order to enhance or maintain effectiveness? How effective were the partnerships developed with GoSL at national, district, and Chiefdom level as well as other development actors in delivering intended results? In addition to the planned effects or outcomes of the project, what other effects, positive or negative, were achieved? How effective was conditional food aid in promoting behavior change? How has the discontinuation of food aid impacted on participation in care groups and support to pregnant women in the communities? Did the project maintain minimum standards beneficiaries meet expected standards? If not, what were the issues? Was the farmer field school approach using community facilitators effective in promoting improved agricultural technologies to individual farmers? Was provision of agricultural inputs & market linkages effective in promoting farmers to diversify and grow more nutritious crops Efficiency How were activities delivered to produce the desired results, relative to timeliness and the project implementation plans? How efficient has the mother-care group structure been for building local capacities to be able to How efficient has the farmer field school cluster structure been for for increasing farm productivity and returns to farming SNAP Evaluation Plan (PRE-FIELDWORK DRAFT) 7 October 2016 11 How effective was the project management structure and the management of the project in providing vision, leadership, timely decision-making and problem-solving? What was the relationship between partners and how did this impact on achieving project objectives, especially capitalizing on experience and expertise of the different partners? How do project costs compare to project impact? How efficient were financial management systems in ensuring adequate cash flow for implementation and timely reporting? How efficient were commodity management systems in delivering food to project beneficiaries? understand and use the essential nutrition and essential hygiene actions? for targeted beneficiaries? How have the FFS cluster and VSLAs been used in terms of providing platforms for other activities, including behavioral change communications associated with SO1? Sustainability Which interventions are recognized by participants as having the greatest impact? What organizations, services or relationships are required to sustain the impact being achieved by the SNAP project? What are the threats to these organizations, services and relationships that will constrain them from being maintained and how did the SNAP project address these? For materials and commodities distributed by the project, what exit plans have been implemented to ensure that the impact generated by these distributions is sustained after the project ends? How do beneficiaries expect to interact with and be supported by non-project intermediaries (e.g., community groups, government departments, private sector entities, etc.) after the project ends? As part of its efforts to ensure sustainability of the Care Group model SNAP has trained and worked with lead mothers – to what extent will these actors be able to carry on with activities after the project ends? How will the material support provided by the project to PHU and DHMTs be sustained after the project ends? How are the services provided by these departments likely to change after the project ends? What evidence is there to suggest that cluster groups will continue working together as well as expand their agricultural activities after SNAP ends? How will the work of Community Facilitators change after the project ends? What likelihood is there that they will continue to be a source of knowledge for farmers in their communities? How will material support (e.g., seed and tools) provided by the project to farmers be replaced after the project ends? How likely are the marketing relationships (input supply and product) to be maintained after the project ends? SNAP Evaluation Plan (PRE-FIELDWORK DRAFT) 7 October 2016 12 D. IMPLEMENTATION SCHEDULE The full implementation schedule for the evaluation and the specific schedule for the field work are shown in Annex B. Field work in Sierra Leone will commence on October 10 and is expected to be completed by November 4. More detail on some of the key events in the process is provided below. 1. Initial Team Meeting, October 10. The initial meeting of the evaluation team will occur immediately after team members have assembled in Sierra Leone. In this meeting, team members will be introduced to each other, and the evaluation team leader will provide an overview of the evaluation process and answer any questions that team members may have. 2. Project Orientation Meeting, October 11. In this meeting, SNAP management staff based in Freetown will present an orientation to the project for the evaluation team. The purpose of this meeting is to obtain clarity on the types of outputs produced by the project, the stakeholders that need to be interviewed to understand the impact of these outputs, and additional data sources for information to support the evaluation. Discussions will also be held around the site selection for the qualitative interviews and the logistics plans. Attendees to this meeting include the SNAP management and technical team, selected representatives from implementing partners, and anyone else from the project likely to be involved in implementing or supporting the final evaluation. Following this meeting, the evaluation operational plan will be updated and circulated to stakeholders. 3. Meetings in Freetown with Stakeholders, October 12 to 14. During this phase of the evaluation, three types of meetings will be organized with various stakeholders in the SNAP Project. The first of these meetings will be with USAID/FFP at which the evaluation team will be introduced, an overview of the evaluation process will be provided by the team leader and the evaluation team will answer any questions that USAID/FFP may have about the evaluation. In addition, the evacuation team will seek to understand the expectations of USAID and any specific interests that they would like to see covered by the evaluation. The second type of meeting is with representatives of the SNAP implementing partners who are based in Freetown. The purpose of these meetings is also to provide an overview of the evaluation process, answer any questions that consortium representatives may have, discuss their expectations for the evaluation and to begin obtaining information on project performance. These meetings will be attended by the evaluation team leader and members of the team who are responsible for the investigations associated with the particular partner being visited. The third type of meeting is with representatives of government ministries, short-term technical partners, or businesses in Freetown that are familiar with or have been SNAP Evaluation Plan (PRE-FIELDWORK DRAFT) 7 October 2016 13 engaged in the project. These meetings are attended by the members of the evaluation team who are responsible for the investigations associated with the particular organization being visited. 4. Information Gathering, October 15 through 27 (13 Days). Key informant interviews, focus group discussions and direct observations to obtain qualitative data will be undertaken by the evaluation team over the period from October 15 through 27. A provisional schedule for this field work is provided in Annex B. The section below on sampling describes the number of sites that will be visited and the different types of participants that will be interviewed during this period. 5. Data Processing and Preparation for the Verification Workshop, October 28 through 31. During the field work, evaluation team members will process data as it is obtained and the period from October 28 to 31 will be used for the team to continue analyzing information to identify major findings that will be presented for discussion at the Verification Workshop. 6. Verification Workshop, November 1 & 2. During and immediately following the field work, the evaluation team will have started formulating preliminary observations on outputs produced, the outcomes and impact being achieved, and the effectiveness and efficiency of project implementation systems. The team will have also started identifying key lessons learned and good practices. All of these will be shared in the Verification Workshop with SNAP implementation staff and discussed further to ensure that they reflect reality and are described appropriately. An illustrative Workshop Plan for the Verification Workshop is attached in Annex D. Three major outputs are targeted for the workshop. These are (1) agreement on key observations assembled by the evaluation on the outputs, outcomes and impact of the program, cross-cutting themes and implementation systems, (2) identification of lessons learned and good practices, and (3) identification of topics that need further investigation by the evaluation team after the workshop. Participants in the workshop will be SNAP implementation staff and the evaluation team. This is an in-house event, only for those participants who are fully engaged in the project. The purpose of this event is to be able to hold frank discussions between the project and the evaluation team, as one step toward finalizing the product produced by the evaluation. This is not an event at which the product is presented to a wider group of stakeholders. 7. Follow-Up Information Gathering & Preparation for Freetown Debriefing, November 2 & 3. The Verification Workshop will identify areas in which additional information gathering is necessary to achieve full clarity. Beginning on the afternoon of the November 2 and through November 3, evaluation team members will obtain information through further interviews to address these topics, as well as prepare for the USAID debriefing, as described below. SNAP Evaluation Plan (PRE-FIELDWORK DRAFT) 7 October 2016 14 8. USAID Debriefing, November 4 (Two hours). A summary of the final evaluation observations and findings will be presented to USAID. In addition to the full evaluation team, participants in this meeting will be the ACDI-VOCA Country Representative, SNAP Chief-of-Party, and representatives from USAID. 9. Draft Report Finalization (Target Date: November 30). The target date for completion of the draft report is November 30. Writing assignments for evaluation team members and page targets will be finalized in the first week of the evaluation with the finalization of this Evaluation Plan. 10. Final Report Completion (December 15). The draft report will be circulated to ACDI￾VOCA and comments are expected back on the draft by December 10. The target date for completing the final report is December 15. Annex A contains an illustrative format for the report. E. SAMPLING APPROACH 1. Site Selection. The SNAP Project is working in 18 chiefdoms in the four districts with a total of around 2,000 communities (villages or towns)2. Table 3 below summarizes the number of villages which have residents who have participated in various components of the project as indicated. The evaluation period has allocated a total of 13 days for information gathering in all four districts. Taking into account travel time and allowing time for staff interviews and data processing for evaluation team members, it is expected that the team will be able to visit a target of 27 villages in this period, or one to two villages per day, depending on travel distance and the number of interviews/FGDs/work site visits scheduled. The number of villages selected for each district are 5 in Bombali, 10 in Tonkolili, 6 in Koinadugu, and 6 in Kailahun. The team will be working together for data collection in Bombali, taking turns facilitating discussions and interviews to observe each other, consequently a slower process reaching relatively fewer villages. In Tonkolili, the four￾person evaluation team will divide into two teams of two for data collection in this district, consequently a relatively larger number of villages will be visited. The teams will then separate, with one team of two going to Kailahun and the other team of two going to Koinadugu. The full list of villages is attached in Annex E. Table 3. Disaggregation of Communities by Intervention and Integration of SOs Interventions Number of Communities Bombal Koinadugu Tonkolili Kailahun Total                                                              2 The SNAP Program has multiple lists of villages who participated in various activities, with those lists for SO1 managed by IMC and those lists for SO2 managed by ACDI/VOCA.  For the purposes of the final evaluation, the ACDI/VOCA M&E unit consolidated the two lists into one to be able identify the number of villages in each of the three categories required by the evaluation, i.e.,  Villages with SO1 Only, with  SO2 Only, and with Both SOs.        SNAP Evaluation Plan (PRE-FIELDWORK DRAFT) 7 October 2016 15 i Participating Chiefdoms 3 7 3 5 18 Total Participating Villages 333 634 706 374 2,047 SO1 Mother-Care Groups 231 514 684 315 2074 SO1 PHU Support, GMP & Food Distribution 22 45 33 23 123 SO1 Vegetable Production 0 33 0 20 53 SO2 Farmer Field Schools 140 280 86 125 631 SO2 VSLA 29 51 31 23 134 SO2 Literacy & Numeracy Training 5 12 4 8 29 SO3 Youth Vocational Training 17 5 5 9 36 SO2 Input Supplier Training 3 3 1 3 10 SO2 Infrastructure Investment 7 5 9 5 26 SO2 Crop Storage Investments (Drying Floors) 22 35 18 35 110 Communities with only SO1 Interventions 157 340 606 224 1,327 Communities with only SO2 Interventions 104 122 43 61 330 Communities with Interventions for both SOs 72 172 57 89 390 Given the relatively small sample of villages that will be visited, no more than 27 out of a total of 2074, the selection of villages was done strategically to ensure that the evaluation team sees at least some examples of of each of the major outputs produced by the project. The evaluation originally intended to select villages using two criteria, (a) the intensity of intervention and (b) accessibility3. For the intensity of intervention, communities that participated in the SNAP Project were stratified into those in which interventions from only SO1, only SO2, or Both SOs were implemented. For accessibility, given the time available and the absence of an existing village database with accessibility information, it was not possible to categorize all participating SNAP villages according to an accessibility characteristic, such as distance from a market center or chiefdom headquarters. To capture at least some dimension of accessibility, the selection of villages for the SNAP Final Evaluation was done in collaboration with SNAP field staff (District Operations Managers for each district) to get a mix of accessible and remote villages. Because of the time available for the evaluation, villages for which it took more than thirty minutes to reach by either motorbike or on foot were eliminated                                                              3 Accessibility, or distance from a market center, for example can be used as a proxy for a number of important factors, including village size, distance from a main market, coverage of other development programs, access to health services, proximity to government offices, and value chain infrastructure.    SNAP Evaluation Plan (PRE-FIELDWORK DRAFT) 7 October 2016 16 from consideration. Table 3 shows the disaggregation of participating communities into six strata. Normally, a comparison would be provided of the percentage of the sample in each stratum to the percentage of all villages in each stratum, but the information was not available to be able to indicate the latter. Table 4. Disaggregation of Selected Villages for the SNAP Final Evaluation by Intensity of Intervention and Accessibility Intervention Intensity Accessibility Reasonably Accessible Somewhat Difficult to Reach Only SO1 (13) 8 5 Only SO2 (4) 2 2 Both SOs (10) 6 4 3. Participant Selection for Focus Group Discussions. Focus group discussions will be held with groups that have been formed by the project, including Mother-Care Groups, Farmer Field Schools, Farmer Field School Clusters and Village Savings and Loan Associations. The selection of these groups will be dependent on the selection of villages in the sample. In addition to these focus group discussions, additional groups will be organized that are less formalized. These include, for example, food recipients, mothers with children identified with SAM, recipients of post-harvest kits, youth beneficiaries from vocational training activities, and others. The evaluation team, working with SNAP implementers, will ensure that at least some villages are selected in the sample that have these groups. Village leadership will be asked to assemble a representative sample of these participants for the focus group discussions. Focus groups will be limited to six to ten participants. 4. Community-Level Key Informant Interviews. In addition to having focus group discussions with groups of similar participants, there will also be interviews with individual key informants. These include lead mothers, community agriculture facilitators, private service providers in the VSLA component, input suppliers who have been engaged by the project, Block Extension Supervisors and Frontline Extension Workers from the Ministry of Agriculture, and others who have participated in the project. The selection of villages will determine which of these participants will be interviewed in specific sites. F. Data Collection Tools SNAP Evaluation Plan (PRE-FIELDWORK DRAFT) 7 October 2016 17 Topical outlines will be used to guide interviews and focus group discussions for qualitative information gathering. Annex C contains the tools that will be used for information gathering. The questions in any topical outline are fairly general and used to stimulate discussion. In addition, the list of questions in some topical outlines is quite long. Interviews and focus group discussions will generally be limited to no longer than two hours in duration. Evaluators will keep in mind at all times that the purpose of the information gathering is to understand the impact that has been achieved by the project and to extract lessons learned and good practices that should be highlighted by the evaluation. The discussion facilitator will be free to explore in more depth any interesting topics that may come up during each discussion, related to these objectives. Before beginning the discussion, an introduction and explanation of the purpose of the evaluation will be provided with stress put on the importance of obtaining useful information that reflects reality. G. Data Analysis The purpose of analysis in the SNAP final evaluation is to identify truth through triangulation and identification of patterns of response from multiple sources. Triangulation will occur when a piece of information is verified from alternative sources. For example, interviews with a VSLA group may indicate that certain financial benefits have accrued as a result of participation in the project. The project's VSLA database and the findings from the quantitative final evaluation survey will both be examined to determine whether they support the information obtained in the interviews. Patterns of response will emerge when similar responses are heard from different respondents. If one VSLA group, for example, reports that the greatest benefit from the project has been a particular type of intervention, and similar responses are heard from other VSLA groups or other project participant, then there is likely to be significant truth to the response. The information gathered by the evaluation team will be analyzed at multiple points during the evaluation process. As the evaluators are holding interviews and focus group discussions, they will probe and explore topics in more depth with respondents to ensure clear understanding. This represents the first level of analysis. A second level of analysis occurs when evaluators cross-reference responses from interviews and discussions with existing data bases to verify truth. In addition, the evaluator will query project implementation staff as a third source of information. The result of this level of analysis is that each evaluator will formulate his/her own key observations relative to the assigned topics. A third level of analyses occurs immediately before the Verification Workshop when the evaluation team shares with each other the presentations planned for the Verification SNAP Evaluation Plan (PRE-FIELDWORK DRAFT) 7 October 2016 18 Workshop. This allows other team members to present relevant information to be included in the analysis. The evaluator who is interviewing mother-care groups, for example, may have heard useful information about VSLA activities that should be considered by the evaluator responsible for the VSLA investigations Finally, a fourth level of analysis occurs in the Verification Workshop in which preliminary observations and findings are presented to project implementation staff. If project implementation staff seriously question a particular finding that has been presented by the evaluation team, the opportunity exists in the workshop to further discuss and analyze the finding to reach the truth. Table 5 illustrates the different levels of analysis. Table 5. Levels of Qualitative Data Analysis Analysis Level Data Source Analysis Purpose Level 1 Key Informant Interviews, Focus Group Discussions and Personal Observation Primary qualitative data analyzed to look for patterns across different sources of information Level 2 Secondary data sources, additional interviews and discussions Cross-referencing to secondary sources and continuing to look for patterns Level 3 Information obtained by other team members Cross-referencing with observations and findings from other team members Level 4 Verification workshop to draw out knowledge from project implementers Ground-truthing of patterns and other findings H. Evaluation Team & Field Data Collection The final evaluation of the SNAP Project will be a team effort undertaken by four external evaluators. Each will be responsible for coordinating certain parts of the evaluation based on her/his technical expertise, but all will be expected to contribute observations and insights to all parts of the evaluation. Two team members, Anna Afferri and Mabinty Kamara will work together on investigations for SO1, with Anna having lead responsibility for SO1. The two other evaluation members, Mike DeVries and Braima James will work together on investigations for SO2, with Mike having lead responsibility for SO2. When data collection begins in the field, the full team will work together in Bombali District to gather information so that they can support each other in implementing and refining the data collection processes. Then, the evaluation team will divide into pairs, with one member responsible for SO1 and the other member SNAP Evaluation Plan (PRE-FIELDWORK DRAFT) 7 October 2016 19 responsible for SO2, to collect data in separate locations in Tonkolili District. The team will still be residing in the same location (Makeni) over this period and will share experiences after returning from the field to further refine the data collection processes and begin analyzing data. After completing data collection in this second district, one pair of evaluators will travel to Koinadugu to collect data there and the other pair of evaluators will travel to Kailahun to collect data there. The full evaluation team will reassemble in Makeni to process information and prepare for the Verification Workshop, which will also be held in Makeni since it represents a central location for the project. The Evaluation and Field Work Schedules provided in Annex B provide more detail. The evaluation team is composed of the following persons: 1. Mike DeVries (Team Leader) – Mike will be responsible for coordinating overall implementation of the evaluation, coordinating investigations under IR 2.2 (Finance & Marketing), project targeting, and the cross-cutting theme of resilience, as well as investigations on program management and finance systems. He will also be responsible for facilitation of meetings and workshops, preparation of the USAID debriefing, and completion of the final report. He will be in-country from October 9 through November 4. 2. Anna Afferri – Anna will be responsible for coordinating investigations around IR 1.2 (Behavioral Change) and the cross-cutting theme of gender equity, as well as partnership implementation systems, program integration and commodity management systems. She will be in-country from October 9 through November 4. 3. Mabinty Kamara – Mabinty will be responsible for coordinating investigations around IR 1.1 (health and Nutrition Services) and the cross-cutting theme of youth productivity as well as the program human resource management systems. 4. Braima James – Braima will be responsible for coordinating investigations around IR 2.1 (Agricultural Productivity), IR 2.3 (Community Assets) and the cross-cutting themes of environmental stewardship and governance. He will also coordinate investigations around the program's knowledge management systems. 5. Interpreters – To support the information gathering, the evaluation team has contracted Dalan Consultants to provide local language translators for the villages selected for field visits. . Table 6 shows the evaluation team member assignments as they relate to the report that the team intends to produce (see Annex A for a format). The target page length for the report is 40 pages, excluding annexes. Table 6. Evaluation Team Member Assignments INVESTIGATION TOPIC/REPORT SECTION Report Lead Team SNAP Evaluation Plan (PRE-FIELDWORK DRAFT) 7 October 2016 20 Page Target Member I. EXECUTIVE SUMMARY 3 Pages I. BACKGROUND (3 Pages) A. Overview of SNAP Theory of Change & Project Strategy B. Project History and Operating Context 3 Pages C. Evaluation Methodology III. OUTPUTS UNDER INTERMEDIATE RESULTS A. IR 1.1 Access to Improved Quality Health and Nutrition Services 12 Pages B. IR 1.2 Improved Health and Nutrition Practices C. IR 2.1 Agricultural Productivity D. IR 2.2 Access to Financial and Marketing Resources E. IR 2.3 Improved Community Assets IV. PROJECT TARGETING 2 Pages V. PROJECT OUTCOMES & IMPACT A. Outcomes Under SO1 and its IRs 15 Pages B. Outcomes Under SO2 and its IRs C. Impact at the Goal level D. SNAP Theory of Change & Results Framework E. Cross-Cutting Theme: Resilience F. Cross-Cutting Theme: Youth Productivity G. Cross-Cutting Theme: Gender Equity H. Cross-Cutting Theme: Environmental Stewardship G. Cross-Cutting Theme: Governance IV. PROGRAM IMPLEMENTATION SYSTEMS A. Program Management 5 Pages B. Partnership C. Knowledge Management (including M&E) D. Program Integration and Complementarity E. Resource Management - Finance E. Resource Management - Commodities E. Resource Management - Human Resources Table 7 summarizes the different sources of information that the evaluation team intends to consult for each intermediate result, cross-cutting themes and implementation systems. Table 7. Participants & Sites for Qualitative Interviews Type of Participant/Site Evaluation Team Member IR 1.1: Access to Improved Quality Health and Nutrition Services SNAP Evaluation Plan (PRE-FIELDWORK DRAFT) 7 October 2016 21 SNAP SO1 Implementation Staff District Health Management Team Ministry of Health and Sanitation Staff Community Health Workers Traditional Birth Attendants Staff of Participating Peripheral Health Units (Hygiene & Sanitation support, SAM referrals, GMP) Households with children identified with SAM and referred for treatment IR 1.2: Improved Health and Nutrition Practices SNAP SO1 Implementation Staff Community-based Health Promoters (Established by the Project) Mother-Care Groups/Lead Mothers Caregivers participating in cooking demonstrations Distributed food recipients Households trained on vegetable production IR 2.1: Increased Agricultural Productivity SNAP SO2 Implementation Staff for FFS Clusters Farmer Field Schools Farmer Field School Clusters Community Agricultural Field Agents MAFFS Frontline Extension Workers Recipients of Seeds, Tools, and Other Inputs IR 2.2: Increased Access to Financial and Marketing Resources SNAP SO2 Implementation Staff for VSLAs and Markets VSLA Groups Private Service Providers Input Supply Groups P4P Product Collectors Participants in Literacy and Numeracy trainings Participants in Youth Vocational and business skills training IR 2.3: Improved Community Assets SNAP SO2 Implementation Staff for Infrastructure Infrastructure Management Committees Crop Storage Management Committees Program Implementation Systems - Management and Finance Country Directors/Country Representatives or Program Directors for each partner SNAP COP SNAP Project District Operations Managers SNAP Evaluation Plan (PRE-FIELDWORK DRAFT) 7 October 2016 22 ACDI-VOCA Finance Manager Program Implementation Systems - Partnership, Integration and Commodities Country Directors/Country Representatives or Program Directors for each partner SNAP COP SNAP Project District Operations Managers ACDI-VOCA Commodity Manager Field-Based Commodity Management staff Commodity Warehouses Program Implementation Systems - Knowledge Management Country Directors/Country Representatives or Program Directors for each partner SNAP M&E Coordinator SNAP Field-based M&E Staff Program Implementation Systems - Human Resource Management Country Directors/Country Representatives or Program Directors for each partner SNAP COP ACDI-VOCA HR Director Partner HR Directors I. EVALUATION LOGISTICS ISG has contracted Cerra Automotive for transport for the evaluation team. The team will have two vehicles from October 15 through November 2 for the field work up￾country. ACDI/VOCA SNAP staff will facilitate the organization of participants for key informant interviews and focus group discussions in collaboration with former IMC staff who have been contracted by ACDI/VOCA specifically for this purpose4.                                                              4 Sub‐Recipient Agreements with IMC and OICI were completed in September for SNAP, so staff for these partners have already disengaged from the project. SNAP Final Evaluation Plan (Draft 2) Annex A – Report Format 23 ANNEX A: FINAL REPORT FORMAT ACDI/VOCA Sierra Leone Sustainable Nutrition and Agricultural promotion (SNAP) Project FINAL EVALUATION REPORT (Maximum 40 pages in Times New Roman 12pt font, excluding annexes) COVER PAGE TABLE OF CONTENTS ACRONYMS & ABBREVIATIONS I. EXECUTIVE SUMMARY II. BACKGROUND INFORMATION A. Overview of the SNAP Theory of Change & Project Strategy B. Project History and Operating Context C. Evaluation Methodology 1. Data Collection Tools and Processes 2. Data Analysis Tools and Processes 3. Evaluation Limitations III. OUTPUTS UNDER INTERMEDIATE RESULTS (Each section will provide an overview of the major achievements, key observations, and specific lessons learned.) A. Outputs Produced Under IR 1.1: Nutrition & Health Services 1. 2. 3. 4. B. Outputs Produced Under IR 1.2: Health & Nutrition Practices 1. 2. 3. 4. C. Outputs Produced Under IR 2.1: Agricultural Productivity 1. 2. 3. 4. D. Outputs Produced Under IR 2.2: Financial and Marketing Resources 1. 2. SNAP Final Evaluation Plan (Draft 2) Annex A – Report Format 24 3. 4. E. Outputs Produced Under IR 2.3: Community Assets 1. 2. 3. 4. IV. PROJECT TARGETING (This section will describe the evaluation team's observations on who is benefitting from the project, with particular attention to women and youth, including observations on what percentage of the project's impact groups are benefitting from multiple interventions.) V. PROJECT OUTCOMES AND IMPACT ASSESSMENT(Each section, except D, will provide an assessment of the life-of-project outcomes or impact, likelihood of this impact being sustained after the project ends, and specific lessons learned. Section D will summarize the evaluation team's observations on the relevance of the interventions that have been implemented by the program, observations on the project's theory of change, and lessons learned specifically relating to the theory of change.) A. Impact under SO 1: Maternal and Child Health and Nutrition B. Impact under SO 2: Livelihoods C. Collective Impact at the Goal-Level D. SNAP Theory of Change E. Cross-Cutting Theme: Resilience F. Cross-Cutting Theme: Youth Productivity G. Cross-Cutting Theme: Gender Equity H. Cross-Cutting Theme: Environmental Monitoring I. Cross-Cutting Theme: Governance VI. PROJECT IMPLEMENTATION SYSTEMS (Each section will provide an overview of the specific system, major observations from the final evaluation, and lessons learned) A. Project Management B. Partnerships C. Knowledge Management 1. Bringing New Approaches and Other Knowledge into the Project 2. Monitoring and Evaluation 3. Capturing and Disseminating Knowledge Generated by the Project D. Integration & Complementarity (including integration within the SNAP Project as well as integration and complementarity with other programs) E. Resource Management 1. Financial Resources 2. Commodity Management 2. Human Resources SNAP Final Evaluation Plan (Draft 2) Annex A – Report Format 25 VII. GOOD PRACTICES IN THE PROJECT (The evaluation team will work together to identify these.) A. B. C. D. E. VIII. PROJECT-LEVEL LESSONS LEARNED (This section will document the evaluation team's observations on any other lessons learned that have not already been described in previous sections.) A. B. C. D. E. X. CONCLUDING REMARKS (This section will provide an overall judgement by the evaluation team on the quality of the project, citing major achievements and shortcomings.) ANNEXES Annex A: Evaluation SOW Annex B: Evaluation Operational Plan Annex C: Field Work Schedule and Interviews/Discussions Held Annex D: Final Indicator Performance Tracking Table Annex E: Annex F: Annex G: SNAP Final Evaluation Plan (DRAFT 2) Annex B – Schedules 26 ANNEX B: EVALUATION SCHEDULES Full Evaluation Schedule The Table B-1 below describes the full evaluation schedule beginning with the assembling of the team in Freetown. Table B-1. Evaluation Schedule, October 10 through December 15 Activity Date Team Members Interpreters Evaluation team (Mike & Anna) arrives in Freetown October 9 (Monday) X X Initial Evaluation Team Meeting in Freetown October 10 (Monday) X X X X Project Orientation Meeting October 11 (Tuesday) X X X X Meetings in Freetown with Stakeholders October 12-14 (Wednesday to Friday) X X X X Travel to the Field October 15 (Sunday) X X X X X Information Gathering in the Field (SEE SEPARATE SCHEDULE) October 15 - 27 (Saturday - Thursday) X X X X X Data Processing & Preparation for Verification Workshop October 28 - 31 (Friday & Monday) X X X X X Verification Workshop (Makeni) November 1 & 2 (Tuesday & Wednesday) X X X X Follow-Up Information Gathering & Preparation for Debriefing November 2 & 3 (Wednesday & Thursday) X X X X USAID Debriefing November 4 (Friday) X X X X Evaluation Team Disburses November 5 X X X X Draft Report Submission to ACDI-VOCA November 30 X Stakeholder Comments on Draft Report sent to Evaluation Team December 10 X SNAP Final Evaluation Plan (DRAFT 2) Annex B – Schedules 27 Final Report Submitted to ACDI-VOCA December 15 X Field Work Schedule Table B-2 on the following page describes the schedule for stakeholder and site visits in the program area over the period October 15 through 28. SNAP Final Evaluation Plan (DRAFT 2) Annex B – Schedules 28 SNAP Final Evaluation DETAILED FIELD WORK SCHEDULE SHOWING KI INTERVIEWS & FGDS DATE DISTRICT LOCATION VILLAGE STRATA KEY INFORMANTS/FGDS Evaluation Team TRANSLATORS Oct 15 (Sat) ‐‐‐‐  Freetown to Makeni ‐‐‐  ‐‐‐  Travel   ‐‐‐  Bombali Makeni  ‐‐‐  ‐‐‐  Meeting with SNAP Staff ‐‐‐  Oct 16 (Sun) Bombali Sella Limba Chiefdom Kathimbo 1 (Both) LM/Care Unit, FFS/VSLA, CAF Limba (2) Kamankay 2 (SO1) LM/Care Unit, MAP, Oct 17 (Mon) Bombali Makeni  ‐‐‐  ‐‐‐  MAFFS (DAO, NGO Desk, SLARI), MOHS (DMO), District Council, Nutrition Officer, Environmental Specialist, SLIBA ‐‐‐  Gbanti Kamaranka Chiefdom Chiefdom HQ  ‐‐‐  Temne (1) Royema 3 (SO1) PHU staff, WASH Committee, HPs Temne (1) Oct 18 (Tue) Bombali Sanda Loko Chiefdom Mothoi 4 (SO2) FFS/VSLA, CAF/FEW, OICI Trainees Temne (2) Makapa 5 (Both) LM, MCGs, non‐MCHN benef Oct 19 (Wed) Tonkolili Magburaka  ‐‐‐  ‐‐‐  MAFFS (DAO, NGO Desk), MOHS (DMO), District Council   ‐‐‐  Kalansogoia Chiefdom Kathombo 6 (Both) PHU staff, WASH Comm, LMs VSLA, Limba (2) Kadanda 7 (SO1) LM/Care Unit, HPs, Limba (2) Oct 20 (Thu) Tonkolili Kafe Simira Chiefdom Maforeka 8 (Both) LM, MAP, VSLA, PSP, DF Committee Temne (2) Tonkolili 9 (SO2) FFS/VSLA, CAF, PSP, Fonthaneh 10 (Both) MAP, LM/Care Unit, FFS, CAF Mabontor 11 (SO1) LM, PHU staff, HPs, Temne (2) Octo 21 (Fri) Tonkolili Yoni Chiefdom Konta 2 12 (Both) MAP, FFS/VSLA Temne (2) Mayogbor/ 13 (SO1) LM, SNAP Final Evaluation Plan (DRAFT 2) Annex B – Schedules 29 Manjehun Gbombana 14 (SO1) MCG Temne (1) Makundor 15 (SO1) Non‐MCHN Beneficiaries Gaindema (SO2) FFS/VSLA, Cassava Processors Temne (1) Mile 91 to Bo  ‐‐‐  ‐‐‐  Travel ‐‐‐  Oct 22 (Sat) ‐‐‐  Makeni to Kabala ‐‐‐  ‐‐‐  Travel   ‐‐‐  Koinadugu Kasunko Chiefdom Fadugu 16 (Both) MAP, non‐MCHN benef, FFS, OICI Trainees Limba (2) ‐‐‐  Bo to Kailahun  ‐‐‐  ‐‐‐  Travel ‐‐‐  Oct 23 (Sun) Koinadugu Folosaba Dembelia Chiefdom Batahun 17 (SO1) LMs, MCGs, Jaluka (2) Kailahun Kissi Tongi Chiefdom Gblama 18 (SO2) FFS, CAF, Kissi (2) Voahun 19 (SO1) HPs, WASH Committee, Non‐ PM2A Beneficiaries Oct 24 (Mon) Koinadugu Kabala ‐‐‐  ‐‐‐  MOHS (DMO), MAFFS (DAO, NGO Desk, SLARI), District Council, Nutrition Officer, Environmental Specialist   ‐‐‐  Diang Chiefdom Heremakono 20 (SO1) MAP, HPs, MCGs Kuranko (2) Lenghekoro PHU staff Kailahun Kailahun  ‐‐‐  ‐‐‐  SNAP Staff Meeting ‐‐‐  MOHS (DMO), MAFFS (DAO, NGO Desk, SLARI), MOSW, District Council, Nutrition Officer, Environmental Specialist ‐‐‐  Oct 25 (Tue) Koinadugu Kabala to Mongo ‐‐‐  ‐‐‐  Travel ‐‐‐  Kailahun Mandu Chiefdom Manda Tawahun 21 (Both) LM, MAP, VSLA, OICI Trainees Mende (2) Makka 22 (SO1) MCGs, SNAP Final Evaluation Plan (DRAFT 2) Annex B – Schedules 30 Oct 26 (Wed) Koinadugu Mongo Chiefdom Mongo Bendugu 23 (SO2) FFS, VSLA, CAF, Kuranko (2) Mariamaya (By Motorbike) 24 (SO1) LM, MCGs, Town Chief, BES Kailahun Peje West Chiefdom Moyamba 25 (Both) Non‐MCHN benef, MAP, FFS, CAF, FEW Mende (2) Kodibu 26 (SO1) LM, HPs, MCGs, Town Chief Oct 27 (Thu) Koinadugu Neya Chiefdom Porpon 27 (Both) PHU staff, WASH Comm, FFS, CAF Kuranko (2) Mongo to Kabala ‐‐‐  ‐‐‐  Travel   ‐‐‐  Kailahun Kenema to Makeni ‐‐‐  ‐‐‐  Travel   ‐‐‐  Oct 28 (Fri) Koinadugu Kabala to Makeni ‐‐‐  ‐‐‐  Travel   ‐‐‐  Village Summary: Villages in Bombali = 5 (Full Team Starting Here) LM: Leader Mother CAF: Comm Ag Facilitator Villages in Tonkolili = 10 (Teams of two begin working) MCG: Mother Care Group FFS: Farmer Field School Villages in Koinadugu = 6 (Team of two only) HP: Health Promoter VSLA: Village Savings/Loan Ass Villages in Kailahun = 6 (Team of two only) WASH Committee DF: Drying Floor MAP: Men as Partner group BES: MAFFS Block Ext Super FGD: group between 6‐10 persons (maximum) FEW: Frontline Ext Worker ABC: Ag Business Center SNAP Final Evaluation Plan (DRAFT 2) Annex C– Data Collection Tools 31 ANNEX C: DATA COLLECTION TOOLS SO1 TOPICAL OUTLINES SNAP Project Final Evaluation Key Informant Interviews & Focus Group Discussions with Community-Level Beneficiaries and Intermediaries TOPICAL OUTLINE This topical outline is used to guide the key informant interviews and focus group discussions with representatives of impact groups (program beneficiaries) and intermediaries who participated in the SNAP Project to be held during the field visits in participating villages. The impact groups and intermediaries covered by this topical outline include: PM2A Beneficiaries and non-beneficiaries Men as Partners Groups Mother Care Groups Leader Mothers WASH Committees The questions are fairly general, used to stimulate discussion. The facilitator will explore in more depth interesting topics that come up during each interview/discussion. Before beginning the interview/discussions, an introduction and explanation of the purpose of the evaluator's visit will be provided with stress put on confidentiality and the importance of obtaining useful information that reflects reality. Each participant should introduce herself/himself before the discussion 1. What kinds of activities have you participated in with the SNAP Project? How were people selected to participate in these activities? The following types of activities may be mentioned: SO1 SO2 Training on detection of MAM and SAM Support for referrals of SAM children Growth Monitoring and Promotion services Training on health-seeking behavior Training on key family nutrition behaviors Cooking demonstrations and recipe book Food Distribution Kitchen Gardens Training on farming as a business Training on production of nutrient-rich crops Training on crop preservation and food processing Provision of inputs Participation in Farmer field schools Participation in Village Savings & Loan Associations Literacy and numeracy education Vocational skills training SNAP Final Evaluation Plan (DRAFT 2) Annex C– Data Collection Tools 32 Participation in mother-care groups Training on essential hygiene actions Training in IMCI/Family Planning and Reproductive Health Ebola response Agri-business and entrepreneurship training for youth Food-for-Work Drying floor construction Market infrastructure development Feeder road rehabilitation Crop storage facilities Labor for Learning for youth 2. Which of the activities undertaken with the SNAP Project have been most useful for preventing and reducing malnutrition? Why 3. Which activities have been least useful? Why? 4. What changes, positive or negative, have occurred in women and children’s lives as a result of the project? 5. Who exactly has benefitted from these changes? 6. Are there people in your village who should have participated in the project but did not? If yes, why did they not participate? 7. What differences exist between men and women relative to the changes induced by the project? How have men and women benefitted from or been treated differently by the SNAP Project? 8. How lasting are the changes induced by the project likely to be? How will things change after the project ends? 9. How do you feel about the way that you have been treated (informed about project activities, informed about changes in the project, selected or not selected for participation, etc.) by the SNAP Project? What suggestions would you make for future projects in this regard? 10. What other information or suggestions do you have that you would like to pass on to the evaluation team? SNAP Final Evaluation Plan (DRAFT 2) Annex C– Data Collection Tools 33 SNAP Project Final Evaluation SO1 Implementation Staff International Medical Corps TOPICAL OUTLINE 1. Please describe your functions and for how long you have been working with the SNAP Project. 2. Who is targeted by the project? How are these people identified? 3. Are there other people in participating villages who should be participating in the project but are not? Who are these people? Why do they not participate? 4. What positive changes have you seen in the people who have been targeted as a result of project activities? Which project activities have been most effective in creating these changes? 5. Which project activities produced the least impact? Why? 6. What differences exist between men and women relative to the changes induced by the project? How have men and women benefitted from or been treated differently by the SNAP Project? 7. How lasting are the changes induced by the project? How will things change after the project ends? For example, five years from now what are we still likely to be able to see from the work done by the SNAP Project? 8. What do you like about the way that the SNAP Project was implemented? What do you not like about the way the SNAP Project was implemented? How should this be done differently in future project? 9. What other information or suggestions do you have that you would like to pass on to the evaluation team? SNAP Final Evaluation Plan (DRAFT 2) Annex C– Data Collection Tools 34 SNAP Project Final Evaluation Government of Sierra Leone Stakeholders for SO1 TOPICAL OUTLINE This topical outline is used to guide the key informant interviews with representatives of GoSL stakeholders at the national, district and chiefdom levels. These include the following: Ministry of Health and Sanitation Representatives – Freetown District-Level Health Management Teams MOHS Nutrition Officers Peripheral Health Units staff Community Health Promoters 1. What do you know about the purpose and strategy of the SNAP Project being implemented by a consortium led by ACDI/VOCA? 2. How has your office been engaged by the project? 3. What has worked well in this relationship in terms of enabling the project to have greater impact or to make better use of resources? 4. What has not worked well in the relationship? How could this have been addressed? 5. What have you heard about or seen relative to the impact being achieved by the project? Which SNAP interventions were most relevant for achieving impact? Which SNAP interventions were less relevant? Why? 6. What plans do you have for continuing to work with SNAP participants after the project has ended? 7. What would you like to see done differently in future projects like SNAP with regard to how your office has been engaged? SNAP Final Evaluation Plan (DRAFT 2) Annex C– Data Collection Tools 35 SO2 TOPICAL OUTLINES SNAP Project Final Evaluation Key Informant Interviews & Focus Group Discussions with Community-Level Beneficiaries and Intermediaries TOPICAL OUTLINE This topical outline is used to guide the key informant interviews and focus group discussions with representatives of impact groups (program beneficiaries) and intermediaries who participated in the SNAP Project to be held during the field visits in participating villages. The impact groups and intermediaries covered by this topical outline include: Farmer Field Schools Farmer Field School Clusters Community Agriculture Promoters (CAPs) - Do these exist? Village Savings and Loan Associations The questions are fairly general, used to stimulate discussion. The facilitator will explore in more depth interesting topics that come up during each interview/discussion. Before beginning the interview/discussions, an introduction and explanation of the purpose of the evaluator's visit will be provided with stress put on confidentiality and the importance of obtaining useful information that reflects reality. Each participant should introduce herself/himself before the discussion. 1. What kinds of activities have you participated in with the SNAP Project? How were people selected to participate in these activities? The following types of activities may be mentioned: SO1 SO2 Training on detection of MAM and SAM Support for referrals of SAM children Growth Monitoring and Promotion services Training on health-seeking behavior Training on key family nutrition behaviors Cooking demonstrations and recipe book Food Distribution Kitchen Gardens Participation in mother-care groups Training on essential hygiene actions Training on farming as a business Training on production of nutrient-rich crops Training on crop preservation and food processing Provision of inputs Participation in Farmer field schools Participation in Village Savings & Loan Associations Literacy and numeracy education Vocational skills training Agri-business and entrepreneurship training for SNAP Final Evaluation Plan (DRAFT 2) Annex C– Data Collection Tools 36 Training on IMCI/Family Planning and Reproductive Health Ebola Response youth Food-for-Work Drying floor construction Market infrastructure development Feeder road rehabilitation Crop storage facilities Labor for Learning for youth 2. Which of the activities undertaken with the SNAP Project have been most useful for reducing food insecurity? Why? 3. Which activities have been least useful? Why? 4. What changes, positive or negative, have occurred in people's lives as a result of the project? 5. Who exactly has benefitted from these changes? How would you describe these people in terms of their food insecurity, e.g., highly food insecure to not so food insecure? 6. Are there people in your village who should have participated in the project but did not? If yes, why did they not participate? 7. What differences exist between men and women relative to the changes induced by the project? How have men and women benefitted from or been treated differently by the SNAP Project? 8. How lasting are the changes induced by the project likely to be? How will things change after the project ends? 9. How do you feel about the way that you have been treated (informed about project activities, informed about changes in the project, selected or not selected for participation, etc.) by the SNAP Project? What suggestions would you make for future projects in this regard? 10. What other information or suggestions do you have that you would like to pass on to the evaluation team? SNAP Final Evaluation Plan (DRAFT 2) Annex C– Data Collection Tools 37 SNAP Project Final Evaluation SO2 Implementation Staff TOPICAL OUTLINE 1. Please describe your functions and for how long you have been working with the SNAP Project. 2. Who is targeted by the project? How are these people identified? 3. Are there other people in participating villages who should be participating in the project but are not? Who are these people? Why do they not participate? 4. What positive changes have you seen in the people who have been targeted as a result of project activities? Which project activities have been most effective in creating these changes? 5. Which project activities produced the least impact? Why? 6. What differences exist between men and women relative to the changes induced by the project? How have men and women benefitted from or been treated differently by the SNAP Project? 7. How lasting are the changes induced by the project? How will things change after the project ends? For example, five years from now what are we still likely to be able to see from the work done by the SNAP Project? 8. What do you like about the way that the SNAP Project was implemented? What do you not like about the way the SNAP Project was implemented? How should this be done differently in future project? 9. What other information or suggestions do you have that you would like to pass on to the evaluation team? SNAP Final Evaluation Plan (DRAFT 2) Annex C– Data Collection Tools 38 SNAP Project Final Evaluation Government of Sierra Leone Stakeholders for SO2 TOPICAL OUTLINE This topical outline is used to guide the key informant interviews with representatives of GoSL stakeholders at the national, district and chiefdom levels. These include the following: Ministry of Agriculture, Forestry and Food Security Representatives – Freetown District-Level MAFFS Representatives in selected districts Selected District-Level Agricultural Officers District- Level Representatives for Public Works, Roads, Youth & Sports, and Women's Affairs SLIBA Representative in Freetown WFP P4P Representative in Freetown Input suppliers in Freetown and the selected Districts 1. What do you know about the purpose and strategy of the SNAP Project being implemented by a consortium led by ACDI/VOCA? 2. How has your office been engaged by the project? 3. What has worked well in this relationship in terms of enabling the project to have greater impact or to make better use of resources? 4. What has not worked well in the relationship? How could this have been addressed? 5. What have you heard about or seen relative to the impact being achieved by the project? Which SNAP interventions were most relevant for achieving impact? Which SNAP interventions were less relevant? Why? 6. What plans do you have for continuing to work with SNAP participants after the project has ended? 7. What would you like to see done differently in future projects like SNAP with regard to how your office has been engaged? SNAP Final Evaluation Plan (DRAFT 2) Annex C– Data Collection Tools 39 CROSS-CUTTING THEMES SNAP Project Final Evaluation Cross-Cutting Theme: RESILIENCE TOPICAL OUTLINE This topical outline provides key questions related to assessing changes in resilience capacities of targeted impact groups. The questions will be asked of SNAP beneficiaries as well as implementing staff and government representatives at the local level.  How did you find out about the EVD outbreak?  How did the outbreak affect your household?  How would you describe the situation now for your household, i.e., have you been able to resume livelihoods activities at the same level as before the outbreak?  What has been most useful for enabling your household to recover from the EVD outbreak?  What other livelihoods shocks have occurred during the life of the SNAP Project that have affected your household?  What sort of early warning did you receive that this shock was about to happen?  How would you describe the situation now for your household, i.e., have you been able to resume livelihoods activities at the same level as before the shock occurred?  What has been most useful for enabling your household to recover from the shock? SNAP Final Evaluation Plan (DRAFT 2) Annex C– Data Collection Tools 40 SNAP Project Final Evaluation Cross-Cutting Theme: YOUTH PRODUCTIVITY TOPICAL OUTLINE This topical outline provides key questions related to assessing changes in the productivity of youth. The questions will be asked during focus group discussions held specifically with groups of youth who have participated in the project. 1. Before the SNAP Project came to your village, what kinds of livelihoods activities were youth engaged in? 2. Has that changed as a result of the project? If so, how? 3. Which activities undertaken with the SNAP Project have been most useful for producing these changes? Why? 4. How were youth selected by the project to participate in these activities? Are there youth in your village who should have participated in the project but did not? If yes, why did they not participate? 5. Which activities implemented by the SNAP Project have been least useful for improving youth productivity? Why? 6. The SNAP Project is nearly finished, how will youth activities likely change after the project is finished? 7. What other information or suggestions do you have that you would like to pass on to the evaluation team? SNAP Final Evaluation Plan (DRAFT 2) Annex C– Data Collection Tools 41 SNAP Project Final Evaluation Cross-Cutting Theme: GENDER EQUITY TOPICAL OUTLINE Questions for Partners/Stakeholders 1. In your opinion, has SNAP advanced women’s equal participation, promoted the rights of women and girls and increased women’s access to and control over development resources and benefits? If so, to what extent? 2. How do the results achieved for women and girls compare to those achieved for men and boys? 3. Are there any unanticipated effects of SNAP on women, men, girls and boys? If so, what are these? 4. How have Gender Equity (GE) results contributed to the overall impact of SNAP? 5. Has SNAP improved the capacities of stakeholders to promote gender equitable practices and policies? If so, to what extent? 6. What can you say about the gender approach implemented by SNAP? a. Do you think it has had any impact on women's empowerment? If yes, how? Do you have some examples to share? b. How do you think SNAP could have done better to improve gender awareness and women participation? Questions for SO1 Participants 1. Has SNAP helped women and children in your community? If so, how? 2. Has SNAP done anything to increase men’s participation in improving the nutrition of their children? How well has this worked? Questions for SO1 Community Health Promoters 1. How did you manage the fact you are a man talking to women about “women's issues” (i.e. breastfeeding, post-partum child care, etc.)? 2. What barriers exist in addressing sensitive topics between the sexes? 3. Do you think women might have been more receptive to ENA messages if they were trained by female CHPs? 4. Have female and male beneficiary and CHPs perceptions about gender changed over time, as a result of the SNAP Project? If so, how? Questions for SO2 Participants SNAP Final Evaluation Plan (DRAFT 2) Annex C– Data Collection Tools 42 1. Have you seen any changes in attitudes about what is acceptable for women farmers to do have occurred as a result of the SNAP Project? If so, what changes have you seen? a. How do men view women as farmers? b. What are the reasons why women joined the project? c. How do farmer groups that are all women or mostly women differ from groups that are of mixed sex? 2. Have you seen any differences in the ways in which men farmers and women farmers have picked up and used the messages that were taught by SNAP? If so, what differences have you seen? a. How does the understanding of business differ between men and women? b. In general, is it considered acceptable for a woman farmer to run her farming activities as a business enterprise, or is farming for market sale considered to be more for men? How has this changed because of the project? 3. How do men learn about improved nutrition, care, and hygiene practices? What are the opportunities and the difficulties in involving men more deeply in health and nutrition activities? 4. Have men influenced the adoption of improved nutrition, care, and hygiene practices? If so, how? SNAP Final Evaluation Plan (DRAFT 2) Annex C– Data Collection Tools 43 SNAP Project Final Evaluation Cross-Cutting Theme: ENVIRONMENTAL STEWARDSHIP TOPICAL OUTLINE The following topics should be covered in interviews with ACDI-VOCA and partner staff responsible for environmental monitoring and impact mitigation. 1. Please describe the tools and processes that you have used in the SNAP Project for monitoring environmental impact. 2. What environmental impact identified at the beginning of the project has been monitored by these systems? 3. What activities have been undertaken by the project to mitigate these impacts? 4. What new environmental impacts emerged over the life of the project that were identified by the systems? 5. What activities were undertaken by the project to mitigate these impacts? 6. What would you suggest be done differently in future programs like SNAP with regard to environmental monitoring and impact mitigation? SNAP Final Evaluation Plan (DRAFT 2) Annex C– Data Collection Tools 44 SNAP Project Final Evaluation Cross-Cutting Theme: GOOD GOVERNANCE TOPICAL OUTLINE The following topics will be covered in interviews with different types of groups organized by the SNAP Project, including mother-care groups, farmer field schools and VSLAs. 1. How have members of the group been selected for inclusion in the group? How are new members selected? If someone wants to join the group, what process is followed? 2. What by-laws or other governing principles exist for the group? How much do group members know about these by-laws? How much do others outside of the group know about these by-laws? 3. How are decisions made by the group? Who makes the decisions? How much do group members know about how decisions are made? How much do outsiders know about how decisions are made by the group? 4. How accountable are group leaders to group members? How are leaders perceived in terms of serving the group? Are they mainly doing their jobs for themselves or do they see themselves serving the group to some degree or another? 5. How would you describe the level of participation for the group? How many members are active participants in planning group activities or making decisions? SNAP Final Evaluation Plan (DRAFT 2) Annex C– Data Collection Tools 45 IMPLEMENTATION SYSTEMS SNAP Project Final Evaluation Context & Operating Environment TOPICAL OUTLINE The following topics should be covered in interviews with ACDI/VOCA and partner managers, implementation staff, and participants.  Since the SNAP Project began implementation, what changes have occurred in the operating context (e.g., major events like the Ebola crisis or slow onset events like climate change) that have affected program implementation, either positively or negatively?  How did these specifically affect program implementation or program participants?  What did the project do to adapt to these changes?  Given these changes in the operating environment, which project activities seem most irrelevant now in terms of having impact on the lives of targeted impact groups?  Given the experience of the SNAP Project with these changes in the operating environment, what advice would you give to future programs to be better able to adapt to similar changes that may occur in the future? SNAP Final Evaluation Plan (DRAFT 2) Annex C– Data Collection Tools 46 SNAP Project Final Evaluation Targeting TOPICAL OUTLINE (All Team Members) Topics for Project Managers and Implementation Staff ∞ Please explain your understanding of the concept of target communities that was used by the SNAP Project and the criteria that were used to select the target communities. ∞ In your opinion, were these criteria appropriate for the project’s overall objective of reducing food insecurity and increasing resilience in Bombali, Koinadugu, Tonkolili and Kailahun Districts? ∞ In your opinion, were these criteria applied as intended? ∞ If in your opinion the criteria were not appropriate or they were appropriate but not applied as intended, do you think this affected the program negatively, or did the selected communities selected nevertheless need the help provided and benefit from it? ∞ Please explain how you arrived at these opinions. ∞ Within the target communities, the main intended beneficiary categories were farmers, pregnant and lactating women, child caregivers of children under 2, and youth. For each of these categories, please explain your understanding of what percentage of the total population in each beneficiary category was meant to be covered by the program, and what the criteria were for selecting actual program participants before and after program redesign. ∞ Please comment on the extent to which procedures followed for selecting actual program participants before and after program redesign conformed to these criteria. ∞ In your opinion, were the procedures followed appropriate or did they lead to inclusion of unintended beneficiaries and exclusion of intended beneficiaries? ∞ In what ways did the project respond to problems with beneficiary selection that were observed? Were these responses effective, or could more have been done to resolve problems? ∞ How does the project monitor and manage targeting issues such as changes in the list of target communities, movement of program beneficiaries out of the target communities, non-participation of selected beneficiaries in program activities? SNAP Final Evaluation Plan (DRAFT 2) Annex C– Data Collection Tools 47 ∞ How would you classify beneficiaries in terms of level of benefits? For example, there will be some beneficiaries in target communities that benefited from multiple program activities and others in communities benefiting from only one or a few activities. What difference, if any, do you think this made to the overall benefit obtained by the individual beneficiaries in the two different types of communities? Topics for Participants and Intermediaries ∞ What proportion of the population of the program’s target communities was selected as actual program beneficiaries? ∞ Who else in the target communities, apart from these intended beneficiaries, has benefitted from the program and how have they benefitted? How should we describe these unintended beneficiaries in terms of food insecurity? ∞ Who else outside the target communities has benefitted from the program and how have they benefitted? Were these benefits intended by the program, or did they come about by chance? How should we describe unintended beneficiaries outside the target communities in terms of food insecurity? ∞ Some beneficiaries live in communities that benefitted from multiple program activities and others live in communities that benefitted from only one or a few program activities. Do you think this made a difference in the benefits obtained by the individual beneficiaries or not? Please give the reasons for your answer. SNAP Final Evaluation Plan (DRAFT 2) Annex C– Data Collection Tools 48 SNAP Project Final Evaluation Project Management TOPICAL OUTLINE ∞ What structures are used to manage implementation of the SNAP Project across all the partners, including staffing of management teams for ACDI/VOCA and implementing partners, steering or advisory committees, meetings of managers, and so on? ∞ What is the history of these management structures, including turnover of personnel in key positions, restructuring or other changes? ∞ Who is responsible for the overall vision for the project? How well has the vision been articulated? How effectively has this vision been imparted to staff within the different agencies? ∞ Who is responsible for the vision within each implementing partner organization? Are the visions of the individual partners generally coherent with each other and with the vision at project level? If not, how are the differences managed? ∞ If there are differences, what impact has this had on project delivery? ∞ How are operational plans developed for the project? What has worked well in this process? What has not worked well? ∞ How are problems with implementation identified, analyzed and solved? How is information generated by the project's M&E systems used for decision-making? What has worked well in terms of problem-solving? What has not worked well? ∞ Please describe the working relationships between Freetown-based project managers and project officers and senior leadership in your organization’s Headquarters. How much management responsibility is devolved and how much is exercised from HQ? ∞ Please describe the working relationships between Freetown-based project managers and managerial staff in the field. How much management responsibility is devolved to the field and how much is exercised from Freetown? Are some specific managerial functions devolved and others retained at the head office, or are all managerial functions devolved, with the head office providing general oversight and managing planning and reporting functions for the project? ∞ What kinds of information are communicated (i) within ACDI/VOCA at different levels, (ii) among implementing partners at different levels, (iii) with USAID/FFP? What has worked well? What has not worked well? SNAP Final Evaluation Plan (DRAFT 2) Annex C– Data Collection Tools 49 ∞ How well has the project communicated with external stakeholders, e.g., ministry officials (national and local), other donors and UN agencies, other NGOs with similar programs and projects in Liberia? What has worked well? What has not worked well? ∞ What have been the biggest challenges relative to administrative support for the program in the areas of financial management, commodity management, human resource management, procurement, transport or anything else related to project management? ∞ What solutions have been devised to address these challenges? What has worked well? What has not worked well? ∞ What changes would you propose to improve project management in future programs? ∞ What are the most important lessons learned from the SNAP Project relative to management of FFP Development Food Assistance Programs. SNAP Final Evaluation Plan (DRAFT 2) Annex C– Data Collection Tools 50 SNAP Project Final Evaluation Knowledge Management (Including M&E) TOPICAL OUTLINE Knowledge In ∞ Over the five-year life of the project, what new ideas and approaches been brought into the SNAP Project from either internal or outside sources? ∞ Who can be credited for bringing each of these into the project? ∞ Which of these have been most useful? Which have not been very useful? Why? ∞ What staff positions are responsible for bringing new ideas and approaches into the program? How would you describe the effectiveness of these positions n tis function? ∞ What more could the project have done to bring in good ideas and approaches from both internal and outside sources to improve the effectiveness or efficiency of the project? M&E Systems ∞ Please describe the systems used to carry out M&E functions for the SNAP Project. How would you describe the relationships that exist between the various units responsible for M&E in SNAP, including ACDI/VOCA, HQ, ACDI/VOCA Freetown, partner M&E units and field office-based M&E staff? ∞ Please describe the annual planning process used to develop program implementation plans, and the way in which M&E results have been used to feed this process. What has worked well? What has not worked so well? Why? ∞ Please describe the tools used to monitor progress against each of the performance indicators in the Indicators Performance Tracking Table (IPTT) each year. What has worked well? What has not worked so well? Why? ∞ What quality control mechanisms have been put in place to ensure consistency and accuracy of reporting on data and information feeding into the M&E unit from different sources? ∞ Who uses information generated through the project's monitoring systems? What has worked well? What has not worked so well? ∞ How has the project strategy or approach changed as a result of information generated by the project's M&E systems? ∞ Relative to the baseline survey, what worked well in getting this organized, implemented and completed? What did not work so well? SNAP Final Evaluation Plan (DRAFT 2) Annex C– Data Collection Tools 51 ∞ Relative to the mid-term evaluation, what worked well in getting this organized, implemented and completed? What did not work so well? How were the recommendations used by the project? ∞ What other types of assessment tools did the project use? What worked well? What did not work so well? Why? ∞ What lessons learned relative to M&E have emerged from the experience of the SNAP Project? Knowledge Out ∞ What systems are in place for ensuring effective communications related to SNAP within ACDI/VOCA and with other partners? What is working well? What problems have occurred relative to communications? How have these problems affected the project? ∞ What staff positions are responsible for capturing, documenting and disseminating knowledge out of the SNAP Project? How would you describe the effectiveness of these positions on this function? ∞ Did the program have an explicit strategy for identifying best practices and lessons learned? If yes, please describe this strategy and provide some examples of best practices or lessons learned that were documented and disseminated. If no, has there been any ad hoc documenting of best practices and lessons learned other than through external evaluations and FFP trip reports? Please describe. ∞ What constraints were encountered as far as identifying, documenting and disseminating best practices and lessons learned from the SNAP Project experience? What solutions were devised to address these constraints? ∞ What lessons have been learned relative to communication of best practices and lessons learned by the SNAP program that could be applied to future programs? ∞ What changes would you recommend to make the capture, documentation and dissemination of knowledge from the project more effective? SNAP Final Evaluation Plan (DRAFT 2) Annex C– Data Collection Tools 52 SNAP Project Final Evaluation Integration & Complementarity TOPICAL OUTLINE With the Government’s Poverty Reduction Strategy and Other Government or Donor Strategies ∞ How well does the strategy of the SNAP Project align with government strategies for poverty reduction, agricultural development, food and nutrition security, health and nutrition and youth employment? ∞ How does the SNAP Project coordinate with relevant government ministries at different levels? What works well? What has not worked well? ∞ How does the SNAP program coordinate with donors, UN agencies and NGOs that are funding and/or implementing similar activities in the four target districts and elsewhere in Sierra Leone? What works well? What has not worked well? ∞ What mechanisms has the SNAP Project used to resolve issues arising from conflicts between its own objectives and activities and those of other relevant actors in the Sierra Leone context? What works well? What has not worked well? With USAID Country Strategy ∞ Which activities in the SNAP Project have made the largest contributions to the USG strategy for Sierra Leone? ∞ Which activities were more marginal in terms of their contributions to the USG strategy? Across SOs within the SNAP Project ∞ The SNAP project was meant to be implemented as an integrated set of activities for different beneficiary categories within each target community. To what extent do you think the project succeeded in this? ∞ What potential synergies within the project were not sufficiently capitalized upon in terms of complementary activities that could produce benefits for all households within a target community, not only those of the beneficiaries of a specific program activity. Why? ∞ What changes in project design and implementation methods might make it possible to realize a greater degree of synergy across different program components at community level in future programs? SNAP Final Evaluation Plan (DRAFT 2) Annex C– Data Collection Tools 53 SNAP Project Final Evaluation Commodity Management TOPICAL OUTLINE The following topics will be covered in interviews with commodity management staff. Pipeline ∞ How does the food pipeline work? Who is responsible for call forwards, port off￾loading and transport to the port warehouse, inland transport, and distribution to end users? ∞ What problems or changes have occurred with call forwards and how were these managed? ∞ What problems have occurred with port commodity management and how were these managed? ∞ What problems have occurred with inland transport and how were these managed? ∞ What problems have occurred with warehousing/final distribution and how were these managed? ∞ What lessons have we learned with our commodity management in the SNAP? ∞ What losses have occurred and how did these occur? What was the impact of the losses on the program as well as on subsequent call forwards? What changes were made in commodity management systems in response to the losses? The following tables should be completed. LOA Commodity Summary (MT) Distribution Monetization Total FY 10 FY 11 FY 12 FY 13 FY 14 FY 15 FY 16 (Through June 2016) Cumulative disbursements through June 2016 Original Proposed LOA Quantities Revised Projected LOA Quantities Percentage of LOA Projection Achieved Through June 2016 Commodity Loss Summary (MT) SNAP Final Evaluation Plan (DRAFT 2) Annex C– Data Collection Tools 54 Fiscal Year Amount Purchased Ocean Losses Amount Received in Country Inland Losses Percent Lost Monetized Commodities FY 10 FY 11 FY 12 FY 13 FY 14 FY 15 FY 16 Thru Qtr 3 TOTAL Distributed Commodities FY 10 FY 11 FY 12 FY 13 FY 14 FY 15 FY 16 Thru Qtr 3 TOTAL Monetization ∞ What has been the history of monetization in the project? ∞ How is monetization currently being done? ∞ What problems have occurred with monetization and how were these managed? ∞ What has been the cost recovery rate on monetization over the life of the project? What is the cost recovery rate on the most recent monetization? ∞ How is monetization being used to support market development? The following table should be completed. Monetization Cost Recovery SALE # FISCAL YEAR COMMODITY QUANTITY ( MT) C&F ($/MT) SALE PRICE ($/MT) COST RECOVERY TOTAL thru FY 2016 QTR 3 * * *Weighted averages by volume SNAP Final Evaluation Plan (DRAFT 2) Annex C– Data Collection Tools 55 Reporting ∞ How are commodity reports prepared? ∞ What problems have occurred in report preparation and how have these been managed? ∞ What has been the response from the donor on commodity reports? Rations ∞ How are ration composition and quantities calculated? ∞ How have they changed over the life of the program? ∞ How have rations been used by recipients? ∞ What changes should be considered relative to ration composition or quantities? Why should these changes be considered? SNAP Final Evaluation Plan (DRAFT 2) Annex C– Data Collection Tools 56 SNAP Project Final Evaluation Partnership TOPICAL OUTLINE  In terms of working with ACDI-VOCA as the lead agency, what did you like about the relationship with them? What didn't you like and how would you like to see that changed in future programs like the MYAP?  What lessons have been learned in terms of partner relationships that should be highlighted by the Final Evaluation for future programs?  What best practices have emerged from the MYAP relative to partnerships that should be highlighted by the Final Evaluation?  Relative to relationships with consortium partners (IMC and OICI), what has worked well in terms of working together to achieve impact or making more effective use of resources? What has not worked well? What would you like to see done differently in future programs like the MYAP with regard to partnership relationships with consortium partners?  What other partnership relationships have been important for SNAP? What has worked well in these relationships? What has not worked well? What would you like to see done differently in future programs like the MYAP with regard to partnership relationships with these other organizations?  How do you feel about how major strategic or problem-solving decisions were made in the program? How much influence did you have on the decisions? How informed were you on the reasoning behind the decisions that were made?  How would you describe your overall level of satisfaction with the SNAP Project? What would you suggest be done differently in future projects like SNAP? SNAP Final Evaluation Plan (DRAFT 2) Annex C– Data Collection Tools 57 SNAP Project Final Evaluation Financial Management TOPICAL OUTLINE The following topics should be discussed with finance managers of ACDI/VOCA and consortium partners in Freetown as well as in the field offices. Budget Questions ∞ What is the current LOA approved budget? How has this been revised since project start-up? ∞ What are expenditures through the most recent reporting period in FY '16? ∞ What percentage of the approved budget will likely be spent by the end of the program in December 2016? ∞ The following table should be completed. Cash Expenditure Summary Projected Through Life-of-Activity (US$) Cost Center Monetization Proceeds 202e ITSH Cost Share Total ACDI/VOCA OICI IMC Total Direct Costs NICRA for ACDI/VOCA NICRA for OICI NICRA for IMC TOTAL Expenses through June 2016 TOTAL LOA Budget at Time of Approval Current Amended LOA Budget Percent of Current LOA Budget Spent by June 2016 Cash Flow SNAP Final Evaluation Plan (DRAFT 2) Annex C– Data Collection Tools 58 ∞ How is cash for project implementation being provided to partners? ∞ How effective has cash flow been managed in the project? Have there been any significant delays in cash flow either from the donor to ACDI/VOCA or from ACDI/VOCA to the consortium partners? What was the cause of the delays? What changes were made in managing cash flow? Reporting ∞ How do financial reports for the project get prepared? ∞ What problems have occurred with financial reporting and how have these been resolved? Cost Share ∞ What is the cost-share commitment and how is this being met? ∞ What percentage of the cost-share commitment will be achieved by the end of the project? ∞ What other forms of cost-share have arisen since the project was initiated? How are these being reported? Audits ∞ What audits have been completed on project funding since the project was initiated? ∞ What have been the audit findings? ∞ How have these been addressed? Lessons Learned and Best Practices ∞ In terms of financial management systems, what has worked especially well over the life of the SNAP Project? ∞ What advice would you give or what changes would you like to see in future programs like SNAP relative to financial management systems? SNAP Final Evaluation Plan (DRAFT 2) Annex C– Data Collection Tools 59 SNAP Project Final Evaluation Administrative Support (Communications, Procurement & Logistics) TOPICAL OUTLINE The following topics should be covered, as relevant, in interviews with staff from all partners who are directly responsible for managing administrative support for the SNAP Project, particularly related to communications, procurement and logistics. 1. What systems are in place for ensuring effective communications related to SNAP within your organization and with other partners? What is working well? What problems have occurred relative to communications? How have these problems affected the project? What changes would you recommend be made to these systems to make communications more effective? 2. How is procurement done for the SNAP Project, including procurement of tools and equipment such as vehicles, as well as procurement of materials used for daily operations? What is working well? What problems have occurred relative to procurement? How have these problems affected the project? What changes would you recommend be made to the procurement procedures to make them more effective? 3. How is logistics done for the SNAP Project, especially logistics related to transportation of staff or materials? What is working well? What problems have occurred relative to logistics? How have these problems affected the program? What changes would you recommend be made to the logistics systems to make them more effective? SNAP Final Evaluation Plan (DRAFT 2) Annex C– Data Collection Tools 60 SNAP Project Final Evaluation Human Resource Management TOPICAL OUTLINE The following topics should be covered in interviews with those responsible at different l evels in the SNAP Project for human resource management. Staff Recruitment and Retention ∞ What significant challenges has the SNAP Project faced relative to human resources in the project? How did the SNAP Project deal with these challenges? ∞ How many positions total are there in the SNAP Project? How many vacancies are there at this point in time? ∞ How would you describe staff turnover in the project? How has this affected project implementation? ∞ In general, what has worked well and what has not worked well relative to hiring and retaining staff in the SNAP Project? Staff Capacity Building and Performance Management ∞ What types of activities has the SNAP Project undertaken to build staff capacities to be able to work effectively? ∞ How would you describe the systems in the SNAP Project for motivating staff and facilitating high levels of performance? What has worked well? What has not worked well? SNAP Final Evaluation Plan (DRAFT 2) Annex D – Verification Workshop 61 SNAP Final Evaluation Verification Workshop Tuesday & Wednesday (Half Day), November 1 & 2 WORKSHOP PURPOSE Based on information obtained from field work and other interviews undertaken thus far, the evaluation team has started documenting observations and findings on the impact and lessons learned from the SNAP Program. These preliminary results of the evaluation will be shared in a Verification Workshop with SNAP implementation staff and discussed further to ensure that the they reflect reality and are stated appropriately. WORKSHOP OUTPUTS Three major outputs are targeted for the workshop. These are (1) agreement on the major achievements for the program, (2) agreement on the key observations assembled so far on the impact of the program, both short-term within the life of the program and long-term, likely to be sustained after the program ends, and (3) lessons learned from the implementation of the program. WORKSHOP FLOW Day 1- Tuesday, November 1 SESSION 1-1: Introductions, Workshop Purpose, Objectives and Plan, Logistics (½ hour, ) The workshop purpose, objectives, and flow will be presented, along with details on logistics for the workshop. Participants will be asked to write on zip cards their response to the following question.  What have been the three biggest changes that you have observed in the lives of targeted beneficiaries as a result of the SNAP Program? The cards will be handed in (not discussed) as participants introduce themselves and will be put on display. As the topics on the cards are discussed during the workshop sessions, the related cards will be removed from the display. At the workshop, any cards remaining on display will be discussed to determine what steps should be taken to get information to understand the impact that has been proposed on the card. SESSION 1-2: History & Operating Context (½ hour, ) Key events in the history of SNAP will be presented and discussed for clarity and completeness, ensuring that major events in the life of the program that have affected implementation have been captured. Observations on contextual factors that have influenced the program will be presented. In preparation for the following sessions, a brief overview of the HANDS strategy will also be presented. SESSION 1-3: SO1 Reducing Malnutrition (2 hours, ) SO1 has two Intermediate Results with associated interventions as indicated below. SNAP Final Evaluation Plan (DRAFT 2) Annex D – Verification Workshop 62  IR 1.1 Improved access to and use of quality nutrition and health services (Training and material support for DHMTs, PHUs, and WASH Committees; MAM and SAM referral and treatment services, growth monitoring services, and food distributions)  IR 1.2 Improved health and nutrition practices at the household level (Mother Care Groups, Behavioral change communications, food preparation and household gardening) For the interventions under each of the IRs, the evaluation team will present the following for further discussion, clarification and modification: An Overview of the Approach Being Used Achievements To Date Key Observations Short Term Impact (During the Life of the Program) Likely Long Term Sustained Impact Key Lessons Learned SESSION 1-4: SO2 Enhanced Livelihoods (2 hours, ) SO2 has three Intermediate Results with associated interventions as shown below.  IR 2.1 Increased agricultural productivity (Farmer Field Schools/Clusters, Farming as a Family Business, Input Distributions)  IR 2.2 Increased access to financial and marketing services (Value chain linkages, VSLAs, Linkages to private sector and service providers, input supplier capacity building)  IR 2.3 Improved community assets (Infrastructure, Crop Storage, Labor for Learning) For the interventions under each of the IRs, the evaluation team will present the following for further discussion, clarification and modification: An Overview of the Approach Being Used Achievements To Date Key Observations Short Term Impact (During the Life of the Program) Likely Long Term Sustained Impact Key Lessons Learned SESSION 1-5: Program Targeting (½ hour, ) In this session, the evaluation team will present a summary of observations on targeting in terms of who is benefitting from the program, including both targeted impact groups, targeted intermediaries and others who have been affected, positively or negatively by the program. Lessons learned from the targeting processes used in the program for selecting communities and participants will also be presented for discussion and refinement. SESSION 1-6: Summary of Program Impact at the SO and Development Objective-Levels (1 hour, ) Previous sessions have described the impact achieved by the HANDS Program for specific interventions that have been implemented under each IR. In this session a summary of the collective impact of these interventions on each Strategic Objective and ultimately on the SNAP Final Evaluation Plan (DRAFT 2) Annex D – Verification Workshop 63 Development Objective for the program will be presented by the evaluation team for discussion and refinement. Day 2 – Wednesday, November 2 SESSION 2-1: Recap of the Previous Day's Discussions and Stage Setting for Day 2 (¼ hour). A brief summary of the previous day's discussions will be provided along with some introductory information to set the stage for day two of the workshop. SESSION 2-2: Cross-Cutting Themes (1½ Hours, Resilience- , Youth- , Gender- , Environment- , and Governance- ) The SNAP Program was designed to have impact on five cross cutting themes, including household and community resilience, youth productivity, gender equity, environmental stewardship, and good governance, and the evaluation team will present a summary of the key observations, impact and lessons learned that have emerged in the evaluation investigations for further discussion and refinement. SESSION 2-2: Program Implementations Systems - Management, Partnership, Knowledge Management, and Program Integration (1 hour, Management- , Partnership- , Knowledge Management- , and Integration- ) Management systems are oriented around vision and leadership, strategic and operational planning, problem-solving and decision-making, and communications. Partnership refers to the relations between various types of partners in the program, including consortium partners, full￾time technical partners, ad hoc technical partners, and any other stakeholders who have roles to play in the program. Knowledge management refers to how new ideas and approaches have been brought into the program, how Monitoring and Evaluation (M&E) systems have been used to provide information for reporting and decision-making, and how best practices and lessons learned have been documented and disseminated. Program integration refers to how the program has integrated internally across components as well as how the program has interfaced with government strategies and programs or the programs/projects being implemented by other organizations in the four targeted districts. The evaluation team will present an overview of the key observations and lessons learned from investigations into these systems. SESSION 2-3. Program Resource Management – Financial Resources, Commodity Resources, and Human Resources (½ hour, Finance- , Commodities- , HR￾Investigations on financial resource management include looking at budgeting, cash flow, reporting, and financial auditing processes. Commodity management investigations examine the commodity management systems for both monetized and distributed commodities from call forward to final sale (monetization) or distribution Human resource management includes discussions on organizational structure, staff recruitment and retention, and staff development. The final evaluation team will present key observations and lessons learned on program resource management for discussion and refinement. SESSION 2-4: Next Steps (¼ hour, ). Over the course of the workshop, some topics may have emerged from the discussions that represent areas that need to be further investigated by the final evaluation team. In this final SNAP Final Evaluation Plan (DRAFT 2) Annex D – Verification Workshop 64 session of the day, participants will discuss these topics and will agree on how best to obtain information to resolve the questions that have arisen. SNAP Final Evaluation Plan (DRAFT 2) Annex E – Community Sample` 65 Table E-1. Communities Selected for Field Visits in the SNAP Final Evaluation VILLAGE CHIEFDOM INTENSITY CLASSIFICATION ACCESSIBILITY (Distance from District HQ) Bombali District Kathimbo Sella Limba Both SOs 2:20 minutes by car Kamankay SO1 Only 2:30 minutes by car Royeama Gbanti Kamaranka SO1 Only 1:20 minutes by car Mothoi Sanda Loko SO2 Only 2:00 minutes by car Makapa Both SOs 2:45 minutes by car Tonkolili District Kathombo Kalansogoia Both SOs 1:40 minutes by car Kadanda SO1 Only 2 hours by car, 10 minutes by foot Maforeka Kafe Simira Both SOs 1 hour by car Tonkolili SO2 Only 1:20 minutes by car Fonthaneh Both SOs 45 minutes by car Mabontor SO1 Only 1 hour by car Konta 2 Yoni Both SOs 3 hours by car Mayorgbo/Manjehun SO1 Only 4 hours by car Gbombana SO1 Only 3 hours by car Makundor SO1 Only 4 hours by car Koinadugu District Fadugu Kasunko Both SOs 45 minutes by car Batahun Folosaba Dembelia SO1 Only 2 hours by car Heremakono/ Lenghekoro Diang SO1 Only 30 minutes by car Mongo Bendugu Mongo SO2 Only 5 hours by car Mariamaya SO1 Only 6 Hours by car and motorbike Porpon Neya Both SOs 7 hours by car Kailahun District (Kabala) Gblama Kissi Tongi SO2 Only 2.5 hours by a car Voahun SO1 Only 2.5 hours by a car Mandu Tawahun Mandu Both SOs 2 hours by a car Makka SO1 Only 2 hours by a car Moyamba Peje West Both SOs 3 hours by a car Kodibu SO1 Only 3 hours by a car TOTAL ALL DISTRICTS 27 14 SO1 Only = 13 SO2 Only = 4 Both SOs = 10 Remote (>2 hours) = 15 Accessible (<=2 hours) = 12 SNAP Final Evaluation Plan (DRAFT 2) Annex E – Community Sample` 66    Sustainable Nutrition and Agriculture Promotion (SNAP) PROGRAM ENDLINE SURVEY This publication was produced for ACDI VOCA, with support from the US Agency for International Development – Office of Food for Peace. It was prepared by NestBuilders International (NBI). September 2016. i Sustainable Nutrition and Agriculture Promotion (SNAP) Program Endline Survey Research Conducted by: NestBuilders International (NBI) NestBuilders International 16 Fifth Street, Juba Hill Freetown, Sierra Leone +232‐25‐239‐495 www.nbiconsultancy.com ii Table of Contents Acronyms........................................................................................................................................v Survey Summary: at‐a‐glance.........................................................................................................vi 1. Introduction to the Evaluation................................................................................................ 1 2. ‘SNAP’ Program Description ................................................................................................... 1 2.1 Program Activities................................................................................................................... 1 2.2 Geographic Coverage.............................................................................................................. 2 3. Survey Details ........................................................................................................................ 2 3.1 Purpose of the Survey............................................................................................................. 2 3.2 Survey Methodology............................................................................................................... 4 3.3 Sampling Procedure................................................................................................................ 4 3.4 Survey Instrument................................................................................................................... 6 3.5 Training of Field Staff.............................................................................................................. 7 3.6 Survey Logistics....................................................................................................................... 8 3.7 Data Entry and Analysis .......................................................................................................... 9 3.8 Limitations of the Survey ........................................................................................................ 9 4. Endline Survey Results...........................................................................................................11 4.1 Characteristics of the Study Population................................................................................ 11 4.2 Household Participation in SNAP Project Activities..............................................................11 Children's Health and Nutrition Indicators ....................................................................................13 4.3 Anthropometrics................................................................................................................... 13 4.4 Infant and Young Child Feeding Practices............................................................................. 15 4.5 Diarrhea and Oral Rehydration Therapy............................................................................... 17 Maternal and Child Healthcare Indicators.....................................................................................18 4.6 Antenatal Care ...................................................................................................................... 19 4.7 Skilled Birth Attendant.......................................................................................................... 20 4.8 Essential Newborn Care ........................................................................................................ 20 4.9 Postnatal Care....................................................................................................................... 21 4.10 Growth Monitoring ............................................................................................................... 22 Household Indicators....................................................................................................................24 4.11 Household Dietary Diversity Score ....................................................................................... 24 4.12 Household Food Provisioning ............................................................................................... 25 4.13 Caregiver Hygiene Practices.................................................................................................. 25 Agriculture and Livelihood Indicators............................................................................................26 4.14 Rural Household Revenue..................................................................................................... 26 4.15 Crop Sales.............................................................................................................................. 27 4.16 Improved Agricultural Practices............................................................................................ 28 5. Summary Statistical Comparison of Baseline and Endline Indicator Values............................30 6. Additional Analysis by Project Participation ..........................................................................34 iii Annex 1: Summary SNAP Performance Indicator Reference Sheet (PIRS) for Endline Indicators....37 Annex 2: Sampled EAs and Households for the Endline Survey......................................................40 Annex 3: Data Collection Tool .......................................................................................................46 Annex 4: Differences between Baseline and Endline Indicators.....................................................47 List of Figures Figure 1: SNAP intervention areas.......................................................................................................... 2 Figure 2: Type of assistance received by SNAP beneficiaries...............................................................12 Figure 3: Initial breastfeeding for last birth among children aged 0‐6 months, baseline vs. endline ..16 Figure 4: Percentage of children aged 0‐59 months with diarrhea in the last two weeks...................17 Figure 5: Percent of child diarrhea cases (n=86) treated with ORS or ZINC, by sex and district..........18 Figure 6: Percent of mothers of children 0‐23months who had four or more ANC visits, by sex and district ...................................................................................................................................................19 Figure 7: Percent of newborns (0‐3 months) receiving essential newborn care, by sex and district...21 Figure 8: Percent of children that have ever attended growth monitoring, baseline vs. endline........23 Figure 9: Percent of children (0‐3 years) regularly attending growth monitoring, by sex and district 23 Figure 10: Households dietary composition, by food group ................................................................25 Figure 11: Percent of households with non‐farm income by district, baseline vs. endline..................27 Figure 12: Percent of households with any farm sales in the 12 months preceding the survey by district, baseline vs. endline.................................................................................................................. 28 Figure 13: Use of improved agricultural practices................................................................................ 29 List of Tables Table 1: Summary IPTT indicators........................................................................................................... 3 Table 2: Survey modules and target respondents.................................................................................. 7 Table 3: Household characteristics....................................................................................................... 11 Table 4: Type of SNAP assistance received by all surveyed households, by district ............................12 Table 5: Prevalence of underweight children under 5 years by sex and district, baseline vs. endline 14 Table 6: Prevalence of stunted children under 5 years by sex and district, baseline vs. endline.........14 Table 7: Prevalence of malnutrition among children 6‐59 months by sex and district, baseline vs. endline ..................................................................................................................................................15 Table 8: Percent of children aged 0‐6 months exclusively breastfed by sex and district, baseline vs. endline ..................................................................................................................................................16 Table 9: Percent of child diarrhea cases (n=86) treated in Hospital/Clinic/PHU by sex and district, baseline vs. endline............................................................................................................................... 18 Table 10: Percent of mothers of children 0‐59 months who had four or more ANC visits by district, baseline vs. endline............................................................................................................................... 19 Table 11: Percent of births with a skilled birth attendant by sex and district, baseline vs. endline ....20 Table 12: Postnatal care received  by mothers by child sex and district, baseline vs. endline ............22 Table 13: Household dietary diversity score (HDDS) by district, baseline vs. endline..........................24 iv Table 14: Average months of adequate household food provisioning (MAHFP) by district, baseline vs. endline ..................................................................................................................................................25 Table 15: Caregiver hygiene practices by district, baseline vs. endline................................................26 Table 16: Average rural household revenue by source, baseline vs. endline.......................................27 Table 17: Average annual value of household crop sales by district, baseline vs. endline ..................28 Table 18: Summary statistical comparison of baseline and endline indicator values..........................30 Table 19: Endline indicator values, by project participation ................................................................34 v Acronyms ANC Antenatal care CAPI Computer‐Assisted Personal Interviewing EA Enumeration Area FANTA Food and Nutrition Technical Assistance FFP Food For Peace HAZ Height‐for‐age Z‐score HDDS Household dietary diversity score IR Immediate result IMC International Medical Corps IPTT Indicator performance tracking table IYCF Infant and young child feeding MCH Maternal and child health NBI NestBuilders International MoHS Ministry of Health and Sanitation OIC Opportunities Industrialization Centers ORS Oral rehydration solution ORT Oral rehydration therapy PIRS Performance Indicator Reference Sheets SNAP Sustainable Nutrition and Agriculture Promotion SO Strategic Objectives USAID United States Agency for International Development WAZ Weight‐for‐age Z‐score WHZ Weight‐for‐height Z‐score WHO World Health Organization vi Survey Summary: at‐a‐glance PERFORMANCE INDICATOR Baseline Result Endline Result Strategic Objective 1: Reduced chronic malnutrition among children under five IMP 2 % of underweight children 0‐59 month of age (weight for age z‐ score (WAZ) <‐2 SD) (FFP SO 1) 20.3% 14.8%* IMP 3 % of stunted children 6‐59 months (height‐for‐age z‐score (HAZ) < ‐2 SD) (FFP SO2) 38.4% 32.3%* OC 1 % of acutely malnourished children 6‐59 months (weight for height z‐score (WHZ) < ‐2 SD or edema) 9.6% 6.5%* IR 1.1: Improved access to quality nutrition and health services for women and children aged 0‐59 months   OC2 % of children 0‐59 months regularly attending growth monitoring1 94.0% 23.6%* OC 3 % of pregnant women completing 4 antenatal visits2 53.2% 76.9* OC 4 Percent of deliveries with a skilled birth attendant 52.3% 91.3%* OC 5 Percent of mothers participating in postpartum visit from skilled attendant within two days of birth 34.2% 75.5%* OC 6 % of mothers receiving 3 postnatal visits from skilled attendant 38.6% 76.3%* OP 5 # of new‐borns (0‐3 month) receiving essential new‐born care3 55.7% 69.6%* OC 7 Percent of cases of child diarrhea treated in Hospital/Clinic/PHU 21.1% 82.6%* OC 7.1 Percent of cases of child diarrhea treated with ORS or zinc in USAID programs4 ‐  82.6% IR 1.2: Improved health and nutrition practices at the household level OC 9 % of children 0‐6 months of age exclusively breastfed   39.5% 75.6%* OC 10 % of caregivers demonstrating proper food hygiene behaviors 67.1% 56.1%* OC 11 % of caregivers demonstrating proper personal hygiene behaviors 68.9% 90.5%* Strategic Objective 2: Enhanced livelihoods for vulnerable people, especially women and youth IM 4 Average household dietary diversity score (HDDS) (FFP SO 4)   6.74 5.44* IM 6 Average # months of adequate household food provisioning (MAHFP) FFP S03) 7.74 9.67* OC 14.1          Average rural household revenue (Leones)5  Selected Crops 2,391,025 899,227*  Livestock 71,229 491,224*  Nonfarm 639,933 1,529,977*  Total 3,128,690 1,768,050* OC15 % of households with non‐farm income 63.4% 53.7%* OC16 % of farmers reporting use of association benefits4   ‐  14.9% OC17 % of farmers who have adopted at least three agricultural technologies4 ‐  25.5% vii IR 2.1: Increased agricultural productivity at the household level OC 19          Average yield per hectare (Kg/ha)  Lowland Rice 609.97 747.55*  Cassava 2,102.75 760.58*  Groundnuts 310.36 710.02*  Sweet Potatoes 3,581.31 580.59* OC 20 Average value of all selected crop sales (Leones)5 437,124 859,841* IR 2.3: Improved community assets OC 24 % of farmers using improved storage       Men 4.60% 19.8%*     Women 4.70% 17.8%* * Significantly different from baseline level 1 The measurement of this indicator differed between baseline and endline making direct comparisons impractical (see Section 4.10) 2 Antenatal care indicator was measured for mothers of children 0‐59 months at baseline; 0‐23 months at endline as the project uses the PM2A approach 3 The methodology for measuring the essential newborn care indicator was modified for endline measurement (see Section 4.8) 4 Indicator not measured at baseline 5 USD$1 = Le. 6,000.00 1 1. Introduction to the Evaluation From June 2010, ACDI/VOCA and partners, the International Medical Corps (IMC) and Opportunities Industrialization Centers (OIC) launched a USAID Food for Peace (FFP) funded project; ‘Sustainable Nutrition and Agricultural Promotion (SNAP)’. The overarching goal of the SNAP project was to reduce food insecurity and increase resiliency among the most food insecure and vulnerable rural populations in target communities. SNAP therefore addressed the root causes of malnutrition, food insecurity, and vulnerability to shocks through improved health, hygiene, and nutrition practices as well as improved livelihood opportunities. Upon the completion of this project in June 2016, an external evaluation commissioned by ACDI/VOCA was undertaken in Sierra Leone by NestBuilders International (NBI), an independent research consulting firm. This population based Endline survey of ACDI/VOCA’s SNAP project measured four key impact indicators and fifteen outcome indicators in order to statistically compare Endline survey results against baseline findings. This report presents an independent final evaluation of the project’s progress in reaching its goals and objectives.   2. ‘SNAP’ Program Description 2.1 Program Activities The SNAP project was organized under two Strategic Objectives (SO):  SO1 aims to reduce chronic malnutrition among children under five  SO2 aims to enhance livelihoods among women and youth in the project communities In addition, the project addressed five crosscutting themes: (1) resiliency to shocks, (2) productive youth, (3) gender equity, (4) environmental stewardship, and (5) good governance. SNAP provided the targeted population with the necessary tools to improve their food security situation by addressing the key factors influencing utilization, availability and access to nutritious and diverse foods. SNAP facilitated linkages among donor, government, and community activities to deepen impact and sustainability. Furthermore, SNAP’s sustainability strategy aimed to strengthen the capacity of communities, as well as public and private stakeholders.   With the 2014‐2015 Ebola outbreak in Sierra Leone; fear of the disease, coupled with restrictions under the State of Emergency within the country, along with the timing during the peak agricultural season impacted the economy and disrupted farming activities and markets ‐ particularly in remote areas. This led to a decline in nationwide economic activities with the potential to impact a wide variety of socioeconomic factors. SNAP activities suffered the same reality and was extended by one year to address the Ebola outbreak and its impact (with an adjusted implementation period of June 2010 to June 2016). 2 2.2 Geographic Coverage The SNAP project was implemented in 18 chiefdoms, spanning four districts, as per the following:  Kailahun District (Kissi Tongi, Kpeje West, Mandu, Penguia, Yawei)  Bombali District (Sella Limba, Sanda Loko, Gbanti Kamaranka)  Tonkolili District (Kafe Simira, Kalasongia, Yoni)  Koinadugu District (Diang, Folosaba Dembelia, Kassunko, Mango, Neya, Neini, Wara Bafodia) Figure 1: SNAP intervention areas 3. Survey Details 3.1 Purpose of the Survey The endline survey was conducted as part of the overall SNAP Final Evaluation. The primary objective of the survey is to assess the status of key quantitative indicators after program implementation. More specifically, the purpose of the endline survey is to measure endline levels of four (4) impact indicators and fifteen (15) outcome indicators listed in the indicator performance tracking table (IPTT) below.  The endline information is compared to the baseline values to measure the extent of changes over the period of SNAP project implementation and their statistical significance.   3.1.1 Indicators The SNAP project's strategic objectives (SOs) and intermediate results (IRs) are outlined in the IPTT.   Due to the geographical context and the importance of disaggregating data, the endline study findings are disaggregated by program district, and by gender (where applicable). Table 1 shows the summary IPTT indicators that are used to estimate program achievement compared to the baseline. The definitions of indicators and means of measurement are provided in the summary SNAP performance indicator reference sheet (PIRS) in Annex 1. 3 Table 1: Summary IPTT indicators PERFORMANCE INDICATOR Type of Respondent Main Disaggregation Strategic Objective 1: Reduced chronic malnutrition among children under five IMP 2 % of underweight children 0‐59 month of age (weight for age z‐score (WAZ) <‐2 SD) (FFP SO 1) Children 0‐59 months Child Sex, District IMP 3 % of stunted children 6‐59 months (height‐for‐age z‐ score (HAZ) < ‐2 SD) (FFP SO2) Children 6‐59 months Child Sex, District OC 1 % of acutely malnourished children 6‐59 months (weight for height z‐score (WHZ) < ‐2 SD or edema) Children 6‐59 months Child Sex, District IR 1.1: Improved access to quality nutrition and health services for women and children aged 0‐59 months   OC2 % of children 0‐3 years regularly attending growth monitoring Caregiver of index child   (0‐35 months) District OC 3 % of pregnant women completing 4 antenatal visits Caregiver of index child   (0‐23 months) District OC 4 Percent of deliveries with a skilled birth attendant Caregiver of index child   (0‐59 months) Child Sex, District OC 5 Percent of mothers participating in postpartum visit from skilled attendant within two days of birth Caregiver of index child   (0‐59 months) District OC 6 % of mothers receiving 3 postnatal visits from skilled attendant Caregiver of index child   (0‐59 months) District OP 5 # of new‐borns (0‐3 month) receiving essential new‐ born care Caregiver of index child   (0‐3 months) Child Sex, District OC 7 Percent of cases of child diarrhea treated in Hospital/Clinic/PHU Caregiver of index child   (0‐59 months) Child Sex, District OC 7.1 Percent of cases of child diarrhea treated with ORS or zinc in USAID programs Caregiver of index child   (0‐59 months) Child Sex, District IR 1.2: Improved health and nutrition practices at the household level OC 9 % of children 0‐6 months of age exclusively breastfed   Caregiver of index child   (0‐5 months) Child Sex, District OC 10 % of caregivers demonstrating proper food hygiene behaviors Caregiver/Adult HH member District OC 11 % of caregivers demonstrating proper personal hygiene behaviors Caregiver/Adult HH member District Strategic Objective 2: Enhanced livelihoods for vulnerable people, especially women and youth IM 4 Average household dietary diversity score (HDDS) (FFP SO 4)   Caregiver/Adult HH member District IM 6 Average # months of adequate household food provisioning (MAHFP) FFP S03) HH Head/Adult HH member District 4 OC14.1 Average rural household revenue (Leones) HH Head/Adult HH member District, Source of revenue OC15 % of households with non‐farm income HH Head/Adult HH member District OC16 % of farmers reporting use of association benefits HH Head/Adult HH member District OC17 % of farmers who have adopted at least three agricultural technologies HH Head/Adult HH member District IR 2.1: Increased agricultural productivity at the household level OC 19 Average yield per hectare (Kg/ha) HH Head/Adult HH member District, Crop OC 20 Average value of all selected crop sales (Leones) HH Head/Adult HH member District, Crop IR 2.3: Improved community assets OC 24 % of farmers using improved storage   HH Head/Adult HH member District, Sex 3.2 Survey Methodology The overall survey design is a non‐experimental pre‐ and post‐test that mirrors the geographical disaggregation by district used at baseline. An inclusive population‐based sample survey was conducted for this quantitative evaluation to measure four (4) key impact indicators and fifteen (15) outcome indicators. The evaluation covers the five‐year period, including the one‐year extension due to the Ebola outbreak from June 2010 to June 2016, and follows a purely quantitative methodology.    3.3 Sampling Procedure The minimum required sample size for the endline was estimated based on the stunting indicator, as it was during the Baseline Survey.  The FANTA Sampling Guidelines1   were used to calculate a sample size capable of detecting a seven percent reduction in the child stunting indicator over the project lifespan. The minimum sample size required was computed as follows:   n = D {(Zα + Zβ) 2 * [((P1 (1 ‐ P1) + (P2 (1 ‐ P2)) /(P2 ‐ P1)2 ]} Where: n required minimum sample size per survey round D design effect = Default value of 2 P1 Stunting rate at baseline, 38.4% = 0.384 P2 the expected level of stunting at endline such that the quantity (P2‐P1) is the size of the magnitude of change it is desired to be able to detect, 38.4% ‐ 7% = 31.4% = 0.314 Zα  the Z‐score corresponding to the degree of confidence with which it is desired to be able to conclude that an observed change of size (P2‐P1) would not have occurred by chance (α ‐ the level of statistical significance), 95% = 1.645 Zβ  the z‐score corresponding to the degree of confidence with which it is desired to be certain of detecting a change of size  (P2‐P1) if once actually occurred (β ‐ statistical power), 80% = 0.840                                                              1 Magnani, Robert. 1999. Sampling Guide. Washington, D.C.: FHI 360/FANTA. 5 Based on these parameter values, the estimated sample size (n) was 1,140. Thus, the minimum required sample size per survey round for the entire program area is 1,140 children under five years of age (U5).   Considering that not all households have U5 children, the sample size was adjusted according to the Addendum to Fanta Sampling Guide to ensure that a sufficient number of U5 children were measured.2    First, the sample size was adjusted to inflate the required sample size by the inverse of the proportion of households that have at least one eligible child. Assuming that the average number of U5s in the population is 16.6 percent and the average households size is 5.93 , the total number of households required to be interviewed to reach 1,140 U5s is 1,825 households. Next, as the Endline survey sought to sample all eligible children in selected households, the sample size was deflated to 1,590 to account for households that contribute two or more children toward the overall required sample size of children. This sample size is adequate to detect a 7% reduction in the stunting rate of children U5 at the program level.   To account for the possibility of non‐response, a 10 percent non‐response factor is applied to the minimum required sample size, to get a target sample size for the survey (number of households to be selected) of 1,750 households.   Selecting the Sample The sampling frame was constructed from the 2015 Sierra Leone General Population and Housing Census enumeration areas (EAs) in the selected districts for each program chiefdom. The EA is the lowest census administrative level. Based on the endline target sample size of 1,750 households, NBI proposed a design with 108 clusters of 17 households each.   The sample selection was done in four stages. For the first stage, the same 108 EAs selected for the baseline survey were retained for the endline survey (see Annex 2 for details on surveyed EAs). From a statistical point of view, retaining the same clusters was the preferred strategy.4 A separate sample of alternate EAs was selected using simple random sampling in case an EA in the primary sample was inaccessible and needed to be replaced. The sample of alternate EAs was used as a back‐up from which individual replacement EAs were drawn as needed.   The second‐stage selection of dwellings was completed using a random‐walk method selecting the starting point from a boundary (EA) map prior to the start of fieldwork. The random‐walk method is used in EPI (expanded program of immunization) cluster surveys and thus is relatively widely known. The method entails (1) randomly choosing a starting point and a direction of travel within a sample                                                              2 Stukel, Diana; and Deitchler, Megan. 2012. Addendum to FANTA Sampling Guide by Robert Magnani (1999): Correction to Section 3.3.1 Determining the Number of Households That Need to be Contacted. Washington, DC: FHI 360/ FANTA. 3 Figures for both the average household size and the proportion of children <5 years of age were obtained from the 2013 National Demographic and Health Survey (DHS) as the most recent available census data dates back to 2004. 4   Magnani, Robert. 1999. Sampling Guide. Washington, D.C.: FHI 360/FANTA. 6 cluster, (2) conducting an interview in the nearest household, and (3) continuously choosing the next nearest household for an interview until the target number of interviews has been obtained.5 Using the most recent census EA maps, a number of possible starting points were selected at different, easily identifiable locations, and from these a starting point was randomly chosen. The advantage is that survey coordinators choose the starting point before the field work began, reducing any risk of bias that might arise when starting points are chosen on the basis of convenience as opposed to randomly.6 Households in which no survey was conducted due to absence or refusals after three attempts were not replaced; therefore, the target of 17 households per cluster was not always achieved.  Details on the number of sampled households with completed interviews for each program district and chiefdom are provided in Annex 2. The third stage of sampling involved selection of one household when multiple households were living in one dwelling unit or compound. For sampling purposes, a dwelling and a compound were considered the same thing, such as in polygamous situations, where more than one household could be present in the dwelling (or compound). In cases where we found that several polygamous households occupied the dwelling, the interviewer used a Kish grid to randomly select one of the households.   The fourth stage of sampling was done at the individual level to select survey respondents. The household survey questionnaire is broken into several modules with different respondents according to the endline indicators. The four respondent groups targeted were (1) head of household or responsible adult, (2) primary caregiver or mother of children under five years of age7 , (3) household primary caregiver and (4) children under the age of five. Logically, not all respondent groups were present in each sampled household. For the agriculture module, the household head or an informed family member was interviewed. For the child and maternal health questions, the mother or caregiver of youngest child under the age of five in the household was selected. If there was no child under the age of five in the household, these questions were not answered. For household nutrition and hygiene questions, the primary caregiver was interviewed. Lastly, for the anthropometric measurements, all children under the age of five were included. 3.4 Survey Instrument Survey data was collected through face‐to‐face interviews using a well‐structured questionnaire (see Annex 3). The quantitative tool was initially developed by SNAP in collaboration with  USAID. The data collection tool administered at baseline was used for reference, and adapted to reflect changes USAID had made to a limited number of indicators since the SNAP baseline survey was conducted (see Annex 4 for a description of these indicator changes). These indicator changes were taken into account when finalizing the endline questionnaire and are highlighted were relevant in the report.                                                              5 Magnani, Robert. 1999. Sampling Guide. Washington, D.C.: FHI 360/FANTA. 6 Ibid 7 The terms “children under 5” and “children aged 0‐59 months” are used interchangeably in this report.    7 The questionnaire consisted of separate modules and sections as highlighted in Table 2. Survey questions were largely adapted using questions from the FFP Standard Indicators Handbook. 8 The final endline questionnaire was adapted into a computer‐assisted personal interviewing (CAPI) survey using SurveyCTO to allow for mobile data collection.   Table 2: Survey modules and target respondents   Survey Module Survey Section/Related Indicators Surveyed Respondent Module 1 Household Participation in SNAP Activities Primary adult male/female decision‐maker (i.e. household head, principle farmer or primary income earner) Household Food Self Provisioning (MAHFP) Improved Agricultural Productivity Use of FSS Cluster Benefit Module 2 Improved Use of Maternal and Child Health Services (Children 0‐59 months) Mother or caregiver of youngest child in the household.  If there are no children 0‐59 in the household, these survey sections were not asked Child Feeding Practices/Exclusive Breastfeeding (Children 0‐5 months) Household Dietary Diversity Score (HDDS) Mother or caregiver of youngest child in the household.  If there are no children in the household: the primary caregiver for the household was selected (i.e.  primarily responsible for overseeing household chores, cooking, hygiene, etc.)   Food hygiene and Personal Hygiene Module 3 Anthropometric Data All children in the household aged 0‐59 months The finalized draft questionnaire (with revisions made based on inception report feedback) was piloted in a non‐sampled EA to test the survey tool, field logistics and other issues concerning the fieldwork implementation. Necessary changes needed to improve survey functionality were made and the data collection tools and procedures were finalized for the field staff training workshop and data collection.    3.5 Training of Field Staff The NBI core evaluation team was composed of a Team Leader and a Research Manager with expertise in social research methods, agriculture and nutrition, as well as evaluative data collection/analysis. A total of 23 skilled and qualified NBI field enumerators and six supervisors conducted primary data collection at community‐level.   NBI maintains an extensive database of field staff who have undertaken numerous national and regional multi‐topic surveys. Field staff that have ties in the identified research districts and communities were recruited as it ensured familiarity with communities, local languages, customs and roadways.                                                                8 USAID. (2011). Food for Peace Standard Indicators Handbook (Baseline‐Final Indicators). Available at http://pdf.usaid.gov/pdf_docs/pnadz580.pdf. 8 Furthermore, NBI worked with trained health professionals from the Ministry of Health and Sanitation (MoHS) to conduct anthropometric data collection. Health professionals with facility‐level experience conducting anthropometric measurements were recruited to join training and data collection alongside NBI's field staff. MoHS staff were recruited from the districts and chiefdoms sampled for the endline to ensure familiarity with the survey target population. Prior to data collection, NBI organized and facilitated a five‐day training workshop in Freetown. All field staff were required to participate in the workshop which focused on the study background, sampling procedures, interviewing techniques and familiarization with the data collection tool.   NBI also facilitated a simultaneous two‐day session for the MoHS staff who conducted anthropometric data collection. Training for the MoHS field staff specifically focused on:  Training and review on taking anthropometric measurements9  How to properly use the hand‐held device for data collection  Field procedures (sampling, building rapport, recording progression of work, supervision, etc.) In order to minimize any data collection errors, ample training time ensured that all field staff understood the overall objectives of the assignment and their roles and responsibilities. Practice sessions, group discussions and mock interviews also ensured that all interviewers asked questions correctly and consistently, and that data abstraction was reliably done. During the overview of the questionnaire and practice sessions, participants conducted local language translations to questions in order to ensure that valid data was collected from the field. In addition, the field supervisors were trained on how to ensure the quality of data gathered by monitoring research in the field and conducting observations and back checks in line with the established quality assurance measures and code of conduct.   3.6 Survey Logistics Following the training workshop, fieldworkers were deployed to the field to conduct household‐level data collection. Field staff were supported by NBI’s Team Leader and Research Manager, who also spent time in the field to supervise the field exercise. Data collection took place from August 15 ‐ 27, 2016. For quality control, supervisors kept fieldwork control sheets and conducted random spot checks to assess the enumerator's proper administration of the various sections of the questionnaire, and their general adherence to professional standards. As a part of this quality control process, supervisors also conducted back checks with at least 15 percent of all interviews conducted to verify that an interview was completed and to ask several questions from the interview to measure the reliability of responses. Supervisors also reviewed every completed questionnaire on the same day of data collection and checked for adequate completion of all fields in the questionnaires, missing data and legibility of open‐ended items. Enumerators were required to make corrections or return for re‐interview, if necessary.                                                              9 The USAID 'Anthropometric Indicators Measurement Guide' was used to guide this portion of the training. (See: Cogill, Bruce. 2003. Anthropometric Indicators Measurement Guide. Washington, DC: Food and Nutrition Technical Assistance (FANTA) Project, FHI 360).   9 Android phones were used for data collection, complemented with SurveyCTO software. The use of mobile devices and an electronic questionnaire strengthened data quality by allowing data validation rules and consistency checks that were integrated in the SurveyCTO software program. The mobile‐ based data collection process eliminated the data entry burden, as data was entered at the enumerator level and records were uploaded to a cloud server. This allowed the data analysis team to review data consistency periodically throughout the data collection phase. 3.7 Data Entry and Analysis The SurveyCTO dataset (CSV format) was converted into an SPSS (Version 21) database for data management and analysis. SPSS statistical software was used to analyze the dataset, supplemented by ENA software for the anthropometric data analysis using WHO 2006 child growth standards. The data analysis and tabulation followed the definition of the indicators in the IPTT and baseline data analysis (where possible) to allow for valid comparisons of indicator values. All FFP indicators were calculated using tabulation methods documented in the FFP Standard Indicators Handbook. As per the IPTT and PIRS, the analysis includes descriptive statistics to summarize and describe the changes in the four key impact indicators and fifteen outcome indicators since baseline measurement. Endline data have been disaggregated according to the levels required by USAID/FFP. Furthermore, endline indicator estimates have been presented along with the confidence intervals associated with the indicators; and have been compared to baseline data using appropriate statistical test to detect a population level difference at 95% level of significance (see Table 18, Section 5) 3.8 Limitations of the Survey This section summarizes study limitations and issues encountered during the endline survey. Logistics and Transportation Constraints Field teams experienced significant challenges with the geography and road conditions in some sampled EAs. Heavy rains throughout the fieldwork period meant that two EAs in Koinadugu district and three EAs in Tonkolili district were replaced due to flooding and impassable rivers and roads. Comparability to Baseline Indicators Changes to indicator definitions and measurements following the SNAP baseline survey meant that comparisons between baseline and endline values were not possible for a number of indicators (e.g. ORT therapy for child diarrhea cases, growth monitoring, etc.). While these changes to the indicators were taken into account during data analysis and are highlighted were relevant in the report, comparisons to baseline data in these cases must be viewed with caution. (See Annex 4 for details on indicator differences).   Geographic Location of Surveyed EAs at Baseline versus SNAP Operational Areas Following the baseline survey, a number of planned targeted SNAP communities and health centres were dropped from program implementation due to low utilization and inaccessible roadways. This meant that four EAs in Koinadugu district sampled at baseline were no longer a part of the project's operational areas and had to be replaced from the list of replacement EAs for endline data collection prior to the start of fieldwork. 10 Validity and Reliability of Self‐reported Data   Most of the data collected for the indicators related to agricultural productivity, sales and yields; as well as maternal and child health rely on self‐reporting. Self‐reporting has several limitations, such as the possibility of exaggeration or omission of information, inaccurate recollection of experiences or events, social‐desirability bias or reporting of untruthful information and reduced validity when respondents do not fully understand a question. Thus findings (especially in regards to the agricultural data which rely on numeracy skills) should be interpreted with caution.    11 4. Endline Survey Results The endline survey findings are organized in six content categories: (1) characteristics of the population; (2) household participation in SNAP project activities; (3) children’s health and nutrition indicators; (4) maternal and child healthcare indicators; (5) household indicators, which include food access indicators and WASH indicators; and (6) agricultural and livelihood indicators. Each section includes results for FFP and program‐specific indicators. The findings are presented as per the required disaggregations.   4.1 Characteristics of the Study Population The SNAP endline survey completed interviews with 1,816 households. Table 3 shows the characteristics of these households. The average household includes 6.4 household members. Children under five years of age are household members in just over three‐quarters (78.7%) of all households. Nearly seven out of ten (69.8%) heads of household have received no formal education; while 4.2% have completed primary school and 2.7% have completed secondary education.    The majority of households (91.8%) included an adult male and female and 7.8% of households included an adult female but no adult male. Table 3: Household characteristics Household Characteristic Total Average household size (number of persons) 6.4 Percent of households with children under five years 78.7% Household headship (respondent percent male) 69.6% Education level of head of household (percent of households) No formal education 69.8% Some primary 5.1% Completed primary 4.2% Some secondary 15.0% Completed secondary 2.7% Vocational or university 3.1% Gendered household type (percent of households) Both male and female adults 91.8% Adult female with no adult male 7.8% Adult male with no adult female 0.4% Children no adults 0.0% 4.2 Household Participation in SNAP Project Activities In order to identify project beneficiaries, survey participants were asked if they, or anyone in their household, had ever participated in SNAP project activities. Overall, 41.2% of the sampled households reported that they had ever participated any SNAP activity. Among these households, 39.4% reported that they had participated in at least one of the following SNAP intervention's: (1) SNAP Farmer Field School/agriculture activity; (2) SNAP Mother Care Group/child care or nutrition activity; (3) literacy and numeracy or vocational training; (4) SNAP village savings and loans association (VSLA);  or (5)  had received food from SNAP. 12 Overall, the majority of surveyed households  reported that they had participated in a SNAP Farmer Field School/agriculture activity (28.1%),  a SNAP Mother Care Group/child care or nutrition activity (27.8%) or had received food from SNAP (27.5%). Analysis by location revealed that households in Tonkolili district were generally less likely to report participation in any SNAP activities when compared to households in the other program districts. Furthermore, whereas households in Bombali district were most likely to report receiving food assistance, they were least likely to report participation in literacy and numeracy or vocational training.   Table 4: Type of SNAP assistance received by all surveyed households, by district Member of a SNAP Farmer Field School OR other SNAP agriculture activity Member of SNAP Mother Care Group OR other SNAP activity about child care or nutrition Participated in SNAP Literacy and Numeracy or Vocational Training Member of SNAP VSLA Group Received food from SNAP District Bombali 32.4% 30.9% 2.2% 14.7% 38.3% Kailahun 28.4% 31.2% 13.7% 18.4% 24.7% Koinadugu 35.5% 31.6% 20.0% 29.6% 29.0% Tonkolili 15.9% 17.5% 5.3% 8.6% 17.9% TOTAL (among all surveyed respondents) 28.1% 27.8% 10.3% 17.9% 27.5% Among beneficiary households, the majority reported that they had participated in a SNAP Farmer Field School/agriculture activity (68.7%),  a SNAP Mother Care Group/child care or nutrition activity (68.0%) or had received food from SNAP (67.2%). Analysis by location revealed that households in Koinadugu district were most likely to report agriculture assistance (73.8%); whereas as those in Bombali were most likely to report receiving food (81.4%). Figure 2: Type of assistance received by SNAP beneficiaries 68.7% 68.0% 67.2% 43.7% 25.3% SNAP farmer field school / agriculture activity SNAP mothercare group / childcare activity SNAP food assistance SNAP VSLA group SNAP literacy & numeracy / vocational training 13 Children's Health and Nutrition Indicators 4.3 Anthropometrics Reducing malnutrition among children under five years of age is a key goal of the SNAP project, and measuring changes in the anthropometrics of the target population is a major focus for evaluating the impact of the project. Child malnutrition is primarily measured using three indictors: stunting, wasting and underweight.   To this end, data were collected on the age, height and weight of all children under five years of age in the 78.7% of sampled households which had children aged 0‐59 months. Overall, 1,756 children had their anthropometric measurements taken ‐ meeting the required minimum sample size.  These measurements were used to calculate the following three indicators:  Prevalence of underweight children under five years of age (weight‐for‐age)    Prevalence of stunted children 6‐59 months (height‐for‐age)    Prevalence of acutely malnourished children 6‐59 months (weight‐for‐height) Anthropometric measurements were taken using standardized procedures and compared with the 2006 WHO child growth standards. These standards are based on an international sample of ethnically, culturally and genetically diverse healthy children living under optimum conditions that are conducive to achieving a child’s full genetic growth potential.10 Use of the 2006 WHO child growth standards is based on the finding that well‐nourished children of all population groups for which data exist follow similar growth patterns before puberty. 4.3.1 Results for Underweight (Weight‐for‐Age) Underweight, or weight‐for‐age, is an overall indicator of a population's nutritional status.  Weight‐ for‐age identifies children who are of inadequate weight compared to a healthy, well‐nourished child of the same age. It is a measure that reflects both stunting and wasting, reflecting both past (chronic) and/or acute under nutrition.11 The prevalence of underweight children is calculated by first combining weight and age data to compute a child's weight‐for‐age z‐score (WAZ). Children whose WAZ is below minus two standard deviations (‐2 SD) from the median of the WHO reference population are classified as underweight. According to endline findings, there has been a significant reduction of 5.5%  in the prevalence of underweight children across the sampled SNAP operational chiefdoms (from 20.3% at baseline to 14.8% at endline). The gender disaggregated data shows that the reduction in underweight was higher among female children (‐6.3%) then the male children (‐4.5%). Overall, all districts reported a reduction in the prevalence of underweight among sampled children. However, despite having the greatest reduction rate, Koinadugu district continues to have higher than average rates of underweight children (found in both the baseline and endline survey).                                                              10 WHO Multicentre Growth Reference Study Group. (2006). WHO Child Growth Standards: Length/height‐for‐ age, weight‐for‐age, weight‐for‐length, weight‐for‐height and body mass index‐for‐age: Methods and development. Geneva: World Health Organization.    11 See Cogill, Bruce (2003). Anthropometric indicators measurement guide. Food and Nutrition Technical Assistance Project, Academy for Educational Development, Washington, D.C.    14 Table 5: Prevalence of underweight children under 5 years by sex and district, baseline vs. endline Baseline Endline Change(+/‐) Child Sex Male 22.6% 18.1%** ‐4.5% Female 18.0% 11.7%** ‐6.3% District Bombali 21.8% 14.9% ‐6.9% Kailahun 19.0% 13.1% ‐5.9% Koinadugu 23.7% 16.5% ‐7.2% Tonkolili 16.7% 14.6% ‐2.1% TOTAL 20.3% 14.8%*** ‐5.5% *** p<.001 4.3.2 Results for Stunting (Height‐for‐Age) Stunting is a form of growth failure. A child who is stunted often appears to be normally proportioned but is actually shorter than normal for his/her age. Stunting starts before birth and is caused by poor maternal nutrition, poor feeding practices, poor food quality as well as recurrent and chronic illness. Stunting, therefore, represents the long‐term effects of malnutrition and is not sensitive to recent, short‐term changes in dietary intake (such as those related to the season of data collection). Stunting is calculated by first combining height and age data to compute a child's height‐for‐age z‐ score (HAZ). Children whose HAZ is below minus two standard deviations (‐2 SD) from the median of the WHO reference population are considered to be stunted, or chronically malnourished.   Survey results show that at endline,  32.3% of children 6‐59 months of age were stunted. This suggests a significant reduction of ‐6.1% in the stunting rate in the SNAP intervention area over the six years of project implementation (from 38.4% at baseline). While there was a noted reduction in stunting rates across all of the SNAP operational districts, the reduction was significantly higher in Bombali (‐8.4%) and Kailahun (‐8.0%) districts, compared to Koinadugu (‐4.4%) and Tonkolili (‐2.9%). Again, sampled children in Koinadugu district recorded higher than average rates of stunting. Futher, the reduction in stunting was higher among male children (‐7.7%), compared to female children        (‐4.4%). Lastly, as measurements were taken for all children under the age of five, the prevalence of stunting among children 0‐59 months was found to be 30.4% (33.1% for boys and 27.8% for girls). Table 6: Prevalence of stunted children 6‐59 months by sex and district, baseline vs. endline Baseline Endline Change(+/‐) Child Sex Male 42.8% 35.1%* ‐7.7% Female 34.0% 29.6%* ‐4.4% District Bombali 37.0% 28.6% ‐8.4% Kailahun 40.0% 32.0% ‐8.0% Koinadugu 43.2% 38.8% ‐4.4% Tonkolili 34.5% 31.6% ‐2.9% TOTAL 38.4% 32.3%*** ‐6.1% *** p<.001 4.3.3 Results for Malnutrition (Weight‐for‐Height) The last measure of malnutrition is wasting, or weight‐for‐height. At an individual level the weight‐ for‐height index measures body mass in relation to body height or length, and describes current nutritional status.  Weight‐for‐height  is the most commonly used nutrition index in emergency nutrition surveys as it represents the failure to receive adequate nutrition in the period immediately 15 preceding the survey ‐ which may be the result of inadequate food intake or a recent episode of illness causing loss of weight and the onset of malnutrition. If the weight‐for‐height z‐score (WHZ) is less than ‐2 standard deviations below the median of the (adequately nourished) reference population, the child is considered to be wasted. Overall, the wasting rate shows significant improvements at endline: decreasing from 9.6% at baseline, to 6.5% at endline among children aged 6‐59 months. The gender disaggregated data shows that the reduction in malnutrition was higher among female children (‐4.3%) than male children (‐1.9%). Furthermore, while sampled children in all SNAP surveyed districts recorded a reduction in wasting, the reduction rate was smallest in Tonkolili district which had higher than average rates of malnutrition at 7.7% compared to other districts.   Table 7: Prevalence of malnutrition among children 6‐59 months by sex and district, baseline vs. endline Baseline Endline Change(+/‐) Child Sex Male 9.3% 7.4% ‐1.9% Female 9.9% 5.6% ‐4.3% District Bombali 9.9% 6.8% ‐3.1% Kailahun 7.8% 5.2% ‐2.6% Koinadugu 10.8% 5.6% ‐5.2% Tonkolili 9.8% 7.7% ‐2.1% TOTAL 9.6% 6.5%*** ‐3.1% *** p<.001 4.4 Infant and Young Child Feeding Practices Child nutritional status is directly related to infant and young child feeding (IYCF) practices. Breastfeeding is one such important practice which can predict the future health of children.  Using the indicators created by the WHO (2008)12, the following IYCF practices were measured in SNAP project areas:    Percent of children breastfed within first hour of birth  Percent of children under 6 months old exclusively breastfed 4.4.1 Early Initiation of Breastfeeding Provision of mother’s breast milk to infants within one hour of birth is referred to as 'early initiation of breastfeeding' and ensures that the infant receives the colostrum, or 'first milk' which is rich in protective factors. The endline survey collected information on children under six months of age who were ever breastfed and who were breastfed in the first hour after birth. As shown in Figure 3, almost all surveyed children under the age of six months have reportedly ever been breastfed (98.2%). This is a significant increase from the baseline figure of 69.4%.13 Furthermore, the reported prevalence of early initiation of breastfeeding has also substantially increased from 54.6% at baseline to 86.6% at endline.                                                              12 WHO (2008). Indicators for assessing infant and young child feeding practices. Part I: Definitions. World Health Organization, Geneva.    13 It is important to note that the baseline report does suggest that survey finding on breastfeeding were largely inconsistent.   16 Figure 3: Initial breastfeeding for last birth among children aged 0‐6 months, baseline vs. endline 4.4.2 Exclusive Breastfeeding As an important predictor of future health, the WHO recommends that children be exclusively breastfed (with no other liquid or solid food or plain water given) during the first six months of life. Substitutes, such as formula, other kinds of milk and porridge, provide too few calories and essential nutrients and can easily be contaminated during preparation. It is believed that introducing breast milk substitutes to infants before six months of age can limit breastfeeding, which has negative implications for a child’s health and development.   Table 8 compares the results of the baseline and endline survey for the prevalence of exclusive breastfeeding in the 24 hours preceding the survey for children 0‐6 months surveyed (n=225). Breastfeeding in the preceding 24 hours serves as a proxy for long‐term breastfeeding behavior, which is difficult to measure due to recall issues. According to survey findings, three quarters (75.6%) of children under the age of six months were reportedly exclusively breastfed in the 24 hours preceding the endline survey. This is a significant improvement from the baseline findings where only 39.5% of children under the age of six months were found to be exclusively breastfed. According to the district disaggregated data, more children in the Bombali SNAP program area are exclusively breastfed (82.0%) compared to other districts. However, it is worthwhile to note that the largest increase in exclusive breastfeeding was reported in Tonkolili district (+46.1%). Table 8: Percent of children aged 0‐6 months exclusively breastfed by sex and district, baseline vs. endline Baseline Endline Change(+/‐) Child Sex Male n/a 73.6% n/a Female n/a 77.3% n/a District Bombali 45.7% 82.0% +36.3% Kailahun 54.7% 77.3% +22.6% Koinadugu 34.0% 64.3% +30.3% Tonkolili 29.5% 75.6% +46.1% TOTAL 39.5% 75.6%*** +36.1% n/a = not collected at baseline *** p<.001 69.4% 54.6% 98.2% 86.6% Ever breastfed Breastfed within one hour after birth Baseline Endline 17 4.5 Diarrhea and Oral Rehydration Therapy The baseline and endline surveys collected and analyzed data on diarrhea treatment for children under the age of five years. At baseline, the indicator focused on the percent of child diarrhea cases treated in a healthcare facility. The IPTT was later revised to include an indicator on the percent of child diarrhea cases treated with oral rehydration therapy (ORT) or zinc supplements. Therefore, the endline survey expanded data collection and analysis to measure the number of children with diarrheal illness receiving the required treatment ‐ an indicator which was not measured at baseline. In order to collect data on the incidence of child diarrhea cases and treatment, caregivers were asked whether the index child under five years of age had diarrhea at any time during the two‐week period preceding the survey. If the child had diarrhea, the caregiver was asked whether they sought advice or treatment, and whether ORT was given to the child. 4.5.1 Incidence of Child Diarrhea Cases As shown in Figure 4, there has been a significant decrease in diarrheal prevalence from 29.9% at baseline to 6.0% percent at endline among surveyed children aged 0‐59 months (n=86). This large decrease must be viewed with caution as the baseline findings were well above 16.0% average percentage of children aged 0‐59 months who reportedly had diarrhea across all rural areas in Sierra Leone in the two weeks preceding the 2010 Multiple Indicator Cluster Survey (MICS) in Sierra Leone.14  It is also important to note that the baseline and MICS were conducted in the dry season (which may be linked to higher incidences of diarrhea), while the endline survey was conducted in the rainy season.    Figure 4: Percentage of children aged 0‐59 months with diarrhea in the last two weeks 4.5.2 Treatment of Child Diarrhea Cases at a Healthcare Facility Results in Table 9 show there has been a significantly large increase in the treatment of child diarrhea cases at an appropriate healthcare facility from 21.1% at baseline to 82.6% at endline. The noted increase may largely be explained by the existence of government programs and projects supported by development partners aimed at promoting the Sierra Leone Free Health Care Initiative (FCHI) which was implemented in April 2010. The objective of the FHCI is to provide free health care to pregnant and lactating mothers and children under five years of age. Under the initiative,                                                              14 Statistics Sierra Leone and UNICEF‐Sierra Leone. 2011. Sierra Leone Multiple Indicator Cluster Survey 2010, Final Report. Freetown, Sierra Leone: Statistics Sierra Leone and UNICEF‐Sierra Leone. 29.9% 16.0% 6.0% Child diarrhea case in the last 2 weeks Baseline (2011) MICS (2010) Endline (2016) 18 pregnant women, mothers and young children can receive free maternal and child healthcare at any public clinic or hospital within the country.   Table 9: Percent of child diarrhea cases (n=86) treated in Hospital/Clinic/PHU by sex and district, baseline vs. endline   Baseline Endline Change(+/‐) Child Sex Male n/a 81.8% n/a Female n/a 83.3% n/a District Bombali 23.1% 72.2% +49.1% Kailahun 21.3% 100.0% +76.9% Koinadugu 21.4% 92.3% +70.9% Tonkolili 18.5% 65.0% +46.5% TOTAL 21.1% 82.6%*** +61.5% n/a = not collected at baseline *** p<.001 4.5.3 Treatment of Child Diarrhea Cases with ORS or Zinc Dehydration caused by severe diarrhea is a major cause of morbidity and mortality among young children, although the condition can be easily treated with oral rehydration therapy (ORT). Types of ORT include oral rehydration solution (ORS) or homemade sugar‐salt water solution. Routine therapy also includes zinc supplements. Overall, among caregivers who reported a case of child diarrhea, 82.6% reported that they had treated the diarrhea with ORS or Zinc supplements in the two weeks preceding the survey. The prevalence of ORT was highest in Kailahun district (100.0%) and lowest in Koinadugu district (76.9%).   Figure 5: Percent of child diarrhea cases (n=86) treated with ORS or ZINC, by sex and district Maternal and Child Healthcare Indicators Access to quality healthcare services for mothers and children is important for ensuring maternal health and reducing maternal and infant morbidity and mortality. The endline survey collected information on maternal and child healthcare from women who had given birth to at least one child in the five years preceding the survey.  In this section, indicators related to antenatal care, delivery, postnatal care and growth monitoring are presented. 88.9% 100.0% 76.9% 80.0% 84.1% 81.0% 82.6% Bombali Kailhaun Koinadugu Tonkolili Males Females Overall Bombali Kailhaun Koinadugu Tonkolili Males Females Overall 19 4.6 Antenatal Care The antenatal care (ANC) policy in Sierra Leone follows the WHO approach to promoting safe pregnancies, recommending at least four ANC visits with a skilled attendant for women without complications. Women with a child between 0‐23 months of age were asked about the care they received while pregnant with that child, including how many visits they made. According to endline survey results,76.9% of mothers of children 0‐23 months had four or more antenatal visits when they were pregnant with their youngest child. The percent of mothers who reported four or more visits was substantially lower in Bombali district (61.4%) compared to the other districts. Figure 6: Percent of mothers of children 0‐23months who had four or more ANC visits, by sex and district Comparisons to baseline findings are problematic as the baseline survey measured the ANC indicator for all children 0‐59 months, whereas the endline survey limited the data analysis to children 0‐23 months (as per the PM2A project approach). However, to allow for comparisons, endline survey data for children 0‐59 months is presented in Table 10. The proportion of mothers with children 0‐ 59 months that received four ANC visits is very similar to the findings for mothers with children 0‐23 months. Overall, the percent of these mothers who reported four or more visits during pregnancy increased from 53.2% at baseline to 77.3% at endline, also a significant increase. The noted increase in ANC coverage must be viewed within the larger context of the FHCI ‐ which began in 2010 and provided pregnant women with free maternal health care at any public clinic or hospital.   Table 10: Percent of mothers of children 0‐59 months who had four or more ANC visits by district, baseline vs. endline   Baseline Endline Change(+/‐) District Bombali 52.1% 61.7% +9.6% Kailahun 58.4% 78.2% +19.8% Koinadugu 53.1% 84.1% +31.0% Tonkolili 49.3% 84.9% +35.6% TOTAL 53.2% 77.3%*** +24.1% *** p<.001 61.4% 79.4% 81.3% 84.0% 74.6% 79.2% 76.9% Bombali Kailhaun Koinadugu Tonkolili Males Females Overall Bombali Kailhaun Koinadugu Tonkolili Males Females Overall 20 4.7 Skilled Birth Attendant Attendance at birth by a medically trained professional provides the opportunity to administer life‐ saving preventive and curative care at the time of greatest vulnerability for the mother and the newborn. Therefore, delivery under the supervision of trained health providers can reduce the risk of complications and infections that may cause death or serious illness to the mother or child. In the context of Sierra Leone, a 'skilled birth attendant' includes: a doctor, nurse, midwife, maternal and child health (MCH) aide or a community health officer (CHO). Table 11 reveals that the proportion of mothers whose last delivery was assisted by a trained health professional has increased significantly from 52.3% at baseline to 91.3% at endline. The increase was observed across all SNAP project areas, with Kailahun district reporting near complete coverage of skilled birth attendance at 99.7%. Again, these gains in maternal health must be understood within the wider context of the FHCI.   Table 11: Percent of births with a skilled birth attendant by sex and district, baseline vs. endline   Baseline Endline Change(+/‐) Child Sex Male n/a 90.4% n/a Female n/a 92.3% n/a District Bombali 54.8% 95.0% +40.2% Kailahun 70.8% 99.7% +28.9% Koinadugu 45.1% 86.1% +41.0% Tonkolili 39.6% 86.0% +46.4% TOTAL 52.3% 91.3%*** +39.0% n/a = not collected at baseline *** p<.001 4.8 Essential Newborn Care Studies have shown that many newborn lives can be saved by the use of simple interventions immediately after birth. Referred to as 'essential newborn care' these interventions include: immediate and thorough drying, skin to skin contact of the newborn with the mother, cord clamping and cutting after the first minutes after birth and early initiation of breastfeeding.   The measurement of essential newborn care differed greatly between the baseline and endline survey ‐ making direct comparisons impractical. Whereas the baseline survey simply asked: 'Did respondent use any essential newborn care (including thermal care)' to  mothers with children aged 0‐1 months; the endline survey applied a much more nuanced approach. First, any child 0‐3 months who was born in a health facility and had an under‐five card with their birth weight recorded on it was assumed to have received essential newborn care (as per the established essential package of health service protocols in Sierra Leone). Next, in the case where a child was not born in a health facility and/or did not have their birth weight recorded, the mother was asked a series of questions to ascertain if the child had received clean cord care, thermal care and breastfeeding immediately after birth.15                                                                15 Comparisons between baseline and endline data are further complicated by the fact that the baseline survey had over 50% of missing data on this indicator, and limited data collection and analysis to children aged 0‐1 month, while the indicator is concerned with children 0‐3 months.  Lastly, the low number of children aged 0‐1 month in the endline survey (n=28) makes it difficult to generalize findings; while significance testing revealed 21 As highlighted in Figure 7 below, approximately seven out of ten (69.6%) newborns aged 0‐3 months reportedly received essential newborn care. Interestingly, newborns in Koinadugu district were significantly less likely to receive essential newborn care (33.3%) compared to other SNAP operational districts. Figure 7: Percent of newborns (0‐3 months) receiving essential newborn care, by sex and district 4.9 Postnatal Care A large proportion of maternal and neonatal deaths occur during the first 48 hours after delivery. Thus, postnatal care is important for both the mother and child. The endline survey gathered data on the following indicators related to postnatal care across the SNAP project areas:    Percent of mothers participating in postpartum/newborn visits within two days of birth  Percent of mothers receiving three postnatal visits Overall, there was a noted increase in both indicators. The proportion of mothers who reported receiving a postpartum visit within two days of their most recent birth by a skilled health worker (i.e. doctor, nurse, midwife, MCH aide or CHO) increased from 34.2% at baseline to 75.5% at endline (representing a 41.3% increase). Similarly, the proportion of mothers who reported receiving at least three postnatal visits after their last birth from a skilled health worker increased from 38.6% at baseline to 76.3% at endline.                                                                                                                                                                                           that comparisons between baseline and endline values are not statistically significant for this indicator. For these reasons, the endline data on essential newborn care is presented without comparisons being made to baseline findings. 65.5% 83.3% 33.3% 87.8% 65.3% 73.0% 69.6% 0.0% 10.0% 20.0% 30.0% 40.0% 50.0% 60.0% 70.0% 80.0% 90.0% 100.0% Bombali Kailhaun Koinadugu Tonkolili Males Females Overall Bombali Kailhaun Koinadugu Tonkolili Males Females Overall 22 Table 12: Postnatal care received  by mothers by child sex and district, baseline vs. endline   Baseline Endline Change(+/‐) Received a postpartum visit within 2 days of last birth from a skilled health worker Child Sex Male n/a 73.1%* n/a Female n/a 77.7%* n/a District Bombali 28.7% 66.9% +38.2% Kailahun 38.3% 81.3% +43.0% Koinadugu 35.4% 71.2% +35.8% Tonkolili 34.3% 81.1% +46.7% TOTAL 34.2% 75.5%*** +41.3% Received at least 3 postnatal visits after last birth from a skilled health worker Child Sex Male n/a 76.2% n/a Female n/a 76.3% n/a District Bombali 38.0% 76.9% +38.9% Kailahun 58.3% 84.5% +26.2% Koinadugu 34.3% 75.2% +40.9% Tonkolili 23.5% 70.8% +47.3% TOTAL 38.6% 76.3%*** +37.7% n/a = not collected at baseline *** p<.001 4.10 Growth Monitoring Growth monitoring is the process of following the growth rate of a child in comparison to a standard by periodic, frequent anthropometric measurements. Growth monitoring is not the infrequent or one‐time anthropometric measurement of a child to assess nutritional status ‐ rather it is recognized as an effective means of detecting faltering growth early, providing a critical opportunity for preventative or curative actions to be taken.   As frequent, continuous measurements are key to effective growth monitoring, the endline survey assessed growth monitoring among children aged 0‐3 years by examining under five cards for proof of attendance at growth monitoring for the past three consecutive month.16 In the case where no under five card was available, data on growth monitoring was not collected for the child. This approach differed from the data collection and analysis of growth monitoring at baseline which included all children 0‐5 years, accepted self‐ reporting of mothers and did not require proof of growth monitoring for the past three consecutive months (rather, any involvement in growth monitoring was included in the tabulation of the indictor).   Although comparisons to baseline data are problematic given the differing approaches to data collection and analysis for this indicator, Figure 8 suggests that there may have been an increase in the proportion of children ever participating in growth monitoring over the six year implementation period of the SNAP project.                                                                16 Data collection was limited to 0‐3 years as this is the age‐range for which growth monitoring data is collected on the under five cards. 23 Whereas a review of under five cards at baseline showed that 53.5% of children under the age of five had ever participated in growth monitoring; a review of under five cards at endline showed that 61.2% of children under the age of three had ever attended growth monitoring. Figure 8: Percent of children that have ever attended growth monitoring, baseline vs. endline However, as growth monitoring requires an assessment of growth velocity over time, survey results related to infrequent participation do not adequately capture the value of the project indicator related to growth monitoring. As such, endline values are presented without comparison to baseline findings in Figure 9 below. As highlighted, less than one‐quarter (23.6%) of children under the age of three were found to be regularly attending growth monitoring (based on an examination of under five cards for proof of attendance at growth monitoring for the past three consecutive months). Attendance at growth monitoring was considerably below average in Koinadugu district where only 13.0% of children have regularly attended growth monitoring. Figure 9: Percent of children (0‐3 years) regularly attending growth monitoring, by sex and district 40.5% 53.5% 61.2% 0.0% 10.0% 20.0% 30.0% 40.0% 50.0% 60.0% 70.0% Has ever attended growth monitoring Baseline‐Mother self‐reporting (<5 years) Baseline ‐U5 card verified (<5 years) Endline ‐ U5 card verified (<3 years) 25.4% 33.7% 13.0% 22.1% 24.2% 23.0% 23.6% 0.0% 5.0% 10.0% 15.0% 20.0% 25.0% 30.0% 35.0% 40.0% Bombali Kailhaun Koinadugu Tonkolili Males Females Overall Bombali Kailhaun Koinadugu Tonkolili Males Females Overall 24 Household Indicators 4.11 Household Dietary Diversity Score The household dietary diversity score (HDDS) is a proxy measure for household food access. The   HDDS is based on the number out of 12 food groups consumed by any household members in the   24 hours preceding the survey.17   The resulting HDDS ranges from 0 to12, with lower numbers indicating less dietary diversity and access to food. Table 13 presents the results for the HDDS at baseline and endline. The overall score of 5.44 at endline indicates that there has been a slight decrease of 1.3 points in the HDDS, from 6.74 at baseline. However, the endline HDDS finding is in line with the results of the most recent SMART survey in SNAP targeted districts from early 2016 which found the average HDDS to be 5.1.18 Nonetheless, the noted decrease in food access at the endline is somewhat expected given that data collection was conducted during the lean season. Table 13: Household dietary diversity score (HDDS) by district, baseline vs. endline   Baseline Endline Change(+/‐) District Bombali 6.59 5.01 ‐1.58 Kailahun 6.55 5.18 ‐1.37 Koinadugu 7.31 5.98 ‐1.33 Tonkolili 6.50 5.61 ‐0.89 TOTAL 6.74 5.44*** ‐1.3 *** p<.001 As outlined in Figure 10 below, there is a clear emphasis on rice, palm oil and fish in the Sierra Leonean diet. Endline survey results suggest an overall decrease in the access to all food groups among the surveyed households when compared to baseline values. Again, it is important to keep in mind that data collection was conducted during the lean season, whereas the baseline was conducted after the harvest.                                                              17 This set of 12 food groups is derived from the U.N. Food and Agriculture Organization (Food and Agricultural Organization. Food Composition Table for Africa. Rome, Italy, 1970. As viewed at www.fao.org/docrep/003/X6877E/X6877E00.htm.  The 12 food groups include: cereals, roots and tubers, vegetables, fruits, meat/poultry/offal, eggs, fish and seafood, pulses/legumes/nuts, milk and milk products, oil/fats; sugar/honey and miscellaneous condiments. 18 See SNAP (2016) 'Report on Nutrition and Mortality Survey in SNAP Targeted Districts: Bombali, Tonkolili, Kailahun and Koinadugu, Sierra Leone during 27/01/2016 to 3/02/2016'. Available upon request. 25 Figure 10: Households dietary composition, by food group 4.12 Household Food Provisioning The number of months of adequate household food provisioning (MAHFP) is a proxy indicator that captures the seasonality of food security. Significant changes in the MAHFP between baseline and endline were reported among SNAP project districts. Over the six years of SNAP project programming, there has been an overall increase of 1.93 months of adequate household food provisioning reported across all surveyed districts. At endline, Koinadugu district continues to have the highest average MAHFP (10.20); while Bombali district continues to have the lowest (8.91). Table 14: Average months of adequate household food provisioning (MAHFP) by district, baseline vs. endline   Baseline Endline Change(+/‐) District Bombali 7.02 8.91 +1.89 Kailahun 7.72 9.71 +1.99 Koinadugu 8.48 10.20 +1.72 Tonkolili 7.72 9.85 +2.13 TOTAL 7.74 9.67** +1.93 *** p<.001 4.13 Caregiver Hygiene Practices Caregiver hygiene practices were assessed based on two standard indicators:  Percent of caregivers demonstrating proper food hygiene behaviours  Percent of caregivers demonstrating proper personal hygiene behaviours Given time and resource constraints, the measurement of the hygiene‐related indicators at endline followed the methodology employed at baseline: caregivers were asked to self‐report on their 8.5% 6.0% 19.0% 36.8% 24.5% 68.6% 76.4% 57.7% 95.3% 89.8% 91.3% 97.1% 6.1% 6.5% 17.5% 23.0% 23.2% 42.9% 46.6% 58.5% 66.7% 78.5% 82.5% 92.4% Milk/Milk Products Eggs Meat/Poultry/Offal Fruits Sugar/Honey Pulses/Legumes/Nuts Vegtables Roots/Tubers Condiments Fish/Seafood Oils/Fats Cereals/Rice Endline Baseline 26 handwashing practices. Caregivers who self‐reported washing their hands during at least two or more critical handwashing junctions related to food hygiene (i.e. before eating, before preparing food, before feeding babies) were assessed as demonstrating proper food hygiene. Conversely, caregivers who self‐reported washing their hands during at least two or more critical handwashing junctions (i.e. before eating, before preparing food, before feeding babies, after cleaning baby's bottom and after defecation) were assess as demonstrating proper personal hygiene.   Table 15 suggests that there has been a significant decrease in self‐reported proper food hygiene behaviours from 78.5% at baseline to 56.1% at endline. Disaggregation by district suggests that this overall decrease is largely driven by the findings in Koinadugu and Tonkolili districts ‐ which reported drastic reductions in self‐reported handwashing behavior related to food hygiene. On the other hand, there has been a slight increase (+3.1) in self‐reported proper personal hygiene behaviours at endline when compared to baseline findings (90.5% compared to 87.4%, respectively). Nonetheless, these findings should be viewed with caution as they are solely based on self‐reported handwashing behaviour. Table 15: Caregiver hygiene practices by district, baseline vs. endline   Baseline Endline Change(+/‐) Percent of caregivers demonstrating proper food hygiene behaviours District Bombali 78.9% 73.6% ‐5.3% Kailahun 70.2% 66.5% ‐3.7% Koinadugu 88.2% 53.6% ‐34.6% Tonkolili 74.1% 36.3% ‐37.8% TOTAL 78.5% 56.1%*** ‐22.4% Percent of caregivers demonstrating proper personal hygiene behaviours District Bombali 89.9% 90.4% ‐0.5% Kailahun 79.8% 95.6% ‐15.8% Koinadugu 97.4% 83.1% ‐14.3% Tonkolili 82.6% 92.0% ‐9.4% TOTAL 87.4% 90.5%*** ‐3.1% ** p<.001 Agriculture and Livelihood Indicators The agricultural component of the survey collected data related to household revenue, crop sales and the use of improved agricultural technologies over the 12 month period preceding the survey. Given the complexity involved in accounting for income and sales over a 12‐month period, data in this section must be viewed with caution. 4.14 Rural Household Revenue As per the project IPTT, average rural household revenue is tabulated as the total of revenue from farm and non‐farm activities for the last 12 months for all households surveyed. Table 16 suggests that there has been a significant decrease in the overall rural household revenue from Le.3,128,690 (USD$812)19  at baseline to Le.1,768,051(USD$295)20 at endline.                                                                19 As per the exchange rate at baseline (USD$ 1.00 = Le 3,855.00) 20 As per the exchange rate at endline (USD$ 1.00 = Le. 6,000.00) 27 Table 16: Average rural household revenue by source, baseline vs. endline   Baseline Endline Source of income Crops Le. 2,391,025 Le. 899,228 Livestock Le. 71,229 Le. 491,224 Fish Le. 26,503 n/a Non‐farm Le. 639,933 Le. 1,529,977 TOTAL Le. 3,128,690 Le. 1,768,051*** n/a = not collected at endline *** p<.001 Interestingly, while there was a noted increase in the average revenue  from non‐farm sources, there has been a significant decrease in the percentage of households reporting non‐farm income from 63.4% households at baseline to 53.7%  households at endline.   Figure 11: Percent of households with non‐farm income by district, baseline vs. endline 4.15 Crop Sales The agricultural component of the household survey was completed by 1,815 households. Overall, 1,662 households (91.6%) reportedly engaged in any farming activity in the 12 months preceding the endline survey.  Of these farmers, 63.5% reported that they had many at least one sale of an agricultural crop in the previous year. This represents a decrease from the baseline value of 70.1%. Interestingly, all districts (except Bombali) reported a decrease in any crop sales over the previous year. This finding should be viewed with caution as self‐reporting may have been biased by the respondent's desire to elicit future support or aid. 71.4% 51.5% 59.8% 71.2% 63.4% 47.0% 57.0% 53.6% 57.3% 53.7% 0.0% 10.0% 20.0% 30.0% 40.0% 50.0% 60.0% 70.0% 80.0% Bombali Kailahun Koinadugu Tonkolili Overall Baseline Endline 28 Figure 12: Percent of households with any farm sales in the 12 months preceding the survey by district, baseline vs. endline The IPTT defines average value of crop sales as the total sales of all farm production sold in the previous 12 months divided by the total number of farming households. According to survey findings, there has been a significant increase in the value of household crop sales over the SNAP project implementation period: from Le.437,124 (USD$113)21 at baseline, to Le.849,842 (USD$143)22 at endline. This findings seems counterintuitive to the baseline findings presented in Table 16, and may be due to tabulation errors at baseline. Table 17: Average annual value of household crop sales by district, baseline vs. endline   Baseline Endline District Bombali Le. 461,857 Le. 492,035 Kailahun Le. 361,669 Le. 1,101,419 Koinadugu Le. 511,580 Le.1,064,137 Tonkolili Le. 412,049 Le. 827,245 TOTAL Le. 437,124 Le. 859,842*** *** p<.001 4.16 Improved Agricultural Practices The survey sought to measure the changes in three main agricultural improvements; namely: (1) improved storage; (2) association benefits; and (3) adoption of agricultural technologies. In order to assess changes over the lifespan of the SNAP project, respondents were first asked if they had engaged in any agricultural activities in the last five years. Those that reported 'yes', were then asked a series of questions related to the three above mentioned agricultural improvements. Survey findings revealed that only 52.3% of respondents reported that they had in fact engaged in any agricultural activities in the previous five years. This finding is contradictory when viewed in the context of the 91.6% of households that reported growing any crop in the previous year. This                                                              21 As per the exchange rate at baseline (USD$ 1.00 = Le 3,855.00) 22 As per the exchange rate at endline (USD$ 1.00 = Le. 6,000.00) 66.5% 87.5% 69.6% 57.2% 71.3% 69.7% 70.1% 51.9% 60.7% 63.5% 0.0% 10.0% 20.0% 30.0% 40.0% 50.0% 60.0% 70.0% 80.0% 90.0% 100.0% Bombali Kailahun Koinadugu Tonkolili Overall Baseline Endline 29 suggests that survey data on improved agricultural practices lack validity and should be kept in mind when assessing the survey findings in this section.   As shown in Figure 13, the percentage of households using improved storage techniques for their crops has increased significantly from 4.7% at baseline to 19.1% at endline. While baseline data on the use of association benefits and adoption of agricultural technologies was not collected, endline survey findings suggest a moderate level of adoption among the surveyed households. For instance, 14.9% of farmers reported that they use at least one association benefit related to collective storage, collective marketing and bulk purchasing. Furthermore, one‐quarter (25.5%) of farmers reported that they have adopted at least three agricultural technologies in the last five years which they were not using before, including: crop production practices, post‐harvest processing practices or improved crop storage practices. Figure 13: Use of improved agricultural practices 4.7% 19.1% 14.9% 25.5% Improved stroage Association benefits Adopted at least 3 agricultural technologies Baseline Endline 30 5. Summary Statistical Comparison of Baseline and Endline Indicator Values A comparison of baseline with endline values of project indicators was conducted using both z‐tests and t‐tests (where appropriate) to detect a population level difference at a 95% level of significance. Both tests were used to determine if there was a statistically significant difference between the two study groups (i.e. baseline and endline). Z‐tests for proportions were used on all categorical binary indicators; while t‐tests of significance were used to compare the means of interval dependent variables for the two study groups. Table 18: Summary statistical comparison of baseline and endline indicator values Ind Indicator BASELINE RESULT ENDLINE RESULT 95% confidence interval Z‐Test / T‐Test LCL UCL (Significance Test) IM2 % of underweight children 0‐59 month of age (weight for age z‐score (WAZ) <‐ 2 SD) (FFP SO 1) 20.3% 14.8% 13.1% 16.7% z = ‐5.24, p = 0.000 IM3 % of stunted children 6‐59 months (height‐for‐age z‐score (HAZ) < ‐2 SD) (FFP SO2) 38.4% 32.3% 30.0% 34.7% z = ‐4.82, p = 0.000 OC1 % of acutely malnourished children 6‐ 59 months (weight for height z‐score (WHZ) < ‐2 SD or edema) 9.6% 6.5% 5.2% 8.0% z = ‐3.70, p = 0.000 OC 2 % of children 0‐3 years regularly attending growth monitoring 94.0% 23.6% 21.1% 26.1% z = ‐100.09, p = 0.000 31 OC 3 % of mothers of children age 0‐23 months who had four or more antenatal visits when they were pregnant with the youngest child 53.2% 76.9% 74.1% 79.8% z = 13.82, p = 0.000 OC 4.1 % of births with a skilled birth attendant (SBA) in USG assisted programs 52.3% 91.3% 89.9% 92.8% z = 29.55, p = 0.000 OC 5 % of mothers participating in postpartum/newborn visit within 2 days of birth in USG programs 34.2% 75.5% 73.2% 77.7% z = 32.89, p = 0.000 OC 6 % of mothers receiving 3 postnatal visits 38.6% 76.3% 74.1% 78.5% z = 29.28, p = 0.000 OP 5.1 % of newborns (0‐3 months) receiving essential newborn care through USG supported programs n/a 69.6% 61.1% 78.2% n/a OP 5.1b % of newborns (0‐1 months) receiving essential newborn care through USG supported programs 55.7% 50.0% 31.5% 68.5% z = ‐0.61, p = 0.544 OC 7.1 % of child diarrhea cases treated with ORS or Zinc in USG programs n/a 82.6% 74.5% 90.6% n/a OC 7.1B % of child diarrhea cases treated in hospital/clinic/PHU 21.1% 82.6% 74.5% 90.6% z = 13.97, p = 0.000 OC 9 % of children 0‐5 months of age exclusively breastfed 39.5% 75.6% 69.9% 81.2% z = 11.06, p = 0.000 OC 10 % of caregivers demonstrating proper food hygiene behaviours 78.5% 56.1% 53.5% 58.7% z = ‐20.63, p = 0.000 OC 11 % of caregivers demonstrating proper personal hygiene behaviours 87.4% 90.5% 89.0% 92.0% z = 3.52, p = 0.000 32 IM 4 Average Household dietary diversity score (HDDS) (FFP SO 4) 6.74 5.44 5.34 5.55 t(1826) = ‐23.75, p = 0.000 IM 6 Average # of months of adequate household food provisioning (MAHFP) (FFP S03) 7.74 9.67 9.60 9.74 t(1828) = 55.03, p = 0.000 OC 14 Average rural household revenue Le.3,128,690 Le.1,768,050.79 Le.1,569,315.37 Le.1,966,786.21 t(1399) = ‐13.42, p = 0.000 OC 14A Average rural household revenue from CROPS Le.2,391,025 Le.899,227.94 Le.811,725.81 Le.986,730.05 t(1055) = ‐33.41, p = 0.000 OC 14B Average rural household revenue from LIVESTOCK Le.71,229 Le.491,224.22 Le.285,101.22 Le.697,347.22 t(445) = 3.99, p = 0.000 OC 14C Average rural household revenue from NON‐FARM Le.639,933 Le.1,529,977.05 Le.1,264,698.21 Le.1,795,255.89 t(853) = 6.58, p = 0.000 OC 15 % of households with non‐farm income 63.4% 53.7% 51.5% 56.0% z = ‐8.57, p = 0.000 OC 16 % of farmers reporting use of association benefits n/a 14.9% 12.7% 17.2% n/a OC 17 % of farmers who have adopted at least three agricultural technologies n/a 25.5% 22.7% 28.2% n/a OC 19.1A Average yield per hectare (Kg/ha) Lowland Rice 609.97 747.55 704.91 790.19 t(977) = 6.32, p = 0.000 OC 19.1B Average yield per hectare (Kg/ha) Groundnut 310.36 710.02 668.77 751.27 t(938) = 18.99, p = 0.000 OC 19.1C Average yield per hectare (Kg/ha) Cassava 2102.75 760.58 696.69 824.47 t(419) = ‐41.18, p = 0.000 OC 19.1D Average yield per hectare (Kg/ha) Sweet Potato 3581.31 580.59 482.22 678.96 t(200) = ‐59.79, p = 0.000 33 OC 20.1 Average value of crop sales 437124.00 859841.82 778672.30 941011.34 t(1044) = 10.21, p = 0.000 OC 24 % of farmers using improved storage* * Improved storage = Storage that provide adequate protection against rain, insects, pests 4.7% 19.1% 16.6% 21.5% z = 20.90, p = 0.000 34 6. Additional Analysis by Project Participation While the evaluation design did not include a statistically representative comparison (or control) group, the population based survey design did include a large proportion of households that did not participate directly in SNAP activities. Using the endline survey data, a limited amount of analysis comparing beneficiaries and non‐beneficiaries was carried out. This analysis is meant to ascertain if there is any (non‐statistically representative) indication that program activities might be influencing the program results reported in this report. Beneficiary status was established according to the following:  Households that reported to have participated in farmer field school (FSS) or agriculture activities, were considered to be 'beneficiaries' for the agriculture, livelihood and food security indicators  Households that reported to have participated in Mothercare group activities, were considered to be 'beneficiaries' for the nutrition, maternal and child health indicators Table 19: Endline indicator values, by project participation Ind Indicator OVERALL Result Beneficiary (Mothercare Group) Beneficiary (FFS/ agriculture) No Yes No Yes IM2 % of underweight children 0‐59 month of age (weight for age z‐score (WAZ) <‐2 SD) (FFP SO 1) 14.8% 15.6% (p=0.424) 12.9% (p=0.424) n/a n/a IM3 % of stunted children 6‐59 months (height‐ for‐age z‐score (HAZ) < ‐2 SD) (FFP SO2) 32.3% 33.3% (p=0.204) 30% (p=0.204) n/a n/a OC1 % of acutely malnourished children 6‐59 months (weight for height z‐score (WHZ) < ‐2 SD or edema) 6.5% 6.4% (p=0.991) 6.5% (p=0.991) n/a n/a OC 2 % of children 0‐3 years regularly attending growth monitoring 23.6% 20.7%*** (p=0.000) 31%*** (p=0.000) n/a n/a OC 3 % of mothers of children age 0‐23 months who had four or more antenatal visits when they were pregnant with the youngest child 76.9% 76.7% (p=0.807) 77.4% (p=0.807) n/a n/a 35 OC 4.1 % of births with a skilled birth attendant (SBA) in USG assisted programs 91.3% 89.7%*** (p=0.000) 95.4%*** (p=0.000) n/a n/a OC 5 % of mothers participating in postpartum/newborn visit within 2 days of birth in USG programs 75.5% 72.3%*** (p=0.000) 83.9%*** (p=0.000) n/a n/a OC 6 % of mothers receiving 3 postnatal visits 76.3% 73.8%*** (p=0.000) 82.7%*** (p=0.000) n/a n/a OP 5.1 % of newborns (0‐3 months) receiving essential newborn care through USG supported programs 69.6% 66.7% (p=0.172) 80% (p=0.172) n/a n/a OP 5.1b % of newborns (0‐1 months) receiving essential newborn care through USG supported programs 50.0% 44.4% (p=0.276) 66.7% (p=0.276) n/a n/a OC 7.1 % of child diarrhea cases treated with ORS or Zinc in USG programs 82.6% 78.2% (p=0.172) 90% (p=0.172) n/a n/a OC 7.1B % of child diarrhea cases treated in hospital/clinic/PHU 82.6% 74.5%* (p=0.011) 96.7%* (p=0.011) n/a n/a OC 9 % of children 0‐5 months of age exclusively breastfed 75.6% 78.2% (p=0.159) 69% (p=0.159) n/a n/a OC 10 % of caregivers demonstrating proper food hygiene behaviours 56.1% 50.4%*** (p=0.000) 69.4%*** (p=0.000) n/a n/a OC 11 % of caregivers demonstrating proper personal hygiene behaviours 90.5% 88.4%*** (p=0.000) 95.7%*** (p=0.000) n/a n/a IM 4 Average Household dietary diversity score (HDDS) (FFP SO 4) 5.44 5.41 (p=0.267) 5.56 (p=0.267) 5.40 (p=0.081) 5.61 (p=0.081) IM 6 Average # of months of adequate household food provisioning (MAHFP) (FFP S03) 9.67 n/a n/a 9.69 (p=0.061) 9.54 (p=0.061) OC 14 Average rural household revenue Le.1,768,050.79 n/a n/a 1732156.43 (p=0.599) 1847413.53 (p=0.599) OC 14A Average rural household revenue from CROPS Le. 899,227.94 n/a n/a 768118.48*** (p=0.000) 1138309.89*** (p=0.000) OC 14B Average rural household revenue from LIVESTOCK Le. 491,224.22 n/a n/a 390667.94 (p=0.254) 634407.61 (p=0.254) 36 OC 14C Average rural household revenue from NON‐ FARM Le.1,529,977.05 n/a n/a 1745128.76* (p=0.015) 1027396.09* (p=0.015) OC 15 % of households with non‐farm income 53.7% n/a n/a 54% (p=0.866) 54.5% (p=0.866) OC 16 % of farmers reporting use of association benefits* *Associations benefits: collective storage; collective marketing; bulk purchasing; etc. 14.9% n/a n/a 8.4%*** (p=0.000) 29.1%*** (p=0.000) OC 17 % of farmers who have adopted at least three agricultural technologies* *Agricultural technologies include: any 3 of the crop production practices, post‐harvest processing practices or crop storage practices noted in Q554, Q556 or Q558 respectively. 25.5% n/a n/a 16.4%*** (p=0.000) 45.2%*** (p=0.000) OC 19.1A Average yield per hectare (Kg/ha) Lowland Rice 747.55 Kg/Ha n/a n/a 710.52* (p=0.017) 819.93* (p=0.017) OC 19.1B Average yield per hectare (Kg/ha) Groundnut 710.02 Kg/Ha n/a n/a 681.25 (p=0.060) 764.39 (p=0.060) OC 19.1C Average yield per hectare (Kg/ha) Cassava 760.58 Kg/Ha n/a n/a 737.87 (p=0.360) 799.81 (p=0.360) OC 19.1D Average yield per hectare (Kg/ha) Sweet Potato 580.59 Kg/Ha n/a n/a 528.42 (p=0.188) 664.62 (p=0.188) OC 20.1 Average value of crop sales Le. 859,841.82 n/a n/a 732268.65*** (p=0.000) 1090639.52*** (p=0.000) OC 24 % of farmers using improved storage* * Improved storage = Storage that provide adequate protection against rain, insects, pests 19.1% n/a n/a 13.2%*** (p=0.000) 31.8%*** (p=0.000) *p < 0.05; **p < 0.01; *** p< 0.001 (Statistical significance between beneficiary and non‐beneficiaries was evaluated using chi‐square and ANOVA) 37 Annex 1: Summary SNAP Performance Indicator Reference Sheet (PIRS) for Endline Indicators Ind. # Source Performance Indicator Tabulation/Definition IM 2 FFP % of underweight children 0‐59 month of age (weight for age z‐score (WAZ) <‐2 SD)  (FFP SO 1) # of children age 0‐59 months with weight for age below ‐2z score _________________________________________________________________________________________________________________________________________________________________ total # of children 0‐59 months old measured IM 3 FFP % of stunted children 6‐59 months (height‐for‐age z‐ score (HAZ) < ‐2 SD) (FFP SO2) # of children age 6‐59 months with height for age below ‐2 z score ________________________________________________________________________________________________________________________________________________________________ total # of children 6‐59 months old measured OC 1 FFP % of acutely malnourished children 6‐59 months (weight for height z‐score (WHZ) < ‐2 SD or edema) * percent of children (6‐59 months) falling below ‐2 standard deviations for weight‐for‐height or with bilateral pitting edema.    Numerator: # of children 6‐59 months old with weight for height below ‐2 z score or edema _______________________________________________________________________________________________________________________________________________________________ total # of children 6‐59 months old measured OC 2 SNAP % of children 0‐59 months regularly attending growth monitoring # of children 0‐59 months participating in regularly growth monitoring _______________________________________________________________________________________________________________________________________________________________ total # of children 0‐59 months surveyed OC 3 SNAP % of pregnant women completing 4 antenatal visits Total number of mothers of children aged 0‐23 months that saw a skilled provider four or more times during last pregnancy ______________________________________________________________________________________________________________________________________________________________ Total number of mothers of children aged 0‐23 months OC 4.1 F/ SL USAID % of births with a skilled birth attendant (SBA)* in USG assisted programs * SBA includes: medically trained doctor, nurse, midwife, CHO or MCH Aide . It does NOT include traditional birth attendants (TBA). Total number of children age 0‐59 months whose birth was attended by a doctor, nurse, midwife, MCH Aide or CHO ______________________________________________________________________________________________________________________________________________________________ Total number of mothers of children age 0‐59 months in the survey OC 5 F/ SL USAID % of mothers participating in postpartum/newborn   visit within 2 days of birth in USG programs Total number of mothers of children aged 0‐59 months participating in postpartum/newborn visit within 2 days of birth ______________________________________________________________________________________________________________________________________________________________ Total number of mothers of children age 0‐59 months in the survey 38 OC 6 SNAP % of mothers with children months receiving 3 postnatal visits * SBA includes: medically trained doctor, nurse, midwife, CHO or MCH Aide . It does NOT include traditional birth attendants (TBA). Total number of mothers of children aged 0‐59 months that saw a skilled provider three times ______________________________________________________________________________________________________________________________________________________________ Total number of mothers of children aged 0‐ 59 months in the survey OP 5.1 F % of newborns receiving essential newborn care through USG supported programs Number of children aged 0‐3 months who receive essential newborn care (clean cord care, drying and wrapping, immediate breastfeeding) from trained facility, outreach or community health workers through USG‐ supported programs _____________________________________________________________________________________________________________________________________________________________ Total number of children aged 0‐3 months in the survey OC 7.1 SL USAID % of diarrhea treated with ORS or Zinc in USG programs Number of children aged 0‐59 months with diarrhea in the last two weeks who were treated with ORS or zinc supplements _____________________________________________________________________________________________________________________________________________________________ Total number of children aged 0‐59 months in the survey OC 9 FFP % of children 0‐5 months* of age exclusively breastfed *(age in days < 183)   Number of infants 0‐5 months of age who are fed exclusively with breast milk _____________________________________________________________________________________________________________________________________________________________ Total number of children aged 0‐5 months in the survey OC 10 FFP % of caregivers demonstrating proper food hygiene behaviours # of caregivers observed practicing 2 or more SNAP promoted food hygiene behaviors _____________________________________________________________________________________________________________________________________________________________ Total number of caregivers surveyed OC 11 FFP % of caregivers demonstrating proper personal hygiene behaviours # of caregivers  observed carrying out 2 or more proper personal hygiene behaviors   _____________________________________________________________________________________________________________________________________________________________ Total number of caregivers surveyed   IM 4 F/SL USAID/SNAP Average Household dietary diversity score  (HDDS)   (FFP SO 4) Total number of food groups consumed by members of the household (Sum HDDS) _____________________________________________________________________________________________________________________________________________________________ Total number of household surveyed IM 6 FFP/SL USAID/SNAP Average # of months of adequate household food provisioning (MAHFP)  (FFP S03) Household MAHFP:   Twelve months minus total number of months out of the previous twelve months that the household was unable to meet food needs.    Average MAHFP:   Sum of the MAHFPs for all households in the sample   _____________________________________________________________________________________________________________________________________________________________ Total number of household surveyed 39 OC 14 SNAP Average rural household revenue Total of revenue from farm and non‐farm activities of all households surveyed _____________________________________________________________________________________________________________________________________________________________ Total number of household surveyed OC 15 SNAP % of participants with non‐ farm income % of households reporting non‐farm income _____________________________________________________________________________________________________________________________________________________________ Total number of household surveyed OC 16 SNAP % of farmers reporting use of association benefits* *Associations benefits: collective storage; collective marketing; bulk purchasing. # of farmers reporting use of association benefits _____________________________________________________________________________________________________________________________________________________________ Total number of famers surveyed   OC 17 FFP % of farmers who have adopted at least three agricultural technologies* *Agricultural technologies include: any 3 of the crop production practices, post‐ harvest processing practices or crop storage practices   Number of farmers  who have adopted at least 3 of agricultural technologies   _____________________________________________________________________________________________________________________________________________________________ Total number of famers surveyed   OC 19.1 SNAP Average yield per hectare (Kg/ha) Total yield of all farm production in per crop   _____________________________________________________________________________________________________________________________________________________________ Total number of famers surveyed   OC 20.1 SNAP Average value of crop sales Total sales of all farm production sold per crop _____________________________________________________________________________________________________________________________________________________________ Total number of famers surveyed   OC 24 SNAP % of farmers using improved storage* * Improved storage: Storage that provide adequate protection against rain, insects, pests   # of farmers who use improved storage techniques _____________________________________________________________________________________________________________________________________________________________ Total number of famers surveyed      40 Annex 2: Sampled EAs and Households for the Endline Survey Ea Number District Chiefdom Section Community # Of Households Sampled 1 Koinadugu Wara Wara Bafod Taelia Taleya 17 2 Koinadugu Wara Wara Bafod Pampakoh Old Panpakoh 18 3 Koinadugu Wara Wara Bafod Kamanikie Kama Kumba‐Kapakie 17 4 Koinadugu Wara Wara Bafod Kadanso Kakonso 17 5 Koinadugu Nieni Nieni Gbenikoro 17 6 Koinadugu Nieni Nieni Funebakura 17 7 Koinadugu Nieni Seradu Dabinya 17 8 Koinadugu Nieni Kalian Alikalia Town 17 9 Koinadugu Nieni Kalian Kumala Town 17 10 Koinadugu Nieni Barawa Koikoro 17 11 Koinadugu Neya Lower Neya I Masofinia 17 12 Koinadugu Nieni Barawa Moria 17 13 Koinadugu Mongo Morifindugu Bockaria 17 14 Koinadugu Diang Kondembaia Kondembaia 19 15 Koinadugu Mongo Mongo Iii Karawani 19 16 Koinadugu Wara Wara Bafod Taelia Kabalekeh 17 17 Koinadugu Mongo Benadugu Modolforia 17 41 18 Koinadugu Folosaba Dembelia Musaia Musaia Town 17 19 Koinadugu Kasunko Kayaka Kanasinkenara 17 20 Koinadugu Kasunko Kasunko Kaduwokoh (8) 17 One‐Mile (9) 21 Koinadugu Kasunko Gbonkobor Kamayara 17 22 Koinadugu Kasunko Kayaka Kakuna 16 23 Koinadugu Folosaba Dembel Musaia Musaia Town 17 24 Koinadugu Folosaba Dembel Kalia Gbentu Town 17 25 Koinadugu Folosaba Dembel Dogoloya Dogoloya Town 17 26 Koinadugu Mongo Mongo Iii Karawanie 15 27 Koinadugu Diang Kondembaia Kondembaia 15 28 Bombali Sella Limba Samia Kamalai / New Site 17 29 Bombali Sella Limba Manonkoh Kamayibin 16 30 Bombali Sella Limba Manonkoh Magborlontor 17 31 Bombali Sella Limba Magbonkoni Ii Kamabayi 17 32 Bombali Sella Limba Magbonkoni I Kagberay 17 33 Bombali Sella Limba Kayimbor Kagigranah 17 34 Bombali Sella Limba Kayimbor Small Kathirie 17 35 Bombali Sella Limba Kamankoh Kadarie 17 36 Bombali Sella Limba Kamakwie Kayawuyea 17 37 Bombali Sella Limba Kamakwie Brima Sanda Street 17 38 Bombali Sella Limba Kamakwie H. Plan 17 42 39 Bombali Sella Limba Kamakwie Kamabonko 17 40 Bombali Sella Limba Kamakwie Madina Street Kamakwie 2 17 41 Bombali Sanda Loko Rothatha Mabanta Ii 17 42 Bombali Sanda Loko Manathi Makoba‐Loko 17 43 Bombali Sanda Loko Makapa Amina 17 44 Bombali Sanda Loko Madina Kawilay Limba 17 45 Bombali Sanda Loko Kamalu Kamalo 17 46 Bombali Sanda Loko Kaindema Makoti / Mathoi 17 47 Bombali Sanda Loko Banka Rochin 17 48 Bombali Gbanti Kamarank Sakuma A Makumbu 17 49 Bombali Gbanti Kamarank Romaneh Masinneh 17 50 Bombali Gbanti Kamarank Makumray A Makumray 17 51 Bombali Gbanti Kamarank Makulon Mamaria 17 52 Bombali Gbanti Kamarank Romaneh Samu 17 53 Bombali Gbanti Kamarank Kambia Gbom‐Fore Road 17 54 Bombali Gbanti Kamarank Gbonkobana Gbonko Bana 16 55 Tonkolili Yoni Gaindema Rochain Kamandao 17 56 Tonkolili Yoni Yoni Yoni Bana 17 57 Tonkolili Yoni Ronietta Mabunkuma (2) 17 Masankay (15) 58 Tonkolili Yoni Ronietta Magboyo Ii 17 59 Tonkolili Yoni Petifu Masesay (3) 18 43 Mashangba (15) 60 Tonkolili Yoni Petifu Bakeh‐Loko I 17 61 Tonkolili Yoni Mayira Madina 17 62 Tonkolili Yoni Mamaka Old Mathoir 17 63 Tonkolili Yoni Mamaka Rokimbie Ii 17 64 Tonkolili Yoni Malompor Mabila‐Lol 17 65 Tonkolili Yoni Makeni Rokefula Makafie (10) 17 Marokie (7) 66 Tonkolili Yoni Makeni Rokefula Makambo 17 67 Tonkolili Yoni Macrogba Macrogba 17 68 Tonkolili Yoni Gaindema Mile 91 17 69 Tonkolili Yoni Gaindema Mile 91 17 70 Tonkolili Yoni Gaindema Mile 91 17 71 Tonkolili Yoni Gaindema Mile 91 17 72 Tonkolili Yoni Gaindema Mashetheh 17 73 Tonkolili Yoni Foindu Foindu 16 74 Tonkolili Kalansogoia Bumbuna Bumbuna Town 17 75 Tonkolili Kalansogoia Songoni Songoni Iii Sokirella 17 76 Tonkolili Kalansogoia Kasokira Kasasi 17 77 Tonkolili Kalansogoia Lower Section Yeben 17 78 Tonkolili Kafe Simiria Mayaso Sangbamaba 17 79 Tonkolili Kafe Simiria Mabontor Moria 17 44 80 Tonkolili Kafe Simiria Makelfa Masumbiri Town 13 81 Tonkolili Kafe Simiria Makelfa Gbafaya 14 82 Kailahun Yawei Kuivawa Madina 17 83 Kailahun Yawei Kuivawa Bandajuma 17 84 Kailahun Yawei Kuiva Jagor Malema 18 85 Kailahun Yawei Kuiva Buima Torbohun (14) 17 Massa Yiema (3) 86 Kailahun Yawei Bendu Bendu 17 87 Kailahun Penguia Nimima Sandaru 18 88 Kailahun Penguia Lombama Kongonani 17 89 Kailahun Penguia Kumatandu Kumatandu 17 90 Kailahun Penguia Bulima Sengama 17 91 Kailahun Mandu Upper Kuiva Ngoo 17 92 Kailahun Mandu Upper Kuiva Njala 17 93 Kailahun Mandu Upper Kuiva Potoru 17 94 Kailahun Mandu Lower Kuiva Kuiva 17 95 Kailahun Mandu Levuma Jeigbla Levuma 17 96 Kailahun Mandu Gbongre Baiima 17 97 Kailahun Kpeje West Kpindima Joikibu 17 98 Kailahun Kpeje West Golama Boama 17 99 Kailahun Kpeje West Kpeje Foiya Bunumbu 17 100 Kailahun Kissi Tongi Upper Tongi Tingi Senga 17 45 101 Kailahun Kissi Tongi Upper Tongi Tingi Voahun 17 102 Kailahun Kissi Tongi Lower Tongi Tingi Mano 17 103 Kailahun Kissi Tongi Lower Tongi Tingi Ndambalu 17 104 Kailahun Kissi Tongi Upper Pokorli Foadu 17 105 Kailahun Kissi Tongi Lower Konio Buedu 17 106 Kailahun Kissi Tongi Lower Konio Ngokodu 17 107 Kailahun Kissi Tongi Upper Konio Taidu 17 108 Kailahun Kissi Tongi Upper Konio Marko 17 46 Annex 3: Data Collection Tool Note: Electronic data collection tool available as a separate file upon request.   47 Annex 4: Differences between Baseline and Endline Indicators Performance Indicator Baseline Measurement Endline Measurement OC2 % of children 0‐59 months regularly attending growth monitoring  Included all children <5 years  Accepted self‐reporting of mothers  Did not require proof of growth monitoring for the past three consecutive months  Included children 0‐3 years  Did not accept self‐reporting of mothers  Required proof of growth monitoring for the past three consecutive months OC 3 % of pregnant women completing 4 antenatal visits  Measured for mothers of children 0‐59 months  Measured for mothers of children 0‐23 months at endline as the project uses the PM2A approach OP 5 # of new‐borns (0‐3 month) receiving essential new‐born care  Limited data collection to mothers with children aged 0‐1 month  Data collection limited to the following survey question: 'Did respondent use any essential newborn care (including thermal care)'  Baseline survey had over 50% of missing data on this indicator  Data collection included mothers with children 0‐3 months old  The endline survey applied a much more nuanced approach to data collection and analysis. Any child 0‐3 months who was born in a health facility and had an under‐five card with their birth weight recorded on it was assumed to have received essential newborn care (as per the established essential package of health service protocols in Sierra Leone). In the case where a child was not born in a health facility and/or did not have their birth weight recorded, the mother was asked a series of questions to ascertain if the child had received clean cord care, thermal care and breastfeeding immediately after birth OC7.1 Percent of cases of child diarrhea treated with ORS or zinc in USAID programs  Not measured at baseline  Measured at endline 48 OC16 % of farmers reporting use of association benefits  Not measured at baseline  Measured at endline OC17 % of farmers who have adopted at least three agricultural technologies  Not measured at baseline  Measured at endline    Sustainable Nutrition and Agriculture Promotion (SNAP) FY16 Annual Results Survey on Agricultural Technology Adoption This publication was produced for ACDI VOCA, with support from the US Agency for International Development – Office of Food for Peace. It was prepared by NestBuilders International (NBI). November 2016. i Sustainable Nutrition and Agriculture Promotion (SNAP) FY16 Annual Results Survey on Agricultural Technology Adoption Research Conducted by: NestBuilders International (NBI) NestBuilders International 16 Fifth Street, Juba Hill Freetown, Sierra Leone +232‐25‐239‐495 www.nbiconsultancy.com ii Table of Contents 1. Background.......................................................................................................................... 1 1.1 The Sustainable Nutrition and Agricultural Promotion (SNAP) Project..................................1 1.2 Geographic Coverage.............................................................................................................. 1 2. Survey Details...................................................................................................................... 2 2.1 Purpose of the Survey............................................................................................................. 2 2.2 Indicators ................................................................................................................................2 2.3 Survey Methodology............................................................................................................... 2 2.4 Sampling Procedure................................................................................................................ 2 2.5 Survey Instrument................................................................................................................... 3 2.6 Survey Logistics....................................................................................................................... 3 2.7 Data Entry and Analysis .......................................................................................................... 3 3. Endline Survey Results ....................................................................................................... 4 3.1 Beneficiaries' Farming Activities............................................................................................. 4 3.2 Adoption of Improved Agricultural Technologies................................................................... 5 3.3 Association Benefits................................................................................................................ 7 4. Summary Statistical Comparison of Baseline and Endline Indicator Values ................ 9 List of Figures Figure 1: SNAP intervention areas.......................................................................................................... 1 Figure 2: Respondent Sex........................................................................................................................ 4 Figure 3: Planting and crop production practices used in the last 12 months.......................................5 Figure 4: Post‐harvest processing practices used in the last 12 months................................................6 Figure 5: Crop storage practices used in the last 12 months.................................................................. 6 Figure 6: Association benefits of SNAP FFS membership ....................................................................... 7 List of Tables Table 1: Summary IPTT indicators........................................................................................................... 2 Table 2: Crops grown and sold by surveyed FFS beneficiaries in the last 12 months............................4 Table 3 : Indicator values related to use of agricultural technolgies and improved storage practices..7 Table 4: Indicator vales related to association benefits......................................................................... 8 Table 5: Summary statistical comparison of baseline and endline indicator values..............................9 iii Acronyms CAPI Computer‐Assisted Personal Interviewing EA Enumeration Area FFS Farmer field school IR Immediate result IPTT Indicator performance tracking table NBI NestBuilders International PIRS Performance Indicator Reference Sheets SNAP Sustainable Nutrition and Agriculture Promotion SO Strategic Objectives USAID United States Agency for International Development iv Survey Summary: at‐a‐glance PERFORMANCE INDICATOR Baseline Result Endline Result Strategic Objective 2: Enhanced livelihoods for vulnerable people, especially women and youth OC16 % of farmers reporting use of SELECT association benefits1,2  ‐  94.2% Oc16b % of farmers reporting use of ANY association benefits1,3  ‐  99.7% OC17 % of farmers who have adopted at least three agricultural technologies1 ‐  100.0% IR 2.3: Improved community assets OC 24 % of farmers using improved storage   4.7% 99.8%* * Significantly different from baseline level 1 Indicator not measured at baseline 2 Association benefits include: collective storage, collective marketing and bulk purchasing 3 Association benefits include: collective storage, collective marketing, bulk purchasing, knowledge exchange, shared labour and greater access to loans   1 1. Background 1.1 The Sustainable Nutrition and Agricultural Promotion (SNAP) Project From June 2010 to June 2016, ACDI/VOCA and partners, the International Medical Corps (IMC) and Opportunities Industrialization Centers (OIC) launched a USAID Food for Peace (FFP) funded project; ‘Sustainable Nutrition and Agricultural Promotion (SNAP)’. The overarching goal of the SNAP project was to reduce food insecurity and increase resiliency among the most food insecure and vulnerable rural populations in target communities. SNAP therefore addressed the root causes of malnutrition, food insecurity, and vulnerability to shocks through improved health, hygiene, and nutrition practices as well as improved livelihood opportunities. The SNAP project was organized under two Strategic Objectives (SO):  SO1 aims to reduce chronic malnutrition among children under five  SO2 aims to enhance livelihoods among women and youth in the project communities In addition, the project addressed five crosscutting themes: (1) resiliency to shocks, (2) productive youth, (3) gender equity, (4) environmental stewardship, and (5) good governance. SNAP provided the targeted population with the necessary tools to improve their food security situation by addressing the key factors influencing utilization, availability and access to nutritious and diverse foods.   1.2 Geographic Coverage The SNAP project was implemented in 18 chiefdoms, spanning four districts, as per the following:  Kailahun District (Kissi Tongi, Kpeje West, Mandu, Penguia, Yawei)  Bombali District (Sella Limba, Sanda Loko, Gbanti Kamaranka)  Tonkolili District (Kafe Simira, Kalasongia, Yoni)  Koinadugu District (Diang, Folosaba Dembelia, Kassunko, Mango, Neya, Neini, Wara Bafodia) Figure 1: SNAP intervention areas 2 2. Survey Details 2.1 Purpose of the Survey The FY16 annual results survey on agricultural technology adoption was conducted as part of the overall SNAP Final Evaluation. The primary objective of the survey is to assess the status of key quantitative indicators after program implementation. More specifically, the purpose of the survey is to measure endline levels of three outcome indicators related to agricultural technology adoption listed in the indicator performance tracking table (IPTT) below.  The endline information is compared to the baseline values to measure the extent of changes over the period of SNAP project implementation and their statistical significance.   2.2 Indicators The SNAP project's strategic objectives (SOs) and intermediate results (IRs) are outlined in the IPTT.   Table 1 shows the summary IPTT indicators that are used to estimate program achievement in terms of agricultural technology adoption. The definitions of indicators and means of measurement are provided in the summary SNAP performance indicator reference sheet (PIRS) in Annex 1. Table 1: Summary IPTT indicators PERFORMANCE INDICATOR Type of Respondent Main Disaggregation Strategic Objective 2: Enhanced livelihoods for vulnerable people, especially women and youth OC16 % of farmers reporting use of association benefits Direct Beneficiary: Member of Farmer Field School Sex OC17 % of farmers who have adopted at least three agricultural technologies Direct Beneficiary: Member of Farmer Field School Sex IR 2.3: Improved community assets OC 24 % of farmers using improved storage   Direct Beneficiary: Member of Farmer Field School Sex 2.3 Survey Methodology A  beneficiary‐based sample survey was conducted for this quantitative evaluation to measure three outcome indicators. The evaluation covers the five‐year period, including the one‐year extension due to the Ebola outbreak from June 2010 to June 2016, and follows a purely quantitative methodology.    2.4 Sampling Procedure 30 X 20 Cluster Design A 30x20 cluster sampling methodology was applied where 30 SNAP farmer field school (FFS) clusters were sampled with 20 farmers per cluster being  interviewed.  (See Annex 2 for a list of the sampled FFS clusters) 3 The sample was randomly selected according to a two‐stage cluster sampling methodology.  In the first stage, a sample size of 30 FFS clusters was selected according to probability proportional to measures of size (PPS); where measure of size was the number of members per FFS cluster. By using a probability proportional to size selection at the first stage, an implicit stratification and proportional allocation was achieved. The second stage involved the selection of twenty beneficiaries in each sampled FFS cluster. Using the list of FFS members for each cluster as the sampling frame, names were selected according to simple random sampling using a random numbers generator in Excel.   2.5 Survey Instrument Survey data was collected through face‐to‐face interviews using a well‐structured questionnaire (see Annex 3). The quantitative tool was developed by SNAP. The questionnaire focused exclusively on the adoption of agricultural technologies and association benefits among SNAP FFS beneficiaries.   The questionnaire was adapted into a computer‐assisted personal interviewing (CAPI) survey using SurveyCTO to allow for mobile data collection.   2.6 Survey Logistics A total of 15 skilled and qualified NBI field enumerators and three supervisors conducted primary data collection at community‐level. Prior to data collection, NBI organized and facilitated a one‐day training workshop in Freetown. All field staff were required to participate in the workshop which focused on the study background, sampling procedures, interviewing techniques and familiarization with the data collection tool.   Following the training workshop, fieldworkers were deployed to the field to conduct data collection according to a clearly developed field plan which had been developed prior to the commencement of field work. Letters had been prepared and delivered to the sampled cluster communities which included details of the field plan, date of interview and the names of the pre‐selected beneficiaries who were to be interviewed. Data collection took place from November 15th ‐ 21st, 2016.   Android phones were used for data collection, complemented with SurveyCTO software. The use of mobile devices and an electronic questionnaire strengthened data quality by allowing data validation rules and consistency checks that were integrated in the SurveyCTO software program. The mobile‐ based data collection process eliminated the data entry burden, as data was entered at the enumerator level and records were uploaded to a cloud server.   2.7 Data Entry and Analysis The SurveyCTO dataset (CSV format) was converted into an SPSS (Version 21) database for data management and analysis. SPSS statistical software was used to analyze the dataset. The data analysis and tabulation followed the definition of the indicators in the IPTT and PIRS, and includes descriptive statistics to summarize and describe the changes in the three outcome indicators since baseline measurement (where applicable).  Furthermore, data have been disaggregated according to the levels required by USAID/FFP, and estimates have been presented along with the confidence intervals associated with the indicators (see Table 5, Section 4). 4 3. Endline Survey Results A total of 600 FFS beneficiaries were sampled. In line with SNAP's approach to gender equity, more than half (55.7%) of the respondents were female. Figure 2: Respondent Sex Survey findings are organized in three content categories: (1) beneficiary farming activities; (2) adoption of improved agricultural technologies; and (3) association benefits. Each section includes results for the program‐specific indicators. The findings are presented as per the required disaggregations.   3.1 Beneficiaries' Farming Activities The survey first sought to gather information on which crops the beneficiaries grew in the last 12 months. For every crop they reported growing, respondents were then asked to report if they had sold any of the harvest. As highlighted in Table 2, the most common crops grown by the surveyed FFS beneficiaries included: groundnuts (95.3%), lowland rice (94.5%) and vegetables (91.8%). Survey results suggest that anywhere from seven to nine out of ten farmers made sales on each crop they grew in the last 12 months. This suggests that the majority of beneficiaries are engaged in agricultural activities beyond subsistence farming. Table 2: Crops grown and sold by surveyed FFS beneficiaries in the last 12 months Beneficiary grew the crop in the last 12 months Beneficiary made sales of the crop (Among respondents that grew the crop) Beneficiary made sales of the crop (Among all surveyed respondents) Crop Groundnuts 95.3% 89.2% 85.0% Lowland rice 94.5% 85.2% 80.5% Vegetables 91.8% 91.5% 84.0% Upland rice 87.5% 87.0% 76.2% Maize 84.8% 82.5% 70.0% Cassava 78.3% 82.3% 64.5% Sesame 75.5% 79.2% 59.8% Sweet potato 73.0% 84.2% 61.5% Cowpea 71.0% 77.2% 54.8% Pigeon pea 70.2% 74.6% 52.3% Sorghum 62.5% 71.5% 44.7% Male 44% Female 56% 5 3.2 Adoption of Improved Agricultural Technologies The survey sought to measure the respondents' adoption of improved agricultural technologies through their participation in the SNAP FFS. More specifically, the survey asked which (i) planting and crop production practices, (ii) post‐harvest processing practices and (iii) crop storage practices the respondent used in the previous 12 months when they planted and/or harvested the aforementioned crops. 3.2.1 Planting and Crop Production Practices The most commonly used planting and crop production practices over the previous year were reportedly: spacing between plants (89.7%), planting in rows (86.8%) and nursery/transplanting vegetables (68.2%). There was little variation between male and female respondents' adoption of new planting and crop production practices ‐ with the exception of females being more likely to practice thinning plants after germination and planting new types of crops. Figure 3: Planting and crop production practices used in the last 12 months 3.2.2 Post‐Harvest Processing Practices The most commonly used post‐harvest processing practices over the previous year were reportedly: shaking dirt off groundnuts (85.2%), harvesting quickly when the crop is mature (76.0%) and drying grains on tarpaulin (72.7%). There was no obvious variation between male and female respondents' adoption of new post‐harvest processing practices. Spacing between plants Planting in rows Nursery / transplant vegetables Earlier or on time planting More frequent weeding Harvest at the right time Thinning plants after germinati on New varieties of crops New types of crops Male 89.1% 87.2% 72.9% 56.8% 51.5% 39.8% 18.0% 18.0% 13.2% Female 90.1% 86.5% 64.4% 61.4% 50.9% 39.2% 24.0% 15.9% 18.6% Total 89.7% 86.8% 68.2% 59.3% 51.2% 39.5% 21.3% 16.8% 16.2% 6 Figure 4: Post‐harvest processing practices used in the last 12 months 3.2.3 Crop Storage Practices The most commonly used crop storage practices over the previous year were reportedly: storing crops in bags (90.8%), storing grain in elevated storage containers (68.7%) and storing crops in a well ventilated area (65.8%). There was little variation between male and female respondents' use of crop storage practices ‐ with the exception of males being more likely to store grain in an elevated storage container. Figure 5: Crop storage practices used in the last 12 months Shake dirt of groundnuts Harvest quickly when crop is mature Dry grains on tarpaulin Dry grains on drying floor Cover grain before threshing Dry grains for 2‐3 days Parboil rice and re‐dry before milling Male 86.5% 77.4% 71.4% 53.0% 38.7% 31.6% 18.0% Female 84.1% 74.9% 73.7% 51.5% 39.2% 33.2% 18.9% Total 85.2% 76.0% 72.7% 52.2% 39.0% 32.5% 18.5% Store crops in bags Store grain in elevated storage container Store crops in well ventilated area Store grain in cow dung covered basket None of the above Male 89.8% 72.9% 67.3% 28.2% 0.4% Female 91.6% 65.3% 64.7% 33.2% 0.0% Total 90.8% 68.7% 65.8% 31.0% 0.2% 7 Taking the above findings into consideration, the total percentage of farmers who reportedly   adopted and used at least three agricultural technologies in the last 12 months is 100%. This includes any of the three crop production practices, post‐harvest processing practices or crop storage practices noted above. While baseline data on the adoption of agricultural technologies was not collected, endline survey findings show that nearly all surveyed FFS members are using improved agricultural technologies. Furthermore, as shown in Table 3, the percentage of farmers using improved storage techniques for their crops has increased significantly from 4.7% at baseline to 99.8% at endline.   Table 3 : Indicator values related to use of agricultural technolgies and improved storage practices PERFORMANCE INDICATOR Baseline Result Endline Result Strategic Objective 2: Enhanced livelihoods for vulnerable people, especially women and youth OC17 % of farmers who have adopted at least three agricultural technologies ‐  100% IR 2.3: Improved community assets OC 24 % of farmers using improved storage   4.7% 99.8%    Male 4.6% 99.6%    Female 4.7% 100.0% 3.3 Association Benefits In order to measure the association benefits of SNAP FFS membership the survey asked beneficiaries a series of questions related to their use of various benefits, including collective storage, marketing and bulk purchasing among others. As highlighted in Figure 6 below, the majority of respondents reported that they had benefited from the exchange of knowledge with farmers in their SNAP FFS cluster (98.5%), from shared labour of a SNAP FFS cluster (96.5%) and from a SNAP FFS cluster group purchase of seeds, tools, or other inputs (90.3%). Figure 6: Association benefits of SNAP FFS membership Benefited from exhange of knowledge w/ farmers in SNAP FFS Benefited from shared labour of SNAP FFS Benefited from SNAP FFS group purchase of seeds, tools, etc. SNAP FFS made it easier to get credit or a loan Marketed crops through SNAP FFS Stored harvested crops in a bulk storage facility managed by SNAP FFS Male 98.5% 95.1% 92.5% 85.0% 68.8% 35.0% Female 98.5% 97.6% 88.6% 83.2% 70.7% 22.8% Total 98.5% 96.5% 90.3% 84.0% 69.8% 28.2% 8 When taken as a whole, nearly all surveyed beneficiaries reported that they were enjoying at least one association benefited through their membership in a SNAP FFS cluster. As highlighted in Table 4, 94.2% reported taking advantage of collective storage, collective marketing or bulk purchasing through the SNAP FFS. When the definition of association benefits is expanded to include knowledge exchange, shared labour and greater access to loans, the proportion reporting use of any association benefits increases to 99.7%.   Table 4: Indicator vales related to association benefits PERFORMANCE INDICATOR Endline Result OC16 % of farmers reporting use of SELECT association benefits *Associations benefits: collective storage, collective marketing and bulk purchasing 94.2% Male 94.7% Female 93.7% OC16b % of farmers reporting use of ANY association benefits *Associations benefits: collective storage, collective marketing, bulk purchasing, knowledge exchange, shared labour and greater access to loans 99.7% Male 99.6% Female 99.7% 9 4. Summary Statistical Comparison of Baseline and Endline Indicator Values Table 5 presents the endline values of the key outcome indicators related to the adoption of agricultural technologies and association benefits. A comparison of baseline with endline values of the one indicator that was measured at baseline (OC24) was conducted using a z‐test to detect a population level difference at a 95% level of significance. The test was used to determine if there was a statistically significant difference between the two study groups (i.e. baseline and endline). As highlighted below, there has been a significantly significant increase in the percentage of farmers using improved storage among project beneficiaries since the start of the intervention. Table 5: Summary statistical comparison of baseline and endline indicator values Ind Indicator BASELINE RESULT ENDLINE RESULT 95% confidence interval Z‐Test / T‐Test LCL UCL (Significance Test) OC 16 % of farmers reporting use of SELECT association benefits n/a 94.2% 92.3% 96.0% n/a OC 16b % of farmers reporting use of ANY association benefits n/a 99.7% 99.2% 100.1% n/a OC 17 % of farmers who have adopted at least three agricultural technologies n/a 100.0% 100.0% 100.0% n/a OC 24 % of farmers using improved storage* * Improved storage = Storage that provide adequate protection against rain, insects, pests 4.7% 99.8% 99.5% 100.2% z = 110.11, p = 0.000 10 Annex 1: Summary SNAP Performance Indicator Reference Sheet (PIRS) for Endline Indicators Ind. # Source Performance Indicator Tabulation/Definition OC 16 SNAP % of farmers reporting use of SELECT association benefits* *Associations benefits: collective storage, collective marketing and bulk purchasing # of farmers reporting use of SELECT association benefits _____________________________________________________________________________________________________________________________________________________________ Total number of famers surveyed   OC16b SNAP % of farmers reporting use of ANY association benefits* *Associations benefits: collective storage, collective marketing, bulk purchasing, knowledge exchange, shared labour and greater access to loans # of farmers reporting use of ANY association benefits _____________________________________________________________________________________________________________________________________________________________ Total number of famers surveyed   OC 17 FFP % of farmers who have adopted at least three agricultural technologies* *Agricultural technologies include: any 3 of the crop production practices, post‐harvest processing practices or crop storage practices   Number of farmers  who have adopted at least 3 of agricultural technologies _____________________________________________________________________________________________________________________________________________________________ Total number of famers surveyed   OC 24 SNAP % of farmers using improved storage* * Improved storage: Storage that provide adequate protection against rain, insects, pests   # of farmers who use improved storage techniques _____________________________________________________________________________________________________________________________________________________________ Total number of famers surveyed      11 Annex 2: Sampled FFS Clusters District Chiefdom Cluster Community No. Cluster Name # of members Koinadugu Diang Sandia 9 90 Koinadugu Diang & Nieni Foria 19 90 Koinadugu Neini Keindeya 28 90 Koinadugu Wara Wara Bafodia Sakuta 39 90 Koinadugu Wara Wara Bafodia Thealia 48 89 Koinadugu Folosaba Dembelia Koromasilaya 57 90 Koinadugu Folosaba Dembelia Malanga 1 67 90 Koinadugu Kasunko Kamadeka 76 90 Koinadugu Kasunko Katimbo 86 90 Koinadugu Neya Kurobola 95 90 Koinadugu Neya Koromandor 104 90 Koinadugu Mongo Sinikoro 113 90 Koinadugu Mongo Kromandor 123 90 Kailahun Mandu Dablama 134 90 Kailahun Mandu Sembehun 143 90 Kailahun Kissi Tongi Saama 155 90 Kailahun Yawei Malema 167 58 Kailahun Penguia Kanjara/Musaloh 177 78 Kailahun Penguia Kongonani 189 90 Kailahun Peje West Pejewa 201 65 Bombali Gbanti Kamaranka Borongor Morata 211 86 Bombali Gbanti Kamaranka Petifu 220 90 Bombali Sanda Loko Yana 230 90 Bombali Sanda Loko Maparay 242 60 Bombali Sella Limba Kathimbo 252 89 Bombali Sella Limba kakuru 262 87 Tonkolili Lower Yoni Makumkarie 271 80 Tonkolili Upper Yoni Matatie 282 90 Tonkolili Kafe Simira Makontaday 291 90 Tonkolili Kalansogoia Kanigba 302 90    12 Annex 3: Data Collection Tool Note: Electronic data collection tool available as a separate file upon request. 1                                                                                  SNAP END‐LINE FOLLOW‐UP SURVEY __‐ __ [Month] 2016 Follow‐Up Survey of Selected Agriculture Indicators   READ THE FOLLOWING STATEMENT TO EACH PERSON BEFORE THE INTERVIEW BEGINS “I am helping conduct a survey for SNAP and would like to ask you some questions about your household. Your participation in this survey is entirely voluntary.  All information you provide is confidential.  We will not tell anyone the answers you give, and your name will not be associated with any answers or any part of the survey process. Do you agree to participate in this interview?” Yes ______      No______ HHN Survey Questionnaire Number   DIST District   CD Chiefdom   SEC Section   COM Community   DAT Date           ______ [Month] 2016 FFS Are you a member of a SNAP Farmer Field School Cluster? Yes..................................................... 1 No...................................................... 2 SEX What is the gender of the person being interviewed Male.................................................. 1 Female .............................................. 2 EN Enumerator’s name _______________________________ TLN Team leader’s Name and signature Name _________________________ Signature_______________________ 2 Q1: Which of the following crops did you, or another household member, produce on your own household farm during the past 12 months? Crop 1Did you produce? Yes=1 No=2 2Did you sell any ____? Yes=1 No=2 1Lowland rice    2Groundnut    3 Cassava    4Sweet potato    5Upland rice     6Cowpea     7Pigeon pea     8Maize     9Sorghum     10Sesame     11Vegetables     QUESTION 2:  During your participation in SNAP Farmer Field School and Cluster activities you learned about new planting and crop production practices, which of those practices did you use when you planted the crops you mentioned above? Do not read the responses. Circle the number after the practices that are mentioned by the farmer. This question can have MORE THAN ONE ANSWER. Circle the number of ALL that the person mentions.  If they don’t mention any of the choices on the right, circle 10 for “None of the above” 1Earlier or on time planting...............1 2Planting in rows...............................2 3Spacing between plants...................3 4More frequent weeding ..................4 5Thinning plants after germination...5 6Nursery/transplanting vegetables...6 7 New types of crops..........................7 8New varieties of crops.....................8 9Harvest at the right time .................9 10None of the above........................10 3 QUESTION 3:  During your participation in SNAP Farmer Field School and Cluster activities you learned about new post‐harvest processing practices, which of those practices did you use after you harvested the crops you mentioned above? Do not read the responses. Circle the number after the practices that are mentioned by the farmer. This question can have MORE THAN ONE ANSWER. Circle the number of ALL that the person mentions.  If they don’t mention any of the choices on the right, circle 8 for “None of the above” 1Harvest quickly when crop   is mature ...........................................1 2Shake the dirt off harvested groundnuts........................................2 3Cover grain before threshing...........3 4Dry grains on tarpaulin ....................4 5 Dry grains on a drying floor.............5 6Dry grains for 2‐3 days.....................6 7Parboil rice and re‐dry   before milling....................................7 8None of the above...........................8 QUESTION 4:  During your participation in SNAP Farmer Field School and Cluster activities you learned about new crop storage practices, which of those practices did you use after you harvested the crops you mentioned above? Do not read the responses. Circle the number after the practices that are mentioned by the farmer. This question can have MORE THAN ONE ANSWER. Circle the number of ALL that the person mentions.  If they don’t mention any of the choices on the right, circle 5 for “None of the above” 1Store grain in cow dung   covered basket..................................1 2 Store grain in elevated storage container..............................2 3Store crops in well ventilated area..3 4Store crops in bags ..........................4 5None of the above...........................5 Ask the following 6 questions and circle 1 for “Yes” or 2 for “No”. Q5 During the past 12 months have you stored any of your harvested crops in an bulk storage facility managed by the SNAP FFS Cluster? Yes..................................................... 1 No...................................................... 2 Q6 During the past 12 months have you benefited from a SNAP FFS Cluster group purchase of seeds, tools, or other inputs ? Yes..................................................... 1 No...................................................... 2 Q7 During the past 12 months have you marketed any of your crops through a SNAP FFS Cluster? Yes..................................................... 1 No...................................................... 2 Q8 Have you benefited from the shared labor of a SNAP FFS Cluster group? Yes..................................................... 1 No...................................................... 2 Q9 Have you benefited from the exchange of knowledge with other farmers in your SNAP FFS Cluster? Yes..................................................... 1 No...................................................... 2 Q10 Has the SNAP FFS Cluster made it easier for you to get credit, or a loan, or loan from a revolving fund? Yes..................................................... 1 No...................................................... 2