November 2013 This publication was produced at the request of the United States Agency for International Development. It was prepared independently by Moneval Solutions Ltd (www.moneval-solutions.com) under contract to ACDI VOCA. Mid-Term Evaluation Report: Sustainable Nutrition and Agriculture Promotion (Sierra Leone) PHOTO CREDIT: All photographs used in this report were taken by the evaluation team members, Ange Tingbo and Daphyne Williams, during the course of the evaluation. MID-TERM EVALUATION REPORT SUSTAINABLE NUTRITION AND AGRICULTURE PROMOTION (SNAP) November 01, 2013 DISCLAIMER The author’s views expressed in this publication do not necessarily reflect the views of the United States Agency for International Development or the United States Government. SNAP MTE – Final Report Moneval Solutions Ltd Confidential Page i Table of Contents 1 EXECUTIVE SUMMARY ....................................................................................................... 1 1.1 MAIN FINDINGS ................................................................................................................................................. 1 1.2 RECOMMENDATIONS........................................................................................................................................ 2 2 BACKGROUND ..................................................................................................................... 3 2.1 OVERVIEW OF PROJECT STRATEGIES .............................................................................................................. 3 2.2 PROGRAM HISTORY AND OPERATING CONTEXT ....................................................................................... 4 2.3 EVALUATION METHODOLOGY ....................................................................................................................... 6 2.3.1 Evaluation Methodology Design ................................................................................................................................ 6 2.3.2 Evaluation Questions and Guides ............................................................................................................................. 7 3 EVALUATION PURPOSE AND OBJECTIVES .................................................................. 8 4 STRATEGIC OBJECTIVE 1 ................................................................................................. 10 4.1 INTERMEDIATE RESULT 1.1: IMPROVED ACCESS TO QUALITY NUTRITION AND HEALTH SERVICES FOR WOMEN AND CHILDREN AGED 0–59 MONTHS.............................................................................................. 11 4.1.1 Description of Interventions.....................................................................................................................................11 4.1.2 Service Delivery Strategies and Approaches .......................................................................................................11 4.1.3 Implementation Progress and Achievements of Results ..................................................................................14 4.1.4 Other Achievements ..................................................................................................................................................16 4.1.5 Discussion of General Evaluation Questions .......................................................................................................16 4.2 INTERMEDIATE RESULT 1.2: IMPROVED HEALTH AND NUTRITION PRACTICES AT THE HOUSEHOLD LEVEL ....................................................................................................................................................... 16 4.2.1 Description of Interventions.....................................................................................................................................16 4.2.2 Service Delivery Strategies and Approaches .......................................................................................................17 4.2.3 Implementation Progress and Achievements of Results ..................................................................................20 4.2.4 Other Achievements ..................................................................................................................................................21 4.2.5 Discussion of General Evaluation Questions .......................................................................................................22 5 STRATEGIC OBJECTIVE 2 ................................................................................................. 23 5.1 INTERMEDIATE RESULT 2.1: INCREASED AGRICULTURAL PRODUCTION AND CROP DIVERSIFICATION THROUGH IMPROVED INPUTS AND AGRICULTURAL PRACTICES............................................ 24 5.1.1 Description of Interventions.....................................................................................................................................24 5.1.2 Service Delivery Strategies and Approaches .......................................................................................................24 5.1.3 Implementation Progress and Achievements of Results ..................................................................................25 5.1.4 Other Achievements ..................................................................................................................................................25 5.1.5 Discussion of General Evaluation Questions .......................................................................................................26 5.2 INTERMEDIATE RESULT 2.2: ENHANCED INTEGRATION INTO AGRICULTURAL VALUE CHAINS ......... 26 5.2.1 Brief Description of Interventions ..........................................................................................................................26 5.2.2 Service Delivery Strategies and Approaches .......................................................................................................28 5.2.3 Implementation Progress and Achievements of Results ..................................................................................29 5.2.4 Other Achievements ..................................................................................................................................................29 5.2.5 Discussion of General Evaluation Questions .......................................................................................................29 5.3 INTERMEDIATE RESULT 2.3: INCREASED HUMAN CAPACITY AND COMMUNITY ASSETS ..................... 29 5.3.1 Description of Interventions.....................................................................................................................................29 5.3.2 Service Delivery Strategies and Approaches .......................................................................................................30 5.3.3 Implementation Progress and Achievements of Results ..................................................................................31 5.3.4 Other Achievements ..................................................................................................................................................31 5.3.5 Discussion of General Evaluation Questions .......................................................................................................32 SNAP MTE – Final Report Moneval Solutions Ltd Confidential Page ii 5.4 INTERMEDIATE RESULT 2.4: INCREASED ACCESS TO FINANCIAL RESOURCES........................................ 32 5.4.1 Description of Interventions.....................................................................................................................................32 5.4.2 Service Delivery Strategies and Approaches .......................................................................................................32 5.4.3 Implementation Progress and Achievements of Results ..................................................................................33 5.4.4 Other Achievements ..................................................................................................................................................34 5.4.5 Discussion of General Evaluation Questions .......................................................................................................34 6 PROGRAM QUALITY AND CROSS-CUTTING THEMES ............................................. 35 6.1 PARTNERSHIP/CONSORTIUM QUALITY........................................................................................................ 35 6.1.1 Program Management, Coordination, and Operations ...................................................................................36 6.2 TARGETING...................................................................................................................................................... 39 6.2.1 Strategic Objective 1 .................................................................................................................................................39 6.2.2 Strategic Objective 2 .................................................................................................................................................39 6.3 GENDER............................................................................................................................................................ 41 6.4 SOCIAL AND BEHAVIORAL CHANGE ............................................................................................................ 43 6.5 INTEGRATION .................................................................................................................................................. 43 6.6 SUSTAINABILITY/EXIT STRATEGIES ................................................................................................................ 44 6.7 ENVIRONMENT STEWARDSHIP ...................................................................................................................... 45 6.8 RESILIENCY TO SHOCKS ................................................................................................................................. 46 6.9 PRODUCTIVE YOUTHS.................................................................................................................................... 47 7 IMPLEMENTATION PROCESSES ..................................................................................... 48 7.1 COMMODITY MANAGEMENT ........................................................................................................................ 48 7.2 MONITORING AND EVALUATION................................................................................................................. 50 7.3 KNOWLEDGE MANAGEMENT........................................................................................................................ 51 7.4 GENERAL MANAGEMENT ............................................................................................................................... 52 7.5 PROGRAM DESIGN AND IMPLEMENTATION................................................................................................. 52 8 LESSONS LEARNED AND BEST PRACTICES................................................................ 53 9 RECOMMENDATIONS ....................................................................................................... 55 9.1 HIGH-PRIORITY RECOMMENDATIONS ......................................................................................................... 55 9.1.1 Strategic Objective 1 .................................................................................................................................................55 9.1.2 Strategic Objective 2 .................................................................................................................................................59 9.2 MEDIUM-PRIORITY RECOMMENDATIONS .................................................................................................... 62 9.2.1 Strategic Objective 1 .................................................................................................................................................62 9.2.2 Strategic Objective 2 .................................................................................................................................................62 9.3 LOW-PRIORITY RECOMMENDATIONS.......................................................................................................... 64 9.3.1 Strategic Objective 1 .................................................................................................................................................64 9.3.2 Strategic Objective 2 .................................................................................................................................................65 10 LIMITATIONS TO THE EVALUATION ........................................................................... 66 ANNEX 1: EVALUATION SOW.............................................................................................. 69 ANNEX 2: EVALUATION PLAN AND SCHEDULE ............................................................. 88 ANNEX 3: COMPOSITION OF THE TEAM ........................................................................... 91 ANNEX 4: EVALUATION METHODS AND TOOLS............................................................ 93 ANNEX 5: LIST OF MEETINGS AND SITES VISITED ....................................................... 121 ANNEX 6: UPDATED INDICATOR PERFORMANCE TRACKING TABLES (IPTT) ...... 125 ANNEX 7: SUMMARY TABLES ON COMMODITIES AND HUMAN RESOURCES ....... 130 ANNEX 8: MESSAGE FROM ONE KII ................................................................................... 130 SNAP MTE – Final Report Moneval Solutions Ltd Confidential Page iii ANNEX 9: SUGGESTED FOOD DISTRIBUTION METHODOLOGY ............................... 131 ANNEX 10: DISCLOSURE OF ANY CONFLICTS OF INTEREST .................................... 133 SNAP MTE – Final Report Moneval Solutions Ltd Confidential Page iv Acknowledgements The Mid-Term Evaluation (MTE) team would like to thank all the beneficiaries and SNAP partners encountered during the field trip. We would like to express our gratitude to the local authorities in the chiefdoms and districts of Bombali, Koinadugu, Kailahun, and Tonkolili, the representatives of the Ministry of Health and Sanitation (MoHS), and the Ministry of Agriculture, Forestry and Food Security (MAFFS) at the district level. We appreciate the time they spent with us and the valuable information they shared. Their input was instrumental in understanding the impact of SNAP program activities. We are also grateful to all staff at the national and district levels for their availability to answer our questions, for supporting us in the field, and for providing the numerous documents requested. This enabled the MTE team to develop an in-depth understanding of the program. As with most evaluations, the people implementing the program were a valuable source for identifying opportunities to adjust program activities. We thank our translators in each chiefdom; their knowledge of the language and culture of the locale was instrumental in communicating with the communities and stakeholders. Our thanks go to Mohamed Kamara and Samuel Jalloh, who drove us around rugged terrain safely during the rainy season in Sierra Leone. They were also valuable cultural guides. The ACDI/VOCA office in Freetown provided outstanding support throughout the MTE. We appreciate the efforts of the Administration Department in providing logistical support from our arrival at Lungi Airport and throughout the field visits to the four districts visited by the MTE team. We thank Sam Juana, the SNAP Monitoring and Evaluation (M&E) Manager, who efficiently helped us keep on track in the field, and Dr. Nirinjaka Ramasinjatovo, the ACDI/VOCA Regional M&E Officer, who spent some days with the team sharing the realities of field conditions. We also thank Jacob Gray and Dr. Nirinjaka Ramasinjatovo from ACDI/VOCA for the constructive feedback provided on the draft and final reports. A final thank you goes to Georgia Beans, Chief of Party (CoP), and Seraphine Olanya, Deputy Chief of Party (DCoP), for their full involvement and support in the MTE process. Ange Tingbo, Team Leader and SO2 Evaluator Daphyne Williams, Health and Nutrition Expert and SO1 Evaluator SNAP MTE – Final Report Moneval Solutions Ltd Confidential Page v ACRONYM LIST ANC Ante Natal Care APRs Annual Project Reports ARRs Annual Results Reports BEAs Bloc Extension Agents CG Care Group CHWs Community Health Workers C-IMNCI Community-Integrated Management of Neonatal and Childhood Illnesses CMAM Community-Based Management of Acute Malnutrition CMT Commodity Management Team CoP Chief of Party CSB Corn Soy Blend CRS Commodity Status Report CTC Compliance Technical Coordinator CU2 Children Under Two Years Old DAO District Agriculture Officer DC District Council DCoP Deputy Chief of Party DHMT District Health Management Team DOMs District Office Managers ESR (annual) Environmental Status Report EHA Essential Hygiene Actions ENA Essential Nutrition Actions FaaB Farming as a Business FaaFB Farming as a Family Business FANTA Food and Nutrition Technical Assistance FBO Farmer Business Organization FIFO First In, First Out FFP Food for Peace FFSs Farmer Field Schools FGDs Focus Group Discussions FOM Field Office Manager FY Fiscal Year GDP Gross Domestic Product GIAP Gender Integrated Action Plan GoSL Government of Sierra Leone HRW Hard Red Winter (wheat) HDI Human Development Index HP Health Promoter HQ Headquarters HR Human Resources IEE Initial Environmental Examination SNAP MTE – Final Report Moneval Solutions Ltd Confidential Page vi IFPRI International Food Policy Research Institute IGA Income Generating Activities IMC International Medical Corps IMNCI Integrated Management of Neonatal and Childhood Illnesses IPM Integrated Pest Management IPTT Indicator Performance Tracking Table IR Intermediate Result ITSH Internal Transportation, Storage and Handling IYCF Infant and Young Child Feeding KII Key Informant Interview LM Lead Mother MAHFP Months of Adequate Household Food Provisioning M&E Monitoring and Evaluation MCG Mother and Child Care Group MCHP Maternal and Child Health Post MAFFS Ministry of Agriculture Forestry and Food Security MoHS Ministry of Health and Sanitation MoU Memorandum of Understanding MT Metric Ton MTE Mid-Term Evaluation MUAC Mid Upper Arm Circumference MYAP Multi-Year Assistance Program NGO Non-Governmental Organization OICI Opportunities Industrialization Centers International OTP Outpatient Therapeutic Program PBF Performance-Based Financing PHU Peripheral Health Unit PLW Pregnant and Lactating Women PM2A Preventing Malnutrition in Children Under Two Approach PMP Performance Management Plan PREP Pipeline and Resource Estimate Proposal RSRs Recipient Status Reports SAM Severe Acute Malnutrition SCP Smallholder Commercialization Program SG Savings Group SNAP Sustainable Nutrition and Agriculture Promotion SO Strategic Objective SOP Standard Operating Procedure SOW Scope of Work TFOs Training Field Officers TOTs Training of Trainers SNAP MTE – Final Report Moneval Solutions Ltd Confidential Page vii USAID United States Agency for International Development VDCs Village Development Committees VSLAs Village Savings and Loan Associations WASH Water, Sanitation and Hygiene SNAP MTE – Final Report Moneval Solutions Ltd Confidential Page 1 1 Executive Summary This report details the results of the Mid-Term Evaluation (MTE) of the Sustainable Nutrition and Agriculture Promotion (SNAP) program in Sierra Leone. This is a five-year United States Agency for International Development (USAID) Food for Peace (FFP) Multi-Year Assistance Program (MYAP). The overall goal of SNAP is to reduce food insecurity and increase resiliency among the most food insecure and vulnerable rural populations. The program targeted just over 400,000 people. The purpose of the MTE was to assess progress towards targets and to provide evidence-based recommendations. The scope of the MTE included administration, operations, management, technical implementation, and cross-cutting themes. A qualitative MTE was conducted over a four￾week period (June-July 2013) to assess the impact of SNAP. The evaluation methodology included discussions with stakeholders, Key Informant Interviews (KIIs), Focus Group Discussions (FGDs), and site visits. Evaluation questions in the SOW were core to the data collection activities and included a focus on the appropriateness of activities, their quality, and adoption of new practices or behavior toward health and post-natal care and nutrition among other issues. 1.1 Main Findings Overall, the program has made good progress in nearly all of its proposed activities. In line with the Indicator Performance Tracking Table (IPTT), and the 2013 Pipeline and Resource Estimate Proposal (PREP) to USAID FFP, the cumulative targets of Year 1 and Year 2 have been met and in some cases been exceeded. SNAP is well on its way to meeting the majority of its objectives, despite being only halfway through its implementation period, and many results are already visible. Beneficiaries have been trained in both Strategic Objectives (SOs), which has led to: 1) changes in behavior at the household and community levels; 2) improved services at health facilities; 3) knowledge of new farming techniques with tangible outcomes; 4) formerly illiterate beneficiaries signing with their own names; and 5) improved health outcomes among children. The gender component is also having a slow but positive impact on household interactions. Important findings include staff turnover which has had a negative impact on the supervision of PHUs. Exiting and transition under SO1 has not been addressed and no plan is complete for financing Care Groups, Lead Mothers (LMs) or Health Promoter (HPs) activities beyond the close of SNAP. Related to exiting and transition from the feeding program, PM2A beneficiaries know they will leave the program when their child reaches two years of age. However, few can articulate a strategy about how they will feed their child beyond this, especially as their child is used to the taste of the Corn Soy Blend (CSB). Some mothers indicate they have difficulty introducing their child to new foods after graduation. Finally, some LMs are beginning to graduate from SNAP but many have expressed the desire to continue in their role and have approached HPs or other SNAP staff to discuss this. Monitoring and evaluation aspects of SNAP could be improved to allow more relevant data to be collected and analyzed. On-the-spot corrections of activities are not recorded. SNAP has no mechanism to assess the degree to which messages are appropriate, received correctly, or require revisions over the entire program. SNAP MTE – Final Report Moneval Solutions Ltd Confidential Page 2 Significant progress has been made with Farmer Field School (FFS) to increase productivity and diversification. SNAP should upgrade the training of FFS facilitators according to the Ministry of Agriculture, Forestry and Food Security (MAFFS) curriculum, and systematize joint visits with MAFFS FFS agents. Although ACDI/VOCA has developed and deployed a comprehensive set of strategies to realize IR 2.2 as a special case, it has been difficult to assess the quality and success because no activities following the training have taken place. Other activities, such as the Village Savings and Loan Associations (VSLAs) are behind schedule. There are a number of issues affecting SO1, including the lack of funds resulting from poor monetization proceeds. SNAP staff have developed an adequate plan in the 2013 PREP to address these issues. Financial constraints resulting from the low cost recovery of the monetization proceeds mean a number of community activities have been placed on hold, including construction of drying floors, grain stores, rehabilitation of farm-to-market roads and water wells. 1.2 Recommendations Of the 33 recommendations formulated, the highlights are summarized below. High Priority Recommendations include introducing a Standard Operating Procedure (SOP) to help staff report more effectively and provide clear and structured training to mitigate staff turnover. Cascade training should be introduced to ensure better health adoption. Transition and exit planning needs to be more clearly developed and announced to stakeholders and local partners to avoid any surprises. SNAP needs to formulate a plan to construct and/or distribute tippy taps more effectively than it has currently. SNAP’s planned tippy tap distribution should be firmed up and systematized. ACDI/VOCA should update the outcomes of the VSLAs census and draft an implementation plan with a timeline for achievement. Many VLSAs are in various stages of early development and need technical and managerial support. Given the time limitations SNAP should focus on supporting existing VSLAs rather than creating new ones. Medium Priority Recommendations include capturing the support system for children under two (CU2) not currently in the program. Within the SNAP implementation area there are CU2s whose mothers were not registered by SNAP, and this group is falling through the cracks. The management team should consider strategies to involve these caregivers in supportive nutrition activities. Given the high staff turnover, field staff should have more instruction as to what constitutes a success story. SNAP has made great strides in changing food taboos, and documentation of this achievement will help maintain institutional knowledge. Some demonstration plots did not display any technical information regarding a farming method or test crop. SO2 staff should agree to change the educational material in line with best practices such as increasing the pictorial usage in the training documentation. SNAP should organize exchange visits between FFSs within the same chiefdoms and/or between districts to stimulate weak groups and assist them to become better farmers. Low Priority Recommendations include SNAP promoting the uptake of tippy taps by demonstrating their construction using locally available materials. In addition, effort to recruit women as HPs should be documented and reviewed to identify challenges and lessons learned that may assist future planning. SNAP MTE – Final Report Moneval Solutions Ltd Confidential Page 3 2 Background Sierra Leone was left in a state of desolation after a decade of civil war between 1991 and 2002, which devastated the country’s infrastructure, economy, and stability. The infrastructure was widely degraded, and food insecurity became a significant issue. National statistics for the first few years of the new century showed that 70% of Sierra Leone’s population lived in poverty, with 26% in extreme poverty. Human capacity was severely impacted, with consequences continuing to be felt today. Many university teachers were killed or went into exile and the school system was marred with sporadic strikes. As a result, technical talents became scarce. In 2009, the International Food Policy Research Institute (IFPRI) ranked Sierra Leone among the top five countries with the highest Global Hunger Index score and among the top six countries most severely affected by, and vulnerable to, the global economic downturn. The United Nations Development Program’s Human Development Index (HDI), which takes a composite of factors including the Gross Domestic Product (GDP) into account, ranked Sierra Leone as the 180th (out of 182) least developed country. Notably, over one third of children under the age of five suffer from chronic malnutrition, with most of the rural population suffering from four-month-long lean seasons. One in eight women in Sierra Leone die from pregnancy-related causes. To combat the above problems, the SNAP program proposes to implement an integrated, multi￾pronged approach addressing the root causes of food insecurity. The overall goal is to reduce food insecurity whilst at the same time increasing resiliency among the most food insecure and vulnerable rural populations in the target districts. 2.1 Overview of Project Strategies SNAP utilizes a multi-pronged approach to address the root causes of food insecurity. Activities focus on health care service delivery; household health, hygiene, and nutrition practices; agricultural production; post-harvest handling; agro-enterprise development; skills building for employment; and access to cash. Monthly food distribution has included targeting Pregnant and Lactating Women (PLW), their children, and an additional allocation (known as “family rations”) given to PM2A beneficiary households during the lean season (from June to September). It is noted that PM2A does not focus on food distribution alone, but combines health capacity building, behavior change, and children’s welfare. Importantly, the approach includes strategies to put in place a long-term, sustainable improvement in health, sanitation, and nutrition practices within target communities. In aiming to find an appropriate mechanism to address the challenges encountered by the poorly performing agricultural sector, SNAP activities have revolved around a training program to develop new skills and practices. The training focuses on improving marketing, processing, storage, and crop retention throughout the year, which can boost lean-season consumption and income. SNAP was designed to enhance marketing channels for farmers by establishing and strengthening producer and market associations and training a network of village-based traders. Through this agricultural objective, SNAP intends to implement activities that boost agriculture and livelihood opportunities. This would increase agricultural production, productivity, and post-harvest handling whilst safeguarding the link between agriculture and nutrition. Initially, the program was SNAP MTE – Final Report Moneval Solutions Ltd Confidential Page 4 meant to strengthen existing FFSs and establish new ones in communities where this strategy was not known or perhaps poorly implemented. The program anticipated that profit made from farming would increase by improving their integration and participation in existing agricultural value chains, such as those for rice, cassava, sorghum, pigeon pea, sesame, oil palm, groundnuts, sweet potato, and garden vegetables. The Farming as a Family Business (FaaFB) approach was subsequently adopted to enable farmers to become more competitive while addressing gender inequity at the household level. In addition, SNAP has provided literacy, numeracy, and vocational training to youths to better prepare them for jobs. In exchange, the youths have provided unskilled labor to rebuild their communities at large. Finally, the establishment and improvement of VSLAs is intended to increase the cash supply for both social and productive purposes and to provide more opportunities for women as equal partners in community development. SNAP has two essential SOs, with each having a number of IRs, and four major cross-cutting themes run throughout the program. The SOs, IRs, and cross-cutting themes are summarized below: SO 1: Reduce chronic malnutrition among children under five:  IR 1.1: Improved access to quality nutrition and health services for women and children aged 0–59 months; and  IR 1.2: Improved health and nutrition practices at the household level. SO 2: Enhance livelihood opportunities:  IR 2.1: Increased agricultural production and crop diversification through improved inputs and agricultural practices;  IR 2.2: Enhanced integration into agricultural value chains;  IR 2.3: Increased human capacity and community assets; and  IR 2.4: Increased access to financial resources. Key Cross-cutting themes  Gender equity;  Environmental stewardship;  Resiliency to shocks; and  Productive youth. 2.2 Program History and Operating Context In 2009, ACDI/VOCA and IMC conducted an assessment, supplemented by a desk review, which concluded that while various causes have contributed to food insecurity in Sierra Leone, poor utilization of food is the greatest underlying cause. Limitations in women’s empowerment and general knowledge of health, nutrition, and hygiene practices led to poor utilization. Exacerbating these limitations were low yields, high post-harvest losses due to improper infrastructure and SNAP MTE – Final Report Moneval Solutions Ltd Confidential Page 5 technologies, and a lack of market channels. Whilst agriculture as a livelihood did not generate adequate income, few other means existed to generate an income. The timing of cash requirements was regarded as a key reason why farmers were selling their produce, rather than saving it to combat seasonal shortages. In addition, unemployment has continued to be a significant barrier to food security, with many people lacking the marketable skills needed to start a business or even find a job.1 Following on from its assessment, ACDI/VOCA submitted an application to USAID FFP to deliver a comprehensive and integrated program to reduce food insecurity and chronic malnutrition in Sierra Leone. The Office of FFP works to reduce hunger and malnutrition and ensure that all people at all times have access to sufficient food for a healthy and productive life. It monitors regional food insecurity and works with field-based partners to save lives, tackle chronic under-nutrition, and help the most vulnerable break the cycle of poverty and hunger. Food for Peace programs combine relief and development efforts so that humanitarian assistance contributes to the long-term benefits for local populations. The application was successful and the SNAP program began in July 2010, and is due to end in June 2015. The SNAP program intends to reach 405,049 vulnerable people in 18 of the most malnourished and impoverished chiefdoms in four districts of Sierra Leone: Bombali (three chiefdoms), Kailahun (five chiefdoms), Koinadugu (seven chiefdoms), and Tonkolili (three chiefdoms). It will provide the targeted population with the necessary tools to improve their food security situation by addressing the key factors influencing utilization, availability, and access to nutritious and diverse foods. SNAP is being implemented by a consortium led by ACDI/VOCA, which includes IMC to lead on health and nutrition activities and Opportunities Industrialization Centers International (OICI) to lead youth vocational education activities, while ACDI/VOCA leads on food insecurity and agricultural development. The total program budget request of USAID FFP was $25,425,946 to be generated from commodity monetization, $7,757,334 from Section 202(e), and $5,568,377 from Internal Transportation, Storage, and Handling (ITSH). Additionally, there was a non-U.S. Government contribution from ACDI/VOCA estimated at $1,447,723, and other in-kind resources estimated to be $1,250,000. The approved level of food commodities for distribution by USAID FFP was 2,093 metric tons (MT) of CSB, 4,904 MT of vegetable oil, 6,183 MT of bulgur, and 2,473 MT of lentils. With bilateral agreements with foreign countries in place, the GoSL has made attempts to change the operating context and improve the living conditions of its population. Many roads are still in an extremely poor condition, especially during the rainy season. It is widely accepted that in Africa, rural road construction has led to significant increases in agricultural production and the uptake of related services. The consequence of poor roads and a lack of transport services is that transport costs are high in the targeted villages, particularly during the rainy season and especially on poor-quality roads. Geographic distances and road conditions pose challenges for staff in the monitoring of some communities and for PM2A food recipients who must travel long distances to the food distribution points. Fuel scarcity in the capital and the districts is frequent, with a severe impact on the smooth running of many services and operations, including 1 Source: Original SNAP Proposal, January 2010 SNAP MTE – Final Report Moneval Solutions Ltd Confidential Page 6 those of SNAP. Field visits are often suspended while vehicles and motorcycles are grounded and repaired at high cost. Communication is unsurprisingly difficult, with many interruptions and dropped lines. Internet and phone coverage are reasonable when they function, with generally good communication with Freetown and external parties. However, communication with the interior is sporadic.. 2.3 Evaluation Methodology 2.3.1 Evaluation Methodology Design The MTE of the SNAP program was designed as a qualitative exercise. A key aspect of the approach was to ensure that the evaluation was conducted objectively, impartially, and in an open and participatory manner based on empirically verified valid and reliable evidence. The structured collection and management of the evaluation data allowed the MTE team to 1) attain an overview of data collected, 2) identify trends or patterns within the data related to the evaluation criteria, and 3) check for and identify any information gaps. The MTE activities were arranged in the following three phases: Preparation: 19 June–2 July 2013  Inception meeting;  Desk review;  Stakeholder discussions;  Inception report; and  Evaluation questions and guides. Data collection and preliminary analysis: 4 July–17 July 2013 The data collection activities consisted of discussions with stakeholders, FGDs, KIIs, and site visits. The MTE team held discussions with six key stakeholders prior to the fieldwork. The fieldwork covered 29 locations, while the team met with 647 beneficiaries, 160 informants, and visited 20 program sites. Stakeholder discussions:  Jacob Gray, Deputy Director, Community Development;  Maria Bina Palmisano, M&E Coordinator;  Georgia Sheehy Beans, CoP;  Niry Ramasinjatovo, Regional M&E Officer;  Emily Byron (Previous Director/Coordinator for the program under food security); and  Seraphine R. Olanya, Deputy Chief of Party (DCoP). FGDs:  For SO1 these were: 109 beneficiaries and participants consisting of four neighborhood mothers groups (55 people), five LM groups (47 women), and two Men as Partners groups (7 people); and SNAP MTE – Final Report Moneval Solutions Ltd Confidential Page 7  For SO2 these were: 538 beneficiaries and participants consisting of ten groups from FFS/communities (360 people), six groups representing vocational education services (58 people), and ten groups representing informal VSLAs (120 people). KIIs:  For SO1 these were: 26 staff members and 57 from government; and  For SO2 these were: 43 staff members and 34 from government. Site visits:  Villages: Bunumbu, Levuma, Mamaka, Karamanka, Kondembaia, Kafogo, Mandy Tawahuh, Ndgba, and Bumbuna; and  PHUs and Government: Kailahun DHMT, Kailahun District Council, Levuma PHU, Tonkolili DHMT, Mamaka PHU, Kaponkie MCHP, Bumbuna Chiefdom Clinic, Kamabaio, Koinadugu District Council, Bombali DHMT, and Kondembaia PHU. Interpretation and Reporting: 22 July–30 August 2013 This phase focused on analyzing the data collected through stakeholder discussions, a country visit, and program document review. The team members worked together to arrive at a common interpretation of findings and recommendations, with preliminary findings presented in country prior to the development of the draft and final reports. Please refer to the annex sections for further details on the methodology, tools, schedule, key informants, and communities and sites visited, and refer to Section 10 for limitations of the methodology. 2.3.2 Evaluation Questions and Guides The MTE team developed a series of questions and guides in collaboration with the SNAP team to support FGDs and KIIs. The 24 questions detailed in the SOW were central, and an additional 36 questions were identified by the MTE team. The final set of evaluation questions was reviewed by the SNAP team at HQ and in Freetown. The questions and guides were tailored for the various groups involved in the SNAP program (health/nutrition, agriculture, and vocational education), and used to steer the MTE team during interactions with the beneficiaries, communities, informants, and authorities. Please refer to Annex 4 for further details of the evaluation methodology and tools. SNAP MTE – Final Report Moneval Solutions Ltd Confidential Page 8 3 Evaluation Purpose and Objectives The purpose of the MTE was to assess progress towards meeting SNAP program targets and to provide evidence-based recommendations to strengthen implementation in the remaining years of the program. Global Objectives: The global objectives of the SNAP MTE were to:  Assess the program’s progress in meeting its expected results and the likelihood of attaining its outcomes and impact as defined in the agreed Performance Management Plan (PMP);  Re-examine the validity of the SNAP proposed hypothesis and assumptions set in the strategic objectives and results framework, and how the PMP appropriately reflects the capture of these;  Identify both challenges and successes across the program, and provide new insight into how to strengthen the program based on the successes and challenges; and  Document the program’s lessons learned and best practices. Specific Objectives: The specific objectives of the MTE were to:  Complete a comprehensive desk review of key documentation to assess the effectiveness of both programmatic and operational activities, processes, and procedures of SNAP ;  Complete and finalize a set of data collection instruments to be used in the field-based evaluation;  Input, process, and analyze the data collected;  Submit a written report providing initial analysis and recommendations to ACDI/VOCA’s CoP, DCoP, and M&E Manager;  Provide additional comment, support, answers to questions, and feedback to the ACDI/VOCA regional and HQ team in developing the final analysis and MTE report; and  Present a summary of findings to the SNAP program partners and USAID. Scope of Work: The MTE was expected to consider the SNAP program from implementation onwards, and document the impact to date, identify key lessons, and make actionable recommendations to support future planning and implementation. The key areas for the MTE were:  Progress towards program outcomes;  Program design and implementation;  Program management, coordination, and operations;  Monitoring and evaluation; and SNAP MTE – Final Report Moneval Solutions Ltd Confidential Page 9  Cross-cutting themes: o Gender equity; o Environmental stewardship; o Resiliency to shocks; and o Productive youth. Please refer to Annex 1 for further details regarding the scope of the work. SNAP MTE – Final Report Moneval Solutions Ltd Confidential Page 10 4 Strategic Objective 1 Strategic Objective 1 was designed to address the root causes of food insecurity for PLW and children under the age of five. It focuses on the prevention and treatment of malnutrition and capacity building for district and local government health care staff. It also includes health education and capacity building for 50,000 PLW and community members. Originally, SO1 operated with three IRs, but following program realignment during the course of implementation this was scaled down to two IRs. The current IRs are:  IR 1.1: Improved access to quality nutrition and health services for women and children aged 0–59 months; and  IR 1.2: Improved health and nutrition practices at the household level SNAP aims to address malnutrition in pregnant women and children under two through the implementation of PM2A. This is a food-assisted approach to reducing the prevalence of child malnutrition by targeting a package of health and nutrition interventions to all pregnant women and mothers of children aged between 0 and 23 months. It also includes children under two in food￾insecure program areas, regardless of nutritional status.2 SNAP combines the provision of supplementary food rations with health education, and also includes capacity-building of government health staff to increase the provision of quality health care services. SNAP uses the CG model as its community health education methodology, and trains and funds health promoters to act as trainers for LMs. Beneficiaries selected by other beneficiaries act as leaders and a primary source of delivering health education messages. Lead mothers are organized into CGs based on geography, while each LM has 10–15 neighborhood mothers with whom they conduct health education sessions. In addition to providing health education to PM2A beneficiaries, LMs help to identify pregnant women eligible for SNAP participation. The results for SO1 are generally good, with successes in developing access to better post-natal services, raising awareness of the importance of adopting a good feeding regime for children, and utilizing CG and LM instruments for technical and social support. These are beginning to lead to the sustained adoption of food security approaches and improved health practices. Good links with government have been created, and collaboration has been working well, with positive practices being considered for adoption. The achievements and findings for each SO1 IR are detailed in the sections below. 2 Title II Technical Reference Materials. TRM-01: Preventing Malnutrition in Children Under 2 Approach (PM2A): A Food-Assisted Approach. Revised November 2010. Washington DC: AED 2010. SNAP MTE – Final Report Moneval Solutions Ltd Confidential Page 11 4.1 Intermediate Result 1.1: Improved Access to Quality Nutrition and Health Services for Women and Children Aged 0–59 Months 4.1.1 Description of Interventions SNAP implements complementary health interventions to work in synergy to improve the quality of health and the nutrition service for women and children aged 0–59 months. These interventions include:  Training DHMT staff on pre-natal and post-natal care services for PLW;  Training MoHS staff on Integrated Management of Neonatal and Childhood Illnesses (IMNCI) and Community Health Workers (CHWs) on Community- Integrated Management of Neonatal and Childhood Illnesses (C-IMNCI);  Improving PHU hygiene and sanitation conditions;  Training community members in the detection of Severe Acute Malnutrition (SAM) and referral to treatment facilities;  Supporting PHUs through capacity building and the donation of drugs and other pharmaceuticals in the form of gifts in-kind; and  Educating communities on proper health-seeking behavior. 4.1.2 Service Delivery Strategies and Approaches ACDI/VOCA has developed a comprehensive set of strategies and approaches for the success of IR 1.1. An assessment of their quality, successes, and challenges is presented below. The key feature of this has been capacity building in health service delivery and increased access to appropriate health care at the community level. Quality: Strengthening and capacity building for government health structures is a major intervention and success of the SNAP implementation. SNAP has developed close and symbiotic working relationships with DHMTs that include training, capacity building, and joint supervisory (of DHMT and SNAP technical staff) visits and planning. This relationship with DHMTs increases their capacity to provide adequate and appropriate supervision for PHUs. Joint visits also allow SNAP staff to provide leadership and examples of how to conduct, supervise, and develop follow-up plans and document supervisory visits. SNAP improves access to quality nutrition and health services by engaging in capacity building and strengthening activities and training for three layers of service provider. This occurs at national, district, and local levels. All layers are engaged in a technical capacity through training to increase the quality of services for women and children under five years old. This includes Essential Nutrition Actions (ENA), IMNCI, pharmaceutical waste management, growth monitoring, and SNAP's programmatic strategies and theories of change. Successes: Since the start of SNAP activities, community participation Ante Natal Care (ANC), postnatal care, growth monitoring, and facility-based deliveries has increased. This is documented in both SNAP and government data collection, and is likely due to the community’s increased SNAP MTE – Final Report Moneval Solutions Ltd Confidential Page 12 awareness of the promotion, educational activities, and improved health care service delivery implemented by SNAP. Mothers now attend clinics and there are anecdotal reports that more men, fathers, and community members are encouraging women to attend. Challenges: Implementation of IR 1.1 is complicated by government ministry staff turnover and the turnover of SNAP staff. Although SNAP has provided significant and widespread training, those trained may leave for other employment opportunities. Furthermore, DHMTs and PHU staff have varying capacities and experience with Maternal and Child Health (MCH) strategies and implementation, health training, data collection, and management. This is likely to impact the speed with which new strategies can be implemented and monitored. Once trained, PHU staff are expected to cascade the training to other staff members, but there is no set structure to ensure this happens or a strategy to monitor the quality and content of information cascaded. 4.1.2.1 Linkages with Government SNAP has developed good relationships with the MoHS, which has helped enormously in the delivery of post-natal services. The relationship has provided a platform for sharing knowledge and information related to SNAP activities and the way in which national health services function and operate. At the national level, SNAP engages the MoHS in ongoing discussions relating to the implementation of the program, and participates in ministry-led NGO coordination meetings. Based on its awareness and involvement with SNAP programming, the MoHS has rolled out CG models in communities where SNAP is not operating. It has also adopted the Men as Partners strategy as best practice. Through KIIs undertaken by the MTE team, MoHS representatives expressed an interest in learning from SNAP how to integrate nutrition and agriculture strategies more effectively, given that this is a key strategy to improve long-term health and nutrition sustainability. SNAP field staff work closely with DHMTs, engaging them in training and increasing their capacity and ability to perform supportive supervision for PHUs. In addition, SNAP staff and DHMTs conduct joint supportive supervision visits, with SNAP providing transportation and mentoring for DHMTs. At the field level, SNAP participates in district council and DHMT-NGO coordination meetings and presents the progress of the program when requested; SNAP also plays a role in national campaigns, such as immunization days and breastfeeding weeks, and supports PHUs. It is also involved with other levels of government health Ministries by providing gifts in-kind, including drugs and other medical supplies, training, and routine supportive supervision. 4.1.2.2 Training and Activities SNAP has been delivering training to a range of target beneficiaries, including community workers and government health service providers. The program has enjoyed great success and attendance at sessions has been encouraging. SNAP and government (DHMT) joint visits have also been rewarding in terms of skills transfers and raising awareness. Health care education appears to be successful, and behavioral changes are beginning to be seen in health service delivery and uptake by mothers and other household members. Strengthen community services – SNAP has trained government CHWs to provide community screening for malnutrition. Identified cases are referred to PHUs that have been trained by SNAP in growth monitoring and referral to government emergency feeding facilities. SNAP MTE – Final Report Moneval Solutions Ltd Confidential Page 13 Train DHMT staff on pre-natal and post-natal care services for PLW – SNAP conducted pre-natal and post-natal care refresher training for the DHMTs, including Community-based Management of Acute Malnutrition (CMAM), reproductive health, emergency obstetric care, Infant and Young Child Feeding (IYCF), and IMNCI. In FY11, 1,881 individuals, and in FY12 1,978 individuals, were trained in maternal and child health strategies, including DHMT staff. However, the quality and experience of DHMTs does vary; three out of four DHMTs had members who had received SNAP training and participated in SNAP and DHMT joint supervisory visits to PHUs. They were more knowledgeable in subject areas and the program strategies and objectives, and better able to articulate the impact of the program on PLW as a result. Train MoHS staff on IMNCI and CHWs on C-IMNCI – SNAP has trained four DHMTs and 140 PHUs on IMNCI, and has trained more than 70 CHWs on C-IMNCI. In FY12, refresher training was conducted, with additional refresher sessions planned for FY13. In FY12, 93% of CHWs were trained in IMNCI protocols. Improve PHU hygiene and sanitation conditions – SNAP has improved sanitation and hygiene at health facility and household levels. It has provided and rehabilitated PHU latrines, hand-washing stations, and bio-sand filters for potable water where appropriate. In addition, SNAP also installed tippy taps in PHUs. In FY12/13, Rainbow Family (another NGO) began distributing bio-sand filters to all government health centers. As such, SNAP has discontinued its plan to distribute bio-sand filters to prevent duplication of services. Each PHU visited during the MTE had a SNAP bio-sand filter, which had been distributed prior to the arrival of Rainbow Family. Train community members in detection of SAM and referral to treatment facilities – SNAP has trained CHW to conduct monthly community-wide screenings for SAM. Cases are referred to the PHU for confirmation, and then to health centers that provide emergency, rehabilitative feeding. In FY13, SNAP conducted training in 433 Outpatient Therapeutic Program (OTP) feeding sites with a district nutritionist. In the second quarter of FY13, nearly 7,000 children were screened in the community for malnutrition, with 213 moderate or severe cases identified. The MoHS provided 100 Mid Upper Arm Circumference (MUAC) tapes for recently SNAP-trained CHWs to help in community screening. Support PHUs through capacity building and the donation of drugs and other pharmaceuticals in the form of gifts in-kind – SNAP provided PHUs with donated drugs and other pharmaceuticals and equipment as gifts in-kind. These inputs included incinerators, cold boxes, hand-washing stations, and antenatal cards where appropriate. Educate communities on proper health-seeking behaviors – SNAP provided extensive training to PHUs and DHMTs to increase their capacity to provide quality services. This action is reinforced through the activities of IR 1.2, which includes SNAP educating communities on proper health-seeking behaviors in care groups and working with community leaders and Men as Partners groups. Additionally, SNAP participation in, and support of, national health days has also helped reinforce health-seeking behavior messages. These messages appear effective in the promotion of facility-based births, growth monitoring, immunizations, and treatment of fever conditions such as malaria. 3 SNAP_FY13_Q2 Progress Report_ Final_5-3-2013 SNAP MTE – Final Report Moneval Solutions Ltd Confidential Page 14 4.1.3 Implementation Progress and Achievements of Results Despite challenges in terrain, staff turnover, and varying pre-intervention experience, SNAP appears to be successfully working to achieve its goals. It has conducted training and provided health-related inputs to PHUs and DHMTs. Of the 123 PHUs within the SNAP service area, the MTE team visited eight selected by SNAP staff (two in each district). Each PHU had received inputs through capacity-building and gifts in-kind. Despite a high staff turnover, each PHU visited had at least one member of staff that had participated in a SNAP capacity building training event, and each PHU reported participating in at least seven SNAP training sessions. Each PHU reported receiving regular, at least monthly, supportive supervision visits from SNAP district supervisors. The district supervisors generally complete a detailed checklist, including the availability of items and equipment, proper systems for SAM referrals, births, deaths, and other materials to facilitate documentation and delivery of services. The checklist, through its design, also provides an opportunity to identify action items. Field staff rely heavily on “on-the-spot” correction, which may or may not be documented and shared with senior staff or incorporated into program improvement. This strategy is intended to increase the uptake and utilization of the information conveyed in the training, resulting in the improvement of services. There are several challenges to supportive supervision, including PHU staff turnover, SNAP staff turnover, SNAP staff capacity, training to provide appropriate and relevant supportive supervision, and SNAP program data use for program improvement. In addition, SNAP staff participate in joint supportive supervision of PHUs with DHMT staff. During this supervision, district supervisors complete a checklist designed to capture the implementation of the training received, identify opportunities for “on-the-spot” corrective support, collect data, and provide a record of SNAP interaction with the PHU. Peripheral Health Unit staff and SNAP field staff have been trained on SAM screening, growth monitoring, and referral strategies to identify children at risk of malnutrition, even if they are not eligible to participate in SNAP. This system helps identify children at risk. However, children under 59 months often fall through the “cracks” in the system due to a variety of factors both within and outside the purview of SNAP. While SNAP staff provide supportive supervision, their main opportunity to review the accuracy of growth monitoring occurs during food distribution sessions. This does not allow for the careful review of growth charts or providing corrective supervision. Field observation during the MTE found beneficiaries’ growth-monitoring charts were inconsistent and often inaccurate. Table 1 shows the indicator description, current achievement towards its target, and an assessment by the MTE team. Of the 19 indicators listed, 12 can be said to have been achieved, it is too early to formulate an opinion on three of the indicators, two were revised in 2012 and show mixed progress, and two were unverified. All achieved indicators actually surpassed the expected target, with the number of trained CHWs nearly double the number originally planned. Table 1: Indicator Description, Current Achievement, and Assessment by the MTE SNAP MTE – Final Report Moneval Solutions Ltd Confidential Page 15 4 IPTT Attachment A FY12. Indicator Type Indicator Current Achievement (2012) Assessment4 IM 2 % of underweight children 0–59 months of age (weight-for-age z-score (WAZ) <-2 SD) (FFP SO1) - Too early to assess IM 3 % of stunted children 6–59 months (height￾for-age z-score (HAZ) < -2 SD) (FFP SO2) - Too early to assess OC 1 % of acutely malnourished children 6–59 months (weight-for-height z-score (WHZ) < - 2 SD or edema) - Too early to assess OP 1 # of targeted direct beneficiaries reached during the FY (F) 161% Achieved OP 2 # of PLW reached by US Government￾supported nutrition programs 151% Achieved OC 2 % of children 0–59 months regularly attending growth-monitoring sessions 110% Achieved OC 3 % of pregnant women completing four antenatal visits 134% Achieved OC 4 # of deliveries with a skilled birth attendant in USG-assisted programs 117% Achieved OC 5 % of mothers participating in postpartum/newborn visit within two days of birth in USG programs (F) 114% Achieved OC 6 % of mothers receiving three post-natal visits 120% Achieved OP 3 # of people trained in child health and nutrition through USG-supported health area programs (F) 120% Achieved OP 4 # of people trained in maternal/newborn health through USG-supported programs - Unverified OP 5 # of newborns receiving essential newborn care through USG-supported programs - Unverified SNAP MTE – Final Report Moneval Solutions Ltd Confidential Page 16 4.1.4 Other Achievements SNAP’s work in DHMT capacity building is having a positive impact on the way DHMTs function within their implementation areas. In most districts, DHMTs report utilizing strategies implemented by SNAP in other geographic areas to address the issues of malnutrition, promotion of locally available foods, use of the CG model, and engaging men in improving their family’s nutrition. 4.1.5 Discussion of General Evaluation Questions The evaluation questions detailed in the SOW were core to the data collection activities, with a number of further questions added by the MTE team during the preparation phase. The final set of evaluation questions was reviewed by the SNAP team at HQ and in Freetown. The target groups for these questions were the PHUs, DHMTs, LMs, and CGs. In addition, village leaders were interviewed in FGDs. Staff involved in implementing SO1 were interviewed using the KII approach, as were Men as Partners. Evaluation questions revolved around the level of service received and the degree to which this met the needs of the target group. Please refer to Annex 4, “Evaluation Methods and Tools,” for further details of the evaluation questions. 4.2 Intermediate Result 1.2: Improved Health and Nutrition Practices at the Household Level 4.2.1 Description of Interventions Using a household approach and PM2A beneficiaries as the base, SNAP has adopted a three-pronged approach to improve household practices. This includes utilization of HPs and LMs, establishment of Men as Partners groups, and capacity building for government health workers. In addition, SNAP identifies and hires community members as HPs. They must be literate, able to travel to cover an entire CG area, ideally use a SNAP-provided bicycle, and must be able, and ready, to train LMs. Health promoters are in fact key data collectors, providing written reports for SNAP OP 6 % of health staff members trained in IMNCI protocol 135% Achieved OP 7 % of IMNCI-trained staff observed correctly applying skills during supportive supervision 110% Achieved OP 8 % of CHW trained in C-IMNCI protocol 186% Achieved OP 9 % of health facilities with standard growth￾monitoring equipment 129% Achieved OP 10 % of health facilities with functional bio-sand filters 87% Revised in 2012 OP 11 % of health facilities with new or rehabilitated WASH infrastructure (latrines, water points, or hand-washing stations) 31% Revised in 2012 SNAP MTE – Final Report Moneval Solutions Ltd Confidential Page 17 district supervisors, and monitors of LM training sessions. Currently, all SNAP health promoters are male. Lead Mother is a voluntary position, with candidates identified by other beneficiaries to receive training and additional responsibility. Currently, SNAP has 1,7375 LMs, each responsible for the education of 10–15 neighborhood mothers on key ENA, Essential Hygiene Actions (EHA), and nutrition education conveyed in a four-module SNAP curriculum. While SNAP has a target of reaching 50,064 PM2A beneficiaries over the life of the program, its food ration enrolment will cease for new cases in November 2013, allowing the last enrolled beneficiaries to exit when their child reaches 24 months, as was the case for all previous enrollees. Complementing this, Men as Partners are men from the community who volunteer to undergo education and capacity building in health messaging for PLW and children under two years old. While not a requirement, many participants have partners that are PM2A beneficiaries. 4.2.2 Service Delivery Strategies and Approaches SNAP has developed a set of strategies and approaches for the success of IR 1.2. An assessment of the quality, successes, and challenges of this IR are presented below. The formation of mother CGs and the training of LMs has been particularly successful, although there have been some challenges with regards organization. Quality: Mother CGs have received significant high-quality training following the health education modules. SNAP synchronizes training so that each group covers the same material; one complete rotation of the modules has been completed, and another is already underway. Meanwhile, LMs are trained by HPs and the district supervisors that monitor their training, and lead lessons with neighborhood mothers during supportive supervision. Through FGDs, Mother CGs reported that they appreciate the information provided during the training and that they have changed their behaviors accordingly, especially in breastfeeding, hand washing, and growth monitoring. Successes: SNAP beneficiaries report changes in traditional beliefs around appropriate food for children, breastfeeding behavior, and seeking health care services. Some reported changes are documented through evidence collected by SNAP’s own M&E activities. Generally, the districts are on target to meet the expected 50,000 beneficiaries, and some have already started to wind down their services. Most districts will end enrolment of new beneficiaries in November 2013, although through discussions with the various teams the exact date is not yet clear6 . Challenges: SNAP’s implementation of the PM2A program is being challenged due to the high staff turnover of both GoSL and SNAP. There are also many problems during food distribution, including eligible beneficiaries being removed from lists by SNAP SO2 staff because of monitoring errors and paperwork mix-ups. This provides challenges for SNAP field staff, HPs, and LMs. During KIIs and FGDs conducted by the MTE team, LMs and staff (male and female) reported verbal abuse and the refusal of beneficiaries to participate, resulting in a decreased personal motivation to continue. 5 SNAP_FY13_Q2 Progress Report_ Final_5-3-2013 6 The MTE was made aware in post mission discussions that SNAP may continue to enroll beneficiary’s post November 2013 with the intention of phasing the responsibilities to the MoHS however this appears not to have been captured in program documents shared with the mission. SNAP MTE – Final Report Moneval Solutions Ltd Confidential Page 18 Discussions with district supervisors indicated there is no common understanding of who to report such issues to within the senior management team. SNAP purports to invite others to community-based cooking demonstrations, but this is limited to households enrolled in the program.7 However, the decision to utilize this strategy is determined by individual district supervisors, or IMC HPs, as indicated during discussions with field staff, rather then there being a uniform programmatic strategy for this. An additional safety net could include inviting those identified for education through CMAM, especially those with Children Under Two (CU2). 4.2.2.1 Linkages with Government As with IR 1.1, SNAP’s relationship with the MoHS under IR 1.2 is effective in the areas of post-natal services, wider social services, and health data collection. This has helped reinforce within MoHS the need to work with households as a unit and formalize health care. SNAP works closely with the MoHS line ministries, with MoHS reviewing and approving modules and participating in training and monitoring. The MoHS also supports the implementation by providing growth-monitoring tools for annual assessments, while PHU staff verify pregnancies and play a key role in the identification and promotion of the program for newly pregnant women. 4.2.2.2 Training and Activities As for IR 1.1, training under IR 1.2 is producing positive results and a good uptake in the target communities. Mothers in particular appear enthusiastic in adopting better nutrition practices, eager to ensure that their very young children receive the best food, while nutrition and cooking education is well received. There is naturally a keen interest in benefitting from nutritious foods, especially amongst the most vulnerable families in the communities. Train households on key family nutrition behaviors – Lead Mothers convey nutrition messages and education to neighborhood mothers through bi-monthly health education sessions that include nutrition education and advice on improving hygienic food preparation. SNAP encourages LMs to conduct cooking demonstrations with neighborhood groups and to invite their husbands and community members to attend. Expand caregiver cooking training through lessons on preparing nutritionally rich and diverse foods – SNAP trains households on key family nutrition messages through monthly cooking demonstrations with LMs using locally available foods. In addition, SNAP has developed a recipe book that includes traditional recipes modified to make them more nutritious. The cooking demonstrations and nutritional education have a whole-family approach rather than a strict focus on the nutritional requirements for children under two. The lessons provided by SNAP emphasize food diversity, hygienic preparation, storage, and sanitation. Distribution of nutritious foods to vulnerable groups and household members – SNAP provides “conditional” monthly food rations for women during their pregnancy, (confirmed through 7 Cooking demonstrations occur at agreed venues decided by LMs and beneficiaries within the community. The limitation to including other non-PM2A beneficiaries is that SNAP is not providing the food – the beneficiaries bring donations supported by minimal input from SNAP (at a value of Le 2,000 per beneficiary) to provide complementary items to support food diversification. SNAP MTE – Final Report Moneval Solutions Ltd Confidential Page 19 PHU testing and verification), and up until the child reaches 24 months. During the lean season, SNAP provides an additional ration for the household. The household ration is not determined by the number of household members, but is something set for the program as a whole. Food rations are conditional and require attendance at CG meetings and regular participation in food distribution. For example, if a beneficiary or her designee does not collect their allotted food ration for two consecutive months, the woman, child, and household will be ineligible for further rations. Table 2 below provides details on what constitutes a food ration for each category of person or family group. Table 2: Food Ration by Beneficiary and Content Train households in vegetable growing for increased diversity of foods – SNAP educates PM2A beneficiaries in the importance of growing foods necessary for a diverse diet and promotes household gardens. This supports the utilization of diverse foods and increases the availability of foods for household use. Form mother CGs – As of the second quarter of FY13, SNAP had formed 192 mother CGs8 across the four districts based on geographical considerations according to CG methodology. This limits the distances LM have to travel to visit neighborhood mothers. However, LMs may still have to travel long distances for training. Train mother CGs – Mother CGs have been trained in the four existing SNAP modules. Each mother CG has completed the full SNAP health education program and has started to deliver training using the four modules. Provide educational messages on EHA to mother CGs – SNAP defines EHA in the following ways, which is covered in training modules two and three: 8 SNAP_FY13_Q2 Progress Report_ Final_5-3-2013 Beneficiary Type Ration Size Pregnant mother or lactating mother (child aged 0–6 months) Corn soy blend 5 kg/month Vegetable oil 1 L/month Lentils 1 kg/month Child (6–24 months) Corn soy blend 2.25 kg/month Vegetable oil 0.5 L/month Family ration (per family per month) Vegetable oil 4 L/month Bulgur 10 kg/month Lentils 2 kg/month SNAP MTE – Final Report Moneval Solutions Ltd Confidential Page 20  Diarrhea: home treatment and danger signs;  Hand washing: how and when;  Latrine guidance;  Safe disposal of children’s feces;  Safe water usage, how to make a tippy tap, purification, and storage;  Safe household food storage;  Dishwashing and cleaning;  Feeding guidelines for sick children;  De-worming children every six months; and  Danger signs and when to take a sick child to the clinic. During the cholera outbreak of 2012, SNAP had the flexibility and foresight to reorganize itself to focus on hygiene and cholera prevention, and provided cholera medicines as gifts in-kind. This substantially reduced the spread and lessoned the impact of the outbreak, at least in SNAP implementation areas. Provide basic supplies via mother CGs to improve household-level hygiene and sanitation – SNAP has provided tippy tap materials, jerry cans, nylon rope, and soap to LMs, HPs, and PHUs. In the time remaining, SNAP plans to provide additional beneficiaries with tippy tap materials. 4.2.3 Implementation Progress and Achievements of Results Table 3 below shows the progress and achievements of IR 1.2; of the eight indicators, four have been achieved, with one nearly doubling the anticipated target level. There are two indicators for which advisory comments have been noted relating to food hygiene and personal health demonstrations for caregivers. Two indicators, relating to the number of PLW completing 12 months of PM2A programming and the number of children over 23 months who have completed the full PM2A program, can be achieved, but further attention is required to ensure this. Table 3: Progress and Achievements of IR 1.2 with MTE Team Comments Indicator Type Indicator Current Achievement Assessment9 OC 9 % of children 0–5 months of age 169% Achieved 9 IPTT Attachment A FY12. Photo credit: Daphyne Williams, 2013 Above: A Tippy Tap Display A Tippy Tap system outside of a SNAP donated latrine system, Sierra Leone SNAP MTE – Final Report Moneval Solutions Ltd Confidential Page 21 exclusively breastfed OC 10 % of caregivers demonstrating proper food hygiene behaviors 81% Needs improvement according to baseline; however, anecdotal MTE evidence suggests the baseline may be inflated OC 11 % of caregivers demonstrating proper personal hygiene behaviors 73% Needs improvement according to baseline; however, anecdotal MTE evidence suggests the baseline may be inflated OP 12 # of MCGsactive 140% Achieved OP 13 # of mother CGs trained 137% Achieved OP 14 # of PLW completing 12 months of PM2A programming 47% Achievable OP 15 # of children over 23 months who have completed the full PM2A program - Achievable OP 16 # of children reached by USG￾supported nutrition programs (F) 106% Achieved The PM2A beneficiaries are more knowledgeable about appropriate foods, and the MTE indicates they are practicing these behaviors. They are also knowledgeable about the importance of diverse diets and report improved hygiene during and before food preparation, a behavior likely to be sustained after the end of SNAP activities. Cooking demonstrations are being held by LMs, with most neighborhood mothers encountered during the MTE reporting that they had participated. There does appear to be some confusion over what (and how) to prepare for children under two if there is no CSB, for a variety of reasons. Some are advised to buy Benemix, whilst others are told to make “country” Benemix or a more nutritious version of pap (weaning food for babies). During the MTE discussions it was uniformly agreed that there were not more nutritious foods available than before SNAP, but because of the education provided the people now know the importance of these foods and use them differently. Beneficiaries clearly attribute the changes in their children to improved nutrition, which increases the likelihood that the use of the promoted foods will continue after the end of SNAP. 4.2.4 Other Achievements SNAP MTE – Final Report Moneval Solutions Ltd Confidential Page 22 Other notable achievements identified by the MTE team include:  More informed community leadership;  The MoHS indicates it is using the SNAP recipe book and promotion of locally available foods in other areas;  Recognizing the importance of the SNAP strategy of integrating nutrition and agriculture, the MoHS has requested training in this;  The MoHS has established Men as Partners groups in other areas to encourage their involvement in, and support of, maternal and child health;  SNAP nutrition education is clearly changing traditionally held food taboos. In communities where children were not given eggs, mangoes, meat, and other nutritious foods, SNAP education is convincing women to change their behavior and equipping them with strategies to help educate those who still hold to traditional beliefs;  As children are healthier and bigger, PM2A beneficiaries and Men as Partners members report more men are encouraging their wives to attend the clinic, proudly display their children, and follow other health-promoting behaviors; and  Other NGOs are recognizing the efforts of SNAP and are requesting assistance in implementing similar strategies, including the promotion of tippy taps 4.2.5 Discussion of General Evaluation Questions The evaluation questions for IR 1.2 followed the same approach as for IR 1.1 and are detailed in the SOW. The target groups were the same as IR 1.1 but with a focus on post-natal activities that supported health, particularly nutrition, leading to changes in behaviors and attitudes. Evaluation questions were designed to examine the degree of change in parents’ feeding practices. Please refer to Annex 4, ‘Evaluation Methods and Tools,’ for further details of the evaluation. Photo credit: Daphyne Williams, 2013 Above: Men as Partners Men as Partners Focus Group Discussion with the MTE mission, in Sierra Leone. SNAP MTE – Final Report Moneval Solutions Ltd Confidential Page 23 5 Strategic Objective 2 The overall goal of SO2 is to “Enhance Livelihood Opportunities” through the following four IRs:  IR 2.1: Increased agricultural production and crop diversification through improved inputs and agricultural practices  IR 2.2: Enhanced integration into agricultural value chains  IR 2.3: Increased human capacity and community assets  IR 2.4: Increased access to financial resources As per the SNAP proposal, the first entry point into the communities for SO2 activities is through the PM2A component. As food availability is the first pillar of the food security concept, the rationale is that improved production on family farms will sustain the nutritional gains obtained using the PM2A strategy. SNAP has planned to reach a total of 45,375 direct participants. The core activities of this strategic objective include 1) good agricultural practices (farming, agribusiness), 2) vocational education (tailoring, carpentry, soap-making, hairdressing, tie-dyeing, vehicle maintenance, literacy), and 3) VSLAs and assets construction. In its original SNAP design, ACDI/VOCA planned to apply its value chain approach to improve the production and sale of staple foods and cash crops. The objective was to enhance the performance and competitiveness of smallholder farming households and progress them beyond subsistence farming. At its inception, SNAP’s strategy was to review existing value chain analyses (rice, cassava, palm oil, and sesame) from both a gender and food-security perspective. This was to ensure women participated in, and benefitted from, the program, and that their time, labor, and other resource constraints and opportunities were taken into consideration. The strategy also included a value chain analyses for sweet potato and pigeon pea. The MTE team was able to meet with all different categories of SO2 beneficiaries and stakeholders. This included tailoring, hairdressing, carpentry, soap-making, tie-dyeing, FFS and VSLA farmers, and mechanics. The results for SO2 as a whole are generally mixed, with some successes in training but poorer results in terms of developing greater access to finance through, for example, the VSLA approach. Agricultural practices amongst target beneficiaries appear to have changed, with good adoption of new techniques. Training has also been delivered to almost 700 people in a range of vocational and subject areas. In all cases, gender balance has been a key message area. There remain concerns that value chain integration could be behind target somewhat, particularly as some activities, such as Farming as a Business (FaaB) and youth vocational training, have not been implemented. The FFSs have been a successful vehicle for SO2 and across its IR areas. The achievements and findings for each IR are detailed in the sections below. SNAP MTE – Final Report Moneval Solutions Ltd Confidential Page 24 5.1 Intermediate Result 2.1: Increased Agricultural Production and Crop Diversification Through Improved Inputs and Agricultural Practices 5.1.1 Description of Interventions The target beneficiaries for livelihood sustainability activities are vulnerable individuals, both male and female, from PM2A communities, with 9,526 farmers enrolled in 321 FFSs across the 18 chiefdoms. Adult literacy, numeracy, and vocational skills training are also components of this strategic objective, and 70% of the beneficiaries of these activities are female. Participants targeted for vocational training were also targeted for numeracy training. To improve the quality of its interventions and reach its target goal, SNAP has had to recruit additional program staff in both its field and Freetown offices. Additionally, SNAP has completed a comprehensive review of the FFS training curriculum by integrating nutrition, health, and gender into FFS training modules, and through including modules on business development and marketing. The latter modules are integrated as part of its vocational skills training program. SNAP is laying the foundations for clustering FFSs, strengthening relationships, and sharing learning between FFSs through exchange visits and linking SNAP clusters with existing marketing groups. SNAP is also mapping the existing VSLA communities, with the next step to organize and train selected VSLA members. 5.1.2 Service Delivery Strategies and Approaches As with SO1, ACDI/VOCA has developed a set of strategies and approaches to implement SO2. The following presents an assessment of the quality, successes, and challenges relating to IR 2.1. Generally, appropriate farm training have been developed, although improvements are required to line this up more effectively with the government’s approach. Quality: The activities promoted by SNAP to improve household agricultural production are relevant to the needs of beneficiary farmers and have resulted in increased production and productivity. Land measuring and agroecosystem analysis is one of the themes of the FFS curriculum, with the dissemination of new farming technologies tailored to the nature of the soil of his/her farm to help achieve better results. Successes: Crop diversification, including staple food crops, cash crops, and vegetables, is the main goal of SNAP. The FFS training curriculum focuses on the most common group of crops found in the target areas, and through this training farmers come to understand and adopt improved sustainable agricultural practices. This includes how to optimize Photo credit:: Ange Tingbo, 2013 Above: An example of displaying technical information Clear displays of technical information help farmers to replicate practices and inputs that do well. Sierra Leone SNAP MTE – Final Report Moneval Solutions Ltd Confidential Page 25 the use of improved seed variety, optimize spacing, improve weeding and water management, and enhance plant nutrition. The FFS participatory training approach provides farmers with the opportunity to actively observe and absorb the effects of new practices, and uses the “learning by doing” principle to allow farmers to discover solutions themselves. Their yield has increased, and as suggested by one farmer, “when the old farming way” yields one bag of cassava or rice, the “newly adopted technologies yield three bags” for the same acreage. The FFS curriculum also promotes gender equity, which has had tangible social benefits in the communities. Challenges: Although MAFFS agents agree with the FFS approach, they suggest that the curriculum used by SNAP to train its facilitators is not complete as per the ministry’s protocol (see 5.1.2.1 below). Additionally, they indicate that the five-day duration of the training offered is not sufficient to cover all themes in the required detail. Organizing joint visits by SNAP field agents and the MAFFS’s extension agents is problematic due to logistical issues such as scheduling access to vehicles, and approval for government staff to travel. 5.1.2.1 Training and Activities The FFS training is conducted by a FFS facilitator who has gone through a five-day training session delivered by the SNAP district trainer. The MTE team found the training material designed by SNAP required some improvement. The document is in a plain text format without visual aids. Pictures of the themes being taught as good practices (spacing, weeding, fencing, and so on), and of comparisons with the results of traditional ways of farming, would be a useful means of illustrating the benefits of the training, thus increasing its appeal to the farmers. 5.1.3 Implementation Progress and Achievements of Results To date, SNAP has reached 20,439 of the 45,375 vulnerable individuals targeted for livelihood sustainability activities in PM2A communities10(). It has enrolled 9,526 farmers in 321 FFSs across the 18 target chiefdoms, and has increased female enrolment in FFSs by 1,155, to reach a total of 5,059 (a 30% increase from enrolment in FY11 FFSs). Meanwhile, 1,024 individuals received adult literacy, numeracy, and vocational skills training in FY13, of which 714 beneficiaries, or 70%, were female. The total enrolment of FFS farmers in SNAP during the time covered by the MTE was 9,526 beneficiaries, of which 5,059 are women. 5.1.4 Other Achievements Other achievements identified by the MTE team include:  Farmers are better informed on improved farming technologies;  By being exposed to the same teaching at FFSs and PM2A sessions, husbands and wives indicate that their life is better at home. On the farms, men are no longer imposing their own ways of doing business; 10 Source: PREP FY13 SNAP MTE – Final Report Moneval Solutions Ltd Confidential Page 26  The gender equity component has brought a new dynamic in daily family life. All SNAP beneficiaries suggest there is more “happiness at home,” as men are less domineering;  Men are more interested in the nutritional status of their children, especially those under two. Even during community meetings, the MTE team observed many fathers attended with their children; and  The interaction between communities, local authorities, and the SNAP team is extremely positive and is a conducive strategy to promote sustained change. 5.1.5 Discussion of General Evaluation Questions The evaluation questions for IR 2.1 were detailed in the SOW, although these were elaborated on during the preparation phase. The target groups were the FFSs, government, and other agricultural support agents. Evaluation questions were designed to explore the general uptake of agricultural practices, results of improved farming techniques, and approaches. Target groups were interviewed through FDGs and KIIs, and farm site visits were also carried out. Please refer to Annex 4, ‘Evaluation Methods and Tools,’ for further details of the evaluation questions for SO2. 5.2 Intermediate Result 2.2: Enhanced Integration into Agricultural Value Chains 5.2.1 Brief Description of Interventions Agricultural value chains in Sierra Leone are plagued by many constraints, of which the main ones are poor storage facilities and lack of access to processing equipment. The consequences are increased food loss, limited volumes for trade, and no value addition that could subsequently lead to increased income. Moreover, farmers have limited information about market prices in Freetown or district capital cities, and instead have to rely on conservative prices or the “tricks” of middle men. Smallholder farmers have no or little interaction with the larger traders, and consequently become price-takers. Budget constraints and the high turnover of SNAP staff at the country office and district levels have proved the main obstacles to implementing the activities of IR 2.2. Activities under FaaFB, youth agribusiness and entrepreneurship training, storage, and processing are yet to be implemented by SNAP. Working in FFS groups has already created a sense of cohesion as membership is decided through self-selection. Weekly meetings on FFS farms and activities based on the existing VSLAs have built a new sense of togetherness and shared values. SNAP has developed a plan, as outlined in the 2013 PREP, to address this IR by providing machines available as in-kind grants to FFS groups that are organized and have developed viable business plans. These micro￾enterprises will be required to cover a percentage of the operating costs up front, which will gradually become 100% as their businesses grow. Training in storage and warehousing will teach the basic principles of Title II food commodities management, e.g. ledgers, stacking, spacing, inventories, First In First Out (FIFO), and so on, as ACDI/VOCA’s food team possesses the necessary skills to deliver this. SNAP MTE – Final Report Moneval Solutions Ltd Confidential Page 27 The ACDI/VOCA value chain approach is likely to help achieve the objectives of IR 2.2 in the remaining years of the program. The value chain strengthening activities incorporate the following concepts:  Using a market system perspective;  Focusing on end markets;  Understanding the role of value chain governance;  Recognizing the importance of relationships within the family;  Facilitating changes in firm behavior;  Transforming family relationships and those within the community;  Targeting leverage points; and  SNAP’s ultimate goal is to take clustered FFS groups to the farmer business organization (FBO) level The value chain approach is a strategy to move the smallholder farmer into an emerging “farmer status.” As production and productivity increase with the adoption in FFSs of new methodologies, famers will either equip the groups with light processing equipment or link them with other structured entities. SNAP can use the steps below as a guide to reinforce the value chain approach:  Sensitize and train groups in agriculture as business (production, productivity, and profit);  Train groups in bulk warehousing, governance, business, and financial management; and  Identify some food processors in the districts or in Freetown, or provide light equipment to facilitate sustainable links between producers and processors; facilitate a stronger relationship with food exporters or large retailers in the main cities. A breakdown of current vocational training is shown in Table 4 below. This illustrates the large number of areas from which the target beneficiaries could select or be supported in. In addition, the table shows that nearly 700 people have been trained so far, with Kailahun accounting for 38% of the total. Those in Tonkolili represent only 13% of all trained beneficiaries. Table 4: Breakdown of Beneficiaries Trained in Vocational Skills (FY12 Annual Results Report) District Tailoring Masonry Hairdressing Gara tie dyeing Soap-making Carpentry Auto mechanics Sub Total M F M F M F M F M F M F M F M F SNAP MTE – Final Report Moneval Solutions Ltd Confidential Page 28 Bombali 20 10 0 0 0 0 2 48 18 82 37 0 0 0 77 140 Tonkolili 28 65 0 0 0 25 0 0 0 0 0 0 56 2 84 92 Koinadugu 4 26 0 0 0 0 0 83 6 81 30 0 0 0 40 190 Kailahun 6 40 49 0 1 49 0 0 20 180 0 0 0 0 76 269 Total 58 141 49 0 1 74 2 131 44 343 67 0 56 2 277 691 5.2.2 Service Delivery Strategies and Approaches Although a comprehensive set of strategies and approaches for the success of IR 2.2 have been devised, it is difficult to assess the related quality, successes, and challenges given the lack of activity. Quality: The quality of the strategy or approach is difficult to comment on as this IR has not been implemented. Successes: Not applicable as of yet; however, SNAP retains this in its PREP and plans to address it. Challenges: Budget constraints have been a challenge as monetization proceeds have not met expectations. The provision of light processing equipment in the months to come will also be challenging. The equipment will be made available as in-kind grants to organized FFS groups that have developed viable business plans. It will also involve substantial effort in delivering further training. The following two areas under this IR are commented upon below: Storage – SNAP emphasizes improved storage at the household, community, and regional market levels. Improved storage facilities and systems at various levels will contribute to food security during months of shortages, and allow farmers to sell when prices are higher. However, in many communities visited during the MTE there were empty storage buildings owned by the MAFFS, and the MTE team found most farmers stored their produce at home due to a lack of trust. There is insufficient knowledge of how best to structure and govern group storage arrangements in a way that would not jeopardize an individual household’s stocks. Both the FFSs and FaaFB will address issues related to establishing community-based storage systems. Processing – The SNAP proposal suggested that farmers carry out some light on-farm processing, but IR 2.2 is focused more on industrial-level processing that requires machinery. Rice and cassava will be the principal crops targeted for processing, although there will be some vegetable drying. 5.2.2.1 Links with Government There are no links with any government entity as of yet. However, from the discussions the MTE team held with the MAFFS authorities, the strategy is in line with the GoSL’s development vision. Clustering the FFS groups into FBOs remains a work in progress. SNAP MTE – Final Report Moneval Solutions Ltd Confidential Page 29 5.2.2.2 Training and Activities ACDI/VOCA’s plan is for the SNAP team to provide training in basic business management practices and handle inputs to improve processing and marketing. They will also provide market-based studies and give special attention to women and youth farmers. The studies will focus on the highly specific nutritious crops and vegetables promoted by the program. This has not happened during the period covered by this MTE. 5.2.3 Implementation Progress and Achievements of Results The implementation of IR 2.2 has one important prerequisite – clustering the FFSs into FBOs – and 102 clusters have been organized involving the FFSs implemented in 2011. However, for these clusters to become formal FBOs they must satisfy various criteria of the DC and MAFFS, for example providing FFS graduation certificates and having a mandated constitution. The staff are working on sensitizing the farmer’s groups to the benefits of working together as a commercial group. From the evaluator’s discussions with the FFS beneficiaries, it is clear they understand the general principles of clustering FFSs and are waiting for SNAP staff to provide the practical details and implementation. SNAP’s agronomists and field staff are confident that this will begin in the coming months. 5.2.4 Other Achievements Although nothing concrete has happened with regards this component yet, ACDI/VOCA anticipates that other activities will contribute to its success. The vision in the PREP is to organize exchange visits between SNAP clusters with successful producer groups, marketing associations, and the GoSL smallholder commercialization programs. SNAP will also develop links between its clusters and existing marketing groups that are successfully contracting and looking to increase their raw material acquisition to meet their contractual commitments. 5.2.5 Discussion of General Evaluation Questions The evaluation questions for IR 2.2 were detailed in the SOW and elaborated on during the preparation phase. The target groups were the FFSs, traders, and youths, with questions designed to understand whether farmers are now becoming more connected to processors and thereby the markets they hope to serve. In addition, the questions were designed to elicit whether vocational training is having an impact on the livelihood chances of previously untrained or unskilled labor. Target groups were interviewed through FDGs and KIIs, with visits to a number of enterprises undertaken. Please refer to Annex 4, ‘Evaluation Methods and Tools,’ for further details of the evaluation questions for SO2. 5.3 Intermediate Result 2.3: Increased Human Capacity and Community Assets 5.3.1 Description of Interventions There are two components embedded in IR 2.3: 1) strengthening human capacity and 2) building community assets. SNAP MTE – Final Report Moneval Solutions Ltd Confidential Page 30 The programmatic activities to strengthen human capacity and diversify livelihood opportunities range from training on literacy and numeracy to vocational skills. This involved training in tailoring, masonry, hairdressing, gara tie-dyeing, soap-making, carpentry, and auto mechanics. The FY12 ARR suggested this component had reached a total of 968 beneficiaries so far. 5.3.2 Service Delivery Strategies and Approaches A comprehensive set of strategies and approaches for the success of IR 2.3 have been devised, and the quality, successes, and challenges assessed. Generally, the quality of training provided is good, although not all elements of capacity building under this IR have been implemented yet, making it difficult to review the performance. Quality: The main activities involve tailoring, soap-making, carpentry, hairdressing, and gara tie-dyeing. To ensure quality training, SNAP hires master trainers in each trade to deliver the teaching. Beneficiaries are encouraged to cluster in a business association after their graduation, and are monitored by SNAP field agents. The other sub-IR of this component, namely activities in FaaFB, youth agribusiness and entrepreneurship training, storage, and processing, have yet to be implemented by SNAP and cannot be qualified at this time. However, SNAP has a plan to address the issue in its PREP. Successes: SNAP provides literacy, numeracy, and vocational training to youths and older persons to prepare them to earn a living, take care of themselves, and benefit their communities at large. Business development and marketing modules are integrated in vocational skills training, giving trainees new tools to utilize in their endeavor. Challenges: The first challenge is the three-month duration of the tailoring training. It was assessed by the MTE team that three months are insufficient to learn all the intricacies of tailoring, e.g. measurement, evaluation of fabric texture, ironing, buttonhole making, operating the sewing machine using the feet, and so on. SNAP is planning to extend the training to five months in agreement with the teacher seamstress and tailor. 5.3.2.1 Linkages with Government There were no links with the government during the period covered by the MTE, and the MTE team did not meet with any authority from the Ministry of Youth. However, the DC and MAFFS authorities encountered by the MTE team approved this component of SNAP as it targeted youths in the program districts. Photo credit:: Ange Tingbo, 2013 Above: Vocational Training Success A visit to a vocational training success enterprise – tailoring, Sierra Leone Photo credit:: Ange Tingbo, 2013 Above: Vocational Training Success School uniforms sewn by SNAP Beneficiaries, Sierra Leone SNAP MTE – Final Report Moneval Solutions Ltd Confidential Page 31 5.3.2.2 Training and Activities The training activities in literacy/numeracy, tailoring, masonry, hairdressing, gara tie-dyeing, soap￾making, carpentry, and auto mechanics are held in the beneficiaries’ home regions. In addition, the teachers selected to take charge of the training come to stay in the community throughout its duration, which creates a sense of community ownership. The training duration varies according to the trade; three months for auto mechanics, carpentry, masonry, hairdressing, and tailoring, two months for gara tie-dyeing, and a month for soap-making. In the PREP, ACDI/VOCA suggests increasing the three-month period to five months in the coming years. 5.3.3 Implementation Progress and Achievements of Results In the breakdown of beneficiaries of vocational training, the FY12 ARR indicates a total of 968 took part, of which 691 were women. The MTE team met with the beneficiaries of the various trades, who explained the vocational school has transformed their lives; they can now earn a living and have a role in their community. The interview with Mrs. Gandi, a SNAP auto mechanic trainee, is an example of such a success story (see Annex 8 for details). It was reported that during the period covered by the MTE, SNAP staff continued assessing the needs of the communities through the Village Development Committees (VDCs). The community assets range from grain stores, drying floors, water wells, and crop processing equipment. Financial constraints resulting from the low cost recovery of the monetization proceeds compelled ACDI/VOCA to realign its program activities. The activities of IR 2.3 were most affected by the cash shortage, including the construction of drying floors, grain storages, farm-to-market road rehabilitation, water wells, cassava processing equipment, and rice mills. As per the FY12 ARR, SNAP field staff continued to work with the VDCs to identify and prioritize community asset development needs. Across SNAP districts, 210 community assets have been identified for rehabilitation or construction. The MTE team did not observe any ongoing activities related to this IR during the field visit. However, in its FY13 PREP, SNAP plans to engage communities in a participatory approach to identify and prioritize community asset initiatives that would best suit the needs of the 110 cluster groups. Rehabilitation of agricultural production assets, including but not limited to grain stores, drying pads, market structures, and minor maintenance of farm-to-market roads would be prioritized. SNAP has an excellent funding approach for the achievement of IR 2.3, this being that communities would be given grants after pledging that they contribute their labor for unskilled tasks. 5.3.4 Other Achievements A number of additional achievements were identified during the MTE these were: Impact on Youth - The MTE team met with two young former tailoring trainees who are now working with two co-graduates in their ex-trainer’s shop. They were previously dropouts, but are now taking evening classes and paying for their own school tuition. Another carpentry trainee is proud of the house he built himself in his village. SNAP MTE – Final Report Moneval Solutions Ltd Confidential Page 32 A further anecdotal story involves a carpentry trainee who is now working in a group of ex￾graduates. He stated that he previously used to be brought before the village chief regularly due to unpaid debts. However, now that “he can make some money from time to time, the village chief doesn’t see him anymore”. Agricultural Productive Assets – The plan was for SNAP to engage communities to identify and prioritize community asset initiatives to link cluster groups to value chains in rice, cassava, pigeon pea, ground nuts, and maize. Priority would be given to the rehabilitation of assets to aid agricultural production, such as grain stores, drying floors, market structures, and farm-to-market roads. 5.3.5 Discussion of General Evaluation Questions The evaluation questions for IR 2.3 were detailed in the SOW and elaborated on during the preparation phase. The target groups were the non-farmers and youths, with questions designed to understand how far people’s capacity to offer their services in the market has been achieved, and whether enterprise creation was taking place. Target groups were interviewed through FDGs and KIIs, community sites and enterprises were also visited. Please refer to Annex 4, ‘Evaluation Methods and Tools,’ for further details of the evaluation questions for SO2. 5.4 Intermediate Result 2.4: Increased Access to Financial Resources Given that, SNAP’s VSLA coordinator was only recruited in January 2013, his task for the past seven months has been restricted to mapping out and conducting a census of the communities interested in savings and loans activities. As this component is at an early stage of full implementation, it was decided to drop VSLAs from the SOW of the MTE. 5.4.1 Description of Interventions The main activity of IR 2.4 was the creation of VSLAs with the intention to strengthen access to finance by community members of the association. 5.4.2 Service Delivery Strategies and Approaches Intermediate Result 2.4 was dropped from the MTE SOW as SNAP is currently assessing VSLAs that could potentially be willing and strong groups to support. No services, strategies, or approaches implemented by SNAP were available to the MTE team, other than the exercise of mapping the VSLAs. However, SNAP has a helpful training plan for the viable groups in the months to come. As suggested in the PREP, 73 existing VSLA groups will be activated in FY14, and 12 new VSLA groups will be established. The MTE team reviewed the existence and mechanisms of several VSLAs. These appear to function well without any external capital or seed money from SNAP or other institutions. The quality, successes, and challenges experienced under IR 2.4 are presented below. Quality: The VSLA was dropped from the MTE SOW. At the time of the field visit, SNAP staff were conducting a census of potential VSLA groups to strengthen in the remaining years of the program. SNAP MTE – Final Report Moneval Solutions Ltd Confidential Page 33 Successes: Considering the farmers’ high level of interest in VSLA activities they are implementing themselves without external seed capital, the MTE team foresees that SNAP will play a positive role in strengthening the managerial capacities of the existing and future VSLA groups. Challenges: Culturally, people do not want to disclose how much money they have. All attempts by the MTE team to assess how much money beneficiaries have in their coffers or at a bank were unsuccessful. Another challenge facing SNAP is taking successful VSLAs to another level by linking them to financial institutions. 5.4.2.1 Linkages with Government There was no link to any government structure during the period covered by the MTE. SNAP is currently mapping FFS activities and the communities with a VSLA in place prior to providing strengthening support. As the VSLA principles are widely known in the country, there is a tacit recognition by the local authorities that they have a valuable contribution to make. 5.4.2.2 Training and Activities SNAP is carrying out a census of the VSLAs, after which period there will be a training plan developed to strengthen the capacity of selected groups. Training will be in drafting MoUs, record keeping, drafting investment plans, and most importantly, the provision and management of a “metal box” or safe. 5.4.3 Implementation Progress and Achievements of Results Intermediate Result 2.4 involves the creation of savings clubs (with between 15 and 30 members) with weekly meetings, regular cash contributions, and locally determined internal control mechanisms that dictate the savings and lending interest rates. Groups together on membership criteria and lending limits using mutually determined rules and regulations based on the principles of transparency, responsibility, and participation (see Box 1 on feelings towards money). They have their own MoU with compelling clauses, one of which is the payment of a fine when a member cannot justify their absence from two meetings. In two different communities visited by the MTE team, fines had been utilized due to unjustified absences from scheduled meetings. This is an indication that the MoU clauses set by the members themselves are respected and reinforced. Arriving late to meetings also triggers fines. The use of these fines has a deep social benefit, as they are used to cook a meal shared by all group members. This is a good sign of solidarity and group cohesiveness that should be retained as an example of good practice. Box 1: NO MORE MONEY WORRIES: In the community of Kafogo (Koinadugu), a young village chief confirmed that with the VSLA in place, there are no more people brought before him for uncollected loans. This presents a huge opportunity for SNAP to get the VSLAs organized and take them to another level. Interestingly, all the communities have been asking for “the boxes” where they can keep money. SNAP has these boxes; they are metal and superior to the wooden boxes or cloth bags currently used by communities. SNAP MTE – Final Report Moneval Solutions Ltd Confidential Page 34 5.4.4 Other Achievements The informal banking system is made up of two components: 1) the business unit, where members can take out loans to hire laborers to meet, for example, farming needs, to pay school tuition fees for their children, acquire personal assets, and so on (this attracts an interest charge) and 2) that which can be issued to pay for medical treatment or bereavement in the family (with no interest on loans charged in this case). The impact of the village-owned VSLAs can be viewed as outstanding. The MTE team heard stories from some farmers who would use their crops as collateral even before the harvest. One villager said that before his membership with the VSLA he was summoned before the tribal chief frequently due to unpaid debts. That farmer is now debt free. Many villagers indicated that they no longer have illness or money worries (especially the latter). Please refer to the text box associated with this section for an example of the impact of VSLAs. 5.4.5 Discussion of General Evaluation Questions The evaluation questions for IR 2.4 were detailed in the SOW, with general reference to access to finance and the training delivered. However, the MTE team discovered that little had been developed in this regard and instead focused on existing VSLAs.. Please refer to Annex 4, ‘Evaluation Methods and Tools,’ for further details of the evaluation questions for SO2. SNAP MTE – Final Report Moneval Solutions Ltd Confidential Page 35 6 Program Quality and Cross-cutting Themes 6.1 Partnership/Consortium Quality SNAP is implemented by a consortium of organizations, and whilst it is never easy to manage such an arrangement, the program history and current situation suggests that by and large the relationships are amicable and working well. The division of sectoral responsibilities is clear, and activities are being implemented in line with the original plan. Nevertheless, a number of areas have been identified as potential concerns. In summary, these relate to varying partner employment and contractual details of employed staff, and the common pooling of physical resources, such as vehicles. Greater efficiency could be generated by discussing and agreeing a common approach and position in this respect, which may reduce the discontent currently found amongst implementation team staff. This relates to management and administration rather than any technical issues. The Monday meetings of partners are instrumental in harmonizing the viewpoints and program strategies of the three partners (see below). Above all, ACDI/VOCA carries out quarterly reviews of the sub-recipients to obtain reasonable assurance that program implementation is complying with the clauses of the sub-agreement as indicated in the USAID Cooperative Agreement #AID-FFP-A￾10-00014. As per the sub-agreement, ACDI/VOCA conducts compliance monitoring visits to IMC and OICI, and its CTC carries out audits of the partners. Between July 2 and July 18 2012, the CTC carried out an audit on IMC in Freetown. The audit SOW that covered the main aspects of a well-run program read as follows:  Review of financial documents (July 2010–June 2011);  Review of internal control systems;  Review of progress towards quarterly targets set;  Review of monitoring tools for data quality and accuracy; and  Assess documentation and filing system of key program documents. The MTE team found the audit report to be a beneficial piece of work in its findings and presentation, and a useful tool to help ACDI/VOCA and partners implement program activities according to FFP’s regulations. Eight recommendations were to be addressed in the final report, although the MTE team is unsure if these are being addressed. In addition, ACDI/VOCA designed a “Compliance Monitoring Checklist for SNAP Program” to cover the following areas: 1) financial accountability, 2) internal controls, and 3) previous findings. This is good practice and should be maintained to ensure that the program is run according to the regulations and principles of USAID. Unfortunately, ACDI/VOCA’s CTC has no additional staff and the volume of the task is immense. SNAP MTE – Final Report Moneval Solutions Ltd Confidential Page 36 6.1.1 Program Management, Coordination, and Operations The management of program operations is structured around four distinct levels inseparable in their functionality. These are:  The head office in Washington D.C.;  The country office in Freetown;  The district offices; and  The field staff. The deputy director for community development has direct supervision of the program, thus ensuring activities are carried out as per the approved proposal and liaising with USAID FFP in Washington D.C. The management team in the country office in Freetown is composed of the CoP, the DCoP, the program director, the CTC, the food management officer, the M&E coordinator and the team leaders of the two SOs. This collective team is in charge of the coordination of the operations/activities with IMC and OICI, the two technical partners in the consortium with ACDI/VOCA. The country office provides the financial and logistical support for food distribution. It coordinates the implementation of the activities with the DOMs, who in turn ensure the implementation of program activities in collaboration with a district staff sub-divided between the two SOs. The final level is the field staff that are in direct contact with the communities to carry out and then monitor program activities. To coordinate the operations, ACDI/VOCA, IMC, and OICI meet every Monday in Freetown. Discussions revolve around the field activities of each partner and any communication from ACDI/VOCA HQ. Minutes of meetings are kept by ACDI/VOCA HR and shared by email with partners. Ad-hoc meetings are held when necessary. All DOMs meet with the Freetown management team on a monthly basis (usually the last Thursday of every month) to discuss common and specific issues. Between the management team meetings, DOMs meet with the field agents who come to the district office and seize the opportunity to write their field reports. Their reports are handwritten and contribute to the DOMs’ individual presentations in Freetown. Following question and answer sessions conducted by the MTE team with field agents and colleagues in the office, field agents should now be able to use a computer to compile their reports. This is considered staff capacity building. Moreover, it is easy for a field agent to correct his/her computerized report when the supervisor finds discrepancies therein or if there are statements to clarify. For SO1, district supervisors meet monthly with all team members to discuss issues and work plans. The MTE team found the weekly and monthly meetings in Freetown to be relatively effective forums for sharing information. This typically includes updates on progress and activities, discussing and resolving issues, and providing training for new staff members. However, the MTE team also noted that many important issues that should have been dealt with in the Monday meeting were still pending during the team visit. These included: SNAP MTE – Final Report Moneval Solutions Ltd Confidential Page 37 Inaccurate lists of food beneficiaries – At times this puts the SNAP distribution teams in a difficult situation with the communities as they do not have any guidance from senior management nor a policy response for this event. This has caused some embarrassment to SNAP, and can have a reputational knock-on effect on the implementation consortium:  Non-payment of stipends to FFS facilitators – This has been seen as an issue which not only reduces implementation but can also lead to a loss of morale;  Vehicles and motorcycles grounded due to maintenance procedure difficulties – This will lead to a reduced implementation of activities and the consequence of not meeting targets set in the IPTT; and  Non-functioning databases – This results in information not being available, decisions taken which may not be appropriate, and a consequent loss of ability to demonstrate impact and change. There are a number of recommendations targeted at improving the effectiveness of the meetings to deal with the issues mentioned above. Please refer to Section 9 for further details of the MTE recommendations (recommendations 10, 11, and 16 in particular). Initially, SNAP staff were provided with training and an in-depth awareness of the SNAP implementation, methodology, and capacity-building strategies. However, SNAP has faced an ongoing challenge of high staff turnover at all levels. Of the 26 staff interviewed during the MTE, 54% reported that they had been in their current position less than two years, and only 10 had received training on ENA, hygiene, and/or gender. SNAP has adopted the strategy of providing week-long orientation sessions to new SO1 staff in Freetown. Following this, field based District Supervisors are given on-the-job training by more experienced staff to provide them with sufficient skills to provide supportive supervision to health promoters, LMs, and PHUs, while mitigating the expense incurred in continually training new staff. However, on-the-job training in the subject matter, procedures, and reporting strategies are determined by the staff member leading the training. No district visited by the MTE team had a set timetable for completing the training, a list of what needed to be covered, or an evaluation period to determine when new staff had met all the requirements. Inconsistencies in PM2A beneficiary roles during food distribution have a significant impact on program implementation. The inconsistency stems from problems in the M&E system; once SO1 identifies beneficiaries, SO2 decides who is eligible with very little comment or review from SO1. The system is managed from Freetown, from where lists are sent to SO2 staff, and SO1 do not know who has been left off the list until the day of distribution. Meanwhile, LMs report being ridiculed by the community for working on the program but not receiving benefits, and staff report having received verbal threats from those in the community who are eligible but do not receive rations. Program activities are included in the strategic plans of the MAFFS and the development plans of the DCs; SNAP district offices have monthly meetings with the two entities, along with other international NGOs operating in the district. As proof of this, the MTE team was able to view notes taken during the previous two meetings. Minutes are shared with the participants prior to the next meeting, allowing them to make comments or correct some incorrectly transcribed ideas. The SNAP MTE – Final Report Moneval Solutions Ltd Confidential Page 38 meeting report is ultimately approved by the group when everybody is in agreement. The systematic monthly meeting with MAFFS, FFS, and DC members is a good example of collaboration with the official bodies. In the field, the Bloc Extension Agents (BEA), who are MAFFS and FFS extension agents, mainly work with SNAP field agents and the FFS facilitators. Together, they participate in training sessions organized either by the government or by SNAP. However, joint visits of SNAP field agents and extension agents to monitor the activities of FFS facilitators or villagers happen infrequently. The only area of uncertainty with regards relationships with the GoSL local authorities is the Bombali District Agriculture Officer (DAO). During an interview with the MTE team, he stated that too much bureaucracy is “hampering the smooth implementation of the activities.” The bone of contention is that “SNAP district staff always want to refer to Freetown or Washington D.C. before addressing any requests.” He added that as agricultural activities are seasonal and time-bound, the delay in obtaining authorization from the highest level creates a damaging delay to agricultural activities. He was not specific, but said that the other NGOs operating in his district were more flexible. Motorbikes that are grounded pending repair are impacting implementation. This is due to the procedures for maintenance services that often take time to organize. Some motorbikes are also simply unfit to cover the rugged terrain. Another issue is the delay in the payment of stipends to FFS training facilitators. There was an attempt by Freetown and district offices to make payments by phone, but this proved unsuccessful as the SIM cards of the owners were not recognized by the phone company, having been purchased on the “parallel market.” Some staff members of the partners that share the office with ACDI/VOCA receive different treatment in terms of travel allocations for monitoring visits. This has caused disaffection amongst many workers. The contract with the FFS facilitators has a clause for the provision of a bicycle due to the distance to be covered from one school to another. However, the bicycles have not yet been provided due to SNAP’s financial constraints. The MTE team found that the high turnover of staff at the country office in Freetown and at district level negatively impacted the smooth running of operations and the implementation of activities. There have been five CoPs in three years, and morale could be affected. Many teachers and students have left the country, and the national universities are yet to resume normal academic years. There is little in the way of competent human resources to satisfy the needs of the many international NGOs working in the country, and ultimately they resort to “poaching” staff from each other. To illustrate this, the program director was hired as recently as February 2012. The M&E manager was hired in May 2012 but released in July; the position was advertised in September 2012 but remains vacant. The warehouse managers in Makeni and Kabala were hired in July 2012, and the commodity manager in Freetown was only hired in November 2012. The VSLA coordinator was hired in January 2013, while the SO2 M&E officers in Kailahun and Bombali both resigned this year (2013). SNAP MTE – Final Report Moneval Solutions Ltd Confidential Page 39 Although some positions were filled in 2012, we must note that it takes time for any individual to understand his/her new work environment and get to know colleagues, the “policy,” and “USAID” rules of operation. His/her real performance and contribution might take three to six months to take effect. 6.2 Targeting The targeting in general appears to be well guided, and serves the purpose of identifying those most in need of health or agricultural support. The geographical focus has also been carefully selected, as have the institutions, whether government or local, with which SNAP collaborates. Whilst selection and targeting are effective, actual participation within the SOs is limited due to the resources available, which means not everyone can benefit from the full program of activities. 6.2.1 Strategic Objective 1 IR 1.1: Targeting – Although SNAP targets four geographic districts, it does not target the entire district; implementation areas were selected by identifying the chiefdoms within each district with the highest rates of malnutrition and stunting. As a result, DHMTs from each district participate in the capacity building and training opportunities, but PHU participation is limited because of this to only 143 PHUs within the geographic targeted areas. IR 1.1: Selection Criteria – All DHMTs are targeted for SNAP participation, but selection for participation in activities may be determined by the function of the DHMT member. For example, the district M&E officer may participate in the review of data collection, while the nutrition officer may participate in supervising cooking demonstrations. The current SNAP strategy is for each PHU in the targeted areas to identify one PHU staff member to participate in training which, through the cascade model, filters through to other PHU staff. IR 1.2: Targeting – PM2A beneficiaries are pregnant women, confirmed through PHU testing and verification. Although SNAP is only in the third year of implementation, the MTE team saw results that indicate it is improving health and nutrition practices at the household level and changing traditionally held beliefs about infant and young child feeding practices. For example, LMs and neighborhood mothers reported increased breastfeeding, hand washing, and removal of traditional food taboos. IR 1.2: Selection Criteria – As for IR 1.1 targeting, SNAP aims to support households that could benefit from health and food education, as identified by DHMTs, LMs, and CGs. 6.2.2 Strategic Objective 2 IR 2.1: Targeting – Targeting of SNAP beneficiaries is carried out at three different levels. Despite the entire country being negatively impacted by the civil war, ACDI/VOCA and its partners targeted the most affected districts based on reliable statistics. The first targeting occurred at the district level, with four districts chosen due to their health, poverty, and child nutritional status. The second level was to identify the 18 most vulnerable chiefdoms within these districts, and the third level involved selection of the ultimate beneficiaries. In all, 50,064 mother-child units were targeted for the PM2A health and nutrition activities, with an additional 350,448 family members targeted for feeding in the lean season. An estimated 45,375 targeted to participate in the livelihood activities, including FFS, literacy, numeracy, and vocational training. The different targeting levels followed by SNAP MTE – Final Report Moneval Solutions Ltd Confidential Page 40 ACDI/VOCA illustrate that this was not a random action but based on strategies by which the most vulnerable and neediest chiefdoms and beneficiaries could be reached. IR 2.1: Selection Criteria – The first sensitization messages communicated by SNAP to beneficiary communities appear highly effective. Beneficiaries selected themselves based on mutual trust, work ethic, reputation, and a shared common vision. This is a good practice that works toward the cohesiveness of the groups. More importantly, bringing people to work and live together again after a decade in a war-torn country is a key component of the SNAP proposal. The FFS system has the expected intangible benefit of “building trust and solidarity in communities still recovering from the trauma of war.” The self-selected groups typically have between 15 and 20 members. As reported in the food distribution section of this report (see below), the authority of the Paramount Chief of Karamanka was instrumental in defusing a tense situation that resulted from a problematic list of beneficiaries at a food distribution PHU. IR 2.2: Targeting – The targeting strategies remain the same as indicated in the IR 2.1 context. The beneficiaries of IR 2.2 activities are not new to SNAP interventions, and are therefore part-and￾parcel of the program. These communities are targeted for livelihood interventions because SO2 activities are designed to address the challenges of rural households with regards food access and availability. Agribusiness and youth employment are also the priorities of the government. IR 2.2: Selection Criteria – This will involve clustering FFSs into FBOs. Essentially this means pulling two to three FFSs together to create a critical mass suitable as an agribusiness center. This would be similar to the MAFFS’s Smallholder Commercialization Program (SCP). IR 2.3: Targeting – Vocational skills training targets youths and offers teaching in the different skill areas mentioned above. All trades are open to all youth based on their own volition and preference. There is a pre-sensitization in the community to explain what the vocational school involves, and prior to enrollment the SNAP team provides trainees with counseling. IR 2.3: Selection Criteria – There are essentially three selection criteria for vocational training. Beneficiaries should:  Belong to a district and community involved in SNAP activities;  Be a dropout or somebody willing to take a new direction to increase their potential for employment; and  Be willing to join a group or association at the end of the training to start a business. IR 2.4: Targeting – The targeted beneficiaries include all participants in SNAP activities and additional groups, and individuals not part of the SNAP program. SNAP activities indiscriminately target both males and females in the intervention areas, and internal MoUs regulate the functions of individual groups. IR 2.4: Selection Criteria – Beneficiaries are self-selected through a participatory community mobilization process and include both men and women involved in PM2A and FFS activities. SNAP MTE – Final Report Moneval Solutions Ltd Confidential Page 41 6.3 Gender Gender equity has been an important theme throughout SNAP’s interventions and is seen both in the fieldwork it conducts and its overall management approach. Reference to gender issues and equity is therefore mentioned throughout this report but dealt with in its own right here. Gender Integration Action Plan (GIAP): ACDI/VOCA and its partners held a workshop on August 2–3 2011 that resulted in the establishment of a Gender Integration Action Plan (GIAP). The main goal was to decide how best to engage women and men equitably, allowing them both to participate in, and benefit from, program interventions. The intended results were to: Allow both women and men to participate equally;  Increase participation by women and men, with special emphasis on increasing female involvement;  Increase the overall knowledge that could lead to positive behavior change; and  Improve female ability to participate in development practices. SNAP expended effort in creating this plan designed to promote gender awareness through staff training and workshops and raise awareness in target communities. In terms of implementation however, the GIAP has not been widely effective. The inclusion of gender awareness within training manuals, such as those for FFS and VSLA activities, did not take place, and there has been no significant integration of this approach. One reason for this might simply be a lack of gender-enforced prioritization by SNAP management, despite GIAP being designed to be taken forward by management. The demanding workload of SNAP officers, relatively high staff turnover, and financial constraints will undoubtedly have had a negative impact on realizing the intentions of the plan. As the GIAP is structured into a large number of management and project activities, the realization of these may prove ambitious to implement and contribute to the overall poor progress. In terms of the key outputs and activities of the plan, with regards both internal and program￾focused activities, the following summary can be noted: Gender training workshops, like all staff training, is negatively affected by the level of staff turnover experienced by SNAP. While gender training is scheduled in the future, many staff currently implementing the program are yet to be trained;  Feeding and child care has occurred while the mother attends training;  Men as Partners groups have been created, although they often differ in structure. Such groups focus on engaging men in health education and awareness, with little attention paid to establishing how men could be engaged in child care. Father CGs were not taken forward, and seem unlikely to proceed; and SNAP MTE – Final Report Moneval Solutions Ltd Confidential Page 42  Participatory gender training and sensitization for women and men has taken place as part of FFS training, father support groups, youth groups, and women’s groups. This generally focuses on health education rather than gender or child care. Whilst the GIAP is a useful and meritorious document, the MTE mission doubts whether all the activities can be implemented satisfactorily. SNAP will, however, continue to roll out and implement a streamlined version of the plan through FFSs, VSLAs, mother CGs, and vocational training in the coming year. Recommendation 29 in Section 9 provides suggestions for realizing some of the plan’s most important elements. Other Activities under Gender: SNAP has made progress in engaging men and women in SO1 activities, and in encouraging women to take leadership roles. Under the health/nutrition component, SNAP has made progress in involving men in MCH and nutrition through the establishment of Men as Partners groups. These are separate from the mother CGs but involve training and health education, and many members are partners of PM2A beneficiaries. These groups, though small in number, are having an impact on the belief and behavior of the participants. Men appear well informed on health and nutrition messages, and report engaging other women in education. Currently, however, SNAP does not have any female HPs. Staff have reported that they are unable to recruit them because of the distances HPs must cover, literacy requirements, and husbands that refuse to allow them to be away from the home for extended periods. The SNAP staff could not articulate specific strategies for recruiting female HPs. Overall, activities demonstrate how SNAP staff are paying increased attention to gender issues, not only in the field but also in the Freetown office. The implementation of the gender action plan mentioned above includes activities through FFS, VSLAs, mother CGs, and vocational training in the field, and transcends both SO1 and SO2. SNAP amended the FFS training material to include updated messages on health, nutrition, and gender. The SO1 team trained the community farmer facilitators on how to use the revised curriculum. The mid-term impacts are quite impressive, with male and female beneficiaries reporting to the MTE team how much better their marital life has become with the introduction of greater gender sensitivity (Box 2: text highlights this finding with reference to the village of Madina). The only challenge is how to measure social behavior with quantitative indicators, and the SNAP team will most likely need to use anecdotal and success stories to illustrate progress. Box 2: THE CASE OF THE VILLAGE OF MADINA: In the village of Madina, a couple beneficiaries reported that before SNAP, they had six children but now they have made the joint decision to have no more children following their exposure to gender teaching. Many participating women said that being exposed to the same teachings with their husbands was a source of “harmony” in their family: the husband does not to “yell orders“ on what or how to plant crops. They have learned the same things from attending the FSS. SNAP MTE – Final Report Moneval Solutions Ltd Confidential Page 43 6.4 Social and Behavioral Change SNAP’s use of FFSs appears to have brought both qualitative and quantifiable benefits in terms of increasing yields and incomes and the transformation of social behavior in communities in which they take place. The FFS instrument has built trust and solidarity in communities still recovering from the trauma of war, and seems to encourage farmers to work and share a development vision under the leadership of their peers. The perception of “nothing good can come out of another fellow farmer” is perhaps diminishing. In addition, both male and female community members that participate in FFS activities experience the same curriculum and are likely therefore to have emerged with a common understanding of how to continue to cooperate and collaborate in both their lives and the farms they run. The establishment of VSLAs without external financial and managerial support provides another good indication of solidarity and behavioral change with regards finances and management of money. Groups themselves decide on the membership criteria, lending limits, and so on, and use mutually determined and accepted rules of practice based on transparency, responsibility, participation, and respect. There are provisions in the MoUs viewed by the MTE team for members to “be there for one another” in case of life shocks such as ill health, bereavement, or school financing difficulties. It appears that both men and women participate in such decision-making events. The introduction of the gender component into SNAP has also brought a change in social behavior. During MTE interviews, some women suggested that on their farms, husbands no longer “order” wives, nor decide what crops to grow and how, because they (men) have received the same messages. Many men and women confess they now live happier lives than before the arrival of SNAP. During FGDs and KIIs, women were seen to lead and respond to questions more vocally than men, and were able to effectively inform the mission of achievements they felt important. This was a clear success in changing social behavior in a largely male-dominated country. 6.5 Integration Whilst integration is not a cross-cutting theme, the relationship and integration of SO1 and SO2 is nevertheless important and therefore discussed below. As per SNAP’s FY12 ARR, the integration of SO1 and SO2 received increased attention through strengthening coordination and the impact of activities, minimizing redundancies, and increasing the consistency of communicating with beneficiaries. SNAP amended the FFS training to include updated health and nutrition elements, while the SO1 team trained the community farmer facilitators on how to use the revised curriculum. The MTE team was impressed during FGDs and individual interviews with how well FFS and vocational education beneficiaries could cite how to prevent cholera and the benefits of using tippy taps, latrines, and utensil racks. They had received these health lessons from the SO2 field agents and/or the FFS facilitators, but there was no input from health staff. Of the 150 neighborhood mothers and LMs who participated in the MTE FGDs, 79 reported that they or someone in their household participated in agriculture or VSLA activities. Six of the seven Men as Partners members interviewed either participated in agriculture or their partners were PM2A beneficiaries. SNAP MTE – Final Report Moneval Solutions Ltd Confidential Page 44 SNAP has therefore, clearly defined entry points in its target communities. This approach has many benefits, as everybody is exposed to the same knowledge. There are households where the wife is a PM2A beneficiary and the husband an FFS member, and cases where the woman belongs to the two groups. In its FY13 PREP, SNAP will continue to strengthen the program of integration, including the continuing use of health and nutrition education curricula delivered to both SO1 and SO2 beneficiaries. Program integration in the coming years will support strengthening the capacity of the existing VSLA groups and the establishment of new ones composed of both SO1 and SO2 beneficiaries. 6.6 Sustainability/Exit Strategies Sustainability and exit strategies are discussed below. The former adopts a long-term perspective on the issue of change, whilst the latter deals with the exit mechanisms that SNAP will adopt as it disengages from beneficiary communities. Sustainability: With regards sustainability, it is hoped many of the activities SNAP has initiated or developed further can continue into the future. As with any program, sustained change and growth are difficult to identify during implementation, but rather should be examined some years after the end of activities. Nevertheless, at this point the MTE team can already see that behavioral changes have taken place with regards health, awareness of nutrition, and child care amongst the program’s target population. In other respects, there are also good prospects for sustained agrarian development, albeit at a smallholder or even household level, in terms of better farming practices, techniques, and relationships with markets and commodity buyers. Some of the instruments SNAP has promoted, such as the FFS and FaaB, are already seen to have a knock-on effect in non-SNAP communities or on non-directly participating farmers and households. This replication may continue in its current form or even be modified to suit local preferences and attitudes. Exit Strategy: Most SNAP partners and beneficiaries know when it will end. However, SNAP has not started to work with all stakeholders (DHMTs, MAFFS, DCs, FFS/PM2A beneficiaries, and so on) to develop or implement a transition strategy or plan. SNAP field staff do not appear to have the necessary specialized skillset to engage in transition and exit planning (please refer to the recommendation 3 in Section 9 for guidance on addressing this finding). During MTE FDGs, some communities stated that they could still provide incentives for the FFS facilitators when SNAP has gone because “the teachings are very good and they are one of us.” SNAP staff can approach the communities with the following options: Option 1: Each participating member gives the FFS facilitator one kilo of maize, groundnuts, rice, a root of cassava, and so on after the harvest Option 2: Cluster groups/associations to set aside a token monthly cash amount from their VSLA SNAP MTE – Final Report Moneval Solutions Ltd Confidential Page 45 account to be given to the facilitator Option 3: Community leaders could send some people to work on a facilitator’s farm during plating or harvesting as free labor for a period of time to continue the learning process The VSLAs will undoubtedly be one of the paramount sustainability strategies of SNAP. As a result, a focus on assisting the existing groups, as opposed to creating new ones, will be a crucial task to take on as soon as possible. As for the vocational education endeavors, sustainability and exit/strategies should mainly rely on:  Facilitating contact between beneficiaries and potential markets, e.g. private contractors, schools (for uniforms), government entities, and so on;  Assessing with them the possibility of refresher courses to improve the quality of their products and their presentation (carpentry, soap-making, tailoring, tie-dyeing, and so on);  Helping them develop a robust and realistic business plan for the replacement or purchase of additional tools; and  More practical monitoring of field visits. 6.7 Environment Stewardship ACDI/VOCA submitted an Initial Environmental Examination (IEE) plan to FFP along with the SNAP proposal. The IEE lists the mitigating measures to be taken during the implementation of a MYAP program to minimize the environmental risks in compliance with USAID FFP policy. In 2012, SNAP updated its IEE with the assistance of ACDI/VOCA’s director for environmental compliance, and ACDI/VOCA submitted an environmental monitoring plan along with the IEE to USAID/FFP. The SNAP team has also developed a set of environmental monitoring tools and guidance for the program, and initiated training in, and utilization of, tools by relevant staff. In turn, the staff have trained SNAP beneficiaries in the field. Environmental stewardship essentially concerns SO2, with little SO1 involvement. During discussions with the various groups, the MTE team learned how the communities are conscious of environmental issues in the domain of their respective activities. They appear to follow what they have learned from the SNAP field staff to the letter. Unfortunately, the template for the monthly reports by SNAP field staff does not have a specific section to report on environmental issues. The MTE strongly suggests a review of the template to include reporting on different environmental aspects. This approach will make the annual ESR (Environmental Status Report) easier to compile. With respect to SO1 beneficiaries – SNAP provides new jerry cans and nylon rope to construct tippy taps. The jerry cans are widely available and commonly used. It is common practice for non-SNAP jerry cans to be recycled and used to collect water or palm wine. With respect to SO2 beneficiaries (and FFS beneficiaries in particular) – The following lists a range of issues that are environmentally important and which impact agricultural and farming communities: SNAP MTE – Final Report Moneval Solutions Ltd Confidential Page 46  Fencing the farms with local materials is recommended to protect against rodents and goats, instead of using mechanical traps or rodent poison. In addition, farmers are advised to take their dogs to their farms to chase away rodents;  The use of scarecrows is recommended instead of slings or other traps for birds or other animals;  Fertilizers and pesticides are not used and slash￾and-burn is not promoted. The new farming methods yield good results, and the production is organic by default;  For Integrated Pest Management (IPM), farmers are encouraged to use ash as a defense against pests that attack cassava or groundnut leaves. An original way to fight against grasshopper invasions has been to kill and crush some of them and mix these with papaya leaves diluted with water, which is then sprayed over the farm. The farmers are extremely satisfied with the results. Some farmers said the addition of cigarette butts to the mixture also gave excellent results; and  Field agents use dead wood as a support to display technical information on the demo plot. Environmental issues and vocational education (with respect to tailoring, carpentry, soap-making, and tie-dyeing):  The environmental risks from tailoring and carpentry are minimal. From observation, the MTE team did not see any small pieces of cut materials littering the vicinity of the tailor shops visited or clinging to trees. The carpenters do not cut down trees for their work, but instead buy their wood from dealers in the community;  Soap-making and tie-dyeing is potentially harmful to the environment and to human health. The beneficiaries took the MTE team through a wide range of measures they were taught during their training to protect the environment and themselves;  All residues from caustic soda during soap-making are to be dumped in latrines and not in open nature; and  All of the above-mentioned findings are good practices to maintain and replicate. On a negative note, it was observed that the demonstration farms did not have technical information on display relating to the crops except the spacing measurements. Some of them had the information in plain view on cardboard that could not stand the vagaries of the weather. 6.8 Resiliency to Shocks With regards food security, food availability is the first pillar in resiliency to shocks, followed by income diversification to sustain accessibility. Photo credit:: Ange Tingbo, 2013 Above: Novel way of protecting your farm from pests. Scarecrows dressed in male and female clothing to ward off monkeys. Clothes are changed regularly to keep the monkeys from believing that these are just scarecrows! Sierra Leone SNAP MTE – Final Report Moneval Solutions Ltd Confidential Page 47 Throughout the MTE field visit, stakeholders, including the prime beneficiaries, indicated that new farming methodologies introduced by SNAP had helped them increase their yield threefold (for cassava, rice, sweet potatoes, and so on). New crops, such as pigeon peas, beans, and sesame, have been introduced to help farmers diversify their crops. Undeniably, their Months of Adequate Household Food Provisioning (MAHFP) have increased as a result of increased production and productivity. It is well known that sustained resiliency against shocks requires appropriate crops, and the variety of crops the community can now grow has broadened. SNAP increases household resiliency to food shocks through promoting the use of locally available foods and household gardening. Through FGDs, we learned these foods are not necessarily more widespread, but the people now know how to use them to create more nutritious meals. SNAP cooking demonstrations are helping households diversify their diets, which will help in times of food shocks. As for income diversification, the food processing component is still in the assessment phase. However, the VSLAs operating in the SNAP areas (although not started by SNAP) contribute greatly to community resiliency to shocks. Individual shocks are currently taken care of by the community through a well-organized safety net mechanism. The increased income from the sale of surplus crop production is also an important contributor to resiliency to shocks. For the remaining years of the program, SNAP will enhance the establishment of integral value chains. Current VSLAs should be strengthened in all aspects of their needs. Overall, existing VSLAs, or new VSLAs, should be taken to another level by linking them to marketing associations, microfinance institutions, or even banks. 6.9 Productive Youths The youths involved in vocational education (e.g. tailoring, carpentry, and soap-making) should be more closely monitored to heighten the success of their own activities and allow them to be cited as models for future students. The achievement of SNAP thus far has been outstanding. There are young men and women who now have a wider purpose in their lives; they possess a skill, and most of them look forward to contributing financially to their households. SNAP MTE – Final Report Moneval Solutions Ltd Confidential Page 48 7 Implementation Processes Implementation processes refer to an examination of commodity management, the SNAP M&E system, knowledge management, general management, and program design and implementation. To this end, the issue of commodity monetization and the poor realized commodity price has undoubtedly been a key concern for SNAP management, and is correctly receiving the highest level of attention. However, food management and distribution appear to be performing extremely well overall. Issues relating to the M&E system and practices require consideration, especially if it is to capture the qualitative good work of SNAP. Given the higher than expected staff turnover, institutional knowledge is compromised as staff leave and new staff take over, and the result is a team with little experience of the program. In terms of the general management, reference is made to SNAP’s warehouse system practices, which are functioning to the highest standard. In addition, program design is good and implementation is progressing satisfactorily. This said, there are some poorly performing areas within SNAP, illustrated by SO2 IR 2.3 and 2.4 discussed in Sections 5.3 and 5.4. 7.1 Commodity Management Monetization – The total monetization tonnage to June 2013, according to SNAP management provided data is 13,194 MT of rice, 350 MT of vegetable oil, 1,895 MT of bread flour, and 11,130 MT of (hard red winter) HRW wheat. The total proceeds stand at $10,430,117. The actual cost recovery rates for these commodities were in the range of 46.8% to 67.5%, which are lower than the 70% envisaged by SNAP. There was a monetary loss of $5,134,013 over the first three years of the program, which was anticipated. Based on the PREP document and discussions with SNAP management in Freetown, the future of monetization in Sierra Leone is not bright in terms of the satisfactory cost recovery of proceeds. To make up the monetization shortfall and continue SNAP activities in the coming fiscal year, the ACDI/VOCA team has been in negotiation with USAID FFP to replace as much as $4,348,013 using 202(e) resources. Without this additional funding, SNAP will be unable to continue to finance its program operations sufficiently, and some of its objectives, such as community assets, will be unattainable. The situation is delicate as 202(e) funds are limited. The agricultural lobby behind monetization and food commodities in Congress is extremely strong, which is evident from the press release below from Congress in July 2013 in reaction to USAID’s plan to make cash available for local food purchase and program implementation. “The decision is seen as a blow for the proposed new strategy for delivering food aid that envisions gradually scaling back the traditional strategy of buying American surplus and using those funds to purchase locally￾sourced food to both deliver it faster and spur local business. U.S. Agency for International Development Administrator Rajiv Shah has insisted that the plan is not to overhaul Food for Peace but make it ‘more rapid, cost-effective, and life-saving.’ SNAP MTE – Final Report Moneval Solutions Ltd Confidential Page 49 Shah however failed to persuade the powerful agribusiness and shipping lobbies, as well as legislators from rural states.” Food for distribution – By the end of FY12, SNAP had received 448 MT of CSB, 300 MT of vegetable oil, 1,097 MT of bulgur, and 467 MT of lentils for distribution. The balance is scheduled to arrive at the beginning of the first quarter of FY13 (October–December). These commodities are currently stored in six warehouses: two in the port of Freetown, one in Makeni, one in Mile 91, one in Bombali, and one in Kabala. There are also other small community warehouses that SNAP is planning to use to pre-position food allocation in some difficult localities. Evidently, the large number of warehouses that require management by a single program is problematic. It is a logistical challenge, and compiling the Recipient Status Reports (RSR) and the Commodity Status Reports (CSR) to USAID can be a particularly daunting exercise. Moreover, renting so many sites is costly, and has a toll on the budget. Food commodities management team – The team is knowledgeable and dedicated in both the Freetown and field offices. The MTE team participated in the monthly physical inventory of the warehouses in the port, which is conducted by the CTC. The MTE team concluded that the CTC and the warehouse keeper in the port have an outstanding knowledge of Title II food management, including security, cleanliness, space from the walls, space between stacks, keeping ledgers, stacking on pallets, procedures for disposing of unfit foods, FIFO, and the presence of warehouse equipment, e.g. ladders, scales, extinguishers, and so on. The physical inventory conducted should be retained as good practice. Contrary to the way it is usually carried out, the CTC counts the commodities first before checking against what was written on the stack cards and in the ledgers. The CTC used a ladder to reach the top of the last pile to check the famous “donut,” which is an empty space created in the middle and surrounded on all sides by sound bags or cartons of food. Additionally, the CTC used a notebook to jot down calculations per pile/stack instead of using loose sheets of paper. In case of any future discrepancies a year from now, the CTC can pull out these calculations for verification. The MTE team witnessed the same level of good management on visits to other warehouses. However, a few observations are provided below:  The warehouses all have the three-padlock system for front door security. However, one person keeps all sets of keys instead of requiring two persons to be present to enter the warehouse;  There is no column on the right of the ledger or the stack card for the signature of the person carrying out the food movement (receipt/delivery). It is difficult to trace back to the person involved in the operations when there is a replacement; and Photo credit:: Ange Tingbo, 2013 Above: Form Used by SNAP The current form used for Commodity Stack recording could benefit from an additional column for signatures improving accountability and tracking of record keeping, Sierra Leone SNAP MTE – Final Report Moneval Solutions Ltd Confidential Page 50  The warehouse managers all work alone in their duties, which can be problematic in cases of leave or sickness. PM2A food distribution – This is an SO cross-cutting issue. The PM2A food distribution is facing organizational problems inherent to food programs, but this is more challenging for SNAP due to the logistical difficulties encountered in Sierra Leone. The most critical problem is the inaccuracy of the distribution lists established by SO1 and SO2 staff. Beneficiaries would be invited to the PHUs for food distribution to find out they are no longer on the list, even when they meet the criteria in their category. An incident that occurred in Karamanka is illustrative of this, where many women were inexplicably removed from the list, which led to a very confused situation. The ACDI/VOCA food distributor decided to call off the operation and the Paramount Chief was called in. Fortunately, there was no violence and the matter was amicably resolved. Distance is also a major problem for some beneficiaries, coupled with organizational challenges that mean women spend a number of hours at a distribution site and go home late at night, sometimes empty-handed. As PHUs are used as a distribution site, there is often not enough room to accommodate all participating mothers. At a number of centers visited by the MTE team there were no outdoor shelters for mothers and their children. How empty containers were used after distribution was not clear. Moreover, the rations are indicated in kilos and liters, and in their interactions with the MTE team the beneficiaries were keen for the rations to be expressed using their own local measurement system. It is difficult for an outsider to associate the local gauges with the approved USAID FFP rations, and indeed converting the measurements communicated to them is a capacity-building strategy. They will eventually have to adopt the metric system if they are to continue to belong to a clustered FFS or an association dealing with large crop buyers. Food that remains undelivered to women who are not on the list is returned to the warehouse, which is not a particularly efficient use of resources. Under USAID/FFP policy food allocations cannot be provided retrospectively. As a result, SNAP might have an unexpected high tonnage of food on their hands, an issue that must be discussed with USAID FFP. Please refer to Section 9 (Recommendation 13) for further guidance on this. 7.2 Monitoring and Evaluation The MTE team reviewed the M&E system used by SNAP, which is mainly quantitative in nature with little qualitative data being recorded. The “why’s and how’s” of the varying IRs and identified shortcomings need to be equally captured by the system. SNAP is now using a household census survey to assist with food distribution. This is designed to inform SNAP of the number of individuals within a household receiving support through SO1 and SO2, allowing for a more accurate assessment of the crossover between the SOs. The establishment of a centralized database and clarity in the data collection methodology is critical in assessing the number of beneficiaries in each SO. The current methodology counts the same beneficiary participating in the different activities of the two SOs as two different persons. This might result in SNAP MTE – Final Report Moneval Solutions Ltd Confidential Page 51 an artificial inflation of the total number of SNAP beneficiaries as people are counted twice, and therefore must be avoided. In addition to the quantitative bias of the system, the tools (i.e. the actual forms themselves) used to collect SO2 data, for example, are not uniform within all SNAP districts. This is concerning as it will lead to inconsistencies and an inability to make accurate comparisons of progress. Subheadings viewed by the MTE team are not specific enough in terms of information gathered, and instead are restricted to simple general headings, such as “Achievement/Outcome (please report at outcome level),” for each IR being reported on. It was also found that the field staff responsible for M&E only collect data and are not involved in analyzing it. This amounts to enumerator work, and does not fully utilize the skills of the M&E staff. Please refer to Section 9 for further details of the MTE recommendations (recommendations 4, 22, and 23 in particular). The creation of a new database will be instrumental in updating the list of beneficiaries in each component of SNAP. While SO1 collects a great deal of data for the indicators used in reporting, the system is laborious and multi-layered. Lead Mothers report on activities using a pictorial data collection form. This is then given to HPs, who review it and compile their report. District supervisors collect HPs’ reports and then compile their own report for submission to field supervisors, who review and submit it to the M&E team, who then submit it to the Freetown M&E team. Within this system, staff could not articulate how the information they collect is used for program review or improvement. For SO2, the current system of reporting begins with information being passed from field FFS facilitators to field agents. Both quantitative and qualitative information is then shared with agronomists, from them it is passed to DOMs, then to team leaders in Freetown. It appears that the data is not systematically entered by the M&E manager in Freetown.. The reporting forms used in the four districts by field agents are not systematically the same across the board. Furthermore, the forms are not specific in terms of information other than “Achievement/Outcome (please report at outcome level)” for each IR. This does not mean that the data is erroneous, but rather it is suggested by the MTE team that SNAP put in place a system by which the M&E manager receives the information at the same time as the team leaders. The monthly meetings with MAFFS and DC members offer suitable channels through which to report program achievements and progress to the national and local authorities. However, joint visits should be systematized and SNAP should find ways to inform communities themselves of the progress and challenges of activities. This could be a quarterly or semi-annual exercise. The M&E of the vocational enterprises has to be more practical. Some statements in the report, such as “effective and successful monitoring was done by the MED officer” or “the soap-making groups are also doing well in their various communities,” might not help in gauging the actual progress of the activities. 7.3 Knowledge Management Staff turnover is having a negative impact on the retention of knowledge in SNAP. In addition, after orientation, field staff do not appear to have been trained uniformly. There is no SOP to instruct staff on how to document reports, success stories, or challenges. SNAP relies heavily on on-the-spot correction, and while this is an effective strategy to train illiterate LMs and provide on-the-spot SNAP MTE – Final Report Moneval Solutions Ltd Confidential Page 52 mentoring, it does not allow the program to view its challenges longitudinally. For example, there is no record to show whether LMs are consistently facing challenges with a particular subject area that might require the review or revision of material (please refer to recommendation 4 in Section 9 for suggestions on how to remedy this). 7.4 General Management The detail below provides comment on general management, with reference to commodity management, staff turnover and general data collection. Commodity management – Warehousing of the Title II food commodities by SNAP is in full compliance with the basic principles of USAID FFP’s Title II food management in terms of building security, food handling and stacking, sanitation, use of pallets, and accountability. Unfortunately, logistical problems (e.g. bad road conditions, unavailability of trucks at certain times), combined with problems in updating the lists of mother beneficiaries and their children, mean making timely deliveries to the distribution sites is challenging. In addition, the practical organization of food distribution at the PHUs has seen some issues that result in beneficiaries being at the sites for longer than actually needed. It is critical to address these weaknesses during the remaining SNAP implementation period. High staff turnover – The high expatriate and local staff turnover is impacting the smooth implementation of SNAP activities. Approach to Collection and Management of Data - The current approach to collecting information and the treatment of the data is problematic. An example of this involves IMC, the medical partner of the consortium, being in charge of the registration of PM2A beneficiaries (SO1), while ACDI/VOCA’s staff are responsible for organizing the physical distribution of the food commodities (SO2). Unfortunately, a lack of coordination and communication ultimately results in many beneficiaries being taken off the list or not appearing on the list at all. 7.5 Program Design and Implementation The design of SNAP is an appropriate response to the main problems of a country torn apart by a decade-long civil war. The situational analysis identified the root causes of the problems, and the activities of the two SOs are designed to address them. Although problems were evident nationwide, ACDI/VOCA and its partners (IMC and OICI) choose to work in the most impoverished districts of the country. The choice of beneficiaries corresponds to criteria of vulnerability: men and women eager to improve their living conditions through better farming, youths who have not completed the regular school curriculum, and children under two that require sustained nutrition. In spite of the difficulties faced by SNAP (e.g. staff turnover, serious logistical challenges, and financial constraints due to problematic monetization), activities are being implemented according to the approved proposal by USAID FFP. ACDI/VOCA has not requested any drastic change in the course of the activities in its last two ARRs to FFP. SNAP MTE – Final Report Moneval Solutions Ltd Confidential Page 53 8 Lessons Learned and Best Practices The following key lessons have been identified through the MTE: Lesson 1: SNAP improving health service delivery at PHUs can help but only if there is a clear strategy in place - DHMTs and PHU staff report positive changes in their ability to deliver quality services to communities. SNAP training and supportive supervision is positively affecting the way DHMTs interact with PHUs and improving supervision, accountability, and reporting. There is no documented strategy to help PHUs cascade the training further into the community or to other PHUs. The cascade of training ranges from a 20-minute lecture on what was learned during the training to longer, more intensive, training sessions. This goes unmonitored by SNAP. Lesson 2: Changing nutrition and health behavior can lead to huge improvements in the health status of targeted community members - SNAP beneficiaries have detailed knowledge of nutrition-promoting behavior and practices, including understanding the importance of dietary diversity and the proper and hygienic feeding of children. Meanwhile, PM2A beneficiaries report changes in breastfeeding practices and cultural taboos. While beneficiaries have stated that food is not more abundant, they now are beginning to accept that the difference in their children’s health is attributable to SNAP training in helping to make their families healthier. Lesson 3: Learning through discovery – The teaching and learning through discovery in the FFS activities is a remarkable approach. Farmers are encouraged to implement experimental farming techniques alongside their own traditional practices, and they see the difference during the germination phase and in the increased volume of the yields. The default cascade training approach of non-participating beneficiaries who have adopted new methodologies through observation is encouraging. SNAP should attempt to capture this information through field agent reports and consider incorporating it into the ARR to USAID FFP. During FGDs and KIIs, SNAP beneficiaries indicated that their life at home has become more pleasant due to adhering to the new methods on their own farms. There are fewer disputes among married couples over how to plant crops or which crop to plant as they learned the improved land husbandry and plant management techniques together. Lesson 4: Gender awareness lead to improving social cohesion – SNAP’s work on gender issues is changing behavior. Throughout interviews with the MTE team, the following statement was frequently repeated by beneficiaries: “No more fight in the house; we live now in peace.” If this is happening in individual households, it is anticipated that widespread support for behavioral change will benefit all communities where SNAP is intervening. This social impact on the lives of community members can never be fully captured by an indicator in an IPTT when it is the pivotal foundation for the existence of a community, let alone a family. Lesson 5: Vocational education for the youth can help set in place early incentives for enterprise development and local economic prosperity –Vocational education, which focuses directly on youths as a primary target, is a relevant component of the SNAP spectrum. The MTE team visited varying micro-enterprises, including tailoring, carpentry, soap-making, and tie-dyeing SNAP MTE – Final Report Moneval Solutions Ltd Confidential Page 54 activities, and female hairdressers. With the exception of the soap-makers and tie-dyers, the background information suggests SNAP is correctly targeting beneficiaries. In the majority of these cases, the youths were dropouts from school because their parents were unable to pay their tuition fees, they had challenging family relationships (e.g. polygamous parents), or were orphaned at an early age. Lesson 6: Integration of heath and livelihood activities whilst adopting a whole household approach can promote life changing opportunities - SNAP has demonstrated success in the integration of SO1 and SO2 beneficiaries, while the “whole household approach” is strengthening the overall impact and uptake of promoted behaviors. The approach of exposing both men and women to the same development themes in the SO activities is good practice, and should be continued during the remainder of the program. SNAP MTE – Final Report Moneval Solutions Ltd Confidential Page 55 9 Recommendations The following sections categorize 33 recommendations of high, medium, and low priority. Of these, eight are high-priority recommendations for SO1, two medium priority, and two low priority. For SO2, there are 12 high-priority recommendations, seven medium priority, and two low priority. Some of these also include cross-cutting elements that deal with implementation or resource use. The SO1 and SO2 recommendations were prioritized according to their urgency and feasibility of being implemented to deliver benefits within the remaining time of the SNAP program. To this end:  High-priority recommendations generally aim at managerial/administrative/ project issues that can be addressed in the very near future and which if not implemented may have a negative impact on the program achieving its intended outcomes;  Medium-priority recommendations refer to activities or situations that require attention but are not as urgent as higher priority recommendations; and  Low-priority recommendations refer to issues that can currently be kept on the “back burner,” but require action in the future. 9.1 High-Priority Recommendations 9.1.1 Strategic Objective 1 Recommendation 1: SNAP field staff – Staff require a Standard Operating Procedure (SOP), which includes reporting procedures, documentation instruction, operational plans, and programmatic background. On-the-job training is an economical way to mitigate staff turnover. However, the training should be more structured, with specific programmatic interventions covered, documented, and signed off on. By having a SOP, SNAP can ensure that all new staff understand, and can implement, program activities in line with accepted strategies. The SOP should only require gathering documents and compiling them for field office use. On-the￾job training subjects should be developed with field office staff and include the main SNAP tenets and strategies. The creation of a SOP will help mitigate the impact of staff turnover while maintaining and ensuring program quality. Senior program staff can educate field staff on how and when to use the SOP during regular scheduled monthly meetings. The SOP and specified on-the-job training subjects should be developed in the next quarter. Recommendation 2: PHU training cascade – To facilitate the adoption of improved and sustainable health techniques, SNAP training should include strategies on how PHU staff can improve cascade training, such as including a timeline, helping attendees identify a concrete strategy to educate others, and having PHUs and DHMTs agree to set time aside for cascade training. This form of training is essential as PHU staff also have high rates of turnover. Implementation of training cascades will increase the likelihood of more staff being trained and higher adoption rates. SNAP has scheduled follow-up training in the final years of the program. As part of the output it should consider including training in action plan development so that district supervisors can appropriately follow up with PHU staff. Monitoring of action plans will require additional training for district supervisors, which can be carried out at regularly scheduled meetings. SNAP MTE – Final Report Moneval Solutions Ltd Confidential Page 56 Recommendation 3: Exiting and transitioning – SNAP will end in 2015, barring any extension. As such, it will need to work with the MoHS and its target communities to develop an exit plan (should the program end in that geographic area completely) or transition plan (should the program modify its implementation and continue for some time) so that DHMTs can lead the implementation of the mother CGs. “Establishing and maintaining clear communication with communities about the exit process helps lead to successful exit and sustainable program impacts. Post-program evaluation is a valuable tool for understanding the sustainability of program outcomes and for improving the design and implementation of exit strategies.” 11 These are specialized skills and will likely require SNAP field staff training and capacity building to engage DHMTs in exit and transition activities. First, SNAP needs to determine its exit strategy; FANTA recommend12 that an exit strategy needs to include the following core elements:  Identification of approaches to be used for different program components;  Specific criteria for graduation (of communities) and exit (of the program from the region);  Measurable benchmarks for assessing progress toward meeting the criteria;  A timeline, recognizing flexibility may be required;  Identification of action steps to reach the stated benchmarks and identification of parties responsible for taking these steps; AND  Mechanisms for periodic assessment of progress toward exit and for possible modification of the exit plan. The MTE suggest SNAP will have to consider exiting at three levels. These are: SNAP partners – ACDI/VOCA, IMC, and OICI have to start strategizing their plan in their different technical domains; ACDI/VOCA with agriculture/livelihoods, IMC with PHUs/health, and OICI with vocational education activities. The overall strategies will be coordinated by ACDI/VOCA as the lead partner of SNAP Local authorities – In their regular meetings with the district or council authorities, SNAP agents should initiate the conversation regarding the completion date of the program. During the MTE, district authorities knew that SNAP has an end but they were not aware of the actual date. In Kabala and Kailahun, the DC expected to receive vehicles and equipment at the end of the program, as they had from other international NGOs. Donating vehicles and equipment so that business can be carried out as usual is useful, but it might not increase sustainability. It is critical to start conducting joint visits now so that SNAP can share what is being implemented for the community in agriculture and livelihoods. More importantly, it might take some time to include operations in their respective budget to cover new resources for fuel or maintenance. They need to be aware now, and inform local or national government about such future budget implications 11 Rogers, Beatrice Lorge, and Kathy E. Macías. Program Graduation and Exit Strategies: A Focus on Title II Food Aid Development Programs. Washington, D.C.: Food and Nutrition Technical Assistance Project, Academy for Educational Development, 2004. 12 Ibid. SNAP MTE – Final Report Moneval Solutions Ltd Confidential Page 57 Communities – Field staff and team leaders (for SO1 and SO2) have to start informing communities about how to prepare for continuation at the completion of SNAP activities. This can only happen once there is a clear exit strategy Given the constraints found within Sierra Leone, it is recommended that SNAP considers using “phase over” as its exit strategy, which refers to the transfer of responsibility for activities aimed at accomplishing program goals (current activities, or other activities aimed at achieving the same outcomes) to another entity. SNAP leadership will need to develop benchmarks and train field staff to document the exit strategy implementation. Working with the DHMTs, SNAP should begin to plan the transition of responsibility to DHMTs. To facilitate sustainability, this should be gradual, realistic, and time-bound. SNAP should begin with discussions and planning for the results-based financing if it is a viable option. In addition, DHMTs require basic information on how funding works to realistically plan for the future. Exit and transition planning will require considerable expertise. To minimize costs, the SNAP consortium should identify such expertise and experience in-house or, if none can be identified, hire an external consultant to develop an exit plan of action on its behalf. Please refer back to Section 6.5 for further detail on this issue. Recommendation 4: Monitoring for program improvement – Field staff need a mechanism through which they can analyze data collected for continuous program improvement, which would allow SNAP to systematically and intentionally improve services for beneficiaries and for them to be trained to use that system. The checklist currently used does not allow the staff a longitudinal view of issues to identify problems. Adjusting the existing checklist through training and expecting staff to identify longitudinal “big picture issues” will allow continuous quality improvement. Once an issue is identified, SNAP field staff should develop a plan to address it and document the process. Senior SNAP program staff should analyze the issues across districts to understand if they are localized or systemic. To institute continuous program improvement, supportive supervision and monitoring “checklist” training should be modified to specify challenges in health education subject matter, training implementation, or on-the-spot correction that might help SNAP develop a more nuanced response to implementation variances. This information should then be analyzed in the field to assess and identify subject matter material specific to districts, with the SNAP office in Freetown undertaking analysis to assess its quality. Noting such challenges will not add costs to the program, although staff will need training on what to note and how to make such additional notations. Staff will also likely require further training on how to analyze data for program improvement. This can be achieved during the monthly meetings for example, with such information providing a longitudinal picture of SNAP’s progress and responsiveness. Recommendation 5: Tippy taps – SNAP currently does not have a plan to construct or distribute tippy taps in terms of the prioritization of beneficiaries. In KIIs, some respondents thought there were challenges with the module on tippy tap construction. SNAP MTE – Final Report Moneval Solutions Ltd Confidential Page 58 SNAP’s tippy tap planned distribution should to be firmed up and systematized across districts. In the development of the plan, SNAP should consider prioritizing neighborhood mothers who are graduating; a tippy tap could be provided as recognition of their achievement and participation in the program. SNAP field staff must be aware of the plan and accurately inform communities of the basis for distribution. This plan should also include strategies for how tippy tap construction will occur. It is recommended that SNAP refrain from using HPs as tippy tap constructors, but rather use them as a means of engaging the community and male partners in the uptake of tippy taps. Developing a plan of action with field supervisory staff should not increase program costs as it can be achieved during regularly scheduled meetings and documented in the SOP. Recommendation 6: Exiting and transitioning from feeding – SNAP should include instruction on how to wean children off of CSB. Mothers should be given instruction on how to slowly introduce new foods around three or four months before graduation from the program to ensure a smooth transition from CSB to household food. Like all cooking demonstrations, this should include a focus on diverse diets and hygiene. This will require additional education for field supervisors and HPs, which can be provided during the monthly meetings. Recommendation 7: LM graduation – Like all PM2A beneficiaries, LMs graduate from receiving food rations when their child reaches 24 months. As they graduate, SNAP will have to identify additional strategies to ensure their continued participation. SNAP should consider additional motivation for LMs, such as prioritized participation in skills building and literacy, community recognition, certificates, and literacy classes. This means LMs could then be evaluated and considered for HP positions. Recognition for LMs will not cost a great deal, and SNAP should work to expand its work with community leaders to express this recognition; it could be achieved through community forums in which their husbands participate. Prioritization or inclusion of LMs in building skills and literacy activities will require senior managers across the SNAP consortium to develop prioritization plans. As it is unlikely that the more than 1,73713 LMs could be incorporated into each activity, the consortium should develop requirements for LM participation, prioritize participation, and inform all field staff that this will require time and planning. SNAP should develop this plan before the end of 2013. Recommendation 8: Cooking demonstrations – The MTE findings indicate that there may be challenges with cooking demonstration implementation and instruction. However, given the limitations of the MTE and the lack of documentation of challenges in this area, SNAP should monitor cooking demonstrations, noting the challenges with materials or variances in implementation or understanding. With adoption of the recommendation for better monitoring of program improvement, SNAP should discover if there are challenges with the cooking demonstrations within three to four months, and determine if this is a localized problem or a program-wide issue. 13 SNAP FY13 Q2 Progress Report Final (5-3-2013). SNAP MTE – Final Report Moneval Solutions Ltd Confidential Page 59 9.1.2 Strategic Objective 2 Recommendation 9: Establishment of VSLAs – Ideally, ACDI/VOCA should update the outcomes of the VSLAs census and draft an implementation plan with a timeline for achievement. Many VLSAs are currently in an embryonic stage of development at best, and need technical and managerial support to provide communities with the tools to decide how they could evolve and options for their improved functioning. This will only involve management training and require no additional SNAP finance, which will effectively mean the program focuses on supporting existing VSLAs rather than creating new ones. This would appear both feasible and sensible given the limited time and resources available to contribute to some form of sustained financial management at community level. Following on from this, VSLAs will need continued support to become more sophisticated and organized in their management and operation. This will require training and specialized support that SNAP could consider providing immediately. Recommendation 10: Operations (cross-cutting) – Freetown and district offices should review procedures to speed up the purchase of spare parts, and put in place a reliable system for motorbike and vehicle maintenance. The current system for equipment and vehicle purchase and maintenance must be carefully examined, as the inefficiencies found by the MTE team in this area can be reversed. District Office Managers (DOMs) should provide the Freetown office with a list of reliable mechanic shops in their area and sign a contract with them for light maintenance. This does not preclude submission of quotations to the DOMs before any actual repairs are carried out. “Heavy repairs” should be handled by the Freetown office in consultation with the DOMs. Some motorbikes are currently grounded because they are inadequate for the rugged terrain. If there are capital/equipment purchases in the coming year, SNAP partners should coordinate buying logistical equipment based on field realities, and ACDI/VOCA should take responsibility for coordinating the purchases on the consortium’s behalf. These are activities that need to be actioned immediately. Recommendation 11: Administration (cross-cutting) – The FFS facilitators are not receiving their stipends in a timely manner. This is frustrating field staff, and some SNAP field agents have become very vocal on behalf of the facilitators. Both Freetown and the DOMs should work out a plan for the timely payment of the stipends to FFS facilitators. It is suggested they can either:  Pursue the payment by phone and see how facilitators can validate their SIM cards; and/or  Explore the possibility of money transfers through local financial institutions. Recommendation 12: Staffing (cross-cutting) – ACDI/VOCA should identify the root causes of high expatriate turnover, which inevitably is hindering progress and damaging morale. One option to explore is allocating five additional days of vacation in the sub-region in return for a three-year irrevocable commitment. This is a long-term solution, but ACDI/VOCA should ensure the current management staff is on board for the remaining duration of the program. The MTE team recognizes this is a sizeable problem, but urges SNAP to find a solution that will stabilize the team until the end of the program. SNAP MTE – Final Report Moneval Solutions Ltd Confidential Page 60 The MTE team recommends that SNAP recruit qualified M&E staff to fill field-level positions in Kailahun and Bombali/Tonkolili as soon as possible. Given that the recruitment of qualified staff is a real challenge, ACDI/VOCA should start the process immediately. In addition, SNAP should hire an assistant Compliance Technical Coordinator (CTC) as soon as possible to assist the current CTC with the monthly inventory of warehouses and the audit or management review of the SNAP partners (the MTE team was informed that this process had been initiated). Recommendation 13: Funding/Monetization – Due to the problematic status of funding the program with monetization proceeds, ACDI/VOCA should pursue its negotiation with USAID FFP and obtain written confirmation so that an agreement can be reached to supplement or replace monetization proceeds with 202(e) resources. The transfer modification should be obtained from USAID FFP in the first quarter of the next fiscal year (October–December 2013), or be guaranteed in the approval of the pending PREP. The last resort is to explore the possibility of third-country monetization, which will be far from easy and needs to be addressed immediately by senior management and consortium partners alike. Recommendation 14: Warehousing – Use one or two warehouses in the port of Freetown as a transit storage, and use the Mile 91 warehouse as the main field warehouse for storing and dispatching food allocations to other districts. The road conditions are not seen to be a problem with the current location of the six warehouses. Food transfers can take place within the next quarter. The two-padlock system for the warehouse doors is a security measure following FFP’s food management principles, and at least two persons with keys should be present to open the doors of the warehouse. Should there be a break in, there would be two people to testify to this. The Commodity Management Team (CMT) should ensure that two people keep a set of keys. Spare keys should be securely kept with the DOM, Human Resources (HR), or commodity manager. ACDI/VOCA should immediately appoint assistant warehouse managers to the warehouses while moving to a point where there are fewer warehouses in the SNAP system. Recommendation 15: Vocational Education – SNAP staff should help vocational education beneficiaries explore different markets for their products. Starting immediately, the Training Field Officers (TFOs) and their assistants should facilitate contact with schools as potential buyers of school uniforms in their respective district for tailoring graduates, and FFS and PM2A beneficiaries for soap makers and other district markets. SNAP staff should help active vocational graduates write attractive investment plans (the CTC and TFOs should be able to carry this out). This should start immediately and be part of the regular monitoring visits. Meanwhile, M&E visits should have an exit strategy built into them where, for example, practical ideas for sustainability are discussed, such as improving the quality of their livelihood enterprise (e.g. soap-making, tailoring, and carpentry). Recommendation 16: Database (cross-cutting) – Both SO1 and SO2 team leaders and the M&E manager must come up with a more effective data collection methodology to develop more accurate lists of food beneficiaries. The new database will also help with reporting results. This activity involves the DOMs, the agronomists and International Medical Corps (IMC) health SNAP MTE – Final Report Moneval Solutions Ltd Confidential Page 61 promoters. The inaccuracy of the current food distribution lists has created much frustration in the field. This activity should start immediately with the establishment of a new data collection methodology as a first step. All involved parties across the SNAP components will have to communicate updates more frequently and in a systematic manner. As with other high-priority recommendations, SNAP consortium partners and senior management must be instrumental in ensuring this happens. Recommendation 17: Food Distribution (cross-cutting)– To reduce the time spent by women and their children organizing efficient food distribution at the PHUs, ACDI/VOCA, IMC, and the PHUs should review the overall distribution methodology. The revision should be simultaneous to the development of the new data collection methodology. Women and children often have to wait in the sun to receive their food rations if there is no shelter available at the PHUs. It is suggested that SO1 field staff should sensitize the husbands to build sun and weather shelters using local materials at each distribution PHU. Recommendation 18: SNAP staff to hold monthly meetings with the District Council and Ministry of Agriculture authorities – SNAP field agents interact with Government of Sierra Leone (GoSL) agriculture extension agents. However, it transpired during the KIIs held by the MTE team that joint visits are not carried out in the field. One of the reasons is the financial implication given the system for payment of per diems. This needs to be discussed by SNAP management and clear instruction provided to the agents. The requirement is for SNAP field agents to have joint visits with MAFFS extension agents on a quarterly basis; that DOMs, M&E officers, and executive managers should have joint field visits with local district/MAFFS authorities on a semi-annual basis cannot be overemphasized. A solution may be for SNAP to agree to use the GoSL per diem rate when civil servants are out in the field with ACDI/VOCA officers, but revert to ACDI/VOCA rates when they are not. An alternative is to take an actual cost position for such joint activities. Recommendation 19: The FFS training document – This is in a plain text format without any visual aids. Given the audience, the approach for adult teaching would recommend that training material be more pictorial and visual. It should contain pictures of what is being taught as good practice (spacing, weeding, seed population, germination, pest management, fencing, harvesting, and so on) that can be easily compared visually with the results of traditional farming practices. This means SO2 staff must agree on changing the material in line with the best practices suggested. Recommendation 20: The phrasing of Strategic Objective 2 – This and the subsequent IRs vary in the original proposal, the Annual Results Report (ARR), and the PREP; IR 2.4 is omitted altogether in the latter two documents, while activities are reported on in IR 2.3. The SNAP team must bring clarity and ensure they use the same definition of each SO and IR throughout its documentation. The MTE team recommends that activities for agribusiness and farmers’ cooperatives, and VSLAs, should be captured by two different IRs. SNAP MTE – Final Report Moneval Solutions Ltd Confidential Page 62 9.2 Medium-Priority Recommendations 9.2.1 Strategic Objective 1 Recommendation 21: Strengthen the support system for children under two not currently in the program – Within the SNAP implementation area there are children under two whose mothers were not registered by SNAP, and this group is falling through the “cracks” in the system. They are referred to in the OTP feeding points but have no opportunities for follow-ups or capacity building. These children may be maternal orphans and cared for by elderly grandparents, making them extremely vulnerable to malnutrition. The SNAP senior management team should immediately consider strategies for the involvement of these caregivers in supportive nutrition activities, such as cooking demonstrations and nutrition education lessons. This is in addition to referrals for emergency feeding. These mothers could be visited by LMs or paired with new graduates for support and encouragement. Recommendation 22: Knowledge management – Given the high staff turnover, field staff should be given more instruction as to what constitutes a success story. SNAP has made great strides in changing food taboos. However, when staff were asked about positive changes they have witnessed, many (who may not have been present from the beginning) were not able to recognize progress or positive changes that can highlight achievements not captured through monitoring indicators. SNAP should consider documentation of the changes in food taboos and the Men as Partners development for sustainability purposes. Changes in cultural or traditional beliefs also represent a huge accomplishment and success story for SNAP, and can contribute to the long-term sustainability of behavior change. Documentation of this achievement will help maintain institutional knowledge of the program in the area. Additionally, SNAP should document changes in breastfeeding habits, another impressive program achievement. 9.2.2 Strategic Objective 2 Recommendation 23: Technical information – Some demonstration plots visited did not display any technical information in the field regarding a farming method or the crop being tested. This is a lost opportunity to instill the learning that has taken place. It is suggested that SO2 staff need to agree on changing the educational material in line with best practices such as increasing the pictorial usage in the training documents SNAP field agents should ensure the systematic display of technical information in all the FFS fields, including the type of seeds, date of plantation, height reached at a given time, probable time of harvest, and so on. Checks should also be carried out to ensure all field agents use environmentally friendly techniques to display the technical information on the demonstration plots (see the model of Madina detailed in section 6.3. Agronomists should exchange ideas with field agents and FFS facilitators, and share good practices observed at a demonstration plot with other FFSs. Technical information written on cardboard should be protected using a transparent plastic sheet to help avoid weather damage. SNAP MTE – Final Report Moneval Solutions Ltd Confidential Page 63 Recommendation 24: Exchange visits – An intangible benefit of SNAP has been to bring people together in the aftermath of the civil war. SNAP should organize exchange/study visits between FFSs within the same chiefdoms and/or between different districts. This will hopefully stimulate weak groups and assist them in becoming better farmers. Above all, people of different tribes sharing what they have in common would contribute to national unity. Recommendation 25: SNAP resources (cross-cutting) – The FFS facilitators have many demonstration plots under their supervision, although these are often well spread geographically. Therefore, the lack of transportation observed by the MTE team may jeopardize the facilitators’ productivity. SNAP should provide FFS facilitators with bicycles as stipulated in their contract (it should be clarified in a MoU that the primary user of the vehicle has the ownership, responsibility for maintenance, what to do in case of theft, and so on). This should be initiated when the financial situation of SNAP becomes more stable. However, the facilitators should be informed immediately about any cause of delay. Recommendation 26: Common remuneration (cross-cutting concerning implementation) – In the field, it was reported that some staff members from the different SNAP consortiums are performing more or less the same activities but have different contractual working conditions. An example illustrates this point; some are entitled to a travel allowance while others are not. Even though each member of the consortium has its own identity and policy, SNAP should make sure that some basic employment conditions are shared by all staff, and ACDI/VOCA should include this point on the agenda of the next meeting with the consortium. This can be coupled with a review of the local staff salary. Due to the “poaching” syndrome experienced in Sierra Leone by international NGOs, it might be useful to explore how annual salary increases can be based on performance within a scale of perhaps 1% to 5%. Recommendation 27: Warehouse management (cross-cutting) – The current stack cards used in warehouses do not have a column for the person who is actually moving the food (i.e. entry/delivery/balance) to sign off. This is a risk in case staff, basically the warehouse manager, leave or fall ill. It is recommended therefore that:  The food commodity manager should instruct the storekeepers to add a new column for signatures in the ledgers (by the remarks section) and stack cards (the MTE team was told there was currently a large supply of the cards and ledgers; when this supply is finished, new forms should have the additional column pre-printed). Action should be taken now to implement this; and  The food management team should ensure that passes are obtained for SNAP-designated staff to enter the port premises at any time, as the experience of the MTE team showed that access to the food stock can easily be denied by the port authority. Recommendation 28: Bulk storage management by farmers – Most farmers store their produce at home because of a lack of trust in any common community store. There is insufficient knowledge of how best to structure and govern group storage arrangements in a way that would not jeopardize an individual household’s stocks (as discussed in Section 5.2). SNAP should help the SNAP MTE – Final Report Moneval Solutions Ltd Confidential Page 64 group or association draft a MoU containing the provisions required to safeguard the cohesion of the group. An issue which contributes to the problem of trust or lack thereof in crop bulk storage is the phenomenon of “desiccation.” For example, a 50 kg bag of rice or cassava brought to the warehouse does not weigh the same 30 days later due to moisture content changes. This has been a source of conflict in other countries, and it will become more evident in SNAP target areas if the issue is not explained upfront. SNAP should explore two options to incentivize storekeepers: 1) pay a salary agreed by the associations or 2) encourage a monthly agreed in-kind (i.e. a percentage of the crop stored) donation. In the latter case, the storekeeper is free to monetize, keep the crop allocation for his/her own consumption, or both. Recommendation 29: Gender Integration Action Plan (GIAP) (cross-cutting) – Although the GIAP development appears well structured it is nevertheless an ambitious plan. To ensure at least some of its salient elements can be effectively realized at community level, whilst also allowing some management changes to occur, the following recommended actions should be taken:  Dissociate the gender field activities portion from the overall plan design, so to also address gender issues at the Freetown office level;  Update the GIAP from page 7 onwards, entitled “Project Activities,” according to the field realities and available resources; and  Include the revised gender themes found in the FFS, VSLA, and Men as Partners curricula, and hold refresher courses for the field agents. 9.3 Low-Priority Recommendations 9.3.1 Strategic Objective 1 Recommendation 30: Tippy taps – Tippy taps are successful because they can be made with readily available materials by anyone. SNAP should promote uptake by demonstrating the construction of tippy taps using a variety of materials, for example gourds, one-liter water bottles, and locally made/resourced twine. SNAP should also consider using recycled jerry cans that are currently used to collect and carry water and palm wine. Rather than purchasing soaps from external distributors, SNAP should consider sourcing soap for tippy taps from within the program. Recommendation 31: Gender – The MoHS, communities, and field staff had difficulty articulating strategies to increase male participation in maternal and child health strategies. Men as Partners groups are an encouraging step in this direction, and should be studied closely to assess their motivation, impact, involvement, and contribution in the community. To improve male involvement, SNAP should consider inviting men to CG sessions, especially during the discussion of sensitive subjects or when addressing cultural taboos. SNAP should also consider adjusting the requirements for HPs to engage women in these positions. Field staff reported that a critical barrier to women’s participation as HPs is the long distances they are expected to cover. Therefore, SNAP should weigh up the value of having female HPs against the distance requirement, perhaps reducing the size of catchment areas to increase the chances of female participation. Another barrier field staff raised is that women do not normally ride bicycles. The MTE team noted, however, that in some villages PM2A beneficiaries felt this was untrue and women do in fact ride bicycles and could become HPs. SNAP MTE – Final Report Moneval Solutions Ltd Confidential Page 65 Efforts to recruit women as HPs should be documented and reviewed to identify challenges and lessons learned that may assist in future planning. 9.3.2 Strategic Objective 2 Recommendation 32: Empty containers – Please refer to FFP’s Regulation 11 for the policy on empty containers, as it contains guidelines on their use. Previously used, but now empty, containers can either be sold and proceeds used by distribution centers to buy utensils and cooking demonstration ingredients, or else thoroughly cleaned and used for temporarily storing commodities. It was also seen by the MTE team that they could be used to build shelters for women and children. Recommendation 33: Use of computers – Field agents who are all more or less computer literate still have to write their reports by hand in preparation for the monthly meetings with the DOMs. The MTE team suspects that presenting the reports to the supervisors or DOMs must be extremely challenging. The field agent cannot rewrite a whole paragraph or two when some handwritten information is not accurate or difficult to understand. The SNAP management team and the DOMs should equip district offices with up to two computers for field agent use. This can be initiated immediately if funds are available. SNAP MTE – Final Report Moneval Solutions Ltd Confidential Page 66 10 Limitations to the Evaluation A number of limitations were noted in respect of the MTE methodology, which will have some impact on the interpretation of the findings and recommendations. However, the MTE team is confident they were able to gather representative data through the triangulation of information gleaned from various sources. These included FGDs, KIIs, field agents, DOMs, site visits, observations, and program documentation. The main limitations are summarized below: The selection of beneficiaries, informants, and locations – The MTE beneficiaries, informants, and locations were pre-selected by the SNAP team in Freetown. Consequently, there is the possibility of a bias and limitation in the results. However, based on the FGDs, KIIs, and site visits, the MTE team has concluded that the selection of participants and locations was made objectively. There was no evidence or indication of a bias towards a particular type of beneficiary or location. The team attempted to mitigate potential bias to some degree by informing the participants that they were not part of the SNAP program and that it was important for the success of the MTE and the SNAP program that they were open and objective in their discussions. The MTE team is confident that evaluation findings and recommendations would have been similar even in the case of a more refined sampling and data-collection procedure. The number of beneficiaries, informants, and locations – The MTE team covered 29 locations and met with 647 beneficiaries, 160 informants, and visited 20 program sites. This represents only a fraction of the more than 400,000 beneficiaries targeted by SNAP, and this may therefore prejudice the representativeness of data collection. Interpreters – A major constraint of the MTE was the need to work through interpreters. This was necessary and unavoidable; nevertheless, given the variety of experience in qualitative research and translation abilities, there is a risk that some elements of the investigation may have been lost in translation. Quantitative survey – The SOW did not include a quantitative study of the SNAP program. This might have helped to introduce greater statistical and quantitative rigor to what amounted to a qualitative review. Control group – The SOW did not allow the MTE team to visit reference groups or sites, which might have added further insight to the evaluation. Therefore, it was not possible to compare the progress and development of SNAP beneficiaries against the non-SNAP population. VSLA groups – The VSLA coordinator recruited in January 2013 is currently mapping out the communities that are interested in savings activities. This component is at an early stage, and it was decided to exclude it from the MTE. Description of SO2 IRs – A major challenge and limitation in the MTE was the discovery of a variation in the phrasing of the SO2 IRs in key project documents. This is evident in the original proposal, the ARRs, and the PREP. This caused some confusion and inconsistency in understanding during the MTE, which is illustrated by the following wording from the various documents mentioned above: SNAP MTE – Final Report Moneval Solutions Ltd Confidential Page 67 From the SNAP proposal: SO2: Enhance Livelihood Opportunities From FY12 ARR SO2: Enhanced livelihoods for vulnerable people, especially women and Youth From the PREP 2013 SO2: Enhanced livelihoods for vulnerable people, especially women and Youth IR 2.1: Increased agricultural production and crop diversification through improved inputs and agricultural practices IR 2.1: Increased agricultural productivity at the household level. Improved agricultural practices (FFS) IR 2.1: Increased agricultural productivity at the household level IR 2.2: Enhanced integration into agricultural value chains IR 2.2: Increased access to financial and marketing resources IR 2.2: Increased access to financial and market resources IR 2.3: Increased human capacity and community assets IR 2.3: Improved Community Assets IR 2.3: Improved Community Assets IR 2.4: Increased access to financial resources No IR 2.4 No IR 2.4 At times, the activities of the last two IRs are combined. Please refer to recommendation 20 in Section 9 for further details. Travel conditions – Travelling to the MTE locations was a challenge due to the poor road infrastructure and the condition of the roads due to the rainy season. This impacted the MTE by limiting the possibility to widen the field of data collection. Timeframe – The timeframe for the MTE was challenging, with 18 days allocated for data collection involving FGDs with 647 beneficiaries, KIIs with 160 individuals, and 20 site visits in 29 locations. The MTE plan included one day for the preparation of preliminary findings and ten days for report writing. SNAP MTE – Final Report Moneval Solutions Ltd Confidential Page 68 ANNEXES SNAP MTE – Final Report Moneval Solutions Ltd Confidential Page 69 11 Annex 1: Evaluation SOW Appendix C Sustainable Nutrition and Agriculture Promotin (SNAP) Mid –Term Evaluation and Annual Results Survey Scopes of Work Consultancy Title: Mid-Term Evaluation and Annual Results Survey Consultancy Project Name: Sustainable Nutrition and Agriculture Promotion (SNAP) Location: Sierra Leone Timeframe: April – June 2013 Level of effort: Up to 55 work days; number of work days to be extended only upon approval from project supervisor A. INTRODUCTION ACDI/VOCA and partners, the International Medical Corps (IMC) and Opportunities Industrialization International (OICI) are implementing since July 2010 a 5-year USAID Food for Peace (FFP) funded program, Sustainable Nutrition and Agricultural Promotion (SNAP). SNAP is a new PL 480 title II USAID Multi Year Assistance Program (MYAP) which is being implemented in 18 chiefdoms in 4 districts of Sierra Leone. SNAP has passed the middle of the project term and will undertake a mid-term evaluation (MTE) in order to take lessons from the past and better prepare the future, according to the project M&E plan. Along with the mid-term evaluation exercise, SNAP will undertake its FY13 Annual Results Survey (ARS). For this purpose, SNAP is recruiting a firm or pool of consultants to help the project undertake the two separate but linked processes. B. DESCRIPTION OF THE PROJECT SNAP seeks to achieve the following two objectives which will contribute to the reduction of food insecurity among the targeted vulnerable rural populations of Sierra Leone: 1) reduce chronic malnutrition among children under five (strategic objective 1or SO1); and 2) enhance livelihood opportunities (strategic objective 2 or SO2). In addition, the program will address four crosscutting themes: resiliency to shocks, productive youth, gender equity, environmental stewardship. SNAP will provide the targeted population with the necessary tools to improve their food security situation by addressing the key factors influencing utilization, availability and access to nutritious and diverse foods. SNAP will also facilitate linkages among donor, government and community activities to deepen impact and sustainability. SNAP’s sustainability strategy strengthens the capacity of communities as well as public and private stakeholders. SNAP’s core activities include: reducing the prevalence of child malnutrition through integrating food distribution with promotion of good maternal and child health, sanitation and hygiene practice through the PM2A approach, strengthening local health service provision SNAP MTE – Final Report Moneval Solutions Ltd Confidential Page 70 through capacity building of health professionals and community health workers, health facility rehabilitation and equipment and supportive supervision of clinicians in rural health units, enhancing rural livelihoods through training of farmers in target PM2A communities in improved agricultural production and post-harvest handling technologies and practices, increasing availability of high-nutrition foods and income from the sale of cash crops, and income diversification through training in vocational skills to support alternative livelihoods. While ACDI/VOCA is the prime contractor and leads the SO2 activities, IMC implements the SO1 ones and OIC the vocational training strategies. To date, SNAP has provided: food distribution and good maternal and child, hygiene, and sanitation training to more than 35,000 Pregnant and Lactating Mothers, currently providing services to approximately 26,000 through 1736 Care Groups (CG), led by 183 Lead Mothers (LM), enhanced agricultural production and postharvest handling training to more than 19,000 farmers, currently providing services to 9,526 farmers in 321 Farmer Field Schools (FFS) lead by 108 community facilitators and 321 Agricultural Field Agents (AFA) , adult literacy, numeracy, and vocational skills training to more than 1024 individuals, and capacity building to health service providers in 113 Peripheral Health Units (PHUs) across SNAP Districts. C. SCOPES OF WORK (SOW) UNDER THIS TASK ORDER There are two Scopes of Work (SOW) under this Task Order – an SOW for a Mid Term Evaluation (MTE), and an SOW for an Annual Results Survey (ARS). Any proposal submitted must be responsive to both SOWs. Mid Term Evaluation SOW 1. Evaluation Purpose and Objectives The overall purpose of the MTE is to assess the extent of progress of SNAP in achieving its targets and to provide evidence-based recommendations and identify practical measures, both in terms of Program Administration, Operations, and Management, as well as Program Technical Implementation, to improve and strengthen implementation in the out-years. More specifically, the MTE’s objectives are to:  Assess the project’s progress in meeting its expected results and the likelihood of attaining its outcomes and impact as defined in the agreed Performance Monitoring Plan (PMP);  Re-examine the validity of the SNAP proposed hypothesis and assumptions set in the strategic objectives, results framework, and how the PMP appropriately reflects the capture of these;  Identify both challenges and successes across the program, and provide new insight into how to strengthen the program based on these successes and challenges;  Document the project’s lessons learned and best practices.  The MTE must comply with USAID’s Evaluation Policy (January 2011), found at http://transition.usaid.gov/evaluation/USAIDEvaluationPolicy.pdf 2. Objective of the consultancy and tasks to perform SNAP MTE – Final Report Moneval Solutions Ltd Confidential Page 71 The firm/consultants will lead all the data collection and analysis of the MTE with respect to the methodology described in the following sections. They will more specifically:  Complete a comprehensive desk review of key documentation to assess the effectiveness of both programmatic and operational activities, processes, and procedures of SNAP;  Complete and finalize a set of data collection instruments to be used in the evaluation;  Lead in the recruitment and training of local enumerators to carry out the bulk of data collection activities;  Lead and carry out the data collection activities;  Impute, process and analyze the data collected;  Submit written report with initial analysis and recommendations to ACDI/VOCA Field M&E Manger;  Provide additional comment, support, answer to questions, feedback to the ACDI/VOCA; Regional and HQ team in developing the final, complete analysis and report writing of the MTE;  Present the findings to the staff and the summary of the findings to SNAP’s partners and donor;  Participate in the presentation of the summary of the findings to SNAP’s beneficiaries. The consultants/firm must comply with USAID’s Evaluation Policy (January 2011), found at http://transition.usaid.gov/evaluation/USAIDEvaluationPolicy.pdf 3. Data collection methodology The mid-term evaluation will be formative in design, which will include a number of methodologies including: desk review and analysis of the SNAP quantitative data and core programmatic and operational documents, surveys, key informant interviews, focus group discussions, and direct observation. The M&E Staff, SO1 and SO2 teams and team leads, and project beneficiaries will be consulted in the development and finalization the MTE methodology, instrumentation, and selection of project sites and persons to interview. Donors and project beneficiaries will also be given the opportunity to participate in the process. Sampling Frame SNAP as developed the following DRAFT sampling frame. This is only a DRAFT sampling frame – SNAP expects the bidding firms/consultants to review this sampling frame and propose modifications and adjustments based on their experience and technical expertise. The contacted firm / consultants must work with the SNAP M&E, SO1, SO2, and team leads to finalize the MTE sampling frame and sample. SNAP MTE – Final Report Moneval Solutions Ltd Confidential Page 72 Chiefdoms Communiti es FFS CG PHU DHMT SNAP Teams (So1, So2, Commodity , M&E) Local Gov. (District) Local Gov. (Chiefdom) Local Gov. (Communit y) Unit 2 Per Distri ct 2 Per Chief dom 1 Per Comm unity 1 Per Com munit y 1 Per Chiefd om 1 Per Dis trict 1 Per Distric t 1 Per Distri ct 1 Per Chief dom 1 Per Commun ity District 1 2 4 4 4 2 1 1 1 4 8 District 2 2 4 4 4 2 1 1 1 4 8 District 3 2 4 4 4 2 1 1 1 4 8 District 4 2 4 4 4 2 1 1 1 4 8 Total 8 16 16 16 8 4 4 4 16 32 This illustrative sampling frame would result in aggregate totals for data collection activities of approximately 164 surveys/guided interviews, 28 focus groups, 104 key Informant Interviews, and 48 site visits / direct observations. See Attachment B: Illustrative Mid Term Evaluation (MTE) Sampling Matrix for detailed breakdown of these estimations. Final numbers will be finalized by the winning firm and SNAP team. Desk Review and Analysis:  Review and analysis of key project documents. Documents must include but are not limited to:  The IPTT/PMP, baseline report, annual reports, routine monitoring reports, outcome monitoring reports;  Past PREPS, Quarterly Reports, and Annual Results Reports.  2012 households data collection findings;  Other special study reports (gender assessment, SO1 LQAS, etc.);  Other relevant qualitative and quantitative available information that:  help determine the targeting, project strategies, service delivery mechanisms and the quality of services, quality of project results  evaluate the development assumptions, quality of program management, cross cutting issues (gender, integration, environmental compliance), and the likelihood of achieving the targets. Qualitative Survey / Guided Interviews A number of qualitative survey instruments will be developed for different SNAP stakeholders. Where necessary, such as with beneficiaries who may or may not have functional literacy and numeracy skills, surveys will be conducted verbally using a guided interview protocol to operationalize the survey. The sampling schedule for Qualitative Surveys / Guided Interviews may include but not limited to: Qualitative Survey / Guided Interview Schedule SNAP MTE – Final Report Moneval Solutions Ltd Confidential Page 73 Qualitative Survey / Guided Interview Schedule Farmer Field Schools (FFS) Vocational Skills Trainings (VST) Care Groups (CG) FFS Participants VST Participants CG/PM2A participants FFS Facilitators VST Facilitators CG/PM2A graduates FFS Graduates VST Business Cluster Members Lead Mothers Producer Group Cluster Members Health Promoters PHU staff Qualitative Survey/Guided Interview instruments must be developed and finalized in consultation with the local M&E staff, the management staff including the SO1 and SO2 team leaders, and implementing partners. Qualitative Surveys/Guided Interviews will cover at a minimum the following elements and dimensions:  Effectiveness, strengths, weaknesses, new opportunities and suggested improvements to the design of SNAP activities, including coordination between SO1 and SO2 components  Effectiveness, strengths, weaknesses, new opportunities and suggested improvements to program resources used in implementing SNAP activities (FFS, CG, VSLA, and VST materials and curricula)  Effectiveness, strengths, weaknesses, new opportunities and suggested improvements to the management and operational implementation of SNAP activities including coordination between SO1 and SO2 components  Effectiveness, strengths, weaknesses, and suggested improvements to the administration and operations of SNAP activities Focus Group Discussions The MTE will include a number of focus group discussions engaging different stakeholders. The primary objective of the focus group discussions will be to analyze and assess SNAP program and operational planning, coordination, management, implementation, and reporting. The interview schedule will include at a minimum: Focus Groups will cover at a minimum the following elements and dimensions: Focus Group Schedule Leadership/Management Team Lead Mothers Head Office SO1 and SO2 Teams FFS Facilitators Field SO1 and SO2 Teams VSLA Coordinators Commodity Management Team M&E Team Health Promoters SNAP MTE – Final Report Moneval Solutions Ltd Confidential Page 74  Effectiveness, strengths, weaknesses, new opportunities and suggested improvements to the design of SNAP activities, including coordination between SO1 and SO2 components  Effectiveness, strengths, weaknesses, new opportunities and suggested improvements to program resources used in implementing SNAP activities (FFS, CG, VSLA, and VST materials and curricula)  Effectiveness, strengths, weaknesses, new opportunities and suggested improvements to the management and operational implementation of SNAP activities including coordination between SO1 and SO2 components  Effectiveness, efficiency, and suggested improvements to the management and operational implementation of SNAP commodity distribution to PM2A including coordination between SO1 and SO2 components Key Informant Interviews Key Informant interviews will be conducted with critical stakeholders across the span of SNAP programming to provide additional insight into SNAP programming and activities. These interviewed program beneficiaries/stakeholders and staff will be selected purposively in both high-functioning and low-functioning programming areas in order to gleam additional insight and perspective into the SNAP program. Gender balance and locations will also be considered. We estimate the total number of interviewed persons not to exceed 20. The interview schedule may include but is not limited to: Key Informant Interview Schedule Select Staff within each SO at HQ and Field Local Government and Line Ministry Officials Select Staff within HQ and Field Commodity Management Teams Commodity Vendors Select Lead Mothers USAID and FFP Representatives Select Farmer Field School Trainers Youth Committee Members Select PHU staff Key Informant Interviews may include the following elements and dimensions:  Overall perception of the SNAP program and SNAP Partners  Effectiveness, strengths, weaknesses, new opportunities and suggested improvements to the design of SNAP activities, including coordination between SO1 and SO2 components  Effectiveness, strengths, weaknesses, new opportunities and suggested improvements to program resources used in implementing SNAP activities (FFS, CG, VSLA, and VST materials and curricula)  Effectiveness, strengths, weaknesses, new opportunities and suggested improvements to the management and operational implementation of SNAP activities including coordination between SO1 and SO2 components  Effectiveness, efficiency, and suggested improvements to the management and operational implementation of SNAP commodity distribution to PM2A including coordination between SO1 and SO2 components SNAP MTE – Final Report Moneval Solutions Ltd Confidential Page 75 Site Visits and Direct Observation As part of the MTE, trained enumerators will carry out direct observations of a number of SNAP activities. These visits will include some “Scheduled” and some “Unscheduled” visits. Site visits and Direct Observations should include but are not limited to: Site Visits and Direct Observation Schedule: FFS Training Sessions CG Training Sessions Commodity Direct Distributions Direct Distribution Loading/Unloading at Warehouse Vocational Skills Training Sessions 4. Key Evaluation Questions A number of Key Evaluation Questions have been identified to which the MTE must provide a response within the final report. In the analysis and assessment of each of the key questions, the firm should take into consideration the following dimensions and their effects/impact.  Relevant FFS, CG, and VSLA methodologies and curricula  Program and activity design  Leadership and management  Logistical and operational management & implementation  Coordination between the partners and their programming  M&E and learning systems Progress Towards Program Outcomes14 1. Are we seeing any increase in usage of preventative and curative services by pregnant and lactating mothers? Why or why not? If YES, is it likely to be sustained without program resources? Why or why not? 2. Are we seeing any increase in access to nutritional foods in target communities? Why or why not? If YES, will it likely be sustained without program resources? Why or why not? 3. Are we seeing any increase in consumption of nutritional foods in target communities? Why or Why not? If YES, will it likely be sustained without program resources? Why or why not? 4. Are we seeing any transfer of new knowledge, skills, and practice/behavior change from program PM2A participants to mothers NOT enrolled in PM2A program? Why or why not? 14 Questions around progress towards program outcomes are not to be answered by any population based, quantitative survey, and are not intended to measure impact. Instead, they are intended to test a number of core assumptions imbedded within the SNAP TOC, and should be answered through qualitative surveys and interviews with relevant stakeholders, such as Lead Mothers, PM2A beneficiaries, local leaders, FFS beneficiaries, and community members. SNAP MTE – Final Report Moneval Solutions Ltd Confidential Page 76 5. Are we seeing any increase in agricultural yields in target communities? Why or why not? If YES, will it be sustained without program resources? Why or why not? 6. Are we seeing any transfer of new knowledge, practices, and skills from program FFS participant farmers to farmers NOT participating in FFS? Why or why not? 7. Are farmers adopting the new technologies and practices imparted by the project? Why or why not? If YES, will it be sustained without program resources? Why or why not? 8. Are we seeing any increase in income for SNAP beneficiary households? Why or why not? If YES, will it be sustained without program resources? Why or why not? 9. If we are seeing any increased income, is it being used to strengthen household health? If YES, will it be sustained without program resources? Why or why not? 10. Has new knowledge translated into adoption of new practices and recommended behaviors? Why or why not? What factors are contributing to this? If YES, will it be sustained without program resources? Why or why not? Program Design and Implementation 1. Which strategies and activities have been more effective so far and why? 2. Are we targeting the right beneficiaries? 3. Are the right beneficiaries being enrolled/enrolling in the program? Why or Why not? 4. What explains the low achievement rates on the following indicators:  Use of improved storage  Women’s organizations/associations assisted  New technologies or management practices made available  Yield of some crops  Crop sales  Youth receiving vocational training  Community-assets projects completed  New targeted beneficiary reached  Health facilities with functional bio-sand filters  Outreach to PLW 5. Do producer groups show the capacity to sustain continued, increased production of cash-crops and nutritious foods? Why or why not? Dimensions should include:  Knowledge of improved production and post-harvest handling skills and technologies  Use of improved production and post-harvest handling skills and technologies  Good practice in business management of producer groups  Established relationships with input suppliers, buyers, and markets  Access to technical support related to agriculture without program assistance 6. What would Care Group members like to see happen to the Care Groups after the program ends? How can we best help to achieve this? 7. Do PHUs show long-term sustainable capacity to provide services to pregnant and lactating mothers? Why or why not? 8. Why has full recommendations of the gender assessment recommendations not yet been implemented? What opportunities should we work on to make this happen? 9. Do we foresee any issue in addressing FFP Environmental Compliance requirements? 10. Do we foresee any issue to address or any opportunity to implement the project’s WASH strategy? Program Management, Coordination, and Operations SNAP MTE – Final Report Moneval Solutions Ltd Confidential Page 77 1. How effective and efficient is coordination between the two SO activities. What has worked well? What has not? Dimensions under consideration should include at a minimum:  Program Planning and Activity Coordination between HQ (Washington) and the Head offices (Freetown)  Program Planning and Activity Coordination between SO1 and SO2 Head Offices (Freetown)  Program Planning and Activity Coordination between Head Office and Field Offices  Program Planning and Activity Coordination between SO1 and SO2 teams in the field  Program Implementation of SO1 and SO2 Integrated Activities 2. How effective and efficient is integration between the two SO activities. What is working well? What is not? 3. How have implementation related problems been identified, analyzed and solved? What has worked well in terms of problem solving? What has not worked well? 4. Is SNAP staff qualified and knowledgeable of the purpose and methods used in the program and equipped to properly implement the activities? 5. Does the management structure/culture support staff spending time creating and sharing knowledge (i.e., documentation, report outs in meetings, brown bags, mentoring). Do job descriptions identify clear knowledge sharing activities that should take place? 6. What are areas of excellence, and areas of improvement in terms of donor/awardee relationship including at a minimum:  Reporting  Communication  Donor/Mission/Awardee Coordination Monitoring and Evaluation 1. How well is the output monitoring system functioning? What is working well? What is not? Where, if any, are their bottlenecks in this system 2. Does the output monitoring it have well developed data collection forms, data flow diagram, defined responsibilities for each partner and/or sector, aggregation levels, and a clear link showing how the information will be aggregated and reported onto the Annual Results Report and SAPQ? 3. Does the monitoring system measure outcome level changes? Does the system use appropriate methods and tools? How often it measures outcome? Is the system too burdensome to the M&E team? 4. Does the outcome measurement system also investigates and present reasons behind the successes and/ or under achievements? How useful these findings are to the users? How this can be improved? 5. How is the output and outcome monitoring systems and monitoring activities linked/coordinated between SO1 and SO2 M&E Teams? Between the field and head office? Between head office and DC office? What is working well? What is not working well? SNAP MTE – Final Report Moneval Solutions Ltd Confidential Page 78 5. Roles and Responsibilities The consultants/firm will be supported by the SNAP staff and the ACDI/VOCA Regional and HQ team. For that purpose: The ACDI/VOCA Regional and HQ M&E Team Will:  Provide input and guidance in the development of the data collection instruments  Provide guidance when needed in the collection and analysis of the data;  Provide comments to the draft report. The ACDI/VOCA Local M&E Team Will:  Make available all SNAP documents and data needed by the firm to fulfill the desk review activity;  Provide sensitization in preparing Farmer Field School (FFS), Care Groups (CG), Provincial Health Units (PHUs), communities, and other stakeholders around the MTE;  Provide inputs into the firm/consultants’ initial drafting, and finalization of the instrumentation;  Provide Logistical support to the firm/consultants in implementing the Mid Term Evaluation Data Collection activities;  Provide oversight and management of the consultant/firm, and provide any necessary support and guidance in order to ensure all deadlines are met in time, and that the quality of deliverables meets expectations as will be detailed within the final contract issued to the firm/consultants;  Review the initial analysis presented within the report submitted by the firm/consultants, providing comments, opinions, questions and general feedback on the data/findings by the firm/consultants in order to ground the analysis in experience;  Provide comments to ACDI/VOCA;  Provide comments on the MTE report;  Participate in the results presentation meetings. Annual Results Survey SOW 1. Purpose of the Annual Results Survey The purpose of the annual survey is to capture and document progress made by SNAP in SNAP 18 targeted chiefdoms towards achievement of the IPTT targets for fiscal year 2013 (FY13) relating to:  agricultural/livelihood activities as well as food security in target communities;  health facilities/services in target communities;  nutritional status of women and children (0-59) months in the SNAP targeted households. The annual outcome monitoring survey will be used to quantitatively measure the changes in impact and outcome indicators of the SNAP program to inform management decisions. 2. Objectives of the Annual Results Survey SNAP MTE – Final Report Moneval Solutions Ltd Confidential Page 79 The specific objectives are: a) To measure outcome indicators as specified in the PMP and IPTT during FY13; b) To update IPTT and performance indicator reference sheet (PIRS) with outcome indicators values for FY13; c) To track progress of program implementation and provide possible road blocks that would inform management decisions. As the annual survey will be done in parallel with the MTE, the annual survey results will serve as inputs to the MTE process. 3. Description of the work to contract The purpose of the work to be contracted is to undertake the measurement survey of SNAP’s impact and outcome indicators by the related data for the fiscal year 2013 4. Activities The contracted entity will be in charge of: a) The development of a detailed survey protocol with detailed sampling procedures and results, data collection methodology and questionnaire; b) The recruitment and training of the enumerators and data entry clerks; c) The fulfillment of the data collection activities; d) The data entry. It is recommended that the contracted entity uses SPSS or Access for data entry and that data are entered twice to limit typo errors; e) The data cleaning and quality control at all stages of the survey process; f) Data processing and analysis; g) The reporting and presentation of the survey results. At all stage of the activities, the contractor will regularly communicate and engage in consultation with SNAP. Activity #1., #2, #3. will request SNAP’s approval before the start of the next activities. SNAP will provide the contractor with the project’s data in order to draw the individual beneficiaries to survey and to make them better understand the project. SNAP will also help facilitate the introduction of SNAP to the beneficiaries to survey. 5. Data to Collect The data to be collected during this annual survey is related to the 24 impact and outcome indicators as specified in the SNAP PMP and IPTT and listed in Attachment D: Impact and Outcome Indicator Table. The collection methodology of the FFP, FtF and F indicators shall follow the guidance found at http://transition.usaid.gov/our_work/humanitarian_assistance/ffp/ffpstdindicatorhb.pdf, http://www.feedthefuture.gov/sites/default/files/resource/files/ftf_handbookindicators_apr201 2.pdf and www.state.gov. Please note that 3 of the 24 indicators are anthropometric ones and will require the use of a medical doctor and specific equipment (e.g. height board and scale) in the data collection, and must follow WHO data collection procedures. 6. Sampling Frame The annual survey data will be collected through a one-time face to face interview of a representative sample of the project household beneficiaries. The households to survey will be identified randomly from the SNAP household beneficiaries database. Those will include households with: a) Pregnant women; SNAP MTE – Final Report Moneval Solutions Ltd Confidential Page 80 b) Mothers of children less than 24 months old; c) Caregivers of children 0-59 months; d) Farmers. Sample shall consider the farmers’ and children’s sex and location (district level) features as representativeness criteria. The sample size should be the smallest one while representative of the beneficiaries and respecting the statistics and scientific standards. Three scenarios might be considered: a) Sample size based on the LQAS methodology; b) Sample size based on the use of the confidence interval and margin of errors only (95% and 5 %); c) Sample size based on the magnitude of change. The firm should include their recommendation as to the best scenario from among the 3 different scenarios with clear explanation as to why they consider the recommended scenario as the best option. D. DELIVERABLES UNDER THIS TASK ORDER The firm or consultant will deliverable the following documents and deliverables, following recommended guidance where indicated: 1. Mid Term Evaluation (MTE) Deliverables a) Final MTE Sampling Plan b) Inception Report (report outlining initial findings based on documentation/desk review) c) Finalized data collection instruments (forms and guides) d) Curricula for training of the enumerators e) Draft MTE report f) Final MTE report15 g) PowerPoint presentation summarizing MTE results h) All MTE data files (electronic and paper versions), including completed questionnaires, focus group and interview notes or matrix analysis 2. Annual Results Survey (ARS) Deliverables a) Final detailed ARS protocol with the sampling plan b) Draft ARS report c) Final ARS report d) All ARS data files (electronic and paper versions) E. RECOMMENDED TIMEFRAME 15 The MTE report must be developed as outlined in Attachment E: MTE Report Guidance and Outline, which is consistent with FFPs SOW development guidance. SNAP MTE – Final Report Moneval Solutions Ltd Confidential Page 81 The mid-term evaluation process is planned to start in April and the final report should be available by the end of May 2013. The MTE findings and recommendations will be presented first and in details to the whole project staff in order to get any remaining feedback and comments. Short presentation sessions will also be organized for the donor. Below is a recommended Time Frame for activities related to completion the SNAP MTE and Annual Survey. Activity Date(s) Mid Term Evaluation Documentation review April 22 Inception Report (preliminary findings based on documentation review) April 26 Finalization of data collection instrumentation April 29 Qualitative survey administration May 20 Focus groups, key informant interviews, and direct observation May 27 Data compilation and analysis June 10 Draft report writing June 17 Collection of comments of the draft report June 21 Final report writing June 26 Presentation to the whole staff June 28 Presentation to the USAID/FFP and partners July 1 Annual Survey Finalization of the survey protocol including the survey questionnaire, the sample design/sampling frame April 29 Piloting of draft questionnaire May 1 Review and finalize Annual Survey questionnaire May 3 Logistics arrangement Recruitment of data collectors May 7 Sample selection May 15 Training of data collectors May 20 Field data collection June 10 Data entry, cleaning and analysis June 24 Draft reporting writing July 1 Collection of comments of the draft report July 5 Final report writing July 12 Presentation to the whole staff June 15 F. FIRM / CONSULTANTS REQUIREMENTS Institutional Requirements  Over 5 years of legal existence; SNAP MTE – Final Report Moneval Solutions Ltd Confidential Page 82  Successful track records of evaluation activities;  Successful track records of qualitative surveys design and implementation with at least 5 years of experience in the domain;  Successful track records of quantitative surveys design and implementation with at least 5 years of experience in the domain;  Records of LQAS based surveys design and implementation;  Experience with Title II projects and activities;  Experience in the SNAP’s districts preferred. MTE and ARS Leadership Requirements The firm MUST include two separate team leaders to carry out the two SOWs outlined in this Task Order: one for the MTE and one for the annual survey. The MTE Team Leader:  At least 5 years of experience in the design and implementation of evaluation projects;  Strong qualitative data collection and analysis skills;  Demonstrated knowledge of Title II projects and activities specificities and requirements;  Excellent English writing skills;  Demonstrated evaluation management capacities;  Prior experience in leading evaluation teams;  A Master’s degree in Social Sciences, Development studies or similar studies preferred. The Annual Survey Team leader will have:  At least 5 years of experience in the design and implementation of quantitative surveys;  Experience in implementing LQAS based surveys;  Strong quantitative data collection and analysis skills;  Demonstrated knowledge of Title II projects and activities specificities and requirements;  Excellent English writing skills;  Demonstrated evaluation management capacities;  Prior experience in leading evaluation teams;  A Master’s degree in Social Sciences, Development studies or similar studies preferred. Local Staff and Enumerators Requirements  At least several years of experience in the design and implementation of evaluation projects;  At least several years of experience in the design and implementation of surveys;  Strong quantitative data collection and analysis skills;  Strong qualitative data collection and analysis skills;  Demonstrate knowledge of the SNAP’s districts cultural specificities and dialects; SNAP MTE – Final Report Moneval Solutions Ltd Confidential Page 83  Experiences with Title II projects and activities preferred. G. SUBMISSION REQUIREMENTS All interested firms or consultants shall submit a detailed proposal including the following core components by 12 P.M. EST, April 11, 2012, to Awards at awards@acdivoca.org: 1. Detailed introduction to the proposing firm including capabilities and experience relevant to the SOWs under this Task Order; 2. Technical design responsive to the activities and objectives described within the two SOWs under this Task Order. The technical design must include a description of the firm’s overarching strategy, approaches, and methodologies to be used, and must demonstrate a comprehensive understanding of the objectives and activities of the SOWS under this Task Order; 3. Detailed, adjusted timeframe (if proposing a different timeframe then the one presented in this Task Order) for the completion of each activities and submission of each deliverable outlined in this Task Order 4. Detailed budget ; 5. Past references of the previous work conducted relevant to the SOWs under this Task Order; 6. Curriculum Vitae of the consultants or firms’ employees that will be mobilized for this contract; 7. Sample of report of similar work previously done and the relating data collection instruments. H. Evaluation Criteria ACDI/VOCA will evaluate proposals based on a best-value determination; Offerors should submit their most competitive price proposal. Proposals will be evaluated using the following criteria: Technical Scores Points* Delivery and Performance 25 Key Staff & Quality Control Mechanisms 25 Past Performance/References 10 Company Capabilities 15 Price or Cost Proposal 25 Total Technical Score 100 The evaluation committee will review the technical proposal based upon the technical criteria listed above. The cost proposals will be reviewed to ensure they are complete and free of computational errors. The committee will also assess the reasonableness of costs and the cost-effectiveness of the budget, and will determine whether the costs reflect a clear understanding of project requirements. A contract will be offered to the responsible Offeror whose proposal follows the RFP instructions and is judged to be the most advantageous to ACDI/VOCA. SNAP MTE – Final Report Moneval Solutions Ltd Confidential Page 84 11.1 Moneval Solutions Understanding of the SOW and Responses The core purpose of the evaluation is to assess progress toward meeting program targets and to provide evidence-based recommendations and practical measures to strengthen implementation in the out-years. 11.2 Global Objectives The global objectives of the SNAP MTE are to:  Assess the program’s progress in meeting its expected results and the likelihood of attaining its outcomes and impact as defined in the agreed Performance Management Plan (PMP);  Re-examine the validity of the SNAP proposed hypothesis and assumptions set in the strategic objectives, results framework, and how the PMP appropriately reflects the capture of these;  Identify both challenges and successes across the program, and provide new insight into how to strengthen the program based on these successes and challenges; and  Document the program’s lessons learned and best practices.  The MTE must comply with USAID’s Evaluation Policy (January 2011), found at http://transition.usaid.gov/evaluation/USAIDEvaluationPolicy.pdf 11.3 Specific Objectives The specific objectives of the evaluation are to:  Complete a comprehensive desk review of key documentation to assess the effectiveness of both programmatic and operational activities, processes, and procedures of SNAP;  Complete and finalize a set of data collection instruments to be used in the field based evaluation;  Impute, process and analyze the data collected;  Submit a written report with initial analysis and recommendations to ACDI/VOCA Field Chief of Party (CoP), Deputy Chief of Party (DCoP) and M&E Manager;  Provide additional comment, support, answer to questions, feedback to the ACDI/VOCA, Regional and HQ team in developing the final, complete analysis and report writing of the MTE; and  Present the findings to the staff and the summary of the findings to SNAP’s partners and USAID.  The consultants/firm must comply with USAID’s Evaluation Policy (January 2011), found at http://transition.usaid.gov/evaluation/USAIDEvaluationPolicy.pdf SNAP MTE – Final Report Moneval Solutions Ltd Confidential Page 85 11.4 Scope of Work In responding to the SOW, the MTE will cover the SNAP program from its implementation, document the impact to date and key lessons learned, and make actionable recommendations that support future planning for implementation activities. The key areas for the MTE are:  Progress towards program outcomes  Program design and implementation  Program management, coordination, and operations  M&E  Cross-cutting themes:  Resiliency to shocks  Productive youths  Gender equity  Environmental stewardship 11.5 Clarifications and Amendments An inception meeting was held with the MTE team on 26 June 2013 at ACDI/VOCA’s HQ in Washington D.C. The meeting was presided over by Jacob Gray, deputy director for SNAP, and Maria Bina Palmisano, M&E director for ACDI/VOCA. Sia Iscandari, program director (Freetown), Niry Ramasinjatovo, regional M&E support, Georgia Sheehy-Beans, CoP, and Joseph Marton, director of Moneval Solutions, joined via a conference call. The MTE team held further discussions on the work plan and evaluation questions with the SNAP team in Freetown. The discussions were instrumental in clarifying the SOW and ACDI/VOCA’s expectations of the MTE. The key points are highlighted below: Key questions for ACDI/VOCA are: Question 1: What would CG members like to see happen to the groups after the program ends? How can we best help achieve this? What do these groups see themselves becoming? Do these groups plan to engage in other programs/activities following SNAP? In what areas could SNAP provide additional support? SNAP aims to train the HPs to become full-time staff and then continue outreach, but also wants the staff to be inducted into a system to continue working in that same structure. A “sensitization” program is also a useful exit strategy (as shared by Ange). Question 2: Why have the full recommendations of the gender assessment not been implemented? What opportunities should we work on to make this happen? SNAP aims to promote gender via the training manuals of, for example, FFSs, VSLAs, and so on. However, the window of opportunity to integrate gender into the FFS manual has passed since the training is already underway. The answer to this question is rather open-ended, and can perhaps be attributed to a lack of gender prioritization amongst the plethora of other activities. Which gender activities can be implemented at this time? Which ones are no longer feasible, and why not? Is it a SNAP MTE – Final Report Moneval Solutions Ltd Confidential Page 86 staffing issue or a lack of prioritization? Overall, what are the challenges and what can SNAP do improve the integration of gender into its programs? Expectations of the MTE – ACDI/VOCA also expects this MTE to be an opportunity to gauge the relationship with its partners in the field and how SNAP is working with partners both programmatically and strategically, i.e. are these the most suitable activities to reach the targets? Is the tracking of the indicators being conducted properly? How is the program integration? Is health messaging being integrated? Are communities adopting agriculture practices? As SNAP has two more years left to run, ACDI/VOCA believes it is an opportune time to assess how adjustments to current activities, or initiating new ones, will be essential to better meet the objectives and reach the indicator targets. As per the meeting report drafted by its staff, ACDI/VOCA suggests “a 360-degree perspective of the program, both programmatically and operationally, will be extremely helpful.” Another expectation of ACDI/VOCA from the MTE is the entry/exit criteria for food distribution. Overall, the evaluation will assess whether the food commodities are being used as suggested in the SNAP proposal, and if they are managed as per USAID’s requirements. Desk review – More than 170 documents were provided for the desk review, and from these ACDI/VOCA highlighted the following for the MTE team to focus on. The team also reviewed other documents in addition to those listed below.  Review the proposal  Review the FY13 IPTT/PMP – the most updated version of the indicator performance and definitions  Review the PREP narratives to inform on how the program design has been altered  Focus on the FY13 PREP narrative, IPTT, PMP, low-performing indicators sheet, and question and answers to USAID questions (attached here again for your reference)  Review the ARR narratives, as these will provide information on the achievements, challenges, and successes of each year  Review the DQA report compiled last year by USAID  Review the quarterly reports Control groups – The use of control groups was removed from the MTE as a similar task has already been carried out by SNAP (Freetown). However, the questions in the SOW that relate to control groups will remain, and the question will be asked of SNAP beneficiaries as they may have useful insights into the non-SNAP population. VSLA groups – The VSLA coordinator recruited in January 2013 is currently mapping the communities interested in savings activities. This component is at an early stage, and it was decided to remove it from the MTE. Gender strategy – ACDI/VOCA has recently developed a strategic plan for gender; this was shared with the MTE team and included in the desk review. SNAP MTE – Final Report Moneval Solutions Ltd Confidential Page 87 Country work plan – The work plan was developed by the SNAP team in Sierra Leone and discussed with the MTE team, with the aim of including as many meetings as possible. The final work plan includes 71 meetings (including 15 community meetings that are expected to be short). It is the intention for the team to cover the maximum number of FGDs, KIIs, site visits, and other field data collection activities detailed in the plan. However, as with all best-laid plans, unforeseen events, both human and natural, often can upset them. Whilst the plan has been designed to cover the full range of stakeholders and partners involved with SNAP, we recognize that real constraints can intervene and are beyond our control. The country work plan reflects our best intentions to capture appropriate and relevant information on which to base the evaluation findings and conclusions, draw lessons, and develop sound recommendations. These collectively will add value to program management as it implements activities in what is undoubtedly a hugely challenging environment. Qualitative survey – The plan is to gather this information through additional interviews with individuals selected by SNAP. The data will be included in the overall analysis, findings, and recommendations. Other points:  The hope is that after the evaluation, the HQ team can submit a program modification to FFP based on the outcome/findings of the MTE  There may be a link-up between SNAP program graduates and the existing farmers’ associations SNAP MTE – Final Report Moneval Solutions Ltd Confidential Page 88 12 Annex 2: Evaluation Plan and Schedule The evaluation plan and schedule was based on a 19 June start date, with the country visit, data collection, and presentation of the preliminary findings being carried out between 30 June and 19 July. The draft evaluation report was submitted on 09 August, and the final report on 30 August. The work plan was divided into the following three phases: Phase 1 Preparation: 19 June – 02 July  Draft evaluation questions and Guides: 01 July  Updated evaluation questions and guides: 03 July  Inception report and final evaluation questions and guides: 05 July Phase 2 Data collection and preliminary analysis: 04 July – 19 July  Fieldwork: 04 July – 17 July  Data compilation and analysis: 18 July  Presentation of preliminary findings: 19 July  MTE Team depart Sierra Leone: 19 July Phase 3 Interpretation and Reporting:  Draft report: 09 August  ACDI VOCA and SNAP to review and provide feedback: 12 – 23 August  Final report and PowerPoint presentation: 30 August 12.1 Overall Work Plan SNAP MTE – Final Report Moneval Solutions Ltd Confidential Page 89 12.2 Country Work Plan 12.3 Fieldwork Plan SNAP MTE – Final Report Moneval Solutions Ltd Confidential Page 90 12.4 Logistical Arrangements: Accommodation for the consultants in Freetown and the four districts of intervention (Bombali, Kailahun, Koinadugu, and Tonkolili) will be arranged by SNAP staff, although payment and expenses will be covered by Moneval Solutions in line with the contract agreement. SNAP will ensure that vehicles and drivers are provided for the field visits. Upon arrival in Freetown, the consultants will be provided with a cell phone or SIM card each. SNAP MTE – Final Report Moneval Solutions Ltd Confidential Page 91 13 Annex 3: Composition of The Team The evaluation team consists of two sector experts, with backstopping and quality assurance support from Moneval Solutions. The table below summarizes the profiles, roles, and responsibilities of the team members, and their credentials for undertaking the evaluation. Expert Title Role and Responsibilities Ange Tingbo Agriculture and livelihoods expert Team leader; responsible for SO1 and cross-cutting themes Daphyne Williams Health and nutrition expert Team member; responsible for SO2 and cross-cutting themes Paul Schoen Agricultural economist Backstopping and quality assurance Ange Tingbo: Team leader – agriculture and livelihoods Ange has strong management knowledge of food security programs, strategic planning, risk reduction, and community development, together with Title II food commodities management knowledge. He has worked for Catholic Relief Services, the American Red Cross, and Africare, with progressive responsibilities over the years, in close collaboration with FFP/USAID MYAP and Title II activities. He is therefore extremely familiar with the relevant principles and guidelines. His work has evolved around the intricacies of food security and livelihood interventions, i.e. agriculture, health/nutrition/HIV/AIDS, microcredit (including environmental issues), community mobilization, literacy, disaster preparedness response and humanitarian assistance, and community capacity building and strengthening. He has proven leadership ability in program design in both stable and post-conflict states. Activities have ranged from agriculture, water sanitation, MCH/nutrition, HIV/AIDS, microcredit, literacy/numeracy, and education. Ange has previously worked in Sierra Leone and has supervised food security activities in a number of other countries, including Benin, Burkina Faso, Niger, Mali, Ghana, Chad, Rwanda, Burundi, Uganda, Malawi, Mozambique, Democratic Republic of Congo, Liberia, Togo, Eritrea, and Vietnam. He has also backstopped Africare’s MYAP in Uganda, where ACDI/VOCA was the lead consortium partner, and was previously interim country director for Africare in Sierra Leone for four months. Daphyne Williams: Team member – health and nutrition Daphyne is a skilled public health expert with more than 12 years of experience with Title II programming in the technical areas of health, nutrition, HIV, and reproductive health, and in cross￾cutting areas including capacity development, community engagement, gender integration, program management, design, technical advice, and M&E. She has worked on public health programs in more than a dozen countries across sub-Saharan African and Asia, most recently covering six countries in SNAP MTE – Final Report Moneval Solutions Ltd Confidential Page 92 East Africa. Daphyne was also a member of the team that recently completed the MTE of ACDI/VOCA’s LAUNCH program in Liberia, where she was responsible for health and nutrition assessment and involved with cross-cutting issues relating to gender. Daphyne recently led the development of the nutrition component of the Title II DFAP proposal for Zimbabwe that included focusing on mitigating negative nutrition impacts (both under- and over￾nutrition), increasing male involvement, improving cross-sectoral integration, and the integration of the CG model. She also served as a regional technical advisor for the Catholic Relief Services (CRS) Title II programs in East Africa for two-and-a-half years, providing technical assistance, guidance, and local capacity building for holistic public health programming. In addition, Daphyne has previously conducted evaluations and assessments in Kenya, Tanzania, Uganda, Ethiopia, Ghana, Bangladesh, and Zimbabwe. Daphyne is fluent in English and Pidgin, and has basic French language skills. Her regional experience includes East, West, and Southern Africa, and South-East and South Asia. Paul Schoen: Backstopping and quality assurance Paul Schoen will provide backstopping and quality assurance for the duration of the assignment. Paul is a highly experienced M&E specialist, and an economist with over 22 years of international experience in development programs (Africa, Asia, Pacific, Eastern Europe and the Balkans, and Latin America). He combines his evaluation skills with socio-economic analysis, program identification, program preparation, and appraisal. He has led missions and participated in studies and economic advisory work for the E.U., Tacis, Phare, UNDP, DFID (British government), IFC (World Bank), and Dutch Aid-Senter. Sectoral areas covered have included livelihood development reviews, gender and empowerment, and agricultural development, together with health, education, financial systems analysis, and institutions. Paul will ensure appropriate standards are met in undertaking the work SNAP MTE – Final Report Moneval Solutions Ltd Confidential Page 93 14 Annex 4: Evaluation Methods and Tools A key aspect of our approach is ensuring that the evaluation is conducted in an objective, impartial, open, and participatory manner, based on empirically verified evidence that is valid and reliable. The team will apply their subjective analysis of the data and provide professional opinions. The structured input and management of evaluation data will allow the team to (i) attain an overview of data collected, (ii) identify trends or patterns within the data collected that relate to the evaluation criteria, and (iii) check for and identify any information gaps. It will also aim to:  Assess the relevance and attainability of the objectives: Are program activities addressing food insecurity and improving nutrition? Are the project objectives still relevant? (questions related to program outcomes)  Assess what project activities are currently working to achieve program goals, and establish what is not working: What changes could be made to current activities to enhance their contribution to the fulfillment of objectives? (questions related to program design and implementation)  Analyze how efficiently project planning and implementation are carried out: Are the management arrangements of the project such that efficient execution and implementation is guaranteed? (questions related to efficiency in program management coordination and operations)  Assess the direct impact prospects of the project in terms of its overall goals: What planned and unplanned positive or negative impacts might the project achieve? What are the prospects of inducing behavioral changes in terms of the adoption of technology and changes in the care given by mothers (questions related to program effectiveness)  Determine whether gender issues have been adequately incorporated in the implementation of the program  Analyze the likelihood of better practices being sustainable with regards the adoption of improved farming and post-harvest handling techniques, knowledge and adoption of good nutritional practices, maintaining the capacity and services of local administrative bodies or local government structures, and private sector arrangements in support of improved livelihoods  Assess how, and to what extent, the varying components of SNAP are sufficiently integrated and mutually supportive so that the activities as a whole are seen as integrated. This is an important element in examining the ways in which both design and implementation of a project lead to a successful outcome for the key objectives  Asses the relationships between stakeholders (both positive and negative) to ascertain the effectiveness of the project’s impact. Understanding these relationships, and building on them, could influence and strengthen the opportunities that allow for sustainable livelihoods beyond the lifetime of project activities The following sections detail the MTE methodology for the evaluation; the key phases in undertaking the assignment are preparation, data collection, analysis, and interpretation and reporting. SNAP MTE – Final Report Moneval Solutions Ltd Confidential Page 94 14.1 Preparation The purpose of this phase is to discuss the approach, clarify any issues relating to the SOW, and finalize the work plan. This is aimed at gaining a shared understanding and agreement on the scope, methodologies, and timelines of the evaluation within the team, and between the team and ACDI/VOCA and SNAP management. Key activities in the preparation phase include: Desk review of the following documents:  SNAP Approved Proposal and Budget – by ACDI VOCA, January 15, 2010  Performance Management Plan (PMP - by ACDI VOCA, January 15, 2010  Indicators Performance Tracking Table (IPTT) by ACDI VOCA, January 15, 2010  Targeted Chiefdom Profiles and Rationale for Selection by ACDI VOCA, Jan. 15, 2010  SNAP Cross-Cutting Themes -Annual Results Report (ARR) – by Bernie Runnebaum, Acting Chief Of Party - Nov. 5, 2012  Responses to Issues Letter of the ARR by ACDI VOCA FY 2012  Results Framework - by ACDI VOCA, January 15, 2010  Pipeline and Resources Estimate Proposal (PREP)- M. Brazill, June 12, 2013  Responses to Issues Letter of the PREP ACDI VOCA, No date  Assessment of Food Insecurity in Sierra Leone by ACDI VOCA, January 15, 2010  Gender Integration Action Plan by ACDI VOCA/Freetown FY 12  Inland Transport Material - by ACDI VOCA, January 15, 2010  AER - by ACDI VOCA, January 15, 2010FY13 IPTT/PMP  PREP narratives  ARR narratives  DQA report  Quarterly Reports  Briefing session with ACDI/VOCA and SNAP management;  Discussions with the SNAP team in Sierra Leone  Finalization of the evaluation questions and guides  Finalization of the country work plan SNAP MTE – Final Report Moneval Solutions Ltd Confidential Page 95  Identification of the program staff that will participate in the evaluation and develop the country work plan. Pre Country Interview – The MTE will meet and hold a teleconference call with the following people prior to departure and writing the inception report/evaluation plan:  Jacob Gray, deputy director, community development  Maria Bina Palmisano, M&E coordinator  Georgia Sheehy-Beans, CoP  Niry Ramasinjatovo, regional M&E support  Emily Byron, previous director/coordinator for the program under food security  Seraphine R. Olanya, DCoP 14.2 Data Collection and Analysis There will be three main levels of data collection: 1) the key players of the SNAP consortium, 2) the beneficiaries, 3) other agencies and government entities. The analysis of the information collected will be based on facts and cross-checked between the different sources. Data collection and preliminary analysis – The data collection activities will consist of a number of FGDs, KIIs, and site visits. The MTE team will meet with 647 beneficiaries and participants through the FGDs, conduct 160 KIIs, and visit 29 project locations. FGDs:  For SO1 these are: 109 beneficiaries and participants, consisting of four groups of neighborhood mothers (55 people), five LM groups (47 people), and two Men as Partners groups (7 people)  For SO2 these are: 538 beneficiaries and participants, consisting of ten groups from FFS/communities (360 people), six groups representing vocational education services (58 people), and ten groups representing informal VSLAs (120 people) KIIs:  For SO1 these are: 26 staff members and 57 from government  For SO2 these are: 43 staff members and 34 from government Site visits:  Villages: Bunumbu, Levuma, Mamaka, Karamanka, Kondembaia, Kafogo, Mandy Tawahuh, Ndgba, Bumbuna  PHUs and government: Kailahun DHMT, Kailahun DC, Levuma PHU, Tonkolili DHMT, Mamaka PHU, Kaponkie MCHP, Bumbuna Chiefdom Clinic, Kamabaio, Koinadugu DC, Bombali DHMT, Kondembaia PHU The following SNAP groups will be included in data collection phase:  Mother Care Groups (CGs)  Peripheral Health Unit (PHU)  District Health Medical Team (DHMT) SNAP MTE – Final Report Moneval Solutions Ltd Confidential Page 96  Farmer Field School (FFS)  Vocational and Literacy  Ministry of Agriculture and Food Security (MoA)  Community  Chiefdom  District Council (DC)  Sustainable Nutrition and Agricultural Promotion team (SNAP) 14.3 Data Collection Tools The evaluation team will develop evaluation questions and guides for use during FGDs and KIIs. The questionnaire will be conducted according to the audience, e.g. management team, beneficiaries of SO1 and SO2 activities in general, farmers, women’s groups, local authorities, food commodities team, vocational training group, and so on. 14.3.1 Multi-criteria and Cross-checking Analysis Data collection tools will include studying documents, interviews, and visual examinations during field visits, and the process will include cross-checking results from different methods to confirm their validity and identify any contradictions between information sources. A further analysis of the degree of reliability of these information sources will then be carried out to validate the most objective conclusion. In this context, baseline data will prove important and, where possible, will be collected from government sources. 14.3.2 Forms and Guides The field data collection will be carried out during interviews with SNAP staff (ACDI, IMC, and OICI) at the Freetown office and the district offices. The second level will involve interviews with the beneficiaries, such as mother CGs, PHU agents, FFS participants, FFS trainers, other district authorities, and so on. These informants will not be given any forms to fill out, but the consultants will use the questionnaire to lead the discussion. The discussion will not involve only open-ended questions, but the consultants will use the questionnaire as a guide to delve deeper into areas of interest. 14.4 Key Informants and Agencies The MTE team will seek to interview and hold discussions with the officials from various agencies/organizations. The consultants will meet with key players from the three institutions of the consortium: ACDI/VOCA, IMC, and OICI. The meetings will be with:  SO1 and SO2 program managers and their assistants (for activities implementation)  CoP, DCoP, and program director (for operations and integration and funding mechanisms)  Commodity manager, commodity support coordinator, and CTC (for Title II food commodities management and monetization) SNAP MTE – Final Report Moneval Solutions Ltd Confidential Page 97 Additional institutions to meet include government departments (health, youth education, and MoA, all at district level) and the DC members. As suggested above, there will be a meeting with a MoH official in Freetown. This does not exclude meeting with the DHMT in each district in the field. Other key informants will be the beneficiaries of the activities: farmers in FFS groups, mother CGs, PHU agents, and youths in vocational training. Given that the meeting with USAID in Freetown is no longer happening prior to the MTE field visit, it is anticipated that the feedback from the mission’s authorities, whom the team is now meeting on July 17, may shed some light on the overall analysis and recommendations. 14.5 Interpretation and Reporting This phase will focus on analyzing the data collected from the stakeholder discussions, country visit, program documents, and other sources. The team members will work closely to arrive at a common interpretation of findings, conclusions, and draft recommendations. The preliminary findings will be presented in country, with the full report developed off site. The key findings and recommendations will be documented in line with the report structure defined in section 4.7. This will include the program’s progress towards meeting objectives and targets, lessons learned, and actionable recommendations. 14.6 Backstopping and Quality Assurance Paul Schoen will provide backstopping and quality assurance for the duration of the evaluation. This will include reviewing, analyzing, and approving all deliverables before they are submitted to ACDI/VOCA. Paul will be involved in all phases of the evaluation, will provide guidance and support to the team in terms of approach, and ensure delivery compliance. He will maintain close supervision of mission progress through regular QA activities, as detailed below. Pre-fieldwork in Freetown:  Team support – to assess progress, address issues, QA input, and so on  Conference call with the team leader and team member daily when possible  SNAP management – to gather feedback on any issues or potential issues and ensure the team is aligned with expectations  Regular calls with CoP, M&E experts, and other key resources Fieldwork:  Regular contact with the team while they are in the field  Regular contact with SNAP management Post-fieldwork in Freetown:  Review the preliminary findings and presentation  Ensure the findings and recommendations are sound  Ensure the presentation is of a high standard  Contact with SNAP management Reporting: SNAP MTE – Final Report Moneval Solutions Ltd Confidential Page 98  Review the draft and final report  Ensure the draft and final report meet the expectations and standards of ACDI/VOCA and USAID  Contact with SNAP management 14.7 Evaluation Questions The evaluation questions in this section will form the basis for the data collection and guide the MTE team as they collect information. The questions detailed in the SOW form the core of the questions and additional questions were identified during the preparation phase. The final set of evaluation questions was reviewed by the SNAP team at HQ and in Freetown. The table below provides a summary of the number of questions allocated to each category. Category SO 1 SO2 SO1 1& SO2 Resiliency to shocks Productive Youth Gender Equity Environmental Stewardship Total Progress towards Program Outcomes 4 4 2 10 Program Design and Implementation 2 1 3 1 1 8 Additional questions from the MTE team 1 10 1 2 14 Program Management, Coordination, and Operations 6 6 Additional questions from the MTE team 4 10 1 1 1 17 Monitoring and Evaluation 5 5 Total 6 10 26 10 1 3 4 60 14.7.1 Progress Towards Program Outcomes Data Collection Methodology: FGD/KII/Interviews with SNAP Staff/Review of FFS Training Sessions/CG Training Sessions/Vocational Skills Training Sessions. Questions SO/ Cross Cutting Are we seeing any increase in usage of preventative and curative services by pregnant and lactating mothers? Why or why not? If YES, is it likely to be sustained without program resources? Why or why not? SO1 Are we seeing any increase in access to nutritional foods in target communities? SO1 SNAP MTE – Final Report Moneval Solutions Ltd Confidential Page 99 Why or why not? If YES, will it likely be sustained without program resources? Why or why not? Are we seeing any increase in consumption of nutritional foods in target communities? Why or Why not? If YES, will it likely be sustained without program resources? Why or why not? SO1 Are we seeing any transfer of new knowledge, skills, and practice/ behavior change from program PM2A participants to mothers NOT enrolled in PM2A program? Why or why not? SO1 Are we seeing any increase in agricultural yields in target communities? Why or why not? If Yes, will it be sustained without program resources? Why or why not? SO2 Are we seeing any transfer of new knowledge, practices, and skills from program FFS participant farmers to farmers NOT participating in FFS? Why or why not? SO2 Are farmers adopting the new technologies and practices imparted by the project? Why or why not? If YES, will it be sustained without program resources? Why or why not? SO2 Are we seeing any increase in income for SNAP beneficiary households? Why or why not? If YES, will it be sustained without program resources? Why or why not? SO2 If we are seeing any increased income, is it being used to strengthen household health? If YES, will it be sustained without program resources? Why or why not? SO1/SO2 Has new knowledge translated into adoption of new practices and recommended behaviors? Why or why not? What factors are contributing to this? If YES, will it be sustained without program resources? Why or why not? SO1/SO2 14.7.2 Program Design and Implementation Data Collection Methodology: FGD/KII/Interviews with SNAP Staff/Review of FFS Training Sessions/CG Training Sessions/Vocational Skills Training Sessions Key Questions SO/ Cross Cutting Program Design: Which strategies and activities have been more effective so far and why? Are we targeting the right beneficiaries? Are the right beneficiaries being enrolled/enrolling in the program? Why or Why not? SO1/SO2 SNAP MTE – Final Report Moneval Solutions Ltd Confidential Page 100 Implementation: What would Care Group members like to see happen to the Care Groups after the program ends? How can we best help to achieve this? SO1 Implementation: Do PHUs show long-term sustainable capacity to provide services to pregnant and lactating mothers? Why or why not? SO1 Implementation: Do producer groups show the capacity to sustain continued, increased production of cash-crops and nutritious foods? Why or why not? Dimensions should include:  Knowledge of improved production and post-harvest handling skills and technologies  Use of improved production and post-harvest handling skills and technologies  Good practice in business management of producer groups  Established relationships with input suppliers, buyers, and markets  Access to technical support related to agriculture without program assistance SO2 Implementation: What explains the low achievement rates on the following indicators:  Use of improved storage  Women’s organizations/associations assisted  New technologies or management practices made available  Yield of some crops  Crop sales  Youth receiving vocational training  Community-assets projects completed  New targeted beneficiary reached  SO2  SO2  SO2  SO2  SO2  SO2  SO2  SO2 SNAP MTE – Final Report Moneval Solutions Ltd Confidential Page 101  Health facilities with functional bio-sand filters  Outreach to PLW  SO1  SO1 Why has full recommendations of the gender assessment recommendations not yet been implemented? What opportunities should we work on to make this happen? Gender Do we foresee any issue in addressing FFP Environmental Compliance requirements? Environment WASH: Do we foresee any issue to address or any opportunity to implement the project’s WASH strategy? SO1/ SO2 14.7.3 Additional Questions for Program Design and Implementation Data Collection Methodology: FGDs and KIIs/Review of FFS Training Sessions/CG Training Sessions/Vocational Skills Training Sessions. Agriculture and Livelihoods (SO2) SO/ Cross Cutting Impact: Are communities better prepared now than they were before 2010 for potential shocks? If so, how so? Resilience Resilience/ Impact/ Sustainability: Which SNAP activities have been most effective at mitigating risk and why? Resilience Impact/Effectiveness: Which activities have been the least effective and why? Resilience Impact/Sustainability: What have community members done to improve disaster preparedness, mitigation of impacts and response to shocks? Resilience Impact: How have household coping strategies changed since 2010? Resilience Impact: SNAP MTE – Final Report Moneval Solutions Ltd Confidential Page 102 Were there any infrastructures i.e. storage, classrooms, roads, etc. rehabilitated? Impact/Sustainability: Did infrastructure rehabilitated or constructed by the program improve community or household resiliency and food security? If so, how? If not, why not? Resilience Impact/Sustainability: How has natural resource management practices changed (if at all) over the life of the program? Resilience Impact: How could this contribute to improved health/nutrition and livelihood outcomes? Resilience Impact: Do both men and women from different types of households have access to and control over the resources required to participate in resilience activities? Do they benefit equally from these activities? What types of decisions do women and men make with regards to resilience strategies? Gender and Resilience Effectiveness: To what extent has the management of community assets been effective? If it has been effective, what factors have contributed to that? If not, why not? SO2 Resilience Effectiveness: What effect has leadership and management of community assets had on community capacity to respond to shocks and on community resiliency? Resilience Impact: Do community members have the technical and managerial capacity, the material and human resources, sufficient motivation and the partnerships required to maintain these impacts after the program ends? Resilience 14.7.4 Program Management, Coordination, and Operations Data Collection Methodology: KII/Interviews with SNAP Staff/Discussion with Partner Organizations/Observation of Commodity Direct Distributions/Direct Distribution Loading/Unloading at Warehouse. Question SO/ Cross￾Cutting SNAP MTE – Final Report Moneval Solutions Ltd Confidential Page 103 Management – Coordination Planning Efficiency (includes discussion with Partner Organizations):  How effective and efficient is coordination between the two SO activities. What has worked well? What has not?  Dimensions under consideration should include at a minimum:  Program Planning and Activity Coordination between HQ (Washington) and the Head offices (Freetown)  Program Planning and Activity Coordination between SO1 and SO2 Head Offices (Freetown)  Program Planning and Activity Coordination between Head Office and Field Offices  Program Planning and Activity Coordination between SO1 and SO2 teams in the field  Program Implementation of SO1 and SO2 Integrated Activities SO1 and SO2 Management – Integration/ Efficiency/ Effectiveness (includes discussion with Partner Organizations): How effective and efficient is integration between the two SO activities. What is working well? What is not? SO1 and SO2 Management – Efficiency/ Effectiveness How have implementation related problems been identified, analyzed and solved? What has worked well in terms of problem solving? What has not worked well? SO1 and SO2 Management – Efficiency Is SNAP staff qualified and knowledgeable of the purpose and methods used in the program and equipped to properly implement the activities? SO1 and SO2 Management – Efficiency Does the management structure/culture support staff spending time creating and sharing knowledge (i.e., documentation, report outs in meetings, brown bags, mentoring). Do job descriptions identify clear knowledge sharing activities that should take place? SO1 and SO2 Management – Efficiency What are areas of excellence, and areas of improvement in terms of donor/awardee relationship including at a minimum: SO1 and SO2 SNAP MTE – Final Report Moneval Solutions Ltd Confidential Page 104  Reporting  Communication  Donor/Mission/Awardee Coordination 14.7.5 Additional Questions for Program Management, Coordination, and Operations Data Collection Methodology: FGD/KII/SNAP Staff/ Observation of Commodity Direct Distributions/ Direct Distribution Loading/Unloading at Warehouse. Evaluation Questions SO/ Cross￾Cutting Program Management and Integration: Management: What key best practices can be identified which have allowed the program to succeed? SO1 and SO2 Management/ Relevance: What are the key lessons learned (successes or failures) of the program? SO1 and SO2 Impact: What is the impact of the program on overall gender equity? SO1 and SO2 Gender Impact: What is the impact of the program on overall Youth? SO1 and SO2 Youth Impact: How has the SNAP program impacted the environment in target communities? SO1 and SO2 Environment Relevance and Impact: What improvements in local governance resulted from this program i.e. local authorities and communities relationship, less road taxes on farm produce, etc.? SO1 and SO2 Impact: How did the program respond to community concerns? SO1 and SO2 Sustainability: Do communities feel that they had a voice in program management? SO1 and SO2 SNAP MTE – Final Report Moneval Solutions Ltd Confidential Page 105 Efficiency and Effectiveness: Was the program responsive to their needs and requests? What mechanisms were established for the community to express their concerns? SO1 and SO2 Management – Efficiency Are the food distribution mechanisms well understood by all the parties involved - partners and beneficiaries? i.e. food requests, approval, waybills, inland transportation, primary and secondary warehouses, etc. SO2 Management – efficiency and effectiveness Is the inland transportation system adequate? if not what are the main challenges? SO2 Management – efficiency and effectiveness What are the main challenges for food distribution for participating mothers? SO2 Management – efficiency Are there end-use checking and physical inventories? If yes, how often? SO2 Management – efficiency/ effectiveness/ impact What are the main challenges? Any suggestions? SO1 and SO2 14.7.6 Monitoring and Evaluation Data Collection Methodology: SNAP Staff/Observation Questions How well is the output monitoring system functioning? What is working well? What is not? Where, if any, are their bottlenecks in this system? Does the output monitoring have well developed data collection forms, data flow diagram, defined responsibilities for each partner and/or sector, aggregation levels, and a clear link showing how the information will be aggregated and reported onto the Annual Results Report and SAPQ? Does the monitoring system measure outcome level changes? Does the system use appropriate methods and tools? How often it measures outcome? Is the system too burdensome to the M&E team? Does the outcome measurement system also investigate and present reasons behind the successes and/ or under achievements? How useful are these findings to the users? How this can be improved? How is the output and outcome monitoring systems and monitoring activities linked/coordinated SNAP MTE – Final Report Moneval Solutions Ltd Confidential Page 106 between SO1 and SO2 M&E Teams? Between the field and head office? Between head office and DC office? What is working well? What is not working well? 14.8 Evaluation Guides 14.8.1 Strategic Objective 1 This sections details guides for the following Focus Group Discussions and Key Informant Interviews:  PHUs and DHMTs  Lead Mothers and Care Group mothers  Community Based Meetings (Chiefs and Village Leaders)  Field Staff  Men as Partners 14.8.1.1 PHUs and DHMTs PHU visits will include a review of the inputs, logs (if appropriate) and tour of the facility Questions Which strategies and activities have been more effective so far and why? What trainings, inputs, interactions has this PHU had with SNAP? Thinking of the mothers in the care groups, which activities, education has been most effective? Why do you think that is so? Of the activities that SNAP does, what is most important? Why? Are we seeing any reported increase in usage of preventative and curative services by pregnant and lactating mothers? Why or why not? If YES, is it likely to be sustained without program resources? Why or why not? How is this information tracked? Reported to SNAP? Do PHUs show long-term sustainable capacity to provide services to pregnant and lactating mothers? Why or why not? Of the activities/inputs and trainings by SNAP, which will continue at the end of the program? Which will not continue? Why or why not? How often does SNAP staff visit this PHU? What happens when they visit? How many staff that were trained by SNAP are still here? After attending SNAP trainings, did they SNAP MTE – Final Report Moneval Solutions Ltd Confidential Page 107 cascade the training to other staff? How does the PHU monitor the cascade of training? How does SNAP verify the cascade of trainings? What barriers does the PHU face in providing services to PLW? How has the PHU planned to address them? Are we seeing any transfer of new knowledge, skills, and practice/behavior change from program PM2A participants to mothers NOT enrolled in PM2A program? Why or why not? Are we seeing reported increase in male engagement in MNCH health? Why or why not? If Yes, is this likely to be sustained? Are we seeing reported increase in male engagement in MNCH health? Why or why not? If Yes, is this likely to be sustained? Do you see changes in the behavior of women in SNAP? If so what? How can SNAP help men to be more involved in the care of PLW? What does the PHU do to engage men? What would Care Group members like to see happen to the Care Groups after the program ends? How can we best help to achieve this? When SNAP ends, will the care groups continue in your community? How/Why? What challenges will they face in continuing? 14.8.1.2 Lead Mothers and Care Group mothers Core Tasks:  Review care group cards and lead mother records.  Introductions - Name, purpose of the evaluation, what I will do with the information. Introduction - Participant name and age of child (or month of pregnancy), Length as beneficiary Questions Which strategies and activities have been more effective so far and why? What does SNAP do? Why is this important? Of the activities that SNAP does, what is most important? Why? How has SNAP helped you and your children? Are we seeing any reported increase in usage of preventative and curative services by pregnant and lactating mothers? Why or why not? If YES, is it likely to be sustained without program resources? Why or why not? SNAP MTE – Final Report Moneval Solutions Ltd Confidential Page 108 1. In your last pregnancy, how many of you visited the clinic? Why did you visit the clinic? If you did not visit, why? 2. For your child under two years, have you taken them to the clinic? For growth monitoring? For vaccinations? 3. Is it important to take your child under two to the clinic? When is it important? Why is it important? 4. What is EBF? How long should you practice EBF? 5. What challenges do you face when trying to practice EBF? 6. How has SNAP helped you overcome those challenges? (this will also hopefully get at some of the gender changes as a result of the program, primarily the cultural taboos that inhibit optimal BF). Are we seeing any reported increase in consumption of nutritional foods in target communities? Why or Why not? If YES, will it likely be sustained without program resources? Why or why not? What are nutritious foods? If you were going to prepare a nutritious meal what is important to include? Where did you learn this? Why are nutritious foods important for CU2? PLW? How do you prepare CSB? Why is CSB important for PLW and CU2? Have you participated in a cooking demonstration for CSB? Do you know how to prepare a nutritious meal for PLW and CU5? Where did you learn this? What is important to include in nutritious meals for PLW and CU5? Does your husband support your efforts to make more nutritious meals? Does your mother-in-law? If so how (for both husband and mother-in-law)? Are we seeing any reported increase in access to nutritional foods in target communities? Why or why not? If YES, will it likely be sustained without program resources? Why or why not? Where do you find nutritious foods? Are there more nutritious foods available now? Where did they come from? Are we seeing any transfer of new knowledge, skills, and practice/behavior change from program PM2A participants to mothers NOT enrolled in PM2A program? Why or why not? Are we seeing reported increase in male engagement in MNCH health? Why or why not? If Yes, is this likely to be sustained? Are we seeing reported increase in male engagement in MNCH health? Why or why not? If Yes, is this likely to be sustained? Do you share the things you learn in the group with others? Who are they? Why do you share? What do you share? Do you think they changed their behavior as a result? Do you share what you learn with your husband? Your mother-in-law (will need the help of SL staff SNAP MTE – Final Report Moneval Solutions Ltd Confidential Page 109 for the best phrasing). What kinds of things do you share? Why? What happens when you share the information? Is anyone else in your household participating in a SNAP program? If so what? Do they share with you, what they learn from that activity? What would Care Group members like to see happen to the Care Groups after the program ends? How can we best help to achieve this? What does it mean to graduate from the program? What have you learned from the program that you will continue? What can SNAP do now to help you continue those behaviors? What can SNAP do to improve care groups for mothers who are not yet members? What do you plan to do when you have graduated from SNAP? Are these things different than before? Lead Mothers: When you have graduated from food benefits, will you continue participating as Lead Mother? Why? If not, why not? Which strategies and activities have been more effective so far and why? Do you have hand-washing facilities at home? Please describe them. Do you have a tippy tap? Who uses the hand-washing facilities? Is this different than before SNAP started? MCG members ONLY: What do Lead Mothers do well? Is there a particular module that has really helped you benefit? If so, which? How have you benefited? MCG members ONLY: What could Lead Mothers do better? Have you participated in MCG competitions? Why? Why do you participate in MCG? Lead Mothers: Do you face challenges in your task as Lead Mother? If so, what? What has SNAP done to help you address those challenges? What could snap do to help address those challenges? Do you have specific challenges with any of the modules? What makes you most proud about being a Lead Mother? 14.8.1.3 Community Based Meetings (Chiefs and Village Leaders) Core Tasks:  Review community involvement with respect to services and access to nutritional foods.  Introductions - Name, purpose of the evaluation, what I will do with the information. SNAP MTE – Final Report Moneval Solutions Ltd Confidential Page 110 Introduction - Participant name and age of child (or month of pregnancy), Length as beneficiary. Questions Are we seeing any increase in usage of preventative and curative services by pregnant and lactating mothers? Why or why not? If YES, is it likely to be sustained without program resources? Why or why not? Are we targeting the right beneficiaries? Are the right beneficiaries being enrolled/enrolling in the program? Why or Why not? What is the purpose of SNAP? What does SNAP do? Why is it important? (If they say nutrition) How does better nutrition for PLW and CU2 help your community? Are the women in the program correct for the goal of the program? Are we seeing any increase in access to nutritional foods in target communities? Why or why not? If YES, will it likely be sustained without program resources? Why or why not? Since SNAP has started, are there more nutritious foods in your community? What kinds of nutritious foods are now available that weren’t available before? Do you think these foods will be available after the program ends? Why? Has new knowledge translated into adoption of new practices and recommended behaviors? Why or why not? What factors are contributing to this? If YES, will it be sustained without program resources? Why or why not? What do you see women in SNAP doing now that they didn’t do before the program started? Do you see their men doing things differently than they did before? If so, what? In your home, are you doing anything differently that you learned from SNAP or a SNAP beneficiary? Which strategies and activities have been more effective so far and why? Of all of the SNAPS activities, what is having the greatest positive change in your community? Why? How do you know? Why has full recommendations of the gender assessment recommendations not yet been implemented? What opportunities should we work on to make this happen? How are men involved in SNAP PLW and CU2 programming? Is this important? Why? How could male involvement be increased? Have you seen negative results of men being involved? If so what? What would Care Group members like to see happen to the Care Groups after the program ends? How can we best help to achieve this? SNAP MTE – Final Report Moneval Solutions Ltd Confidential Page 111 When SNAP is finished, what activities will your community continue to use? How will these be used? What do you do as a community leader to support the improvement of the health of PLW and CU2? 14.8.1.4 KII Field Staff The review will consist of interviews with staff as well as observation. Additional questions may be asked during field visits and interactions with staff. Question How effective and efficient is coordination between the two SO activities. What has worked well? What has not? Dimensions under consideration should include at a minimum:  Describe the coordination between SO activities? What works well? What hasn’t?  Program Planning and Activity Coordination between HQ (Washington) and the Head offices (Freetown)  Program Planning and Activity Coordination between SO1 and SO2 Head Offices (Freetown)  Program Planning and Activity Coordination between Head Office and Field Offices  Program Planning and Activity Coordination between SO1 and SO2 teams in the field  Program Implementation of SO1 and SO2 Integrated Activities How have implementation related problems been identified, analyzed and solved? What has worked well in terms of problem solving? What has not worked well? How are implementation problems solved? What steps are involved? How do you know the problems have been addressed? Describe a time it has worked well Is SNAP staff qualified and knowledgeable of the purpose and methods used in the program and equipped to properly implement the activities? What health and nutrition trainings have you undergone? What is ENA? Why is nutrition critical for PLW and CU2? How do you explain this to community members? Does the management structure/culture support staff spending time creating and sharing knowledge (i.e., documentation, report outs in meetings, brown bags, mentoring). Do job descriptions identify clear knowledge sharing activities that should take place? How often are you engaged in learning activities? How do you report/share your lessons learned? Is this encouraged? How is it encouraged? SNAP MTE – Final Report Moneval Solutions Ltd Confidential Page 112 Are we seeing any increase in usage of preventative and curative services by pregnant and lactating mothers? Why or why not? If YES, is it likely to be sustained without program resources? Why or why not? Are we targeting the right beneficiaries? How do you know there have been increases? How is this information shared with management? The community? Are we seeing any increase in access to nutritional foods in target communities? Why or why not? If YES, will it likely be sustained without program resources? Why or why not? How do you know there is increased access? How is this information tracked? How is it reported back? Has new knowledge translated into adoption of new practices and recommended behaviors? Why or why not? What factors are contributing to this? If YES, will it be sustained without program resources? Why or why not? What do you see women in SNAP doing now that they didn’t do before the program started? Which strategies and activities have been more effective so far and why? Of all of the SNAPS activities, what is having the greatest positive change in your community? Why? How do you know? How do you share/report this information back? Why has full recommendations of the gender assessment recommendations not yet been implemented? What opportunities should we work on to make this happen? What are the main points of SNAP gender integration strategy? How is this reflected in your work (internal)? How are men involved in SNAP PLW and CU2 programming? Is this important? Why? How could male involvement be increased? Have you seen negative results of men being involved? If so what? What have you/SNAP done to address any negative results? 14.8.1.5 KII – Men as Partners The review will consist of interviews with beneficiaries as well as through observation with respect to men as partners in health changes. Additional questions may be asked during field visits and interactions with staff. Are we seeing any increase in usage of preventative and curative services by pregnant and lactating mothers? Why or why not? If YES, is it likely to be sustained without program resources? Why or why not? Are we targeting the right beneficiaries? Are the right beneficiaries being enrolled/enrolling in the program? Why or Why not? What is the purpose of SNAP? What does SNAP do? Why is it important? (If they say nutrition) How does better nutrition for PLW and CU2 help your community? SNAP MTE – Final Report Moneval Solutions Ltd Confidential Page 113 Are the women in the program the correct for the goal of the program? How does your participation in Men as Partners support the goal of the program? Are we seeing any increase in access to nutritional foods in target communities? Why or why not? If YES, will it likely be sustained without program resources? Why or why not? Since SNAP has started, are there more nutritious foods in your community? What kinds of nutritious foods are now available that weren’t available before? Do you think these foods will be available after the program ends? Why? Has new knowledge translated into adoption of new practices and recommended behaviors? Why or why not? What factors are contributing to this? If YES, will it be sustained without program resources? Why or why not? What do you see women in SNAP doing now that they didn’t do before the program started? What do you see the partners of these women doing now that they did not do before? Which strategies and activities have been more effective so far and why? Of all of the SNAPS activities, what is having the greatest positive change in your community? Why? How do you know? Are we seeing any transfer of new knowledge, skills, and practice/behavior change from program PM2A participants to mothers NOT enrolled in PM2A program? Why or why not? Do women in the community NOT in SNAP do any of the promoted behaviors in SNAP? If so what? Have you started doing anything SNAP promotes? If so what? Why? Why has full recommendations of the gender assessment recommendations not yet been implemented? What opportunities should we work on to make this happen? How are men involved in SNAP PLW and CU2 programming? Is this important? Why? How could male involvement be increased? Have you seen negative results of men being involved? If so what? What would Care Group members like to see happen to the Care Groups after the program ends? How can we best help to achieve this? When SNAP is finished, what activities will your community continue? How? SNAP MTE – Final Report Moneval Solutions Ltd Confidential Page 114 14.8.2 Strategic Objective 2 This sections details guides for the following Focus Group Discussions and Key Informant Interviews:  Farmer Field Schools  Vocational Training 14.8.2.1 Farmer Field Schools The FFS visits will involve participating farmers and the facilitators. This will include a visit to the farms to see what it’s been grown on demonstration plots. Which strategies and activities have been more effective so far and why? In particular: Since when have you been benefiting from the SNAP program? How did you hear about the program? Sensitization in the community? A friend? A spouse? What training, inputs, interactions have been effective with the facilitators? Are farmers adopting the new technologies and practices imparted by the SNAP program? Why or why not? If YES, will these practices continue do you think, without program resources? Why or why not? Which technologies are the most appropriate and being used? Why? Name a couple of those technologies. Are we seeing any increase in agricultural yields in target communities? Why or why not? Are we seeing any increase in income for SNAP beneficiary households? Why or why not? If YES, will it be sustained without program resources? Why or why not? How do participants and facilitators track this information? On a proxy basis? Can we see them having radios? bicycles? iron roofing? Are children’s tuition being paid for? Are medical fees taken care of? Etc. How did increase of yield lead to an increase their living standards? How does this information get reported to SNAP SO2 Agriculture Manager? Do the FFS facilitators show long-term sustainable capacity to provide services to farmers? Why or why not? Of the activities/inputs and training that takes place by SNAP-trained facilitators, which will continue at the end of the program? Which will not continue? Why or why not? How many Facilitators trained by SNAP are there? After attending a SNAP training session, did they cascade the training to other lead farmers? How does SNAP monitor the cascade of training? How does SNAP verify the cascade of trainings? SNAP MTE – Final Report Moneval Solutions Ltd Confidential Page 115 How often does staff of SNAP visit the FFS? What do they do during the visit? Were there agricultural technical issues the facilitators were able to solve by themselves? Give a couple of examples. Are we seeing any transfer of new knowledge, skills, and change in practice of FFS program participants to non-participants? Do you see changes in the farming practices of farmers in SNAP? If so what? Give a few examples. Are we seeing any transfer of new knowledge, practices, and skills from program FFS participant farmers to farmers NOT participating in FFS? Why or why not? Are we seeing any increase in farming nutritional foods in target communities? Why or why not? If YES, will it likely be sustained without program resources? Why or why not? Are we seeing any increase in farming of cash crops in target communities? Why or why not? What would FFS participants like to see happen to the schools after the SNAP program ends? How can these best be helped to achieve this? When SNAP ends, will the FFS continue in your community? How/Why? What challenges will they face in continuing? Please give any suggestions? Are we seeing women well represented in FFS beneficiary groups supported by SNAP program? On average, what is the ratio between men and women in an FFS participating group? Do both men and women from different income level households have access to and control over the resources required to participate in livelihoods activities? What are the obstacles? Do they benefit equally from livelihood activities? What types of decisions do women and men make with regard to household livelihoods? Give a couple of examples. How many participating women in FFS are PM2A beneficiaries? 14.8.2.2 Vocational Training The vocational training visits will involve participating Youths and their Trainers/Facilitators. This will include a visit to the schools or the shops to see what the young beneficiaries have been doing/learning and what they have accomplished. Are the right beneficiaries being enrolled/enrolling in the program? Why or Why not? Since when have you been benefiting from the SNAP program? How did you hear about the program? Sensitization in the community? A friend? A spouse? What is the purpose of SNAP for vocational schools? What does SNAP do for the youth? Why is this important? SNAP MTE – Final Report Moneval Solutions Ltd Confidential Page 116 Are girls and boys involved in the program the right goal of the program? Why and why not? If yes, what is the ratio between boys and girls? In the last two years, are we seeing any increase in access to vocational training in target communities? Why or why not? If YES, will it likely be sustained without program resources? Why or why not? Since SNAP started, how have living conditions of participating youths changed in the community? What kinds of trades have they been involved in? i.e. masonry, hair dressing, gara tie-dyeing, soap￾making, and auto- mechanic, etc.? How are they Trainers/Facilitators motivated? Monthly/Quarterly stipends by SNAP? Do you think these incentives will be available after the program ends? Why and how? Have the participating youths acquired new knowledge translated into new skills? Why or why not? What factors are contributing to this? Why or why not? What do you see boys and girls in SNAP doing now that they did not do before the program started? Which strategies and activities have been more effective so far and why? Of all of the SNAP activities with the vocational education component, what trade is having the greatest positive change in your community? Why? Are we seeing any transfer of new knowledge, skills, and practice with the enrolment in other parts of the SNAP program? Why or why not? Do boys and girls in the community NOT in SNAP do any of the promoted change in SNAP? If so what? Are there boys and girls in the vocational education program members of FFS? and/or PM2A? or Mother Care Groups? Have you started doing anything SNAP promotes? If so what? Why? How many have become entrepreneurs on their own? Doing what? How the trainings have improved your income? By how much? or by proxy? How the trainings have improved your nutritional status? What are you eating now and did not have before SNAP? What would Graduate Members of the Vocational Education program like to see happen to the boys and girls after the program ends? How can we best help to achieve this? When SNAP finishes, what activities will your community continue do you think? and how will this SNAP MTE – Final Report Moneval Solutions Ltd Confidential Page 117 continue? 14.8.3 Common Groups The following guide deals with groups which are common to SO1, SO2 and Crosscutting themes, these are:  Leadership Management  Monitoring and Evaluation Staff  Title II Food Commodities Management Team 14.8.3.1 KII – Leadership Management This guideline is designed correspond to Program, Management, Coordination and Operations evaluation questions. The review will consist of interviews with staff as well as observations and cross-checking information. Additional questions may be asked during field visits and interaction with staff from SNAP. How effective and efficient is coordination between the three partners. What has worked well? What has not? Dimensions under consideration should include at a minimum: Describe the coordination between the 3 partners (ACDI, IMC and OICI). What works well? What has not? Program Planning and Activity Coordination between HQ (Washington) and the Head offices (Freetown); Program Planning and Activity Coordination between the Country offices (Freetown); and Program Implementation of SO1 and SO2 integrated activities. How effective and efficient is coordination between the two SO activities? What has worked well? What has not? Describe the coordination between the partners and the SO activities? What works well? What hasn’t? Which strategies and activities have been effective so far and why? Are the right beneficiaries being targeted? Are the right beneficiaries being enrolled in the SNAP program? Why or Why not? How have implementation related problems been identified, analyzed and solved? What has worked well in terms of problem solving? What has not worked well? How are implementation problems solved? What steps are involved? How do you know the problems have been addressed? Describe a time it has worked well. SNAP MTE – Final Report Moneval Solutions Ltd Confidential Page 118 High staff turnover has been an issue that has been a challenge to smooth implementation of the activities. How do you think such problems in the remaining two years of SNAP can be addressed? Are SNAP staff qualified and knowledgeable of the purpose and methods used in the program and equipped to properly implement the activities? What project management, M&E health and nutrition, food management, VSLA training, staff retreat, etc.? have they undertaken? Does the management structure/culture, support staff spending time creating and sharing knowledge (i.e., documentation, meeting reports, coaching). Do job descriptions clearly specify roles and responsibilities? How often are you engaged in learning activities? How do you report/share your lessons learned? Is this encouraged? How is it encouraged? Which strategies and activities have been more effective so far and why? Of all SNAP activities, which one is having the greatest positive change in the communities? Why? How do you know? How do you share/report this information back to senior management? Joint monitoring visits? Monthly/quarterly visits or meetings? The cost recovery of the monetization transaction has been problematic and has actually cost SNAP to scale down/cancel some field activities i.e. storage, drying floors, etc. Do you still see monetization as a funding source? Why? Why not? If not, what are your suggestions? What other possibilities are you exploring? 14.8.3.2 KII – Monitoring and Evaluation Staff Between October 2012 and May 2013, the SNAP Team made a number of significant advancements in relation to the program’s Monitoring and Evaluation. Along with routine data collection, the SNAP M&E team accomplished a number of additional activities including: Designed, developed data collection instruments, and initiated a household beneficiary’s census to update data and link SO1 and SO2 databases so that they can be integrated into a single, comprehensive database (PREP of June 2013). Furthermore, in its response to FY12 ARR, USAID/FFP wrote: “In the IPTT, FY12 beneficiaries reached total is 160,000, but the FY12 beneficiary table notes 178,000. In the PREP, meanwhile, the FY13 target almost triples to 453,000 beneficiaries”. This indicates that something needs to be done with SNAP M&E system. The discussions will be held at both the country office and district office levels. It necessitates the reading of some report forms. How well is the output monitoring system functioning? What is working well? What is not? Where, if any, are their bottlenecks in this system? Does the output monitoring have well developed data collection forms, data flow diagram? defined responsibilities for each partner and/or sector? aggregation levels? and a clear link showing how the SNAP MTE – Final Report Moneval Solutions Ltd Confidential Page 119 information will be aggregated and reported onto the Annual Results Report and PREP? Does the monitoring system measure outcome level changes? Does the system use appropriate methods and tools? How often it measures outcome? Is the system too burdensome to the M&E team? Does the outcome measurement system also investigates and present reasons behind the successes and/ or under achievements? How useful these findings are to the users? How this can be improved? How is the output and outcome monitoring systems and monitoring activities are linked/coordinated between SO1 and SO2 M&E Teams? Between the field and the country offices in Freetown office? What is working well? What is not working well? The creation a single collection database has become critical. Does ACDI/VOCA have the technical and Managerial capacity to carry out the exercise? If YES, is there a timeline? Is there a plan with the other partners to have a training or retreat to harmonize the indicators and the co0llection methodology? If YES, when is it happening? How often does the M&E staff of the 3 partners get together on a routine basis to discuss about data collection methodology of the 2 SOs activities in the field? If it is not systematic, why? Does the current M&E system imply systematic joint visits with local authorities or government agents? If YES, how of often? If NO, why not? Does the current M&E imply systematic restitution of progress/achievements to the beneficiaries and/or the local authorities? If not why? Where does the M&E system “sit” and what IT program is used for the data input? Is this working well? Who data inputs and who has access to it? What are the reports that the system generates? Is this adequate? 14.8.3.3 KII – Title II Food Commodities Management Team The review consists of interviews with commodity staff as well as observation of existing warehouses, documentation and commodity distribution system. Additional questions will be asked during field visits and interaction with staff. At distribution sites, the MTR will interact with women beneficiaries to assess if they understand their rations and how far they travel to pick up food. Effectiveness:  Are the food distribution mechanisms well understood by all the parties involved – staff, partners and beneficiaries?  Food request approval system?  Waybills, inland transportation? SNAP MTE – Final Report Moneval Solutions Ltd Confidential Page 120  Does primary and secondary warehousing respond to USAID’s requirements?  Does the staff understand the ration allocations?  Do participating women know their rations?  Is the inland transportation system adequate? If not, what are the main challenges?  Are trucks available and reliable?  Are all distribution sites reachable in all seasons? Weaknesses:  What are the main challenges for the food distribution to participating mothers?  Distance: how far do women travel to the distribution sites?  What are the conservation/storage facilities? Handling of unfit foods? Etc.  What was done to address the issues?  Are there end-use checking and physical inventories in place? If yes, how often?  What are the main challenges? Any suggestions? Opportunities:  There were cases of food mismanagement in the past.  What did you do to prevent this from happening then or again?  Was the monetization process a success story? If not, why?  How do you foresee the funding of SNAP in the next 2 years? Continue with monetization or look for other funding mechanisms and strategies?  If monetization remains the first option, what can you do to address the cost recovery issue?  If monetization is not the option, what other alternatives are you reviewing?  What types of training would your team and you need to do this job effectively? SNAP MTE – Final Report Moneval Solutions Ltd Confidential Page 121 15 Annex 5: List of meetings and sites visited List of Key Informants, Communities visited and site visits undertaken Name Function Institution Aiah .J. Thorlie District Agriculture Officer – MAFFS, Bombali District Agriculture Officer Amadu B.J Sesay District Extension Officer￾Bombali District Extension Officer Charles Mansaray District Training Officer- Bombali District Training Officer Alie A. Fofanah M & E Officer - Bombali M & E Officer Ibrahim M. Kargbo Assistant District Forest Officer Assistant District Forest Officer Cmlex Mammy District Forest Officer- Bombal District Forest Officer Abu A. Kalokoh Block Extension Supervisor Block Extension Supervisor Ambrose Bindi Kamara District Agriculture Officer MAFFS Gibson Foray District Training Officer MAFFS Augustine P.S Kamara District Crop Officer MAFFS Simeon Jambawai District Office Manager ACDI/VOCA Francess Sia Saquee Agronomist ACDI/VOCA Moses Kainyande Field Agent ACDI/VOCA Aruna Bangura Warehouse Officer ACDI/VOCA Rev. Claude Senesie Training Field OIC Sheku Gbla District Operations Manager ACDI/VOCA Momodu N. Massaquoi M & E Officer ACDI/VOCA Alpha Yayah Mansaray Agronomist ACDI/VOCA Andrew Pemagbi Agric. Field Agent - Yoni ACDI/VOCA Nancy Turay Agric. Field Agent – Kalansogoia ACDI/VOCA SNAP MTE – Final Report Moneval Solutions Ltd Confidential Page 122 Alhaji F. Mansaray Agric. Field Agent – Kafe Simera ACDI/VOCA Victoria Ann Sandi Assistant Distribution and Data Officer -Bombali ACDI/VOCA Margret Kalokoh Food Aid Monitor ACDI/VOCA Fallah Fomba Warehouse Assistant ACDI/VOCA Shamsu Mustapha Assistant Training officer OICI Abdul S.Koroma Micro Enterprise Trainer OICI Mustapha M. Ishmail M & E Officer – Bombali District IMC MoHSamed S.T. Kamara M & E Officer – Tonkolili District IMC Francess K. NyandeMoHS VSLA Officer ACDI/VOCA Peter H. D. Karimu Team Leader – Tonkolili District IMC Henry J. Tucker District Supervisor – Mile 91, IMC Mustapha Bawoh District Technical Adviser IMC Victor K. Harding District Supervisor IMC Morris Coomber District Supervisor – Mile 91, IMC Richard Cortu Warehouse Officer ACDI/VOCA Sidikie T. Brima VSLA Coordinator ACDI/VOCA Princess H. Sowa Team Leader – Bombali District IMC Tewa Bandabla District Supervisor – Gbanti Kamaranka IMC Agnes M. Gibateh District Supervisor – Kafe Simira IMC Mamoud Fofanah Driver IMC Sheku Conteh Driver IMC Umaro S. Kamara VSLA Officer ACDI/VOCA Georgia Bean Chief Of Party ACDI/VOCA Seraphine Olanya Deputy Chief Of Party ACDI/VOCA SNAP MTE – Final Report Moneval Solutions Ltd Confidential Page 123 Patrick Lebbie Compliance Technical Officer ACDI/VOCA Joseph Turay Commodity Support Coordinator ACDI/VOCA Michael Ayoola Warehouse Officer ACDI/VOCA Tigabu Tarakegn Commodity Manager ACDI/VOCA Sam Juana M & E Manager ACDI/VOCA Sia Iscandari Program Director ACDI/VOCA MoHSamed Kano Chief Tailor Vocational Education John Kamara Graduate Tailor Vocational Education Abdul K. Koroma Graduate Tailor Vocational Education Kadiatou K. Sesay Graduate Tailor Vocational Education Nancy Gandi Graduate Mechanic Vocational Education Mirancy Kamara Graduate Soap-making Vocational Education Saley Koroma Graduate tie-dyeing Vocational Education Ibrima Bonoura Graduate Carpentry Vocational Education Sundu Dabot Graduate Tie-dyeing/Soap Vocational Education Sarah Marah Graduate Tie-dyeing/Soap Vocational Education MoHSomad Kondoh Warehouse Officer ACDI/VOCA David S. Sallu District M & E and 13 members of his entourage MAFFS Issatou Kamarou FFP Representative FFP/USAID Kelly Barracks Country Program Manager FFP/USAID Leonard Banoh District Office Manager ACDI/VOCA Anthony Foday Agronomist ACDI/VOCA Edward V. Sefoi Field Agent ACDI/VOCA Sheku Samuel Kamara Chairman, Koinadugu Dist Council (KDC) SNAP MTE – Final Report Moneval Solutions Ltd Confidential Page 124 Michael M Kamara Deputy Chairman KDC Almamy B Koroma Chief Administrator, KDC KDC Umaru Jalloh Deputy Chief Administrator KDC Marie S Sesay Human Resource Officer KDC Eric H Tommy Environmental Officer KDC Alhassan H Jalloh Development & Planning Ofr KDC Komba Ngandi Procurement Officer KDC Councilor Morlai Samura Councilor, Folosaba Chiefdom KDC Councilor Haja Bintu Mansaray Councilor, W. Yagala chiefdom KDC Konta Marah Training Field Assistant OICI Farmer Field Schools Communities Kailahun, Bunumbu I, Bunumbu II, Kigbai, Mandu Tawahun, Levuma, Magburaka, Gaindema, Mamaka, Makonthandae, Kamaranka Rowula, Kaponkie, Kamabaio, Kabala, Kondebaia, Madina, Kafogo Name of Community Bunumbu Levuma Mamaka Karamanka Kondembaia Mandy Tawahuh Ngdba Bumbuna Karamanka Kafogo SNAP MTE – Final Report Moneval Solutions Ltd Confidential Page 125 16 Annex 6: Updated Indicator performance tracking tables (IPTT) Please see overleaf for the details of the IPTT for SNAP Fiscal Year 2013. This is available as an Excel file as well. SNAP MTE – Final Report Moneval Solutions Confidential Page 126 SNAP MTE – Final Report Moneval Solutions Ltd Confidential Page 127 SNAP MTE – Final Report Moneval Solutions Ltd Confidential Page 128 SNAP MTE – Final Report Moneval Solutions Ltd Confidential Page 129 SNAP MTE – Final Report Moneval Solutions Confidential Page 130 17 Annex 7: Summary Tables on Commodities and Human Resources Not available to the MTE team. 18 Annex 8: Message from One KII “My message to the American People” By Nancy Gandi “Please tell the American people that I’m very grateful. I say thank you for what they have done for me. They have made a mark in my life.” These are the words of Nancy Gandi, a 37-year old woman interviewed in Sierra Leone during the SNAP MTE. Nancy is now a female mechanic with a mining company, Base Construction Mining (BCM International), in Kalonsogoia Chiefdom in the District of Tonkolili. “I was nothing before,” said Nancy. “In 2011, when SNAP came up with the sensitization on vocational training in my community, I told myself I had to show I was worth something and chose mechanic instead of the usual hairdressing or tailoring that women do. We were two females among fifty-eight males for our training. After the authorized three months of training, I graduated with honors among my peers. With the start-up kits provided by SNAP to each graduate, the pliers, screwdrivers, and other tools, I decided to open an auto shop and asked other graduates to join me; but they were dragging their feet. In the meantime, I heard about a job offer on the radio; BCM, the mining company, was looking for some mechanics at its newly built complex in Kalonsogoia. From twenty-five candidates in the first step of the recruitment process, we were screened down to thirteen, and three were to be retained after the final interview. It happened that all three were graduates of SNAP. The other two were Alassane Kamara and Maliki.” “I’ve been in the job for the last three months and my life has changed for the better. Now, I have a steady salary on a monthly basis. I can take care of my child. My boyfriend just lost his job and what I’m bringing home is enough provision for the three of us.” Nancy has a dream to fulfill in her life; she would like to have her own auto shop, and become the first female in Sierra Leone to do so in the process. She is already working on it, as she invests any spare money in buying tools. The other ambition she cherishes is to drive her own car one day, as mechanical problems on the road during long trips are no longer one of her worries. “Thank you America, thank you SNAP,” were Nancy’s closing words. Photo credit:: Ange Tingbo, 2013 Above: Nandy Gandi A success story – from vocational training to construction work as a mechanic! SNAP MTE – Final Report Moneval Solutions Ltd Confidential Page 131 19 Annex 9: Suggested Food Distribution Methodology The main goal of this approach, as suggested by the MTE, is to organize the women and empower them with the distribution of food. The LMs will still lead their groups. The advantages are as follows:  The groups are formed and headed by their LM  The women know their rations, and understand why they are entitled to such a ration of CSB, bulgur oil, and lentils. Assuming that the sensitization is well done, the rations cannot be tampered with and are displayed at the front of the site  The delimitation of the distribution area with rope creates an environment that keeps intruders at bay  Gauges will be built to measure a content level equal to the authorized rations (see picture). Gauges would have been “technically” measured and tested in the presence of beneficiaries to confirm their capacity in kilos or liters.  The change suggested is not drastic compared to the current modus operandi. Two CGs can be put together on the day of distribution; however, they will remain under the leadership of their respective LMs. No more than twenty persons per group is a manageable size.  The LMs are given the  distribution list for their CGs; the first thing to do is verify attendance and have the members put down their fingerprints. Cartons of oil can be used as tables. The PHU agents’ verification of the health cards for weighing and vaccinations can also be carried out  The ACDI/VOCA food distributor will instruct the helpers to take out to each group the correct number of bags and cartons according to her/his calculations.  In this scenario, six groups can distribute/receive food at the same time. For a group of twenty, everything is generally completed in less than two hours Photo credit:: Ange Tingbo, 2013 Above: Beneficiaries taking care of their own distribution Female community members taking care of food distribution, Sierra Leone SNAP MTE – Final Report Moneval Solutions Ltd Confidential Page 132  Distribution lists are given back to the food distributor for compilation  Meanwhile, HPs and IMC managers or field agents will be present in their supervisory role, making sure that the knowledge transferred to the LMs actually has passed down to the other mothers. This is also an opportunity to check (in theory) for achievements in terms of hygiene and sanitation  This approach is less expensive than any other, with the advantage of empowering the beneficiaries. It is capacity building in food distribution and, following rapid training and two operations, it runs smoothly in the presence of supervisors SNAP MTE – Final Report Moneval Solutions Ltd Confidential Page 133 20 Annex 10: Disclosure of any Conflicts of Interest Name Ange Tingbo Title Mr Organization Moneval Solutions Ltd Evaluation Position? Team Leader Evaluation Award Number (contract or other instrument) USAID Project(s) Evaluated (Include project name(s), implementer name(s) and award number(s), if applicable) SNAP I have real or potential conflicts of interest to disclose. No If yes answered above, I disclose the following facts: Real or potential conflicts of interest may include, but are not limited to: 1.Close family member who is an employee of the USAID operating unit managing the project(s) being evaluated or the implementing organization(s) whose project(s) are being evaluated. 2.Financial interest that is direct, or is significant though indirect, in the implementing organization(s) whose projects are being evaluated or in the outcome of the evaluation. 3.Current or previous direct or significant though indirect experience with the project(s) being evaluated, including involvement in the project design or previous iterations of the project. 4.Current or previous work experience or seeking employment with the USAID operating unit managing the evaluation or the implementing organization(s) whose project(s) are being evaluated. 5.Current or previous work experience with an organization that may be seen as an industry competitor with the implementing organization(s) whose project(s) are being evaluated. 6.Preconceived ideas toward individuals, groups, organizations, or objectives of the particular projects and organizations being evaluated that could bias the evaluation. I certify (1) that I have completed this disclosure form fully and to the best of my ability and (2) that I will update this disclosure form promptly if relevant circumstances change. If I gain access to proprietary information of other companies, then I agree to protect their information from unauthorized use or disclosure for as long as it remains proprietary and refrain from using the information for any purpose other than that for which it was furnished. Signature Date 25 October, 2013 SNAP MTE – Final Report Moneval Solutions Ltd Confidential Page 134 Name Daphyne Williams Title Miss Organization Moneval Solutions Ltd Evaluation Position? Team Member Evaluation Award Number (contract or other instrument) USAID Project(s) Evaluated (Include project name(s), implementer name(s) and award number(s), if applicable) SNAP I have real or potential conflicts of interest to disclose. No If yes answered above, I disclose the following facts: Real or potential conflicts of interest may include, but are not limited to: 7.Close family member who is an employee of the USAID operating unit managing the project(s) being evaluated or the implementing organization(s) whose project(s) are being evaluated. 8.Financial interest that is direct, or is significant though indirect, in the implementing organization(s) whose projects are being evaluated or in the outcome of the evaluation. 9.Current or previous direct or significant though indirect experience with the project(s) being evaluated, including involvement in the project design or previous iterations of the project. 10. Current or previous work experience or seeking employment with the USAID operating unit managing the evaluation or the implementing organization(s) whose project(s) are being evaluated. 11. Current or previous work experience with an organization that may be seen as an industry competitor with the implementing organization(s) whose project(s) are being evaluated. 12. Preconceived ideas toward individuals, groups, organizations, or objectives of the particular projects and organizations being evaluated that could bias the evaluation. I certify (1) that I have completed this disclosure form fully and to the best of my ability and (2) that I will update this disclosure form promptly if relevant circumstances change. If I gain access to proprietary information of other companies, then I agree to protect their information from unauthorized use or disclosure for as long as it remains proprietary and refrain from using the information for any purpose other than that for which it was furnished. Signature Date 30 November 2013 U.S. Agency for International Development 1300 Pennsylvania Avenue, NW Washington, DC 20523