1 FINAL EVALUATION FOOD FOR EMERGENCY EBOLA VIRUS DISEASE SUPPORT (FEEDS) PROJECT IMPLEMENTED IN KAILAHUN DISTRICT BY SAVE THE CHILDREN SIERRA LEONE FINAL REPORT Save the Children - Sierra Leone Prepared by: MSD Consulting Limited - Sierra Leone Evaluation Consultants: Baimba Abdulai Koroma – Team Leader (Food Security and Cash Transfer Evaluation Expert) Momoh Thomas Bockarie (Socio-economic and Livelihood Expert) December, 2017 Award Information: Funded by USAID/Office of Food For Peace; AID-FFP-G-15-00024 2 Acknowledgements This final evaluation report is made possible by the generous support of the American people through the United States Agency for International Development (USAID). The end-line evaluation team members remain grateful to Save the Children (SC) - Sierra Leone program staff and partners for the ownership they exhibited and for providing support to MSD through the entire evaluation exercise. To this end, MSD Consulting Limited appreciates your inputs into the production of this report. Our sincere thanks go to the United States Agency for International Development (USAID) for providing the funds to design and implement the FEEDS intervention and its evaluation process through SC - Sierra Leone. Without this funding, the FEEDS intervention would have not been initiated and subsequently contributed to addressing the acute food insecurity, reduced income and malnutrition situations created by the Ebola Virus Disease (EVD) among households in Kailahun district. We are grateful to the beneficiary participants and implementing partners, including; Ministry of Agriculture, Forestry and Food Security (MAFFS), Kailahun district Local Council, National Commission for Social Action (NaCSA), Anti Corruption Commission (ACC), representative from SPLASH electronic money transfer agency, community chiefs, and Cash Transfer Committees (CTC) who participated in this evaluation process by providing relevant information used for the production of this report. We also recognize the important role played by field data collectors whose names are attached to this report. We hope that findings, lessons learned and recommendations generated by conducting this end-line evaluation will gainfully be utilized by stakeholders, especially in the design and implementation of cash transfer related future program in Sierra Leone and/or other developing countries. DISCLAIMER The contents of this evaluation report are the responsibility of MSD consulting limited – Sierra Leone and do not necessarily reflect the views of USAID or the United States Government. 3 Table of Contents Acknowledgements 2 Table of Contents 3 Acronyms 4 List of Tables 4 List of Figures 4 Executive Summary 6 Constraining Factors of Project Implementation 9 Lessons Learned 10 1.1 Introduction 12 1.2 Justification of the Final Evaluation 12 2.0 Methodology of the Evaluation 14 2.3 Data Collection 14 2.3.1 Primary Data Collection Tools 14 2.3.2 Secondary Data Collection Tools 15 2.4 Sampling Techniques and Sample Size Determination 16 2.4.1 Sampling Techniques 16 2.4.2 Determination of Sample Size 16 2.5 Data Management and analysis 18 2.5.1 Data capturing and Processing 18 2.5.2 Data Analysis 18 2.6 Data Quality Assurance 19 2.8 Evaluation Ethics 20 2.9 Limitations of the Evaluation 20 3.0 Analysis and Findings 21 3.3.1 Household Dietary Diversity Score (HDDS) 28 3.3.5 Average Household Meal Frequency 34 3.3.6 Utilization of FEEDS Cash by Beneficiaries 34 3.4 Capacity Supports to FEEDS Cash Beneficiaries 36 3.4.1 Capacity Supports to Traders 36 3.4.2 Capacity Supports to Women in Agriculture Groups (WAGs) 38 3.5 Effectiveness of Implementation - Cash Transfer Processes 39 3.7 Intervention Relevance and Strategic Fit 42 3.8 Resource Mobilization and Efficiency of FEEDS Implementation 43 3.9 Partnerships, Collaborations and Coordination 44 3.10 Mainstreaming Gender and Sustainability in FEEDS Project Activities 45 3.11 Monitoring, Evaluation, Accountability and Learning 46 3.12 Constraining and Facilitating Factors of Project Implementation 47 4.0 Lessons Learned and Recommendations 48 4.1 Lessons Learned 48 4.2 Recommendations 49 References 50 Appendices 51 4 Acronyms CFSVA Comprehensive Food Security and Vulnerability assessment CTC Cash Transfer Committees EVD Ebola Virus Disease FAO The United Nations Food and Agriculture Organization FCS Food Consumption Score FEEDS Food for Emergency Ebola Virus Disease Support FGDs Focus Group Discussions HDDS Household Dietary Diversity Score HHS Household Hunger Scale KIIs Key Informants Interviews MAFFS Ministry of Agriculture, Forestry and Food Security M&E Monitoring and Evaluation MDGs Millennium Developments Goals MSD Management for Sustainable Development NaCSA National Commission for Social Action NGOs Non-Governmental Organizations PDM Post Distribution Monitoring PPS Probability Proportional to Size PRSP Poverty Reduction Strategy Paper RCSI Reduced Coping Strategies Index SDGs Sustainable Development Goals SLDHS Sierra Leone Demographic Health Survey SPSS Statistical Package for Social Scientists UN United Nations USAID United States Agency for International Development WASH Water Sanitation and Hygiene WFP The United Nations World Food Program WHO World Health Organization List of Tables Table 1: Summary of Respondents’ Category and Response Rate Table 2: Food Consumption Score Benchmarks Table 3: Meal Frequency (for last 24hrs) Table 4: Satisfaction with Distance Travelled to Pay Points Table 5: Adequacy of Beneficiary Cash Amount Table 6: Satisfaction with Waiting Time to Receive Cash Table 7: Satisfaction with Safety at pay Points Table 8: Level of Beneficiary Awareness about the Feedback Mechanisms List of Figures Figure 1: Percentage Distribution of Sex of Household Respondents Figure 2: Percentage Distribution of Age of Household Respondents Figure 3: Percentage Distribution of Respondents Level of Education Figure 4: Employment Status of Household Heads Figure 5: % of Primary Sources of Respondents Household Income 5 Figure 6: % of Secondary Sources of Respondents Household Income Figure 7a: % of Tertiary Sources of Respondents Household Income Figure 7b: % of Overall Sources of Beneficiary Household Income Figure 8: % of Average Normal Household Expenditure Pattern Figure 9: Average Household Dietary Diversity Score (HDDS) Figure 10: % of Respondents Household Dietary Diversity (Multiple Responses) Figure 11: % of Average Household Hunger Scale Figure 12: Average Household Food Consumption Score Figure 13: Average Household Reduced Coping Strategies Index Figure 14: % of Average FEEDS Cash Utilized on Food and Non-Food Items Figure 15: % of Food types Purchased with FEEDS Cash (Multiple Responses) Figure 16: % of Household Member Participation in Decision Making 6 Executive Summary The May 2014 Ebola Virus Disease (EVD) outbreak in Sierra Leone and the accompanied control measures, including quarantine of households, communities, chiefdoms and districts restricted the movement of many household members. The restrictions interrupted households’ participation in livelihoods and agricultural activities, food production levels and access to markets. These impacts eventually worsened households’ economic and food security situations, leading to reduced diversified food consumption and increased negative coping strategies such as selling household productive assets, child labor, begging, and skipping of meals. This left over 88% of Sierra Leonean households food insecure; in Kailahun, 55.1% of the population was food insecure (9.5% were severely food insecure and 45.6% were moderately food insecure). The need to complement government efforts to address the acute food insecurity and livelihood situation prompted Save the Children, in partnership with its funding agency USAID, to design and implement a cash-based food program for Emergency Ebola Virus Diseases Support (FEEDS) intervention in the nine chiefdoms of Kailahun District. The FEEDS intervention was mainly focused on distributing conditional and unconditional cash transfers to enable the most vulnerable affected households to recover from the shocks of EVD and meet their food security needs. To measure progress and results achieved, as well as demonstrate learning and accountability, Save the Children and USAID commissioned this independent evaluation to generate evidence that would permit them conclude what results had been achieved, and to enhance future design and implementation of related interventions. The evaluation was conducted to: ❖ Assess the extent to which households in targeted district participated in the FEEDS intervention to achieve food security as measured by key outcome indicators, including household income and livelihoods, household dietary diversity score, household hunger scale, household food consumption score, reduced coping strategies index, and household meal frequency. ❖ Assess the extent to which conditional and unconditional cash transfers contributed to improving the ability of project participants to mitigate, adapt, and recover from the economic impacts of the Ebola emergency. ❖ Assess whether new management practices were adopted, and the extent to which these affected the activities of female traders and women’s agricultural groups. ❖ Assess the extent to which the systems, processes, and procedures used by FEEDS for delivering cash transfers and non-cash assistance to participants were effective, efficient, and sustainable during the period of program implementation. ❖ Assess the extent to which collaboration and coordination with public and private sector stakeholders contributed towards enhancing effective delivery of cash-based food assistance to vulnerable households during the emergency and recovery phases. ❖ Identify lessons learned from the implementation of FEEDS intervention to be used for future programming. A non-experimental pre-test and post-test mixed-method evaluation design with a participatory approach was conducted using purposive and three-stage cluster sampling techniques. Both quantitative and qualitative data collection methods were used. A mobile-based household survey questionnaire designed on the KOBO collect open-source data collection platform was used to collect quantitative data on key intervention outcomes and other indicators. Focus group discussions (FGD) and key informant interview (KII) guides were used as qualitative tools to collect in-depth information, particularly for explanations and triangulation of evaluation findings. Quantitative data was analyzed using statistical techniques, which 7 included tests of significance, while content analysis was used to analyze qualitative data. Below are key findings, lessons learned, and recommendations of the evaluation. Key Findings Household Income/Livelihood Sources ❖ Overall, even though casual labor remains the third most prominent source of household income, the average reliance of households on crop farming/sales, petty trading, and livestock farming as main sources of income have increased. While petty trading marginally increased by 0.7%, crop farming increased more substantially, from baseline at 32% to an endline value of 58.9%. Livestock farming increased from 3% at baseline to 6.1% at endline. The evaluation also revealed that casual labor has markedly reduced, from 22% at baseline to 16% at endline, while other sources of income (begging, firewood and charcoal sales, cash for work, or ‘no income’) have largely reduced, from 18.7% to 6.3% at endline. ❖ One possible explanation for these findings could be the fact that cash received by household beneficiaries has helped households to reduce their reliance on temporary casual labor, begging, and remittances. FEEDS cash-supported beneficiaries seemed to have focused on engaging themselves in crop and livestock farming on their own plots, as well as petty trading activities. They have thus eventually been enhanced to have access to better own production and purchase of food from the market in other to meet their food security and nutritional needs. Normal Household Expenditure ❖ Overall, average normal expenditure of beneficiary households on food is seen to have increased from 44% at baseline to 45.0% at end-line. Average expenditure on non-food items are also noted to have increased. In particular, expenditure on education have increased from 10% at baseline to 18.3% at end-line, expenditure on health care increased from 8% at baseline to 14.7% at end line, expenditure on agriculture increased from 7% at baseline to 12% at end-line, and expenditure on savings at had an end-line value of 4.7% compared to 0% at baseline. Additionally, the prominence of expenditure on other items (including household items, rent/shelter, gifts, etc) have been reduced from 15% at baseline to 1% at endline, which provides a possible explanation that households now spend such moneys on agriculture, food, or savings to improve on their dietary and food intake. The increases in expenditure on the above non-food items could be attributed to the fact that at baseline, while movement was restricted during Ebola, households were found to be recovering from reduced access to their farmlands. Once the restriction was over, participants’ household extensive engagement in crop farming for own production of food has once again increased. Thus, while diverting excess funds available to expenditure on non-food items, beneficiary households still continue to have a 1% increase in food expenses, which seem to be used as part of inputs for a more diversified food production. Household Dietary Diversity Score (HDDS) ❖ Overall, the aggregate HDDS determined was 6.3. This value is noted to be greater than the FEEDS intervention baseline and target HDDS values of 4.5 and 5.0 respectively, which indicates very good achievement as measured from baseline. This is confirmed by results from a one-tailed Independent Sample T-test of statistical significance conducted at 0.05 significance level and 95% confidence level on average household dietary diversity score, which revealed a statistically significant (p<0.001) difference between baseline and endline values with respect to average HDDS. ❖ In addition, Chi-Square tests conducted at 0.05 significance level and 95% confidence level on each of the individual food groups revealed evidence of a statistically significant 8 (p≤0.001) difference between baseline and endline values with respect to the individual food groups, except for condiments and fat/oil, for which results show existence of statistically insignificant differences between baseline and endline for the two groups. Household Hunger Scale (HHS) ❖ The HHS scores are categorized into three groups: Little or no hunger, Moderate hunger and Severe Hunger. ❖ Severe hunger among all households has reduced from 1.0% at baseline to 0.5% at endline. ❖ On a whole, moderate hunger in all sampled households markedly reduced from 68% at baseline to 34.3% at endline, while little to no hunger largely increased from 32% at baseline to 65.2% at endline. This is a huge achievement of the FEEDS intervention. The marked reduction in the proportion of beneficiary participants in severe and moderate hunger, which was the basis of their selection at baseline, accompanied by the noticeable increase in the proportion of households with little to no hunger, could be attributed to contributions by the FEEDS intervention, which in effect demonstrates value for money with regards the intervention. ❖ The above findings are supported by Chi-Square tests conducted at 0.05 significance level and 95% confidence level on three HHS parameters, which revealed the difference between baseline and endline values with respect to the three disaggregated categories to be statistically significant (p<0.001). Household Food Consumption Score (FCS) ❖ Measurement of this indicator is based on International Food Consumption Score Benchmarks presented in the United Nations World Food Program (WFP) Vulnerability Analysis and Mapping Technical guideline (2008), and the Sierra Leone WFP Comprehensive Food Security and Vulnerability Analysis (CFSVA) report (2015). ❖ Overall, the average household food consumption score was found to be 52.7 at endline, which is considered to be greater than the minimum acceptable consumption of 35. Household Reduced Coping Strategies Index (RCSI) ❖ The overall average household coping strategies index was found to be 17.5. This is far below the average FEEDS intervention target and baseline household coping strategies indices of 23.0 and 25.0 respectively. ❖ The above findings show marked achievement to which the FEEDS intervention would have contributed in reducing negative coping strategies among beneficiary households. ❖ The findings are supported by results from an Independent Sample T-test of statistical significance conducted at 0.05 significance level and 95% confidence level on an average of the five categories/parameters measurement of reduced coping strategies index (rCSI) presented in the body of this report. The test revealed a statistically significant (p<0.001) difference between baseline and endline values with respect to the average rCSI. This is because the 1-tailed significance value (p<0.001) is far less than the significance level (0.05). Household Meal Frequency ❖ On average, every household member across phases and age brackets was found to have eaten one meal or more within the last 24 hours before their participation in the current endline survey. ❖ In the absence of baseline values for comparison, the evaluation team considers this value a good achievement. This is evident from taking into account criteria used select target beneficiaries and the level of coping strategies beneficiary households employed before their selection into the FEEDS intervention. ❖ Overcoming the difficulties households would have encountered to get a meal per day provides evidence that the FEEDS intervention directly improved the frequency of meal intake by all categories of household members. 9 FEEDS Cash Utilization by Beneficiaries ❖ The evaluation found that the FEEDS cash was spent on diverse goods and services, including food. It was found that out of the total cash given to beneficiaries, 44.3% was spent on food while the remaining was spent on several non-food items including education (18.0%), health care (15.0%), agriculture (12.0%), and savings (5.7%). ❖ Regarding the different types of food commodities beneficiaries buy, it was found that overall, a minimum of 90% of households spend the cash on rice, fish, vegetable oil/palm oil, and condiments. Foods like vegetables, meat, fruits, and eggs did not command much expenditure from the FEEDS cash. Support to Female Traders and Women in Agriculture Groups (WAGs) ❖ The evaluation found that support given to Female Traders has enhanced their capabilities to continue in business and some have even expanded their businesses. Almost all of them reported that they have increased their commodity stocking and restocking capacities. As a result, most of the WAG-target communities now have almost all the basic food stuffs they would need on daily basis. ❖ The cash and capacity building training support given to the WAGs enhanced their capacities to expand cultivation through use of high quality seeds and increased knowledge. Their groups have now been strengthened through group management and are well positioned to perform better in the coming years. Cash Transfer Processes ❖ The evaluation assessed four key process parameters (distance travelled to pay points, cash adequacy, waiting time, and safety at pay points) to help understand beneficiaries’ overall satisfaction level about the adequacy of the cash transferred to them and the processes that led to the cash distribution itself. Tables 4 to 7 present findings relating to this. ❖ It was found that almost all beneficiaries were satisfied with all parameters assessed, except cash adequacy, where 62.3% reported that the quarterly cash transferred to them was not enough to cover all their food needs for that duration. Reasons given for this included large family sizes and the fast rise of food prices during the intervention period, which eventually eroded the value of the cash transferred to them at the time. However, desk review findings suggest that there was some misunderstanding on the part of the beneficiaries, who thought the cash amount given to them was to cover all their food needs for a three-month period. In actuality, the cash was supposed to cover only 40% of their food needs for that period. This variance in understanding was probably the major reason for the level of dissatisfaction reported about the adequacy of the cash transferred. ❖ Regarding distances travelled to access pay point, 88.8% of respondents reported to be satisfied, because the majority of them walked two miles or less to access their designated pay points. Only 3.8% reported walking over three miles to their pay point. For waiting time, 95.7% reported that they were satisfied with the time taken at pay points before they received cash. Most beneficiaries waited for less than an hour to receive cash. Only 5.1% reported that they sometimes waited for over two hours before they received cash. For safety, both at pay points and after leaving with the cash, the evaluation found that almost all beneficiaries were satisfied with this, at 97.4% and 95.1% respectively. Even when they got back to their communities, the evaluation assessed whether beneficiaries received any pressure from community leaders or CTC members to give part of the cash to them. 99.4% reported never experiencing that. Constraining Factors of Project Implementation ❖ Bad road networks and poor mobile communications networks slowed down the pace of activities, including the cash transfer itself as well as monitoring activities. ❖ Some Cash Transfer Committee (CTC) members reported that in some cases they felt the urge to quit the project because ‘they were not getting any direct benefit (tokens, etc.) in 10 return for their services’. This was confirmed from KIIs with community leaders and field program staff, who reported that they sometimes had to spend a long time encouraging CTC members to continue working and not quit. According to them, it would have taken too much time to get new CTC members onboard and get them understand the intervention. Rather, they were encouraged to maintain current CTC members partly because the CTC members involved already understood the project concept and partly because they already knew the cash beneficiaries. This problem of CTC members quitting the project was largely mitigated by giving them FEEDS customized T-shirts. Facilitating Factors of Project Implementation ❖ The FEEDS project had enough staff to execute all key functions. Assigning specific staff to specific chiefdoms ensured that the project team was always close to the beneficiaries. This enhanced monitoring activities and flagged up issues that were addressed promptly. ❖ The evaluation noted that all the partners that the FEEDS team worked with (including the CTC and local authorities) performed their respective duties. This enhanced the level of results that the evaluation found. Note: Though some members of the CTC sometimes complained of “no tokens”, their involvement played a very crucial role in achieving the overall result that the FEEDS project produced. ❖ The project was well funded. This was a huge source of timely and full implementation of activities, which permitted the achievement of desired results. ❖ Effective sensitizations and house-to-house visits by field staff on the correct use of cash received, coupled with the complementary trainings, ensured that beneficiaries used more of the cash to access food items, rather than non-food items. This helped improve their food security statuses, as evidenced by changes in the respective indicators. The special attention paid to household decision-making regarding the use of FEEDS cash also helped avert most potential conflicts, which may have been the source of the level of women empowerment the evaluation found. Lessons Learned ❖ When the income of households increases in cash, they are motivated to increase their expenditure on other important items that were not previously prioritized as expenditure items when there was less or no income in the household. The FEEDS cash expenditure patterns seem to be contrary to the expectations of the FEEDS intervention that at the end of the intervention, 70% of the EFSP cash received by households would be spent on food items. However, the observed decrease in EFSP cash expenditure on food (44.3%) was compensated for by expenditures on other important and productive household needs like education, health care, agriculture, and savings, all of which contribute to long-term household food security. ❖ Save the Children deployed staff directly in the target chiefdoms. This provided for a closer link between implementing staff and the beneficiaries throughout the project. This was very helpful for effectively monitoring market prices, cash utilization, and nutrition￾sensitive messaging. This strategy created high SC/USAID visibility, which in turn helped improve implementation through prompt and firsthand information gathering that was used to make informed decisions about the project. ❖ Establishing the right partnerships with state and non-state actors adds special value to quick-impact projects like cash transfer projects. The presence of NaCSA, ACC, KDC, and CTCs added integrity and visibility to the whole process. ❖ When the right beneficiary targeting is done, cash interventions could have a very big impact on the daily lives of the beneficiaries. Their food security situation will be significantly impacted in the shortest possible time, as measured by the statistically significant changes that occurred in the values of target household’s HDDS, HHS, and rCSI scores. 11 ❖ The involvement of CTCs created a meaningful impact, especially in targeting the right beneficiaries and conveying information quickly to beneficiary households. However, when they are not allowed to benefit directly, they will have problems with total commitment to the program. ❖ In the absence of effective mobile technology, a well-deployed and monitored offline cash transfer project can adequately replace technology-based cash transfers. Recommendations ❖ The average meal frequency of households per day was noted to be approximately two during the final evaluation. There is need for this figure to reach a minimum of three for better nutrition outcomes. Therefore, the Government of Sierra Leone through the Ministry of Health and Sanitation, represented by the District Health Management Team (DHMT) and other related community structures, should continue with activity-based and monitoring program strategies to increase or maintain the community’s access to sufficient nutritious foods in the right varieties. ❖ The evaluation found that many households have diversified their diets because of the EFSP intervention. This has prompted most to start or increase their backyard gardening to have continuous access to fresh vegetables, fruits etc. On that note, the Government of Sierra Leone, through the Ministry of Agriculture, Forestry and Food Security (MAFFS), should undertake regular sensitizations on the benefits of backyard gardening and the importance of eating these foods to help improve household nutritional status. Additionally, MAFFS block extension officers should continue monitoring the participants who were trained in post-harvest management technology to ensure they continue to employ it in their farming activities. They should also provide technical assistance where necessary. ❖ The cash transfer processes and systems were largely effective, as noted by the beneficiary’s satisfaction levels with the distances covered to access pay points, waiting time to receive cash, and safety both at pay points and after leaving pay points. It is recommended that the strategies employed to achieve these must be maintained by Save the Children in similar future interventions. ❖ The direct deployment of Save the Children staff in the target chiefdom helped significantly in achieving results. This strategy should be replicated and/or scaled up in future similar interventions. ❖ There was some level of misunderstanding on the part of the beneficiaries about the percentage of household food needs the transferred cash was supposed to cover. Most thought the cash was meant to cover all the household food needs. Therefore, most households (62.1%) reported that the cash was not adequate for their household food needs. It is recommended that in the future, a very clear sensitization must precede the transfer of cash, which should continue throughout the life of the project. ❖ The evaluation noted that there was limited involvement of the private sector in the overall implementation. Though there was not observed direct negative impact on the project, the high potential of this sector could be tapped into if more organizations from the sector were drawn into the project. We therefore recommend that in similar future projects, Save the Children should partner with more than one private sector organization. 12 1.1 Introduction The May 2014 unprecedented Ebola Virus Disease (EVD) outbreak in Sierra Leone resulted in a worsening of the economic and food security situations for households in the country. To contain the rapid spread of the disease, control measures were identified and implemented by government and its development partners. These control measures, including quarantine of households, communities, chiefdoms, and districts graduated into restricting movement of household members from one point to the other. These measures interrupted household access to markets and negatively impacted their investment and participation in groups and agricultural activities. Consequently, households suffered reduced household economic access to diversified food and to food production level. This eventually resulted in reduced food consumption stability of households over time, increased negative coping strategies such as selling of household productive assets, child labor, begging, and skipping meals. These dismal situations left Sierra Leone with only about 11.2% of all poor households’ food secure, and only 3.3% in Kailahun district (CFSVA, 2015). The need to urgently address the acute food insecurity situation prompted the identification, designing, and implementation of cash transfer intervention, which was intended to contribute to improved food security situations among vulnerable beneficiaries. Save the Children, in partnership with USAID, took the challenge to complement government effort towards achieving implementation of the National Ebola Recovery Strategy through cash-based Food for Emergency Ebola Virus Diseases Support (FEEDS) intervention in nine chiefdoms in Kailahun District, (Dea, Njaluahun, Jawei, Kpeje Bongre, Kissi Teng, Kissi Kama, Luawa, Malema and Upper Bambara). The FEEDS project, implemented in three different phases (reaching a total of 77,211 individuals in FY 2017 and an estimated 91,154 over the life of the award) was mainly focused on distributing conditional and unconditional cash transfers to enable the most vulnerable affected EVD households recovered from the shocks of Ebola and meet their food and nutrition needs. As implementation of the project came to an end, SC commissioned an independent evaluation to assess the impact and extent to which results were achieved, contracting MSD Consulting Limited to undertake the current final evaluation to which this report serves as a primary deliverable. This report is a key deliverable for the endline evaluation exercise. After the introduction, sections on the evaluation objectives and justification, methodology, analysis of findings, and lessons learned and recommendations are presented. 1.2 Justification of the Final Evaluation In line with standards and best practices emerging from the 2005 Paris Declaration and 2008 Accra Agenda for Action on enhancing development effectiveness, the cash-based FEEDS final evaluation demonstrates learning and accountability to stakeholders for SC services delivered to beneficiaries. This permits project stakeholders to make conclusions on achievement (or non-achievement) of intervention results based on evidence. The evaluation findings allow SC, its funding partner, and other stakeholders to properly plan, implement, monitor, and evaluate future related food security-based cash transfer interventions to address issues affecting vulnerable/marginalized groups in emergencies. 1.3 Purpose and Objectives of the End-line Evaluation The overall objective of the end-line evaluation was to conduct a performance evaluation of the Save the Children USAID-funded FEEDS program in other to assess its effectiveness, efficiency, sustainability, and impact on improving the food security of beneficiary households 13 in an emergency context after the EVD outbreak. The endline quantitative survey primarily focused on assessing the following key indicators for a sample of unconditional cash transfer beneficiary households: (1) income and livelihood status, (2) the household dietary diversity score, (3) the household hunger scale, (4) the household food consumption score, (5) the reduced coping strategies index, and (6) the household meal frequency. The end-line evaluation also assessed whether new management practices were adopted and the extent to which these have affected the sales of two groups of beneficiaries, namely female traders and the women’s agricultural groups. Information gathered from the endline evaluation was used to measure the FEEDS program achievements over the course of its implementation period and report the results against predefined indicators set forth in the agreement between USAID and Save the Children. Additionally, the final evaluation provides documented learning evidence on SC implementation approaches and will inform future programming of similar interventions. The specific objectives of the endline evaluation were to answer the following questions: 1) To what extent did households that participated in the FEEDS program achieve food security and improved resilience as measured by the following quantitative variables? a. Income and Livelihood b. The household hunger scale c. Household dietary diversity score d. Reduced coping strategies index e. Food consumption score f. Meal frequency In addition to analysis of the quantitative data, the evaluator was expected to address the extent to which the below components contributed to improving the ability of project participants to mitigate, adapt, and recover from the economic impacts of the Ebola emergency: ● Unconditional cash transfers ● Conditional cash transfer support to Female Traders ● Conditional cash transfer support to Women in Agriculture ● Agronomic and financial management / business skill training 2) To what extent were the systems, processes, and procedures used by FEEDS for delivering cash transfers and non-cash assistance to participants effective, efficient, and sustainable during the period of program implementation? In answering this question, the evaluator should address the following: a) Accountability, timeliness, and cost of delivering the cash and non-cash assistance to participating households b) Assess the participants’ perception about the cash transfer and non-cash delivery processes and procedures c) Explore the gender implications of cash transfer and non-cash assistance relating to o Decision making o Gender roles and responsibilities o Gender based violence d) What were the key lessons learnt during the FEEDS program implementation? 3) To what extent did the collaboration and coordination with public and private sector stakeholders resulted in enhanced capacity to deliver cash transfers as a form of social 14 protection and food assistance to vulnerable households during emergency and recovery phases? In answering this question, the evaluator should address the following: a. Connectedness and coherence with other actors such as other private sector, NGOs, Donors, UN organizations, and multilateral agencies. 2.0 Methodology of the Evaluation 2.1 Approach A performance and summative evaluation with mixed and participatory approaches were used in this endline evaluation. The approaches permit stakeholder involvement as well as the use of quantitative and qualitative data collection techniques and tools. In addition, these approaches ensured effective triangulation of information and generalization of the FEEDS evaluation findings. 2.2 Evaluation Design A non-experimental pre-test and post-test mixed-method evaluation design with a participatory approach was used. The design allowed effective triangulation of information gathered from different stakeholders. Since project implementation was done in phases, the evaluation design strategies used accordingly took into account variations in the different extension phases of the EFSP intervention. This design provides an approximate estimate of intervention impacts in terms of changes in outcome indicators over the project lifecycle. If carefully implemented, significant difference will be obtained in each test; and this consequently would provide evidence of the intervention effects. 2.3 Data Collection The designed FEEDS logical framework, progress monitoring report and other documents, evaluation objectives, and data collection tools used at phase I baseline and midline were used to guide the current evaluation exercise. These were used to develop data collection tools to assess the extent to which project objectives were achieved, and consequently to determine the performance of the FEEDS intervention in Kailahun district. The content of phase I baseline and midline tools were largely maintained, especially for the key outcome indicators, to allow comparison of baseline, midline and endline values. A combination of quantitative and qualitative methods was employed to collect data using the different data collection tools from the respective sources. Both primary and secondary data were collected. Secondary data was collected by document review. Primary data was collected using a quantitative online mobile-based household survey questionnaire designed on KOBO software platform and qualitative tools, including focus group discussions and key informant interview guides. Beneficiary (household) surveys were used by enumerators to collect data from randomly selected beneficiaries in randomly selected clusters and respective communities. Qualitative data was used to collect data from randomly selected cluster communities across targeted chiefdoms in the FEEDS intervention. Descriptions of both primary and secondary data collection tools used are presented below: 2.3.1 Primary Data Collection Tools ❖ Online Mobile-based Household Survey Questionnaire: The questionnaire was designed on the Kobo software platform and administered to direct project beneficiaries. The tool was used to collect data on the following indicators, including; household demographic characteristics, household income and livelihoods, expenditure patterns (on food and non-food items), changes in agricultural and financial 15 assets owned by households, household hunger scale, household dietary diversity score, food consumption score, reduced coping strategies index, household meal frequency, cash transfer processes and utilization, capacity building and practices (on agronomic practices, nutrition, hygiene behavior, and financial/business management skills), relevance of FEEDS intervention to beneficiaries and communities, monitoring and accountability, and gender integration into the FEEDS intervention planning and implementation and its related effects. On average, administering the tool to one beneficiary respondent took 25 minutes. ❖ Focus Group Discussion (FGD): This was a group discussion guide designed and used to collect detailed information from direct SC FEEDS beneficiaries in small groups of 8 – 10 participants each. The guide was used to conduct group discussions separately for men and women. These included direct unconditional cash transfer beneficiaries, women farmers (agricultural), and small-scale traders who benefitted from the FEEDS intervention. The tool was comprised of open￾ended questions, administered by trained enumerators, to have respondents freely express their opinions and perceptions about the FEEDS intervention. The tool collected data on household income and livelihoods, household expenditure (on food and non-food items), relevance of FEEDS intervention, capacity building trainings and practices (on agronomic practices, nutrition, hygiene related issues, and financial/business management skills), cash transfer processes and utilization, monitoring and accountability, and coordination and sustainability of the FEEDS intervention. ❖ Key Informant Interview (KII) Guide: This guide was designed and used to collect detailed information from different categories of individual respondents, including community chiefs, Local Council, Ministry of Agriculture, Forestry and Food Security (MAFFS) at district level, National Commission for Social Action (NaCSA), Anti Corruption Commission (ACC), representatives from SPLASH electronic money transfer agency, Community Identification Committees (CICs) or cash transfer committee members, SC FEEDS intervention staff, and other partners. The tools comprised of open-ended questions, administered to the above stakeholders by trained data collectors to collect detailed information on relevance of FEEDS intervention (to beneficiaries, communities, and country strategic priorities), efficiency of the intervention, cash transfer processes and utilization, monitoring and accountability, impacts of the intervention, coordination and sustainability of the intervention, and successes, challenges, and lessons learnt. 2.3.2 Secondary Data Collection Tools The major tool used here was desk review of documents, mainly to collect secondary data from FEEDS intervention documents and other existing studies and literature relevant to conditional and unconditional cash transfers, as well as food security (particularly in emergencies) across developing countries including Sierra Leone. FEEDS collected and reported data periodically on the following indicators, although the definition and methodology of collection and analysis varied slightly for an emergency context: - FFP#28 Prevalence of households with moderate or severe hunger (Household Hunger Scale) - FFP#29 Average Household Dietary Diversity Score (HDDS) - FFP#16 Value of incremental sales attributed to program implementation - FFP#11 Number of individuals who have received USG supported short‐term agricultural sector productivity or food security training 16 -FFP#9 Number of farmers and others who have applied new technologies or management practices as a result of USG assistance. Documents prioritized for review included: SC FEEDS narrative proposal and Save the Children country strategic plan, FEEDS needs assessment report, baseline(s) and mid-line evaluation reports, project internal routine and annual monitoring reports, the Sierra Leone Smallholder Commercialization Program, Post Ebola Recovery Plan, Comprehensive Food Security Vulnerability Assessment (CSFVA, 2015), cash based FEEDS implementation and impact documents (including best practices and lessons learnt), post distribution and market monitoring reports, PRSP II & III, emergency food security technical and monitoring documents by USAID, World Food Program (WFP) and FAO and other organizational documents on household dietary diversity score, food consumption score, household hunger scale, meal frequency, and reduced coping strategies index. 2.4 Sampling Techniques and Sample Size Determination 2.4.1 Sampling Techniques At baseline (phase I) and midline (phases I and II), a stratified random sampling was followed, using a statistical formula developed by Krejcie and Morgan (1970) to determine representative sample sizes. This sampling was not, however, based on a phased-program design. Per the CDC guidance for comparable evaluations, the FEEDS program stratified each sample for the Phase I baseline survey, midline of Phase I and II survey, and baseline of Phase III survey per chiefdom for all three datasets considered. Since the phased-program design and implementation within the life cycle of the FEEDS intervention only emerged after the phase I baseline and the midline were conducted, conducting a pipeline design final evaluation would be challenging. Additionally, using the Krejcie and Morgan (1970) formula for each phase’s endline seemed to be impractical given the time and resource limitations for the final evaluation. 2.4.2 Determination of Sample Size Given the sampling challenges described above, the evaluation team used a statistical formula developed by Magnani (1999) presented as addendum in FANTA III Sampling Guide (2012) at 95% confidence level to determine a representative survey sample size of household beneficiary participants to whom the online mobile-based quantitative survey data collection tool was administered. The sample size determination took into account the design effect and key intervention outcomes indicators for which adequate information was available. Below is the sample size formula used: Where, n = required sample size of cash beneficiary respondents D = Design effect (for a two stage cluster sampling, D=2.0 for FFP Program) P1 = the value of the key indicator at baseline (or a proxy value), expressed as a proportion between 0 and 1 17 P2 = the planned target value of the key indicator at the end-line/final evaluation, expressed as a proportion between 0 and 1 Zα = the Z-score corresponding to the desired confidence level (α = 1 - a). At 95% confidence level, a = 0.05, which gives α =0.95, and Zα = 1.645 Zβ = the Z-score corresponding to the desired statistical power (β = 1 - b). Consistent with FANTA III Sampling Guide (2012), for FFP program, b is typically set at b = 0.2, which gives β=0.80 and Zβ = 0.840 ❖ Design effect is the factor by which the sample size for a cluster sample would have to be increased to produce survey estimates with the same precision as a simple random sample. Based on information provided about baseline and target values for the three key indicators [Household Hunger Scale (HHS), Household Dietary Diversity Score (HDDS), and reduced Coping Strategies Index (rCSI)], and keeping a large enough number of clusters to provide representative sample size, a mix of sample sizes was determined. For HHS, very small non￾representative sample size was determined (46 sample respondents), while very large sample sizes (almost census values) were determined for both HDDS and rCSI (3,443 and 4,932 sample respondents respectively). With these determined sample sizes, in addition to the limited time and other resources, the evaluation team assumed at least a 10% estimated change in each of the key outcome indicators during the life span of the project. This allowed for the determination of a minimum required sample size of 605 beneficiary participants. However, accounting for non-response participants and sample representation, provisions were made for an additional 48 response participants for flexibility. In line with the principle of Probability Proportional to Size (PPS) and consistent with the number of households served in the three FEEDS phases with three different groups of beneficiaries across chiefdoms and respective cluster points, the above sample size was proportionally distributed among these phased groups as follows: Phase I: 6,445 Beneficiary unique households Phase II: 1,492 Beneficiary unique households Phase III: 7,194 Beneficiary unique households1 According to Magnani (1999) as Addendum in FANTA III Sampling Guide (2012), and as used in this evaluation, a minimum of 30 clusters represents a figure adequate to ensure that sample target groups are sufficiently spread across enough clusters of intervention coverage area. In this evaluation, a total of 33 randomly selected clusters out of the 77 clusters for the FEEDS intervention with a minimum of 21 beneficiary respondents per cluster was used. In addition, three (3) communities per cluster with a minimum of seven (7) household beneficiary participants (per community) were randomly selected and surveyed. Also, a total of 22 selected focus group discussions (one per randomly selected cluster) of 8 to 10 participants per group were selected and conducted across the intervention chiefdoms. These included six (6) FGDs each for women and men beneficiaries participants, five (5) for traders, and five (5) for women in agriculture were conducted across the intervention coverage 1 The additional households, that entered the program in Phase I and II, but who were also benefitting in Phase III, were not in the list of unique Phase III households. 18 area. FGDs for the different groups were conducted in same randomly selected communities. Additionally, 51 key informant interviews were conducted with the different categories of individual respondents identified earlier. In particular, KII for Cash Transfer Committees (CTCs) and community chiefs were conducted in same randomly selected communities where the FGDs were conducted. Below is a table, presented by tools, a summary of respondent category and response rate. Table 1: Summary of Respondents’ Category and Response Rate Respondent Type Expected and Actual Number of Tools Administered Questionnaire KIIs FGDs Expected Actual Expected Actual Expected Actual Cash Participants 605 653 - - 22 22 Local Council - - 1 1 - - MAFFS Reps - - 1 1 - - Community Chiefs - - 22 22 - - Anti Corruption - - 1 1 - - Community Transfer Committees (CTCs) - - 22 22 NaCSA - - 1 1 - - SPLASH - - 1 1 - - FEEDS Program Management Staff - - 2 2 - - 2.5 Data Management and analysis 2.5.1 Data capturing and Processing Quantitative data was collected through the online mobile-based Kobo data collection system with incorporated skip logic to enhance data accuracy. The data was downloaded through MS excel and subsequently imported into SPSS Version 23.0 database, where it was cleaned to avoid duplication and other errors to ensure quality. Qualitative data was collected and captured using paper and, in some cases, recorders. The recorded data was transcribed and merged with paper-recorded data. Data was captured by thematic indicators and entered into MS Word by themes, response category, and chiefdom. This made it easier for data screening to ensure quality assurance with regards accuracy and consistency. 2.5.2 Data Analysis Quantitative data was analyzed by gender (sex) using SPSS version 23.0, which allowed for descriptive statistical analysis and presentation of evaluation findings. For selected outcome indicators (HDDS, rCSI, and HHS) with available baseline datasets, Chi-Square statistical tests were performed to estimate, based on changes observed, whether there were statistically significant differences between endline and baseline values particularly for HHS. 19 In addition, a one-tailed Independent Sample T-test was performed for average HDDS and rCSI, to estimate, based on changes observed, whether there were statistically significant differences between endline and baseline values with respect to the mean value of the indicator. Qualitative data was analyzed using thematic content analysis of evaluation information gathered from identified sources, which took into account analysis framework and plans agreed upon by an evaluation team and Save the Children. The analysis also took into account identification of emerging themes and establishment of relationships between these themes. The data was analyzed and findings presented within the framework of household demographic characteristics, household income and livelihoods, intervention relevance and appropriateness of its design, efficiency of resource utilization, and effectiveness and impact of the intervention. Effectiveness and impacts were analyzed and discussed with specific reference to Household Dietary Diversity Score (HDDS), Household Hunger Scale (HHs), reduced Coping Strategies Index (rCSI), FEEDS cash transfer implementation processes, cash utilization, and technical assistance provided by FEEDS and the accompanied beneficiary practices. In the absence of baseline values, additional food security indicators that are key to the FEEDS intervention, including Household Meal Frequency and Food Consumption Score (FCS), were also analyzed. The analysis framework took into account presentation of endline findings on partnership and collaboration, mainstreaming of gender and sustainability into the intervention, and monitoring, evaluation, accountability, and learning of the intervention. 2.6 Data Quality Assurance For quality and safety of data collected, the evaluation team established reporting lines of supervisors and enumerators assigned to them. This maintained a free flow of communication between enumerators and supervisors and between them and lead consultants, through which barriers encountered were easily overcome. Daily summary report forms were prepared and given to supervisors to be completed by enumerators and submitted on a daily basis to supervisors before the start of next day data collection. Additionally, provisions were made on the Kobo platform for names of enumerators and supervisors to enhance the ability to easily trace any anomaly. To verify the presence of data collectors in their designated locations, lead consultants were constantly engaged in random spot checks on supervisors and enumerators, visiting some of the beneficiary households where administration of survey tools was ongoing or had already been administered. This was to enhance data quality through monitoring of field data collection and to ascertain that data collectors actually visited identified sample respondents as planned. In few cases, callbacks were made at households where the FEEDS intervention beneficiaries were not met on first visits. However, data collectors were successful in those few cases. At the end of field exercise, the downloaded data from Kobo into MS Excel was subsequently exported into SPSS version 23.0. Consistent with SC data quality protocols, the following steps were taken to clean the quantitative data: ➢ Saved the original (downloaded) excel data file and created a copy of the downloaded data to serve as reference point. ➢ Checked to verify beneficiary participants’ full name, location, and their Segovia ID number given to them for receipt of cash. This was to ensure that data was collected from direct beneficiaries. ➢ Added a filter to the top column of the downloaded dataset and used the filter to check for beneficiary consent status, particularly for those that did not give consent to provide information. This was done to follow up with any data collector who could have collected information without the respondent’s consent. 20 ➢ A further check was done by running frequencies for beneficiary response rates to confirm the sample representativeness, particularly for the calculated minimum sample size that was determined to be adequate for the endline evaluation, so that inferences drawn from the large enough sample size would be meaningful. ➢ Data was then exported into SPSS version 23.0 for further cleaning, followed by analysis. ➢ During data analysis, there were cases where average comprehensive values of indicators needed to be calculated. In those cases, the required portion of the data set was carefully copied into excel from SPSS and subsequently create columns for total, averages and percentages. 2.7 Recruitment and Training of Data Collectors Fifteen experienced data collectors (supervisors and enumerators) who have participated in several prior evaluation data collections, including working with MSD were recruited. A two￾day training, facilitated by lead consultants and one SC FEEDS staff, was conducted for all data collectors, who were trained on the background overview of the FEEDS intervention, evaluation design, sampling techniques used, and evaluation ethics. Facilitators took trainees through each of the data collection tools, and trainees were divided into smaller groups for role-play sessions. This was followed by field pre-testing of the tools. In the role-play sessions and field pre-testing exercises, enumerators were assessed on the practical use of the tools for effective and ethical data collection from respondents. 2.8 Evaluation Ethics Development research ethics were maintained throughout the entire evaluation process. In this way, written or verbal informed consent from evaluation participants was obtained. FGD and individual survey participants were also told the confidentiality of their names and information would not be shared with unauthorized users. Furthermore, participants were instructed by the evaluation team that they (evaluation participants) were expected to be confidential about all that was said during interviews and discussions. In addition, the rights of respondents to end their participation in the evaluation at any point during the course of information gathering was made clear to them. The evaluation team constituted of both males and females to take into account asking questions dealing with sensitive topics where necessary, from which participants’ total anonymity was assured. Linked with SC ethical rules, complaint mechanisms were also established with identified subjects to contact in events where any concerns were to be raised. Although no sensitive cases were identified, referral form templates were prepared for completion by enumerators on sensitive cases or serious child protection issues, to be submitted to supervisors for onward submission to lead consultants and eventually to SC. 2.9 Limitations of the Evaluation 1. The evaluation team used only a 10% estimated change in each of the key outcome indicators during the life span of the project, rather than the targets set by the project. This provided the team with a large minimum required sample size of intervention beneficiaries. The sample size for the evaluation was larger (653) than that used for phase I and III baselines and midline (400 each), but was comparatively smaller than having an endline with 400 households for each of the intervention phases. This was because the calculation using the statistical formula developed by Magnani (1999), and using project determined baseline and target information, resulted in sample sizes that were on one hand too small and on the other hand too large relative to the number of targeted beneficiaries. See Section 2.4.2 for details. 21 2. Chi-Square tests were only conducted for HHS, and t tests for HDDS and rCSI outcome indicators, to test whether there are statistically significant differences between baseline and endline values with respect to these indicators. No statistical tests were run on income and livelihoods data. Furthermore, the baseline datasets did not include Food Consumption Score and Meal Frequency. This limited the evaluation team to conducting similar Chi-Square and t tests to estimate and ascertain whether the associated changes in values of these indicators from baseline to endline were statistically significant. 3. The absence of baseline data for certain outcome indicators made it impossible for the evaluation team to have perform a direct comparison of baseline and endline indicator values. The evaluation could therefore only compare these endline values with national survey data, which likely resulted in an under/over estimation of attributable changes estimated in this report. 4. Field workers were challenged with accessing randomly selected target communities due to poor road networks and crossing of rivers/streams. In these instances, data collectors are strictly advised by MSD Consulting Limited not to cross streams/rivers, so they had to travel long distances to access the randomly selected survey communities and respective beneficiaries, which took more time and resources. 3.0 Analysis and Findings 3.1 Household Demographic Characteristics ● Sex and Age of Household Beneficiary Respondents A total sample of 653 beneficiary respondents were randomly selected and participated in the survey. Figure 1, below, shows that among these respondents, more females were surveyed (72.4%) than their male counterparts (27.6%). Evaluation respondents are thus dominated by females. Figure 1: Percentage Distribution of Sex of Household Respondents In Figure 2, it can be seen that over 98% of all household respondents that participated in this end-line survey fall in the adult age range of 25 to 50 years and beyond. As revealed in the figure below, more respondents within the age range of 25-49 years were selected as FEEDS participants. 22 Figure 2: Percentage Distribution of Age of Household Respondents ● Average Households Size of Beneficiary Participants Overall, the average household size of survey respondents is 5.7. Comparatively, the value is consistent with the Sierra Leone CFSVA (2015) and the 2013 Sierra Leone Demographic Household Survey results at both national and district levels, especially Kailahun District where this end-line evaluation was conducted. ● Educational Level of Household Respondents This is presented in Figure 3 below. Figure 3: Percentage Distribution of Respondents Level of Education It can be seen from the Figure 3 above that the vast majority of household participants have no formal education; over 80% of the respondents in this evaluation fell in the category of participants with no formal education. Figure 3 clearly shows that next to the high proportion of respondents with no formal education are respondents who could only be educated up to primary level (8.3% of all respondents) and others who could attain some form of koranic 23 education (4.9%). Despite the above, a minimal proportion of respondents were found with Junior Secondary School levels of education (3.8%). Of the total respondents, about 64.9% (424) of females and 15.9% (104) of males do not have any formal education. The findings indicate that women remain more vulnerable than men in terms of education out of the FEEDS beneficiary participants. The low level of education among women is evidence of their increased vulnerability to poverty and limited access to resources in these settings. 3.2 Household Socio-economic Characteristics 3.2.1 Income and Livelihood Sources ● Employment Status of Household Heads Figure 4 presents the employment status of household heads. The overall average shows that almost all household heads are either unemployed (57.9%) or self-employed (41.7%). No meaningful evidence of employment in formal institutions is observed in this evaluation. Although evidence of changes that could be attributed to the intervention with regards to a reduction in unemployment and an increase in self-employment of household heads cannot directly be determined in this evaluation, connecting these evidences to baseline findings on sources of household income may yield interesting results that resonate with the above findings. At baseline, most respondents across study chiefdoms were found selling some of their crops, as well as engaging in casual labor and petty trading in the informal markets for dependable income sources. Figure 4: Employment Status of Household Heads ● Household Income Sources The evaluation revealed that beneficiary households depend on diverse sources of income. Presented below are sections of findings on three main (primary, secondary, and tertiary) income sources. Findings presented in this section on proportions of household income used data that was analyzed outside of cash transfers, to ensure comparison of endline and baseline values. Primary Income Sources: Figure 5 presents findings on beneficiary household primary sources of income. As revealed in the figure, crop farming/sales remain the major primary income source among households in the FEEDS intervention coverage area. 89.5% of beneficiaries reported that they rely on crop farming/sales. 24 Figure 5: % of Primary Sources of Beneficiary Household Income Comparatively, the degrees to which beneficiary households rely on crop farming/sales seem to be higher than baseline value of 66%. Following crop farming, are petty trading and casual labor. Overall, petty trading increased from 4.0% at baseline to 5.7% at endline, while casual labor reduced from 9% at baseline to 2.9% at endline. Other primary sources of beneficiary household income include begging, mining, charcoal, etc. Findings could be attributed to the fact that cash received by participant beneficiaries has given households the opportunity to work on their own plots and produce for themselves, rather than having to go out and look for temporary casual labor to get by. “In FGDs conducted with women and men in all randomly selected locations across FEEDS intervention chiefdoms in Kailahun district revealed crop farming (production/sales) as their main occupation and source of household income. Followed by crop farming were petty trading and casual labour, mentioned in particularly Folu, Woloma and Baoma communities in Jawei, Peje West, and Upper Bambara Chiefdoms.” “In FGDs conducted with men and women in Pendembu Njaigbla and Gbaima Koboihun communities in the Njaluahun Chiefdom, and Manowa and Golahun communities in the Peje Bongray Chiefdom, participants emphasized that the main activity that they do in these communities to have their daily living is farming. These included; rice production, garden work, oil palm production, etc.” Secondary Income Sources: The evaluation findings revealed evidence of casual labor, petty trade, and livestock farming apart from crop farming/sales as secondary sources of household income. On average, Figure 6 reveals marked improvement in the reduction of casual labor, reducing from 35% at baseline to 21.3% at endline, while livestock farming increases from 4% at baseline to 12.2% at endline. Petty trading relatively remains the same as a secondary source of income among beneficiary households. Both begging and remittances have been reduced from 2% at baseline to 0.3% at endline. One possible explanation for these findings could be the fact that cash received by household beneficiaries has allowed households to reduce their reliance on temporary casual labor and begging and 25 remittances, while focusing on engaging themselves in livestock and crop farming on their own. Figure 6: % of Secondary Sources of Beneficiary Household Income Although casual labor was mentioned, “When asked about secondary income sources in separate FGDs with FEEDS men and women beneficiaries, participants in all the discussions across the randomly selected FGD locations made much mentioned of petty trading and livestock farming among others.” Tertiary Income Sources: Figure 7a shows that crop farming takes the lead for tertiary beneficiary income sources, followed by casual labor, others, and petty trading. Casual labor specifically grew from 22% at baseline to 28.5% at endline. While petty trading remained relatively the same with a baseline value of 14%, begging has largely reduced from 12% at baseline to 0% at endline. These are complemented by reduced utilization of other diversified sources of income, including charcoal sales and cash for work. A possible explanation for the above evidence could be the fact that with FEEDS cash, households that previously were engaged in begging, charcoal sales, or had no income source have increased rates of temporary casual labor by providing their services to households engaged in farming and trading, thereby increasing farming and petty trading activities among FEEDS-supported households. 26 Figure 7a: % of Tertiary Sources of Beneficiary Household Income Overall Sources of household income: As seen in Figure 7b, the average reliance of households on crop farming/sales, casual labor, petty trading, and livestock farming as the main sources of income has increased. While petty trading and livestock farming marginally increased by 0.7% and 3.1% respectively, crop farming increased largely from the baseline at 32% to an endline value of 58.9%. Concurrently, evidence revealed that casual labor markedly reduced from 22% at baseline to 16% at endline, while other sources of income (begging, firewood and charcoal sales, cash for work, or no income) largely reduced from 18.7% to 6.3% at endline. 27 Figure 7b: % of Overall Sources of Beneficiary Household Income Linking the marked improvement in the degree to which households depend on each of the primary, secondary, and tertiary income sources presented above, there is likely possibility for the evidence to remain associated with contributions from the SC FEEDS cash in the intervention district, causing positive changes in sources of income. As they can be seen to spend a little more on food and agriculture (see Figure 8 below), FEEDS-supported households seem to have eventually been able to have access to better own production and purchase of food from the market in other to meet their food security and nutritional needs. 3.2.2 Normal Household Expenditure Pattern Figure 8 presents the orderly magnitude of average normal household expenditure patterns on food and non-food items. Overall, average expenditure on food is seen to have increased from 44% at baseline to 45.0% at endline. Figure 8 also reveals that average expenditures on non-food items have increased since baseline. In particular, expenditures on education moved from 10% at baseline to 18.3% at endline, expenditure on health care increased from 8% at baseline to 14.7% at endline, expenditures on agriculture went from 7% at baseline to 12% at endline, and expenditure on savings at endline was 4.7% compared to 0% at baseline. Debt repayment and transportation reduced from baseline values of 11% and 5% down to 1.7% and 2.7% respectively. Expenses on other items (including household items, rent/shelter, gifts, etc) have also reduced from 15% at baseline to 1% at endline. The reduction in expenditure on debt repayments and other items could be possibly explained due to households spending funds on agriculture, food, or savings to improve on their diet and food intake. 28 The increases in expenditure on the above non-food items could be attributed to the fact that at baseline, while movement was restricted during Ebola, households were recovering from reduced access to their farmlands. Once the restriction was over, participants’ household engagement in crop farming for their own production of food has once again increased. While diverting excess funds available to expenditure on non-food items, beneficiary households continued to have a 1% increase in food expenses, which seemed to be used as part of inputs for a more diversified food production. Figure 8: % of Average Normal Household Expenditure Pattern 3.3 FEEDS Intervention Outcomes This section presents evaluation findings on key outcomes indicators of the FEEDS intervention. These are given in averages, and include the Household Dietary Diversity Score (HDDS), Household Hunger Scale (HHS), Food Consumption Score (FCS), reduced Coping Strategies Index (rCSI), and the Meal Frequency. 3.3.1 Household Dietary Diversity Score (HDDS) Household dietary diversity is defined as the number of food groups consumed by members of a given household within a 24 hour period. This gives an indication of the household’s access to food. Respondents were asked to recall all the different foods that household 29 members had consumed within the last 24 hours prior to the survey. These different foods were grouped into the standard 12 food groups used in the calculation of the HDDS. The maximum possible score is 12, which would indicate that a household ate all the 12 different food groups within the recall period. Figure 9 presents evaluation findings relating to the average household dietary diversity score of the target beneficiaries. It can be seen that at baseline the average score was 4.5, which the FEEDS intervention aimed at increasing to 5. At the endline, the evaluation found that the average HDDS for all households was 6.3. Figure 9: Average Household Dietary Diversity Score (HDDS) Results from an Independent Sample T-test of statistical significance conducted on average household dietary diversity score revealed a statistically significant (p<0.001) difference between baseline and endline values with respect to average HDDS. This is because the 1- tailed significance value (p<0.001) is far less than the significance level 0.05 measured at 95% confidence level. See appendix A. These results show a good achievement relative to the intended target. From a review of PDM reports, program field staff intermittently conducted nutrition sensitive sensitizations and awareness-raising campaigns across all target communities. These activities may have also contributed to the good achievements noted on the HDDS. The different kinds of food groups that each beneficiary household consumed in the last 24 hours prior to including them into the survey were analyzed. Figure 10 reveals the proportions at which beneficiaries consumed the following food groups at endline: cereals at 97.1%, fish at 97.2%, and roots and tubers at 90.8%. These are followed by more than 77% of fruits and vegetables each, and a minimum of 41% for both condiments and pulses. Over one-third of participants reported to have consumed fats and oils as well as sugar or honey. Other food groups that had limited consumption among all households included meat, poultry and other organ meats, dairy products, and eggs. From KIIs with CTC members and SC program staff, it was revealed that SC field staff intermittently conducted nutrition-sensitive messaging in the target communities. This finding was confirmed by FGDs with participants, who reported that these sensitizations played a role in helping them diversify their foods, as seen in the values reported above. 30 Figure 10: % of Respondents that consumed each Household Dietary Diversity Score food group (Multiple Responses) In a similar manner, Chi-Square tests conducted on all individual food groups revealed that the difference between baseline and endline values with respect to all other individual food groups are statistically significant (p≤0.002), except for condiments and fat/oil, for which results showed existence of statistically insignificant differences between the baseline and endline for the two groups. See appendix A on results from statistical tests of significance. The test results, apart from the exceptions of condiments and fat/oil, are considered statistically significant because the individual p-values are each far less than the significance level (0.05). 3.3.2 Household Hunger Scale (HHS). The Household Hunger Scale is a particular household food insecurity indicator used to measure household hunger in food insecure areas. This indicator was determined using information provided by beneficiary respondents in response to questions on the extent to which, in the past one month or four weeks:  Household members had no food to eat at all in the house  Household members go to sleep at night hungry  Household members go a whole day and night without eating anything at all because there was not enough food in the house. 31 Analysis of results from household responses to these three parameter questions is presented in Figure 11 below. These have been categorized into three hunger levels: ‘Severe hunger’, ‘Moderate hunger’, and ‘Little to no hunger’ among beneficiary households. Figure 11: % of Average Household Hunger Scale Severe hunger among all households reduced from 1.0% at baseline to 0.5% at endline (this is supported by the statistical test result). A second data collection exercise would have been necessary to investigate why 0.5% of households remain severely hungry, but time and resources could not permit the evaluation team to revisit these households to further investigate these results. In addition, the average moderate hunger among beneficiary households significantly reduced from 68% at baseline to 34.3% at endline, and on average, little to no hunger showed marked increase from 32% at baseline to 65.2% at endline. The consistent reductions in both moderate and severe hunger resulted in the marked increase in percentages of households with little to no hunger, from 32.0% at baseline to endline value of 65.2%. A very limited proportion of beneficiary households continued to face severe hunger situations in FEEDS intervention target areas. The marked reduction of the proportion of beneficiary participants facing severe and moderate hunger (which was the basis of their selection at baseline), accompanied by a noticeable increase in the proportion of households with little to no hunger, could be attributed to contributions by the FEEDS intervention, which in effect demonstrates value for money with regards to intervention. These findings are supported by Chi-Square tests (presented in appendix A) conducted on three HHS parameters. It was revealed that the difference between baseline and endline values with respect to the three parameters (given below) were statistically significant because the significance values (p<0.001) for the parameters are each less than the significance level (0.05) measured at 95% confidence level: ● The extent to which household members in the past four weeks go a whole day and night without eating anything at all because there was not enough food in the house ● The extent to which household members in the past four weeks have no food of any kind to eat at all because of lack of food in the house ● The extent to which household members in the past four weeks go to sleep at night hungry because there was not enough food in the house. 32 3.3.3 Household Food Consumption Score (FCS) The Food Consumption Score (FCS) is mostly used as a proxy indicator for access to food. It is a weighted score based on dietary diversity, food frequency, and the nutritional importance of food groups consumed by households. The data for this score was collected by asking respondents to recall the number of days in the last 7 days preceding the survey that the household ate different food types. These food types were then grouped into the standard 12 food groups for the calculation of FCS. The higher the score, the better the food security situation of the household. In the FEEDS intervention, this indicator was not monitored and so there was no baseline value to measure it against. Results from the endline are presented here without comparison to the baseline or midline surveys. As shown in Table 2, the Comprehensive Food Security and Vulnerability Analysis (CFSVA) report 2015, conducted by WFP, gives international benchmarks for poor, borderline, and acceptable food consumption scores. Table 2: Food Consumption Score Benchmarks Food Consumption Profile Food Consumption Score Poor 1 – 21 Borderline › 21.5 – 35 Acceptable › 35 Source: WFP’s CFSVA Report 2015 When the evaluation consumption scores were compared with these international benchmarks, it was found that the overall average household score of 52.7 was far above the minimum acceptable consumption level of 35+ (see Figure 12). Figure 12: Average Household Food Consumption Score The households that participated in the FEEDS project were selected after a rigorous poverty and vulnerability screening exercise for phases I and II. Save the Children used a participatory targeting approach to identify these beneficiaries, and community members were directly responsible for selecting eligible HHs as per the targeting criteria through a public and transparent process. In phase III, the FEEDS program used the HEA baseline and outcome analysis data extensively to re-structure the targeting, community selection, and beneficiary registration procedures. The steps in the process used for the re-targeting were geographical targeting to identify which communities were to participate, community ranking (identification by community members of HH matching the selection criteria), scoring - which implies that all identified HH were pre-registered and surveyed (scoring tool integrated to pre-registration questionnaire), selection, which is ranking and selection of beneficiary households based on scoring tool (higher scores), and verification of the HH information for those with high scores. 33 One would assume that household food consumption scores were poor, since only 3.3% of all households in Kailahun were food secure and another 26.1% were marginally food secure (CFSVA, 2015). Therefore, the evaluation concluded that the FEEDS intervention has contributed immensely to improving the food consumption score of target households. 3.3.4 Household Reduced Coping Strategies Index (rCSI) Another key indicator the evaluation assessed was the reduced Coping Strategies Index (rCSI), which measures the frequency of different strategies adopted by households to cope with shortages of food, and calculates a score weighted by the severity of each strategy. In Figure 13 (below), it can be seen that at baseline, this index was 25 out of a possible index score of 56; for the rCSI, the lower the score the better. The FEEDS intervention aimed at reducing this score from 25 to 23. The endline evaluation found an average household coping strategies index to be 17.5. This is far below the FEEDS intervention target household coping strategies index of 23.0, which shows a marked achievement in reducing negative coping strategies for households in FEEDS intervention coverage areas (this claim is supported by the statistical test result given in appendix A). From KII with program staff, it was found that Save the Children, during their nutrition sensitizations to beneficiaries, always stressed the fact that households should not employ the coping strategy of starving adults so children could eat. Save the Children’s point here was that if adults are starving continuously to let children eat, there will come a time when they will not be healthy enough to provide for the children and the whole household will be in food security crisis. The target households may have adopted this message in their coping strategies, so that both adults and children always have something to eat. Figure 13: Average Household reduced Coping Strategies Index The findings above are supported by results from a one-tailed Independent Sample T-test of statistical significance conducted at 5% degree of accuracy and 95% confidence level on average of the five categories/parameter measurement of reduced coping strategies index (rCSI) outlined below, revealed that there is a statistically significant (p<0.001) difference between baseline and endline values in respect to average rCSI. This is because the 1-tailed significance value (p<0.001) is be far less than the significance level (0.05). See appendix A. ● How many days did your household have to rely on less preferred foods or less expensive food? ● How many days did your household have to borrow food or rely on help from friends and relatives? ● How many days did your household have to reduce the portion size of meals? 34 ● How many days did your household have to reduce the quantities of food consumed by adults so that young children could eat? ● How many days did your household have to reduce the number of meals eaten per day? 3.3.5 Average Household Meal Frequency Table 3 presents findings relating to the number of meals eaten by target households in the last 24 hours before they were included in the survey. This is another key measure of food access at the household level. The data was disaggregated by age to present the complete picture of the household meal situation in the last 24 hours before the survey. It was found that, on average, every household member ate more than one meal within that last 24 hours prior to the survey. Meal frequency was not computed at baseline. However, noting the criteria used to select the target beneficiaries, and taking into consideration the results of the reduced coping strategies index measured at the baseline and throughout the program, it is known that some households reduced the frequency of meals, at least for specific household members, in order to cope with reduced access to food. At endline, findings suggest that every household had a secured one meal plus within the last 24hrs before the survey. Table 3: Meal Frequency (for last 24hrs) All Total Children less than 5yrs 1.7 Children 5-17yrs 1.8 Adults 18yrs and above 1.7 Overall Total 1.7 3.3.6 Utilization of FEEDS Cash by Beneficiaries The evaluation found that a total of 91,154 unconditional cash transfer participants were targeted by the FEEDS project, distributing a total cash amount of approximately $4.9 million. Figure 14 presents the orderly magnitude of the average proportion of FEEDS cash expenditure patterns on food and non-food items. Overall, 44.3% of cash received was spent on food, followed by education, health care, agriculture, and savings at 18%, 15%, 12% and 5.7% respectively. This stresses the fact that in vulnerable communities like those targeted by the FEEDS project, households spend a quite a large portion of their income on food. However, the FEEDS cash may have given them some flexibility to spend on non-food items. Although food remains the major item for household expenditure, the above expenditure pattern of FEEDS beneficiaries is consistent with the theoretical fact that once income of households increases, members are prompted to also prioritize other expenditure lines that are important for their health and welfare. These in effect account for strengthening vulnerable households’ human and financial capital, which are necessary for future sustainability of the FEEDS intervention. 35 Figure 14: % of FEEDS Cash Utilized on Food and Non-Food Items Types of Foods Purchased with FEEDS Cash Figure 15 presents findings related to the different food items on which households spent the cash received from FEEDS support. On a whole, it can be seen that in all phases, a minimum of 90% of beneficiary households spend the cash on rice, fish, vegetable oils/palm oil, and condiments. Other key food items bought frequently were tubers/roots (cassava, potatoes, etc.), sugar/honey, and pulses (beans, pears, lentils, etc.). Comparatively, vegetables, meat, fruits, and eggs seem to attract less attention for purchase by households. This is because these food items are generally produced locally by the households themselves. 36 Figure 15: % of Food types Purchased with FEEDS Cash (Multiple Responses) 3.4 Capacity Supports to FEEDS Cash Beneficiaries 3.4.1 Capacity Supports to Traders The evaluation conducted several Focus Group Discussions (FGDs) with traders, primarily women, who were supported to do petty trading in their communities. The evaluation found that the FEEDS project supported these women with direct cash and capacity building training in Business/Financial management. Findings from the FGDs are summarized below. ● Cash Support The evaluation found that all the female traders who participated in the FGDs reported that they were given cash by the FEEDS project to do business. Most of them were already doing petty trade in their various communities (though according to them their capital had drastically dwindled) and some had gone out of business due to the Ebola Virus Disease (EVD) outbreak. All of them reported that they were given two tranches of cash to support their businesses. “Yes, we were given money by Save the Children to start our businesses again after we lost all the little monies we had during the EVD.” [FGD Participant, Baiwala Community, Mandu Chiefdom] 37 This evaluation therefore concludes that the FEEDS intervention supported female traders with cash to continue or restart their businesses. A total of approximately $137,878 was disbursed to 400 traders (399 female and 1 male) in two tranches across the project target communities. ● Capacity Building Trainings The evaluation also examined whether the female traders were given any form of capacity building training in business/financial management. Findings suggest that most (not all) of the traders who participated in the FGDs had received training from Save the Children in business and financial management. According to most of the traders, these trainings were conducted before they were given the cash, although for some the trainings were conducted after the cash was given to them. Whichever way it was done, results found that the duration between the receipt of cash and training was not long enough to affect the activity intent either by using the cash incorrectly before the training came or forgetting the training before the cash came. “Because of the training given to us by Save the Children, we have now got some business techniques like writing report on all our transactions” [FGD Participant, Kambama Community, Jawei Chiefdom] “The training has taught us to be able to engage in the right business good for us” [FGD Participant, Manowa Community, Peje Bongrey Chiefdom] A total of 400 traders were provided conditional cash grants and trainings were held across the target communities. ● Impact of Cash and Capacity Building Training on Trade The evaluation assessed the opinion of traders about how the cash and capacity building training affected their trade volumes and subsequently their profit margins. It found that all (100%) of traders reported that the cash and training support boosted their trade volumes and, therefore, profit margins. Traders reported that once they started receiving the cash from the FEEDS project, they were able to increase their stocking capacity of basic food stuff. Some even reported having used part of the cash to improve their storage facilities, in order to accommodate the increments in their stock. During implementation, the program measured the value of incremental sales recorded by traders and found that, within 6 to 9 months, sales had increased by $248,839, implying that for every $1 received $2 was generated. However, the evaluation was not able to quantitatively confirm these claims because there was no monitoring data that had been tracking changes in the traders’ stock level over time after this initial survey. However, findings from FGDs with participants who were not traders, revealed that most of their basic foodstuff needs were met by their community markets, which was not the case before the FEEDS intervention according to the baseline findings. Most traders also reported that the FEEDS intervention (unconditional cash transfer to households) increased demand for basic food stuff in their communities. Since traders were also simultaneously capacitated, most reported that they now not only buy commodities in their towns (when vendors bring them), but also go out to periodic markets outside their communities to buy goods and restock. According to most, this helped to increase their profit margins further and made most required basic food stuffs in their local markets available. “The cash support has also helped us go out and buy goods to sell in our community here, which people now buy.” [FGD Participant, Manowa Community, Peje Bongrey Chiefdom] “It has also help us to buy in large quantity now. Before this time, we were unable to buy goods in large quantity because of limited money. But with the intervention of Save the Children, we 38 now buy goods in large quantity to sell in this community. We say thanks to Save the Children.” [FGD Participant, Kambama Community, Jawei Chiefdom] “The demand for basic food stuff has increased because all what people were going out to buy are now with us here in the community.” [FGD Participant, Gbama Koboihun Community, Njaluahun Chiefdom] “Demand has now increased significantly. They now come to buy and not to loan from us.” [FGD Participant, Nyanyahun Community, Upper Bambara Chiefdom] The evaluation concluded that the FEEDS support to female traders helped restore their businesses (and for some, even expanded it). This had a positive ripple effect on the communities in terms of availability of basic food commodities in most local markets. 3.4.2 Capacity Supports to Women in Agriculture Groups (WAGs) Like the support to female traders, the evaluation conducted several Focus Group Discussions (FGDs) with members of the WAGs to assess the support given to them and to see how it helped them in their agricultural activities. The evaluation found that the FEEDS project supported these women with direct cash and capacity building training in organizational management, improved agricultural techniques, and basic business skills. Findings from the FGDs are summarized below. ● Capacity Building Trainings The evaluation found that all the WAG members who participated in the FGDs were jointly trained by Save the Children and MAFFS Kailahun District. According to the participants, some of the topics covered included: ● Farming as a business ● Development of business plans ● Conflict management ● Leadership principles ● Record keeping ● Group formation and the different responsibilities of leaders The above topics were consistent with the planned training topics except for one: the component that was supposed to deal with “improved agricultural techniques”, meaning the focus of the capacity building was on the business aspect of agriculture (farming as a business). ● Cash Support The evaluation found that the FEEDS intervention gave out cash to the members of the eighteen WAGs across the target chiefdoms in Kailahun district. According to participants, they used the cash to buy agricultural inputs, especially seed and payment for labor. Few of them reported having used the money on some non-agricultural items like school fees and medication. “Yes, we received cash from Save the Children to do our farm work. We used the money to buy seeds, fertilizer and food for the workers.” [FGD Participant, Baoma Community, Luawa Chiefdom] 39 “Save the Children gave us a specific amount and told us to use it for our farming activities to expand the size of land we cultivate. We thank them very much.” [FGD Participant, Kambama Community, Jawei Chiefdom] ● Impact of Cash and Capacity Building Training on Cultivation All the participants said the cash and training support helped them to do better farming than they were doing before. On further examination, they some said this was the first time their households had been able to cultivate all the land that belonged to them. “Before now, we were just cultivating part of our land, because we don’t have money to pay labour and buy enough seeds. But after we received the cash from Save the Children, we cultivated all the land.” [FGD Participant, Golahun Community, Peje Bongrey Chiefdom] Others said the cash support also helped them to get access to quality rice and vegetable seeds, which they planted. For some other participants, they said the knowledge they gained has helped their group become more united than before. “With the training, we all now know our roles and responsibilities in the group. So, there is peace now and we have focused on our objectives.” [FGD Participant, Pendembu Njaigbla Community, Njaluahun Chiefdom] As part of the impact of the FEEDS support to the WAGs, some participants reported that they have now become role models in their communities that other women are copying from. “We have become role models to other women who are now trying to form their own women’s groups.” [FGD Participant, Baiwala Community, Mandu Chiefdom] The evaluation therefore concludes that the cash and capacity building training support given to the WAGs had a positive impact on the operations of their groups, as well as individually. Additionally, there has been some spillover effect on the community, which has been motivated to start more women groups. 3.5 Effectiveness of Implementation - Cash Transfer Processes Cash interventions are becoming increasingly popular. As such, it is necessary that an assessment be made at the end of an intervention of not only the food security indicators that the program sought to address, but also the process that led to the changes observed in the indicators. The evaluation therefore looked at four key aspects of the cash transfer process, and the findings are presented below. ● Satisfaction with Distance Travelled to Pay Points The evaluation attempted to understand how the beneficiaries felt about the distances they travelled to access the various pay points. Table 4 shows that 88.8% of the beneficiaries reported they were satisfied with the distances they covered to access their pay points. The table also shows that 54.7% of the beneficiaries walked less than 1 mile to access their pay points, while 27.4% walked between 1 and 2 miles. Only a small proportion (3.8%) of beneficiaries reported that they walked above 3 miles to access their pay points. This was a reasonable distance to cover to a pay point, taking cognizance of the target terrains, which means the FEEDS project positioned enough pay points to avoid beneficiaries being forced to walk long distances. “The distance between the paying points to the communities is around 2 kilometres on average, which is less than 40 minutes walking distance.” [KII with town chief, Baiwala]. 40 Table 4: Satisfaction with Distance Travelled to Pay Points Total (n=653) % How far do you travel to access the FEEDS pay point? Less than 1miles 357 54.7 1miles - 2miles 179 27.4 2.1miles - 3miles 92 14.1 Above 3miles 25 3.8 Are you satisfied with the distance travelled to the pay point? No 73 11.2 Yes 580 88.8 ● Adequacy of Cash Transferred The evaluation assessed the perception of beneficiaries regarding the adequacy of the cash amount the FEEDS project was giving them to cover their food need for three months. Table 5 shows that 62.3% of all beneficiaries held the perception that the cash given to them was not adequate to cover all their food needs for a three month period,2 leaving only 37.7% who reported that the cash they received was adequate for their food needs. The reasons given by most households for holding the view that the cash amount was not adequate to cover their three month food needs were large family size and the rapid increase in food stuff prices during the cash transfer period. However, the evaluation found that almost all households were also spending the cash received on several other things including health, education, transportation, and agriculture, meaning the amount left to be spent on food would be lower than expected. Additionally, it was noted there was some misunderstanding on the part of beneficiaries, because the FEEDS intention was that the cash was meant to cover only 40% of the household food needs, not all their food needs for the three months period. Table 5: Adequacy of Beneficiary Cash Amount Do you consider the amount of cash you receive adequate for your household food needs for three months? Frequency % Yes 246 37.7 No 407 62.3 Total 653 100.0 ● Satisfaction with Waiting Time to Receive Cash The evaluation also examined the opinion of beneficiaries on the time taken at the pay point to receive their cash. Table 6 shows that almost all beneficiaries (95.7%) reported that they were satisfied with the length of time taken at the pay points to receive cash. It was found that 42.3% of all beneficiaries were spending less than 30 minutes at pay points to receive cash, while 27.7% spent between 30 to 59 minutes and 25% spent between 1 to 2 hours. Only a small fraction of beneficiaries (5.1%) reported that they were waiting more than for 2 hours or 2 Outside of social safety net transfers, transfers were intended to cover 40% of a households monthly food needs, distributed in tranches of two or three months of assistance at a time. Therefore it is reasonable that the cash did not meet all household food needs, but further information would be required to understand this finding. 41 more at pay points before they received their cash. This indicates that the activities that lead to the actual transfer of cash from SPLASH to the beneficiaries was fast enough to avoid significant delays at the pay point. Table 6: Satisfaction with Waiting Time to Receive Cash Total (n=653) % How long do you wait to receive the cash pay point? Less than 30 min 276 42.3 30 min - 59 min 181 27.7 1hr - 2hrs 163 25.0 2 hrs. and above 33 5.1 Are you satisfied with the waiting time for you to receive the cash at the pay point? No 28 4.3 Yes 625 95.7 ● Satisfaction with Safety at Pay Points Cash is attractive to all, including criminals. The evaluation therefore investigated the views of beneficiaries regarding their safety when they were at the pay points to collect cash, as well as their safety after leaving. The evaluation also revealed beneficiary’s opinions regarding any pressure from other sources, including local authorities, politicians, or CTC. Table 14 shows that 97.4% of all beneficiaries reported that they felt safe when they came to collect their cash at the pay points. After leaving the pay point with their cash, the table shows that almost all of them (95.1%) still felt safe as they returned to their respective communities. Regarding pressure or harassments from local leaders or CTC members to give them part of their cash, results indicate that this almost never happened (99.4%) as reported by beneficiaries. These were good indications of safety. “We feel safe. The payment is normally done at the town barray which is located at the center of town. This makes it safe for all of us.” [FGD with beneficiaries, Pendembu Njeigbla]. Table 7: Satisfaction with Safety at Pay Points Total (n=653) % When you are at the pay point, do you feel safe? No 17 2.6 Yes 636 97.4 After leaving the pay point with cash, do you still feel safe? No 32 4.9 Yes 621 95.1 Have you ever been pressured by local authorities (Chiefs, Councilors); CICs, extended family or community members to give money you did not want to give? No 649 99.4 Yes 4 0.6 3.6 Appropriateness of Project Design 42 The FEEDS project was designed to primarily be a direct unconditional cash support project to vulnerable food insecure households affected by EVD in the Kailahun district. The overall objective was to increase the speed at which these households would recover from the shock of EVD. The strategy was to ensure that all targeted households had continued access to nutritious food, especially throughout the lean season, so that they could reduce recourse to negative coping strategies. The evaluation noted that these design strategies were all appropriate in enhancing the achievement of project results. Cash transfer projects are cost effective to implement since they do not require too many program activities, and the transport and logistics costs associated with other alternatives are eliminated. Additionally, cash transfers (especially when made unrestricted) allow beneficiaries a great deal of latitude to choose what to buy. This brings about some sense of dignity and control over household expenditures, which is a basic duty of a household head. Thirdly, the design also allows for greater nutrition-sensitive messaging and messaging on how to utilize the cash received to improve their food security situation and to cope with any future shock. To this end, the staff were based directly in the chiefdoms to implement the activities associated with this design aspect. Though the project was largely based on unconditional cash transfers, there were two conditional complementary activities implemented during the program: conditional cash grants to traders (primarily female) and conditional transfers to Women in Agriculture Groups (WAGs). To receive cash assistance, these beneficiaries were required to attend trainings and demonstrate application of the skills and methods. This design aspect was very apt because it emphasizes the important aspect that food security can only be sustainably realized through enhancing a households or a nations own food production. 3.7 Intervention Relevance and Strategic Fit The Ebola Virus Disease (EVD) entered Sierra Leone through Kailahun district. From all indications, it took the government unawares, with no response plan in place, which is why the virus killed many people in Kailahun district and stayed there for a longer period than most other parts of the country that it affected. It was here that government and its emergency response partners first imposed quarantines. Whole communities and sometime whole chiefdoms were put under quarantine, disrupting farming activities, markets, schools, and all other social activities. This had a prolonged negative impact on the livelihoods of people in Kailahun district, and created a shock that was reported in the baseline report as follows: “The Prevalence of households with moderate or severe hunger in Kailahun district is 69%”. The Reduced Coping Strategies Index score in Kailahun district is 25.” These two indicators showed that there was a high level of food insecurity in the district at the time the FEEDS baseline was conducted. Therefore, an intervention aimed at improving continuous access to nutritious foods by target households was relevant to Kailahun district. The beneficiaries themselves echoed the relevance of FEEDS project; almost all of the beneficiaries said if the FEEDS project had not come to their aid after EVD, they would have coped with life in an extremely difficult manner, which highlights the relevance of the FEEDS project to the target beneficiaries. The current strategic plan of Save the Children in Sierra Leone sets out its objectives in five thematic areas: Health and Nutrition, Education, Child Poverty, Child Protection, and Child Rights Governance. The evaluation found that the FEEDS project strategically fit into the first thematic area, Health and Nutrition. The Strategic Goal 1 and Strategic Objective 1 of USAID states, “Strengthen America’s Economic Reach and Positive Economic Impact” and “Promote Inclusive Economic Growth, Reduce Extreme Poverty and Improve Food Security.” The 43 evaluation found that the FEEDS project fit into these two strategic guiding statements of the donor, USAID. Following the end of EVD, the government of Sierra Leone drew up its National Ebola Recovery Plan (2015 – 2017) to guide and coordinate all recovery efforts. One of the key objectives of the plan was, “Managing and Mitigating Immediate Ebola Impact in the Social Sector, Gender, Children, and Social Protection”. In order to achieve the objectives of the recovery plan, the government formulated certain strategies, including “Provide livelihood support for EVD affected children, women, orphans, widows, widowers, and the elderly and disabled” and “Provide cash transfers to poor households and vulnerable groups, therefore benefitting local economies.” The FEEDS project undertook cash transfers to vulnerable EVD affected households and supported women to do business and agriculture (which are livelihood activities). The evaluation therefore concluded that the FEEDS project also directly fit into the government of Sierra Leone’s strategic plan for recovering from the shock of EVD, thus making it relevant at national level. 3.8 Resource Mobilization and Efficiency of FEEDS Implementation The evaluation found that all three categories of resources (human, material, and financial) were mobilized within the required time frame and were adequate. The FEEDS project recruited staff for all required positions; field staff were based in all targeted chiefdoms and support staff were based in the Kailahun field office. The field staff were in direct contact with beneficiaries on a day-to-day basis to implement and monitor project activities, such as market prices, cash usage, and supporting MAFFS in conducting improved agriculture trainings. The field office in Kailahun provided backstopping support to the field staff in all their activities, including liaising with MAFFS and the country office in Freetown. Because there was adequate staff for the required project activities, the evaluation did not find instances of staff being over￾stretched to cover large areas of implementation, or moved from one chiefdom to the other to support because of work overload. The evaluation believes that the human resource of the FEEDS project was wisely utilized. The evaluation did not look at budgets and expenditure documents because of limited time. However, a review of the activity implementation against planned timelines revealed that almost all activities were implemented within the planned timeline, and there was no instance of activity backlogs due to lack of funds. All staff were paid their remunerations on time. All planned cash transfers (across all three phases) were completed and the required adjustments due to price changes were all fully accommodated without a break. The evaluation noted that there were no activities implemented outside the planned budgets, indicating that FEEDS project finances were wisely utilized to achieve the desired results. Materials resources mobilization and utilization were also briefly assessed. The evaluation did not look at the project assets register, but found that adequate materials were mobilized to facilitate implementation: all staff required to use computers had them, motor bikes were pooled for field trips, and the Kailahun field office had a photocopier and printers assigned to the FEEDS project. At the time of evaluation, all of this equipment were in good working order. Regarding alternative approaches to get the same results, it was found that an online money transfer would have been better and faster. However, the availability of reliable mobile networks to support mobile transfers was a problem. Additionally, the number of target households with mobile phones was very low (6%) at baseline, meaning overall technology literacy was low. Therefore, the approach used was the most feasible under the circumstances. It is the conviction of the evaluation team that the FEEDS project materials were adequate and were utilized efficiently in the implementation of the project. 44 3.9 Partnerships, Collaborations and Coordination The evaluation assessed how Save the Children related with other state and non-state actors in the implementation of the FEEDS project. This section presents the findings. The evaluation noted that Save the Children was the sole implementing partner of the FEEDS project, but they collaborated with several agencies to increase efficiency and effectiveness. The National Commission for Social Action (NaCSA) was brought on board to support beneficiary targeting at the community level. This was because NaCSA has a long-standing reputation for supporting vulnerable households through cash transfers and other interventions. In phases I and II, SC conducted registration using vulnerability criteria determined with communities, and in consultation with NaCSA, in order to select the most vulnerable Ebola-affected households. However, in phase III, Save the Children used the HEA to select target beneficiaries. NaCSA also trained Save the Children staff in the appropriate use of this tool. This was a capacity building activity conducted by NaCSA. In addition, the evaluation noted NaCSA also intermittently participated in joint monitoring activities of the project. Cash handling and transfer was outsourced to a leading mobile money transfer agency, SPLASH Mobile Money Limited. SPLASH specializes in both electronic and manual money transfer services and has operated in Sierra Leone for quite some time. Their experience with the local terrain was heavily leveraged upon and enabled the necessary speed and accuracy to the cash transfer process. Their effectiveness was confirmed by beneficiaries in focus group discussions, most of whom said they waited for less than 30 minutes at pay points before receiving cash. The evaluation also confirmed from beneficiaries that they had not been experiencing shortages of cash transferred to them, because “the SPLASH and Save the Children staff always tell us to count our cash before leaving the pay point.” The Anti-Corruption Commission (ACC), the government agency responsible for fighting corruption in Sierra Leone, collaborated with Save the Children to address issues of complaints (from both beneficiaries and non-beneficiaries) and to monitor the cash transfer process across the board. The evaluation found that the ACC carried out a wide range of sensitizations on corruption issues, its dangers, and how/why to prevent it. The evaluation noted they were sometimes present at cash distribution points to get first-hand information from beneficiaries. This was a very good strategy to add integrity to the process, and ACC has built a strong reputation for fighting corruption over the years despite their own challenges. The Ministry of Agriculture Forestry and Food Security (MAFFS), Kailahun district was brought on board to support the Women in Agricultural Groups (WAGs) with a series of trainings in improved agricultural techniques. The evaluation noted that these trainings, conducted jointly by MAFFS and Save the Children, enhanced the self-food production capacities of the WAGs members and were noted to be one of the two key sustainability activities incorporated into the project design. Other key actors that Save the Children collaborated with included the Kailahun District Local Council, the Ministry of Social Welfare, Gender and Children’s Affairs, and the Presidential Recovery Team Kailahun district. The evaluation found that these other agencies were involved with intermittent joint monitoring on the project. Regarding linkages of the FEEDS project with other Save the Children projects, the evaluation found that the nutrition-sensitive awareness raising messages that were sent out integrated with health, child safeguarding, education, and child rights governance messages, thus making the FEEDS project interface with all the other Save the Children thematic areas. However, there was limited involvement of the private sector, as only SPLASH was involved. 45 3.10 Mainstreaming Gender and Sustainability in FEEDS Project Activities ● Gender Mainstreaming The evaluation assessed how gender was integrated into the project design and how that played out on the ground during implementation. The evaluation found that, at design stage, key gender issues were incorporated into the project. The design made way for a minimum of 50% of all beneficiaries to be women. In phase I, the design made way for targeting traders (mainly females) with cash support and capacity￾building trainings to continue and even expand their businesses. In phase III, the design targeted Women in Agriculture groups with cash and capacity building trainings to improve their agricultural activities. In these ways, the design allowed most of the FEEDS support, especially the cash for the household, to be put into the hands of more women than men. The project did not have a dedicated gender focal person at the field level in Kailahun. However, the gender coordinator based in the country office in Freetown provided relevant trainings and intermittent field visits during the course of implementation. Additionally, the program staff who were assigned to different chiefdoms also sensitized the target households about the benefits of joint decision-making regarding the FEEDS cash utilization, an aspect of gender equality that probably helped in mitigating intra-household conflicts. Figure 16, below, presents findings relating to how decisions were made about how to spend the FEEDS cash. Most decisions about how to spend the cash were made solely by those who received the cash themselves (67.2%). Even though this was against the notion of joint decision making by key household members (preferably the husband and wife), it was good for female beneficiaries as it means most of them had the authority to decide on how to spend the money, which was a form of women empowerment. Nevertheless, joint decision making about how to spend the cash was observed in an appreciable number of households (19.3%), compared to decisions made by head of households alone. Figure 16: % of Household Member Participation in Decision Making ● Sustainability of Project Activities/Achievement The FEEDS project main activity was direct cash transfers to vulnerable, food insecure households. The evaluation did not find any current measures taken by any organization or agency to sustain this, even though there are plans by Save the Children to handover the FEEDS beneficiaries to NaCSA. However, NaCSA currently operates in only 3 of the 14 chiefdoms in Kailahun district, and an immediate scaling up by NaCSA is not in sight. There are also plans to hand over the WAGs to MAFFS for continuation of project activities. Again, 46 MAFFS capability to sustain these activities (conducting trainings, providing regular extension services at their own expense) is not currently available. However, the complimentary activities provided a good sustainability framework. The evaluation found that, through the support of Save the Children and the other stakeholders (USAID, MAFFS etc.), beneficiaries have made investments in petty trading, VSL, and increments in land areas currently cultivated. With their enhanced knowledge of organization management including conflict management, roles and responsibilities, record keeping, business planning, and good agronomic practices skills such as nursery management, transplanting, spacing, weed management, integrated pest management, soil and water management, harvesting and post-harvest management, beneficiaries self-food production capabilities will be enhanced long after the end of the FEEDS project, which will hopefully prepare them for any other shocks in the short or long term. 3.11 Monitoring, Evaluation, Accountability and Learning The evaluation found that the FEEDS project was monitored both internally and externally. Internally, the field staff assigned to the chiefdoms were responsible for first line monitoring of project activities. They conducted post distribution monitoring (PDMs) exercises after each phase of cash distribution. These were meant to get feedback from beneficiaries about several issues, including their satisfaction with the adequacy of cash amount given to them to cover their food needs for three months, the waiting time at pay points before receiving cash, distances covered to access the pay point, cash utilization to ensure they were spending more on food, etc. The evaluation found that these monitoring exercises were conducted regularly and feedback was shared with the Kailahun field office, which in turn shared it with the MEAL team in the Freetown country office. Staff also conducted regular monthly market price monitoring of basic foodstuffs to ensure the purchasing power of beneficiaries was not seriously affected by price increases. This data was also shared with the respective offices for detail analysis. The program staff based in the field office in Kailahun, and the MEAL team in Freetown, provided direct monitoring support to the field staff. The Save the Children MEAL team based in the USA also provided remote monitoring support from time to time. The evaluation noted that this approach made internal monitoring more robust. At the external level, the evaluation found that the FEEDS project had several monitoring mechanisms. The first mechanism was the Community Feedback and Response Mechanism (CFRM). This was set to provide community members (beneficiaries and non-beneficiaries), including children, with access to a safe and confidential means of passing feedback and complaints on issues relating to the FEEDS implementation. Another monitoring mechanism employed for external monitoring of the project was the ACC Grievance Redress Mechanism (GRM). This mechanism allowed beneficiaries and non-beneficiaries to call directly on a toll￾free line and raise any issues of concern relating to the FEEDS project. These concerns and complaints were then collated and analyzed by the respective organizations and shared with stakeholders for decision-making. The evaluation assessed the awareness level of beneficiaries about these feedback mechanisms. Table 8 below shows that 89.1% of beneficiaries were aware of these monitoring mechanisms. The evaluation noted that there was an appreciable level of awareness about the feedback mechanisms instituted by the FEEDS team and their partners. Regarding the channels they were mostly aware of, the evaluation results presented in Table 8 below found telephone hotlines to be ranked the highest at 76.4%, followed by suggestion boxes at 42.7% and letters at 19.6%. Table 8: Level of Beneficiary Awareness about the Feedback Mechanisms 47 Frequency % If you are treated badly by a cash out/field agent, are you aware of somewhere to complain him/her? Yes 582 89.1 No 71 10.9 Which channel or means do you report your concerns to the above institutions or persons? (Multiple responses) Suggestion Box 279 42.7 Telephone Hotlines 499 76.4 Letters 128 19.6 Others (Specify) 34 5.2 Other external monitoring mechanisms employed included intermittent cash transfer working group meetings and food security group meetings at the national level to share lessons learnt and best practices. These informed field-level decisions as project implementation progressed, and the meetings were replicated at the district level. Outcomes were shared with key stakeholders, including line ministries and district local council. The evaluation therefore concludes that the FEEDS project was adequately monitored. As a supplement to the routine monitoring exercises, the evaluation found that phase I implementation was evaluated by the FEEDS Program Team at midline. The evaluation notes that the midline review findings, which were not assessed in this evaluation, showed positive progress towards intended results and were in line with the findings of this evaluation, even though the midline was not an independent review. The other two phases were not evaluated at midline. However, the PDMs were used as proxies to keep track of progress and learn from what was going on in the field. 3.12 Constraining and Facilitating Factors of Project Implementation The evaluation noted most constraining factors of the FEEDS implementation were general operational constraints. There weren’t specific constraints that prevented the achievement of results. ● Constraining Factors ● Bad road network and poor mobile communication network slowed down the pace of activities, including cash transfer itself and monitoring activities. ● Some Cash Transfer Committee (CTC) members reported that in some cases they felt the urge to quit the project because ‘they were not getting any direct benefit (tokens, etc.) in return for their services.’ This was confirmed from KIIs with community leaders and field program staff, who reported that they sometimes had to spend a long time encouraging CTC members to continue working and not quit. According to them, committed new CTC members onboard and get them understand the intervention would take too much time. Because the original CTC members involved already understood the project concept, and partly because they already knew the cash beneficiaries, it was preferable to keep them on. This problem of CTC members quitting the project was largely mitigated by giving them FEEDS customized T-shirts. ● Facilitating Factors ● The FEEDS project had enough staff to execute all key functions. Assigning specific staff to specific chiefdoms ensured that the project team was always close 48 to beneficiaries. This enhanced monitoring activities and flagged any issues that were addressed promptly. ● The evaluation noted that all the partners the FEEDS team worked with (including the CTC and local authorities) performed their respective duties. This enhanced the level of results that the evaluation found. Note: Though some members of the CTC sometimes complained of “no tokens”, their involvement played a very crucial role in achieving the overall result that the FEEDS project produced. ● The project was well funded. This was a huge source of timely and full implementation of activities and permitted the achievement of desired results. ● Effective sensitizations and house-to-house visits by field staff about the correct use of cash received, coupled with complementary trainings, ensured that beneficiaries used more of the cash to access food rather than non-food items. This helped in improving their food security statuses, as evidenced by changes in the respective indicators. The special attention paid to household decision￾making regarding the use of FEEDS cash also helped avert most potential conflicts. This may have been the source of the level of women empowerment the evaluation found. 4.0 Lessons Learned and Recommendations 4.1 Lessons Learned ❖ When the income of households increases in cash, they are motivated to increase their expenditure on other important items that were not previously prioritized as expenditure items when there was less or no income in the household. The FEEDS cash expenditure patterns seem to be contrary to the expectations of the FEEDS intervention that at the end of the intervention, 70% of the EFSP cash received by households would be spent on food items. However, the observed decrease in EFSP cash expenditure on food (44.3%) was compensated for by expenditures in other productive categories, including education, health care, agriculture, and savings, which will promote long-term food security. ❖ Save the Children deployed staff directly in the target chiefdoms. This provided for a closer link between implementing staff and the beneficiaries throughout the life of the project, which was very helpful for effective monitoring of market prices, cash utilization and nutrition-sensitive messaging. This strategy created high SC/USAID visibility, which helped improve implementation through prompt and firsthand information gathering that was used for informed decision-making about the project. ❖ Establishing the right partnerships with state and non-state actors adds special value to quick – impact projects like cash transfer projects. The presence of NaCSA, ACC, KDC and CTCs added integrity and visibility to the whole process. ❖ When the right beneficiary targeting is done, cash interventions could have a very big impact on the daily lives of the beneficiaries. Their food security situation will be significantly impacted in the shortest possible time, as measured by the statistically significant changes that occurred in the values of the target household’s HDDS, HHS, and rCSI. ❖ The involvement of CTCs had a meaningful impact, especially in targeting the right beneficiaries and conveying information quickly to beneficiary households. However, when they are not allowed to benefit directly, they could have problems with total commitment. ❖ In the absence of effective mobile technology, a well-deployed and monitored offline cash transfer project can adequately replace the technology-based cash transfers. 49 4.2 Recommendations 1. The average meal frequency of households per day was noted to be approximately 2 at the final evaluation. This figure should reach a minimum of 3 for better nutrition outcomes. Therefore, the Government of Sierra Leone, through the Ministry of Health and Sanitation and represented by the District Health Management Team (DHMT) and other related community structures, should continue with activity-based monitoring program strategies to increase or maintain the community’s access to sufficient nutritious foods in the right varieties. 2. The evaluation found that many households have diversified their diets as a result of the EFSP intervention. This has prompted most to start or increase their backyard gardening to keep up a steady stream of fresh vegetables, fruits, etc. On that note, the Government of Sierra Leone, through the Ministry of Agriculture, Forestry and Food Security (MAFFS), should undertake regular sensitizations on the benefits of backyard gardening and the importance of eating these foods instead of selling them to help improve household nutritional status. Additionally, MAFFS block extension officers should continue monitoring the participants who were trained in post-harvest management technology to ensure they continue to employ it in their farming activities. They should also provide technical assistance where necessary. 3. The cash transfer processes and systems were largely effective, as noted by the satisfaction levels of beneficiaries with the distances covered to access pay points, waiting time to receive cash, and safety both at pay points and after leaving the pay points. It is recommended that the strategies employed to achieve these must be maintained by Save the Children in similar future interventions. 4. The direct deployment of Save the Children staff in the target chiefdom helped extensively in the achievement of results. This strategy should be replicated and/or scaled up in future similar interventions. 5. There was some misunderstanding on the part of the beneficiaries about the percentage of household food needs that the transferred cash was supposed to cover. Most of them thought the cash was meant to cover all the household food needs, so most households (62.1%) reported that the cash was not adequate for their household food needs. It is recommended that in the future, a very clear sensitization must precede the transfer of cash and this should continue throughout the life of the project. 6. The evaluation noted that there was limited involvement of the private sector in the overall implementation. Though there was not observed direct negative impact on the project, the high potential of this sector could be tapped into if more organizations from the sector were drawn into the project. It is recommended that in similar future projects, Save the Children should partner with more than one private sector organization. 50 References 1. Ballard, Terri; Coates, Jennifer; Swindale, Anne; and Deitchler, Megan. 2011. Household Hunger Scale: Indicator Definition and Measurement Guide. Washington, DC: FANTA-2 Bridge, FHI 360 2. Bamberger, M., Rugh, J. & Mabry, L., 2012. RealWorld Evaluation: Working Under Budget, Time, Data, and Political Constraints. (Second edition). Thousand Oaks: Sage. 3. CARE International, 2009. Guidelines for the implementation of Baseline study for women’s empowerment programs funded by Norad (2009 - 2013) 4. Kennedy, Gina; Ballard, Terri; and Dop, MarieClaude. 2013. Guidelines for Measuring Household and Individual Dietary Diversity. EC-FAO “Linking Information and Decision Making to Improve Food Security” Program. 5. Krejcie and Morgan, 1970. Determining Sample Size in Research Activities. Educational and Psychological Measurement, No. 30, pp.607-610. 6. Magnani, R., 1999. Sampling guide. Food and Nutrition Technical Assistance Project (FANTA) and Academy for Educational development. The document can be downloaded from http://www.fantaproject.org 7. Maxwell, D., and Caldwell. R, 2008. Coping Strategies Index: Field Methods Manual. A Tool for Rapid Measurement of Household Food Security and the Impact of Food Aid Programs in Humanitarian Emergencies. Second Edition, Cooperative for Assistance and Relief Everywhere, Inc. (CARE). 8. Poverty Reduction Strategy Paper (PRSP) or the Agenda for Prosperity (PRSP III – 2013-2018) 9. Sierra Leone Demographic and Health Survey (2013) 10. Sierra Leone National Ebola Recovery Plan (2015 – 2017) 11. Sierra Leone World Food Program (WFP) Comprehensive Food Security Vulnerability Assessment (CSFVA), 2015. 12. Stukel, Diana; and Deitchler, Megan. 2012. Addendum to FANTA Sampling Guide by Robert Magnani (1999): Correction to Section 3.3.1 Determining the Number of Households That Need to be Contacted. Washington, DC: FHI 360/ FANTA. 13. Swindale, Anne, and Paula Bilinsky. 2006. Household Dietary Diversity Score (HDDS) for Measurement of Household Food Access: Indicator Guide (v.2). Washington, D.C.: FHI 360/FANTA. 14. WFP, 2008. Food Consumption Analysis: Calculation and Use of the Food Consumption Score in Food Security Analysis. Technical Guidance Sheet on Vulnerability Analysis and Mapping 51 Appendices Appendix A: Results from Statistical Test of Significance Chi-Square Test Results on Household Hunger Scale (HHS) Group * In the past one month, was there ever no food to eat of any kind in your house because of lack of resources to get food? Crosstab In the past one month, was there ever no food to eat of any kind in your house because of lack of resources to get food? No Yes Total Group Baseline Count 134 266 400 Expected Count 217.7 182.3 400.0 Endline Count 439 214 653 Expected Count 355.3 297.7 653.0 Total Count 573 480 1053 Expected Count 573.0 480.0 1053.0 Chi-Square Tests Value df Asymptotic Significance (2- sided) Exact Sig. (2- sided) Exact Sig. (1- sided) Pearson Chi-Square 113.760a 1 .000 Continuity Correctionb 112.405 1 .000 Likelihood Ratio 115.302 1 .000 Fisher's Exact Test .000 .000 N of Valid Cases 1053 a. 0 cells (0.0%) have expected count less than 5. The minimum expected count is 182.34. b. Computed only for a 2x2 table Group * In the past one month, did you or any household member go to sleep at night hungry because there was not enough food? Crosstab In the past one month, did you or any household member go to sleep at night hungry because there was not enough food? No Yes Total Group Baseline Count 110 290 400 Expected Count 189.2 210.8 400.0 Endline Count 388 265 653 Expected Count 308.8 344.2 653.0 Total Count 498 555 1053 Expected Count 498.0 555.0 1053.0 52 Chi-Square Tests Value df Asymptotic Significance (2- sided) Exact Sig. (2- sided) Exact Sig. (1- sided) Pearson Chi-Square 101.380a 1 .000 Continuity Correctionb 100.104 1 .000 Likelihood Ratio 104.203 1 .000 Fisher's Exact Test .000 .000 N of Valid Cases 1053 a. 0 cells (0.0%) have expected count less than 5. The minimum expected count is 189.17. b. Computed only for a 2x2 table Group * In the past one month, did you or any household member go a whole day and night without eating anything at all because there was not enough food? Crosstab In the past one month, did you or any household member go a whole day and night without eating anything at all because there was not enough food? No Yes Total Group Baseline Count 183 217 400 Expected Count 240.8 159.2 400.0 Endline Count 451 202 653 Expected Count 393.2 259.8 653.0 Total Count 634 419 1053 Expected Count 634.0 419.0 1053.0 Chi-Square Tests Value df Asymptotic Significance (2- sided) Exact Sig. (2- sided) Exact Sig. (1- sided) Pearson Chi-Square 56.286a 1 .000 Continuity Correctionb 55.317 1 .000 Likelihood Ratio 56.079 1 .000 Fisher's Exact Test .000 .000 N of Valid Cases 1053 a. 0 cells (0.0%) have expected count less than 5. The minimum expected count is 159.16. b. Computed only for a 2x2 table 53 A one-tailed Independent Sample T-Test Results on Average Household Dietary Diversity Score (HDDS) Group Statistics Group N Mean Std. Deviation Std. Error Mean Average HDDS Baseline 400 4.60 2.084 .104 Endline 653 6.44 1.971 .077 Independent Samples Test Levene's Test for Equality of Variances t-test for Equality of Means F Sig. t df Sig. (2- tailed) Mean Difference Std. Error Difference 90% Confidence Interval of the Difference Lower Upper Average HDDS Equal variances assumed .770 .380 -14.401 1051 .000 -1.842 .128 -2.053 -1.631 Equal variances not assumed -14.210 807.770 .000 -1.842 .130 -2.055 -1.629 By dividing two-tailed test significance value in half, after running the test at 90% CI will result into a one-tailed test significance value as presented in the body of the report. Chi-Square Tests Results on Individual Food Groups Group * Corn/maize, sorghum, millet, wheat, bread, biscuits, rice, rice noodles Crosstab Corn/maize, sorghum, millet, wheat, bread, biscuits, rice, rice noodles No Yes Total Group Baseline Count 93 307 400 Expected Count 42.5 357.5 400.0 Endline Count 19 634 653 Expected Count 69.5 583.5 653.0 Total Count 112 941 1053 Expected Count 112.0 941.0 1053.0 Chi-Square Tests Value df Asymptotic Significance (2- sided) Exact Sig. (2- sided) Exact Sig. (1- sided) Pearson Chi-Square 107.972a 1 .000 Continuity Correctionb 105.843 1 .000 Likelihood Ratio 107.926 1 .000 Fisher's Exact Test .000 .000 N of Valid Cases 1053 a. 0 cells (0.0%) have expected count less than 5. The minimum expected count is 42.55. b. Computed only for a 2x2 table 54 Group * Potatoes, sweet potatoes, yams, manioc, cassava, coco yams or any other foods made from roots or tubers Crosstab Potatoes, sweet potatoes, yams, manioc, cassava, coco yams or any other foods made from roots or tubers No Yes Total Group Baseline Count 103 297 400 Expected Count 61.9 338.1 400.0 Endline Count 60 593 653 Expected Count 101.1 551.9 653.0 Total Count 163 890 1053 Expected Count 163.0 890.0 1053.0 Chi-Square Tests Value df Asymptotic Significance (2- sided) Exact Sig. (2- sided) Exact Sig. (1- sided) Pearson Chi-Square 52.003a 1 .000 Continuity Correctionb 50.745 1 .000 Likelihood Ratio 50.438 1 .000 Fisher's Exact Test .000 .000 N of Valid Cases 1053 a. 0 cells (0.0%) have expected count less than 5. The minimum expected count is 61.92. b. Computed only for a 2x2 table Group * Vegetables like spinach, tomato, onion, okra, eggplant, pepper, or pumpkin Crosstab Vegetables like spinach, tomato, onion, okra, eggplant, pepper, or pumpkin No Yes Total Group Baseline Count 138 262 400 Expected Count 70.7 329.3 400.0 Endline Count 48 605 653 Expected Count 115.3 537.7 653.0 Total Count 186 867 1053 Expected Count 186.0 867.0 1053.0 Chi-Square Tests Value df Asymptotic Significance (2- sided) Exact Sig. (2- sided) Exact Sig. (1- sided) Pearson Chi-Square 125.715a 1 .000 Continuity Correctionb 123.855 1 .000 Likelihood Ratio 123.523 1 .000 Fisher's Exact Test .000 .000 N of Valid Cases 1053 a. 0 cells (0.0%) have expected count less than 5. The minimum expected count is 70.66. 55 b. Computed only for a 2x2 table Group * Fruits like mango, guava, papaya, banana, avocado, orange, lemons or plantains Crosstab Fruits like mango, guava, papaya, banana, avocado, orange, lemons or plantains No Yes Total Group Baseline Count 180 220 400 Expected Count 123.8 276.2 400.0 Endline Count 146 507 653 Expected Count 202.2 450.8 653.0 Total Count 326 727 1053 Expected Count 326.0 727.0 1053.0 Chi-Square Tests Value df Asymptotic Significance (2- sided) Exact Sig. (2- sided) Exact Sig. (1- sided) Pearson Chi-Square 59.493a 1 .000 Continuity Correctionb 58.438 1 .000 Likelihood Ratio 58.610 1 .000 Fisher's Exact Test .000 .000 N of Valid Cases 1053 a. 0 cells (0.0%) have expected count less than 5. The minimum expected count is 123.84. b. Computed only for a 2x2 table Group * Beef, pork, lamb, goat, rabbit wild game, chicken, duck, or other birds, liver, kidney, heart, or other organ meats Crosstab Beef, pork, lamb, goat, rabbit wild game, chicken, duck, or other birds, liver, kidney, heart, or other organ meats No Yes Total Group Baseline Count 378 22 400 Expected Count 359.7 40.3 400.0 Endline Count 569 84 653 Expected Count 587.3 65.7 653.0 Total Count 947 106 1053 Expected Count 947.0 106.0 1053.0 Chi-Square Tests Value df Asymptotic Significance (2- sided) Exact Sig. (2- sided) Exact Sig. (1- sided) Pearson Chi-Square 14.857a 1 .000 Continuity Correctionb 14.055 1 .000 Likelihood Ratio 16.083 1 .000 Fisher's Exact Test .000 .000 N of Valid Cases 1053 56 a. 0 cells (0.0%) have expected count less than 5. The minimum expected count is 40.27. b. Computed only for a 2x2 table Group * Fresh or dried fish or shellfish Crosstab Fresh or dried fish or shellfish No Yes Total Group Baseline Count 145 255 400 Expected Count 61.9 338.1 400.0 Endline Count 18 635 653 Expected Count 101.1 551.9 653.0 Total Count 163 890 1053 Expected Count 163.0 890.0 1053.0 Chi-Square Tests Value df Asymptotic Significance (2- sided) Exact Sig. (2- sided) Exact Sig. (1- sided) Pearson Chi-Square 212.689a 1 .000 Continuity Correctionb 210.137 1 .000 Likelihood Ratio 218.899 1 .000 Fisher's Exact Test .000 .000 N of Valid Cases 1053 a. 0 cells (0.0%) have expected count less than 5. The minimum expected count is 61.92. b. Computed only for a 2x2 table Group * Foods made from beans, peas, lentils, or nuts Crosstab Foods made from beans, peas, lentils, or nuts No Yes Total Group Baseline Count 322 78 400 Expected Count 268.2 131.8 400.0 Endline Count 384 269 653 Expected Count 437.8 215.2 653.0 Total Count 706 347 1053 Expected Count 706.0 347.0 1053.0 Chi-Square Tests Value df Asymptotic Significance (2- sided) Exact Sig. (2- sided) Exact Sig. (1- sided) Pearson Chi-Square 52.840a 1 .000 Continuity Correctionb 51.863 1 .000 Likelihood Ratio 55.280 1 .000 Fisher's Exact Test .000 .000 N of Valid Cases 1053 57 a. 0 cells (0.0%) have expected count less than 5. The minimum expected count is 131.81. b. Computed only for a 2x2 table Group * Cheese, yogurt, milk or other milk products Crosstab Cheese, yogurt, milk or other milk products No Yes Total Group Baseline Count 398 2 400 Expected Count 390.5 9.5 400.0 Endline Count 630 23 653 Expected Count 637.5 15.5 653.0 Total Count 1028 25 1053 Expected Count 1028.0 25.0 1053.0 Chi-Square Tests Value df Asymptotic Significance (2- sided) Exact Sig. (2- sided) Exact Sig. (1- sided) Pearson Chi-Square 9.775a 1 .002 Continuity Correctionb 8.515 1 .004 Likelihood Ratio 12.145 1 .000 Fisher's Exact Test .001 .001 N of Valid Cases 1053 a. 0 cells (0.0%) have expected count less than 5. The minimum expected count is 9.50. b. Computed only for a 2x2 table Group * Foods made with oil, fat, or butter Crosstab Foods made with oil, fat, or butter No Yes Total Group Baseline Count 260 140 400 Expected Count 249.2 150.8 400.0 Endline Count 396 257 653 Expected Count 406.8 246.2 653.0 Total Count 656 397 1053 Expected Count 656.0 397.0 1053.0 Chi-Square Tests Value df Asymptotic Significance (2- sided) Exact Sig. (2- sided) Exact Sig. (1- sided) Pearson Chi-Square 2.005a 1 .157 Continuity Correctionb 1.823 1 .177 Likelihood Ratio 2.013 1 .156 58 Fisher's Exact Test .169 .088 N of Valid Cases 1053 a. 0 cells (0.0%) have expected count less than 5. The minimum expected count is 150.81. b. Computed only for a 2x2 table Group * Eggs Crosstab Eggs No Yes Total Group Baseline Count 391 9 400 Expected Count 369.6 30.4 400.0 Endline Count 582 71 653 Expected Count 603.4 49.6 653.0 Total Count 973 80 1053 Expected Count 973.0 80.0 1053.0 Chi-Square Tests Value df Asymptotic Significance (2- sided) Exact Sig. (2- sided) Exact Sig. (1- sided) Pearson Chi-Square 26.273a 1 .000 Continuity Correctionb 25.059 1 .000 Likelihood Ratio 30.982 1 .000 Fisher's Exact Test .000 .000 N of Valid Cases 1053 a. 0 cells (0.0%) have expected count less than 5. The minimum expected count is 30.39. b. Computed only for a 2x2 table Group * Sugar or honey Crosstab Sugar or honey No Yes Total Group Baseline Count 346 54 400 Expected Count 289.8 110.2 400.0 Endline Count 417 236 653 Expected Count 473.2 179.8 653.0 Total Count 763 290 1053 Expected Count 763.0 290.0 1053.0 Chi-Square Tests Value df Asymptotic Significance (2- sided) Exact Sig. (2- sided) Exact Sig. (1- sided) Pearson Chi-Square 63.719a 1 .000 59 Continuity Correctionb 62.589 1 .000 Likelihood Ratio 68.464 1 .000 Fisher's Exact Test .000 .000 N of Valid Cases 1053 a. 0 cells (0.0%) have expected count less than 5. The minimum expected count is 110.16. b. Computed only for a 2x2 table Group * Other foods, such as condiments, coffee, tea Crosstab Other foods, such as condiments, coffee, tea No Yes Total Group Baseline Count 239 161 400 Expected Count 232.9 167.1 400.0 Endline Count 374 279 653 Expected Count 380.1 272.9 653.0 Total Count 613 440 1053 Expected Count 613.0 440.0 1053.0 Chi-Square Tests Value df Asymptotic Significance (2- sided) Exact Sig. (2- sided) Exact Sig. (1- sided) Pearson Chi-Square .625a 1 .429 Continuity Correctionb .527 1 .468 Likelihood Ratio .626 1 .429 Fisher's Exact Test .440 .234 N of Valid Cases 1053 a. 0 cells (0.0%) have expected count less than 5. The minimum expected count is 167.14. b. Computed only for a 2x2 table A one-tailed Independent Sample T-Test Results for Average Reduced Coping Strategies Index (RCSI) Group Statistics Group N Mean Std. Deviation Std. Error Mean Average rCSI Baseline 400 3.206 1.3638 .0682 Endline 653 2.442 1.2778 .0500 60 Independent Samples Test Levene's Test for Equality of Variances t-test for Equality of Means F Sig. t df Sig. (2- tailed) Mean Difference Std. Error Difference 90% Confidence Interval of the Difference Lower Upper Average rCSI Equal variances assumed 3.574 .059 9.181 1051 .000 .7643 .0832 .6273 .9014 Equal variances not assumed 9.039 801.643 .000 .7643 .0846 .6251 .9036 By dividing two-tailed test significance value in half, after running the test at 90% CI will result into a one-tailed test significance value as presented in the body of the report. Appendix B: List of Evaluation Team members No. Name Designation 1 Baimba Abdulai Koroma Team Leader (Food Security and Cash Transfer Evaluation Expert) 2 Momoh Thomas Bockarie Socio-economic and Livelihood Expert 3 Kenneth Lansana Mangow Field supervisor 4 Aiah John Kellie Field supervisor 5 Jesse Yamba Online Kobo Mobile-based Data Collection Platform System Designer 6 Joseph Kallon Enumerator 7 Mohamed Daramy Enumerator 8 Sahr Emmanuel Koademba Enumerator 9 Andrew Senesie Enumerator 10 Kenneth Maada Lansana Enumerator 11 David Foday Enumerator 12 Mohamed Gbabai Enumerator 13 Edward W. Baion Enumerator 14 Hawa Musa Enumerator 15 Haja M. Musa Enumerator 16 Mohamed Brima Enumerator 17 Manu Katta Enumerator 18 Mohamed Konneh Enumerator 19 Salieu Koroma Enumerator 20 Mamusu Gordon Enumerator