i Sierra Leone Emergency Food Security Program (EFSP) Supporting Vulnerable Households Recover from the Ebola Crisis Sierra Leone Emergency Food Security Program (EFSP) FINAL PROJECT PERFORMANCE EVALUATION This publication was produced at the request of the United States Agency for International Development. It was prepared independently by NestBuilders International (NBI) under contract to ACDI/VOCA. December 2017. Final Project Performance Evaluation Evaluation Conducted by: NestBuilders International (NBI) NestBuilders International 16 Fifth Street, Juba Hill Freetown, Sierra Leone +232-25-239-495 www.nbiconsultancy.com DISCLAIMER The authors' views expressed in this publication do not necessarily reflect the views of the United States Agency for International Development or the United States Government. EFSP Final Project Performance Evaluation Page i Table of Contents 1. Acronyms ................................................................................................................................... iv 2. Executive Summary..................................................................................................................... v Introduction ............................................................................................................................................1 1. The Emergency Food Security Program..........................................................................................2 2.1 Background .........................................................................................................................2 2.2 Description of the Emergency Food Security Program (EFSP)............................................2 2. Evaluation Objectives and Scope....................................................................................................5 2.1 Evaluation Objectives..........................................................................................................5 2.2 Evaluation Scope.................................................................................................................5 2. 3. Evaluation Questions..........................................................................................................5 3. Methodology...................................................................................................................................7 3.1 Study Design........................................................................................................................7 3.2 Survey Instruments.............................................................................................................8 3.3 Sampling..............................................................................................................................9 3.4 Data Collection..................................................................................................................11 3.5 Data Processing and Analysis............................................................................................12 3.6 Limitations.........................................................................................................................13 4. Evaluation Findings.......................................................................................................................15 4.1 Food Security and Improved Resilience............................................................................15 4.2. Program Systems, Processes and Procedures ..................................................................41 4.3. Program Coordination and Collaboration.........................................................................70 5. Achievement of Project Targets....................................................................................................76 6. Conclusion and Recommendations...............................................................................................78 List of Tables Table 1: Program Overview.....................................................................................................................3 Table 2: Evaluation questions and corresponding survey instrument used...........................................8 Table 3: Moderate or severe hunger, by VSLA membership (Phase 1)................................................18 Table 4: Moderate or severe hunger, by IGA (Phase 1)........................................................................18 Table 5: Moderate or severe hunger, by VSLA membership (Phase 2)................................................20 Table 6: Moderate or severe hunger, by IGA (Phase 2)........................................................................20 Table 7: Moderate or severe hunger among beneficiaries that attended nutrition training, by use of nutrition messages (Phase 2)................................................................................................................21 EFSP Final Project Performance Evaluation Page ii Table 8: Moderate or severe hunger among beneficiaries that attended financial management training, by use of financial management messages (Phase 2)............................................................22 Table 9: Household Dietary Diversity Score, by VSLA membership (Phase 1)......................................23 Table 10: Household Dietary Diversity Score, by IGA (Phase 1) ...........................................................24 Table 11: Household Dietary Diversity Score, by VSLA membership (Phase 2)....................................25 Table 12: Household Dietary Diversity Score, by IGA (Phase 2) ...........................................................26 Table 13: Household Dietary Diversity Score among beneficiaries that attended nutrition training, by use of nutrition messages (Phase 2).....................................................................................................26 Table 14: Household Dietary Diversity Score among beneficiaries that attended financial management training, by use of financial management messages (Phase 2)......................................27 Table 15: Household coping strategy index (Phase 1 and 2)................................................................33 Table 16: Meal frequency, by age group (Phase 1 and 2).....................................................................35 Table 17: CIC Criteria for their Initial Nomination of Households as Potential EFSP Beneficiaries (Phase 2) ...............................................................................................................................................44 Table 18: Revised Eligibility Criteria according to Household Food Security and EVD-impact (Phase 2) ..............................................................................................................................................................45 Table 19: Household selection score, LPMT vs. HFS (Phase 2).............................................................46 Table 20: EFSP Indicator Performance Tracking Table, Achievement of Indicators.............................76 List of Figures Figure 1: Number of beneficiary households in Phase 1 and Phase 2....................................................4 Figure 2: Household Hunger Scale, at baseline and endline (Phase 1).................................................15 Figure 3: Percentage of households with moderate or severe hunger, at baseline and endline (Phase 1) ...........................................................................................................................................................16 Figure 4: Household Hunger Scale, at baseline and endline (Phase 2).................................................18 Figure 5: Household Dietary Diversity Score, at baseline and endline (Phase 1).................................23 Figure 6: Household Dietary Diversity Score, at baseline and endline (Phase 2).................................24 Figure 7: Percentage of cash spent on selected types of expenditures last month (Phase 1).............28 Figure 8: 'Other' most common household expenditures in the last month (Phase 1) .......................29 Figure 9: Percentage of last cash transfer spent on selected types of expenditures (Phase 2)...........30 Figure 10: 'Other' most common household expenditures in the last month (Phase 2) .....................30 Figure 11: Prevalence of negative coping strategies used at least once in the last month (Phase 1 and 2) ...........................................................................................................................................................32 Figure 12: Negative coping strategies adopted in the last three months (Phase 1 and 2)...................33 Figure 13: Did you use any of the money from the cash transfer to start or continue an income generating business? ............................................................................................................................36 EFSP Final Project Performance Evaluation Page iii Figure 14: Types of IGAs beneficiaries invested in with the cash transfer money (Phase 1 and 2) .....37 Figure 15: Number of beneficiaries participating in each module for the Household Economics and Financial Management Training ...........................................................................................................60 Figure 16: HHS and HDDS based on use of household economics training messages to make spending decisions................................................................................................................................................61 Figure 17: Number of beneficiaries participating in each of the maternal/child nutrition and basic hygiene training sessions......................................................................................................................63 Figure 18: HHS and HDDS based on use of household economics training messages to make spending decisions................................................................................................................................................64 Figure 19: HHS and HDDS based on VSLA membership........................................................................66 Figure 20: Did you discuss how to spend the money with other household members after you received it?............................................................................................................................................69 EFSP Final Project Performance Evaluation Page iv 1. Acronyms ACC Anti-Corruption Commission CAPI Computer-Assisted Personal Interviewing CFSVA Comprehensive Food Security and Vulnerability Analysis CTP Cash Transfer Program CTWG Cash Transfer Working Group EA Enumeration Area EFSP Emergency Food Security Program EVD Ebola Virus Disease FANTA Food and Nutrition Technical Assistance FFP Food For Peace FGD Focus Group Discussion GoSL Government of Sierra Leone GRM Grievance Redress Mechanism HDDS Household Dietary Diversity Score HFS Household Food Security HHS Household Hunger Scale IGA Income Generating Activity IR Intermediate Result KII Key Informant Interview LPMT Light Proxy Means Test MIS Management Information System MSWGCA Ministry of Social Welfare, Gender and Children Affairs NaCSA National Commission for Social Action NBI NestBuilders International RE-SSN Rapid Ebola-Social Safety Net SNAP Sustainable Nutrition and Agriculture Promotion SO Strategic Objectives SOP Standard Operating Procedures SSN Social Safety Net (program) USAID United States Agency for International Development UCT Unconditional Cash Transfer VSLA Village Savings and Loans Association EFSP Final Project Performance Evaluation Page v 2. Executive Summary Background and Program Objectives The Ebola Virus Disease (EVD) in West Africa began in Guinea in February 2014 and quickly spread to the neighboring countries of Liberia and Sierra Leone. In response to the sudden and rapid spread of the disease, the Government of Sierra Leone (GoSL) declared a State of Emergency in July 2014, and restrictions on all types of public gatherings and the closure of schools and markets were codified into law. Due to their inability to pursue their normal livelihood activities, many households lost their basic livelihood resources and were unable to recover and stabilize themselves to pre-Ebola levels once the State of Emergency was lifted. To address the economic and social impacts of Ebola in Sierra Leone, the United States Agency for International Development (USAID) Food for Peace (FFP) awarded funding for ACDI/VOCA’s Sustainable Nutrition and Agriculture Promotion Plus (SNAP+) program in April 2015. Designed as an Ebola response and recovery program, a key component of the program was direct unconditional cash transfers to EVD affected households. The program was implemented in two phases: Phase 1 (June 2015 through December 2016) interventions targeted Ebola impacted households with unconditional cash transfers (UCTs) and, Phase 2 (January through December 2017) was developed as an extension to the original program term to address recovery needs of communities that were not covered during Phase 1. The specific program objective was “to support vulnerable households to recover from the social and economic shocks as a result of the Ebola Viral Disease emergency.” Evaluation Objectives, Scope and Methods The focus of this final performance evaluation is strategically linked to the objective of determining to what extent the program, particularly cash transfers, contributed to recovery of household food security and fostered resilience against future shocks from similar emergencies. The overarching objective is to provide answers to a specific set of research questions to investigate the extent to which direct cash transfers to households was an efficient, effective mechanism for enabling beneficiaries to restore food security, and their livelihoods, following the sudden and widespread epidemic of Ebola Virus Disease (EVD) in Sierra Leone. The evaluation considers progress from April 2015 through to December 2017 and therefore assesses results for both Phase 1 and Phase 2 activities. The specific objective of the evaluation is to answer the following key questions: 1) To what extent did households that participated in the EFSP achieve food security and improved resilience? 2) To what extent were the systems, processes, and procedures used by the program for delivering cash transfers and non-cash assistance to participants effective, efficient, and sustainable during the period of program implementation? EFSP Final Project Performance Evaluation Page vi 3) To what extent did the collaboration and coordination with public and private sector stakeholders result in enhanced capacity to deliver cash transfers as a form of social protection to vulnerable households and contribute to effective delivery of cash transfers during the emergency and recovery phases? NestBuilders International (NBI), a research consulting firm with operations in Sierra Leone, was recruited to conduct a final evaluation of the program. The evaluation was carried out according to a non-experimental design for simple pre-post comparison of results, using a combination of qualitative and quantitative research methods, as well as primary and secondary sources. Information provided has been gathered through review of program records and existing data, a quantitative survey with program beneficiaries as well as through interviews and FGDs with program staff, key sector stakeholders and beneficiaries. Therefore, this evaluation synthesizes multiple data sources to ensure a high degree of triangulation. Key Findings Evaluation Question 1: To what extent did households that participated in the EFSP achieve food security and improved resilience? In the context of the EVD crisis, the evaluation found that the cash transfer intervention was appropriate and essential to addressing the food security and livelihood needs of affected households. This is especially true in the case of Phase 1 of the program when many households lost their basic livelihood resources and were unable to recover and stabilize themselves to pre-Ebola levels once the State of Emergency was lifted. The economic relief provided by the cash transfer program (CTP) was critical to supporting improvements in food security and longer-term recovery during this post-emergency phase. The programmatic shift to focus more on recovery was relevant in regard to the timing of Phase 2 implementation. By the time this phase began, and certainly by its end, Sierra Leone was no longer in an emergency. In this way, beneficiaries were able to spend more of the UCT on income generating activities, rather than food alone, to support their recovery and future resilience. Across both program phases, the cash transfer improved household food security and reduced hunger in the short and medium term. Quantitative data revealed a marked decrease in the proportion of households reporting moderate to severe hunger and an increase in the household dietary diversity score for all program beneficiaries. Qualitative research reinforced these findings as beneficiaries reported being able to buy more food, increase their food intake, consume a wider variety of food items and reduce their reliance on negative coping strategies because of the UCT. Beneficiaries also reported prioritizing the cash transfer for expenditures on education and investments in income generating activities with a view to plan for their future. Overall, the impacts of the cash transfer were positive, and in the case of Phase 1 beneficiaries, shown to be sustained one year after the last cash transfer. Recipients were generally found to use the cash wisely and in a manner that helped their families recover from the shock of EVD. Measures of resiliency for beneficiary households show that the CTP enabled a number of protective effects which have supported beneficiaries to become more resilient to future shocks. Households have invested in livestock, engaged in non-farm income generating activities, joined Village Savings and EFSP Final Project Performance Evaluation Page vii Loans Associations, are reportedly taking less debt and are maintaining greater food stocks because of their receipt of the cash transfer. In the case of Phase 1 beneficiaries, many of these activities are still ongoing a year later. Evaluation Question 2: To what extent were the systems, processes, and procedures used by the program for delivering cash transfers and non-cash assistance to participants effective, efficient, and sustainable during the period of program implementation? The evaluation found that the systems, processes and procedures employed by the program were highly efficient and effective in supporting the objectives of the program. First, the targeting approach employed by the program worked well. Following the post-emergency, poverty-focused approach of Phase 1, the targeting was successfully improved in Phase 2 to focus on food security indicators of heavily impacted households. While this helped to clarify the targeting criteria and reduce inclusion errors, there were noted exclusion errors related to set quotas and lower coverage rates. Second, the cash transfer delivery mechanisms performed well across both phases, with significant improvements in data collection and monitoring being introduced in Phase 2. While initially proposing to use e-transfers, ACDI/VOCA was careful to consider the realities on the ground (e.g. poor roads, limited mobile phone service coverage, etc.) and engaged local service providers to provide a direct cash delivery mechanism which was appropriate for the operating context in the project’s target areas. While e-transfers would have provided an ideal solution to efficiently deliver the cash at scale, ACDI/VOCA effectively worked with the service providers to minimize any negative impacts on the delivery of the cash to beneficiaries. Overall, beneficiaries praised the process of cash distribution for its speed and efficiency. Despite challenges related to poor roads and liquidity of service providers, ACDI/VOCA ensured that payments were made on time, and beneficiaries reported being treated courteously by program staff. ACDI/VOCA also established well-functioning feedback mechanisms, community engagement and accountability measures throughout the delivery of the cash transfers. Lastly, the incorporation of non-cash, complementary activities for participating households significantly improved the effectiveness of the program. Households that used the economic and nutrition training messages to make spending decisions were found to have significantly improved their household food security across a range of measures. Beneficiaries linked to Village Savings and Loans Associations (VSLA) also demonstrated improved food security and higher potential for sustainable recovery through their increased access to savings and credit, reducing their reliance on debt and moneylenders. The introduction of the VSLAs was highly valued by beneficiaries as it provided them with an opportunity to save beyond the life of the program and established a longer￾term, local safety net. Evaluation Question 3: To what extent did the collaboration and coordination with public and private sector stakeholders result in enhanced capacity to deliver cash transfers as a form of social protection to vulnerable households and contribute to effective delivery of cash transfers during the emergency and recovery phases? Strong coordination and collaboration with government officials, NGOs, donors and the private sector was a major strength of the CTP. The establishment of the Cash Transfer Working Group was EFSP Final Project Performance Evaluation Page viii integral to providing a platform for shared learning and coordination to avoid duplication and overlap of efforts. In the context of limited experience with humanitarian cash transfers in the country, the strong coordination was integral to establishing harmonization of critical elements such as targeting, monitoring and transfer amounts to ensure consistency of approach across development partners. The close coordination with the Government of Sierra Leone also ensured cohesion between ACDI/VOCA’s activities and the national Social Safety Net program framework – effectively avoiding the development of parallel systems. Harmonization of critical elements also ensured ACDI/VOCA’s CTP was properly aligned with the national Social Safety Net program framework. Lastly, the direct leadership and support provided by the donor assisted the effective delivery of the CTP from the outset by providing adaptive management, technical support and oversight. Recommendations Moving forward, a series of recommendations can be made, which in turn can help boost the efficiency and effectiveness of future cash transfer program interventions: ● Ensure context-appropriate cash transfer delivery mechanisms. The use of e-transfers was initially planned, and would have provided an ideal solution to efficiently deliver the cash at scale in a manner that would greater promote financial inclusion for beneficiaries. However, given the limited capacity of the mobile money networks in Sierra Leone, this was not a feasible option at the time. While the short-term nature of the program never justified a longer-term solution to cash delivery through other means, future long term CTPs can gain from the experience of the EFSP to look at other solutions that can be brought to scale and potentially bring participating households into the formal banking system. Since the start of the program, a number of service providers have built their capacity in delivering e-transfers (such as Orange and Africell), warranting further engagement. Stakeholders engaged in CTP need to leverage their aggregate demand to make the business case for financial service providers to expand their services into CTPs to remote areas in the country. ● Cash transfers should be considered as a suitable response to future emergencies in Sierra Leone. A clear benefit of cash transfers is the flexibility it provides. The cash transfer provided an opportunity to households that could not be achieved through non-cash assistance. It enabled them to choose the best options for their own households and capacities, so that the investments they made in their futures suited their own needs and abilities. Humanitarian actors with sector-specific mandates should seek to build on the success of the EVD response cash transfer intervention by assessing its suitability in addressing future emergencies. As with any intervention, it will be important for humanitarian actors to consider the appropriateness of CTPs on a case-by-case basis by examining factors such as the needs of their target group, market functionality as well as an assessment of governance and security considerations. ● Use of appropriate targeting criteria aligned with project objectives is essential for cash-based emergency responses to food insecurity. The targeting approach used in Phase 1 was based on a standard poverty means test which did not directly address the program objective of Ebola recovery. Standard criteria more appropriate for long-term poverty reduction needs to be EFSP Final Project Performance Evaluation Page ix considered alongside suitable indicators related to the impact of the emergency and food security. These are more directly linked to the shorter-term recovery objectives of cash-based emergency responses to food insecurity and have the potential to better limit inclusion and exclusion errors. Furthermore, while a quota system may be necessary to manage limited resources, future programs should review village quotas on a case-by-case basis to minimize the potential for exclusion errors. For instance, in the case where a village quota excludes five households (out of a potential 40), consider including all households as beneficiaries. This will not only serve to minimize exclusion errors, but will also reduce the potential for inter￾household conflict while encouraging community buy-in and support for the program. ● More effective ways of handling polygamous households in future cash transfer programs need to be explored. Polygamous households were more likely to experience intra-household conflict if only one wife received the cash. As beneficiary targeting focused on individual household economies and not marital relationships (i.e. each ‘pot’ was treated as a separate household as each wife is traditionally responsible for her own children), splitting the cash was not always a feasible solution. While the ACDI/VOCA approach of focusing on household economics served to reduce inclusion and exclusion errors within polygamous households, there is a need to consider how future CTPs will address the potential for intra-household conflict among polygamous households. As conflicts are traditionally resolved by local authorities such as Town Chiefs (whose default resolution was to split the money with other wives), future programs should provide local authorities with training on the targeting requirements and the purpose of the cash transfers. This will help those tasked with resolution of conflict to respond in ways that promote the goals of the program. ● Complementary activities should be implemented alongside future cash transfer programs to promote recovery and resilience. The inclusion of training activities and increased access to VSLAs enabled beneficiaries to use the cash better, generating an impact beyond the duration of the project. Regardless of the timeframe of the response, an effort should be made to provide high-impact complementary activities, such as training on household economics and financial management during program orientation. However, as evidenced by the evaluation, participation in training activities can be impacted by the timing and rollout of activities. Future CTPs should carefully plan the phasing in of complementary activities according to the needs of beneficiaries based on the cash transfer schedule and disbursements. For instance, if cash transfers are being made at the start of the planting season, training on key crop production practices and the prioritization of funds to maximize impact should be provided from the outset (rather than later in the disbursement schedule). Providing essential training during an orientation session is also most likely to result in high attendance as the expectation of receipt of cash and interest in the program will be high. ● The use of digital data gathering technology should be integrated in future CTPs to improve accuracy, efficiency and monitoring. In Phase 2, ACDI/VOCA effectively leveraged digital technology to register beneficiaries, track payments, confirm beneficiary identity and improve monitoring systems. This overcame challenges with mobile money providers’ data capacity, improving accountability and allowing for the integration of multiple service providers. EFSP Final Project Performance Evaluation Page x Technological solutions should be sought at the outset of any CTP for implementers to have direct control over their data collection. ● Ensure robust coordination mechanisms are built from the outset of any future cash-based emergency response: The close coordination with the GoSL during the emergency phase and the ongoing coordination through the CTWG were critical to the success of the CTP. The harmonization of critical elements such as targeting, monitoring and transfer amounts enhanced the program's effectiveness and communication, and safeguarded against the development of parallel systems. This level of coordination is essential for any future cash-based emergency responses to food insecurity to improve coordination and accountability. ● There is a strong case for follow-up development activities in order to ensure the sustainability of cash-based emergency responses to food security. The impact findings indicate that continued follow-up complementary activities would enable ACDI/VOCA to increase the potential for sustainable impact and reduce the need for future social protection among their beneficiaries. Follow-up development activities could capitalize on the outcomes of the CTP by offering complimentary services aligned to the livelihood choices that beneficiaries have made (e.g. training on improved agricultural practices, business skills, SME formation, etc.). ACDI/VOCA has an established presence in these communities, and would have a readily identifiable and eager audience for follow-up learning. Providing such follow-up development activities would be directly aligned with the original objectives of the CTP by continuing to build the resilience of these households against future shocks. ● The monitoring and evaluation design of future cash transfer programs should incorporate a representative comparison group and additional behavioral measures of food security to analyze impact. The addition of a control group would allow for a more thorough evaluation of the effectiveness of program activities and provide robust data on the causal relationship between the intervention and outcomes. Furthermore, beyond the main impact indicators of HHS and HDDS to measure food insecurity, assessing coping strategies should be added to future CTP monitoring and evaluation to better understand the behavioral response to food insecurity. This will support better program design as data on behavioral coping strategies goes beyond the ‘access’ dimension of food insecurity and is immediately useful for programmatic decision making, monitoring and evaluation. EFSP Final Project Performance Evaluation Page 1 Introduction This document outlines findings from the final end-of-project summative performance evaluation for the Emergency Food Security Program (EFSP) implemented by ACDI/VOCA for post-Ebola recovery in Sierra Leone between April 10, 2015 and December 31, 2017. The performance evaluation was implemented by NestBuilders International (NBI), an independent research consulting firm based in Sierra Leone. The report describes the extent to which the program, particularly cash transfers, contributed to recovery of household food security and fostered resilience against future shocks from similar emergencies. In conducting this evaluation, researchers considered two phases of the program: one of which ended 12 months ago (Phase 1), and another which ended at the time of the evaluation (Phase 2). Therefore, results from Phase 1 can be interpreted to measure the impact of the program one year after Phase 1 beneficiaries’ final cash distribution; while findings from Phase 2 serve as an endline measurement for the final phase of the program. Structure of the Report This report is presented in five sections. Section 1 provides the introduction, while Section 2 presents the background of the program including the context in which it was implemented. Section 3 presents the evaluation purpose and objectives including the scope of the assignment with specific evaluation questions. Section 4 provides the evaluation methodology and sources of data and information. The key findings are provided in Section 5 and Section 6. Lastly, Section 7 concludes the evaluation findings and provides recommendations for the way forward. EFSP Final Project Performance Evaluation Page 2 1.The Emergency Food Security Program 2.1 Background The Ebola Virus Disease (EVD) in West Africa began in Guinea in February 2014 and quickly spread to the neighboring countries of Liberia and Sierra Leone. Once the disease had entered Sierra Leone the lack of basic infrastructure including health infrastructure, logistics, and communications resulted in a rapid spread of the virus to every district in the country. In response to the sudden and rapid spread of the disease, the Government of Sierra Leone (GoSL) declared a State of Emergency in July 2014, and restrictions on all types of public gatherings and the closure of schools and markets were codified into law. During this emergency period, the movement of people was restricted, households and villages were quarantined, farmers were unable to attend to their farms, and small business owners were unable to conduct their business as usual or resupply their stocks of goods. Due to their inability to pursue their normal livelihood activities, many households lost their basic livelihood resources and were unable to recover and stabilize themselves to pre-Ebola levels once the State of Emergency was lifted. As evidence, in September 2015 a joint WFP FAO Comprehensive Food Security & Vulnerability Analysis (CFSVA) reported that 54.2% of rural households reported less income than previous years.1 Following the EVD emergency, other important socio-economic factors, such as healthcare systems and local markets, also faced slow progress toward recovery. By the time Sierra Leone was declared Ebola free by the World Health Organization on November 7, 2015, the country had experienced 14,601 cases of EVD resulting in 3,955 fatalities. 2.2 Description of the Emergency Food Security Program (EFSP) To address the economic and social impacts of Ebola in Sierra Leone, the United States Agency for International Development (USAID) Food for Peace (FFP) awarded funding for ACDI/VOCA’s Sustainable Nutrition and Agriculture Promotion Plus (SNAP+) program in April 2015. Designed as an Ebola response and recovery program, a key component of the program was direct unconditional cash transfers to EVD affected households. The SNAP+ program also included activities designed to complement ACDI/VOCA’s pre-existing Title II Sustainable Nutrition and Agriculture Promotion (SNAP) which was a multi-year development project implemented from 2010-2016. The SNAP+ program was implemented in three of the four Districts of the SNAP program. Koinadugu District with only one Chiefdom impacted by EVD was not selected to be part of the cash transfer program but participated in agricultural and VSLA recovery activities. 1.2.3. Two Phase Implementation Phase 1 of SNAP+ activities (June 2015 through December 2016) were implemented in Bombali, Tonkolili, and Kailahun Districts. Phase 1 interventions targeted Ebola impacted households with 1 WFP & FAO (2015). State of Food Security in Sierra Leone 2015: Comprehensive Food Security and Vulnerability Analysis (September – October 2015). EFSP Final Project Performance Evaluation Page 3 unconditional cash transfers (UCTs) and, additionally, provided support to the on-going SNAP program’s development activities that included: ● Restoring agricultural productivity of commercial farming groups with emergency seed packages: 23,800 farmers ● Village Savings and Loan Associations (VSLAs) recovery grants: 5,100 members ● Value chain agribusinesses recovery grants: 2,989 smallholders ● Nutrition support to children aged 6-59 months at risk of malnutrition: 1,225 children During Phase 1, a total of 13,584 beneficiary households were identified for enrolment in cash transfers (UCTs). Phase 2 (January through December 2017) was developed as an extension to the original program term to address recovery needs of communities that were not covered during Phase 1 in Bombali and Tonkolili Districts. Phase 2 provided households with unconditional cash transfers and, additionally, included beneficiary-based training activities related to: ● Maternal and child nutrition, household nutrition, and water, sanitation and hygiene (WASH) ● Household economic and financial planning ● Access to Village Savings and Loan Associations (VSLAs) During Phase 2, a total of 8,783 households were identified for enrolment in cash transfers (UCTs). After ACDI/VOCA’s much larger development program, SNAP, came to an end in December 2016, the continuation of SNAP+ Ebola recovery activities were rebranded as the Emergency Food Security Program (EFSP) to avoid confusion. Table 1: Program Overview Phase 1 Phase 2 Program Title Sustainable Nutrition and Agriculture Promotion Plus (SNAP+) (SNAP+) – rebranded as EFSP Implementer ACDI/VOCA as prime and International Medical Corps (IMC) as sub-recipient ACDI/VOCA as prime Regions in Sierra Leone Kailahun, Bombali, Tonkolili, Koinadugu Bombali, Tonkolili Total Funding Requested $9,000,000 $4,740,816 Start and End of Award Apr 14, 2015 – Dec 31, 2016 Jan 1, 2017 – Dec 31, 2017 Resources Requested by Interventions(s) Cash Transfer $7,921,794.00 Local Procurement $702,642 Vouchers $375,564 Cash Transfer $4,740,816 Number of Beneficiaries (total, per intervention) Cash Transfer 67,920 Local Procurement 24,098 Voucher 23,800 Grants 8,089 Cash Transfer 42,441 1.1.2. EFSP Objective and Intermediate Results Program Objective: To support vulnerable households to recover from the social and economic shocks as a result of the Ebola Viral Disease emergency. EFSP Final Project Performance Evaluation Page 4 Intermediate Result 1: Increase agricultural production at the community level to support early recovery through the increased availability of food and stable prices resulting in increased access to food in local markets (Phase 1 only) Intermediate Result 2: Reduce negative coping strategies by ensuring vulnerable households can access food available in local markets as well as strengthen nutrition of children at risk of malnutrition through locally procured complementary foods and locally available produce (Phase 1 and Phase 2) Intermediate Result 3: Strengthen local markets/enterprises through increasing production and therefore increase the availability of food in low surplus areas (Phase 1 only) 1.1.3. Geographic Coverage Phase 1 cash transfers were implemented in 65 sections within 14 Chiefdoms in Kailahun (5), Bombali (5) and Tonkolili (4) Districts.2 Phase 2 cash transfers and complementary activities were implemented in 26 sections of 6 chiefdoms within Bombali (3) and Tonkolili (3) Districts and targeted sections that were not covered during Phase 13 . A map showing the number of unconditional cash transfer beneficiaries per district is presented below. Communities and beneficiaries served during each Phase of program implementation are unique, and there was no overlap in the beneficiary count for Phase 1 and Phase 2. Figure 1: Number of beneficiary households in Phase 1 and Phase 2 2 Sections are political sub-divisions of chiefdoms, which are, in turn, political sub-divisions of districts. 3 Koinadugu District, with low EVD caseload, did not benefit from cash transfers. At suggestion of USAID, Kailahun District was not carried into Phase II. EFSP Final Project Performance Evaluation Page 5 2.Evaluation Objectives and Scope 2.1 Evaluation Objectives The focus of this final performance evaluation is strategically linked to the objective of determining to what extent the program, particularly cash transfers, contributed to recovery of household food security and fostered resilience against future shocks from similar emergencies. The overarching objective is to provide answers to a specific set of research questions to investigate the extent to which direct cash transfers to households was an efficient, effective mechanism for enabling beneficiaries to restore food security, and their livelihoods, following the sudden and widespread epidemic of Ebola Virus Disease (EVD) in Sierra Leone. Evaluation findings will contribute to the body of knowledge concerning the use of direct cash transfers as a recovery strategy following these types of extreme health emergencies. 2.2 Evaluation Scope The final end-of-project summative performance evaluation considers progress from April 2015 through to December 2017 and therefore assesses results for both Phase 1 and Phase 2 activities. Study locations cover Kailahun (Phase 1), Bombali and Tonkolili (Phase 1 and Phase 2) Districts. At the time of the final evaluation field work, over one year had passed since Phase 1 beneficiaries received their final cash transfer payment (July 2016); and approximately 30 days since Phase 2 beneficiaries received their final payment (Sept/Oct 2017). This required that the two groups be treated as separate populations to ensure that each is represented adequately in the field research. Data was collected at the end of Phase 1 and Phase 2 through exit interviews conducted during the final cash transfer - providing a consistent endline point for Phase 1 and Phase 2. Supplemental endline data was collected by NBI for Phase 1 beneficiaries as part of this evaluation to determine their status one year following their final cash transfer receipt. The exit survey questionnaire in Phase 2 (Sept/Oct 2017) and supplemental endline for Phase I was expanded to collect additional indicator data as agreed with USAID and FFP Implementing Partners that would contribute to broader comparison of impact data with other EFSP programs in Sierra Leone. 2. 3. Evaluation Questions There are three broad research areas for the final performance review: 1) impact on food security and improved resilience, 2) program systems, processes and procedures and 3) program coordination and collaboration. The specific objective of the evaluation is to answer the following key questions on these topics: Evaluation Question 1: To what extent did households that participated in the EFSP achieve food security and improved resilience? Evaluation question 1 entails a (i) quantitative and a (ii) qualitative component. (i) Food security and improved resilience to be measured by these five quantitative variables: 1. The household hunger scale (HHS) 2. Household dietary diversity score (HDDS) EFSP Final Project Performance Evaluation Page 6 3. Expenditure on food and non-food 4. Reduced coping strategies index 5. Meal frequency In addition, quantitative analysis of the HHS and HDDS will be disaggregated by participant/non￾participant in a number of key program activities. This will allow the evaluation to address the extent to which beneficiary participation in program activities contributed to improving the ability of households to mitigate, adapt, and recover from the economic impacts of the Ebola emergency. (ii) Key questions to be answered by beneficiaries through qualitative research: a. What are some of the activities you have undertaken to keep or grow the cash transfer money you received between (first cash transfer date and last cash transfer date)? b. Do you believe these activities will continue when you are no longer receiving assistance? c. Are you purchasing goods and services at prices similar to before cash transfer? If not, why do you believe prices have increased? d. From your perspective, what are some negative things that households do when food is scarce? Do you believe that the program was effective in reducing these negative behaviors? e. If there was another shock today, has the program prepared you to cope with the shock? How long could you cope without assistance, 1 month, 3 months, 6 months, longer? Evaluation Question 2: To what extent were the systems, processes, and procedures used by the program for delivering cash transfers and non-cash assistance to participants effective, efficient, and sustainable during the period of program implementation? In answering this question, the performance review will address the following areas: • Accountability, timeliness, and cost of delivering the cash and non-cash assistance to participating households • Assess the participants’ perception about the cash transfer and non-cash delivery processes and procedures • Explore the gender implications of cash transfer and non-cash assistance in terms of decision making, gender relations and protection Key questions to be answered include: a. What mechanisms were used to ensure that the right beneficiaries were targeted in all phases? Was there a change in approach or modalities due to emerging conditions/situations? If yes, can you please describe? b. During implementation, what were the mechanisms used to ensure beneficiaries, the public, and management were fully informed about what was happening at field level? c. What was the beneficiaries experience with the cash distribution process such as waiting time, walking distance, safety and grievance redress? d. Were beneficiaries ever pressured by local authorities, community identification committees, extended family or community members to give money they did not want to give? If so, how did they handle this? e. Has there been any other kind of conflict over the beneficiaries’ receipt of cash? f. Were there conditions that caused beneficiaries not to receive the cash at any point in time? How was this situation handled? How long did it take for ACDI/VOCA to respond to the situation? EFSP Final Project Performance Evaluation Page 7 Evaluation Question 3 - To what extent did the collaboration and coordination with public and private sector stakeholders result in enhanced capacity to deliver cash transfers as a form of social protection to vulnerable households and contribute to effective delivery of cash transfers during the emergency and recovery phases? In answering this question, the performance review will address the following areas: • Compliance with national standards and policies on social safety nets • Connectedness and coherence with other actors such as NGOs, donors, UN organizations, multilateral agencies, Government agencies, etc. Key questions to be answered include: a. How did ACDI/VOCA coordinate activities with national and local government authorities? b. How did ACDI/VOCA coordinate with other Food for Peace (FFP) implementing partners? How did that collaboration enhance achievement of results? c. How did ACDI/VOCA address the issues of parallel systems and structures? How did this impact results? d. How did other interventions in targeted communities impact results (leveraging funds, enforced or complement implementation, and sustainability)? 3.Methodology 3.1 Study Design The research methodology for the final performance review adhered to a mixed methodology using a combination of quantitative and qualitative methods to address the key evaluation questions outlined above. The study design was conducted in two separate components with findings combined in this report. Component One included a review of quantitative endline survey data which was previously collected during the last cash transfers in Phase 1 and Phase 2 (July 2016 for Phase 1 and September 2017 for Phase 2). Consisting of a representational sample of households receiving cash transfers taken at the last cash disbursements, this endline data measured change against the two primary impact indicators under the EFSP Program, household dietary diversity score (HDDS) and the household hunger scale (HHS). Raw data was provided to the evaluators for analysis and calculation of the final endline indicators presented in this report. In addition, the Evaluation Team undertook a supplemental endline of Phase 1 beneficiaries to measure the impact of the program one year after their final cash distribution. Component Two was a purely qualitative evaluation that sought to answer the key research questions on the effectiveness and efficiency of the program’s systems, procedures and coordination. A series of focus group discussions and key informant interviews were conducted to gather information on beneficiaries’ and key stakeholders’ attitudes and practices related to the EFSP. Respondents were sampled from beneficiaries in the SNAP+/EFSP intervention area, as well as key stakeholders from national, district and local government authorities. The evaluation also drew on a review of program documents and interviews with ACDI/VOCA and FFP implementing partner staff. EFSP Final Project Performance Evaluation Page 8 3.2 Survey Instruments The evaluation adopted participatory approaches which combined both qualitative and quantitative study methods. These were triangulated through a household survey (quantitative); focus group discussions; and key informant interviews (qualitative). Supportive document reviews and secondary quantitative data analysis were also carried out. The following data collection methods were employed during the final performance evaluation: ● A quantitative household questionnaire was administered to direct project beneficiaries of cash transfers from Phase 1. The structured questionnaire sought to solicit and assess project impact on household food security and resilience (namely data on HDDS and HHS), as well as the beneficiaries’ perceptions of the cash transfer processes. Using a similar survey to the one administered during the Phase I endline in July 2016, this supplemental Phase I endline data collection allowed for direct comparisons to be made against previous endline values. ● Focus group discussions (FGDs) were held with program beneficiaries, and community-level actors and stakeholders. The FGDs collected qualitative information and provided explanation to project impact, processes, challenges, best practices and community participation. Information collected from FGDs was also used to compliment and validate information collected through the other data collection techniques. ● Key informant interview guides were administered to program implementers, partners and stakeholders. Key informant interviews focused mainly on project coordination and collaboration. The interviews also solicited information on project challenges, lessons learned and best practices. Table 2: Evaluation questions and corresponding survey instrument used Evaluation Question Research Methods Question 1: To what extent did households that participated in the EFSP achieve food security and improved resilience? • Household questionnaire with Phase 1 beneficiaries (supplemental endline survey) • Secondary data analysis of previously collected endline data (Phase 1 and Phase 2) • Focus group discussions (FGDs) with program beneficiaries Question 2: To what extent were the systems, processes, and procedures used by the program for delivering cash transfers and non-cash assistance to participants effective, efficient, and sustainable during the period of program implementation? • Quantitative endline surveys (supplemental and Phase 1 and Phase 2 endlines) • Focus group discussions (FGDs) with program beneficiaries • Key informant interviews (KIIs) with major stakeholders and partners Question 3: To what extent did the collaboration and coordination with public and private sector stakeholders result in enhanced capacity to deliver cash transfers as a form of social protection to vulnerable households and contribute to effective delivery of food assistance during the emergency and recovery phases? • Key informant interviews (KIIs) with major stakeholders and partners EFSP Final Project Performance Evaluation Page 9 3.3 Sampling As stated above, there was no overlap between Phase 1 and Phase 2 beneficiaries (although there was overlap among higher level key informants such as government stakeholders at the national and local level). In order to ensure that each was adequately represented in the field research, beneficiaries from each phase were considered as two separate populations. The sampling design was developed with this in mind. 3.3.1 Quantitative Sampling Existing Endline Survey Data (Phase 1 and Phase 2) Originally an evaluation was not anticipated for the one-year emergency program. However, baseline data was collected and quantitative monitoring data was collected through an ‘exit survey’ administered during the cash transfers that included interim collection of the HDDS and HHS indicator data. The information collected was used to monitor progress against program targets; determine how money was being used and whether program messages were being adopted. When an extension of the program for a Phase 2 to include new beneficiaries was approved, it was agreed with USAID that the indicator data collected during the exit survey at the final cash transfer (July 2016) could be used for the endline data for what was now considered Phase 1 for purposes of a final evaluation. Further, it was suggested by FFP/Dakar that the same sampling and data collection methodology be used for the endline for Phase 2 beneficiaries for consistency. Exit surveys were administered to a representative sample of participants who received their cash transfer, using a 95% confidence interval with 5% margin of error (Phase I sample 1,333 and Phase II sample 866). SPSS tables and raw data collected during the final cash transfer exit/endline survey from Phase 1 and Phase 2 were provided to the evaluators for analysis. Supplemental Phase 1 Endline Survey The Evaluation Team undertook a supplemental endline of Phase I beneficiaries to measure the impact of the program one year after their final cash distribution. The minimum required sample size for the supplemental endline was estimated based on the household dietary diversity score (HDDS). The indicator value and the standard deviation were obtained from the Phase 1 endline dataset. The FANTA Sampling Guidelines4 were used to calculate a sample size capable of detecting a 0.7 point increase in the HDDS indicator over the one-year period since the last cash transfer.5 The minimum sample size required per survey round was computed as follows: n = D [(Zα + Zβ) 2 * (sd1 2 + sd2 2 ) / (X2 – X1) 2 ] *Nf n required minimum sample size per survey round D design effect, default value = 2.0 4 Magnani, Robert (1999). Sampling Guide. Washington, D.C.: FHI 360/FANTA 5 The expected value of HDDS is based on a 2017 study conducted by Welthungerhilfe and Concern Worldwide in the program district of Tonkolili which showed a HDDS of 7.4. EFSP Final Project Performance Evaluation Page 10 X1 HDDS last recorded at Phase 1 endline (last known measurement of HDDS among Phase I beneficiaries at the time of their last cash transfer, July 2016) = 6.7 X2 the expected value of the HDDS indicator one year later such that the quantity (X2-X1) is the size of the magnitude of change it is desired to be able to detected (0.7) = 7.4 sd1 observed standard deviation of the HDDS measured during the Phase 1 endline =2.34 sd2 expected standard deviation calculated using the ratio of X2/X1/sd1 observed in the Phase 1 endline survey = 2.58 Zα the Z-score corresponding to the degree of confidence with which it is desired to be able to conclude that an observed change of size (X2-X1) would not have occurred by chance (α – the level of statistical significance for one-tailed test), 95% = 1.645 Zβ the z-score corresponding to the degree of confidence with which it is desired to be certain of detecting a change of size (X2-X1) if one actually occurred (β - statistical power), 80% = 0.840 Nf Non-response factor (assuming a 10% non-response rate) = 1.10 Based on these parameter values, the estimated sample size (n) was 338 beneficiaries. The optimum sampling allocation was determined to be 25 clusters, with 15 households sampled for each, resulting in a total sample size of 375 beneficiaries. Sample Size Allocation Sample selection for the supplemental Phase 1 endline survey followed a two-stage cluster sampling methodology. For the first stage, beneficiary localities across Kailahun, Bombali and Tonkolili Districts were used as the sampling frame for probability proportional to size (PPS) cluster selection. Using a list of beneficiary households receiving cash at PPS selected localities, simple random selection was carried out to identify a total of 20 beneficiaries to be interviewed in each cluster.6 The list of selected beneficiaries was shared with local chiefs in each cluster a week prior to field work to ensure that sampled beneficiaries would be assembled at a central location for interview on a specified date. In total, 417 completed surveys were administered during the fieldwork. 3.3.2 Qualitative Sampling The qualitative evaluation also considered Phase I and Phase II beneficiaries as separate populations. Respondents were sampled from beneficiaries in the SNAP+/EFSP intervention areas as well as key stakeholders from national, district and local government authorities according to the sampling design outlined below. Focus Group Discussions (FGDs) Seven FGD categories were included in the research: female beneficiaries, male beneficiaries, pregnant/lactating women from beneficiary households, beneficiaries participating in a Village Savings and Loans Association (VSLA), beneficiaries with income-generating activities (IGA) started or supported by cash transfers, local leaders and community members involved in pre-identifying 6 An additional five names were included in the respondent selection to serve as replacements in the case where selected beneficiaries were not available for interview. All respondents that showed up for the survey were interviewed. EFSP Final Project Performance Evaluation Page 11 beneficiaries. Prior to implementation of research, these groups were identified in collaboration with ACDI/VOCA as most important to addressing the key evaluation questions. FGDs with pregnant/lactating women, beneficiaries participating in a VSLA and beneficiaries with IGAs were only held in Phase 2 communities. Research sites for FGDs were selected according to simple random sampling, using a number generator to select sites from a list of beneficiary localities (villages) in each of the targeted districts (separated by program phase). Selection of FGD participants relied on a combination of two forms of purposeful sampling: convenience sampling and maximum variation sampling. Using convenience sampling for FGDs, the evaluation leveraged focus group participants that were available at the time of data collection. However, convenience samples were also stratified by key characteristics to ensure maximum variation between groups; that is, heterogeneity of persons sampled (e.g. targeting beneficiaries participating in a VSLA and beneficiaries with IGAs in separate FGDs). Furthermore, sex composition was considered in all FGDs, to ensure, as much as possible, representation of both males and females. In total there were 29 FGDs conducted, with 305 participants. (See Annex 1 for a list of FGDs conducted). Key Informant Interviews Key informants were purposively sampled. A list of relevant individuals and their contact information was compiled in collaboration with ACDI/VOCA. Key informants included ACDI/VOCA staff implementing the program, key national government stakeholders (National Commission for Social Action (NaCSA) and the Anti-Corruption Commission (ACC)) and Food for Peace implementing partners. (See Annex 2 for a list of individuals consulted). 3.4 Data Collection 3.4.1. Training Training of field staff took place in Freetown from 25 – 28 October 2017. The training focused on the study background, sampling procedures, interviewing techniques and familiarization with the data collection tools. Practice sessions, group discussions and mock interviews also ensured that all interviewers asked questions correctly and consistently, and that data abstraction was reliably done. During the overview of the questionnaire and practice sessions, participants conducted local language translations to questions in order to ensure that valid data was collected from the field. 3.4.2. Fieldwork and Data Capture The NBI core evaluation team was composed of a Project Manager, a Research Manager, a Food Security and Nutrition Specialist and a Research Assistant/Field Coordinator. A total of 20 field staff and three supervisors conducted primary data collection at community-level, supervised by the core evaluation team. Field data collection was conducted from 30 October – 7 November 2017. Android phones were used for data collection, complemented with SurveyCTO software. The use of mobile devices and an electronic questionnaire strengthened data quality by allowing data validation rules and consistency checks that were integrated in the SurveyCTO software program. The mobile-based data collection process eliminated the data entry burden, as data was entered at the enumerator level and records EFSP Final Project Performance Evaluation Page 12 were uploaded to a cloud server. This allowed the data analysis team to review data consistency periodically throughout the data collection phase. Focus group discussions were voice-recorded during fieldwork and were accompanied by short-hand notes to support transcription. In order to address scheduling conflicts, key informant interview guides were translated into online surveys which were emailed to all selected key informants to complete at their own convenience. In the case where key informants were available for interview, in-person and phone interviews were conducted. In order to ensure quality control during data collection, supervisors kept fieldwork control sheets and conducted random spot checks to assess the field staff's proper administration of the various sections of the questionnaire and FGD guides. Supervisors also reviewed every completed questionnaire on the same day of data collection and checked for adequate completion of all fields, missing data and legibility of open-ended items, in addition to checking FGD short hand notes for completeness. Enumerators were required to make corrections or return for subsequent interviews and discussions after the reviews if necessary. 3.5 Data Processing and Analysis Quantitative Survey Data As data collection was conducted using mobile devices in the field, data entry occurred in real time during fieldwork. The SurveyCTO dataset (CSV format) was converted into an SPSS (Version 21) database for data management and analysis. Data was cleaned for logical consistency, skimming, skip patterns, outlying variables, and invalid and missing responses. The data analysis and tabulation of key indicators for impact measurement (HHS and HDDS) followed the tabulation methods documented in the USAID FFP Standard Indicators Handbook. 7 As per the evaluation criteria, quantitative analysis of the key impact indicators HHS and HDDS were disaggregated by participant/non-participant in a number of key program activities. Statistical significance between participant and non-participant was evaluated using chi-square (HHS) and independent samples t test (HDDS) at a 95% level of significance. A comparison of baseline with endline values of project indicators was conducted using both z-tests and t-tests (where appropriate) to detect a population level difference at a 95% level of confidence.8 Qualitative Interview Data All FGDs were audio recorded and short hand notes were written while in the field. Recordings were then transcribed and translated by the research assistant who conducted the interview. Key informant interview responses to the online survey were downloaded into Excel format for analysis. Qualitative data underwent a thorough and systematic analysis to draw out the major themes and findings as they relate to the evaluation objectives. The framework approach was used to analyze the qualitative data collected through the evaluation. Narratives of the qualitative data along with illustrative quotes are presented along with the quantitative findings in the report. To ensure validity 7 FFP Indicators Handbook Part I: Indicators for Baseline and Final Evaluation Surveys. April 2015. Washington, DC: Food and Nutrition Technical Assistance III Project (FANTA III), 2015. 8 Z-tests for proportions were used on all categorical binary indicators; while t-tests of significance were used to compare the means of interval dependent variables for the two study groups. EFSP Final Project Performance Evaluation Page 13 and allow for generalization, data were triangulated across all quantitative and qualitative data collection tools, to build consensus on the findings. 3.6 Limitations This section summarizes study limitations and issues encountered during the performance evaluation. Validity and Reliability of Self-reported Data The evaluation relied primarily on information provided by the beneficiaries themselves to determine the project’s effectiveness. Self-reporting has several limitations, such as the possibility of exaggeration or omission of information, inaccurate recollection of experiences or events, acquiescence bias or reporting of untruthful information and reduced validity when respondents do not fully understand a question. The evaluation’s mix of quantitative and qualitative approaches aimed to mitigate these risks. Influence of Expectations Prior to the start of fieldwork, former ACDI/VOCA staff members familiar with, but unrelated to the cash transfer program, visited each sampled community to introduce the objectives of the evaluation to the town chiefs. A key aspect of the messaging was to reiterate that no further registration would be taking place during the evaluation data collection period. This message was repeated as part of the field entry strategy by all field workers. Nonetheless, in some communities, field workers were met with large gatherings of community members who were under the misconception that the team was there to register them. This created serious time delays, and in some cases, difficulties for field staff during data collection. Local leaders were enlisted to support the field staff in reiterating the objectives of the data collection exercise in order to better manage expectations among the community members. Further, during data collection with beneficiaries the field team noted that respondents in Phase 2 (who just received their last cash transfer) were frequently more likely to respond to questions in a manner intended to elicit further assistance from ACDI/VOCA. In these cases, field staff repeatedly reminded the respondents of the evaluation objectives and the importance of answering truthfully. The field team also used triangulation from other questions during the discussion to identify potentially misleading responses. This issue was not frequently encountered with Phase 1 beneficiaries, who receiving their last cash transfer over a year ago, were less likely to voice expectations for future assistance. Timing of Data Collection The Phase 1 baseline and endline were conducted during the lean season when households may be less food secure and negative coping strategies are more likely to occur. In contrast, the baseline for Phase 2 took place shortly after the harvest season, while the endline took place during the early harvest season – both times when food is more available and accessible. Where applicable, the evaluation explicitly discusses these differences when examining data on household food security as the timing of data collection can impact the household’s access to food and overall achievement on key food security measures. EFSP Final Project Performance Evaluation Page 14 Absence of a Comparison Group The prescribed evaluation design did not include a comparison (or control) group of households that did not participate directly in the cash transfer program. This makes it difficult to draw direct conclusion regarding the causal relationship between the intervention and outcomes. Where possible, the evaluation team relied on the evaluation’s mix of quantitative and qualitative approaches to triangulate findings in order to make the attribution of change to the initiative more robust. EFSP Final Project Performance Evaluation Page 15 4.Evaluation Findings The evaluation findings are organized according to the three main evaluation areas: (1) food security and improved resilience; (2) program systems, processes and procedures; and (3) program coordination and collaboration. Each section is organized by the relevant key evaluation questions. Where appropriate, the findings are presented for Phase 1 and Phase 2 separately. 4.1 Food Security and Improved Resilience This section concerns the extent to which beneficiary households achieved food security and improved resilience. As impact is largely measured by five key quantitative variables, endline data is analyzed and compared to baseline levels (where possible). The achievement of food security and improved resilience is also measured through a number of key qualitative measures. Information provided has been gathered through quantitative endline surveys and focus group discussions with different groups of beneficiaries. 4.1.1. Household Hunger Scale (HHS) The Household Hunger Scale (HHS) is one of the main indicators for assessing the impact of the UCT on household food security. It measures the percent of households experiencing hunger according to the following categories of food deprivation: (1) little to no hunger; (2) moderate hunger; and (3) severe hunger. As Phase 1 and Phase 2 beneficiaries were treated as separate populations, the data is presented for each phase separately to allow for comparisons between baseline and endline. Phase 1 There has been a dramatic improvement in the HHS across beneficiary households since the start of the program (see Figure 2). Data from (a) the endline survey conducted at last cash transfer and (b) the supplemental endline conducted more than one year after the last cash transfer revealed improvements across all categories of food deprivation from baseline levels. For instance, the proportion of households reporting ‘little to no hunger’ grew from 4.7% at baseline, to 66.2% at endline to 69.1% one year post-program (Supplemental Endline (Phase 1)). Overall, the findings suggest that improvements to household food security have been sustained a year after the last cash transfer. Figure 2: Household Hunger Scale, at baseline and endline (Phase 1) 4.7% 66.2% 69.1% 33.1% 30.5% 8.7% 0.7% 0.4% 86.6% Baseline (July 2015) Endline (July 2016) Supplemental Endline (Oct 2017) Little to no hunger Moderate hunger Severe hunger EFSP Final Project Performance Evaluation Page 16 In order to better understand the changes that have taken place since baseline, the standard FFP indicator on the ‘percentage of households with moderate or severe hunger’9 was analyzed more closely (see Figure 3). During the SNAP+ baseline assessment conducted prior to the start of the program, nearly all (95.3%) of the beneficiary households were classified as having ‘moderate or severe’ hunger. By the end of the program, this had decreased to 33.8%. More promising, the supplemental endline found that these improvements have been sustained more than a year after the last cash transfer. In fact, there has been a further slight decrease in the proportion of households reporting moderate or severe hunger from 33.8% at endline, to 30.9% one year later. This suggests that the considerable improvement in food security since baseline has been sustained a year on without the assistance of the cash transfers. The overall change in the HHS from baseline to the supplemental endline represents a statistically significant decrease of 64.4 percentage points (from 95.3% at baseline to 30.9% more than one year after the last cash transfer).10 Figure 3: Percentage of households with moderate or severe hunger, at baseline and endline (Phase 1) The extent to which the improvements in the HHS can be attributed to the cash transfer program (CTP) is not straightforward as quantitative data does not allow a comparison between beneficiaries and non-beneficiaries. Furthermore, there were other external changes taking place during the post￾Ebola emergency period which may have also impacted on household food security. For instance, quarantines and travel-bans were largely lifted as the cash transfers began, supporting farming and market activities to reach near pre-EVD levels by the end of Phase 1. Nevertheless, qualitative research with beneficiaries suggests that the cash transfers were integral to enabling households to restore food security, making the attribution of change to the initiative more robust. Respondents reported using the cash transfer to start-up income generating activities which were negatively impacted by the EVD emergency. One year later, many attributed the continuation of these businesses and the resulting profit as the key to their household’s food security. Others mentioned their membership in a Village Savings and Loan Association (VSLA) set up through SNAP as enabling them to save money and providing a safety net when loans are needed. As proof of the 9 To tabulate the standard FFP indicator, only the cutoff values for moderate and severe hunger need to be applied to the data, as the indicator to be reported to FFP is the percentage of households with moderate or severe hunger. For more information see: FFP Indicators Handbook Part I: Indicators for Baseline and Final Evaluation Surveys. April 2015. Washington, DC: Food and Nutrition Technical Assistance III Project (FANTA III), 2015. 10 z=-62.10, p=0.001. This p-value corresponds to a less than 0.1% probability that the differences measured between the baseline and the supplemental endline (1-year post intervention) were due to chance. 95.3% 33.8% 30.9% Baseline (July 2015) Endline (July 2016) Supplemental Endline (Oct 2017) EFSP Final Project Performance Evaluation Page 17 interventions impact on their lives, others pointed to the situation of non-recipient households in their community which are still struggling to recover from the impact of Ebola one year later. “The participation in SNAP[+] has helped me recover from the loss I experienced during the Ebola crisis by starting new business, our farm and purchasing cattle. During the Ebola I was unable to do any farming or business which led us to stop all business and farm production. But with the help of SNAP[+] I was able to start new business in livestock, pay fees, eat good food and do farming. I also used the money to pay medical bills and save some. Now, I am still using the profit from the business and farming to provide for my family today.” (Female beneficiary, Bombali District, Kunthai village, Phase 1) “I can now buy food from the profit I made from my palm oil sales to buy more food stuff for my household which was not possible before I started participating in the SNAP [+] project. Since I join the SNAP box I can now save some cash and loan money to buy food stuffs when needed and thereby making my household food secured which was not possible before becoming a participant in the SNAP [+] project.” (Female beneficiary, Kailahun District, Geihun village, Phase 1) “I have livestock now because of the SNAP [+] program, which the non-beneficiaries cannot afford to buy. Because of the SNAP [+] money, I am able to provide food for my own household which the other households that did not get the money still are struggling to do since Ebola.” (Male beneficiary, Bombali District, Fandehun village, Phase 1) 4.1.2. HHS by Beneficiary Participation in Program Activities As part of the performance evaluation, a more in-depth analysis of the standard FFP indicator on the ‘percentage of households with moderate or severe hunger’11 for Phase 1 data was conducted to include a: • comparison of households with moderate or severe hunger reporting Village Savings and Loans Association (VSLA) membership and those who are not VSLA members • comparison of households with moderate or severe hunger reporting to have started or strengthened an income generating activity (IGA) using cash transfer income and those reporting that they do not have an IGA HHS by Village Savings and Loans Association Membership (Phase 1) During Phase 1, ACDI/VOCA supported the establishment of Village Savings and Loans Associations (VSLAs). It was hypothesized that supporting increased access to credit for cash transfer beneficiaries’ would support their sustainable recovery by limiting their dependence on debt, providing a safe mechanism for saving cash and encouraging investments in IGAs. The supplemental endline asked Phase 1 beneficiaries if they had joined a VSLA anytime between their first cash transfer and their last. Nearly four out of ten (38.4%) respondents reported that they had. Findings suggest that beneficiaries that reported VSLA membership had significantly lower levels of moderate to severe hunger (23.1% compared to 35.8% for non-VSLA members) one year after their last cash transfer. This is an encouraging finding as it suggests that VSLA membership had a lasting impact on the household’s longer term food security and resilience. 11 Only the cutoff values for moderate and severe hunger need to be applied to the data, as the indicator to be reported is the percentage of households with moderate or severe hunger. EFSP Final Project Performance Evaluation Page 18 Table 3: Moderate or severe hunger, by VSLA membership (Phase 1) Moderate or Severe Hunger VSLA membership VSLA 23.1%** (p=.006) Non-VSLA 35.8%** (p=.006) Supplemental Endline Value 30.9% *p < 0.05; **p < 0.01; *** p< 0.001 HHS by Income Generating Activities (IGA) (Phase 1) Unconditional cash transfers (UCTs) provided an opportunity for beneficiaries to use the cash according to their own household’s needs. In many cases, this meant investing a proportion of the cash in income generating activities (IGAs) to contribute to their longer term recovery and resilience. According to supplemental endline findings, the majority (68.3%) of respondents reported that they had used a portion of the money to start or continue an IGA. A year since the last cash transfer, these households were found to have substantially lower levels of moderate to severe hunger. For instance, while 26.7% of households that reported starting or supporting an IGA with the cash transfer were classified as having moderate to severe hunger, 40.2% were classified as the same among non-IGA households. These findings suggest IGAs have the potential to improve the effectiveness of cash transfers to impact long-term recovery and food security. Table 4: Moderate or severe hunger, by IGA (Phase 1) Moderate or Severe Hunger IGA status IGA 26.7%** (p=.006) Non-IGA 40.2%** (p=.006) Supplemental Endline Value 30.9% *p < 0.05; **p < 0.01; *** p< 0.001 Phase 2 Phase 2 beneficiaries reported significant improvements in HHS by the end of the cash transfer program (CTP) intervention. The percentage of households reporting moderate or severe hunger significantly decreased from 99.7% at baseline to 13.1% at endline, representing a statistically significant reduction of over 86 percentage points.12 Figure 4: Household Hunger Scale, at baseline and endline (Phase 2) 12 z = -471.37, p = 0.001. This p-value corresponds to a less than 0.1% probability that the differences measured between the baseline and endline were due to chance. 0.3% 86.9% 52.1% 13.1% 47.6% 0.0% Baseline (Apr 2017) Endline (Oct 2017) Little to no hunger Moderate hunger Severe hunger EFSP Final Project Performance Evaluation Page 19 The relatively shorter duration of the Phase 2 cash transfer program suggests that in the immediate￾term, the EFSP was able to dramatically impact the food security of beneficiary households. Again, while a direct correlation cannot be ascertained in the absence of a comparison group of non￾beneficiaries, FGD participants overwhelmingly credited their receipt of the cash transfer as directly linked to their increased food security. Much like the Phase 1 beneficiaries, respondents reported that the cash transfers supported income generating activities, paid school fees and provided adequate income to improve household food security. “For me since Ebola ended in the country, we have struggled even to get our daily living, let alone to pay fees for the children. But since we started receiving this money, it has helped us a lot, in paying our children’s fees and increasing our consumption habits. We used to manage to cook two cups of rice a day. Due to the money we now cook four to five cups, we buy seeds, pay our loans etc.” (Pregnant beneficiary, Tonkolili District, village, Phase 2) After Ebola we didn't have money to send our children back to school. Through this cash transfer we send our children to school and I start up my own business which is doing well. I use part of the money to repair my house, and use the remaining to buy food stuff. As I am speaking to you now before Ebola me and my family use to eat one time per day but since the starting of the cash transfer we are now eating three time per day thanks to the people of American and the SNAP [+]. (Male beneficiary, Bombali District, village, Phase 2) HHS by Beneficiary Participation in Program Activities As part of the performance evaluation, a more in-depth analysis of the standard FFP indicator on the ‘percentage of households with moderate or severe hunger’13 for Phase 2 data was conducted to include a: • comparison of households with moderate or severe hunger reporting VSLA membership and those who are not VSLA members • comparison of households with moderate or severe hunger reporting to have started or strengthened an Income Generating Activity (IGA) using Cash Transfer income and those reporting that they do not have an IGA • comparison of households with moderate or severe hunger reporting to have used program nutrition messages to make dietary decisions and those who reported that they did not consider program nutrition messages for household dietary decisions • comparison households with moderate or severe hunger reporting to have used program economic and financial management messages to make household spending decisions and those who reported that they did not consider program economic and financial management messages for household spending decisions HHS by Village Savings and Loans Association Membership (Phase 2) The approach to VSLAs shifted during Phase 2. In Phase I, ACDI/VOCA introduced a private service provider (PSP) methodology to expand VSLA services in SNAP operational areas. While a number of cash transfer households participated, it was not intentional. Observing the participation of cash transfer beneficiaries in VSLA activities, ACDI/VOCA took a more intentional approach training PSPs 13 Only the cutoff values for moderate and severe hunger need to be applied to the data, as the indicator to be reported to FFP is the percentage of households with moderate or severe hunger. EFSP Final Project Performance Evaluation Page 20 in new operational areas to offer VSLAs as a complimentary activity for Phase II participants. The premise was the same – this form of non-cash assistance was introduced to build the resilience of cash transfer beneficiary households by breaking the cycle of debt, while increasing access to credit and savings. The overwhelming majority of endline respondents reported joining a VSLA (81.4%). Much like the findings in Phase 1, beneficiaries who reported VSLA membership in Phase 2 showed lower levels of moderate or severe hunger, although the relationship was not statistically significant. It is important to point out that given the relatively shorter implementation period of the second phase of the EFSP, this could potentially limit the measurable effectiveness of VSLA membership at endline. In fact, many FGD respondents reported that their own community’s VSLAs was just being set up at the time of the evaluation. This would restrict the ability of the VSLA to properly establish the fund and engage in sufficient rounds of lending. Table 5: Moderate or severe hunger, by VSLA membership (Phase 2) Moderate or Severe Hunger VSLA membership VSLA 12.3% (p=.167) Non-VSLA 16.4% (p=.167) Endline Value 13.1% *p < 0.05; **p < 0.01; *** p< 0.001 HHS by Income Generating Activities (Phase 2) Nearly nine out of ten (88.1%) Phase 2 beneficiaries reported that they had started or continued an income generating activity (IGA) with the money from the UCT. When the HHS for this group is calculated, we see that households with IGAs supported by the cash transfers were more likely to be classified as having moderate to severe hunger. However, as the difference in HHS between IGA and non-IGA households was not statistically significant, it is difficult to draw any robust conclusions from this data. Table 6: Moderate or severe hunger, by IGA (Phase 2) Moderate or Severe Hunger IGA Status IGA 13.7% (p=.144) Non-IGA 8.6% (p=.144) Endline Value 13.1% *p < 0.05; **p < 0.01; *** p< 0.001 HHS by Program Nutrition Messages to make Dietary Decisions (Phase 2) Phase 2 included beneficiary-based training activities related to maternal, child and household nutrition. Regular delivery of Infant and Young Child Feeding (IYCF) best practice messaging during distributions was coupled with cooking demonstrations on preparation of nutritious complementary feeding foods made from locally-available ingredients. In addition, Community Facilitators and Lead EFSP Final Project Performance Evaluation Page 21 Mothers were engaged to deliver key nutrition messages at community level outside of cash distribution points. Overall, six out of ten (59.7%) Phase 2 endline survey respondents reported that they had participated in any nutrition training during the program. The percent of household reporting ‘moderate to severe hunger’ among those that reported attending nutrition training was 5.5% (compared to 13.1% for the overall total sampled population). During the endline survey, participants were also asked if they had used program nutrition messages to make dietary decisions. Among households that attended the training, those that reportedly used the nutrition messages to make household dietary decisions had significantly lower levels of moderate to severe hunger (5.0%) when compared to those who reportedly did not use the messages (12.9%). This suggests that the program nutrition messages were relevant and effective at addressing key food security issues when they were used by beneficiaries. (See discussion in Section 5.2.3 on the nutrition training). Table 7: Moderate or severe hunger among beneficiaries that attended nutrition training, by use of nutrition messages (Phase 2) Moderate or Severe Hunger Use of program nutrition messages Used program nutrition messages to make dietary decisions 5.0%*** (p=.000) Did NOT use program nutrition messages to make dietary decisions 12.9%*** (p=.000) Adjusted Endline Value14 5.5% *p < 0.05; **p < 0.01; *** p< 0.001 HHS by Program Financial Management Messages to make Spending Decisions (Phase 2) Building on the learning from Phase 1, ACDI/VOCA introduced training on household economic and financial management for all registered beneficiaries. The training focused on messages about the prioritization of the cash for recovery of food security as well as basic household budgeting and decision-making. Sixty Four percent (64%) of Phase 2 endline survey respondents reported that they had participated in any financial management training during the program. Among those that reported attending financial management training, the percent of households reporting ‘moderate to severe hunger’ was 8.1% (compared to 13.1% for the overall total sampled population). During the endline survey, those households that reported attending financial management training were also asked whether they used the messages to make household spending decisions. Those that reported using the economic and financial management messages reported a lower prevalence of moderate to severe hunger (8.0%) when compared to those that attended the training but did not use the messages (11.1%). (See discussion in Section 5.2.3 on the household economic training). 14 Comparison of HHS for households reporting to have used program nutrition messages to make dietary decisions was conducted only for households that reported attending training. In this case, the percent of household reporting ‘moderate to severe hunger’ among those that reported attending nutrition training was 5.5% (compared to 13.1% for the overall total sampled population). EFSP Final Project Performance Evaluation Page 22 Table 8: Moderate or severe hunger among beneficiaries that attended financial management training, by use of financial management messages (Phase 2) Moderate or Severe Hunger Use of program financial messages Used program economic and financial management messages to make household spending decisions 8.0%*** (p=.000) Did NOT use program economic and financial management messages to make household spending decisions 11.1%*** (p=.000) Adjusted Endline Value15 8.1% *p < 0.05; **p < 0.01; *** p< 0.001 Household Dietary Diversity Score (HDDS) The Household Dietary Diversity Score (HDDS) is the other main program indicator for assessing the impact of the CTP on household food security. The HDDS is a proxy measure for household food access and is based on the number out of 12 food groups consumed by any household member in the 24 hours preceding the survey.16 The resulting HDDS ranges from 0 to 12, with lower numbers indicating less dietary diversity and access to food. As Phase 1 and Phase 2 beneficiaries were treated as separate populations during quantitative data analysis, the data is presented for each phase separately to allow for comparisons between baseline and endline. Phase 1 Figure 5 presents the results of the HDDS for Phase 1 beneficiaries at baseline, endline and one year after program implementation. Dietary diversity substantially improved throughout program implementation, and in a sustained way. The average HDDS significantly grew from 1.9 at baseline, to 6.7 at endline, to 7.4 one year post-program implementation. This represents a statistically significant increase of five food groups on average since the start of the program.17 Overall, the findings suggest that improvements to household food access have been sustained a year after the last cash transfer – implying that longer-term household food security has improved without requiring a cash transfer. It is important to note, that while the baseline and endline surveys were both administered during the lean season, the supplemental endline took place at the start of the harvest season – which may have influenced the slightly higher score. 15 Comparison of HHS for households reporting to have used financial management messages to make household decisions was conducted only for households that reported attending training. In this case, the percent of household reporting ‘moderate to severe hunger’ among those that reported attending financial management training was 8.1% (compared to 13.1% for the overall total sampled population). 16 The set of 12 food groups is derived from the U.N. Food and Agriculture Organization Food Composition Table for Africa. The 12 food groups include: cereals, roots and tubers, vegetables, fruits, meat/poultry/offal, eggs, fish and seafood, pulses/legumes/nuts, milk and milk products, oil/fats; sugar/honey and miscellaneous condiments. 17 t=52.18, p=0.001. This p-value corresponds to a less than 0.1% probability that the differences measured between the baseline and the supplemental endline (1-year post intervention) were due to chance. EFSP Final Project Performance Evaluation Page 23 Figure 5: Household Dietary Diversity Score, at baseline and endline (Phase 1) The extent to which the improvements in the HDDS can be attributed to the cash transfer program (CTP) is not possible to assess on the basis of this quantitative data alone. In the absence of a comparison to non-beneficiaries, FGD respondents noted that they were able to broaden the number of food items they consumed as a result of the cash transfers. As proof of the UCT’s impact, some respondents reported that the food consumption of non-recipients did not improve as significantly as that of recipients. In many cases, households reported making additions to their diets (adding fish) and increasing the frequency of consumption of certain food groups. The qualitative fieldwork indicates that these improvements were introduced during the initiative but have also continued since it ended. “For me it has changed my household diet. Now we eat better and more food every day. Before we couldn't always have rice or fish. Now we have more food.” (Male beneficiary, Kailahun District, village, Phase 1) “The way I am eating at home is different from those that didn’t participate in the program. We are now able to eat more different foods because of our participation in the SNAP [+]. I have been able to start business and farming for my household and those who didn’t take part have not been able to get as much as us.” (Male beneficiary, Bombali District, village, Phase 1) HDDS by Beneficiary Participation in Program Activities (Phase 1) As part of the performance evaluation, a more in-depth analysis of the HDDS for Phase 1 was conducted to include a comparison of VSLA membership and IGA status among beneficiaries. HDDS by Village Savings and Loans Association Membership (Phase 1) Beneficiaries that reported VSLA membership had significantly higher household dietary diversity scores one year after their last cash transfer (8.1 compared to 6.9 for non-VSLA members). Coupled with the effect of VSLA membership on the HHS, this is a promising finding as it suggests that VSLA membership had a lasting impact on the household’s longer-term food security and food access. As discussed in greater detail in Section 5.2.3, VSLA members reported having greater access to credit and savings which has served to build their resilience over the last year. Table 9: Household Dietary Diversity Score, by VSLA membership (Phase 1) Total VSLA membership VSLA 8.1*** (p=.000) Non-VSLA 6.9*** (p=.000) Supplemental Endline Value 7.4 *p < 0.05; **p < 0.01; *** p< 0.001 1.9 6.7 7.4 Baseline (July 2015) Endline (July 2016) Supplemental Endline (Oct 2017) EFSP Final Project Performance Evaluation Page 24 HDDS by Income Generating Activities (Phase 1) Phase 1 beneficiaries who reported that they had started or continued an income generating activity (IGA) with the money from the UCT were found to have higher HDDS one year after the last cash transfer, compared to those who had not (7.7 compared to 6.7 respectively). This finding is notable because although the data collection took place in the harvest season, IGA households were more likely to have diversified their food basket at a time when there is greater availability of food items for all households. As discussed in Section 5.1.7, beneficiaries who have started or continued an IGA with the cash transfer reported that the returns on their investments have increased their household’s access to food and longer-term resilience. Table 10: Household Dietary Diversity Score, by IGA (Phase 1) Total IGA status IGA 7.7*** (p=.000) Non-IGA 6.7*** (p=.000) Supplemental Endline Value 7.39 *p < 0.05; **p < 0.01; *** p< 0.001 Phase 2 The average number of food groups consumed by Phase 2 beneficiary households doubled throughout program implementation. As outlined in Figure 6, the average HDDS for Phase 2 beneficiaries at baseline was 4.3 out of 12 possible food groups compared to 8.2 at endline.18 The slightly higher endline HDDS reported in Phase 2 (compared to Phase 1) is likely owing to the fact that these households have been actively receiving cash transfers.19 While there may be an anticipated slight drop in the HDDS for Phase 2 beneficiaries after the last cash transfer, findings from Phase 1 suggest that program beneficiaries have been largely able to sustain higher access to food groups one year after their last cash transfer relative to pre-intervention levels. Figure 6: Household Dietary Diversity Score, at baseline and endline (Phase 2) The substantial growth in the HDDS suggests that the CTP led to immediate improvements in household food access. While direct causation cannot be ascertained in the absence of a comparison 18 t=71.95, p=0.001. This p-value corresponds to a less than 0.1% probability that the differences measured between the baseline and the endline were due to chance. 19 Furthermore, the slightly higher HDDS reported at Phase 2 baseline (compared to Phase 1) is likely owing to the fact that Phase 2 data was collected shortly after the harvest season and during the fruit season, while Phase 1 data was collected during the lean season. 4.3 8.2 Baseline (April 2017) Endline (Oct 2017) EFSP Final Project Performance Evaluation Page 25 group of non-beneficiaries, FGD participants cited the receipt of the cash transfer as directly linked to their increased access to food. However, the link was not always explicitly focused on increases to their dietary diversity. The qualitative research suggests that households were able to increase their meal frequency and ensure more regular access to food after they received the cash transfer through their engagement in IGAs. Respondents reported diversifying how they spent the cash transfer to ensure they could meet their longer term food access needs. As highlighted by the female beneficiary below. “During the Ebola, I lost my husband and I have two children that are presently with me. The first money I was able to pay their school fees, buy food and also do some business. Before I received the money, my children were not eating sufficient food at home but since I received the money, they are all eating two big meals every day. The balance money I was able to do business to generating more income and also invest in my farm. The last payment the money received, I was able to continue doing business and put more money into the farm. This helps me to continue providing school fees and food for my home.” (Female beneficiary, Bombali District, village, Phase 2) HDDS by Beneficiary Participation in Program Activities (Phase 2) As with the HHS, a more in-depth analysis of the HDDS for Phase 2 data was conducted to include a comparison of VSLA membership and IGA status, as well as household use of program nutrition and financial management messages. HDDS by Village Savings and Loans Association Membership (Phase 2) There was very little difference in the average number of food groups consumed by beneficiary households with VSLA membership, versus those without (8.2 versus 8.6 respectively). Previously discussed, as VSLAs were still being established at the time of the endline, their potential impact on beneficiary’s food security may not yet be fully realized. If findings from Phase 1 are to be any indication, we can expect to see greater impact of the VSLAs in the medium-term. Table 11: Household Dietary Diversity Score, by VSLA membership (Phase 2) Total VSLA membership VSLA 8.2** (p=.002) Non-VSLA 8.6** (p=.002) Endline Value 8.2 *p < 0.05; **p < 0.01; *** p< 0.001 HDDS by Income Generating Activities (Phase 2) Beneficiaries who reportedly started or strengthened an income generating activity (IGA) using funds from the cash transfer had a higher average HDDS when compared to non-IGA beneficiaries. This is an interesting finding as an inverse relationship was found when analyzing the HHS. It can be recalled that IGA-households in Phase 2 were more likely to report moderate to severe hunger on the Household Hunger Scale. While the HDDS and HHS are reasonably well correlated, they likely measure different dimensions of food security.20 This may explain the difference in findings. 20 Vaitla, B. et al. Comparing Household Food Security Indicators to Inform IPC Acute Food Insecurity Phase Classification. Available at: https://www.fantaproject.org/sites/default/files/resources/HFCIS-poster-Oct2015.pdf EFSP Final Project Performance Evaluation Page 26 Table 12: Household Dietary Diversity Score, by IGA (Phase 2) Total IGA status IGA 8.4*** (p=.000) Non-IGA 7.1*** (p=.000) Endline Value 8.2 *p < 0.05; **p < 0.01; *** p< 0.001 HDDS by Program Nutrition Messages to make Dietary Decisions (Phase 2) Recall that 59.7% of Phase 2 endline survey respondents reported that they had participated in any nutrition training during the program. The average HDDS among those that reported attending nutrition training was 8.1 (compared to 8.2 for the overall total sampled population). Beneficiaries that received and used the nutrition messages to make dietary decisions were more likely to have a higher average HDDS than households that did not. These households were found to consume, on average, one more food group compared to those who did not use program nutrition messages to make dietary decisions (8.2 compared to 7.3, respectively). Table 13: Household Dietary Diversity Score among beneficiaries that attended nutrition training, by use of nutrition messages (Phase 2) Total Use of nutrition messages Used program nutrition messages to make dietary decisions 8.2*** (p=.001) Did NOT use program nutrition messages to make dietary decisions 7.3*** (p=.001) Adjusted Endline Value21 8.1 *p < 0.05; **p < 0.01; *** p< 0.001 HDDS by Program Financial Management Messages to make Spending Decisions (Phase 2) Of the 64.0% of Phase 2 endline survey respondents that reportedly participated in any financial management training, the average HDDS among this group was 8.1 (compared to 8.2 for the overall total sampled population). As highlighted in Table 15, the largest difference in household dietary diversity was observed between beneficiaries who did, and did not reportedly use program economic and financial management messages to make household spending decisions. Beneficiaries who used the training messages had an average HDDS of 8.2. On the other hand, beneficiaries who received, but reportedly did not use the messages had a HDDS far below the average at 6.0. This is an encouraging finding which suggests that the complimentary training provided has had a positive impact on the beneficiaries’ food security. 21 Comparison of HDDS for households reporting to have used program nutrition messages to make dietary decisions was conducted only for households that reported attending training. In this case, the average HDDS among those that reported attending nutrition training was 8.1 (compared to 8.2 for the overall total sampled population). EFSP Final Project Performance Evaluation Page 27 Table 14: Household Dietary Diversity Score among beneficiaries that attended financial management training, by use of financial management messages (Phase 2) Total Use of program financial messages Used program economic and financial management messages to make household spending decisions 8.2*** (p=.000) Did NOT use program economic and financial management messages to make household spending decisions 6.0*** (p=.000) Adjusted Endline Value22 8.1 *p < 0.05; **p < 0.01; *** p< 0.001 Expenditure on Food and Non-Food In order to assess household expenditure on food and non-food items throughout the program, ACDI/VOCA collected data using a ‘bean counting’ protocol. Administering this protocol involved asking beneficiaries to imagine that 10 beans represented all the money they had received during the previous cash transfer. By placing the number of beans (i.e. money) spent on four main spending categories (food, health care, school expenses and other), the survey was able to provide details on the percentage of cash that was spent on each category. As Phase 1 and Phase 2 beneficiaries were treated as separate populations, the data is presented for each phase separately to allow for comparisons between endline and previously collected data. Phase 1 As part of the supplemental endline, beneficiary households were asked to report on the percentage of money they spent last month on the four main spending categories. As the same exercise was used during exit interviews with Phase 1 beneficiaries after their second and third cash transfer, the data provides an interesting glimpse into how beneficiary households spent the cash while receiving the UCT versus how households are spending their money a year later. The results show that while there has been very little change in terms of spending on food and health care (accounting for over half of costs), there has been a marked increase in the household’s expenditure on school expenses one year later. This suggests that in the absence of receiving the cash transfer, households have not only been able to maintain their spending on food, they have been able to put a larger proportion of their resources towards education. This finding confirms the results from a previous assessment undertaken by ACDI/VOCA which found that the CTP had a significant impact on children’s access to education.23 22 Comparison of HDDS for households reporting to have used financial management messages to make household spending decisions was conducted only for households that reported attending training. In this case, the average HDDS among those that reported attending financial management training was 8.1 (compared to 8.2 for the overall total sampled population). 23 “Results determined that 90 percent of children in CTP recipient households indicated that they were able to return to school because of the program, which was a significant change because some of them had dropped out for as long as two years. In contrast, 97 percent of children from the non-CTP households indicated that they had dropped out of school due to lack of money for education.” See: ACDI/VOCA (2016). Gender impact assessment of cash transfers support to vulnerable households in Sierra Leone. Available upon request. EFSP Final Project Performance Evaluation Page 28 Figure 7: Percentage of cash spent on selected types of expenditures last month (Phase 1) The evaluation found that while the emphasis in the first phase was on the prioritization of the cash for recovery of food security, beneficiaries prioritized spending a significant portion on education. The qualitative data confirms that education was a high priority for Phase 1 beneficiaries as nearly all FGD respondents reportedly spent some of the cash transfer on their children’s school fees. In some cases, paying for education was prioritized alongside buying food with the first cash transfer (this was the case for both male and female beneficiaries). The qualitative research suggests that the cash transfers directly enabled beneficiaries to pay school expenses by providing funds that they otherwise would not have had. Many viewed the investment in their child’s education as safeguard for their own future livelihood – as a way to guarantee a better future for themselves, and their children. Many respondents indicated while the transfer made it possible for them to pay school fees during the intervention, the returns on their investments and VSLA savings have continued to support their children’s education cost over the last year since the program ended. According to some, non￾recipients in their community have not yet been able to recovery sufficiently to support their children’s education. “This money helped me to solve the problems of school fees as the money came after the Ebola outbreak and there was no source of money for me and the household to pay my children's schooling.” (Female beneficiary, Kailahun District, village, Phase 1) “I sat down with my wife we decided to pay school fees and school materials for our children and the balance money we bought seed rice for farming and we also used some for food.” (Male beneficiary, Bombali district, village, Phase 1) “I was able to send my children back to school because I received the cash transfer. While my friends who did not get the cash transfer still cannot pay for some of their children to school because they cannot pay their fees still today since the Ebola outbreak.” (Female beneficiary, Kailahun District, village, Phase 1) “For me I used to struggle to pay my children school fees. But now due to the money I received I started a business. Today, I can easily pay their fees and still have the remaining which I contribute to the box and improve on my business.” (Male beneficiary, Kailahun District, village, Phase 1) 35.00% 24.00% 21.00% 20.00% 35.0% 28.0% 21.0% 16.0% 33.6% 30.4% 20.7% 15.3% Food School Expenses Health Care Other Rount 2 Exit Interview (Nov 2015) Round 3 Exit Interview (Feb 2016) Supplemental Endline (Oct 2017) EFSP Final Project Performance Evaluation Page 29 In order to assess expenditure on ‘other’ items, supplemental endline respondents were asked to rank the top three ‘other’ expenditures from among a list (i.e. indicating with 1, 2, and 3 for the first, second, and third high-ranking items on the list). As highlighted in Figure 8 below, in terms of ‘other’ expenses, beneficiaries reported spending the most (i.e. ranked as number one) on income generating activities (28.4%) followed by home repairs (26.7%) and farm labor (15.8%). As only 15.3% of household expenditure was reportedly spent on ‘other’ expenses (down from 20% after the second cash transfer), it can be hypothesized that one year after the last cash transfer, Phase 1 beneficiaries have not sufficiently diversified their spending to include a large percentage of household cash on business, home repair and farm expenses. Rather, food and school expenses are prioritized and account for nearly two-thirds (63.6%) of all spending. Figure 8: 'Other' most common household expenditures in the last month (Phase 1) Phase 2 As part of the second and last (endline) payment exit interview, Phase 2 beneficiaries were asked about how they spent their last cash transfer using the bean counting exercise described above. Results are reported in Figure 9. The findings show a stark difference between how Phase 1 and Phase 2 beneficiaries spent their cash during the cash transfer period. For instance, while food and education were prioritized by Phase 1 beneficiaries, Phase 2 beneficiaries, on average, spent a higher percentage of their cash transfer on ‘other’ expenses, followed by food. This is largely in line with the objectives of the Phase 2 intervention which sought to address the recovery (rather than emergency) needs of EVD-impacted households. IGA 28.4% Home repairs 26.7% Agricultural inputs 16.5% Farm Labor 15.8% Animals 4.9% Savings 3.5% Cultural 2.8% Debt repayment 1.4% EFSP Final Project Performance Evaluation Page 30 Figure 9: Percentage of last cash transfer spent on selected types of expenditures (Phase 2) The lower proportion of cash transfers being spent on food in Phase 2 may be due to the timing of the intervention. As many of the Ebola-era restrictions on farming and market activities were lifted by April 2017, Phase 2 beneficiaries’ access to food was likely less restricted than the food security situation that Phase 1 beneficiary faced (as evidenced by the substantially higher baseline HDDS reported by Phase 2 beneficiaries compared to Phase 1, 4.3 versus 1.9 respectively). Nonetheless, Phase 2 beneficiaries were found to more greatly diversify their spending to include other expenses. As highlighted in Figure 10, these expenses were mostly related to income generating activities (58.1%) and agricultural inputs (25.0%). Figure 10: 'Other' most common household expenditures in the last month (Phase 2) The priority placed on spending a large proportion of the cash transfer on business was confirmed in the qualitative research. FGD respondents reported that they largely sought to increase the impact of the transfer by diversifying how they spent the cash and investing in business and income generating activities. There was a keen understanding of the potential of the cash transfer to support the household’s recovery through investments that would contribute to their longer-term needs and resilience. This approach to spending the cash transfer is directly in line with the main recovery objectives of Phase 2, which were promoted to all beneficiaries through the economic and financial management training messages delivered throughout the intervention (See Section 5.2.3 for a greater discussion on the training objectives and outcomes). 38.1% 30.3% 19.0% 12.6% 38.6% 30.8% 18.0% 12.6% Other Food School Expenses Health Care Round 2 Exit Interview (July 2017) Endline (Oct 2017) IGA 58.1% Agricultural inputs 25.0% Home repairs 5.7% Livestock 4.8% Farm Labour 3.9% Savings 2.0% Debt repayment 0.3% Cultural 0.2% EFSP Final Project Performance Evaluation Page 31 “The first money came in when there was nothing at home. We didn’t even have anything to eat at home. When I received that money, we used it to buy seeds to do our farming, and also we paid people to work for us at the farm. The balance money was used to buy food for my family. The second payment we received, we used the money to buy animals for rearing. I also used the money to pay for my younger brother school fees. The balance money was used to start business in the community. The final payment, I used the money to add to my business and also to feed my home.” (Female beneficiary, Bombali District, village, Phase 2) “They trained us on how to manage the money. Before I received the money, I was thinking about how to repair my house and pay school fees for my child because those were the pressing needs for my family. After I received the money, I changed that plan. I decided to invest some in business, buy some seeds for my farming and I even bought some animals for rearing in the community. Now I use the profit to take care of my family's needs. The money would have all been spent without these activities.” (Female beneficiary, Tonkolili District, village, Phase 2) Coping Strategies Coping strategies refer to the specific efforts that households employ in response to stressful events. When faced with food insecurity, households generally adopt numerous coping strategies that they would not normally undertake when they cannot access enough food. In this sense, ‘coping’ is what households do when they do not have enough food—the more households have to cope, the less food secure they are. When these strategies have negative consequences in the short or long-term on the individual or their families, they are known as negative coping strategies. These strategies include, but are not limited to: reducing the number of meals household members consumer each day, limiting portion sizes, begging and eating less expensive food. In both the Phase 1 supplemental endline and the Phase 2 endline survey, beneficiaries were asked to report whether their household had engaged in a list of negative coping strategies to access food in the last month. As this data was not collected at baseline for either group, it was agreed with USAID/FFP Implementing Partners to harmonize on indicators collected for potential broader analysis of the UCT impact by FFP. Prevalence of Negative Coping Strategies (Phase 1 and Phase 2) Beneficiaries from both phases were asked to report on the frequency of negative coping mechanisms practiced by their household in the last month. Responses ranged from: (1) never, (2) seldom (1 day a week), (3) once in a while (1-2 days a week), (4) fairly often (3-6 days a week), and (5) almost every day. Figure 11 shows the percentage of households in each phase which reported to have practiced each negative coping strategy at least once in the previous month (i.e. those who reported practicing the strategy seldom, once in a while, fairly often or almost every day). Results show that Phase 1 beneficiaries were far more likely to report engaging in negative coping strategies one year after receiving their last cash transfer when compared to Phase 2 beneficiaries. More than one year after receiving their last cash transfer, nearly two-thirds of these households were more likely to limit portion sizes (68.6%), reduce the number of meals eaten per day (62.6%) and rely on less expensive or preferred foods (60.2%) at least once in the previous month to access food. Therefore, while Phase 1 beneficiaries have largely sustained the gains made to their food security since the end of the cash transfer in terms of their HHS and HDDS, they are more likely to engage in negative coping strategies when faced with shortfalls in their food requirements. In the EFSP Final Project Performance Evaluation Page 32 absence of baseline data, it is difficult to ascertain if reliance on these negative coping strategies have decreased or increased since the intervention. On the other hand, Phase 2 beneficiaries reported very low reliance on negative coping strategies as the overwhelming majority (upwards of 70%) reported that they never practiced any of the listed strategies to access food in the last month. This is most likely due to these households active engagement with the CTP and their receipt of cash transfers which helped to improve their access to food and minimize their reliance on coping mechanisms. Figure 11: Prevalence of negative coping strategies used at least once in the last month (Phase 1 and 2) Additional coping strategies related to the diversion of funds from productive assets, education and health care to meet household food needs were also assessed (see Figure 12). In regards to the last three months, Phase 1 beneficiaries were more likely to report spending less money on agriculture/livestock inputs (36.7%) and health care (26.4%) in order to buy food compared to Phase 2 beneficiaries (14.1% and 16.9% respectively). Interestingly, Phase 1 households were least likely to divert funds from education in order to address food insecurity. This supports the earlier findings regarding the high priority these households place on education. 68.6% 62.6% 60.2% 39.8% 39.6% 39.3% 38.4% 38.1% 37.6% 21.8% 14.4% 7.4% 27.3% 29.0% 29.3% 23.0% 9.0% 12.3% 21.3% 6.1% 12.2% 3.3% 4.2% 3.4% Phase 1 Phase 2 EFSP Final Project Performance Evaluation Page 33 Figure 12: Negative coping strategies adopted in the last three months (Phase 1 and 2) Household Coping Strategy Index (Phase 1 and 2) The 12 individual coping strategies described in Figure 11 were combined into a household coping strategy index. The index was developed by FFP implementing partner World Vision. Each of the strategies was weighted according to the perceived severity of that behavior and then summed for each surveyed household. The coping strategy index as described is thus a measure of food insecurity— the higher the score, the greater the food insecurity (i.e. a higher score indicates a greater level of coping, and hence increased food insecurity). The maximum score possible on the index is 174, indicating all 12 coping strategies were used ‘almost every day’ over the last month. Conversely, the lowest score possible is 34.6, indicating that all 12 strategies were ‘never’ used over the last month. The average score for surveyed Phase 1 beneficiaries was 51.4 versus 43.2 for Phase 2 beneficiaries. This finding indicates that Phase 1 households are more likely to be food insecure and require a greater level of coping. However, in the absence of baseline data, it is difficult to ascertain whether the level of coping has improved as a result of the CTP. Nonetheless, an average score of 51.4 out of a possible 174 suggests that the level of negative coping (i.e. food insecurity) is quite low among Phase 1 beneficiaries. Table 15: Household coping strategy index (Phase 1 and 2) Average Score Minimum Score Maximum Score Phase Phase 1 51.4 34.6 110.8 Phase 2 43.2 34.6 95.2 From the perspective of beneficiaries, what are some negative things that households do when food is scarce? Do they believe that the program was effective in reducing these negative behaviors? The qualitative research largely confirms the findings of the quantitative survey. According to FGD participants in both phases, households are most likely to (1) limit portion size, (2) reduce the number of meals consumed per day and (3) rely on less preferred/expensive food when faced with food shortages. Phase 1 beneficiaries also reported taking loans to purchase food as a common coping mechanism in their community. 36.7% 13.7% 26.4% 14.1% 14.5% 16.9% Spend less money on agriculture/livestock inputs in order to buy food Spend less money on education in order to buy food Avoid spending money on health care in order to buy food Phase 1 Phase 2 EFSP Final Project Performance Evaluation Page 34 Overall, while the quantitative findings suggest that Phase 1 beneficiaries were more likely to rely on negative coping strategies, the qualitative research suggests that households have reduced their use of negative coping strategies since receiving the cash transfer. Respondents reported that a year after their last cash transfer they are less like to rely on the coping mechanisms they used prior to the intervention. Respondents cited increased food stores as supporting them through periods of food shortages, although they still have to rely on negative coping mechanisms at times. “Before I used to take food on credit from friends to feed my family. We would always eat small amounts of rice every day because food was short. But this time around it has happened much less. I don’t take as much loan anymore. My participating in SNAP [+] has improved greatly my life in terms of food security. I can now boast of having a larger quantity of food in my store than before SNAP+ came”. (Male beneficiary, Bombali District, village, Phase 1) “The negative things we do when there is food scarcity at home is that we reduce the amount we cook, and cut down the number of times we eat. The SNAP [+] program has helped us to reduce these behaviors. By participating in the SNAP [+] program it helped me to know how to manage our food better and how to prepare it well. This has helped us to store more food better so we now have something for when we have food shortage. Like, now we normally eat three times per day, but when there is less food, we can at least make it two times per day.” (Female beneficiary, Tonkolili District, village, Phase 1) Phase 2 beneficiaries reported that the CTP has been overwhelmingly effective in ending their reliance on negative coping strategies. Respondents reported that they no longer practice negative coping strategies since their participation in the EFSP. As this data was captured weeks after these beneficiaries received their last cash transfer, it is likely that the positive response is due to the immediate impact of the increased cash flow in the household, increasing their access to food. “The cash transfer has brought a big change in our life. We used to cook one cup of rice, but we are now cooking four cup of rice per day and my child can now eat enough. Before I use to cook plenty of empty sauce and a small amount of rice. It used to cause our children to get sick. This has changed because of my participation with the cash transfer program. We cook more rice and we are able to get small fish and other things to put in the sauce.” (Lactating mother, Bombali District, village, Phase 2). “For my household when food is scarce, we used to go to the bush to find yam to eat with my family. We also reduced the amount of food we eat per day. Like, we the parents will only eat once a day, so the kids can eat more. Since we received this money, we thank the America people we are not doing these negative things again in our household. Now I can cook 3 cups of rice in the morning and 6 cups in the evening.” (Female beneficiary, Bombali District, village, Phase 2) Based on a review of the quantitative and qualitative data the findings are mixed on the lasting effect of the CTP on negative coping strategies. In the absence of baseline data, it can be hypothesized that reliance on negative coping strategies has drastically reduced due to the intervention in the immediate term. A year later, while beneficiaries do still have to revert to relying on these negative strategies, they do so rarely (consider for instance the low average score on the coping index of 51.4). EFSP Final Project Performance Evaluation Page 35 Meal Frequency Self-assessment of the number of meals eaten per day is another indicator to assess the food security situation of beneficiary households. Both the Phase 1 supplemental endline and the Phase 2 endline survey asked respondents to report on the number of meals adult and child members of the household had eaten the day before the survey (focusing on the youngest person in each age group). The descriptive results are summarized below. Meal Frequency for Adults and Children (Phase 1 and 2) Beneficiaries from both Phase 1 and Phase 2 reported similar average meal frequencies for each age group. While the average number of meals consumed among Phase 2 beneficiaries was slightly higher, it is difficult to draw conclusions from these results in the absence of baseline figures. Qualitative research suggests that households were able to increase their meal frequency as a result of their participation in the program. Table 16: Meal frequency, by age group (Phase 1 and 2) Phase 1 Phase 2 Age group Youngest child 6 months to <5 years 2.4 2.5 Youngest child 5 years to 17 years 2.1 2.3 Youngest person 18+ years 2.0 2.2 “The cash from SNAP [+] helped me to change my level of living. In those days we would only eat two times per day. But today we eat three times per day. Thanks to the SNAP[+] program. We are so grateful.” (Male beneficiary, Tonkolili District, village, Phase 1) What are some of the activities beneficiaries have undertaken to keep or grow the cash transfer money they received? Do they believe these activities will continue when they are no longer receiving assistance? The majority of beneficiaries from Phase 1 (68.3%) and Phase 2 (88.1%) reported that they invested some portion of the cash transfer in an income generating activity. However, in line with the findings on cash transfer expenditures, Phase 2 beneficiaries were more likely to report using the cash transfer for an IGA. EFSP Final Project Performance Evaluation Page 36 Figure 13: Did you use any of the money from the cash transfer to start or continue an income generating business? Yes 68.3% No 31.7% Phase 1 Yes 88.1% No 11.9% Phase 2 The frequency of investing the cash transfer in income generating activities was confirmed in FGD discussion with beneficiaries. The majority of participants reported using the cash to restart an existing business that had been negatively impacted by the EVD crisis, or to start an entirely new income generating activity. Respondents voiced a clear understanding of the potential of these investments to grow the cash transfer and to provide longer-term impact for their family’s recovery and well-being. “I have invested my money in the VSLA box. I'm doing business so that my money can generate more income. I have bought some animals for rearing. When they give birth our money will be increased.” (Female beneficiary, Bombali District, village, Phase 2) “I am engaged in petty trading and I began doing the business when I received the SNAP [+] cash transfer. The first money they gave me was used to buy food for the household as it was paid when the Ebola outbreak had just ended and there was no food at home. But the second money I collected was the one I started doing my business with. Since then my business has flourished. From the profit I make, I have been able to provide food for the household.” (Female beneficiary, Kailahun District, village, Phase 1) The qualitative data provides further insight into why some beneficiaries did not invest a portion of their cash transfer in business. The data suggests that beneficiaries with larger households and those dealing with health issues where less able to spend on business as they had higher food, education and health care costs. In these instances, the transfer was perceived to be worth considerably less as a proportion of the households needs. “We have a large family so it was difficult to take any money for business. When the first money came we were in a bad place from Ebola and there was nothing in the house. We had to first take care of feeding the household and paying school fees. After that, every time the money came, it would just go fast to buy food, medicine, school fees for the household. For this reason I don't have any business activity from the SNAP+ money.” (Female beneficiary, Kailahun District, village, Phase 1) According to endline survey data, beneficiaries in both phases were most likely to invest their money in petty trading, followed by raising livestock and buying and selling palm oil or fish (see Figure 14). EFSP Final Project Performance Evaluation Page 37 The qualitative data further confirmed that these were the most common activities undertaken by beneficiaries in both phases to grow the cash transfer money they received. Many beneficiaries reported that raising livestock and selling palm oil had the greatest potential for longer term wealth. Figure 14: Types of IGAs beneficiaries invested in with the cash transfer money (Phase 1 and 2) In order to assess the sustainability of the IGAs, FGD participants were asked: “Do you believe your income generating activities will continue when you are no longer receiving assistance?” Phase 1 beneficiaries were able to draw on their own experiences over the last year and overwhelmingly reported that they have been able to continue their businesses in the absence of receiving the cash transfer. There were numerous success stories shared by beneficiaries who were able to boast of profitable businesses one year after the intervention. “Yes, I am doing business and I started it when I received my second pay of the cash transfer. I am buying and selling palm oil. I started with five gallons of palm oil and selling at home. I can now boast of a palm oil business of six drums. I am now the head of the ‘palm oil business women’ in our community.” (Female beneficiary, Kailahun District, village, Phase 1) “I was a petty trader. When the second money was paid, I got advice from my friend who did not benefit [from the SNAP+] to set up a palm oil business as she was already in it. So I decided to join her in the business. As God would have it, my business has multiplied ten times from where I started. I hope and pray that the business will continue for years to come. (Female beneficiary, Kailahun District, village, Phase 1) “When I got the money I bought a bag of rice which I cook to sell. Now a year later this business has grown to a level that I now have workers to do the cooking for my food business.” (Female beneficiary, Kailahun District, village, Phase 1) Given that Phase 2 beneficiaries just stopped receiving the cash transfer, they were less certain, yet hopeful that their businesses’ will continue in the absence of the cash transfer program. There was a sense of pride in their business activities to date, and optimism that their IGAs will continue to provide future positive returns. Many suggested that their diverse income generating activities act to safeguard future income flows. However, while not widespread, some beneficiaries voiced skepticism over the potential to continue gaining from their investments without further support in the context of the ongoing needs of their household. “Yes, if we are no longer receiving assistance our business will continue because we have invested the money into petty trading business and we are receiving income from the sales. From this money, 68.1% 28.4% 21.4% 19.6% 19.3% 2.5% 70.6% 25.5% 19.2% 9.2% 4.6% 1.2% Petty trading Raise livestock Buy and sell palm oil Buy and sell fish Sell cooked food Other Phase 1 Phase 2 EFSP Final Project Performance Evaluation Page 38 we usually give out loan to people in the community. They then will give us interest because of the loan given to them. This also helps the money to grow and develop. We have also invested in animal rearing. When they give birth we will be able to get more income from them if we decide to sell them.” (Female beneficiary, Bombali District, , Phase 2) “For me the activities I invested in are oil palm and cashew nut. These products should produce high yield, in time to come. But we still need support to grow these businesses while we are trying to also provide for our families.” (Male beneficiary, Tonkolili District, village, Phase 2) Are beneficiaries purchasing goods and services at prices similar to before cash transfer? ACDI/VOCA reported that periodic market checks and assessments indicated that cash transfers did not cause significant price distortions for food commodities. This was largely confirmed by FGD participants from both phases. The qualitative data indicates that the majority of beneficiaries were purchasing goods and services at prices similar to before the cash transfer. In cases where there was a reported difference in price, beneficiaries reported that the price increases were due to increases in transportation costs, rather than as a result of the cash transfer. “I am purchasing goods and services the same as it was before the cash transfer. I still buy a cup of rice at Le. 1,000 and palm oil at Le. 2,000 for a pint as it was before the cash transfer.” (Female beneficiary, Kailahun District, village, Phase 1) “Now the prices have increased, but it's not because of the money that has been given to us. The business people says it’s because of an increase in transportation cost. That is why they have to increase the price of their goods.” (Male beneficiary, Tonkolili District, village, Phase 2) Nonetheless, there were isolated reports of ‘artificial’ increases to the price of goods and services. Beneficiaries reported that these price hikes were often initiated by non-recipients looking to benefit from the cash transfer money. This was mostly reported to result in an increase for local farm labor, transportation from the pay point or in the cost of goods. Some beneficiaries pointed out that these increases were short-lived and generally imposed around the time of the cash transfer, only to return to normal shortly thereafter. “The price did not stay the same for food stuff at the time we were receiving the money. At any payment, the price of foodstuff would increase. The reason being most of the petty traders are not part of the SNAP [+] program, so they did this so that they can benefit someway from the SNAP [+] program. Every time we got SNAP [+] payment, the price of foodstuff would increase and then a few days later, it would go back to the normal price.” (Female beneficiary, Tonkolili District, village, Phase 1) “Since the start of the first payment business people increased their price at first. After a few days, they price would return back to normal as it was before. Some of the traders would say that we have money now so they will increase the cost of the foodstuff.” (Pregnant/lactating beneficiary, Tonkolili District, village, Phase 2) However, one community targeted in Phase 1 reported that price increases introduced during the CTP have remained since the end of the intervention. Respondents indicated that the cost of farm labor has increased since receiving the cash transfers (on average by USD$0.66/day). It was believed EFSP Final Project Performance Evaluation Page 39 that the price increases were introduced because, as one beneficiary reported, people in the community “think we are now rich people”. It is not clear from the qualitative data whether the increase in farm labor extended to non-cash transfer households or was limited to cash transfer households that may have been targeted for the increases knowing they had received ‘free’ money. However, it may be argued that any increase in farm labor costs would have promoted a positive knock-off effect on the wider community as day laborers would have benefited from the cash transfer as well – potentially compensating for any negative impact on beneficiary households due to higher labor costs. “The price for even farm labor has increased since we got the first cash transfer. Before the cash transfer, we used to pay Le. 10,000 for farm labor but after the cash transfer we now pay Le. 15,000 for brushing a piece of land per person. Even now because they think we are now rich people and money has been given to us free.” (Female beneficiary, Kailahun District, village, Phase 1) It is important to reiterate, however, that self-reports of permanent price increases since the cash transfer intervention were rare. The majority of FGD respondents indicated that there was either no difference in the price of goods and services; or that price increases were short-lived and often returned to normal days after each cash disbursement. ACDI/VOCA program monitoring data largely confirmed the qualitative evaluation findings on price distortions. In Phase 1 six local and imported items (rice, fish, salt, petrol, charcoal, and labor) were regularly surveyed at weekly markets in cash and non-cash transfer areas. On average, prices in cash transfer areas were 2-6% higher than non-cash transfer areas with exception of local labor which varied as much as 77% between cash and non-cash transfer areas. In Phase 2, cash transfer beneficiaries were asked at exit interviews and during post distribution monitoring whether they paid more for items than before the cash last transfer. 84% of respondents indicated they had paid more. However, beneficiaries in Phase 1 attributed higher costs to inflation, higher costs of fuel and transportation and impact of Ebola. Whereas, Phase 2 beneficiaries were more likely to attribute the higher cost to the cash transfer; reporting a temporary price increase immediately after the cash transfer with prices returning to normal within a few days post-cash transfer. If there was another shock today, has the program prepared beneficiaries to cope with the shock? How long do they think they cope without assistance, 1 month, 3 months, 6 months, longer? Focus group discussions with beneficiaries sought to better understand their perceptions regarding the impact of the CTP on their resilience and ability to cope with future shocks in the absence of assistance. While respondents from both phases overwhelmingly reported that the program prepared them to cope with future shocks more effectively, there were differences in their perception of how long they could cope without assistance. Overall, beneficiaries from both phases reported that through various impacts of the CTP, they felt better equipped to manage any future shock. Most referred to their investment in IGAs, membership in VSLAs, increased food stocks and knowledge gained through the financial training as key outcomes from the CTP which have strengthened their future resilience. EFSP Final Project Performance Evaluation Page 40 “The advantages I had in participating in SNAP [+] over the ones that didn’t take part is that I learned how to be able to manage the situation when there is any emergency. I learnt how to save for the future which those that didn’t take part in the SNAP [+] don’t know. I started my palm oil business with the money given to me by SNAP [+]. Without this money, I would have no business, and I would not be well prepared for any future emergencies like Ebola again.” (Female beneficiary, Bombali District, village, Phase 1) “I will not suffer like we did during the previous Ebola outbreak because I have now learned the culture of saving food and money which will take me and my family for a long time without suffering without food. I will not suffer because I have engaged in different business like petty trading and even have a backyard garden now.” (Female beneficiary, Bombali District, village, Phase 1) “For me this money has well prepared us for the future. We are also trained on how to manage the money and what to do to have more reserve. We have cultivated large farms, and are doing business with our own money now and are contributing to the VSLA box as reserve. This has helped us to be better prepared for any future emergency.” (Female beneficiary, Bombali District, village, Phase 2) Overall, Phase 1 beneficiaries were far more optimistic about their ability to cope without assistance for longer periods of time. Having spent the last year without receiving any cash transfers, these respondents were able to refer to their own experience as proof of their improved resilience. On average, these beneficiaries reported that they would be able to cope with any future shock for an average of four to five months. “My participating in SNAP[+] has helped me greatly to prepare for any other shock in terms of food security. Me and my household are well prepared for 4 to 5 months because now I have food in store for my children. During the cash transfer I used part of the money to invest in business so this is why I am better prepared now than I was in terms of the Ebola crisis.” (Female beneficiary, Tonkolili District, village, Phase 1) “If there is any kind of such emergency I am now more prepared. Having been a little self-sufficient in income and food from my farm land for the last year, I will be in position to feed my household for at least four months.” (Male beneficiary, Kailahun District, village, Phase 1) On the other hand, while the majority of Phase 2 respondents reported that the CTP has prepared them to cope with future shocks; they were less optimistic in terms of the duration they would be able to cope without assistance. In fact, the majority of FGD respondents estimated that they would not be able to manage beyond two to three months without assistance. Again, given that Phase 2 beneficiaries just received their last cash transfer, they have not had an opportunity to experience the sustainability of the CTP impacts compared to Phase 1 beneficiaries who have had over a year to put their resilience into practice and cope in the absence of assistance. Overall, while both groups reported that the program prepared them to cope with future shocks more effectively, the degree of their perceived preparedness differed slightly (i.e. four to five months for Phase 1 beneficiaries; two to three months for Phase 2 beneficiaries); most likely based on their actual versus anticipated ability to cope without assistance. EFSP Final Project Performance Evaluation Page 41 Nonetheless, Phase 2 respondents did feel more prepared to deal with an emergency than they had prior to the CTP. Again, their investments, VSLA membership and financial training were all cited as key elements in their improved ability to deal with future shocks. “The program has really prepared us for any future shock. For me I bought goats and sheep. I also harvest a large farm, and right now I am doing some business. Because of these activities I believe I will be able to cope for no more than 3 months.” (Pregnant/lactating beneficiary, Tonkolili District, village, Phase 2) “For me I think I’ll be able to cope for 2 months because we have invested a lot in our farm. If we harvested everything that we have planted we will be able to cope without any assistance from anyone for at least 2 months. We have also saved some money in the VSLA box. I also bought some animals that I can convert to cash if the needs arise.” (Female beneficiary, Bombali District, , Phase 2) Interestingly, a number of beneficiaries in both phases raised the issue of household size resulting in their diminished ability to cope with any future shock. As each household received the same cash transfer amount, beneficiaries from larger households reported that they have higher costs to meet in terms of household immediate needs, resulting in a less sustainable impact on their longer-term ability to cope with shocks. “I will not withstand if there is a shock. There are many people in my households and we have spent all the savings on food and health care. So there is nothing to withstand any shock at this time.” (Female beneficiary, Kailahun District, village, Phase 1) “As for me if there is any other shock in our country again I will able to manage for only one month because I have so many children to take care of. I have up to six children to feed every day and pay school fees. This has used a lot of the cash transfer. Their father died during the Ebola crisis, so this has made it more difficult for me to grow the money that has been given.” (Pregnant/lactating beneficiary, Bombali District, village, Phase 2) However, it is important to point out that household size was considered during program design and beneficiary registration. As the average household size of registered beneficiaries was 4.7, the cash transfer amount of USD$30/month based on a basket of goods for a family of six was well aligned with the needs of beneficiaries. Further, it is very likely that reports related to the higher costs of ‘larger households’ are linked to the cultural impact of extended family members and the additional responsibilities that arise when beneficiary households received the cash. Lastly, given the short￾term nature of the emergency programming, providing variable cash transfer amounts on the basis of household size would have posed a series of challenges in terms of overall management and verification of beneficiaries. 4.2. Program Systems, Processes and Procedures This section concerns the extent to which the systems, processes and procedures used by the program for delivering cash transfers and non-cash assistance to participants was effective, efficient and sustainable during the period of program implementation. Information provided has been gathered through FGDs with beneficiaries, local leaders and community members involved in pre- EFSP Final Project Performance Evaluation Page 42 identifying beneficiaries, as well as through interviews with project staff and a review of program documents. What mechanisms were used to ensure that the right beneficiaries were targeted in all phases? Was there a change in approach or modalities due to emerging situations? Geographic Targeting ACDI/VOCA had been implementing the Sustainable Nutrition and Agriculture Promotion (SNAP) program in Sierra Leone from June 2010 to December 2016 in Kailahun, Bombali, Tonkolili, and Koinadugu districts. The SNAP program was aimed at addressing the causes of malnutrition and building resiliency at the household level. The SNAP program was implemented by a consortium of development partners, with ACDI/VOCA as prime responsible for agriculture, livelihood, and access to credit. With the onset of the EVD outbreak, ACDI/VOCA applied for funding under the Food For Peace Emergency Food Security (EFSP) Annual Program Statement (APS) to address Ebola recovery in highly impacted areas of Sierra Leone. Implemented from June 2015 to December 2016 and rebranded as SNAP+, the intervention areas in Bombali, Tonkolili and Kailahun were prioritized as highly impacted by EVD in terms of case and fatality rates as well as high projected food insecurity. Koinadugu less impacted, was not included in the application for funding. Activities to increase agricultural productivity were prioritized to SNAP commercial farming groups to restore productivity likely to be sold in local markets and SNAP VSLA groups were targeted to receive recovery assistance. With the EFSP funding, efforts were made to ensure the appropriateness of geographic targeting for unconditional cash transfers in the in pre-existing SNAP operational areas. Using data from poverty maps provided by Statistics Sierra Leone (and used by the World Bank Social Safety Net program) overlaid with indicators of Ebola affectedness and projected cases of malnutrition,24 ACDI/VOCA applied rigorous analysis to ensure that the right geographic areas were being targeted for the CTP. In this way, cash transfer target areas were based on independent assessments unrelated to participation in SNAP, and based on the Statistics Sierra Leone extreme poverty mapping. During Phase 1, SNAP operational areas in Kailahun, Bombali and Tonkolili were targeted for UCTs based on a careful review of this data and considerations related to the impact of EVD on agricultural production and timing within the agricultural calendar. For instance, Kailahun was most impacted by loss of agricultural production particularly rice production; Bombali, a highly market￾dependent district, was food-insecure prior to EVD; and Tonkolili had the highest rates of Global Acute Malnutrition and Stunting prior to onset of EVD. All three areas had been subject to frequent, wide-spread and sustained quarantining that impacted on agriculture and markets. Targeting these districts for unconditional cash transfers was thus appropriate and based on real needs. Following the implementation of Phase 1, assessments indicated that Phase 1 beneficiary households had overcome the shocks associated with EVD and were back to pre-Ebola levels of security or better. Furthermore, while Phase 1 activities provided highly valuable recovery support to existing SNAP communities, 36% of the chiefdom sections within ACDI/VOCA’s operational areas 24 Projected cases of malnutrition were based on World Food Programme (WFP) and UNICEF caseload from 2014 EFSP Final Project Performance Evaluation Page 43 had not received assistance. The extension in Phase 2 was planned to directly close this gap in areas with moderate to severe food insecurity in Bombali and Tonkolili that received no assistance under the program in Phase 1.25 In Phase 2, ACDI/VOCA conducted a comprehensive mapping of chiefdoms, sections and villages not previously covered under Phase 1 in chiefdoms with high levels of food insecurity (based on WFP food insecurity data). Verified data on areas and populations were collected as the previously sourced data from the Statistics Sierra Leone Extreme Poverty Mapping on villages and number of households proved to be too limited for the purposes of Phase 2 targeting.26 The data collection on geographic targeting took considerable time to complete and delayed the registration of Phase 2 households. However, the resulting findings confirmed that the EFSP could reach the anticipated additional 8,800 households within the existing program chiefdoms in Bombali and Tonkolili. Overall, Phase 2 targeted an additional 163 previously un-served communities across 13 sections. The geographic targeting implemented by ACDI/VOCA during both phases resulted in a highly customized approach to ensuring that the right areas were targeted for support. While the EVD outbreak created varying degrees of impact across the operational areas, the rigorous analysis employed by ACDI/VOCA supported the identification and targeting of the most impacted chiefdoms and sections across the operational areas. Furthermore, the changes to geographic targeting that took place at the outset of Phase 2 were relevant and responded to the emerging recovery situation in the operational areas. There was a keen understanding among program implementers that continuing support to Phase 1 beneficiaries had the potential to lead to dependency given that the project had accomplished its emergency relief objectives in these areas. Furthermore, an analysis by ACDI/VOCA of the situation at the close of Phase 1 suggested that the un-served communities were at increased risk of becoming chronically poor and more vulnerable to future shocks in the absence of recovery support. By expanding the program coverage in Phase 2, ACDI/VOCA ensured a more equitable distribution of assistance to previously un-covered communities. Furthermore, the approach to program expansion was efficient. Rather than expand into new chiefdoms, Phase 2 provided more comprehensive coverage in the geographic areas originally targeted. Beneficiary Targeting While households were the target of the UCTs, women in charge of household nutrition were intentionally targeted to receive the cash transfers on behalf of their households (irrespective of the headship). This modality was well aligned with the GoSL’s national Social Safety Net (SSN) program where funds are given to women. ACDI/VOCA implemented this aspect of beneficiary targeting 25 According to ACDI/VOCA project documents, Kailahun District was not included in Phase 2 because: (1) the previous high coverage of Kailahun District under SNAP+ meant that nearly 80% of the chiefdoms and sections had been served; and (2) the logistical challenges and costs related to serving un-served households in a small population spread over a large geographic area was prohibitive. Therefore, in discussions with USAID when planning for Phase 2, Kailahun District was dropped in order to focus on un-served areas in Bombali and Tonkolili Districts demonstrating high levels of projected food insecurity. 26 According to ACDI/VOCA, the mapping was not comprehensive and limited to potential NaCSA areas for social safety nets consideration and did not cover the new geographic areas for Phase 2. Will the mapping data was aligned well for Phase I, it proved to be too limited for Phase II targeted ‘recovery’ communities. EFSP Final Project Performance Evaluation Page 44 exceptionally well as 85.0% and 93.7% of beneficiaries targeted in Phase 1 and Phase 2 respectively, were women. The overall targeting approach employed by the EFSP assumed that while the number of people directly impacted by EVD was relatively small in relation to the size of the population, the indirect impacts of EVD affected the larger population27. As a result, households directly and indirectly affected by Ebola were targeted to receive UCTs. Beneficiary targeting largely followed the guidance presented in the GoSL’s ‘Ebola Virus Disease (EVD) Response-Social Safety Net Cash Transfers Standard Operating Procedures’. 28 The Standard Operating Procedures (SOPs) provided clear guidance on the targeting procedures for cash transfers and served as the basis for ACDI/VOCA’s approach in Phase 1. As per the SOPs, poverty was a key determinant for beneficiary selection. During Phase 1, the beneficiary selection process relied on a community identification process using the National Committee for Social Action’s (NaCSA)29 Light Proxy Means Test (LPMT) to identify vulnerable households. This survey ranks household poverty through a series of question responses, and was administered through an open community forum that allowed on-the-spot information verification and household selection. Households pre-selected by locally formed community identification committees (CICs) were surveyed using the LPMT questionnaire and those with highest LPMTs were selected for the program. A sample of those households was later verified for response accuracy though a follow-up survey conducted by ACDI/VOCA. During Phase 1, a total of 13,584 beneficiary households were identified for program enrollment. In Phase 2, the existing beneficiary targeting approach was enhanced to more directly address the program objective of Ebola recovery (rather than an outright focus on poverty measures which were characteristic of the initial emergency phase). To this end, initial candidates for beneficiary household selection were nominated by communities based on a set of EVD impact criteria provided by ACDI/VOCA. These criteria are shown in Table 18. Similar to the Phase 1 community-based approach, community committees were provided with training and held accountable for the accurate pre-identification of potential beneficiaries under the criteria stipulated. Pre-identified households then submitted to a questionnaire for verification. Table 17: CIC Criteria for their Initial Nomination of Households as Potential EFSP Beneficiaries (Phase 2) 1) Household food security • Families that can only manage 1 meal per day 2) Spouse or other worked aged adult(s) died during the Ebola emergency • Families that lost the household head/main bread winner to Ebola 27 For instance, in response to the unprecedented EVD outbreak, the GoSL declared a State of Public Health Emergency across the country. The state of emergency banned public gatherings, imposed quarantines, border closures and restrictions on movement – ultimately hampering economic activities and leading to an increase in poverty, hardship and vulnerability for many families and communities in the larger society. 28 Government of Sierra Leone (May 2015). Ebola virus disease (EVD) Response-Social Safety Net Cash Transfers Standard Operating Procedures’. Available at: https://www.humanitarianresponse.info/system/files/documents/files/standard_operating_procedures_sop_08.05.15.pdf 29 NaCSA had been operating a national Social Safety Net (SSN) program prior to the Ebola outbreak. The LPMT was established under the SSN program for targeting vulnerable households. EFSP Final Project Performance Evaluation Page 45 4) Household suffered loss of income generating activity or sold income earning assets during the Ebola emergency • Families that were farming or had businesses that did not resume after Ebola • Families that lost economic asset due to Ebola During Phase 2, the NaCSA LPMT poverty identification survey was augmented with a set of questions developed by ACDI/VOCA to more clearly identify households that were EVD-impacted and demonstrating high levels of household food insecurity (HFS). As outlined in Table 19 below, rather than simply identifying households based on their poverty level, the modified survey better identified highly impacted households that were food insecure (while still collecting the standard poverty data as per the LPMT). Households were then selected based on high scores evidencing EVD impact and food insecurity. Registration in Phase 2 was conducted by NBI, an independent third party, using mobile devices and software to allow for rapid scoring and registration. During Phase 2, a total of 8,783 households were identified for program enrollment. Table 18: Revised Eligibility Criteria according to Household Food Security and EVD-impact (Phase 2) 1) Household size increased during the Ebola emergency • Household is caring for children who were orphaned during the Ebola emergency • Household is caring for children who were abandoned during the Ebola emergency • Household is caring for adults(s) who were widowed during the Ebola emergency • Household is caring for adult(s) who were abandoned during the Ebola emergency 2) Spouse or other worked aged adult(s) died or left during the Ebola emergency • Husband or wife of the household died during the Ebola emergency • Some other working aged household member died during the Ebola emergency • Working aged person(s) left during the Ebola emergency and has not returned 3) Children did not resume schooling after Ebola emergency was lifted • Children were attending school before Ebola but did not return to school after school resumed 4) Household suffered loss of income generating activity or sold income earning assets during the Ebola emergency • Household stopped an important income generation activity during the Ebola emergency and has not resumed the activity • Household farm size decreased during the Ebola emergency • Household that stopped farming during the Ebola emergency • Household member lost their employment during the Ebola emergency • Household sold income earning assets (motorcycle, sound system, etc.) during the Ebola emergency and has not replaced them • Household sold majority of their cattle or goats or their last female animal during the Ebola emergency and has not replaced them Based on a review of the revised criteria, the changes introduced to the selection survey and overall scoring of households in Phase 2 may have served to reduce inclusion and exclusion errors.30 The modified questionnaire scored a number of measures related to EVD impact and the household’s capacity to recover with the assistance offered in the timeframe provided. For instance, questions related to the number of household members, changes in household composition since Ebola, the 30 Inclusion error refers to the proportion of beneficiaries that did not deserve to be registered as beneficiaries. Exclusion error is the reverse of inclusion error. EFSP Final Project Performance Evaluation Page 46 composition of different age groups (i.e. productive working age vs. dependents), food insecurity measures and the impact of the EVD outbreak on livelihoods were all scored. This added rigor resulted in a strengthened targeting approach in Phase 2 that allowed for meaningful selections beyond standard poverty measures. As part of this revised beneficiary selection criteria, the households that were enrolled in Phase 2 had a clearly demonstrable potential for post-EVD economic recovery. Furthermore, the revised scoring questions were not always obvious measures for targeting (e.g. number of pregnant women in the household). These ‘blind’ targeting techniques were effective in reducing the level of response bias inherent in a standard poverty measure, where respondents are more aware of the types of responses to give to solicit support. The revised approach to beneficiary targeting in Phase 2 was relevant and appropriate given the shift in focus to recovery efforts in Phase 2. The evaluation found that the original LMPT approach relied too heavily on identifying households that were poor, rather than examining the impact of Ebola and food security measures. ACDI/VOCA staff reported that the changes in beneficiary targeting in Phase 2 adequately addressed the limits of the prior poverty indicators to provide more comprehensive screening of eligible households and reduced exclusion errors. According to one member of the ACDI/VOCA project team, “the changes ensured more meaningful evaluation of households and that high-need households were less likely to be overlooked”. While the NaCSA methodology proved effective in identifying beneficiaries during the emergency phase, the limited focus of the LPMT on extreme poverty did not provide sufficient information to segregate impacted households affected by high levels of food insecurity in the Phase 2 recovery program. As evidence, an analysis of the resulting LPMT and HFS data collected on selected and non￾selected households in Phase 2 demonstrates marginal differences between household as measured by level of poverty (i.e. LPMT score). On the other hand, measuring EVD impact and food insecurity through the newly introduced HFS score resulted in greater distinction between eligible and non￾eligible households. Table 19: Household selection score, LPMT vs. HFS (Phase 2) Selected Households Non-selected Households Selection Criteria LPMT 7.7 6.9 HFS 8.8 3.1 The evaluation found that overall there was little evidence of inclusion error. A review of the qualitative and quantitative registration data31 indicates that virtually all households selected for the CTP met the eligibility criteria. The robust registration methodology ensured that households were selected according to their fulfillment of requirements set under each Phase. The addition of electronic data capture in Phase 2 also helped to strengthen the registration process. While a sample of registered households were back-checked and verified in Phase 1, the delay in registration meant that there was not enough time to carry out a thorough verification process in Phase 2. However, community facilitators did conduct verification of a small sub-sample of 31 Only quantitative registration data for Phase 2 was reviewed as part of the evaluation. EFSP Final Project Performance Evaluation Page 47 beneficiaries through post-distribution monitoring after the first cash transfer. ACDI/VOCA and ACC staff also conducted regular on-the-spot verifications of registration data during the data collection and payment period and flagged a number of cases which were handled individually by ACDI/VOCA regional and Freetown based staff. Both processes helped to minimize the potential for inclusion error. However, as discussed in greater detail below, there was substantial perceived exclusion error related to the quotas set for the number of cash transfer beneficiaries per village. In Phase 1, village quotas were established by NaCSA through the Social Safety Net program on the basis of Statistics Sierra Leone poverty mapping. In Phase 2, quotas were established based on the World Food Program’s 2015 Comprehensive Food Security Vulnerability Analysis (CFSVA) data on the percentage of projected food insecurity at chiefdom level. Village level quotas were then computed on the basis of the number of households per village (numbers which were verified through a comprehensive mapping exercise at the outset of Phase 2). For instance, if a village had 30 households, and the projected food insecurity in the chiefdom was 80% according to the CFSVA, the village would have a quota of 24 beneficiaries. According to FGD participants, the quota system across both phases resulted in under-coverage of eligible beneficiaries. At community level, FGDs with beneficiaries and CICs from both phases revealed that participants broadly accepted the beneficiary targeted process as fair. The community-based system was viewed to be transparent and participatory insofar as the pre-identification of potential beneficiaries was indicated to be largely free from exclusion errors. Participants stated that those that were initially listed were in fact the most vulnerable in their communities and that selection was done free from external influence. The primary concern raised by all participants was related to the number of people registered in their community relative to the scope of need. According to the respondents, households with similar qualities to the registered beneficiaries were left out of the CTP due to the quota system. Although there was broad acceptance of the eligibility criteria and the targeting decisions, households that were pre-listed, but later excluded, were reportedly more likely to perceive the targeting process as unfair. In these instances, the CICs played an integral role in explaining the selection process and increasing acceptance of the quota system. “I think if there is another opportunity in our community please increase the number of beneficiaries. The quota was so small in this community and so many people who are in the exact same situation as me did not benefit.” (Female beneficiary, Tonkolili District, village, Phase 1) “We suggest that any other time the people come to distribute the money, let them register everyone. We don't mind if it is not enough money but as long as everybody receives. We are all living the same here in this community. More people deserved to receive the money, but they didn't.” (Female beneficiary, Tonkolili District, village Phase 2) “As the quota for the eligible beneficiaries was far below the number of people who needed the cash transfer, those who did not benefit initially considered us as enemies and bias. People of similar qualities as the registered beneficiaries but left out because of the quota system expressed dissatisfaction of not being selected.” (Local leader, Kailahun District, village, Phase 1) EFSP Final Project Performance Evaluation Page 48 The issues raised regarding the quota system were particularly pronounced in smaller villages where the quota meant that a handful of households were excluded from the CTP; in some cases this meant that only five households were excluded from the whole village. As noted above, these outcomes of the quota system were a cause for concern among communities and warrants further consideration in terms of targeting approaches for future CTPs. Interestingly, having two scores to review in Phase 2 meant that the exclusion error was substantially reduced. In cases where the number of households classified as eligible by the HFS was greater than the maximum quota allocated to the village, beneficiaries were then selected on the basis of their LPMT score. In similar cases during Phase 1, a lottery system was used to select beneficiaries from the pool for eligible individuals with equal LMPT scores; resulting in a higher potential for exclusion error. Whereas the lottery system was commonly applied in Phase 1, there were very few incidents where it had to be used in Phase 2. During implementation, what were the mechanisms used to ensure beneficiaries, the public, and management were fully informed about what was happening at field level? Evaluation finding suggest that the mechanisms used to ensure proper communication between the various project stakeholders and beneficiaries were well-executed, highly effective and changed little between project phases. Discussion with ACDI/VOCA staff, local leaders and beneficiaries revealed that clear communication channels were established from the outset of each phase, and sustained throughout the implementation. This served to ensure that beneficiaries, the public and management were well informed of the project objectives, processes and outcomes. ACDI/VOCA conducted sensitization with local authorities and governance structures including District Council Officers, Ward Councilors, Paramount Chiefs, Section Chiefs and Town Chiefs at the beginning of each phase. This early engagement supported the development of strong relationships with local authorities which proved integral for project coordination, sensitization and conflict resolution activities at community level. However, it was the interaction between ACDI/VOCA Community Facilitators and the CICs which proved to be the centrally organized means for communicating to and from field level. ACDI/VOCA Community Facilitators served at chiefdom/village level to maintain contact with local authorities and community members throughout program implementation. The facilitators visited the target communities regularly to answer question and resolve issues and were present at all cash payouts. In FGDs, CIC members discussed the two-way communication channel that was set up between themselves and the beneficiaries/public, and the ACDI/VOCA Community Facilitators. They explained in clear detail their responsibilities of communicating with beneficiaries on the basis of directives provided by ACDI/VOCA; in addition to providing a channel for updates from the field to be delivered to community facilitators and on to ACDI/VOCA management. “During registration and payment time, community members were informed by calling town barry meetings using mega phones, town crier and house-to-house visitation to explain to them how the selection process will be done and who is eligible to be registered as a beneficiary for the cash transfer. Facilitators were recruited and given the responsibility to work with the registration committee to report the day-to-day work to the community leaders. We also appointed a community member to be a liaison between the community and ACDI/VOCA with his primary EFSP Final Project Performance Evaluation Page 49 responsibility to report the day to day work to ACDI/VOCA.” (CIC member, Kailahun District, village, Phase 1) “In most cases, one of the SNAP [+] staff visit us regularly to pass on the information. I then pass this information to the community beneficiaries. I also give a monthly report to the SNAP [+] staff or I call SNAP [+] staff through mobile to communicate anything happening in the field.” (CIC member, Bombali District, village, Phase 2) Discussions with beneficiaries also revealed that CIC members served as a main point of contact for them throughout project implementation. Beneficiaries not only looked to them for information on key dates and processes for the cash transfers, they also sought them out to report any issues or complaints they had. CIC members were also regularly present at cash distribution points to support beneficiaries and ACDI/VOCA with coordination and to resolve any issues. Beneficiaries appreciated having someone familiar involved in the process. According to one recipient: “Having our own people in the process has made it easier for us to have direct contact and always be clear on what is happening.” (Female beneficiary, Tonkolili District, village, Phase 1). Community radio was also used as a tool to keep beneficiaries and the general public informed about the program objectives and activities. Several call-in shows were held at the beginning of the program and during or after cash disbursement days. Topics included the selection process, payment dates, the purpose of the cash transfers and the Anti-Corruption Commission’s (ACC) role in the program among others. The use of local radio was highly relevant in the rural targeted areas and served to enhance the program’s transparency and accountability. For instance, in Phase 2, a local radio call-in program provided the opportunity for ACDI/VOCA to speak directly with households that were not selected and to answer their questions. As noted by ACDI/VOCA, at the conclusion of the program, many callers thanked ACDI/VOCA for the transparency in the process. In Phase 2, beneficiaries attended a program orientation session before the cash distribution began. Among other things, they were informed about the amounts of money they would be collecting at each cash transfer and the month when the three cash transfers would occur. This communication at the outset of beneficiary enrollment served to enhance clarity on the program’s objectives, policies and timeline for the beneficiaries. Overall, evaluation data suggest that the mechanisms employed by ACDI/VOCA to ensure clear lines of communication were effective. Beneficiaries reported a clear understanding of the cash distribution amount, schedule and timeline. Messaging about what was happening in the field proved effective as cash transfer payouts had an average completion rate of 99% throughout both phases. Furthermore, the program ensured that all authorities and governance structures at chiefdoms and community level were involved and well informed from the outset. This community based approach and engagement with local leaders and CICs served to build trust and enhance the accountability of the CTP. What was the beneficiaries experience with the cash delivery systems, processes, and procedures? Payment processes and delivery systems are important components of any cash transfer program as weak delivery and payout mechanisms can drastically limit the program’s impact. The evaluation EFSP Final Project Performance Evaluation Page 50 sought to assess the effectiveness and efficiency of the systems, processes, and procedures used by the program for delivering cash transfers and non-cash assistance. Areas assessed included: (1) the delivery mechanisms; (2) accountability; and (3) the beneficiaries experience with the cash distribution process. The findings suggest that across numerous aspects of program implementation, the EFSP performed exceptionally well. Delivery Mechanisms Cash transfers were made in 3 month increments equivalent to USD$30 a month. Phase 1 included four payments over an 11-month period, while Phase 2 included three payments over a nine-month period. Quarterly cash transfers were seen to be more beneficial as it provided households with sufficient cash to move beyond short-term financial planning and the ability to buy food in bulk. As many of the target locations were remote, quarterly payments also reduced transaction costs associated with delivering cash to isolated rural locations. The EFSP relied on direct physical transfer of funds during both program phases. While the program had initially proposed to use electronic transfer through mobile networks, assessments and early trials revealed that the mobile money networks in Sierra Leone did not have the capacity or geographic coverage to meet recipient transfer needs. This was especially the case in the large proportion of remote and rural areas targeted by the program. Furthermore, there was a limited supply of service providers in Sierra Leone that could be called on to provide the level of service required to deliver the cash (in any form). In response, a local service supplier, SPLASH, was engaged to provide the physical transfer of funds using a system which was previously developed for the World Bank. SPLASH was able to conduct ‘off-line’ transfers using SIM Cards as unique identifiers of beneficiaries with photographic record keeping that was correlated with a beneficiary database. Beneficiaries would present their unique SIM Card at pay points, and once verified, SPLASH agents directly paid the beneficiaries in cash. While the use of direct physical transfers overcame the impracticality of e-transfers in the program areas, it was not without its challenges. By design of the program’s objectives, the majority of the targeted areas were remote rural locations. Although a ‘cluster’ approach was followed to designate centrally located pay points, poor road networks and limited accessibility to local financial institutions created serious challenges for the direct delivery mechanism. For instance, in Phase 1, SPLASH had to work with a total of 68 local financial institutions32 in order to provide their agents with sufficient liquidity to meet agreed payment schedules. This created additional issues for coordinating payouts and the movement of funds. In one instance, a local financial institution had cash flow issues and payment had to be delayed by one day as only small currency notes were available. ACDI/VOCA had to develop precise payout schedules and coordinate with SPLASH very closely to ensure payments were appropriately timed and delivered. This required considerable resources and time to support. Further, due to the remote locations and accompanying security concerns with carrying large sums of money, SPLASH agents had to be incentivized with commissions to travel to the difficult-to-reach areas. 32 There were 17 local financial institutions in Kailahun, 29 in Bombali and 22 in Tonkolili. EFSP Final Project Performance Evaluation Page 51 As described in greater detail below, while the direct delivery mechanism remained the same in Phase 2, changes introduced to the program’s information management system in Phase 2 allowed for additional payment vendors to support the cash transfers. To this end, ACDI/VOCA engaged Rokel Commercial Bank to deliver cash transfers in the more highly populated cash transfer areas, while SPLASH continued to focus on smaller, more remote pay points. Rokel Commercial Bank had established protocols and security procedures for delivering mobile banking services to their customers where they did not have a branch location. This made Rokel an ideal partner as they were fully insured and able to carry larger sums of cash - reducing risk and allowing for more flexibility in the planning and scheduling of payouts with SPLASH. According to ACDI/VOCA, this allowed for payments to be scheduled more efficiently as multiple vendors were available. Overall, the delivery mechanism employed by the program is evaluated to be applicable and relevant. ACDI/VOCA was careful to consider the realities on the ground and engaged local service providers to provide a delivery mechanism which was appropriate for the operating context in the project’s target areas. While e-transfers would have provided an ideal solution to efficiently deliver the cash at scale, but as this was not available in the operational areas, ACDI/VOCA effectively worked with the service providers to minimize any negative impacts on the delivery of the cash to beneficiaries. In fact, as discussed in greater detail below, there was never a reported incident where beneficiaries did not receive their cash despite the challenges associated with delivering direct cash in remote areas in Sierra Leone. This experience demonstrates that it is possible to do direct cash distribution in fragile post-emergency environments. It also highlights the importance of conducting feasibility assessments during project design of cash transfer programs. Accountability A number of mechanisms were employed by the EFSP to ensure accountability and transparency throughout the cash transfer process. At program level, a robust program Management Information System (MIS) served as a key instrument for accountability, regular program monitoring activities tracked implementation and outputs systematically, and measured the effectiveness of program, while donor-led monitoring trips provided oversight and monitoring. At the beneficiary level, the ACC established grievance redress system provided a formal appeals process, and the communication mechanisms described in 5.2.2 provided channels for ongoing feedback about program rollout. Program Management Information Systems From the outset of Phase 1 ACDI/VOCA established an information system for internal monitoring of UCT beneficiaries and payments. During this phase, the SIM Cards issued by SPLASH served as identification for all registered beneficiaries, as well as a means for record keeping as unique SIM numbers correlated with the beneficiary database. The card contained personal information and photos which were integral to beneficiary verification and tracking payment. ACDI/VOCA augmented the SIM cards with additional laminated photo ID cards which were more easily identifiable to the beneficiaries and SPLASH agents. As SPLASH was responsible for managing the data collection for all payments, their records were reconciled against data collected by ACDI/VOCA staff to validate payments at the cash points. Subsequently, data was verified through submission of beneficiary pictures and individual pay point records (including separate thumbprint files from SPLASH and ACDI/VOCA). Completed payment files EFSP Final Project Performance Evaluation Page 52 were uploaded to ACDI/VOCA’s EFSP MIS Dashboard which provided an overall accounting of unconditional cash transfers and the processing of payments made to program beneficiaries. However, throughout Phase 1 implementation, ACDI/VOCA noted a number of limitations in SPLASH’s ability to provide complete data, specifically regarding data capture and payment verification picture uploads. For instance, it was noted that payment monitoring data provided by SPLASH was missing between 1 – 10% of verification pictures. While ACDI/VOCA developed back-end systems for accountability during Phase 1 (e.g. reconciliation processes), the situation ultimately impacted on the accountability of the program as the system for tracking beneficiary payments through photographic identification was compromised. In Phase 2, ACDI/VOCA, recognizing the challenges in SPLASH’s data collection, amended the data collection system to include: (1) A QR-code system to enhance identification of beneficiaries and (2) an independent third party to monitor and provide data collection for all payments via mobile phones using SurveyCTO. The changes increased the efficiency at pay points as SIM cards were no longer used and the QR-codes integrated on the beneficiary ID cards allowed for quicker accessing of personal information and photos. Moreover, as SPLASH was no longer responsible for managing the data collection for all payments, the new system provided the flexibility to accept different payment vendors (i.e. Rokel Commercial Bank). Finally, the quality of data on the distribution, collection and processing of payments was greatly improved as there was virtually no missing data or missing payment verification pictures in the Phase 2 MIS. The MIS implemented by ACDI/VOCA was one of the main enablers of the accountability and control processes of the CTP. The MIS provided a systematic, secure system to link and reconcile information on beneficiary registration and payment. In effect, the MIS ensured that every beneficiary household met all the eligibility criteria and was receiving the right amount of cash at each distribution. This was integral to minimizing and managing risk, and to ensuring effective program implementation and sustainability. Lastly, ACDI/VOCA efficiently leveraged technology to strengthen the program’s information management system in a relevant and context-appropriate way by relying on easily deployable QR-code technology and improved software monitoring systems. Periodic Program Monitoring Program implementation and impact was regularly monitored through data collected at exit interviews, post distribution monitoring and routine monitoring to oversee community-led trainings. Exit interviews taken at the cash transfer points collected data on the household dietary diversity score and household hunger score to measure progress towards targets set for these indicators. Data was also collected on how decisions were made for expenditures, how money from the cash transfer was used and beneficiary feedback on budgeting and nutrition messages. Post distribution monitoring also measured how long the cash transfer money lasted and whether money was being prioritized to ensure sufficient food is available in the household. These regular and systematic monitoring activities served to enhance the program’s internal and accountability of the resources used and the results obtained. Donor Monitoring USAID/SL, USAID/FFP/W/SL USAID/Dakar, and the US Embassy conducted a number of monitoring and site visits throughout the life of the program. Between June 2015 and October 2017 a total of 12 field visits including direct observations of cash transfer disbursements, VSLA groups, training and EFSP Final Project Performance Evaluation Page 53 IGA groups. Data Quality Assessments and meetings with beneficiaries were also conducted. This high-level of monitoring by the donor served to increase the accountability and collaboration throughout the life of the CTP. Grievance Redress As outlined in the SOPs issued by the GoSL, ACDI/VOCA relied on the Grievance Redress Mechanism (GRM) established by the ACC for reporting instances of corruption during both phases of the project. Additionally, the project established various community-level feedback mechanisms which provided numerous opportunities for interaction and information flows between beneficiaries, implementing staff and GoSL stakeholders. Central to the ACC GRM was a national toll-free line which reports of corruption or fraud could be routed to and directly resolved by ACC. Information on the toll free line and its purpose was shared with all beneficiaries through sensitization activities. All beneficiary ID cards issued by the program included details on the toll free number. Moreover, representatives from NaCSA and ACC were present at all cash distribution points to reinforce message regarding fraud and corruption and to serve as a help/complaint desk.33 The role and presence of ACC was clearly recognized by beneficiaries in Phase 1. FGD participants frequently cited the messages they received from the ACC on the purpose and appropriate use of the money. Interestingly, Phase 2 beneficiaries were more likely to cite CICs as the source of messaging and resolution for issues relating to fraud and corruption. Additional feedback mechanisms were established at community level which could be used by beneficiaries to ask questions, provide feedback or voice complaints. These included ACDI/VOCA Community Facilitators who visited communities regularly to share information, gather feedback and complaints; as well as radio call-in shows and direct contact with program staff, CICs and community leaders throughout implementation. The Evaluation Team found that these community￾level accountability mechanisms were very effective in providing a robust beneficiary interface where beneficiaries were provided with context-appropriate opportunities for grievance redress and channels for ongoing feedback. Discussions with beneficiaries, local leaders and CIC suggest that the feedback mechanism most used by beneficiaries was related to direct contact with ACDI/VOCA program staff, CICs and community leaders. Cash transfer payment days provided the greatest opportunity for interaction between all actors, while CICs were viewed as being the most constant feedback channel. The use of interpersonal communication was viewed as being the most accessible grievance mechanism to beneficiaries. Anecdotal evidence suggests that any reported grievance was promptly reported to CIC, ACC or ACDI/VOCA representatives and dealt with in the field. On the other hand, no beneficiaries in either phase mentioned use of the toll free line; likewise ACDI/VOCA indicated that no calls were reported through this mechanism. High levels of illiteracy and limited phone ownership and coverage in the target areas likely rendered the toll-free line less accessible to beneficiaries. 33 During Phase 1, representatives from the Ministry of Social Welfare Gender and Children’s Affairs (MSWGCA) were also present during cash distribution days. EFSP Final Project Performance Evaluation Page 54 Beneficiaries Experience with Cash Distribution Process Focus group discussions with beneficiaries provided a great deal of insight on beneficiary perceptions of the cash distribution process. Beneficiaries reported highly satisfactory experiences with the payment system, waiting time, staff treatment and safety. There was, however, one significant cause for concern: distance to pay points. Walking Distance As previously mentioned, ACDI/VOCA implemented a ‘cluster’ approach to payouts in order to efficiently coordinate payments across targeted rural communities. This meant that villages within close proximity to each other would be assembled in a central village for cash transfers. To this end, the majority of pay points were established within a 5 km (3 mile) radius of cluster communities. According to data collected during Phase 1, 42% of beneficiaries reported walking less than one mile to the pay point, while a further 34% reported walking between one and three miles to the pay point. ACDI/VOCA also encouraged households to designate an ‘able bodied’ woman to collect the cash to ensure that the walking distance would not pose an issue to the selected representative.34 Nonetheless, the distance was deemed to be too far for some beneficiaries – especially those crossing difficult terrain/poor road conditions, the elderly, sick and disabled. In the most extreme cases, FGD participants reported walking upwards of four hours to reach the pay point (a challenge that was exacerbated in the rains). Many of these beneficiaries reported having to sleep in the pay point village prior to returning to their home the next day. “It was not easy considering the distance I covered to get to the paying point. I would have to walk the whole day if I want to return back home, so in most cases I spent the night at the paying point.” (Female beneficiary, Tonkolili District, village, Phase 1) While ‘far distance to pay points’ was the most common grievance with the cash distribution process, this was largely limited to beneficiaries living in communities which were over three miles to the pay point. In most cases, the distance was seen as being tolerable given the reward of the cash transfer. “The distance is far to reach where we must go to collect our money. We do not wait for long to receive the money. Both going to collect the money and returning home is safe; but the fact is that the road network is not good. It is just because of the money that we go and manage the distance. Had it been for any other business we will not go.” (Male beneficiary, Tonkolili District, village, Phase 2) Overall, given the limitations in regards to available delivery mechanisms and the need to do direct cash delivery, ACDI/VOCA did an exceptional job of balancing accessibility concerns with the need for reasonable transaction costs. As travelling to every beneficiary community to deliver the cash would be unrealistic, ACDI/VOCA went to great lengths and worked with local communities to designate carefully mapped out central pay points. 34 However, according to ACDI/VOCA, elderly heads of households in some cases still insisted to be registered and indicated they could manage the trip to the pay points. EFSP Final Project Performance Evaluation Page 55 Waiting Time In order to ensure that payments could be completed in a reasonable amount of time, ACDI/VOCA had to manage a complex network of pay points and actors. The number of beneficiaries to be served at each pay point varied greatly. For instance, in Phase 2, anywhere from 20-1,579 people were served across 29 different pay points. Once the field team was experienced, the average time for processing a beneficiary’s payment was two minutes per person using the enhanced QR-code system in Phase 2. Based on the feedback received from FGD participants, the systems and processes used during cash disbursement days were highly effective in minimizing wait times. According to reports from ACDI/VOCA field staff and pay point enumerators, each pay point took an average of four hours to complete. In fact, beneficiaries praised ACDI/VOCA for their efficiency and convenience. Overall, beneficiaries in both phases reported that the wait times were acceptable, even in the case where they had to wait over two hours for payment. Interestingly, beneficiaries were less inclined to complain about the wait times at pay points as they focused more on the fact that payment happened on the days they were scheduled. There was an overwhelming appreciation for the planning and commitment of ACDI/VOCA to show up when they said they would. “I was so convinced that the way and manner the distribution goes it was so transparent. When we arrive at the paying point the SNAP [+] staff checked the list and found out that we covered the furthest distance - so they start with us first. So we were the first community to be paid. Thanks to the SNAP [+] for their understanding.” (Male beneficiary, Tonkolili District, village, Phase 1) “The waiting time was ok because the specific date and time given to me for the payment was the exact time they will pay me.” (Female beneficiary, Bombali District, village, Phase 1) Safety Discussions with beneficiaries and community leaders revealed that security was not an issue during the cash distribution process. In fact, there were no reported incidents of theft or assault throughout program implementation. While FGD participants pointed out that the security and safety at pay points was enhanced by the presence of police, findings suggest that for the most part, communities largely assumed responsibility over the safety of payouts. There were numerous references made to the general peace maintained in their communities as safeguarding against the potential for conflicts or theft. Furthermore, beneficiaries reported travelling in groups to and from pay points for added protection (a practice encouraged through ACDI/VOCA sensitization messages). In many cases, female beneficiaries reported feeling protected as their husbands would accompany them to the pay point. “I never felt unsafe when receiving the cash transfer because our community is safe from criminals. (P1). I did not feel unsafe as I was paid in my community and nothing to be worried about as our community is safe even before and after the cash transfer. (P2) I did not feel unsafe because our community has never experienced people being attacked or breaking into their houses and having things or money stolen from them (P3).” (Male and female beneficiaries, Kailahun District, village, Phase 1) “We feel safe during the payments. Most of the payments were done in the community. Our husbands were with us. There was huge security presence in the community. We have police and EFSP Final Project Performance Evaluation Page 56 also Anti-corruption members around. Because of that, we felt that we were safe.” (Female beneficiary, Bombali District, , Phase 2) “For my safety it was not an issue because we the women were protected by the men each time we collected the money as we always walk in groups to our village after receiving the money. (Female beneficiary, Kailahun District, village, Phase 1) Concerns for beneficiary safety were also built into the payment procedures. For instance, distributions always began with beneficiaries from the village furthest away from the pay point and the elderly were often prioritized. This was meant to prevent beneficiaries from further villages from having to travel at night. Were beneficiaries ever pressured by local authorities, CICs, extended family or community members to give money they did not want to give? The overwhelming majority of surveyed beneficiaries reported that they were never asked to pay a commission or pressured to give money they did not want to give. The evaluation found that the involvement of ACC messaging during early sensitization and cash disbursements served as a largely effective deterrent. Representatives from the ACC observed cash transfers and were on hand to resolve any complaints with regard to corruption or improper requests for tribute from community authorities. In fact, many beneficiaries across Phase 1 and Phase 2 referenced the sensitization they received from ACDI/VOCA, the CIC and ACC staff regarding the reporting mechanisms established to safeguard against corruption as a contributing factor to them never being asked to pay a commission. “I was never being pressured by anybody, not even the CIC to give them money because they registered me. I did not give them and they did not ask me for a single Leone because the SNAP [+] staff and the ACC staff were with us and told us that the money is for us and we need not give a Leone to any staff or chief (P1). Nobody will even ask me for a Leone because the ACC told us to report anybody who will ask us for a Leone as it is meant for us and not to be given to chiefs neither the SNAP [+] staff (P2).” (Female beneficiaries, Kailahun District, village, Phase 1) “Nobody asked us for money and we are not pressured by any local chief or CIC. None of them ask us for money. The CIC always used to tell us that 'the money was sent by the American people and it’s for us. Let no one ask us to pay anything' and we stand by that.” (Pregnant/lactating woman, Bombali District, village, Phase 2) However, the evaluation did uncover a few instances where surveyed beneficiaries reported being asked to pay a commission. However, the follow-on question whether they paid was not explored. The overall reporting was ambiguous and after investigation by ACDI/VOCA was unable to be corroborated. On the other hand, qualitative data revealed that Phase 1 FGD participants in one location reported paying a commission; while none of the Phase 2 FGD participants reported the same during the FGDs. EFSP Final Project Performance Evaluation Page 57 “We were asked to contribute Le. 20,000 each for the office. The CIC collected the money from us. We decided to pay because the CIC said if we don't pay they will not come with this kind of help again in this community and any other opportunity coming they will not reregister us. So we paid to be on the safer side.” (Male beneficiary, Tonkolili District, village, Phase 1) It is nonetheless important to further reiterate that these were found to be isolated incidents. Overall, very few beneficiaries reported being pressured to give money they did not want to give. Rather they stated that they voluntarily shared with extended family members and community members who were not selected to receive the UTC (see the discussion on sharing below in Section 5.2.5). Interestingly, one group of male beneficiaries from Phase 1 reported on what they believed to be an indirect commission levied against them by local chiefs. According to the respondents, local chiefs in their area were prone to levying heavy fines against beneficiary households for community by-law violations because they were aware of the household’s receipt of the cash transfer. This is an unexpected finding which may warrant additional targeted sensitization for chiefs in any future cash transfer programs. “Involve the local chiefs as beneficiaries because if you do not involve them, indirectly they will heavily fine anybody that does something bad or wrong in the community - especially if they know you are one of the beneficiaries. By levying a heavy fine, they know you can afford to pay the fine. That is how indirectly they take their own share of the SNAP [+] cash.” (Male beneficiary, Tonkolili District, village, Phase 1) Has there been any other kind of conflict over the beneficiaries’ receipt of cash? As rural areas in Sierra Leone characteristically have high degrees of social cohesion, the evaluation found that conflict among extended families and the community-at-large was not common as a result of the cash transfer. While some FGD respondents reported that non-beneficiaries in their community felt jealousy and frustration for not being selected, this had not resulted in any discernible conflict. The possibility of conflict may have been lessened by the beneficiaries regularly sharing the benefits of the cash transfer with both extended family and neighbors. Examples of sharing an amount of the cash transfer, or food purchased with the cash were regularly mentioned by participants. While sharing was reported to be a regular occurrence, it is difficult to quantify its extent. Anecdotal evidence suggests that the closer the family relationship, the higher the amount shared. “I did not experience any change in relationship with my extended family because we do things in common. So when the money was paid, I bought food and other items like maggi, onions and shared it amongst my extended family members.” (Female beneficiary, Kailahun District, village, Phase 1) “Even though many people did not benefit from our community that does not create any conflict among us. We all remain peaceful and help each other when needs arise. Most of us would share small amounts of the money with those that didn't receive anything.” (Male beneficiary, Tonkolili District, village, Phase 2) EFSP Final Project Performance Evaluation Page 58 “Our community is a peace loving one and we all lived as a family so even when the cash transfer was made there was no conflict amongst those that benefited and those who did not get the money. Even at one time of payment, beneficiaries gave some amount to people who never benefited from the cash transfer. Every member of the community was either a beneficiary or a member of the household that a beneficiary came from so this made our community have no grievances.” (Community leader, Kailahun District, village, Phase 1) Where there conditions that made beneficiaries not to receive the cash at any point in time? How was this situation handled? How long did it take for ACDI/VOCA to respond to the situation? The evaluation did not find any instances where the beneficiaries did not receive their cash during project implementation. As highlighted above, beneficiaries and communities in both phases reported overwhelmingly positive reviews of the payment process, and all surveyed beneficiaries reported that they received their payments on time. ACDI/VOCA had established clear protocols to deal with missed and non-standard payments. The clear reporting and tracking mechanisms established ensured that any missed payments were clearly identified and earmarked for double payment during the following cash disbursement. Program staff were diligent in following up on any missed payments and even organized for the project accountant to meet three beneficiaries who missed the last cash transfer to deliver their cash. In the case of non-standard payments, a system was established and clear guidelines were provided to beneficiaries who could not attend on cash payout days (for instance, due to illness or death of the registered household beneficiary). To ensure that all beneficiary households received their money, designates were allowed if they satisfied two conditions: (1) presented a letter indicating the reason why the registered beneficiary couldn’t attend; and (2) had the situation and the identity of the designate verified by a local authority (usually the town chief), who would then be photographed alongside the designate. Beneficiaries reported a clear understanding of the process and appreciated having such systems available to accommodate unforeseen circumstances. “No there was no problem. All of us got the money on the right day. Only if an individual is not well, they will ask their representative to collect their money. They have to write a letter and the local chief will be around to identify the individual. They will then take a photo of the representative together with chief.” (Pregnant/lactating beneficiary, Bombali District, village, Phase 2) What was the beneficiaries experience with the non-cash assistance? In the first phase, the program addressed multiple areas of intervention aimed at Ebola recovery to improve agricultural production and markets. While these activities were successful in meeting their EFSP Final Project Performance Evaluation Page 59 objectives, they were not designed to be complementary to the cash transfer intervention and did not overlap with cash transfer households.35 In the second phase, ACDI/VOCA introduced a number of targeted complementary activities aimed at strengthening the resilience of beneficiary households. Given the revised focus on recovery during the second phase, these activities largely promoted income generating activities and recovery priorities set at household level, while still maintaining a focus on nutrition and food security. Phase 2 complimentary activities included: (1) household economics and financial management training; (2) nutrition and household hygiene training; and (3) access to savings and credit. Details on the beneficiaries’ experience with these complimentary activities are discussed in the sub-sections below. Household Economics and Financial Management Training According to ACDI/VOCA, a main lessons learned from Phase 1 was that beneficiary households needed more support to effectively plan for the cash they were to receive, and how to use it to maximize its impact. To address this ACDI/VOCA introduced training in household economics and financial management in Phase 2. The main objective of this training was to promote recovery and resilience through the opportunity the cash transfer provided. Community Facilitators were enlisted to deliver the training at community level. Trainings were kept to a minimum of two hours and focused on targeted messages. In order to encourage the most optimal use of the UCTs, targeted messages included households making money decisions together and identifying financial priorities for immediate, medium and long-term goals. The first training module was delivered during the one-day orientation session which all registered households were invited to prior to receiving their first cash transfer.36 During this session beneficiaries plotted the timing of income and expenses incurred over the course of a year against a calendar separated into the rainy and dry seasons. The timing and amounts of cash transfers were also plotted on the calendar as a tool to reinforce the concepts of planning and saving. The importance of inclusive household decision making was also discussed and emphasized during this session. Beneficiaries were encouraged to discuss and plan how they would use the cash transfer with the main decision makers in their household. Building on these key messages, the second module focused on the importance of eating different types of foods and strategies for using the transfer money to help beneficiaries fill important nutritional gaps in household diets. Examples of common foods in important food categories were discussed according to how they benefit the body. Key messages on the importance of household consumption of foods typically grown for sale were also presented. Lastly, the third module was delivered after the last cash transfer and focused on sustainability and the way forward. Beneficiaries were engaged to share what they had learned to date, how they have 35 For instance, agricultural interventions in Phase 1 targeted SNAP Farmer Field Schools and provided emergency seed recovery packages. However, not all CTP beneficiaries benefited from this intervention. 36 During the orientation session, beneficiaries learned about USAID, ACDI/VOCA and the cash transfer program. Sensitization on the selection process was also reiterated and beneficiaries were informed about the amounts of money they would be collecting at each cash transfer and the month when the three cash transfers would occur. EFSP Final Project Performance Evaluation Page 60 used the cash transfer to make their households more food secure and how they have used the cash transfer to plan for the future. Overall, 96.1% of all participating households attended at least one of the household economic trainings during Phase 2. However, it is worthwhile to note that participation in the household economics training declined substantially after the first session (see Figure 15). The initial high participation in the first module was largely driven by beneficiary early anticipation to learn more about the program and their receipt of cash. As ACDI/VOCA did not offer any incentives to participate in the trainings, and considering competing household priorities, attendance dropped off after the CTP orientation session. Given the initial high attendance, and the fact the key messages from the training were delivered at pay points, it can be assumed that virtually all beneficiaries had been reached by the economic and financial management training messages to some degree. Figure 15: Number of beneficiaries participating in each module for the Household Economics and Financial Management Training 8221 4430 5251 Module 1 (Jun/Jul) Module 2 (Sept) Module 3 (Oct/Nov) Quantitative endline data indicated that among the beneficiaries that attended the training, the majority (95.2%) reported that the training helped them to make decisions about how to spend the money. The influence of the training messages on spending decisions ultimately impacted the household’s recovery. It can be recalled that the quantitative research indicated that Phase 2 beneficiaries that used the financial management training to make household spending decisions were less likely to report ‘moderate to severe hunger’ (8.0% compared to 11.1%) and have higher HDDSs (8.2 compared to 6.0) when compared to beneficiaries that reportedly did not use the training messages. (See Section 5.1.1 and 5.1.2). EFSP Final Project Performance Evaluation Page 61 Figure 16: HHS and HDDS based on use of household economics training messages to make spending decisions 8.0% 11.1% Moderate to Severe Hunger HHS: Used HH economic training HHS: Did not use HH economic training 8.2 6.0 Household Dietary Diversity Score HDDS: Used HH economic training HDDS: Did not use HH economic training Given the ‘recovery’ rather than ‘emergency response’ orientation of Phase 2, the financial management training encouraged households to diversify their spending with a view to improve their longer-term resilience and ability to cope with future shocks. Qualitative research indicated that Phase 2 beneficiaries were in fact more likely to diversify how they spent the cash transfer after the training. Many respondents reported that they altered their initial plans for the cash transfer after receiving the training with a view to grow the cash and save for future emergencies. There was an acknowledgement that this advice has led to positive impacts on the household’s food security as returns on their investments increased their access to food. “When we received the money, most of us in the community didn’t have money to buy seeds for our farming and we were planning on how to buy seeds for farming. When we received the money, the first thing that came to our mind is to buy seeds for farming. When we attended that training everything changed. They told us that we should not use all of the money received on farming, we should use the money to pay for our child’s school fees, do some business and save some in the [VSLA] box for the future. This has helped us to now be stronger and take care of the household's need. We can now get more food and medicine when we need it.” (Female beneficiary, Bombali District, village, Phase 2) FGD participants also reported learning about inclusive decision making among household members on how to use the money. There was overwhelming support for this concept among both male and female beneficiaries. As the discussion on gender implications below found (see Section 5.2.8), the belief that the cash was intended for household expenditure supported women to have a voice in household decision making on how the UCT would be used. “The training helped us a lot about how to manage the money. They told us that the money is not only meant for individual people. They told us that the money is for the entire household and that we should plan together how to use the money for the future. So whoever receives the money, if it is the husband or the wife, they should come together to work things out.... After I received the training I was able to prioritize the things we need for the household all year.” (Female beneficiary, Tonkolili District, village, Phase 2) EFSP Final Project Performance Evaluation Page 62 Many beneficiaries reported that the training had provided them with new financial management skills they did not previously have. There were numerous references to prior mismanagement of cash and doing things differently after receiving the training with a view to plan for the future. Many beneficiaries reported on the importance of building household food stocks and diversifying how they spent the cash to maximize profits and plan for future emergencies. “We attended the training before we were given the money. The training helped me a lot on how I should use the money. I was told in the training that I should use some of the money for the household needs, but to also use the money to make more profit. Before I used to just spend the money anyhow, but this training has really helped me to use the money correctly.” (Male beneficiary, Tonkolili District, village, Phase 2) “Since participating in the training, I now know you must be able to keep like three bags after harvest for future emergency. You can eat some, but this way you can sell some and then use that money to solve household problems. We can even use part of it to pay laborers the next planting season. Most of us never used to do this because we did not have a plan for the next month.” (Female beneficiary, Tonkolili District, village, Phase 2) Overall, the evaluation findings indicate that beneficiaries benefited greatly from the household economics and financial management training. The messages resonated well with current realities and the planning tools (e.g. calendar) employed by the program were effective in demonstrating the need to properly plan the use of the cash transfers. Based on feedback from beneficiaries, the training impacted how they decided to use the cash and many reported changing their household decision making processes with a view to planning for the future. Maternal/Child Nutrition and Hygiene Training A number of short training modules focused on maternal/child nutrition and basic hygiene were delivered at community level. The training was organized in five modules: (1) maternal, infant and child nutrition; (2) hygiene; (3) complementary feeding; (4) essential hygiene; and (5) household nutrition. The EFSP used the materials developed for SNAP to deliver the maternal/child and basic hygiene training. A review of the topics covered suggests that there was some overlap in training materials. For instance, the key messages delivered as part of Module 5 “household nutrition” was very similar to the material presented in Module 2 under the household economics and financial management training. Furthermore, hygiene messages delivered in Module 2 and Module 4 were interlinked and largely served to reinforce the same messages. Unlike the economic training, module topics were combined and delivered through a total of three separate trainings according to the following: ● Hygiene and sanitation training delivered in June 2017 focused on key messages for handwashing; unsafe water sources and boiling water; and clean water storage. ● Nutrition training delivered in July/August 2017 focused on key messages including households eating from a minimum of four food groups; frequency of meals for pregnant/lactating women; and eating from a minimum of four food groups; immediate and exclusive breastfeeding for children under six months of age; and complementary feeding for children 6-59 months of age. EFSP Final Project Performance Evaluation Page 63 ● Nutrition training delivered in August/September 2017 focused on household diet and consumption of a minimum of four food groups; own food production and rearing of livestock for consumption (not just sale). Overall, 83.6% of registered households attended at least one of the nutrition and hygiene trainings. However, as highlighted in Figure 17 below, participation in the training sessions dramatically declined after the first hygiene/sanitation session in June 2017. While the nutrition sessions were substantially less attended, the messages did have a larger reach as the subject matter in Module 2 of the household economics training covered many of the topics in the second nutrition training focused on household diet and consumption. Furthermore, cash transfer events were used to provide cooking demonstrations featuring complimentary foods for children 6-24 months using locally-available ingredients coupled with nutrition messaging delivered by lead mothers. Nonetheless, a number of FGD respondents reported that they did not receive targeted nutrition training at community-level. Figure 17: Number of beneficiaries participating in each of the maternal/child nutrition and basic hygiene training sessions 7309 3872 1815 Hygiene/Sanitation (Jun) Nutrition 1 (Jul/Aug) Nutrition 2 (Aug/Sept) Quantitative endline data indicated that among the beneficiaries that attended the training, the majority (94.1%) reported that the training helped them to make dietary decisions for their family. The impact of the training messages on dietary decisions ultimately improved the household’s food security. It can be recalled that the quantitative research indicated that Phase 2 beneficiaries that used the nutrition training to make household dietary decisions were significantly less likely to report ‘moderate to severe hunger’ (5.0% compared to 12.9%) and have higher HDDSs (8.2 compared to 7.3) when compared to beneficiaries that reportedly did not use the training messages. (See Section 5.1.1 and 5.1.2). EFSP Final Project Performance Evaluation Page 64 Figure 18: HHS and HDDS based on use of household economics training messages to make spending decisions 5.0% 12.9% Moderate to Severe Hunger HHS: Used nutrition training HHS: Did not use nutrition training 8.2 7.3 Household Dietary Diversity Score HDDS: Used nutrition training HDDS: Did not use nutrition training Focus group discussions with women that participated in the nutrition training revealed very high recall for messages related to hand washing, household diet and consumption of important food groups, frequency of meals for pregnant/lactating women, exclusive breastfeeding and complimentary feeding. Women reported being given practical tools to implement what they learned and were able to give specific examples of how they have changed the food they prepare for themselves and their children. For instance, a number of women reported adding more local ingredients to their children’s diet, introducing more foods in their household diet and moving away from serving plain rice as the main meal. Respondents also reported receiving messages encouraging them to consume food which they would normally grow for sale, rather than for consumption. Women repeatedly reported that they had no previous knowledge on the importance of preparing other foods for their family. They reported being very open to these messages, and applying what they learned in the training to diversify their household’s food consumption. Many reported that they have since noticed a difference in their own health and that of their children. “Before the training, I used to give my child local rice made pap with no ingredient added to it. But since this training, I’m preparing different varieties of foods and also adding different ingredients in my child's food like, palm oil, fish etc. Now I can prepare biscuit for my child especially my son who is over 6 months old. Even though, things are very difficult but since I attended the training, we are not only eating rice at home. Now I can prepare cassava in the morning, in the afternoon rice and other things for my entire family.” (Female beneficiary, Bombali District, village, Phase 2) “When I attended the training I learned so many things about the types of food we need to eat when we are pregnant. They also taught us how to prepare the food we eat during pregnancy. Thanks to SNAP [+] for this kind of training, I really benefited from it. As you can see, I am pregnant but I never knew the kind of food pregnant women need to eat and how to prepare it. Now I know that I need to eat meat and fish and other good foods, not only cassava, at least three times a day. I was never doing this before, but because of the training, I now prepare these foods for me and my family's good health. (Pregnant beneficiary, Bombali District, village, Phase 2) “We were told that we should not be only selling our yams, beans, cassava, fruits. That we should be eating them ourselves. Since we started eating them, we have really seen the changes. (P1) We were told that most of these things we plant we should be eating them and not only selling them. The EFSP Final Project Performance Evaluation Page 65 people when they came they told us that we should be eating cocoa, beans, egg, palm oil, potato, cassava leaf, oranges, and later cook rice in the evening to eat which will be our food before bed (P2).” (Female beneficiaries, Bombali District, village, Phase 2) Taken together, the quantitative and qualitative data suggests that the nutrition training was overwhelmingly successfully in conveying practical messages that had an impact on the nutrition and food security of households that received the training. However, there were issues with the rollout of the training as participation in all sessions quickly dropped off after the initial trainings due to availability of beneficiaries and lack of incentives to participate. Village Savings and Loans Associations (VSLAs) During Phase 1, ACDI/VOCA established Village Savings and Loans Associations (VSLAs) to increase access to financial services for the rural poor in their target areas. The VSLA methodology was based on best practices and included the member-managed groups holding weekly meetings for members to purchase shares, make social fund payments and loan re-payments. Low interest loans were provided to members from the savings fund, and social fund grants and interest-free loans were granted to members for emergencies. As a self-sustainable and self-replicating mechanism, shares and returns from interest income on loans are shared out to members at the end of the saving cycle (typically between 6-12 months). While the VSLAs were managed by community members, ACDI/VOCA supported the formation of the groups by providing training, a ledger, calculator and a lock box for the safe keeping of the money. At the end of SNAP, a private service provider model was introduced to increase access to this highly valued intervention. Many cash recipients in Phase I were able to access VSLA services through these private service providers. In Phase 2, the approach was repeated to include private service providers (PSPs) as the main implementers of community based VSLAs targeted to cash transfer beneficiary households. A total of 75 community members from target areas with numeracy and literacy skills were selected and trained as PSPs in the VSLA methodology developed under SNAP. Once trained, each PSP was provided with two toolkits (metal box, locks, ledgers, calculator, bowls/bags) to start two VSLAs each. PSPs received no compensation from ACDI/VOCA. Instead, PSPs entered into agreements with VSLAs to reimburse them for their time and the cost of the toolkit. According to ACDI/VOCA this approach has showed signs of sustainability as 19 PSPs to date have used the payments from VSLAs to purchase toolkits and start additional VSLAs. Overall, a total of 150 VSLAs with 4,477 members were established in Phase 1. However, as Phase 1 addressed multiple areas of intervention, not all cash transfer beneficiaries were targeted by the VSLA component. In fact, more than half of the Phase 1 FGD participants reported that they did not participate in any VSLA activities; and only 38.4% of endline respondents reported joining a VSLA during the CTP implementation. The coverage of VSLA membership among cash transfer beneficiaries was much higher in Phase 2. Accounting for non-participant households and multiple members from the same household, a total of 4,088 beneficiary households were enrolled in the 150 VSLA groups formed in Phase 2 (from a total number of 4,170 households that joined the VSLA groups). The greater scope of VSLA coverage during this phase was also found during the evaluation data collection. All but one of the FGD groups EFSP Final Project Performance Evaluation Page 66 reported that a VSLA had been established in their community; and 81.4% of Phase 2 endline respondents indicated that they had joined a VSLA since receiving their first cash transfer. An assessment of the quantitative data revealed that cash transfer households linked to VSLAs demonstrated lower levels of ‘moderate to severe hunger’ and higher household dietary diversity scores (See Section 5.1.1 and 5.1.2 for greater details). Figure 19: HHS and HDDS based on VSLA membership Qualitative research suggests that beneficiaries valued the positive impact of their VSLA membership almost as much as the cash transfers. There was an overwhelming appreciation for the complimentary benefits provided through the cash transfers and the credit facility. Phase 1 beneficiaries were able to attribute positive gains to their household livelihood as a result of their VSLA membership one year after joining; and participants from Phase 2 were able to realize the immediate gains months after joining. Beneficiaries reported using loans for investing in income generating activities, school fees and medical expenses. Having access to credit and savings was reported to be one of the biggest benefits of the VSLA as it allowed households to maximize the impact of the original cash transfer. The social fund (“problem money”) was highly valued as a true safety net by beneficiaries. Many reported having no recourse to respond to emergencies in the past, and were grateful for the facility. “Since I have joined the VSLA, it has helped me to add on to my business. I started doing business with two jerry cans palm oil, now my business has increased to a drum of palm oil. I will then add more to the business after the sale. This has helped me to provide more food for my family without taking debt.” (Female beneficiary, Kailahun District, village, Phase 1) “When my son was sick, at that time, I didn’t have any money with me, but with the help of the box I was able to take loan from the box and took my son to the nearest health centre for treatment.” (Female beneficiary, Bombali District, village, Phase 2) “I use loans from the box to improve on my business, and it really helped. We also thank God for the box. Even though we were given that money, had it not been for the box, the money would have been finished before this time. By contributing to the box one has hope of reserve cash for after every six months.” (Female beneficiary, Tonkolili District, village, Phase 2) “The VSLA BOX is so important in our community. It helped me greatly to pay my son's final college fees in Makeni and I used a little part of it to buy foodstuff for the household.” (Female beneficiary, Bombali District, village, Phase 2) 23.1% 12.3% 35.8% 16.4% Phase 1 Phase 2 Moderate to Severe Hunger VSLA non-VSLA 8.1 8.2 6.9 8.6 Phase 1 Phase 2 Household Dietary Diversity Score VSLA non-VSLA EFSP Final Project Performance Evaluation Page 67 In addition to these benefits, the longer term impact of the VSLAs on reducing the beneficiaries’ debt cycle were often cited as the most significant change experienced since joining the VSLA. In fact, the endline survey found that a very small proportion of VSLA members across both phases reported taking loans since their last cash transfer (9.4% in Phase 1 and 6.9% in Phase 2). In the absence of baseline data, it is difficult to measure the impact of the VSLAs on the beneficiaries’ debt burden (and the individual circumstances which have led to the debt being taken). However, the qualitative data does suggest that VSLA members have substantially reduced their reliance on debt to meet their household needs. FGD participants from both phases reported that they have reduced their reliance on taking loans from moneylenders since joining the VSLAs. Access to the low interest loans through the VSLA were praised as being fair, accessible and reliable. Many respondents reported that the access to credit has assisted their households to break the cycle of debt to money lenders and attain sustainable recovery. “For me joining this box has really changed my life. Many people knew me for always taking loans, and [money lenders] were always running after me for their money. Since the box came, I have never had to take a loan again as I am able to meet my family’s needs.” (Male beneficiary, Kailahun District, village, Phase 1) “The first benefit [of participating in SNAP+] is the VSLA box. This box has really helped my family. Before, I used to take heavy loans and always struggled to pay back that money. But now that we have this box, it is one of the biggest benefits as I have been able to quickly attend to emergencies in the house and when there is not enough food, I can use my savings instead of taking heavy debt from the moneylenders.” (Male beneficiary, Kailahun District, village, Phase 1) What were the gender implications of the cash transfer in terms of gender relations, decision making and protection? ACDI/VOCA made concerted efforts to integrate gender considerations in the CTP design. Beyond targeting women for the cash transfers, a number of measures aimed at improving beneficiary protection were included in CTP planning and implementation. Approaches included: engaging NaCSA, ACC and MSWGCA to deliver sensitization on targeting and objectives of the UCTs; ensuring CIC members of both genders were involved in beneficiary selection; setting up accessible grievance redress mechanisms; limiting the travelling distance to pay points; paying beneficiaries who were furthest away first to prevent travelling at dark; and providing training to all beneficiaries on the use of the funds and how to withstand pressure from spouses, extended families, and neighbors. Qualitative results indicate that the effects of the cash transfer on women were largely positive. At the individual level, female beneficiaries self-reported increased self-esteem and confidence in making decisions and handling money to meet their household’s needs. Further, beneficiaries of both sexes spoke about feeling “proud” and “accomplished” in the changes brought about by the cash transfer. EFSP Final Project Performance Evaluation Page 68 “At the time I received the money, I had no hope to start any business because my only hope was my uncle and he died during the Ebola crisis. But I used some of the money to buy goats and I managed on my own to start a business. Everyone was surprised that I did this alone because I never did this before and my uncle always took care of money. I am so proud today of how I did this and changed my life.” (Female beneficiary, Bombali District, village, Phase 1) At the household level, nearly all beneficiaries reported that the cash transfer created “peace” among household members and improved relations between husband and wife. This was largely reported to be as a result of household food needs being met with the cash. As the pressure of providing food for the family was lessened, beneficiaries reported less tension between spouses and across generations. “There is more unity and love between me and my wife because there is food at home to eat.” (Male beneficiary, Bombali District, Fandehun village, Phase 1) “Since the starting of the payment, there is peace in my house. My husband and I used to quarrel before because it was difficult to provide food for the children. Now there is food so there is more peace. The children are even happier.” (Female beneficiary, Bombali District, village, Phase 2) There were a number of interesting reports from both male and female beneficiaries regarding the changes to gender relations within the home during the cash transfer period. According to some female beneficiaries, receipt of the cash transfer (temporarily) made them more attractive to their husbands who would come closer to them when they wanted money. Conversely, a number of male beneficiaries reported that the receipt of the cash made their wives have more respect for them. “When I was receiving the cash it brought me and my husband closer. At that time, my husband would always stay at home. Since the start of the first payment he would always come closer to me whenever he needed money.” (Female beneficiary, Tonkolili District, village, Phase 1) “I have to say thanks to SNAP [+] because before the cash transfer I was only living by the power of the Almighty. I had no respect from my wife because I had no money to even buy a day’s meal as my income had dropped since of the Ebola. When I received the cash and bought some food stuffs and paid the debts, the relationship with my wife grew to another level. She now sees me as a better man and has great respect.” (Male beneficiary, Kailahun District, village, Phase 1) Overall, beneficiaries reported that the cash transfer resulted in a strengthening of gender relations and even reduced the amount of intra-family conflict. However, as households are unique in character, different scenarios were captured by the research team. Anecdotal evidence suggests that polygamous households were more likely to experience intra-household conflict if only one wife received the cash. As beneficiary targeting focused on individual household economies and not marital relationships, splitting the cash was not always a feasible solution. Nonetheless, ACDI/VOCA field staff did uncover cases where this happened, resulting in a greater potential for conflict. Conversely, there were reports of sharing among polygamous households which did not result in conflict. This suggests that family dynamics is an important element in the context of cash transfers and polygamous households. Even so, in the case presented below where conflict was minimized EFSP Final Project Performance Evaluation Page 69 due to sharing the cash transfer, splitting the UCT ultimately reduces the impact of the cash on the registered household’s recovery. “When SNAP [+] gave that money, I have two wives and they both have four children. So I divided the money into two. I gave half to each wife for them to send children to school. I also gave the remaining money to them so they can start a business. From that business they can be able to give lunch and pay some school expenses for the children.” (Male beneficiary, Tonkolili District, village, Phase 1) Within the household, beneficiaries reported overwhelming agreement on the use of cash between key decision makers. The qualitative research revealed that most beneficiaries from both phases reported consulting with their families on spending decisions. In these cases, both male and female beneficiaries reported seeking out the advice of their spouse, and in the case of widows, older children and extended family, on how to spend the cash. However, the data does suggest that Phase 2 beneficiaries were slightly more likely to report following a more family-oriented approach to decision making, while some female beneficiaries reported simply “handing over the cash” to their husband as he is the “head of the household”. “I sat down with my wife and we decided to pay school fees and school materials for our children and the balance money we bought seed rice for farming and we also used some for food.” (Male beneficiary, Bombali District, village, Phase 1) “When I received the money, I took it straight to my husband as he is the household head.” (Female beneficiary, Kailahun District, village, Phase 1) “For me I discussed with my son, daughter and husband. I told them that this money belongs to all of us and not me alone. I told them that we should not only eat the money but we should try to do something with some of the money so that it will serve as a reserve in case things become rough.” (Female beneficiary, Tonkolili District, village, Phase 2) The quantitative findings confirmed that inclusive decision-making was in fact practiced by the majority of beneficiaries across both phases. However, as highlighted in Figure 20 below, a slightly higher proportion of Phase 2 beneficiaries reported that they discussed how to spend the money with other household members after they received it (95.3% compared to 88.2%). Taken together with the qualitative data, this suggests that the incorporation of modules on household economics and decision-making in Phase 2 promoted a slightly more family-orientated, gender inclusive decision making process on household spending. Nonetheless, across both phases, the CTP encouraged inclusive decision making and gave women a voice in household economic decisions. Figure 20: Did you discuss how to spend the money with other household members after you received it? 88.2% 95.3% 11.8% 4.7% Phase 1 Phase 2 Yes No EFSP Final Project Performance Evaluation Page 70 Targeting females for the UCT was overwhelmingly accepted by both male and female beneficiaries. This was largely based on a perception that women and men have different spending priorities – with women being responsible for daily household needs and management. It was reported that women were most likely to manage the cash responsibly and in line with household needs as they are the ones who are “staying at home to run the affairs of the household, do the cooking, check on the children’s health, monitor them from school, clean, etc.” In this way, the female’s receipt of the cash transfer tended to reinforce, rather than challenge, women’s traditional roles. Qualitative research indicates that while women were often looked to as sensible, honest and righteous in their spending; men were often described with negative stereotypes of misspending and diverting household funds to drinking or “taking another wife”. In light of these binary distinctions between male and female spending, the belief that the cash was intended for household expenditure supported women to largely control the UCT unchallenged. “The women should be handed the money because she is the one who takes care of the household affairs. Women are very much careful in managing money compared to we the men. Some of us will drink or smoke half of the money. So it is better we give to the women to manage and take care of the household. We the men, we should just provide but the women do every other thing.” (Male beneficiary, Kailahun District, village, Phase 1) “For me there is a great difference when a man received money for the home and a woman received money for the home. Me as the woman I received money for the home, because I know that the home responsibility is on my shoulder. I mean I take care of everyone in the home. I will take my time to manage the money as compared to man who will only collect the money and give it to you for you to run the home. Some men will even go to the street and misuse the money and later come back in regret.” (Female beneficiary, Tonkolili District, village, Phase 2) Overall, while there were many noted positive benefits for women, the research suggests that the CTP was not transformative in terms of gender relations. Targeting the women for cash transfers did empower them to fulfill their role as caregiver more effectively; however it did not necessarily translate into changed or improved gender relations, roles or perceptions. Women’s empowerment is a long-term goal which includes societal and cultural changes at various levels. The short duration of the program means that it is unlikely that any longer term changes would have emerged in terms of gender relations or women’s long term empowerment. Nonetheless, the project was not found to introduce any negative gender impacts, and did manage to successfully empower women to participate in decision making process and fulfill their roles more effectively during the cash transfer period. 4.3. Program Coordination and Collaboration This section concerns the extent to which the coordination and collaboration with public and private sector stakeholders impacted program effectiveness and efficiency. Information provided has been gathered through interviews with project staff, key sector stakeholders and partners as well as through a review of program records and existing documentation. As the project was coordinated with the same actors and institutions across both phases, findings are presented as a whole for the entire project duration. EFSP Final Project Performance Evaluation Page 71 How did ACDI/VOCA coordinate activities with government authorities, FFP implementing partners and other key stakeholders?To a large extent, the Cash Transfer Working Group (CTWG) provided the primary platform for ACDI/VOCA to coordinate activities with government authorities, FFP implementing partners and other key stakeholders. From the outset, efficient coordination mechanisms and standard operating procedures were established to guide the cash transfer interventions of various partners and stakeholders. As Phase 1 activities began during the Ebola response, a number of actors were showing interest in providing cash transfers. In response, the CTWG was established in November 2014 to provide a forum for coordination and discussion among the private sector, GoSL, NGOs and donors considering, or already implementing CTPs in the country. Initially led by the UN Food and Agriculture Organization (FAO), ACDI/VOCA later assumed sole responsibility of chairing the bi￾monthly CTWG meetings. All FFP implementing partners participated in the CTWG throughout CTP implementation. It was agreed through the President’s Recovery Priorities that all cash transfer activities would be reported under the Rapid Ebola-Social Safety Net (RE-SSN) Pillar through NaCSA. As NaCSA had implemented a national Social Safety Net (SSN) program prior to the Ebola outbreak, all CTPs in the country were bound to operate within the existing social protection structures. Further, in order to provide cash transfer actors with coherent guidelines for implementation, the GoSL released the ‘Ebola Virus Disease (EVD) Response-Social Safety Net Cash Transfers Standard Operating Procedures’37in May 2015. The Standard Operating Procedures (SOPs) provided clear guidance on a number of integral aspects of cash transfer programming, including: (a) targeting strategy; (b) grievance redress mechanism; (c) transfer value and mechanism; (d) duration and transition; (e) stakeholder and implementation arrangements; and (f) stakeholders’ responsibilities and roles. A memorandum of understanding (MoU) was signed with NaCSA at the beginning of program activities, aligning ACDI/VOCA’s cash transfer interventions with the SOPs. ACDI/VOCA also submitted detailed implementation plans and weekly reports to the President's Recovery Team through NaCSA in line with the MoU. In Phase 2, as the RE-SSN Pillar fully achieved its targets the coordination of cash transfer interventions was coordinated with NaCSA through the CTWG. During this second phase of implementation, there was less responsibility to report on activities beyond NaCSA as the emergency phase had ended and cash transfers were coordinated under the national entity responsible for social safety net programming. Overall, throughout Phase 1 and Phase 2, ACDI/VOCA collaborated with a variety of organizations during the implementation of the CTP. Each organization’s respective involvement in the EFSP is outlined below. ● National Commission for Social Action (NaCSA): Overall coordination of cash transfer programming. Initial training on targeting and registration of participants in accordance with the 37 Government of Sierra Leone (May 2015). Ebola virus disease (EVD) Response-Social Safety Net Cash Transfers Standard Operating Procedures’. Available at: https://www.humanitarianresponse.info/system/files/documents/files/standard_operating_procedures_sop_08.05.15.pdf EFSP Final Project Performance Evaluation Page 72 GoSL social safety net criteria. Thereafter, representatives from the District NaCSA office were on hand at cash distributions to oversee and monitor the cash transfer process. ● Anti-Corruption Commission (ACC): Participated in the sensitization of CICs used for targeting eligible beneficiaries. Thereafter, participated at distribution sites and post distribution monitoring and resolution of any findings. ● Ministry of Social Welfare, Gender and Children Affairs (MSWGCA): Assisted with the registration to provide messaging on gender-based violence or sexual exploitation that may be linked to cash transfers. Thereafter, representatives from the District Social Welfare office were on-hand at cash distributions to further reinforce messaging and provide problem-resolution. (Phase 1 only) ● Office of the President: Coordination of cash transfer and reporting against the President’s Recovery Plan. (Mainly during Phase 1) ● SPLASH and Rokel Commercial Bank: Offline mobile money providers ● Nestbuilders International (NBI): Provided electronic data collection during beneficiary registration and verification of payments during all cash distributions. To what extent did the coordination and collaboration with public and private sector stakeholders enhance capacity to deliver cash transfers and impact program effectiveness? Coordination and collaboration with public and private sector actors was a major strength of the EFSP. The coordination mechanisms provided through the CTWG ensured harmonization of activities, alignment with government strategies and shared learning. This ultimately enhanced the capacity of ACDI/VOCA to effectively deliver cash transfers during the emergency and recovery phases. The coordination was praised by ACDI/VOCA and FFP implementing partners’ project staff as being a major accomplishment of the CTP. Evidence from interviews suggests that the following coordination activities directly impacted on program effectiveness and enhanced the achievement of results: ● Coordination through the CTWG allowed ACDI/VOCA and FFP implementing partners to harmonize the value of cash transfers as well as targeting criteria, ensuring consistency of approach across development partners. This provided clear guidance to NGOs who were implementing cash transfers for the first time in Sierra Leone. What resulted was enhanced capacity, and a consistent, streamlined approach to delivering cash transfers to beneficiaries across various operational areas. ● The CTWG provided a platform for anticipated areas of program intervention to be discussed and validated with NaCSA and FFP implementing partners to ensure that there was no overlap or duplication of program activities. For instance, through the CTWG, geographic areas were affirmed with FFP Implementing partner CARE and NaCSA both providing cash transfers in Bombali/Tonkolili Districts to avoid duplication or overlap with other social safety net programs in the area. EFSP Final Project Performance Evaluation Page 73 ● Coordination with NaCSA through the RE-SSN, and later the CTWG, ensured cohesion between ACDI/VOCA’s activities and the national SSN program framework. Harmonization of critical elements such as targeting, monitoring and transfer amounts ensured ACDI/VOCA’s CTP was properly aligned with the national Social Safety Net program framework. (The role played by the coordination mechanism in aligning the CTP with national strategies is discussed in greater detail below in 5.3.3) ● The collaboration with local district government officials (e.g. NaCSA, ACC) during all cash distributions allowed cash transfer recipients to access each government ministry’s respective redress mechanisms in person. This not only enhanced the accountability of the CTP, but supported the efficient implementation of the GRM. ● As all FFP implementing partners were using SPLASH for cash distribution, the CTWG provided a platform for all partners to directly engage SPLASH on technical issues. Meetings were held to discuss bottleneck and strategies for improving cash transfer modalities with SPLASH. This allowed implementing partners to leverage their collective requirements and make the business case for improvements in the timeliness of reconciliation and delivery of payment information to implementing partners. ● The CTWG provided a platform for information sharing and collective learning among government authorities, private sector service providers, donors and FFP implementing partners. The CTWG meetings regularly included presentations and information sharing on completed CTP studies, identification of new technologies and opportunities for participation with the Cash Learning Partnership (CaLP) and other CTP actors to continue to improve the cash transfer programming in Sierra Leone. ● In Phase 2, the CTWG provided a platform for FFP implementing partners to re-visit the cash transfer amount in light of rising inflation and dramatic currency depreciation in Sierra Leone.38 As the purchasing power of UCT beneficiaries was dramatically reduced, all partners coordinated their budgets to continue providing the USD$30 per month, however a new exchange rate was set.39 ● Lastly, implementing partners reported that USAID/FPP maintained direct involvement and supported coordination from the outset of the CTP. By providing valuable technical input, remaining flexible to changing circumstances and discussing solutions with implementing partners, the substantial involvement of USAID/FPP was integral to supporting the effective delivery of the CTP. How did ACDI/VOCA address the issues of parallel systems and structures? One of the hazards when implementing cash transfer systems in emergencies is the potential for humanitarian actors to set up parallel systems which undermine government social-welfare systems, 38 According to the World Bank: Rising from a base of 9.5% in December 2015, inflation reached 17.41% in December 2016. The local currency (the Leone) had depreciated by 28.73% in December 2016 (year-on-year). See: World Bank (2017). Sierra Leone Overview. See: http://www.worldbank.org/en/country/sierraleone/overview 39 Using an exchange rate of 7000SLL/$1, cash transfers were set at 210,000 SLL per month or 630,000 SLL per quarter as opposed to the 390,000 SLL per quarter distributed under Phase I. EFSP Final Project Performance Evaluation Page 74 or remove government responsibility to provide them.40 While NaCSA made the first cash transfer under the SSN program in 2014, there was limited capacity to scale the national program rapidly in response to the Ebola emergency. Recognizing the scope of need, and the importance of avoiding potential parallel systems, the GoSL developed the SOPs with the aim of ensuring a smooth transition from short term emergency interventions to long term sustainable government-led social safety net approaches. ACDI/VOCA, along with the other FFP implementing partners, effectively addressed the issue of parallel systems and structures by ensuring alignment with government policies. By entering into a MoU with NaCSA and operating according to streamlined modalities (e.g. targeting, transfer amounts, etc.), ACDI/VOCA implemented a program which was compatible with the government’s national social protection strategy. Through direct coordination with NaCSA, ACDI/VOCA and FFP implementing partners also contributed to system strengthening of ongoing national social safety net programming. First, ACDI/VOCA operated according to modalities which were similar and compatible with the pre￾existing social protection instruments and operational capacity of NaCSA. Following NaCSA’s LPMT targeting approach, ACDI/VOCA collected information which was in line with national targeting approaches and could feed into NaCSA’s database of extremely poor households. Furthermore, the dollar amount was selected in close consultation with NaCSA who set a monthly cash transfer value of USD$30 during the RE-SSN response period. In these ways, ACDI/VOCA’s activities were aimed to contribute to and be in alignment with the President’s National Recovery Plan for EVD. Furthermore, mapping of geographic target areas and data was organized through a World Bank Consultant to ensure that any data collected by the CTP was consistent with the National Social Safety Net Program MIS system and could be integrated into NaCSA database. The use of the social safety net MIS platform developed by World Bank ensured that government had information on Ebola recovery actors including areas of operations, number of households served and could provide accurate information to the President Priorities; but also could inform new entrants to cash transfer programming regarding un-served areas for potential intervention. The information provided by partners could also be used for transition from the recovery program to the longer-term social safety net program, as appropriate. In these ways, ACDI/VOCA worked through a government institution with a view to avoid the development of a parallel system. There was a focus on strengthening pre-existing, state-run, social protection instruments and building upon them, rather than creating parallel interventions. This ultimately supported greater sustainability of the intervention and government ownership. According to the SOPs, households that qualified under the LPMT “may be absorbed and captured under the national social safety net project which has a 24 month duration”.41 However, at the time of the evaluation, there was no indication that this had happened. 40 Oxfam (2006). Cash Transfer Programming in Emergencies. Available at: http://www.cashlearning.org/downloads/resources/guidelines/oxfam_guidecash_trsfer_prog.pdf 41 Government of Sierra Leone (May 2015). Ebola Virus Disease (EVD) Response-Social Safety Net Cash Transfers Standard Operating Procedures’. EFSP Final Project Performance Evaluation Page 75 How did other interventions in targeted communities impact results? Evidence from interviews with ACDI/VOCA suggest that, during Phase 1, many of the development activities operating in the program areas were suspended during the Ebola emergency and had not resumed. In the second phase, during geographic mapping other activities were reviewed for overlap. It was found that there was little other development activities being undertaken in the areas where ACDI/VOCA was providing cash transfers so this was not a factor in the program. Furthermore, as stated above, a number of mechanisms were put in place to ensure that the EFSP avoided duplication and overlap with other cash transfer programs. Efforts were also made to review interventions similar to the complimentary services provided under the CTP. This included reviewing target communities to determine if any areas were currently covered by implementers providing VSLA services to avoid overlap. Conversely, PSPs in un-served areas were identified and trained to deliver VSLA services. EFSP Final Project Performance Evaluation Page 76 5.Achievement of Project Targets The table below presents a summary of achievement of the project’s indicator targets according to the project’s Indicator Performance Tracking Table (IPTT). Overall, the program has performed exceptionally well, with all indicator values at or above 85% of the proposed targets. Out of the 11 total indicators, 5 were fully achieved (>100%) and a further 3 indicators were within 99% of the proposed targets. More specifically, the ACDI/VOCA CTP largely met its targets in terms of the total number of beneficiaries benefiting from the program, cash transfers and complimentary services (>90%). The greatest achievement of targets was observed in the improvement of key food security indicators (HHS and HDDS), resulting in targets being surpassed (>100%). Table 20: EFSP Indicator Performance Tracking Table, Achievement of Indicators Disaggregation Baseline Value Target Actuals % Target Achieved # of people benefiting from SNAP + programming Gender Phase 1 + Phase 2 0 173,056 166,348 96% Phase 1 Total42 123,907 Phase 2 Male no target set 19,350 Female no target set 23,091 Household Dietary Diversity Score Phase 1 (Baseline June 2015/Endline July 2016) 1.9 6.5 6.7 103% Phase 2 (Baseline April 2017/Endline October 2017) 4.3 6.5 8.2 126% Household Hunger Score (% of HHS reporting little or no hunger during the past 30 days) Phase 1 (Baseline June 2015/Endline July 2016) 4.7% 60% 66.2% 110% Phase 2 (Baseline April 2017/Endline, October 2017) 0.3% 60% 86.9% 145% Objective 1 - Increased Agricultural Productivity IR1.1 Increased Agriculture at HH Level Phase 1 (April 2015 to December 2016 – Producer Groups) 0 27,980 23,800 95.7% Phase 1 (June 2015 to December 2016 – Smallholders) Not Applicable 0 No target Set 2,989 42 Gender disaggregation not available for Phase 1 EFSP Final Project Performance Evaluation Page 77 Objective 2 - Reduce Negative Coping Strategies IR2.1 Deliver Cash Transfers to Vulnerable HHs Phase 1 + Phase 2 0 22,448 22,367 99.6% Phase 1 0 13,640 13,584 99.6% Phase 2 0 8,860 8,783 99.1% Strengthen Child Nutrition % of HHs participating in Health / Nutrition Training Phase 2 (January 2017 to December 2017) 0% 100% 91.47% 91% Build Resilience at HH Level % of HHs participating in Home Economics/Decision Making Training Phase 2 (January 2017 to December 2017) 0% 100% 98.3% 98% # of HHs that form VSLAs (both recipient and non-recipient combined) Phase 2 (January 2017 to December 2017) 0% 4,500 4,170 92.7% Number of VSLAs formed District Phase 2 (January 2017 to December 2017) 0 150 150 100% Bombali No target set 64 Tonkolili No target set 86 EFSP Final Project Performance Evaluation Page 78 6.Conclusion and Recommendations Conclusion As is evidenced throughout this report, the ACDI/VOCA cash transfer program has been successfully implemented to meet its objectives. The UCTs had a positive impact on the majority of beneficiaries and supported affected households to recover, obtain food security and become more resilient to future shocks. The systems developed throughout project implementation, coupled with the evidence of short and medium term positive impacts make a case for future consideration of cash￾based emergency responses to food insecurity in Sierra Leone. In the context of the EVD crisis, the cash transfer intervention was appropriate and essential to addressing the food security and livelihood needs of affected households. This is especially true in the case of Phase 1 of the program when many households lost their basic livelihood resources and were unable to recover and stabilize themselves to pre-Ebola levels once the State of Emergency was lifted. The economic relief provided by the CTP was critical to supporting improvements in food security and longer-term recovery during this post-emergency phase. The programmatic shift to focus more on recovery was relevant in regard to the timing of Phase 2 implementation. By the time this phase began, and certainly by its end, Sierra Leone was no longer in an emergency. In this way, beneficiaries were able to spend more of the UCT on income generating activities, rather than food alone, to support their recovery and future resilience. Across both program phases, the cash transfer improved household food security and reduced hunger in the short and medium term. Quantitative data revealed a marked decrease in the proportion of households reporting moderate to severe hunger and an increase in the HDDS for all program beneficiaries. Qualitative research reinforced the causal impact of these outcomes as beneficiaries reported being able to buy more food, increase their food intake, consume a wider variety of food items and reduce their reliance on negative coping strategies because of the UCT. Beneficiaries also reported prioritizing the cash transfer for expenditures on education and investments in income generating activities with a view to plan for their future. The targeting approach employed by the program worked well. Following the post-emergency, poverty-focused approach of Phase 1, the targeting was successfully improved in Phase 2 to focus on food security indicators of heavily impacted households. While this helped to clarify the targeting criteria and reduce inclusion errors, there were noted exclusion errors related to set quotas and lower coverage rates. The cash transfer delivery mechanisms performed well across both phases, with significant improvements in data collection and monitoring being introduced in Phase 2. While initially proposing to use e-transfers, ACDI/VOCA was careful to consider the realities on the ground (e.g. poor roads, limited mobile phone service coverage, etc.) and engaged local service providers to provide a direct cash delivery mechanism which was appropriate for the operating context in the project’s target areas. While e-transfers would have provided an ideal solution to efficiently deliver the cash at scale, ACDI/VOCA effectively worked with the service providers to minimize any negative impacts on the delivery of the cash to beneficiaries. EFSP Final Project Performance Evaluation Page 79 Beneficiaries praised the process of cash distribution for its speed and efficiency. Despite challenges related to poor roads and liquidity of service providers, ACDI/VOCA ensured that payments were made on time, and beneficiaries reported being treated courteously by program staff. ACDI/VOCA also established well-functioning feedback mechanisms, community engagement and accountability measures throughout the delivery of the cash transfers. The incorporation of complementary activities for participating households significantly improved the effectiveness of the program. Households that used the economic and nutrition training messages to make spending decisions were found to have significantly improved their household food security across a range of measures. Beneficiaries linked to VSLAs also demonstrated improved food security and higher potential for sustainable recovery through their increased access to savings and credit, reducing their reliance on debt and moneylenders. The introduction of the VSLAs was highly valued by beneficiaries as it provided them with an opportunity to save beyond the life of the program and established a longer-term, local safety net. In general, the evaluation found that the effects of the cash transfer on women were largely positive. Targeting the women for cash transfers did empower them to fulfill their role as caregiver more effectively; however it did not necessarily translate into changed or improved gender relations, roles or perceptions. For the most part, intra-household relations reportedly improved during program implementation as household food needs were being met. Anecdotal evidence suggests that polygamous households were more likely to experience intra-household conflict if only one wife received the cash. At the community level, the possibility for conflict may have been lessened by the beneficiaries regularly sharing the benefits of the cash transfer with non-recipient extended family and neighbors. Strong coordination and collaboration with government officials, NGOs, donors and the private sector was a major strength of the CTP. The establishment of the CTWG was integral to providing a platform for shared learning and coordination to avoid duplication and overlap of efforts. In the context of limited experience with humanitarian cash transfers in the country, the strong coordination was integral to establishing harmonization of critical elements such as targeting, monitoring and transfer amounts to ensure consistency of approach across development partners. The close coordination with the GoSL also ensured cohesion between ACDI/VOCA’s activities and the national SSN program framework – effectively avoiding the development of parallel systems. Harmonization of critical elements also ensured ACDI/VOCA’s CTP was properly aligned with the national Social Safety Net program framework. Lastly, the direct leadership and support provided by the donor assisted the effective delivery of the CTP from the outset by providing adaptive management, technical support and oversight. Overall, the impacts of the cash transfer were positive, and in the case of Phase 1 beneficiaries, shown to be sustained one year after the last cash transfer. Recipients were generally found to use the cash wisely and in a manner that helped their families recover from the shock of EVD. Measures of resiliency for beneficiary households show that the CTP enabled a number of protective effects which have supported beneficiaries to become more resilient to future shocks. Households have invested in livestock, engaged in non-farm IGAs, joined VSLAs, are reportedly taking less debt and are maintaining greater food stocks because of their receipt of the cash transfer. In the case of Phase 1 beneficiaries, many of these activities are still ongoing a year later. EFSP Final Project Performance Evaluation Page 80 Recommendations ● Ensure context-appropriate cash transfer delivery mechanisms. The use of e-transfers was initially planned, and would have provided an ideal solution to efficiently deliver the cash at scale in a manner that would greater promote financial inclusion for beneficiaries. However, given the limited capacity of the mobile money networks in Sierra Leone, this was not a feasible option at the time. While the short-term nature of the program never justified a longer-term solution to cash delivery through other means, future long term CTPs can gain from the experience of the EFSP to look at other solutions that can be brought to scale and potentially bring participating households into the formal banking system. Since the start of the program, a number of service providers have built their capacity in delivering e-transfers (such as Orange and Africell), warranting further engagement. Stakeholders engaged in CTP need to leverage their aggregate demand to make the business case for financial service providers to expand their services into CTPs to remote areas in the country. ● Cash transfers should be considered as a suitable response to future emergencies in Sierra Leone. A clear benefit of cash transfers is the flexibility it provides. The cash transfer provided an opportunity to households that could not be achieved through non-cash assistance. It enabled them to choose the best options for their own households and capacities, so that the investments they made in their futures suited their own needs and abilities. Humanitarian actors with sector-specific mandates should seek to build on the success of the EVD response cash transfer intervention by assessing its suitability in addressing future emergencies. As with any intervention, it will be important for humanitarian actors to consider the appropriateness of CTPs on a case-by-case basis by examining factors such as the needs of their target group, market functionality as well as an assessment of governance and security considerations. ● Use of appropriate targeting criteria aligned with project objectives is essential for cash-based emergency responses to food insecurity. The targeting approach used in Phase 1 was based on a standard poverty means test which did not directly address the program objective of Ebola recovery. Standard criteria more appropriate for long-term poverty reduction needs to be considered alongside suitable indicators related to the impact of the emergency and food security. These are more directly linked to the shorter-term recovery objectives of cash-based emergency responses to food insecurity and have the potential to better limit inclusion and exclusion errors. Furthermore, while a quota system may be necessary to manage limited resources, future programs should review village quotas on a case-by-case basis to minimize the potential for exclusion errors. For instance, in the case where a village quota excludes five households (out of a potential 40), consider including all households as beneficiaries. This will not only serve to minimize exclusion errors, but will also reduce the potential for inter￾household conflict while encouraging community buy-in and support for the program. ● More effective ways of handling polygamous households in future cash transfer programs need to be explored. Polygamous households were more likely to experience intra-household conflict if only one wife received the cash. As beneficiary targeting focused on individual household economies and not marital relationships (i.e. each ‘pot’ was treated as a separate household as each wife is traditionally responsible for her own children), splitting the cash was not always a feasible solution. While the ACDI/VOCA approach of focusing on household economics served to reduce inclusion and exclusion errors within polygamous households, there EFSP Final Project Performance Evaluation Page 81 is a need to consider how future CTPs will address the potential for intra-household conflict among polygamous households. As conflicts are traditionally resolved by local authorities such as Town Chiefs (whose default resolution was to split the money with other wives), future programs should provide local authorities with training on the targeting requirements and the purpose of the cash transfers. This will help those tasked with resolution of conflict to respond in ways that promote the goals of the program. ● Complementary activities should be implemented alongside future cash transfer programs to promote recovery and resilience. The inclusion of training activities and increased access to VSLAs enabled beneficiaries to use the cash better, generating an impact beyond the duration of the project. Regardless of the timeframe of the response, an effort should be made to provide high-impact complementary activities, such as training on household economics and financial management during program orientation. However, as evidenced by the evaluation, participation in training activities can be impacted by the timing and rollout of activities. Future CTPs should carefully plan the phasing in of complementary activities according to the needs of beneficiaries based on the cash transfer schedule and disbursements. For instance, if cash transfers are being made at the start of the planting season, training on key crop production practices and the prioritization of funds to maximize impact should be provided from the outset (rather than later in the disbursement schedule). Providing essential training during an orientation session is also most likely to result in high attendance as the expectation of receipt of cash and interest in the program will be high. ● The use of digital data gathering technology should be integrated in future CTPs to improve accuracy, efficiency and monitoring. In Phase 2, ACDI/VOCA effectively leveraged digital technology to register beneficiaries, track payments, confirm beneficiary identity and improve monitoring systems. This overcame challenges with SPLASH’s data capacity, improving accountability and allowing for the integration of multiple service providers. Technological solutions should be sought at the outset of any CTP for implementers to have direct control over their data collection. ● Ensure robust coordination mechanisms are built from the outset of any future cash-based emergency response: The close coordination with the GoSL during the emergency phase and the ongoing coordination through the CTWG were critical to the success of the CTP. The harmonization of critical elements such as targeting, monitoring and transfer amounts enhanced the program's effectiveness and communication, and safeguarded against the development of parallel systems. This level of coordination is essential for any future cash-based emergency responses to food insecurity to improve coordination and accountability. ● There is a strong case for follow-up development activities in order to ensure the sustainability of cash-based emergency responses to food security. The impact findings indicate that continued follow-up complimentary activities would enable ACDI/VOCA to increase the potential for sustainable impact and reduce the need for future social protection among their beneficiaries. Follow-up development activities could capitalize on the outcomes of the CTP by offering complimentary services aligned to the livelihood choices that beneficiaries have made (e.g. training on improved agricultural practices, business skills, SME formation, etc.). EFSP Final Project Performance Evaluation Page 82 ACDI/VOCA has an established presence in these communities, and would have a readily identifiable and eager audience for follow-up learning. Providing such follow-up development activities would be directly aligned with the original objectives of the CTP by continuing to build the resilience of these households against future shocks. ● The monitoring and evaluation design of future cash transfer programs should incorporate a representative comparison group and additional behavioral measures of food security to analyze impact. The addition of a control group would allow for a more thorough evaluation of the effectiveness of program activities and provide robust data on the causal relationship between the intervention and outcomes. Furthermore, beyond the main impact indicators of HHS and HDDS to measure food insecurity, assessing coping strategies should be added to future CTP monitoring and evaluation to better understand the behavioral response to food insecurity. This will support better program design as data on behavioral coping strategies goes beyond the ‘access’ dimension of food insecurity and is immediately useful for programmatic decision making, monitoring and evaluation.