Final Performance Evaluation of the Kore Lavi DFAP in Haiti Final Performance Evaluation of the Kore Lavi Development Food Assistance Project in Haiti April 2020 IMPEL | Implementer-Led Evaluation & Learning Associate Award ABOUT IMPEL The Implementer-Led Evaluation & Learning Associate Award works to improve the design and implementation of Food for Peace (FFP) funded development food security activities (DFSAs) through implementer-Led evaluations and knowledge sharing. Funded by the USAID Office of Food for Peace (FFP), the Implementer-Led Evaluation & Learning Associate Award will gather information and knowledge in order to measure performance of DFSAs, strengthen accountability, and improve guidance and policy. This information will help the food security community of practice and USAID to design projects and modify existing projects in ways that bolster performance, efficiency and effectiveness. The Implementer-Led Evaluation & Learning Associate Award is a two-year activity (2019-2021) implemented by Save the Children (lead), TANGO International, and Tulane University, in Haiti, the Democratic Republic of Congo, Madagascar, Malawi, Nepal, and Zimbabwe. RECOMMENDED CITATION IMPEL. (2020). Final Performance Evaluation of the Kore Lavi Development Food Assistance Project in Haiti. Washington, DC: The Implementer-Led Evaluation & Learning Associate Award COVER PHOTO CREDITS Fruit and Frescoes: A selection of goods sits in stalls at the at the inauguration of a USAID-funded market. In the background, rich frescoes painted by local artists flank the market's walkways. 5/27/2011, Port-au-Prince, Haiti. Photo by Ben Edwards/USAID. DISCLAIMER This report is made possible by the generous support of the American people through the United States Agency for International Development (USAID). The contents are the responsibility of the Implementer-Led Evaluation & Learning (IMPEL) award and do not necessarily reflect the views of USAID or the United States Government. CONTACT INFORMATION IDEAL Activity c/o Save the Children 899 North Capitol Street NE, Suite #900 Washington, DC 20002 www.fsnnetwork.org info@fsnnetwork.org PREPARED BY: Final Performance Evaluation of the Kore Lavi DFAP in Haiti Table of Contents i TABLE OF CONTENTS Figures ................................................................................................................................. iii Acronyms.............................................................................................................................. v Executive Summary .............................................................................................................viii 1. Introduction .................................................................................................................... 1 1.1 Country Setting...............................................................................................................................1 1.2 Project Overview............................................................................................................................4 1.3 Project Goals, Objectives, Results Framework, and Theory of Change .........................................4 1.4 Description of Activities and Implementing Partners....................................................................5 SO1 Activities: National systems for vulnerability targeting strengthened...................................7 SO2: Access to local nutritious food...............................................................................................8 SO3: Maternal and child nutritional status improved....................................................................9 SO4: Haitian institutions’ capacity to lead and manage safety net programming.......................10 Gender..........................................................................................................................................11 2. Overview of Evaluation.................................................................................................. 12 2.1 Evaluation Purpose and Intended Uses .......................................................................................12 2.2 Evaluation Approach ....................................................................................................................12 2.3 Evaluation Questions ...................................................................................................................13 3. Design and Methodology ............................................................................................... 15 3.1 Desk Review ................................................................................................................................. 15 3.2 Consultation with Implementing Partners and USAID................................................................. 15 3.3 Qualitative Design ........................................................................................................................16 3.4 Identification of Key Informants ..................................................................................................17 3.5 Selection of Focus Group Sites.....................................................................................................19 3.6 Identification of Focus Group Participants...................................................................................20 3.7 Ethics............................................................................................................................................22 3.8 Evaluation Training and Fieldwork...............................................................................................22 3.9 Data Analysis................................................................................................................................23 3.10 Data Limitations...........................................................................................................................25 4. Findings......................................................................................................................... 26 4.1 EQ1: To what extent has the project met its defined goals, purposes and outcomes? ..............26 EQ1.1: Project performance on indicators measured against targets set by the partners and FFP indicators......................................................................................................................................26 EQ1.2: Factors that promoted or inhibited the achievement of the project objectives. ............31 EQ1.3: Plausibility of pathways and the determinants of achieving the key outcomes..............35 EQ1.4: Targeting strategies and their contribution to achieving project goals...........................39 EQ1.5: The appropriateness and effectiveness of interventions for the poorest individuals and households...................................................................................................................................43 4.2 EQ2: Based on the evidence, which project outcomes are likely to be sustained?.....................44 IMPEL | Implementer-Led Evaluation and Learning ii Table of Contents EQ2.1: The quality of the processes, systems, and institutional arrangements..........................44 EQ2.2: Communities’ perceptions on the quality, frequency, effectiveness, and sustainability of the services provided by the project............................................................................................47 EQ2.3: The likelihood that service providers will continue providing services after the project ends. .............................................................................................................................................49 EQ2.4: The motivation of the community and beneficiaries to demand and pay (or invest time) for the services.............................................................................................................................50 EQ2.5: Whether the necessary resources and capacity strengthening will exist to sustain service providers. .....................................................................................................................................51 EQ2.6: The extent to which the projects leveraged other USG and non-USG investments to achieve sustained outcomes as identified in the theories of change..........................................52 EQ2.7: Evidence of enhanced linkages with other service providers. .........................................53 4.3 EQ3: What are the strengths and challenges of the selected interventions and their implementation, and how are these received by the target communities?........................................54 EQ3.1: Factors in the implementation and context associated with greater or lesser effectiveness in producing outputs of higher or lower quality....................................................55 EQ3.2: Interventions and implementation processes deemed more/less acceptable to members of the target communities............................................................................................................56 4.4 EQ4: What are the key lessons learned and best practices that should inform future projects in the country? .........................................................................................................................................57 EQ4.1: The unintended positive and/or negative consequences of the projects........................57 EQ4.2: Ways to minimize potential unintended negative consequences and systematically capture positive consequences....................................................................................................59 4.5 EQ Cross Cutting...........................................................................................................................61 EQ Cross Cutting: Gender Integration across the project............................................................61 5. Conclusions and Recommendations............................................................................... 64 Annex 1: Evaluation Scope of Work ..................................................................................... 72 Annex 2: Questions of USAID Interest for Haiti Final Evaluation ........................................... 81 Annex 3: List of Key Informants ........................................................................................... 83 Annex 4: Site selection (communes) for data collection........................................................ 84 Annex 5: Discussion Guides (English) ................................................................................... 85 Annex 6: KII and FGD Tools (French/Creole)....................................................................... 103 Annex 7: Project Performance Summary Tables................................................................. 128 Annex 8: Qualitative Coding .............................................................................................. 133 Final Performance Evaluation of the Kore Lavi DFAP in Haiti Table of Contents iii FIGURES Figure 1: Integrated Phase Classification, Acute Food Insecurity Maps, 2017-Present ............................... 2 Figure 2: Allocation of Responsibility for SOs and IRS among Kore Lavi Consortium Partners (first four years, pre-extension) .................................................................................................................................... 5 Figure 3: Map of Project Intervention Zones (2013) .................................................................................... 7 Figure 4: Kore Lavi fresh food voucher......................................................................................................... 9 Figure 5: Key Informant Interview, Boucan Carre, Center Department.....................................................18 Figure 6: Key Informant Interview Sample ................................................................................................. 19 Figure 7: Assembling a focus group, Boucan Carre, Center Department...................................................21 Figure 8: Focus Group Discussion Interview Sample ..................................................................................21 Figure 9: Summary IPTT Reporting 2014 – 2017.........................................................................................28 Figure 10: Summary IPTT Reporting 2018 – 2019 ......................................................................................30 Figure 11: Qualitative data topic coding summary on Achievements (EQ1.2)...........................................32 Figure 12: Changes in IPTT Capacity Building Reporting 2014 – 2019........................................................33 Figure 13: Qualitative data topic coding summary on Kore Lavi’s contributions to reducing food insecurity and vulnerability (EQ1.3)............................................................................................................36 Figure 14: Qualitative data topic coding summary on pathways/determinants of achieving key outcomes (EQ1.3) ........................................................................................................................................................36 Figure 15: Changes in IPTT safety net graduation reporting 2014 – 2019 .................................................37 Figure 16: Qualitative data topic coding summary on Kore Lavi’s contributions to establishing a safety net and expanding government capacity (EQ1.3) ......................................................................................38 Figure 17: MAST Global Institutional Capacity Index..................................................................................39 Figure 18: Qualitative data topic coding summary on targeting strategies (EQ1.4) ..................................41 Figure 19: Changes in IPTT targeting Reporting 2014 – 2019.....................................................................42 Figure 20: Qualitative data coding summary on intervention appropriateness, effectiveness (EQ1.5) ....44 Figure 21 MFI office in Anse-a-Galets, La Gonave, West Department.......................................................45 Figure 22: Qualitative data topic coding summary on quality of process, systems, arrangements (EQ2.1) ....................................................................................................................................................................47 Figure 23: IPTT complaint response rate 2014 – 2017 ...............................................................................47 Figure 24: Qualitative data topic coding summary on community perceptions of project services (EQ2.2) ....................................................................................................................................................................48 Figure 25: IPTT VSLA group participation and establishment 2014-2017, beneficiary perceptions and complaint response rate 2018-2019...........................................................................................................49 Figure 26: Qualitative data topic coding summary on services after project end (EQ2.3).........................50 Figure 27: Qualitative data topic coding summary on communities motivated to demand and invest in services (EQ2.4)...........................................................................................................................................51 Figure 28: IPTT VSLA members starting income generating activities 2018-2019 .....................................51 Figure 29: Qualitative data topic coding summary on Leverage of other investments (EQ2.6) ................53 Figure 30: Qualitative data topic coding summary on linkages with other service providers (EQ2.7) ......53 Figure 31: IPTT complementary services 2014-2017, linkages and community structures 2018-2019.....54 Figure 32: Qualitative data topic coding summary on context and factors that impacted outputs (EQ3.1) ....................................................................................................................................................................55 IMPEL | Implementer-Led Evaluation and Learning iv Table of Contents Figure 33: Qualitative data topic coding summary on intervention acceptability to communities (EQ3.2) .................................................................................................................................................................... 56 Figure 34: IPTT voucher redemption and number of vendors, 2014-2017, 2018-2019.............................56 Figure 35: Qualitative data topic coding summary on unintended positive/negative consequences (EQ4.1) ........................................................................................................................................................59 Figure 36: Qualitative topic coding summary on ways to minimize negative consequences, capture positive (EQ4.2)...........................................................................................................................................60 Figure 37: Gender specific IPTT indicators..................................................................................................62 Final Performance Evaluation of the Kore Lavi DFAP in Haiti Acronyms v ACRONYMS ACF Action Contre La Faim International BFS Bureau for Food Security BMI Body mass index CADEP Participatory Development Support Council (Conseil d’Appui au Développements Participatif) CASEC Collective Territorial Administrative Councils (Conseils d’Administrations des Collectivités Territoriales) CBO Community Based Organization CDF Community Development Fund CPI Consumer price index CASE Centre d’appui au suivi et à l’evaluation CNSA National Coordination for Food Security (Coordination Nationale de l Securite Alimentaire) COP Chief of Party DAI Development Alternatives, Inc. DDL USAID’s Development Data Library DFAP Development Food Assistance Project DFSA Development Food Security Activity DHS Demographic and Health Survey ECVMAS Enquête sur les conditions de vie des ménages après le séisme EU European Union FANTA Food and Nutrition Technical Assistance III Project FAO U.N. Food and Agriculture Organization FCS Food consumption score FEWS NET Famine Early Warning System Network FFP USAID’s Office of Food for Peace FGD Focus Group Fiscussion FY Fiscal Year GBV Gender Based Violence GOH Government of Haiti GPS Global positioning system GSIS Groupe de Support en Informatique et en Statistiques HAZ Height-for-age Z-score HDDS Household dietary diversity score HHS Household hunger scale HDVI Haitian Deprivation and Vulnerability Index HTG Haitian Gourde IMPEL | Implementer-Led Evaluation and Learning vi Acronyms ICF ICF International IFPRI International Food and Policy Research Institute IHE Institut Haïtien de l’Enfance IHSI L’Institut Haïtien de Statistique et d‘Informatique IR Intermediate Results IPTT Indicator Performance Tracking Table KII Key Informant Interview IYCF Infant young child feeding LCU Local currency unit LSMS Living Standards Measurement Study LOA Life of Activity MAD Minimum acceptable diet MAST Ministère des Affaires Sociales et du Travail MCFDF Ministry for Women’s Affairs and Women’s Rights (Ministère à la Condition Féminine et aux Droits des Femmes) MCHN Maternal & Child Health & Nutrition M&E Monitoring and Evaluation MOU Memorandum of Understanding MSPP Ministère de la Santé Publique and de la Population NGO Non-governmental organization ODAV Vulnerability Analysis and Mapping Branch ORT Oral rehydration therapy OLS Ordinary least squares PGI Poverty gap index PPP Purchasing power parity SDE Sections d’Énumération SD Standard deviations SBCC Social and Behavioral Change Communications SIMAST Systeme Informatique du Ministère des Affaires Sociales et du Travail SO Strategic Objective SPSS Statistical Package for the Social Sciences UNDP United Nations Development Agency USAID U.S. Agency for International Development USD United States dollar USG United States Government VSLA Village Savings and Loans Association WASH Water, sanitation and hygiene WDDS Women’s dietary diversity score Final Performance Evaluation of the Kore Lavi DFAP in Haiti Acronyms vii WB World Bank WFP U.N. World Food Programme WHO World Health Organization WVI World Vision International IMPEL | Implementer-Led Evaluation and Learning viii Executive Summary EXECUTIVE SUMMARY Background In FY 2013, the US Agency for International Development’s (USAID) Office of Food for Peace (FFP) issued an award to a consortium of three organizations, CARE, Action Contre La Faim (ACF), and the United Nations World Food Programme (WFP) to implement a Development Food Assistance Project (DFAP). World Vision (WV) later joined the consortium. The four-year project, titled ‘Kore Lavi’ (‘supporting life’ in Haitian Creole) started in August 2013. Following a two-year extension granted in 2017, the projected ended in September 2019. The overall purpose of Kore Lavi was to support the Haitian Government in creating a social safety net for food and nutrition security that prioritizes consumption of locally grown quality products. The overall goal was to contribute to reducing food insecurity and vulnerability in targeted communities by establishing a replicable safety net system and expanding government capacities to prevent child under nutrition. Kore Lavi covered five departments (Upper Artibonite, Central Plateau, Northwest, Southeast and West￾La Gonave Island) and 23 communes, targeting over 18,000 households under the SO2 component, and more than 205,000 pregnant and lactating women and children under two years old. Kore Lavi directly supported the Ministry of Social Affairs & Labor (MAST) to deliver social services to vulnerable families. It had two safety net components: distribution of food vouchers to targeted families, and a Mother and Child Nutrition (MCHN) component, which includes the distribution of nutrition supplements to PLW and provision of counseling services on health and food safety. The nutrition components of Kore Lavi were phased out in October 2017. The project budget was approximately USD $80 million for the first four years and USD $24 million for the two years of extension. At the end of its implementation, the project was expected to achieve the following key results: • National systems for vulnerability targeting strengthened. • Access of extremely vulnerable households to local and nutritious food increased. • Maternal and child nutritional status improved. • Haitian institutions’ capacity to effectively lead and manage safety net programming improved. The activities of KORE LAVI fall under four strategic objectives (SOs), with activities for promoting gender equality integrated into each component of the project design. Under SO1, Kore Lavi developed, tested, and adjusted a national Haitian Deprivation and Vulnerability Index (HDVI), which allows the project to measure vulnerability in a multidimensional way using proxy means calculated through a customized algorithm. Kore Lavi defined a specific data collection methodology implemented by other organizations, collecting household data from all households within a targeted commune. Kore Lavi developed and implemented a database application hosted by the Ministry of Social Affairs and Labor (MAST), which is a tool to store, analyze and manage collected data. The main function of this system, SIMAST (Information System of MAST), is to answer key questions about the structure and distribution of household deprivation or vulnerability so as to allow decision makers to develop an appropriate frame of social assistance for future safety net interventions. Final Performance Evaluation of the Kore Lavi DFAP in Haiti Executive Summary ix Under SO2, Kore Lavi developed voucher transfer modalities prioritizing the use of local foods so as to deliver food vouchers to the poorest and most vulnerable 10% of households in each targeted commune. To promote good nutritional practices through a balanced diet by fresh food, and at the same time to make a direct contribution to local production, Kore Lavi used two types of vouchers: paper vouchers, which are allocated for the purchase of local fresh foods (fruits, vegetables, fresh meat, fish, eggs, etc.) accepted by select vendors in local community markets, and electronic vouchers allocated for the purchase of local staple foods (local rice, maize, sorghum, beans, flour, and oil) from small community shops selected by the project. Also under SO2, Kore Lavi promoted and facilitated participation in village savings and loan associations (VSLAs). A VSLA is a group of people who save together and take small loans from those savings. The activities of the group run in cycles of one year, after which the accumulated savings and the loan profits are distributed back to members. Kore Lavi attempted to integrate as many project safety net beneficiaries as possible into VSLA groups. Under SO3, Kore Lavi implemented a social behavior change communications (SBCC) strategy in line with the preventive approach focusing on the window of opportunity in the first 1,000 days of life. The project provided supplementary conditional rations to pregnant and lactating women and children aged 6-23 months with a goal to reach 76,567 women and 96,597 children during the life of the project. The SBCC strategy was mainly implemented through a Care Group approach. In the context of Kore Lavi, a Care Group is a group of 10 women named “Lead Mothers” (or in limited cases “Lead Fathers”) who are identified by the community and come together for mutual support and learning and who are regularly trained by the local community health agents (CHAs, also known as Agentes de Sante Communautaire Polyvant, ASCPs). Each Lead Mother had responsibility for approximately 10 households and conducts visits to each household on monthly bases to share and promote key infant and young children feeding (IYCF2) practices and other appropriate health and nutrition information. In order to reinforce the knowledge of the community members regarding proper health and nutrition behavior and practices, Kore Lavi collaborated with the existing network of CHAs belonging to the Ministry of Health (Ministère de la Santé Publique et de la Population, or MSPP), while sensitizing through various community public awareness raising campaigns – such as media campaigns, nutrition fairs, special days and events, etc. – key community structures, civil society organizations and traditional health workers. CHAs also played a central role in the project’s moderate and acute malnutrition intervention. A network of nurse supervisors was responsible for training and overseeing the work of CHAs. CHAs in turn are responsible for approximately10 to 20 Lead Mothers. Moreover, to reinforce health and nutrition services provided by the health facilities, the project conducted trainings for health professionals and primary health care workers. Specifically, the project supported and facilitates active and passive screening of children through its network of CHAs so as to identify new cases of MAM and refer them to healthcare facilities to receive nutritional treatment. SO3 was terminated as of October, 2017. CARE and WV continued some of the nutrition awareness under SO2 and cross-cutting gender activities. Under SO4, Kore Lavi sought to enhance the leadership capacity of MAST and other entities of the safety net programming. This was done by the following activities: (1) implement a joint USAID/MAST capacity building and resource mobilization plan; (2) work with MAST national and department-level staff on coordinating project safety net activities; (3) develop MAST safety net reference documentation (policies, governance, coordination, implementation); (4) support Participatory Development Support Councils (Conseil d’Appui au Développements Participatif, or CADEPs) and other civil society organizations to develop capacity building plans and carry out social audit processes; (5) implement IMPEL | Implementer-Led Evaluation and Learning x Executive Summary gender-sensitive annual social audits with local stakeholders; and (6) provide training to community structures (local authorities, civil society organizations) on social accountability focusing on the application of the Community Score Card (CSC), which is a tool to enhance community influence on the quality, effectiveness and accountability of the services offered by Kore Lavi at community level. Methodology In August 2019, a final qualitative performance evaluation of Kore Lavi was conducted by Tulane University, under the umbrella of the Implementer-led Evaluation and Learning award (IMPEL). The objective of this evaluation is to assess the development outcomes of the Kore Lavi project and implications for sustainability. The evaluation was guided by five main evaluation questions, each with specific sub-questions, as well queries related to gender integration and other cross-cutting and interests. The ET was made up of four members from Tulane University: two in country, and two additional members providing report support. Team members had expertise in qualitative and quantitative research design, project evaluation, food security, gender, nutrition, livelihoods, and voucher-based programs. Tulane partnered with the local research company IFOS, who provided a six-member qualitative field data collection team with backgrounds in qualitative research and project evaluation, and fluent in French and Haitian Creole, as well as basic English skills. The evaluation employed a mixed of qualitative methods, which was complemented by a document review and secondary quantitative data from project monitoring and other surveys. Fieldwork began with initial meetings and briefings in Port-au-Prince, followed by field data collection that included interviews with 63 key informants/in-depth interviews, and 26 focus group discussions with a total of 257 participants. The evaluation concluded with a validation workshop to present and discuss initial evaluation results and a debriefing with the USAID mission. Key Findings, Conclusions, and Recommendations Since the official start of Kore Lavi in August 2013, the project has achieved success in achieving most of its strategic objectives, and the activities were well received by stakeholders and participants. Some key areas where Kore Lavi has been particularly successful in achieving its goals and in providing a model for future social protection activities in Haiti include: • The inclusion of local micro-finance institutions, food vendors, and use of locally produced foods as part of the food voucher system appears to have provided an important multiplication of inputs and impacts. The use of local MFIs in the voucher reimbursement and vendor monitoring was very successful. The MFIs gained more visibility and business with local vendors in need of revolving credit, and at better interest rates that are often otherwise available. The local vendors were viewed not only as service providers in the project, but also as beneficiaries. The vendors, particularly the fresh food vendors, while not in extreme poverty, are still often quote poor. Participation in the project appears to have allowed these vendors not only to gain more business, but also benefit from training in hygiene and basic accounting practices. The restriction of the applicability of the vouchers to local products ensured the infusion of money into the economy would not immediately leak out of the community or country. Final Performance Evaluation of the Kore Lavi DFAP in Haiti Executive Summary xi • The SIMAST has been a very successful component of Kore Lavi. The census approach to targeting used by Kore Lavi was considered by some as difficult to impossible to implement in Haiti, until it was successfully implemented by Kore Lavi. The SIMAST is impressively projected to have collected information on 30% of Haiti’s population by 2020. The SIMAST has opened the eyes of many stakeholders about the significant possibilities that actually exist in this regard. As of Q3 2019, SIMAST is being used in three other projects outside of Kore Lavi. Additionally, concrete plans are in place to fund and use the SIMAST in other projects, such as European Union (EU) funded 11th European Development Fund (11th EDF, or 11eme FED), as well as interested expressed by the Swiss Cooperation and the World Bank. • The focus on capacity building at MAST was strongly appreciated by the GOH staff, and viewed as a vital support to helping them make progress towards the government’s social protection goals. • The addition of VSLAs as a complementary component of the project was very successful in Kore Lavi. It appears that VSLAs were most successful when combined with other activities, such as the voucher distribution. The VSLAs are considered a very sustainable part of Kore Lavi that will continue to have an impact long after the life of the project for many households. • The drafting of the National Policy for Social Protection (PNSP) was a positive consequence of Kore Lavi, despite the fact that it was not originally envisioned at the start of the project. While in theory, it would be ideal to have the policy in place before the piloting of a social protection program, it was in fact because of the Kore Lavi activities that the various social protection stakeholders were more receptive to the drafting of the policy. Kore Lavi in effect was a necessary precursor to set the stage for work on the social protection policy document. Furthermore, this policy discussion and drafting was consistently viewed as a high-quality, inclusive process. • Learning from the initial years of the project, and recognizing the needs to develop a plan and metric to monitor project impacts on governance and instutionalization, Kore Lavi developed and included the global institutional capacity index to monitor the capacity of MAST. The evaluation also found several areas where the project has an opportunity to improve and learn, informing future social protection projects in Haiti and globally. These can be loosely grouped in four categories: • Project conception and design • Use of adaptive management at all stages of the project cycle • Handover of the project to the GOH • Monitoring and management Project Conception and Design Although very successful in demonstrating the possibilities of what social protection in Haiti could be, Kore Lavi was lacking in some areas from the conception of the project. To promote sustainability of impacts, social protection projects like Kore Lavi should include livelihood/agriculture support components, or partner closely with complementary projects. The Kore Lavi project design lacked such a component, which likely limited the impacts the project was able to have on long-term resilience. The assumption that households would use the voucher support coupled with the VSLAs to help lift themselves out of poverty (or at least put the household one run further up IMPEL | Implementer-Led Evaluation and Learning xii Executive Summary the economic ladder) relied on the presumption that these households would find and develop new/improved livelihood sources, without actually formally supporting that important pathway step. Kore Lavi failed to adequately incorporate graduation out of the voucher program into the project design, and never clearly defined what ‘graduation’ entails. If graduation is not envisioned as a likely or common outcome of the project, then this should be clearly spelled out in the project design. Kore Lavi monitoring included the indicator “number of safety net beneficiary households that graduated from the safety net program” throughout the life of the project, and yet reported zero or nearly zero graduations. The inclusion of this indicator was ill-conceived by both FFP and the consortium, due to the lack of a clear pathway to graduation, a clear definition of ‘graduation’, or a discussion on if graduation was an appropriate goal for the project at all. It may be in the Haiti context that focusing on graduation to any large degree ignores the chronic nature of poverty among the poorest in Haiti. It was also a lost opportunity to think about and better define what graduation means for Kore Lavi during the extension period. Kore Lavi also lacked any clear route to enroll new voucher recipients once the initial voucher recipient selection process was completed. This missing aspect of the project did not allow Kore Lavi to act as a true safety net. The targeting focused on the chronically poor at the time of the HDVI survey. Households in targeted communities that later suffered a shock such as a death of an income generating household member, or a family member developing a chronic illness, could not be added to the voucher recipient lists. This means that the food voucher program can better be described as long-term assistance for chronically poor, rather than as a safety net put in place to catch households and prevent them from falling (further) into poverty. In fact, one of the IPs identified this drawback of the Kore Lavi system, and rather than design a way to update the Kore Lavi rosters (which was described as ‘rigid’), they attempted to provide support to households that were struck by idiosyncratic shocks by accessing other programs they ran in the communities. Finding a feasible solution to this challenge is admittedly challenging. One approach may be to increase the regularity of data collection, updating the household information in SIMAST in order to refine/adapt the beneficiary list on a more regular basis. Such adaptability of the beneficiary lists will require more resources, not only for the collection of more real￾time targeting data, but also potentially if the number of those meeting the criteria to receive vouchers increases over time. Regional differences in prevalence of poverty should be accounted for in targeting. Kore Lavi made the decision to target the poorest 10% (approximately) of households in each commune where the voucher program was implemented. Ten percent was chosen because that was the national prevalence of extreme poverty. However, this approach discounted the difference in prevalence of extreme poverty between communes. This means that households considered less poor (as measured by the HDVI) may have been included in the voucher program in one commune, while poorer households may have been excluded in another. While there are difficulties in drawing an absolute threshold of the poverty metric for targeting, as the numbers of beneficiaries may exceed resources, the system of Kore Lavi did not attempt to address this. Monitoring and Management There must be close monitoring and consistent management of payments to the micro-finance institutions (MFIs) in future projects. The most frequently cited complaint in the project process quality Final Performance Evaluation of the Kore Lavi DFAP in Haiti Executive Summary xiii was timeliness of the payments to the MFIs (who then in turn reimbursed merchants for the fresh food vouchers they took as payment), leading to late payments and frustration among the participating vendors. This was likely aggravated in part by the shift in management going into the two year extension period There was a lack of solid evidence measuring impacts on vendors, such as increase in vendor income, job creation, improvements on vendors’ business practices, increase of the food and economic security of the vendor households, etc. Future projects should include this kind of data in the monitoring and evaluation design in order to better monitor, refine, and improve this potentially important and impactful facet of the activities. While the consortium model appeared to have more benefits that drawbacks, Kore Lavi did experience some struggle in maintaining a collaborative environment between consortium partners while still keeping strong and dynamic leadership. It is important, to ensure an appropriate combination of partners that will make up a consortium, and to carefully define their roles and responsibilities from the start, informed by well thought out project design, and their roles and responsibilities are well defined. In the case of Kore Lavi, some of the benefits of the consortium were not fully utilized. For example, CARE has extensive experience in social protection projects in other countries, such as Ethiopia. Yet there was very little opportunity provided by CARE to their Haiti office which would have allowed them to benefit from this. Additionally, some members of the consortium seemed to have slightly different priorities. Some were more concerned with ‘checking the boxes’ in terms of completing outputs, while others were more willing to sacrifice some degree of project performance progress in order to move forward only with the GOH and other partners in stride. Use of Adaptive Management FFP should continue to work closely with the IPs to include and encourage adaptive management over the life of a project, including at the RFA stage. There were several examples where the IPs effectively adapted their work in collaboration with FFP, such as the work on the national policy for social protection, or the adjustments of voucher value to adapt to inflation and food price increases. Although adaptive management is now standard for FFP projects, this only became the norm after the start of Kore Lavi, so it was not written specifically into the project design. This left the IPs without clear processes to adapt the project during its life. Adjustments to the project were more ‘ad hoc’ than systematic as a result. The ET found that there was some tendency to ‘check the boxes’ of project outputs (in some cases, not across the board), focusing more on compliance than strategic thinking. The lack of adaptive management may have led to the perception of some that the reduction of funding and subsequent decision to not continue the SO3 component of Kore Lavi after the mid-term evaluation was somewhat abrupt, and that there was not sufficient time to propose adaptations to that component of the project. Handover of the Project The overly ambitious objectives of the project related to institutionalization/handover to government, particularly coupled with the initial short time frame, set up the project to fall short by those standards. As noted in the mid-term evaluations, programs to build social safety nets in Ethiopia and Brazil have IMPEL | Implementer-Led Evaluation and Learning xiv Executive Summary taken many years of financial and technical assistance, despite having much more stable, strong governments that Haiti. The perceived expectations of what Kore Lavi was to achieve had the unintended consequence of setting up the project to be perceived as less of a success. The government hand-over in particular was unrealistic, and led to some sense of failure or shortcoming at MAST. The initial project design aimed at institutionalization of the social protection activities without considering the need for a national policy on social protection to ensure long-term viability of social protection projects. FFP needs to think well ahead of time on the role of/need for national policy when piloting prototypes of social protection programs, and identify which partnerships FFP and the IPs would be most strategic for this work. However, it should be noted that Kore Lavi did recognize this need, and added policy support/advocacy later in the life of the project. Another challenge to the handover of the project to the government was the fact that the HDVI indicator in Haiti was considered a ‘black box’ by all consortium and other stakeholder staff interviewed. The statistics used to design the HDVI are complicated, and difficult to understand for all partners. Furthermore, it appears that Kore Lavi does not have access to the datasets or complete set of analyses to create/recreate the HDVI. The statistical development was conducted by a consultant at the start of the project, and the complete dataset and syntax used to derive the HDVI algorithm were never made available to Kore Lavi. This means that not only can Kore Lavi not fully own and advocate for the HDVI as a targeting tool, it is impossible for MAST to take over ownership of the HDVI. Any future targeting metric should be developed transparently and in close collaboration with the GOH, and balance simplicity with sensitivity and specificity. Final Performance Evaluation of the Kore Lavi DFAP in Haiti Introduction 1 1. INTRODUCTION 1.1 Country Setting Haiti has historically suffered from very high poverty levels, as measured by socio-economic indicators such as levels of undernourishment, stunting, and underweight children, coupled with poor performance in gross domestic product. Haiti was ranked 169th out of 189 countries on the United Nations 2018 Human Development Index.1 The Global Hunger Index (GHI) for 2018 ranks Haiti 113 out of 119 countries, with ‘alarming’ levels of undernourishment, child wasting, child stunting and child mortality. 2 Chronic food insecurity is a significant challenge for Haiti. Almost one-third of Haiti’s population is considered food insecure.3 Moreover, the high incidence of climatic shocks, environmental hazards and the government’s limited capacity to mitigate risks or respond to disasters further impedes the country’s ability to improve socio-economic development. As a result, a majority of the Haitian population still faces significant challenges in all four pillars food security: availability, access, utilization and stability. It is important to underline the deteriorating food security situation in Haiti recently before and during the time of the evaluation. Looking at the integrated phase classifications (IPC) over the past three years, a worsening trend can be seen across the country. The most recent IPC4 , published in October 2019, indicates the cause of the deteriorating situation is due to rising food prices (annual inflation of about 23%), depreciation of the gourde against the dollar, the socio-political unrest and the worsening security situation. Additionally, a drought that lasted from 2018 into the first half of 2019 negatively impacted agriculture production in many parts of the country. This deteriorating situation must be taken into consideration when assessing the impacts of food and nutrition security interventions. 1 United Nations Development Programme (UNDP). (2018). Human Development Index. Available at http://hdr.undp.org/en/composite/HDI 2 International Food Policy Research Institute (IFPR). (2013). 2013 Global Hunger Index: Country case study: Haiti. Available at https://www.globalhungerindex.org/haiti.html. 3 United Nations World Food Programme (WFP). Haiti: Overview. Available at http://www.wfp.org/countries/haiti/overview. 4 http://www.ipcinfo.org/fileadmin/user_upload/ipcinfo/docs/IPC_Haiti_AcuteFoodSec_2019Oct2020Feb_English.pdf IMPEL | Implementer-Led Evaluation and Learning 2 Introduction Figure 1: Integrated Phase Classification, Acute Food Insecurity Maps, 2017-Present Figure 1a: June – September 2017 Figure 1b: October 2017 – February 2018 Final Performance Evaluation of the Kore Lavi DFAP in Haiti Introduction 3 Figure 1c: October 2018 – February 2019 Figure 1d: October 2019 – February 2020 Maps taken from various IPS reports, found at http://www.ipcinfo.org/ IMPEL | Implementer-Led Evaluation and Learning 4 Introduction 1.2 Project Overview Kore Lavi – “Support to the National Food Security and Nutrition Program” – is a FFP-funded DFAP implemented in Haiti by a consortium of partners comprised of CARE Haiti (as prime), Action Against Hunger, World Food Programme and World Vision. The project started in August 2013, and a two year extension was granted from October 2017 to September 2019. The purpose of Kore Lavi is to support the Haitian Government in creating a social safety net for food and nutrition security that prioritizes consumption of locally grown quality products. The overall goal is to contribute to reducing food insecurity and vulnerability in targeted communities by establishing a replicable safety net system and expanding government capacities to prevent child under nutrition. Kore Lavi covers five departments (Upper Artibonite, Central Plateau, Northwest, Southeast and West-La Gonave Island) and 23 communes, targeting over 18,000 households under the SO2 component, and more than 205,000 pregnant and lactating women and children under two years old. Kore Lavi directly supports the Ministry of Social Affairs & Labor (MAST) to deliver social services to vulnerable families5 . It has two safety net components: distribution of food vouchers to targeted families, and a Mother and Child Nutrition (MCHN) component, which includes the distribution of nutrition supplements to PLW and provision of counseling services on health and food safety. The nutrition components of Kore Lavi were phased out in October 2017. The project budget was approximately USD $80 million for the first 4 years and USD $24 million for the two years of extension. At the end of its implementation, the project is expected to achieve the following key results: • National systems for vulnerability targeting strengthened. • Access of extremely vulnerable households to local and nutritious food increased. • Maternal and child nutritional status improved. • Haitian institutions’ capacity to effectively lead and manage safety net programming improved. 1.3 Project Goals, Objectives, Results Framework, and Theory of Change The overall project goal of Kore Lavi is to strengthen the national social safety net and improve the food security and nutritional status of deprived populations in Haiti. To achieve this Goal, core programming under Kore Lavi is endeavoring to: • Address the urgent needs of extremely poor households in the five departments of Artibonite, Center, Northwest, Southeast, and West/La Gonave, and • Build Haitian institutional capacity to replicate and scale-up, at the national level, the knowledge, behavioral changes and models of social assistance for preventing food insecurity and malnutrition. Kore Lavi’s Theory of Change (TOC) holds that positive and lasting transformation happens within interrelated domains: 5 The original project description included direct support to a number of ministries, but in the end the only ministry that got direct support was MAST. Final Performance Evaluation of the Kore Lavi DFAP in Haiti Introduction 5 • Where effective social safety net (SSN) programming and complementary services reach the most vulnerable populations and protect their access to food while building self-reliance, • That achieve breadth and depth in behavior and social change needed to tackle under-nutrition among vulnerable women and children, • That institutionalize accountability, transparency and quality of delivery for mutually reinforcing social protection programs under the leadership of MAST • Where a targeted effort at the national level to improve programming for those in extreme poverty in Haiti can also help them to escape chronic food and nutritional insecurity and experience a dignified recovery. Towards these ends, Kore Lavi has four strategic objectives (SOs) supported by 10 intermediate results (IRs): • SO1: National systems for vulnerability targeting strengthened o IR1.1: MAST-led equitable vulnerability targeting methodology developed, tested and implemented • SO2: Access of extremely vulnerable households to local and nutritious foods increased o IR 2.1: MAST-led, gender-responsive food voucher-based safety net model developed and implemented o IR 2.2: Inclusion of local foods in the voucher-based safety net increased o IR 2.3: Access to complementary services for safety net households increased (including particularly access to village savings and loan groups, or VSLAs) • SO3: Maternal and child nutritional status improved o IR3.1: Household practice of appropriate nutrition behaviors to prevent malnutrition increased o IR 3.2: Capacity of community-based entities to promote appropriate nutrition practices to prevent malnutrition improved o IR3.3: Capacity of health facilities to deliver appropriate nutritional services strengthened o To achieve the third strategic objective, Kore Lavi implements a Social Behavior Change Communication (SBCC) strategy in line with the preventative approach focusing on the window of opportunity in the first 1,000 days of life. • SO4: Haitian institutions’ capacity to effectively lead and manage safety net programming improved o IR4.1: Institutional capacity of various levels of government to lead, coordinate and implement safety net programs reinforced o IR4.2: Capacity of civil society to monitor and support safety net programs reinforced o IR4.3: Government capacity to respond to food emergencies expanded. 1.4 Description of Activities and Implementing Partners Activities and consortium partner roles were mainly delineated by the various SOs of Kore Lavi, with the addition of the cross-cutting component of gender (see table below). When the project was extended by two years, SO3 activities were not continued, and ACF was no longer an implementing consortium partner. Their activities under SO2 and SO4 in the NW and Artibonite were shifted over to Care. Figure 2: Allocation of Responsibility for SOs and IRS among Kore Lavi Consortium Partners (first four years, pre-extension) IMPEL | Implementer-Led Evaluation and Learning 6 Introduction Project Component CARE ACF WFP World Vision SO1 N/A N/A Technical lead & implementation N/A SO2 Technical lead and guidance (holistically) Implementation: Southeast & Center Implementation/fie ld activities: Northwest & Artibonite (under leadership of CARE) N/A Implementation/field activities: West/La Gonave (under leadership of CARE) SO3 Implementation: Southeast & Center Technical lead & implementation: Northwest & Artibonite Commodity management N/A SO4/IR4.1 N/A N/A Technical lead & implementation N/A SO4/IR4.2 Implementation: Southeast & Center Implementation: Northwest & Artibonite (under leadership of CARE) Technical lead Implementation/field activities: West/La Gonave (under leadership of CARE) SO4/IR4.3 N/A N/A Technical lead & implementation N/A Gender* Technical lead & implementation: Southeast & Center Implementation: Northwest & Artibonite N/A* Implementation/field activities:: West/La Gonave (under leadership of CARE)* * As a cross-cutting component, all consortium partners worked to incorporate gender into all components of Kore Lavi The map below shows the areas (communes) covered by Kore Lavi, including the level of food insecurity at the start of the project. Final Performance Evaluation of the Kore Lavi DFAP in Haiti Introduction 7 Figure 3: Map of Project Intervention Zones (2013) The activities descriptions here are updated from those provided in the mid-term evaluation.6 SO1 Activities: National systems for vulnerability targeting strengthened SO1 covers three main outputs related to targeting, which are closely related to each other and which are supposed to be achieved through coordination and joint efforts of Kore Lavi and MAST. Briefly, these activities and outputs are: • Following on the work done by Kore-Fanmi (see below), Kore Lavi developed, tested and adjusted a national Haitian Deprivation and Vulnerability Index (HDVI), which allows the project to measure vulnerability in a multidimensional way using proxy means calculated through a customized algorithm. • Kore Lavi defined a specific data collection methodology implemented by other organizations on a door-to-door basis, so as to allow Kore Lavi to benefit from already-existing data. • Kore Lavi developed and implemented a database application hosted by MAST, which is a tool to store, analyze and manage collected data. The main function this system is to answer key questions about the structure and distribution of household deprivation or vulnerability so as to allow decision makers to develop an appropriate frame of social assistance for future safety net interventions. In an effort to harmonize and improve the efficiency of social services in Haiti, in 2012/3 the Government of Haiti, in collaboration with the World Bank, UN agencies, and several NGOs, created the 6 “Kore Lavi Mid-Term Evaluation,” April 2016, https://pdf.usaid.gov/pdf_docs/PA00M5HG.pdf IMPEL | Implementer-Led Evaluation and Learning 8 Introduction Kore-Fanmi Program. To implement this project, the Kore-Fanmi Vulnerability Index (KFVI), a preliminary targeting tool, was developed, in order to identify, rank, and classify households. In 2013, analysis of the KVFI found that the selection of the different dimensions that make the KFVI was not the most adequate, and that the aggregation of the dimensions was also problematic. Ultimately, this finding supported the need to conduct a more in-depth and comprehensive design of a targeting tool, which was then proposed as a nationwide instrument to determine household eligibility for social programs (social safety nets). This analysis was conducted in 2013 and 2014, giving birth to ‘Haiti’s Deprivation and Vulnerability Indicator’, the HDVI also known as the Indice Nationale de la Privation et Vulnérabilité (INPV). The methodology, tool, and score were refined over time, and remain in use today. The HDVI is a proxy means test type indicator that takes into account both aspects of deprivation and consumption/expenditure based poverty, drawing from the 2012 ECVMAS (poverty) national survey data to construct. The population considered as most deprived by the HDVI, and therefore seeks to be targeted by Kore Lavi social protection programs (most importantly, the food vouchers), are households that exhibit the larger number of deprivations at the same time, as well as being considered as poor from an expenditure point of view. Twenty indicators are derived from the Kore Lavi survey data to construct the HDVI score, which fall into seven dimensions categories. To combine these indicators, a complex formula is used, including weights for each indicator that differ depending on if the household lives in the Metropolitan Area of Port-au￾Prince, other urban areas of the country, or rural areas. The selection of indicators and their individual weights were optimized to predict the economic poverty of households, based on the 2012 ECVMAS survey data. Data is collected census-style within a targeted commune, and the index is calculated for all households. The index is then used to rank households by relative poverty/deprivation (as described by the HDVI score) within the commune, and the poorest 10% of households are targeted with SO2 (food voucher) assistance. SO2: Access to local nutritious food Under SO2, Kore Lavi developed voucher transfer modalities prioritizing the use of local foods so as to deliver food vouchers to a target of 18,150 extremely vulnerable households. To promote good nutritional practices through a balanced diet by fresh food, and at the same time to make a direct contribution to local production, Kore Lavi uses two types of vouchers: paper vouchers, which are allocated for the purchase of local fresh foods (fruits, vegetables, fresh meat, fish, eggs, etc.), and electronic vouchers allocated for the purchase of local staple foods (local rice, maize, sorghum, beans, flour, and oil). The vouchers totaled the equivalent of about USD$25, and took into account the inflation of the Haitian Gourde over the life of the project, though the adjustments were frequently delayed. The fresh food is purchased in the community markets (which function every week, often more than once a week), and the staple food is purchased in small community shops, in both cases from vendors selected by the project. Final Performance Evaluation of the Kore Lavi DFAP in Haiti Introduction 9 Figure 4: Kore Lavi fresh food voucher Photo Credit: Cesare Dieudonne, CARE HAITI To select the participating fresh food and staple vendors, Kore Lavi applies the following selection criteria: (1) capacity to deliver a variety of fresh foods and a commitment to charge stable and fair price, (2) agreement to sell only local products approved by the project, and (3) agreement to be registered on a regular basis in the market in which they do business. Kore Lavi also provided training to vendors in hygiene, basic accounting, and other basic market skills. Under SO2, Kore Lavi is also promoting and facilitating participation in village savings and loan associations (VSLAs). A VSLA is a group of people who save together and take small loans from those savings. The activities of the group run in cycles of one year, after which the accumulated savings and the loan profits are distributed back to members. Kore Lavi is facilitating the creation of VSLAs in order to help integrate as many program safety net beneficiaries as possible into VSLA groups. Although the idea of the VSLAs was to support graduation of the less vulnerable beneficiaries from the safety net program, the project was not designed to formally incorporate such graduation. SO3: Maternal and child nutritional status improved Under SO3, Kore Lavi was implementing a social behavior change communications (SBCC) strategy in line with the preventive approach focusing on the window of opportunity in the first 1,000 days of life. The project provided supplementary conditional rations to pregnant and lactating women and children aged 6-23 months with a goal to reach 76,567 women and 96,597 children during the life of the project. The conditions that beneficiaries needed to meet in order to receive food rations are: (1) take part in primary health care activities (such as antenatal and post-natal consultation, growth monitoring and immunization) and (2) participate in project SBCC interventions. SO3 ended in October, 2017 and was not continued during the extension period. However, CARE and WV carried forward some of the nutrition awareness activities in connection to the voucher distributions after that date, conducted under SO2 and cross-cutting gender activities. IMPEL | Implementer-Led Evaluation and Learning 10 Introduction The SBCC strategy was mainly implemented through a Care Group approach. In the context of Kore Lavi, a Care Group is a group of 10 women named “Lead Mothers” (or in limited cases “Lead Fathers”) who are identified by the community and come together for mutual support and learning and who are regularly trained by the local community health agents (CHAs, also known as Agentes de Sante Communautaire Polyvant, ASCPs). Each Lead Mother had responsibility for approximately 10 households and conducts visits to each household on monthly bases to share and promote key infant and young children feeding (IYCF2) practices and other appropriate health and nutrition information. In order to reinforce the knowledge of the community members regarding proper health and nutrition behavior and practices, Kore Lavi collaborated with the existing network of CHAs belonging to the Ministry of Health (Ministère de la Santé Publique et de la Population, or MSPP), while sensitizing through various community public awareness raising campaigns – such as media campaigns, nutrition fairs, special days and events, etc. – key community structures, civil society organizations and traditional health workers. CHAs also played a central role in the project’s moderate and acute malnutrition intervention. A network of nurse supervisors was responsible for training and overseeing the work of CHAs. CHAs in turn are responsible for approximately10 to 20 Lead Mothers. Moreover, to reinforce health and nutrition services provided by the health facilities, the project conducted trainings for health professionals and primary health care workers. Specifically, the project supported and facilitates active and passive screening of children through its network of CHAs so as to identify new cases of MAM and refer them to healthcare facilities to receive nutritional treatment. SO4: Haitian institutions’ capacity to lead and manage safety net programming Under SO4, Kore Lavi seeks to enhance the leadership capacity of MAST and other entities of the safety net programming. This was done by the following activities: (1) implement a joint USAID/MAST capacity building and resource mobilization plan; (2) work with MAST national and department-level staff on coordinating project safety net activities; (3) develop MAST safety net reference documentation (policies, governance, coordination, implementation); (4) support Participatory Development Support Councils (Conseil d’Appui au Développements Participatif, or CADEPs7 ) and other civil society organizations to develop capacity building plans and carry out social audit processes; (5) implement gender-sensitive annual social audits with local stakeholders; and (6) provide training to community structures (local authorities, civil society organizations) on social accountability focusing on the application of the Community Score Card (CSC), which is a tool to enhance community influence on the quality, effectiveness and accountability of the services offered by Kore Lavi at community level. This component also involves support to the National Coordination for Food Security (Coordination National de la Securite Alimentaire, or CNSA). Kore Lavi has actively contributed to CNSA’s involvement in the implementation of vulnerability targeting in the project target areas. Kore Lavi also had a seat at the table and actively participated in the CNSA-led Integrated Phase Classification (IPC) process, the collaborative stakeholder process to pool pertinent food security information in Haiti on a regular basis. 7 CADEPs are commune-based organizations comprised of local government officials and civil society and private sector representatives for the primary purpose of planning and coordinating community development. Final Performance Evaluation of the Kore Lavi DFAP in Haiti Introduction 11 Gender Activities for promoting gender equality are integrated into each component of the project design. One of the gender goals related to women is to enhance the quality, and not just the quantity, of female participation at different levels, from participation in strategic planning at the national level to more effective participation in decision-making at the household level. In addition, the project provided was designed to support to the Haitian Ministry of Women (Ministère à la Condition Féminine et aux Droits de la Femme, or MCFDF) to promote and support gender integration in food security and social assistance programming in Haiti IMPEL | Implementer-Led Evaluation and Learning 12 Overview of Evaluation 2. OVERVIEW OF EVALUATION 2.1 Evaluation Purpose and Intended Uses The objective of this qualitative performance evaluation is to assess the development outcomes of the Kore Lavi project and implications for sustainability. Evaluation results speak to challenges and lessons that can be better understood through qualitative inquiry and increase learning across partners and USAID. Findings should also inform the design and implementation of future USAID investments in Haiti. Particular attention should be directed at the processes and collaborations of stakeholders around the use of SIMAST. To address the process portion of the evaluation, there is an additional focus on “implementation methods and outputs” and on exploring “how well the project is following implementation plans and meeting targets, the acceptability of the methods employed to the beneficiary population, and signs of changes that beneficiaries associate with project interventions.”8 The primary audience of the evaluation report is USAID, as well as CARE and their consortium partners. The report will also be shared with the Government of Haiti and other stakeholders, particularly the Ministry of Social Affairs. USAID will make use of the findings from the evaluation to make presentations as part of a wider dissemination of best practices and lessons learned. The evaluation recommendations may be used by FFP to refine proposal guidelines and project policy. USAID/Haiti and GoH may use the evaluation recommendations for design of future activities. The qualitative study assesses all four SOs of the Kore Lavi project (although SO3 ended in October 2017, and only a few of the activities received minor support under SO2 during the extension award, efforts were still made to attain meaningful information about that objective). Particular attention is directed at the processes and around the use of SIMAST (the GoH proxy means testing database). Changes in participants’ lives as a result of the project are also discussed as well as issues around participant exclusion and any effects of those dynamics. The design also seeks to understand the context and community perceptions about key practices and behaviors. 2.2 Evaluation Approach The approach to the qualitative performance evaluation conducted in Haiti 2019 is unique. The Kore Lavi final performance evaluation foregoes the population-based quantitative data collection used in the baseline and midterm studies. Instead, the evaluation approach utilizes project performance monitoring data, project documents, baseline and midterm reports, as well as qualitative interview methods, to robustly answer evaluation questions, in order to inform future programming at CARE, FFP, USAID Haiti, and the Ministry of Social Affairs (MAST) Haiti. The final evaluation covers all of the project’s SOs, including trying to glean information on lessons from SO3, even though those interventions were not implemented in the two extension years. Prior evaluations include a baseline study and a midterm evaluation. The baseline evaluation consisted of the following: 1) a mixed-method qualitative and population-based quantitative study of the Kore Lavi 8 “USAID’s Office of Food for Peace Policy and Guidance for Monitoring, Evaluation, and Reporting for Development Food Assistance Projects: Draft for Public Comment,” September 2015, p. 12. Final Performance Evaluation of the Kore Lavi DFAP in Haiti Overview of Evaluation 13 implementation area, and 2) an impact evaluation baseline of SO2 to study the impact of the food voucher safety net component of the Kore Lavi project. Various changes occurred during project implementation, most significantly a decision to end interventions around SO3, maternal and child nutrition. FFP decided not to conduct a population-based final evaluation, and instead focuses on conducting a qualitative performance evaluation. However, FFP will be conducting an impact evaluation of SO2 in 2020, the results of this evaluation will feed into that impact evaluation design and data interpretation. Primary data collection for the qualitative evaluation took place in August 2019. 2.3 Evaluation Questions Consistent with the above methods, the evaluation answers the following questions as defined by USAID as outlined below. These questions largely align to common evaluation criteria: Relevance, Effectiveness, Efficiency, Impact, and Sustainability. EQ1: To what extent has the project met its defined goals, purpose, and outcomes? The ET evaluates the contribution of Kore LAVI to USAID’s efforts to reducing food insecurity and vulnerability in targeted communities by establishing a replicable safety net system and expanding government capacities to prevent child under nutrition. The ET supports these determinations using qualitative methodologies and document reviews when discussing the following: • EQ1.1 Project performance on indicators measured against targets set by the partners and FFP indicators;9 • EQ1.2 Factors that promoted or inhibited the achievement of the project objectives, including but not limited to the effectiveness of interventions including MAST systems transfer to GoH, food voucher safety net, and earlier maternal and child health interventions; • EQ1.3 Plausibility of pathways and the determinants of achieving the key outcomes; • EQ1.4 Targeting strategies and their contribution to achieving project goals (especially with regard to gender and reaching the most vulnerable) including analyzing if the HDVI methodology systematically identified the poorest households; • EQ1.5 The appropriateness and effectiveness of interventions for the poorest individuals and Households. Q2: Based on the evidence, which project outcomes are likely to be sustained? The ET evaluates the functionality and performance of systems and processes established independently by the projects, as well as in collaboration with the private sector, Government of Haiti, non￾governmental organizations, and academic organizations to achieve project outcomes and sustainability. It supports its determination using qualitative methodologies that explore the following: EQ2.1 The quality of the processes, systems, and institutional arrangements developed and/or strengthened to sustain the necessary and critical services; EQ2.2 Communities’ perceptions on the quality, frequency, effectiveness, and sustainability of the services provided by the project; EQ2.3 The likelihood that service providers will continue providing services after the project ends; 9 Monitoring indicators that represent proxies along the pathway to reductions in Depth of Poverty, stunting and underweight will be identified and studied. IMPEL | Implementer-Led Evaluation and Learning 14 Overview of Evaluation EQ2.4 The motivation of the community and beneficiaries to demand and pay (or invest time) for the services; EQ2.5 Whether the necessary resources and capacity strengthening will exist to sustain service providers; EQ2.6 The extent to which the projects leveraged other USG and non-USG investments to achieve sustained outcomes as identified in the theories of change; EQ2.7 Evidence of enhanced linkages with other service providers. Q3: In each technical sector, what are the strengths and challenges of the selected interventions and their implementation, and how are these received by the target communities? The ET evaluates the effectiveness and relevance of the technical interventions to achieve project outcomes, and discuss those findings in relation to the theory of change. Determinations are supported when discussing the following: EQ3.1 Factors in the implementation and context associated with greater or lesser effectiveness in producing Outputs of higher or lower quality; EQ3.2 Interventions and implementation processes deemed more/less acceptable to members of the target communities. Q4: What are the key lessons learned and best practices that should inform future projects in the country? During the course of research, the ET identified best practices, strengths, and challenges in the project designs (including theories of change), that support project achievements, as well as approaches that should be considered favorably in designing future food and nutrition security projects and strengthening household and communities’ resilience capacities. The ET supports determinations using evidence when discussing the following: EQ4.1 The unintended positive and/or negative consequences of the projects; EQ4.2 Ways to minimize potential unintended negative consequences and systematically capture positive consequences. In addition to the above EQs, two cross-cutting questions were considered when presenting the findings across all EQs: • Did the Consortium fully integrate gender across the project as was originally conceived in the project? • What were the benefits and lost opportunities of the consortium approach? The ET also considered the efficacy of the following cross-cutting interests when investigating the EQs: • Project management, • Performance monitoring, • Strategies to improve gender equality at the participant and project management levels, • Conflict sensitivity, • Government system-strengthening, • Lessons learned, best practices, and facilitators and inhibitors of sustainability. Finally, FFP provided a list of SO-specific questions/areas they considered of particular interest. While not formal EQs, the ET attempted to take these questions into account. Those additional evaluation questions can be found in Annex 6. Final Performance Evaluation of the Kore Lavi DFAP in Haiti Design and Methodology 15 3. DESIGN AND METHODOLOGY The Evaluation Team consulted with the Kore Lavi implementing partners in Haiti, USAID/Haiti Mission, and FFP/Washington teams to identify the most appropriate data collection and analysis methods. The methodology chosen by the Evaluation Team (ET) was based on the following data collection activities: • A desk review of documents and data relevant to the project, • A purposively-sampled qualitative study based on Key Informant Interviews and Focus Group Discussions. Analysis began with a desk review of project documents, validating their understanding of the project through consultations with USAID and the implementing partner. While analysis primarily draws on qualitative data from interviews, the Evaluation Team also uses available quantitative data (e.g. annual reporting data and secondary data) to answer the evaluation questions and to cross reference findings from the fieldwork. 3.1 Desk Review The desk review serves two primary purposes. One purpose is to inform the evaluation design and implementation. Another purpose is to integrate information from project documents and other secondary sources with the primary source information collected during the fieldwork in order to generate a set of evaluation findings and conclusions. The desk review also includes an inventory of other reports and studies that may provide relevant information to the project themes or contextual information about Haiti, focusing on the areas of project implementation. The review includes a range of project documents describing different aspects of project design, implementation, monitoring and evaluation, and results. Documents reviewed contextualize the evaluation questions, as well as gain an in-depth understanding about the project design, implementation and the food security situation in general. Annual monitoring data was reviewed to prepare for qualitative research, considered in relation to the evaluation findings, and incorporated into the report as evidence of evaluation findings. The below documents represent the literature review. • Project RFA and proposals, • Annual Results Reports (ARRs), Quarterly reports including Indicator Performance Tracking Tables (IPTTs) for performance against targets, • Midterm review and corresponding action plans developed by the project, • Baseline Study for the Title II Development Food Assistance Project in Haiti, • Haiti Demographic and Health Survey, • Monitoring data and field reports, • Sustainability and Exit Strategy Plan. 3.2 Consultation with Implementing Partners and USAID As a supplement to the desk review, and in preparation for qualitative research, consultations with FFP and partner staff in Washington, DC and Haiti allowed the ET to corroborate its understanding of the IMPEL | Implementer-Led Evaluation and Learning 16 Design and Methodology design, approaches and interventions employed by the DFAP and acquired through the desk review. Prior to the beginning of field research, USAID and implementing partners had the opportunity to review and provide input and feedback on the draft evaluation protocol and on data collection tools. Equally important, the ET consulted with FFP/Haiti and implementing partner (IP) staff post-data collection to validate initial findings. After an initial briefing with USAID FFP staff, the ET consulted with the main IP (CARE) to discuss their expectations of what would be assessed in the evaluation, to brief the team on the overall activities of Kore Lavi, and to help identify other key information such as monitoring data that may inform the evaluation. The ET also requested lists of project implementation locations and specific activities, in order to inform the purposive sampling for the qualitative research in the field. Additional discussions with CARE and other Kore Lavi stakeholders continued following these initial consultations. Initial in-country discussions helped the ET gain an understanding the context and to gather basic information about the project. In addition, the ET sought information regarding the following: • Areas of success, • Missed opportunities, • Evidence of sustainability, • Initial thoughts/conclusions on Kore Lavi from the partners, • Specific challenges that may be hindering implementation of activities, • Success stories or aspects of the implementation, • Adoption of practices by community members not formally participating in FFP activities, • Approaches to promote adoption of practices by non-participants, • Areas of learning and mechanism for knowledge sharing. The ET conducted initial meetings with key consortium staff members during the first few days of fieldwork. Regular consultations with implementing partners (most notably CARE and FFP) over the course of the evaluation ensured that key stakeholders kept informed of progress and findings as they happened, rather than at the end of the mission. 3.3 Qualitative Design Qualitative research involved a mix of methods including key informant interviews (including in-depth interviews) and focus group discussions (including observations and follow-up interviews). Qualitative methods were used to collect information to answer evaluation questions, and also to support the interpretation of findings from the desk review. These methods — to the maximum extent possible — ensured that if a different, well-qualified evaluator were to have undertaken the same evaluation, he or she would have arrived at the same or similar findings and conclusions. The ET used methodological and data triangulation to validate and interpret the research findings. The research protocol submitted to FFP presented specific methodologies along with the proposed sample, the number and location of interviews, type of respondent and type of interview (KII, FGD, etc.) Following feedback, discussion and agreement (between ET, FFP/W, FFP/Haiti, and CARE), the ET finalized the research protocol and methods. Final Performance Evaluation of the Kore Lavi DFAP in Haiti Design and Methodology 17 The interview guides are found in Annexes 5 (English) and 6 (French/Creole). The ET team was responsible for interviewing direct, indirect, and non-participant community members and households as well as looking for evidence of ongoing learning and activities. The ET was also responsible for analyzing the qualitative data. The ET engaged a local survey firm, IFOS, to complement the data collection effort, however they did not replace the Evaluations Team’s role in data collection and analysis. The ET began by collecting data at the central level with FFP staff and implementing partners to get an overview of project activities, contextual factors that have influenced implementation, and changes in project implementation. Subsequently, the ET and IFOS travelled to communities located in the identified departments where Kore Lavi activities have taken/are taking place, to assess these interventions. Through department and commune-level KII and FGD interviews, the ET assesses: • The range and quality of activities being carried out, • The appropriateness and effectiveness of interventions including cross-cutting themes such as gender and sustainability, • The extent to which the projects have been implemented according to plan, • The incorporation of findings and recommendations from the baseline study and mid-term evaluation, • Perceptions related to strengths and weaknesses of the interventions, as well as positive and negative consequences of the Kore Lavi activities. The combination of data collection at the central, department, and community levels allows the ET to identify factors that have enhanced or constrained achieving the desired objectives and longer-term sustainability. Key informant interviews were carried out with FFP staff and implementing partners in Port-au-Prince as well as in the field. In-depth interviews, key informant interviews, and focus groups discussions were conducted in community settings. 3.4 Identification of Key Informants The ET implemented in-depth one-on-one and small group Key Informant Interviews with purposively selected project stakeholders. To the extent possible, interviews were conducted in-person using a combination of prepared questions and follow-up probing questions, taking an average of 90 minutes to complete. The KII sample was selected in an attempt to be representative by gender, department, respondent group, and respondent type. Although the ET made a concerted effort to identify women respondents, approximately three quarters of KII respondents were men (47 of 63). IMPEL | Implementer-Led Evaluation and Learning 18 Design and Methodology Figure 5: Key Informant Interview, Center Department Photo Credit: John Berry Key Informant Interviews were held in each of the five departments where Kore Lavi was active including: Artibonite, Center, Northwest, Southeast and West (La Gonave). Respondent groups included the three implementing partners (IP), two international organizations (IO), project partners, and central and departmental Government of Haiti staff (GOH) and community leaders (CASEC and CADEC). A full list of KII broken down by gender, department and respondent group is included below. The full list of entities (IPs, GOH, etc.) is found in Annex 3. Final Performance Evaluation of the Kore Lavi DFAP in Haiti Design and Methodology 19 Figure 6: Key Informant Interview Sample KI Interviews, respondents Number of KII 46 Number of KI respondents 63 KII Respondents by Gender Female respondents 16 Male respondents 47 KIIs by Department Center 5 Northwest 5 West (including PaP) 22 Artibonite 8 Southeast 6 KIIs by Respondent Groups International Organization 3 Implementing Partner 17 Project Partner 14 GOH 12 KIIs by Organization ACF 2 CNSA 1 MFIs 2 EU 1 CADEC 3 MAST 5 CADEP 2 MSPP 1 CARE 10 USAID 2 CASEC 5 Vendors 7 3.5 Selection of Focus Group Sites The focus group sampling strategies at community-level and respondent-level purposively targeted groups who experienced a range of the variability in project outcomes. Sampling criteria were designed to capture experiences across the operating context for different communities and respondent groups with the goal of selecting communities that demonstrate a representative range of project activities and beneficiary characteristics. The sample included groups in all five departments (Center, Northwest, West, Artibonite, and Southeast) and in a range of communes in different geographic zones (mountainous, island, bordering the Dominican Republic, non-border, coastal, and non-coastal plane). Two to three focus groups were conducted in each selected commune. Sampling also captured respondents from different socio￾economic backgrounds with a range of perspectives and experiences. This included relatively poor/vulnerable households as well as extremely poor households. The sample was designed with the goal of selecting communities that experienced a range of project activities such as food security and IMPEL | Implementer-Led Evaluation and Learning 20 Design and Methodology livelihoods. Focus groups had different perspectives based on their experience with the project including voucher beneficiaries, VSLA beneficiaries (with and without vouchers), mère and père leaders. Sampling for the evaluation included both individuals who directly participated in the Kore Lavi project and those not targeted with any intervention (non-participants). Implementing partners were asked to provide lists beneficiaries as well as the sites where interventions were/are being carried out and information related to the project activities. Potential community sites were identified during the first days of fieldwork planning, though an attempt was made to prevent excessive forewarning of the communities to prevent potential bias. The assistance of implementing partners was requested in organizing meetings for KII and FGD interviews. The following decision-making factors were used in selecting communes: • Achieving, to the extent possible, a representative sample of project locations and stakeholder groups • To work in at least one commune in each of the five departments; 19 communes in total • Commune population variation and ensuring coverage of both peri-urban and more rural communes • Climatic and geographical variations • Include communes where multiple interventions took place and where a range of respondent groups is present (vouchers, VSLA, vendors, non-beneficiaries, etc.) • The desire to follow-up on findings from previous FGDs with specific stakeholder groups • Include communes that border the Dominican Republic to ensure consideration of the impacts of migration and cross-border trading • The availability of key informants to participate when the evaluation team is in a particular location • The difficulties in finding respondents familiar with discontinued SO3 activities • Travel, logistics, ease of access, security and working within a tight budget and a short timeframe. A list of all the communes covered by Kore Lavi, those selected for study in this evaluation, and commune-specific information that helped guide the selection of data collection sites, is found in Annex 4. 3.6 Identification of Focus Group Participants Focus group discussions were facilitated discussions conducted with groups of 6-10 participants. FGDs were conducted using a combination of pre-prepared interview questions/guides, and facilitator follow￾up probing questions, and took on average around 60-90 minutes to complete. Final Performance Evaluation of the Kore Lavi DFAP in Haiti Design and Methodology 21 Figure 7: Assembling a focus group, Center Department Photo Credit: John Berry Focus Group Discussions included the following respondent groups: Voucher recipients, VSLA members, Lead mothers and fathers, and Non-participating members of the communities. In order to identify non￾beneficiaries, the evaluation team worked with CARE to develop selection criteria to identify individuals in communities where project activities took place who were not direct project beneficiaries, but who could offer informed opinions about their outside perceptions of project activities. Figure 8: Focus Group Discussion Interview Sample Focus Group Discussions and Respondents Number of FGD 26 Number of respondents 257 FGD respondents by Gender Female respondents 173 Male respondents 84 FGDs by Department Center 6 Northwest 8 West 3 Artibonite 5 Southeast 4 Beneficiary Groups Voucher beneficiaries 9 VSLA beneficiaries 8 VSLA/No Vouchers beneficiaries 1 Mère and Père Leaders 5 Non-beneficiaries (in voucher communities) 3 IMPEL | Implementer-Led Evaluation and Learning 22 Design and Methodology 3.7 Ethics Tulane received approval from its Institutional Review Board (IRB) to conduct the evaluation fieldwork. Verbal informed consent was obtained from all key informants and focus group participants before being interviewed. Additionally, verbal consent was obtained from all photographed subjects in this report. 3.8 Evaluation Training and Fieldwork Evaluation fieldwork took place from 4-24 August, 2019. It commenced with kick-off meetings with USAID at the US Embassy and with senior management teams from CARE and other implementing partners at the Kore Lavi headquarters in Port-au- Prince. The ET worked closely with IFOS prior to field work to identify a group of skilled and experienced qualitative data collectors. The qualitative field data collection team from IFOS was composed of six people. Three (two men, one women) served as the leaders of the FGDs and the KIIs in the field sites. Each of these interview leaders were paired with a note-taker (one woman, two men), responsible for recording the FGDs and taking notes during the FGDs and the KIIs. Each pair served as a ‘team’. One data collector also served as the overall field work supervisor/coordinator for the teams. Technical and logistical support/supervision from the IFOS office in Port-au-Prince provided technical and logistical support/supervision to the entire ET. After approval of the protocol and tools, the Evaluation Team worked with the IFOS staff and the six data collectors to refine the FGD guides and KII questions and to translate them from English to French and/or Creole (one member of the Tulane team has a basic command of Haitian Creole). A two day training was conducted with the qualitative data collectors, reviewing the goals and objectives of the evaluation, an overview of Kore Lavi activities, review and discussion of the interview guide, and general principles of qualitative data collection to gather, accurate and unbiased data. Additionally, the training covered principles of ethically-sound data collection, including how to request verbal consent to participant, respect for privacy, anonymization of data (names not recorded on interview notes), and how to approach sensitive topics to ensure the participants confidence and safety. The team then conducted a one-day field testing of the instruments in Boucan Carre10 (a community not far from Port-au-Prince where Kore Lavi had conducted SO2 voucher activities), under the supervision of the Tulane ET and IFOS management. This was followed by a discussion with the field team of challenges, doubts, and questions they had regarding the instruments and field work in general. The Tulane ET members then began conducting a series of initial Key Informant Interviews in Port-au￾Prince while also providing guidance/supervision supervising the fieldwork of IFOS interview teams. Two person teams from IFOS conducted Focus Group Discussion in the following departments: Center, Northwest, Artibonite, Southeast and West/La Gonave. The ET worked closely with IFOS to oversee the data collection process, making field visits and when feasible in order to participate in interviews and focus group discussions. Additionally, the ET and IFOS worked collaboratively during the data collection 10 Although the data collected in Boucan Carre was not formally included in the analysis, the information was still reviewed for salient points. Final Performance Evaluation of the Kore Lavi DFAP in Haiti Design and Methodology 23 phase to troubleshoot any logistical challenges and ensure the smooth transition from data collection to transcription and translation. The ET also worked closely with the Kore Lavi staff to develop the fieldwork interview schedule and to receive appropriate introductions and contact information. The ET requested the support of the Kore Lavi team to make the appropriate arrangements with the intended Key Informants and FGD participants. All interviews and FGDs were conducted ns in the language in which the participant was most comfortable in speaking (Creole for all FGDs, KIIs Primarily in French or Creole, with some also in English). A digital recorder was used to record all Focus Group Discussions (following their informed consent). In the event that a participant refused to have their interview recorded, the interviewer notes were used to create a summary of the respondents’ answers to the questions. All digitally recorded interviews were shared with IFOS managers and the ET within 24 hours of completing the interview. IFOS transcribed all digitally recorded interviews in Creole and then translated them into English. Quality checks of the transcription occurred in the field. IFOS produced the first set of transcriptions for examination one week following the start of interviewing. The transcription and translation process logically extended beyond the data collection period; and was competed approximately two weeks after the end of data collection. The ET implemented quality assurance and quality control activities before, during, and following data collection. Quality control procedures included clear documentation on how the data was collected, translated and transferred to the ET, in-field monitoring, supervision and additional training as necessary, daily debriefings and problem solving sessions, and the consistent review of the collected data. 3.9 Data Analysis At the end of fieldwork, a validation workshop was held at Kore Lavi headquarters with the participation of key members of the Kore Lavi and Kore Lavi consortium member management teams, USAID and other Port-au-Prince based project stakeholders. The initial findings were presented for discussion, and the remaining analysis plan outlined. While the analysis at that point was only very preliminary, it allowed for initial feedback from the key stakeholders and input on the interpretation of key findings to date. To analyze the qualitative data, the evaluation team created detailed written summaries of all KIIs and FGDs drawing on their detailed interview notes and transcripts (of FGDs). These detailed notes from each KII and FGD were main set of data that were then analyzed. The ET then reviewed all these notes and coded them using the following steps. This work was done by the team lead and assistant team lead, in close collaboration to ensure the work was unbiased and both were approaching the data in similar fashion. 1) Information was organized by which of the EQs and sub-EQs the data informed/related to. At times certain points/subject related to multiple sub-EQs, and so the data was classified as IMPEL | Implementer-Led Evaluation and Learning 24 Design and Methodology informing multiple EQs. This in essence was coding the data by which sub-EQ(s) they pertained to. 2) Then, the data under each of the sub-EQs were reviewed and initial creation of ‘topic-codes’ was done under each sub-EQ. These codes consisted of a specific theme, statement, concept, relationship, or meaning. These topic-codes were added to a code-book. 3) The two ET members coding the data merged their codebooks iteratively to ensure that both were using the same set of topic-codes, and coding results similarly under topic-code and sub￾EQ. They also periodically reviewed a set of the other’s coding work to be sure that they were consistent with each other. 4) Iterations of coding were done to combine similar topic-codes under each evaluation question/sub-question and ensure consistent coding throughout. 5) Once the topic-coding was complete under all sub-EQs, all topic-codes were then classified into “themes”, which were groups of codes that were thematically related. Some examples of these themes are: Nutrition, Training, GoH capacity, livelihoods, targeting, project quality. Codes that fell under more than one theme were assigned both theme. 6) This resulted in three sets of codes: topic-codes (the most granular, detailed set of codes), which were disaggregated by which sub-EQ they fell under. The topic-codes were also separately grouped into thematic codes. 7) The data were then tabulated into an excel spreadsheet (‘Tally Sheet’). Each interview (KII, FGD) had all of the topic-codes under each of the sub-EQs (and the theme that code fell under) checked off when that topic-code had occurred in that particular interview. 8) To summarize the topic-codes that occurred most frequently, the data from the tally sheet was tabulated according to the frequency of occurrence of each topic-codes under each sub-EQ. The most frequent were retained for including in tabular results in the report. Additional analysis was done on by the type of respondent (gender, participant, government, consortium staff, etc.) and included in the findings text where pertinent. 9) Additionally, throughout these analysis steps, the ET extracted key findings, insightful comments, or other pieces of information or quotes from the KII and FGD notes which were considered important and informative despite their rare or singular occurrence. These were simply recorded in a separate document and used during the writing of the report. A list of the questions, topic-codes, and themes used for the coding of the qualitative data can be found in Annex 8. In the findings sections of this report, the most frequently occurring topic codes under each sub-EQ are provided in tables. The frequency of each key theme/phrase is the number of respondents that were coded for that key theme/phrase. ‘N’ presented at the top of each table is the sum of the respondent frequencies for all the key phrases/themes under that sub-EQ. As some of the sub-question themes were simply discussed less (fewer questions, or limited to specific key informants), the relative frequency of a key theme/phrases to the total N for the sub-question is important to take into account. The findings from the qualitative study were compared with findings from the desk review and analysis of performance data and are combined into this final evaluation report. Final Performance Evaluation of the Kore Lavi DFAP in Haiti Design and Methodology 25 3.10 Data Limitations The Kore Lavi mid-term evaluation relied almost exclusively on qualitative data collection methods. By its nature, qualitative data lacks representativeness. In this case, and owing to time and logistical constraints, the evaluation team was not able to cover all of the project communes in the five project departments. While the evaluation team was conscious of avoiding bias in the selection of study communes, logistical conditions made it impossible to visit certain communes within the timeframe allotted for the fieldwork. In those communes visited by the evaluation team it was not possible to meet with all primary stakeholders. While the evaluation team worked with Kore Lavi to identify a set of representative communes and stakeholders in each department, these may not have provided a full picture of the project. Because the final evaluation comes at the end of the project implementation period, this may have limited the ability of the evaluation team to contact informants who were no longer engaged with the project, particularly those involved in SO3 activities that ended in 2017. However, through close collaboration with the Kore Lavi staff, the ET endeavored to define a broad sample of respondent groups and to interview as many respondents as possible. The evaluation chose to focus the data collection on the project participants and other stakeholders in Haiti. However, many people had been involved in KL early on in the process, but only a limited number of these people were included in the key informant interviews. The evaluation is careful in its analysis to separate unique occurrences from generalizable trends. Some findings, particularly those based on respondent anecdotes, may be true in certain cases, but are not necessarily representative of the project as a whole. To mitigate the risk of reporting anecdotal evidence, the evaluation team captured findings that occur with sufficient frequency across different data collection methods to indicate that they provide useful insights into project operations and results. In those cases where the ET is required to draw on more limited anecdotes, or where the ET cannot comfortably assert that something generalizable in occurring, it is made explicit. IMPEL | Implementer-Led Evaluation and Learning 26 Findings 4. FINDINGS The four evaluation questions and the two cross-cutting questions laid out in the Scope of Work for this evaluation provided the framework for the research design, the interview tools, and the findings that follow. The purpose of these findings is to present the evidence that the ET gathered through the desk review and through Key Informant and Focus Group Discussion interviews. The goal of presenting this evidence is to offer the ET’s assessment of the development outcomes of the Kore Lavi project and the implications for sustainability, with the intention of informing the design and implementation of future USAID investments in Haiti. Particular emphasis was placed on outcomes regarding improvements to the national social safety net and to the food security and nutritional status of deprived populations in Haiti. The conclusions and recommendations offered by the ET in subsequent sections of this report are based on these findings. 4.1 EQ1: To what extent has the project met its defined goals, purposes and outcomes? In order to assess Kore Lavi’s progress towards achieving the project goal of strengtheningthenational socialsafetynet andimprovingthefoodsecurity and nutritionalstatusofdeprivedpopulationsinHaiti, the ET reviewed project reporting, including IPTT tables, and discussed project outcomes with a range of individuals, including GOH officials, and groups, including voucher and VSLA beneficiaries, as well as project partners. In addition, the ET sought information regarding the project’s progress toward and potential for achieving large-scale impact on malnutrition. Finally, the ET investigated the extent to which Kore Lavi leadership integrated gender across project activities as well as the benefits and lost opportunities of the project’s approach. Specific areas of inquiry regarding achievement of project goals included review of project performance data, as well as assessment of factors influencing achievement of those objectives. The ET also inquired about the appropriate and effectiveness of Kore Lavi’s targeting and their interventions to support development of a robust social safety net and reduce malnutrition. As laid out in detail in the discussion below, Kore Lavi largely achieved its goals as defined by indicators captured in IPTT reporting. In addition, many of the direct project participants that took part in the focus groups indicated that the project achieved its purpose of reducing malnutrition, especially in children, and in improving the diversity and quality of food available to beneficiaries. Additionally, the national safety net policy (whether promulgated or simply as a regulatory framework), and the development of the computer-based poverty targeting system (SIMAST), have the potential to be lasting legacies of the project. EQ1.1: Project performance on indicators measured against targets set by the partners and FFP indicators The main tool used by Kore Lavi for measurement of progress against performance targets was the Indicator Performance Tracking Table (IPTT table). These tables were updated each quarter to reflect performance data for the quarter and cumulatively, compared to baseline and final targets. Because the Final Performance Evaluation of the Kore Lavi DFAP in Haiti Findings 27 project underwent a major revision of its targets after the midterm evaluation in April 2016, the IPTT tables for 2014-2017 are different from, and not entirely comparable to the IPTT tables for 2018-2019. Therefore, performance against targets for the two periods will be discussed separately. (A summary of these tables is presented below and the full tables are found in Annex 7.) Although the Strategic Objective (SO) level framework remained the same, there was significant change in Intermediate Result (IR) level indicators between 2014-2017 and 2018-2019. These changes reflected a shift in the focus of project activities to emphasize building the institutional capacity of MAST. The significant increase in focus on SO1 (National systems for vulnerability targeting strengthened) is demonstrated in the increase in number of IR indicators under this objective from the original 5 to 11. There was a similar increased emphasis on SO4 as shown in the increased in IR indicators from 10 to 15. The transition between the first phase of the project and the extension also saw the end of SO3 activities. During both the 2014-2017 timeframe and the 2018-2019 period, the project reported on 45 IPTT performance indicators. Many basic FFP indicators were captured in both periods (i.e. “Prevalence of households with moderate or severe hunger” and “Prevalence of underweight children under five years of age”). However, in 2018 the project added indicators to capture progress towards GOH capacity building and localization such as “MAST Institutional Capacity Index progression” and “Percentage of food accepted for voucher redemption that is locally grown.” Performance against targets was fairly consistent over the two periods. Performance against Targets 2014-2017 During the 2014 - 2017 period, Kore Lavi made significant progress towards achievement of a range of IPTT indicators11. During 2014-2017 the project reported achieving 100% or more of its target for 16 indicators, 75% to 99% for 23 indicators, 50% to 75% for two indicators, and less than 50% for three indicators. In analyzing the project’s overall performance, there were several areas where the project had notable success (as indicated by achievement of 100% or more of targets). These include, training, targeting and outreach to poor HH, distribution of food vouchers, and capacity building. For example, Kore Lavi reached all of its targets in training for child health and nutrition (102,180 people trained, including assigned households to LM/LF care groups), Gender Based Violence (199 people trained) and social audit methods (32 organizations trained). In addition, the project reached 100% or more of its targets in communes using vulnerability targeting (18 communes) and in outreach (18,158 HH enrolled in voucher program, 1,241 VSLA established and 998 vendors accredited). Final, the project was also successful in its capacity building goals, for both the GOH (5 departments developed social safety net plans and 20 MAST staff trained to manage the SIMAST database) and civil society organizations (16 CADEPs who developed capacity building plans). 11 IPTT table, FY 2014-FY2017, Kore Lavi Project, 2017. IMPEL | Implementer-Led Evaluation and Learning 28 Findings During 2014-2017 period the project reached 0% on two indicators12, for reasons mainly tied to the feasibility of the indicator design, according to IP interviews. Furthermore, a key indicator with a target but no data on actual achievements for this period in the IPTT tables was “Number of safety net beneficiary households graduated from safety net program”, which according to multiple interviews proved to be an unrealistic expectation for extremely vulnerable households (see further discussion on this in the 2018-19 performance against targets section, below). The table below details a selection of project indicators for 2014 – 2017 (the full list is found in Annex 7). Figure 9: Summary IPTT Reporting 2014 – 2017 LOA Target LOA Actual % of Target Achieved Indicator 81,318 102,180 126% Number of people trained in child health and nutrition through USG-supported programs. (cumulative, 8,376 female and 93,804 male, includes household assigned for care groups LM/LF) 80% 83% 104% Percentage of food accepted for voucher redemption that is locally grown 1,200 1,241 103% Number of VSLA groups established 17,700 18,158 103% Number of households enrolled in the food voucher￾based safety net 95% 95% 100% Percentage of vouchers redeemed 1,000 998 100% Number of vendors who are providing food under the safety net 178,051 172,493 97% Number of rural households benefiting directly from USG interventions 179,251 173,554 97% Number of vulnerable households benefiting directly from USG interventions 80% 72% 90% Percentage of CADEPs (or other identified civil society structures) supporting public awareness campaigns on ENA, IYCF, GMP or CMAM 163,861 143,978 88% Number of individuals receiving supplementary, conditional rations 12 Indicators during the 2014-2017 period with 0% performance against target include Number of safety net participants linked with complementary services (other than VSLA groups), Number of DPC staff trained to use vulnerability database. Final Performance Evaluation of the Kore Lavi DFAP in Haiti Findings 29 LOA Target LOA Actual % of Target Achieved Indicator 30,723 24,550 80% Number of children under two (0-23 months) reached with community-level nutrition interventions through USG-supported programs 10,152 7,911 78% Number of pregnant women reached with nutrition interventions through USG supported programs 50% 38% 77% % of safety net participants in VSLA groups (non￾cumulative) 40% 14% 35% Percentage of eligible children in a health center catchment area enrolled in GMP services 3,600 0 0% Number of safety net participants linked with complementary services (other than VSLA groups) Performance against Targets 2018-2019 In the 2018 - 2019 period, Kore Lavi continued to make important progress towards achievement of a range of IPTT indicators13. During the 2018-2019 period the project reached 100% or more of target for 17 indicators, 75% to 99% for 10 indicators, 50 to 75% for 10 indicators, and less than 50% for eight indicators. Areas where the project achieved 100% or greater of its targets varied somewhat from the first to second phases of the project. From 2018 – 2019 the project performed particularly well on training, gender inclusion and capacity building. For example, Kore Lavi more than doubled its targets for training MAST departmental staff (154 staff trained), and more than 100% of its targets for nutrition-related professional training (1,144 trainees). Gender was also a major area of success for the project (greater than 100% of targets reached) with 18 community activities and 150 group discussions organized around gender topics. In addition, the project purposively selected women vendors who represented 87% of all project vendors (109% of target). The project continued to perform on its capacity building goals for the GOH, particularly regarding the SIMAST system (training MAST staff, and uploading beneficiary data and integrating SIMAST with other systems). In addition, Kore Lavi developed an innovative measurement system for the institutional capacity building at MAST, achieved more than 100% of its goal for MAST staff capacity building. During the 2018-2019 period the project reported six indicators at 0% performance against target. Lack of achievement for some of these indicators was impacted by the inability of the project to directly influence the desired outcomes. Two examples of indicators over which the project did not have direct influence, both of which were reported a 0% achievement against targets, are the “Number of interns from Haitian IT institutions providing technical support to SIMAST day-to-day operations” and the 13 IPTT table, FY 2018-FY2019, Kore Lavi Project, 2019. IMPEL | Implementer-Led Evaluation and Learning 30 Findings “Percentage of Commune-level Agricultural offices that conduct the regular technical supervision/support visits to the local food producers.”14 The latter was reported to be due to the difficulty for the project M&E to effectively monitor the technical support that was provided by the commune-level agricultural offices to the local producers that were associated with the project. However, the project did create linkages between certain local producers and commune-level agricultural offices. An important indicator that was that was reported at 0% during the 2018-19 period was “Number of safety net beneficiary households graduated from safety net program” The challenge with this indicator is that FFP and the consortium never agreed on the definition of ‘graduation’, and the lack of a livelihood component in the design of Kore Lavi means that there was no clear programmatic pathway to graduation. Some reports of ‘self-graduation’ were reported by the consortium staff, but the definition of this indicator was also unclear- it appears to include households that were removed from the voucher lists because they were considered ‘fraudulent’, and possibly households that simply relocated out of the project area. The inclusion of this indicator was ill-conceived by both FFP and the consortium without a clear pathway to graduation, a clear definition, or a discussion on if graduation was an appropriate goal for the project at all. It was also a lost opportunity to define this better during the extension period. The table below details a selection of project indicators for 2018 – 2019 (the full list is found in Annex 7). Figure 10: Summary IPTT Reporting 2018 – 2019 LOA Target LOA Actual % of Target Achieved Indicator 72 154 214% Number of Departmental MAST staff trained in complementary trainings (ICT, program management and administration) 2 2.14 107% Progression of the SIMAST staff capacity, as measured on the institutionalization scale 22% 23% 105% Percentage of beneficiary households in the MAST-hosted Household Vulnerability Database who have been resurveyed 70% 72% 103% Percentage of VSLA groups integrating nutrition in their regular activities 83% 83% 100% Percentage of food accepted for voucher redemption that is locally grown 19,158 18,158 95% FFP 51a: Number of households benefiting directly from USG assistance under Food for Peace 14 IPTT table, FY 2018-FY2019, Kore Lavi Project, 2019. Final Performance Evaluation of the Kore Lavi DFAP in Haiti Findings 31 LOA Target LOA Actual % of Target Achieved Indicator 44,063 41,763 95% FFP 32: Number of people benefiting from USG-supported social assistance programming 150 136 91% Number of Safety net beneficiaries who start income generating activities through VSLA 4 2.98 75% MAST Global Institutional Capacity Index progression (related to social safety net programming) 90% 58% 64% Percentage of autonomous field visits performed by MAST staff at Departmental and Communal levels 5 3.14 63% Community Structures Capacity Index progression 0.8 0.5 63% MAST (central and departmental-level) Institutional Capacity Index progression 50% 26% 52% Percentage of Safety net beneficiaries (VSLA Members) who start income generating activities 1.30 0.36 28% Progression of the Vulnerability Targeting MAST staff capacity, as measured on the institutionalization scale 1,000 2 0% Number of safety net beneficiary households graduated from safety net program EQ1.2: Factors that promoted or inhibited the achievement of the project objectives. Among the numerous factors influencing project achievements cited in KII and FGD interviews several key themes emerged. Some of these themes represent decisions internal to the project and some external factors beyond the control of the project. The most frequently cited theme revolved around issues regarding the project’s exit strategy. In addition, the influence of choices in project design and management were mentioned. Finally, external factors related to macro conditions were cited as impacting the ability of the project to achieve its objectives. IMPEL | Implementer-Led Evaluation and Learning 32 Findings Figure 11: Qualitative data topic coding summary on Achievements (EQ1.2) What FACTORS PROMOTED OR INHIBITED achievement of the project objectives including EFFECTIVENESS? N = 88 (total topic code frequency under this question) The handover strategy to the GOH was overly ambitious and needed more time to ensure sustainability and ownership of activities. 21 The participation of CASEC and CADEC in the planning and implementation of the project helped to achieve the objectives 9 Macro conditions such as inflation and increased food prices eroded positive gains in reducing vulnerability. 8 Kore Lavi focused on survival, not protecting/promoting livelihoods. The level of support needed to protect livelihoods is much more than what the coupons provided. 4 In response to the specific question of factors influencing achievement of project objectives, close to a quarter of all mentioned issues with exit strategy. Two elements of the exit strategy were cited as particularly ambitious, the institutional capacity building of MAST and the handover of project activities to the GOH. One implementing partner clearly noted that: “The project’s goal of a complete transfer of social safety programming to the GOH in 4 years was ambitious and overly optimistic. They should have questioned of what level of institutionalization equals success – planning control or operational control. MAST achieved the former and not the latter.“ This sentiment that the designers of Kore Lavi were overly ambitious in what they thought could be achieved in four years was expressed most frequently by GOH and implementing partner staff during KIIs, as well as other partners (the handover strategy was not brought up in any of the FGDs), particularly concerning the project hand over strategy and its potential impact on the sustainability of project-supported activities. One specific example of this cited in interviews with GOH staff is that the MAST department level staff indicated that they would be unemployed when the project ends. While the project’s theory of change focused on both institutionalizing effective social safety nets and reducing vulnerability, achievements in capacity building proved harder to measure than decreases in vulnerability. As one of the IP staff members put it: “The consortium brought a food delivery and food security mentality and approach to a capacity building and governance project.” Implementing partners further described the difficulties in measuring institutional capacity with more nutrition-focused indicators from the ‘required if applicable’ list of standard indicators from FFP. Although FFP asks partners to create a comprehensive monitoring strategy and include custom indicators as well, there was a lack of indicators to measure institutional capacity or policy-level progress in the original M&E plan (though more were added during the two-year extension period). The disconnect between performance indicators and the projects achievements is perhaps best demonstrated in their singular success in supporting the development of a national social safety net Final Performance Evaluation of the Kore Lavi DFAP in Haiti Findings 33 policy. While the design and potential promulgation of a social safety net policy was cited as one of the most important and sustainable achievements of the project, there were no IPTT performance indicators to capture policy-level progress according to IP interviews. Yet according to a GOH respondent: “The project had a major influence on national level social protection policy on social protection, and to help to create a policy framework for a national-level social safety net.” Understanding both the limitations of the existing performance indicators in capturing capacity building and its importance to ensuring the sustainability of project activities, Kore Lavi refocused its performance measurement system to include a greater emphasis on capacity building. The two tables below offer an indication of this shift from the first to the second phases of the project, with a major increase in the number and type of indicators captured. It is also important to mention that in response to a dearth of measurement tools for capacity building, the WFP supported KL in the development of an innovative institutionalization measurement system for MAST (see more on this indicator in subsequent sections). This system scores progress on discrete aspects of institutional capacity such as the ability to plan, coordinate and implement social safety net activities. Scores were updated regularly and progress towards institutionalization measured against benchmarks organized into five levels. 15 Figure 12: Changes in IPTT Capacity Building Reporting 2014 – 2019 Indicator Target Actual % met IPTT 2014 – 2017 43 Number of departments where local governmental structures have a plan to support implementation of safety net programs 5 5 100% 44 Number of MAST staff trained to coordinate and monitor safety net and other social protection programs 20 18 90% IPTT 2018 – 2019 11 Progression of the Vulnerability Targeting MAST staff capacity, as measured on the institutionalization scale 1.3 0.36 28% 12 Percentage of Information System (IS) staff recruited by MAST (MAST Planning unit) or transfer under MAST contract 50% 25% 50% 15 Level 5 - normative and operational responsibility; Level 4 - normative responsibility; Level 3 - involved in monitoring and key decisions; Level 2 - consulted for key decisions; Level 1 - informed of developments; and Level 0 - not involved in decision making. IMPEL | Implementer-Led Evaluation and Learning 34 Findings Indicator Target Actual % met 13 Progression of the SIMAST staff capacity, as measured on the institutionalization scale 2 2.14 107% 14 Number of MAST/CAS staff trained on information systems, information analysis and communication 5 5 100% 38 MAST Global Institutional Capacity Index progression (related to social safety net programming) 4 2.98 75% 39 Percentage of completed activities of the specific central and departmental-level institutionalization/transfer plans 90% 63% 70% 40 MAST (central and departmental-level) Institutional Capacity Index progression 0.8 0.5 63% 41 Percentage of MAST department and central-level organizational units that are able to produce quarterly activity reports 90% 36% 40% The increased focus on capacity building at MAST indicated by the above was strongly appreciated by the GOH staff interviewed as a vital support to helping them achieve the government’s social protection goals. GOH staff expressed their appreciation for different forms of support offered by the project, particularly the department level MAST staff. In kind support in the form of generators, batteries, chairs, computer and paper was also appreciated. In addition, project support in the form of fuel for GOH vehicles “helped MAST staff to go to isolated communities that they had not been able to reach before” and “improved MAST’s visibility on the ground to promote social protection.” Another frequently mentioned positive factor promoting the achievement of project social protection objectives was the participation of the GOH and Community Based Organizations (CBOs) in project planning. When asked specifically about the project’s achievement, more than one out of seven respondents cited the involvement of GOH and CBOs. Areas of involvement mentioned include planning with CASEC and CADECs and regular meetings between MAST and Kore Lavi staff. The impact of external factors was mentioned frequently by respondents, both in response to the specific question of project achievements, and to overall issues regarding project impact. Among both KII and FGD respondents there was a general feeling that even while the project was able to temporary improve food security, macro conditions such as inflation, devaluation, drought and civil unrest eroded positive gains in reducing vulnerability. Specifically, project participants expressed in FGDs the opinion that: Final Performance Evaluation of the Kore Lavi DFAP in Haiti Findings 35 “Inflation reduced the value of coupons over the course of the project” and that despite the efforts of the project “food insecurity and malnutrition have not gone away.” While there were adjustments made to the value of the vouchers to adjust for inflation, KIIs indicated that the adjustments were often slow, and did not keep up with the quickly devaluation Gourde. The limitations on how many beneficiaries Kore Lavi could enroll was frequently cited as inhibiting achievement project objectives. FGD participants as well as many KIs frequently mentioned that there were many more vulnerable households in their communities than the project could reach. The consortium members often discussed this as a budgetary challenge, but this response reveals the poor messaging that should have emphasized that Kore Lavi was in fact a model on which the GoH (and perhaps other donors/stakeholders) could build its approach and expand coverage. Another limiting factor to achieving project objectives was the inability of the project to capture/target newly vulnerable/poor households. According to an implementing partner the inability of the project to take on new beneficiaries meant that it was not a true social safety net, saying: “KL isn’t a true social safety net, because a true social safety net also needs to catch people when they fall by responding to [idiosyncratic] shocks.” Finally, another factor influencing achievement of the overarching goal of decreasing vulnerability was the focus on improving food distribution rather than livelihoods promotion. During an interview with an international organization, it was explained that the project “couldn’t do everything,” and that not focusing on livelihoods was an intentional design choice. (Although it could be argued that VSLA is a livelihoods approach.) Several GOH mentioned the impact of this design choice, one indicated “Kore Lavi focused on temporary survival, not on promoting sustainable livelihoods. The level of support needed to protect livelihoods is much more than the coupons provided.” EQ1.3: Plausibility of pathways and the determinants of achieving the key outcomes. When evaluating the impact on household food security and resilience, the effects of the project during the implementation phase (particularly SO2 activities) tend to be different than the lasting effects of the project post-implementation. The impact on the knowledge, attitude, and practice of consuming a diverse diet appears to be successful in both the short and long term. Although the fact that a portion of the vouchers could only be used for fresh local products (fruits, vegetables, meat, fish, etc.) may have obligated households to adopt a more diverse diet (rather than a less diverse diet consisting of the cheaper-calorie foods) during the life of the project, the FGDs and KIIs indicated that as a result of the awareness training and focus activities on diversification of diet, they had gained an improved understanding of the value of a more diversified diet on health and their intentions to continue eating a more diversified diet where possible after the end of the project. This was salient across most FGDs, particularly voucher recipients, but also among non-voucher recipients and VSLA group FGDs. However, in some FGD discussions, the voucher IMPEL | Implementer-Led Evaluation and Learning 36 Findings recipients indicated that their ability to continue to eat a more diverse diet will greatly diminish after the fresh food vouchers stop being distributed, despite their increased knowledge and attitude, and they may need to return to eating a less diverse (and cheaper) diet.16. Figure 13: Qualitative data topic coding summary on Kore Lavi’s contributions to reducing food insecurity and vulnerability (EQ1.3) What has been Kore LAVI’s contribution to reducing food insecurity and vulnerability in targeted communities? N = 171 (total topic code frequency under this question) Reduce malnutrition especially in children 27 Improve the variety of nutritious products accessible to the most vulnerable 20 Improve diversity and balance in the diet 20 Facilitate access to local products 19 Reduced food insecurity 17 Training on the preparation of balanced and nutritional meals 12 Figure 14: Qualitative data topic coding summary on pathways/determinants of achieving key outcomes (EQ1.3) What is the plausibility of PATHWAYS and the DETERMINANTS of achieving the key outcomes? N = 34 (total topic code frequency under this question) Project Kore Lavi has achieved its objectives because thanks to VSLAs. VLSAs have improved the economic situation of many women. 6 The project safety net was able to respond to co-variant shocks, such as droughts or hurricanes, but not to household level idiosyncratic shocks. 4 The project’s goal of a complete transfer of social safety programming to the GOH in 4 years was ambitious and overly optimistic. 4 VSLAs helped people avoid the use of credit with high interest coming from other sources. 3 Communities didn’t just need access to food, they needed to know how to increase local production. 3 One of the weaknesses of SO3 was its theory of change that beneficiaries would escape insecurity 3 There also appear to be some small improvements related to child nutrition. Project monitoring reports that exclusive breastfeeding increased from 38% at baseline to 45% at Q3 2019, and the use of ORT increased from 67% to 72% during the same period (a very minor improvement). However, FGDs often 16 The quantitative SO2 impact survey, with data collection planned to take place between late 2019 and late 2020, may yield more specific information on the long-term (post-project) impact of Kore Lavi on household food consumption/dietary diversity. Final Performance Evaluation of the Kore Lavi DFAP in Haiti Findings 37 cited positive impacts of training on child and maternal health. Project data indicate that 102,000 people were trained in MCHN. The beneficiaries often linked the improved knowledge and practice of diversification of their diets to improved child health. One of the much-spoken assumed impacts of the Kore Lavi activities was that households would take the opportunity the long-term assistance provided to improve their self-reliance and resilience by improving their social capital, developing new livelihood strategies or improving existing ones, etc. However, the vouchers alone have a limited effect on increasing household self-reliance. According to project reports, only a very small fraction of voucher recipients started income generating activities (of 17,000 hhs receiving vouchers at the end of the life of the projectme, only 136 were reported to have started income generating activities in Q3 2019). And very few households ‘graduated’ from the safety net program. Figure 15: Changes in IPTT safety net graduation reporting 2014 – 2019 Indicator Target Actual % met 2014-2017 IPTT 12 Number of safety net beneficiary households graduated from safety net program 1,650 NA 0% 2018-2019 IPTT 18 Number of safety net beneficiary households graduated from safety net program17 1,000 2 0% The lack of a livelihood component to the project, either as part of the project or in close collaboration with other projects, was a missed opportunity for Kore Lavi. FGDs and KIIs alike reported that the vouchers alone were not enough to bring households out of poverty, additional support in agriculture or other livelihood activities was required if households were to be brought up, rather than just sustained, by the overall project. However, it should be noted that not all households may have the ability to ‘graduate’, such as elderly households, handicapped, chronically ill, etc. Some portion of poverty in Haiti will be constant, and some may only improve on a generational time scale. Integration with other livelihood projects and/or inclusion of these livelihood projects should be spelled out from the initial conception of the project. Additionally, the IPs should be encouraged to make justified changes to the project (deviations from the RFA) in their initial proposals without concern that this may lessen their chances of being granted the project. Finally, FFP should work more closely with the IPs to encourage adaptive management over the life of the project. Adaptive management only became the official standard for FFP after the start of the KL project, but future projects should include specific language defining this process in the project design. The lack of a clear mechanism in the Kore Lavi food voucher program allowing households to ‘graduate’ from the project was a design flaw, eclipsed by the more concerning fact that the project did not allow for the incorporation of ‘newly poor’ households. Here, the term ‘newly poor’ refers to household that may have been above the threshold for inclusion in the Kore Lavi voucher program at the beginning of 17 CARE indicated that an additional 228 households had ‘auto-graduated’ (or self-graduated). The definition of this indicator is unclear as discussed under EQ1. IMPEL | Implementer-Led Evaluation and Learning 38 Findings the project (not in extreme poverty), but due to household specific shocks (e.g. - death of a household head, additional children in the household, loss of livelihood source, sickness, etc.) fell into a situation of more extreme poverty. This means that the food voucher program can better be described as long-term assistance for chronically poor, rather than as a safety net put in place to catch households and prevent them from falling (further) into poverty. Finding a feasible resolution to this is admittedly challenging. Adaptability requires not only more data, but potentially more resources if the number of those meeting the criteria to receive vouchers increases over time. One approach may be to increase the regularity of data collection, updating the household information in SIMAST in order to refine/adapt the beneficiary list on a more regular basis. There seemed to be an assumption among some of the IPs that when given the additional resources for the extended period of time, poor households would establish new livelihoods/income sources. However, FGD participants often indicated a belief that increased food security among the extreme poor was temporary, and recovery (exit from extreme poverty) was not broadly achieved. Although the data indicate that the vouchers reduced the day-to-day struggle of poor households, the evaluation interviews (both KIIs and FGDs) generally indicated that the vouchers allowed for maintenance but not improvement of the households’ situation. According to project documents, only two households officially ‘graduated’ from receiving vouchers (out of a goal of 1,000)... Some FGDs and KIIs indicated that some beneficiaries were dependent on the vouchers, and beneficiaries often compared the end of Kore Lavi to a shock equivalent to that of a death in the family. The limited long-term impact of the vouchers was noted by an interviewee from the GoH, who said that, “The level of income required to protect and augment livelihoods is much higher than what the vouchers provided.” Figure 16: Qualitative data topic coding summary on Kore Lavi’s contributions to establishing a safety net and expanding government capacity (EQ1.3) How has Kore LAVI contributed to establishing a replicable SAFETY NET and EXPAND GOVT CAPACITY to prevent child undernutrition? N =68 (total topic code frequency under this question) Training and capacity building support for anchoring MAST capacity increased ability to govern social safety net programs. 9 The project helped to improve MAST visibility and linkages between MAST staff and communities. 7 MAST is understaffed, overworked, with high levels of turn over, and central level capacity is weak 6 The project had a fundamental influence national level policy on SP and to help to create a policy framework for SSN. 5 The project’s capacity building diagnostic tools were precise and useful. 4 While not a complete success when measured by the overly ambitious project goals, the major progress in institutionalizing accountability, transparency, and quality of delivery for social protection programs at MAST should be considered an area of great success of Kore Lavi. The Global Institutional Capacity Final Performance Evaluation of the Kore Lavi DFAP in Haiti Findings 39 Index was created specifically to measure progress in this area18. This shows a steady trend from the start of the project to mid-year 5 of the project (see figure below). Respondents also indicated that the project greatly enhanced MAST’s visibility and engagement with communities, and helped give them a ‘seat at the table’. However, capacity of MAST for operational self-sufficiency remains a work in progress. Respondents indicated that MAST remains understaffed, overworked, and lacking capacity at the central level. Figure 17: MAST Global Institutional Capacity Index19 Level 3 – Involved in supervision and key decisions Level 2- Consulted in key decisions EQ1.4: Targeting strategies and their contribution to achieving project goals Considerable effort was made in Kore Lavi under SO1 to develop and refine an accurate targeting methodology that could efficiently identify the most poor, minimize inclusion and exclusion error, and gain community acceptance of the targeting approach.20 Generally, respondents indicated that the SIMAST system did a good job of identifying vulnerable populations. However, this positive opinion of targeting was most frequently heard from consortium 18 This MAST Global Institutional Capacity Index indicator measures the overall level of advancement of the institutionalization of MAST through the institutionalization scale, which is from 0 to 5 (5 being the top score in institutionalization). It combines a number of criteria related to seven aspects of institutional capacity: human resources, participation, planning, management systems and procedures, coordination, monitoring and implementation, and material resources. The scores for each of these aspects are used to create a weighted average, which is the global index value. 19 Graph taken from the Kore Lavi project powerpoint titled “Résultats du diagnostic sur l’institutionnalisation – 2e Trimestre, Année 5 “ dated 8 June, 2018. This graph contains an backwards projection, as the indicatyor was only developed during the extension period. The estimated level for 6/2019 was received verbally in a KII with a consortium member staff. 20 The Kore Lavi voucher targeting system consisted of identifying the poorest/most vulnerable 10% of households a target commune. The measure (indicator) of poverty/vulnerability is the HDVI, which is a score derived from a complex formula drawing from household interview data, as collected on the HDVI ‘scorecard’ from all households in a targeted commune. The data from the HDVI scorecards are entered into the SIMAST system for managemet, storage, calculation of the HDVI (or other indicators), and the creating of the lists of the 10% poorest/most vulnerable 0 0.4 0.8 1.2 1.6 1.98 2.43 2.76 3.05 2.83 0 0.5 1 1.5 2 2.5 3 3.5 MAST Global Institutional Capacity Index Estimated Level 06/2019 = 2.98 IMPEL | Implementer-Led Evaluation and Learning 40 Findings and GoH key informants and from focus groups of voucher recipients. Respondents from all levels frequently indicated that the demand was bigger than the supply in terms of breadth of coverage. As of June 2019, the SIMAST system contained data on 22% of the population of Haiti, or about 2 million people in 350,000 households. They expect to reach 30% of the population by June 202021. Whatever the minor critiques are of the system, the SIMAST system and data is a large and impressive undertaking, and has clearly demonstrated that this type of approach (census-type data for targeting) is possible in Haiti. The success of the SIMAST system is evidenced by the fact that, according to Q3 2019 project reports, SIMAST is being used in three other projects. Additionally, concrete plans are in place to fund and use the SIMAST in other projects, such as European Union (EU) funded 11th European Development Fund (11th EDF, or 11eme FED), as well as interested expressed by the Swiss Cooperation and the World Bank. Occasional critiques of the targeting system did surface during interviews however. Despite careful attention to any indication of targeting bias in the interviews, only a few indicated that there was room to manipulate the system and identify participants through favoritism. This was most frequently cited in the KIIs with the CADEP and CASEC as well as in several of the FGDs. These same respondents also often stated that they felt the targeting approach in general was not very accurate. However, this general type of criticism were more the exception rather than the rule. In some cases, respondents specifically stated that the SIMAST system was preferable as it avoided the risks of favoritism that come from community listing type targeting. A few respondents did comment the fact that, although they felt that the targeting system adequately identified chronically poor people, it did not allow for the addition of newly-poor households who suffered a loss/shock that changed their circumstances and brought them into extreme, chronic poverty after the voucher targeting exercise was completed. One IP interview indicated that they had thought the targeting census data would be more regularly updated, but that this was not always the case, and many voucher recipient lists were still based on what was becoming rather outdated information. One potential source of bias in the targeting methodology was identified by several respondents. They indicated that the poorest households may not have been honest when responding to the initial data collection questions, possibly out of fear or shame of their impoverished status. This perception was heard from key informant interviews (consortium staff as well as Kore Lavi participating food vendors), as well as in some of the FGDs (two with Kore Lavi direct participants, one with non-Kore Lavi￾participants). This may have led to their exclusion from the voucher recipient lists. However, it should be noted that there is not a quantification of how often this actually occurred. Although frequently cited in FGDs and KIIs, this does not necessarily indicate that the exclusion itself was a frequent occurrence, but rather a noteworthy one to the respondents. The project made positive use of a ‘bureau de recours’ (appeals office), which opened for a few weeks after targeting was completed in a community. This allowed households who felt they were incorrectly excluded from receiving vouchers to present themselves for review and re-evaluation. A few respondents indicated that the bureau de recours (appeal office) was an important part of the targeting approach that helped reduce exclusion errors in particular. 21 According to an interview with a consortium staff member. Final Performance Evaluation of the Kore Lavi DFAP in Haiti Findings 41 Figure 18: Qualitative data topic coding summary on targeting strategies (EQ1.4) Did TARGETING strategies identified the poorest households and reach the most vulnerable and including women? N = 154 (total topic code frequency under this question) Overall, targeting indicators and SIMAST system did a good job of identifying vulnerable populations 32 Reached some of the poorest households but not all. Demand for coupons was greater than supply. 16 The targeting system was skewed, some people were able to manipulate it and others benefited from favoritism 14 Weaknesses in targeting census prevented the most vulnerable people in the area from integrating the project 13 Women benefited the most (especially SO2 and SO3) 12 When surveyed, some people interviewed did not give relevant personal information, they were ashamed to show how poor they were. 10 Vendors benefited the most from the project. 9 One of the common issues respondents highlighted when discussing the targeting approach was that the difference between the extreme poor that were selected to receive vouchers and the others in the community was often extremely slight. The difference between those receiving vouchers and many of those not within the same community was often negligible. While this sometimes led to a feeling that the targeting was poor, it mostly was discussed around the fact that there were more poor households in the communities that would benefit from the vouchers than could be covered by Kore Lavi. Looking specifically at the SIMAST system and the HDVI tool, it should be noted that the SIMAST system and the HDVI are not synonymous; the success of the SIMAST system is not tied to the HDVI. While the SIMAST system currently collects and houses the indicators for the HDVI calculation, it is easily adaptable to collect and analyze other indicators. For example, the project 11eme Fed, supported by the EU, is making use of the SIMAST system by adding additional indicators such as MUAC of children under 5, in a system they are referring to as ProcSIMAST. This is important for the other actors to understand. If there is increased criticism of the HDVI as it becomes outdated and/or insufficient to inform the targeting of other projects using SIMAST, the risk of ‘throwing the baby out with the bathwater’ should be avoided. Indeed, one of the strengths of the SIMAST is its adaptability to adapt what kind of information it collects. A review of the HDVI was conducted in 201822. This report concluded that the HDVI is a good proxy for chronic food insecurity, but not a good proxy for acute food insecurity. This being said, people interviewed as part of this review generally agreed that the HDVI functioned adequately as a targeting 22 “HDVI Review: Complementary and Alternative Approaches to Targeting”, September 2018. Reporte written by consultant to WFP, with the goals to 1) Review the HDVI (tool, score, methodology), 2) Compare the HDVI to other targeting tools (PMTs, scorecard, frequency listing) 3) Recommend possible changes/adaptations to the HDVI tool, score, and methodology, 4) Advise on possible complimentary modules to the HDVI tool, particularly in the areas of livelihoods, food security, and nutrition. IMPEL | Implementer-Led Evaluation and Learning 42 Findings tool. Although informants said that there continues to be some degree of inclusion and exclusion error, these errors are not huge nor are they of large concern. However, the review also concluded that the HDVI methodology/score is a ‘black box’ statistically. This was reflected in evaluation interviews, where some of the IPs expressed their lack of clarity in general on the HDVI methodology, and the difficulty they had in advocating for its use when they did not fully grasp it themselves. There is only general documentation of the statistical process, but the database used and the detailed specifics of the statistical approaches are unavailable even to Kore Lavi and partners. Additionally, the basis for the HDVI (the 2012 ECVMAS) may be becoming outdated. Unfortunately, the black-box nature of the HDVI prevents it from being fully re-assessed or redesigned. Kore Lavi may have benefited from using slightly more transparent proxy measures of poverty/vulnerability. Existing proxy measures of poverty/vulnerability for Haiti, such as the Poverty Probability Index23 could be considered and tested. The project monitoring data indicates relatively good achievement of output targets (see tables below). The way that the percentage of beneficiaries incorrectly included/excluded (1.6%) is questionable, however. The definition/calculation of that indicator was not clear to the ET. Figure 19: Changes in IPTT targeting Reporting 2014 – 2019 Indicator Target Actual % met 2014-2017 IPTT 7 Percentage of beneficiaries incorrectly included/excluded 5% 1.6% 310% 8 Number of communes in which vulnerability targeting methodology is implemented 17 18 106% 9 Percentage of vulnerability recourse committee members who are women 50% 49% 98% 11 Number of beneficiary verification reports 21 16 76% 17 Number of activities (meetings, media campaigns, awareness raising sessions) to sensitize stakeholders on gender issues 275 279 101% 47 Percentage of CADEP (or other identified civil society structures) representatives who are women 50% 27% 54% 23 https://www.povertyindex.org/country/haiti Final Performance Evaluation of the Kore Lavi DFAP in Haiti Findings 43 Indicator Target Actual % met 2018-2019 IPTT 27 Percentage of vendors providing food under the safety net who are women 80% 87% 109% 30 Number of community-level activities integrating dialogues on gender equality and women’s empowerment 16 18 113% 61 Number of group discussions organized on gender equity and equality with participation of the safety net beneficiaries 144 150 104% 48 Percentage of participating community structures' members who are women 45% 34% 76% EQ1.5: The appropriateness and effectiveness of interventions for the poorest individuals and households As discussed in previous sub EQ sections, the overall project was generally well viewed by project stakeholders and participants alike. Respondents frequently indicated that the beneficiary households were generally satisfied with the appropriateness and quality of the services. VSLAs continued to be discussed very positively, for the whole community and not just the most extremely poor. Respondents indicated that the combination of vouchers and VSLAs was a synergistic approach. The vouchers allowed even the poorest households to participate in VSLAs, which they otherwise may have been excluded from, due to lack of disposable income to put towards savings. Respondents discussed the benefits of having vouchers limited to local foods vs. simply being given cash. While some indicated that they may have preferred cash because that would allow them to pay for other necessities such as medical care or schooling, many expressed and understanding that having vouchers restricted to the purchase of fresh/local foods also had benefits, even for the poorest households. It should be noted that some expression by participants of a preference for cash is inevitable in any food-voucher based intervention; this should not be considered a shortcoming of the project. Indeed, it is an indication of success that many participants described the positive impacts of the local-food voucher system on diet quality. One important shortcoming that was brought up repeatedly throughout the interviews was the lack of a livelihoods or agricultural support component to allow the communities/households to increase production and income, which could have had a synergistic effect with the voucher portion of the project. IMPEL | Implementer-Led Evaluation and Learning 44 Findings Figure 20: Qualitative data coding summary on intervention appropriateness, effectiveness (EQ1.5) Were interventions for the poorest individuals and HH’s APPROPRIATE AND EFFECTIVE? N = 19 (total topic code frequency under this question) All vulnerable beneficiary households were satisfied with the quality and appropriateness of services 5 Linking coupons and VSLA was successful. 3 It would be better to give cash to the beneficiaries 2 Coupons were more appropriate than cash. 2 Communities didn’t just need access to food, they needed to know how to increase local production. 2 4.2 EQ2: Based on the evidence, which project outcomes are likely to be sustained? The ET evaluated the functionality and performance of systems and processes established independently by the projects, as well as in collaboration with the private sector, Government of Haiti, non-governmental organizations, and academic organizations to achieve project outcomes and sustainability. The following themes were explored as part of this EQ: • The quality of the processes, systems, and institutional arrangements developed and/or strengthened to sustain the necessary and critical services; • Communities’ perceptions on the quality, frequency, effectiveness, and sustainability of the services provided by the project; • The likelihood that service providers will continue providing services after the project ends; • The motivation of the community and beneficiaries to demand and pay (or invest time) for the services; • Whether the necessary resources and capacity strengthening will exist to sustain service providers; • The extent to which the projects leveraged other USG and non-USG investments to achieve sustained outcomes as identified in the theories of change; • Evidence of enhanced linkages with other service providers. EQ2.1: The quality of the processes, systems, and institutional arrangements Overall, respondents indicated very frequently that they felt the quality of Kore Lavi services was acceptable to excellent, and indicated they had no serious complaints about Kore Lavi activities. This perception was noted in nearly all the FGDs, but was mentioned much less frequently among KIIs (one CADEP, one CADEC, and one MFI). While to some extent, project participants may have been reticent to critique the project because they hoped for an extension/continuation, this indicates an overall appreciation of the project nonetheless. Final Performance Evaluation of the Kore Lavi DFAP in Haiti Findings 45 The most frequently cited complaint in the project process quality was timeliness of the payments to the MFIs (who then in turn reimbursed merchants for the fresh food vouchers they took as payment). Despite the fact that the consortium generally communicated that the payment delay issues were eventually resolved, this challenge appeared to be chronic/recurring. One interviewed MFI indicated that their payments had been regular and timely until the transition in IP management that happened during the extension period, during which time their payments became consistently late, leading to late payments and frustration among the participating vendors. While the ET was not able to fully understand why this problem persisted, it was likely due in part this shift in management. The monitoring of payments in future projects must be very carefully done. Figure 21 MFI office in Anse-a-Galets, La Gonave, West Department Photo Credit: John Berry However, despite the payment timeliness, the two MFIs interviewed discussed many good aspects of their work with Kore Lavi, and described many long-term positive impacts. The MFIs’ participation in Kore Lavi gained them better recognition and respect within the communities. This allowed them to expand their services greatly, they expressed that they will continue to benefit from Kore Lavi long after the project ends. While good for their businesses, these MFIs are also usually providing credit to people at much lower rates that other forms of credit often available in rural communities, allowing households to take greater advantage of credit. The MFIs also complemented well with the VSLAs. The MFIs interviewed indicated that the VSLAs were more appropriate for the small savings/credit of individuals/poor households, while the MFIs were better for small businesses that needed access to credit, such as merchants. One important contribution of Kore Lavi to the development of social protection policy in Haiti is the work on the draft National Policy for Social Protection (PNPS). Although not part of the initial project IMPEL | Implementer-Led Evaluation and Learning 46 Findings design, the necessity of such a policy to ensure long-term viability of any social protection program managed by the government was recognized over the course of the project, and this was added during the two-year extension period. Work began formally on the PNPS in January of 2018, and was near completion at the time of the evaluation fieldwork24. The development of the PNPS included a nation￾wide set of stakeholder consultations (the first public political consultation since 2009), including all GoH line ministries, as well as local and civil society authorities in all 10 departments, with excellent participation from MAST from the start of the process. According to one IP staff interview: “A social safety net cannot be built without an overall social protection government policy” Another big success of Kore Lavi in the area of sustainability is the creation of SIMAST, which promises to have long term impact beyond Kore Lavi. Interviews indicated that the SIMAST database, once handed over, will cost between $80,000 and $100,000 per year to maintain. Government funds were included in the GoH budget for 2018/19, but not approved. Local computer technicians can maintain the system if they have the resources, however, it will cost more money and require more staff if a national level system is to be achieved. However, continued financial support for SIMAST for the immediate future seems likely. According to Q3 2019 project reports, SIMAST is being used in three other projects. Additionally, concrete plans are in place to fund and use the SIMAST in other projects, such as European Union funded 11th European Development Fund (11th EDF, or 11eme FED), as well as interested expressed by the Swiss Cooperation and the World Bank. Despite these successes, however, the handover of SIMAST to MAST and passage of the PNPS are still works in progress. Although the end of Kore Lavi just short of these goals could be considered a missed opportunity, there is reason for optimism that WFP’s continued support to the SIMAST and the PNPS process will allow for the full handover to take place, and help support the passage of the PNPS. Challenges to this goal are the frequent government turnover, and divergent visions of the purpose/role of the SIMAST. While designed for social protection targeting, there was some indication that the vision of some stakeholders for the SIMAST was broader than that, expanding into targeting of agriculture, livelihood, health, or other types of interventions. While this speaks well of how the SIMAST is viewed, it risks blurring the purpose and yielding an overly complicated yet non-specific information system. Additionally, if the SIMAST were expanded to include more information that just that pertinent to targeting social protection safety nets, and possibly into a unique registry of beneficiaries (registre unique de beneficiares, or RUB) for Haiti, then the question of where to house the system and who would manage it would need to be addressed. VSLAs were cited more frequently by FGDs and KIs than any other Kore Lavi activity as a positive, sustainable contribution to support food security in Haitian communities. Officially, Kore Lavi supported the formation of about 1,200 VSLAs. Additionally, there are indications of numerous other non-project, community-initiated VSLAs, which indicates the appeal of VSLAs to the Haitian population. Kore Lavi was successful in providing a complaint line for project participants, allowing them to officially register any concerns or complaints they had. This was brought up as a positive point by a few 24 It should be noted that the bulk of the development of the social protection policy was conducted through a separate award to WFP. However, this work was very integrated into KL as a whole, and was generally viewed as an activitiy that came out of KL. Final Performance Evaluation of the Kore Lavi DFAP in Haiti Findings 47 respondents, who indicated that the vendor or voucher recipient complaint had been addressed following their reporting of the issue. The project monitoring also indicated that 76% of complaints were addressed. This was short of the 100% goal, but the very existence and general positive view of the complaint line suggests its success. Beyond the success of the use of the complaint line, several data points indicated that Kore Lavi did an excellent job of treating the project participants (referring to the treatment of the vendors and voucher recipients) with respect and dignity. Figure 22: Qualitative data topic coding summary on quality of process, systems, arrangements (EQ2.1) What was the QUALITY OF THE PROCESSES, systems, and institutional arrangements? N = 80 (total topic code frequency under this question) Quality of services of a satisfactory / acceptable quality 14 The quality of services was good / excellent quality 12 No complaints against the project 11 Delays in paying vendors and MFIs 10 The treatment of beneficiaries and vendors by the project staff was very open; beneficiaries were treated with respect and dignity 7 Coupons were distributed on time and highly appreciated 7 The project did a satisfactory job at responding to complaints and called the vendor or beneficiary to explain the situation 3 Figure 23: IPTT complaint response rate 2014 – 2017 Indicator Target Actual % met 2014-2017 IPTT 15 Percentage of complaints cases addressed 100% 76% 76% EQ2.2: Communities’ perceptions on the quality, frequency, effectiveness, and sustainability of the services provided by the project. As observed in other sections, the respondents cited the VSLAs most frequently as a success, followed by their appreciation for the vouchers. Most FGD/KIIs observed a positive impact on child health and nutrition status, and that households were consuming a more diverse diet. However, FG participants often mentioned that the improvements in food security and vulnerability were temporary, and that despite the vouchers, they still considered themselves vulnerable. They indicated that after the end of the voucher distributions, many of the poorest households would no longer be able to participate in the VSLAs very easily. The significant loss that households were feeling at the end of the project was described by several respondents. One voucher recipient stated “For the beneficiaries of the Kore Lavi IMPEL | Implementer-Led Evaluation and Learning 48 Findings project in the commune of Port-de-Paix, the closer of the project feels like one’s spouse has died or moved out of the country”. In general, it appears that respondents are indicating that VSLAs and food vouchers were very well received by the communities, and very successful overall. However, either alone or when paired together, are still not sufficient to build long-term resilience of households. The 2018 Kore Lavi commissioned study entitled “Kore Lavi Safety Net Beneficiary Resilience Assessment provided an important learning opportunity to capture positive consequences of the project activities. This report showed that the food voucher had a very positive impact, particularly in response to both covariate and idiosyncratic household shocks. Additionally, the voucher recipients interviewed indicated that the vouchers allowed them to save for school fees (59% of respondents), for medical expenses (28%), and for participation VSLAs (36%). However, the study also found that 10% of the voucher recipients interviewed stated that they had no other source of income, and that many voucher recipients indicated that they would no longer be able to afford to participate in VSLAs when the vouchers stopped. The report also states that “the end of the project is even perceived a “major shock” because the people are still not able to guarantee the basic food security on their own. The respondents expressed a high uncertainty about what might happen after Kore Lavi ends.” The inclusion of other community structures, such as the CADEPs (CBO planning/coordination groups) was viewed as a positive way to ensure a smoothly operating project. One CADEP member stated: “The involvement of all actors in the project has had a mitigating effect on problems that could have prevented the project from progressing” Two FGDs were conducted specifically with people who did not receive vouchers (in areas where vouchers were being distributed), to gain their perspective on Kore Lavi’s activities. All the FGD participants were aware of the voucher distribution. They had positive feedback on their perceptions of impact, including reductions in child malnutrition and improvements in diet quality among the participating households. They generally thought the targeting was fair. In one focus group, the participants brought up the fact that their neighbors that received the vouchers shared with them every month, so they also indirectly benefited. However, one focus group discussed how they had noted that some of the poorest households may not have given true answers in the survey, resulting in their possible exclusion from voucher lists. Figure 24: Qualitative data topic coding summary on community perceptions of project services (EQ2.2) What were COMMUNITY PERCEPTIONS on the quality, frequency, effectiveness, and sustainability of the services provided by the project? N = 25 (total topic code frequency under this question) VSLA was a success 7 Everything worked well in the project, particularly supervision by project staff 4 Beneficiaries greatly appreciated the voucher system 3 Coupons will not have a lasting impact on the community 3 Final Performance Evaluation of the Kore Lavi DFAP in Haiti Findings 49 What were COMMUNITY PERCEPTIONS on the quality, frequency, effectiveness, and sustainability of the services provided by the project? N = 25 (total topic code frequency under this question) Beneficiaries complained that the end of the project was like the death of their mother and father 2 There were small disagreements between the sellers and the beneficiaries at the start; the problem was solved through additional training 2 The IPTT data showed positive improvements in community perceptions meeting targets as well. The data indicate that that 80% of beneficiaries reported improvements of safety net operations during the previous 12 months, and that 99% of community member complaints on safety net programming were responded to under MAST management. Figure 25: IPTT VSLA group participation and establishment 2014-2017, beneficiary perceptions and complaint response rate 2018-2019 Indicator Target Actual % met IPTT 2014 -2017 20 Percentage of safety net participants in VSLA groups 50% 38% 77% 21 Number of VSLA groups established 1,200 1,241 103% IPTT 2018 -2019 20 Percentage of beneficiaries reporting improvement of safety net operations in the past 12 months 80% 80% 100% 52 Percentage of community members complaints on the safety net programming responded to under MAST management 100% 99% 99% EQ2.3: The likelihood that service providers will continue providing services after the project ends. Despite the planned discontinuation of activities under Kore Lavi and the well-communicated end date, the likelihood that service providers will continue providing services is tenuous in some areas. The ability of MAST to manage the SIMAST and continue providing the data and methodology as project targeting tools to others is dependent on future funding/payment for SIMAST data from other donors/NGOs/organizations, as well as the completion of the hand-over of the SIMAST system to the government. Vendors indicated that they will continue to provide services (which is simply the continuation of their business of selling goods, so not unexpected), though many respondents say that their revenue will be greatly diminished, as well as their ability to provide a wide range of locally produced goods as during the project implementation period. Additionally, On a positive note, the links between MAST and other donor agencies have been strongly reinforced by Kore Lavi activities, and MAST has had improved visibility, reputation, and relations with other actors. If IMPEL | Implementer-Led Evaluation and Learning 50 Findings the hand-over of SIMAST is completed as currently planned, then these linkages will help ensure that the data from SIMAST will continue to be in demand. Many respondents indicated that VSLAs will endure well beyond the end of the project. During the first four years of the project, Kore Lavi help start just over their target of 1,200 VSLAs. A 2018 study looking at the impacts of Kore Lavi on resilience25 reported that “resilient and vulnerable respondents who participate in a VSLA feel better prepared to deal with similar situations in the future”. This report also found that 17% of those interviewed in the study area expressed a desire to join a VSLA (if they were not already a member) during the phase-out period of the project (June-Sept. 2019). The general interest in VSLAs was already made evident, according to project staff, by the ‘numerous’ non-targeted communities had taken the initiative to set up VSLAs without external support26. The CADEC/CADEPs indicated they would continue beyond the end of the project, as they are deeply involved in the communities. The MFIs also indicated that their businesses have benefited in the long-term from the Kore Lavi activities, and they will continue to provide microfinancing after the end of the project. Figure 26: Qualitative data topic coding summary on services after project end (EQ2.3) What is the likelihood that service providers will CONTINUE PROVIDING SERVICES after the project ends? N = 45 (total topic code frequency under this question) Vendors will continue to provide service to the public 18 CADECs will continue beyond the end of this project because they are deeply involved in the community 10 MAST will continue to use HDVI and SIMAST, but will need funding for updates and maintenance. 5 Meres et peres leaders will continue to assist the beneficiaries 3 EQ2.4: The motivation of the community and beneficiaries to demand and pay (or invest time) for the services The VSLAs are the main activity/service that communities will continue to invest time and money in after the end of the project. VSLAs are, by their nature, voluntary organizations. As described elsewhere, VSLAs were already being spontaneously replicated in communities outside the Kore Lavi intervention areas. The knowledge gained through trainings was frequently cited by FGD participants as likely to have a lasting impact. The attitude and behavior change around nutrition, hygiene, breastfeeding, gender will be retained, as well as the training given to the mother leaders. However, the impacts on actual practice of some of these may not be sustained. For example, FGD participants frequently indicated that after the end of the voucher distribution, they will no longer be able to buy the more expensive local produce, and will revert to the less expensive, less diverse foods. 25 “Kore Lavi Beneficiary Assessment Report”, 2018. 26 Although the project staff was quite confident that the number of these community-initiated VSLAs were ‘numerous’, a precise estimate of the number could not be ascertained by the ET. Final Performance Evaluation of the Kore Lavi DFAP in Haiti Findings 51 One vendor indicated in an interview: “The beneficiaries will not be able to maintain the consumption habits after the end of the project. They won’t have the money to pay for these [fresh/local] food on a regular basis anymore.” One sentiment that came up in several unrelated KI interviews was the general sentiment towards social protection safety net programs. The success of Kore Lavi increased awareness of social protection safety nets as a concept and possibility within the government as well as the population in general. This is exemplified by the work on the PNPS (as discussed in other sections). Figure 27: Qualitative data topic coding summary on communities motivated to demand and invest in services (EQ2.4) Are communities and beneficiaries motivated to DEMAND AND PAY (or invest time) services? N = 98 (total topic code frequency under this question) VSLA is the only sustainable aspect of the project 40 Behavioral change promoted by training on nutrition, hygiene, breastfeeding, gender equity will be sustainable 19 Beneficiaries lack of economic means to continue buying local produce / vendors will lose customers 14 The financial capacity of VSLA will decrease significantly after project closure. They will continue to need support 7 Training given to meres leaders will be sustainable 6 Raising goats, poultry and pigs 4 Project monitoring data from the 2018-19 IPTT indicated that 26% of VSLA members started income generating activities. While impressive, this was short of the goal of 50%, and the definition of this indicator was unclear to the ET. Additionally, the data on the actual numbers of safety net beneficiaries who started income generating activities through VSLAs was only 136, which while nearing the goal of 150, seems to be a very low number considering the large number of beneficiaries overall. Figure 28: IPTT VSLA members starting income generating activities 2018-2019 Indicator Target Actual % met IPTT 2018 -2019 28 Percentage of Safety net beneficiaries (VSLA Members) who start income generating activities 50% 26% 52% 60 Number of Safety net beneficiaries who start income generating activities through VSLA 150 136 91% EQ2.5: Whether the necessary resources and capacity strengthening will exist to sustain service providers. It is difficult to accurately predict if the resources will exist to sustain service providers. It is clear that overall, the project would require many more years of investment for any social protection system to IMPEL | Implementer-Led Evaluation and Learning 52 Findings gain legs in Haiti and for MAST (and the SIMAST) to expand and improve to take full responsibility over the management of a national level social protection system27. This cannot be attributed to the early end of the project, but rather to the overly ambitious goals of sustainability set at the start. It should be noted that the sustainability goals were not so ambitious as to aim to establish a fully-functioning national level system in four years. Indeed, the initial proposal for Kore Lavi states “it will likely take a decade to establish a well0functioning, sustained voucher-based safety net”28. IPs indicated that they will continue to support the VSLAs and CADECs, though numbers around this continued support were not available. WFP has funds to continue to support the SO1 and SO4 activities that will hopefully result in the eventual hand-over of the SIMAST. Finally, the EU and other donors appear to be interested in funding SIMIST and the Kore Lavi model. Data around the perceptions relative to this question was only discussed in KIIs (mainly CADEC, CASEC, and vendors). Though no singular viewpoint stood out in these discussions, they generally indicated that additional resources and training will be needed to continue providing services such as the SIMAST. EQ2.6: The extent to which the projects leveraged other USG and non-USG investments to achieve sustained outcomes as identified in the theories of change. Kore Lavi has been relatively successful in gaining the interest and potential funding from other actors. As discussed in other sections, other donors and organizations such as the EU, Swiss Cooperation, and possible the WB and others have committed to using the SIMAST for their food security projects (or expressed interest in doing so). While these projects may be beyond the scope of strict social protection activities, they are taking advantage of the work of Kore Lavi/MAST and expanding on it for their projects. The drawback of this approach currently is that there is not a smooth transition from Kore Lavi to MAST management, following the path towards expanded social safety net programs managed and funded by the GoH that was originally envisioned by Kore Lavi. The shorter-term funding of SIMAST for project-specific purposes risks shifting/altering the focus of the system. The active engagement of many actors by Kore Lavi/MAST in the drafting of the PNPS is another way in which the project has been very strategic in leveraging non-financial investments of other actors. The respondents consistently spoke about the well-executed, inclusive process in drafting the policy. Despite the current political turmoil in Haiti, the hopeful official passage of the document seems likely, and if so, this will be a significant accomplishment and legacy of the work done under the project. The multiplier effect of the money that was injected into the local economy by the vouchers and the impact of the VSLAs was another way that respondents identified when asked about leveraging other investments for sustainable outcomes. This was particularly interesting, as this implied that some 27 The Kore Lavi mid-term evaluation also made it clear that a four-year donor initiative is insufficient to establish a well￾functioning national safety net program, and acknowledges that a national level program was not the goal of Kore Lavi. It cites Brazil’s Bolsa Familia social safety net initiative, as an example, which has been receiving significant financial ($572 million from 2004 to 2009) and technical assistance support for the past twelve years. Source: http://www.worldbank.org/projects/P087713/br-bolsa-familia-1st-apl?lang=en&tab=overview 28 Annex 20: A VULNERABILITY TARGETING & SAFETY NET VISION FOR 2023 Final Performance Evaluation of the Kore Lavi DFAP in Haiti Findings 53 respondents viewed the various service providers and project participants (producers, vendors, VSLA members etc.) were also investors in the system. According to one IP interview, “There was a multiplier effect by injecting money into the local economy. The market for local produce grew, as did the sales of food vendors”. Figure 29: Qualitative data topic coding summary on Leverage of other investments (EQ2.6) To what extent did the project LEVERAGE other USG and non-USG investments to achieve sustained outcomes? N = 40 (total topic code frequency under this question) Post disaster, the project provided tools for sanitation activities, seeds and increased coupon money 13 Other donors will use and support SIMAST (WFP, EU, Cooperation Suisse, CRS, Mercy Corps) 9 The national policy on social protection was well done and will be a sustainable change. 4 VSLA groups are supported with funds from members, public and private institutions 3 There was a multiplier effect by injecting money into the local economy. The market for local produce grew, as did the sales of food vendors. 2 GOH has taken over SIMAST and will ensure its sustainability, sharing it with other ministries 2 EQ2.7: Evidence of enhanced linkages with other service providers. Kore Lavi achieved some successes l in creating linkages with other partners and service providers, enhancing the project through these synergistic collaborations, and helping to lift up many stakeholders. Kore Lavi worked with many different ministries, local government entities, and local CADEP, CASEC, CADEC groups. The visibility of MAST was greatly increased, and the project served to give MAST a ‘seat at the table’ in collaborative decision making, such as the IPC work led by CSA. The work with the MFIs and vendors is also an area where enhanced linkages allowed efficient responses to emergencies. One IP interview indicated “The project was able to respond after hurricane Matthew by identifying and targeting beneficiaries for cash distributions. Thanks to existing partnership with a local MFI, the project was able to cash to out to the village level quickly.” Figure 30: Qualitative data topic coding summary on linkages with other service providers (EQ2.7) What is the evidence of enhanced LINKAGES with other service providers? N = 9 (total topic code frequency under this question) CADEP will continue new partnerships developed thanks to KORE LAVI, especially with the state authorities such as MAST 2 The seller will continue relationships with voucher recipients and suppliers. 3 IMPEL | Implementer-Led Evaluation and Learning 54 Findings What is the evidence of enhanced LINKAGES with other service providers? N = 9 (total topic code frequency under this question) Technical working groups created by the project will continue to allow for knowledge sharing between organizations involved in SP 2 However, CARE as consortium lead did not successfully define a common approach to working with community structures/volunteers for the project that was agreed upon, understood, and adopted by all consortium members. There was inconsistent approaches by each partner (and even between geographic areas managed by the same member) in how community leaders were approached and how communities were engaged. For example, identifying inclusions and exclusions of participants was done very differently between partners, leading to difficulties in interpreting and comparing this information. Furthermore, according to the IPTT data, the outputs related to linkages and community structures were not fully successful. There was no reported linking of safety net participants with complementary services (other than VSLAs) in the 2014-2017 IPTT data. Additionally, some of the output indicators of community structures participating in monitoring or engaging in leadership processes with MAST and local authorities did not achieve the targets. Figure 31: IPTT complementary services 2014-2017, linkages and community structures 2018-2019 Indicator Target Actual % met IPTT 2014 – 2017 22 Number of safety net participants linked with complementary services (other than VSLA groups) 3,600 0 0% IPTT 2018 – 2019 17 Number of applications integrated and modifications made to SIMAST to improve linkages with other systems 1 1 100% 24 Number of community-based structures participating in monitoring of frontline safety net activities 150 97 65% 46 Percentage of community structures (CADEPs and other similar structures) engaged in community social audit (SA) and leadership processes in collaboration with MAST and local authorities 88% 54% 61% 4.3 EQ3: What are the strengths and challenges of the selected interventions and their implementation, and how are these received by the target communities? The ET evaluated the effectiveness and relevance of the technical interventions to achieve project outcomes, and relates those findings to the theory of change. Determinations are supported when discussing the following: • Factors in the implementation and context associated with greater or lesser effectiveness in producing Outputs of higher or lower quality; • Interventions and implementation processes deemed more/less acceptable to members of the target communities. Final Performance Evaluation of the Kore Lavi DFAP in Haiti Findings 55 • This EQ evaluates the effectiveness and relevance of the technical interventions to achieve project outcomes, and discusses those findings in relation to the Theory of Change. EQ3.1: Factors in the implementation and context associated with greater or lesser effectiveness in producing outputs of higher or lower quality. The ET interpreted this question as an investigation of the internal and external contextual factors that had an impact on Kore Lavi’s activities. Broadly, FGDs and KIIs consistently stated their appreciation for the quality of project services. Looking at internal factors, the most frequently cited was the good involvement of key players, such as the CADEC, local authorities, vendors, etc. which contributed positively to the project. Some respondents discussed challenges in the project management, including tensions within the consortium, stemming from different visions and approaches. In many interviews, when discussing these sorts of internal collaboration challenges, the respondents would clarify that these problems were more prominent at the start of the project, and lessened with time and effort. Other respondents highlighted internal factors linked to project design, such as insufficient coverage, the voucher package being too small, the lack of a livelihoods component, or the lack of entrance/exit mechanism for the voucher benefits. Finally, the low institutional capacity of MAST at the start of the project was a challenging internal factor, particularly at the beginning. The main external factor impacting the project was the deteriorating food security situation, particularly towards the end of the project. While Kore Lavi was able to adjust the voucher value to account for inflation of the Haitian gourde and increasing food prices, and in response to shocks such as hurricanes, this was not always timely nor did it fully compensate for these shocks. Furthermore, other contextual factors such as heightened insecurity and rising fuel prices negatively impacted the economic situation of all Haitians. According to one vendor interviewed: “Insecurity and political unrest have been the biggest constraints in … business. …stabilizing the country would facilitate the continuity of our economic activities.” Figure 32: Qualitative data topic coding summary on context and factors that impacted outputs (EQ3.1) What factors in the IMPLEMENTATION AND CONTEXT were associated with greater or lesser EFFECTIVENESS in producing outputs of higher or lower quality? N = 92 (total topic frequency under this question) The involvement of key players (CADEC, local elected officials, salespeople, etc.) contributed to the success of the project 10 Tensions in the consortium resulted from different visions and approaches 8 The project should have reached more people 6 There were problems initially, but the project has improved over time 7 Vendors often raised the price of produce 5 Lacking a livelihoods component 6 The project did not have entrance/exit mechanisms where beneficiaries could join or leave the project. 4 The coupon package was too small/too limited 4 IMPEL | Implementer-Led Evaluation and Learning 56 Findings EQ3.2: Interventions and implementation processes deemed more/less acceptable to members of the target communities. When asked what interventions the target communities found more or less acceptable, the vouchers and the VSLAs were the most frequently mentioned. VSLAs were often discussed in the context of women’s empowerment through access to credit and income. Voucher recipients felt that vouchers were a more dignified and flexible way to receive food distribution. There was also a great appreciation of the local food, stemming from the education on the benefits of a diverse diet, as well as the appreciation of the financial inputs staying in the community and benefiting more than just the voucher recipients. However, as discussed in previous sections, there was a strong demand to increase the value of the voucher package as well as the coverage. Some respondents also cited SO3 activities as positively viewed by members of the community, such as the capacity building of the system of mother and father leaders. Figure 33: Qualitative data topic coding summary on intervention acceptability to communities (EQ3.2) What interventions and implementation processes deemed more/less ACCEPTABLE to members of the target communities? N = 73 (total topic frequency under this question) The distribution of coupons was appreciated 16 VSLAs have enabled pooling of savings, women's empowerment, access to credit and the creation of income-generating activities 15 The system of meres and peres leaders 12 Capacity building of vendors, meres and peres leaders 9 The IPTT figures show the outputs in terms of percentage of food accepted for voucher redemption that is locally grown, and the number of vendors providing food under the safety net, met their targets. Figure 34: IPTT voucher redemption and number of vendors, 2014-2017, 2018-2019 Indicator Target Actual % met IPTT 2014 – 2017 18 Percentage of food accepted for voucher redemption that is locally grown 80% 83% 104% 19 Number of vendors who are providing food under the safety net 1,000 998 99.8% IPTT 2018 – 2019 25 Percentage of food accepted for voucher redemption that is locally grown 83% 83% 100% Final Performance Evaluation of the Kore Lavi DFAP in Haiti Findings 57 4.4 EQ4: What are the key lessons learned and best practices that should inform future projects in the country? During the course of research, the ET sought to identify the following: • best practices, • strengths, • challenges in the project designs (including theories of change), • approaches that should be considered favorably in designing future food and nutrition security projects and strengthening household and communities’ resilience capacities. These issues are presented in brief in the following two questions on the two sub-evaluation questions. The key findings in terms of recommendations for future project design, drawing from critiques of Kore Lavi as well as areas of success, are presented in the section on conclusions and recommendations. EQ4.1: The unintended positive and/or negative consequences of the projects. Several unintended consequences were identified during the evaluation research, both positive and negative. However, the majority related to either unintended positive spillover effects, or to other aspects of the project that were not in the original design. When asked about consequences of the projects, FGD participants often indicated that the vouchers allowed them to save money for other non-food expenses, such as school fees or livelihood related expenses. The fact that they were consistently receiving the voucher support over a long period of time was discussed as being of particular importance; the regularity allowed the participants to plan with more certainty. One regional MAST staff member noted this positive impact when he stated: “People used to line up at departmental MAST offices every Monday morning to ask for food and money. Since the project has been operating in the commune, they don’t come by the office for handouts anymore.” Respondents also identified the positive effects on agricultural and animal production from the increased demand for local fresh foods, and a reinvigoration of the markets. Some respondents indicated that non-participating vendors in the markets expanded their selection, including the addition of more local, fresh goods when they saw the demand increasing among the participating vendors. MFIs cited increased business from better visibility and recognition in the communities. Specific to the impact of the projects on improving livelihoods of the participants, the FGD participants generally (though not unanimously) agreed that the vouchers were insufficient to allow them to engage in new/improved livelihoods (such as the possibility to invest in starting new business, expanding existing ones, or acquiring productive assets). However, the FGD participants did not express this insufficiency as a failure of the project, but simply that greater investments would be needed to have marked positive long-term impacts on improving livelihoods. Some respondents indicated that the vouchers created ‘laziness’ or ‘dependency’ among the recipients. However, this was far from the majority of respondents. Though there may have been isolated cases or IMPEL | Implementer-Led Evaluation and Learning 58 Findings examples of this, the data indicate this was not widespread, and the ET did not consider this an issue of concern. Indeed, participants indicated that the vouchers were not actually sufficient to address all needs, and that other opportunities, such as livelihood or agriculture support, were lacking in the project design. The problem, according to one IP interviewed, is that “there are no other opportunities for [the voucher recipients] to move on to.” As discussed previously, voucher recipients described the end of the project as a significant shock-type event. Although the communication around the end of the project was well done, some participants still did not feel they were prepared to continue better off than before they received vouchers. One vendor indicated: “Since the project closed, I think that people's vulnerability problems are resurging because they cannot really support themselves.” It should be noted, however, that this perception, while unintended, appears to be linked to the worsening food security situation in Haiti at the time of the end of Kore Lavi and the evaluation, rather than necessarily a project design flaw that brought about these unintended consequences. The unrealistic expectations set up at the inception of Kore Lavi had the unintended consequence of setting up the project to be perceived as less of a success. The government hand-over in particular was extremely unrealistic, and led to some sense of failure or shortcoming at MAST. A few KIs identified the source of these unrealistic expectations as originating in the initial design process itself. They indicated that the design was in large part already spelled out in the RFA, and it was simply up to the IP to figure out how to implement it. However, other key informants described a very inclusive process in the initial design of the project. It is difficult to accurately evaluate how the process that took place seven years prior was in truth carried out. Additionally, the sharing of food was not well planned out in the design of the project. Project documents indicated that the food rations distributed as part of SO3 did not take sharing into account. Sharing is a very common and expected practice in Haiti, and yet certain SO3 activities were critiqued because of this sharing of food. It should be noted, however, that at the time of the design of the project, the latest guidance from FFP29 recommended individual rations as opposed to family rations. The same was true to a lesser extent of the SO2 vouchers, which were also frequently shared. Indeed, the sharing of benefits can be considered a positive consequence in some ways. Some respondents indicated that households gained social capital when sharing their voucher benefits with others. One IP indicated that: 29 Food Aid and Food Security Assessment II, produced by FANTA, was officially released in March 2013. Even if the guidance was not officially provided by Food for Peace through the RFA, the timing of this report during the development of the proposal for Kore Lavi would have influenced the way in which the submission was designed. The report found that there was little improvement in nutritional impact with the provision of family rations, and at times less impact. There was a recommendation that supplementary nutritional rations be provided to mothers and children under two until/unless additional research provided an indication that the contrary were true as there were questions regarding the cost effectiveness of preventive rations. Final Performance Evaluation of the Kore Lavi DFAP in Haiti Findings 59 “People in the community know who project beneficiaries are and when they get food [vouchers]. There is a sense of pride among beneficiaries that they are able to share food with others, rather than have to borrow food.” Figure 35: Qualitative data topic coding summary on unintended positive/negative consequences (EQ4.1) What were the unintended positive and/or negative CONSEQUENCES of the projects? N = 132 (total topic code frequency under this question) Because of the vouchers, beneficiaries were able better plan for expenses and to save money to start a business, buy animals, fix up their houses and pay school fees. 11 Increased agricultural and animal production and reinvigorated local markets 11 Thanks to the strengthening of the financial capacity of the VSLA, beneficiaries learned to save money for business, gardens animal raising and family emergencies. 10 Access to credit at a low interest rate 8 Many improvement in the promotion of women's rights (less domestic violence, more responsibility in the home) 8 Creation of income generating activities 6 Distributed of vouchers contributed to strengthening the financial capacity of VSLA groups 6 Vouchers created dependency on free food and laziness among beneficiaries. 6 The drafting of the PNSP was a positive consequence of Kore Lavi that was not originally envisioned at the start of the project. It is interesting to note that while key informants indicated that it would have been ideal to have the policy in place before the Kore Lavi project, it was in fact because of the Kore Lavi activities that the various social protection stakeholders were more receptive to the drafting of the policy. Kore Lavi in effect was a necessary precursor to set the stage for work on the policy document. EQ4.2: Ways to minimize potential unintended negative consequences and systematically capture positive consequences. This sub-question is interpreted as a discussion of best practices, strengths, and challenges in project design, specific to Kore Lavi and lessons learned that can be applied to future food and nutrition security projects in Haiti and elsewhere. Several points related to this are presented more in-depth elsewhere in the findings, but are summarized here. The findings on best practices, strengths, and challenges are further expanded upon in the conclusions and recommendations section. Several approaches surfaced during the evaluation as recommendations for future food and nutrition security projects. These fall under a few different categories: 1) some are lost opportunities of Kore Lavi which future projects can learn from, 2) some are areas where Kore Lavi learned lessons and applied them successfully, and 3) some were considered successes from the conception through the execution, and merit recognition and replication. IMPEL | Implementer-Led Evaluation and Learning 60 Findings Overall, the ET found the project design and implementation to be innovative (and/or using developing new best practices and applying them to Haiti), and successful in meeting many of the ambitious objectives. Furthermore, when some of the overly ambitious objectives are not used as the absolute benchmark for success, then Kore Lavi has many areas where it has excelled. Indeed, the project has gained the interest and investment of other donors/actors, who are using it as a model to help with targeting of other projects. The respondents often cited lacking or missing components of the project when discussing this topic. The most frequent comments were around the lack of support for livelihood activities (agriculture, livestock, etc.) as a component of Kore Lavi. Others discussed the need for more value of the vouchers. One important aspect discussed with a few of the key informants was the use of adaptive management in the project. Because adaptive management was only officially adopted as standard practice after the start of KL, it was not written explicitly into the project documents. A potential consequence was some perceived reticence of the IPs to approach FFP with proposed alterations/modifications to the project, despite the best efforts of FFP to encourage this, and instead they focused more on on ‘checking the box’ in terms of pre-established output goals. This focus on ‘checking the box’ also may have contributed to some exclusion of MAST from the processes, because there was a feeling in some cases that fully involving MAST would slow down the process too much. One IP staff interviewed indicated that when they came on to the project, MAST was not fully involved at the strategic level, so they made the decision to involve MAST at EVERY step, accepting the consequence of slower progress, indicating “they would succeed or fail together”. Despite moving slower, the respondent was very clear that that sacrifice was worth it. Figure 36: Qualitative topic coding summary on ways to minimize negative consequences, capture positive (EQ4.2) What are the ways to MINIMIZE NEGATIVE CONSEQUENCES and systematically capture positive consequences? N = 169 (total topic code frequency under this question) Support to agriculture and livestock would have an impact on sustainability and facilitate access to local products 19 Create "cash for work" jobs, especially for women 16 Strengthening VSLA (administration, materials and meeting space) and training of members 10 Add in livelihood and credit fund to the project 9 Focus on women-headed households, elderly, disabled, single parents, malnourished and city dwellers 8 The increase the number of beneficiaries and departments reached 8 Contact community leaders to implement needs analysis prior to project execution 7 The GOH should have a larger role in the design and management of future projects. 7 Training on equality and gender equity and women's empowerment 5 Final Performance Evaluation of the Kore Lavi DFAP in Haiti Findings 61 What are the ways to MINIMIZE NEGATIVE CONSEQUENCES and systematically capture positive consequences? N = 169 (total topic code frequency under this question) Activities to protect the environment such as soil conservation 5 Increase the amount of vouchers 5 No graduation built into the system 5 Consortium should be managed more transparently, and flexibly. Roles should be more clearly defined. 5 Further discussion related to unintended negative consequences and ways to capture positive consequences are further discussed in the section on Conclusions and Recommendations. 4.5 EQ Cross Cutting In addition to the above EQs covered above, two cross-cutting questions were considered when presenting the findings across all EQs: • Did the Consortium fully integrate gender across the project as was originally conceived in the project? • What were the benefits and lost opportunities of the consortium approach? While the ET attempted to address these cross-cutting questions across all the EQs, this section is included to consolidate the findings and highlight the key findings specific to gender integration across the project. EQ Cross Cutting: Gender Integration across the project The project originally conceived to integrate gender equality into each component of the project design, as a cross-cutting component to the project. This included goals related to female participation at all levels, from planning to project management and implementation, to decision making within the household. From the conception, Kore Lavi benefited from a beneficial partnership with MCFDF (Ministry of Women and Women’s Rights). MCFDF supported Kore Lavi in drafting a strategy for gender equality for Kore Lavi, and they provided recommendations on project operations and targeting from a gender perspective. As also noted in the mid-term evaluation, however, there appeared to remain some tensions related to funding at MCFDF, and their role during the extension period was not clear to the ET30. To assess gender roles among the direct and non-direct participant populations, the ET made an effort to focus on women where possible with the FGDs. 173 of the 257 people that took part in FGDs were women. Of the 25 focus groups conducted, four were with men only, 14 with women only, and 5 were with men and women together. To assess gender roles within Kore Lavi, the ET made an effort to identify and interview female key informants for interview. However, for the key informants interviewed by the ET, only about one quarter were with women (16 out of 63 KIIs). Although these women came from the range of different types of key informants, they were always fewer in number in each organization/entity than the men interviewed (with the exception of USAID). While this sample is not necessarily indicative of the gender 30 The ET was not able to meet and interview a representative of MCFDF despite all due diligence. IMPEL | Implementer-Led Evaluation and Learning 62 Findings ratios within the project and project partner staff, it does indicate that men tend to be in key leadership positions more than women across the project. Staff at CARE in Port-au-Prince underlined the importance of the role of the staff focused on gender within Kore Lavi, but lamented the fact that no gender focused staff were hired at the department level. One GOH staff member interviewed made it very clear that the project’s gender inclusion strategy was very important, that this focus cannot be left to happen ‘naturally’. The KI went further to suggest that fixing gender quotas for project staff may give better results, rather than simply making such an attempt without fixed targets to comply with. This suggestion may be valuable in future project design to ensure gender balance within the staff and leadership roles, either through target values (which the project success can be measured against), or through enforced quotas in hiring. However, organization-specific gender hiring policies will have to be considered and accounted for if a quota system is desired. While the project monitoring data from the IPTT specific to gender reached or made significant progress towards their targets, few monitoring indicators were designed specifically to monitor gender balance in the project and project activities. There were only five total reported-on indicators in the IPTT tables that were specific to gender between the two reporting periods, and only three of those measured gender balance in project activities. Two female KIs from two of the IPs separately indicated that they felt gender related objectives were not fully integrated into the project from the beginning. Rather, the project was adapted over time to better incorporate a gender perspective and focus on the needs of women. Figure 37: Gender specific IPTT indicators IPTT Reporting 2014-2017 LOA Target LOA Actual % of Target Achieved Indicator 50% 49% 98% Percentage of vulnerability recourse committee members who are women IPTT reporting 2018-2019 LOA Target LOA Actual % of Target Achieved Indicator 16 18 113% Number of community-level activities integrating dialogues on gender equality and women’s empowerment 80% 87% 109% Percentage of vendors providing food under the safety net who are women 144 150 104% Number of group discussions organized on gender equity and equality with participation of the safety net beneficiaries 45% 34% 76% Percentage of participating community structures' members who are women However, the IPTT did report several of their indicators disaggregated by gender (though without gender-specific targets). The large majority of people that received nutrition-related professional Final Performance Evaluation of the Kore Lavi DFAP in Haiti Findings 63 training (2018/19 reporting period) and trained in child health and nutrition (2014-17 reporting period) were women. Women made up 69% of the people benefiting from social assistance programming during the 2014-17 reporting period, and about 54% during the 2018/19 reporting period. The gender trainings were identified by several KIs as well as in the majority of the FGDs in a positive light. Many people indicated that they appreciated the gender trainings, and that it was a welcome component in a generally male-dominated society. However, this sentiment was usually expressed in the FGDs with only women (only once was it discussed in a mixed male/female FGD). The VSLAs were often discussed in FGDs and some KIIs as having a positive impact on women, giving them better access to credit and improving their economic autonomy31. The Kore Lavi mid-term evaluation also found similar sentiments from VSLA members, who underlined the VSLAs as creating opportunities for women to take on leadership roles in their communities. Both this evaluation and the mid-term noted that despite this, however, women were still underrepresented in VSLA leadership. The project monitoring data indicate that 87% of vendors providing food under the safety net are women (2018/19 reporting). However, the majority of vendors in Haiti are women, so while the fact that more women vendors benefited from the Kore Lavi voucher program, this is not necessarily due to the project design itself, but simply consequence of existing gender roles. Despite this fact, however, benefiting vendors means supporting and empowering women whether or not the project influenced the gender balance in the vendor beneficiaries. 31 It should be noted, however, that this was almost uniquely expressed by women IMPEL | Implementer-Led Evaluation and Learning 64 Conclusions and Recommendations 5. CONCLUSIONS AND RECOMMENDATIONS The conclusions and recommendations presented here are broadly organized into three topics: 1) areas where Kore Lavi was particularly successful from design to implementation and had positive impacts (even if areas still exist for improvement), 2) areas where Kore Lavi adapted its programming to address challenges that came up during the life of the project, providing lessons learned that can inform future projects, and 3) areas of missed opportunities, where Kore Lavi may have come up short or learned the lesson too late to fully implement meaningful and impactful changes. Positive Consequences of Kore Lavi 1) The inclusion of local micro-finance institutions, food vendors, and use of locally produced foods appears to have provided an important multiplication of inputs and impacts. However, the vendor related outcomes were not carefully measured/quantified. Future programming should better monitor this level of impact. The use of local MFIs in the voucher reimbursement and vendor monitoring was very successful. The MFIs gained more visibility and business with local vendors in need of revolving credit, and at better interest rates that are often otherwise available. The local vendors were viewed not only as service providers in the project, but also as beneficiaries. The vendors, particularly the fresh food vendors, while not in extreme poverty, are still often quote poor. Participation in the project appears to have allowed these vendors not only to gain more business, but also benefit from training in hygiene and basic accounting practices. While the vendors will lose business after the end of the project, it was generally thought that the positive impacts on them would be sustained in the long term. The restriction of the applicability of the vouchers to local products ensured the infusion of money into the economy would not immediately leak out of the country. In some cases, the immediately local production could not keep up with the increased demand, forcing vendors to source from neighboring areas of Haiti. Supporting agriculture would further ensure the assistance remains in the targeted communities and has even more impact. Despite the information indicating these successes, it should be noted that they are primarily anecdotal/qualitative. There is a lack of solid evidence measuring impacts on vendors, such as increase in vendor income, job creating, impact on vendors’ business practices, impact on the food and economic security of the vendor households, etc. Future projects should include this kind of data in the monitoring and evaluation design in order to better monitor, refine, and improve this potentially important and impactful facet of the activities. 2) Collaboration with many stakeholders, internal and external, on the construction of the information system is important to get buy-in. The SIMAST has been a very successful component Kore Lavi. The process may have been even more successful if there was a clearer long-term vision shared by stakeholders from the start. However, there are clearly challenges at gaining interest and buy-in when a system is not yet proven. The census approach to targeting used by Kore Lavi was considered by some as difficult to impossible to implement in Haiti, until it was successfully implemented by Kore Lavi. The SIMAST is impressively projected to have collected information on 30% of Haiti’s population by 2020. The SIMAST has opened the eyes of many stakeholders about the significant possibilities that actually exist in this regard. As of Q3 2019, SIMAST is being used in three other projects outside of Kore Lavi. Additionally, concrete plans are in place to fund and use the SIMAST in other projects, such as European Union (EU) Final Performance Evaluation of the Kore Lavi DFAP in Haiti Conclusions and Recommendations 65 funded 11th European Development Fund (11th EDF, or 11eme FED), as well as interested expressed by the Swiss Cooperation and the World Bank. 3) The use of the HDVI as a targeting indicator performed well overall in the Kore Lavi context. A review of the HDVI conducted in 2018 concluded that the HDVI is a good proxy for chronic food insecurity, although not an ideal proxy for acute food insecurity. This being said, people interviewed as part of this evaluation generally agreed that the HDVI functioned adequately as a targeting tool. Although informants said that there continued to be some degree of inclusion and exclusion error, these errors were not considered to be excessive or of large concern. There are also some lessons learned related to the HDVI (see recommendation further below). 4) Allow for adaptable voucher value to adjust for inflation or respond to acute shocks. The Kore Lavi voucher system provided a quick and efficient way to provide additional assistance to beneficiaries following acute shocks, such as drought or storms. Additionally, Kore Lavi was able to adjust the value of the vouchers to follow the inflation/increase in food prices that were experienced during the life of the project. FGD respondents expressed their appreciation that this adjustment was made. There was a need, however, to better streamline this adjustment of voucher value so that it could respond more nimbly to inflation/prices changes. 5) Include formal and accessible communication channels with the participants, such as a complaint line. Kore Lavi was very successful at providing complaint channels and addressing all complaints that came through these channels. This helped increase transparency and accountability. 6) Include CBOs in planning to ensure community engagement. Kore Lavi generally included the CBOs in the planning, though some critiques of the exclusion of these partners at the beginning were expressed. One of the reasons this may be is that the call for proposals itself outlined some of the project components, thus not allowing a collaborative engagement to envision the project to start. 7) The increased focus on capacity building at MAST was strongly appreciated by the GOH staff, and viewed as a vital support to helping them make progress towards the government’s social protection goals. 8) Continue to use VSLAs as a complementary intervention. The addition of VSLAs as a complementary component of the project was immensely successful in Kore Lavi. It appears that VSLAs are most successful when combined with other activities, such as the voucher distribution, or possibly with livelihood building activities, with which they have synergistic effects. As a stand-alone project, respondents indicated that their impact may not be as significant, particularly on the poorest households. Indeed, some households indicate that after the end of the voucher program, the feared they would no longer have the available resources to participate in the VSLAs. However, the extent to which this would happen was not quantified. The VSLAs are considered a very sustainable part of Kore Lavi that will continue to have an impact long after the life of the project, though they may not benefit the poorest after the end of the voucher benefits. IMPEL | Implementer-Led Evaluation and Learning 66 Conclusions and Recommendations 9) A well-defined consortium provides benefits by creating synergies between partners. Despite some challenges with the consortium, most respondents indicated that overall the consortium had more benefits than drawbacks. Kore Lavi did experience some struggle in maintaining a collaborative environment between consortium partners while still keeping strong and dynamic leadership. It is important, to ensure an appropriate combination of partners that will make up a consortium, and to carefully define their roles and responsibilities from the start, informed by well thought out project design, and their roles and responsibilities are well defined. In the case of Kore Lavi, some of the benefits of the consortium were not fully utilized. For example, CARE has extensive experience in social protection projects in other countries, such as Ethiopia. Yet there was very little opportunity provided by CARE to their Haiti office which would have allowed them to benefit from this. Additionally, some members of the consortium seemed to have slightly different priorities. Some were more concerned with ‘checking the boxes’ in terms of completing outputs, while others were more willing to sacrifice some degree of project performance progress in order to move forward only with the GOH and other partners in stride. 10) Final evaluations should take place before end of project. The final evaluation was extremely well timed, taking place at the very end of the project, but before key staff had left and beneficiaries and service providers became less identifiable. This greatly enhanced the quality of the data, and the ability to collect it. Challenges, Adaptations, and Lessons Learned 11) The use of IPTT template needs to have adaptability/flexibility built in to measure outputs and impacts of the governance portions of social protection projects. While the template has a set of ‘required if applicable’ indicators, it also asks the IPs to propose other indicators that fully capture progress/success of the activities. In the case of Kore Lavi, the use of the IPTT appeared to have been done from a food and nutrition security perspective, but then used to measure what was in large part a governance project. This failed to capture important aspects of project impact. The incorporation of additional indicators of institutionalization and government capacity growth during the extension period of Kore Lavi, however, was an important modification to address this problem. 12) Develop an appropriate plan and metric to monitor project impacts on governance and institutionalization. Related to the above recommendation on the use if the IPTT, the addition of a capacity measurement by Kore Lavi was a very positive innovation. The project recognized the need for an objective measurement of the capacity of MAST, and so developed the MAST global institutional capacity index. This (or similar) indicator may be of interest to measurement of other governance capacity building projects. 13) The drafting of the PNSP was a significant acheivement of Kore Lavi, despite the fact that it was not originally envisioned at the start of the project. While in theory, it would be ideal to have the policy in place before the piloting of a social protection program, it was in fact because of the Kore Lavi activities that the various social protection stakeholders were more receptive to the drafting of the policy. Kore Lavi in effect was a necessary precursor to set the stage for work on the social protection policy document. Implementation of the policy is the next step, one that will be challenging given the weak national government. Final Performance Evaluation of the Kore Lavi DFAP in Haiti Conclusions and Recommendations 67 14) Prioritize early strengthening of government policy related to social protection, and place the responsibility with the appropriate type of organization. The initial project design aimed at institutionalization of the social protection activities without considering the need for a national policy on social protection. However, Kore Lavi did recognize this need and added policy support/advocacy later in the life of the project. Furthermore, this policy discussion and drafting was consistently viewed as a high-quality, inclusive process. The critique of the late start to the policy work should be tempered with the fact that many respondents indicated that although the policy work would ideally come before or jointly with implementation, it is very likely the policy work would not have progressed in the absence of Kore Lavi’s example of what a successful social protection program can offer. One of the challenges at the start of the social protection policy work of Kore Lavi was who, within the Kore Lavi consortium, was leading the process. NGOs, with the support of FFP, are the appropriate entities to lead the prototyping process of social protection delivery systems. NGOs may also have an important role in advocating for national policy. However, the drafting and writing of a national social protection policy is not generally a role of most NGOs (being non-governmental organizations). The ET felt that the choice of WFP to take the lead on policy development guidance was correct (among the consortium partners). However, the question remains if there was a more appropriate organization (outside the Kore Lavi consortium) that would have been a more natural fit, if this had been considered when first identifying the consortium partners. FFP needs to think well ahead of time on the role of/need for national policy when piloting prototypes of social protection programs, and identify which partnerships FFP and the IPs would be most strategic for this work. 15) Consider cultural context (e.g. food sharing) in project design. There were some inconsistencies in the perception of sharing of benefits between project participants and non-participants, particularly in the case of SO3 activities. Sharing of food in Haiti is a known and very strong cultural norm, particularly between family members (when the food assistance is targeted to an individual), but also between households within the community. The sharing of individually￾targeted food assistance between family members (or with other households) risks diluting the impact on the targeted individual. Further research is needed to accurately assess if food assistance projects would be better served to supply family rations even when targeting individuals such as pregnant/lactating women, to better understanding if/how sharing between families might dilute the impacts of family-targeted food assistance, and if/how between￾household sharing may grant the household more social capital and thus building their resilience. 16) Project monitoring and evaluation should take into consideration the external influences that may affect impacts. Kore Lavi was unfortunately scheduled to end during what was a very challenging period of food insecurity in Haiti. In additional to adding to the severity of the perceived ‘shock’ by the project participants at the end of the project , this may also result in some of the food security outcome indicators showing less impact that under a more ‘normal’ food security situation. The SO2 impact evaluation (planned to take place from late 2019 to late 2020) will take place in a context where the negative impacts of the current crisis may far outweigh any of the improvements in food security/resilience status of households that had benefited from vouchers and other complementary activities. As such, any findings of the impacts of Kore Lavi need to be carefully interpreted; and the SO2 impact evaluation question itself may need to be modified to ask “What are the impacts on the on the resilience of voucher- IMPEL | Implementer-Led Evaluation and Learning 68 Conclusions and Recommendations recipient households after the end of the project, during a period of increased food insecurity?” This SO2 impact evaluation is planned to take place from November 2019 to December 2020. 17) The most frequently cited complaint in the project process quality was timeliness of the payments to the MFIs (who then in turn reimbursed merchants for the fresh food vouchers they took as payment). Despite the fact that the consortium generally communicated that the payment delay issues were eventually resolved, this challenge appeared to be chronic/recurring. One interviewed MFI indicated that their payments had been regular and timely until the transition in IP management that happened during the extension period, during which time their payments became consistently late, leading to late payments and frustration among the participating vendors. While the ET was not able to fully understand why this problem persisted, it was likely due in part this shift in management. The monitoring of payments in future projects must be very carefully done. 18) When asked about consequences of the projects, FGD participants often indicated that the vouchers allowed them to save money for other non-food expenses, such as school fees or livelihood related expenses. The fact that they were consistently receiving the voucher support over a long period of time was discussed as being of particular importance; the regularity allowed the participants to plan with more certainty. This being said, the FGD participants generally (though not unanimously) agreed that the voucher amount was insufficient to allow them to readily engage in new/improved livelihoods (such as the possibility to invest in starting new business, expanding existing ones, or acquiring productive assets). However, the FGD participants did not express this insufficiency as a failure of the project, but simply that greater investments would be needed to have marked positive long-term impacts on improving livelihoods. 19) In challenging context and ambigious situations, leadership from FFP and the KL COP is very important. When there is active involvement of USAID/FFP mission staff in all aspects of the project, this fosters collaboration, trust, and communication. Strong leadership at the COP level improves both the upward and downward flow of strategic thinking. The leadership at the mission and in Care improved over the life of the project, particularly in the last years. For example, respondents generally spoke very highly of the FFP Office Direcctor when discussing the presence and role of FFP in meetings, workshops, and other discussions, particularly in the last years of KL. This was very highly appreciated by the IPs and GOH, and their respect for the USAID/FFP mission staff certainly improved the project overall. Missed Opportunities and lessons learned for future programming 20) Take a realistic approach to the project goals and objectives, particularly when working in a difficult setting such as Haiti. This is particularly true of institutionalization/handover to government aspects of a project. The overly ambitious objectives of the project, particularly coupled with the initial time frame, set up the project to fall short by those standards. As noted in the mid-term evaluations, projects to build social safety nets in Ethiopia and Brazil have taken many years of financial and technical assistance, despite having much more stable, strong governments that Haiti. The perceived expectations of what Kore Lavi was to achieve had the unintended consequence of setting up the project to be perceived as less of a success. The Final Performance Evaluation of the Kore Lavi DFAP in Haiti Conclusions and Recommendations 69 government hand-over in particular was unrealistic, and led to some sense of failure or shortcoming at MAST. 21) To promote sustainability of impacts, social protection projects like Kore Lavi should include livelihood/agriculture support components, or at alternativelypartner closely with complementary projects. The Kore Lavi project design lacked a livelihoods/agriculture component, which likely limited the impacts the project was able to have on long-term resilience. The assumption that households would use the voucher support coupled with the VSLAs to help lift themselves out of poverty (or at least put the household one run further up the economic ladder) relied on the presumption that these households would find and develop new/improved livelihood sources, without actually formally supporting that important pathway step. This assumption is also linked to the lack of a safety net ‘graduation’ component (see specific recommendation on that). This integration with other livelihood projects and/or inclusion of these livelihood projects should be spelled out from the early stages of the conception of the project. Additionally, the IPs should be encouraged to make justified changes to the project (deviations from the RFA) in their initial proposals without concern that this may lessen their chances of being granted the project. Finally, FFP should work more closely with the IPs to encourage adaptive management over the life of the project. 22) Incorporate graduation into the project design, with a clear and defined definition of what ‘graduation’ entails. If graduation is not envisioned as a likely or common outcome of the project, then this should be clearly spelled out in the project design. Kore Lavi monitoring included the indicator “number of safety net beneficiary households that graduated from the safety net program” throughout the life of the project, and yet reported zero or nearly zero graduations. The inclusion of this indicator was ill-conceived by both FFP and the consortium, as it lacked a clear pathway to graduation, a clear definition of ‘graduation’, or a discussion on if graduation was an appropriate goal for the project at all. It may be in the Haiti context that focusing on graduation to any large degree ignores the chronic nature of poverty among the poorest in Haiti. It was also a lost opportunity to think about and better define what graduation means for Kore Lavi during the extension period. 23) Include a way to enroll new voucher recipients. Kore Lavi lacked any clear route to enroll new voucher recipients once the initial census data collections were done, the HDVI values calculated, and the appeal office period ended. This was a key aspect of the project that did not allow Kore Lavi to act as a true safety net. The targeting focused on the chronically poor at the time of the HDVI survey. Households in targeted communities that suffered a shock such as a death of an income generating household member, or a family member developing a chronic illness, could not be added to the voucher recipient lists. This means that the food voucher program can better be described as long-term assistance for chronically poor, rather than as a safety net put in place to catch households and prevent them from falling (further) into poverty. In fact, one of the IPs identified this drawback of the Kore Lavi system, and rather than design a way to update the Kore Lavi rosters (which was described as ‘rigid’), they attempted to provide support to households suffering from idiosyncratic shocks by accessing other programs they ran in the communities. Finding a feasible solution to this challenge is admittedly challenging. One approach may be to increase the regularity of data collection, updating the household information in SIMAST in order to refine/adapt the beneficiary list on a more regular basis. Such adaptability of the beneficiary lists will require more resources, not only for the collection of IMPEL | Implementer-Led Evaluation and Learning 70 Conclusions and Recommendations more real-time targeting data, but also potentially if the number of those meeting the criteria to receive vouchers increases over time. 24) Any prototype/pilot social protection program in Haiti (or elsewhere) should have as an output a detailed operational/tactical manual (program guidance document) for social protection activities that is context/country specific. This would serve as concrete and lasting support to government capacity, and would help inform the social protection policy development. Additionally, unstable government such as Haiti’s risk capacity development being lost if the operational and tactical are not well documented. This sort of guidance can be used to resume social protection project work following a period of government instability. Additionally, there was some question as to who was leading this process at the start. NGOs may not be the ideal leaders in supporting the government in drafting policy. Kore Lavi made the appropriate decision to put the policy guidance leadership under WFP rather than CARE, However, without critiquing the work done by WFP, the question remains if there was a more appropriate organization (outside the Kore Lavi consortium) that leads in strategy and policy that would have been a more natural fit, particularly if this policy portion of the project was designed during the conception rather than along-the-way. 25) FFP should continue to work closely with the IPs to include and encourage adaptive management over the life of a project, including at the RFA stage. There were several examples where the IPs effectively adapted their work in collaboration with FFP, such as the work on the national policy for social protection, or the adjustments of voucher value to adapt to inflation and food price increases. Although adaptive management is now standard for FFP projects, this only became the norm after the start of Kore Lavi, so it was not written specifically into the project design. This left the IPs without clear processes to adapt the project during its life. Adjustments to the project were more ‘ad hoc’ than systematic as a result. The ET found that there was some tendency to ‘check the boxes’ of project outputs (in some cases, not across the board), focusing more on compliance than strategic thinking. The lack of adaptive management may have led to the perception of some that the reduction of funding and subsequent decision to not continue the SO3 component of Kore Lavi after the mid-term evaluation was somewhat abrupt, and that there was not sufficient time to propose adaptations to that component of the project. 26) Consider the exit strategy/hand-over, and take into account long term, even inter￾generational poverty, in project design. While Kore Lavi hoped to help the extremely poor rise out of poverty during the project, it did not envision that all the extremely poor targeted with assistance would benefit from long-term improved food security resilience, and it recognized that chronic extreme poverty cannot be resolved with a few years of food vouchers and complementary programming. Kore Lavi was a short-term solution to a long-term problem. As such, the end of the project had a certain amount of ‘departure shock’ to many of the participants, even though they were well informed that benefits were for a limited time period. 27) The targeting metric should be developed transparently, and balance simplicity with sensitivity and specificity. The HDVI indicator in Haiti was considered a ‘black box’ by all consortium and other stakeholder staff interviewed. The statistics used to design the HDVI are complicated, and difficult to understand for all partners. Furthermore, it appears that Kore Lavi does not have access to the datasets or complete set of analyses to create/recreate the HDVI. The statistical development was conducted by a consultant at the start of the project, and the Final Performance Evaluation of the Kore Lavi DFAP in Haiti Conclusions and Recommendations 71 complete dataset and syntax used to derive the HDVI algorithm were never made available to Kore Lavi. This means that not only can Kore Lavi not fully own and advocate for the HDVI as a targeting tool, it is impossible for MAST to take over ownership of the HDVI. Furthermore, the HDVI was derived from data collected in 2012, and risks becoming outdated. Future targeting tools should more transparently consider other existing tools. One example in Haiti is the PPI, which was being used in Haiti prior to the development of the HDVI (this is an example of a possible alternative to explore, the ET has not fully evaluated the actual potential of the PPI or other targeting indicators/approaches in Haiti). 28) Regional differences in prevalence of poverty should be accounted for in targeting. Kore Lavi made the decision to target the poorest 10% (approximately) of households in each commune where the voucher program was implemented. 10% was chosen because that was the national prevalence of extreme poverty. However, this approach discounted the difference in prevalence of extreme poverty between communes. This means that households considered less poor (as measured by the HDVI) may have been included in the voucher program in one commune, while poorer households may have been excluded in another. While there are difficulties in drawing an absolute threshold of the poverty metric for targeting, as the numbers of beneficiaries may exceed resources, the system of Kore Lavi did not attempt to address this. IMPEL | Implementer-Led Evaluation and Learning 72 Annex 1: Evaluation Scope of Work ANNEX 1: EVALUATION SCOPE OF WORK Introduction The final evaluation of the 2013 Haiti Kore Lavi Food for Peace (FFP) project is the final phase of qualitative data to be collected to evaluate this project. The baseline study, employed a mixed-method approach, and was designed to provide information on all four aspects of food security – availability, access, utilization and stability. The study investigated household food access, sanitation and hygiene, agriculture, household expenditures and assets, dietary diversity, and anthropometry among women and children. The Haiti final performance evaluation will forego the population-based quantitative data collection, but will utilize project performance monitoring data and qualitative methods to robustly answer evaluation questions. A quantitative final impact evaluation will later be conducted only on the SO2 component of the Kore Lavi project. Overview of DFAPs The Office of Food for Peace (FFP) uses Title II and/or community development funds to support multi￾year development food assistance projects (DFAPs) around the world that improve and sustain the food and nutrition security of vulnerable populations. Through the DFAPs, partners implement activities across technical sectors, layering and sequencing interventions at both the individual and household levels. FFP’s partners improve food access and incomes through agriculture and other livelihoods initiatives; enhance natural resource and environment management; combat undernutrition, especially for children under two and pregnant and lactating women; and mitigate disaster impact through early warning and community preparedness activities. Development activities are intended to build resilience in populations vulnerable to chronic hunger and repeated hunger crises, and to reduce future need for ongoing or emergency food assistance. FFP’s efforts are increasingly integrated with other USAID efforts to promote resilience and reduce extreme poverty, and the FFP development activities contribute to the USAID Bureau of Food Security as well as the Global Food Security Strategy (GFSS). Haiti’s CARE Kore Lavi DFAP • Kore Lavi “Support to the National Food Security and Nutrition Program” is a FFP-funded DFAP implemented in Haiti by a consortium of partners comprised of CARE Haiti as prime, Action Against Hunger, World Food Programme and World Vision. The project started in August 2013 and two additional years of extension was granted from October 2017 to September 2019. • The purpose of Kore Lavi is to support the Haitian Government in creating a social safety net for food and nutrition security that prioritizes consumption of locally-grown quality products. The overall goal is to contribute to reducing food insecurity and vulnerability in targeted communities by establishing a replicable safety net system and expanding government capacities to prevent child undernutrition. • Kore Lavi covers 5 geographic departments (North-West, Artibonite, West-La Gonave Island, Central Plateau and South East) and 23 communes, targeting over 17,000 households. Kore Lavi collaborates directly with the Ministry of Social Affairs & Labor (MAST) and the Ministry of Health to reach out and deliver social services to vulnerable families. It has two safety net components that consist in distribution of food vouchers to targeted families and nutrition supplements to pregnant and lactating women. The nutrition components of Kore Lavi were Final Performance Evaluation of the Kore Lavi DFAP in Haiti Annex 1: Evaluation Scope of Work 73 phased out in October 2017. The project budget was approximately 80 million for the first 4 years and 24 million for the two years of extension. • At the end of its implementation, the project is expected to achieve the following key results: o National systems for vulnerability targeting strengthened. o Access of extremely vulnerable households to local and nutritious food increased. o Maternal and child nutritional status improved. o Haitian institutions’ capacity to effectively lead and manage safety net programming improved. FFP Approach to Final Evaluation in Haiti The approach to the final evaluation to be conducted in Haiti 2019 is unique. The baseline evaluation consisted of the following: 1) a mixed-method qualitative and population-based quantitative study of the Kore Lavi implementation area, and 2) an impact evaluation baseline of SO2 to study the impact of the food voucher safety net component of the Kore Lavi project. Various changes occurred during project implementation, most significantly a decision to end interventions around SO3, maternal and child nutrition. Largely for this reason, FFP has decided not to conduct an population-based final evaluation, and will instead focus on conducting a qualitative performance evaluation that will study all SO’s, including trying to glean information on lessons from SO3, even though those interventions were not implemented in the two extension years. The data collection for the qualitative evaluation will be scheduled around May 2019, pending an improved security situation. This will occur well-before quantitative data collection on the impact evaluation for SO2 in order to a.) meet the evidence needs of USAID Mission Haiti and the GoH, and b.) ensure the presence of the implementing partner staff in the communities. The qualitative performance evaluation will use project documents, monitoring data, and qualitative primary data collection to evaluate the DFAP and document learning that can be used to inform future programming at Care, FFP, USAID Haiti, and the Ministry of Social Affairs (MAST) Haiti. Evaluation Purpose and Intended Uses The objective of the purposively sampled qualitative final evaluation is to assess the development outcomes of the Kore Lavi project and implications for sustainability. Evaluation results will speak to challenges and lessons that can be better understood through qualitative inquiry and increase learning across partners and USAID. Findings will also inform the design and implementation of future USAID investments in Haiti. The primary audience of the evaluation report will be CARE and their sub-partners, and USAID. The report will also be shared with the Government of Haiti and other stakeholders, particularly the Ministry of Social Affairs. USAID will make use of the findings from the evaluation to make presentations and bulletins as part of a wider dissemination of best practices and lessons learned. The evaluation recommendations may be used by FFP to refine proposal guidelines and project policy. The evaluation recommendations may be used by USAID/Haiti and GoH for design of future activities. The firm must design the qualitative study to assess all four objectives of the Kore Lavi project. The main objectives are to analyze perceptions about practices and behaviors; quality of and access to services, organizations, systems, and safety nets; targeting of vulnerable households; delivery systems of the food voucher; and quality of and lessons learned from the government handover process of MAST systems. Particular attention should be directed at the processes and collaborations of stakeholders around the use of SIMAST. Changes in participant’s lives as a result of the project should also be discussed as well as IMPEL | Implementer-Led Evaluation and Learning 74 Annex 1: Evaluation Scope of Work issues around participant exclusion and any effects of those dynamics. In addition, although the project underwent large changes and SO3 was severely reduced during the award, efforts to attain meaningful information about that objective should be sought through the evaluation. The firm is responsible for the design and execution of all aspects of the qualitative study including designing the methodology and questions to understand the context and community perceptions about key practices and behaviors. Study Design The final evaluation will consist of the following data collection activities: a desk review of documents and data relevant to the project, and a purposively-sampled qualitative study. The evaluation team (ET) will begin with a desk review of project documents, validate their understanding of the project through consultations with USAID and the implementing partner, and lead a purposively-sampled qualitative study. Desk Review The evaluation team should review the following documents to contextualize and refine the evaluation questions, as well as to gain an in-depth understanding about the project design, implementation and the food security situation in the area. Partner annual monitoring data should be reviewed when preparing for qualitative research, considered in relation to the evaluation findings, and incorporated into the report as evidence of evaluation findings. While FFP recommends the below documents for pre￾evaluation learning, the literature review should not be limited to the following: A. Project proposals B. Pipeline Resource Estimate Proposals (PREPs) C. Annual Results Reports (ARRs), Quarterly reports including Indicator Performance Tracking Tables (IPPTs) for performance against targets D. Midterm review and corresponding action plans developed by the project E. Baseline Study for the Title II Development Food Assistance Project in Haiti F. Haiti Demographic and Health Survey G. Partner formative research H. Monitoring data and field reports I. Sustainability and Exit Strategy Plan Consultations As a supplement to the desk review, and in preparation for qualitative research, consultations with FFP and partner staff in Washington, DC and Haiti will allow the ET to corroborate its understanding of the design, approaches and interventions employed by the DFAP and acquired through the desk review. It is recommended that the ET engage with the staff at each organization prior to beginning fieldwork. FFP/W, FFP/Haiti, and CARE staff will provide input and feedback on the draft evaluation protocol and on specific tools, questions and/or outlines to be used in the collection of data. Equally important to engaging pre-data collection is to reconnect post-data collection to “ground-truth” findings with FFP/Haiti and the partner staff. In the case of major disagreements, the project staff should provide evidence in support of the argument, and pending time constraints, the ET may revisit the field. Final Performance Evaluation of the Kore Lavi DFAP in Haiti Annex 1: Evaluation Scope of Work 75 Qualitative Study Qualitative methods will be used to collect information to answer evaluation questions, and also to support the interpretation of findings from the desk review. The ET will design the overall qualitative study approach and should consider a variety of primary data collection methods, such as semi￾structured in-depth interviews, group discussions, key informant interviews, direct observations and case studies (the ET may choose to use the most significant change methodology to identify a selective set of case studies). These methods - to the maximum extent possible - will ensure that if a different, well-qualified evaluator were to undertake the same evaluation, he or she would arrive at the same or similar findings and conclusions. The ET should decide on specific methodologies before traveling to Haiti, and present those to FFP along with estimated numbers of interviews, FGDs, etc., in the protocol. Following feedback, discussion and agreement (between ET, FFP/W, FFP/Haiti, and CARE) the ET will finalize the methods. The evaluation team leader and members will be responsible for interviewing direct, indirect, and non-participant community members and households as well as look for evidence of ongoing learning and activities (such as home gardens etc.). The ET will be responsible for interviewing relevant stakeholders for the evaluation and analyzing the qualitative data. Should the ET decide to hire additional researchers to complement the data collection effort, they cannot replace the evaluations team members’ role of collecting primary data using qualitative methods. In addition to providing evidence and interpreting findings around the factors specified in the evaluation questions (Table 1), the ET should also consider the efficacy of the following cross-cutting interests: project management; performance monitoring; strategies to improve gender equality at the participant and project management levels; environmental considerations; conflict sensitivity; and government system-strengthening. Lastly, it is expected that the qualitative study will speak to lessons learned, best practices, and facilitators and inhibitors of sustainability. Sampling for the evaluation should include both individuals who directly participated in the DFAP and those not specifically targeted with any intervention (non-participants). The importance of this cannot be understated to evaluate spillover effects, community equity issues, and to understand the broader impacts of the project at the community level. In addition, the qualitative team should interview USAID personnel, project staff, knowledgeable people from the community, local government staff, community leaders, host Government officials and other agencies and individuals as appropriate. A complete list/map of these key stakeholders should be developed thoughtfully in collaboration with FFP/Haiti and CARE. The sampling strategies at both the community-level and the respondent-level should be done independently by the ET. The sampling strategy is critical to capture the range of variability of interest in a deliberate way. The ET should clearly define sampling criteria to capture potential range of experience across the operating context for communities, participant respondents, staff and other stakeholders. Evaluation Questions The table below presents a guide to the evaluation questions and fundamental elements that should shape the Evaluation Team’s (ET) research. It is expected that the ET use these questions as a guideline, but is not limited to working within this guidance. Annex 1 provides additional ideas in form of preliminary research questions that are of particular interest to USAID/Haiti and organized by Kore Lavi IMPEL | Implementer-Led Evaluation and Learning 76 Annex 1: Evaluation Scope of Work SO. The areas of inquiry presented in the annex by SO should be considered when designing the Evaluation Questions based on intervention/respondent type. Q1: To what extent has the project met its defined goals, purpose, and outcomes? The ET will evaluate the contribution of Kore LAVI to USAID’s efforts to reducing food insecurity and vulnerability in targeted communities by establishing a replicable safety net system and expanding government capacities to prevent child undernutrition. The ET will support these determinations using qualitative methodologies and document reviews when discussing the following: (1) Project performance on indicators measured against targets set by the partners and FFP indicators*; (2) Factors that promoted or inhibited the achievement of the project objectives, including but not limited to the effectiveness of interventions including MAST systems transfer to GoH, food voucher safety net, and earlier maternal and child health interventions. (3) Plausibility of pathways and the determinants of achieving the key outcomes; (4) Targeting strategies and their contribution to achieving project goals (especially with regard to gender and reaching the most vulnerable) including analyzing if the HDVI methodology systematically identified the poorest households; (5) The appropriateness and effectiveness of interventions for the poorest individuals and HH’s. * Monitoring indicators that represent proxies along the pathway to reductions in Depth of Poverty, stunting and underweight will be identified and studied Q2: Based on the evidence, which project outcomes are likely to be sustained? The ET will evaluate the functionality and performance of systems and processes established independently by the projects, as well as in collaboration with the private sector, Government of Haiti, non-governmental organizations, and academic organizations to achieve project outcomes and sustainability. It will support its determination using qualitative methodologies that explore the following: (1) the quality of the processes, systems, and institutional arrangements developed and/or strengthened to sustain the necessary and critical services; (2) communities’ perceptions on the quality, frequency, effectiveness, and sustainability of the services provided by the project; (3) the likelihood that service providers will continue providing services after the project ends; (4) the motivation of the community and beneficiaries to demand and pay (or invest time) for the services; (5) whether the necessary resources and capacity strengthening will exist to sustain service providers; (6) the extent to which the projects leveraged other USG and non-USG investments to achieve sustained outcomes as identified in the theories of change; (7) evidence of enhanced linkages with other service providers. Q3: In each technical sector, what are the strengths and challenges of the selected interventions and their implementation, and how are these received by the target communities? Final Performance Evaluation of the Kore Lavi DFAP in Haiti Annex 1: Evaluation Scope of Work 77 The ET will evaluate the effectiveness and relevance of the technical interventions to achieve project outcomes, and discuss those findings in relation to the theory of change. Determinations will be supported when discussing the following: (1) factors in the implementation and context associated with greater or lesser effectiveness in producing Outputs of higher or lower quality; (2) the interventions and implementation processes deemed more/less acceptable to members of the target communities. Q4: What are the key lessons learned and best practices that should inform future projects in the country? During the course of research, the ET should identify best practices, strengths, and challenges in the project designs (including theories of change), that support project achievements, as well as approaches that should be considered favorably in designing future food and nutrition security projects and strengthening household and communities’ resilience capacities. The ET will support determinations using evidence when discussing the following: (1) The unintended positive and/or negative consequences of the projects; (2) Ways to minimize potential unintended negative consequences and systematically capture positive consequences. Data Analysis and Interpretation A plan to record data and the role of translators in the data collection process should be described, in addition to methods that will be used in the field among the ET. A description of the method of analysis that the team will use to analyze qualitative data should be detailed in the protocol. Report The ET will produce one report in English, not to exceed 60-pages. The draft report will be shared within FFP, USAID/Haiti, and with CARE for review and comment over a two-week period. The final report should include a table of contents, table of figures (as appropriate), acronyms, executive summary, introduction, purpose of the evaluation, research design and methodology, limitations, findings, conclusions, lessons learned and recommendations. All evaluation questions should be answered, and the evaluation methodology should be explained in detail. To ensure a high quality deliverable, the report should reflect a thoughtful, well-researched and well￾organized effort to objectively evaluate what worked in the project, what did not, and why. Where noteworthy, the discussion should particularly highlight gendered outcomes and impacts and sustainability. The ET should draw from partner annual monitoring data to contextualize qualitative findings and situate/interpret the findings within the context of project achievements, taking particular care around the fact that the qualitative findings will not be representative of the experience of the whole of participants and stakeholders of the DFAP. For example, if a qualitative finding is that beneficiaries report increased yields, the ET should compare/contrast that finding against data collected by the partners. Findings should be specific, concise, and supported by strong quantitative and/or qualitative evidence, and presented as analyzed facts/evidence/data, and not be based on anecdotes, IMPEL | Implementer-Led Evaluation and Learning 78 Annex 1: Evaluation Scope of Work hearsay or a compilation of people’s opinions. The report should include analytical methods and interpretation of findings. The report should describe limitations to the evaluation, with particular attention to the limitations associated with the evaluation methodology. Recommendations should be supported by a specific set of findings, and be action-oriented, practical and specific. It is expected that the final reports will address and incorporate feedback, as appropriate, from the reviewers. Should the ET disagree with any of the comments, it should raise this with the COR immediately for discussion. Deliverables The ET shall produce the following deliverables during the evaluation and submit to Food for Peace for review and approval. All draft documents should be submitted in Microsoft Word or Microsoft Excel. Food for Peace must approve all deliverables. Deliverable Timeline Work Plan • brief synthesis and timeline for the Haiti performance evaluation Performance Evaluation Review and Protocol (a protocol that includes document review and qualitative components) (no more than 40 pages) • brief synthesis of literature review o Presents evaluation questions contextualized based on the literature review, types of respondents • describes the evaluation questions to be answered; • introduces the local partner (data collection firm); • describes the analysis method and plan; o includes site selection criteria, sampling plan, research design and plan, interview guides • presents the fieldwork plan (including trainings and field support/supervision, data management, quality control, recording, analysis and reporting. Pertinent Permissions and approvals • demonstrate official approval from all relevant institutional review boards (IRBs) and from host country institutions to collect data, conduct the evaluation, and release data and reports, as required, as well as a statement affirming adherence to all requirements specified in USAID’s Scientific Research Policy. Qualitative Methods and Tools • outlines, methods/tools developed for data collection for each evaluation question. Final Performance Evaluation of the Kore Lavi DFAP in Haiti Annex 1: Evaluation Scope of Work 79 Deliverable Timeline Draft Evaluation Report • will be no more than 60 pages • include as an annex appropriate methodological templates such as matrices to allow for transparency of analysis method In-country briefings to USAID/Haiti and FFP stakeholders in Haiti • present the major findings of the evaluation to USAID/Haiti; • present the major findings of the evaluation to FFP stakeholders in Haiti, including DFAP partners, Government of Haiti (invited by USAID/Haiti and partner), and other relevant donors, via a 60-minute PowerPoint presentation • provide an opportunity for immediate stakeholder feedback that can be considered for the revision (as appropriate and without compromising the validity or independence of the evaluation) Briefing to FFP/Washington • provide a 60-minute PowerPoint presentation on the evaluation, major findings, lessons learned and recommendations Final Evaluation Reports • include items identified in the draft report as well as a three- to five-page executive summary of the purpose, background of the project, methods, findings, conclusions and recommendations and the following annexes: the scope of work, tools used in conducting the evaluation ( checklists, discussion guides, interview guides), and any substantially dissenting views by any Team member, USAID or the PVOs on any of the findings or recommendations; • must be uploaded to the Development Clearinghouse following COR approval ; • include analysis files of qualitative data Team Composition and Qualifications The Senior Evaluation Specialist will be responsible for designing and managing the evaluations and supervising the evaluation team members; coordinating with the implementing partners, USAID Mission and other stakeholders; analyzing the findings and ensuring the quality of the report. The qualitative study will require in-country participation from the Evaluation Team Leader and up to three subject matter specialists. As FFP projects are multi-sectoral, the evaluation team must possess expertise in food security, nutrition and field experience with rural integrated programming. The team must demonstrate an in-depth knowledge of the following technical sectors and cross-cutting areas: MCHN; social safety net systems, WASH; gender, youth, resilience, and disaster risk management. The subject matter specialists must have experience and knowledge about the specific processes used by the projects (e.g., SIMAST institutionalization, VSLAs, Care Groups, etc.) The qualitative study should also, to the extent possible and appropriate, include local representation. IMPEL | Implementer-Led Evaluation and Learning 80 Annex 1: Evaluation Scope of Work Management and Logistics The ET is responsible to arrange and pay for all logistics, including any equipment and transportation. The ET should request assistance from CARE on making introductions, as necessary, to local ministry representatives and community leaders. The ethical review process should be detailed in the protocol and report and adhere to GoH requirements including IRB processes. Intellectual Property USAID shall, solely and exclusively, own all rights in and to any work created in connection with this agreement, including all data, documents, information, copyrights, patents, trademarks, trade secrets or other proprietary rights in and to the work. The firm is not allowed to withhold any information related to this agreement, as this will become public information. Final Performance Evaluation of the Kore Lavi DFAP in Haiti Annex 2: Questions of USAID Interest for Haiti Final Evaluation 81 ANNEX 2: QUESTIONS OF USAID INTEREST FOR HAITI FINAL EVALUATION In addition, the Evaluation Team places particular attention to areas of specific interest expressed by USAID. These include the following: • SO2: Food Voucher-based Safety Net o To what extent and in what ways is the project’s food voucher-based safety net model shock-responsive when a natural disaster strikes. (Kore Lavi voucher provided support for the drought response and Hurricane Matthew. How did the project respond? What were the challenges? What was the impact of the response?). o What structural, governance obstacles exist that caused challenges to achieving this IR? How did the project try to overcome them? What would have been a more effective approach to establish more sustainability in the project to achieve the goal of replicating and scaling up Kore Lavi? o What data has the project gathered that might show the economic or food security impacts of the Kore Lavi project had at the household level? How might the experience of the project, supported by data, demonstrate to the Government the benefits of a longer-term food-based social safety net supported by the GOH? o What are some of the most notable economic multiplier effects and how can these be quantified or demonstrated in terms of unanticipated positive outcomes? For example, what was the impact of the project on the market for fresh and dried foods? Will changes to these markets be sustainable after the end of the project? How did the voucher system impact partner market vendors? How did it impact non-partner vendors and non-beneficiaries? • SO3: Maternal and Child Health and Nutrition o How was CARE’s work training community health workers, traditional birth attendants and other health professionals to improve nutrition practices coordinated with other donor activities and USAID activities? o Did the health workers find this useful or was it duplicative of other efforts? o How did they use the information? o Are they continuing to use the information? o Did the end of the food distribution under this SO dis-incentivize the work in the community on changing behavior/continuing to use the nutrition information provided? • SO4: Haitian institutions’ capacity to effectively lead and manage safety net programming improved o The Consortium measured capacity building progress against an internally developed evaluation system that showed improvements in readiness to assume leadership over the Kore Lavi project.  Was this evaluation system effective?  Did it provide an accurate account of the 'readiness" of the MAST counterparts?  Did the system take into account external, non-Kore Lavi, factors in 'readiness" of MAST to manage and assume ownership of Kore Lavi activities?  Did CARE meet its objectives of civil society engagement to monitor safety nets?  Was this aspect of the project implemented evenly across the project implementation areas (i.e. across all 5 departments)? If not why not? IMPEL | Implementer-Led Evaluation and Learning 82 Annex 2: Questions of USAID Interest for Haiti Final Evaluation  What were the main achievements of this work and how did it benefit the overall results of the project?  The results of IR 4.3 are unclear. Did the Consortium have a plan for this IR and how to achieve it? Final Performance Evaluation of the Kore Lavi DFAP in Haiti Annex 3: List of Key Informants 83 ANNEX 3: LIST OF KEY INFORMANTS KEY INFORMANT INTERVIEWS (KII) Implementing Partners (IP) CARE Action Contre la Faim (ACF) World Food Programme (WFP) World Vision International Organizations (IO) USAID World Bank European Union Government of Haiti (GOH) Ministère des Affaires Sociales et du Travail (MAST) Ministère de la Santé Publique and de la Population (MSPP) Ministère à la Condition Féminine et aux Droits des Femmes (MCFDF) Coordination Nationale de l Securite Alimentaire (CNSA) Conseils d’Administrations des Collectivités Territoriales (CASEC) FOCUS GROUP DISCUSSIONS (FGD) Project Beneficiaries (PB) Village Savings and Loan Association (VSLA) members Voucher beneficiaries (SO2) Nutrition beneficiaries (SO3) Lead mothers Lead fathers Non-beneficiaries Project Partners (PP) Vendors Conseil d’Appui au Développements Participation (CADEP) Microfinance Institutions (MFIs) IMPEL | Implementer-Led Evaluation and Learning 84 Annex 4: Site Selection (Communes) for Data Collection ANNEX 4: SITE SELECTION (COMMUNES) FOR DATA COLLECTION Department Data Collection team Commune IP (Care, WV, ACF/Care, etc.) (peri) Urban or rural SO2 (voucher) activities in the commune? Selected for sample? Other considerations for selection South East Cote de CARE Rural N C Lavallee CARE Rural N Y VSLA only, no coupons C Cayes Jacmel CARE Rural Y Y Beneficiaries in mountainous areas C Belle Anse CARE Rural Y Y Somewhat vulnerable Grand Gosier CARE Rural Y Thiotte CARE Rural N Anse a Pitres CARE Rural Y Center A,B,C Boucan Carre CARE Rural Y (pre-test) Pre-test (closest to PaP) Thomonde CARE Rural N B Hinche CARE Rural N Y Somewhat urban in character B Thomassique CARE Rural Y Y Border area Cerca Lassource CARE Rural Y B Cerca Carvajal CARE Rural Y Y non-frontalier North West A Baie de Henne ACF Rural Y Y Poorest area in NW A Bombardopolis ACF Rural Y Y Furthest from the coast A Mole Saint Nicolas ACF Rural Y Y Coastal zone A Port de Paix Urbain ACF Urban Y Y Urban zone Port de Paix Rural ACF Rural Y Bassin Bleu ACF Rural N Artibonite B Gonaives Urbain ACF Urban Y Y Urban area Gonaives Rural ACF Rural Y A Anse Rouge ACF Rural Y Y Very Vulnerable Terre Neuve ACF Rural N West C Anse a Galet WV Rural Y Y C Pointe a Raquette WV Rural Y Y Note: All communes had VSLAs established by Kore Lavi, and had SO3 project participants. Final Performance Evaluation of the Kore Lavi DFAP in Haiti Annex 5: Discussion Guides (English) 85 ANNEX 5: DISCUSSION GUIDES (ENGLISH) DISCUSSION GUIDES KORE LAVI QUALITATIVE EVALUATION The purpose of the evaluation is to obtain information through qualitative instruments (key information interviews, focus group discussions) to provide context and explanations for the performance results indicated in the quantitative household survey. The guides are designed to address the following evaluation questions: EVALUATION QUESTIONS EQ1: To what extent has the project met its defined goals, purpose, and outcomes? EQ2: Based on the evidence, which project outcomes are likely to be sustained? EQ3: In each technical sector, what are the strengths and challenges of the selected interventions and their implementation, and how are these received by the target communities? EQ4: What are the key lessons learned and best practices that should inform future projects in the country? PROJECT OBJECTIVES Information will be gathered in conversation with project participants, interviews with project implementation staff and managers, partners and beneficiaries. Towards these ends, Kore Lavi has four Strategic Objectives (SOs) supported by 10 Intermediate Results (IRs): • SO1: National systems for vulnerability targeting strengthened o IR1.1: MAST-led equitable vulnerability targeting methodology developed, tested and implemented • SO2: Access of extremely vulnerable households to local and nutritious foods increased o IR 2.1: MAST-led, gender-responsive food voucher-based safety net model developed and implemented o IR 2.2: Inclusion of local foods in the voucher-based safety net increased o IR 2.3: Access to complementary services for safety net households increased (including particularly access to village savings and loan groups, or VSLAs) • SO3: Maternal and child nutritional status improved o IR3.1: Household practice of appropriate nutrition behaviors to prevent malnutrition increased o IR 3.2: Capacity of community-based entities to promote appropriate nutrition practices to prevent malnutrition improved IMPEL | Implementer-Led Evaluation and Learning 86 Annex 5: Discussion Guides (English) o IR3.3: Capacity of health facilities to deliver appropriate nutritional services strengthened • SO4: Haitian institutions’ capacity to effectively lead and manage safety net programming improved o IR4.1: Institutional capacity of various levels of government to lead, coordinate and implement safety net programs reinforced o IR4.2: Capacity of civil society to monitor and support safety net programs reinforced o IR4.3: Government capacity to respond to food emergencies expanded RESPONDENT GROUP: IMPLEMENTING PARTNERS KEY INFORMANT INTERVIEWS IMPLEMENTING PARTNERS • CARE • Action Contre la Faim (ACF) • World Food Programme (WFP) • World Vision DISCUSSION GUIDE: IMPLEMENTING PARTNERS EQ1: To what extent has the project met its defined goals, purpose, and outcomes? 1. What is your current position? What is your association with the Kore Lavi Project? How long have you worked with project? 2. Explain your understanding of the strategy for the KORE LAVI Project, i.e., how is the project expected to achieve its objectives? 3. What are the main activities being implemented? 4. What is working well and why? (Promoting achievement of project objectives) 5. What is not working well and why? (Inhibiting achievement of project objectives) 6. How are the challenges being addressed? 7. Have you seen changes in food insecurity as a result of the project? What kind? 8. Have you seen changes in vulnerability as a result of the project? What kind? Final Performance Evaluation of the Kore Lavi DFAP in Haiti Annex 5: Discussion Guides (English) 87 9. Have you seen changes among poor or extreme poor households as a result of the project? 10. How did the project engage communities in the planning and implementation of its activities? 11. Do you think project interventions were in alignment with community priority needs? 12. Are project activities appropriate to reach the poorest households? Why or why not? 13. Are project activities effective at reaching the poorest households? Why or why not? 14. Did targeting strategies achieve project goals of outreach to women and the most vulnerable? 15. Are you familiar with the Haitian Deprivation and Vulnerability Index (HDVI)? 16. If so, what do you feel are its strengths and weaknesses? 17. What is your opinion of its effectiveness in systematically identified the poorest households? 18. Who, in your opinion, has benefitted most from the project? 19. Are there other people who should be benefitting from the project but are not? Please describe them for us. 20. Who should have been included in the activities but were not? Who? 21. Who should NOT have been included in the activities? Who? 22. How did the food voucher program respond to shocks (Hurricane Matthew, drought)? 23. What were the challenges to the voucher program during these events? 24. What was the impact of the voucher system during these events? 25. What was the project’s contribution to establishing a social safety net system? 26. What were the challenges to strengthening safety net programs? How did the project overcome these challenges? 27. How did your organization promote civil society engagement to monitor social safety nets? 28. What approaches would have been more effective? 29. Are you familiar with the MAST capacity building evaluation system? Do you think it was effective? Why or why not? Was it accurate? Did is take into account external factors? 30. What data did the project collect regarding HH level impact? 31. Were these data be used to improve social safety nets? How? 32. What were the achievements in safety net monitoring? What were the benefits? 33. Was monitoring implemented in the same way across all 5 departments? Why not? EQ2: Based on the evidence, which project outcomes are likely to be sustained? IMPEL | Implementer-Led Evaluation and Learning 88 Annex 5: Discussion Guides (English) 1. How permanent are the changes facilitated by the project? 2. What are the biggest challenges to sustaining the changes facilitated by the project? How can these challenges be addressed? 3. What is the opinion of beneficiaries of the activities promoted by the project? 4. What is the opinion of project partners of the activities promoted by the project (i.e. vendors, health centers, etc.)? 5. What were communities’ perceptions on the quality of the services provided by the project? 6. How motivated will participants and partners be to continue to maintain or support these activities after the end of the project? 7. Do you think services providers will have the capacity to continue providing services after the project ends? Which services? 8. What has been the involvement of other non-project organizations during KORE LAVI? (Government, donor, NGO) 9. Do you think that these organizations will continue the activities started under KORE LAVI? (Government, donor, NGO) 10. How was the project end date been communicated to the beneficiaries? What was their reaction? 11. What exit strategies are in place to ensure sustainability? Are you confident in the strategy? EQ3: In each technical sector, what are the strengths and challenges of the selected interventions and their implementation, and how are these received by the target communities? 34. What do you feel beneficiaries most appreciated about the project? 35. What do you feel beneficiaries least appreciated about the project? 36. Were some beneficiaries much more successful than others? Which and why? 37. Did any conflicts take place in the communities during implementation? How did the project respond/handle the situation? 38. How effective were the complaint mechanisms established by the project? 39. What were the unexpected positive and/or unintended negative consequences of the projects? 40. How could the project have predicted and addressed these? EQ4: What are the key lessons learned and best practices that should inform future projects in the country? 41. What do you think were the most innovative approaches of the Kore Lavi project? Final Performance Evaluation of the Kore Lavi DFAP in Haiti Annex 5: Discussion Guides (English) 89 42. Describe the major changes in the project/activities since start up in 2013? Since the extension 2016? Give examples. 43. Did the project become more effective in achieving its outputs and outcomes over time? How? 44. How could the project have been more effective? 45. What were the lessons learned or to be learned from KORE LAVI? Cross-Cutting Theme: Women's Empowerment and Gender Equity 46. What activities are you engaged in on gender? 47. What have been some of the greatest successes to date regarding gender inclusion? For whom? 48. What were the most successful gender-related activities? What were less successful training activities? Why? What has been happening after the trainings? 49. We were the greatest challenges for gender inclusion? 50. Were gender objectives integrated across the project? If not, why? 51. How were they monitored? How were they evaluated? 52. Do you think the project has been able to empower women in the project area? How? 53. In your opinion, were the right gender activities identified? Have the activities fit the needs of the communities? 54. Have gender norms/cultural beliefs in the community impacted outcomes? How? 55. Have there been any negative gender-related outcomes? [Probe for possible increase in gender￾based violence as a result of vouchers given to women]. 56. Do you think programming has had a sustainable impact on gender equity within the communities? How? 57. What has been the key lessons learned from the implementation of the project as well as partnering with different stakeholders to achieve gender equality? 58. What project activities should be included to promote gender and equity for future projects? Why? SUBGROUP: M and E Staff 59. What was/is the role of your organization in the design, roll-out or implementation of the HDVI? 60. What was particularly effective in the design and rollout of the HDVI? 61. What was particularly challenging in the design and rollout of the HDVI? 62. How can the implementation of HDVI be improved? IMPEL | Implementer-Led Evaluation and Learning 90 Annex 5: Discussion Guides (English) 63. What elements of HDVI do you find most useful? 64. Is the HDVI data collection methodology consistently used across geographies and beneficiary types/groups? 65. What is your experience with the household score card/household vulnerability scoring? 66. What is working well with the household scoring methodology? What can be improved? 67. Is MAST working effectively towards a singular beneficiary registry? If not, what are the constraints that may be impeding progress towards this stated objective? 68. What are the key/critical emerging learning from the HDVI that could improve quality of social protection in Haiti and FFP projects globally? RESPONDENT GROUP: INTERNATIONAL ORGANIZATIONS KEY INFORMANT INTERVIEWS INTERNATIONAL ORGANIZATIONS • USAID • World Bank • European Union DISCUSSION GUIDE: INTERNATIONAL ORGANIZATIONS EQ1: To what extent has the project met its defined goals, purpose, and outcomes? 1. Do you think the Kore Lavi project had impacts on food security: a. At beneficiary household level? b. In the community? c. With market effects on vendors in the community and in the region? 2. Do you believe that the Kore Lavi voucher program is preferable to a cash transfer? Why or why not and under what circumstances? 3. What is your assessment of the extent to which Kore Lavi provided a scalable model for targeting and resource transfers for both chronically vulnerable and emergency contexts? 4. What are the strengths and limitations of the targeting tool? 5. Has it been superseded at MAST by another approach? If so, why? Final Performance Evaluation of the Kore Lavi DFAP in Haiti Annex 5: Discussion Guides (English) 91 EQ2: Based on the evidence, which project outcomes are likely to be sustained? 6. Do you feel that MAST has the capacity to implement the poverty targeting tool in the future? 7. Do you feel that MAST has the capacity to manage resource transfers for the chronically vulnerable and transitorily vulnerable in the future? 8. If not MAST, is there another governmental entity that should administer emergency resource transfers? 9. Do you think any aspects of Kore Lavi have led to sustainable effects in households, communities, systems (government programs)? Which? EQ4: What are the key lessons learned and best practices that should inform future projects in the country? 10. How could Kore Lavi have been designed and implemented to have better results? 11. If USAID were to plan a follow on activity to Kore Lavi, what do you recommend should be the key elements? Who should be its partners? RESPONDENT GROUP: GOH STAKEHOLDERS KEY INFORMANT INTERVIEWS GOVERNMENT OF HAITI • Ministry of Social Affairs and Labor (MAST) • Ministry of Public Health and Population (MSPP) Directorate of Nutrition • Ministry for Women’s Affairs and Women’s Rights (MCFDF) • National Coordination for Food Security (CNSA) • Collective Territorial Administration Councils (CASEC) • Health center staff DISCUSSION GUIDE: GOH STAKEHOLDERS EQ1: To what extent has the project met its defined goals, purpose, and outcomes? 1. What is your current position? What is your association with the Kore Lavi Project? How long have you worked with project? 2. Explain your understanding of the strategy for the KORE LAVI Project, i.e., how is the project expected to achieve its objectives? IMPEL | Implementer-Led Evaluation and Learning 92 Annex 5: Discussion Guides (English) 3. What are the main activities being implemented? 4. What is working well and why? (Promoting achievement of project objectives) 5. What is not working well and why? (Inhibiting achievement of project objectives) 6. How are the challenges being addressed? 7. Have you seen changes in food insecurity as a result of the project? What kind? 8. Have you seen changes in vulnerability as a result of the project? What kind? 9. Have you seen changes among poor or extreme poor households as a result of the project? 10. How did the project engage communities in the planning and implementation of its activities? 11. Do you think project interventions were in alignment with community priority needs? 12. Are project activities appropriate to reach the poorest households? Why or why not? 13. Are project activities effective at reaching the poorest households? Why or why not? 14. Did targeting strategies achieve project goals of outreach to women and the most vulnerable? 15. Are you familiar with the Haitian Deprivation and Vulnerability Index (HDVI)? 16. If so, what do you feel are its strengths and weaknesses? 17. What is your opinion of its effectiveness in systematically identified the poorest households? 18. Who, in your opinion, has benefitted most from the project? 19. Are there other people who should be benefitting from the project but are not? Please describe them for us. 20. Who should have been included in the activities but were not? Who? 21. Who should NOT have been included in the activities? Who? 22. How did the food voucher program respond to shocks (Hurricane Matthew, drought)? 23. What were the challenges to the voucher program during these events? 24. What was the impact of the voucher system during these events? 25. What was the project’s contribution to establishing a social safety net system? 26. What were the challenges to strengthening safety net programs? How did the project overcome these challenges? 27. What were the achievements in safety net monitoring? What were the benefits? 28. Was monitoring implemented in the same way across all 5 departments? Why not? Final Performance Evaluation of the Kore Lavi DFAP in Haiti Annex 5: Discussion Guides (English) 93 29. Are you familiar with the MAST capacity building evaluation system? Do you think it was effective? Why or why not? Was it accurate? Did is take into account external factors? EQ2: Based on the evidence, which project outcomes are likely to be sustained? 30. How permanent are the changes facilitated by the project? 31. What are the biggest challenges to sustaining the changes facilitated by the project? How can these challenges be addressed? 32. What is the opinion of beneficiaries of the activities promoted by the project? 33. What is the opinion of project partners of the activities promoted by the project (i.e. vendors, health centers, etc.)? 34. What were communities’ perceptions on the quality of the services provided by the project? 35. How motivated will participants and partners be to continue to maintain or support these activities after the end of the project? 36. Do you think services providers will have the capacity to continue providing services after the project ends? Which services? 37. What has been the involvement of other non-project organizations during KORE LAVI? (Government, donor, NGO) 38. Do you think that these organizations will continue the activities started under KORE LAVI? (Government, donor, NGO) 39. How was the project end date been communicated to the beneficiaries? What was their reaction? 40. What exit strategies are in place to ensure sustainability? Are you confident in the strategy? EQ3: In each technical sector, what are the strengths and challenges of the selected interventions and their implementation, and how are these received by the target communities? 41. What do you feel beneficiaries most appreciated about the project? 42. What do you feel beneficiaries least appreciated about the project? 43. Were some beneficiaries much more successful than others? Which and why? 44. Did any conflicts take place in the communities during implementation? How did the project respond/handle the situation? 45. How effective were the complaint mechanisms established by the project? 46. What were the unexpected positive and/or unintended negative consequences of the projects? 47. How could the project have predicted and addressed these? IMPEL | Implementer-Led Evaluation and Learning 94 Annex 5: Discussion Guides (English) EQ4: What are the key lessons learned and best practices that should inform future projects in the country? 48. What do you think were the most innovative approaches of the Kore Lavi project? 49. Describe the major changes in the project/activities since start up in 2013? Since the extension 2016? Give examples. 50. Did the project become more effective in achieving its outputs and outcomes over time? How? 51. How could the project have been more effective? 52. What were the lessons learned or to be learned from KORE LAVI? Cross-Cutting Theme: Women's Empowerment and Gender Equity 53. What activities are you engaged in on gender? 54. What have been some of the greatest successes to date regarding gender inclusion? For whom? 55. What were the most successful gender-related activities? What were less successful training activities? Why? What has been happening after the trainings? 56. We were the greatest challenges for gender inclusion? 57. Were gender objectives integrated across the project? If not, why? 58. How were they monitored? How were they evaluated? 59. Do you think the project has been able to empower women in the project area? How? 60. In your opinion, were the right gender activities identified? Have the activities fit the needs of the communities? 61. Have gender norms/cultural beliefs in the community impacted outcomes? How? 62. Have there been any negative gender-related outcomes? [Probe for possible increase in gender￾based violence as a result of vouchers given to women]. 63. Do you think programming has had a sustainable impact on gender equity within the communities? How? 64. What has been the key lessons learned from the implementation of the project as well as partnering with different stakeholders to achieve gender equality? 65. What project activities should be included to promote gender and equity for future projects? Why? Final Performance Evaluation of the Kore Lavi DFAP in Haiti Annex 5: Discussion Guides (English) 95 SUBGROUP: MAST and CNSA 66. How your institution been strengthened by its involvement with the MAST project? 67. Do you feel the support provided by the Kore Lavi project was appropriate to your needs? 68. What are the key strengths and weaknesses of the project related to capacity development of the Haitian government institution? 69. Has coordination between national-level and department-level MAST staff improved/worsened as a result of Kore Lavi activities? 70. What is the key/critical emerging learning that can improve social protection in Haiti? SUBGROUP: MCFDF 71. Can you tell us about your experiences with the development and implementation of the gender integration to safety net programs? 72. What worked best and what did not work so well? What are your suggestions for improving the process and outcomes with regards to gender integration in Safety Net program? 73. Overall, how would you describe the relevance, usefulness and impact of your collaboration with Kore Lavi? 74. What were the tools developed? 75. What has been the outcome of using these tools, particularly with regards to how the government institutions are changing how they do things with regards to gender integration? 76. What are your suggestions for improving the gender aspects of this project? For instance, what difference might the presence of MCFDF at the oversight and management level had? 77. From a gender perspective, what are the key/critical emerging learning from this intervention related to social protection in Haiti? Within Food for Peace programs more generally? IMPEL | Implementer-Led Evaluation and Learning 96 Annex 5: Discussion Guides (English) RESPONDENT GROUP: PROJECT BENEFICIARIES FOCUS GROUP DISCUSSION The following persons or groups at the community-level will be engaged in interviews or focus group discussions (this is not a complete list): PROJECT BENEFICIARIES Voluntary Saving and Loan Association (VSLA) members Voucher beneficiaries Training beneficiaries Lead mothers Lead fathers Non-beneficiaries DISCUSSION GUIDE: PROJECT BENEFICIARIES EQ1: To what extent has the project met its defined goals, purpose, and outcomes? 1. Are you familiar with the KORE LAVI Project? 2. How would you describe what this project seeks to accomplish? 3. For how long have you/they been involved with the project? 4. How has food security changed in your household as a result of the project? In your community? 5. How has child malnutrition changed as a result of the project? 6. How has your ability to manage negative shocks changed as a result of the project (bad harvest, flooding, health crisis, etc.)? 7. What do you think went well with the project? 8. What do you think went not so well? 9. Were there ways that the project did not help you, or negatively affected you, your family and community? Give examples. 10. How else could the project have helped you/your family/your community? 11. Who, in your opinion, has benefitted most from the project? (In your household, in your community) 12. Do you feel the project reached the right people in your community? 13. Are there people who should have been included in the activities but were not? Who? Final Performance Evaluation of the Kore Lavi DFAP in Haiti Annex 5: Discussion Guides (English) 97 14. Is there anyone you feel should NOT have been included in the activities? Who? 15. Have there been changes in project activities since start up in 2013? Give examples. 16. What aspects of the voucher system worked best for you and others in your community; what aspects did not work so well? Explain. 17. What are your suggestions on how to the voucher program? 18. Are there aspects of the voucher system that affected women different than men? 19. If ‘Yes,’ what were they and how did they affect the outcomes? 20. How can aspects of the voucher system be changed to improve the outcomes for both women and men? EQ2: Based on the evidence, which project outcomes are likely to be sustained? 21. How motivated are you to continue to maintain or support these activities? 22. Now that the project is ending, what do you think will be sustainable? (At household level, community level) 23. What project activities should be continued by future projects? 24. What project activities should NOT be continued by future projects? 25. How could these activities be improved? EQ3: In each technical sector, what are the strengths and challenges of the selected interventions and their implementation, and how are these received by the target communities? 26. What did you most appreciate about the project? 27. What did you least appreciate about the project? 28. What is your opinion of the quality of the services provided by the project? 29. What constraints do you believe prevented the project from accomplishing its objectives? 30. What suggestions do you have for addressing these constraints? EQ4: What are the key lessons learned and best practices that should inform future projects in the country? 31. If you had any problem with any activity/staff, what did you do/would have done? (hotline, staff, box, external person, etc.) 32. Are you aware that the project has system for making complaints? 33. Have you ever made a complaint to the project? If so, was the complaint resolved to your satisfaction? Why or why not? IMPEL | Implementer-Led Evaluation and Learning 98 Annex 5: Discussion Guides (English) SUBGROUP: Lead Mothers and Lead Fathers 34. How/why did you get selected as a Lead Mother/Father? 35. Did the training you received though Kore Lavi meet your needs and priorities? 36. What did you most appreciate about the training? 37. What did you least appreciate? 38. What changes within the household, particularly, in terms attitudes and behaviors, are you observing within target households as a result of your work? 39. How do you assess the food security situation in your community? 40. Is food insecurity is a problem in your community? How? 41. What factors trigger the food insecurity? 42. Do you think that malnutrition is a problem in your household? In your community? 43. How has the overall living situation in your community changed over the last 5 years? (Security, markets, food, health, women’s rights, and agricultural production) 44. Who is active in improving food security in your community? 45. What has been done to address food insecurity in your community? 46. What needs to be done to address this problem more effectively? SUBGROUP: VSLA 47. How/why did you get into the VSLA program? 48. How long ago did your savings group form? 49. How many members are in your savings group? 50. Has involvement in the project enhanced food security in your household? How? 51. Has involvement in the project enhanced your status in your household? In your community? 52. Overall, what would you say are the key strengths and weakness of the project? 53. Are there business development opportunities that you believe would help build food security? Please explain your thoughts? 54. How would improvements in food security help businesses in your community? 55. Have there been other USAID programs to promote livelihoods in your community? What was there impact on your community? Final Performance Evaluation of the Kore Lavi DFAP in Haiti Annex 5: Discussion Guides (English) 99 56. Is it common for individuals or families to save money? Why or why not? And if so, what are savings commonly directed towards? 57. Is it possible to secure business loans in your community? What types of loans? 58. What are some of the roadblocks to securing loans? 59. How best can communities be supported to improve on their livelihoods? IMPEL | Implementer-Led Evaluation and Learning 100 Annex 5: Discussion Guides (English) RESPONDENT GROUP: PROJECT PARTNERS KEY INFORMANT INTERVIEWS PROJECT PARTNERS • Vendors • Community Health Agents (CHA) • Participatory Development Support Council (CADEP) DISCUSSION GUIDE: PROJECT PARTNERS EQ1: To what extent has the project met its defined goals, purpose, and outcomes? 1. Are you familiar with the KORE LAVI Project? 2. How would you describe what this project seeks to accomplish? 3. For how long have you/they been involved with the project? 4. How has food security changed in your household as a result of the project? In your community? 5. How has child malnutrition changed as a result of the project? 6. How has your ability to manage negative shocks changed as a result of the project (bad harvest, flooding, health crisis, etc.)? 7. What do you think went well with the project? 8. What do you think went not so well? 9. Were there ways that the project did not help you, or negatively affected you, your family and community? Give examples. 10. How else could the project have helped you/your family/your community? 11. Who, in your opinion, has benefitted most from the project? (In your household, in your community) 12. Do you feel the project reached the right people in your community? 13. Are there people who should have been included in the activities but were not? Who? 14. Is there anyone you feel should NOT have been included in the activities? Who? 15. Have there been changes in project activities since start up in 2013? Give examples. EQ2: Based on the evidence, which project outcomes are likely to be sustained? 16. Did you undertake new activities as a result of your work with the project? 17. What is the likelihood that you will continue providing services after the project ends? Final Performance Evaluation of the Kore Lavi DFAP in Haiti Annex 5: Discussion Guides (English) 101 18. Did you establish new relationships as a result of your work with the project? (New clients, new partners) 19. What is the likelihood that you will continue these relationships after the project ends? 20. Now that the project is ending, what activities are sustainable? (HH level, community level) 21. What project activities should be continued by future projects? 22. What project activities should NOT be continued by future projects? 23. How could these activities be improved? EQ3: In each technical sector, what are the strengths and challenges of the selected interventions and their implementation, and how are these received by the target communities? 24. What did you most appreciate about the project? 25. What did you least appreciate about the project? 26. What is your opinion of the quality of the services provided by the project? 27. What constraints do you believe prevented the project from accomplishing its objectives? 28. What suggestions do you have for addressing these constraints? EQ4: What are the key lessons learned and best practices that should inform future projects in the country? 29. If you had any problem with any activity/staff, what did you do/would have done? (hotline, staff, box, external person, etc.) 30. Are you aware that the project has system for making complaints? 31. Have you ever made a complaint to the project? If so, was the complaint resolved to your satisfaction? Why or why not? SUBGROUP: Community Health Agents 1. How has the overall context and living situations changed within the last 5 years? (Security, markets, food, health, women’s rights, and agricultural production) 2. How do you assess the food security situation in your community? 3. Is food insecurity is a problem in your community? How? 4. What factors trigger the food insecurity? 5. What has been done to address food insecurity in your community? 6. What were some of the strengths of those projects? And weaknesses? 7. What needs to be done to address this problem more effectively? IMPEL | Implementer-Led Evaluation and Learning 102 Annex 5: Discussion Guides (English) 8. Do you think that malnutrition is a problem in your household? In your community? 9. What type of support from the Kore Lavi project did you find most useful? 10. How did you use the information provided in the training? Are you still using it? 11. How have your clients used the information provided in the training? Are they still using it? 12. How did the end of food distribution change motivation to use nutrition practices? 13. Who is active in improving food security in your community? 14. How was training on improved nutrition practices coordinated with other donor activities? SUBGROUP: Vendors 15. How do you feel about your experience as a vendor for the Kore Lavi project? 16. Did working with the project change the number of clients you serve? Did it increase the amount of money your business earned? 17. Did you experience a change in prices for your goods as a result of the project? 18. Did you experience a change in demand for your goods as a result of the project? 19. Were you able to meet all of the demand for goods? 20. What were the best selling items purchased with project vouchers? 21. Did you experience a change in your supply goods as a result of the project? Final Performance Evaluation of the Kore Lavi DFAP in Haiti Annex 6: KII and FGD Tools (French/Creole) 103 ANNEX 6: KII AND FGD TOOLS (FRENCH/CREOLE) Bénéficiaires (Coupons) GWOUP KAP REPONN NAN: BENEFISYÈ PWOJÈ YO DISKISYON AN GWOUP Benefisyè yo nan pwojè a/SO2 Pami sijè nou kouvri: BENEFISYÈ PWOJÈ EQ1: Nan ki nivo pwojè a reyalize bi, objektif ak rezilta li te defini yo? EQ2: Baze sou done yo, eske rezilta pwojè a gen chans pou’l dire? EQ3: Nan chak sektè teknik pwojè a, ki sa ki fòs ak defi entèvansyon yo te chwazi ak ekzekite yo, epi ki jan kominote ki sible yo te resevwa entevansyon sa yo? EQ4: Ki leson prensipal nou te tire e ki pi bon pratik ki ta dwe rete kom enfòmasyon pou lòt pwojè kap vini nan peyi a? GID POU DISKISYON YO : BENEFISYÈ PWOJÈ YO EQ1: Nan ki nivo pwojè a reyalize bi, objektif ak rezilta li te defini yo? 1. Ki jan ou ta dekri kisa pwogram KORE LAVI a ap chèche reyalize/akonpli? 2. Depi Konbyen tan ou/yo te patisipe/enplike nan pwojè a? 3. Ki chanjman pwoje a pote nan zafè lamanjay nan kay la? Nan kominote w la? 4. Ki chanjman pwojè a pote nan zafè malnitrisyon timoun? 5. Ki jan pwojè a rive ogmante kapasite nou nan jere pwoblèm ke nap fè fas (move rekòt, inondasyon, kriz sante, elatriye)? 6. Ki sa nou panse ki te mache byen nan pwojè a? 7. Ki sa nou panse ki pa t’ byen mache nan pwojè a? 8. Ki bagay ou tap tann e ke pwojè a pa’t kapab fè? 9. Ki konsekans bagay sa (ki pa fet oswa ki pa fet ase) te gen sou fanmiw oswa nan kominote w? 10. Ki lòt fason pwojè a te ka ede w / fanmi w / kominote w la? 11. Kiyès nou panse ki benefisye pi fò nan pwogram lan? (Nan kay ou, nan kominote w la ?) 12. Èske nou panse pwojè a te rive jwenn moun ki pi frajil/vilnerab yo nan kominote w la? 13. Èske gen moun ki ta dwe patisipe nan aktivite yo, men ki pa ladann? Ki moun? IMPEL | Implementer-Led Evaluation and Learning 104 Annex 6: KII and FGD Tools (French/Creole) 14. Ki kategori moun ou santi ki pa ta dwe patisipe nan aktivite yo? 15. Èske nou te remake gen anpil chanjman nan aktivite pwojè a depi l’ konmanse nan lane 2013? Bay kek egzanp. 16. Ki pati nan sistèm sèvis koupon an ki te mache pi byen pou ou ak pou lòt moun nan kominote w la ? ki pati ki pa’t mache byen? Eksplike. 17. Ki sijesyon ou genyen pou pwogram koupon an? 18. Kisa nou we kom diferans ant le se gason oswa se fanm ki resevwa koupon pou fanmi an? 19. Ki chanjman ou wè ki ta dwe fèt nan sistèm koupon an pou amelyore rezilta pou fanm ak gason? EQ2: Baze sou done yo, eske rezilta pwojè a gen chans pou’l dire? 20. Kisa ou panse ou ka fe pouw kontinye kenbe abitid konsomasyon pwoje a te edew kreye lakay ou ? (Nan sa nap manje ? Nan kantite nap manje ?) 21. Kounye a ke pwojè a fini, ki sa ou panse ke li genyen kòm enpak/chanjman (pozitif oswa negatif) ? (Nan nivo fanmi yo, nan nivo kominote a). 22. Ki aktivite pwojè a yo ta dwe kontinye fè nan lot pwojè ki gen pou vini? Poukisa ? 23. Ki aktivite pwojè a yo pa ta dwe tounen avè ankò nan lot pwojè ki gen pou vini? Poukisa ? 24. Kijan aktivite sa yo ka amelyore? EQ3: Nan chak sektè teknik pwojè a , ki sa ki fòs ak defi entèvansyon yo te chwazi ak ekzekite yo, epi ki jan kominote ki sible yo te resevwa entevansyon sa yo? 25. Ki sa ou te pi renmen nan pwojè sa a? 26. Ki sa ou pa’t renmen ditou nan pwojè sa a? 27. Kisa ou panse sou kalite sèvis yo bay nan pwojè a? 28. Daprè ou menm, ki kontrent/pwoblem ki te anpeche pwogram lan reyalize objektif li yo? 29. Ki sijesyon ou genyen pou fè fas (adrese) ak kontrent/pwoblem sa yo? EQ4: Ki leson prensipal nou te tire e ki pi bon pratik ki ta dwe rete kom enfòmasyon pou lòt pwojè kap vini nan peyi a? 30. Si ou te gen yon pwoblèm ak nenpòt aktivite oswa moun ki t ap travay nan pwojè a, ki sa ou te fè? (Liy Dirèk, anplwaye, bwat, moun deyò, elatriye) 31. Èske w konnen ke pwojè a te gen yon sistèm plent? 32. Èske w te janm fè yon plent nan pwojè a? Nan ka sa a, èske w te satisfè jan yo te trete plent lan ? Poukisa? Final Performance Evaluation of the Kore Lavi DFAP in Haiti Annex 6: KII and FGD Tools (French/Creole) 105 Bénéficiaires (Mother & Father Leaders) GWOUP KAP REPONN NAN: BENEFISYÈ PWOJÈ YO DISKISYON AN GWOUP Manman kap dirije Papa kap dirije Pami sijè nou kouvri: Manman ak papa kap dirije EQ1: Nan ki nivo pwojè a reyalize bi, objektif ak rezilta li te defini yo? EQ2: Baze sou done yo, eske rezilta pwojè a gen chans pou’l dire? EQ3: Nan chak sektè teknik pwojè a, ki sa ki fòs ak defi entèvansyon yo te chwazi ak ekzekite yo, epi ki jan kominote ki sible yo te resevwa entevansyon sa yo? EQ4: Ki leson prensipal nou te tire e ki pi bon pratik ki ta dwe rete kom enfòmasyon pou lòt pwojè kap vini nan peyi a? GID POU DISKISYON YO : BENEFISYÈ PWOJÈ YO EQ1: Nan ki nivo pwojè a reyalize bi, objektif ak rezilta li te defini yo? 33. Ki jan ou ta dekri kisa pwogram KORE LAVI a ap chèche reyalize/akonpli? 34. Depi Konbyen tan ou/yo te patisipe/enplike nan pwojè a? 35. Ki chanjman pwoje a pote nan zafè lamanjay nan kay la? Nan kominote w la? 36. Ki chanjman pwojè a pote nan zafè malnitrisyon timoun? 37. Ki jan pwojè a rive ogmante kapasite nou nan jere pwoblèm ke nap fè fas (move rekòt, inondasyon, kriz sante, elatriye)? 38. Ki sa nou panse ki te mache byen nan pwojè a? 39. Ki sa nou panse ki pa t’ byen mache nan pwojè a? 40. Ki bagay ou tap tann e ke pwojè a pa’t kapab fè? 41. Ki konsekans bagay sa (ki pa fet oswa ki pa fet ase) te gen sou fanmiw oswa nan kominote w? 42. Ki lòt fason pwojè a te ka ede w / fanmi w / kominote w la? 43. Kiyès nou panse ki benefisye pi fò nan pwogram lan? (Nan kay ou, nan kominote w la ?) 44. Èske nou panse pwojè a te rive jwenn moun ki pi frajil/vilnerab yo nan kominote w la? 45. Èske gen moun ki ta dwe patisipe nan aktivite yo, men ki pa ladann? Ki moun? IMPEL | Implementer-Led Evaluation and Learning 106 Annex 6: KII and FGD Tools (French/Creole) 46. Ki kategori moun ou santi ki pa ta dwe patisipe nan aktivite yo? 47. Èske nou te remake gen anpil chanjman nan aktivite pwojè a depi l’ konmanse nan lane 2013? Bay kek egzanp. 48. Ki pati nan sistèm sèvis koupon an ki te mache pi byen pou ou ak pou lòt moun nan kominote w la ? ki pati ki pa’t mache byen? Eksplike. 49. Ki sijesyon ou genyen pou pwogram koupon an? 50. Kisa nou we kom diferans ant le se gason oswa se fanm ki resevwa koupon pou fanmi an? 51. Ki chanjman ou wè ki ta dwe fèt nan sistèm koupon an pou amelyore rezilta pou fanm ak gason? EQ2: Baze sou done yo, eske rezilta pwojè a gen chans pou’l dire? 52. Kisa ou panse ou ka fe pouw kontinye kenbe abitid konsomasyon pwoje a te edew kreye lakay ou ? (Nan sa nap manje ? Nan kantite nap manje ?) 53. Kounye a ke pwojè a fini, ki sa ou panse ke li genyen kòm enpak/chanjman (pozitif oswa negatif) ? (Nan nivo fanmi yo, nan nivo kominote a). 54. Ki aktivite pwojè a yo ta dwe kontinye fè nan lot pwojè ki gen pou vini? Poukisa ? 55. Ki aktivite pwojè a yo pa ta dwe tounen avè ankò nan lot pwojè ki gen pou vini? Poukisa ? 56. Kijan aktivite sa yo ka amelyore? EQ3: Nan chak sektè teknik pwojè a , ki sa ki fòs ak defi entèvansyon yo te chwazi ak ekzekite yo, epi ki jan kominote ki sible yo te resevwa entevansyon sa yo? 57. Ki sa ou te pi renmen nan pwojè sa a? 58. Ki sa ou pa’t renmen ditou nan pwojè sa a? 59. Kisa ou panse sou kalite sèvis yo bay nan pwojè a? 60. Daprè ou menm, ki kontrent/pwoblem ki te anpeche pwogram lan reyalize objektif li yo? 61. Ki sijesyon ou genyen pou fè fas (adrese) ak kontrent/pwoblem sa yo? EQ4: Ki leson prensipal nou te tire e ki pi bon pratik ki ta dwe rete kom enfòmasyon pou lòt pwojè kap vini nan peyi a? 62. Si ou te gen yon pwoblèm ak nenpòt aktivite oswa moun ki t ap travay nan pwojè a, ki sa ou te fè? (Liy Dirèk, anplwaye, bwat, moun deyò, elatriye) 63. Èske w konnen ke pwojè a te gen yon sistèm plent? 64. Èske w te janm fè yon plent nan pwojè a? Nan ka sa a, èske w te satisfè jan yo te trete plent lan ? Poukisa? Final Performance Evaluation of the Kore Lavi DFAP in Haiti Annex 6: KII and FGD Tools (French/Creole) 107 SOUGWOUP: MANMAN AK PAPA DIRIJAN 65. Kijan / poukisa yo te chwazi w kòm Manman / Papa lidè? 66. Èske fòmasyon ou te resevwa nan men Kore Lavi satisfè bezwen w ak priyorite w? 67. Ki sa ou te plis renmen nan fòmasyon yo? 68. Ki sa ou pa’t tèlman renmen? 69. Ki chanjman ou obsève nan fanmi yo, an patikilye, nan zafè atitid ak konpòtman, nan menaj/fanmi ki te benefisye aktivite pwojè a gras ak travay ou? 70. Kijan ou wè sitiyasyon lamanjay la nan kominote w la? 71. Èske koze lamanjay la se yon pwoblèm nan kominote w la? Ki jan? 72. Ki faktè ki deklanche/pwovoke pwoblèm lamanjay la? 73. Èske ou panse malnitrisyon se yon pwoblèm nan fanmiw? Nan kominote w la? 74. Kijan sitiyasyon lavi anjeneralnankominote w la chanje nan 5 dènye ane yo? (Sekirite Sosyal, mache, manje, sante, dwa fanm ak pwodiksyon agrikòl) 75. Ki moun ki aktif nan travay pou amelyore pwoblèm lamanjay nan kominote w la? 76. Kisa ki te fèt pou konbat pwoblèm lamanjay nan kominote w la? 77. Kisa ki dwe fèt pou abòde pwoblèm sa a pi byen? Bénéficiaires (VSLA) GWOUP KAP REPONN NAN: BENEFISYÈ PWOJÈ YO DISKISYON AN GWOUP Manm Asosyasyon Volontè Epay ak Kredi VSLA Pami sijè nou kouvri: Manm VSLA EQ1: Nan ki nivo pwojè a reyalize bi, objektif ak rezilta li te defini yo? EQ2: Baze sou done yo, eske rezilta pwojè a gen chans pou’l dire? EQ3: Nan chak sektè teknik pwojè a, ki sa ki fòs ak defi entèvansyon yo te chwazi ak ekzekite yo, epi ki jan kominote ki sible yo te resevwa entevansyon sa yo? EQ4: Ki leson prensipal nou te tire e ki pi bon pratik ki ta dwe rete kom enfòmasyon pou lòt pwojè kap vini nan peyi a? GID POU DISKISYON YO : BENEFISYÈ PWOJÈ YO EQ1: Nan ki nivo pwojè a reyalize bi, objektif ak rezilta li te defini yo? IMPEL | Implementer-Led Evaluation and Learning 108 Annex 6: KII and FGD Tools (French/Creole) 78. Ki jan ou ta dekri kisa pwogram KORE LAVI a ap chèche reyalize/akonpli? 79. Depi Konbyen tan ou/yo te patisipe/enplike nan pwojè a? 80. Ki chanjman pwoje a pote nan zafè lamanjay nan kay la? Nan kominote w la? 81. Ki chanjman pwojè a pote nan zafè malnitrisyon timoun? 82. Ki jan pwojè a rive ogmante kapasite nou nan jere pwoblèm ke nap fè fas (move rekòt, inondasyon, kriz sante, elatriye)? 83. Ki sa nou panse ki te mache byen nan pwojè a? 84. Ki sa nou panse ki pa t’ byen mache nan pwojè a? 85. Ki bagay ou tap tann e ke pwojè a pa’t kapab fè? 86. Ki konsekans bagay sa (ki pa fet oswa ki pa fet ase) te gen sou fanmiw oswa nan kominote w? 87. Ki lòt fason pwojè a te ka ede w / fanmi w / kominote w la? 88. Kiyès nou panse ki benefisye pi fò nan pwogram lan? (Nan kay ou, nan kominote w la ?) 89. Èske nou panse pwojè a te rive jwenn moun ki pi frajil/vilnerab yo nan kominote w la? 90. Èske gen moun ki ta dwe patisipe nan aktivite yo, men ki pa ladann? Ki moun? 91. Ki kategori moun ou santi ki pa ta dwe patisipe nan aktivite yo? 92. Èske nou te remake gen anpil chanjman nan aktivite pwojè a depi l’ konmanse nan lane 2013? Bay kek egzanp. 93. Ki pati nan sistèm sèvis koupon an ki te mache pi byen pou ou ak pou lòt moun nan kominote w la ? ki pati ki pa’t mache byen? Eksplike. 94. Ki sijesyon ou genyen pou pwogram koupon an? 95. Kisa nou we kom diferans ant le se gason oswa se fanm ki resevwa koupon pou fanmi an? 96. Ki chanjman ou wè ki ta dwe fèt nan sistèm koupon an pou amelyore rezilta pou fanm ak gason? EQ2: Baze sou done yo, eske rezilta pwojè a gen chans pou’l dire? 97. Kisa ou panse ou ka fe pouw kontinye kenbe abitid konsomasyon pwoje a te edew kreye lakay ou ? (Nan sa nap manje ? Nan kantite nap manje ?) 98. Kounye a ke pwojè a fini, ki sa ou panse ke li genyen kòm enpak/chanjman (pozitif oswa negatif) ? (Nan nivo fanmi yo, nan nivo kominote a). 99. Ki aktivite pwojè a yo ta dwe kontinye fè nan lot pwojè ki gen pou vini? Poukisa ? Final Performance Evaluation of the Kore Lavi DFAP in Haiti Annex 6: KII and FGD Tools (French/Creole) 109 100. Ki aktivite pwojè a yo pa ta dwe tounen avè ankò nan lot pwojè ki gen pou vini? Poukisa ? 101. Kijan aktivite sa yo ka amelyore? EQ3: Nan chak sektè teknik pwojè a , ki sa ki fòs ak defi entèvansyon yo te chwazi ak ekzekite yo, epi ki jan kominote ki sible yo te resevwa entevansyon sa yo? 102. Ki sa ou te pi renmen nan pwojè sa a? 103. Ki sa ou pa’t renmen ditou nan pwojè sa a? 104. Kisa ou panse sou kalite sèvis yo bay nan pwojè a? 105. Daprè ou menm, ki kontrent/pwoblem ki te anpeche pwogram lan reyalize objektif li yo? 106. Ki sijesyon ou genyen pou fè fas (adrese) ak kontrent/pwoblem sa yo? EQ4: Ki leson prensipal nou te tire e ki pi bon pratik ki ta dwe rete kom enfòmasyon pou lòt pwojè kap vini nan peyi a? 107. Si ou te gen yon pwoblèm ak nenpòt aktivite oswa moun ki t ap travay nan pwojè a, ki sa ou te fè? (Liy Dirèk, anplwaye, bwat, moun deyò, elatriye) 108. Èske w konnen ke pwojè a te gen yon sistèm plent? 109. Èske w te janm fè yon plent nan pwojè a? Nan ka sa a, èske w te satisfè jan yo te trete plent lan ? Poukisa? SOUGWOUP: VSLA 110. Kijan / poukisa yo te chwazi w nan pwogram VSLA a? 111. Konbyen tan ou genyen nan sere lajan/epaye nan yon gwoup? 112. Konbyen manm gwoup epay/fè ekonomi/sere lajan ou a genyen? 113. Èske patisipasyon ou nan pwogram nan amelyore pwoblèm lamanjay nan fanmiw? Ki jan? 114. Èske patisipasyon ou nan pwogram lan amelyore sitiyasyon/estati/jan lòt moun wèw nan kay ou a? Nan kominote w la? 115. An jeneral, ki sa ou ta ka idantifye kòm fòs ak feblès prensipal pwogram nan ? 116. Dapre ou, èske gen opòtinite pou devlopman biznis kita ede rezoud pwoblèm lamanjay? Tanpri eksplike panse ou? 117. Kijan Pwogram koze lamanjay la ka ede amelyore devlopman biznis nan kominote w la? 118. Èske USAID te gen lòt pwogram li pou ankouraje mwayen pou viv (mwayen sibzistans) nan kominote w la? Ki enpak yo te genyen sou kominote w la? IMPEL | Implementer-Led Evaluation and Learning 110 Annex 6: KII and FGD Tools (French/Creole) 119. Èske moun yo konnen ke yo menm oswa fanmi yo dwe fè ekonomi lajan? Poukisa? E si se sa, ki sa yo fè ak lajan sa a? 120. Èske li posib pou jwenn kob prete pou w fè biznis nan kominote w la? Ki kalite prè? 121. Di kèk nan barye/obstak ki genyen pouw jwenn kob prete? 122. Ki pi bon mwayen yo ka sipòte kominote yo pi byen pou amelyore mwayen pou yo viv? 123. Pandan pwoje a fini la, kisa nou prevwa fe pou nou kontinye mennen aktivite epay/pre nan kominote a oswa nan oganizasyon nou an? CADEP & CADEC GWOUP KAP REPONN: PATNE PWOJÈ YO ENTEVYOU AK ENFOMATE YO • Konsèy Sipò pou Devlopman Patisipatif (CADEP) Pami sijè yo kouvri: CADEP/CADEC EQ1: Nan ki nivo pwojè a reyalize bi, objektif ak rezilta li te defini yo? EQ2: Baze sou done yo, eske rezilta pwojè a gen chans pou’l dire? EQ3: Nan chak sektè teknik pwojè a, ki sa ki fòs ak defi entèvansyon yo te chwazi ak ekzekite yo, epi ki jan kominote ki sible yo te resevwa entevansyon sa yo? EQ4: Ki leson prensipal nou te tire e ki pi bon pratik ki ta dwe rete kom enfòmasyon pou lòt pwojè kap vini nan peyi a? GID DISKISYON YO: PATNE PWOJÈ A EQ1: Nan ki nivo pwojè a reyalize bi, objektif ak rezilta li te defini yo? 1. Ki jan ou ta dekri kisa pwogram KORE LAVI a ap chèche reyalize/akonpli? 2. Depi Konbyen tan ou/yo te patisipe/enplike nan pwojè a? 3. Ki chanjman pwoje a pote nan zafè lamanjay nan kay la? Nan kominote w la? 4. Ki chanjman pwojè a pote nan zafè malnitrisyon timoun? 5. Ki jan pwojè a rive ogmante kapasite nou nan jere pwoblèm ke nap fè fas (move rekòt, inondasyon, kriz sante, elatriye)? 6. Ki sa nou panse ki te mache byen nan pwojè a? 7. Ki sa nou panse ki pa t’ byen mache nan pwojè a? 8. Ki bagay ou tap tann e ke pwojè a pa’t kapab fè? 9. Ki konsekans bagay sa (ki pa fet oswa ki pa fet ase) te gen sou fanmiw oswa nan kominote w? Final Performance Evaluation of the Kore Lavi DFAP in Haiti Annex 6: KII and FGD Tools (French/Creole) 111 10. Ki lòt fason pwojè a te ka ede w / fanmi w / kominote w la? 11. Kiyès nou panse ki benefisye pi fò nan pwogram lan? (Nan kay ou, nan kominote w la ?) 12. Èske nou panse pwojè a te rive jwenn moun ki pi frajil/vilnerab yo nan kominote w la? 13. Èske gen moun ki ta dwe patisipe nan aktivite yo, men ki pa ladann? Ki moun? 14. Ki kategori moun ou santi ki pa ta dwe patisipe nan aktivite yo? 15. Èske nou te remake gen anpil chanjman nan aktivite pwojè a depi l’ konmanse nan lane 2013? Bay kek egzanp. EQ2: Baze sou done yo, eske rezilta pwojè a gen chans pou yo rete, dire? 1. Èske pwojè a te pèmèt ou antreprann lòt aktivite? 2. Ki chans ki genyen pou ou kontinye bay sèvis apre pwojè a fini? 3. Èske pwojè a te pèmèt ou tabli/devlope nouvo relasyon ? (Kliyan nouvo, nouvo patnè) 4. Ki chans pou ou kontinye relasyon sa yo apre pwojè a fini? 5. Kounye a ke pwojè a fini, ki aktivite ki dirab, k ap rete? (Nivo Fanmi, nivo kominote a) 6. Ki aktivite nan pwojè yo ta dwe pouswiv nan lòt pwojè kap vini? 7. Ki aktivite nan pwojè a ki pa ta sipoze kontinye nan lòt pwojè kap vini? 8. Kijan aktivite sa yo ta dwe amelyore? EQ3: Nan chak sektè teknik pwojè a , ki sa ki fòs ak defi entèvansyon yo te chwazi ak ekzekite yo, epi ki jan kominote ki sible yo te resevwa entevansyon sa yo? 9. Ki sa ou te pi renmen nan pwojè sa a? 10. Ki sa ou pa’t renmen ditou nan pwojè sa a? 11. Kisa ou panse sou kalite sèvis yo bay nan pwojè a? 12. Daprè ou menm, ki kontrent/pwoblem ki te anpeche pwogram lan reyalize objektif li yo? 13. Ki sijesyon ou genyen pou fè fas (adrese) ak kontrent/pwoblem sa yo? EQ4: Ki leson prensipal nou te tire e ki pi bon pratik ki ta dwe rete kom enfòmasyon pou lòt pwojè kap vini nan peyi a? 14. Si ou te gen yon pwoblèm ak nenpòt aktivite oswa moun ki t ap travay nan pwojè a, ki sa ou te fè? (Liy Dirèk, anplwaye, bwat, moun deyò, elatriye) 15. Èske w konnen ke pwojè a te gen yon sistèm plent? 16. Èske w te janm fè yon plent nan pwojè a? Nan ka sa a, èske w te satisfè jan yo te trete plent la ? Poukisa? IMPEL | Implementer-Led Evaluation and Learning 112 Annex 6: KII and FGD Tools (French/Creole) CASEC RÉPONDANT: CASEC ENTREVUES AVEC LES INFORMATEURS LE GOUVERNEMENT DE HAÏTI • Conseils d’Administrations des Collectivités Territoriales (CASEC) QUESTIONS CLES DE L’EVALUATION QE1: Dans quelle mesure le projet a atteint le but, ses objectifs définis, et les résultats? QE2: D'après les données, quels sont les résultats du projet qui sont susceptibles d'être soutenus/durables? QE3: Dans chaque secteur technique, quels sont les points forts et les défis des interventions sélectionnées et leur mise en œuvre, et comment sont-elles reçues par les communautés cibles? QE4: Quelles sont les principales leçons apprises et les meilleures pratiques qui devraient informer les futurs projets dans le pays? GUIDE DE DISCUSSION: CASEC QE1: Dans quelle mesure le projet a atteint ses objectifs définis, le but et les résultats? 1. Quelle est votre position actuelle? Quelle est votre relation avec le Programme Kore Lavi? Depuis combien de temps travaillez-vous avec le programme? 2. Expliquez votre compréhension de la stratégie du programme KORE LAVI, à savoir, comment le programme devrait atteindre ses objectifs? 3. Comment votre institution a été renforcée par sa participation au projet KORE LAVI? 4. Comment le projet a-t-il engagé votre institution dans la planification et la mise en œuvre de ses activités? 5. Est-ce que la coordination entre les niveaux national et les départements est améliorée / aggravée à la suite des activités Kore Lavi? 6. Qu’est ce qui a bien marché et pourquoi? (Promouvoir la réalisation des objectifs du projet) 7. Qu'est-ce qui n’a pas bien marché et pourquoi? (Paralyserait l'atteinte des objectifs du projet) 8. Comment les défis ont été abordés? 9. Est-ce que les stratégies de ciblage ont permis d’atteindre des femmes et des personnes les plus vulnérables? 10. Pensez-vous que les interventions du projet ont été en conformité avec les besoins prioritaires de la communauté? Pourquoi ou pourquoi pas? Final Performance Evaluation of the Kore Lavi DFAP in Haiti Annex 6: KII and FGD Tools (French/Creole) 113 11. Comment le projet a-t-il engagé les communautés ? Est-ce-que le projet a pris au sérieux les opinions et l’engagement de la communauté ? 12. Avez-vous observé/constaté des changements dans la vulnérabilité et l'insécurité alimentaire à la suite du projet? Quells genres de changements? 13. Qui, selon vous, a le plus bénéficié du programme? 14. Y a-t-il d'autres personnes qui devraient être bénéficiaires du programme, mais ne le sont pas? S'il vous plaît décrivez-les pour nous. 15. Y a-t-il d'autres personnes qui devraient être bénéficiaires du programme Qui ne devrait pas être inclus dans les activités? Qui? 16. Comment le programme de coupons / « bons » alimentaires a-t-il répondu aux chocs (ouragan Matthew, la sécheresse)? 17. Quel a été l'impact du système de coupons / « bons » au cours de ces événements? QE2: D'après les données, les résultats du projet qui sont susceptibles d'être soutenus/durables? 18. Quels sont les changements durables facilités par le programme? 19. Quelles sont les stratégies de sortie mises en place pour assurer la durabilité? Êtes-vous confiant dans ces stratégies? 20. Quels sont les plus grands défis pour soutenir les changements facilités par le projet? 21. Quelle sera la motivation des participants et des partenaires pour continuer à maintenir ou soutenir ces activités après la fin du projet ? 22. Pensez-vous que organisations « non-projet » (Gouvernement, donateurs, ONG) continueront les activités qui ont été entreprises par KORE LAVI? (Gouvernement, donateurs, ONG) QE3: Dans chaque secteur technique, quels sont les points forts et les défis des interventions sélectionnées et leur mise en œuvre, et comment sont-ils reçus par les communautés cibles? 23. Que pensez-vous que les bénéficiaires ont apprécié le plus dans le projet? 24. Que pensez-vous que les bénéficiaires ont apprécié le moins dans le projet? 25. Quel est l'avis des partenaires du projet des activités promues par le projet (fournisseurs, centres de santé, etc.)? 26. A-t-il eu des conflits au sein des communautés au cours de la mise en œuvre? Comment le programme a adressé /a géré la situation? 27. Est-ce que les mécanismes de plainte mis en place par le projet ont-ils été efficaces ? Quelle est la nature des plaintes soumises au projet ? Est-ce qu’elles ont été résolues ? 28. Quelles ont été les conséquences inattendues, non intentionnelles, positives et / ou négatives des projets? IMPEL | Implementer-Led Evaluation and Learning 114 Annex 6: KII and FGD Tools (French/Creole) 29. Comment le projet a prévu et tenu compte de ces imprévus? QE4: Quelles sont les principales leçons apprises et les meilleures pratiques qui devraient informer les futurs projets dans le pays? 30. Selon vous, quelles sont les approches les plus innovantes du projet Kore Lavi? 31. Décrivez pour nous les principaux changements dans le projet / activités depuis le démarrage en 2013. Depuis l'extension 2016? Donnez des exemples. 32. Est-ce que le projet était devenu plus efficace dans la réalisation de ses produits et les résultats au fil du temps? Comment? 33. Comment le projet aurait pu être plus efficace? 34. Quelles ont été les leçons apprises ou à tirer de KORE LAVI qui peut améliorer la protection sociale en Haïti? Thème transversal : Autonomisation des femmes et Equité de genre 35. Les objectifs de genre étaient-ils intégrés dans toutes les activités du projet? Sinon, pourquoi? 36. Quels ont été les plus grands succès du projet en ce qui concerne l'inclusion de genre? 37. Quelles ont été les activités ayant moins de succès? Pourquoi? 38. Quels ont été les plus grands défis pour l'intégration du genre? 39. Pensez-vous que le programme a été en mesure d’autonomiser les femmes dans le domaine du programme? Comment? 40. Y a-t-il eu des résultats négatifs liés au genre? [Faites un sondage pour l'augmentation possible de la violence fondée sur le sexe en raison des coupons/« bons » donnés aux femmes]. 41. Pensez-vous que la programmation a eu un impact durable sur l'équité entre les sexes au sein des communautés? Comment? 42. Quels ont été les principaux enseignements tirés de la mise en œuvre du programme pour parvenir à l'égalité entre les sexes? 43. Quelles sont les activités qui devraient être incluses au programme pour promouvoir l'égalité et l'équité de genre au niveau des futurs programmes? MAST RÉPONDANT: MAST ENTREVUES AVEC LES INFORMATEURS LE GOUVERNEMENT DE HAÏTI • Ministère des Affaires Sociales et du Travail (MAST) Final Performance Evaluation of the Kore Lavi DFAP in Haiti Annex 6: KII and FGD Tools (French/Creole) 115 Parmi les sujets abordés: MAST QE1: Dans quelle mesure le projet a atteint le but, ses objectifs définis, et les résultats? QE2: D'après les données, quels sont les résultats du projet qui sont susceptibles d'être soutenus/durables? QE3: Dans chaque secteur technique, quels sont les points forts et les défis des interventions sélectionnées et leur mise en œuvre, et comment sont-elles reçues par les communautés cibles? QE4: Quelles sont les principales leçons apprises et les meilleures pratiques qui devraient informer les futurs projets dans le pays? GUIDE DE DISCUSSION: MAST QE1: Dans quelle mesure le projet a atteint ses objectifs définis, le but et les résultats? 1. Quelle est votre position actuelle? Quelle est votre relation avec le Programme Kore Lavi? Depuis combien de temps travaillez-vous avec le programme? 2. Expliquez votre compréhension de la stratégie du programme KORE LAVI, à savoir, comment le programme devrait atteindre ses objectifs? 3. Quelles sont les principales activités mises en œuvre? 4. Qu’est ce qui a bien marché et pourquoi? (Promouvoir la réalisation des objectifs du projet) 5. Qu'est-ce qui n’a pas bien marché et pourquoi? (Paralyserait l'atteinte des objectifs du projet) 6. Comment les défis ont été abordés? 7. Avez-vous observé/constaté des changements dans l'insécurité alimentaire en raison du projet? Quels genres de changements? 8. Avez-vous observé/constaté des changements dans la vulnérabilité à la suite du projet? Quels genres de changements? 9. Avez-vous observé/constaté des changements parmi les ménages pauvres ou extrêmement pauvres en raison du programme? 10. Comment le projet a-t-il engagé les communautés dans la planification et la mise en œuvre de ses activités? 11. Pensez-vous que les interventions du projet ont été en conformité avec les besoins prioritaires de la communauté? 12. Les activités du projet sont-elles appropriées pour atteindre les ménages les plus pauvres? Pourquoi ou pourquoi pas? 13. Les activités du projet sont-elles efficaces pour atteindre les ménages les plus pauvres? Pourquoi ou pourquoi pas? IMPEL | Implementer-Led Evaluation and Learning 116 Annex 6: KII and FGD Tools (French/Creole) 14. Est-ce que les stratégies de ciblage ont permis d’atteindre les objectifs du projet en matière de sensibilisation des femmes et des personnes les plus vulnérables? 15. Êtes-vous au courant de l'indice de Privation et la vulnérabilité d'Haïti (HDVI)? 16. Si oui, qu'est-ce que vous sentez comme étant ses forces et ses faiblesses? 17. Quelle est votre opinion de son efficacité dans l’identification systématique des ménages les plus pauvres? 18. Qui, selon vous, a le plus bénéficié du programme? 19. Y a-t-il d'autres personnes qui devraient être bénéficiaires du programme, mais ne le sont pas? S'il vous plaît décrivez-les pour nous. 20. Qui aurait dû être inclus dans les activités mais ne l’était pas? Qui? 21. Qui ne devrait pas être inclus dans les activités? Qui? 22. Comment le programme de coupons / « bons » alimentaires a-t-il répondu aux chocs (ouragan Matthew, la sécheresse)? 23. Quels ont été les défis au programme de coupons / « bons » au cours de ces événements? 24. Quel a été l'impact du système de coupons / « bons » au cours de ces événements? 25. Quelle a été la contribution du projet à l'établissement d'un système de filet de sécurité sociale? 26. Quels ont été les défis à renforcer les programmes de filet de sécurité sociale? Comment le projet a-t-il-surmonté ces défis? 27. Quelles ont été les réalisations dans le suivi du filet de sécurité sociale? Quels sont les avantages? 28. La mise en œuvre de la surveillance/du suivi a-t-elle été identique à travers les 5 départements? Pourquoi pas? 29. Êtes-vous familier avec le système d'évaluation de renforcement des capacités du MAST? Pensez￾vous qu'il a été efficace? Pourquoi ou pourquoi pas? A-t-il été authentique? A-t-il pris en compte les facteurs externes? QE2: D'après les données, les résultats du projet qui sont susceptibles d'être soutenus/durables? 30. Comment les changements permanents sont facilités par le programme? 31. Quels sont les plus grands défis pour soutenir les changements facilités par le projet? Comment peut-on aborder ces défis? 32. Quel est l'avis des bénéficiaires des activités promues par le projet? 33. Quel est l'avis des partenaires du projet des activités promues par le projet (fournisseurs, centres de santé, etc.)? 34. Quelles étaient les perceptions des communautés sur la qualité des services fournis par le projet? Final Performance Evaluation of the Kore Lavi DFAP in Haiti Annex 6: KII and FGD Tools (French/Creole) 117 35. Quelle sera la motivation des participants et des partenaires pour continuer à maintenir ou soutenir ces activités après la fin du projet ? 36. Pensez-vous que les fournisseurs de services auront la capacité de continuer à fournir des services après la fin du projet? Quels services? 37. Quelle a été la participation d'autres organisations « non-projet » au cours de l’implémentation du projet KORE LAVI? (Gouvernement, donateurs, ONG) 38. Pensez-vous que ces organisations continueront les activités qui ont été entreprises par KORE LAVI? (Gouvernement, donateurs, ONG) 39. Comment la date de fin du programme a-t-elle été communiquée aux bénéficiaires/communautés cibles ? Quelle a été leur réaction? 40. Quelles sont les stratégies de sortie mises en place pour assurer la durabilité? Êtes-vous confiant dans ces stratégies? QE3: Dans chaque secteur technique, quels sont les points forts et les défis des interventions sélectionnées et leur mise en œuvre, et comment sont-ils reçus par les communautés cibles? 41. Que pensez-vous que les bénéficiaires ont apprécié le plus dans le projet? 42. Que pensez-vous que les bénéficiaires ont apprécié le moins dans le projet? 43. Y-avait-il des bénéficiaires qui ont eu beaucoup plus de succès que d'autres? Qui sont –ils et pourquoi? 44. A-t-il eu des conflits au sein des communautés au cours de la mise en œuvre? Comment le programme a adressé /a géré la situation? 45. Comment les mécanismes de plainte mis en place par le projet ont-ils été efficaces ? 46. Quelles ont été les conséquences inattendues, non intentionnelles, positives et / ou négatives des projets? 47. Comment le projet a prévu et tenu compte de ces imprévus? QE4: Quelles sont les principales leçons apprises et les meilleures pratiques qui devraient informer les futurs projets dans le pays? 48. Selon vous, quelles sont les approches les plus innovantes du projet Kore Lavi? 49. Décrivez pour nous les principaux changements dans le projet / activités depuis le démarrage en 2013. Depuis l'extension 2016? Donnez des exemples. 50. Est-ce que le projet était devenu plus efficace dans la réalisation de ses produits et les résultats au fil du temps? Comment? 51. Comment le projet aurait pu être plus efficace? 52. Quelles ont été les leçons apprises ou à tirer de KORE LAVI? IMPEL | Implementer-Led Evaluation and Learning 118 Annex 6: KII and FGD Tools (French/Creole) Thème transversal : Autonomisation des femmes et Equité de genre 53. Quelles sont les activités que vous exercez sur le genre? 54. Quels ont été quelques-uns des plus grands succès à ce jour en ce qui concerne l'inclusion de genre? Pour qui? 55. Quelles ont été les activités liées au genre ayant plus de succès? Quelles ont été les activités de formation ayant moins de succès? Pourquoi? Qu’est-ce qui se passe après les formations? 56. Quels ont été les plus grands défis pour l'intégration du genre? 57. Les objectifs de genre étaient-ils intégrés dans le programme? Sinon, pourquoi? 58. Comment ont-ils été suivis? Comment ont-ils été évalués? 59. Pensez-vous que le programme a été en mesure d’autonomiser les femmes dans le domaine du programme? Comment? 60. À votre avis, les activités sur la problématique /la prise en compte de genre ont-elles été identifiées? Les activités ont-elles pu répondre aux besoins des communautés? 61. Est-ce que les normes/croyances culturelles de genre ont eu un impact sur les résultats de la communauté? Comment? 62. Y a-t-il eu des résultats négatifs liés au genre? [Faites un sondage pour l'augmentation possible de la violence fondée sur le sexe en raison des coupons/« bons » donnés aux femmes]. 63. Pensez-vous que la programmation a eu un impact durable sur l'équité entre les sexes au sein des communautés? Comment? 64. Quels ont été les principaux enseignements tirés de la mise en œuvre du programme, ainsi que des partenariats avec différentes parties prenantes pour parvenir à l'égalité entre les sexes? 65. Quelles sont les activités qui devraient être incluses au programme pour promouvoir l'égalité et l'équité de genre au niveau des futurs programmes? Pourquoi? SOUS-GROUPE: MAST et AEIC 66. Comment votre institution a été renforcée par sa participation au projet KORE LAVI? 67. Pensez-vous que le soutien apporté par le projet Kore Lavi était approprié à vos besoins? 68. Quelles sont les principales forces et faiblesses du programme relatif au renforcement des capacités des institutions du gouvernement haïtien? 69. Est-ce que la coordination entre les niveaux national et le personnel MAST au niveau des départements est améliorée / aggravée à la suite des activités Kore Lavi? 70. Quelle est l'apprentissage clé / critique émergeant du projet qui peut améliorer la protection sociale en Haïti? Final Performance Evaluation of the Kore Lavi DFAP in Haiti Annex 6: KII and FGD Tools (French/Creole) 119 Non-Bénéficiaires (Coupons) GWOUP KAP REPONN NAN: NON-BENEFISYÈ PWOJÈ YO DISKISYON AN GWOUP NON-Benefisyè yo nan pwojè a/SO2 (pas de coupon) Pami sijè nou kouvri: BENEFISYÈ PWOJÈ EQ1: Nan ki nivo pwojè a reyalize bi, objektif ak rezilta li te defini yo? EQ2: Baze sou done yo, eske rezilta pwojè a gen chans pou’l dire? EQ3: Nan chak sektè teknik pwojè a, ki sa ki fòs ak defi entèvansyon yo te chwazi ak ekzekite yo, epi ki jan kominote ki sible yo te resevwa entevansyon sa yo? EQ4: Ki leson prensipal nou te tire e ki pi bon pratik ki ta dwe rete kom enfòmasyon pou lòt pwojè kap vini nan peyi a? GID POU DISKISYON YO : NON-BENEFISYÈ PWOJÈ YO EQ1: Nan ki nivo pwojè a reyalize bi, objektif ak rezilta li te defini yo? 1. Ki jan ou ta dekri kisa pwogram KORE LAVI a ap chèche reyalize/akonpli? 2. Depi Konbyen tan moun yo nan kominote nous la te patisipe/enplike nan pwojè a? 3. Ki chanjman pwoje a pote nan zafè lamanjay nan kay la? Nan kominote w la? 4. Ki chanjman pwojè a pote nan zafè malnitrisyon timoun? 5. Ki jan pwojè a rive ogmante kapasite nou nan jere pwoblèm ke nap fè fas (move rekòt, inondasyon, kriz sante, elatriye)? 6. Ki sa nou panse ki te mache byen nan pwojè a? 7. Ki sa nou panse ki pa t’ byen mache nan pwojè a? 8. Ki bagay ou tap tann e ke pwojè a pa’t kapab fè? 9. Ki konsekans bagay sa (ki pa fet oswa ki pa fet ase) te gen nan kominote w? 10. Ki lòt fason pwojè a te ka ede kominote w la? 11. Kiyès nou panse ki benefisye pi fò nan pwogram lan? (nan kominote w la ?) 12. Èske nou panse pwojè a te rive jwenn moun ki pi frajil/vilnerab yo nan kominote w la? 13. Èske gen moun ki ta dwe patisipe nan aktivite yo, men ki pa ladann? Ki moun? 14. Ki kategori moun ou santi ki pa ta dwe patisipe nan aktivite yo? 15. Èske ou panse seleksyon patisipan yo te fè byen? Èske li te kòrèk ak jis? Poukisa? IMPEL | Implementer-Led Evaluation and Learning 120 Annex 6: KII and FGD Tools (French/Creole) 16. Èske aktivite pwogram yo te koze pwoblèm ant benefisyè ak benefisyè ki pa nan kominote w la? 17. Èske fanmi ki te resevwa koupon yo te pataje avèk lòt fanmi nan kominote a? 18. Pou moun nan kominote a tankou ou ki pa te resevwa koupon, te gen lòt pwogram oswa aktivite ki te ede fanmi ou? Kisa? 19. Èske nou te remake gen anpil chanjman nan aktivite pwojè a depi l’ konmanse nan lane 2013? Bay kek egzanp. 20. Ki pati nan sistèm sèvis koupon an ki te mache pi byen pou ou ak pou lòt moun nan kominote w la ? ki pati ki pa’t mache byen? Eksplike. 21. Ki sijesyon ou genyen pou pwogram koupon an? 22. Kisa nou we kom diferans ant le se gason oswa se fanm ki resevwa koupon pou fanmi an? 23. Ki chanjman ou wè ki ta dwe fèt nan sistèm koupon an pou amelyore rezilta pou fanm ak gason? EQ2: Baze sou done yo, eske rezilta pwojè a gen chans pou’l dire? 24. Kounye a ke pwojè a fini, ki sa ou panse ke li genyen kòm enpak/chanjman (pozitif oswa negatif) ? (Nan nivo fanmi yo, nan nivo kominote a). 25. Ki aktivite pwojè a yo ta dwe kontinye fè nan lot pwojè ki gen pou vini? Poukisa ? 26. Ki aktivite pwojè a yo pa ta dwe tounen avè ankò nan lot pwojè ki gen pou vini? Poukisa ? 27. Kijan aktivite sa yo ka amelyore? EQ3: Nan chak sektè teknik pwojè a , ki sa ki fòs ak defi entèvansyon yo te chwazi ak ekzekite yo, epi ki jan kominote ki sible yo te resevwa entevansyon sa yo? 28. Ki sa ou te pi renmen nan pwojè sa a? 29. Ki sa ou pa’t renmen ditou nan pwojè sa a? 30. Kisa ou panse sou kalite sèvis yo bay nan pwojè a? 31. Daprè ou menm, ki kontrent/pwoblem ki te anpeche pwogram lan reyalize objektif li yo? 32. Ki sijesyon ou genyen pou fè fas (adrese) ak kontrent/pwoblem sa yo? EQ4: Ki leson prensipal nou te tire e ki pi bon pratik ki ta dwe rete kom enfòmasyon pou lòt pwojè kap vini nan peyi a? 33. Si moun yo nan komunite sa a te gen yon pwoblèm ak nenpòt aktivite oswa moun ki t ap travay nan pwojè a, ki sa moun yo te fè? (Liy Dirèk, anplwaye, bwat, moun deyò, elatriye) 34. Èske w konnen ke pwojè a te gen yon sistèm plent? Final Performance Evaluation of the Kore Lavi DFAP in Haiti Annex 6: KII and FGD Tools (French/Creole) 121 35. Èske w te janm fè yon plent nan pwojè a? Nan ka sa a, èske w te satisfè jan yo te trete plent lan ? Poukisa? Partenaires RÉPONDANT: Partenaire (ACF/CARE/WORLD VISION/PAM) ENTREVUES AVEC LES INFORMATEURS ACF/CARE/WORLD VISION/PAM Parmi les sujets abordés: ACF/CARE/WORLD VISION/PAM QE1: Dans quelle mesure le projet a atteint le but, ses objectifs définis, et les résultats? QE2: D'après les données, quels sont les résultats du projet qui sont susceptibles d'être soutenus/durables? QE3: Dans chaque secteur technique, quels sont les points forts et les défis des interventions sélectionnées et leur mise en œuvre, et comment sont-elles reçues par les communautés cibles? QE4: Quelles sont les principales leçons apprises et les meilleures pratiques qui devraient informer les futurs projets dans le pays? GUIDE DE DISCUSSION: ACF/CARE/WORLD VISION/PAM QE1: Dans quelle mesure le projet a atteint ses objectifs définis, le but et les résultats? 1. Quelle est votre position actuelle? Quelle est votre relation avec le Programme Kore Lavi? Depuis combien de temps travaillez-vous avec le programme? 2. Expliquez votre compréhension de la stratégie du programme KORE LAVI, à savoir, comment le programme devrait atteindre ses objectifs? 3. Quelles sont les principales activités mises en œuvre? 4. Qu’est ce qui a bien marché et pourquoi? 5. Qu'est-ce qui n’a pas bien marché et pourquoi? (Paralyserait l'atteinte des objectifs du projet) 6. Comment les défis ont été abordés? 7. Avez-vous observé/constaté des changements dans l'insécurité alimentaire en raison du projet? Quels genres de changements? 8. Avez-vous observé/constaté des changements dans la vulnérabilité à la suite du projet? Quels genres de changements? 9. Avez-vous observé/constaté des changements parmi les ménages pauvres ou extrêmement pauvres en raison du programme? 10. Comment le projet a-t-il engagé les communautés dans la planification et la mise en œuvre de ses activités? IMPEL | Implementer-Led Evaluation and Learning 122 Annex 6: KII and FGD Tools (French/Creole) 11. Pensez-vous que les interventions du projet ont été en conformité avec les besoins prioritaires de la communauté? 12. Les activités du projet sont-elles appropriées pour atteindre les ménages les plus pauvres? Pourquoi ou pourquoi pas? 13. Les activités du projet sont-elles efficaces pour atteindre les ménages les plus pauvres? Pourquoi ou pourquoi pas? 14. Est-ce que les stratégies de ciblage ont permis d’atteindre les objectifs du projet en matière de sensibilisation des femmes et des personnes les plus vulnérables? 15. Êtes-vous au courant de l'indice de Privation et la vulnérabilité d'Haïti (HDVI)? 16. Si oui, qu'est-ce que vous sentez comme étant ses forces et ses faiblesses? 17. Quelle est votre opinion de son efficacité dans l’identification systématique des ménages les plus pauvres? 18. Qui, selon vous, a le plus bénéficié du programme? 19. Y a-t-il d'autres personnes qui devraient être bénéficiaires du programme, mais ne le sont pas? S'il vous plaît décrivez-les pour nous. 20. Qui aurait dû être inclus dans les activités mais ne l’était pas? Qui? 21. Qui ne devrait pas être inclus dans les activités? Qui? 22. Comment le programme de coupons / « bons » alimentaires a-t-il répondu aux chocs (ouragan Matthew, la sécheresse)? 23. Quels ont été les défis au programme de coupons / « bons » au cours de ces événements? 24. Quel a été l'impact du système de coupons / « bons » au cours de ces événements? 25. Quelle a été la contribution du projet à l'établissement d'un système de filet de sécurité sociale? 26. Quels ont été les défis à renforcer les programmes de filet de sécurité sociale? Comment le projet a-t-il-surmonté ces défis? 27. Quelles ont été les réalisations dans le suivi du filet de sécurité sociale? Quels sont les avantages? 28. La mise en œuvre de la surveillance/du suivi a-t-elle été identique à travers les 5 départements? Pourquoi pas? 29. Êtes-vous familier avec le système d'évaluation de renforcement des capacités du MAST? Pensez-vous qu'il a été efficace? Pourquoi ou pourquoi pas? A-t-il été authentique? A-t-il pris en compte les facteurs externes? QE2: D'après les données, les résultats du projet qui sont susceptibles d'être soutenus/durables? 30. Comment les changements permanents sont facilités par le programme? Final Performance Evaluation of the Kore Lavi DFAP in Haiti Annex 6: KII and FGD Tools (French/Creole) 123 31. Quels sont les plus grands défis pour soutenir les changements facilités par le projet? Comment peut-on aborder ces défis? 32. Quel est l'avis des bénéficiaires des activités promues par le projet? 33. Quel est l'avis des partenaires du projet des activités promues par le projet (fournisseurs, centres de santé, etc.)? 34. Quelles étaient les perceptions des communautés sur la qualité des services fournis par le projet? 35. Quelle sera la motivation des participants et des partenaires pour continuer à maintenir ou soutenir ces activités après la fin du projet ? 36. Pensez-vous que les fournisseurs de services auront la capacité de continuer à fournir des services après la fin du projet? Quels services? 37. Quelle a été la participation d'autres organisations « non-projet » au cours de l’implémentation du projet KORE LAVI? (Gouvernement, donateurs, ONG) 38. Pensez-vous que ces organisations continueront les activités qui ont été entreprises par KORE LAVI? (Gouvernement, donateurs, ONG) 39. Comment la date de fin du programme a-t-elle été communiquée aux bénéficiaires/communautés cibles ? Quelle a été leur réaction? 40. Quelles sont les stratégies de sortie mises en place pour assurer la durabilité? Êtes-vous confiant dans ces stratégies? QE3: Dans chaque secteur technique, quels sont les points forts et les défis des interventions sélectionnées et leur mise en œuvre, et comment sont-ils reçus par les communautés cibles? 41. Que pensez-vous que les bénéficiaires ont apprécié le plus dans le projet? 42. Que pensez-vous que les bénéficiaires ont apprécié le moins dans le projet? 43. Y-avait-il des bénéficiaires qui ont eu beaucoup plus de succès que d'autres? Qui sont –ils et pourquoi? 44. A-t-il eu des conflits au sein des communautés au cours de la mise en œuvre? Comment le programme a adressé /a géré la situation? 45. Comment les mécanismes de plainte mis en place par le projet ont-ils été efficaces ? 46. Quelles ont été les conséquences inattendues, non intentionnelles, positives et / ou négatives des projets? 47. Comment le projet a prévu et tenu compte de ces imprévus? QE4: Quelles sont les principales leçons apprises et les meilleures pratiques qui devraient informer les futurs projets dans le pays? 48. Selon vous, quelles sont les approches les plus innovantes du projet Kore Lavi? IMPEL | Implementer-Led Evaluation and Learning 124 Annex 6: KII and FGD Tools (French/Creole) 49. Décrivez pour nous les principaux changements dans le projet / activités depuis le démarrage en 2013. Depuis l'extension 2016? Donnez des exemples. 50. Est-ce que le projet était devenu plus efficace dans la réalisation de ses produits et les résultats au fil du temps? Comment? 51. Comment le projet aurait pu être plus efficace? 52. Quelles ont été les leçons apprises ou à tirer de KORE LAVI? Thème transversal : Autonomisation des femmes et Equité de genre 53. Quelles sont les activités que vous exercez sur le genre? 54. Quels ont été quelques-uns des plus grands succès à ce jour en ce qui concerne l'inclusion de genre? Pour qui? 55. Quelles ont été les activités liées au genre ayant plus de succès? Quelles ont été les activités de formation ayant moins de succès? Pourquoi? Qu’est-ce qui se passe après les formations? 56. Quels ont été les plus grands défis pour l'intégration du genre? 57. Les objectifs de genre étaient-ils intégrés dans le programme? Sinon, pourquoi? 58. Comment ont-ils été suivis? Comment ont-ils été évalués? 59. Pensez-vous que le programme a été en mesure d’autonomiser les femmes dans le domaine du programme? Comment? 60. À votre avis, les activités sur la problématique /la prise en compte de genre ont-elles été identifiées? Les activités ont-elles pu répondre aux besoins des communautés? 61. Est-ce que les normes/croyances culturelles de genre ont eu un impact sur les résultats de la communauté? Comment? 62. Y a-t-il eu des résultats négatifs liés au genre? [Faites un sondage pour l'augmentation possible de la violence fondée sur le sexe en raison des coupons/« bons » donnés aux femmes]. 63. Pensez-vous que la programmation a eu un impact durable sur l'équité entre les sexes au sein des communautés? Comment? 64. Quels ont été les principaux enseignements tirés de la mise en œuvre du programme, ainsi que des partenariats avec différentes parties prenantes pour parvenir à l'égalité entre les sexes? 65. Quelles sont les activités qui devraient être incluses au programme pour promouvoir l'égalité et l'équité de genre au niveau des futurs programmes? Pourquoi? SOUS-GROUPE: Personnel de Suivi/Evaluation (S/E) 66. Quel a été / est le rôle de votre organisation dans la conception, le déploiement ou la mise en œuvre de l'indice de Privation et la vulnérabilité d'Haïti (HDVI)? Final Performance Evaluation of the Kore Lavi DFAP in Haiti Annex 6: KII and FGD Tools (French/Creole) 125 67. Qu’est-ce qui a été particulièrement efficace dans la conception et le déploiement du HDVI? 68. Qu’est-ce qui a été particulièrement difficile (a représenté un défi) dans la conception et le déploiement du HDVI? 69. Comment la mise en œuvre de HDVI doit être améliorée? 70. Quels sont les éléments de HDVI trouvez-vous plus utiles? 71. Quelle est la méthodologie de collecte des données HDVI systématiquement utilisée dans les zones géographiques et les types / groupes bénéficiaires? 72. Quelle est votre expérience avec le score de ménage /notation de la carte de vulnérabilité de ménage? 73. Qu’est-ce qui fonctionne bien avec la méthode de notation des ménages? Comment peut-elle être améliorée? 74. Le MAST fonctionne-t-il efficacement vers un registre bénéficiaire singulier? Sinon, quelles sont les contraintes qui nuisent au progrès vers cet objectif déclaré? 75. Quels sont les apprentissages clés qui émergent de l’implémentation du HDVI qui pourraient améliorer la qualité de la protection sociale dans les programmes en Haïti et FFP globalement? Vendeurs GWOUP KAP REPONN: PATNE PWOJÈ YO ENTEVYOU AK ENFOMATE YO • Konpayi kap bay sèvis yo ( Founisè yo ) Pami sijè yo kouvri: VANDÈ YO EQ1: Nan ki nivo pwojè a reyalize bi, objektif ak rezilta li te defini yo? EQ2: Baze sou done yo, eske rezilta pwojè a gen chans pou’l dire? EQ3: Nan chak sektè teknik pwojè a, ki sa ki fòs ak defi entèvansyon yo te chwazi ak ekzekite yo, epi ki jan kominote ki sible yo te resevwa entevansyon sa yo? EQ4: Ki leson prensipal nou te tire e ki pi bon pratik ki ta dwe rete kom enfòmasyon pou lòt pwojè kap vini nan peyi a? GID DISKISYON YO: PATNE PWOJÈ A EQ1: Nan ki nivo pwojè a reyalize bi, objektif ak rezilta li te defini yo? 1. Ki jan ou ta dekri kisa pwogram KORE LAVI a ap chèche reyalize/akonpli? 2. Depi Konbyen tan ou/yo te patisipe/enplike nan pwojè a? IMPEL | Implementer-Led Evaluation and Learning 126 Annex 6: KII and FGD Tools (French/Creole) 3. Ki chanjman pwoje a pote nan zafè lamanjay nan kay la? Nan kominote w la? 4. Ki chanjman pwojè a pote nan zafè malnitrisyon timoun? 5. Ki jan pwojè a rive ogmante kapasite nou nan jere pwoblèm ke nap fè fas (move rekòt, inondasyon, kriz sante, elatriye)? 6. Ki sa nou panse ki te mache byen nan pwojè a? 7. Ki sa nou panse ki pa t’ byen mache nan pwojè a? 8. Ki bagay ou tap tann e ke pwojè a pa’t kapab fè? 9. Ki konsekans bagay sa (ki pa fet oswa ki pa fet ase) te gen sou fanmiw oswa nan kominote w? 10. Ki lòt fason pwojè a te ka ede w / fanmi w / kominote w la? 11. Kiyès nou panse ki benefisye pi fò nan pwogram lan? (Nan kay ou, nan kominote w la ?) 12. Èske nou panse pwojè a te rive jwenn moun ki pi frajil/vilnerab yo nan kominote w la? 13. Èske gen moun ki ta dwe patisipe nan aktivite yo, men ki pa ladann? Ki moun? 14. Ki kategori moun ou santi ki pa ta dwe patisipe nan aktivite yo? 15. Èske nou te remake gen anpil chanjman nan aktivite pwojè a depi l’ konmanse nan lane 2013? Bay kek egzanp. EQ2: Baze sou done yo, eske rezilta pwojè a gen chans pou yo rete, dire? 1. Èske pwojè a te pèmèt ou antreprann lòt aktivite? 2. Ki chans ki genyen pou ou kontinye bay sèvis apre pwojè a fini? 3. Èske pwojè a te pèmèt ou tabli/devlope nouvo relasyon ? (Kliyan nouvo, nouvo patnè) 4. Ki chans pou ou kontinye relasyon sa yo apre pwojè a fini? 5. Kounye a ke pwojè a fini, ki aktivite ki dirab, k ap rete? (Nivo Fanmi, nivo kominote a) 6. Ki aktivite nan pwojè yo ta dwe pouswiv nan lòt pwojè kap vini? 7. Ki aktivite nan pwojè a ki pa ta sipoze kontinye nan lòt pwojè kap vini? 8. Kijan aktivite sa yo ta dwe amelyore? EQ3: Nan chak sektè teknik pwojè a , ki sa ki fòs ak defi entèvansyon yo te chwazi ak ekzekite yo, epi ki jan kominote ki sible yo te resevwa entevansyon sa yo? 9. Ki sa ou te pi renmen nan pwojè sa a? 10. Ki sa ou pa’t renmen ditou nan pwojè sa a? 11. Kisa ou panse sou kalite sèvis yo bay nan pwojè a? Final Performance Evaluation of the Kore Lavi DFAP in Haiti Annex 6: KII and FGD Tools (French/Creole) 127 12. Daprè ou menm, ki kontrent/pwoblem ki te anpeche pwogram lan reyalize objektif li yo? 13. Ki sijesyon ou genyen pou fè fas (adrese) ak kontrent/pwoblem sa yo? EQ4: Ki leson prensipal nou te tire e ki pi bon pratik ki ta dwe rete kom enfòmasyon pou lòt pwojè kap vini nan peyi a? 14. Si ou te gen yon pwoblèm ak nenpòt aktivite oswa moun ki t ap travay nan pwojè a, ki sa ou te fè? (Liy Dirèk, anplwaye, bwat, moun deyò, elatriye) 15. Èske w konnen ke pwojè a te gen yon sistèm plent? 16. Èske w te janm fè yon plent nan pwojè a? Nan ka sa a, èske w te satisfè jan yo te trete plent la ? Poukisa? SOUGWOUP: • Vandè yo 1. Kijan ou santi ou sou eksperyans ou kòm yon founisè pou pwojè Kore Lavi a? 2. Èske travay ak pwojè a chanje kantite kliyan ou konn sèvi? Èske li ogmante kantite lajan ou fè nan biznis ou a? 3. Èske pwojè a te okazyone yon chanjman pri nan pwodwi ke wap vann yo? 4. Èske pwojè a te pèmèt yon chanjman nan demand pou pwodwi ke wap vann yo? 5. Èske ou te kapab reponn a tout demann pou pwodwi’w tap vann yo? 6. Nan pwodwi ou konn achte pou revann, kisa ki te plis vann ak koupon pwojè a? 7. Èske pwojè a te okazyone yon chanjman nan pwodwi ou te konn achte pouw revann yo ? IMPEL | Implementer-Led Evaluation and Learning 128 Annex 7: Project Performance Summary Tables ANNEX 7: PROJECT PERFORMANCE SUMMARY TABLES IPTT Reporting 2014 – 2017 LOA Target LOA Actual % of Target Achieved Indicator 5% 1.61% 310% Percentage of beneficiaries incorrectly included/excluded 81,318 102,180 126% Number of people trained in child health and nutrition through USG-supported programs 170 199 117% Number of MDAs trained on GBV and GBV referral processes 17 18 106% Number of communes in which vulnerability targeting methodology is implemented 70% 73.7% 105% Percentage of people reached by SBCC messages 80% 83% 104% Percentage of food accepted for voucher redemption that is locally grown 1,200 1,241 103% Number of VSLA groups established 31 32 103% Number of civil society organizations trained to use social audit processes 17,700 18,158 103% Number of households enrolled in the food voucher-based safety net 275 279 101% Number of activities (meetings, media campaigns, awareness raising sessions) to sensitize stakeholders on gender issues 95% 95% 100% Percentage of vouchers redeemed 20 20 100% Number of MAST staff trained to manage database 12 12 100% Number of MCFDF recommendations integrated by governmental institutions working on vulnerability targeting and safety net programs 5 5 100% Number of departments where local governmental structures have a plan to support implementation of safety net programs 16 16 100% Number of CADEPs (or other identified civil society structures) who developed capacity building plan to support safety net programs 1,000 998 100% Number of vendors who are providing food under the safety net 50% 49% 98% Percentage of vulnerability recourse committee members who are women 178,051 172,493 97% Number of rural households benefiting directly from USG interventions 179,251 173,554 97% Number of vulnerable households benefiting directly from USG interventions 98 94 96% Number of trainings on ENA, IYCF, GMP and CMAM Final Performance Evaluation of the Kore Lavi DFAP in Haiti Annex 7: Project Performance Summary Tables 129 LOA Target LOA Actual % of Target Achieved Indicator 135 129 96% Number of MAST-led activities (meetings, forums, recommendations given) to coordinate implementation of safety net and other social protection programs 89,294 84,871 95% Number of children under five whose parents/caretakers received behavior change communication interventions that promote essential infant and young child feeding behaviors 603 568 94% Number of community public awareness campaigns on ENA, IYCF, GMP, and CMAM implemented 114,353 104,968 92% Number of children under five reached by USG-supported programs 821,026 752,049 92% Number of people benefiting from USG-supported social assistance programming 80% 72% 90% Percentage of CADEPs (or other identified civil society structures) supporting public awareness campaigns on ENA, IYCF, GMP or CMAM 20 18 90% Number of MAST staff trained to coordinate and monitor safety net and other social protection programs 163,861 143,978 88% Number of individuals receiving supplementary, conditional rations 220 189 86% Number of MDAs completing MSPP nutrition training curricula 147,475 124,900 85% Number of households reached through Care groups 16,500 13,732 83% Number of households in the database with a CIN 25,059 20,097 80% Number of children under five who were admitted for treatment of moderate acute malnutrition 30,723 24,550 80% Number of children under two (0-23 months) reached with community-level nutrition interventions through USG-supported programs 10,152 7,911 78% Number of pregnant women reached with nutrition interventions through USG supported programs 18,000 13,885 77% Number of children under 5 (6-59 months) referred to health facilities for treatment of MAM 50% 38% 77% % of safety net participants in VSLA groups (non-cumulative) 100% 76% 76% Percentage of complaints cases addressed 21 16 76% Number of beneficiary verification reports 29 22 76% Number of civil society organizations engaged in social audit processes IMPEL | Implementer-Led Evaluation and Learning 130 Annex 7: Project Performance Summary Tables LOA Target LOA Actual % of Target Achieved Indicator 50% 27% 54% Percentage of CADEP (or other identified civil society structures) representatives who are women 2 1 50% Number of contingency plans incorporating vulnerability targeting tool 40% 14% 35% Percentage of eligible children in a health center catchment area enrolled in GMP services 50% 12% 23% Percent of children identified with MAM receiving food who have met GoH recuperation standards 3,600 0 0% Number of safety net participants linked with complementary services (other than VSLA groups) 2 0 0% Number of DPC staff trained to use vulnerability database IPTT Reporting 2018 – 2019 LOA Target LOA Actual % of Target Achieved Indicator 72 154 214% Number of Departmental MAST staff trained in complementary trainings (ICT, program management and administration) 295 113 116% FFP-78: Number of individuals receiving nutrition-related professional training through USG-supported programs 16 18 113% Number of community-level activities integrating dialogues on gender equality and women’s empowerment 1,536 1,697 110% Number of participatory meetings, discussion sessions, follow-up sessions between community structures, CBO networks, local authorities and community leaders to follow the implementation of the safety net 80% 87% 109% Percentage of vendors providing food under the safety net who are women 2 2.14 107% Progression of the SIMAST staff capacity, as measured on the institutionalization scale 22% 23% 105% Percentage of beneficiary households in the MAST-hosted Household Vulnerability Database who have been resurveyed 144 150 104% Number of group discussions organized on gender equity and equality with participation of the safety net beneficiaries 70% 72% 103% Percentage of VSLA groups integrating nutrition in their regular activities Final Performance Evaluation of the Kore Lavi DFAP in Haiti Annex 7: Project Performance Summary Tables 131 LOA Target LOA Actual % of Target Achieved Indicator 3 3 100% Number of projects implemented in the country benefiting from the MAST-led vulnerability targeting systems 5 5 100% Number of MAST/CAS staff trained on information systems, information analysis and communication 2 2 100% Number of other caseloads of beneficiaries included in SIMAST by Kore Lavi and other programs during the last 12 months 1 1 100% Number of applications integrated and modifications made to SIMAST to improve linkages with other systems 80% 80% 100% Percentage of beneficiaries reporting improvement of the safety net operations in the past 12 months 4 4 100% Number of Department in which food voucher based safety net is implemented by MAST 83% 83% 100% Percentage of food accepted for voucher redemption that is locally grown 4 4 100% Number of events promoting social safety net programs convening key institutional and external partners of MAST 100% 99% 99% Percentage of community members complaints on the safety net programming responded to under MAST management 19,158 18,158 95% FFP 51a: Number of households benefiting directly from USG assistance under Food for Peace 44,063 41,763 95% FFP 32: Number of people benefiting from USG-supported social assistance programming 150 136 91% Number of Safety net beneficiaries who start income generating activities through VSLA 800 660 83% Number of safety net beneficiary households with P&L women and CU2 who receive MCHN local food￾based vouchers 63% 51% 81% Percentage of valid community (CADEPs, local leaders and authorities) feedbacks on the safety net programming that are implemented 5 4 80% Number of updates of the Safety net beneficiary data completed in the Safety net targeted communes 5 4 80% Number of mechanisms of operationalization of the safety net by MAST, formalized (developed and/or agreed-upon) through consultative frameworks/meetings with engagement of the program and central/departmental-level MAST staff 45% 34% 76% Percentage of participating community structures' members who are women 4 2.98 75% MAST Global Institutional Capacity Index progression (related to social safety net programming) 90% 63% 70% Percentage of completed activities of the specific central and departmental-level institutionalization/transfer plans 82 56 68% Number of quarterly community feedback reports developed by Community leaders (CADEPs and other CBO networks) and submitted to MAST for follow-up IMPEL | Implementer-Led Evaluation and Learning 132 Annex 7: Project Performance Summary Tables LOA Target LOA Actual % of Target Achieved Indicator 150 97 65% Number of community-based structures participating in monitoring of frontline safety net activities 90% 58% 64% Percentage of autonomous field visits performed by MAST staff at Departmental and Communal levels 5 3.14 63% Community Structures Capacity Index progression 24 15 63% Number of broadcasts of special messages (related to markets, nutrition, food availability, standards of safety net services, disaster preparedness etc.) through VSLA network 0.8 0.5 63% MAST (central and departmental-level) Institutional Capacity Index progression 88% 54% 61% Percentage of community structures (CADEPs and other similar structures) engaged in community social audit (SA) and leadership processes in collaboration with MAST and local authorities 50% 26% 52% Percentage of Safety net beneficiaries (VSLA Members) who start income generating activities 50% 25% 50% Percentage of Information System (IS) staff recruited by MAST (MAST Planning unit) or transfer under MAST contract 90% 36% 40% Percentage of MAST department and central-level organizational units that are able to produce quarterly activity reports on 1.30 0.36 28% Progression of the Vulnerability Targeting MAST staff capacity, as measured on the institutionalization scale 11 0 0% Number of commune for which food voucher distribution beneficiary lists are integrated in SIMAST 2 0 0% Number of internes from Haitian IT institutions providing technical support to SIMAST day-to-day operations 1,000 2 0% Number of safety net beneficiary households graduated from safety net program 90% 0% 0% Percentage of safety net beneficiary vendors who use new technologies for the food voucher operations 30% 0% 0% Percentage of Commune-level Agricultural offices that conduct the regular technical supervision/support visits to the local food producers 64 0 0% Number of quarterly updated joint work plans between Central and Departmental MAST unit Coordinators IMPEL | Implementer-Led Evaluation and Learning Annex 8: Qualitative Coding 133 ANNEX 8: QUALITATIVE CODING Evaluation Questions* Themes Topic-Codes** EQ1 (general1): What has been Kore LAVI’s contribution to reducing food insecurity and vulnerability in targeted communities?* Malnutriton Reduire la malnutrition particulièrement chez les enfants Food access and quality Améliorer la qualité nutritionelle des produits accessible aux plus vulnérables Food access and quality Améliorer diversité et l'équilibre dans l’alimentation Food access and quality Facilitant l’accès aux produits locaux Food security Reduite insécurité alimentaire Food access and quality Formation sur la préparation des repas équilibrés et nutritionels Impact The project didn't have/couldn’t prove the sustainability of its impact on malnutrition, food security, vulnerability or resilience EQ1 (general1): What has been Kore LAVI’s contribution to reducing food insecurity and vulnerability in targeted communities?* Malnutriton Reduire la malnutrition particulièrement chez les enfants Food access and quality Améliorer la qualité nutritionelle des produits accessible aux plus vulnérables Food access and quality Améliorer diversité et l'équilibre dans l’alimentation Food access and quality Facilitant l’accès aux produits locaux Food security Reduite insécurité alimentaire Food access and quality Formation sur la préparation des repas équilibrés et nutritionels Impact The project didn't have/couldn’t prove the sustainability of its impact on malnutrition, food security, vulnerability or resilience Food access and quality Donner la nourriture aux femmes enceintes, aux mères et aux enfants Social conditions Améliorer les conditions sociales Economic conditions Améliorer les conditions économiques Sanitary conditions Formation sur l'hygiene Training Donner les conseils sur l’alimentation aux femmes enceintes, aux mères et aux enfants Health conditions Recommander les enfants malnourite et femmes enceintes aux centres de santé Sanitary conditions Améliorer les conditions sanitaires Health conditions Améliorer les conditions de santé Food access and quality Facilitant l’accès aux produits de qualité Training Formation sur l’importance de l’allaitement maternel Food access and quality People eat fresh foods more Food access and quality Created dependence on food assistance Economic conditions Coupons led to more stable market (no negative impacts on markets, market prices) Health conditions Diminuer le taux de mortalité des femmes enceintes IMPEL | Implementer-Led Evaluation and Learning 134 Annex 8: Qualitative Coding Evaluation Questions* Themes Topic-Codes** Health conditions Beaucoup plus d’enfants ont été vaccinés et ont été bien nourris grâce aux paniers alimentaires Food security Thanks to the project beneficiaries no longer practice negative coping strategies like borrowing food from neighbors. EQ1 (general2): How has Kore LAVI’s contributed to establishing a replicable SAFETY NET and EXPAND GOVT CAPACITY to prevent child undernutrition? Capacity building Training and capacity building support for anchoring MAST capacity increased ability to govern social safety net programs. GOH capacity The project helped to improve MAST visibility and linkages between MAST staff and communities. GOH capacity MAST is understaffed, overworked, with high levels of turn over, and central level capacity is weak Social protection policy The project had a fundamental influence national level policy on SP and to help to create a policy framework for SSN. Capacity building The project’s capacity building diagnostic tools were precise and useful. GOH capacity Grâce a ce projet les autorités locales ont eu moins de pression par rapport aux responsabilités qu’elles ont vis-à-vis des plus pauvres GOH capacity Le CASEC ont développé un excellent partenariat pour planifier et superviser toutes les activités GOH capacity The project made lots of recommendations for institutionalization, but didn’t keep promises or meet expectations. GOH capacity Un meilleur rapport entre le niveau central et le niveau départemental du MAST GOH capacity The biggest help that MAST offered was office equipment and fuel for office vehicles GOH capacity The departmental office works much better now thanks to the project. GOH capacity GOH employees working for the project should have gotten salary support. GOH capacity The SIMAST system was successfully integrated into MAST, but they do not have the technical capacity to manage it. GOH capacity There was no common definition of what institutionalization meant. GOH capacity Le programme a pu jouer le rôle que l’Etat devrait jouer dans la vie des gens Food access and quality Mis en place des filières de distribution de nourritures locales et équilibrées GOH capacity N'a pas bien traité les employés de MAST; ne donnait pas le per diem, ni une voiture appropriée pour le déplacement Communications Prior to the project, there was an institutional and operational fragmentation in implementation of SP activities and little coherence between donor and GOH activities. GOH capacity MAST’s capacity for strategic planning exceeds its capacity for operational management. GOH capacity Capacity building support for MAST will be lost because there will no longer be any project-funded staff in the MAST office. Final Performance Evaluation of the Kore Lavi DFAP in Haiti Annex 8: Qualitative Coding 135 Evaluation Questions* Themes Topic-Codes** Social safety net The project tried to develop an advocacy component, but there was confusion about the objectives. GOH capacity MAST should have coordinated with and involved other ministries in the project, but didn’t. Women Working with the Ministry of Women made for better project results and more sustainable impact. Communications MAST now implicated in the IPC, particularly the chronic IPC Project design MAST wasn't involved at the strategic level for the first part of Kore Lavi, GOH capacity Referred to SIMAST as SIPAM (WFP). MAST not sufficiently involved in the technical system GOH capacity MAST feels a positive ownership of SIMAST Exit strategy There was a push to handover to MAST even when MAST said they were not ready Social protection policy Should have started the social protection policy work at the BEGINNING Social protection policy Lack of political will for the GoH to use the model of social protection in the fture. EQ1.1 How do project PERFORMANCE INDICATORS measure against targets set by the partners and FFP indicators? Training Capacity building is hard to measure using standard FFP indicators. It's hard to measure the outcome of trainings. Project design The consortium brought a food delivery/food security mentality and approach to a capacity building/governance project. GOH capacity At first, people thought the SIMAST data would be regularly updated, but not always done, so the data in SIMAST are sometimes really old. EQ1.2: What FACTORS PROMOTED OR INHIBITED achievement of the project objectives including EFFECTIVENESS? Exit strategy The handover strategy to the GOH was overly ambitious and needed more time to ensure sustainability and ownership of activities. Project design La participation des CASEC et CADEC dans la planification et la mise en œuvre du projet a favorisé l’atteinte des objectifs External factors Even as food security increased, macro conditions such as inflation eroded positive gains in reducing vulnerability. Project design Kore Lavi focused on survival, not protecting/promoting livelihoods. The level of support needed to protect livelihoods is much more than what the coupons provided. Project coverage Le projet avait des limites budgétaires car il ne pouvait pas toucher tout le monde External factors L’insécurité alimentaire et la malnutrition battaient son plein dans la communauté Project management Project was important, interesting and innovative. Project management Il y a beaucoup de changment et le programme s’est beaucoup amélioré depuis son lancement IMPEL | Implementer-Led Evaluation and Learning 136 Annex 8: Qualitative Coding Evaluation Questions* Themes Topic-Codes** Project design The project stopped just short of its goal. It should be redesigned using what we know to make it better. Project design No way to enroll newly poor Project design Involving the GOH from the beginning in design and implementation of project activities was a strength. Capacity building Institutional reinforcement did not work well in Kore Lavi Project management The project practiced adaptive management, but not strategically. Project design The monthly technical working group meetings led by the GOH were a strength. Project management Focus on deadlines, targets, logframes, and checking boxes detracted from trying to work more with MAST Consortium management The lead of the consortium wasn't always at liberty to say what they really thought. Project management Au cours de route que des changements ont été opérés pour ajouter d’autres personnes vulnérables Project design World Vision et CARE ont impliquer les autorités locales dans la planification du projet Project management Parfois les coupons restent pendant trop de temps entre les mains des bénéficiaire Infrastructure La région a un problème d’irrigation terrible. External factors L’insécurité et les troubles politiques ont constitué les plus grandes contraintes dans l’entreprise External factors Les catastrophes naturelles représentent un défi majeur pour le projet Project design La communauté locale se sentait écartée du pilotage du projet. Targeting It was the first time for VSLAs in Haiti and the HDVI offered the first verifiable means test for SP. Project design The project’s approach worked, and should be used again. Gender/women Collaboration with the Ministry of Women was a strength. Project design Combining objectives of helping the GOH to establish a SSN and increasing access to local food for poor households was a strategic weakness. Targeting SO1 (targeting) and SO2 (SSN) worked together in perfect harmony. Targeting GOH insisted that Gonave be included because it is so poor. Project management The project work on new things all the time, but didn’t capitalize on their innovations and combine them with learning. Project design Kore Lavi didn't forsee the implications of needing to work with the local authorities Project management Health workers viewed as extra, unpaid work. Made collaboration with Centre de Sante difficult. EQ 1.3: What is the plausibility of PATHWAYS and the VSLA Projet Kore Lavi a atteint ses objectifs car grâce à TIPA TIPA il a amélioré la situation économique de nombreuses femmes Final Performance Evaluation of the Kore Lavi DFAP in Haiti Annex 8: Qualitative Coding 137 Evaluation Questions* Themes Topic-Codes** DETERMINANTS of achieving the key outcomes? Social safety net The project safety net was able to respond to co-variant shocks, such as droughts or hurricanes, but not to household level indiosyncratic Social safety net The project’s goal of a complete transfer of social safety programming to the GOH in 4 years was ambitious and overly optimistic. VSLA VSLAs helped people avoid the use of credit with high interest coming from other sources. Livelihoods Communities didn’t just need access to food, they needed to know how to increase local production. Project design One of the weaknesses of SO3 was its theory of change that beneficiaries would escape insecurity Consortium management The consortium brought a food delivery/food security mentality and approach to a capacity building/governance project. Project design There was tension between short-term performance outcomes and long-term goal of building MAST’s institutional capacity. Project design Sharing culture not taken into account Impact Il n’y aucune amélioration des conditions de vie chez les personnes qui recevaient les coupons Livelihoods The problem with graduating beneficiaries from the project is that there are no other opportunities for them to move on to. Food access and quality Beneficiary interviews indicated they would have bought cheaper, imported, unvaried food if given cash rather than coupons. Project design Culture of sharing was taken into account in initial design, but then taken out Project design The idea of graduation was not an initial expectation of the designers. EQ1.4 Did TARGETING strategies identified the poorest households and reach the most vulnerable and including women? Targeting Overall, targeting indicators and SIMAST system did a good job of identifying vulnerable populations Project coverage Touché certains les ménages les plus pauvres mais pas tous. La demande etait plus importante que l'offre. Targeting Le système de ciblage a été biaisé, les plus malins l’ont contourné et il y avait du favoritisme Targeting Une faille dans le recensement a empêché aux personnes les plus vulnerables de la zone d’intégrer le projet Targeting Les femmes qui en ont le plus bénéficié (en particulier SO2 et SO3) Targeting People interviewed didn't give pertinent personal information, they sometimes were ashamed to show how poor they were. Targeting Les vendeurs ont le plus bénéficié du programme. IMPEL | Implementer-Led Evaluation and Learning 138 Annex 8: Qualitative Coding Evaluation Questions* Themes Topic-Codes** Targeting Les enfants sont les plus grands bénéficiaires de ce programme Targeting The bureau de recours was important for people to register themselves if they felt they were excluded. Targeting Travailler avec des agents sur le terrain et les CASECs dans le choix des bénéficiaires pour identifier les véritables vulnerables Project design No way to add the newly poor during the project. Gender/women Les objectifs de genre n’étaient pas intégrés dans le programme au départ; c’est plutôt une adaptation par rapport des besoins Targeting MFIs benefited a LOT Targeting Les hommes ont le plus bénéficié du projet car le projet Targeting At start up there were many problems with the first list of beneficiaries Targeting SIMAST system is flexible and modular and can be adapted for use by other ministries. Targeting The SIMAST targeting system created accountability Gender/women Encouragement of women did not mean excluding men (ex. peres leader) Targeting Targeting the poorest 10% of HHS in each commune when communes have very different levels of poverty may have been an easy way out for Kore Lavi targeting. Targeting The HDVI wasn't the best targeting tool. Targeting Little difference between poor, so some problems with solliciting help from beneficiaries, Targeting Mères leaders en sont les plus grands bénéficiaires Gender/women The role of mere leaders increased women’s responsibility and leadership. Gender/women Gender outreach was problematic with SO2 because worked with head of HH, who were often men. Gender/women One of the challenges to promoting a gender approach is resistance by women to discuss sensitive or taboo topics. Gender/women Gender training focused on changing the role of women in the HH Gender/women MCFDF worked with project in all departments Targeting Digicel was the biggest beneficiary of the project Targeting Targeting 10% of the most vulnerable in a particular commune may be richer or poorer than the bottom10% in a different commune. Targeting The selection of food vendors was problematic. Selection criteria were too stringent EQ1.5: Were interventions for the poorest individuals and HH’s APPROPRIATE AND EFFECTIVE? Quality Tous les ménages vulnérables bénéficiaires sont satisfaits des services fournis et de plus ces derniers ont été appropriés VSLA Linking coupons and VSLA was successful. Vouchers Ill serait préférable que l’on donne du cash aux bénéficiaires Final Performance Evaluation of the Kore Lavi DFAP in Haiti Annex 8: Qualitative Coding 139 Evaluation Questions* Themes Topic-Codes** Vouchers Coupons were more appropriate than cash. Livelihoods Communities didn’t just need access to food, they needed to know how to increase local production. Vouchers Les coupons n’ont pas permis de réduire la vulnérabilité des gens car la nourriture qu’on donne ne peut satisfaire qu’une semaine Vouchers Ils devaient se rendre très loin pour s’approvisionner avec les coupons Quality World Vision a aider pour mettre des vendeurs a proximité des bénéficiaires Project design Le projet n’a pas eu des interventions en conformité avec les besoins des zones Project design The First 1,000 days strategy of SO3 was appropriate and effective. EQ2.1 What was the QUALITY OF THE PROCESSES, systems, and institutional arrangements? Quality La qualité des services d'une qualite satisfaisante/acceptable. Quality La qualité des services étaitde bonne/excellente qualite Quality Aucune plainte contre le projet Quality Les retards enregistrés dans le paiement des vendeurs et des IMF Quality Le comportement du staff technique du projet et les vendeurs etait tres ouvert; ils ont traité les bénéficiaires avec respect et dignité Quality Les coupons sont distribués à temps et très appréciés Quality En cas de plainte, le point focal du projet arrive à donner satisfaction en appelant le vendeur ou le bénéficiaire pour lui expliquer. Quality Beneficiaries were informed in advance that the project was ending and offered a final “resilience package” worth 3 month’s distribution. Quality Les vendeurs ont travaillé en étroite collaboration avec le staff technique du partenaire d’implémentation Project design MAST was given the project after it was developed, not implicated in the process Quality Il remarquait une certaine amélioration au fil du temps dans les doleances des beneficiaries Quality Le traitement donné à elles était mauvais. Elles étaient traitées en parent pauvre Quality Les services fournis était de piètre qualité Project coverage The main complaint was that there were more vulnerable people than expected and not all of them could be beneficiaries. Quality Ils ignoraient le système de plaintes du projet Quality Les mécanismes de plaintes n’étaient pas efficaces car les gens ne pouvaient pas se déplacer pour aller faire les plaintes Quality Il y a eu de la régularité dans la distribution des coupons. Quality Les retards enregistrés dans la distribution des coupons IMPEL | Implementer-Led Evaluation and Learning 140 Annex 8: Qualitative Coding Evaluation Questions* Themes Topic-Codes** Quality Au commencement, les bénéficiaires se plaignaient de la qualité des produits auprès des vendeurs ; mais cela s’est résolu au fil du temps Quality Il y a une manque de motivation des Agents Villageois (AV) pour l’encadrement des groupes TIPA EQ2.2: What were COMMUNITY PERCEPTIONS on the quality, frequency, effectiveness, and sustainability of the services provided by the project? VSLA Le TIPA etait un success Project management Tout a bien fonctionné dans le projet; elle a apprécié aussi la supervision par les membres du projet Vouchers Les bénéficiaires ont beaucoup apprécié le système des coupons Vouchers Les coupons n’ont pas vraiment un impact durable sur la communauté Exit strategy Beneficiaries complained that the end of the project was like the death of their mother and father Project management Il y a eu de petites mésententes entre les vendeurs et bénéficiaires au départ ; on a fait des formations et le problème a été résolu Training La formation les intéressait beaucoup également, surtout celle des mères leaders. Training Le projet a formé des leaders pour lancer les associations villageoises d’épargne et de crédit Project management Support for the project was demonstrated by how easy it was to organize focus groups. People, were willing to give up their time on a Sunday. Project management Projet KORE LAVI dans le Nord-Ouest accorde plus d’importance aux bénéficiaires de coupons qu’au vendeur, EQ2.3: What is the likelihood that service providers will CONTINUE PROVIDING SERVICES after the project ends? Sustainability Les vendeurs pourront continuer à fournir leur service à la population Sustainability Les CADEC vont continuer au delà de la fin de ce projet, car elles sont impliquée à fond dans la communauté Sustainability MAST will continue to use HDVI and SIMAST, but will need funding for updates and maintenance. Sustainability Pères et mères leaders formés continueront a accompanier les beneficiaires Sustainability Les fournisseurs de services ne pourront plus continuer Sustainability La durabilité des services est incertaine Sustainability Implementing partner will continue to implement food distribution, nutrition and hygiene training and support VSLAs Sustainability La production locale des vivres frais continuera Sustainability Le personnel médical continueront a accompanier les beneficiaires Sustainability Les CADEC ne pourront pas continuer avec leurs activités d’appui et d’accompagnement Sustainability Magasin communautaire Final Performance Evaluation of the Kore Lavi DFAP in Haiti Annex 8: Qualitative Coding 141 Evaluation Questions* Themes Topic-Codes** EQ2.4: Are communities and beneficiaries motivated to DEMAND AND PAY (or invest time) services? VSLA Les VSLA constituent l’unique aspect de durabilité du projet Training Le changement du comportement apres la formation sur la nutrition, l’hygiène, l’allaitement, l’équité de genres sera durable Food access/quality Les bénéficiaires ne pourront pas consommer des produits locaux par faute de moyens économiques/les clients des vendeurs vont chute VSLA La capacité financière des VSLA/TIPA diminuera considérablement après la clôture du projet. They will continue to need support Training Les formations données aux mères leaders seront les plus durables Livelihoods L’élevage de caprins, volailles et de porcins Sustainability Les ménages bénéficiaires sont toujours motivés pour demander et payer des services, mais ce dernier ne sera pas facile Food access/quality La consommation des produits locaux Quality Manque de motivation des mères et pères leaders, et des Agents Villagois car ils ne perçoivent aucun salaire Sustainability Les problèmes de vulnérabilité des personnes resurgissent car elles ne peuvent pas vraiment subvenir à leurs besoins Gender/women Le changement dans la participation des hommes dans le foyer apres la formation sur le genre EQ2.5: Do necessary RESOURCES AND CAPACITY BUILDING exist to sustain service providers? Sustainability Local computer technicians can maintain the SIMAST system, if they have resources. Sustainability SIMAST will require more money and more staff if it is to scale to a national level system. Sustainability In order to ensure the sustainability of SIMAST, donors and NGOs should be asked to pay for reporting from the database. Sustainability Les CADEC disposent des ressources nécessaires à la fin du projet pour continuer à fournir des services à court terme et à moyen terme VSLA Aux formations s’ajoute le besoin de financement des TIPA Training La formation sur la gestion d’entreprise leur sera d’une grande utilité pour continuer après le projet. VSLA Les groupes VSLA représentent l’unique recours pour les vendeurs de produits pour financer leurs activités Markets Il n’existe pas de ressources disponibles pouvant soutenir les vendeurs Markets The project also supported food vendors by building a platform for cooperation rather than competition. IMPEL | Implementer-Led Evaluation and Learning 142 Annex 8: Qualitative Coding Evaluation Questions* Themes Topic-Codes** EQ2.6: To what extent did the project LEVERAGE other USG and non-USG investments to achieve sustained outcomes? Natural disasters Lors des catastrophes, le projet fournissait des outils pour des activités d’assainissement, les semences et augmentait l’argent des coupons Sustainability Other donors will use and support SIMAST (WFP, EU, Cooperation Suisse, CRS, Mercy Corps) Social protection policy The national policy on social protection was well done and will be a sustainable change. VSLA Les groupes de VSLA / TIPA sont soutenu avec les fonds de ses membres et des institutions publique et prive Markets There was a multiplier effect by injecting money into the local economy. The market for local produce grew, as did the sales of food vendors. Sustainability GOH has taken over SIMAST and will ensure its sustainability, sharing it with other ministries Mere Leaders Mere leaders are acknowledged by the community and will be used by other NGOs. Natural disasters Le projet n’aidant pas vraiment lors des catastrophes naturelles Sustainability The project strengthened the Sector Roundtable on Social Protection and improved its sustainability. Leverage Leverage funds from other projects and source to reach vulnerable population not included in the project. Leverage Project funds allowed organization to expand coverage of nutrition and hygiene training to new communities. Leverage With project resources created 35 VSLAs. Beneficiaries created 11 VSLAs on their own without project support. Sustainability The GOH has taken over the project’s school feeding pilot. EQ2.7: What is the evidence of enhanced LINKAGES with other service providers? Sustainability CADEP va garder les nouveaux partenariats développés grâce à KORE LAVI, surtout avec les instances étatiques telles le MAST Sustainability Le vendeur gardera le contact avec les bénéficiaires de coupons et les fournisseurs. Sustainability Technical working groups created by the project will continue to allow for knowledge sharing between organizations involved in SP Sustainability CNSA will maintain linkages with community agents to gather SP data. Sustainability Created partnerships- MSPP, MoAg, CNSA, Marie, Maires, CASEC, etc. Learned to work together EQ3.1: What factors in the IMPLEMENTATION AND CONTEXT were associated with greater or lesser EFFECTIVENESS in producing Project design L’implication des acteurs clés (CADEC, élus locaux, vendeurs, etc.) a contribué à la réussite du projet Consortium management Tensions in the consortium resulting from different visions and approaches created problems w decision-making and communications Final Performance Evaluation of the Kore Lavi DFAP in Haiti Annex 8: Qualitative Coding 143 Evaluation Questions* Themes Topic-Codes** outputs of higher or lower quality? Project coverage On aurait pu toucher plus Project management Il y avait des problemes au commencement, mais le projet s’est amélioré au fil de son exécution Markets Les vendeurs ont souvent tendance à augmenter le prix des produits Livelihoods Lacking a livelihoods component Exit strategy The project did not have entrance/exit mechanisms where beneficiaries could join or leave the project. Impact The coupon package was too small/too limited Consortium management Working with a consortium was very positive. Project management There were problems with paying vendors and MFI partners after end of SO3 Vouchers The system for electronic coupons was very efficient. Vouchers Le comportement des vendeurs envers les bénéficiaires de coupons a favorisé la réalisation des objectifs du projet Communications La communication n’était pas du tout efficace car les messages étaient toujours en retard. Project management Over time, the project moved from a deliverables focused approach towards a more learning based adaptive management approach. Consortium management Each consortium member was chosen for their complementary expertise in specific areas Project management There was very little diversion of food because of training, monitoring and the threat of being kicked out of the project. Project design The combination of e coupons for dried food and paper vouchers for fresh food was effective Targeting Project start up it was difficult because didn't have a usable database of vulnerable population. Exit strategy The exit at the end of SO3 was abrupt. Could have been done better if the close down ran through the extension period. Markets A monthly market survey was implemented in order to ensure vendors were respecting project rules. External factors The current difficult situation (inflation, insecurity, etc.) may have eroded any gains from Kore Lavi. Project design Adaptive management was not a policy, but operationally project responsive to opportunities that arose during implementation. Capacity building Renforcement de capacités du CADEC en termes de formation, matériels Project management A chaque période, le projet adapte les prix des produits en fonction de l’inflation. Project management Not enough staff for monitoring and evaluation. Didn’t get out to field often enough. IMPEL | Implementer-Led Evaluation and Learning 144 Annex 8: Qualitative Coding Evaluation Questions* Themes Topic-Codes** Consortium management Consortium members worked together with the same goal. Project management The cost of operations for coupons was very high. Project management When SO3 activities were closed down the project closed all of the offices in the NW and Artibonite departments. Project management There were frequent problems with distributing food during the rainy season. Project management High turnover among gender promotion staff was a problem. Project design SO3 should have had a family planning component Project design There problems with operationalizing and managing SO2 from the beginning. EQ3.2: What interventions and implementation processes deemed more/less ACCEPTABLE to members of the target communities? Vouchers La distribution de coupons VSLA TIPA a permis la mise en commun de l’épargne, l’autonomisation des femmes, l’accès au crédit et la création des activités génératrices de revenu Meres leaders Le systeme des pères et mères leaders Capacity building Renforcement de capacités des vendeurs, des pères et mères leaders. Food access and quality La promotion de la consommation des produits locaux Training Les activités de formation (nutrition et hygene, et activites generatrices de revenus) vont de pair avec les objectifs du projet Livelihoods La formation reçue par les mères allaitantes et les activités génératrices de revenus Project management Les meres leaders disent n’avoir disposé d’aucun matériel de travail ni d’aucun frais Project management Beneficiaries appreciated final “resilience package” worth 3 month’s distribution. Livelihoods Financier via ses activités génératrices de revenu Project design La cantine scolaire Quality La qualité des produits livrés Vouchers Il n’a pas aimé les sites de distribution de coupons Meres leaders Les meres leaders sentaient traitées en parents pauvre Project design Kore Lavi is the best model for social protection in Haiti. EQ4.1 What were the unintended positive and/or negative CONSEQUENCES of the projects? Impact Because of the vouchers, beneficiaries were able better plan for expenses and to save money to start a business, buy animals, fix up their houses and pay school fees. Markets Increased agricultural and animal production and reinvigorated local markets VSLA Grace au au renforcement de la capacité financière des « TIPA TIPA », elles ont appris à économiser pour le commerce, l’entretien des jardins, l’achat de bétail, la gestion des urgences familiales. VSLA Accès au crédit à un taux dérisoire Final Performance Evaluation of the Kore Lavi DFAP in Haiti Annex 8: Qualitative Coding 145 Evaluation Questions* Themes Topic-Codes** Gender/women Beaucoup d’avancées dans la promotion des droits de la femme (moins de violences conjugales, plus de responsabilité dans les foyers) Livelihoods Création des activités génératrices de revenu Vouchers Les coupons distribués aux bénéficiaires ont contribué au renforcement de la capacité financière des groupes de VSLA Strengths and weaknesses Mettre les bénéficiaires dépendant sur la nourriture gratiute et creer la paresse chez les bénéficiaires. VSLA TIPA TIPA favorise l’accès au crédit aux femmes, facilite leur autonomisation Strengths and weaknesses Le programme aide les gens à vivre en commun; les conflits des les foyers et communaute ont diminue Vouchers Avec les hommes qui ont plusieurs femmes, les rations alimentaire sont détournés a d’autres fins et séparée dans plusieurs foyers. Institutionalization Institutionalization of SP in the GOH was a weakness. Social Safety Net Accès à certains services sociaux est plus ou moins amélioré Strengths and weaknesses L’augmentation du taux de natalité Vendors Les vendeurs ont pratiqué du marché noir au mépris des droits des bénéficiaires. Vendors Le vendeur augmenter sa clientèle, de varier et changer les produits offerts, de répondre aux demandes des bénéficiaires de coupons. Sanitary conditions L’hygiène communautaire s’est installe. Il y a plus de latrines; lors de l’épidémie de Cholera, il a aidé à isoler les malades et à prévenir d’autres cas. Markets L’augmentation de la demande pour la nouriture locale, surtout le riz. VSLA The weakness with VSLAs was that the project didn’t create enough and didn’t create a federation of VSLA. Project management Performance related data were captured and analyzed, but not assimilated to create learning. Children Because kids eat better before they go to school, they are able to perform better in school. Strengths and weaknesses The project increased partner organizations' capacity and visibility in the community Exit strategy The end of the project will push vendors into vulnerability. Exit strategy Some people won't be able to participate in VSLAs when the coupons finish Exit strategy Le projet « KORE LAVI » tombait à point au moment de sa réalisation Markets Had invigorating effect on market overall, no negative impact on market prices seen IMPEL | Implementer-Led Evaluation and Learning 146 Annex 8: Qualitative Coding Evaluation Questions* Themes Topic-Codes** GOH capacity The end of the project highlights the weakness of the gov't, leads peole to question the gov't. Gave people an idea of what they could get if they had good government… Vouchers Le montant alloué aux paniers alimentaires était devenu insignifiant à cause de la cherté de la vie Malnutrition La consommation des produits alimentaires a améliorer la santé des mères et à la prévention de la malnutrition chez les enfants. Sustainability Les coupons arrivent à compenser momentanément la vulnérabilité des bénéficiaires par rapport aux catastrophes naturelles Gender/women Il y a moins de femmes mourant pendant l’accouchement Gender/women Meilleures planifications familiales, le contrôle des grossesses par les femmes. Gender/women Kore Lavi did NOT increase pregnancy/birth rate, this is a stigmitization Quality Il n’y a eu aucune conséquence négative imprévue Vouchers There was a sense of pride among beneficiaries that they are able to share food with others, rather than have to borrow food. Quality The school feeding pilot was a success, but the project didn’t have enough time to scale up its implementation. GOH capacity Not getting GOH ministries to work together was a big missed opportunity. Vouchers Coupons created more jobs (help at boutiques, for example) Malnutrition La consommation des produits locaux a améliore la santé des bénéficiaires et contribue à la prévention de la malnutrition chez les enfants. VSLA La plupart d’entre elles ont laissé FONKOZE pour s’adonner uniquement aux groupes AVEC. Natural disasters Les inondations et les grèves a répétition ont été les principaux obstacles pour le projet selon eux. Project design 4 years plus a 2 year extension is not the same as having 6 years from the start. EQ4.2 What are the ways to MINIMIZE NEGATIVE CONSEQUENCES and systematically capture positive consequences? Agriculture Un appui a l’agriculture et l’élevage aurait un impact sur la durabilité et faciliterait l’accès aux produits locaux Livelihoods Créer des emplois « cash for work », surtout pour les femmes VSLA Le renforcement des TIPA TIPA (administration, matériels et un espace de rencontre) et formation des membres Livelihoods Add in livelihood component and credit fund to the project Targeting Focus on women-headed households, elderly, handicapped, single parents, malnourished and city dwellers Project coverage L’augmentation du nombre de bénéficiaires et de departments touchees Final Performance Evaluation of the Kore Lavi DFAP in Haiti Annex 8: Qualitative Coding 147 Evaluation Questions* Themes Topic-Codes** Project design Contacter les leaders communautaires pour faire l’analyse des besoins avant l’exécution des projets Project design The GOH should a larger role in the design and management of future projects. Training Formations sur l’égalité et de l’équité de genre et l’autonomisation de la femme Infrastructure Activités pour proteger l'environnement tel que le conservation de sol Vouchers L’augmentation du montant des coupons Social Safety Nets No graduation built in to the system Consortium management Consortium should be managed more transparently, and flexibly. Roles should be more clearly defined. Vouchers Reprendre la distribution des coupons Children Si le projet pouvait également financer la scolarité des enfants Communications Improve precision consistency and correctness of communications and data sharing, particularly between consortium members. Training Formation des leaders communautaires Training Formations données aux mères leaders Infrastructure Add in component of small community basic infrastructure creation (water, environmental protection, etc.) Targeting Le système de plaintes a été efficace car elles ont eu des suivis VSLA VSLAs should be legally recognized, federated, and linked with community banks Gender/women The project should have done more to promote role of women. Training L’encadrement des vendeurs Training Formations sur tous les aspects de commerce Infrastructure Supporter le logement pour les gens Gender/women Use a community-based approach to addressing gender issues that include men as well as women. Training L’importance des formations dans un tel projet comme catalyseur de changement de comportement. Mere leaders Les meres leaders devaient disposaient de moyens pour se déplacer, matériels, frais de déplacements, des produits de traitement de l’eau. Food access/quality La consomation des produits locaux contribuent à résoudre les problèmes nutritionnels Project design MAST should have a greater leadership role and greater role in distribution of coupons GOH capacity Strengthening the capacity of MAST should continue, particularly training and material support (fuel, vehicles and equipment). Consortium management Consortium approach of institutional and operational collaboration is worth repeating. Gender/women Women included should have conditions such as child vaccinations, maternal care, etc. IMPEL | Implementer-Led Evaluation and Learning 148 Annex 8: Qualitative Coding Evaluation Questions* Themes Topic-Codes** Social Safety Nets Le mieux aurait été de considérer d’autres personnes en cours de route. Training Sensibiliser les gens sur les bonnes habitudes alimentaires Project design Il faut écarter dans l’avenir les vendeurs, ils sont des obstacles à la réalisation des objectifs du projet Vouchers Need to rethink the coupon platform to reduce the cost of operations. Consortium management It takes time to develop synergies among different consortium members. Consortium management Technical working groups should continue and be strengthened. Project design The private sector should be involved in future activities. Project design The project stopped just short of its goal. It should be redesigned using what we know to make it better. Project design The project should practice adaptive management and create forum for making adjustments to implementation. *Under the EQ1 questions, the overall EQ1 question has been formulated into two additional questions for coding, in addition to the EQ1.1 to 1.5. These are labelled as 'general'. ** The topic codes were created when analyzing both French and English notes (by bilingual analysts). The codes were kept here in the language first used in for that topic￾code, but all are translated to English in the body of the report.