Mid-Term Evaluation of the South Kivu Food Security Project (FSP)-Enyanya Development Food Security Activity in the Democratic Republic of the Congo (DRC) February 2020 IMPEL | Implementer-Led Evaluation & Learning Associate Award ABOUT IMPEL The Implementer-Led Evaluation & Learning Associate Award works to improve the design and implementation of Food for Peace (FFP) funded development food security activities (DFSAs) through implementer-led evaluations and knowledge sharing. Funded by the USAID Office of Food for Peace (FFP), the Implementer-Led Evaluation & Learning Associate Award will gather information and knowledge in order to measure performance of DFSAs, strengthen accountability, and improve guidance and policy. This information will help the food security community of practice and USAID to design projects and modify existing projects in ways that bolster performance, efficiency and effectiveness. The Implementer-Led Evaluation & Learning Associate Award is a two-year activity (2019-2021) implemented by Save the Children (lead), TANGO International, and Tulane University, in Haiti, the Democratic Republic of Congo, Madagascar, Malawi, Nepal, and Zimbabwe. RECOMMENDED CITATION IMPEL. (2020). Mid-Term Evaluation of the South Kivu Food Security Project (FSP)-Enyanya Development Food Security Activity in the Democratic Republic of the Congo (DRC). Washington, DC: The Implementer-Led Evaluation & Learning Associate Award PHOTO CREDITS Photo Credit: Annette Fay. Women using FSP-Enyanya constructed water point and laundry stations DISCLAIMER This report is made possible by the generous support of the American people through the United States Agency for International Development (USAID). The contents are the responsibility of the Implementer-Led Evaluation & Learning (IMPEL) award and do not necessarily reflect the views of USAID or the United States Government. CONTACT INFORMATION IDEAL Activity c/o Save the Children 899 North Capitol Street NE, Suite #900 Washington, DC 20002 www.fsnnetwork.org info@fsnnetwork.org PREPARED BY: Mid-Term Evaluation of the FSP-Enyanya DFSA in the DRC Table of Contents i TABLE OF CONTENTS List of Tables and Figures ..................................................................................................................iii Acronyms .........................................................................................................................................iv Executive Summary...........................................................................................................................vi Background............................................................................................................................................ vi Methodology ......................................................................................................................................... vi Findings................................................................................................................................................. vii Timeliness of Implementation and M&E.............................................................................................. vii Transmission of Information to Participants....................................................................................... viii Initial Signs of Activity Impact and Sustainability.................................................................................. ix Lessons Learned from the R&I Approach............................................................................................... x Recommendations.................................................................................................................................. x 1. Introduction ................................................................................................................................1 1.1 Background and Setting: Eastern DRC ...........................................................................................1 1.2 FSP-Enyanya Activity Overview......................................................................................................2 2. Overview of Mid-term Evaluation.................................................................................................5 2.1 Purpose ..........................................................................................................................................5 2.2 DRC DFSA MTE Objectives..............................................................................................................6 2.3 Methodology..................................................................................................................................6 2.4 Limitations and Delimitations ........................................................................................................9 3. Findings.....................................................................................................................................11 3.1 Evaluation Objective 1: Service Delivery......................................................................................11 3.1.1 P1 – Agriculture...........................................................................................................11 3.1.2 P2 – Health and Nutrition ...........................................................................................15 3.1.3 P2 – WASH...................................................................................................................17 3.1.4 P3 – Governance .........................................................................................................18 3.1.5 Monitoring and Evaluation (M&E)..............................................................................19 3.2 Evaluation Objective 2: Evidence of Changes..............................................................................21 3.2.1 P1 – Agriculture...........................................................................................................21 3.2.2 P2 – Health and Nutrition ...........................................................................................22 3.2.3 P2 – WASH...................................................................................................................22 3.2.4 P3 – Governance .........................................................................................................24 3.2.5 Across purposes..........................................................................................................25 3.3 Evaluation Objective 3: Activity Design........................................................................................26 3.3.1 P1 – Agriculture...........................................................................................................26 3.3.2 P2 – Health and Nutrition ...........................................................................................27 3.3.3 P2 – WASH...................................................................................................................28 3.3.4 P3 – Governance .........................................................................................................29 3.3.5 M&E ............................................................................................................................30 3.4 Evaluation Objective 4: Coordination, Collaboration, and Convergence.....................................30 3.4.1 Within the FSP-Enyanya Consortium..........................................................................30 3.4.2 With other USAID-funded activities............................................................................31 IMPEL | Implementer-Led Evaluation and Learning ii Table of Contents 3.4.3 With the Government of DRC .....................................................................................32 3.4.4 With other INGOs........................................................................................................33 3.5 Evaluation Objective 5: Sustainability..........................................................................................33 3.5.1 P1 – Agriculture...........................................................................................................33 3.5.2 P2 – Health & Nutrition...............................................................................................34 3.5.3 P2 – WASH...................................................................................................................34 3.5.4 P3 – Governance .........................................................................................................35 3.5.5 Across Purposes..........................................................................................................35 3.6 Evaluation Objective 6: Refine and Implement............................................................................36 3.6.1 General Impressions ...................................................................................................37 3.6.2 P1 – Agriculture...........................................................................................................38 3.6.3 P2 – Health & Nutrition...............................................................................................38 3.6.4 P2 – WASH...................................................................................................................38 3.6.5 P3 – Governance .........................................................................................................39 3.7 Evaluation Objective 7: Cross-Cutting Themes............................................................................39 4. Conclusions and Recommendations............................................................................................42 4.1 Evaluation Objective 1: Service Delivery......................................................................................42 4.1.1 P1 – Agriculture...........................................................................................................42 4.1.2 P2 – Health & Nutrition...............................................................................................44 4.1.3 P2 – WASH...................................................................................................................45 4.1.4 M&E ............................................................................................................................46 4.2 Evaluation Objective 2: Evidence of Changes..............................................................................46 4.2.1 P3 – Governance .........................................................................................................46 4.2.2 Across purposes..........................................................................................................47 4.3 Evaluation Objective 3: Activity Design........................................................................................48 4.3.1 P1 – Agriculture...........................................................................................................48 4.3.2 P2 – Health & Nutrition...............................................................................................48 4.3.3 P2 – WASH...................................................................................................................50 4.3.4 P3 – Governance .........................................................................................................50 4.4 Evaluation Objective 4: Coordination, Collaboration & Convergence.........................................51 4.5 Evaluation Objective 5: Sustainability..........................................................................................52 4.5.5 P2 – WASH...................................................................................................................52 4.5.6 Across Purposes..........................................................................................................52 4.6 Evaluation Objective 6: Refine and Implement............................................................................53 4.7 Evaluation Objective 7: Cross-cutting Themes.............................................................................53 4.8 General Recommendations for FSP-Enyanya...............................................................................54 5. Annexes.....................................................................................................................................56 5.1 FSP-Enyanya Organizational Chart...............................................................................................56 5.2 Theory of Change .........................................................................................................................56 5.3 FSP-Enyanya Intervention Descriptions.......................................................................................60 5.4 MTE Objectives and Complete Evaluation Questions..................................................................70 5.5 Data Collection Sites Visited.........................................................................................................73 5.6 Evaluation Team Composition .....................................................................................................75 Mid-Term Evaluation of the FSP-Enyanya DFSA in the DRC Table of Contents iii 5.7 Mid-Term Evaluation Scope of Work ...........................................................................................76 5.8 Consent Forms & Data Collection Tools.......................................................................................91 5.8.1 Informed Consent Forms – English .............................................................................91 5.8.2 Guide for Farmers' Focus Groups – French.................................................................95 5.8.3 Nutrition/Health Guides – French.............................................................................100 5.8.4 In-Depth Interview (Natural Leaders) – French ........................................................116 5.8.5 In-Depth Interview Promoteurs WASH – French......................................................118 5.8.6 Interview Guide for CLD Members/Members of Justice and Peace Commissions – French 120 5.8.7 Focus Group Discussion Guide for Participants 14-18 Years of Age – French ..........122 5.8.8 Management KII & In-Depth Interview Tool.............................................................123 5.8.9 M&E KII & In-Depth Interview Tool...........................................................................127 LIST OF TABLES AND FIGURES Figure 1: Mercy Corps South Kivu Food Security Project Intervention Areas..............................................2 Table 2: FSP-Enyanya Purposes, sub-purposes, and related interventions .................................................3 Table 3: Data collection table for Mercy Corps (MC) FSP-Enyanya MTE per study area (village)................8 Table 4: FSP-Enyanya R&I Studies ..............................................................................................................37 IMPEL | Implementer-Led Evaluation and Learning iv Acronyms ACRONYMS AEA Agriculture Extension Agent AOR Agreement Officer’s Representative ARR Annual Results Report APC Action pour la Paix et la Concorde BCZ Bureau Central de la Zone de Santé CARM Community Accountability & Responsibility Mechanism CG Care Group CLA Collaborating, Learning, and Adapting CLTS Community-Led Total Sanitation CODESA Comité de Développement de l'Aire de Santé CoH Channels of Hope CoP Chief of Party CRS Catholic Relief Services CSC Community Score Card CSO Civil Society Organization CVA Citizen Voice in Action DFSA Development Food Security Activity DHS Demographic and Health Survey DIP Detailed Implementation Plan DMC Dialogue and Mediation Committee DPS Provincial Directorate of Health (DPS) DRC Democratic Republic of Congo ET Evaluation Team FANTA Food and Nutrition Technical Assistance FFA Food for Assets FFS Farmer Field School FGD Focus Group Discussion FH Food for the Hungry FFP Food for Peace FSP South Kivu Food Security Project FY Fiscal Year GBV Gender Based Violence GoDRC Government of DRC HH Household IDI In-depth Interview IHP Integrated Health Project INGO International Non-Governmental Organization INERA National Institute for Agronomic Studies and Research IP Implementing Partner IPAPEL Provincial Inspectorate for Agriculture, Fisheries and Livestock Mid-Term Evaluation of the FSP-Enyanya DFSA in the DRC Acronyms v IPM Integrated Pest Management IPTT Indicator Performance Tracking Table IT Head Nurse (Infirmière Titulaire (de l’Aire de Santé) in French) JFFS Junior Farmers’ Field School KII Key Informant Interview KSPH Kinshasa School of Public Health LDC Local Development Committee M&E Monitoring and Evaluation MC Mercy Corps MFI Microfinance Institution MoU Memorandum of Understanding MTE Mid-Term Evaluation NGO Non-Governmental Organization NL Natural Leader OD Open Defecation ODF Open Defecation Free PO Producers’ Organization PPC Positive Peer Couple R&I Refine and Implement RMNCH Reproductive, Maternal, Newborn and Child Health RECO Community Health Worker (Relais communautaire in French) SBC Social & Behavior Change SD Standard Deviation SENASEM National Seed Service SNHR National Rural Hydraulic Service (Service National de Semences in French) SVC Strengthening Value Chains TOC Theory of Change TP II Tuendelee Pamoja II UEA Université Evangélique de l’Afrique VSLA Village Savings and Loan Association WASH Water, Sanitation and Hygiene WMC Water Management Committee (COGEP in French) WV World Vision YBG Youth Business Group IMPEL | Implementer-Led Evaluation and Learning vi Executive Summary EXECUTIVE SUMMARY Background In September 2016, the U.S. Agency for International Development’s (USAID) Office of Food for Peace (FFP) issued an award to Mercy Corps (MC) Democratic Republic of Congo (DRC) to implement a Development Food Security Activity (DFSA). It is a five-year activity (October 2016 to September 2021) with a budget of $38 million, being carried out in the South Kivu province. The overall program goal of the South Kivu Food Security Project (FSP)-Enyanya is to “improve food and nutrition security and economic well-being of vulnerable households in South Kivu.” FSP-Enyanya has three primary purposes:  P1: Agriculture: Household incomes are increased  P2: Health, Nutrition, and WASH: Improved nutrition status in communities  P3: Governance: Socio-economic status is stable and inclusive FSP-Enyanya is being implemented by Mercy Corps as the prime, World Vision as both the main sub￾partner and the lead on health interventions; Harvest Plus as the supplier of bio-fortified seeds; Université Evangélique de l’Afrique (UEA) leading activities related to soil fertilization, erosion control, and providing technical leadership to implement the hill approach; and Action pour la Paix et la Concorde (APC) as the conflict management interventions, with an emphasis on land access. FSP-Enyanya is targeting 80% of the population living in its zone of influence or 35,000 households, with particular attention on women, youth and children under two years of age. The first year of implementation was a pilot of FFP’s Refine and Implement (R&I) approach that was largely dedicated to formative research. Methodology A mid-term process evaluation of FSP-Enyanya was conducted in August and September 2019, led by Tulane University, under the umbrella of the Implementer-Led Evaluation and Learning award (IMPEL). The mid-term evaluation (MTE) aimed to assess the quality of program service delivery, identify evidence of changes to date due to FSP-Enyanya’s interventions, assess the quality, relevance and efficacy of FSP-Enyanya’s design, assess the degree and benefits of efficacious coordination, collaboration, and convergence with locally-based partners and external organizations, assess early evidence of sustainability produced by FSP-Enyanya interventions, assess the appropriateness and quality of the R&I approach, determine the appropriateness and effectiveness of interventions focused on cross-cutting themes and recommend adjustments to FSP-Enyanya implementation or design. The MTE field team was a seven member team composed of consultants from Tulane University, the Kinshasa Public School of Health, and Food for Peace Washington. Team members had expertise in qualitative research design, implementation and analysis, program evaluation, agriculture, Mid-Term Evaluation of the FSP-Enyanya DFSA in the DRC Executive Summary vii livelihoods, food security, nutrition, WASH, and monitoring and evaluation. The DFSA AOR participated as an observer. The team had support from five data collectors/translators. The MTE used a mixed-methods approach consisting primarily of qualitative data collection methods, such as key informant and in-depth interviews, focus group discussions, and direct observation. Water quality samples were taken and tested using the Aquagenx Compartment Bag Test. Quantitative data collected from the FSP-Enyanya M&E team included monitoring data, annual reporting data and secondary data. Data were collected from August 17 to September 5, 2019 in six health areas and in eight villages. On August 29, a one-day validation workshop was held in Bukavu to ‘ground-truth’ the preliminary evaluation findings. This allowed the MTE team, the AOR observer from FFP Washington, and an observer from FFP Kinshasa to discuss initial findings from the MTE’s fieldwork with Mercy Corps, World Vision, Harvest Plus, UEA and APC technical and field staff. Findings The evaluation identifies several areas where FSP-Enyanya has had important successes that should be highlighted as lessons learned, as well as other areas where opportunities to improve and learn can inform FSP-Enyanya as well as other DFSAs. These areas include:  Timeliness of activity implementation and M&E  Transmission of information/technology comprehension to participants  Initial trends in impact and sustainability  Lessons learned from the R&I approach Timeliness of Implementation and M&E Given the R&I approach, FSP-Enyanya had limited time to ramp up activities. Year one of the activity exclusively conducted formative research. Most field staff were hired during FY2 (October 2017 to September 2018) after which FSP-Enyanya began ramping up implementation, including recruiting the volunteers across FSP-Enyanya’s zone of influence that support the activity’s cascade model. Even taking into account the relatively short implementation period at the time of the evaluation, some interventions are behind schedule or in need of improved service delivery quality across interventions. For example, the Producer Organization (PO) intervention (creating/strengthening Producer Organizations, post-harvest handling/storage, linkages to inputs, financial or technical services and market linkages) is either weak, behind schedule or non-existent. Additionally, a number of agricultural activities started only at the beginning of season A in the last quarter of FY18 and start of FY19. To address this, all the unimplemented interventions were carried forward and re-planned for FY19. The R&I pilot introduced significant delays and inefficient programming. Significant delays in review and approval of key M&E Plan documents on the part of Food for Peace was an important contributor to delays in setting up the M&E system at the start of the project. The M&E team, with assistance from Mercy Corps regional staff, spent several months revising and re-revising the activity Theory of Change at the request of Food for Peace. This process eroded the relationship between the FSP-Enyanya M&E team and technical teams and contributed to the technical teams’ decisions to IMPEL | Implementer-Led Evaluation and Learning viii Executive Summary create their own monitoring systems that they could put in place in the interim. Only in the 12 months prior to the MTE did FSP-Enyanya work with technical teams to discard those systems in favor of a centralized, digital data collection platform and a system of qualitative data aggregation. However, despite the initial challenges in setting up the M&E processes, Mercy Corps has successfully built internal staff capacity (both on the M&E team and on technical teams) to carry out most key M&E functions. Importantly, there appears to be a strong culture of learning at FSP￾Enyanya among activity staff (which even extended to learning about better ways to document and use learning). There is not, however, a systematic method or practice for feeding these lessons learned from the field into the DIP (annual work-planning) process. Transmission of Information to Participants One of the cross-purpose challenges of FSP-Enyanya is quality and completeness of the transmission of information (knowledge, technology, practice, etc.) to the participants. The general impression given by FSP-Enyanya participants was that the overarching activity design focused on too many initiatives (especially trainings), whereby program implementation seemed to be best represented by a long list of trainings without adequate in-depth follow-up to ensure the achievement of longer￾term outcomes. The MTE team recommends that FSP-Enyanya think critically about the most important interventions to scale and those that should be cut back or eliminated in order to have an impact on food security in these three health zones. One key finding is that farmers are very enthusiastic to learn about and adopt new practices. This enthusiasm should continue to be encouraged however, it has resulted in some challenges in information transmission. For example, the transmission of knowledge from Agriculture Extension Agents (AEA) and Farmer Field School (FFS) committee members to farmers is uneven and the technologies are not being applied accurately or consistently. Perhaps the metaphor of the “cart before the horse” is significant here, where participants gain only introductory information about an agricultural technique then attempt to perform it immediately before they have a full comprehension. Similar enthusiasm in adoption of the hill approach to reduce soil erosion and improve soil quality risks failure related to quality assurance, as it requires a consistent and complete combination of practice, making it vulnerable to even small deficiencies. Comprehension by rabbit recipients of how to raise, manage, and breed the animals is limited and varied. Increased education prior to receiving animals (rabbits) as well as frequent monitoring visits by FSP-Enyanya staff and Village Livestock Agents during the first few months after reception should improve the livestock intervention. To improve transmission of information and knowledge to participants without sacrificing enthusiasm, FFS may benefit from adopting more of an "experiential learning" methodology as the core learning process. Using this methodology the FFS would try all new crops, inputs, or techniques in the designated learning plot and also on a small scale in participants’ plots to test before making a decision about adoption and scale-up. This may result in an initial increase in farmer’s workload, as was found by the MTE for women who became involved in the permagarden intervention while also being involved in numerous other FSP-Enyanya interventions. It is to be expected that participants in this ZOI are eager to become involved in the opportunities presented by FSP-Enyanya and that their Mid-Term Evaluation of the FSP-Enyanya DFSA in the DRC Executive Summary ix workload will increase, at times significantly, due to this enthusiasm. However, FSP-Enyanya can mitigate this, as it is currently doing, by continuing to conduct analyses of participant workload and having field staff check in with participants on how the increased labor is being distributed amongst their households. Most of the community-based health and nutrition interventions under P2 are considered best practices by FFP, and the MTE team was quite impressed with FSP-Enyanya’s commitment to addressing key underlying determinants of malnutrition and food security, making the connection between the conceptual and the practical. Two examples of this were the 1) FSP-Enyanya’s focus on adolescent nutrition and food security via the Safe Spaces approach and 2) the attention given to family planning through multiple approaches: Safe Spaces, PPC, CoH and both Care Groups. The main design problem noted with the Safe Space approach is that its coverage targets of the adolescent population is not ambitious enough. Currently, FSP-Enyanya reaches about 3500 adolescents via Safe Spaces (with plans to target more than double this amount by the end of the activity), which is a small percentage of the adolescents in the target health areas. Expanded coverage of the adolescent population may be more likely to lead to population-based impact in nutrition and knowledge, attitude, and practice of sexual health behaviors. Initial Signs of Activity Impact and Sustainability Although the short period of implementation to date for FSP-Enyanya makes it challenging to identify which interventions are most likely to have a sustainable, long-term impact, the ET was able to identify several early positive signs of impact and sustainability. Farmer field schools, VSLAs, Safe Spaces, Natural Leaders, CVA and tax trainings are all well-received and clearly benefiting participants. However, variation in quality of service delivery and delays in timing threaten the sustainability of the activity effects. In particular, interventions such as FFS and VSLA do not have a clear transition plan. Water Management Committees require additional training in order to effectively ensure the long-term functioning of the water points being constructed and rehabilitated by FSP-Enyanya as they are unlikely to receive necessary financial support from local government in the case of a breakdown. Other important issues threaten the longer-term view of sustainability. Because of the weak and delayed activities related to POs, the MTE found it unlikely that the POs will survive after activity close. To remedy this, FSP-Enyanya may consider loosening restrictions on memberships to POs to allow for more members to join, to consolidate the existing POs into fewer, stronger groups, and better clarifying the role of the Feed the Future’s Strengthening Value Chains (SVC) activity to better support the most vulnerable and empower small-holders (rather than seeking higher-level market participants), or eliminate the SVC activity. Layering of other USAID activities needs to be clarified and finalized. The role of IHP will require definition regarding health system strengthening. The coordination of Strengthening Value Chains Activity and FSP-Enyanya must be strengthened in order for FSP-Enyanya to best target interventions. FSP-Enyanya also needs to collaborate with government and other partners for legitimacy of interventions and to invest in sustainability after activity closure. Good progress has been made with local governments and informal leaders to date. Despite the difficult and evolving IMPEL | Implementer-Led Evaluation and Learning x Executive Summary context in S. Kivu, continued efforts in collaboration will be an important factor in short and long￾term sustainability. FSP-Enyanya would benefit from a more developed, and unified sustainability strategy planned across the different purposes at the intervention level. To date, the MTE team observed that there have been good linkages of activities across purposes, using this model will help inform an improved sustainability strategy. Lessons Learned from the R&I Approach The MTE found that application of R&I had mixed effects on the implementation of FSP-Enyanya while benefiting design. As one of the pilot countries for R&I, the activity suffered from confusion both on the part of FFP and program staff as to how exactly R&I should work. One key point of confusion was how much research versus implementation should be happening during the first year. The IP understood that implementation should only begin in FY2. Because of this delay in project activities, FSP-Enyanya is still viewed by some community members as an extractive program that collects information rather than helps local populations. In addition, the process of approvals for R&I research and results was very slow. This was primarily due to the large number of studies that required review and approval in a short period of time. As an example, the Health, Nutrition and WASH Services Assessment Report collected data in September of 2017, but the report was finalized in March 2019, well after WASH activities began. On the other hand, the formative research conducted on FSP-Enyanya interventions allowed for several important pivots in programming. Examples include the inclusion of the hill approach came out of the R&I formative research, as did interventions focusing on land access, localized conflict, and tax laws. The MTE team recommends that FFP continues to clarify with IPs how R&I should be implemented and allow for some interventions to begin during the first year. Initial program design can include a set of core elements from which formative research can build out program intervention strategies and components. Recommendations This evaluation has yielded specific recommendations for each of the evaluation objectives, further divided by activity purpose. These include operational and strategic recommendations specific for the activity purposes, M&E, for the Refine and Implement approach (R&I), as well as cross-cutting and cross-sectoral recommendations. The evaluation team presented preliminary recommendations during a validation workshop with FSP-Enyanya consortium members and FFP Washington and Kinshasa at the end of the field data collection. FSP-Enyanya should begin collaborating on common lessons learned and other challenges with the two other DFSAs being implemented simultaneously in DRC, especially the one based in Bukavu. Other FFP programs (both future and those being currently implemented) stand to learn from FSP-Enyanya’s experience and the consortium already has a vast network through which to seek solutions and disseminate lessons learned. Mid-Term Evaluation of the FSP-Enyanya DFSA in the DRC Introduction 1 1. INTRODUCTION 1.1 Background and Setting: Eastern DRC Eastern Democratic Republic of Congo (DRC) has experienced decades of conflict, poor governance, and, as a result, widespread extreme poverty. The genocide that took place in neighboring Rwanda unfortunately spilled over into Eastern DRC, as an easy and convenient place for Hutus to flee to.1 The region suffered tremendously during the aftermath of that conflict which then evolved into the war in eastern DRC between 1998 and 2007, during which an estimated 5.4 million people died as a result of the conflict and protracted humanitarian crisis.2 Mortality rates were higher in eastern DRC, demonstrating the effect of insecurity, with most deaths from easily preventable and treatable illnesses rather than violence. 3 Millions of more citizens were pushed into poverty due to displacement and loss of economic livelihoods. By 2012, most of the country was relatively stable, although armed factions have persisted. In 2017, 52 armed groups were active in South Kivu alone, committing thefts, attacks, collecting illegal taxes and exploiting natural resources (forests and mines.)4 Coupled with the longstanding conflict, DRC experiences significant political tensions: both the 2011 and 2018 presidential and legislative elections were marred by violence and disputes leading up to the elections and regarding the results across the country. Gender inequalities are prevalent; in 2018 DRC ranked 147 out of 166 countries on the Gender Development Index. The Demographic and Health Survey of 2013-2014 found that 27% of Congolese women aged 15-49 have experienced sexual violence in their lifetime, and more than half had experienced some form of physical violence.5 The 2013-14 DHS reported that over 40% of Congolese children suffer from chronic malnutrition indicated by stunting (-2 Standard Deviation (SD) height-for-age), with more than half of these children falling into the severe range (-3 SD height-for-age), signifying a very serious public health problem according to WHO classifications. Eastern DRC is also grappling with the world’s second largest Ebola epidemic on record, with more than 2000 lives lost and 3000 confirmed infections since the outbreak was declared on 1 August 2018.6 Other issues of note in the region are high rates of alcoholism and land access is very difficult. The combination of the rapidly growing population, traditional practices related to land inheritance, which causes division of family properties, and land degradation, is decreasing the availability of arable land and the size of agricultural fields. A Comprehensive Food Security and Vulnerability Analysis conducted in 2011-12 by the World Food Programme found 64% of the rural population in South Kivu to be food insecure, with Kalehe ranking 3rd (72.2% of the population food insecure) and Kabare ranking 4th (70.8%) of the 12 South Kivu territories. 1 https://www.theguardian.com/world/2008/may/16/congo 2 (Moszynski, 2008) 3 (Coghlan 2006) 4 Fonds Social de la RDC, Cartographie des conflits de la province de Sud Kivu, Février 2017. 5 (DHS 2013-14) 6 https://www.who.int/emergencies/diseases/ebola/drc-2019 IMPEL | Implementer-Led Evaluation and Learning 2 Introduction 1.2 FSP-Enyanya Activity Overview To assist vulnerable populations of South Kivu given the challenges above, USAID’s Office of Food for Peace (FFP) awarded Mercy Corps implementation of a five-year Development Food Security Activity (DFSA) from September 2016-September 2021. Mercy Corps’ South Kivu Food Security Project Enyanya7 is operating in three health zones8 (Kalehe (territory of Kalehe), Miti-Murhésa and Katana (territory of Kabare)) with a budget of $38 million.9 FSP-Enyanya10 is being implemented by Mercy Corps as the prime, World Vision as both the main sub-partner and the lead on health interventions; Harvest Plus supplies bio-fortified seeds; Université Evangélique de l’Afrique (UEA) is responsible for soil fertilization and erosion control as well as technical leadership to implement the hill approach; and Action pour la Paix et la Concorde (APC) is implementing conflict management interventions, with an emphasis on land access. Figure 1: Mercy Corps South Kivu Food Security Project Intervention Areas Source: Mercy Corps FSP-Enyanya’s overall program goal is “improved food and nutrition security and economic well￾being of vulnerable households in South Kivu”11 and it is targeting approximately 35,000 households. 7 Enyanya means “go forward” in Mashi, one of the main languages spoken in FSP-Enyanya’s intervention areas. 8Health zone is the larger administrative level, covering 50,000-100,000 people in rural areas. Health area is the next level down, covering 5000-10000 people in rural areas and typically have one health center per health area. 9Mercy Corps was originally awarded $35 million but this was increased to $38. 10 FSP will be used as the primary name of the activity throughout this report. 11 FSP ARRNarrative FY 2018 Final Mid-Term Evaluation of the FSP-Enyanya DFSA in the DRC Introduction 3 This activity has three purposes that are then further divided by sub-purpose and addressed through the related interventions12 as follows: Table 2: FSP-Enyanya Purposes, sub-purposes, and related interventions Purposes Sub-Purposes Related Interventions 1 - Agriculture: Household incomes are increased 1.1: Agricultural productivity, especially of nutritious, lower-risk food crops, increased  Farmer Field Schools  Junior Farmer Field Schools  Tree Nurseries  Village Savings and Loan Associations 1.2: Women, men, youth actively participate in income-generating activities  Producer Organizations (PO)  Youth Business Groups (YBG)  Microfinance Institutions  Seed Multipliers  Literacy and Numeracy 2 – Health, Nutrition and WASH: Improved nutrition status in communities 2.1: Improved Healthy Timing and Spacing of Pregnancy  Safe Spaces  Positive Peer Couples  Channels of Hope  Health Service Providers 2.2: Women, men, children eat optimal diets  Care Groups  Permagarden/Livestock 2.3: Community members, mothers & Children Under 2 are healthy  Care Groups  Community-Led Total Sanitation (CLTS) and Water Point Construction/Rehab  Health Services and Community Health Workers (RECOs) 2.4: Children whose growth is faltering identified and referred for care and counseling  Care Groups  CLTS  Health Services and Community Health Workers (RECOs) 3 – Governance: Socio-economic status is stable and inclusive 3.1: Women, youth participate in community planning and decision￾making  Local Development Committees (LDC)  Care Groups, Safe Spaces, YBGs, POs, VSLAs 3.2: Communities have the info, tools & processes to monitor and improve the socio-economic environment  LDCs  Food for Assets (FFA)  Civil Society Organizations (CSO) 3.3: Communities are resilient against conflict  Land Access Securitization  Dialogue & Mediation Committees  CSO 3.4: Taxation practices perceived as transparent & fair  CSO  Local government, traditional authorities 12 For a detailed description of FSP-Enyanya’s interventions, please refer to Annex 5.3 IMPEL | Implementer-Led Evaluation and Learning 4 Introduction The leadership team, based in Bukavu, provides managerial, administrative and technical support to the field teams based in the regional bases in each health zone and implementing field activities. Mercy Corps runs the Miti-Murhésa and Katana bases, while World Vision runs the Kalehe base. Program staff rely on a network of volunteers across purposes to reach a larger number of participants. Under each purpose, program field staff train volunteers who are directly associated with FSP-Enyanya on technologies or modules who then cascade out learnings to the larger group of FSP-Enyanya participants. Please refer to Annex 5.1 for FSP-Enyanya’s Organizational Chart. Mid-Term Evaluation of the FSP-Enyanya Development Food Security Activity in the DRC Overview Of Mid-Term Evaluation 5 2. OVERVIEW OF MID-TERM EVALUATION 2.1 Purpose Between August and September 2019, mid-term evaluations (MTE) were conducted for all three DFSAs13 being implemented in the DRC. The purpose of these three mid-term process evaluations is to improve the quality of activity interventions and increase the likelihood of sustainable and positive impacts on communities and individuals in the implementation areas. These three evaluations used the same methodology. As with the other evaluations, the FSP-Enyanya evaluation aims to:  Assess the overall program strategy in relation to the respective Theory of Change (TOC);  Review the quality of program service delivery;  Examine the relevance and appropriateness of activities according to the local needs and context;  Analyze intended and unintended consequences of the activities at the individual- and community-levels;  Examine the extent to which the activities have successfully collaborated with partners and other stakeholders;  Identify sustainability pathways and inform the strategic plan to achieve sustained, positive outcomes; and  Determine the appropriateness and effectiveness of cross-cutting themes such as gender, youth and governance in each activity. A critical component of the MTE is to assess FFP’s recently introduced Refine and Implement (R&I) approach. R&I can be summarized as requiring IPs to conduct formative research during the first year of a DFSA award and to then use the findings from this research to adapt the activity’s programming at the beginning of the second year, where necessary. The DFSAs that were awarded in 2016 in DRC and Liberia are the first two countries to apply R&I to FFP-funded activities, and the approach itself is still in a refinement period. The DRC MTE sought to pay particular attention to:  The appropriateness and quality of R&I activities;  How the R&I activities were used to inform activity design and implementation;  The extent the R&I approach impacted the appropriateness and effectiveness of field interventions; and  Whether activities should be eligible for a 1-5 year cost extension based on the R&I model. The MTE team provides recommendations on ways to strengthen the R&I approach based on findings under Evaluation Objective 6. 13 CRS Budikadidi DFSA in Kasai Oriental, Mercy Corps DFSA FSP-Enyanya in South Kivu and Food for the Hungry DFSA Tuendelee Pamoja II in South Kivu & Tanganyika. IMPEL | Implementer-Led Evaluation and Learning 6 Overview Of Mid-Term Evaluation 2.2 DRC DFSA MTE Objectives The overall objectives guiding these three MTE evaluations are following and the full evaluation objectives as well as accompanying evaluation sub-questions are available in Annex 5.4. In summary, the eight evaluation objectives for this evaluation are: 1. Review the quality of program service delivery related to the different themes and systems in addressing chronic food insecurity and child malnutrition with the targeted clients, taking into account contextual changes that may have occurred since the inception of the activities. 2. Identify evidence of changes (positive and negative, intended and unintended) associated with program interventions. This will include identifying factors that appear to promote or hinder women’s, men’s, and young people’s food security and safety. 3. Assess the quality, relevance and efficacy of the DFSA design, taking into account whether activities are contextually appropriate, address critical needs, and maintain standards that can impact on positive change. Evaluate relevant programmatic principles, such as whether approaches are human centered, evidence based, systems oriented, focused on the most vulnerable, designed for multiple interventions to target the same household, and focused on quality. 4. Assess the degree and benefits of efficacious coordination, collaboration, and convergence with locally-based DFSA partners and external organizations that are critical to achieve DFSA goals and purposes. 5. Assess early evidence of sustainability produced by the DFSA activities thereby determining the extent to which outcomes, systems, and services are designed and being implemented to continue after the DFSA ends. 6. Assess the appropriateness and quality of the R&I approach and activities. How well did the R&I approach work in DRC across the 3 DFSAs. 7. Determine the appropriateness and effectiveness of interventions focused on cross-cutting themes for the activities. This includes cross-cutting interventions designed to improve gender and equity in decision making to achieve food security outcomes and targeting youth to improve their access to, participation in, and benefit from DFSA interventions. This also includes analysis of the effectiveness of activities designed to strengthen governance at the local level, in an overall effort to empower communities to sustain improvements in food security and nutrition, and environmental risk and mitigating the impacts of climate change. 8. Recommend adjustments to program implementation or design and explain how these changes would improve program outcomes and sustained impact. 2.3 Methodology The FSP-Enyanya MTE is a process evaluation that uses a mixed methods approach. This approach consists primarily of qualitative data collection methods, such as key informant and in-depth interviews, focus group discussions, and direct observation. Water quality samples were taken and tested using the Aquagenx Compartment Bag Test. Quantitative data was also collected from the FSP-Enyanya M&E team, including monitoring data, annual reporting data and secondary data. This MTE, as the other DRC MTEs, were conducted using a participatory approach and the team was composed of both female and male evaluators with a range of technical and research expertise from Tulane University, the Kinshasa School of Public Health (KSPH), FFP Washington and Mercy Corps. Please see Annex 5.6 for a complete list of the FSP-Enyanya evaluation team members. Mid-Term Evaluation of the FSP-Enyanya DFSA in the DRC Overview Of Mid-Term Evaluation 7 The process to conduct the MTE began with an extensive review of FSP-Enyanya program documentation, development of interview guides and site selection. Prior to fieldwork, the evaluation teams for the concurrent Catholic Relief Services (CRS) and Mercy Corps evaluations participated in a two-day training on August 12 and 13, 2019 to finalize data collection tools and delineate roles and responsibilities across team members. Final versions of interview guides and the data collection sites are presented in Annexes 5.8 and 5.5. Once in Bukavu, four translators were tested for language competency and hired to accompany the MTE team with Mashi, Kihavu, Swahili, French and English14 language skills. Prior to fieldwork, the translators were briefed on FSP-Enyanya and participated in a half-day training focusing on qualitative data collection methods led by the KSPH qualitative specialist and one of the Co-Team Leads. An emphasis was put on asking open￾ended questions, techniques to remain unbiased during interviews and ensuring full participation during focus group discussions. Once the full MTE team was in Bukavu, the Mercy Corps-led FSP-Enyanya consortium held a half-day meeting to present FSP-Enyanya’s interventions and allow MTE team members to ask questions of technical staff. Data collection was conducted from August 17- 28, 2019 by the full evaluation team accompanied by the four translators and an Agreement Officer Representative (AOR) observer from FFP Washington and from August 29 to September 5 by remaining team members. Data was collected in six health zones and in seven villages. These villages were selected for data collection purposively based on proximity to the regional base (selecting one no more than five kilometers away, and one at least five kilometers away), as well as the package of interventions being implemented in the village. Regarding interventions, at least two villages had to feature the hill approach, two had to feature road rehabilitation, and two had to feature WASH infrastructure; in each case this had to be one per IP. A balance was sought between villages where Mercy Corps versus World Vision is the lead implementer. This approach allowed the MTE team to observe all of the types of FSP-Enyanya’s interventions and see how they are being implemented by each IP. Complete details on the selected villages and total numbers of interviews conducted can be found in Annex 5.5. Due to ongoing security concerns in the intervention areas and in order to respect both USAID’s and Mercy Corps’ security regulations, the relevant FSP-Enyanya chief of the regional base was contacted up to two days prior to data collection to confirm there was no security risk in or near each village proposed for the sample. Field staff working in the selected village were informed the day before 14 There was only one translator with Kihavu, Mashi, Swahili and French language skills and one with Mashi, Swahili, French and English language skills. Photo Credit: Diana Caley Validation Workshop Discussion IMPEL | Implementer-Led Evaluation and Learning 8 Overview Of Mid-Term Evaluation data collection. This allowed the MTE team to arrive without giving staff or participants too much warning or time to prepare responses. Table 3: Data collection table for Mercy Corps (MC) FSP-Enyanya MTE per study area (village) Data collection method Kashunguri, Cinjoma, Cishanga￾Kalwa (MC) Munanira (WV) Luhihi (MC) Lemera (WV) Bushumba (MC) Bukavu Totals KIIs 6 7 7 2 1 14 37 IDIs 11 14 11 12 16 1 65 FGDs 12 9 7 12 5 - 45 Water Samples 2 - - 2 1 - 5 Observations 7 6 7 10 8 - 38 Once in the selected village, the MTE team coordinated with field staff to schedule key informant and in-depth interviews, focus group discussions and observations. Key informant interviews were conducted with FSP-Enyanya staff and in-depth interviews were conducted with participants, local leaders and health center personnel. All interview questions were asked by an evaluation team member, translated into the local language (Mashi, Swahili, Kihavu, and French) and responses were translated back in order for the evaluation team member to take notes. Note-taking was conducted on paper in the field and on computers in Bukavu. Electronic versions of note summaries were produced by each sector lead at the end of fieldwork. During the first few days of data collection, MTE team members were encouraged to adjust interview guides where appropriate based on their improved understanding of FSP-Enyanya interventions. Consent was obtained for all of the interviews and FGDs that were conducted and an audio recording was made for which additional consent was provided by respondents. There were some cases of refusal of the audio recording. Direct observations consisted of ET members assessing road rehabilitation, demonstration plots, water infrastructure, markets, permagardens, household latrines, and the hill approach or observing a gender training. Water samples were taken of sources improved by FSP-Enyanya and from household storage containers. They were tested using the Aquagenx Compartment Bag Test in order to determine most probable number of fecal contamination present in the sample. The MTE team worked in each village for two days except in the case of Kashunguri, Cinjoma and Cishanga-Kalwa, where two days were spent in total for the three villages. Following the second day of data collection, the team met to share impressions during a nightly briefing. The nightly briefings consisted of a report-out from each technical sector (agriculture, health and nutrition, WASH, governance, M&E, sustainability) highlighting key impressions or questions based on the data collected the previous two days. On August 29, a one-day validation workshop was held in Bukavu that allowed the MTE team, the AOR observer from FFP Washington and an observer from FFP Kinshasa to discuss initial findings Mid-Term Evaluation of the FSP-Enyanya DFSA in the DRC Overview Of Mid-Term Evaluation 9 from the MTE’s fieldwork with Mercy Corps, World Vision, Harvest Plus, UEA and APC staff, from both managerial and field teams. Evaluators from each technical sector presented findings via a PowerPoint in the morning and later participants were split per technical sector to discuss findings related to FSP-Enyanya design, implementation and sustainability of outcomes. This exchange enabled each group to discuss what the MTE team had observed thus far, some of the challenges on the IP’s side, and to identify next steps within each group. Following the validation workshop, FFP Washington members of the MTE team as well as the FFP Washington AOR observer departed, and the Mercy Corps participant began preparing a planning workshop for Mercy Corps technical staff. The remaining members of the MTE team along with the four translators continued data collection for two more days in an additional village and then for three more days in Bukavu with FSP-Enyanya staff. The selection of the final village (Bushumba) and prioritization of the final interviews to be conducted were guided by the information gained through the validation workshop. The above table briefly summarizes the data collection completed for this MTE and a more detailed version of this table is included in Annex 5.5. After fieldwork ended, technical sector leads of the MTE team analyzed the various types of data and triangulated information gathered from documentation review, nightly briefings held during fieldwork and field notes taken during data collection. Through this analysis process, each lead prepared a summary report per the eight evaluation objectives relevant to their sector. Given the compressed timeline to produce first initial recommendations and later a draft report, transcription of the audio recordings was not an option due to the associated time commitment. The Team Leads were asked to provide initial recommendations in advance of the final report submission due to the timing of the fiscal year end and overall planning for year 4. These initial recommendations were based on the summaries provided by the technical leads and were limited to the highest priority recommendations. 2.4 Limitations and Delimitations The MTE encountered several challenges in its execution, primarily related to the time allowed for the ET pre-planning and fieldwork. The text below discusses limitations (things out of the control of the ET, such as security restrictions) and delimitations (evaluation design decisions). The period allotted for planning of the MTE was short due to a delay in signing of the contract and time constraints related to limited travel availability of some evaluation team members. These constraints restricted the timing of data collection. The FSP-Enyanya MTE team consists of seven technical members with varying levels of experience conducting evaluations and familiarity with Food for Peace programming; given time limitations, protocols and training were adaptive during the field work. Fieldwork was limited to fourteen days, primarily for budgetary reasons. The MTE was originally scheduled in June 2019, but due to delays in contract signing and evaluation team member availability, the MTE took place at the same time as the Mercy Corps COP’s annual leave. Mercy Corps leadership covered the CoP’s absence, but given his role and history with programming in DRC the fieldwork phase was not able to benefit from his wealth of knowledge. Also due to the delay in data collection, Mercy Corps had two other major studies planned for the same time period. The risk IMPEL | Implementer-Led Evaluation and Learning 10 Overview Of Mid-Term Evaluation of survey fatigue for FSP-Enyanya participants was a concern. Security concerns in South Kivu during the time of fieldwork limited the team movement in the field. The team was not allowed to leave Bukavu before 7 a.m. and had to be back within city limits by 5 p.m. This restricted the time available in each village for data collection to a maximum of five hours per day because the travel time to sites was at least 1.5 hours by car. Additionally, due to budgetary constraints and the wide array of sectors covered by FSP-Enyanya interventions, subject matter experts were not available for all intervention areas. For example, there was no specialist dedicated specifically to governance, gender, or youth. The ET did not have sufficient information at the time the sample was created of the timeline of group creation for each intervention in order to use this as additional criteria for sampling villages included in the qualitative data collection. The MTE team acknowledges that some technical oversight also resulted from the short planning period and extensive sectoral interventions of the activity. Insufficient assessment of agricultural input systems, FFA for water infrastructure and the literacy component of the activity are most notable. Mid-Term Evaluation of the FSP-Enyanya DFSA in the DRC Findings 11 3. FINDINGS 3.1 Evaluation Objective 1: Service Delivery This section presents the findings related to the quality of FSP-Enyanya’s service delivery observed by the evaluation team. Findings for objective 1 are presented by purpose (agriculture, health and nutrition/WASH, and governance) and grouped by intervention. An analysis of these findings is shared in the following section, Conclusions and Recommendations. Across the FSP-Enyanya activity, interventions are operating with acceptable but varying quality and intensity across program sites. The MTE team would like to emphasize that implementation in general for all three DRC DFSAs is behind where an activity with three years of implementation should be because the first year of these DFSAs has been dedicated to formative research—not to hiring staff and beginning implementation. FSP-Enyanya operated in the first year with the technical team in Bukavu but did not hire many field staff until FY2 (October 2017-September 2018). Only after all of the field staff were hired was FSP-Enyanya able to begin recruiting the volunteers across FSP-Enyanya’s zone of influence that support the activity’s cascade model. Implementation, therefore, was ramping up throughout FY2. This has had an impact on the findings for both evaluation objective 1 (service delivery) and evaluation objective 2 (evidence of changes due to the activity) and is not a reflection of FSP-Enyanya, but is reflective of the real amount of time the activity has been implementing each intervention. 3.1.1 P1 – Agriculture As described above in the background on FSP-Enyanya, the goal of purpose 1 is to increase household income of participants and the two main sub-purposes are to increase agricultural productivity and to have participants (women, men, youth) engage in income-generating activities. There are several interventions under P1 being implemented to meet these goals. Farmer Field Schools (FFS) Farmer Field Schools have been established by FSP-Enyanya as a way to transfer knowledge to a group of 25 “lead farmers” over the course of 12 months. Participants receive seeds and are taught new techniques by Agriculture Extension Agents at the FFS demonstration plot. Each AEA is responsible for 10 FFS at a time. In Y2, 144 FFSs were established, and in Y3, FSP-Enyanya planned to establish an additional 96 schools.15 The ET found that the FFS were set up in the field and had distributed seeds that lead farmers used for one season in the demonstration plot before receiving seeds that they used in their individual plots. Uniformly, the FFS all comprised 25 producers with each group supported by an AEA. For both Mercy Corps and World Vision, only the FFS committees received direct training from FSP-Enyanya staff. (These committees are usually composed of six to eight people.) The training sessions were held in classrooms during workshops, and committee members were expected to share the knowledge with other FFS members. It appeared that these committee members have worked with the other FFS members under the supervision of the AEAs. 15 FY19 Q3 Report, April-June 2019 USAID South Kivu Food Security Project IMPEL | Implementer-Led Evaluation and Learning 12 Findings The ET found that the quality of the transmission of knowledge from committee members to other FFS members is uneven (e.g., not all individual fields showed that the techniques taught had been adopted). The evaluation team was able to visit several individual plots of FFS participants, and the techniques taught were not systematically applied. Each AEA was responsible for monitoring participant’s fields in about 20 FFS (i.e., a group of about 400 individuals). Taking into account the distance between FFS, it can be challenging for the AEAs to do their job well. Their heavy workload means they cannot spend enough time in each FFS and then in the individual fields to monitor the application of the practices adopted and to give advice to producers. The ET learned that AEAs are managing two to three times as many groups as they are committed to managing. All of the FFS visited referred to regular interaction with the AEAs during implementation of the FFS; however, the interaction was reported to be less regular after the FFS were implemented. Junior Farmer Field Schools (JFFS) and Youth Business Groups (YBGs) JFFS use the same model as FFS except their target is youth instead of adults. In practice, little has been done on this front, only six JFFS had been set up as of Q316. This is due to the fact that the activities of the JFFS are the same as those of the FFS and that the AEAs are already overburdened. Youth Business Groups are youth associations that emulate POs. These groups of select young men and women receive business training for agribusiness opportunities and their groups are eligible for small grants to boost the development of their income generating activities based on business plans. The ET found that some YBG participants shared business plan concepts but have no notion of how to transform a plan into action nor how to establish market links. YBG participants were able to list trainings attended, but the majority did not demonstrate skills which should result from such trainings. Producer Organizations (PO) Upon completion of (J)FFS, farmers automatically “graduate” into a Producer Organization. These POs typically share the same market goals (in terms of value chain) and they are designed to build social and economic cohesion among farmers attempting to connect goods to market, as well as explore storage and transport options. FSP-Enyanya is meant to support POs with production and the USAID activity Strengthening Value Chains (SVC) is meant to support POs connection to markets. The first 30 POs were initiated in Y2, and another 42 had joined the project in Y3 by Q3, reaching the project target of 72 POs. The MTE team found that FFS participants “graduate” to Producer Organizations with little understanding of what that means or how to activate and, of the POs contacted during data collection, none were officially certified at that time. No one interviewed (in FFSs, YBGs or POs) knew the difference between profits and sales, an important concept for successful business operation. Given the short period of existence of these POs, these weaknesses are understandable and point to specific areas for reinforcement both with newer and older POs. At the time of data collection POs had yet to experience support from SVC market linkages and were uniformed as to how to attain meaningful market competence and engagement. There are layers of confusion within this program, including ambiguity for POs on the certification process and PO functions (which SVC has indicated must precede their efforts), and SVC’s understanding of their own role compared to FSP-Enyanya staff’s expectations of SVC. 16 FY19 Q3 Report, April-June 2019 USAID South Kivu Food Security Project Mid-Term Evaluation of the FSP-Enyanya DFSA in the DRC Findings 13 Bio-pesticides & Integrated Pest Management (IPM) One of the technologies female and male farmers are being taught through the permagarden and (J)FFS interventions is application of bio-pesticides as an alternative to chemical fertilizer. Some of the farmers who have been taught this skill and were interviewed by the ET indicated they are adopting use of “bio-pesticides” before understanding how to correctly mix ingredients, correctly identify pests, or accurately choose which mix to use for which pest (hence efficacy is limited). Agroforestry, Soil Erosion Control & the Hill Approach As presented in the background section on FSP-Enyanya, consortium partner UEA is introducing agroforestry through the hill approach and soil erosion control techniques on select hillsides to improve soil quality and erosion conditions, and in turn improve crop productivity. Trees are provided through the tree nursery intervention. “Hill” owners are subsequently enticed to grant longer-term leases with farmers as conditions improve, and farmers should feel more confident about investing in conservation measures with longer leases. Of the 18 hills targeted for re-design by FSP-Enyanya, nine had begun work in Y2, and another nine were planned for Y4. The ET observed both well-executed examples of the hill approach and examples that require improvement. Permagardens The permagarden intervention is a cross-purpose intervention with Purpose 2 Health and Nutrition. It consists of Care Group participants (female farmers) being taught to cultivate small gardens (typically next to their homes) for both nutrient-rich food production and income-generation. CG participants receive a starter kit of vegetable seeds and fruit tree seedlings as well as training on resilient agriculture techniques. Permagarden agricultural techniques include specific surface water management, soil water conservation and soil building techniques to be considered permagardens. The term “Permagarden” is often used interchangeably with “backyard gardens” amongst FSP￾Enyanya participants however practices specific to permagardens are generally absent or done incorrectly. It appears that thus far gardening techniques are being applied in the permagardens which do not result in the same water conservation as if permagarden techniques were being used. The confusion introduced previously related to IPM is also having a negative effect on the success of permagardens. Although FSP-Enyanya has developed handbooks that indicate local “recipes” for controlling certain diseases and pests, in the field participants concoct a unique mixture of chili and extracts of some plants to fight all kinds of diseases and pests. Crop losses for participants can be enormous, especially when it comes to virus diseases. The ET also learned that, as expected, the workload of participants, female farmers, increased when they became involved in this intervention. Small Livestock (Rabbits) The livestock raising intervention is also a cross-purpose intervention between P1-Agriculture and P2-Health & Nutrition. The intervention consists of distributing rabbits to Mother Leaders who multiply them in their homes then distribute the offspring amongst the Care Groups they lead. The ET found that comprehension by rabbit owners of how to raise, manage, and breed the animals varies between participants, and is at times limited. The majority of first litters died due to illnesses, participants’ initially limited understanding of veterinary services, and lack of understanding of rabbit management. For example, owners were uninformed as to why offspring should be separated from parents (i.e., inbreeding). IMPEL | Implementer-Led Evaluation and Learning 14 Findings Guidelines for rabbit offspring were found to be unclear and inconsistent among participants. The concept was for landless Mother Leaders to multiply rabbits and share with their neighbors, but offspring management is seen as an individual choice instead and rabbit offspring are not consistently being shared. Furthermore there is limited support from vets and lack of medicine for rabbits in the intervention areas. Finally, in combination with the permagarden intervention, the MTE team learned that some participants are receiving both rabbits and permagarden starter kits which was not the intent for either of these interventions and raises questions of favoritism. Tree Nurseries Twelve tree nurseries were contracted to produce 810,000 seedlings in total for distribution to FSP-Enyanya Hill Approach sites, permagarden sites, and small-holder plots. The nurseries are managed by 360 young people who receive training in production and business from FSP-Enyanya, and Food for Asset17 commodities for their work in the FSP-Enyanya tree nurseries. The nurseries are supposed to also begin receiving 20% of the cost of production for each tree seedling, however this had not begun at the time of the MTE. Seedlings are provided for free from FSP-Enyanya to the hill approach, permagarden participants and FSP-Enyanya smallholders. The MTE team found no issues with the quality of service delivery for this intervention. Village Savings and Loan Associations (VSLA) Under FSP-Enyanya, VSLAs are groups of on average 25 members who create a savings group from which to generate loans to one another (for agricultural inputs, school fees, etc.). Groups are guided to produce constitutions, structuring agreements, governance, etc. by a FSP-Enyanya VSLA Agent. 17 Food for Assets is described in detail under the following purpose. It is the FFP program that provides participants with American-origin food commodities in exchange for work. Photo Credit: Eva Christensen Small Livestock Intervention Example Photo Credit: Eva Christensen Tree Nursery in Luhihi Mid-Term Evaluation of the FSP-Enyanya DFSA in the DRC Findings 15 Each VSLA agent is responsible for nine VSLAs and each group is provided a box and lock in which to store their savings. Groups meet once a week and participants are typically active in other FSP￾Enyanya interventions. The ET noted a high level of female participants in the VSLAs. VSLAs charge 10% interest per month to all loans to make profits for members and borrowing is contingent upon repayment by other members. Loans are on average 150,000 Congolese francs and borrowers have three months to repay plus one month grace period. Members also contribute weekly to a social fund for emergencies or for culturally important ceremonies. 72 VSLAs had been set up in Y2, and another 130 as of Q3 of Y3. However, according to the IPTT figures from July 2019, the number of individuals participating in VSLAs was well below targets. VSLA principles seem well-received by participants and the underlying methodology appears to be understood and applied by members. VSLAs interviewed by the ET function well, fulfill the intended purpose of making small-scale credit available, and have also created social cohesion among members (mentioned several times). 3.1.2 P2 – Health and Nutrition To achieve the Purpose 2 overarching goal of improved nutritional status, the P2 team implements several interventions to address the issue of healthy pregnancy under sub-purpose 2.1, two other interventions to ensure participants are consuming a nutritious diet under 2.2, several others that aim to keep participants healthy related to 2.3 and 2.4 focuses on preventing and treating stunting in FSP-Enyanya’s zone of influence. Care Groups One of the backbone P2 interventions is Care Groups, which consist of 10 to 15 volunteers (Mother Leaders) who, once trained by FSP-Enyanya on a Care Group module, ensure the transfer of their knowledge to the women in their neighborhood (neighbor women). A service delivery quality concern was noted by the ET with regards to quality of social and behavior change (SBC) implementation under this intervention. Several instances were observed where Mother Leaders had challenges in providing accurate information to neighbor women. In one observed example, a neighbor woman expressed concerns with having to pay for services, it was revealed that the mother leader indicated that women should only go to the hospital (at over 40 Km away) to meet their needs for family planning services, even though these services could be met at a local health center. One Mother Leader supervisor interviewed (known as Mere de Mere) shared that she spends a great deal of time explaining to her fellow Mother Leaders how to best understand the Care Group modules, citing that Mother Leaders could not solve basic problems with infant and young child feeding. She also shared that few Mother Leaders are able to read and write, and she is required to fill out their registers for the field staff at the end of the month. In addition, Mother Leaders are being trained to identify the most severe cases of malnutrition, but not moderate forms, which limits effectiveness of the program. Health Center Support At health center level, FSP-Enyanya supports health centers through capacity strengthening of health staff and provision of materials, equipment and supplies. With regards to capacity strengthening of health staff and community health workers (RECO in French), the activity limited itself to date to IMPEL | Implementer-Led Evaluation and Learning 16 Findings training of up to two facility-based staff and a select number of RECOs in each health area. Given that there are currently limited partners supporting the health centers in nutrition, maternal child health and family planning, these activities are relevant, but are limited as training is not adequate to achieve full capacity building. This is in part due to the late start of USAID’s Integrated Health Project, which is intended to support health systems as a partner of FSP-Enyanya. Mentorship, accompaniment and supportive supervision is inadequate. FSP-Enyanya staff were not able to describe what capacity building of health staff entails beyond training. In addition, some of the trainings have been ill-timed, namely the training on Consultation Pre-scolaire, since the MTE team learned that the materials for growth monitoring and promotion arrived several months after the training was completed. The MTE team also learned from field and managerial team members that some youth who had sought family planning or contraception services from local health centers were either turned away or reported to parents by clinicians or other staff at those facilities. Finally, at the time of data collection, a clear understanding of what role IHP will play at the health center level was lacking. Service Delivery Observations Some activities had been taking place for between 12 and 18 months at the time of MTE data collection, such as Mother Care Groups, Safe Spaces18 and training of health staff and community health workers. Other activities have more recently been implemented by FSP-Enyanya, in the 3-4 months preceding data collection. These include Positive Peer Couples19 and Father Care Groups. In some health areas, however, certain interventions have not started or were in the very early stage of implementation. As such, the level of advancement of these interventions varied across health areas. Kalehe health zone appeared quite advanced in implementation of a more complete package of interventions, while others are less advanced. This may be due to the fact that World Vision, the P2 leader for the project, is the implementer in Kalehe Health Zone and it may be due to the previous existence of another FFP-funded development program in the same area. The consortium did not decide to physically place World Vision technical staff across all three health zones, instead the World Vision technical staff retain technical oversight of the Mercy Corps P2 staff in Miti-Murhésa and Katana. The MTE team also noted variation of service quality across the FSP-Enyanya health zones. There was variation in the number of project staff dedicated to monitoring and supervision of 18 In the Safe Spaces intervention youth participate in weekly activities where they are able to express their opinions and learn about topics such as nutrition, hygiene, reproductive health, gender, and leadership. Groups are separated by gender. 19 Positive Peer Couples are married couples who use or have used family planning successfully to delay or space pregnancies. The community identifies PPCs by selecting couples that implement the expected collaborative decision￾making in family planning and gender equity. Photo Credit: Mike Manske Health Center in Luhihi Mid-Term Evaluation of the FSP-Enyanya DFSA in the DRC Findings 17 certain interventions, with fewer numbers of project staff and volunteers dedicated to monitoring and supervision in some cases. For example in one area there was only one Safe Space Coordinator dedicated per Health Zone and some Safe Space mentors were responsible for 30 adolescents, while the Safe Spaces should not have more than 15 participants. However, as of Q3 in FY19, the Safe Spaces had provided nutrition and hygiene activities and exercises to the participant target of 3,555 youth. There was similar unevenness with the quality of implementation of other interventions, such as Care Groups. 3.1.3 P2 – WASH Water Infrastructure As with the other purposes addressed thus far, the late implementation start has impacted the status of FSP-Enyanya’s WASH component. Related to water, FSP￾Enyanya plans to construct/rehabilitate 24 water points across FSP-Enyanya’s zone of influence. WASH staff support the creation of Water Management Committees (WMC or COGEP in French) for each water point and then train them on water point cleanliness, maintenance and financial management. The Bureau Central de la zone de santé (BCZ) leads these trainings. Service delivery for water infrastructure is progressing and water infrastructure was in the process of being constructed during the MTE. However, as of July 2019, only seven had been constructed/rehabilitated, well behind the target of 13. WMC have been formed following a consistent democratic process inclusive of gender across the intervention areas and have been trained on water point maintenance as well as the need to collect funds from users in order to pay for repairs. However WMCs have limited understanding of how to collect funds to pay for repairs of the water points they are responsible for. This is to be expected considering they have only participated in one training and users had not been paying for water previously. Community-Led Total Sanitation (CLTS) To reduce open defecation and therefore reduce participant’s risk of waterborne disease, namely diarrhea, FSP-Enyanya is using the Community-Led Total Sanitation (CLTS) approach. CLTS consists of “triggering” a village by showing residents that they have been consuming feces by allowing open defecation in their village and then getting the village to agree to abandon OD through construction and use of household latrines, along with best hygiene practices. Similar to the process taken to establish the WMCs, FSP-Enyanya has worked with communities to nominate Natural leaders (NL) to implement CLTS across the three health zones. These NL, in turn, identify WASH promoters to assist with their work on sensitization around latrine use and in teaching the key WASH messages across their communities. Both NL and WASH promoters are volunteers. Interviewed NLs reported that Photo Credit: Annette Fay Water point constructed by FSP-Enyanya IMPEL | Implementer-Led Evaluation and Learning 18 Findings communities are receptive to the WASH messages and adopting most of the behaviors FSP-Enyanya is promoting. CLTS is being implemented in 147 FSP-Enyanya villages. However, the majority of villages had been triggered in the six months prior to the MTE, and at the time of data collection no villages had reached ODF declaration or certification. With an activity that only has 1.5 years of implementation remaining (at best), this is slightly alarming because it gives FSP-Enyanya limited time to accompany these communities towards abandoning and maintaining Open Defecation Free status. The accompaniment period is key to understanding the factors that make maintaining ODF difficult for a community. These factors will contribute to slippage if not addressed while FSP￾Enyanya is present. Based on interviews and FGDs with NLs and WASH promoters, it appears that FSP-Enyanya is conducting regular supervision of this intervention. 3.1.4 P3 – Governance The P3 component’s overarching goal is to establish a stable and inclusive socio￾economic environment in FSP-Enyanya interventions areas. The sub-purposes FSP￾Enyanya has divided this into are (4.1) inclusive decision-making, (4.2) that participants have the tools to monitor and improve their socio-economic environment, (4.3) that communities are resilient in the face of conflict, and (4.4) that taxation is perceived as transparent and fair. Generally speaking, under P3 service delivery is adequately being implemented after roughly a year and a half of FSP￾Enyanya implementation. All four interviewed Local Development Committees (LDCs are committees representative of all major social strata and sub-villages within the village that were created by FSP-Enyanya to coordinate food security and development activities at the village level) possess written Local Development Plans. LDPs are documents that outline a village’s development priorities. All communities observed by the ET related to P3 have evidence of public works projects supported with FFA. Over half of the communities visited by the ET have already participated in a Citizen Voice in Action health center campaign. CVA is a process by which users, civil society organizations and government employees evaluate government services. The approach helps to define performance norms, a Community Score Card (CSC) and action plan, allowing government actors to implement recommendations and stakeholders to evaluate them after a 1-year period. With the support of FSP-Enyanya consortium partner Action pour la paix et la concorde, the P3 team has also created Dialogue and Mediation Committees (DMCs) to guide communities in conflict resolution. DMCs are active in nearly all communities visited by the ET. For mediation specifically, all observed DMCs had visitation hours clearly posted within their office and the majority of DMCs had clear mechanisms for monitoring and reporting on conflicts and mediation processes. All interviewed Photo Credit: Justin Colvard FGD with Local Development Committee Mid-Term Evaluation of the FSP-Enyanya DFSA in the DRC Findings 19 community members suggested that they were extremely satisfied with the free mediation services provided by these offices, although across sites the mediation process seemed to have slight variations in implementation. These variations could be due to intentional localized adaptations to the process, or else could be due to a lack of quality control of DMC approaches. Food for Assets The FFA intervention consists of compensating participants for work to improve public infrastructure through payment with food commodities from the USA. Participants work in cycles of 22 days and receive 180g20 of vegetable oil, 720g of split peas and 2400g of rice (this was changed from cornmeal earlier in FSP-Enyanya) for their labor. Under FSP-Enyanya, FFA is being given in exchange for work in the tree nurseries, community work on water infrastructure and road rehabilitation. FSP-Enyanya supports the creation of a FFA committee to manage and maintain each FFA public works project. These mixed gender committees appear to be created fairly. The selection of FFA recipients under the P1 tree nursery intervention appears to be only targeting some nursery employees and not all, bringing up questions related to a clear definition of FFA criteria. However, FSP-Enyanya’s declared targeting was consistent with FFP Strategy for road rehabilitation and water infrastructure: Those, “…most severely food insecure…an opportunity for the neediest households to access food, especially during the lean season”. Activities to be performed by participants for food were also consistent with the Strategy: To be used, “to improve public infrastructure through community work for food”. 3.1.5 Monitoring and Evaluation (M&E) The quality of program service delivery has been supported by a well-designed and well-executed M&E system, support from senior-level management, and a culture of learning among activity staff. FSP-Enyanya staff interviewed for the MTE generally had a firm understanding of their roles and responsibilities with respect to data collection, although there was some inconsistency when it came to staff members’ understandings of how data can and should be used in practice to make adaptive management decisions. All field staff interviewed expressed that key features of the M&E system￾most notably the quarterly atelier (workshops) and weekly emails to line managers-enabled all technical team members to better understand and communicate to others what was happening on the ground in their respective sectors. Several individuals noted that the workshops in particular helped individual staff members to see how their efforts (e.g. leading trainings) contributed to the wider team’s efforts to support purpose-level results. Additionally, it was clear that Mercy Corps has successfully built internal staff capacity (both on the M&E team and on technical teams) to carry out most key M&E functions. Importantly, there appears to be a strong culture of learning at Mercy Corps among activity staff (which even extended to learning about better ways to document and use learning). Several field staff, for example, said they looked forward to attending the quarterly workshops (led by the M&E team) because these workshops provide an opportunity to learn how they are doing and how they can do better. Staff conveyed that the COP has consistently supported building internal M&E capacity (i.e. as opposed to 20 FFA portions per 22 days of labor are typically communicated by individual (30g of vegetable oil, 120g of split peas and 400g of rice) however the amount provided at the end of 22 days is multiplied by 6 using the assumption that the individual is part of at least a six person household. IMPEL | Implementer-Led Evaluation and Learning 20 Findings hiring external consultants). The M&E team lead, for example, also described learning as the most important function of the M&E system, followed by tracking progress, reporting, and accountability. The M&E team has also set up a formal system to document, aggregate, and track lessons learned from the field, e.g. from ‘pause and reflect’ sessions and other fora. There is not, however, a systematic method or practice for feeding these lessons learned into the DIP (annual work-planning) process. The M&E system has for the most part been designed and set up in twelve months leading up to the midterm evaluation. There are several reasons that caused delays, including those within and outside the manageable control of the partner. The M&E budget included in the final award was drastically less than what has been proposed at proposal (since the final award was half of the proposed value). When the FSP-Enyanya activity received a $3 million top-off in FY1 the M&E budget was partially restored, which supported a pivot in approach from reliance on short-term, external technical assistance to building in-house staff capacity. The M&E team members (who were brought on after the budget was realigned), with support of senior leadership, were able to re-create problematic systems that were set up initially in FY1. For example, the technical teams had created informal monitoring systems (complete with their own indicators and data collection forms) in light of the fact that they had little or no formal M&E institutional support (due to the lack of staffing). An important factor that contributed to delays in setting up the M&E system initially were significant delays in review and approval of key M&E Plan documents on the part of Food for Peace. The M&E team, with assistance from Mercy Corps regional staff who were reassigned to this task, spent at least six months revising and re-revising the activity Theory of Change21 at the request of Food for Peace. This process eroded the relationship between the FSP-Enyanya M&E team and technical teams and contributed to the technical teams’ decisions to create their own monitoring systems that they could put in place while the M&E documents went back and forth for final approval. The M&E team in the 12 months prior to the MTE worked with technical teams to discard those systems in favor of a centralized, digital data collection platform (with some complementary paper forms) and adopt a system of qualitative data aggregation (e.g. via email and in-person meetings). The MTE team found that field staff varied in their ability to provide examples of qualitative or process types of data or information. Two different field staff members (in two separate interviews) gave “number of women versus men” as examples of qualitative information, for example. The MTE team did not have the time or resources to interview a sufficiently large sample of Mercy Corps and World Vision staff to achieve thematic saturation to support a comparison of staff from the two organizations; however, according to KIIs with Mercy Corps staff, the MTE team found they generally had a stronger understanding of qualitative data or qualitative monitoring practices (i.e. as demonstrated by being able to provide examples) while World Vision staff interviewed were unable to provide additional examples even when prompted with illustrative qualitative monitoring practices or data sources). FSP-Enyanya’s M&E plan does include qualitative methods, however. One of the most difficult aspects of the FSP-Enyanya activity to monitor program quality (in support of good implementation) is P3 and, more specifically, governance. The M&E system under P3 is primarily set up to track and report on indicators required or requested by USAID, however, these 21 The original Theory of Change and the updated TOCs for each purpose are available in Annex 5.2. Mid-Term Evaluation of the FSP-Enyanya DFSA in the DRC Findings 21 are not necessarily the data points that support timely and effective performance monitoring. As noted above, the focus on quantitative measures--which in the case of governance are largely output-based measures rather than outcome-based-generally don’t enable effective monitoring of these interventions. The need for improvement of qualitative monitoring methods under P3 has undermined the P3 team’s capacity to monitor performance and communicate progress to others both within and outside of FSP-Enyanya. Community Accountability and Response Mechanism (CARM) The Community Accountability and Response Mechanism (CRM in French), the operational complaints and feedback mechanism, appears to be working well and supporting FSP-Enyanya’s capacity to track complaints (i.e. including unintended consequences). However some areas for improvement were noted by the ET. Mercy Corps and World Vision operate parallel systems. The FSP-Enyanya team is currently working on trying to create a unified system but it is not clear when this will be completed and what the implications for these dual systems might be. Secondly, the highest level of complaints that go into the Mercy Corps system go up to a centralized tracking and sorting system run by specialists in Goma and Mercy Corps headquarters (i.e. outside of the FSP￾Enyanya program office). While the system is designed like this to improve accountability and transparency and reduce the possibility of bias, FSP-Enyanya staff are unable to see patterns and levels of complaints in a timely way to support adaptive management, although the CoP is generally informed. Additionally, there is a long delay between when complaints (or positive feedback) are received and when this information reaches the FSP-Enyanya team. 3.2 Evaluation Objective 2: Evidence of Changes This section presents the findings related to Objective 2: evidence of changes (positive and negative, intended and unintended) due to FSP-Enyanya’s interventions to date. Both positive and negative changes were detected across purposes. Before presenting them in detail, the MTE team calls attention to the fact that FSP-Enyanya interventions started very late due to the introduction of R&I in the first year and the activity approval process. As such, some interventions may not have had sufficient implementation time to show evidence of change. Detailed intervention descriptions for each purpose are available in Annex 5.3. 3.2.1 P1 – Agriculture Starting with P1, several positive changes are apparent in relation to FSP-Enyanya interventions. Mother Leaders are creating permagardens to augment consumption of nutritious foods (as intended by design), and have indicated purchasing of food has decreased while consumption of nutritious foods held constant or improved. The hill approach is appreciated by beneficiaries Photo Credit: Mike Manske Gardening tools for a Mama Leader IMPEL | Implementer-Led Evaluation and Learning 22 Findings (farmers) and hill owners alike, where executed successfully, and neighboring farmers are interested in replicating. Owners are expressing interest in more flexible and longer rental terms thanks to the benefits from the hill approach, which would enable farmers to remain multiple years and invest in sustainable land management. Technologies being used through the hill approach appear successful at reducing soil erosion and improving soil fertility. In another example, a hill owner from Luhihi Centre is discussing rent reduction and has arranged with farmers how they choose to pay—either part of harvest or a fixed amount (in cash). VSLA principles seem well-received: participants understand the methodology and apply it. VSLAs interviewed function well, fulfill the intended purpose of making small scale credit available, and have also created social cohesion among members (mentioned several times). One woman interviewed in Munanira reported that she used a VSLA loan to start a business selling cookies. Once she had saved 50,000 CDF (about $30 USD), she was able to borrow three times this amount (150,000 Congolese Francs, or about $90 USD). She paid 10% interest, had one month grace period, and was able to pay the loan off in three months. She said it was easy because now she had profits with the expanded business. 3.2.2 P2 – Health and Nutrition Related to P2 Health and Nutrition, positive intended changes that were documented by the MTE team include changes in behavior at the individual level, as well as improvements in use of health services documented at the health facility level. The mother care group participants reported increased awareness of best practices in infant and young child feeding. Specifically, improvements in exclusive breastfeeding until 6 months were cited on several occasions. At the health facility level, all Infirmier Titulaire (IT) or head nurse health staff cited increased or sustained uptake in use of family planning methods over the past 3 months, while several mentioned increases in use of pre￾school consultations22 and antenatal care visits. In addition, several IT also mentioned reduced cases of diarrhea and acute malnutrition. Finally, Safe Space mentors and adolescent participants were able to confidently describe areas where their lives have improved, including children who have gone back to school, a large number who have cited changes in their outlook towards gender roles, others who have been able to seek health services where they were otherwise afraid to do so. One lauded success story observed by the ET was that of a Safe Space participant who decided to turn away from joining a militia group as a result of discussions with his mentor. 3.2.3 P2 – WASH Water Infrastructure The water infrastructure intervention consists of building or rehabilitating 24 water points across Mercy Corps and World Vision zones. This intervention was delayed and only two capped springs were visited, as well as one water system that had been completed only days before the MTE began. Water management committees have been assembled for each water point. Functional water points 22 Consultation pré-scolaire in French. Mid-Term Evaluation of the FSP-Enyanya DFSA in the DRC Findings 23 have increased water access, although overall coverage remains a challenge in FSP-Enyanya’s intervention area because the villages span vast distances. The MTE team was not able to visit many examples of constructed or rehabilitated water points due to the limited data collection time and the need to send the entire MTE team to the same village for two days, however in Munanira samples were taken from both a capped spring that was improved by FSP-Enyanya and a household water storage container. Both water samples tested within acceptable levels of fecal coliform (1.523 for the spring and 0 for the HH storage.) This indicates the construction and rehabilitation work conducted by FSP-Enyanya is providing safe drinking water to participants. Community-Led Total Sanitation (CLTS) The P2 CLTS intervention is credited by interviewed natural leaders and WASH Promoters as increasing latrine usage and construction. Observations in the sampled villages corroborated usage of latrines; however of the thirty observed household latrines only three had a handwashing station. Participants are said to know the five key times for handwashing as taught through FSP￾Enyanya. Natural leaders and WASH promoters monitor the progress of adoption of latrines on a bi-weekly basis by conducting door to door visits. Barriers to latrine construction remain for more vulnerable households, e.g. widows and the elderly, however NL and WASH promoters reported addressing this problem by assisting these HH with latrine construction. The HH visits help remind community members who did not previously have a latrine of the village’s commitment to move towards ODF and the interaction enables the NL to identify specific barriers to adoption, and, where necessary, to assist these households in building their latrines. Natural leaders and WASH promoters explained that drinking water treatment has not been adopted because people don’t want to spend energy to boil water or to spend money on water treatment tablets available in the area. This was expected by FSP-Enyanya and both NL and WASH promoters seem to agree that it is more important they continue working on behavior change regarding other behaviors. 23 According to WHO guidelines, 1-10 E. coli CFU per 100 mL is an « intermediate risk/probably safe. » Photo Credit: Annette Fay Latrine in good condition IMPEL | Implementer-Led Evaluation and Learning 24 Findings 3.2.4 P3 – Governance For P3, Citizen Voice in Action processes (where implemented), FSP-Enyanya’s intervention facilitating acquisition of customary land titles24, DMC-led mediations, and trainings on tax law25 showed the most signs of positive change of all P3 interventions studied. Not only did trainings result in increases in knowledge, but the interventions appeared to result in improved outcomes for participants. One health center in Munanira made multiple improvements to their services after having participated in a CVA process; the head nurse showed the MTE team examples of doors that have been added to showers, and toilets that had been improved since the start of the CVA campaign, and mentioned that these structures would be repainted in the following days. FSP￾Enyanya participants praised customary land titles as being easier and more transparent to obtain, community members felt confident that localized conflicts were adequately resolved, and local leaders and community members shared several stories of successful elimination of illegal taxes. Some LDCs mentioned positive results of advocacy efforts. For example, Munanira’s LDC successfully convinced a non-partner INGO to integrate some development plan activities—primarily water and sanitation system rehabilitation—into their project. However, many LDCs did not have any success stories of convincing external actors to prioritize their LDP-defined activities. Furthermore, nearly all LDCs admitted that they perceived themselves as having no influence on FSP-Enyanya activities or strategy. Social inclusion efforts were not sufficient. Evidence of majority-dominated groups was accentuated when examining participation of people with disabilities and other typically marginalized groups, such as pygmy populations. The MTE team encountered two individuals with disabilities during FGDs; in both instances, people with disabilities were systematically prevented by LDC participants from openly sharing their views with the evaluators. Although many committees admitted to the existence of a large proportion of pygmies in their village, not a single pygmy￾identifying individual was encountered in a FGD. Some interventions are not yet having the effect desired by FSP-Enyanya. For example, following Inclusive Decision-Making gender training, women are systematically integrated into various committees and community groups, but their level of attendance does not appear to consistently translate into increased active participation and influence in these committees. In many instances, female committee members present in mixed gender FGDs did not speak, and when they did speak they were often interrupted, or their opinions were “corrected” by male committee members. Although there were some positive examples of women’s views being respected and considered (and the MTE identified a handful of champions of women’s voice), observations suggest that in many communities female committee members might be seen as symbolic additions to ensure gender balance as promoted by FSP-Enyanya. In nearly all P3 committees and community volunteer positions, participant selection was successfully conducted through a democratic election process. This democratic process, however, appears to have resulted in a lack of representation of marginalized groups in some cases. The MTE 24 FSP-Enyanya facilitates access to more equitable land through the acquisition of customary land titles with legal recognition that protects smallholders from speculation on land. 25 FSP-Enyanya builds the organizational capacities of 22 civil society organizations to support democracy, good governance, inclusiveness and advocacy efforts to sensitize communities about official and illegal taxes. FSP-Enyanya also works with governments and local authorities to improve the transparency of taxation practices. Mid-Term Evaluation of the FSP-Enyanya DFSA in the DRC Findings 25 team found multiple examples of “community leaders” holding several roles within different committees. While democratic processes ensure that there is consensus amongst the majority, they risk silencing the voices of the most vulnerable while concentrating power in more popular, vocal, or well-off individuals or families, and reinforcing potentially harmful community dynamics. FSP￾Enyanya’s efforts to include quotas in committees appears to have mitigated this risk to some extent, but interviews with committee members suggested that these group politics still have significant influence on decision-making. In two FGDs, LDCs expressed a lack of agency and an overwhelming dependency upon the program to push forward development initiatives. Finally, election-based recruitment to some volunteer positions may result in choosing individuals that lack the core competencies necessary to adequately perform their duties, especially when electors are not aware of the capacity requirements to sit on certain committees. Food for Assets Among FFA-rehabilitated roads, the quality of the works seems to be acceptable overall. In most areas where a rainy season had occurred since the public works, significant soil erosion seems to have already negated maintenance efforts and additional work will be needed before the closure of FSP￾Enyanya to further rehabilitate these roads. None of these roads are paved and thus require ongoing maintenance, especially considering that the region has a rainy season that extends from mid-August to mid-May. In addition, FFA-supported public works (where labor is exchanged for commodities) solicited skepticism amongst FFA committees, who were concerned that an unintended consequence of providing FFA for community works (especially outside of road rehabilitation) is the result that in the future such works will not be accomplished without FFA support. Finally an unintended consequence observed by the MTE team regarding FFA was evidence of the same FFA food commodities for sale (which may or may not be from FSP-Enyanya, as other USAID projects are active in the evaluated areas) in the local markets within intervention areas. 3.2.5 Across purposes An unintended consequence that emerged across FSP-Enyanya’s purposes is jealousy, either between participants or between participants and non-participants. However, considering the high level of poverty in this post-conflict setting, such consequences are difficult to avoid. For example under P2 the ET learned of the creation of tension among household members due to limited shared understanding of selection criteria for participation in various health and nutrition activities. In particular, FSP-Enyanya has not provided ample and clear communications to the community members with regards to who will receive permagarden seeds and tools and rabbits. FSP-Enyanya participants stated that, by their understanding, only some benefitted from seeds/tools/rabbits because they were on a project list, while others were not. Four participants interviewed stated that Photo Credit: Justin Colvard Example of road rehabilitation IMPEL | Implementer-Led Evaluation and Learning 26 Findings food received from FFA, and donations of seeds, tools and rabbits have caused conflict with non￾participant neighbors, even when recipients offered to share knowledge and skills gained with those neighbors. Natural Leaders and WASH Promoters also described being met by jealousy from their non-participant neighbors. 3.3 Evaluation Objective 3: Activity Design This section presents the findings related to the third evaluation objective: assess the quality, relevance and efficacy of the DFSA design. Overall, the design of individual interventions seemed to be rigorous and adapted to the context, but the general impression given by FSP-Enyanya participants was that the overarching activity design focused on too many initiatives (especially trainings), whereby program implementation seemed to be best represented by a long list of training without adequate in-depth follow-up to ensure the achievement of longer-term outcomes. Under each purpose there is a mixture of positive and negative findings on FSP-Enyanya design. It is worth noting that there were several cases where the overlap between interventions had a positive, synergistic effect. 3.3.1 P1 – Agriculture Regarding intervention design, the majority of FFS groups interviewed indicated practices offered are locally appropriate and feasible. The hill approach is appreciated by participating male and female farmers and hill owners alike. Hill owners are expressing interest in flexible rental terms, enabling farmers to remain multiple years and invest in sustainable land management. The technologies being used appear successful at reducing soil erosion and improving soil fertility when applied correctly. The MTE team found that VSLAs are well received by FSP-Enyanya participants. However, groups hit a barrier once they get to the microfinance institution (MFI) stage. The savings group aspect works in supporting participants to save and gain access to small loans. However VSLA participants haven’t demonstrated willingness to risk taking a larger loan and there is a lack of MFI presence in FSP￾Enyanya’s intervention area. The design of the tree nursery support in exchange for FFA is questioned by the MTE team under P1. As mentioned related to findings for Evaluation Objective 1, these tree nurseries visited pre-existed the activity and are managed by youth. At the time of the MTE, employees had been receiving Food for Assets from FSP-Enyanya in exchange for work done at the nursery and will in the future receive revenue from 20% of the production costs of seedlings grown (not nearly the full cost to produce the seedlings). FSP-Enyanya provides these seeds to the nurseries, the seedlings are grown by the nurseries, then given back to FSP-Enyanya and finally distributed to local FSP-Enyanya interventions (permagardens and smallholder farmers) for free. According to some nursery participants, this arrangement has harmed their financial status, and may also result in market distortion when those seedlings are donated by FSP-Enyanya to locals within the Nursery’s previously established market area. The MTE team also questions whether FFA + 20% is a fair market price for these seedlings and, more importantly, how FSP-Enyanya will ensure a smooth transition for the nurseries upon program close when the nurseries will only be successful if the local market has developed demand for their products. Mid-Term Evaluation of the FSP-Enyanya DFSA in the DRC Findings 27 3.3.2 P2 – Health and Nutrition At the community level, FSP-Enyanya has designed multiple approaches to increase uptake of practices and use of health services, namely Mother Care Groups, Father Care Groups, Channels of Hope26 (CoH), Safe Spaces for adolescents and Positive Peer Couples. Given that Positive Peer Couples and Channels of Hope are relatively new activities, the MTE team was not able to go into depth on these approaches, but the limited initial data are positive, these interventions appear to be locally relevant and appropriate. Most of the community-based interventions, with some exceptions, are viewed as best practices by Food for Peace. 27 In addition, the MTE team was quite impressed with FSP-Enyanya’s commitment to addressing key underlying determinants of malnutrition and food security, making the connection between the conceptual and the practical. Two examples of this were the 1) FSP-Enyanya’s focus on adolescent nutrition and food security via the Safe Spaces approach and 2) the attention given to family planning through multiple approaches: Safe Spaces, PPC, CoH and both Care Groups. The main design problem noted with the Safe Space approach thus far is that it is not ambitious enough in its coverage of the adolescent population. Currently, FSP￾Enyanya reaches about 3500 adolescents via Safe Spaces (with plans to more than double this amount by the end of the activity), which is a small percentage of the adolescents in the target health areas. Expanded coverage of the adolescent population may be more likely to lead to population-based impact in nutrition and knowledge, attitude, and practice of sexual health behaviors. The MTE team also noted that Safe Space materials were not well adapted to meet the needs of the youngest adolescents (aged 10-14), or mentors were not comfortable sharing information. Specifically, several mentors mentioned that they skip over certain content related to sexual health, including avoiding showing images of the human sexual anatomy. The mentors felt that they can only share this with the older adolescents. While Safe Space activities have been effective in creating demand for services, the team noted that health facilities are not currently addressing the unique needs of adolescents. In one instance, an IT described that she must meet the adolescent at her home or after health center hours given that adolescents are not often comfortable coming to health centers in plain day due to fear of shame or negative consequences. Other providers described that they are not reaching adolescents, which they believe may be due to the fact that adolescents do not feel comfortable coming to them. Several instances were also noted where the selection of community volunteers to participate in interventions did not appear appropriate or could send the wrong message to other FSP-Enyanya participants. For example, the team noted that in some health areas Positive Peer Model Couples were selected based on criteria which included that they possess a tin roof or had a productive livelihood. The MTE team felt that this could confuse participants into thinking that only wealthier individuals could access family planning. This observation was confirmed upon further discussion with the P2 staff, who explained this criteria was used because they wanted the other community members to emulate people who were positive deviants. The MTE team noted several areas of 26Channels of Hope is a P2 Health & Nutrition intervention that aims to obtain support of religious leaders in order to promote the use of family planning and ideal birth spacing within FSP-Enyanya zones of influence. 24 According to information from FFP Nutrition Subject Area Expert IMPEL | Implementer-Led Evaluation and Learning 28 Findings concern where P2 approaches were either poorly adapted to the context or lacked appropriateness. First, the timing and sequencing of the activities is questionable, if not problematic given the complex nature of the factors affecting behavior change in family planning, health and nutrition. For example, Father Care Groups and PPC potentially should have begun earlier to address family planning issues to reach community members before the Mother Care Groups were trained on the family planning module. While gender was addressed via Safe Spaces, gender (as well as gender-based violence) had not been addressed with regards to Care Group activities at the time of the MTE, arguably the largest resource investment FSP-Enyanya is making under Purpose 2 Health & Nutrition, despite being planned in FSP￾Enyanya’s Detailed Implementation Plan as early as October 2018. The MTE team learned of several instances where Care Group neighbor women experienced conflict at the household level. During FGDs and IDIs, the MTE team learned of at least two cases (in two different health areas) where this led to gender-based violence from the husband of the neighbor women (women Care Group participants). The topics that appeared to create the conflict at the household included the desire to begin family planning and the request to purchase more diverse (typically more expensive) complementary foods. 3.3.3 P2 – WASH Water Infrastructure The MTE team found the general design of WASH infrastructure to be appropriate, as is the intent to train Water Management Committees to accompany these structures following activity close. CLTS CLTS is also appropriate for the main sanitation intervention, especially when using the GoDRC Village Assaini phrase "chacun ses moyens, chacun sa latrine28." This promotes construction of a latrine and working with communities to move past open defecation, without the pressure to move up the sanitation ladder immediately. Considering the number of vulnerable people in FSP-Enyanya’s intervention areas that struggle to afford a simple latrine, this emphasis to have any latrine is important. The MTE team does not encourage FSP-Enyanya to begin too many new interventions. However, the design adaptation for the WASH intervention to introduce WASH messaging in schools during FY4 is a good sign, as this will contribute to sustainability and full community adoption. Sanitation Marketing The MTE team found the Sanitation Marketing intervention to be inappropriate given the context of FSP-Enyanya’s intervention areas and where FSP-Enyanya is in terms of implementation. Sanitation Marketing is intended to accompany villages where CLTS has been introduced once they have reached the point of being ODF-certified, as this intervention creates supply for improved latrines that should match the demand created through CLTS. However in the zones where FSP-Enyanya is operating, purchasing power is incredibly low and of the 100+ latrines observed, very few were improved. It should also be noted that none of the villages participating in FSP-Enyanya were near ODF-certification at the time of the MTE; several of them had not even been triggered in CLTS by 28 Everyone builds a latrine that is within their means. Mid-Term Evaluation of the FSP-Enyanya DFSA in the DRC Findings 29 then. This means, if successful, they will not reach certification until halfway through year 4. If Sanitation Marketing is introduced this late in FSP-Enyanya’s implementation, or if communities are only ready to start thinking about moving up the sanitation ladder by then, it gives that intervention very little time. It may have made sense at the time of designing FSP-Enyanya before R&I and the resulting time implications were fully understood. 3.3.4 P3 – Governance Land access activities and DMCs appear to have contextually appropriate and rigorous designs based on key findings from program research (FSP-Enyanya proposal & Global STRESS report.) Processes, outputs, and outcomes for these activities were clear to interviewed community members. The evaluation team recognized that there is an opportunity to expand on these trainings to increase their coverage (to be discussed further in the conclusions and recommendations section.) Although tax law trainings seem to have successfully responded to community needs, the volition of grassroots organizations to do more presents an opportunity to leverage their motivation and train them on community mobilization, project management, and governance. In an attempt to ensure that Local Development Plans included all possible development activities for a particular village, these plans cover a vast array of initiatives and give a sense that there is no clear understanding/prioritization of the most essential actions; mentioned activities include general high budget items such as “improve water and sanitation structures,” “build a center for people living with handicaps,” and “improve school infrastructure.” Certainly, evaluated villages have numerous barriers to development that need to be addressed, but the exhaustive nature of LDPs risks diluting a clear and unified vision for local development. Furthermore, development plans are aggregated at the chiefdom level, and LDCs seem ill-equipped to present simple planning and advocacy documents to external actors in order to push for local development priorities, instead offering 30+ page documents that contain priorities of all neighboring villages. The MTE team was concerned that the all-inclusive design of LDPs hampers communities’ abilities to “pitch” their top development priorities. Finally, given the extensive scope of these documents, it is unclear that LDCs will be able to conduct additional needs assessments and prioritization on their own (this will be discussed more under findings for evaluation objective #5). After interviewing LDCs and participants in the CVA campaigns, the MTE team observed the link between Citizen Voice in Action processes and LDPs is not apparent, and many community members conflated the two initiatives without being able to clearly articulate how they are unique and/or might reinforce one another. In addition, the program chose to commence CVA processes at health centers, and at the time of the evaluation there were no alternative public services outside of health centers for which a CVA campaign had occurred. Although some CVA participants mentioned that the process could also be run on other services (e.g. markets, schools), no interviewee could describe what this process would look like for these services. This has resulted in CVA participants that have difficulties describing how CVA might be used for other public services. Food For Assets Finally, the MTE team recognized that using FFA as compensation for community contributions to public works seems to have dis-incentivized locally-led development initiatives. When questioned about the FFA intervention, the majority of FSP-Enyanya participants doubted its efficacy. Many cited IMPEL | Implementer-Led Evaluation and Learning 30 Findings that they were unclear of how FSP-Enyanya prioritized participants when nearly all households were perceived to be vulnerable to food insecurity. The use of food commodities as a modality for distributing aid was largely seen as inappropriate. Local markets were mentioned as having strong supply chains and locally produced food seemed to be preferred as improving purchasing power of FFA participants would allow farmers to make money off of their products.29 Cash for Work distributions were cited as preferable by nearly all interviewed community members (both male and female), who claimed that this would lead to less risk of conflict (as there were multiple stories of neighbors pressuring FFA participants to share their food commodities).30 3.3.5 M&E The design of the M&E system is impressive and supporting good implementation, particularly in light of the fact that they spent the first year to year and a half ‘moving around boxes on the TOC’ and waiting for approvals. FSP-Enyanya’s M&E plan was approved in November 2018, which is the second month of the third year of implementation. During the early days of implementation technical teams simply skirted the M&E team and created their own monitoring systems so they could get a sense of what was happening on the ground, but then the M&E team under the new leadership had to come in and re-socialize the theory of change and other ‘official’ tools with all team members. This also involved switching from paper to digital systems, including CommCare, which ended up being a lot easier to use so technical teams wanted to switch. They also created a well￾designed system in terms of feedback loops, like tracking lessons learned, holding these quarterly workshops by technical sector, and doing independent verification of their data quality. While the M&E team has been collecting sex-disaggregated data on all person indicators (as is required by USAID), they are not necessarily analyzing the outcome data to identify possible gender discrepancies. This is, however, challenging given that the activity participants are predominantly women and the sampling design (i.e. for household surveys) are not stratified. As such, the datasets are not designed to detect differences between women and men. 3.4 Evaluation Objective 4: Coordination, Collaboration, and Convergence In this section we present findings on evaluation objective 4: Coordination, collaboration, and convergence between FSP-Enyanya and government partners, other USAID programs as well as those internal to the consortium. Rather than present the findings by the three activity purposes, the findings here are presented by the different types of partners. 3.4.1 Within the FSP-Enyanya Consortium Mercy Corps is the prime implementer and World Vision is the main sub-partner on this award, with UEA, APC and Harvest Plus playing smaller roles. Coordination and collaboration could be improved between Mercy Corps and World Vision. Staff members from both organizations attributed tension 29 The MTE team did not have enough time in field to assess the availability of locally produced food. 30 Cash for Work was suggested by participants because of the cash distributions happening through other programs in the area – not because this option was proposed by the MTE team or is a viable alternative to FFA. Mid-Term Evaluation of the FSP-Enyanya DFSA in the DRC Findings 31 between the two IPs to two primary factors: personality differences between individuals in the senior management of both organizations and organizational differences between the two organizations. There were also observable differences between how implementation of the same activities were carried out in the World Vision versus Mercy Corps sites, for example, regarding policies on rabbit distribution and acceptable use of rabbit offspring. One senior FSP-Enyanya staff member also inadvertently drew attention to discrepancies in the approaches used in the two different areas by saying “on the Mercy Corps side...and on the World Vision side,” which suggests the on-the-ground strategy and designs are conceived of and implemented differently. The one example of where the Mercy Corps and World Vision staff appeared to be working together seamlessly as one team was on the M&E team. The MTE team noted limited collaboration between APC, the Congolese mediation and land access partner, and the two INGO partners of the consortium. A lack in field coordination and mutual understanding of interventions could present a missed opportunity for finding synergies between interventions; for example, mediation trainings could offer value to other stakeholders outside of the DMCs. Similarly, UEA indicated the World Vision leadership had limited knowledge of UEA’s activities in World Vision zones. Mercy Corps is responsible for communication across the consortium as the prime and this absence of communication raises the question of whether the current approach should be modified. The FSP-Enyanya consortium appears to have some challenges operating as ‘one team’ in practice. 3.4.2 With other USAID-funded activities Outside of the consortium, FSP-Enyanya is meant to collaborate with several USAID-funded activities.31 Related to agriculture, Tetratech’s Strengthening Value Chains Activity is meant to collaborate with FSP-Enyanya on market access. However, there is ambiguity between SVC, FSP￾Enyanya and Producer Organizations about roles and responsibilities; and possibly a gap between those responsibilities. According to a discussion with the SVC representative, FSP-Enyanya is responsible for transforming Farmer Field School farmers into market-oriented, business-trained POs. Once achieving that level of qualification, SVC is to link producers to markets. However, according to FSP-Enyanya DFSA documents, “FSP-Enyanya is in charge of the production part, while the marketing part is handled by the SVC-Feed the Future initiative.” There is no role assigned for closing the gap between skilled production and skilled marketing—each group assumes the other is doing it. PO members remain untrained on market engagement beyond local village sales, and have had, to date, no significant contact with SVC, nor guidance from FSP-Enyanya. There are layers of confusion within this program, including lack of information on the certification process for POs (which legitimizes them), and organization functions once certified; and SVC’s understanding of their own role compared to IP staff’s expectations of SVC is also ambiguous. With such ambiguity about roles and responsibilities (including a potential gap between them), benefits resulting from collaboration will remain limited. FSP-Enyanya is not the only DFSA headquartered in Bukavu: Food for the Hungry’s Tuendelee Pamoja II DFSA is also based there and was awarded for the same time period. While the two DFSAs are 31 Under P2, FSP-Enyanya is supposed to be collaborating with Integrated Health Project on strengthening health centers, but IHP had not started implementation at the time of MTE data collection. IMPEL | Implementer-Led Evaluation and Learning 32 Findings being implemented in different areas of South Kivu (FH TP II is also present in Tanganyika), these remain an extremely similar context. They are both partnering with both UEA and Harvest Plus, as well. Clearly they can learn from each other given the similar challenges they are facing in post￾conflict South Kivu and given the number of staff that re-locate from one IP to the other. At the time of the MTE no regular coordination meetings were scheduled between these two DFSAs. Another USAID-funded activity, Integrated Health Project, was supposed to be supporting health systems and to relieve some of the burden of FSP-Enyanya in health system strengthening. This activity has been met with numerous delays due to a change in the TIC and had not begun implementation at the time of MTE data collection. 3.4.3 With the Government of DRC FSP-Enyanya also needs to collaborate with government for legitimacy of interventions and to invest in sustainability after activity closure. The MTE team observed that collaboration with state structures in the field is not optimal related to agriculture; for example, SENASEM (the National Seed Service), and INERA (National Institute for Agronomic Studies and Research), are involved but the cost is fully borne by FSP-Enyanya instead of contributed by GoDRC. IPAPEL (Provincial Inspectorate for Agriculture, Fisheries and Livestock) involvement to date appears almost non-existent, according to key informant interviews. Limited ownership of community health and nutrition activities by Ministry of Health staff at both the health center and health zone level exists. These health staff viewed the community activities as something outside of the health system and the FSP-Enyanya staff do not systematically implicate these staff. In many cases, the health staff viewed FSP-Enyanya staff as being extractive: only asking for information that they can put in their reporting. The MTE team found some variation across health areas. In particular, there appeared to be much stronger support and implication by health staff in community efforts in some health areas, especially in the Kalehe Health Zone. At the health zone level, the Provincial Health Office (Bureau Centrale) shared that they have no formal relationship with the project, citing a lack of Protocole d’Accord between FSP-Enyanya and the Health Zone. The Bureau Centrale went on to describe how they are not able to do any supervision of activities and that they expect other USAID partners like IHP to provide them with commodities and other inputs, as well as funds to support their office operations, including payment of fuel and per diems which they want to manage themselves. The MTE team learned from FSP-Enyanya that they did prepare and share MoUs with all three health zones in 2018, but the Provincial Directorate of Health (DPS) never countersigned these and instead asked for a combined MoU which leaves control over material and financial resources at the DPS level. The DPS and Mercy Corps signed this MoU, but the DPS did not share it back with the health zones. Related to P2 WASH, the Provincial Health Offices participate in trainings and collaborated with FSP-Enyanya to identify intervention areas at the beginning of implementation. However KIIs with Service National d’Hydraulique Rurale (SNHR) revealed that they have extremely limited resources and although they have been reviewing WASH infrastructure that is constructed or rehabilitated by FSP-Enyanya per the DRC Water Law, they will be unable to provide any real support in terms of maintenance or repairs in the future. The MTE team also learned from FSP-Enyanya technical staff that collaborating with the government (above a local level) has been difficult for them in the post-electoral period because the government changes; they spent time establishing an MoU, for example, with one group of individuals and then with the Mid-Term Evaluation of the FSP-Enyanya DFSA in the DRC Findings 33 election all of these people changed and did not accept the MoU with their predecessors, forcing FSP-Enyanya to have to begin the same process anew. Specific to P3, it appears that the majority of collaborative opportunities exist with local governments and informal leaders and FSP-Enyanya is making significant efforts to engage them in interventions. Village and sub-village chiefs seemed especially involved in FSP-Enyanya implementation, serving on different committees and supporting FSP-Enyanya with community awareness-raising and coordination amongst sub-villages. Trainings on tax transparency were noted as an extremely positive example of linking civil society advocates to the public sector, with CSO leaders citing close relationships with administrative authorities as a positive outcome of these sessions. 3.4.4 With other INGOs While other INGOs and UN agencies are operating in most FSP-Enyanya intervention zones, the scope of their programming is often not aligned with FSP-Enyanya, and thus convergence seems to be minimal. One problematic instance, however, was identified due to insufficient collaboration with the World Food Programme, who is conducting food and/or cash distributions in parallel with FSP￾Enyanya’s FFA programming. It appears that distribution modalities and amounts are not coordinated, and there is overlap in FSP-Enyanya intervention zones. This parallel programming runs the risk of creating duplication in participants; field staff have expressed concerns that program participants have on occasion been absent from FSP-Enyanya trainings when per diems or transportation allowances provided by WFP have been more attractive. In addition, overlapping participants also risk generating conflict in the communities, by creating an impression that some locations or households are preferred over others by aid agencies. 3.5 Evaluation Objective 5: Sustainability In this section findings related to Objective 5, sustainability of outcomes produced by FSP-Enyanya to date, are presented. The MTE noted some early signs of sustainability in some of the intervention areas, despite the early stages of their implementation. However, the FSP-Enyanya activity would benefit from a more developed, and unified sustainability strategy planned across the different purposes or technical sectors. While some teams seem to have made headway in establishing relationships with local entities (including local non-governmental groups and various local government entities), these efforts do not appear to be part of a broader, FSP-Enyanya -wide strategy. World Vision staff implementing FFA explained, for example, how they have worked with the local communities to create a memorandum of understanding to articulate how road maintenance will continue (beyond the life of the award). However, this effort was not part of a broader sustainability strategy and FSP-Enyanya staff were not able to articulate what potential impacts might be on women versus men, for example, who would actually end up carrying out the road maintenance in the coming years (i.e. under this MOU that was signed by male community leaders). 3.5.1 P1 – Agriculture In the case of P1, for most interventions it is too early to determine whether changes are sustainable or not due to the short implementation period of FSP-Enyanya. However the sustainability of Village IMPEL | Implementer-Led Evaluation and Learning 34 Findings Savings and Loan Associations as a means to save money and provide small loans under FSP-Enyanya appears to be established. As indicated under the findings for evaluation objectives 1-3, VSLAs are so popular amongst participants and their neighbors that in response to additional demand, one VSLA Coordinator independently created 6 additional VSLAs (as a fee for service income generating activity). 3.5.2 P2 – Health & Nutrition In an example from P2, FSP-Enyanya has emphasized technical capacity for activities such as Safe Spaces. The methodology for training mentors appears sound and is being implemented by many of the Safe Space mentors the team met with. In addition, the activity has put in place a structure which focuses on continued capacity improvement via supervision of Mother Leaders, although MLs often need additional coaching and may not always meet the minimum criteria of being able to read and write. In addition, FSP-Enyanya has emphasized a volunteer model for Care Groups, Safe Spaces and Positive Peer Couples which may allow for community members to continue seeking counsel of these individuals after activity close. However, stronger linkages with existing health and community structures are paramount for the future success of the community-based activities. 3.5.3 P2 – WASH As with P1, most WASH interventions are also too early in their implementation to predict their potential for long-term sustainability. During MTE data collection, water infrastructure was still in the process of being constructed in FSP-Enyanya zones. However two threats to their sustainability had already manifested: the lack of government support and the weakness of the water community management committees. Generally speaking, the limited ownership of activities by government entities represents a threat to the long-term success of FSP-Enyanya’s interventions. However, specific to water infrastructure, government officials made it clear that they will never have the funds to assist if a water point requires a large repair and that communities will be best placed to identify another INGO should they require help. Second, while water management committees have been formed, they have limited understanding of their responsibilities and how to convince community members to contribute financially for a water source that was given to them for free. Regarding the sustainability of the Community-Led Total Sanitation intervention: while some positive behavior changes have been observed by natural leaders and WASH promoters thus far, this is typical in the months following the “triggering” of a village (which was the case for most of the villages in the MTE sample) but the real factors contributing to sustainability will manifest once a village has been certified as Open Defecation Free if latrine usage and replacement continues. For example, CLTS may be sustained if participants see the value in spending time and money to replace their latrines. This may be because FSP-Enyanya has assisted participants to identify an efficient method to replace latrines or has over enough time socialized the expectation across the community that they should replace their latrine once it is full. A successful CLTS program will have identified the barriers to making that easy during implementation so that whomever is responsible for ensuring CLTS continues after FSP-Enyanya implementation (NL, WASH promoters, health professionals or local leaders) is not saddled with the burden of identifying both problems and solutions.. Mid-Term Evaluation of the FSP-Enyanya DFSA in the DRC Findings 35 3.5.4 P3 – Governance Related to governance, early signs of sustainability were most evident in CVA and transparent tax activities. Infirmiers titulaires (ITs) interviewed by the MTE Team expressed their desire to continue to improve their services according to community feedback, and two ITs in particular considered that they were not dependent upon FSP-Enyanya to improve their services. Trainings on taxes have led to multiple advocacy successes amongst grassroots organizations, who expressed that they feel confident that they can continue to serve a “watchdog” function in their communities after activity closure. Of all FSP-Enyanya-promoted services, the gender training-enhanced LDCs show the most positive signs of serving an immediate benefit to community members beyond the life of the program. LDCs are seen as an effective way of coordinating advocacy and village-level initiatives, and community members are acutely aware of their important role in representing civil society. However, LDCs were not able to adequately explain the process for post-activity LDP elaboration, and not a single LDC demonstrated concrete plans for activity-independent sustainability. Income generating activities were often cited as an appropriate exit strategy for the intervention, but LDCs could not provide any evidence of reflection on appropriate business models, nor how they would effectively manage these activities alongside their existing responsibilities in the committee. This lack of planning, combined with warnings from APC on committee-managed income generating activities, causes the MTE Team to question whether side businesses are the appropriate sustainability model for this important service and calls again for the need for FSP-Enyanya to establish a clear exit and sustainability strategy. While members of DMCs expressed an interest in continuing to facilitate mediation processes after activity closure, they did not have a clear plan for doing so. The idea of imposing tariff structures proved problematic; DMC members were strongly against fee-for-service models as this would limit their ability to provide services to the ultra-poor, and pay-what-you-want models would likely introduce bias and potentially corruption to the mediation service. Furthermore, community donation strategies are not likely to be effective, as Catholic and Protestant churches act as market competitors, offering alternative mediation services (exclusively for members of their congregation), funded with donations to the church. When discussing exit strategies with APC team members, one cited prior experience of introducing income generation activities to DMCs as a limited success, explaining that income generation was modest or ineffective and also created conflicts between DMC members due complex intra-group power dynamics as well as poor and unclear management practices. Mediation in particular was flagged as an outcome that must be focused on for sustainability; numerous community members and mediators described fears that conflict would increase in their community if the DMCs were to close down, and some mediators expressed that their neighbors were pressuring them to keep these services open after activity closure. FSP-Enyanya technical staff may recognize that with some interventions, targeted research will be necessary to identify an appropriate exit strategy. 3.5.5 Across Purposes While FSP-Enyanya has incorporated some measures of sustainability into the overall activity, many of the specific interventions require further reflection by FSP-Enyanya staff with regards to IMPEL | Implementer-Led Evaluation and Learning 36 Findings sustainable impact. The FANTA Sustainability and Exit Strategies32 paper highlights the following elements that should be addressed to increase the likelihood that activities will be sustained beyond the life of the project: sustained motivation, linkages to government organizations and other entities, and technical and managerial capacity. In parallel to the need to establish an exit and sustainability strategy for all of FSP-Enyanya’s interventions, there are a few issues threatening sustainability of positive outcomes. The first of these is the question of long-term motivation of community volunteers. FSP-Enyanya is relying on volunteer Safe Space mentors, Mother Leaders, Father Leaders, Model Couples, Natural Leaders, Farmer Leaders, WASH Promoters, etc. to execute interventions, which on the one hand is an investment in the community as FSP-Enyanya trains these individuals. However given the context and history of South Kivu, the presence of emergency programs that are currently distributing aid (cash and goods) in the same area and the many other pressures on these individuals, FSP-Enyanya has to consider seriously how to maintain the commitment of these volunteers post-implementation or how to design an exit that does not depend on their support. 3.6 Evaluation Objective 6: Refine and Implement This section presents findings on how the application of Refine and Implement (R&I), an approach unique to FFP that was developed in 2014, has impacted FSP-Enyanya. It is being piloted in Liberia and DRC with the DFSAs that were awarded in these countries in 2016. According to the final report on the DRC R&I Pilot, “The R&I model builds in a one-year refinement period during which time the awardee (a) engages program participants, government, and local stakeholders through participatory consultations; (b) carries out prioritized formative and implementation research; and (c) deliberately builds staff and local capacity.”33 The idea behind this is that instead of launching immediately into implementation, because FFP activities are implemented in complex contexts and they target the most vulnerable populations, IPs should use the first year to adapt to the realities of the intervention areas. IPs are expected to modify their Theory of Change and their interventions based on the research that is produced during the first year. Learnings from this pilot can inform FFP of what has worked well thanks to its application to a DFSA and what aspects of it need to be further modified. The ET reviewed eight reports produced by FSP-Enyanya as part of the initial formative and implementation research, with field work conducted between March 2017 and May 2018. 32 https://www.fantaproject.org/research/exit-strategies-ffp 33 Refine and Implement Study DRC Pilot Final Report Mid-Term Evaluation of the FSP-Enyanya DFSA in the DRC Findings 37 Table 4: FSP-Enyanya R&I Studies Report Title Methodology Data Collection Dates DR Congo FSP-Enyanya Program Strategic Resilience Assessment STRESS Methodology (qualitative and quantitative) March to September 2017 Report on Erosion Control Practices and the Determinants of their Adoption Literature review, Observation, interviews, focus group discussions, and cartography. August 2017 (report date) South Kivu Food Security Project- Land security and conflicts in the health areas of Kalehe, Miti Murhesa and Katana in South Kivu STRESS methodology and Focus Group Discussions (qualitative and quantitative) August, 2017 Health, Nutrition and WASH Services Assessment Report Barrier analysis, document review, observation, focus group discussions, key informant interviews, and surveys. September. 2017 South Kivu Food Security Project- Market Systems Assessment STRESS methodology and Focus Group Discussions (qualitative and quantitative) May-18 South Kivu Food Security Project - Gender Analysis Report Qualitative- Focus Group Discussions May, 2018 South Kivu Food Security Project - Youth Analysis Report STRESS methodology and Focus Group Discussions (qualitative and quantitative) May, 2018 3.6.1 General Impressions Based on interviews with FSP-Enyanya technical and field staff, there was general confusion on the side of the IP as to how much research versus implementation should be happening in this first year. As a new concept, FFP was trying to figure out how best to introduce R&I at the same time as Mercy Corps was trying to implement it. As one FFP staff member put it, the initial introduction of R&I was like asking the IPs “to fly an airplane before the wings were fully attached.” The IPs obtained more clarity during the R&I workshop that was held in February 2017 in Kigali, Rwanda on what FFP’s expectations were at that stage although even then FFP technical staff were still working out some kinks. The MTE team notes the timing of this workshop as taking place at least four months after FSP￾Enyanya began implementation (Sep. 2016). For FSP-Enyanya in particular, seven individual R&I studies (called Strategic Resilience Assessment or STRESS Reports for Mercy Corps) were produced as well as one Global STRESS Report including findings from all of the R&I studies. FSP-Enyanya staff that were interviewed regarding R&I shared both positive and negative impressions. They generally felt that the R&I period lasted too long and that it ended up delaying implementation due to the associated approval process. At the community level, the activity was seen as “doing nothing” and “a year lost” during the first year. Community members pointed to FSP￾Enyanya’s extractive nature, conducting numerous assessments and organizing too many meetings without sharing any results or conducting any activities that would have tangible benefit to the community, as a source of frustration and doubt about the intentions of the consortium. Because the refinement year is also the first year of the activity and the IP did not have all staff hired, at least half of the formative research studies were not conducted by the staff members who are currently IMPEL | Implementer-Led Evaluation and Learning 38 Findings implementing FSP-Enyanya. They were largely conducted by consultants, of varying quality34 and in some cases by staff that have since left FSP-Enyanya. 35 Some interviewed staff suggested future iterations of R&I favor staff conducting the formative research, however this view is not held universally; some staff members are quick to point out that they are not researchers. 3.6.2 P1 – Agriculture Related to agriculture in particular, although staff shared with the MTE team that they felt that much of the resultant information generated through R&I was already known in the area, the ecosystem report increased FSP-Enyanya’s knowledge about drivers of soil infertility and informed the successful design of the Hill Approach. In addition, the land use practices report highlighted stresses around crop theft and plant diseases specific to South Kivu that lead to critical pivots in the TOC and the P1 and P3 operational strategies. 3.6.3 P2 – Health & Nutrition One of the key outputs of R&I should be a better-informed Theory of Change. In FSP-Enyanya’s case the FFP Washington M&E Team asked FSP-Enyanya’s M&E team to modify the TOC multiple times (see Annex 5.2 for the original TOC and the current TOC.) Several field staff shared with the MTE Team that they had only seen the TOC once during an M&E workshop several months before the MTE team arrived. Upon review of World Vision’s consolidated formative research for health and nutrition36 , it does not appear that this formative research effort addresses the most critical needs to truly refine the project’s TOC. The MTE team felt that the project is attempting to implement far too many activities and the TOC appears to be all encompassing and working at multiple levels (health system, community health system, community nutrition, etc.). The report does not seem to examine critical systemic and structural knowledge gaps under P2, beyond behaviors, which could help the project to better prioritize interventions or systems that could lead to lasting change. The formative research report focuses primarily on barriers to key behaviors and says little about the broader systemic, societal and structural challenges that vulnerable Congolese face with regards to optimal health nutrition and well-being. Moreover, the formative research report does not provide any recommendations whatsoever as to how the TOC should be modified. Sustainability factors were not assessed adequately during the formative research. FSP-Enyanya is not addressing all of these factors and these aspects do not appear to be incorporated into the activity’s Theory of Change, as requested by FFP during the R&I Inception Workshop in 2017. 3.6.4 P2 – WASH The World Vision formative research is the only formative research relevant to WASH that was completed during the R&I period. This formative research consists of an access and availability services assessment and a barrier analysis study on health, nutrition, water and sanitation (WASH) practices in the three FSP-Enyanya health zones. It should be noted that during the pre-design phase of FSP-Enyanya, Mercy Corps completed a WASH knowledge and skills assessment of the proposed 34 FSP-Enyanya staff explained the Gender Report was largely plagiarized and had to be redone. 35 The MTE team learned of several cases of technical staff moving from one DFSA IP to another. 36 World Vision Health, Nutrition and WASH Services Assessment Mid-Term Evaluation of the FSP-Enyanya DFSA in the DRC Findings 39 intervention areas. Unfortunately, there is no clear indication that the findings from that assessment differed from those of the study completed during R&I. Several of the WASH-focused recommendations made in the barrier analysis aspect of the World Vision Report have not been implemented by FSP-Enyanya. For example, given that covering water collection containers is not common, it was proposed to offer a subsidy to provide water containers in intervention areas. This was not observed during the MTE. In another example, the barrier analysis proposes to use the activity Citizen Voice for Action to motivate community members to maintain public latrines. This was not observed during the MTE. The disconnect between implementation and findings in the World Vision report may be due to the time it took for the World Vision report to be approved and finalized. The report was finalized in late March 2019 and implementation for WASH activities began in FY2: between October 2017 and September 2018. Overall the study produced during the R&I phase does not provide significant new information regarding WASH practices in FSP-Enyanya areas nor does it link concretely to the pre-design WASH assessment. 3.6.5 P3 – Governance For P3, there are multiple examples of how the R&I formative research year facilitated a deeper understanding of the context, tailoring intervention approaches to respond to unmet needs across the geography covered by the program. A primary example can be found in the Global STRESS Report, which enabled FSP-Enyanya staff to map out socio-economic and ecological systems the activity was aiming to influence, identify with community members the shocks and stresses that posed a threat to those systems, and ideate program initiatives to harness resilient capacities and respond to those threats at different levels in the results chain. Concretely, FSP-Enyanya’s technical team cited barriers to land access, localized conflict, and tax laws as three shocks and stresses that were intentionally integrated into the activity thanks to resilience design thinking that was facilitated through R&I. Throughout interviews with FSP-Enyanya staff, community stakeholders, and activity participants, it is clear that risk mitigation and basic resilience concepts are well understood, and these aspects of the activity are seen to be a direct response to this research. The FSP-Enyanya Gender Lead in particular attributes her ability to quickly understand the context in which FSP-Enyanya is operating and the specific challenges related to gender to be due to the existence of the Gender STRESS Report that was conducted during R&I. Overall, the MTE team has found that R&I produced helpful research for the goals of FSP-Enyanya for P3 and Gender, but not for P1 or P2. This may be for several reasons: agriculture, health, WASH are better studied in FSP￾Enyanya’s zones than governance and gender, for example. The MTE team also found that conducting R&I over the course of one year is too long and that FFP’s approval process caused delay for the IP. 3.7 Evaluation Objective 7: Cross-Cutting Themes This section presents the findings related to evaluation objective 7: determine the appropriateness and effectiveness of interventions focused on cross-cutting themes. In the case of FSP-Enyanya, governance, gender and youth are all addressed under purpose three. Governance is addressed across all four sub-purposes and gender and youth specifically under sub-purpose 4.1, women and youth participate in community planning and decision-making. The appropriateness and IMPEL | Implementer-Led Evaluation and Learning 40 Findings effectiveness of governance interventions was treated under evaluation objective 3 while gender and youth is addressed here. Findings related to accountability have also been included under this cross￾cutting theme section. At the time of the MTE, there did not appear to be a coherent or comprehensive gender and youth integration strategy as part of FSP-Enyanya. While the gender team is working with each of the other technical teams, gains in one aspect of the program may be undermining those in another because there is no overarching strategy for targeting participants (i.e. women and girls may be targeted by multiple interventions but the FSP-Enyanya staff leading each of those interventions may not be aware of which participants are already engaged with other time-consuming activities). Additionally, while the gender team appears to be collecting a lot of qualitative data from field visits and staff interviews, there does not appear to be a comprehensive data collection strategy to support better program design. Further, several FSP-Enyanya staff members expressed concern about the activity’s targeting approach with respect to women, i.e. female participants are being targeted by multiple arms of the program. It is unclear if there is a coherent strategy in place to ensure that FSP-Enyanya activities are not over-burdening female participants. There does not appear to be a system through which P1 staff, for example, can check to see if a potential P1 participant is already enrolled in other P2 or P3 activities. Regarding gender equality, it was observed during the MTE that FSP-Enyanya is aiming to achieve gender balance in the structure of committees. This is noted across P2 WASH and P3 interventions. As mentioned earlier, the quota system that FSP-Enyanya seems to be promoting does not necessarily mean that at this stage FSP-Enyanya has achieved equality in decision making due to a more even distribution of male and female committee members. However, it is possible that FSP￾Enyanya is helping break down pre-existing barriers and improving equality overall. While gender trainings and work on equitable decision-making has shown some output-level effects of equitable attendance in committee meetings, higher-level outcomes of increased influence and enhanced benefit from the program are not yet apparent. Women and youth are participating in virtually all committees created or revived by the program, but their actual decision-making power within these committees was difficult to ascertain. Another cross-cutting theme addressed by the evaluation was that of accountability. It appears in large part that community members are well-informed of program methodologies, selection criteria, and timelines, and can identify focal points within the community that can address their questions and concerns with program implementation. However, the vast majority of those interviewed were unable to answer foundational “why” questions around methodologies, selection criteria and timelines. When asked, “Why is the modality of activity X as such?” community members generally responded “because Mercy Corps/World Vision have chosen to do it that way.” This type of response points to a possible lack of understanding of the theory of change, a lack of genuine two-way dialogue on programmatic approaches, and/or a perceived inability of participants to influence changes to program implementation. On the other hand, community members were almost unanimously able to answer why “resilience-linked” interventions were implemented as such, citing appropriate shocks and stresses and discussing at length how mediation, transparent taxes, and land rights would enable them to become more food secure and productive. In addition, interviewees were able to adequately explain why equitable decision-making and gender equality were important Mid-Term Evaluation of the FSP-Enyanya DFSA in the DRC Findings 41 to development. These two positive trends indicate strongly that the community is capable of understanding complex concepts that feed into the theory of change, but an opportunity exists to redouble efforts in promoting community dialogue around FSP-Enyanya’s outcomes, and to reinforce learning and adaptive management practices as the program continues. IMPEL | Implementer-Led Evaluation and Learning 42 Conclusions and Recommendations 4. CONCLUSIONS AND RECOMMENDATIONS Objective 8 of the evaluation is to recommend adjustments to both program implementation and design and to explain how these changes would improve program outcomes and sustained impact. These recommendations are presented here by purpose, followed by recommendations that are applicable across sectors. 4.1 Evaluation Objective 1: Service Delivery 4.1.1 P1 – Agriculture Farmer Field Schools One of the key findings regarding the FFS is that transmission of knowledge from AEA and FFS committee members to farmers is uneven and the technologies are not being applied accurately or consistently. There appears a common thread of “cart before the horse”, where participants gain only introductory information about an agricultural technique then attempt to perform it immediately. The enthusiasm is commendable and should be encouraged, however when technologies are adopted prematurely without full comprehension, they are more likely to fail. A central reason for this may be the limited implementation time. Incomplete training leads to unsustainable outcomes. In this case, farmers may attempt the technology, see that it isn’t working, then discard it permanently. The enthusiasm of farmers to learn about and adopt new practices is a commendable success of FSP-Enyanya and this enthusiasm should continue to be encouraged. However, at times this led participants to adopt and attempt to practice new methods without full comprehension. This may cause the new technologies to fail, and farmers may then see it as unsuccessful and discard it permanently. Work to ensure the participants are fully trained in bio￾pesticide use will help ensure efficacy. This applies to FFS in general as well as UEA’s hill approach. The hill approach requires a consistent and complete combination of practices to reduce soil erosion and improve soil quality. This makes it particularly vulnerable to even small deficiencies in consistency and/or intervention completion, which was observed in some instances. FFS may benefit from adopting more of an "experiential learning" methodology as the core learning process. Using this methodology the FFS would try all new crops, inputs, or techniques in the designated learning plot and also in a small scale in participants’ plots to test before making a decision about adoption and scale up. A related issue noticed by the ET is that some Agricultural Extension Agents are responsible for monitoring more lead farmers than is realistic, given the distance between them. Recommendation 1: FSP-Enyanya may need to revisit the workload of AEAs and either reduce the number of (J)FFS they are supporting or see if it’s possible to do something else to support them. It is crucial in FY4 that the (J)FFS solidify their skills and the POs are well supported if we want them to continue practicing what they’ve been taught after FSP￾Enyanya closes. Mid-Term Evaluation of the FSP-Enyanya DFSA in the DRC Conclusions and Recommendations 43 Recommendation 2: Define and share the entire learning and graduation process with the (Junior) Farmer Field School (FFS) participants and provide them with multiple pathways. At the beginning, all FFS participants should learn the FFS process, their time commitment, the curriculum, and potential pathways after completion so that they can make an informed decision. Producer Organizations The transition from (J)FFS to PO could stand to be improved. In general, the producer activities (creating/strengthening Producer Organizations, post-harvest handling/storage, linkages to inputs, financial or technical services and market linkages) are either weak, behind schedule or non-existent. The MTE team recommends that FSP-Enyanya makes a concerted effort to improve their intervention related to commercialization of farmers. Specifically, the MTE team found the PO intervention to be generally weak and unlikely that the POs will survive after activity close. As one step to strengthen the POs, the MTE team recommends loosening restrictions on membership to POs. Currently, FSP-Enyanya only allows farmers who have undergone a FSP-Enyanya Farmer Field School to become members of a PO. This seems overly rigid and does not take into account that there are producers who are already proficient enough to join. The MTE team also recommends consolidating POs into fewer groups. There is also a high level of confusion regarding SVC’s role in FSP-Enyanya which needs to be clarified in order to ensure value chain linkages with POs. SVC appears to be seeking higher-level market participants instead of focusing on empowering small holders, and this is contrary to the DFSA (and FFP) objectives of helping the most vulnerable. If agreement on SVC’s role cannot be agreed upon and solidified quickly, FSP-Enyanya should consider eliminating SVC’s role. Recommendation 3: Map out the process to becoming a formal producer organization (PO), conduct economic analysis, define functions of a PO, and share these with participants from the beginning of the FFS so participants can make an informed decision on the path they would like to follow. POs will then have greater mutual focus on business and will be clear on the path to attain success and long-term sustainability. Permagarden The permagarden intervention is increasing the amount of vegetables being grown at home for some participants, but it is not teaching permagarden techniques. Ensuring that participants apply true permagarden techniques will enable them to reap the benefits of this intervention, namely a reduction in necessary water inputs. Permagarden participants indicated that their workload is currently higher since beginning the intervention. While that is expected because most FSP-Enyanya participants are involved in multiple interventions plus the inaccurate application of permagarden techniques not actually abating the quantity of water necessary to maintain their permagardens, this burden will taper as participants master techniques and find a better distribution of labor across their household. The ET finds it understandable that FSP-Enyanya participants are involved in numerous interventions, as FSP-Enyanya is presenting them with opportunities that they otherwise will not have. Bio-pesticide application has also been confused as it is currently taught and is contributing to crop loss. IMPEL | Implementer-Led Evaluation and Learning 44 Conclusions and Recommendations Recommendation 4: Revisit the knowledge of staff, and training curriculum of permagarden and integrated pest management for permagardens; provide refresher training to staff about the content, and facilitation of permagarden training; and monitor the quality of training for permagarden development and implementation. Small Livestock (Rabbits) The MTE team observed that guidelines for rabbit offspring are unclear and inconsistent among participants. Comprehension by rabbit recipients of how to raise, manage, and breed the animals is limited and varied. Increased education prior to receiving animals as well as frequent monitoring visits by FSP-Enyanya staff and Village Livestock Agents during the first few months after reception should improve this intervention. Landless households who want to develop an income stream from rabbit raising need more guidance on rabbit breeding, rabbit husbandry, and disease maintenance. These households should learn the nutritional benefits of rabbit meat and economics of rabbit production and marketing. FSP-Enyanya will also have to address the issue of limited medicine being available in the areas where participants have received rabbits, or prioritize distribution of livestock in areas where that is less of a problem. Recommendation 5: Livestock recipients need increased information on relevant husbandry practices and more frequent monitoring for recent recipients to ensure comprehension of rabbit management. 4.1.2 P2 – Health & Nutrition Care Groups The MTE observed suboptimal quality of social and behavior change (SBC) session implementation. Several instances were observed by the MTE team where Mother Leaders had challenges in providing accurate information to neighbor women. The MTE team recommends evenly distributing P2 technical field staff across the three Health Zones to ensure higher technical quality and evenness of implementation. Facilitating field visits and/or consultations between technical field staff and volunteer staff (Positive Peer Couples, Mother Leaders, Mere de Mere, Mentors, etc.) to document and disseminate shared learning and improve overall quality of implementation. Bringing in regional or Headquarters technical staff to perform a thorough review of current capacity of technical staff to support Mother Leaders and other community volunteers compared against a standard set of basic skills37 needed to implement SBC activities, beyond provision of messages, will also improve P2 implementation. FSP-Enyanya could also improve frequency and quality of training for Mother Leaders on how to identify malnourished children and children at risk of malnutrition before the child becomes a case of severe complicated malnutrition (Kwashiorkor.) 37 The FFP funded TOPS FSN Network developed a self-assessment toolkit which could be utilized for this recommendation. The toolkit which includes materials available in French can be found here: https://www.fsnnetwork.org/core-competency￾series. Mid-Term Evaluation of the FSP-Enyanya DFSA in the DRC Conclusions and Recommendations 45 Recommendation 6: FSP-Enyanya should revise its CG strategy to substantially improve the quality of service delivery to increase adoption of activity promoted practices and behaviors. Recommendation 7: FSP-Enyanya should include teaching MLs to identify moderately as well as severely malnourished children, to reduce the number of cases that reach that stage. Health Center Support The MTE found that beyond training, the current intervention is not adequately strengthening capacity of health centers. Given that the USAID-funded activity IHP will support health centers by strengthening health systems across FSP-Enyanya’s intervention areas, beyond what FSP-Enyanya has been doing thus far through training of health staff and providing materials and equipment, FSP￾Enyanya should seek to establish an agreement between the two activities to allow IHP to take over capacity strengthening in RMNCH and Family Planning. FSP-Enyanya could potentially continue to support capacity strengthening in nutrition at health facility level. In addition, IHP should conduct an analysis of the effective number of RECO staff that should receive training and other support from the project. FSP-Enyanya had limited these numbers but it would be important given IHP’s larger role in health systems strengthening for them to establish a relevant target. Recommendation 8: Establish agreement for USAID’s Integrated Health Project to take over capacity strengthening in Reproductive, Maternal, Newborn and Child Health and Family Planning. 4.1.3 P2 – WASH Water Management Committees Despite the MTE team’s observation that water committees have been trained in most cases, they found that these committees still require capacity building and support from FSP-Enyanya to solidify their role in the sustainability of FSP-Enyanya constructed or rehabilitated water points. Support could take the form of more frequent meetings between program staff and water committee members or through trainings. The specific areas of weakness that were identified during the evaluation are absence of any legal framework, limited knowledge of Congolese water law and limited understanding of how to capture funds for maintenance. At a minimum, assistance in establishing a legal framework and guidance on how to work with communities on how to capture funds for future maintenance should be prioritized. To date interviewed WMCs have yet to establish regular community contributions, which means they have no money saved for use in the case of even a minor breakdown. Recommendation 9: Increase training and monitoring of water management committees to ensure sustainability of the water points that are constructed or rehabilitated through FSP-Enyanya. IMPEL | Implementer-Led Evaluation and Learning 46 Conclusions and Recommendations 4.1.4 M&E The MTE team learned that field staff lack a strong understanding of qualitative methods. This may impact their ability to conduct qualitative data collection. It is also important to note that while there are some USAID resources on qualitative methods and Collaborating, Learning and Adapting (CLA) available, current FFP policy guidance (i.e. the 2016 FFP Policy Guidance for DFSAs) does not emphasize the use of qualitative methods to support performance monitoring or adaptive management, nor does the FFP policy guidance identify best practices, methods, or approaches that can or should be employed. As such, implementing partners are incentivized to focus most of their attention, budgets, and staff training on quantitative methods, not qualitative however these methods often present staff with different types of findings and a deeper understanding of issues the activity is trying to implement. Recommendation 10: Staff will benefit by increased training on qualitative methods. 4.2 Evaluation Objective 2: Evidence of Changes 4.2.1 P3 – Governance Tax Training Members of civil society organizations (CSOs) and local authorities are being trained on official and formal tax codes, which enables them to identify locally enforced taxes that are illegal. This is one of the practices that was identified as a problem specific to South Kivu. Having used the training as a networking opportunity for CSO leaders and local authorities, the CSO leaders are then able to work directly with local leaders to reinforce accountability and limit or eliminate illegal taxes that are identified throughout the community. The MTE team learned of several cases where local leaders and community members were able to eliminate illegal taxes after participating in these trainings. This kind of knowledge is important within the community and improves trust between community members and officials. Recommendation 11: Scale up P3 activity training civil society organizations on illegal/legal taxes. Inclusive decision-making Despite the work that FSP-Enyanya is doing on inclusive decision making and treating all people fairly, marginalized groups are not yet fully integrated. Considering that FSP-Enyanya has identified tension between pygmies and majority groups as an important stress in the global STRESS report, this seems to be a missed opportunity to promote empathy, peaceful coexistence and co-management of community resources between conflicting groups. The FSP-Enyanya technical team may need to hold bi-annual trainings to continue to break down staff prejudices and help promote these behaviors through the activity. The MTE team recognizes that this kind of shift in mentality will take some time. While democratic elections are likely appropriate for certain committees and volunteer positions within the program that require majority buy-in and representation, a strategic review of the Mid-Term Evaluation of the FSP-Enyanya DFSA in the DRC Conclusions and Recommendations 47 recruitment strategy for each position might unveil opportunities to improve inclusion of marginalized groups, equitable participation, and overall performance. Recommendation 12: FSP-Enyanya staff needs to keep working with participants on equality (pygmy & disabilities) as well as with women. Food for Assets FFA was found to not be appropriate for promoting participation in community works. This finding highlights an area where FFA may have an opportunity to improve the impact of projects by promoting increases in self-reliance and community-led action in areas that will need just that to lift themselves out of poverty. As concerns food security of vulnerable FFA participants, it is unlikely that 22 days of food rations per vulnerable household will result in long-term gains in food security. The evaluation team found that FFA activities (e.g. tree nurseries, road rehabilitation) had divergent quality in terms of implementation, engagement with communities, and implications for women’s time. FSP-Enyanya should examine the extent to which FFA activities may be inadvertently exacerbating “aid dependency,” which was identified as a common problem/challenge in the context of the program, and identify possible alternatives for the intervention design and/or process to transition to more community-based decision-making models, if possible. Any proposed revisions to the FFA aspect of the program should be redesigned in close collaboration with FFP (given limitations on how FFA funding may be used). Recommendation 13: Re-examine the Food for Assets intervention and identify potentially new or alternative community-led project design processes. 4.2.2 Across purposes Jealousy was an issue identified across interventions by the MTE team. Although all of the interventions rely on volunteers who are not receiving any financial incentive from FSP-Enyanya (beyond transport to trainings), they have been treated with jealousy from non-participants and to a certain extent even amongst participants. This indicates the difficult context in which FSP-Enyanya is operating and the need for FSP-Enyanya staff to be extremely careful in participant selection and how participants are perceived by the rest of the community. The issue of jealousy either between participants and volunteers in leadership positions, or participants and non-participants is a sensitive and important issue for FSP-Enyanya to deal with. The context of South Kivu is such that jealousy is a natural and expected reaction to any kind of support that people receive. People are incredibly poor and have extremely limited options to survive. It is unlikely that this problem will ever go away completely, but FSP-Enyanya should do its best to navigate it by being clear on criteria for participation in activities. Recommendation 14: FSP-Enyanya staff should be continuously trained on the need to communicate clearly with participants, as well as non-participants, about the work that FSP-Enyanya is doing, the criteria for participating and with community leaders on how to appear fair. IMPEL | Implementer-Led Evaluation and Learning 48 Conclusions and Recommendations 4.3 Evaluation Objective 3: Activity Design 4.3.1 P1 – Agriculture VSLA The VSLA intervention is working as a support to participants to be able to save money as a group and later access it when they need to make small loans. However, the size of the loans remain small and the intervention has not managed to connect participants to microfinance institutions as they are generally absent from FSP-Enyanya’s intervention area. A more appropriate design would have been to introduce VSLAs within the intervention area and conduct formative research on the options for microfinance during R&I. Recommendation 15: The MTE team recommends investigating if there is a way to assist participants in accessing microfinance and determining if there is interest in borrowing larger sums. Tree Nurseries Numerous problems were found with the current version of FSP-Enyanya’s tree nursery intervention. FSP-Enyanya’s model of payment to these nurseries for the seedlings they grow: FFA commodities plus payment for 20% of seedlings produced once all seedlings have been delivered. Receiving no periodic payments puts these businesses in a precarious position. FSP-Enyanya should terminate the current MOUs with the nurseries and take delivery of the seedlings produced thus far. Instead of FFA, a market-based pricing of seedlings should be paid. Free or subsidized inputs undermines sustainability and the MTE sees this as a major risk in the zone of influence where FSP-Enyanya operates. Recommendation 16: Tree Nurseries intervention needs to be modified to pay nurseries for market-price for seedlings and end the distribution of FFA. 4.3.2 P2 – Health & Nutrition Safe Spaces The MTE team found the Safe Spaces intervention to be effective. However, the intervention may not succeed in achieving population-level results without targeting a greater number of participants and/or exploring the possibility of adapting the programming to achieve sufficient coverage of the adolescent population. Currently, FSP-Enyanya reaches about 3500 adolescents via Safe Spaces (with plans to more than double this amount by the end of the activity), which is a small percentage of the adolescents in the target health areas. The limited coverage of the adolescent population will not lead to population-based impact. FSP-Enyanya should explore the possibility (and implications) of scaling up the Safe Space intervention and/or adapt the programming to support a cascade type of model (or other appropriate methods as determined by the Mercy Corps gender & youth team in collaboration with the FFP Gender and Youth team). Further, the MTE team recommends reviewing the content of the existing modules, especially the one on sexual reproduction, with a group of Mid-Term Evaluation of the FSP-Enyanya DFSA in the DRC Conclusions and Recommendations 49 mentors to verify that they are all age appropriate and adjust accordingly. The M&E team should later verify that these adjustments have been well accepted by mentors and both younger adolescents (10-14) and older (15-18) participants. Recommendation 17: Explore the possibility of scaling up the Safe Space intervention and review curriculum to ensure appropriateness for each age group involved. Care Groups The MTE learned of a couple of cases of GBV due to CG participation. According to the most recent Demographic and Health Survey in DRC in 2014, 31.1% of women 15–49 years reported having experienced physical violence against them in the past 12 months in South Kivu. In this region with a long history of gender-based violence, how a project interacts with and engages men and women in program activities in a manner that does not escalate or exacerbate gender-based violence is of paramount importance. It is FFP’s intention that the principles of ‘do no harm’ are met by activities implementing interventions on behalf of FFP. In this context, program design and implementation of social behavior change communication and nutrition education must take into consideration the pre￾existing power dynamics between men and women and the social hierarchy among older and younger women and older and younger men as these directly affect how each participant can interact with the program. This type of programming should focus on first discussing with men their traditional roles, and how this may impede improved nutrition practices, and what role they can play to support women. This requires a shift in the type of programming approach that is being implemented—from one that is didactic to one that is focused on dialogue as a means to enable men to identify and solve problems to improve nutrition practices without escalating conflict in the household in the form of increased gender-based violence. It is concerning that gender-based violence had not been addressed by FSP-Enyanya at the time of the MTE (even if it was planned.) This however does not mean FSP-Enyanya should seek to manage individual cases of GBV that are reported because of the strict need to maintain confidentiality and even the slightest risk of lack of confidentiality can increase the risk of gender-based violence for a woman. Instead, FSP-Enyanya should focus on using a community-based social mobilization approach to target community leaders, elders and work with men’s groups and women’s groups to change social norms and normative views on the use of violence against women as a means to manage household conflict and reduce gender￾based violence in the community as a whole. Evidence shows that where communities condemn GBV, it can be reduced. During the validation workshop, GBV was discussed at length. FSP-Enyanya should support participants to move beyond “gender training” and messaging to shift programming to promote dialogue within a group. FSP-Enyanya should use the Care Group activity as a safe space for women and men to separately dialogue about these issues, solve problems and identify solutions, and support normative shifts in how to use dialogue as a means to mitigate conflict rather than resulting in gender-based violence. Given the observation during the MTE that topics introduced in Care Groups had resulted in some reported instances of GBV once the female participant shared them with her husband, FSP-Enyanya must include a new approach to address GBV within Care Groups. FSP-Enyanya should consider using a rigorous assessment (e.g. with a third party or USAID/FFP gender advisor as an evaluator or consultant) to determine what's really happening on the ground, e.g. with the reported GBV incident/s. Results from this assessment will help inform the IMPEL | Implementer-Led Evaluation and Learning 50 Conclusions and Recommendations development of an informed and coherent strategy. Building on the Safe Spaces recommendation above, it may be beneficial for FSP-Enyanya to schedule a TDY with the FFP Gender and Youth Advisor to review the Care Group and Safe Space activities and work together to create a coherent gender and youth integration strategy. Recommendation 18: Use GBV-sensitive approaches within Care Groups to promote safe discussion of issues and to identify solutions that will mitigate conflict and prevent GBV. 4.3.3 P2 – WASH A late start to implementation was observed across FSP-Enyanya interventions, due to a variety of factors. This is particularly true for the WASH components of the activity and it is evidenced by the fact that at the time of the evaluation, there were still villages that had not been triggered according to CLTS. Triggering is a major step on the path to abandoning open defecation, but it is far from reaching the end of open defecation and even further from reaching certification that a village is truly open defecation free. Considering that the activity only has 1.5 years of implementation remaining, FSP-Enyanya’s technical team needs to be careful and realistic in what is achievable relating to WASH behavior change. Given the lack of demonstrated interest in the intervention areas to invest in sanitation, as well as the study demonstrating that the purchasing power of residents in FSP-Enyanya areas was too low for residents to invest in improved latrines, the technical team will need to decide if it is worth the investment of staff time and resources in pursuing Sanitation Marketing. Many of the intervention areas are former Villages Assainis, meaning they are former open defecation free villages through the GoDRC program, and have already experienced slippage from open defecation free (ODF) in the past. As such, it may be a better investment to focus on studying and addressing the root causes of ODF abandonment within these communities and tailoring CLTS support from Natural Leaders and WASH promoters before FSP-Enyanya closes. Recommendation 19: Abandon the Sanitation Marketing Activity and instead focus on making Community-Led Total Sanitation (CLTS) sustainable within FSP-Enyanya villages. 4.3.4 P3 – Governance Local Development Committees (LDC) are active, have created Local Development Plans (LDP), and have participated in FSP-Enyanya trainings. However, LDC members have not reached a point where they are able to convince external actors to prioritize LDP-defined activities. This may be in part due to the many activities included in LDPs. The MTE team suggests conducting a simplified training-of￾trainers as a participatory diagnostic exercise to identify the specific tools and skills necessary among current LDC members, and to synthesize the current LDPs into community-specific prioritized 2-page adaptations of existing LDPs. This will help LDC members to prioritize activities in LDPs, to identify different audiences for advocacy, and to tailor and pitch messaging to these external stakeholders. As an immediate hands-on exercise following this training, LDC members should create these 2-page versions of their existing LDPs and begin pitching to external stakeholders. Also during the participatory diagnostic exercise, FSP-Enyanya should brainstorm with LDC participants to identify ways to make LDCs more sustainable. FSP-Enyanya should ensure frequent follow-up with the LDCs during this phase to continue building their capacity in community mobilization. This is an extremely Mid-Term Evaluation of the FSP-Enyanya DFSA in the DRC Conclusions and Recommendations 51 difficult intervention to succeed, especially in the context where FSP-Enyanya is working, therefore it will be important for FSP-Enyanya to be as adaptive as possible related to this intervention. The FSP￾Enyanya staff member should be conducting on-going research and seeking mentorship from Mercy Corps and World Vision international leadership on how to succeed this type of program to inform these iterations that will be necessary to be successful. Recommendation 20: Support Local Development Committees to elaborate community￾specific Local Development Plans and identify appropriate audiences to pitch them. 4.4 Evaluation Objective 4: Coordination, Collaboration & Convergence Based on the findings presented above, coordination, collaboration and convergence should be improved between FSP-Enyanya and all partners. The MTE team recognizes the inherent challenges to working with the government in FSP-Enyanya’s intervention zones and encourages FSP-Enyanya to continue following the necessary bureaucratic processes for each intervention. Despite the challenging context it appears FSP-Enyanya has made some inroads on this. On the other hand it appears that internal communication between Mercy Corps and consortium partners could be improved and the involvement of especially the local partners could be increased. According to FSP-Enyanya’s design it is supposed to collaborate with IHP on health systems strengthening and with the Strengthening Value Chains on market linkages for producer organizations. Unfortunately there are layers of confusion in working with SVC, including lack of information on the certification process for POs (which legitimizes them), and organization functions once certified, as well as SVC’s understanding of their own role compared to IP staff’s expectations of SVC. With such ambiguity about roles and responsibilities (including a potential gap between them), benefits resulting from collaboration will remain limited. Recommendation 21: Determine if and how FSP-Enyanya & SVC should work together according to the DFSA design or if a modification will be necessary to ensure there is a functioning post-production component to P1. Recommendation 22: To improve ownership by health service providers of community health activities, FSP-Enyanya should establish (where non existing) or strengthen (where existing) a mechanism to forge a stronger link between community health and nutrition activities and the health facilities which will foster a sense of these activities as part of the health system. IMPEL | Implementer-Led Evaluation and Learning 52 Conclusions and Recommendations 4.5 Evaluation Objective 5: Sustainability 4.5.5 P2 – WASH The context of FSP-Enyanya’s intervention area in South Kivu is challenging in many ways. The absolute lack of government support in some sectors poses a problem for long-term sustainability and requires activities like FSP-Enyanya to either accept that their work cannot be sustained, or find alternatives that allow participants to find ways to operate in parallel to the government system. For example in the WASH sector, SNHR officials were clear with the MTE team that they will not financially assist with the repair of FSP-Enyanya-constructed water infrastructure. Should FFP be investing in infrastructure, in this context? This is an important question for FFP and for IPs. For the time the infrastructure is functioning, it is providing water to participants and ostensibly improving their lives through a reduction in the time it takes to collect water, improved quality and, hopefully, a reduction in waterborne illness. However it is unlikely with such a weak government that any support will be provided for infrastructure constructed by NGOs. The MTE team recommends in the case of FSP-Enyanya to reinforce the community water management committees as much as possible in terms of technical knowledge of how to identify and fix small breakdowns as well as ensuring community ownership of the water infrastructure. The best case scenario is that the water management committee will work with users to understand a regular contribution is necessary to support the long-term life of the infrastructure. Recommendation 23: Strengthen water management committees and continue to attempt to involve the SNHR in WASH interventions. 4.5.6 Across Purposes Although an exit and sustainability strategy existed at the time of the MTE, few staff or participants were aware of its content. FSP-Enyanya’s technical team should aim to communicate this existing strategy and make sure all staff, volunteers and participants have a shared understanding of its goals and content by the end of the second quarter of year four. Once a solid exit and sustainability strategy has been communicated by FSP-Enyanya, all sectors of FSP-Enyanya technical staff will have a clearer vision for what needs to be accomplished during the remaining period of implementation, and what should happen for the many groups created by FSP-Enyanya to function sustainably after activity close. One aspect to consider in particular is that of financial sustainability, as many of the groups created through FSP-Enyanya are not, as is, financially sustainable. How will Mercy Corps work to better integrate a business sensibility in the groups being created? Without some creative source of revenue, these groups are not likely to continue. If capacity building for FSP-Enyanya staff is necessary or deemed a wise investment, FFP may look for opportunities through IDEAL or other resources to make sure the FSP-Enyanya team is equipped with the skills necessary to benefit from the wealth of knowledge existing in current FFP partners to develop a solid strategy. Finally, the strategy will need to be detailed on an intervention level. Recommendation 24: Make and diffuse a clear exit and sustainability strategy considering each intervention. Mid-Term Evaluation of the FSP-Enyanya DFSA in the DRC Conclusions and Recommendations 53 4.6 Evaluation Objective 6: Refine and Implement While the findings section shared information on FSP-Enyanya’s experience of R&I per the evaluation objective, in this section conclusions about that experience and how FFP can improve R&I are presented. There were several clear examples from P1 and P3 where the completion of R&I studies enabled FSP-Enyanya to recognize a need to modify their programming. Due to R&I, they were able to do so. Overall, two promising aspects of R&I are important to highlight. First, the R&I approach is an opportunity for the IP to invest in better understanding the activity context through formative research. Second, it is an opportunity for the IP to adaptively manage the DFSA instead of feeling “stuck” in implementing whatever was initially designed in the proposal. However, there are some drawbacks to the R&I approach as well for these programs. Many of the IPs have already implemented DFSAs in these same areas and are familiar with the context. Their assumptions based on experiences in other programs or based on experience working in the same setting may cause them to think within the box. Adaptive management is a new concept that is complicated by the requirement with FFP that activity modifications are approved by Washington. Finally, a lot of blame for the late start of implementation was placed on R&I. Allowing some key interventions to begin during the first year would both reduce this lateness and would improve relations with residents in the zone of influence, as they will understand the activity is conducting research and implementing (as opposed to just coming for data then leaving.) Recommendation 25: R&I should focus on refinement of scope of activities to avoid duplicating similar DFSA projects which are a “mile wide and an inch deep.” Formative research should seek to address over-arching structural, systemic issues which can help to prioritize the interventions that could have the greatest impact. Recommendation 26: A minimum package of interventions should be implemented in the first year as crucial research is being conducted to mitigate future lateness and improve relations with residents in the intervention zone. 4.7 Evaluation Objective 7: Cross-cutting Themes The lack of a gender and youth integration strategy compounded by the targeting of many participants for multiple interventions and current lack of a system to track this enrollment may be unnecessarily burdening some participants. There does not appear to be a system through which P1 staff, for example, can check to see if a potential P1 participant is already enrolled in other P2 or P3 activities. The MTE team suggests reviewing in which interventions participants are enrolled and examining the potential time burden associated with female participation in numerous activities, especially for mothers of children under five years old. This review should be narrowly focused on the activity (and not a research study). FSP-Enyanya needs to use gender specific and gender sensitive approaches and articulate how/when those respective approaches should be used across the board. The goal is to ensure that participation in FSP-Enyanya is not causing negative impacts on women, such as spending many hours of the week fetching water for permagardens. IMPEL | Implementer-Led Evaluation and Learning 54 Conclusions and Recommendations Recommendation 27: FSP-Enyanya should undertake a review of the activities’ impact on women's time. 4.8 General Recommendations for FSP-Enyanya Mercy Corps’ FSP-Enyanya should review current distribution and capacity of program staff and ensure a more appropriate distribution of trained field staff and greater involvement of technical field staff across the three Health Zones. This will help ensure higher technical quality and parity in implementation. Differences in quality were observed both by implementing partner (IP) and by intervention area during the mid-term evaluation (MTE). FSP-Enyanya should ensure that all staff have equal access to training opportunities and promote a culture of continuous capacity building. FSP-Enyanya should require field visits and/or consultations between technical field staff and Bukavu-based staff to monitor, document and disseminate shared learning and improve overall quality of implementation. In particular, technical leads need to pay closer attention to training quality and knowledge transfer from idea to how concepts are practiced by participants: are participants receiving sufficient training to replicate what they have learned across sectors? This was not observed during the MTE. If necessary, FSP-Enyanya could bring in regional technical staff to perform a thorough review of current capacity of technical staff to support community volunteers compared against a standard set of basic skills needed to implement agricultural and social and behavior change (SBC) activities, beyond provision of messages. This could potentially lead to the project developing a capacity strengthening plan for staff to best support community based workers and volunteers, both during the remaining implementation period and after activity close. Recommendation 28: Harmonize implementation across the activity's intervention areas, in terms of number of staff and improved quality of service delivery. FSP-Enyanya volunteers (MLs, Father Leaders, NL, AEAs, Livestock Agents38s) are used as a vector of change under all purposes. However, it was observed by the evaluation team that volunteers would benefit from increased training or continued capacity building. There is a general lack of mentorship, accompaniment and supportive supervision. In several cases, it was learned that volunteers attempted to pass on a technology to participants and either do not monitor its application or do not teach it well enough for participants to be successful. Concepts being shared are complex and require more than instruction for participants to fully replicate independently; trainees should be pushed to apply lessons learned through practical exercises. FSP-Enyanya staff should verify that volunteers have fully understood a concept or technology before teaching it to participants. As noted in the recommendation above, the evaluation team also found evidence that staff, in addition to volunteers, would also benefit from additional capacity building strengthening, so FSP-Enyanya may be well served to create a cohesive staff/volunteer capacity building initiative/effort to address these two needs together. 38 Agents Villageois d’Elevage, Village livestock Agent in English. Mid-Term Evaluation of the FSP-Enyanya DFSA in the DRC Conclusions and Recommendations 55 Recommendation 29: Reinforce capacity strengthening and acquisition of new skills amongst FSP- Enyanya volunteers. FSP-Enyanya is operating in a post-conflict zone with still-active emergency programs that are currently conducting distributions in neighboring communities to FSP-Enyanya intervention areas. Participants are accustomed to receiving things for free from NGOs; these communities are still in a transition to living in a more stable development context. With this in mind, FSP-Enyanya must anticipate this reflex and identify ways to keep participants interested in the longer-term work of volunteering and investing in their communities. The ET acknowledges that ensuring continued motivation of volunteers is an ongoing challenge for development programs around the world, a matter discussed at length in the FANTA study mentioned under Objective 539. However, some suggestions on promoting this include making community volunteers into associations. Additionally, these options could be identified through focus groups with current volunteers and participants from across the purposes. Recommendation 30: Address the issue of motivation for current FSP-Enyanya volunteers. One of the MTE’s overall observations is that FSP-Enyanya is attempting to implement more interventions than is realistic given the time and funding available. Therefore as a first step the MTE team recommends reviewing the scope of interventions and determining what is essential to continue. As a second step, the MTE team recommends that FSP-Enyanya reduces its targets to have a more robust intervention. For example, they are currently targeting 72 POs. It is next to impossible to really do a good job building capacity of 72 POs and turn them into viable business entities in 2 years (or even 5 years). Recommendation 31: FSP-Enyanya should undertake a careful examination of planned activities to invest more in high-impact interventions and scale down those that are a less efficient or effective use of resources. 39 https://www.fantaproject.org/sites/default/files/resources/FFP-Sustainability-Exit-Strategies-Synthesis-Dec2015_0.pdf IMPEL | Implementer-Led Evaluation and Learning 56 Annexes 5. ANNEXES 5.1 FSP-Enyanya Organizational Chart Field Structure Organizational Chart 5.2 Theory of Change From the Mercy Corps FSP-Enyanya Original Proposal Narrative: The figure to the right illustrates the overall FSP design with Goal, Purposes, cross-cutting themes (outer layer) and approaches (in the center). This design is based on the Theory of Change presented in Annex 7. The hypothesis underlying the TOC is: If an adequate level of stability allows the FSP to be implemented and if participants are able to improve their livelihoods, generate sufficient income, promote gender equality and youth inclusiveness, adopt better health and nutrition practices, benefit from improved services from public and private sector providers, and increase their resilience to shocks and stresses, resulting in an environment more favorable to sustainable development, then participants will have improved access to nutritious food, better overall health and more economic opportunities, resulting in greater food security, improved nutritional status and enhanced well-being. As shown in Annex 7, the TOC addresses a number of systemic and contextual factors contributing to food insecurity in South Kivu. Examples include:  Conflict: Conflict creates uncertainty, hinders mobility and discourages livelihood investments. While the project cannot address major armed conflict, the TOC shows that we will work at the community level and across communities to prevent or mitigate conflict between groups or communities with diverging interests but common goals. We will address Mid-Term Evaluation of the FSP-Enyanya DFSA in the DRC Annexes 57 conflict by i) promoting dialogue and ii) supporting local conflict management mechanisms where they are functional and facilitating the creation of mechanisms where none currently exist.  Poor governance and lack of social accountability: A lack of good governance and accountability may be found on several levels, including: within a social or economic group such as a women’s group or farmers association, between the community and traditional authorities, or between the government and its citizens. As the TOC shows, the design of each of the three Purposes includes activities to improve services and to generate demand. We propose to address this systemic problem by helping participants increase their demand for services while working with service providers to improve supply and build better relations with clients.  Marginalization of women and to some extent, youth: The TOC shows to what degree women and youth are included in every pathway leading to the project Goal. Without their contribution, sustainable change is unlikely to occur, especially as youth represent the future of South Kivu and women play a critical role in agriculture, other livelihoods and family health.  Climate change: We have also included Outcomes for climate risk management in the TOC. These include using a climate-resilient agriculture approach, helping communities prepare contingency plans, and refurbishing or building new infrastructure to mitigate the effects of climate change whether it be an unforeseen weather event such as drought or flooding or simply the long-term effects of changing weather patterns. The TOC in Annex 7 also includes major assumptions that will affect whether the FSP achieves its Goal. Three of the assumptions are:  The planned USAID Activities for South Kivu will take place in the same geographic areas as the FSP and start within the first year of the FSP. These Activities are the FtF South Kivu Value Chain Activity, the IHP, the Integrated Governance Activity and Solutions for Peace and Recovery (SPR). Although the FSP will achieve results without these Activities, the overall impact will be significantly greater with the contribution of the interventions proposed in these initiatives, especially the market linkages (FtF), improvements in the health system (IHP), the promotion of good governance (IGA) and women’s empowerment (SPR).  The GDRC, traditional authorities and the private sector will recognize the benefits of collaborating with the project and target populations to effect transformative change.  There will be an adequate level of stability (political, social and economic) to allow the project to function at the planned level. Mid-Term Evaluation of the FSP-Enyanya Development Food Security Activity in the DRC Annexes 58 Latest Theory of Change graphics as of the MTE Purpose 1 Purpose 2 OP 3.2.1.1 FFA supports LDP implementation SP 1.2 Women, men, & youth actively participate in IGAs Agricultural productivity increased Natural resources protected & enhanced OP 1.1.1.1 Male, female, youth (J)FFS members trained in use of improved crop techniques on demonstration plots IO 1.1.1 Men, women & youth apply improved agricultural techniques Reduced agricultural losses Gender-, age- specific, environmentally sound FFS curricula developed FFS ToT for CARGs, private input suppliers, IPAPEL, AVAs AVAs train LFs LFs train men, women, youth under AVA supervision LFs supported to become PEPs Private input suppliers embed training in offers Communities trained in principles of TOPS/MC land resilience for application on communal lands FFS farmers receive bio￾fortified seeds for trial OC 1.1.4.1 Farmers can find and purchase improved agricultural inputs locally OP 1.1.4.1 Certified biofortified seed multipliers linked with agrodealers and community- based financers Demand for improved inputs increased, sustained Marketing campaigns for improved & biofortified seeds FFS adopters supported to join POs, promote female leadership P1 Households incomes increased SP 1.1 Agricultural production, especially of nutritious, lower-risk food crops, increased IO 2.2.2 Women, men, youth know key MCHN principles Women, men, youth practice gender equity in business UEA students selected to support Junior FFSs, CARGs Male/female youth trained in market, labor analysis Youth join YBGs YBGs helped to access land, provided subisidies/grants to test innovations; linked with mentors Enterprises' access to land negotiated with Mwamis IO 1.2.2 YBGs start and sustain IGAs Increased presence of PEPs Gender-, age-specific motivators, barriers to agricultural activities identified INERA support to research CARGs help FFSs access land for demonstration plots INERA contribution to seed selection, training Loan guarantees for seed multiplication POs with viable business plans provided business training; financial, technical support Scholarships for higher education to committed Junior LFs Youth have skills for future leadership in agriculture Women raise nutrient-rich permagarden, small livestock produce Youth access business incubators OP 1.2.2.1 YBGs trained in financial services, business set up, management PSPs invest in incubators; engage youth as sales agents Issue of illegal "taxes" & other barriers mitigated VC actors identify related barriers and issues & consider solutions Strategies for adapting VC services, products to needs of most vulnerable identified Private service providers adapt services to small- holder farmers' needs OP 1.1.1.2 Farmers trained in use of improved inputs & techniques by private input suppliers and PEPs Faster, safer transportation available Viable community-based financing mechanisms identified OP 1.1.2.2 Women, men, youth trained, supported in VSLA mgmt, other business skills OC 1.1.2.2 Men, Women, youth organize, effectively manage VSLAs IO 1.1.4 increased number of hectares cultivated with nutritious, low-risk, and profitable crops Affordable nutrient-rich food products available in homes, local markets year-round OP 1.1.3.2 Literacy, numeracy training for women & youth, including VSLAs members. Sharecropping arrangements are transparent, fair Affordable water treatment & sanitation supplies & technologies inputs available locally Market research with PSPs identifies ways to increase affordable sanitation supplies in local markets Market research on increasing sanitation supplies shared with YBGs and VSLAs Maintenance, environmental protection of FFA assets incorporated into community development plans Affordable VC services available locally Plan to improve regulatory environment developed & implemented with Govt, Prvt Sector, KVC, CARGs, IGA, CSOs Critera established for FFA activities community selection of assets for rehabilitation & workers FS, natural resources services mapped; action priorities identified Men, women, youth trained, supported to map FS resources, services Gender-sensitive YBG training curriculum developed OP 1.1.3.1 SBCC messages on GBV, household decision- making incorporated into VSLA training Theft reduction Women and youth are supported to establish nurseries Resilience Output SP IO Purpose External to project Outcome In logframe Long-term commercial arrangements between seed multipliers, agrodealers IO 1.1.2 Women, men, youth have savings and access credit OP 1.2.1.1 POs, women's groups trained to develop gender- sensitive, envr. friendly business plans, register as cooperatives Gender mainstreaming Vulnerable women receive permagarden and/or livestock￾inputs Communities select the most vulnerable FFA workers See P3 for more FFA theory; FFA represetned as P1 staff will be managing these activities Male/female youth identify potential revenue opportunities Community assets support HH & community resilience POs negotiate purchases & sales at favorable rates and reduce taxation costs through collective sales POs discuss topics of gender, nutrition OP 1.1.2.1 Community members assisted to access financing Local seed multipliers produce biofortified seed OC 1.1.4.2 Seed multipliers sell biofortified seed in local markets Women, men, youth invest in IGAs IO 3.1.1 Women, youth have confidence in their ability to participate in community decision-making Market research on increasing sanitation supplies shared with women, youth Women, men, youth trained in food processing techniques, receive equipment subsidies Households have good access to markets IO 2.2.1 Increased consumption of nutrient-rich foods by women, children OC 2.3.4 HHs have latrines, handwashing stations, & an adequate supply of potable water, SP 3.2 Communities have the info, tools, processes to monitor, improve the socio- economic environment IO 3.2.1 Community actors implement resilience￾focused LDPs and practice project monitoring IO 2.3.1 Community members seek timely care for themselves, their children YBG concept promoted among male & female youth Business incubator concept promoted among PSPs IO 3.2.1 Community actors implement LDPs and practice projects monitoring Gender & Nutrition SBCC messages incorporated into PO, VSLA trainings Discussions on inclusive gender themes with VSLA members, spouses IO 3.1.2 Increased community acceptance of self-determination and leadership by women, youth Gender SBCC messages incorporated into YBG trainings IO 1.1.3 Women and men in union make household financial decisions jointly OP 3.1.2.1 Women, men, youth have appropriate views of gender equity, positive masculinity, SGBV IO 1.2.1 VC participation increased, esp. among women, youth POs given conditional financial support to fill VC gaps POs & women's groups register as cooperatives and operate envr friendly businesses, including VC businesses OC 2.3.4 Men & women construct HH latrines & handwashing stations and purchase essential WASH supplies Affordable water treatment, sanitation supplies, tech. inputs available locally Theft reduction P3 The socio-economic environment is stable and inclusive IO 2.3.4 Community members observe WASH best practices OC 2.2.1.1 CGs provide demonstrations on conservation, prep of nutrient-rich foods P2 Improved nutritional status in SP 2.1 targeted communities Improved HTSP Men, women, youth know about HTSP, FP services, methods, benefits Women, men, youth use FP services at community level IO 2.1.2 HC, HP staff are competent, confident, equipped to deliver inclusive FP services OP 2.1.2.1 RECOs &HC, HP staff trained to deliver FP services OP 2.1.1.1 Men, women, youth participate in HTSP trainings OP 2.3.1.1 and OP 2.3.2.1 SBCC messages about MCHN and community health practices, services and benefits broadcast via effective media Assessment of RECOs, HC, HP staff competency, confidence PPCs nominated by CGs in every community CGs requested to identify famlies with HTSP and criteria explained PPCs trained to counsel couples about resource sharing, decision-making, HTSP, GBV, make FP referrals Men, women, youth counseled in FP, sharing resources, decision making SP 2.2 Women, men, children eat optimal diets IO 2.1.1 Women, men, youth adopt healthy FP practices OC 2.2.1.2 LMs pass monthly CG messages to women, families in groups, review practices, counsel/refer Community water committees understand how to protect water points Community promoters trained, supported to share WASH messages, implement CLTS OC 2.3.4 HHs have latrines, handwashing stations, & adequate potable water IO 1.2.2 YBGs start and sustain IGAs CLTS resources, training from OXFAM, UNICEF, MOH, Tear Fund IO 2.3.2 Community members know about MCHN services, when needed and benefits HC, HP staff trained to provide quality MCHN, iCCM, GBV services CBOs advocate for nutrition services MCHN resources, services provided by PRONANUT, IHP, UNICEF OP 2.3.5.1 CODESAs supported to improve community nutrition activities RECOS trained to provide iCCM services IO 2.3.5 HC, HP staff provide high quality MCHN, iCCM services MUAC tapes, age-height ratio silhouettes of children distributed to CGs New, existing WUCs (esp women) established, trained, supported as appropriate WASH community action plans implemented by CVCs Market research with PSPs identifies ways to increase affordable sanitation supplies in local markets PSPs market affordable sanitation products in local markets OP 2.2.2.1 FSP staff conduct monthly training and mentorship for CG and receive feedback from LMs SP 2.3 Community members, esp. mothers and CU2s, are healthy SP 2.4 Children whose growth is faltering identified, referred for care and counseling GBV services mapped, gaps identified OP 2.3.6.1 RECOS trained to identify, refer MHCN, iCCM, GBV cases Information about GBV gaps shared with local committees, authorities OP 2.1.1.2 Safe Space groups for adolescents meet regularly Formative research to define special needs, capacities of female youth Safe space mentors identified & trained to support Safe Space groups& make FP referrals Women, youth have increased self-esteem, confidence Community members access preventive, curative health care at health facilities IO 2.3.3 HH, community environments are clean IO 2.2.1 Increased consumption of nutrient-rich foods by women, children Affordable water treatment, sanitation supplies, tech. inputs available locally Affordable nutrient-rich food products available in homes, local markets year round IO 2.2.2 Women, men, youth know key MCHN principles IO 2.3.1 Women, men, youth seek timely care for themselves and their children Couples are trained to make joint decisions about family health care Men, women, youth recognize the nutritional value of biofortified products CLTS implemented in all villages ICRC, others train HC staff to treat, refer GBV cases IO 1.2.2 YBGs start and sustain IGAs FP commodities available in local health facilities IO 2.4.1 Health facilities, CG correctly identify malnourished, growth-faltering children; counsel caretakers Water committee members trained about protection of water points Resilience Liaise with USAID project on acute malnutrition to refer wasting children Liaise with IHP Effective media for communicating SBCC health, MCHN and FP messages identified OP 2.3.4.2 CVCs mentored on WASH action-planning OC 2.3.2 LMs conduct community health sensitizations IO 2.3.6 RECOS refer MHCN, iCCM, GBV cases Output SP IO Purpose External to project Outcome In logframe IO 1.1.3 Women, men in union make household financial decisions jointly LDCs trained about waer treatment, protection of water points OC 3.2 Community assets improved WASH analysis identifies village' needs, appropriate interventions Gender mainstreaming OP 2.3.4.1 Community WASH structures exist, function, are clean IO 3.1.1 Women, youth have confidence in their ability to participate and community decision-making Risk of water-borne disease outbreak reduced at HH, community level Women, men, youth are sensitized to WASH best practices through SBCC activities Faster, safer transportation available POs discuss topics of gender, nutrition IHP, MOH support IO 3.2.1 Community actors implement resilience-focused LDPs and practice project monitoring LDCs with CSOs make gender-sensitive local development & contingency plans to address communities' priorities to build resilience & define monitoring & accountability mechanisms for performance Gender, health, and life skills SBCC messages incorporated into Safe Space mentors trainings IHP, MOH assure availability of FP commodities at HC, HPs Safe Space group members referred for FP services Gender and health SBCC messages incorporated into PPC trainings OP 2.3.1.2 LMs (care groups members) trained to conduct community health sensitizations Market research on increasing sanitation supplies shared with YBGs Market research on increasing sanitation supplies shared with VSLAs Men, women, youth invest in IGAs Malnourished children rehabilitated and growth￾faltering corrected IO 3.1.1 Women, youth have confidence in their ability to participate in community decision- making IO 1.1.3 Women and men in union make household financial decisions jointly OP 3.1.2.1 Women, men, trained about gender equity, positive masculinity, SGBV, gender themes OC 2.3.4 Men & women construct HH latrines & handwashing stations and purchase essential WASH supplies OP 2.1.1.3 SBCC messages on FP and HTSP methods, services, benefits broadcast via effective media OP 2.3.3.1 Community water points exist and function, are clean OP 2.4.1.1 LMs trained to identify and refer malnourished and growth-faltering children Mid-Term Evaluation of the FSP-Enyanya DFSA in the DRC Annexes 59 Purpose 3 OC 3.2.2.2 Marginalized groups, including women, represented in LDCs Gender-, age-inclusive LDCs have community leadership capacities Marginalized groups identified LDCs trained, mentored on community resilience-related topics Capacity building needs of key peace-building actors identified Justice & Peace Commissions, CLPCs, ChefGs, CDMs trained, mentored as appropriate LDC-led consultations identify community info needs, preferred forms of transmission IO 3.2.1 Community actors implement LDPs and practice projects monitoring Local shocks, stressors, capacities, strategies identified OP 3.1.1.1 Women, youth from LDCy, YBGs, CGs, VSLAS, POs trained in leadership OP 1.1.3.2 Literacy, numeracy training for women and youth, including all VSLA members SP 3.1 Women, youth participate in community planning, decision-making IO 3.1.1 Women, youth have confidence in their ability to participate in community decision- making IO 3.1.2 Increased community acceptance of self-determination, leadership by women, youth CSOs with potential for effective partnerships, capacity reinforcement identified CSO capacity reinforcement plan developed, implemented IO 3.2.3 CSOs, esp. youth, women CSOs, have the capacity to represent community members, incl. social accountability, advocacy and inclusiveness. LDCs with CSOs make gender- sensitive local development & contingency plans to address communities' priorities to build resilience & define monitoring &accountability mechanisms for performance High level leaders, ChefGs, LDCs, CSOs identify issues re development plans CSOs & CDMs have capacities to mitigate or resolve conflicts through dialogue or referral to appropriate body IO 3.3.2 Conflicts resolved peacefully through community platforms Communities aware of conflicts, related conflict-mitigation actions Communities trust conflict management processes & structures OP 3.3.1.1 Community actors trained to advocate, negotiate for land access on sharecroppers' behalf Land negotiations supported Discussion on land policies, services & roles; taxes; inheritance issues promoted OP 3.4.1.1 CSOs, local government, traditional authorities are informed about the land code, inheritance rights Existing mechanisms to obtain acceptable proofs of title to land identified OC 3.3.1.1 Women, men, youth smallholders obtain acceptable proofs of access to land OC 3.3.1.2 Men, women, youth enter sharecropping arrangements that are transparent, fair Individuals are willing to improve the land they farm IO 1.1.1 Farmers apply improved techniques appropriate to their land Issue of illegal "taxes", other barriers mitigated Plan to improve regulatory environment developed & implemented with gov, private sector, KVC, CARGs, IGA, CSOs SP 1.2 Women, men, youth actively participate in markets Market operations stimulated P3 The socio-economic environment is stable and inclusive IO 3.2.2 Inclusive LDCs support community efforts toward resilience IO 3.3.1 Land-related sources of conflict reduced, conflicts avoided Community- service provider partnerships exist Peaceful, participatory management of shared natural resources SP 3.2 Communities have the info, tools and processes to monitor and improve the socio-economic environment Scorecards developed to monitor development plan progress Scorecards administered, results shared with all stakeholders Stakeholders revise, reinforce community resilience action plans OP 3.2.3.2 CSOs trained on accountability, leadership, advocacy, community planning, &development plan monitoring using scorecards CSOs have tools to hold actors accountable for planning, resource allocation OP 3.4.1.2 Communities, authorities are sensitized to legal / illegal taxes, fees related to market access IO 3.2.4 Community members, leaders are aware of community interests, needs, challenges OP 3.2.2.1 Community members have information key to pro- resilience decision- making SP 3.3 Communities are resilient against conflict VC actors identify barriers, solutions IO 1.1.4 increased number of hectares cultivated with nutritious, low-risk, and profitable crops Discussions on inclusive gender themes with VSLA members, spouses OC 1.1.2.2 Women and youth organize, effectively manage VSLAs OP 3.2.1.1 FFA supports LDP projects implementation Women and youth establish nurseries Critera exist for community selection of assets for rehabilitation, workers via FFA WASH engineer provide technical oversight to infrastructure work Assessment of safety of access for women & girls to spring sites Youth have skills for future leadership in agriculture WASH analysis identifies village' needs, appropriate interventions Gender & health SBCC messages incorporated into PPC trainings Gender SBCC messages incorporated into VSLA trainings PLD include a taxation advocacy section Women, men youth are trained on crop theft prevention techniques CLDs, local authorities maintain a public list of legal taxes Local authorities restrict informal tax collecting OC 3.3.2.1 Community conflict mitigation/ resolution systems function Resilience Output SP IO Purpose External to project Outcome In logframe Gender mainstreaming OP 3.2.3.1 CSO capacity- building plans, esp. youth & women CSOs, developed Gender SBCC messages broadcast via mass media OP 3.3.2.1 CSO, CDM members are trained on conflict mitigation, resolution Community structures advocate to local authorities to enforce anti-theft laws peacefully and support theft prevention Theft reduction Intergroup, intercommunity dialogues discuss challenges to conflict mitigation/ resolution systems, theft prevention/ resolution systems, identify actions Community level theft prevention and conflict mitigation/ resolution/ avoidance actions implemented Institutional response and early warning for crop pests/disease LDCs develop standards for more secure land rental contracts Women, youth businesses have capacity to refuse to pay illegal taxes and report illegal tax levying to authorities Women, youth businesses can access info re legal/ illegal taxes SP 3.4 Women, men and youth in business perceive taxation practices as transparent and fair IO 1.1.3 Women and men in union make household financial decisions jointly Community assets support HH & community resilience Sensitization of communities of the need for inclusive representation in leadership Reduced agricultural losses Information about GBV gaps shared with local committees, authorities FS, natural resources services mapped; action priorities identified Maintenance, environmental protection of FFA assets incorporated into community development plans Women, men, youth practice gender equity in business and decision-making OP 1.2.1.1 YBGs trained in financial services, business set up, management, gender equity Gender SBCC messages incorporated into YBG trainings OP 1.2.1.1 POs, women's groups trained to develop gender-sensitive, envr. friendly business plans, register as cooperatives Gender SBCC messages incorporated into PO trainings LDCs trained about water treatment, protection of water points Appropriate service providers share info with communities Gender, health, and life skills SBCC messages incorporated into Safe Space mentors trainings Safe space mentors identified & trained to support Safe Space groups PPCs trained to counsel couples about resource sharing, decision-making, HTSP, GBV, make FP referrals Men, women, youth counseled in FP, sharing resources, decision making Formative research to define special needs, capacities of female youth OP 3.1.1.2 Safe Space groups meet regularly OC 2.3.3 Couples make joint decisions about family health care Community assets improved Faster, safer transportation available OP 2.3.4.1 Community WASH structures exist, function, are clean Communities select the most vulnerable as FFA workers OP 3.1.2.1 Women, men, youth trained on gender equity, positive masculinity, SGBV,gender themes IO 3.4.1 Community leaders support improved transparency of regulatory environment IO 1.1.1 Men, women & youth apply improved agricultural techniques IMPEL | Implementer-Led Evaluation and Learning 60 Annexes 5.3 FSP-Enyanya Intervention Descriptions Mid-Term Evaluation of the FSP-Enyanya DFSA in the DRC Annexes 61 IMPEL | Implementer-Led Evaluation and Learning 62 Annexes Mid-Term Evaluation of the FSP-Enyanya DFSA in the DRC Annexes 63 IMPEL | Implementer-Led Evaluation and Learning 64 Annexes Mid-Term Evaluation of the FSP-Enyanya DFSA in the DRC Annexes 65 IMPEL | Implementer-Led Evaluation and Learning 66 Annexes Mid-Term Evaluation of the FSP-Enyanya DFSA in the DRC Annexes 67 IMPEL | Implementer-Led Evaluation and Learning 68 Annexes Mid-Term Evaluation of the FSP-Enyanya DFSA in the DRC Annexes 69 IMPEL | Implementer-Led Evaluation and Learning 70 Annexes 5.4 MTE Objectives and Complete Evaluation Questions OBJECTIVE 1: Review the quality of program service delivery related to the different themes and systems in addressing chronic food insecurity and child malnutrition with the targeted clients, taking into account contextual changes that may have occurred since the inception of the activities. Illustrative questions under this objective may include: 1.1. What factors appear to enhance or detract from the quality? This should include factors that are within the manageable interest/control of the IP as well as those outside the control of IPs. 1.2. What systems are used to capture, document, and share lessons learned? How are these lessons learned then used to continue ensuring/improving program quality? How is stakeholder and/or participant feedback/perspectives included in this system? 1.3. How well have the interventions met planned schedules and outputs? What factors promoted or inhibited adherence to schedules and planned interventions? How were challenges managed? 1.4. What are the strengths of and challenges to the overall design, implementation, management, communication, and collaboration so far? What factors appear to promote or impede activity operations or effective collaboration and cooperation among the various stakeholders? Are DFSA assumptions still relevant? 1.5. In each technical sector, what are the strengths of and challenges to the efficiency of interventions’ implementation and acceptance in the community? How well do implementation processes adhere to underlying principles and activity protocols? OBJECTIVE 2: Identify evidence of changes (positive and negative, intended and unintended) associated with program interventions. This will include identifying factors that appear to promote or hinder women and men’s, and young people’s, food security and safety. Illustrative questions under this objective may include: 2.1 What changes do community members and other stakeholders associate with the DFSA interventions? What factors appear to promote and deter the changes? How do the changes correspond to those hypothesized by the DFSA TOC? 2.2 To what extent are there observable gender discrepancies? Discrepancies between youth and adults? How has the activity design accounted for these discrepancies and “course corrected” to mitigate these discrepancies? 2.3 How could the DFSA be modified to improve its acceptability to targeted communities or the efficiency and effectiveness of its implementation? How should the DFSA TOC be refined or modified? OBJECTIVE 3: Assess the quality, relevance and efficacy of the DFSA design, taking into account whether activities are contextually appropriate, address critical needs, and maintain standards that can impact on positive change. Evaluate relevant programmatic principles such as whether approaches are Mid-Term Evaluation of the FSP-Enyanya DFSA in the DRC Annexes 71 human centered, evidence based, systems oriented, focused on the most vulnerable, designed for multiple interventions to target the same household, and focused on quality. Illustrative questions under this objective may include: 3.1 How do community members and field agents perceive the activities in relation to local relevance and priority needs? To what extent do local perceptions regarding relevance and quality of the activities affect implementation of activities by field agents and participation of community members? 3.2 How could the DFSA design be adjusted to better coincide with local contextual factors and priorities and at the same time meet desired objectives and longer term sustainability? OBJECTIVE 4: Assess the degree and benefits of efficacious coordination, collaboration, and convergence with locally-based DFSA partners and external organizations that are critical to achieve DFSA goals and purposes. Illustrative questions under this objective may include: 4.1 What mechanisms are in place to engage with local organizations, government entities, private sector, and/or other stakeholders? How strong are these relationships? To what extent will these stakeholders be able to sustain or take over interventions currently being implemented by the IP? 4.2 To what extent do local stakeholders help or hinder the implementation of activities? What has the IP done to overcome these challenges? 4.3 How has the prime implementing partner worked with other USAID activities in DRC? What has worked well and what has not worked well? What factors are within the manageable interest of the IP and what are not? How has the IP managed the relationship/s with other USAID activity implementing partners? 4.4 How has information/data been shared across partners? Other stakeholders? OBJECTIVE 5: Assess early evidence of sustainability produced by the DFSA activities thereby determining the extent to which outcomes, systems, and services are designed and being implemented to continue after the DFSA ends. Illustrative questions under this objective may include: 5.1 To what extent has the activity documented and implemented a sustainability strategy? 5.1.1 How familiar with this plan are senior-level management and technical staff? 5.1.2 How familiar with this plan are “frontline” staff? 5.1.3 How familiar with this plan are subs and other partners? 5.1.4 How familiar with this plan are other indirect partners (e.g. local government, local businesses)? 5.1.5 How familiar with this plan are participants? 5.1.6 What activities do participants perceive to be most relevant/critical in regard to longer term sustainability post implementation? 5.2 How will positive outcomes be sustained after the activity ends? Which local systems/structures/entities will sustain these outcomes? IMPEL | Implementer-Led Evaluation and Learning 72 Annexes OBJECTIVE 6: Assess the appropriateness and quality of the R&I approach and activities. How well did the R&I approach work in DRC across the 3 DFSAs. Illustrative questions under this objective may include: 6.1 How have R&I activities been used to inform the design and ongoing modification of activity implementation? How were findings from the studies reflected in the M&E documents (e.g. TOC, LogFrame, IPTT) and DIP? 6.2 To what extent has R&I been used beyond the initial refinement period and during the implementation phase? 6.3 Have the DFSAs used R&I for DFSA staff capacity building and community engagement and development? 6.4 What are some of the challenges faced in implementing R&I activities and what adjustments need to be made to address and minimize these factors in the future? 6.5 How realistic was the budgeting and timeline for R&I? Did the DFSA significantly over or under￾spend? 6.6 How can the R&I model be improved to better ensure that R&I activities are effectively used to strengthen design and implementation of DFSAs? [Key Recommendation]? OBJECTIVE 7: Determine the appropriateness and effectiveness of interventions focused on cross-cutting themes for the activities. This includes cross cutting interventions designed to improve gender and equity in decision making to achieve food security outcomes and targeting youth to improve their access to, participation in, and benefit from DFSA interventions. This also includes analysis of the effectiveness of activities designed to strengthen governance at the local level, in an overall effort to empower communities to sustain improvements in food security and nutrition, and environmental risk and mitigating the impacts of climate change. OBJECTIVE 8: Recommend adjustments to program implementation or design and explain how these changes would improve program outcomes and sustained impact. Illustrative questions under this objective include: 8.1 Assess personnel capacity, management of human resources (including sub-partners), implementation systems, operations and logistics, and M&E systems 8.2 Assess scope and quality of DFSA activities in relation to overall strategy: What aspects of program design, mechanics related to implementation, R&I, and human resource capacity facilitate or interfere with implementation of activities? How can these factors be modified to strengthen implementation of activities?? Mid-Term Evaluation of the FSP-Enyanya DFSA in the DRC Annexes 73 5.5 Data Collection Sites Visited Health Zone Miti￾Murhésa Kalehe Katana Kalehe Miti￾Murhésa Bukavu (Province Capital) Health Area Cifuma Muhongoza Luhihi Kasheke, Lemera Bushumba Village Kashunguri, Cinjoma, Cishanga￾Kalwa (MC) Munanira (WV) Luhihi Centre (MC) Lemera (WV) Bushumba Centre (MC) Bukavu (all IPs) Totals Data collection method M F M F M F M F M F M F KIIs 6 7 7 2 1 15 38 5 1 9 0 15 2 1 1 1 0 28 2 IDIs 11 14 11 12 16 0 64 9 5 9 8 11 3 7 5 10 11 0 0 FGDs 12 9 7 12 5 45 37 104 62 41 46 13 41 55 19 30 Water Samples 2 0 0 2 1 0 5 Observations 7 6 7 10 8 0 38 IMPEL | Implementer-Led Evaluation and Learning 74 Annexes Sector Totals (KII, IDI, FGD, Observation) Health Zone Miti￾Murhésa Kalehe Katana Kalehe Miti￾Murhésa Bukavu (Province Capital) Totals Health Area Cifuma Muhongoza Luhihi Kasheke, Lemera Bushumba Village Kashunguri, Cinjoma, Cishanga￾Kalwa (MC) Munanira (WV) Luhihi Centre (MC) Lemera (WV) Bushumba Centre (MC) Bukavu (all IPs) Agriculture 9 6 4 5 10 4 38 Health & Nutrition 10 12 10 10 7 7 56 WASH 7 9 8 11 8 4 47 Governance 9 8 8 10 6 7 48 M&E 0 0 0 0 0 3 3 Mix 1 2 2 0 0 1 6 Mid-Term Evaluation of the FSP-Enyanya DFSA in the DRC Annexes 75 5.6 Evaluation Team Composition Name Role Organization Fieldwork Responsibilities Annette Fay WASH Expert, Co-team Lead: Mercy Corps Tulane University Mercy Corps & FH Evaluation Design; Fieldwork; Analysis; Report Writing Alexandre Diouf Agriculture Expert, Co-team Lead: Mercy Corps Tulane University Mercy Corps Fieldwork: Analysis; Report Writing Anicet Yemweni Qualitative Research Expert Tulane University (Kinshasa School of Public Health) Mercy Corps & FH Fieldwork; Analysis Diana Caley M&E Advisor FFP Washington Mercy Corps Fieldwork; Analysis; Report Writing Mike Manske Nutrition Advisor FFP Washington Mercy Corps Fieldwork; Analysis; Report Writing Eva Christensen Agriculture & Livelihoods Advisor FFP Washington Mercy Corps Fieldwork; Analysis; Report Writing Aldabert Lumpali Translator, Logistics Expert Tulane University (Kinshasa School of Public Health) Mercy Corps Fieldwork Justin Colvard Representative, Resilience Expert Mercy Corps Mercy Corps Fieldwork; Analysis; Report Writing IMPEL | Implementer-Led Evaluation and Learning 76 Annexes 5.7 Mid-Term Evaluation Scope of Work USAID Food for Peace DRC Midterm Evaluation Illustrative Scope of Work Summary for IDEAL March 28, 2019 Summary Paragraph The Mid-term Evaluation (MTE) for the Development Food Security Activities (DFSA) in Democratic Republic of Congo (DRC) is scheduled to take place from May to September 2019, with data collection scheduled for mid-July to mid-August 2019. The evaluation will examine the 3 DFSA’s programmatic and operational approaches, quality, and effectiveness. The primary purpose of the MTE is to provide practical recommendations to improve the quality of implementation and increase the likelihood of sustained, positive impacts on communities and individuals in the implementation areas. The evaluation will: review the quality of program service delivery, taking into account contextual changes in the operational environment; analyze intended and unintended consequences of the activities at the individual to community level; examine the extent to which the activities have successfully collaborated with partners and other stakeholders; identify sustainability pathways and inform the strategic plan to achieve sustained, positive outcomes; and determine the appropriateness and effectiveness of cross￾cutting themes in each activity. At the end of the field-based evaluation period, the evaluation team will hold a consultative workshop to share preliminary findings and engage various stakeholders. Given the uncertainty of the security situation in DRC, the evaluation team will be primarily led and staffed by external consultants, including a Team Lead and several technical area specialists. However, if the security situation is permissible, additional technical specialists from the Office of Food for Peace and implementing partners’ home or regional offices will also join the Evaluation Team as technical/area specialists. Given the geographic spread of the three DFSA’s, the Evaluation Team will most likely need two separate sub-teams: one to conduct the fieldwork in Kasai Oriental (the CRS activity area); and one to conduct the fieldwork in South Kivu/Tanganika (the Mercy Corps and Food for the Hungry activity areas). Mid-Term Evaluation of the FSP-Enyanya DFSA in the DRC Annexes 77 USAID Food for Peace DRC Midterm Evaluation Illustrative Scope of Work for IDEAL March 28, 2019 Democratic Republic of Congo USAID Office of Food for Peace Development Food Security Activity Midterm Evaluation Illustrative Scope of Work Background USAID/Food for Peace is currently funding three development food security programs (DFSA) in the Democratic Republic of Congo (DRC). The activities were awarded in fiscal year (FY) 2016, and are scheduled to end in FY 2021. The three prime awardees are Catholic Relief Services (CRS), Food for the Hungry (FH), and Mercy Corps (MC). Each of the three activities have similar overarching goals, although specific sub-activities vary across implementation areas. Overview of Development Food Security Activities CRS: Budikadidi, meaning “self-sufficiency” in Tshiluba, implemented in the Kasaï Oriental region, focuses on building local capacity, strengthening service-delivery systems, increasing accountability, and reducing structural, cultural, and gender-based barriers to change. The goal of the project is to ensure that nutrition and food security for households in the Eastern Kasaï regions are sustainably improved for to achieve sustained nutrition, food security, and economic well-being outcomes for 74,992 participants (20,972 women, 17,209 men, 18,324 girls, and 18,487 boys) in three rural health zones (Miabi, Cilundu, and Kasansa) in the province of Kasaï Oriental. The activity theory of change (TOC) includes the following purposes: Foundational Purpose: Communities empowered to improve and sustain food security and nutrition for all members; Purpose 1: Chronic malnutrition in children under five sustainably reduced; and Purpose 2: Households' inclusive social and economic well-being improved. Food for the Hungry: Tuendelee Pamoja II (TPII) is designed to reach about 214,000 households (1,427,487 individuals) in the two provinces of Tanganyika and South Kivu. TPII includes agriculture recovery and development, health, nutrition, water and sanitation interventions, with particular focus on the integration of women and youth, and conflict resolution. The TOC includes the following purposes: Purpose 1: Households have food and income security; Purpose 2: Improved Nutrition and Health Status of WRA, PLW, Adolescent Girls, and Children Under-5; and Purpose 3: Women, youth, and men of all tribes (WMYAT) are social equals and feel safe in their homes and communities. Mercy Corps: The South Kivu Food Security Project (FSP) focuses on improving food and nutrition security and economic well-being of vulnerable households (210,000 participants) in South Kivu province. The TOC includes: Purpose 1: Household incomes increased; Purpose 2: Improved nutritional status in targeted communities; and Purpose 3: The socio-economic environment is stable and inclusive. FFP Approach to the Midterm Evaluation IMPEL | Implementer-Led Evaluation and Learning 78 Annexes The Mid-term Evaluation (MTE) for these activities is scheduled to take place from May to September 2019, with data collection scheduled for mid-July to mid-August 2019. This process evaluation will examine the 3 DFSA’s programmatic and operational approaches, quality, and effectiveness. The primary purpose of the MTE is to provide practical recommendations to improve the quality of implementation and increase the likelihood of sustained, positive impacts on communities and individuals in the implementation areas. The evaluation will: review the quality of program service delivery, taking into account contextual changes in the operational environment; analyze intended and unintended consequences of the activities at the individual to community level; examine the extent to which the activities have successfully collaborated with partners and other stakeholders; identify sustainability pathways and inform the strategic plan to achieve sustained, positive outcomes; and determine the appropriateness and effectiveness of cross-cutting themes in each activity. As process evaluation, the MTE will primarily draw on qualitative data, although the Evaluation Team will also use available quantitative data (e.g. annual reporting data and secondary data, as needed) to answer the evaluation questions. The Evaluation Team will be responsible for consulting with the DFSA partners, USAID/DRC Mission, and FFP/Washington teams to refine the evaluation questions and identify the most appropriate data collection and analysis methods. Given the uncertainty of the security situation in DRC, the Evaluation Team will be primarily led and staffed by external consultants, including a Team Lead and several technical area specialists. However, if the security situation is permissible, additional technical specialists from the Office of Food for Peace and implementing partners’ home or regional offices will also join the Evaluation Team as technical/area specialists. Given the geographic spread of the three DFSA’s, the Evaluation Team will most likely need two separate sub-teams: one to conduct the fieldwork in Kasai Oriental (the CRS activity area); and one to conduct the fieldwork in South Kivu/Tanganika (the Mercy Corps and Food for the Hungry activity areas). Evaluation Purpose and Intended Uses The objectives of the MTE evaluation will be:  To review the quality of program service delivery and systems in addressing chronic food insecurity with the targeted clients, taking into account contextual changes that may have occurred since the inception of the activities.  Identify evidence of changes (intended and unintended) associated with program activities, examine how the changes relate to progress toward program objectives, and identify factors that appear to promote or hinder the program’s progress toward desired objectives and post￾implementation sustainability.  Assess the degree and benefits of efficacious coordination, collaboration, and convergence with external organizations that are critical to achieve project goals and purposes.  Assess early evidence of sustainability produced by the activity thereby determining the extent to which outcomes, systems, and services are designed and being implemented to continue after the project ends.  Determine the appropriateness and effectiveness of interventions focused on cross-cutting themes for the activities. This includes interventions designed and implemented to improve gender and equity in decision making to achieve food security outcomes and targeting youth to improve their access to, participation in, and benefit from project interventions. This also Mid-Term Evaluation of the FSP-Enyanya DFSA in the DRC Annexes 79 includes analysis of the effectiveness of the activities around environmental risk and mitigating the impacts of climate change.  Recommend adjustments to program implementation or design and explain how these changes would improve program outcomes and sustained impact.  Facilitate a workshop to engage various stakeholders (program staff, donors, local and national government, participant communities) in an analysis of the evaluation results to determine potential program adjustments and strategic planning. The primary audience of the evaluation reports will be DFSA partners (including sub awardees) and USAID. The MTE findings will also be disseminated more widely to additional stakeholders, such as local community representatives, and other identified during the MTE process. Evaluation Questions The MTE will include the following evaluation questions, which are organized by Objective. Objective 1: Review quality of program service delivery and systems in addressing chronic food insecurity with the targeted PSNP clients, taking into account contextual changes that may have occurred since the inception of the activities. Look at strengths and weaknesses of activity implementation; the quality of outputs in terms of the effects they are producing, their adherence to terms agreed upon with FFP, and their appropriateness and perceived value to target communities. This will include identifying factors that appear to enhance or detract from the quality; review systems for capturing, documenting, and sharing lessons learned and how they are used in project implementation, including feedback from the perspective of stakeholders and participants. Illustrative questions under this objective may include:  How well have the interventions met planned schedules and outputs? What factors promoted or inhibited adherence to schedules and planned interventions? How were challenges managed?  What are the strengths of and challenges to the overall design, implementation, management, communication, and collaboration so far? What factors appear to promote or impede activity operations or effective collaboration and cooperation among the various stakeholders? Are project assumptions still relevant?  In each technical sector, what are the strengths of and challenges to the efficiency of interventions’ implementation and acceptance in the community? How well do implementation processes adhere to underlying principles and activity protocols? What factors in the implementation and context are associated with greater or lesser efficiency in producing outputs of higher or lower quality? Which complementary interventions are more or less acceptable to PSNP clients and why?  How does the program identify, capture and document lessons learned? How are lessons learned shared, and how often? What evidence does the program have showing that it successfully used lessons learned, monitoring and evaluation evidence to improve program quality or avoid mistakes? Objective 2: Identify evidence of changes (intended and unintended) associated with program activities, examine how the changes relate to progress toward program objectives, and identify factors that appear to promote or hinder the program’s progress toward desired objectives and post￾implementation sustainability. Illustrative questions under this objective may include: IMPEL | Implementer-Led Evaluation and Learning 80 Annexes  What changes do community members and other stakeholders associate with the project’s interventions? What factors appear to promote and deter the changes? How do the changes correspond to those hypothesized by the project’s TOC or LF?  Based on these findings, how could the project be modified to improve its acceptability to targeted communities or the efficiency and effectiveness of its implementation? How should the project’s TOC or RF be refined or modified? Objective 3: Assess the degree and benefits of efficacious coordination, collaboration, and convergence with external organizations that are critical to achieve project goals and purposes. This includes actors that provide complementary services necessary to achieve the project outcomes, actors that will provide essential services to sustain the outcomes, actors that influence people’s access to goods and services, and organizations that promote or impede an “enabling environment”. Illustrative questions under this objective may include:  What mechanisms are in place to engage with other organizations, government entities, the private sector, and other NGOs? How does information get shared by the programs to the different stakeholders?  Given the reliance on GoE counterparts, what works well and what hinders collaboration and complementarity in each technical sector? Objective 4: Assess early evidence of sustainability produced by the activity thereby determining the extent to which outcomes, systems, and services are designed and being implemented to continue after the project ends. Illustrative questions under this objective may include:  Has the project identified in its sustainability strategy which project outcomes (services, goods or structures)?  How will positive results be sustained after the project end? Which entities/organizations will effectively sustain project outcomes? Objective 5: Determine the appropriateness and effectiveness of interventions focused on the cross￾cutting themes for the activities. This includes interventions designed and implemented to improve gender equity in decision making to achieve food security outcomes and targeting youth to improve their access to, participation in, and benefit from project interventions. This also includes analysis of the effectiveness of the activities around environmental risk and mitigating the impacts of climate change. Illustrative questions under this objective may include:  How are program activities integrating, adapting programs to, and measuring the effectiveness of gender equity across the program technical and operational areas? Do activities encourage/deter equity in participation of different community structures and activities?  What measures are put in place to mitigate any negative effects of gender inequities within activities to promote full active participation in program activities? How does the program define and measure “participation in” and evaluate “benefit from” related to gender in the program? Study Design The evaluation will used a mixed methods approach, which should include analysis of existing quantitative data (e.g. routine monitoring and annual survey) and collection and analysis of qualitative data. Qualitative data will be collected primarily through interviews, observation, and group discussions Mid-Term Evaluation of the FSP-Enyanya DFSA in the DRC Annexes 81 with representatives of project implementation staff, project participants, community member non￾participants, and project partners including consortia and government partners. The evaluation team members will also observe DFSA interventions, trainings, community-based meetings, as they are being implemented. The Evaluation Team should propose specific data collection and analysis methods for each evaluation question identified in the final scope of work. The ethical review process should be detailed in the Evaluation Plan and Final Report, and adhere to local ethics requirements and regulations. Desk Review The Evaluation Team will begin with a desk review of the following documents to contextualize and refine the evaluation questions, as well as to gain an in-depth understanding about the activity design, implementation and the food security situation in the area. Partner annual monitoring data should be reviewed when preparing for qualitative research, considered in relation to the resilience findings, and incorporated into the report as evidence of evaluation findings. While FFP recommends the below documents for pre-evaluation learning, the literature review should not be limited to the following:  Activity proposals  Theory of change and M&E Plan  Pipeline Resource Estimate Proposals (PREPs)  Detailed Implementation Plan  Annual Results Reports (ARRs) and Quarterly reports  Including Indicator Performance Tracking Tables (IPTTs)  Baseline Study  DRC Demographic and Health Survey  Partner formative research reports  Monitoring data and field reports  Sustainability and Exit Strategy Plan  Gender Analysis report  Initial Environmental Examination Report and Environmental Status Report Consultation with Implementing Partners and USAID In preparation for the field-based qualitative research, the Evaluation Team will hold consultations with FFP and partner staff in DRC and in Washington, DC, to corroborate its understanding of the design, approaches and interventions employed by the DFSAs and acquired through the desk review. USAID and implementing partners will provide input and feedback on the draft evaluation protocol and on specific tools, questions and/or outlines to be used in the collection of data. Qualitative Study Qualitative methods will be used to collect information to answer evaluation questions, and also to support the interpretation of findings from the desk review. The Evaluation Team will design the overall qualitative study approach and should consider a variety of primary data collection methods, such as semi-structured in-depth interviews, group discussions, key informant interviews, direct observations and case studies. These methods, including criteria for purposive sampling, as well as locations and IMPEL | Implementer-Led Evaluation and Learning 82 Annexes timing of data collection activities, should be finalized and agreed upon during the consultation process and prior to field-based data collection. The evaluation team leader and members will be responsible for interviewing direct, indirect, and non￾participant community members and households, and other relevant stakeholders for the evaluation and analyzing the qualitative data. The qualitative study should use a purposive sample of individuals that includes both those who directly participated in the DFSA and those not specifically targeted with any intervention (i.e. non-participants). The purpose of including non-participants in the qualitative study sample is to help the Evaluation Team examine any spillover effects, community equity issues, and to understand the broader impacts of the activity at the community level. In addition, the qualitative team should interview USAID personnel, activity staff, knowledgeable people from the community, local government staff, community leaders, host Government officials and other agencies and individuals as appropriate. A complete list/map of these key stakeholders should be developed thoughtfully in collaboration with FFP/DRC and the implementing partners. Ground Truthing/Verification Workshop At the end of the data collection fieldwork, and pending security considerations, the Evaluation Team will facilitate a workshop to engage various stakeholders (program staff, donors, local and national government, participant communities) in an analysis of the preliminary evaluation results to determine potential program adjustments and strategic planning. Evaluation Team The MTE will be carried out by a team of evaluators and technical experts. Given the geographic distribution of the DFSAs, most likely two evaluation teams will be staffed to cover all 3 DFSAs. For example, one team will likely evaluate the CRS activity in Kasai Oriental and the other team will evaluate the FH and Mercy Corps activities in South Kivu/Tanganika provinces. The Evaluation Team will consult with DFSA partners, USAID Washington staff, and USAID field staff to identify the aspects of each DFSA that they consider to be working particularly well, and to identify the gaps that they think would be most valuable to address during the midterm evaluation. The final scope of work for the evaluation should be drafted in consultation with partner and USAID input. Team members will develop data collection tools in their areas of responsibility and will also prepare presentations for a final, field-based validation and analysis workshop. The team members will analyze the findings and formulate the recommendations linked to the findings. For the three program components in each project (corresponding to each DFSA’s respective Purposes), one of the review team members will be designated as the lead for that component and will have the final say on the formulation of recommendations in the assigned areas and on the content and recommendations. The main areas of subject matter coverage for both evaluation teams will likely include the following:  Agriculture and NRM/Watershed Restoration  Health and Nutrition  Social and Behavioral Change  Potable water source development, Sanitation and Hygiene Education (WASH)  Livelihoods, Agribusiness, and Markets Development Mid-Term Evaluation of the FSP-Enyanya DFSA in the DRC Annexes 83  Environment or Disaster Risk Reduction  Natural Resource Management  Gender  Youth employment  Social protection and productive safety net  M&E and Learning Evaluation Team Roles and Responsibilities There may be one or two Team Leads, depending on the timing and logistics of the final evaluation design (i.e. taking into consideration the geographic spread of the activities and the timeline for data collection, analysis, and dissemination). The Team Lead/s will ultimately be responsible for delivery of final deliverables and for managing the technical specialists on her/his team/s. The Team Leader will:  Organize and lead the overall evaluation;  Assure a thorough review and analysis of available secondary data by the appropriate team member(s),  Lead the selection of a purposely selected sample of activity sites and outputs for primary data collection and assure adequate triangulation and validation of findings;  Lead the collection and analyses of primary and secondary data to evaluate the program’s M&E processes and the integration of program sectors and activities;  Assure that 1) final report presentation is logical and presented in a way that clearly separates findings, conclusions, and recommendations, and 2) all findings, conclusions and recommendations are based on evidence presented in the report  To liaise, on the part of the evaluation team, with the awardee and USAID The Technical Specialists will:  Lead the collection and analyses of primary and secondary technical data related to his/her field(s) of expertise and form recommendations.  Consider all general aspects of the implementation of all activities related to his/her sector, i.e., resource management, staffing, linkages/partnerships, branding, community involvement, cultural acceptability, gender, exit/sustainability measures, environmental protection, adherence to schedules, and integration with other sectors. Team Member Qualifications The Team Leader should have significant formal education at the postgraduate level (Applicants that do not hold a graduate degree in a field should document relevant formal education in the field.) in a field relevant to evaluation (e.g., program evaluation, statistics, anthropology, applied research, organizational development, sociology, organizational change, etc.) and extensive experience using mixed methods of investigation (qualitative and quantitative) in developing countries. Knowledge in the conceptual framework of food security and nutrition and experience in food security programming is highly desirable. IMPEL | Implementer-Led Evaluation and Learning 84 Annexes Each Technical Specialist should have a postgraduate degree (M.S., M.A., or Ph.D.) in a field related to at least one of the technical sectors of the project, plus extensive practical experience in developing countries. Deliverables The Evaluation Team Lead will provide the following deliverables:  Evaluation Plan: The Evaluation Plan should be developed in consultation with USAID and partners, and include the revised evaluation questions; evaluation methods (data collection and analysis); revised timeline; and ethical considerations.  MTE Scope Finalization Workshop: The Team Lead should hold a 1-day workshop to present the final evaluation plan to all relevant stakeholders.  Enumerator Training and Launch: The Team Lead will facilitate a 2-3 day training and orientation for all team members and/or enumerators, as decided in consultation with partners and USAID.  Ground Truthing/Verification Workshop: The Team Lead will facilitate a one-day workshop to review preliminary results of the field data collection. The location of the workshop should be decided in consultation with USAID and partners, such as each respective partners’ primary field office.  Evaluation Reports: The Evaluation Team will produce three evaluation reports, one for each DFSA, not to exceed 40 pages per report. The draft reports will be shared with FFP, USAID/DRC, and with each prime partner for review and comment. Each report should include, at a minimum, the following: o Executive Summary (3-5 pages) o Introduction/Background (3-4 pages) o Mid-Term Evaluation Objectives (1-2 pages) o Mid-Term Evaluation Methods (including strengths and limitations) (up to 5 pages) o Mid-Term Evaluation Findings o Conclusions (3-4 pages) o Recommendations (up to 5 pages) o Annexes (e.g. list of acronyms; MTE SOW; Team Composition; Interview Guides; Bibliography; list of sites visited)  Final Evaluation Report Presentation: Once the final reports have been completed, the Evaluation Team will present the final report findings. These presentations will be 2 hours long and held in Washington, DC, at the Food for Peace Office with participation from the partner teams. Logistics and Timeline The MTE will be conducted over a period of approximately 5 months.  Phase 1 (remote): Evaluation Preparation and Desk Review(May - July 2019)  Phase 2 (in field): Fieldwork and Ground Truthing/Verification Workshop (mid-July to mid￾August 2019)  Phase 3 (remote): Review Recommendations Finalization and Processing (September 2019) Mid-Term Evaluation of the FSP-Enyanya DFSA in the DRC Annexes 85 The Evaluation Team Lead will be responsible for created a detailed timeline for completing the MTE and completing the deliverables outlined above. DFSA Locations  CRS operates in in the province of Kasaï Oriental, specifically in three health zones (Miabi, Cilundu, and Kasansa).  Food for the Hungry operates in Tanganika province and South Kivu province. In Tanganyika province, the implementation area includes Kalemie territory (Kalemie and Nyemba health zones); and in Moba territory (Moba and Kansimba health zones). In South Kivu, the implementation area includes Walungu territory (Walungu, Mubumbano, and Kaziba health zones).  Mercy Corps operates in South Kivu province across the health zones Miti-Murhesa, Katana and Kalehe (which includes 24 health areas). IMPEL | Implementer-Led Evaluation and Learning 86 Annexes Annex A. Activity Description – CRS Budikadidi, meaning “self-sufficiency” in Tshiluba, implemented in the Kasaï Oriental region, focuses on building local capacity, strengthening service-delivery systems, increasing accountability, and reducing structural, cultural, and gender-based barriers to change. The goal of the project is to ensure that nutrition and food security for households in the Eastern Kasaï regions are sustainably improved for to achieve sustained nutrition, food security, and economic well-being outcomes for 74,992 participants (20,972 women, 17,209 men, 18,324 girls, and 18,487 boys) in three rural health zones (Miabi, Cilundu, and Kasansa) in the province of Kasaï Oriental. The activity theory of change (TOC) includes the following purposes:  Foundational Purpose: Communities empowered to improve and sustain food security and nutrition for all members;  Purpose 1: Chronic malnutrition in children under five sustainably reduced; and  Purpose 2: Households' inclusive social and economic well-being improved. Catholic Relief Services (CRS) works with its partners National Cooperative Business Association Cooperative League of the USA (NCBA CLUSA), Caritas Mbuji-Mayi, Réseaux Femmes et Développement (REFED), Human Network International (HNI), and RACOJ (Réseau des Associations Congolaises de Jeunes) to achieve sustained nutrition, food security, and economic well-being outcomes for 74,992 participants (20,972 women, 17,209 men, 18,324 girls, and 18,487 boys). During FY18, Kasai Oriental enjoyed significant improvements in security, allowing program staff access to all targeted communities without any restrictions. CRS and the consortium successfully deployed 108 field agents in all 425 targeted communities. At the provincial level, Cholera was a major health issue in Kasai Oriental, and most cases were recorded in Mbujimayi. Between February 12 and April 30, 2018, a total of 449 cases and 23 deaths were registered, including 45 cases and 2 deaths of children between 0- 59 months. The provincial government, some NGOs and UNICEF were mobilized to address the epidemic, and it was brought under control during the last week of March (March 26-April 1st). Key assumptions to the activity design remain largely unchanged, except for cell phone ownership and network coverage, which are less than what was initially assumed which affects the use of cell phones and SMS as a major means of communication. As a result, SBCC strategies have been adapted. Refine and Implement (R&I) studies and community consultations conducted during year 1 aided Budikadidi in making critical decisions regarding staffing, staff location and the establishment of program sub offices in Miabi, Cilundu and Kasansa. These decisions helped ensure coordination with the territorial government, decentralized ministries and other key stakeholders at the health zone level. Integrated Governance Activity (IGA) and Integrated Health Program (IHP) were also not visibly present on the field, which impacted planned collaboration. Mid-Term Evaluation of the FSP-Enyanya DFSA in the DRC Annexes 87 Annex B. Activity Description - Food for the Hungry Food for the Hungry (FH), DRC is implementing a Development Food Security Activity (DFSA) – Tuendelee Pamoja II (TPII). The implementation period is September 30, 2016 to September 30, 2021 The activity is designed to reach about 214,000 households in the two provinces of Tanganyika (Kalemie territory [Kalemie and Nyemba health zones] and in Moba territory [Moba and Kansimba health zones]) and South Kivu (Walungu territory [Walungu, Mubumbano, Kaziba health zones]) to directly benefit up to 1,427,487 individuals, through interventions in agriculture recovery and development, health, nutrition, water and sanitation with particular focus on the integration of women and youth. As of November 2018, TP II has worked with 137,812 participant households, which include 331,159 individuals. Since its inception in September 2016, FH has continued to establish a foundation for the program based on the lessons learnt during the Refine and Implement (R&I) period, especially the community consultation processes and interaction with various partners. FH continued to analyze and revise the initial activity designs and processes to incorporate findings and recommendations of the various studies conducted. The Program Management Team (PMT) composed of senior management staff, met on a quarterly basis to review program performance and discuss challenges in the implementation of activities. These meetings a key to identifying challenges to program delivery and solutions to improve implementation. In FY18, partnership agreements were signed with key stakeholders, including government ministries of health, Agriculture, and Gender. Two new partners “Union pour l’émancipation des Femmes Autoctone” (UEFA) and Tear Fund (TF) were approved for inclusion into the consortium. UEFA, particularly to address ethnic conflict among the predominantly Twa communities in Tanganyika province and TF to provide support to the Water Sanitation and Hygiene (WASH) interventions. To enhance collaboration and ensure sustainability of interventions from the onset, FH has continued to hold consultative meetings with other USAID funded projects in TPII activity areas such as South Kivu Value Chains (SVC) project, SPR, FSP and recently with the USAID Integrated Health Program (IHP). FH has also held meetings with communities, leaders, and officials from the government sector ministries who are the main stakeholders in service delivery and community development. The TP II activity engaged with a number of partners to complement its efforts in improving the well-being of households in Tanganyika and South Kivu through various activities. TP II Technical Partners and Roles Program Element Technical Partners Maternal Child Health and Nutrition  Integrated Health Project (IHP)  Johns Hopkins University (JHU)  Catholic University of Bukavu (UCB) WASH  Villages Assainis Program  Department for International Development (DFID)-funded WASH Consortium  Tearfund IMPEL | Implementer-Led Evaluation and Learning 88 Annexes Agriculture, fisheries, livelihoods, natural resource management and biodiversity  CGIAR: IITA, ICRAF, ILRI  Institut National d’Etude et de Recherche Agricole (INERA and HarvestPlus)  Cornell University (fisheries)  Tillers International Civic Participation  Search for Common Ground  Ligue pour la Protection des Enfants et le Développement des Mamans (LIPEDEM)  Union pour l’émancipation des Femmes Autochtone (UEFA) The FY2018 annual results report from FH highlighted the following key challenges, successes, and lessons learned. These should be used to help guide the midterm evaluation scope and direction: Challenges  High levels of insecurity, instability and unpredictability in some project target areas (particularly in Tanganyika) in relation to Inter-ethnic conflict, and activities of armed groups, some villages/locations in Tanganyika remained inaccessible to TPII staff during the beneficiary registration and structuring phase in FY18. This significantly delayed the process of beneficiary database creation as well as start-up of project activities in these locations. Though the security in these communities improved by the 3rd quarter of FY18, there are still households that have remained displaced and inaccessible to the project staff.).  Due to the challenges faced with former staff litigations, TP II experience an overall delay in the start-up of activities such as the assessments which should have been conducted during the R&I period such as EAs, and feasibility studies for feeder roads as well as drainage and irrigation activities being completed in FY18. However, TP II was able to recover lost time and complete the assessments for use in the implementation of the programs.  Beneficiary population who were displaced during inter-ethnic conflict lost their agricultural inputs (seeds and tools) and are now facing a lot of challenges to restart or re-engage in farming and some do not even have enough seeds for the value chain crops promoted by TP II.  TPII Initial Environmental Examination was approved after the end of Season B of the agricultural calendar in Eastern DRC thus a number agricultural activities delayed till the start of season A in the last quarter of FY18 and start of FY19. To address this, all the unimplemented activities have been carried forward and re-planned for FY19.  There was a significant delay in the production of an image box by SFCG, a tool necessary to raise awareness amongst members of discussion groups; SFCG will produce it in the first quarter of the year 3. Successes  High level of community participation in project activities was reported all over the three project sites showing community acceptance of the program. The program has gained an overwhelming support of community leaders and elders;  Some activities are already being adapted by farmers. A typical example is the terrace activities in South Kivu. Before completion of any of TP II’s supported terraces, one community had organized themselves into a group and are now working on their own terraces on a rotational basis; Mid-Term Evaluation of the FSP-Enyanya DFSA in the DRC Annexes 89  After a number of inter-organizational meetings between TP II and SVC (Strengthening Value Chain), another USAID funded program implemented in South Kivu, the two organizations have substantially gained an insight into areas for collaboration in terms of implementation of similar program activities. It is now clearer to each of the two organizations that the collaboration will be a win-win one.  FH signed a Protocol of collaboration with the Provincial Ministry of Agriculture of both South Kivu and Tanganyika, which allows the ministry to collaborate and support FH in the implementation of food security and livelihood activities in the field. The collaboration is working well.  After the training of agriculture and health promoters on gender and conflict sensitivity, there is some change and improvement in how to target CDC, LFG, CG and GDG. FH has documented wide participation of men, women and young people in all the activities that are being carried out in the field, which is believed to contribute to prevention of conflicts and support sustainability.  The installation of GDGs has introduced concepts of gender in communities and has visibly improved gender relations, especially in Tanganyika. Attitudes and behaviors related to traditional customs that do not promote women’s rights are slowly fading away.  The presence of women in food for work committees, (CAPITA) strengthens their active participation, their decision-making power in the implementation of the activities of the Nyalugana marsh. Lessons Learned  While implementing the program, TP II should keep on monitoring the security situation all over its geographical coverage and be proactive and work in advance on alternatives;  It would have been advisable for TP II to plan a backup food and tools distribution in communities affected by inter-ethnic conflict within its geographical coverage in order to raise the beneficiaries to a certain level that can facilitate their re-engagement in farming activities after the shock. The distribution could include food rations, seeds of the promoted value chain crops and hand tools (hoes, fork hoes and pangas). IMPEL | Implementer-Led Evaluation and Learning 90 Annexes Annex C: Activity Description - Mercy Corps Mercy Corps, in partnership with World Vision, Harvest Plus, the Université Evangélique en Afrique (UEA) and Action pour la Paix et la Concorde (APC), is implementing the five-year United States Agency for International Development (USAID) South Kivu Food Security Project (FSP). The activity works towards improved food and nutrition security and economic well-being of vulnerable households in South Kivu in support of USAID’s strategic objectives for the Democratic Republic of the Congo (DRC). USAID-FSP supports USAID’s strategic objectives for the Democratic Republic of the Congo (DRC) by working with households, community leaders, the government of DRC and other development programs in pursuit of the project’s goal of improved food and nutrition security and economic well-being of vulnerable households in South Kivu. Operating in the three health zones of Miti Murhesa and Katana (territory of Kabare) and Kalehe (territory of Kalehe), FSP expects to reach approximately 35,000 households or 210,000 people living in 24 health areas. FSP’s Theory of Change (TOC) is based on the hypothesis that: If an adequate level of stability allows the FSP to be implemented and if participants are able to prepare and respond to main shocks and stress to: ● Improve their livelihoods ● Generate sufficient income ● Promote gender equality and youth inclusiveness ● Adopt better health and nutrition practices, and ● Benefit from improved services from public and private sector providers resulting in an environment more favorable to sustainable development, then participants will have: ● Improved access to nutritious food ● Better overall health and more economic opportunities ● Resulting in greater food security and stability, and ● Improved nutritional status and enhanced well-being. The activity identified six key stresses and shocks facing households in the implementation area: land degradation; pests & plant diseases; crop theft; land disputes; multiple taxation to access markets; and waterborne diseases. To support project participants in absorbing and adapting to these shocks and stresses and achieve transformative change in their socio-economic and regulatory environment, resulting in improved food and nutrition security, FSP implements activities under the following sectoral and cross-cutting purposes: Sectoral Purposes:  Purpose 1 (P1): Household incomes increased  Purpose 2 (P2): Improved nutritional status in targeted communities  Purpose 3 (P3): The socio-economic environment is stable and inclusive Cross-Cutting Purposes and Approaches:  Gender and Youth Inclusivity  Social and Behavioral Change Mid-Term Evaluation of the FSP-Enyanya DFSA in the DRC Annexes 91 5.8 Consent Forms & Data Collection Tools 5.8.1 Informed Consent Forms – English Key Informants Hello! My name is XXX. I am here with an evaluation team collecting information to understand the activities that your organization is implementing as part of the (name of project), programmatic and operational approaches being used, the quality and effectiveness of the services, and what changes need to be made to improve the services. We are asking you to participate as a key informant in the evaluation. During the interview, we will ask you questions related to the (name of project) activities, including your role in activity design and implementation, details of the activities related to your technical expertise, the way in which activities are being monitored and evaluated, and ongoing collaboration with your colleagues and other implementing partners. We will also ask about your views on the quality and effectiveness of activities, any challenges faced in program implementation, and recommendations to strengthen the (project name). Your experience as a team leader or technical advisor or partner in the project activities will be valuable to our understanding and knowledge of the project. The information collected through this evaluation may also be used for additional analysis purposes in the future. The interview will take approximately 1h and 30 minutes. We will likely return for subsequent interviews, which will be of shorter duration. There are no right or wrong answers to the questions we will be asking during this and subsequent interviews. You can choose not to answer questions if you prefer not to. Your participation in the evaluation is completely voluntary. It is your choice whether to participate or not. Non-participation will not lead to any negative repercussions. If you participate in the evaluation and want to stop, you can do so at any time. All information given by you will be strictly treated as confidential. During the interview, we will take hand-written notes and the interview will also be audio-recorded. Hand written notes and audio recordings will be kept in a secure location and will not be shared with anyone outside of the evaluation team. At the end of the evaluation we will analyze the data and produce a report. You will not be identified in any summary, presentation, or reports of the evaluation results. Recorded information will be destroyed after data analysis is over. Take as much time as you like before you make a decision to participate in this evaluation. If you have any questions or concerns about the evaluation, whether before or after signing this form, you can call the principal investigator of the evaluation listed below. You can call about any matter having to do with the evaluation, including complaints or questions about your rights as an evaluation participant. Evaluation site investigator (researcher): xxx Phone: +243 xxxx If you have any questions, please feel free to ask me or another member of the evaluation team. Are you willing to participate in the evaluation? 1. Yes 2. No IMPEL | Implementer-Led Evaluation and Learning 92 Annexes Are you willing to be recorded? 1. Yes 2. No Signature of the interviewer: ______________________ Date: ____/_____/2019 Name of the participant: ________________________ Signature of the participant: ________________________ Date: ____/_____/2019 In-Depth Interview Informants Hello! My name is XXX. I am here with an evaluation team collecting information to understand the activities that your organization is implementing as part of the (name of project), programmatic approaches being used, the quality and effectiveness of the services, and what changes need to be made to improve the services. We are asking you to participate in an interview as part of the evaluation. During the interview, we will ask you questions related to the (name of project) activities, including your role in the activities, details of the activities you are involved in, training you have received, your work schedule and responsibilities, satisfaction with your role and responsibilities, and ongoing collaboration with your colleagues and implementing partners. We will also ask about your views on the quality and effectiveness of activities, any challenges faced in program implementation, and recommendations to strengthen the (project name). Your experience as a field agent in the project activities will be valuable to our understanding and knowledge of the project. The information collected through this evaluation may also be used for additional analysis purposes in the future. The interview will take approximately an hour. If we are unable to complete all of our questioning, we may have to return for a subsequent interview. There are no right or wrong answers to the questions we will be asking during this and subsequent interviews. You can choose not to answer questions if you prefer not to. Your participation in the interview is completely voluntary. It is your choice whether to participate or not. Non-participation will not lead to any negative repercussions related to your work. If you participate in the evaluation and want to stop, you can do so at any time. All information given by you will be strictly treated as confidential. During the interview, we will take hand written notes and the interview will also be audio-recorded. Hand written notes and audio recordings will be kept in a secure location and will not be shared with anyone outside of the evaluation team. At the end of the evaluation we will analyze the data and produce a report. You will not be identified in any summary, presentation, or reports of the evaluation results. Recorded information will be destroyed after data analysis is over. Take as much time as you like before you make a decision to participate in this evaluation. If you have any questions or concerns about the evaluation, whether before or after signing this form, you can call the principal investigator of the evaluation listed below. You can call about any matter having to do with the evaluation, including complaints or questions about your rights as an evaluation participant. Evaluation site investigator (researcher): xxx Phone: +243 xxxx If you have any questions, please feel free to ask me or another member of the evaluation team. Mid-Term Evaluation of the FSP-Enyanya DFSA in the DRC Annexes 93 Are you willing to participate in the evaluation? 1. Yes 2. No Are you willing to be recorded? 1. Yes 2. No Signature of the interviewer: ______________________ Date: ____/_____/2019 Name of the participant: ________________________ Signature of the participant: ________________________ Date: ____/_____/2019 Group Discussion Participants Hello! My name is XXX. I am here with an evaluation team collecting information to understand the services that you receive from the (name of project), your perception of the services and what changes you recommend to improve the services. The overall purpose of the evaluation is to examine the programmatic approaches, quality, and effectiveness of the activities being implemented. We are asking you to participate in a group discussion with other people from your community. During the discussion, we will ask you questions related to the (name of project) activities. Your experience as a participant in the project activities will be very valuable to our understanding and knowledge of the project. The information collected through this evaluation may also be used for additional analysis purposes in the future. Your involvement in a group discussion will take approximately 1h and 45 minutes. There are no wrong answers to the questions we will be asking during the discussion. You can choose not to answer questions if you prefer not to. Your participation in the group discussion is completely voluntary. It is your choice whether to participate or not. Non-participation will not affect the services that you usually receive in any way. If you participate in the evaluation and want to stop, you can do so at any time. You will not lose any of your regular benefits Due to the nature of the group discussion, anonymity cannot be guaranteed. We request you and other group discussion participants to keep the discussions in the group confidential. During the group discussion, we will take hand written notes. Discussion will also be audio-recorded. Ideas and positions being expressed by participants will be shared with fidelity to the best ability of the data collectors. Hand written notes and audio recordings will be kept in a secure location and will not be shared with anyone outside of the evaluation team. At the end of the evaluation we will analyze the data and produce a report. You will not be identified in any summary, presentation, or reports of the evaluation results. Recorded informant will be destroyed after data analysis is over. Take as much time as you like before you make a decision to participate in this evaluation. If you have any questions or concerns about the evaluation, whether before or after signing this form, you can call the principal investigator of the evaluation listed below. You can call about any matter having to do with the evaluation, including complaints or questions about your rights as an evaluation participant. Evaluation site investigator (researcher): xxx Phone: +243 IMPEL | Implementer-Led Evaluation and Learning 94 Annexes If you have any questions, please feel free to ask me or another member of the evaluation team. Are you willing to participate in the evaluation? 1. Yes 2. No Are you willing to be recorded? 1. Yes 2. No Signature of the interviewer: ______________________ Date: ____/_____/2019 Name of the participant: ________________________ Participants who are able to read or write Signature of the participant: ________________________ Date: ____/_____/2019 Participants who are not able to read or write – Signature of Witness The consent form was presented orally to the participant, and the participant has indicated his/her consent for participation in the research. Signature of Witness_____________________________Date: ____/_____/2019 Name of Witness _____________________________________________________ Mid-Term Evaluation of the FSP-Enyanya DFSA in the DRC Annexes 95 5.8.2 Guide for Farmers' Focus Groups – French Guide Pour Les Groupes De Discussion Des Agriculteurs (FFSs, YFFSs, POs, YBGs, Elevage pour femmes et / ou Permagarden) Date: Zone De Santé: Aire De La Santé: Village: ----------------------------------------------- A: Organisation 1) Quel est le nom et le but de votre groupe? 2) Quand ce groupe a-t-il été formé? Avez-vous des documents officiels (règles, règlements, etc.) 3) Combien de membres avez-vous? Femmes Hommes 4) énumérer le nom et le sexe des dirigeants Nom Titre/Position 5) Ont-ils été élus? Si oui, quand? 6) selon l'objectif déclaré de votre groupe, croyez-vous que le fait de faire partie de ce groupe vous aide à atteindre les objectifs? Pourquoi ou pourquoi pas? -------------------------------------------------- B: Défis Actuels Des Producteurs 7) Quels sont les défis auxquels les producteurs sont actuellement confrontés dans votre collectivité? [encerclez tout ce qui s'applique] un. Pas assez de pluie/l'accès à l'eau B. ravageurs/maladies C. intrants non disponibles (semences, engrais, etc.)) D. intrants trop chers IMPEL | Implementer-Led Evaluation and Learning 96 Annexes e. On ne sait pas comment utiliser correctement les intrants F. expérience et connaissances limitées en agriculture (techniques) G. espace D'entreposage et de préservation après la récolte H. techniques de conservation et de conservation après récolte I. connaissance des prix courants j. Pas d'accès aux marchés k. ne sait pas comment commencer le marketing l) Transport vers les marchés M. incapacité d'accéder au crédit ou aux prêts n. Autres--LISTE ------------------------------------------------ C: Programmes 8) Dans quelle (s) activités(s) participez-vous? [POs, J / FFSs, élevage, permagardens, microcredit, Hill Approach; Travail contre nourriture] 9) avez-vous reçu des biens du programme? Si oui, quoi? 10) la saison prochaine, vous ne pouvez pas recevoir les mêmes intrants du programme. Pouvez-vous trouver ces intrants sur le marché? Si oui, vous avez l'intention de les acheter? Pourquoi/pourquoi pas? 11) Avez-vous reçu une formation pour vous aider à réussir dans vos activités? Si oui, Quels enseignement en faites-vous? 12) qui a fourni la formation et comment? 13) combien d'autres personnes ont suivi la formation? 14) [Oui / Non] avez-vous trouvé la formation(s): un. Facile à suivre—si non, pourquoi pas? B. adapté à vos besoins et aux besoins de votre communauté C. Facile à se rappeler et à faire soi—même-si non, pourquoi pas? d. Possible pour vous et votre famille—si non, pourquoi pas? e. Idéalement situé F. tenue à des heures utiles de la journée G. enseignée pendant la saison appropriée Mid-Term Evaluation of the FSP-Enyanya DFSA in the DRC Annexes 97 15) vos pratiques agricoles ont-elles changé à cause de la formation? De quelles manières? 16) quelles nouvelles pratiques utilisez-vous et quelles leçons avez-vous tirées de la formation? Voyez￾vous des changements? Si oui, pensez-vous ils sont bons ou mauvais? 17) Comment déciderez-vous de poursuivre ou non la(les) Nouvelle (S) pratique (s)? 18) quels changements (positifs ou négatifs) ont été apportés à la sécurité alimentaire de votre famille ou de votre collectivité à cause du projet? 19) avez-vous des champs ecoles paysans dans votre communauté? Si Oui, Croyez-vous que les parcelles de démonstration sont utiles? 20) qui décide ce qui est enseigné dans le FFS, et comment? ------------------------------------------------------------------------------------------------------------------------------------------ Guide à l'intention des agents de terrain Poste: IP / institution: Lieu: 1) Quand avez-vous rejoint le projet? Quel type de formation ou d'orientation avez-vous reçu depuis le début du projet? 2) Comment décririez-vous les buts et les objectifs généraux du (nom du projet)? 3) quelles activités liées à l'agriculture sont censées être offertes par (nom du projet)? A. Permagarden B. lutte intégrée contre les ravageurs C. Champs ecoles paysans D. relations avec les fournisseurs d'intrants e. L'élevage F. liens avec le marché G) réduction/atténuation des conflits fonciers h) accès des femmes à la terre I. Collines approche j. Travail contre nourriture 4) quelles activités liées à l'agriculture sont réellement offertes par (nom du projet)? Sonder pour comprendre la mesure dans laquelle: IMPEL | Implementer-Led Evaluation and Learning 98 Annexes les activités censé être offert sont mis en œuvre. Sonder pour savoir si les plans de mise en oeuvre sont pratiques ou non. 5) quelles activités de recherche formative ont été menées avant la mise en œuvre des activités liées à l'Agriculture? Comment la recherche formative été utilisée pour l'élaboration du programme et la mise en œuvre? 6) Dans quelle mesure Avez-vous utilisé l'approche de la R&I pour l'engagement et le développement communautaires? Quels sont certains des difficultés rencontrées dans la mise en œuvre de l'approche de la R&I? 7) Quels sont les groupes et les organismes partenaires qui participent à la prestation des activités en agriculture et de développement économique? Quelles organisations sont responsables pour quelles activités? 8) Quels sont les acteurs clés sur le terrain en charge de la mise en œuvre des activités agricoles et de développement économique? Comment étaient le personnel recruté? 9) Quelle est la motivation offerte aux différents types de personnel? Quelles sont les structures de contrôle mises en place pour suivre les activités? Plus précisément, comment les différents membres du personnel sont-ils supervisés et évalués de façon continue? 10) y a-t-il eu des changements (prévus ou non) dans la mise en œuvre des activités agricoles et de développement économique? depuis le début de (nom du projet)? Si oui, pourquoi ces changements ont été faits. Sonder les changements: A. Planification des activités B. fourniture d'intrants c. De formation D. Supervision des travailleurs E. Rotation du Personnel [tous les membres du personnel ont-ils une description de travail claire?] 11) y a-t-il des changements contextuels (par exemple, les questions liées à la sécurité, les structures gouvernementales nationales et locales, l’environnement, facteurs économiques locaux, etc.) qui se sont produites depuis le début du projet et qui ont pu influencer les activités du projet? Dans l'affirmative, quelles ont été leurs répercussions sur les activités du projet? 12) Comment décririez-vous la coordination et la collaboration avec d'autres organismes partenaires, des institutions gouvernementales; les collaborateurs? Quelles sont les entités responsables de quelles activités? A. Planification des activités B. réunions avec les organisations partenaires C. suivi et évaluation des activités Mid-Term Evaluation of the FSP-Enyanya DFSA in the DRC Annexes 99 D. Mise en place de systèmes pour apprendre/partager les activités en cours, les leçons apprises et les défis sur le terrain 13) Nous savons que la durabilité est l'un des principaux objectifs des activités du programme. Comment le développement durable est intégré dans les activités? Comment les mesurez-vous? Des preuves de durabilité sont-elles apparues dans les activités du projet jusqu'à présent? Selon vous, qu’est ce qui doit être fait pour augmenter la probabilité de durabilité une fois ce projet terminé? 14) Si les objectifs du projet ne sont pas atteints, est-ce dû à des échecs de conception ou de mise en œuvre du programme? 15) pouvez-vous décrire les types de ressources et d'assistance technique du projet, des bureaux régionaux ou de l'administration centrale fourni depuis que vous avez travaillé sur le projet? 16) Quelles sont les faiblesses l’approche du projet (nom du projet)? Quels sont les principaux obstacles au (nom du projet) pour atteindre son plein potentiel? Quelles sont les solutions pour surmonter ces obstacles? 17) dans quelle mesure les perceptions locales à l'égard de la pertinence et de la qualité des activités influent-elles sur la mise en œuvre du programme? la participation des membres de la communauté? IMPEL | Implementer-Led Evaluation and Learning 100 Annexes 5.8.3 Nutrition/Health Guides – French Entretiens approfondis avec les agents de santé/chargés de la nutrition Projet FSP/ ENYANYA : Assistant(e) Santé et Nutrition ; Infirmier(e) Titulaire de CS ; Relais Communautaire 1. Quand les activités relatives à la santé maternelle et à la nutrition et à la santé infantile ont-elles été introduites dans votre région ? En quoi consistent les activités ? D’après l’examen de la documentation et la séance d’information du partenaire d’exécution, posez des questions approfondies pour avoir des informations sur les éléments suivants : a. Groupes de soins b. Démonstrations des pratiques appropriées de transformation des aliments et de préparation d’aliments nutritifs c. Identification des enfants malnutris d. Séances de conseil et d’information e. Jardinage f. Élevage g. Soins prénatals h. Planification familiale i. Consultation préscolaire j. Traitement des cas plus graves de malnutrition k. Prise en charge rapide des cas de diarrhée, de respiration accélérée et de fièvre l. Activités de l’approche SBC m. Activités visant à encourager les pères à jouer un rôle de soutien dans la prise de décision du ménage en matière de nutrition et de santé n. Création/support de VSLAs o. Renforcement des systèmes de santé, l’accent étant mis sur l’ICCM p. Genre q. Jeunesse 2. Quelles sont vos responsabilités concernant les activités de nutrition et de santé du projet (nom du projet) ? Quels types de membres de la communauté participent régulièrement à ces activités ? a. Promoteurs de l’hygiène, Mama Leader, relais, CODESA, poser des questions approfondies sur les activités offertes au niveau communautaire, les participants de la communauté, y compris les caractéristiques et le nombre de personnes qui y participent, et les programmations continues liées aux éléments suivants : i. Visites à domicile ii. Séances de conseil et de groupe avec des groupes d’alimentation de l’enfant iii. Démonstrations de pratiques appropriées de transformation des aliments et de préparation d’aliments nutritifs iv. Jardinage v. Détection des enfants malnutris vi. Référencement d’enfants malnutris ou souffrant de diarrhée, de respiration rapide ou de fièvre vii. Collecte des données, remplissage des formulaires mensuels et suivi des services viii. Supervision (pour les promoteurs de santé) ix. Autres activités de l’approche SBC Mid-Term Evaluation of the FSP-Enyanya DFSA in the DRC Annexes 101 b. Pour les travailleurs en centre de santé, vérifiez les services offerts dans le centre de santé. Posez des questions approfondies pour en savoir davantage sur : i. Le traitement des enfants malnutris dans l’établissement de santé ii. La distribution de RUTF/Plumpy Nut aux enfants souffrant de malnutrition plus sévère. iii. Activités de la CPS iv. Séances d’information ou de conseils sur les stratégies de SBC 3. Quelle formation avez-vous reçue dans le cadre des activités de santé et de nutrition ? a. Quelle a été la durée de la formation et où a-t-elle eu lieu ? b. Qui étaient les formateurs ? Pensez-vous qu’ils étaient qualifiés pour dispenser la formation ? Pourquoi ou pourquoi pas ? c. Quelle a été la nature des thématiques abordées au cours de la formation ? Posez des questions approfondies sur le contenu de la formation en mettant l’accent sur les sujets suivants : i. ANJE ii. Visites à domicile iii. Séances de conseil et de groupe avec des groupes d’alimentation de l’enfant iv. Démonstrations de bonnes pratiques de transformation des aliments et de préparation d’aliments nutritifs v. Jardinage vi. Dépistage des enfants malnutris vii. Soins prénatals viii. Planification familiale ix. Consultation prescolaire x. Référencements d’enfants malnutris ou souffrant de diarrhée, de respiration rapide ou de fièvre xi. Traitement des cas plus graves de malnutrition xii. Collecte des données, remplissage des formulaires mensuels et suivi des services xiii. Supervision (pour les promoteurs de santé) xiv. Approches axées sur le changement de comportement xv. Genre d. Pensez-vous que la formation a été suffisamment longue et adéquate pour que vous puissiez mener à bien les activités de nutrition infantile comme prévu ? Dans la négative, quelles étaient certaines des lacunes de la formation ? Quels sujets manquaient et devraient être modifiés ou ajoutés à la formation ? e. Y a-t-il eu un suivi de la formation ou du recyclage ? Dans la négative, estimez-vous qu’une formation de suivi soit nécessaire ? f. Êtes-vous satisfait de l’ensemble de la formation ? Pourquoi ou pourquoi pas ? 4. Quels matériels et fournitures avez-vous reçus en rapport avec les activités de nutrition et de santé depuis le début du projet (nom du projet) ? (Demander à l’informateur de partager la documentation). a. À quelle fréquence recevez-vous des fournitures, y compris du matériel didactique, des semences et des outils pour le jardinage ? b. Les matériaux et les fournitures sont-ils suffisants pour mener à bien les activités du projet ? Veuillez expliquer. c. Vous arrive-t-il de manquer d’outils et de fournitures ? Si oui, à quelle fréquence ? Que fait-on fait lorsque cela arrive ? IMPEL | Implementer-Led Evaluation and Learning 102 Annexes d. Êtes-vous satisfait du matériel et des fournitures reçus ? Pourquoi ou pourquoi pas ? 5. Quel type de supervision recevez-vous dans le cadre des activités de nutrition infantile du projet (nom du projet) ? a. Qui supervise vos activités professionnelles ? b. À quelle fréquence la supervision est-elle censée avoir lieu et à quelle fréquence a-t-elle réellement lieu ? c. Quand les visites de supervision ont-elles lieu, combien de temps durent-elles ? Que se passe-t-il pendant les visites de supervision ? Dans quelle mesure interagissez-vous avec votre superviseur pendant ces visites ? d. Êtes-vous satisfait de la supervision ? Pourquoi ou pourquoi pas ? e. Quel autre soutien, au-delà de la supervision, le projet vous apporte-t-il ? Posez des questions approfondies sur le mentorat, le coaching, etc. Et s’ils ont reçu des outils en cours d’emploi pour les aider dans leur travail.] 6. Comment assurez-vous le suivi des activités en cours ? Posez des questions approfondies pour en savoir davantage sur : a. Les formulaires à remplir b. La fréquence et le mode de transmission des formulaires aux bureaux des superviseurs/du partenaire d’exécution 7. Nous savons que le genre est inclus dans le projet (nom du projet) et que FFP est en faveur de l’intégration du genre dans les différents secteurs. Pouvez-vous nous dire quelles approches vous utilisez en matière de genre et comment elles sont intégrées dans les activités de santé et de nutrition ? Comment les activités liées au genre sont-elles intégrées aux autres objectifs, tels que les moyens de subsistance, la gouvernance et le WASH ? En quoi le ciblage assure-t-il ou n’assure-t-il pas l’intégration entre les objectifs ? Que pensez-vous de ces approches ? Comment ces interventions se mesurent-elles et quelles sont vos évaluations de l’efficacité de ces activités ? Comment ont-elles été accueillies par les membres de la communauté ? Pouvez￾vous nous faire part de quelques réussites et des difficultés rencontrées dans la mise en œuvre de ces activités ? A votre avis, comment les approches genre peuvent-elles être améliorées ? 8. Nous savons que la jeunesse est incluse dans le projet (nom du projet) et que FFP est en faveur de l’intégration des jeunes dans les différents secteurs. Pouvez-vous nous dire quelles sont les approches que vous utilisez à l’égard des jeunes et comment elles sont intégrées dans les activités de santé et de nutrition ? Comment les activités liées au genre sont-elles intégrées aux autres objectifs, tels que les moyens de subsistance, la gouvernance et le WASH ? En quoi le ciblage assure-t-il ou n’assure-t-il pas l’intégration entre les objectifs ? Que pensez-vous de ces approches ? Comment ces interventions se mesurent-elles et quelles sont vos évaluations de l’efficacité de ces activités ? Comment ont-elles été accueillies par les membres de la communauté ? Pouvez-vous nous faire part de quelques réussites et des difficultés rencontrées dans la mise en œuvre de ces activités ? A votre avis, comment les approches genre peuvent￾elles être améliorées ? 9. Quels sont les facteurs qui vous motivent à mener des activités liées à la nutrition et à la santé dans le cadre du projet ? (Posez des questions approfondies sur les indemnités journalières, l’avancement professionnel, les conditions de travail, la reconnaissance communautaire et les autres avantages non financiers). Que pensez-vous de la façon dont on vous motive à mener des activités ? Avez-vous des documents d’identification officiels qui indiquent que vous travaillez sur le projet de (nom du projet) ? Dans l’ensemble, êtes-vous satisfait de votre rôle ? Pourquoi ou pourquoi pas ? 10. Avec qui collaborez-vous dans le cadre des activités en cours dans le domaine de la nutrition et de la santé ? Comment fonctionne la coordination des activités sur le terrain ? Quels systèmes, Mid-Term Evaluation of the FSP-Enyanya DFSA in the DRC Annexes 103 le cas échéant, ont été mis en place pour partager les enseignements tirés ou les difficultés rencontrées sur le terrain ? En cas de difficultés, comment les surmonter ? Veuillez décrire toutes les réunions de coordination qui ont lieu actuellement. 11. Dans quelle mesure êtes-vous en mesure de mettre en œuvre les services de nutrition et de santé conçus dans le cadre de l’approche du projet ? Quels sont les défis à relever ? Y a-t-il des aspects de l’approche que vous n’êtes pas en mesure de mettre en œuvre ? Si oui, pourquoi ? Comment les surmonter ? 12. Par rapport aux stratégies précédentes auxquelles vous avez participé, comment l’approche a-t￾elle changé la façon dont les mères et les autres donneurs de soins s’occupent de leurs enfants ? Selon vous, l’approche donne-t-elle aux mères, aux autres donneurs de soins et aux membres de la communauté les moyens de s’occuper de leurs enfants ? 13. Quelle est votre évaluation globale des activités de nutrition et de santé dans le cadre de l’approche du projet (nom du projet) ? Êtes-vous satisfait des activités ? Pourquoi ou pourquoi pas ? Les gens de votre communauté sont-ils satisfaits des activités ? 14. Selon vous, quelle est la tendance générale de la malnutrition infantile de votre zone de couverture ? Est-elle à la hausse ou à la baisse ou n’y a-t-il aucun changement ? Qu’est-ce qui explique cette tendance ? 15. Que recommanderiez-vous pour améliorer la nutrition et la santé de l’enfant dans le cadre des activités de DFSA ? Entretien Approfondie Couples Modeles 1. Décrivez vos/votre rôle(s) du couple modèle. Depuis combien de temps avez-vous été impliqué dans les activités du Couple Modèle ? 2. Quels sont les activités principales du couple modèle ? 3. Décrivez-nous les services disponibles pour faire le planning familial dans la communauté. Quels obstacles existent pour accéder aux services et méthode de PF ? Comment êtes-vous choisi/sélectionné(e) pour participer ? 4. Décrivez les thématiques/modules que vous aborder lors des discussions avec les autres membres de la communauté ? Avez-vous convaincu les autre couple a faire le PF ? Si oui, combien y a-t-il de personnes avez-vous convaincu pour faire le PF ? Combien d’hommes ? Combien de femmes ? 5. Quelle formation avez-vous reçue pour participer aux activités du couple modèle ? De la part de qui, et sur quels thématiques ? 6. Si vous rencontrez des difficultés, à qui vous adressez-vous pour obtenir de l’aide ? Quel est votre relation/lien avec le centre de santé ? Entretiens approfondi Leaders Religieux 1. Quelle formation avez-vous reçue pour participer aux initiatives PPC/CoH/SS ? De la part de qui, et sur quels thématiques ? 2. Si vous rencontrez des difficultés, à qui vous adressez-vous pour obtenir de l’aide ? 3. Décrivez les rôles de chaque membre de votre comité ? Combien de temps mettez-vous dans votre activité ? Combien y a-t-il de membres ? combien d’hommes ? Combien de femmes ? Quelles activités entreprenez-vous pour assurer la pérennité du comité ? (Gouvernance) 4. Comment êtes-vous choisi/sélectionné(e) pour participer ? 5. Avez-vous reçu une quelconque formation pour faciliter votre travail ? 6. Quelles sont les activités que vous êtes censé faire dans la communauté ? IMPEL | Implementer-Led Evaluation and Learning 104 Annexes 7. Comment vous sentez-vous dans l’exercice de vos fonctions aux côtés des autres membres de la communauté ? 8. Combien de membre de votre comité ont participé à des formations sur le genre ? 9. Considérez-vous que votre activité ait influencé des changements positifs dans la communauté ? 10. Selon vous, quels sont les principaux obstacles qui empêchent le projet d’atteindre tous ses objectifs ? 11. Que suggéreriez-vous pour surmonter tous ces obstacles mentionnés précédemment ? Entretien Approfondie : Mentors de l’espace de confiance 1. Depuis combien de temps avez-vous été impliqué dans les activités de l’espace de confiance ? À quelle fréquence assistez-vous aux séances ? [Sonder : si vous n'êtes pas en mesure de participer régulièrement, pourquoi ?] 2. Pouvez-vous me dire comment vous avez été choisi pour participer à cette activité ? [Choisi en raison de l'âge, d'autres critères.] 3. Quels sont les principaux sujets de discussion dans les espaces de confiance ? Quels services vous sont offerts, le cas échéant ? [Sonder : les espaces de confiance sont-ils liés aux filles ? à d'autres services, à la planification familiale, etc. Si non, pourquoi pas ? Ces services ne sont-ils pas offerts dans la collectivité ?] 4. Parmi ces sujets/activités que vous venez de mentionner, lesquelles ont eu un impact positif sur votre vie ? Quels sont ceux qui ont eu les effets les plus positifs ? 5. Quels sont les sujets/activités que vous venez de mentionner qui ont eu un impact négatif sur votre vie ? Pourquoi ou pourquoi pas ? 6. Selon vous, quelle est la principale valeur ajoutée (bénéfice) des espaces de confiance dans votre collectivité ? [Sonder : possibilités de soutien social et de création ou de reconstruction d'un réseau social, possibilité d'acquérir des compétences, façon d'accéder à l'information de façon non stigmatisante ; recevoir de l'information sur la santé/les méthodes de planification familiale/les droits des femmes/etc.] 7. Lorsque vous songez aux défis que vous devez relever en matière de soins de santé, d'éducation, de possibilités économiques ou autres, est-ce que l'Espace de confiance répond à tous vos besoins ? Y a-t-il autre chose que le projet pourrait faire pour vous dans ce domaine ? Y a-t-il des leçons qui devraient être interrompues ou modifiées ? 8. J'aimerais maintenant parler de votre point focal de l’espace de confiance. Pouvez-vous décrire ce qu'il fait pour vous faire sentir les bienvenus et à l’aise ? Savez-vous pourquoi il a été choisi comme animateur ? Est-il quelque chose qu'il pourrait faire pour l’améliorer ? 9. Y a-t-il autre chose que vous voudriez me dire à propos des espaces de confiance ? Discussions de groupe Discussions de groupe avec des donneuses de soins aux jeunes enfants Projet FSP/ENYANYA : Femmes de Voisinage ; Hommes de Voisinage ; Adolescents d’Espace de Confiance 1. Connaissez-vous le projet (nom du projet) ? Que pouvez-vous me dire sur le projet ? Quelles sont les activités menées ? Quelles activités liées à la santé et à la nutrition sont-elles menées ? Activités communautaires 2. Quand les activités de nutrition et de santé ont-elles été introduites dans votre région ? En quoi consistent les activités ? Sur la base de l’examen de la documentation et de la séance Mid-Term Evaluation of the FSP-Enyanya DFSA in the DRC Annexes 105 d’information du partenaire d’exécution, interrogez les personnes concernées sur différentes activités, telles que : a. Réunions des groupes de donneuses de soins b. Démonstrations de pratiques appropriées de transformation des aliments et de préparation d’aliments nutritifs. c. Visites à domicile d. Dépistage des enfants malnutris e. Séances de conseil et d’information f. Jardinage g. L’élevage des animaux h. Soins prénatals i. Planification familiale j. Consultation prescolaire k. Traitement des cas plus graves de malnutrition l. Prise en charge rapide des cas de diarrhée, de respiration rapide et de fièvre m. Activités du SBC n. Activités visant à encourager les pères à jouer un rôle de soutien dans la prise de décision du ménage en matière de nutrition et de santé o. VSLAs 3. Avez-vous participé à des activités sur la nutrition et la santé au cours des derniers mois du projet (nom du projet) ? Si oui, à quelles activités avez-vous participé ? Sur la base de l’approche projet, recherchez des activités liées à la nutrition et à la santé telles que : a. Réunions des groupes de donneuses de soins b. Démonstrations de pratiques appropriées de transformation des aliments et de préparation d’aliments nutritifs. c. Visites à domicile d. Identification des enfants malnutris e. Séances de conseil et d’information f. Jardinage g. L’élevage des animaux h. Soins prénatals i. Planification familiale j. Consultation prescolaire k. Traitement des cas plus graves de malnutrition l. Activités de l’approche SBC m. VSLA ou autres programmes de crédit 4. Pour chaque activité mentionnée à la question 3, posez les questions suivantes : a. Qui a dirigé l’activité b. Quelles informations ont été partagées pendant les sessions c. A-t-on utilisé du matériel éducatif pour transmettre des messages ? d. Dans l’affirmative, le matériel éducatif a-t-il amélioré ou diminué votre compréhension des messages ? e. Les séances ont-elles été utiles ? f. En quoi les séances n’ont-elles pas /ont-elles été utiles ? g. En quoi ces séances ont-elles changé vos pratiques ? h. L’une ou l’autre des pratiques modifiées cause-t-elle des problèmes ou entraîne-t-elle des conséquences inattendues ? i. Spécifique aux groupes de donneuses de soins : IMPEL | Implementer-Led Evaluation and Learning 106 Annexes j. Quand avez-vous reçu une visite à domicile pour la dernière fois ? k. De quoi a-t-on discuté ? l. Etait-ce pertinent pour vous ? m. Quand la mère leader vient chez vous, que fait-elle ? Existe-t-il autre chose que la diffusion de messages ? (C.-à-d. observation, conseils sur des questions précises, etc.) n. Pourquoi recevez-vous des visites à domicile ? Pendant combien de temps recevrez￾vous des visites à domicile ? Aussi : 43 + 44 Adolescents d’Espace de Confiance 1. Avez-vous participé à d’autres réunions ou séances dans la communauté (recherche d’approches utilisées dans l’approche SBC) où des informations sur la nutrition et la santé ont été fournies ? Dans l’affirmative, quelles informations ont-elles été transmises ? Avez-vous trouvé ces séances instructives ? Pourquoi ou pourquoi pas ? Et d’autres activites du projet (AVEC, gouvernance, agriculture, etc.) 2. Avez-vous récemment écouté des informations à la radio sur la nutrition et la santé ? Dans l’affirmative, quelles informations ont été fournies ? Quels sont les messages radiophoniques qui vous ont motivé à faire quelque chose de différent ? Pourquoi ou pourquoi pas ? Avez-vous discuté des messages radio avec d’autres membres de la famille ou de la communauté ? 3. Les pratiques alimentaires de votre enfant ont-elles changé depuis que vous avez commencé à participer aux activités du projet ? Dans l’affirmative, quels changements spécifiques ont-ils été apportés à l’alimentation ? Pourquoi avez-vous opéré ces changements ? Selon vous, en quoi ces changements ont-ils affecté la santé de votre enfant ? Posez des questions approfondies sur les changements relatifs à : a. L’allaitement maternel b. La transformation et la préparation des aliments c. L’alimentation complémentaire, y compris d. Les aliments donnés au jeune enfant e. Le nombre de repas quotidiens f. Le lieu où on se procure les aliments complémentaires 4. Existe-t-il des sites de soins communautaires dans votre région ? Si oui, avez-vous eu recours à un site de soins communautaires ? Si oui, dans quelles circonstances ? Quels types de soins ont-ils été fournis ? Avez-vous été satisfait des soins prodigués ? Pourquoi ou pourquoi pas ? Interview Approfondie concernant les activités menées dans un établissement de santé Projet FSP/ ENYANA : Femmes de Voisinage Soins prénatals 1. Avez-vous reçu des soins prénatals(ANC) avant votre dernier accouchement ? Pourquoi ou pourquoi pas ? 2. À quelle étape de votre grossesse avez-vous reçu des soins prénatals pour la première fois ? Pourquoi avez-vous commencé les soins prénatals au moment précis où vous l’avez fait ? 3. Combien de séances de soins prénatals avez-vous suivies pendant votre dernière grossesse ? 4. Comment avez-vous décidé de vous présenter aux soins prénatals ? Qui a participé au processus décisionnel ? Quel rôle votre mari ou votre partenaire a-t-il joué dans la décision de suivre des soins prénatals ? 5. Comment décririez-vous les soins prénatals fournis ? Mid-Term Evaluation of the FSP-Enyanya DFSA in the DRC Annexes 107 6. Qu’avez-vous appris pendant les soins prénatals ? Quel est, selon vous, le but des soins prénatals ? 7. Avez-vous été satisfait de l’ensemble des services fournis pendant les soins prénatals ? Pourquoi ou pourquoi pas ? Qu’est-ce qui vous a particulièrement plu ? Qu’est-ce que vous n’aviez pas apprécié, le cas échéant ? 8. Quelles recommandations, le cas échéant, feriez-vous pour améliorer les soins prénatals fournis ? 9. Après votre accouchement, avez-vous reçu des informations sur la planification familiale ? Dans l’affirmative, quels renseignements avez-vous reçus ? 10. Avez-vous reçu d’autres informations sur la planification familiale ? Si oui, où ? 11. Votre mari participe-t-il à des activités liées aux relations conjugales et à la prise de décisions au sein du ménage ? Visites de puériculture/Soins postnatals 1. Avez-vous bien assisté aux visites chez la puéricultrice avec votre jeune bébé ? Pourquoi ou pourquoi pas ? 2. Si les visites chez la puéricultrice ont été bien suivies, a. À quelle fréquence assistez-vous à ces séances chez la puéricultrice ? b. Quand avez-vous commencé à emmener votre enfant en visite ? c. Est-ce que vous continuez d’emmener votre enfant en visite chez la puéricultrice ? Si vous avez arrêté, pourquoi avez-vous cessé d’y assister ? 3. Pourquoi avez-vous décidé d’emmener votre enfant en visite chez la puéricultrice ? Qui a participé au processus de prise de décision ? Quel rôle votre mari ou votre partenaire a-t-il joué dans la décision de se présenter à la CPS ? 4. Comment les services sont-ils fournis pendant la visite à la CPS ? 5. Qu’avez-vous appris pendant les séances ? À votre avis, quel est le but des visites chez la puéricultrice ? 6. Avez-vous été satisfait de l’ensemble des services fournis pendant la visite chez la puéricultrice ? Pourquoi ou pourquoi pas ? Qu’est-ce qui vous a particulièrement plu ? Qu’est-ce que vous n’avez pas apprécié, le cas échéant ? 7. Quelles recommandations, le cas échéant, feriez-vous pour améliorer les visites des bébés chez la puéricultrice ? Autres questions de Sant’e de l’Enfant 1. Quand votre jeune enfant a-t-il eu pour la dernière fois de la fièvre, de la toux ou de la diarrhée ? Où avez-vous emmené votre enfant pour le faire soigner ? 2. Pourquoi avez-vous emmené votre enfant chez ce prestataire (réponses à la question 1) ? Quels ont été les principaux facteurs qui ont guidé votre décision ? Posez des questions approfondies sur les points suivants : a. Disponibilité des services et de l’équipement médical b. Coût des services c. Établissement/infrastructure de santé d. Relation avec le prestataire de soins de santé e. Qualité des soins de santé 3. Qui a participé au processus décisionnel ? Comment s’est déroulée la prise de décision concernant la recherche d’un traitement ? Quel rôle votre mari ou votre partenaire a-t-il joué dans la décision de consulter un professionnel de la santé ? IMPEL | Implementer-Led Evaluation and Learning 108 Annexes 4. Quelle a été votre évaluation globale du traitement fourni ? Avez-vous été satisfait du traitement fourni ? Pourquoi ou pourquoi pas ? Qu’est-ce qui vous a particulièrement plu ? Qu’est-ce que vous n’avez pas apprécié, le cas échéant ? 5. Le cas échéant, quelles recommandations feriez-vous pour améliorer les services de soins fournis ? 6. Comment votre mari participe-t-il à la prise de décision concernant les décisions familiales, y compris celles liées à la santé et à la garde des enfants dans votre ménage ? Comment, le cas échéant, son rôle a-t-il changé depuis le début du projet (nom du projet) ? Que signifie pour vous le terme « malnutri » ? Posez les questions suivantes : 1. Existe-t-il un mot local pour désigner la malnutrition ? 2. Quels sont les signes et symptômes de la malnutrition chez les jeunes enfants ? 3. Qu’est-ce qui cause la malnutrition chez un enfant ? 4. Comment traite-t-on la malnutrition ? 5. Comment prévenir la malnutrition ? 6. Comment essayez-vous de prévenir la malnutrition ? Entretiens avec les informateurs clés Entretiens avec les informateurs clés à l’aide de question sur la nutrition et la santé posées aux Représentants des organisations partenaires, organismes publiques (au niveau national ou régional) FSP/ENYANYA : PRONANUT ; Infirmier Superviseur de Nutrition ; Medicin Chef de Zone 1. Quand votre organisation/agence a-t-elle commencé à travailler avec le projet ? Quel est le rôle général que votre organisation/agence assume dans le projet ? [Pour le gouvernement : quel sentiment d’appropriation ressentez-vous à l’égard de ce projet ?] 2. Comment décririez-vous les buts et objectifs généraux du projet (nom du projet) ? 3. Y a-t-il eu des changements (prévus ou imprévus) dans la mise en œuvre des activités de nutrition et de santé depuis le début du projet (nom du projet) ? Dans l’affirmative, pourquoi ces changements ont-ils été apportés ? Posez des questions approfondies sur les changements intervenus dans les domaines relatifs à : a. La programmation des activités b. L’approvisionnement en matériel et fournitures c. La formation d. La supervision des travailleurs e. Les changements dans le suivi et l’évaluation des services f. La rémunération ou autres formes de motivation. Posez des questions approfondies sur les différents types de personnel, comme par exemple les : i. Superviseurs/animateurs ii. Promoteurs d’hygiène iii. Mama leaders 4. À quelles activités de nutrition et de santé votre organisation/agence participe-t-elle ? En vous basant sur l’examen de la documentation et la séance d’information du partenaire d’exécution, examinez les éléments suivants : a. Groupes de donneurs de soins Mid-Term Evaluation of the FSP-Enyanya DFSA in the DRC Annexes 109 b. Démonstrations de pratiques appropriées de transformation et de préparation d’aliments nutritifs c. Identification des enfants malnutris d. Séances de conseil et d’information e. Jardinage f. L’élevage g. Soins prénatals h. Planification familiale i. Consultation prescolaire j. Traitement de la malnutrition aiguë sévère k. Prise en charge rapide des cas de diarrhée, de respiration accélérée et de fièvre l. Autres activités de la stratégie de SBC m. Activités visant à encourager les pères à jouer un rôle de soutien dans la prise de décision du ménage en matière de nutrition et de santé n. Création/support de VSLAs o. Renforcement des systèmes de santé, l’accent étant mis sur les ICCM 5. Quelles sont les activités de nutrition et de santé de l’enfant qui sont réellement offertes par l’intermédiaire du projet (nom du projet) ? Posez des questions approfondies pour comprendre dans quelle mesure les activités censées être offertes sont effectivement mises en œuvre. 6. Votre organisation/agence a-t-elle participé à l’une quelconque des études « R&I » au cours de la première année de la mise en œuvre ? (Posez des questions approfondies pour obtenir des informations sur chacune des études de recherche formative portant sur la nutrition et la santé). De quelle manière la recherche formative a-t-elle été utilisée pour l’élaboration et la mise en œuvre des programmes ? 7. (DESTINÉ AUX PARTENAIRES SEULEMENT) Veuillez décrire la théorie du changement (TDC) qui est suivie pour l’ensemble du projet. Dans quelle mesure vous référez-vous à la TDC au fur et à mesure que les activités se poursuivent ? À quoi ressemblait la TDC au début des activités du projet ? À quoi ressemble la TDC à l’heure actuelle et, si des changements ont été apportés, pourquoi ces changements ont-ils été apportés ? 8. Quels groupes et organisations partenaires participent aux activités de nutrition et de santé ? De quelle manière les différents partenaires sont-ils impliqués ? Quelles organisations sont responsables de quelles activités ? 9. (DESTINÉ AUX PARTENAIRES SEULEMENT) Quels sont les acteurs clés de votre organisation sur le terrain en charge de la mise en œuvre des activités de nutrition et de santé dans le cadre de ce projet ? Comment le personnel a-t-il été recruté ? Posez des questions approfondies pour en savoir davantage sur l’implication d’acteurs de terrain tels que a. Mama leaders b. Promoteurs d’hygiène c. Superviseurs et facilitateurs d. Membres du CODESA e. Relais f. Les fournisseurs de services basés sur les établissements IMPEL | Implementer-Led Evaluation and Learning 110 Annexes g. Autres 10. Quels types de motivations sont-elles offertes aux différentes catégories de personnel ? Quelles sont les structures de supervision mises en place pour assurer le suivi des activités ? Comment, concrètement, les différentes catégories de personnel sont-elles supervisées ? Comment le personnel est-il évalué de façon continue ? 11. (DESTINÉ AUX PARTENAIRES SEULEMENT) Veuillez décrire toute capacité technique qui a permis de renforcer les capacités du partenaire technique en chef/principal qui vous a été fourni depuis le début du projet. [Posez des questions approfondies sur la stratégie de SBC] Quels autres domaines techniques aimeriez-vous connaître, s’il y a lieu ? 12. Quelles ont été les premières étapes de la mise en œuvre des activités ? Quelles sont les étapes ultérieures qui ont été réalisées au cours du déroulement des activités ? Quand l’ensemble des activités ont-elles commencé ? 13. Quelles sont les approches de changement social et comportemental qui sont mises en œuvre dans le cadre de l’approche ? Comment les approches de la stratégie de SBC ont-elles été élaborées ? [Question approfondie : comment avez-vous été impliqué ?] Comment les activités de l’approche SBC sont-elles suivies ou supervisées ? {Posez des questions approfondies sur la liste de contrôle ou d’autres outils.} Quels changements, le cas échéant, ont-ils été apportés aux activités de l’approche SBC depuis le début des activités du projet ? Si des changements ont été apportés, expliquez pourquoi. Comment le projet assure-t-il la qualité des activités de l’approche SBC, ainsi que la capacité du personnel et la capacité des mères leaders, etc. Comment les messages de l’approche SBC sont-ils séquencés ? Comment les messages de la stratégie de l’approche SBC sont-ils adaptés aux différents groupes/publics cibles ? [Vérifier si le matériel et les messages du SBC sont différents pour les pères, les hommes, les adolescents, les grands￾parents, etc.] Veuillez décrire le rythme des activités de l’approche SBC, par exemple, quelle est la fréquence 1) des rencontres/interactions avec les mères et 2) des événements radio et représentations théâtrales ? 14. Pour le projet Mercy Corps : Nous avons remarqué que l’une des principales activités de votre projet est la formation de divers personnels de santé, y compris la formation sur la planification familiale. Pouvez-vous décrire de quelle manière le projet assure la qualité de la mise en œuvre par le personnel de santé et les agents de santé communautaires, au-delà de la formation en salles de classe ? [Posez des questions approfondies sur le mentorat, la supervision, l’accompagnement.] Quels obstacles le projet a-t-il rencontrés dans la mise en œuvre des activités de formation et des activités de l’approche SBC axées sur la planification familiale ainsi que sur la planification et l’espacement sains de la grossesse ? 15. Quels sont les systèmes de suivi et évaluation disponibles pour mesurer les activités en cours ? Comment ces systèmes sont-ils utilisés pour éclairer la mise en œuvre du programme ? 16. Y a-t-il eu des changements contextuels (p. ex. des questions liées à la sécurité, aux structures du gouvernement national et des collectivités locales, à l’environnement, à l’économie locale, etc.) depuis le début du projet qui ont pu avoir une incidence sur ses activités ? Dans l’affirmative, comment sont-elles affectées par les activités du projet ? 17. Comment décririez-vous la coordination et la collaboration avec d’autres organismes partenaires, des institutions gouvernementales et des collaborateurs locaux ? Comment fonctionne la coordination et la planification : Posez des questions approfondies sur : a. La programmation des activités Mid-Term Evaluation of the FSP-Enyanya DFSA in the DRC Annexes 111 b. Les réunions avec les organisations partenaires c. Le suivi et évaluation des activités d. Les systèmes mis en place pour apprendre/partager les activités en cours, les leçons apprises et les défis sur le terrain e. D’autres acteurs potentiels susceptibles de présenter un intérêt comprennent : Projet PROSANI/PHI, UNICEF, PAM, autres activités sanitaires et nutritionnelles. 18. Nous savons que la durabilité est l’un des principaux objectifs des projets DSFA. Comment mesurez-vous la durabilité ? Quelles preuves, le cas échéant, de la durabilité produite par les activités du projet sont disponibles à ce jour ? Selon vous, que faut-il faire pour accroître la probabilité de durabilité lorsque l’aide du DFSA arrivera à terme ? 19. Dans quelle mesure le projet a-t-il atteint les objectifs liés aux indicateurs clés de nutrition et de santé ? Posez des questions approfondies sur : a. L’autonomisation des mères dans la prise de décision en matière de nutrition et de santé b. L’identification et la prise en charge des enfants malnutris c. La disponibilité et utilisation accrues d’aliments nutritifs pour l’alimentation des enfants d. La réduction de la prévalence de la malnutrition 20. Si les objectifs du projet ne sont pas atteints, est-ce dû à des échecs liés à la conception ou à la mise en œuvre du programme ? 21. Quels sont certains des obstacles à l’atteinte de ces indicateurs de résultats clés ? Pouvez-vous décrire les types de ressources et d’assistance technique que le projet, les bureaux régionaux ou l’administration centrale vous ont fournis depuis que vous avez travaillé sur le projet ? 22. Quelles sont certaines des faiblesses de l’approche du projet (nom du projet) ? Quels sont les principaux obstacles qui empêchent le projet (nom du projet) d’atteindre son plein potentiel ? Quelles sont les solutions pour surmonter ces obstacles ? 23. Quelles sont vos recommandations pour améliorer les activités futures en matière de nutrition et de santé au cours du projet ? Entretiens avec les informateurs clés Entretiens avec les informateurs clés à l’aide de questions NUTRITION/SANTÉ posées aux Chef de projet (COPs), référents techniques FSP Project/ ENYANYA: Charge de Nutrition World Vision + Mercy Corps; Charge de SBCC; Conseiller Genre et Jeunesse; Charge de Nutrition/Sante (Zone de Sante) 1. Quand avez-vous rejoint le projet ? Quel type de formation ou d’orientation avez-vous reçu depuis le début du projet ? 2. Comment décririez-vous les buts et objectifs généraux du projet (nom du projet) ? 3. Quels sont les principaux indicateurs de résultats liés à la nutrition et à la santé ? 4. Y a-t-il eu des changements (prévus ou imprévus) dans la mise en œuvre des activités de nutrition et de santé depuis le début du projet (nom du projet) ? Dans l’affirmative, pourquoi ces changements ont-ils été apportés ? Posez des questions approfondies sur les changements intervenus dans les domaines relatifs à : IMPEL | Implementer-Led Evaluation and Learning 112 Annexes a. La programmation des activités b. L’approvisionnement en matériel et fournitures c. La formation d. La supervision des travailleurs e. Aux changements dans le suivi et l’évaluation des services f. La rémunération ou autres formes de motivation. Posez des questions approfondies sur les différents types de personnel, comme par exemple les : i. Superviseurs/animateurs ii. Promoteurs d’hygiène iii. Mama leaders g. Le renouvellement du personnel [Est-ce que chaque membre du personnel a une description de poste claire ?] 5. Quelles activités de recherche formative ont-elles été menées avant la mise en œuvre des activités de nutrition et de santé ? Comment les décisions concernant les activités de recherche formative à mener ont-elles été prises ? Quelle est votre évaluation de la qualité et du contenu de la recherche formative ? (Posez des questions approfondies pour obtenir des informations sur chacune des études de recherche formative portant sur la nutrition et la santé). Comment la recherche formative a-t-elle été utilisée dans l’élaboration et la mise en œuvre des programmes ? (Posez des questions approfondies pour déterminer si le projet a élaboré une stratégie de SBC et dans quelle mesure cette stratégie peut ou ne peut pas être utilisée.) 6. Dans quelle mesure le Service de R&I a-t-il été utilisé au-delà de la période de perfectionnement initiale et pendant la phase de mise en œuvre ? Dans quelle mesure avez-vous utilisé l’approche du Service de R&I dans le cadre de la mobilisation et du développement communautaires ? Avez￾vous observé des difficultés à mettre en œuvre l’approche R&I ? Citez-en quelques-unes. 7. Quelles activités de nutrition et de santé sont censées être offertes par l’intermédiaire du projet (nom du projet) ? En vous basant sur l’examen de la documentation et la séance d’information du partenaire d’exécution, examinez les éléments suivants : a. Groupes de donneurs de soins/ Groupes des pères b. Démonstrations de pratiques appropriées de transformation et de préparation d’aliments nutritifs c. Identification, référencement et suivi des enfants malnutris d. Séances de conseil et d’information e. Jardinage f. L’élevage g. Soins prénatals h. Planification familiale et des soins de santé et espacement des grossesses i. Consultation prescolaire j. Traitement de la malnutrition aiguë sévère k. Prise en charge rapide des cas de diarrhée, de respiration accélérée et de fièvre l. Autres activités de la stratégie de SBC, p. ex. les mass media/radio, théâtre/spectacles, mobilisation communautaire Mid-Term Evaluation of the FSP-Enyanya DFSA in the DRC Annexes 113 m. Activités visant à encourager les pères à jouer un rôle de soutien dans la prise de décision du ménage en matière de nutrition et de santé n. Création/support de VSLAs o. Renforcement des systèmes de santé, l’accent étant mis sur l’ICCM p. Mercy Corps : Channels of Hope et Positive Peer Couples q. Santé des jeunes et des adolescents/espaces d’accueil sûrs 8. Quelles sont les activités de nutrition et de santé de l’enfant qui sont réellement menées à bien ? Santé des jeunes et des adolescents/espaces d’accueil sûrs 9. Quelles sont les activités de nutrition et de santé de l’enfant qui sont réellement offertes par l’intermédiaire de (nom du projet) ? Sonder pour comprendre dans quelle mesure les activités censées être offertes sont effectivement mises en œuvre. Vérifier si le plan de mise en œuvre détaillé est pratique ou non. 10. Nous avons remarqué que tous les villages ne mettent pas en œuvre les mêmes activités. Veuillez nous faire savoir comment sont prises les décisions concernant les activités qui sont mises en œuvre et celles qui ne le sont pas. 11. Veuillez décrire la théorie du changement (TDC) qui est suivie pour l’ensemble du projet. Dans quelle mesure vous référez-vous à la TDC en tant qu’activités en cours ? À quoi ressemblait la TDC au début des activités du projet ? À quoi ressemble la TDC à l’heure actuelle et, si des changements ont été apportés, pourquoi ces changements ont-ils été apportés ? 12. Quels groupes et organisations partenaires participent aux activités de nutrition et de santé ? Comment les différents partenaires sont-ils impliqués ? Quelles organisations sont responsables de quelles activités ? 13. Qui sont les acteurs clés sur le terrain en charge de la mise en œuvre des activités de nutrition et de santé ? Comment le personnel a-t-il été recruté ? Posez des questions approfondies pour comprendre le degré d’implication d’acteurs de terrain tels que les : a. Mama leaders b. Promoteurs d’hygiène c. Superviseurs et facilitateurs d. Membres du CODESA e. Relais f. Les fournisseurs de services basés sur les établissements g. Autres 14. Quels types de motivations sont-elles offertes aux différentes catégories de personnel ? Quelles sont les structures de supervision mises en place pour assurer le suivi des activités ? Comment, concrètement, les différentes catégories de personnel sont-elles supervisées ? Comment le personnel est-il évalué de façon continue ? 15. Quelles ont été les premières étapes de la mise en œuvre des activités ? Quelles sont les étapes ultérieures qui ont été réalisées au cours du déroulement des activités ? Quand l’ensemble des activités ont-elles commencé ? 16. Nous savons que les DFSA incluent de nombreux thèmes transversaux et que FFP souhaite que les différents secteurs soient intégrés. Veuillez décrire la manière dont le projet aborde les thèmes transversaux, la manière dont les activités de santé et de nutrition sont intégrées aux IMPEL | Implementer-Led Evaluation and Learning 114 Annexes autres objectifs, Dans quelle mesure le ciblage garantit/ne garantit pas leur intégration entre objectifs, comment les interventions sont évaluées, et vos évaluations de l’efficacité de ces activités. Il serait utile d’en apprendre davantage sur les réussites et les difficultés rencontrées dans la mise en œuvre de ces activités. Posez des questions approfondies pour obtenir des informations relatives à : a. Genre et équité b. Jeunesse c. Atténuation du changement climatique 17. Y a-t-il une personne responsable de la stratégie de SBC au sein du projet ? Si oui, posez les questions suivantes à la personne responsable : Combien de comportements le projet tente-t-il de changer ? Y a-t-il eu des discussions sur la priorisation des comportements ? Dans l’affirmative, comment cela a-t-il été fait ? Combien de membres du personnel connaissent la stratégie de SBC ? Dans quelle mesure le personnel a-t-il participé à l’élaboration de la stratégie de SBC ? Dans quelle mesure le projet parvient-il à bien cerner les obstacles et les facteurs facilitant ou catalyseurs des comportements considérés comme prioritaires ? La stratégie de SBC comprend-elle une combinaison d’approches autres que la communication interpersonnelle et les médias de masse ? [Posez des questions approfondies sur la mobilisation communautaire et le plaidoyer et d’autres approches non liées à la communication.] 18. Quelles sont les approches de changement social et comportemental qui sont mises en œuvre dans le cadre de la stratégie de la stratégie de SBC et/ou de la théorie du changement du projet ? Comment les approches de SBC ont-elles été élaborées ? Comment les activités de la stratégie de SBC sont-elles suivies ou supervisées ? Posez des questions approfondies sur la liste de contrôle ou d’autres outils. Quels changements, le cas échéant, ont été apportés aux activités de la stratégie de SBC depuis le début des activités du projet ? Si des changements ont été apportés, expliquez pourquoi. Comment le projet assure-t-il la qualité des activités de la stratégie de SBC, ainsi que la capacité du personnel et la capacité des mères leaders, etc. Comment les messages SBC sont-ils séquencés ? Comment les messages de la stratégie de SBC sont-ils adaptés aux différents groupes/publics cibles ? Vérifier si le matériel et les messages de la stratégie de SBC sont différents pour les pères, les hommes, les adolescents, les grands-parents, etc. Veuillez décrire le rythme des activités de la stratégie de SBC, par exemple, quelle est la fréquence 1) des rencontres/interactions avec les mères et 2) des événements radio et représentations théâtrales? 19. Pour le projet Mercy Corps : Nous avons remarqué que l’une des principales activités de votre projet est la formation de divers personnels de santé, y compris la formation sur la planification familiale. Pouvez-vous décrire de quelle manière le projet assure la qualité de la mise en œuvre par le personnel de santé et les agents de santé communautaires, au-delà de la formation en salles de classe ? Quels obstacles le projet a-t-il rencontrés dans la mise en œuvre des activités de formation et des activités de SBC axées sur la planification familiale ainsi que sur la planification et l’espacement sains de la grossesse ? 20. Quels sont les systèmes de suivi et d’évaluation disponibles pour mesurer les activités en cours en matière de nutrition et de santé ? Comment ces systèmes sont-ils utilisés pour éclairer la mise en œuvre du programme ? 21. Y a-t-il eu des changements contextuels (p. ex. des questions liées à la sécurité, aux structures du gouvernement national et des collectivités locales, à l’environnement, à l’économie locale, etc.) depuis le début du projet qui ont pu influencer les activités du projet ? Dans l’affirmative, comment ont-elles affecté les activités du projet ? 22. Comment décririez-vous la coordination et la collaboration avec d’autres organismes partenaires, des institutions gouvernementales et des collaborateurs locaux ? Comment fonctionne la Mid-Term Evaluation of the FSP-Enyanya DFSA in the DRC Annexes 115 coordination et la planification : Posez des questions approfondies sur : a. La programmation des activités b. Les réunions avec les organisations partenaires c. Le suivi et évaluation des activités d. Les systèmes mis en place pour apprendre/partager les activités en cours, les leçons apprises et les défis sur le terrain e. D’autres acteurs susceptibles de présenter un intérêt comprennent : le Projet PROSANI/IHP, l’UNICEF, le PAM, les autres activités dans le domaine de la santé et de la nutrition. 23. Nous savons que la durabilité est l’un des principaux objectifs des projets DSFA. Comment mesurez-vous la durabilité ? Quelles preuves, le cas échéant, de la durabilité produite par les activités du projet sont-elles disponibles à ce jour ? Selon vous, que faut-il faire pour accroître la probabilité de durabilité au terme de l’aide fournie par le projet DFSA ? 24. Dans quelle mesure le projet a-t-il atteint les objectifs liés aux indicateurs clés de nutrition et de santé ? Posez des questions approfondies sur : a. L’autonomisation des mères dans la prise de décision en matière de nutrition et de santé b. L’identification et la prise en charge des enfants malnutris c. La disponibilité et utilisation accrues d’aliments nutritifs pour l’alimentation des enfants d. La réduction de la prévalence de la malnutrition 25. Si les objectifs du projet ne sont pas atteints, est-ce dû à des échecs sur le plan de la conception ou de la mise en œuvre du programme ? 26. Quels sont certains des obstacles qui entravent l’atteinte de ces indicateurs de résultats clés ? 27. Pouvez-vous décrire les types de ressources et d’assistance technique que le projet, les bureaux régionaux ou l’administration centrale vous ont fournies depuis que vous travaillez sur ce projet? 28. Quelles sont certaines des faiblesses de l’approche du projet (nom du projet) ? Quels sont les principaux obstacles qui empêchent le projet (nom du projet) d’atteindre son plein potentiel ? Quelles sont les solutions pour surmonter ces obstacles ? 29. Quelles sont vos recommandations pour améliorer les activités futures en matière de nutrition et de santé au cours du projet ? IMPEL | Implementer-Led Evaluation and Learning 116 Annexes 5.8.4 In-Depth Interview (Natural Leaders) – French Nom du village Age d’interview Aire de santé Intervieweur Date Preneur de notes Genre d’interviewé Langue (Formulaire consentement) 1. Qu’est-ce que vous pouvez me dire sur les activités du projet dans votre village? a. SONDER: Toute activité, détail sur WASH 2. Quel est votre rôle dans le projet ? a. SONDER: Confirmer qu’il s’agit d’un leader naturel qui travaille sur l’ATPC entre autres b. Essayer de savoir s’il/elle fait partie des autres activités du projet 3. Comment avez-vous été sélectionné pour être leader naturel? a. C’est quoi un leader naturel? 4. Quels contacts avez-vous eu avec les agents de terrain de Mercy Corps depuis votre sélection? a. SONDER: Participation dans une formation? 5. Si oui formation, ça s’est passé quand? Qui vous a formé? Qu’est-ce que vous avez appris? Qu’est-ce que vous changerez avec la formation? 6. Est-ce que vous avez reçu des supports du projet pour être un leader naturel ? a. SONDER : guide, manuel, etc b. Si oui, qu’est-ce qu’on vous apprend dans le support ? 7. C’est quoi l’approche ATPC du projet? a. Comment vous êtes impliqué dans l’ATPC et la diffusion des messages WASH dans la communauté ? 8. Quelles sont les latrines typiques d’ici ? a. Est-ce que tout le monde a le même type de latrine ? b. SONDER : Quel type de dalle, superstructure, dispositif de lavage des mains 9. Quelle est la perception ici de l’utilisation des latrines? a. SONDER: différences home/femme (enceinte, allaitante), âge, pauvreté b. SUIVI: Cette perception a-t-elle changé depuis le début du projet? 10. Est-ce que les membres de la communauté ont des défis à construire des latrines? a. SONDER: Il y a combien de latrines actuellement dans le village b. SUIVI: Quels sont les défis? Ou pq il n’y en a pas? 11. Comment trouver les matériaux pour construire une latrine ici? a. SONDER: subvention du gouv? b. SUIVI: Y a-t-il des maçons locaux qui peuvent aider? c. SUIVI: Est-ce possible de prêter de l’argent d’un AVEC, MFI, ou autre source? Les gens ont l’habitude de le faire? 12. Quels types de ressources sont disponibles à la communauté pour réparer ou améliorer leur latrine? a. SONDER: subvention du gouv? b. SUIVI: Y a-t-il des maçons locaux qui peuvent aider? c. SUIVI: Est-ce possible de prêter de l’argent d’un AVEC, MFI, ou autre source? Les gens ont l’habitude de le faire? 13. Quels sont les défis à entretenir les latrines ici? Mid-Term Evaluation of the FSP-Enyanya DFSA in the DRC Annexes 117 a. SONDER: Ces défis sont les mêmes pour tout le monde, ou est-ce qu’il y a certaines personnes qui sont plus touches que les autres (âge, genre, pauvreté) b. SUIVI: Qu’est-ce qu’il faut pour affronter ces défis? 14. Revenons au lavage des mains, c’est commun ici comme pratique ? a. Est-ce que les gens ont plus l’habitude de se laver avec du savon ou des cendres ? b. Est-ce difficile de s’approvisionner du savon ou bien des cendres ? c. Diriez vous que la plupart, la moitié ou un peu de gens peuvent facilement se laver les mains ? i. Est-ce qu’il y a une différence pour les femmes enceintes ou allaitantes ? 15. Et le traitement de l’eau de boisson à la maison, ça se fait ? a. Si oui, avec quoi ? (Chlore, aquatabs, autre) b. Depuis quand les membres de la communauté ont l’habitude de traiter l’eau ? (Est-ce qu’il y a un lien avec le projet) 16. Est-ce qu’il y a des autres organisations qui travaille sur l’ATPC/WASH dans ce village? Dans les villages voisins? 17. Pensez-vous que l’approche ATPC du projet pourrait être pérenne ici? IMPEL | Implementer-Led Evaluation and Learning 118 Annexes 5.8.5 In-Depth Interview Promoteurs WASH – French Nom du village Age d’interview Aire de santé Intervieweur Date Preneur de notes Genre d’interviewé Langue (Formulaire consentement) 1. Quelles sont les activités du projet dans votre village? a. SONDER: Toute activité, détail sur WASH 2. Quel est votre rôle dans le projet ? Depuis quand ? a. SONDER: Confirmer qu’il s’agit d’un promoteur WASH b. Essayer de savoir s’il/elle fait partie des autres activités du projet 3. Comment avez-vous été sélectionné pour être promoteur WASH? a. C’est quoi un promoteur WASH? 4. Quels contacts avez-vous eu avec les agents de terrain de Mercy Corps depuis votre sélection? a. SONDER: Participation dans une formation? Contact régulier avec Mercy Corps ? 5. Si oui formation, ça s’est passé quand? Qui vous a formé? Qu’est-ce que vous avez appris? Qu’est-ce que vous changeriez avec la formation? 6. Est-ce que vous avez reçu des supports du projet pour être un promoteur WASH? a. SONDER : guide, manuel, etc b. Si oui, qu’est-ce qu’on vous apprend dans le support ? 7. C’est quoi l’approche ATPC du projet? a. Comment vous êtes impliqué dans l’ATPC et la diffusion des messages WASH dans la communauté ? 8. Quelles sont les latrines typiques d’ici ? a. Est-ce que tout le monde a le même type de latrine ? b. SONDER : Quel type de dalle, superstructure, dispositif de lavage des mains 9. Quelle est la perception ici de l’utilisation des latrines? a. SONDER: différences home/femme (enceinte, allaitante), âge, pauvreté b. SUIVI: Cette perception a-t-elle changé depuis le début du projet? 10. Est-ce que les membres de la communauté ont des défis à construire des latrines? a. SONDER: Il y a combien de latrines actuellement dans le village b. SUIVI: Quels sont les défis? Ou pq il n’y en a pas? 11. Comment trouver les matériaux pour construire une latrine ici? a. SONDER: subvention du gouv? b. SUIVI: Y a-t-il des maçons locaux qui peuvent aider? c. SUIVI: Est-ce possible de prêter de l’argent d’un AVEC, MFI, ou autre source? Les gens ont l’habitude de le faire? 12. Quels types de ressources sont disponibles à la communauté pour réparer ou améliorer leur latrine? a. SONDER: subvention du gouv? b. SUIVI: Y a-t-il des maçons locaux qui peuvent aider? c. SUIVI: Est-ce possible de prêter de l’argent d’un AVEC, MFI, ou autre source? Les gens ont l’habitude de le faire? 13. Quels sont les défis à entretenir les latrines ici? Mid-Term Evaluation of the FSP-Enyanya DFSA in the DRC Annexes 119 a. SONDER: Ces défis sont les mêmes pour tout le monde, ou est-ce qu’il y a certaines personnes qui sont plus touches que les autres (âge, genre, pauvreté) b. SUIVI: Qu’est-ce qu’il faut pour affronter ces défis? 14. Revenons au lavage des mains, quels sont les 5 moments critiques de se laver les mains? a. C’est commun ici comme pratique ? Est-ce que les gens ont plus l’habitude de se laver avec du savon ou des cendres ? b. Est-ce difficile de s’approvisionner du savon ou bien des cendres ? c. Diriez vous que la plupart, la moitié ou un peu de gens peuvent facilement se laver les mains ? i. Est-ce qu’il y a une différence pour les femmes enceintes ou allaitantes ? 15. Et le traitement de l’eau de boisson à la maison, ça se fait ? a. Si oui, avec quoi ? (Chlore, aquatabs, autre) b. Depuis quand les membres de la communauté ont l’habitude de traiter l’eau ? (Est-ce qu’il y a un lien avec le projet) 16. Est-ce que vous avez participé à l’élaboration d’un plan d’action de développement communautaire ? a. Vous pouvez décrire le processus ? Est-ce qu’il y a des aspects portant sur le WASH ? b. Est-ce que la communauté a commencé à le mettre en œuvre ? 17. Est-ce qu’il y a des autres organisations qui travaille sur l’ATPC/WASH dans ce village? Dans les villages voisins? 18. Quel est le rôle du gouvernement local dans le WASH ? Est-ce que le centre de santé joue un rôle dans le WASH ? Si oui, c’est quoi ? 19. Pensez-vous que l’approche WASH du projet pourrait être pérenne ici? Si oui pourquoi et comment ? IMPEL | Implementer-Led Evaluation and Learning 120 Annexes 5.8.6 Interview Guide for CLD Members/Members of Justice and Peace Commissions – French Introduction Texte sur le consentement éclairé Données générales Date: Zone De Santé: Aire De Santé: Ville ou village: 1. Depuis quand votre groupe (CLD ou CDJP ou autre nom de groupe) a-t-il été formé? 2. Combien de membres avez-vous? Qui sont ces membres? 3. Comment devenez-vous membre du (nom du groupe)? Existe-il des critères à remplir? 4. Comment êtes-vous organisés? 5. Quels sont les principaux conflits de développement auxquels font face les membres de votre communauté? a. Quels conflits avez-vous généralement dans la communauté? Quelles sont les sources les plus communes de ces conflits? Qui sont les principaux acteurs? Quand ont-ils commencé? Ce qui a été fait à ce sujet? b. Qu’en est il de l'accès à la terre pour les femmes et les jeunes? Est-il permis? Dans quelles conditions? Qui est responsable de la répartition des terres au niveau communautaire? 6. Qu'avez-vous commencé à faire pour trouver des solutions par rapport à ces problèmes ? 7. Quels sont les principaux acteurs du développement qui vous ont aidés à aborder ces questions jusqu'à présent? Comment? 8. Quand avez-vous commencé à travailler avec le projet (nom du projet)? 9. Quelles activités avez-vous menées jusqu'à présent avec le personnel ou les partenaires du projet? 10. Quelles formations avez-vous reçues? Quand? Combien de personnes y ont participé? Pourcentage d’hommes? Pourcentage de femmes? 11. Quel suivi (supervision) a été fait de ces formations? Combien de fois? Par qui? 12. En tant que membres du (nom du groupe), quelles activités avez-vous menées (ou effectuez￾vous habituellement dans votre collectivité)? 13. Activité 1 14. Activité 2 15. Activité 3 16. Activité 4 17. Pour chacune de ces activités, Quel est l'objectif visé? Qu'essayez-vous de réaliser? 18. Objectif de L'activité 1 19. Objectif de l'activité 2 20. Objectif de L'activité 3 21. Objectif de l'activité 4 Mid-Term Evaluation of the FSP-Enyanya DFSA in the DRC Annexes 121 22. Depuis que vous avez commencé ces activités, pouvez-vous me parler de trois succès que vous avez obtenus jusqu'à présent (ce que vous avez réalisé) 23. Succès 1 24. Succès 2 25. Succès 3 26. Quels ont été les principaux facteurs de ces succès? Ce qui rendait possible ces succes 27. Quels sont les autres enjeux/défis de développement des membres de votre communauté sont confrontés? 28. Défi 1 29. Défi 2 30. Défi 3 31. Défi 4 32. [relevez chaque défi et posez les cinq questions suivantes:] 33. Pourquoi croyez-vous que [citez un défi] prévaut encore dans votre collectivité? 34. Que faites-vous à ce sujet? 35. Est ce que ça marche? 36. Avez-vous des exemples de changement notoire que vous pouvez partager à ce sujet? 37. Sur quoi recommanderiez-vous que le projet soit axé au cours des prochaines années afin d'accroître votre engagement et votre intérêt à l'égard de ses activités? 38. Recommandation 1 39. Recommandation 2 40. Recommandation 3 41. Avez-vous d'autres commentaires à formuler au sujet du projet dans votre communauté? IMPEL | Implementer-Led Evaluation and Learning 122 Annexes 5.8.7 Focus Group Discussion Guide for Participants 14-18 Years of Age – French Jour : Intervieweur : Zone de santé : Preneur de notes : Aire de Santé : Nombre de participants et sexe : Lieu : 1. Depuis combien de temps avez-vous été impliqué dans les activités de l’espace d’écoute? À quelle fréquence assistez-vous aux séances? [Sonder: si vous n'êtes pas en mesure de participer régulièrement, pourquoi?] 2. Pouvez-vous me dire comment vous avez été choisi pour participer à cette activité? [Choisi en raison de l'âge, d'autres critères.] 3. Quels sont les principaux sujets de discussion dans les espaces d’écoute? Quels services vous sont offerts, le cas échéant? [Sonder: les espaces d’écoute sont-ils liés aux filles ? à d'autres services, à la planification familiale, etc. Si non, pourquoi pas? Ces services ne sont-ils pas offerts dans la collectivité?] 4. Parmi ces sujets/activités que vous venez de mentionner, lesquelles ont eu un impact positif sur votre vie? Quels sont ceux qui ont eu les effets les plus positifs? 5. Quels sont les sujets/activités que vous venez de mentionner qui ont eu un impact négatif sur votre vie? Pourquoi ou pourquoi pas? 6. Est-ce que vous avez appris des messages sur le traitement de l’eau de boisson, l’utilisation des latrines ou bien le lavage des mains dans l’espace d’écoute ? Est-ce que vous connaissez le lien entre ces pratiques et les maladies hydriques ? 7. Selon vous, quelle est la principale valeur ajoutée (bénéfice) des espaces d‘écoute dans votre collectivité? [Sonder: possibilités de soutien social et de création ou de reconstruction d'un réseau social, possibilité d'acquérir des compétences, façon d'accéder à l'information de façon non stigmatisante; recevoir de l'information sur la santé/les méthodes de planification familiale/les droits des femmes/etc.] 8. Lorsque vous songez aux défis que vous devez relever en matière de soins de santé, d'éducation, de possibilités économiques ou autres, est-ce que l'Espace d’écoute répond à tous vos besoins? Ou bien aux risques que vous rencontrez dans votre communauté ? Y a t-il autre chose que le projet pourrait faire pour vous dans ce domaine? Y a-t-il des activités qui devraient être interrompues ou modifiées? 9. J'aimerais maintenant parler de l’animateur/facilitateur de l’espace d’écoute. Pouvez-vous décrire ce qu'il/elle fait pour vous faire sentir les bienvenus et à l'aise? Savez-vous pourquoi il/elle a été choisi(e) comme animateur? Est-il quelque chose qu'il pourrait faire pour l'améliorer? 10. Y a-t-il autre chose que vous voudriez me dire à propos des espaces d’écoute? Mid-Term Evaluation of the FSP-Enyanya DFSA in the DRC Annexes 123 5.8.8 Management KII & In-Depth Interview Tool Background: This interview guide will be used by all technical leads on the MTE. These questions, related to sustainability, are “cross cutting” and should be adapted to the specific technical approach for each technical sector. These questions are intended to supplement the technical area specific tools that each technical lead will be using to carry out key informant interviews (KII) and focus groups (FG). The guide is broken into several sections: a section that will be used to interview the COP and other senior￾level technical leads; a section that will be used to interview field staff or other mid-level technical staff; a section that will be used to interview local stakeholders (e.g. local partners, government institutions); and a section that will be used to inform focus group discussions with participants. Evaluation Questions (What broader evaluation questions are we trying to answer using the tool?). [We are asking each technical MTE member to ask these questions to understand WHY the technical approaches are working or not working, and then make recommendations on how to improve the activity and the changes of seeing successful results going forward.]  Assess the degree and benefits of efficacious coordination, collaboration, and convergence with locally-based project partners and external organizations that are critical to achieve project goals and purposes. (Evaluation Objective 4). Sub-questions under this objective include: o What mechanisms are in place to engage with local organizations, government entities, private sector, and/or other stakeholders? How strong are these relationships? To what extent will these stakeholders be able to sustain or take over interventions currently being implemented by the IP? o To what extent do local stakeholders help or hinder the implementation of activities? What has the IP done to overcome these challenges or build on successes? o To what extent is the consortium operating as a single team? Are the Prime and subs collaborating effectively? o How has the prime IP worked with other USAID activities in the implementation area (if any)? What has worked well and what has not worked well? What factors are within the manageable interest of the IP and what are not? How has the IP managed the relationship/s with other USAID activity implementing partners?  Assess early evidence of sustainability produced by the project activities thereby determining the extent to which outcomes, systems, and services are designed and being implemented to continue after the project ends. (Evaluation Objective 5). Sub-questions under this objective include: o To what extent has the activity documented and implemented a sustainability strategy? (Note that partners should have a plan, but the goal is to see how the process is going to implement that plan and see if it is realistic. It will be important to know if there is convergence in opinions on sustainability across the senior IP leadership, subs, partners, local institutions, and even participants.) o How will positive outcomes be sustained after the activity ends? Which local systems/structures/entities will sustain these outcomes? o What mechanisms are in place (formal/informal) that will support these sustainable outcomes? (This should build on what was identified as part of Objective 4 above.)  Assess the appropriateness and quality of the R&I approach/activities. How well did the R&I approach work? (Evaluation Objective 6). Sub-questions under this objective include: o How have R&I activities been used to inform the design and ongoing modification of activity implementation? IMPEL | Implementer-Led Evaluation and Learning 124 Annexes o To what extent has R&I been used beyond the initial refinement period and during the implementation phase? o Do IPs have the capacity to use R&I activities as planned? What adjustments are needed to ensure that R&I activities can be used to their fullest? o What are some of the challenges faced in implementing R&I activities and what adjustments need to be made to address and minimize these factors in the future? o How can the R&I model be improved to better ensure that R&I activities are effectively used to strengthen design and implementation of DFSAs? [Identify internal recommendations for FFP, not partners] o Did the DFSAs meet standards to make them eligible for a cost extension based on the R&I model? What, if any, changes should be made in determining eligibility for a cost extension? [Identify internal recommendations for FFP, not partners] Questions for Partner Senior Staff Key Informant Interviews:  Does the activity have a sustainability plan? What are the key features of the plan? Can we get a copy of the plan?  How far along is the activity (broadly speaking) in implementing the plan?  What kinds of challenges or unexpected hurdles have you encountered in implementing the plan thus far?  What has been going well in implementing the plan thus far?  [Partnerships]: Who or which institutions are critical in realizing the sustainability plan? In other words, what local partners have you identified that will continue providing services and/or doing things to sustain positive results (and mitigate negative ones)? This may include local organizations who will continue providing services/etc. beyond the LOA; also local government and/or chiefdom/traditional leadership; local religious organizations/structures; and others.  [Partnerships]: Has the activity created formalized agreements with these institutions, such as MOUs? Can we review (or can you walk us through/describe) any other documents related to sustainability planning (e.g. MOU’s with partners, local government institutions, chiefdom leaders, care group plans, et cetera).  [Governance]: How is the activity [in X sector] working with the local government? What has been going well and what has not been going well thus far? How will partnership with local government work going forward?  [Governance]: How is the activity [in X sector] working with the local government? What has been going well and what has not been going well thus far? How will partnership with local government work going forward?  [Governance]: To what extent has poor public service delivery or public financial management [in X sector] been a hindrance or enabling factor to implementation? (While PFM may be beyond the manageable interest of the partners, it’s important to know if this is a big potential problem.)  [Partnerships]: How is the activity working with local NGOs [in X sector]? What has been going well and what has not been going well thus far? How will partnership with local NGOs work going forward?  [Partnerships]: How is the activity working with other local institutions [in X sector], e.g. civil society organizations, religious groups, et cetera? What has been going well and what has not been going well thus far? How will partnership with these entities work going forward? Mid-Term Evaluation of the FSP-Enyanya DFSA in the DRC Annexes 125  [Partnerships]: How is the activity working with private sector entities (e.g. input supplies, training institutions, professional service providers)? What has been going well and what has not been going well thus far? How will partnership with these entities work going forward? Questions for Partner Field-Based (or non-senior leadership) Technical Staff Key Informant Interviews:  Does the activity have a sustainability plan? What are the key features of the plan? (In other words, how are activities currently being implemented by staff going to continue after the end of the activity?) [[The goal here is to see if the tech teams and senior leadership are in alignment about what the sustainability planning looks like.]]  Broadly speaking, how do you envision sustainability within your technical sector? (In other words, what would a sustainable [sector X] plan look like? This might include activities with mother care groups, business plans with private sector partners, trainings, linkages with other partners.)  Do you think the activity is currently doing everything needed to achieve this/your vision of what sustainability should look like? Why/why not?  What kinds of challenges or unexpected hurdles has your team encountered in implementing the plan thus far?  What has been going well in implementing the plan thus far?  [Governance]: How is the activity [in X sector] working with the local government? What has been going well and what has not been going well thus far? How will partnership with local government work going forward?  [Governance]: How is the activity [in X sector] working with the local government? What has been going well and what has not been going well thus far? How will partnership with local government work going forward?  [Governance]: To what extent has poor public service delivery or public financial management [in X sector] been a hindrance or enabling factor to implementation? (While PFM may be beyond the manageable interest of the partners, it’s important to know if this is a big potential problem.)  [Partnerships]: How is the activity working with local NGOs [in X sector]? What has been going well and what has not been going well thus far? How will partnership with local NGOs work going forward?  [Partnerships]: How is the activity working with other local institutions [in X sector], e.g. civil society organizations, religious groups, et cetera? What has been going well and what has not been going well thus far? How will partnership with these entities work going forward?  [Partnerships]: How is the activity working with private sector entities (e.g. input supplies, training institutions, professional service providers)? What has been going well and what has not been going well thus far? How will partnership with these entities work going forward? Questions for Partner Organizations (e.g. Subs, Local NGOs, Private Sector Entities, Local Government Institutions) Key Informant Interviews:  What is your organization/institution’s current role on the activity?  Do you know when the activity is ending?  What is your understanding of how the activity’s programming will continue beyond the end of the activity?  What is your organization/institution’s specific role in the continuation of [activities/service delivery/other intervention] after the activity has ended? IMPEL | Implementer-Led Evaluation and Learning 126 Annexes  Does leadership in your institution support taking on these activities/service delivery/other in the future?  Does your institution have sufficient resources (e.g. funding, human resources, capacity, interest, systems, etc.) and interest to do this [i.e. take over activities]? Why or why not?  Questions for Participants (Focus Groups and, if needed, KII’s):  Do you know when [DFSA activity name] will end? Do you know what will happen after the end of [activity name]? (In other words, do you know if another organization in your community will be able to provide X services that you are currently receiving through [DFSA activity name]?)  Do you see much value in adopting the practices/technologies being promoted by [DFSA activity name]?  WASH: use of latrines? Covered latrines?  WASH: handwashing at 5 key times  WASH/Nutrition: food preparation and food storage  Nutrition: diverse eating; cooking  Agriculture: IPM, GAP, etc.  Are you currently receiving services (e.g. veterinary, ag extension) through the program? How willing would you be to pay for this service in the future? How much would you be willing to pay?  [For community-based groups, e.g. water use, farmer groups]:  Does your community group charge user fees?  Is there a management committee?  Can you show us how you keep your books and/or manage the day to day actions required to manage the group finances?  Is there an annual planning process? How does that work and who is involved in that process?  Does the group have a budget?  How is the group managed in terms of leadership and planning?  Are the fees collected enough to sustain the activities/service provision in the long term (i.e. to cover costs of maintenance)?  What is the role of local government in providing this service, if any?  [Governance]: What do you feel is the role of the local government in providing key services? [This should be asked in a way that is specific to the specific service and the relevant level/s of government and/or chiefdom/local leadership]  Water access  Health services  [Governance]: Do you feel that the government is accountable to citizens? For example, if a health clinic is supposed to carry certain supplies but they sometimes do not have those supplies, do community members feel that anything can be done to change this? [We need to refine this one…] Mid-Term Evaluation of the FSP-Enyanya DFSA in the DRC Annexes 127 5.8.9 M&E KII & In-Depth Interview Tool Background: This interview guide will be used by the M&E technical advisors on the MTE. The guide is broken into several sections: a section that will be used to interview the COP and other senior-level technical leads; a section that will be used to interview field staff or other mid-level technical staff; a section that will be used to interview local stakeholders (e.g. local partners, government institutions); and a section that will be used to inform focus group discussions with participants. Evaluation Questions (What broader evaluation questions are we trying to answer using the tool?). [We are asking each technical MTE member to ask these questions to understand WHY the technical approaches are working or not working, and then make recommendations on how to improve the activity and the changes of seeing successful results going forward.]  Review the quality of program service delivery related to the different themes and systems in addressing chronic food insecurity and child malnutrition with the targeted clients, taking into account contextual changes that may have occurred since the inception of the activities. (Objective 1) o What [M&E] systems are used to capture, document, and share lessons learned? How are these lessons learned then use to continue ensuring/improving program quality? How is stakeholder and/or participant feedback/perspectives included in this system?  Identify evidence of changes (positive and negative, intended and unintended) associated with program interventions. This will include identifying factors that appear to promote or hinder women and men’s, and young people, food security and safety. (Objective 2) o What changes do community members and other stakeholders associate with the project’s interventions? What factors appear to promote and deter the changes? How do the changes correspond to those hypothesized by the project’s TOC? o To what extent are there observable gender discrepancies? Discrepancies between youth and adults? How has the activity design accounted for these discrepancies and “course corrected” to mitigate these discrepancies? (How is activity data being used to support these kinds of analyses?) o How could the project be modified to improve its acceptability to targeted communities or the efficiency and effectiveness of its implementation? How should the project’s TOC be refined or modified?  Assess the degree and benefits of efficacious coordination, collaboration, and convergence with locally-based project partners and external organizations that are critical to achieve project goals and purposes. (Evaluation Objective 4). Sub-questions under this objective include: o What mechanisms are in place to engage with local organizations, government entities, private sector, and/or other stakeholders? How strong are these relationships? To what extent will these stakeholders be able to sustain or take over interventions currently being implemented by the IP? o To what extent do local stakeholders help or hinder the implementation of activities? What has the IP done to overcome these challenges or build on successes? o How has the prime IP worked with other USAID activities in the implementation area (if any)? What has worked well and what has not worked well? What factors are within the manageable interest of the IP and what are not? How has the IP managed the relationship/s with other USAID activity implementing partners?  Assess early evidence of sustainability produced by the project activities thereby determining the extent to which outcomes, systems, and services are designed and being implemented to IMPEL | Implementer-Led Evaluation and Learning 128 Annexes continue after the project ends. (Evaluation Objective 5). Sub-questions under this objective include: o To what extent has the activity documented and implemented a sustainability strategy? (Note that partners should have a plan, but the goal is to see how the process is going to implement that plan and see if it is realistic. It will be important to know if there is convergence in opinions on sustainability across the senior IP leadership, subs, partners, local institutions, and even participants.) o How will positive outcomes be sustained after the activity ends? Which local systems/structures/entities will sustain these outcomes? o What mechanisms are in place (formal/informal) that will support these sustainable outcomes? (This should build on what was identified as part of Objective 4 above.)  Assess the appropriateness and quality of the R&I approach/activities. How well did the R&I approach work? (Evaluation Objective 6). Sub-questions under this objective include: o How have R&I activities been used to inform the design and ongoing modification of activity implementation? o To what extent has R&I been used beyond the initial refinement period and during the implementation phase? o Do IPs have the capacity to use R&I activities as planned? What adjustments are needed to ensure that R&I activities can be used to their fullest? o What are some of the challenges faced in implementing R&I activities and what adjustments need to be made to address and minimize these factors in the future? o How can the R&I model be improved to better ensure that R&I activities are effectively used to strengthen design and implementation of DFSAs? [Identify internal recommendations for FFP, not partners] o Did the DFSAs meet standards to make them eligible for a cost extension based on the R&I model? What, if any, changes should be made in determining eligibility for a cost extension? [Identify internal recommendations for FFP, not partners] Questions for Partner Senior-Level and Technical Staff  What function/s does your M&E system serve (e.g. reporting, monitor performance, track progress, accountability)?  Does your team use the data collected though the official M&E system or do you use another data collection/monitoring system to complement or replace that official system?  How useful is the M&E Plan in program management? How useful is the existing data collected?  Do non-M&E staff generally feel that the M&E system is used for performance tracking and improving quality or for reporting?  [If applicable]: How are you conducting PDM? What information is collected and how is it used?  Does the project document lessons learned? How are these lessons documented? What changes have been brought due to these lessons learned? How are reports and other information shared across stakeholders?  [Preface this question with a note about how we want to be objective, and that if the interviewee would rather talk to the Evaluation Team Lead, that’s fine]: What’s been the team’s experience working with FFP M&E guidance (e.g. the P&G)? Refine and Implement and Theory of Change  What was your experience with the R&I approach? What worked well and what didn’t work well? Mid-Term Evaluation of the FSP-Enyanya DFSA in the DRC Annexes 129  How did your team use the findings from the R&I period to inform the technical design?  How did your team find the TOC as a design/management/other tool? To what extent did your team use the R&I process to refine the activity design using the TOC? Data Utilization  How is data being used? Who prepares reports or other products and who gets reports? Is information shared with others? Who?  Is data shared with the community? How?  What data do you find most useful and meaningful?  What challenges do you face to using data more effectively?  How has the project used results from assessments and formative research?  How is the project using data generated from routine monitoring? What decisions could be linked to the M&E system?  How does the M&E team see its role around data utilization? Management and Budget  How do staff see the relationship between the prime and sub? [Ask this question differently based on who we’re talking to.]  To what extent do staff from different organizations operate as “one team”? Questions for Partner M&E Staff (Should include Prime and Subs) Monitoring System  How does the prime partner conduct routine monitoring? How do the subs conduct routine monitoring? (Review the system & walk through process)  How does data flow for different sectors? o Who collects? What are their capacities? What types of training they received related to data collection? Do they have any written guidance?  How often is data being collected? How is data reviewed by the prime? How is quality ensured?  What other responsibilities do the people responsible for M&E also have in the different sectors?  How is qualitative data recorded? What kinds of qualitative information is being systematically collected? What kind of qualitative information is NOT being collected? How is qualitative information being integrated into the M&E system and, more specifically, synthesized with quantitative data (RM or AS data)?  [If applicable]: How are non-implementing partner entities (e.g. local institutions or local NGO partners) involved in the M&E system? To what extent do your data rely on staff from these non-IP entities? What is the capacity of these non-IP entities to perform monitoring? What challenges do they face?  How is data safeguarded? Qualitative v. Quantitative?  Is the data flow working well? Are there any challenges being faced (timeliness, data losses, data manipulation)?  How does the project staff collect annual monitoring data? What are the challenges?  How did they establish the base values? How do they set up targets? [Review some of the key annual monitoring indicators to assess the quality including one behavior change uptake indicator.] How could this process be improved? IMPEL | Implementer-Led Evaluation and Learning 130 Annexes  How are community-based activities monitored for quality? For evidence of behavior change? What has been learned from these? What could be learned from these?  How has the team incorporated new FFP and FTF indicators into their monitoring systems?  How was gender equality considered in designing project monitoring system, indicators, reporting and data utilization?  What else can you share with us about the monitoring system? Complaint & Feedback Mechanism  Does the project have an operational complaints and feedback mechanism?  Who manages the system? Is it independent of project management?  How are complaints and feedback tracked? Are responses given, and if so is response time tracked?  How is information from the system used to inform project implementation?  Does the project have criteria for significant issues (e.g. fraud, waste, and abuse), and if so, how are these issues addressed? Data Utilization  How is data being used? Who prepares reports or other products and who gets reports? Is information shared with others? Who?  Is data shared with the community? How?  What data do you find most useful and meaningful?  What challenges do you face to using data more effectively?  How is the project using data generated from routine monitoring? What decisions could be linked to the M&E system?  How does the M&E team see its role vis-a-vis data utilization?  What else can you share with us about how your team is utilizing data? R&I and Learning  How were the findings from the formative research (e.g. in year 1) incorporated into the M&E Plan? Who was involved in this process and how did it go?  To what extent did the Theory of Change (TOC) change as a result of the R&I process?  What worked well and what did not work well about the R&I approach?  What worked well and did not work well specifically around M&E during and immediately following the first year?  What else would you like to tell us about the R&I approach? Staffing & Staff Capacity  How are training needs and TA determined? For M&E staff vs. non-M&E staff?  Do you feel that field staff are adequately trained and able to collect data in alignment with what is outlined in the M&E Plan? Context Monitoring & Adaptive Management  What changes have occurred in the operating context (e.g., major events like disasters, slow onset events like climate change, government policy changes, etc.) that have affected project Mid-Term Evaluation of the FSP-Enyanya DFSA in the DRC Annexes 131 implementation, either positively or negatively? How did these specifically affect implementation? What did the project do to adapt to these changes?  How have these changes affected the food insecurity or resilience of targeted impact groups for each project? What other groups are becoming more food insecure or are having reduced resilience as a result of these contextual changes?  Given these changes in the operating environment, which project activities seem most relevant now in terms of having impact on the lives of targeted impact groups?  What suggestions do you have for adapting the project's strategy or implementation systems in order to be better able to respond to changes in the operating context?  Since the activity began implementation, have there been any interruptions or delays in the flow of resources, either food or cash or both? If so, what caused these delays, and what was the impact on project implementation? What did the project do to adapt? M&E Questions for Field-Based Staff  What role does performance monitoring play in the activity?  What are your key roles/responsibilities for collecting data? Do you know what happens with the data you collect (after you submit the information)?  Do you feel like the data you’re gathering accurately reflects the situation on the ground? o For example, are those data points you are collecting accurately reflect the participation of women, youth, or other groups? If not, what is missing?  How does your team gather qualitative data? (Are anecdotes about things happening at trainings getting shared with the senior-level technical staff? How are these aggregated? Who analyzes and/or uses this information?)  If you could decide what information to gather about the effectiveness or other characteristics of the program, what information would you gather? Why (what’s different about what you would do as compared to what is currently happening)?  How involved are you in the analysis of monitoring data? What information do you receive, and when? How do you use this information once you have it? o For example, are you given a report or raw data? What are you expected to do with that information? o What do you do if the information/reports you are receiving conflict with what you are seeing on the ground? Do you feel empowered to speak up to share your thoughts? How do you typically do this?  What training did you receive on M&E? o Data collection processes? o Informed consent and research ethics? o Data analysis methods o Other?  [Operating Context] o What changes have occurred in the operating context (e.g., major events like disasters, slow onset events like climate change, government policy changes, etc.) that have affected project implementation, either positively or negatively? How did these specifically affect implementation? What did the project do to adapt to these changes? o How have these changes affected the food insecurity or resilience of targeted impact groups for each project? What other groups are becoming more food insecure or are having reduced resilience as a result of these contextual changes? IMPEL | Implementer-Led Evaluation and Learning 132 Annexes o Given these changes in the operating environment, which project activities seem most irrelevant now in terms of having impact on the lives of targeted impact groups? o What suggestions do you have for adapting the project's strategy or implementation systems in order to be better able to respond to changes in the operating context? o Since the Project began implementation, have there been any interruptions or delays in the flow of resources, either food or cash or both? If so, what caused these delays, and what was the impact on project implementation? o What did the project do to adapt? Local Implementation Office, extension workers, DAs  Assess data collection system, records, data flow and information shared back. Ask for demonstration of what information they have access to.  What data do you collect? Why do you collect this data? How do you record information? Review the paper forms  For those people collecting data, what other roles do they have? Do they receive support to collect data? Do they receive instructions or training?  What support do you need to collect data?  What happens with the data you collect?  What information do you receive from the woreda? From REST? From MoA/MoH?  How is data collected? How do you ensure that it is correct?  What is done with the data you provide? Who do you provide the data to?  How are decisions made using the data you provide?  Demonstrate how measure: hectares, land under improved technologies, health data collected by HEWs- weighing of children, whether women are receiving ANC visits. Trainers and Participants  Inquiry around trainings and behavior change.  Utility of trainings. Have trainings ever changed? What support do you receive? What do you need?  Assess registration logs. Interview HHs about the trainings, what they learned and how they use that information to assess and breakdowns in behavior change occurrence.  If possible, walk through training material with participants to assess understanding and quality of material  What would you like to learn in the trainings? Documents  Review training material or guides if available on M&E  (REST) website, newsletter and learning magazine that have these knowledge dissemination goals  Training material frontline extension agents  USAID DQA reports