Interim Evaluation of the South Kivu Food Security Project (FSP)-Enyanya Resilience Food Security Activity in the Democratic Republic of the Congo May 2022 | Volume I IMPEL | Implementer-Led Evaluation & Learning Associate Award ABOUT IMPEL The Implementer-Led Evaluation & Learning Associate Award (IMPEL) works to improve the design and implementation of Bureau for Humanitarian Assistance (BHA)-funded resilience food security activities (RFSAs) through implementer-led evaluations and knowledge sharing. Funded by the USAID Bureau for Humanitarian Assistance (BHA), IMPEL will gather information and knowledge in order to measure performance of RFSAs, strengthen accountability, and improve guidance and policy. This information will help the food security community of practice and USAID to design projects and modify existing projects in ways that bolster performance, efficiency, and effectiveness. IMPEL is a seven-year activity (2019–2026) implemented by Save the Children (lead), TANGO International, Tulane University, Causal Design,Innovations for Poverty Action, and International Food Policy Research Institute. RECOMMENDED CITATION IMPEL. (2022). Interim Evaluation of the South Kivu Food Security Project (FSP)-Enyanya Resilience Food Security Activity in the Democratic Republic of the Congo (Vol. I). Washington, DC: The Implementer-Led Evaluation & Learning Associate Award. PHOTO CREDITS Annette Fay, Tulane Consultant DISCLAIMER This report is made possible by the generous support of the American people through the United States Agency for International Development (USAID). The contents are the responsibility of the Implementer-Led Evaluation & Learning (IMPEL) award and do not necessarily reflect the views of USAID or the United States Government. CONTACT INFORMATION IDEAL Activity c/o Save the Children 899 North Capitol Street NE, Suite #900 Washington, DC 20002 www.fsnnetwork.org IMPEL@savechildren.org PREPARED BY: Interim Evaluation of the South Kivu FSP-Enyanya RFSA in the DRC (Vol. I) Acknowledgments i ACKNOWLEDGMENTS USAID/BHA: Staff in Washington, DC and at the DRC mission Kinshasa School of Public Health: Pierre Akilimali, Prescillia Visi, John Lhudal, Charles Kasongo, Dynah Kayembe, Steve Mbikayi, Richard Mubikayi, Guy Ngindu, Marc Mchanga Useni, Maurice Kabangu, and Michel Kayembe IMPEL: Tom Spangler, Mohamoud Ahmed, and Corena Sharp Tulane University: Nancy Mock, Peter Horjus, Michelle Lacey, Sarah Truxillo, and Janna Wisniewski Trestle Research and Consulting: Maulik Chauhan and Vikas Arora Independent survey monitors: Elvis Cidoro (supervisor), Innocent Nshombo (South Kivu), Samuel Kataya (Tanganyika), and Roussel Kabeya (Kasai Oriental) IMPEL | Implementer-Led Evaluation and Learning ii Table of Contents TABLE OF CONTENTS Acknowledgments............................................................................................... i List of Tables......................................................................................................iii List of Figures.....................................................................................................iii Acronyms...........................................................................................................iv Executive Summary ............................................................................................ v 1. Introduction................................................................................................... 1 1.1 Background and Setting ................................................................................................................. 1 1.2 Resilience Food Security Activity Goals and Activities................................................................... 2 1.3 Interim Evaluation Purpose and Objectives................................................................................... 3 1.4 Study Team and Partners............................................................................................................... 4 2. Quantitative Evaluation Methods.................................................................. 6 2.1 Overview ........................................................................................................................................ 6 2.2 Sample Design................................................................................................................................ 6 2.3 Sample Size..................................................................................................................................... 8 2.4 Sample Selection............................................................................................................................ 9 2.5 Survey Tools................................................................................................................................... 9 2.6 Fieldwork and Data Collection .....................................................................................................10 2.7 Data Analysis................................................................................................................................11 2.8 Final Sample and Sample Weights...............................................................................................11 2.9 Integration of Secondary Data .....................................................................................................12 2.10 Limitations and Delimitations ......................................................................................................13 3. Findings .......................................................................................................14 3.1 Intervention Exposure and Participation .....................................................................................14 3.2 Water, Sanitation, and Hygiene ...................................................................................................17 3.3 Agriculture....................................................................................................................................19 3.4 Women’s Health and Nutrition....................................................................................................23 3.5 Children’s Health and Nutrition ...................................................................................................24 3.6 Gender..........................................................................................................................................27 3.7 Activity-Specific ............................................................................................................................29 3.8 Shock Exposure and Resilience Indices........................................................................................29 3.9 Food Security................................................................................................................................33 4. Conclusions..................................................................................................35 Interim Evaluation of the South Kivu FSP-Enyanya RFSA in the DRC (Vol. I) List of Tables and Figures iii Volume II Annex A: Protocol for the Quantitative Interim Evaluations of three Resilience Food Security Activities in the Democratic Republic of the Congo Annex B: Survey Questionnaire Annex C: Additional Figures and Tables Annex D: Descriptive Analysis Tables Annex E: Indicator Change Tables LIST OF TABLES Table 1. FSP-Enyanya RFSA purposes, sub-purposes, and related interventions......................................... 2 Table 2. FSP-Enyanya RFSA interim evaluation PBS sampling frame............................................................ 7 Table 3. Population-based survey sampled villages/clusters and sample.................................................... 9 Table 4. FSP-Enyanya RFSA final sample size and response rates..............................................................12 Table 5. Baseline and interim water, sanitation, any hygiene indicator comparisons...............................18 Table 6. Baseline and interim agriculture indicator comparisons..............................................................20 Table 7. Changes in prevalence of sustainable crop practices between baseline (2017) and interim (2021) ....................................................................................................................................................................21 Table 8. Changes in prevalence of livestock practices and technologies between baseline (2017) and interim (2021) .............................................................................................................................................22 Table 9. Baseline and interim women’s health and nutrition indicator comparisons................................23 Table 10. Baseline and interim children’s health and nutrition indicator comparisons.............................25 Table 11. Baseline and interim gender indicator comparisons ..................................................................27 Table 12. Baseline and interim activity-specific indicator comparisons.....................................................29 Table 13. Baseline and interim shock exposure and resilience indicator comparisons .............................30 Table 14. Absorptive capacity index: Changes in component indicators...................................................31 Table 15. Adaptive capacity index: Changes in component indicators ......................................................32 Table 16. Transformative capacity index: Changes in component indicators............................................32 Table 17. Baseline and interim food security indicator comparisons.........................................................33 LIST OF FIGURES Figure 1. Enyanya RFSA intervention areas .................................................................................................. 2 Figure 2. FSP-Enyanya RFSA intervention exposure and household participation.....................................16 Figure 3. Prevalence of reported household shocks...................................................................................30 Figure 4. Absorptive, adaptive, and transformative capacity index by Household Dietary Diversity Score ....................................................................................................................................................................34 IMPEL | Implementer-Led Evaluation and Learning iv Acronyms ACRONYMS APBS Annual Participant Based Surveys BHA Bureau for Humanitarian Assistance BL Baseline CLTS Community-Led Total Sanitation CSO Civil Society Organizations DHS Demographic and Health Survey DRC Democratic Republic of the Congo FFA Food for Assets FIES Food Insecurity Experience Scale FSP Food Security Project HDDS Household Dietary Diversity Score HH Household IMPEL Implementer-Led Evaluation & Learning Associate Award IP Implementing Partner KSPH Kinshasa School of Public Health LDC Local Development Committee MAD Minimum Acceptable Diet MCHN Maternal and Child Health and Nutrition MDD Minimum Dietary Diversity MTE Mid-Term Evaluation NGO Non-Governmental Organization NRM Natural Resource Management ODK Open Data Kit ORT Oral Rehydration Therapy PBS Population-Based Survey PO Producers’ Organization R&I Refine and Implement RFSA Resilience Food Security Activity SD Standard Deviation SFBS Seasonal Farmer Based Surveys USAID United States Agency for International Development WASH Water, Sanitation, and Hygiene VSLA Village Savings and Loan Association YBG Youth Business Groups Interim Evaluation of the South Kivu FSP-Enyanya RFSA in the DRC (Vol. I) Executive Summary v EXECUTIVE SUMMARY Overview To assist vulnerable populations of South Kivu, the United States Bureau for International Development’s (USAID) Bureau for Humanitarian Assistance (BHA) awarded Mercy Corps the implementation of a 5-year Resilience Food Security Activity (RFSA) from September 2016–September 2021, and later granted a 2-year extension through 2023. The South Kivu Food Security Project (FSP)- Enyanya1 RFSA operates in three health zones (Kalehe (territory of Kalehe), Miti-Murhésa, and Katana (territory of Kabare)) with a life of award budget of approximately $62 million. The FSP-Enyanya RFSA’s overall goal is “improved food and nutrition security and economic well-being of vulnerable households in South Kivu”2 and targets approximately 36,000 households. In 2021, under the USAID BHA Implementer-Led Evaluation and Learning (IMPEL) Associate Award with Save the Children, Tulane University School of Public Health and Tropical Medicine (Tulane) was subcontracted to conduct an interim evaluation, including a population-based survey (PBS) and a resilience assessment of the FSP-Enyanya RFSA. Study Purpose and Design The PBS serves as the second phase of a pre-post survey cycle, with data on the same indicators collected in both survey rounds. Statistically detecting changes (if any) for all practice and behavioral change indicators at the population level (the FSP-Enyanya RFSA coverage area) can help inform the performance of the award to date. It should be noted that the evaluation does not include data from areas in which the intervention was not implemented (i.e., a counterfactual). While the evaluation results can be used to help explain differences between the pre- and post-values and may therefore help inform overall performance, true attribution is only possible with experimental and quasi￾experimental evaluation designs, which are not being used in this evaluation. This PBS collected representative data on a number of lower-level outcomes from 1,349 households and nearly 8,000 individuals in July/August 2021. Key Findings Overall Assessment Overall, the interim evaluation, as well as the 2019 mid-term evaluation, show FSP-Enyanya’s RFSA in the Democratic Republic of the Congo (DRC) in a positive light. The interim evaluation shows promising results, with several indicators that appear to be moving in the correct direction. The analysis tends to show that direct participation in interventions is associated with improvements in many lower-level outcome indicators. However, results at the population level do not show consistent significant improvements that were expected to be seen. Coverage rates (numbers of direct participants) of the interventions are often low, covering a small percentage of the total population. Spillover of the impacts from the RFSA interventions to indirect participants may be less than hoped. Consequently, impacts are likely diluted at the 1 Enyanya means “go forward” in Mashi, one of the main languages spoken in the RFSA’s intervention areas. 2 RFSA Annual Report Narrative FY 2018 Final. IMPEL | Implementer-Led Evaluation and Learning vi Executive Summary population level. More programmatic intensity and focus may be required to positively impact the lower-level outcome indicators, and ultimately improve food security and resilience at the population level. It is important for BHA and Mercy Corps to consider, in general, the cost of implementation of an activity and its interventions relative to the saturation that the activity may be expected to reach, in order to determine if it is worth running an intervention that is “a mile wide and an inch deep.” The package of RFSA interventions may need to be streamlined (and/or consolidated) to ensure only sustainable, efficient, and impactful interventions are used. This study only begins to scratch at the surface of these issues. Intervention Exposure and Participation The RFSA achieved moderate levels of self-reported household participation rates in some of their interventions, though none were as high as indicated by the RFSA program data. Program data from 2020 indicate that the RFSA was reaching 52,000 unique direct participant households, or about 70% of the coverage area. The interim Round 2 survey yielded a somewhat lower estimate, with 54% of households that reported direct participation in one or more of the surveyed RFSA interventions. The survey data is not meant to call into question the project data, however. Despite most of the interventions being implemented in all (or most) of the villages sampled in the survey, direct participation, as reported in the Round 2 survey, was generally low. Interventions with the highest coverage of direct participants included WASH trainings/events (24% of households), agricultural training (22% of households), and agricultural inputs (20%). All other interventions surveyed fell below 20%. One example of a potential high-quality/impact intervention with low coverage is safe space groups. This intervention was positively described by the 2019 mid-term evaluation as best practice and likely to have an impact on participant nutrition and knowledge, attitude, and practice of sexual health behaviors. However, in the Round 2 survey, only 4% of households reported direct participation, despite 81% of households in the interim survey living in villages where safe space groups were implemented. This reflects the mid-term findings that the FSP-Enyanya RFSA was reaching about 3,500 adolescents via Safe Spaces in 2019, with plans to double that by the end of the activity. This is a very small percentage of the adolescent population, unable to have any meaningful impact at the population level. Low coverage of many of the interventions means that goals for population-level impacts are unlikely to be achieved. Food Security and Resilience Food security, as measured by the Household Dietary Diversity Score (HDDS) and the Food Insecurity Experience Scale (FIES), showed no significant change between survey rounds. The prevalence of moderate and severe food insecurity (as measured by the FIES) was very high at baseline (95%) and at the interim (96%). However, the HDDS did show some small improvements in the unweighted analysis of the re-sampled areas. The FSP monitoring data also shows little change in the HDDS and the FIES in both the Annual Participant Based Surveys (APBS) and the Seasonal Farmer Based Surveys (SFBS). Interim Evaluation of the South Kivu FSP-Enyanya RFSA in the DRC (Vol. I) Executive Summary vii Considering the volatile food security in the DRC, compounded in the year prior to the survey by the COVID-19 pandemic, stable food security measures could be interpreted as a positive outcome. All three resilience capacity indices (absorptive, adaptive, and transformative) increased significantly between baseline and interim. However, changes in the indices were largely driven by only one (or a few) of their component indicators. The large increase of the absorptive index was driven primarily by the reported increased presence of humanitarian assistance (one of the index components). This may be due to assistance provided through FSP as being “humanitarian” by the respondents (though this cannot be confirmed by available data). Cash savings and informal safety nets also contributed to the increase in the absorptive index. The small increase in the adaptive capacity index was driven mainly by an increase in social safety nets, and to a lesser degree, increases in productive asset ownership. The improvements were offset by significant declines in education/training, livelihood diversity (mainly from a reduction in households reporting remittances/gifts), and improved agricultural practices. The modest increase in the transformative capacity index was driven mainly by improvements reported in formal safety nets, and was offset by small decreases in some of the other component indicators. There is a positive correlation between the HDDS and higher absorptive and adaptive capacity indices. However, this does not hold true for the transformative capacity index, which is defined mainly by community-level indicators rather than household level. Water, Sanitation, and Hygiene Improvements in access to safe drinking water were observed across the surveyed areas, particularly in the time required to access these sources. The percentage of households that could obtain drinking water in less than 30 minutes (round trip) increased significantly (from 39% to 75% of households). Data was not collected in the interim evaluation on water point rehabilitation. However, the mid-term evaluation indicated that the FSP-Enyanya RFSA functional water points had increased water access in some areas, with a recommendation to work to increase coverage. As each water point constructed may benefit an entire community, even modest increases in coverage of related interventions may result in important improvements in these indicators. The use of basic (improved) sanitation facilities remained low (3%), with no significant change. Open defecation remained very low (3%) at the interim. FSP participant monitoring data showed a higher prevalence of improved sanitation in its surveys, but also showed little change over time (2019 to 2021). However, contrary to these findings, Mercy Corps reports the construction of more than 20,000 pit latrines, covering well over 3% of households in the coverage area. Additionally, the interim survey data indicates that 18% of households report participating in toilet-building activities. The predicted improvement in basic (improved) sanitation facilities may be clouded by the survey definition of basic (improved) sanitation facilities. Pit latrines without slabs (considered unimproved by the survey methodology) were the common type of toilet reported in the interim survey (85% of households).3 3 It’s difficult to assess the change in toilet type compared to baseline. The baseline indicates that 82% of HHs had a “water flush to pit latrine” type toilet, which is likely a mistake in data collection (although still classified as unsafe/unimproved sanitation). The interim survey reports that <1% of HHs have flush-to-pit-latrine toilets- similar to the most recent DHS survey, which indicates that 0.5% of urban and 0% of rural have flush-to-pit-latrine toilets. IMPEL | Implementer-Led Evaluation and Learning viii Executive Summary Agriculture There was a significant increase in farmers’ use of financial services between the survey rounds (32% to 50%), as well as the use of improved storage practices (40% to 53%). The use of value chain activities promoted by the project showed a small improvement, which was larger and statistically significant in the analysis of the resampled villages only. The FSP recurrent monitoring data show similar directions of change in these indicators. The 2019 mid-term evaluation (MTE) reported that the producers’ organization (PO) interventions were either weak, behind schedule, or non-existent. However, the 2020 annual report indicates that progress had been made following the MTE in implementing these activities. Participation in village savings and loan associations (VSLAs), as well as participation in farmers’ groups and/or natural resource management (NRM) programs, were strongly associated with farmers’ access to financial services. However, the interim evaluations found no association between participation in farmers’ groups or NRM programs and improved value chain activities or storage practices. The prevalence of farmers reporting using four or more sustainable crop practices/technologies (out of 9 promoted by the activity) by farmers decreased significantly in the interim evaluation survey. There was no change in the prevalence of farmers using at least one sustainable crop practice (88.8% in baseline, 88.5% in interim). However, at baseline, farmers were recorded as practicing a greater number of sustainable crop practices on average at baseline than at interim. FSP monitoring data showed higher prevalences of farmers using four or more sustainable crop practices in their 2019, 2020, and 2021 surveys. However, the percentage point differences between years (+/- 15 percentage points) were similar to that seen in between the PBS survey rounds. It’s challenging to assess whether these differences between rounds are methodology/collection issues, or if they accurately reflect changes. Among livestock practices, the RFSA focused primarily on the breeding and keeping of rabbits. However, this only increased from 2.5% of households to 3.1% of households. The mid-term evaluation found that comprehension by rabbit recipients of how to raise, manage, and breed the animals was limited and varied. The 2020 annual report data indicated improvements, but the intervention remained small, and had low coverage, so this very small population-level change is not surprising. Women’s Health and Nutrition Changes in women’s consumption patterns showed slightly negative changes. The prevalence of women consuming a diet of minimum diversity fell (weakly significant), and the prevalence of women consuming targeted nutrient-rich commodities also dropped, though this change was only significant when looking at the re-sampled villages only. FSP participant monitoring data also showed no large variation in women’s consumption of a diet of minimum diversity or in the consumption of nutrient-rich commodities (among direct participants). However, participation in nutrition training was significantly associated with higher levels of women consuming a diet of at least the minimum dietary diversity. Contraceptive use (both modern and traditional) remained unchanged between survey rounds. FSP participant monitoring data also showed little change between 2019 and 2021. However, the interim survey showed that women participating in mothers’ groups were significantly associated with a higher prevalence of contraception than those who did not. Interim Evaluation of the South Kivu FSP-Enyanya RFSA in the DRC (Vol. I) Executive Summary ix Child Health and Nutrition Diarrhea prevalence in children under 5 had a significant decrease (34% to 20%) between survey rounds. A similar prevalence and change were observed in FSP participant monitoring surveys. Treatment of under-5 diarrhea with oral rehydration therapy (ORT) showed little change (though the sample size was small). Activity monitoring 2021 data from Mercy Corps indicate that among participant households, 71% of children with diarrhea are treated with ORT. Low intervention coverage among the interim survey households does not allow for a similar level of analysis, and changes measured at the overall population level may be too diluted to show any overall change. Exclusive breastfeeding of children under 6 months old did not change significantly between rounds, though the sample was small. FSP monitoring data showed a decline in exclusive breastfeeding between the 2019 and 2020 survey estimates of over 80%, and the 2021 survey estimate of 57% of children among direct participants (similar to the population findings in Round 2). The prevalence of children 6–23 months consuming targeted nutrient-rich foods improved significantly (particularly among girls). However, the consumption of a minimum acceptable diet of children 6–23 months remained largely unchanged. The improvement in consumption of nutrient-rich foods was driven primarily by increased consumption of bio-fortified foods (a value-chain commodity), as well as orange-fleshed sweet potatoes. Although mothers’ groups, nutrition trainings, and home health visits were implemented in all communities, participation rates were low (16% of children under 5 lived in households reporting participation in mothers’ groups, 17% in nutrition training/meetings, and 20% in home health visits). Furthermore, there are no associations between participation in these interventions and improvements in child MAD, breastfeeding practices, or under-5 diarrhea. Gender Little change was noted across the gender indicators, except for a marginally significant improvement in the knowledge of maternal and child health and nutrition (MCHN) practices. Gender norms are deeply entrenched and will likely take years or even decades to change. There was little change in the percentage of adults earning cash in the past year, though the prevalence of men earning cash rose significantly in the resampled areas. FSP monitoring data of participants also show little change in the percentage of men and women earning cash among direct participants. Women had significantly better knowledge of MCHN practices at the interim, though men showed no significant change. FSP monitoring data of direct participants showed similar prevalences, though little change between the 2019, 2020, and 2021 surveys. Among the variables related to decision-making among married/partnered men and women, there was little to no significant change between survey rounds. Women’s participation in decision-making bodies also showed no significant change. FSP monitoring data showed similar evidence of minimal change over time, highlighting the fact that real changes to gender attitudes and norms may take decades to change. IMPEL | Implementer-Led Evaluation and Learning x Executive Summary Methodological Challenges This evaluation had some methodological challenges that should be taken into consideration in future evaluations, revolving primarily around the limitations of the pre-/post-PBS design, as well as some key outcome indicators that may not adequately perform in the context. Modifications in areas of implementation after the baseline PBS is common across RFSAs. The pre-/post￾PBS methodology may not be well suited to adapt to changing areas of implementation. Sampling frame data in the DRC often has large inaccuracies. This results in highly variable probability/population weights, and a loss of statistical power. Alternative PBS sampling strategies should be considered that would have less loss of statistical power with similar sample sizes and budgets. Population-level changes in many of the low-level indicators have a low likelihood of occurring with the given intensity of certain interventions. Furthermore, there is a desire to have evaluation data that can show the impact of interventions on the various outcomes. A PBS does not readily allow this level of analysis. The utility of certain food security indicators (such as the FIES) should be re-evaluated, including an assessment of their functionality in specific locations/contexts. The prevalence of moderate and severe food insecurity (as measured by the FIES) was very high at baseline (95%) and at the interim (96%). This homogeneity of the food security status as measured by this indicator renders a more detailed analysis less useful in assessing change. The FIES has other statistical limitations in the populations surveyed. It may not adequately describe the food security situation in the populations surveyed. The resilience capacity indices are less useful as composite indicators. The sub-components of the indices tend to reveal more useful information. For example, three of the adaptive capacity component indicators are related to agriculture, so households that did not engage in agriculture tend to have lower scores on this index even if they are highly resilient. This indicates that the adaptive capacity index in the aggregate may be of limited use for households not engaged in agriculture. It may benefit from adaptation to reflect adaptability as a function of livelihood. Interim Evaluation of the South Kivu FSP-Enyanya RFSA in the DRC (Vol. I) Introduction 1 1. INTRODUCTION 1.1 Background and Setting Eastern Democratic Republic of the Congo (DRC) has experienced decades of conflict, poor governance, and, as a result, widespread extreme poverty. The genocide that took place in neighboring Rwanda, unfortunately, spilled over into eastern DRC as an easy and convenient place for Hutus to flee to. The region suffered tremendously during the aftermath of that conflict which then evolved into the war in eastern DRC between 1998 and 2007, during which an estimated 5.4 million people died as a result of the conflict and protracted humanitarian crisis. Mortality rates were higher in eastern DRC, demonstrating the effect of insecurity, with most deaths from easily preventable and treatable illnesses rather than violence. Millions more citizens were pushed into poverty due to displacement and loss of economic livelihoods. By 2012, most of the country was relatively stable, although armed factions have persisted. In 2017, 52 armed groups were active in South Kivu alone, committing thefts and attacks, collecting illegal taxes, and exploiting natural resources (forests and mines). Coupled with the longstanding conflict, the DRC experiences significant political tensions: both the 2011 and 2018 presidential and legislative elections were marred by violence and disputes leading up to the elections and regarding the results across the country. Gender inequalities are prevalent; in 2018, the DRC ranked 147 out of 166 countries on the Gender Development Index. The Demographic and Health Survey (DHS) of 2013–2014 found that 27% of Congolese women aged 15–49 have experienced sexual violence in their lifetime, and more than half had experienced some form of physical violence. The 2013–2014 DHS reported that over 40% of Congolese children suffer from chronic malnutrition indicated by stunting (-2 Standard Deviation (SD) height-for-age), with more than half of these children falling into the severe range (-3 SD height-for-age), signifying a very serious public health problem according to WHO classifications. Eastern DRC is also grappling with the world’s second-largest Ebola epidemic on record, with more than 2,000 lives lost and 3,000 confirmed infections since the outbreak was declared on August 1, 2018. Other issues of note in the region are high rates of alcoholism, and land access is very difficult. The combination of the rapidly growing population, traditional practices related to land inheritance, which causes division of family properties, and land degradation is decreasing the availability of arable land and the size of agricultural fields. A Comprehensive Food Security and Vulnerability Analysis conducted in 2011–2012 by the World Food Programme found 64% of the rural population in South Kivu to be food insecure, with Kalehe ranking third (72.2% of the population food insecure) and Kabare ranking fourth (70.8%) of the 12 South Kivu territories. A recent IPC analysis from 20214 indicated that in South Kivu, 53% of the population was in acute food insecurity phase 2, 19% in phase 3, and 3% in phase 4 (none in phase 5). To address these challenges, in Fiscal Year 2016, the United States Agency for International Development’s (USAID) Bureau for Humanitarian Assistance (BHA) funded a multi-year resilience food security activity (RFSA) in the DRC called The South Kivu Food Security Project (FSP)-Enyanya, which was implemented by a consortium led by Mercy Corps. 4 https://www.ipcinfo.org/fileadmin/user_upload/ipcinfo/docs/IPC_DRC_Acute_Food_Insecurity_2021FebDec_ Report_French.Updated.pdf IMPEL | Implementer-Led Evaluation and Learning 2 Introduction 1.2 Resilience Food Security Activity Goals and Activities To assist vulnerable populations of South Kivu, given the challenges described above, BHA awarded Mercy Corps the implementation of a 5-year RFSA5 from September 2016–September 2021, with a 2- year extension later granted through 2023. The South Kivu FSP-Enyanya6 RFSA operates in three health zones7 (Kalehe (territory of Kalehe), Miti-Murhésa, and Katana (territory of Kabare)) with a life of award budget of approximately $62 million. The FSP-Enyanya RFSA8 is implemented by Mercy Corps as the prime, World Vision as both the main sub-partner and the lead on health interventions; Harvest Plus supplies bio-fortified seeds; Université Evangélique de l’Afrique (UEA) was responsible for soil fertilization and erosion control as well as technical leadership to implement the hill approach; and Action pour la Paix et la Concorde (APC) implemented conflict management interventions, with an emphasis on land access. The FSP-Enyanya RFSA’s overall activity goal is “improved food and nutrition security and economic well-being of vulnerable households (HH) in South Kivu”9 and targeted approximately 36,000 HH. This activity has three purposes that were then further divided by sub-purpose and addressed through the related interventions as follows: Table 1. FSP-Enyanya RFSA purposes, sub-purposes, and related interventions Purposes Sub-purposes Related interventions 1 - Agriculture: Household incomes are increased 1.1: Agricultural productivity, especially of nutritious, lower-risk food crops, increased • Farmer Field Schools • Junior Farmer Field Schools • Tree Nurseries • Village Savings and Loan Associations 1.2: Women, men, youth actively participate in income-generating activities • Producer Organizations (PO) • Youth Business Groups (YBG) • Microfinance Institutions • Seed Multipliers • Literacy and Numeracy 5 The activity was previously awarded as a Food for Peace Development Food Security Activity (DFSA). The terminology for these awards changed in 2020. 6 Enyanya means “go forward” in Mashi, one of the main languages spoken in the RFSA’s intervention areas. 7 A health zone is the larger administrative level, covering 50,000–100,000 people in rural areas. A health area is the next level down, covering 5,000–10,000 people in rural areas and typically have one health center per health area. 8 FSP-Enyanya RFSA will be used as the primary name of the activity throughout this report. 9 RFSA Annual Report Narrative FY 2018 Final. Figure 1. Enyanya RFSA intervention areas Interim Evaluation of the South Kivu FSP-Enyanya RFSA in the DRC (Vol. I) Introduction 3 Purposes Sub-purposes Related interventions 2 – Health, Nutrition and WASH: Improved nutrition status in communities 2.1: Improved Healthy Timing and Spacing of Pregnancy • Safe Spaces • Positive Peer Couples • Channels of Hope • Health Service Providers 2.2: Women, men, children eat optimal diets • Care Groups • Permagarden/Livestock 2.3: Community members, mothers & Children Under 2 are healthy • Care Groups • Community-Led Total Sanitation (CLTS) and Water Point Construction/Rehab • Health Services and Community Health Workers (RECOs) 2.4: Children whose growth is faltering identified and referred for care and counseling • Care Groups • CLTS • RECOs 3 – Governance: Socio￾economic status is stable and inclusive 3.1: Women, youth participate in community planning and decision￾making • Local Development Committees (LDC) • Care Groups, Safe Spaces, YBGs, POs, VSLAs 3.2: Communities have the info, tools & processes to monitor and improve the socio-economic environment • LDCs • Food for Assets (FFA) • Civil Society Organizations (CSO) 3.3: Communities are resilient against conflict • Land Access Securitization • Dialogue & Mediation Committees • CSO 3.4: Taxation practices perceived as transparent & fair • CSO • Local government, traditional authorities The leadership team, based in Bukavu, provided managerial, administrative, and technical support to the field teams based in the regional bases in each health zone and implemented field activities. Mercy Corps runs the Miti-Murhésa and Katana bases, while World Vision runs the Kalehe base. Activity staff rely on a network of volunteers across purposes to reach a larger number of participants. Under each purpose, activity field staff train volunteers who are directly associated with the RFSA on technologies or modules who then cascade out learnings to the larger group of the RFSA participants. 1.3 Interim Evaluation Purpose and Objectives In 2021, under the USAID BHA Implementer Led Evaluation and Learning (IMPEL) with Save the Children, Tulane University School of Public Health and Tropical Medicine (Tulane) was subcontracted to conduct an interim evaluation, including a population-based survey (PBS) and a resilience assessment, in the RFSA area. As part of the overall Refine and Implement (R&I) approach, the interim evaluation was originally planned to take place in 2020 so that the findings could help inform decisions on activity extensions. However, the evaluation was delayed until 2021 because of the COVID-19 pandemic. Prior to the interim IMPEL | Implementer-Led Evaluation and Learning 4 Introduction evaluation, BHA made the decision to extend the FSP-Enyanya RFSA for 2 years. 10 The extension took many factors into account, but the design of the interim evaluation to track lower-level outcome indicators remained the same. The PBS serves as the second phase of a pre-post survey cycle, with data on the same indicators collected in both survey waves. This pre-post design allows for the determination of statistically significant change in indicators between baseline (Round 1) and the interim (Round 2) data. In addition, statistically detecting changes (if any) for all practice and behavioral change indicators can help inform the performance of the RFSA to date. Change over time of indicators in the following categories will be assessed: • Food security • Water, sanitation, and hygiene • Agriculture • Women’s health and nutrition • Children’s health and nutrition • Gender • Resilience The interim evaluation will not assess higher-level poverty11 and anthropometry12 impact indicators but rather lower-level outcome indicators. BHA may conduct a third (endline) survey in the future which could collect the HH consumption/expenditure (poverty) data and anthropometric measurements. It should be noted that the evaluation does not include data from areas in which the intervention was not implemented (i.e., a counterfactual). While the evaluation results can be used to help explain differences between the pre- and post-values and may therefore help paint a general picture about overall performance, true attribution is only possible with experimental and quasi-experimental evaluation designs, which are not being used in this evaluation. However, differences in outcomes based on the level of exposure to the activity will be assessed where feasible. 1.4 Study Team and Partners The baseline HH survey was administered in July/August of 2017 by the Mendez England and Associates (ME&A) consortium. Tulane University subcontracted the Kinshasa School of Public Health (KSPH) to conduct the second HH survey in 2021. Tulane has a long-standing relationship with KSPH, going back to the founding of the school in 1986. Dr. Pierre Akilimali, an associate professor at KSPH, directed the fieldwork. Dr. Janna Wisniewski, a faculty member from Tulane, traveled to Kinshasa to work with Dr. Akilimali on supervisor training and pilot testing. Dr. Wisniewski also oversaw the independent survey monitors, who were not associated with KSPH; they were supervised by Elvis Cidoro, a Congolese 10 The vision for the R&I approach is generally to issue up to 2-year performance-based extension to BHA resilience activities that demonstrate “outstanding” performance. The interim evaluation is one component that usually contributes to that determination. 11 The poverty module was deemed by BHA to be too long and labor-intensive relative to its value 12 Anthropometric measurements were not taken during interim data collection due to the risk of close contact during the COVID-19 pandemic. Interim Evaluation of the South Kivu FSP-Enyanya RFSA in the DRC (Vol. I) Introduction 5 national and graduate of the Tulane School of Public Health and Tropical Medicine. The Tulane team who conducted the analysis consisted of Dr. Nancy Mock, Dr. Michelle Lacey, and Peter Horjus. IMPEL | Implementer-Led Evaluation and Learning 6 Quantitative Evaluation Methods 2. QUANTITATIVE EVALUATION METHODS 2.1 Overview The PBS serves as the second phase of a pre-post survey cycle, with data on the same indicators collected in both survey waves. This pre-post design allows for the determination of statistically significant change in indicators between baseline (Round 1) and the interim (Round 2) data; however, it does not allow statements to be made about attribution or causation relating to activity impact. As such, the evaluation results can be used to help explain differences between the pre- and post-values and may therefore help paint a general picture about overall performance. True attribution is only possible with experimental and quasi-experimental evaluation designs, which are not being used in this evaluation. Data were collected in person through a population-based household survey. Baseline data collection took place in July-August 2017. The interim was administered during the same months in 2021 to avoid potential bias from seasonality. The baseline survey used a multi-stage cluster sample design. The initial plan was to have the same clusters (villages) resampled in subsequent surveys. However, the Round 2 sample needed to be modified to account for changes made to the RFSA coverage areas after the baseline was conducted (see additional information on this in the sample design section below). Additionally, activity data on the location of the implementation of various interventions of the activity are used to explore exposure to interventions, and data on self-reported household-level participation in various interventions are used to assess differences in relevant outcome indicators by reported participation. Findings from the PBS are triangulated with findings from the mid-term qualitative evaluation, the project annual reports, and other secondary data sources to develop insight into the reasons why changes in key outcomes were or were not observed and to offer recommendations for future activities. The interim PBS was conducted in compliance with both ethical and human rights standards. Survey procedures were designed to protect participants’ privacy, allowing for anonymity and voluntary participation. Ethical was obtained from the Tulane Internal Review Board as well as the KSPH Comité d'éthique (Ethics Committee) prior to the start of all fieldwork. 2.2 Sample Design The target population for the PBS was all HHs in the current FSP-Enyanya RFSA implementation area. The sampling frame consisted of a list of all villages in which the FSP-Enyanya RFSA implemented activities provided by Mercy Corps. At baseline, the implementing partners (IPs) provided a list of villages where they planned to implement their activities along with the estimated HH counts in each village; villages were grouped by health zones and health areas, and HH counts were obtained from the health zone office. This list was used to draw the baseline sample of villages using probability proportional to size systematic random sampling. Interim Evaluation of the South Kivu FSP-Enyanya RFSA in the DRC (Vol. I) Quantitative Evaluation Methods 7 During initial planning discussions for the interim evaluation, these same baseline villages were planned to be re-sampled for the interim survey in order to reduce inter-cluster sampling variance. However, in the second round of quantitative data collection, the evaluation team conferred with BHA and Mercy Corps to check whether activities had been implemented in the villages from the baseline sampling frame and if other areas that were not covered by the baseline sampling frame had been added to the implementation area. In the FSP-Enyanya RFSA implementation area, some of the originally planned coverage areas had been dropped, while others had been added. After the evaluation team consulted with BHA, BHA decided that it would be preferable to adjust the sampling frames to adjust not only for areas that were removed from the coverage area following the baseline but also to add the areas that were added to the coverage area after the baseline. This decision was based on the expressed need for current population estimates of the entire RFSA coverage area and worth the lessened degree of comparability between rounds. In the data provided for the updated sampling frame of their current implementation area, 13 villages that were in the baseline sampling frame did not end up being covered by the FSP-Enyanya RFSA activities and were therefore excluded in the Round 2 sampling frame. None of the baseline sampled villages were among those 13 villages. Additionally, 28 villages were added to the coverage area after the baseline survey and included in the Round 2 sampling frame. 13 This resulted in a final list of 165 villages in the sampling frame. The total number of villages, households, and population in the FSP-Enyanya RFSA implementation area baseline and interim sampling frames is found in Table 2. Table 2. FSP-Enyanya RFSA interim evaluation PBS sampling frame Survey round Villages Total estimated number of HHs Households added to sampling frame (new coverage areas) Interim 165 57,707** 10,042 (17% of sampling frame) Baseline 148 49,579* * Baseline derived from health zone office estimates of number of HHs in each village in the sampling frame. ** Interim derived from health zone office estimates of number of HHs in each village in the sampling frame, EXCEPT for the added villages. For the added villages, Mercy Corps did not provide the health zone office estimates, but rather provided the HHs in the village according to the FSP database, which were used as the HH counts per village for these villages. It should be noted however that the FSP database numbers for all villages in the interim sampling frame sum to 73,807, which is much greater than the estimated counts according to the health zone office estimates. 13 The 28 villages added to the RFSA after the baseline did not have data for the number of HHs in the Round 2 sampling frame. However, information for the number of HHs for all villages was available in the FSP-Enyanya RFSA’s database. Initially, Round 2 sampling plant estimates were based solely on these RFSA database HH numbers. Later, these were revised to use the RFSA database numbers only for these 28 villages to maintain consistency with the baseline sampling. Those are the numbers used here and in the data analysis weighting. IMPEL | Implementer-Led Evaluation and Learning 8 Quantitative Evaluation Methods 2.3 Sample Size The baseline survey collected data from approximately 1,300 households in 44 villages. The sampling approach/size at baseline was designed to detect an eight-percentage point change in stunting prevalence among children under 5 years of age (0–59 months) in the pre-post comparison. This approach was advised by BHA at baseline as standard practice. However, data to assess stunting was not collected in the interim performance evaluation surveys. The indicators listedin the evaluation protocol (Annex A) are used to assess pre-post improvements in the intervention areas. The RFSAs expected “substantial improvements” for each of the indicators, tentatively defined by BHA as an improvement of 25%14 over baseline levels. In early planning discussions with BHA, it was recommended that a maximum sample of approximately 1,500 households in the RFSA, with allowance for non-response, would be sufficient to measure changes for many of the desired indicators, and would allow for subsequent disaggregation of indicators by various HH characteristics (gender of household head, for example) and more complex multivariate techniques to allow for a deeper analysis of the data. The initial planning also included sampling the same 44 clusters (villages) that were in the baseline survey. However, after the survey planning began, it was found that the RFSA had made changes to their coverage area (villages added and removed from the RFSA coverage area, as described in the previous section). BHA requested to have a representative sample of the current coverage despite the loss of a small degree of comparability between the baseline and Round 2 data collection. All 44 villages sampled at baseline were found in the current Round 2 sampling frame (coverage area) for the FSP-Enyanya RFSA. As such, to maximize comparability, the same 44 villages were selected to be included in the Round 2 data collection. Then, to account for the villages added to the coverage area (and thus to the Round 2 sampling frame), we sampled six additional villages from this sub-set of the sampling frame, 15 using probability proportional to size systematic sampling. This then gave a total sample of 50 clusters/villages. The number of clusters sampled in Round 2 was larger than baseline, which means that the field work travel time between villages would exceed that initially planned. In order to stay within budget, the number of HHs per cluster was decreased slightly from baseline (30 HHs/cluster) to 27 HHs per cluster. 14 This refers to percentage change (applicable to indicators expressed as totals and as percentages), not difference in percentage points (which would refer only to differences between indicators expressed as percentages). 15 The added villages represent 12% of the total HHs in the sampling frame. This would mean adding 5.25 villages to have a proportional number of villages and HHs sampled. This was rounded up to six to ensure sufficient coverage. This slight oversampling will be accounted for in the analysis sampling weights. Interim Evaluation of the South Kivu FSP-Enyanya RFSA in the DRC (Vol. I) Quantitative Evaluation Methods 9 Table 3. Population-based survey sampled villages/clusters and sample Province/health zone Clusters sampled in BL Re-sampled clusters (R2) Newly sampled clusters (R2) Total clusters sampled (R2) Total HHs to be sampled (R2) South Kivu Province 44 44 6 50 1,350 Kalehe 16 16 0 16 432 Katana 15 15 2 17 459 Miti-Murhesa 13 13 4 17 459 TOTAL FSP￾Enyanya RFSA 44 44 6 50 1,350 2.4 Sample Selection The sample selection followed a similar approach as used at baseline. The sample for each RFSA was be selected using multi-stage cluster sampling with three stages of sampling: 1) selection of clusters (or villages), 2) selection of households, and 3) selection of individuals. For the purposes of the household survey, a “household” is defined as “a person or group of people who live together and share meals (i.e., eating from the same pot).” Full sample selection procedures are detailed in the evaluation protocol in Annex A. 2.5 Survey Tools The baseline questionnaire was developed through a series of consultations with BHA, the Food and Nutrition Technical Assistance III Project (FANTA), and the IPs. The same modules were used in Round 2, with the exception of the poverty16 and anthropometry17 modules. Additionally, questions on household-level participation in RFSA interventions were added to the HH questionnaire. The Round 2 questionnaire consists of modules covering the following topics: • Household identification and informed consent • Household roster • Household food security (HDDS and FIES) • Children’s nutrition and health • Women’s nutrition and health • Water, sanitation, and hygiene • Agriculture • Gender – Cash • Gender – Maternal and child health and nutrition • Resilience • Intervention participation 16 The poverty module was deemed by BHA to be too long and labor-intensive relative to its value. 17 Anthropometric measurements were not taken during interim data collection due to the risk of close contact during the COVID-19 pandemic. IMPEL | Implementer-Led Evaluation and Learning 10 Quantitative Evaluation Methods Data on village-level exposure to RFSA interventions was collected from the IPs. 2.6 Fieldwork and Data Collection The baseline questionnaires were programmed in Open Data Kit (ODK) by Trestle Research and Consulting. The surveys were pilot tested in a rural area on the outskirts of Kinshasa. Seven experienced supervisors were hired to oversee fieldwork in Round 2. Dr. Akilimali and Dr. Wisniewski led a supervisor training in Kinshasa in June 2021 to orient them to the topics covered by the surveys (nutrition, agricultural practices, resiliency measurement, etc.) and fieldwork activities, including sampling procedures, research ethics and informed consent, COVID-19 precautions, and electronic data collection. A representative from one of the IPs also gave an overview of the activity. Tulane and KSPH adapted the supervisor manual, enumerator manual, and question-by-question guide that were used at baseline to foster comparability between survey rounds. The Tulane team customized the manuals only to align with the final questionnaire, the general supervision approach of KSPH, and the protocol for using ODK rather than CSPro, which was the activity used at baseline.18 Supervisors practiced administering the questionnaires to each other, and then a full pilot test was held in a rural area on the outskirts of Kinshasa. Three supervisors then traveledto Sud Kivu where they recruited enumerators. All field supervisors and enumerators (including team leaders) were identified from the KSPH roster of experienced survey data collectors. These team members were recruited from the region in which they were working and spoke fluent French as well as the relevant local languages. Data collectors were trained in Bukavu. A representative from the implementing partner attended training and presented the activity overview to the enumerators. Enumerators underwent training similar to that of the supervisors, including practice administering the survey in pairs and a full pilot test. Following training, enumerators deployed in teams of two to selected villages. The “teams” are pairs of enumerators, with one of the enumerators also serving as the “team leader” (but still collecting data). In the KSPH experience, it is less costly to have a team of two stays in a village until all surveys are completed rather than having larger teams working in a single village with more frequent movement. A total of 42 enumerators were deployed in Sud Kivu. To identify the villages selected, the teams used an application called Open Street Map Automated Navigation Directions, which was installed on the tablets. GPS points for each village were provided to the field teams. Once in the villages, enumerators counted and selected HHs to participate in the survey, consented respondents, and collected data. In the case that enumerators needed to replace a village due to insecurity or road conditions, Dr. Akilimali and Dr. Wisniewski worked together to select a new village 18 Originally, the plan was to use the same data entry program and software as used in the baseline (CSPro), with only the minor changes made to the tool where needed. This was to save time and expense, as well as to ensure the data collection tools were the same in both rounds. However, when the survey preparation began, and Tulane was provided with the baseline survey program, it was discovered that it was written in an older version of CSPro that was not compatible with the current version of the software, and updating it would require starting the programming from scratch. The evaluation team chose to use ODK to re-create the data entry program to minimize the additional costs and time required, as the team had a stronger knowledge and experience with that software. Interim Evaluation of the South Kivu FSP-Enyanya RFSA in the DRC (Vol. I) Quantitative Evaluation Methods 11 that was geographically near and of similar population size to the replaced village. Data were transmitted electronically from tablets to a cloud-based server. Data collection occurred in July and August 2021. An independent survey monitor observed 20 HH surveys to assess the extent to which proper procedures were followed. The monitor used standardized data collection forms, which they transmitted electronically to a survey monitor supervisor based in Kinshasa. Data from these reports were aggregated and shared with Dr. Akilimali on an ongoing basis so that adjustments could be made quickly. The survey monitor did not report any major issues with data collection. Additionally, Dr. Akilimali ran quality checks on incoming data, including age pyramids and completion rates by each enumerator. 2.7 Data Analysis Data was exported from ODK into SPSS for initial cleaning. Then further cleaning, indicator calculation, and analysis were conducted in STATA, SPSS, and R. Tulane generated estimates for all BHA and activity-specific indicators, along with additional analyses to explore relationships and plausible determinants for key outcome indicators and a select number of resilience indicators. The syntax/do-files from the baseline were used to ensure all indicators were calculated in the same way. The indicator results tabulated in Annexes D and E take into account the populating weights and sampling design to generate estimates and conduct statistical tests, using the entire sample from both rounds (despite differences in areas of estimation). Annex D has information on the key indicator results, including the confidence intervals, design effects, record numbers, standard errors, and weighted population. Annex E shows the indicator values from both rounds, as well the statistical comparisons between rounds. Although the modifications to the sampling frame were small, to ensure they did not bias results, additional analysis of the baseline-interim changes of the outcome indicators was conducted restricting the Round 2 sample to HHs from the re-sampled villages (1,242 HHs in the Round 2 survey, baseline remains unchanged). This analysis of the re-sampled villages only was conducted without sampling weights in order to also account for the loss of statistical power and potential biases introduced by the large weights (see next section). These analyses are referred to as “resampled” in the report, and are provided only where they show important differences from the analysis of the full weighted results. 2.8 Final Sample and Sample Weights Sampling weights were computed and used in the data analyses, following the same general approach as the baseline. Weights were computed according to the unique sampling scheme that is relevant to the associated sampled household or individual. This involved computing an overall HH sampling weight for each distinct sampling group by taking the inverse of the product of the probabilities of selection from each stage of sampling (village selection and HH selection), also accounting for the correction of sampling probabilities from each sampled village (clusters) estimated number of HHs (from the sampling IMPEL | Implementer-Led Evaluation and Learning 12 Quantitative Evaluation Methods frames), and the actual number of households in the village (collected in the listing exercise during data collection). Weights were calculated for the following distinct sampling groups: • Households (used for indicators derived from Modules B, C, F, and R) • Children under 5 years of age (Module D) • Women 15–49 years of age (Module E) • Farmers (Module G) • Cash-earning adults in a union (Module J) • Parents of children under 2 years of age (Module K) Household-level weights were calculated to adjust for non-response (where HHs declined to be interviewed), or were inaccessible. In each of the individual-level sampling groups, weights were calculated to adjust for non-response based on the number of eligible individuals listed on the HH roster and the number of individuals interviewed. The non-response then accounts for individuals that declined to be interviewed (rare), and individuals that were listed in the HH roster were not located despite the repeat visits to the HH. The HH and individual-level non-response are shown in Table 4. Table 4. FSP-Enyanya RFSA final sample size and response rates Sampling group Number sampled Number interviewed Response rate Households* 1,350 1,349 99.9% Children 0–59 Months** 1,300 1,286 98.9% Women 15–49 years of age** 1,604 1,524 95.0% Farmers** 1,536 1,442 93.9% Cash earning Adults in a union** 1,010 943 93.4% Parents of children under 2 years of age ** 930 824 88.6% * For HHs, the sampled vs. interviewed reflects refusals to participate in the survey. ** For the individual-level data, “number sampled” refers to the total number of eligible HH members as recorded on the HH rosters, and the ‘number interviewed’ reflects the total number interviewed/collected data on. Non-response is both from refusal to participate by the individual (rare) or absence from the HH upon repeated visits. 2.9 Integration of Secondary Data Mercy Corps provided data on the location of the implementation of various components of the activity, which is used to assess associations by the level of exposure. Findings from the PBS are also triangulated with findings from the mid-term qualitative evaluation as well as subsequent project documents outlining adaptations to the mid-term findings in order to develop insight into the reasons why changes in key outcomes were or were not observed, and to offer recommendations for future activities. Data is also triangulated with FSP participant monitoring survey data- the Annual Participant Based Surveys (APBS) and the Seasonal Farmer Based Surveys (SFBS), which provides estimates for 2019, 2020, and 2021 for several of the indicators also reported as part of the interim evaluation. It should be noted that these surveys are representative of direct participants only, and not representative at the population level. Additionally, a full comparison of the survey tools, methodologies, indicator Interim Evaluation of the South Kivu FSP-Enyanya RFSA in the DRC (Vol. I) Quantitative Evaluation Methods 13 construction, etc. was not conducted. Certain indicators may have been collected and/or calculated differently in the recurrent monitoring surveys as compared to the evaluation surveys. 2.10 Limitations and Delimitations This evaluation has several limitations, defined as factors outside the evaluators’ scope of influence. 1. As the evaluation is only based on data from areas that received activity support, the evaluation cannot conclude whether a RFSA caused an observed change in outcomes. 2. The baseline and interim surveys were administered several years apart and by different organizations. While efforts were made to maintain consistency by using the same fieldwork manuals and question-by-question guidance, by incorporating input from IPs, and by rehiring some of the enumerators that worked on the baseline, differences in the administration of the two surveys may have occurred. 3. Although independent survey monitors assessed compliance with fieldwork procedures and data quality checks were run, it is possible that there are quality issues, either in the baseline or interim data sets that cannot be identified. 4. Household exposure to the activity was measured by asking HH members about their participation in various activities. It is possible that participation was under-reported, as heads of HHs may not have been aware of activity names or IPs or whether other members of their HH had participated. 5. Inaccuracies of the sampling frame required large population weights adjustments in some of the sampled villages in both Rounds 1 and 2. The villages with outlying HH weighting values, particularly those with large weights, tend to increase the design effects and reduce the statistical power of the analysis. This also means that the population estimates in the areas of implementation are potentially unreliable. The evaluation also has several delimitations, defined as factors that the evaluation team and BHA agreed to accept, though they were less than ideal. 1. The poverty and anthropometry modules were not administered in Round 2. The poverty module was deemed to be too long and labor-intensive relative to its value, and the physical proximity of enumerators to women and children while taking anthropometric measurements was considered too risky during the COVID-19 pandemic. 2. Where the baseline and interim sampling frames overlapped, villages surveyed in the baseline were re-sampled. However, additional villages—in areas where the RFSA coverage had expanded to—were also included in the sample. This may have some impact on the comparability of the two rounds. Following comments from the IPs after data collection, additional analyses were run comparing only the HHs from the resampled villages, using the sub-set of the Round 2 sample. (See detailed description in the analysis section.) IMPEL | Implementer-Led Evaluation and Learning 14 Findings 3. FINDINGS This section begins with an overview and basic analysis of RFSA activity exposure and participation. Then, results are presented by sector. Indicators are calculated at baseline and interim for the entire sample, and the statistical significance of the difference between the two rounds is calculated. Indicators are also stratified (e.g., by gender) as appropriate. As applicable, analysis of the components of the indicators is conducted, particularly if that analysis provides additional information that may alter the conclusions drawn from the findings. If the findings differ when making comparisons between rounds in the unweighted analysis of the resampled villages only, then this is noted (this occurs only for a few of the indicators). The comparative analysis for each sector is followed by an analysis of the relationship between RFSA intervention self-reported exposure and relevant outcomes. This analysis makes use of the full interim sample with probability weights applied (unless otherwise noted). As such, those are representative of the entire coverage area. Finally, data from the FSP recurrent monitoring data (the APBS and SFBS), the 2019 MTE, the 2020 annual report, and other sources are used throughout the findings to triangulate or shed additional light on the findings. 3.1 Intervention Exposure and Participation To assess the levels of RFSA intervention participation and coverage/exposure in the surveyed, data were collected at the community and household level related to a variety of RFSA interventions. When presenting these data, exposure to an intervention is defined as a household living in a village where the RFSA implemented the intervention (data sourced from the IP). Participation in an intervention is defined (and measured here) as the survey respondent reporting that one or more members of their household had participated in the intervention over the previous 5 years. It is important to note, however, that participation data collected in the Round 2 survey is not meant to be interpreted as providing accurate estimates of coverage by the RFSA, which is better estimated by activity monitoring data. The primary use of the information collected in Round 2 on intervention participation is to allow for analysis of associations (correlations) between lower-level outcome indicators and intervention participation. Overall, 54% of HHs in the Round 2 survey reported participation in one or more of the surveyed RFSA interventions. According the RFSA’s 2020 annual report, the activity was reaching approximately 52,000 unique direct participant HHs. The FSP database indicates that roughly 74,000 HHs are found in the area Interim Evaluation of the South Kivu FSP-Enyanya RFSA in the DRC (Vol. I) Findings 15 of implementation. 19 This would indicate that roughly 70% of HHs in the implementation area were direct participants in one or more RFSA activity. Activity exposure was high for most interventions (Figure 2 below, table in Annex C). The exceptions were toilet building (54% coverage) and food for work activities (15% coverage). Participation ranged from 4% (safe space groups) to 24% (WASH training/events). Low coverage of certain interventions must be considered when assessing population-level changes. For example, the interim evaluation survey showed that even though over 80% of HHs lived in villages where safe spaces were being implemented, less than 5% of HHs reported participation. This reflects the mid-term findings that FSP-Enyanya RFSA was reaching about 3,500 adolescents via Safe Spaces in 2019, with plans to double that by the end of the activity. This is a very small percentage of the adolescent population, unable to have any meaningful impact at the population level. One of the key findings of the mid-term evaluation of the FSP-Enyanya RFSA was that there might be too many initiatives (particularly trainings) without adequate in-depth follow-up to ensure long-term positive impacts. Reporting error is possible, where the respondent (head of household or other decision maker) was simply not aware of the participation of other household members, for example. Respondent bias due to gender did not appear to play a role, however. The reported participation in mother’s groups and nutrition trainings, for example, showed no significant difference between male and female respondents. 19 Note that the FSP database indicates a larger number of HHs in the implementation area than the sampling frame data sourced from the health zone office. Estimations of coverage are challenging to make when the total population estimate is extremely imprecise. IMPEL | Implementer-Led Evaluation and Learning 16 Findings Figure 2. FSP-Enyanya RFSA intervention exposure and household participation *Interventions with no coverage in the sampled villages are excluded. These include: Training in alternative livelihoods/vocational skills, and cash-for-work. Analysis was conducted to assess associations between reported participation in various interventions and relevant outcomes in the following sector results sections. However, this analysis does not account for potential biases that could stem from self-selection and/or intervention targeting (as part of intentional design). Household demographics did not generally differ between participant and non-participant households (male/female headed vs. single female vs. single male, household size). 0% 20% 40% 60% 80% 100% VSLA Employment Training Farmers' Groups Resource Management Agricultural Training Improved Agricultural Production Agricultural Inputs Farming Information Market Support WASH Training Toilet Building Mothers' Groups Nutrition Training Home Visits Safe Space Groups Climate Adaptation Groups Food For Work Programs Literacy Training Couples' Support Groups Adolescent Life Skills Training Youth Leadership Training Intervention Exposure and Participation % HHs reporting one or more members participated in this intervention in the past year (ALL villages) % HHs living in villages where the intervention was done by the RFSA Interim Evaluation of the South Kivu FSP-Enyanya RFSA in the DRC (Vol. I) Findings 17 Differences between participants and non-participants include: • Literate households (households that reported having at least one literate adult member) were at least twice as likely to report participation in any intervention compared to illiterate households. Participation rates among literate households were at least three times greater for many interventions including mothers’ groups, nutrition training, couples’ support, and natural resource management (NRM) programs. • Households with high HDDS (>6) had higher participation rates in nearly all interventions compared to those with lower HDDS. • Involvement with collective actions to benefit the community was strongly associated with higher rates of reported intervention participation. • Households with cash savings were at least twice as likely to report participation in any intervention compared to those with no cash savings. It is important to note that true antecedent variables (those that would remain relatively unaffected by project participation) are limited. Apart from demographic indicators, the rest of the indicators above may have been impacted by participation in RFSA activities. However, the trends seen in the differences in participants vs. non-participants may suggest that participants in many interventions were more likely to be economically better off and more engaged in their communities than those who did not report participation. The RFSA interventions often relied on community leaders (lead mothers, lead fathers, etc.). These leaders are likely to be from more literate, better off households. If large numbers of these leaders’ households were surveyed and identified as intervention participants, then this could be a potential source of bias. The design of the evaluation is insufficient to fully assess these potential biases. 3.2 Water, Sanitation, and Hygiene Table 5 displays a comparison of Water, Sanitation, and Hygiene (WASH) indicators between 2017 and 2021. The percentage of HHs with a basic (improved) drinking water source increased from 41% to 54%, although this was only marginally significant. 20 The unweighted analysis of the resampled areas showed a strongly significant increase from 44% to 52%. The percentage of HHs with an improved source of drinking water available within 30 minutes increased significantly, with nearly 40% of HHs having such access in 2021. Similarly, the percentage of HHs that could obtain drinking water in less than 30 minutes (round trip) increased by 13.4 percentage points, a statistically significant improvement. In terms of sanitation, the percentage of HHs with soap and water at a handwashing station commonly used by family members decreased significantly to only 1% in 2021. 21 Open defecation also remained low, with no significant change. 20 Drinking water source often has a very high design effect in cluster samples. The DEFF for basic (improved) drinking water at baseline was 21, and at interim it was 8.9. The high design effect at baseline reduced the power of the sample to detect significant differences in this variable. 21 This decrease was not significant in the unweighted analysis of the re-sampled areas only. IMPEL | Implementer-Led Evaluation and Learning 18 Findings The percentage of HHs using a basic sanitation facility was low in both rounds, and showed no significant change. However, possible discrepancies in latrine type classifications in both the baseline survey and in the interim survey may be masking change. The baseline data indicates that over 80% of HHs had a “water flush to pit latrine” type toilet, which was later classified as simply unimproved pit latrine, both classified as unsafe/unimproved sanitation. The interim survey reports that <1% of HHs have flush-to-pit-latrine toilets, which is similar to the most recent DHS survey (0.5% of urban and 0% of rural HHs). The interim survey, on the other hand, has a large percentage of unimproved pit latrines (pit latrines without a slab base). The recurrent monitoring data from FSP looking at participant HHs only show a concordance for change in basic drinking water, use of water treatment technologies, basic sanitation facilities, though FSP consistently measured higher prevalence of these indicators (as would be expected from their participant-only survey). Handwashing data from FSP was somewhat variable among the three rounds of the APBS, but also reported higher prevalence of this indicator. Table 5. Baseline and interim water, sanitation, any hygiene indicator comparisons Indicator Indicator value Raw difference (Int.–BL) Significance level* Number of observations 2017 BL 2021 Int. BL Int. 6. Percentage of households using an improved drinking water source 40.8% 54.0% 13.2% † 1,267 1,349 Available on premises 1.6% 1.6% 0.0% ns 1,267 1,349 Available in 30 minutes or less 29.6% 36.8% 7.2% * 1,267 1,349 Available in more than 30 minutes 9.7% 14.1% 4.4% ns 1,267 1,349 7. Percentage of households practicing correct use of recommended household water treatment technologies 3.0% 5.1% 2.1% ns 1,267 1,349 Chlorination 0.8% 1.5% 0.7% ns 1,267 1,348 Flocculent/Disinfectant 0.0% 0.0% 0.0% na 1,267 1,348 Filtration 0.3% 0.1% -0.2% ns 1,267 1,348 Solar 0.1% 0.0% -0.1% ns 1,267 1,348 Boiling 1.9% 3.5% 1.6% † 1,267 1,348 8. Percentage of households that can obtain drinking water in less than 30 minutes (round trip) (GS) 38.6% 74.8% 36.2% *** 1,267 1,337 9. Percentage of households using a basic sanitation facility 3.5% 2.5% -1.0% ns 1,267 1,349 10. Percentage of households in target areas practicing open defecation 2.4% 3.0% 0.6% ns 1,267 1,349 11. Percentage of households with soap and water at a handwashing station commonly used by family members 6.5% 1.0% -5.5% *** 1,267 1,349 Note: ns = not significant, na = not available, † p < 0.1, * p < 0.05, *** p < 0.001, green shading (GS): Indicator updated from value reported in baseline report Interim Evaluation of the South Kivu FSP-Enyanya RFSA in the DRC (Vol. I) Findings 19 WASH training/activities were implemented in most villages, representing 92% of HHs. 27% of HHs reported participation in this activity, although there were no significant associations with handwashing, water treatment, or use of sanitation improved sanitation facilities. Toilet building activities were implemented in villages representing 52% of HHs; however, the proportion of HHs who reported participating in toilet building was nearly identical among HHs where the intervention was present (21%) as in those where it was not implemented (17%). 3.3 Agriculture Changes in agricultural practices between 2017 and 2021 are shown in Table 6. The percentage of farmers who used financial services in the past 12 months increased significantly, from 32% at baseline to 50% in 2021. This significant increase was observed for both male and female farmers, although the increase for male farmers was higher than for females. The percentage of farmers using value chain activities promoted by the project observed a small but non-significant improvement. However, in the unweighted analysis of the re-sampled areas only, the use of value chain activities saw a significant increase, from 40% at baseline to 53% at interim. The percentage of farmers who used improved storage practices in the past 12 months also increased significantly overall and for male and female farmers, again, with male farmers experiencing a larger percentage point increase. The recurrent monitoring data from FSP-Enyanya show similar increases in the percentage of farmers who used one or more financial services, practicing value chain activities, and improved storage practices. The monitoring data tends to show a higher prevalence however, which is expected from their participant only sample. The 2019 mid-term evaluation reported that the PO interventions (creating/strengthening Producer Organizations, post-harvest handling/storage, linkages to inputs, financial or technical services, and market linkages) were either weak, behind schedule, or non-existent. However, the 2020 annual report indicates that progress had been made following the MTE in implementing these activities, which appears to be generally reflected in these findings. Significant decreases were observed in the percentage of farmers who used at least four sustainable agriculture (crop, livestock, and NRM) practices and/or technologies in the past 12 months. 22 This was mainly a result of the drop in the number of sustainable crop practices used by farmers (one of the components used to make this variable). The prevalence of farmers using four or more sustainable crop practices dropped from 14% of farmers at baseline to 3% at interim. No significant changes were seen in the use of sustainable livestock or NRM practices. (see further discussion on these indicators below) The mid-term reported delays in the implementation of many of the agriculture interventions (into 2019). Despite advancements post-mid-term in the implementation of these interventions, this gives, little time for farmers to adopt these practices at the time of the survey. 22 The baseline value for indicator 14 has been recalculated. The baseline calculations used a threshold of 3 or more, but reported it as 4 or more. Both baseline and interim now reflect the correct threshold (4 or more) as defined by BHA and the IP at baseline. IMPEL | Implementer-Led Evaluation and Learning 20 Findings Table 6. Baseline and interim agriculture indicator comparisons Indicator Indicator value Raw difference (Int.– BL) Significance level* Number of observations 2017 BL 2021 Int. BL Int. 12. Percentage of farmers who used financial services (savings, ag credit, &/ or ag insurance) in the past 12 months 32.0% 49.9% 17.9% *** 1,510 1,442 Male 35.2% 52.4% 17.2% ** 700 755 Female 29.2% 42.7% 13.5% *** 810 687 13. Percentage of farmers who practiced value chain activities promoted by the activity in the past 12 months (GS) 43.6% 51.1% 7.5% ns 819 865 Male (GS) 46.3% 51.5% 5.2% ns 420 487 Female (GS) 40.8% 50.5% 9.7% ns 399 378 14. Percentage of farmers who used at least four sustainable agriculture (crop, livestock, and NRM) practices and/or technologies in the past 12 months (GS) 20.1% 4.5% -15.6% *** 1,510 1,442 Male (GS) 23.2% 4.8% -18.4% *** 700 755 Female (GS) 17.3% 4.1% -13.2% *** 810 687 15. Percentage of farmers who used at least 4 sustainable crop practices and/ or technologies in the past 12 months 14.3% 3.1 -11.2% *** 1,492 1,423 Male 16.5% 3.4% -13.1% ** 692 744 Female 12.2% 2.8% -9.4% *** 800 679 16. Percentage of farmers who used at least 1 sustainable livestock practice and/or technology in the past 12 months 2.5% 3.1% 0.6% ns 627 634 Male 4.0% 2.7% -1.3% ns 321 382 Female 0.9% 3.8% 2.9% † 306 252 17. Percentage of farmers who used at least 3 sustainable NRM practices and/ or technologies in the past 12 months 0.1% 0.0% -0.1% na 1,510 1,442 Male 0.2% 0.0% -0.2% na 700 755 Female 0.0% 0.0% 0.0% na 810 687 18. Percentage of farmers who used improved storage practices in the past 12 months 40.3% 52.7% 12.4% * 1,492 1,423 Male 40.7% 54.8% 14.1% ** 692 744 Female 40.0% 50.5% 10.5% ns 800 679 Note: ns = not significant, na = not available, † p < 0.1,* p < 0.05, ** p < 0.01, *** p < 0.001, and green shading: indicator updated from value reported in baseline report Interim Evaluation of the South Kivu FSP-Enyanya RFSA in the DRC (Vol. I) Findings 21 FSP recurrent monitoring data show varying levels of the use of four or more sustainable crop practices among the direct participants they interviewed (46% of farmers in 2019, 38% in 2020, and 54% in 2021). One hypothesis suggested by the IP is that farmers may have interpreted the question about what crop practices/technologies had been “used” in the past 12 months as what crop practices had been “added” in the past 12 months. This hypothesis cannot be tested with the data available, but future surveys may want to consider this possible source of bias when implementing fieldwork. Table 7 below shows the change reported use of each type of sustainable crop practice/technology. There were decreases for each practice/technology except for terracing, which increased by 9.3 percentage points, and tied ridges, which was low and unchanged. The largest decreases were observed in mulching, manure, and weed control, all of which were promoted by the activity. Overall, however, the percentage of farmers that reported using at least one of the sustainable crop practices over the past 12 months remained high, and relatively unchanged (88.8% at baseline, 88.5% at interim). This discrepancy is accounted for primarily by the fact that at baseline, farmers were recorded as practicing a greater number of sustainable crop practices than at interim. It’s difficult to assess whether these differences between rounds stem from methodology/collection issues (differences in how the questions were administered between rounds, or from a misunderstanding of the practices by the enumerators in the baseline and/or interim surveys for example). They may also represent actual changes in the type and number of practices used. Table 7. Changes in prevalence of sustainable crop practices between baseline (2017) and interim (2021) Crop Practice* Baseline Interim Difference Manure 40.8% 22.8% -18.0% Compost 57.7% 51.0% -6.7% Mulching 31.2% 12.6% -18.6% Weed control 40.4% 23.0% -17.4% Ripping into residues 11.1% 6.1% -5.0% Crop rotations 19.1% 10.2% -8.9% Contour planting 1.7% 0.4% -1.3% IPM 0.5% 0.1% -0.4% Improved seeds/crop varieties 0.6% 0.7% 0.1% Dry Planting 7.8% 0.3% -7.5% Clean ripping 2.5% 1.4% -1.1% Tied ridges 0.1% 0.1% 0.0% Zai pits 1.0% 0.0% -1.0% Potholing 0.1% 0.0% -0.1% Intercropping 30.8% 22.0% -8.8% Terracing 3.8% 13.1% 9.3% Land leveling 1.5% 0.2% -1.3% Improved fallow with cover 0.6% 0.2% -0.4% IMPEL | Implementer-Led Evaluation and Learning 22 Findings Crop Practice* Baseline Interim Difference Maintain indigenous trees to improve soil fertility 1.5% 0.1% -1.4% Planting of perennial forage crops 0.3% 0.0% -0.3% Used at least ONE of these practices in the past 12 months 88.8% 88.5% -0.3% *Practices with BOLD font are promoted by the activity. These are used to calculate the relevant outcome indicators. Table 8 shows the breakdown by livestock practice/technology. The prevalence of most livestock practices decreased by small amounts. The only increase was observed in practices for breeding and keeping of rabbits, which was the sole activity promoted by the activity that was collected in the evaluation surveys. However, this practice remained infrequent. The mid-term evaluation found that comprehension by rabbit recipients of how to raise, manage, and breed the animals was limited and varied, and the mortality rate of the rabbits was high. The 2020 annual report indicates that some progress was made during year 4 to support rabbit production, and that mortality rates of the rabbits decreased from 53% in year 3 to 23% in year 4. However, FSP staff indicated that the coverage of the rabbit intervention remained small and had only begun fully as of 2020. The number of farmers breeding/keeping rabbits may continue to grow over time, however, as rabbits enter the market, and more farmers adopt the practice. Table 8. Changes in prevalence of livestock practices and technologies between baseline (2017) and interim (2021) Livestock practice* Baseline Interim Difference Practices for breeding and keeping of rabbits 2.5% 3.1% 0.6% Improved animal shelters 6.9% 2.7% -4.2% Vaccinations 11.0% 9.8% -1.2% Deworming 10.7% 7.9% -2.8% Castration 1.5% 0.2% -1.3% Dehorning 1.0% 0.6% -0.4% Homemade animal feeds from local products 8.6% 4.6% -4.0% Animal feed supplied by stockfeed manufacturer 1.4% 1.2% -0.2% Artificial insemination 0.7% 0.3% -0.4% Pen Feeding 8.8% 8.1% -0.7% Fodder production and/or veld reinforcement 1.9% 1.0% -0.9% Used the services of community animal health workers/para￾veterinarians 2.0% 0.7% -1.3% Make hay or silage to feed animal during the dry season 0.4% 0.4% 0.0% Feed animals with nutritional supplements during the hunger season 4.8% 2.1% -2.7% Used at least ONE of these practices in the past 12 months (B) 38.7% 31.3% 7.4% *Practices with BOLD (B) font are promoted by the activity. These are used to calculate the relevant outcome indicators. Participation in VSLAs was reported by 16.9% of farmers and was associated with a significantly higher rate of access to financial services, while sustainable agriculture practices were positively associated Interim Evaluation of the South Kivu FSP-Enyanya RFSA in the DRC (Vol. I) Findings 23 with participation in farmers’ groups (21.4%) and/or NRM programs (11.8%). There were no significant associations between self-reported participation in any of the agriculture interventions and value chain activities or improved storage practices. 3.4 Women’s Health and Nutrition Indicators of women’s health and nutrition are displayed in Table 9. The prevalence of women of reproductive age consuming a diet of minimum diversity decreased by 8.1 percentage points between 2017 and 2021, a statistically significant change. There was a significant increase in the percentage of women who had eaten orange-flesh sweet potatoes in the past 24 hours and a significant decrease in consumption of cabbage and animal protein. The prevalence of women of reproductive age who consume targeted nutrient-rich value chain and non-value chain commodities also decreased, though the change was not significant. 23 FSP recurrent monitoring data showed no large variation in women’s consumption of a diet of minimum diversity, or in the consumption of nutrient-rich commodities (only measured in 2020 and 2021). However, their estimates for their direct participant￾only sample are higher than observed in either the baseline or interim surveys. Attendance at mother’s groups, which involved activities such as nutrition training and home gardening, significantly improved MDD levels among the 15% of women who participated, among which 25% achieved a MDD, compared to 19% among non-participating women. The contraceptive prevalence rate did not change significantly, though the percentage of women who reported contraceptive use shifted by over 13% among those who attended mothers’ groups. FSP monitoring data also show little change in contraceptive use between 2019 and 2021, though they show higher prevalences (among the participant population they surveyed). The prevalence of underweight women24 and births receiving at least four antenatal care visits were not assessed at interim. 25 Table 9. Baseline and interim women’s health and nutrition indicator comparisons Indicator Indicator value Raw difference (Int.–BL) Significance level* Number of observations 2017 BL 2021 Int. BL Int. 20. Prevalence of women of reproductive age consuming a diet of minimum diversity 24.4% 16.3% -8.1% * 1,327 1,524 21. Contraceptive Prevalence Rate 11.3% 9.3% -2.0% ns 571 718 Modern methods 7.2% 6.2% -1.1% na 571 718 Traditional methods 4.2% 3.7% -0.5% ns 571 718 23 Unweighted analysis of the resampled areas show that this indicator decreased significantly (36% at baseline to 28% at interim). 24 Anthropometry measurements of women were not collected at interim, as the physical proximity of enumerators while taking anthropometric measurements was considered too risky during the COVID-19 pandemic. 25 An error in the skip patterns in the data collection program resulted in excessive missing data for this variable at baseline. IMPEL | Implementer-Led Evaluation and Learning 24 Findings Indicator Indicator value Raw difference (Int.–BL) Significance level* Number of observations 2017 BL 2021 Int. BL Int. 23. Prevalence of women of reproductive age who consume targeted nutrient-rich value chain and non-value chain commodities 32.7% 26.4% -6.3% ns 1,326 1,524 Value chain (VC) commodities 20.6% 20.0% -0.6% ns 1,326 1,524 Bio-fortified (cassava, maize, beans) (VC) 20.6% 20.0% -0.6% ns 1,326 1,524 Non-value chain commodities 12.1% 6.4% -5.7% ** 1,326 1,524 Orange-flesh sweet potatoes (GS) 1.6% 5.1% 3.5% * 1,326 1,524 Soybean (GS) 28.4% 20.2% -8.2% † 1,326 1,524 Cabbage 13.5% 6.2% -7.4% *** 1,326 1,524 Cowpea (GS) 0.5% 0.2% -0.3% ns 1,326 1,524 Carrots 0.0% 0.1% 0.1% ns 1,326 1,524 Moringa 0.0% 0.0% 0.0% ns 1,326 1,524 Orange 1.5% 1.5% 0.1% ns 1,326 1,524 Pineapple (GS) 0.6% 1.0% 0.4% ns 1,326 1,524 Passionfruit 0.9% 1.4% 0.5% ns 1,326 1,524 Mango (GS) 9.9% 8.1% -1.8% ns 1,326 1,524 Okra 0.0% 0.2% 0.2% na 1,326 1,524 Sweet green pepper (GS) 2.4% 10.0% 7.6% ns 1,326 1,524 Eggs (GS) 2.4% 3.0% 0.6% ns 1,326 1,524 Animal protein (GS) 75.3% 49.9% -25.4% *** 1,326 1,524 Note: ns = not significant, na = not available, green shading: indicator value not provided in baseline report, calculated using baseline database, † p < 0.1, * p < 0.05, ** p < 0.01, *** p < 0.001 3.5 Children’s Health and Nutrition Indicators of the health and nutrition of children are shown in Table 10. 26 There was a significant decrease in the percentage of children under 5 years who had diarrhea in the last 2 weeks. This decrease was larger for female children (16.3%) than for male children (12.0%). Diarrhea prevalence was also observed to be dropping in the FSP recurrent monitoring data, and at similar prevalences to the interim findings. Among children that were reported to have had diarrhea, there were no significant changes observed for diarrhea treated with ORT. However, the decreased diarrhea prevalence means that the number of observations at interim was small. Activity monitoring 2021 data from Mercy Corps indicate that among 26 Anthropometric measurements were not taken during interim data collection due to the risk of close contact during the COVID-19 pandemic. If possible, they will be taken at endline. Interim Evaluation of the South Kivu FSP-Enyanya RFSA in the DRC (Vol. I) Findings 25 participant households, 71% of children with diarrhea are treated with ORT. Low coverage among the interim survey households does not allow for analysis of participant vs. non-participant households, and changes measured at the overall population level may be too diluted to show any overall change. The prevalence of children 6–23 months who consumed targeted nutrient-rich value chain and/or non￾value chain commodities increased by 10.2 percentage points, a statistically significant change. This was driven largely by female children, whose consumption increased by 18.0 percentage points compared to a slight decrease among male children. Overall, there were statistically significant increases in the percentage of children who ate bio-fortified foods and orange-flesh sweet potatoes, and a decrease in the consumption of animal protein. There were no significant changes observed for exclusive breastfeeding, or prevalence of a minimum acceptable diet. FSP monitoring data of exclusive breastfeeding among direct participants showed a decrease between the 2019 and 2020 values (81% and 83%) to 57% in their 2021 APBS, which is similar to the interim estimate for the population in the intervention areas. Further investigation to these changes may be useful. Anthropometric measurements of children were only taken at baseline. 27 Table 10. Baseline and interim children’s health and nutrition indicator comparisons Indicator Indicator value Raw difference (Int.–BL) Significance level* Number of observations 2017 BL 2021 Int. BL Int. 27. Percentage of children under age 5 who had diarrhea in the last 2 weeks 33.9% 19.6% -14.3% *** 1,398 1,286 Male 32.7% 20.7% -12.0% *** 660 625 Female 34.8% 18.6% -16.3% *** 738 661 28. Percentage of children under age 5 with diarrhea treated with ORT 41.6% 32.3% -9.4% ns 449 253 Male 47.2% 32.5% -14.7% * 211 133 Female 37.4% 32.1% -5.3% ns 238 120 29. Prevalence of exclusive breast-feeding of children under 6 months of age 60.6% 55.3% -5.4% ns 164 156 Male 59.6% 55.8% -3.9% ns 77 69 Female 61.6% 54.9% -6.7% † 87 87 30. Prevalence of children 6–23 months of age receiving a minimum acceptable diet (MAD) 6.1% 6.7% 0.6% ns 390 388 Male 7.7% 5.1% -2.6% ns 186 186 Female 5.0% 8.0% 3.0% ns 204 202 27 Anthropometric measurements were not taken during interim data collection due to the risk of close contact during the COVID-19 pandemic. If possible, they will be taken at endline. IMPEL | Implementer-Led Evaluation and Learning 26 Findings Indicator Indicator value Raw difference (Int.–BL) Significance level* Number of observations 2017 BL 2021 Int. BL Int. 31. Prevalence of children 6–23 months who consume targeted nutrient-rich value chain and/or non-value chain commodities 22.7% 32.9% 10.2% * 390 388 Male 31.9% 31.8% -0.2% ns 186 186 Female 15.7% 33.8% 18.0% * 204 202 Value chain (VC) commodities 13.6% 26.9% 13.3% ** 390 388 Male 17.9% 24.7% 6.7% ns 186 186 Female 10.3% 28.7% 18.4% ** 204 202 Bio-fortified (cassava, maize, beans) (VC) 13.6% 26.9% 13.3% ** 390 388 Non-value chain commodities 9.1% 6.0% -3.1% ns 390 388 Male 14.0% 7.1% -6.9% † 186 186 Female 5.4% 5.0% -0.4% ns 204 202 Orange-flesh sweet potatoes (GS) 4% 9.4% 5.5% * 390 388 Soybean (GS) 25% 23.5% -1.5% ns 390 388 Cabbage 8.4% 5.0% -3.4% ns 390 388 Cowpea 0% 0.1% 0.1% na 390 388 Carrots 0.0% 0.0% 0.0% na 390 388 Moringa 0.0% 0.0% 0.0% na 390 388 Orange (GS) 0.5% 2.7% 2.2% † 390 388 Pineapple 0.90% 0.7% -0.2% ns 390 388 Passionfruit 1.5% 1.0% -0.5% ns 390 388 Mango (GS) 4.80% 7.9% 3.1% ns 390 388 Okra 0% 0.3% 0.3% ns 390 388 Sweet green pepper (GS) 0.2% 0.0% -0.2% na 390 388 Eggs (GS) 3% 2.4% -0.5% ns 390 388 Animal protein (GS) 59% 38.7% -20.3% *** 390 388 Note: ns = not significant, na = not available, green shading: indicator value not reported in baseline, calculated using baseline database,, † p < 0.1, * p < 0.05, ** p < 0.01, *** p < 0.001 Mothers’ groups, nutrition training, and home health visits were implemented in all communities. However, only 16% of children under 5 years of age lived in HHs that reported participating in mother’s groups, 17% in nutrition trainings/meetings, 20% in home visits for counseling. (12%, 14%, and 18% of HHs reported engaging in each of these activities, respectively). Furthermore, there were no associations between reported participation in these activities and any improvements in child MAD, breastfeeding practices, or frequencies of diarrhea. Interim Evaluation of the South Kivu FSP-Enyanya RFSA in the DRC (Vol. I) Findings 27 3.6 Gender Changes in indicators related to gender are displayed in Table 11. It should be noted that gender norms are deeply entrenched and will take a long time to change, so lack of change is not unexpected in many of the gender indicators. There was a non-significant increase in the percentage of adults earning cash in the past year among both men and women. However, the unweighted analysis of the resampled areas showed a significant increase in adult men earning cash (36% to 43%). The FSP monitoring data indicates that the percentage of adults men and women earning cash has remained unchanged between 2019 and 2021, with some evidence of increase in 2021. The figures collected from these direct-participant surveys are somewhat higher than the population-level findings. There was a marginally significant increase in the percentage of people with children under 2 years who have knowledge of maternal and child health and nutrition practices; this increase was driven by increases among women. Unweighted analysis of the resampled areas showed similar, though more strongly significant changes. However, there was no association between participation in Mothers’ groups, nutrition training, or home health visits and MCHN knowledge. Additionally, FSP monitoring data showed similar levels of MCHN knowledge among the direct participants interviewed, but there was no evidence of change between 2019 and 2021. There was a marginally significant decrease in the percentage of women in union and earning cash who make decisions alone about the use of self-earned cash. No other significant changes were observed in the decision-making indicators. The FSP monitoring data indicates similar findings of little or no change in decision-making indicators in their direct participant surveys. They highlight, however, that there are important construct validity problems with these indicators, as well as the fact that gender norms may take decades to see real change. Table 11. Baseline and interim gender indicator comparisons Indicator Indicator value Raw difference (Int.–BL) Significance level* Number of observations 2017 BL 2021 Int. BL Int. 32. Percentage of men and women who earned cash in the past 12 months 30.7% 36.2% 5.5% ns 3,708 3,788 Male 35.3% 42.7% 7.4% * 1,781 1,762 Female 26.4% 30.5% 4.1% ns 1,927 2,026 33a. Percentage of men in union and earning cash who make decisions alone about the use of self-earned cash 28.9% 28.5% -0.4% ns 491 581 33b. Percentage of women in union and earning cash who make decisions alone about the use of self-earned cash 18.0% 10.1% -7.9% † 377 362 34a. Percentage of men in union and earning cash who make decisions jointly with spouse/partner about the use of self￾earned cash 45.0% 50.9% 5.9% ns 491 581 IMPEL | Implementer-Led Evaluation and Learning 28 Findings Indicator Indicator value Raw difference (Int.–BL) Significance level* Number of observations 2017 BL 2021 Int. BL Int. 34b. Percentage of women in union and earning cash who make decisions jointly with spouse/partner about the use of self￾earned cash 45.1% 52.4% 7.3% ns 377 362 35. Percentage of men and women with children under 2 years who have knowledge of maternal and child health and nutrition (MCHN) practices 55.9% 66.0% 10.1% † 860 854 Male 50.5% 57.8% 7.3% ns 368 376 Female 60.6% 72.2% 11.6% * 492 478 36a. Percentage of men in union with children under 2 years who make maternal health and nutrition decisions alone 18.4% 22.2% 3.8% ns 368 376 36b. Percentage of women in union with children under 2 years who make maternal health and nutrition decisions alone 18.9% 16.1% -2.8% ns 426 448 37a. Percentage of men in union with children under 2 years who make maternal health and nutrition decisions jointly with spouse/partner 38.7% 47.0% 8.3% ns 368 376 37b.Percentage of women in union with children under 2 years who make maternal health and nutrition decisions jointly with spouse/partner 41.4% 38.6% -2.8% ns 426 448 38a. Percentage of men in union with children under 2 years who make child health and nutrition decisions alone 14.2% 13.0% -1.2% ns 368 376 38b. Percentage of women in union with children under 2 years who make child health and nutrition decisions alone 23.1% 18.9% -4.2% ns 426 448 39a. Percentage of men in union with children under 2 years who make child health and nutrition decisions jointly with spouse/partner 44.1% 54.1% 10.0% ns 368 376 39b. Percentage of women in union with children under 2 years who make child health and nutrition decisions jointly with spouse/partner 41.5% 46.3% 4.8% ns 426 448 Note: ns = not significant, † p < 0.1,* p < 0.05 Interim Evaluation of the South Kivu FSP-Enyanya RFSA in the DRC (Vol. I) Findings 29 3.7 Activity-Specific Custom indicators created at baseline are shown in Table 12. No statistically significant changes were seen in any of the indicators. The FSP monitoring data also showed no change in the percentage of women participating in community decision making bodies, but the prevalence among direct beneficiaries was much higher (around 81% for each of the three APBS surveys). Data on livestock penning was not collected at interim. 28 Table 12. Baseline and interim activity-specific indicator comparisons Indicator Indicator value Raw difference (Int.– BL Significance level* Number of observations 2017 BL 2021 Int. BL Int. 46. Percentage of children under 2 whose caregivers who properly disposed of child feces 71.5% 77.8% 6.3% ns 552 542 Male 68.9% 78.2% 9.3% ns 262 254 Female 73.3% 77.5% 4.2% ns 290 288 47. Percentage of men/women who say it is ok for a man to batter his wife for any reason 48.4% 46.8% -1.6% ns 1,512 1,194 Male 50.8% 49.7% -1.1% ns 700 755 Female 46.4% 43.7% -2.7% ns 812 448 48. Percentage of women that report participating in community decision￾making bodies 31.4% 34.4% 3.0% ns 1,327 1,524 Note: ns = not significant, † p < 0.1, *** p < 0.001 3.8 Shock Exposure and Resilience Indices Significant declines in the shock exposure indices (both unweighted and severity weighted) relative to the baseline were predominantly due to sizable drops in the proportion of households reporting economic shocks (Table 13). In 2017, 93% of all HHs overall reported rising food prices as a shock, compared with 70% in 2021, and the proportion of HHs reported shocks related to fluctuating exchange rates and currency devaluation declined from 87% to 42%. Declines of over 10% were also observed for unavailability of agricultural inputs and family illness or death. Interestingly, there were no shocks that significantly increased in magnitude during this period, while rates of drought/flooding, agricultural pests and diseases, and illness or death remained above 40% overall. 28 An error in the ODK skip pattern resulted in excessive missing data. IMPEL | Implementer-Led Evaluation and Learning 30 Findings Figure 3. Prevalence of reported household shocks Reported Shocks in FSP-Enyanya Communities All three Resilience Capacities (absorptive, adaptive, and transformative) increased significantly between baseline and interim. Changes in the indices were largely driven by a single component indicator and/or shifts in the symmetry of the score distribution such that the outlying scores in the tails had more of an impact on the change (rather than an overall shift in the population). Note that the interim evaluation did not collect poverty data, 29 which was used as one of the components in the calculation of the resilience indices. The baseline indicators were re-calculated to match the calculations used in the interim survey to allow for improved comparability between rounds. Table 13. Baseline and interim shock exposure and resilience indicator comparisons Indicator Indicator value Raw difference (Int. - BL) Significance level* Number of observations 2017 BL 2021 Int. BL Int. 40. Shock exposure index 5.7 4.6 -1.1 *** 1,250 1,263 41. Cumulative impact of shock exposure index (severity weighted shock exposure) 33.4 25.7 -7.7 *** 1,250 1,263 42. Absorptive capacity index 28.1 48.8 20.7 *** 1,250 1,263 43. Adaptive capacity index 31.5 36.1 4.6 *** 1,250 1,263 44. Transformative capacity index 17.6 26.8 9.2 *** 1,250 1,263 Note: *** p < 0.001 29 The poverty module was deemed by BHA to be too long and labor-intensive relative to its value Interim Evaluation of the South Kivu FSP-Enyanya RFSA in the DRC (Vol. I) Findings 31 The absorptive capacity index increased significantly between rounds (from 28 to 49). This change was driven largely by the reported increased presence of humanitarian assistance30 (4% at baseline to 31% of HHs at interim), as well as significant changes in informal safety nets and cash savings. The increase in reported humanitarian assistance may be due to any source of additional humanitarian and development assistance, which may include certain types of interventions provided through FSP that were perceived as “humanitarian” by respondents, although it is important to note that this cannot be confirmed by available data. The improvements were offset to some degree by a decline in access to remittances as well as decreases in bonding social capital and preparedness/mitigation. The distribution of the absorptive asset index showed greater symmetry in the interim survey than the baseline, which indicates that the improvement in the mean scores between survey rounds does not necessarily imply an overall population improvement. Additional figures showing the distributions of the index are found in Annex C. Table 14. Absorptive capacity index: Changes in component indicators Indicator Direction of change (BL to Interim) Significance, scale 1. Informal Safety Nets Increase Significant, Large 2. Bonding Social Capital Decrease Significant, small 3. Cash Savings Increase Significant, large 4. Productive Assets Increase Significant, moderate 5. Livestock Assets Decrease Non-significant. 6. Preparedness/Mitigation Decrease Significant, small 7. Agricultural Insurance Increase Significant, small 8. Humanitarian Assistance Increase Significant, large 9. Access to Remittances Decrease Significant, moderate The adaptive capacity index saw a small but significant increase, from 32 to 36. As seen in Table 15, This change was driven primarily by the significant increase in social safety nets, and to a lesser degree, in productive assets. The improvements were offset by significant declines in education/training, livelihood diversity (mainly from a reduction in HHs reporting remittances/gifts, which the the resilience tool considers a livelihood activity), and improved agricultural practices. The increase in the index likely reflects a reduction in skewness due to a shift toward the middle, with fewer households at the upper end of the aggregate scale and little improvement for households at the lower end. Additional figures showing the distributions of the index are found in Annex C. 30 This variable is defined as the respondent indicating that government or non-governmental organization (NGO) emergency food or cash assistance is available in the respondent’s village OR the HH reported receiving emergency food or cash assistance from the government or NGO during the 12 months prior to the survey. IMPEL | Implementer-Led Evaluation and Learning 32 Findings Table 15. Adaptive capacity index: Changes in component indicators Indicator Direction of change (BL to Interim) Significance, scale 1. Bridging Social Capital Decrease Non-significant 2. Social Safety Nets Increase Significant, large 3. Education/Training Decrease Significant, moderate 4. Livelihood Diversity Decrease Significant, large 5. Productive Assets Increase Significant, small 6. Livestock Assets Decrease Significant, moderate 7. Improved Agric. Practices Decrease Significant, moderate The transformative capacity index also increased significantly, from 18 to 27. As seen in Table 16, this change was driven by a significant improvement in access to natural resource31s and formal safety nets. 32 A significant decrease was observed for collective action, with no changes observed for the remaining indicators included in the calculations. While the low value of this index still suggests a high degree of skewness in the distribution, this is a notable improvement that may reflect a meaningful change in the allocation of resources and services within communities. Additional figures showing the distributions of the index are found in Annex C. Table 16. Transformative capacity index: Changes in component indicators Indicator Direction of change (BL to Interim) Significance, scale 1. Access to Natural Resources Increase Significant, small 2. Bridging Social Capital Decrease Non-significant 3. Collective Action Decrease Significant, small 4. Local Gov’t Responsiveness Increase Non-significant 5. Participation In Local Decisions Decrease Non-significant 6. Formal Safety Nets Increase Significant, large Looking at the relationship between coping strategies and resilience indicators, no associations were observed. However, three of the adaptive capacity component indicators are related to agriculture, so households that did not engage in agriculture tend to have lower scores on this index even if they are highly resilient. This indicates that the adaptive capacity index may be of limited use for households not engaged in agriculture. It may benefit from adaptation to reflect adaptability as a function of livelihood. 31 Community-level variable, calculated as the sum of the number of communal natural resources that are available in a village. These resources include: communal grazing land, communal water source for livestock, communal source of firewood, communal source of irrigation water. 32 Community-level variable indicating the number of formal safety nets a HH reports to be present in their village. These safety nets include: Food assistance, non-food items, assistance due to losses in livestock, NGO disaster response program, NGO help when faced with a shock, if the HH received assistance from the gov’t or NGO. Interim Evaluation of the South Kivu FSP-Enyanya RFSA in the DRC (Vol. I) Findings 33 3.9 Food Security Indicators related to food security are summarized in Table 14. No significant changes were observed between 2017 and 2021 in the mean HDDS or the prevalence of moderate/severe food insecurity (FIES). However, unweighted analysis of the resampled areas showed a small but significant increase in the HDDS (3.8 at baseline to 4.2 at interim). The FSP monitoring data also shows little change in the HDDS and the FIES in both the APBS and the SFBS. All of the monitoring survey rounds show a mean HDDS of 3.8 or 3.9, and a prevalence of moderate/severe food insecurity (FIES) above 90% among the direct participants interviewed. Table 17. Baseline and interim food security indicator comparisons Indicator Indicator value Raw difference (Int.–BL) Significance level* Number of observations 2017 BL 2021 Int. BL Int 1. Average Household Dietary Diversity Score (HDDS) 3.6 3.9 0.3 ns 1,180 1,179 2. Prevalence of moderate or severe food insecurity based on 30-day recall (FIES) 95.0% 95.9% 0.9% ns 1,244 1,331 Male and female adults 95.0% 95.6% 0.6% ns 1,041 1,011 Adult female, no adult male 95.9% 98.3% 2.4% † 172 286 Adult male, no adult female 88.6% 84.2% -4.4% ns 30 34 Child, no adults NA na na na 1 0 Note: ns = not significant, na = not available, † p < 0.1 The prevalence of moderate and severe food insecurity was very high at baseline (95%) and at the interim (96%). However, the FIES questions may not be adequately describing the food security situation in this context. Most households answered yes to all (or nearly all) of the eight FIES questions, and there was no observation of the expected delineation between frequency of responses to the less severe questions vs. the most severe questions. Similar patterns are found in the baseline data. Additionally, there is a lower-than-expected correlation between the FIES data and the similar food-related coping strategy questions found in the resilience module. The two indicators use different recall periods (30 days for the FIES, 7 days for the food security coping), which may indicate that things like missing meals may not happen every week, they may be more likely to occur withing the recall period of a month. Additional figures showing the relationship between the individual FIES and food security coping questions are found in Annex C. No associations between the FIES and resilience indices were observed in the Round 2 data, likely due to low variation in the FIES variable. Looking at the full sample from the interim survey, there is some association between the HDDS and the resilience indices, particularly the absorptive and adaptive indices. The transformative capacity index is largely defined by community-level indicators rather than household-level indicators, and so the associations between that index and the HDDS are less likely to be observed. The graphs below show the relationship between the HDDS and the absorptive and adaptive indices. It should be noted that the sample size at the very high end of the HDDS is small, those means and ranges at the high end of the scale should be disregarded. IMPEL | Implementer-Led Evaluation and Learning 34 Findings Figure 4. Absorptive, adaptive, and transformative capacity index by Household Dietary Diversity Score Absorptive capacity index Adaptive capacity index Transformative capacity index Interim Evaluation of the South Kivu FSP-Enyanya RFSA in the DRC (Vol. I) Conclusions 35 4. CONCLUSIONS Overview • Overall, the interim evaluation, as well as the 2019 MTE, show the FSP-Enyanya RFSA in the DRC in a positive light. • The interim evaluation shows promising results. Several indicators appear to be moving in the correct direction, and the analysis tends to show that direct participation in interventions is associated with improvements in many lower-level indicators. • However, results at the population level do not show consistent, significant improvement. o Coverage rates of direct participants of the interventions are often low (a small portion of the total population), and spillover of the impacts from the RFSA interventions to indirect participants may be less than hoped. As such, impacts may be diluted at the population level. o More programmatic intensity and focus may be required to positively impact food security and resilience at the population level. • The important questions for BHA and Mercy Corps are: o Considering the cost of implementation of this activity/these interventions relative to the “saturation” that the activity might be expected to reach, is it worth running an intervention that is “a mile wide and an inch deep”? o How should/can a package of RFSA interventions best be streamlined (and/or consolidated) to ensure only sustainable, efficient, and impactful interventions are used? This study only begins to scratch at the surface of these questions. Intervention Exposure and Participation • Overall, 54% of HHs in the Round 2 survey reported participation in one or more of the surveyed RFSA interventions. o The 2020 annual report data indicate the RFSA was reaching 52,000 unique direct participant HHs, which is roughly 70% of HHs in the coverage area. • The interim survey shows that FSP-Enyanya’s RFSA achieved some moderate levels of self-reported household participation rates of some of their interventions, though none as high as indicated by the RFSA program data. o These include WASH trainings/events (24%), and agricultural training (22%), agricultural inputs (21%), with all others surveyed falling below 20%. o Most of the interventions took place in all (or most) of the villages sampled in the survey. • Other interventions had lower self-reported household participation rates. These include: o Safe space groups, which had only 4% of HHs reporting participation despite 81% of HHs in the interim survey living in villages where safe space groups were implemented. o Youth leadership training (7% of HHs, according to interim survey data, 93% of HHs living in villages where these were implemented). o Several other activities had under 10% of HHs reporting participation despite high coverage at the village level. • Low coverage of several interventions means that goals for population-level impacts are unlikely to be achieved. IMPEL | Implementer-Led Evaluation and Learning 36 Conclusions • Analysis of differences between the participant and non-participant households suggests that participants in the interventions may tend to be economically better off and more engaged in their communities than those who did not report participation. Although antecedent variables are few, it may suggest that this is a result of targeting bias, or self-selection bias, depending on the targeting approach. Water, Sanitation, and Hygiene • Improvements in access to safe drinking water were observed across the surveyed areas, particularly in the time required to access these sources. The percentage of households that could obtain drinking water in less than 30 minutes (round trip) increased significantly (from 39% to 75% of HHs). o Data was not collected in the interim evaluation on water point rehabilitation. However, the mid-term evaluation indicated that the FSP Enyanya RFSA functional water points had increased water access in some areas, with a recommendation to work to increase coverage. As each water point constructed may benefit an entire community, even modest increases in coverage of related interventions both before and after the MTE may have resulted in important improvements in these indicators. • The use of basic (improved) sanitation facilities remained low (3%), with no significant change. o FSP participant monitoring data showed a higher prevalence of improved sanitation in its surveys, but also showed little change over time (2019 to 2021). o Contrary to these findings, the RFSA reports the construction of more than 20,000 pit latrines, covering well over 3% of HHs in the coverage area. The interim survey data indicates that 18% of HHs report participating in toilet-building activities. o The predicted improvement in basic (improved) sanitation facilities may be clouded by the survey definition of basic (improved) sanitation facilities. Pit latrines without slabs (considered unimproved by the survey methodology) were the common type of toilet reported in the interim survey (85% of HHs).33 o Open defecation remained very low (3%) at the interim. Agriculture • There was a significant increase in farmers’ use of financial services between the survey rounds (32% to 50%), as well as the use of improved storage practices (40% to 53%). The use of value chain activities promoted by the project showed a small improvement, which was larger and statistically significant in the analysis of the resampled villages only. The FSP recurrent monitoring data show similar directions of change in these indicators. o The 2019 mid-term evaluation reported that the PO interventions were either weak, behind schedule, or non-existent. However, the 2020 annual report indicates that progress had been made following the MTE in implementing these activities. o Participation in VSLAs, as well as participation in farmers’ groups and/or NRM programs, were strongly associated with farmers’ access to financial services. 33 It’s difficult to assess the change in toilet type compared to baseline. The baseline indicates that 82% of HHs had a “water flush to pit latrine” type toilet, which is likely a mistake in data collection (although still classified as unsafe/unimproved sanitation). The interim survey reports that < 1% of HHs have flush-to-pit-latrine toilets- similar to the most recent DHS survey, which indicates that 0.5% of urban and 0% of rural have flush-to-pit-latrine toilets. Interim Evaluation of the South Kivu FSP-Enyanya RFSA in the DRC (Vol. I) Conclusions 37 • The interim evaluations found no association between participation in farmers’ groups or NRM programs and improved value chain activities or storage practices. • The use of four or more sustainable crop practices/technologies (out of 9 promoted by the activity) by farmers decreased significantly in the interim evaluation survey. o There was no decrease in the prevalence of farmers using at least one sustainable crop practice (88.8% in baseline, 88.5% in interim). However, at baseline, farmers were recorded as practicing a greater number of sustainable crop practices on average at baseline than at interim. o FSP monitoring data showed higher prevalences of farmers using four or more sustainable crop practices in their 2019, 2020, and 2021 surveys. However, the percentage point differences between years (+/- 15 percentage points) were similar to that seen in between the PBS survey rounds. o It’s challenging to assess whether these differences between rounds are methodology/collection issues, or if they accurately reflect changes. • Among livestock practices, the RFSA focused primarily on the breeding and keeping of rabbits. However, this only increased from 2.5% of HHs to 3.1% of HHs. o The mid-term evaluation found that comprehension by rabbit recipients of how to raise, manage, and breed the animals was limited and varied. The 2020 annual report data indicated improvements, but the intervention remained small, and had low coverage, so this very small change is not unexpected. Women’s Health and Nutrition • Changes in women’s consumption patterns showed slightly negative changes. The prevalence of women consuming a diet of minimum diversity fell (weakly significant), and the prevalence of women consuming targeted nutrient-rich commodities also dropped, though this change was only significant when looking at the re-sampled villages only. o FSP participant monitoring data also showed no large variation in women’s consumption of a diet of minimum diversity or in the consumption of nutrient-rich commodities. o However, participation in nutrition training was significantly associated with higher levels of women consuming a diet of at least the minimum dietary diversity. • Contraceptive use (both modern and traditional) remained unchanged between survey rounds. FSP participant monitoring data also showed little change between 2019 and 2021. o However, the interim survey showed that women participating in mothers’ groups were significantly associated with a higher prevalence of contraception than those who did not. Children’s Health and Nutrition • Diarrhea prevalence in children under 5 had a significant decrease (34% to 20%) between survey rounds. A similar prevalence and change were observed in FSP participant monitoring surveys. • Treatment of under-5 diarrhea with ORT showed little change. o However, the decreased diarrhea prevalence means that the number of observations at interim was small. o Activity monitoring 2021 data from Mercy Corps indicate that among participant households, 71% of children with diarrhea are treated with ORT. Low coverage among IMPEL | Implementer-Led Evaluation and Learning 38 Conclusions the interim survey households does not allow for a similar level of analysis, and changes measured at the overall population level may be too diluted to show any overall change. • Exclusive breastfeeding of children under 6 months old did not change significantly between rounds, though the sample was small. o FSP monitoring data showed a decline in exclusive breastfeeding between the 2019 and 2020 survey estimates of over 80%, and the 2021 survey estimate of 57% of children among direct participants (similar to the population findings in Round 2). • The prevalence of children 6–23 months consuming targeted nutrient-rich foods improved significantly (particularly among girls). However, the consumption of a minimum acceptable diet of children 6–23 months remained largely unchanged. o The improvement in consumption of nutrient-rich foods was driven primarily by increased consumption of bio-fortified foods (a value-chain commodity), as well as orange-fleshed sweet potatoes. • Although mothers’ groups, nutrition trainings, and home health visits were implemented in all communities, participation rates were low (16% of children under 5 lived in households reporting participation in mothers’ groups, 17% in nutrition training/meetings, and 20% in home health visits). o There are no associations between participation in these interventions and improvements in child MAD, breastfeeding practices, or under-5 diarrhea. Gender • Little change was noted across the gender indicators, except for a marginally significant improvement in the knowledge of MCHN practices. o Gender norms are deeply entrenched and will likely take years or even decades to change. • There was little change in the percentage of adults earning cash in the past year, though the prevalence of men earning cash rose significantly in the resampled areas. FSP monitoring data of participants also show little change in the percentage of men and women earning cash among direct participants. • Women had significantly better knowledge of maternal and child health and nutrition practices at interim, though men showed no significant change. FSP monitoring data of direct participants showed similar prevalences, though little change between the 2019, 2020, and 2021 surveys. o No association between participation in mothers’ groups, nutrition trainings, and home health visits and MCHN knowledge was observed. • Among the variables related to decision-making, there was little to no significant change between survey rounds. FSP monitoring data showed similar evidence of minimal change over time, highlighting the fact that real changes to gender attitudes and norms may take decades to change. • Women’s participation in decision-making bodies showed no significant change. Interim Evaluation of the South Kivu FSP-Enyanya RFSA in the DRC (Vol. I) Conclusions 39 Food Security, Shock Exposure, and Resilience • All three resilience capacity indices (absorptive, adaptive, and transformative) increased significantly between baseline and interim. However, changes in the indices were largely driven by only one (or a few) of their component indicators: o The large increase in the absorptive index was driven primarily by the increased presence of humanitarian assistance (one of the index components). This may be due to assistance provided through FSP that was perceived as ‘humanitarian’ by the respondents, or provided by other actors (though this cannot be confirmed by available data). Cash savings and informal safety nets also contributed to the increase in the absorptive index. o The small increase in the adaptive capacity index was driven mainly by an increase in social safety nets, and to a lesser degree, increases in productive asset ownership. The improvements were offset by significant declines in education/training, livelihood diversity ((mainly from a reduction in households reporting remittances/gifts), and improved agricultural practices. o The modest increase in the transformative capacity index was driven mainly by improvements reported in formal safety nets, and was offset by small decreases in some of the other component indicators. • Food security, as measured by the Household Dietary Diversity Score (HDDS) and the Food Insecurity Experience Scale (FIES), showed no significant change between survey rounds. o The HDDS did show some small improvements in the unweighted analysis of the re￾sampled areas. o The FSP monitoring data also shows little change in the HDDS and the FIES in both the APBS and the SFBS. o The prevalence of moderate and severe food insecurity (as measured by the FIES) was very high at baseline (95%) and at the interim (96%). This homogeneity of the food security status as measured by this indicator renders a more detailed analysis less useful in assessing change. The FIES has other statistical limitations in the populations surveyed. It may not adequately describe the food security situation in the populations surveyed. o Considering the volatile food security in the DRC, compounded in the year prior to the survey by the COVID-19 pandemic, stable food security measures could be interpreted as a positive outcome. • There is a positive correlation between the HDDS and higher absorptive and adaptive capacity indices. o This does not hold true for the transformative capacity index, which is defined mainly by community-level indicators rather than household-level indicators. Methodology • The pre-/post-PBS design has some important limitations. o Modifications in areas of implementation after the baseline PBS is common across RFSAs. The pre-/post-PBS methodology may not be well suited to adapt to changing areas of implementation. IMPEL | Implementer-Led Evaluation and Learning 40 Conclusions o Sampling frame data in the DRC often has large inaccuracies. This results in highly variable probability/population weights, and a loss of statistical power. Alternative PBS sampling strategies should be considered that would have less loss of statistical power with similar sample sizes and budgets. o Population-level changes in many of the low-level indicators have a low likelihood of occurring with the given intensity of certain interventions. o There is a desire to have evaluation data that can show the impact of interventions on the various outcomes. A PBS does not readily allow this level of analysis. • Some key outcome indicators may not adequately perform in the context. o The utility of certain food security indicators (such as the FIES) should be re-evaluated, including an assessment of their functionality in specific locations/contexts. o The resilience capacity indices are less useful as composite indicators. The sub￾components of the indices tend to reveal more useful information. For example, three of the adaptive capacity component indicators are related to agriculture, so households that did not engage in agriculture tend to have lower scores on this index even if they are highly resilient. This indicates that the adaptive capacity index in the aggregate may be of limited use for households not engaged in agriculture. It may benefit from adaptation to reflect adaptability as a function of livelihood.