Interim Evaluation of the Budikadidi Resilience Food Security Activity in the Democratic Republic of the Congo May 2022 | Volume I IMPEL | Implementer-Led Evaluation & Learning Associate Award ABOUT IMPEL The Implementer-Led Evaluation & Learning Associate Award (IMPEL) works to improve the design and implementation of Bureau for Humanitarian Assistance (BHA)-funded resilience food security activities (RFSAs) through implementer-led evaluations and knowledge sharing. Funded by the USAID Bureau for Humanitarian Assistance (BHA), IMPEL will gather information and knowledge in order to measure performance of RFSAs, strengthen accountability, and improve guidance and policy. This information will help the food security community of practice and USAID to design projects and modify existing projects in ways that bolster performance, efficiency, and effectiveness. IMPEL is a seven-year activity (2019–2026) implemented by Save the Children (lead), TANGO International, Tulane University, Causal Design, Innovations for Poverty Action, and International Food Policy Research Institute. RECOMMENDED CITATION IMPEL. (2022). Interim Evaluation of the Budikadidi Resilience Food Security Activity in the Democratic Republic of the Congo (Vol. I). Washington, DC: The Implementer-Led Evaluation & Learning Associate Award. PHOTO CREDITS Hugh Kinsella Cunningham / Save the Children DISCLAIMER This report is made possible by the generous support of the American people through the United States Agency for International Development (USAID). The contents are the responsibility of the Implementer-Led Evaluation & Learning (IMPEL) award and do not necessarily reflect the views of USAID or the United States Government. CONTACT INFORMATION IDEAL Activity c/o Save the Children 899 North Capitol Street NE, Suite #900 Washington, DC 20002 www.fsnnetwork.org IMPEL@savechildren.org PREPARED BY: Interim Evaluation of the Budikadidi RFSA in the DRC (Vol. I) Acknowledgments i ACKNOWLEDGMENTS USAID/BHA: Staff in Washington, DC and at the DRC mission Kinshasa School of Public Health: Pierre Akilimali, Prescillia Visi, John Lhudal, Charles Kasongo, Dynah Kayembe, Steve Mbikayi, Richard Mubikayi, Guy Ngindu, Marc Mchanga Useni, Maurice Kabangu, and Michel Kayembe IMPEL: Tom Spangler, Mohamoud Ahmed, and Corena Sharp Tulane University: Nancy Mock, Peter Horjus, Michelle Lacey, Sarah Truxillo, and Janna Wisniewski Trestle Research and Consulting: Maulik Chauhan and Vikas Arora Independent survey monitors: Elvis Cidoro (supervisor), Innocent Nshombo (South Kivu), Samuel Kataya (Tanganyika), and Roussel Kabeya (Kasaï Oriental) IMPEL | Implementer-Led Evaluation and Learning ii Table of Contents TABLE OF CONTENTS Acknowledgments............................................................................................... i List of Tables......................................................................................................iii List of Figures.....................................................................................................iv Acronyms............................................................................................................ v Executive Summary ...........................................................................................vi 1. Introduction................................................................................................... 1 1.1 Background and Setting ................................................................................................................. 1 1.2 Resilience Food Security Activity Goals and Activities................................................................... 1 1.3 Interim Evaluation Purpose and Objectives................................................................................... 3 1.4 Study Team and Partners............................................................................................................... 4 2. Quantitative Evaluation Methods.................................................................. 5 2.1 Overview ........................................................................................................................................ 5 2.2 Sample Design................................................................................................................................ 5 2.3 Sample Size..................................................................................................................................... 6 2.4 Sample Selection............................................................................................................................ 7 2.5 Survey Tools................................................................................................................................... 7 2.6 Fieldwork and Data Collection ....................................................................................................... 8 2.7 Data Analysis.................................................................................................................................. 9 2.8 Final Sample and Sampling Weights ............................................................................................10 2.9 Integration of Secondary Data .....................................................................................................11 2.10 Limitations and Delimitations ......................................................................................................11 3. Findings .......................................................................................................13 3.1 Intervention Exposure and Participation .....................................................................................13 3.2 Water, Sanitation, and Hygiene ...................................................................................................15 3.3 Agriculture....................................................................................................................................16 3.4 Women’s Health and Nutrition....................................................................................................20 3.5 Children’s Health and Nutrition ...................................................................................................21 3.6 Gender..........................................................................................................................................22 3.7 Activity-Specific ............................................................................................................................24 3.8 Shock Exposure and Resilience Indices........................................................................................25 3.9 Food Security................................................................................................................................30 4. Conclusions..................................................................................................32 Interim Evaluation of the Budikadidi RFSA in the DRC (Vol. I) List of Tables and Figures iii VOLUME II Annex A: Protocol for the Quantitative Interim Evaluations of three Resilience Food Security Activities in the Democratic Republic of the Congo Annex B: Survey Questionnaire Annex C: Additional Tables and Figures Annex D: Descriptive Analysis Tables Annex E: Indicator Change Tables LIST OF TABLES Table 1. Budikadidi activity purposes ........................................................................................................... 2 Table 2. Baseline and interim evaluation population-based survey sampling frame................................... 6 Table 3. Population-based survey sampled villages/clusters and sample size............................................. 7 Table 4. Final sample size and response rates............................................................................................11 Table 5. Baseline (BL) and interim water, sanitation, and hygiene indicator comparisons (overlapping areas only, unweighted) .............................................................................................................................15 Table 6. Baseline and interim agriculture indicator comparisons (overlapping areas only, unweighted) . 17 Table 7. Changes in prevalence of sustainable crop practices between baseline (2017) and interim (2021) (overlapping areas only, unweighted) ........................................................................................................18 Table 8. Changes in prevalence of sustainable livestock practices between baseline (2017) and interim (2021) (overlapping areas only, unweighted).............................................................................................19 Table 9. Baseline and interim women’s health and nutrition indicator comparisons (overlapping areas only, unweighted).......................................................................................................................................20 Table 10. Baseline and interim children’s health and nutrition indicator comparisons (overlapping areas only, unweighted).......................................................................................................................................22 Table 11. Baseline and interim gender indicator comparisons (overlapping areas only, unweighted) .....23 Table 12. Baseline and interim activity-specific indicator comparisons (overlapping areas only, unweighted)................................................................................................................................................25 Table 13. Baseline and interim shock exposure and resilience indicator comparisons (overlapping areas only, unweighted).......................................................................................................................................27 Table 14. Absorptive Capacity Index: Changes in component indicators (overlapping areas only, unweighted)................................................................................................................................................27 Table 15. Adaptive Capacity Index: Changes in component indicators (overlapping areas only, unweighted)................................................................................................................................................28 Table 16. Transformative Capacity Index: Changes in component indicators (overlapping areas only, unweighted)................................................................................................................................................29 Table 17. Baseline and interim food security indicator comparisons (overlapping areas only, unweighted) ....................................................................................................................................................................30 IMPEL | Implementer-Led Evaluation and Learning iv List of Tables and Figures LIST OF FIGURES Figure 1. Intervention exposure and participation.....................................................................................14 Figure 2. Household-reported shocks.........................................................................................................26 Figure 3. Mean adaptive capacity index by number of coping activities ...................................................29 Figure 4. Absorptive and Adaptive Capacity Index by Household Dietary Diversity Score ........................31 Interim Evaluation of the Budikadidi RFSA in the DRC (Vol. I) Acronyms v ACRONYMS BHA Bureau for Humanitarian Assistance BL Baseline CDCS Country Development Cooperation Strategy CHN Child Health and Nutrition CRS Catholic Relief Services DRC Democratic Republic of the Congo FIES Food Insecurity Experience Scale HDDS Household Dietary Diversity Score HH Household HNW Health, Nutrition, and WASH IHP Integrated Health Program IMPEL Implementer-Led Evaluation and Learning IP Implementing Partner IPM Integrated Pest Management KSPH Kinshasa School of Public Health MAD Minimum Acceptable Diet MCHN Maternal and Child Health and Nutrition ME&A Mendez England and Associates MHN Maternal Health and Nutrition NCBA-CLUSA National Cooperative Business Association Civic Leadership USA International NRM Natural Resource Management ODK Open Data Kit ORT Oral Rehydration Therapy P Purpose PBS Population-Based Survey R&I Refine and Implement RACOJ Réseau des Associations Congolaises de Jeunes REFED Caritas, Réseaux Femmes et Développement RFSA Resilience Food Security Activity SD Standard Deviation SP Sub-Purpose TOC Theory of Change USAID United States Agency for International Development WASH Water, Sanitation, and Hygiene VSLA Village Savings and Loan Association IMPEL | Implementer-Led Evaluation and Learning EXECUTIVE SUMMARY Overview In 2016, the United States Agency for International Development's (USAID) Bureau for Humanitarian Assistance (BHA) issued an award to Catholic Relief Services (CRS) to implement a Resilience Food Security Activity (RFSA) in the Democratic Republic of the Congo (DRC). The activity, named Budikadidi, which means "self-sufficiency" in Tshiluba, is being carried out in the Kasaï Oriental province. The goal of Budikadidi is to ensure that nutrition and food security for households improve to achieve sustained nutrition, food security, and economic well-being outcomes. The start date of the RFSA was December 16, 2016. The violence that erupted in the greater Kasaï region in 2016 forced Budikadidi to alter RFSA implementation plans, which were initially designed to execute activities in both Kasaï Central and Kasaï Oriental. Ongoing security concerns in Kasaï Central influenced a decision to implement activities only in Kasaï Oriental. This change occurred after the activity had already set up offices in Kasaï Central, causing delays in the start of the Refine and Implement (R&I) period, recruitment of staff, and the development of working relationships with local and international partners. With this change, the activity added a third health zone in Kasaï Oriental, not initially included in the RFSA. In 2021, under the Implementer-Led Evaluation and Learning (IMPEL) Associate Award with Save the Children, Tulane University School of Public Health and Tropical Medicine (Tulane) was subcontracted to conduct an interim evaluation, including a population-based survey (PBS) and a resilience assessment, in the three selected RFSA areas. Study Purpose and Design As part of the overall R&I approach, the interim evaluation was originally planned to take place in 2020 so that the findings could help inform decisions on activity extensions. However, the evaluation was delayed until 2021 because of the COVID-19 pandemic. Prior to the interim evaluation, BHA made the decision to extend the Budikadidi RFSA for 2 years. 1 The extension took many factors into account, but the design of the interim evaluation to track lower-level outcome indicators remained the same. The PBS serves as the second phase of a pre-post survey cycle, with data on the same indicators collected in both survey rounds. This PBS collected representative data on a number of lower-level outcomes from 1,224 households in July/August 2021. Statistically detecting changes (if any) for all practice and behavioral change indicators at the population level (the Budikadidi RFSA coverage area) can help inform the performance of the award to date. It should be noted that the evaluation does not include data from areas in which the intervention was not implemented (i.e., a counterfactual). While the evaluation results can be used to help explain differences between the pre- and post-values and may therefore help paint a general picture about 1 The vision for the R&I approach is generally to issue up to 2-year performance-based extension to BHA resilience activities that demonstrate 'outstanding' performance. The interim evaluation is one component that usually contributes to that determination. vi Executive Summary Interim Evaluation of the Budikadidi RFSA in the DRC (Vol. I) Executive Summary vii overall performance, true attribution is only possible with experimental and quasi-experimental evaluation designs, which are not being used in this evaluation. Furthermore, comparisons between baseline and interim are difficult to make given the large changes in the Budikadidi RFSA coverage areas between the baseline and interim surveys. About half of the sample in the interim survey comes from the Kasansa health zone, which was not covered by the baseline survey. In order to ensure comparability between rounds, when making such comparisons in this report, the analysis uses the full baseline sample but restricts the interim analysis to look only at households in the areas covered by the baseline sample (so, excluding Kasansa health zone interim survey data). This is referred to as the ‘overlapping’ areas for the purposes of this report. Furthermore, when comparing the results from these overlapping areas, the analysis is unweighted. (See sampling section for more details.) Key Findings Overall Assessment Overall, the interim evaluation, as well as the 2019 mid-term evaluation, show CRS/Budikadidi’s RFSA activity in the DRC in a positive light. The data presented in this report show promising results. Many indicators appear to be moving in the correct direction, and the analysis tends to show that participation in interventions to be associated with improvements in many lower-level indicators. However, more programmatic intensity (see below) and focus may be required to positively impact food security and resilience at the population level. The important questions for BHA and CRS are: 1. Considering the cost of implementation of this activity with these interventions relative to the “saturation” that the activity might be expected to reach, is it worth running an intervention that is “a mile wide and an inch deep”? 2. How should/can a package of RFSA interventions best be streamlined (and/or consolidated) to ensure only sustainable, efficient, and impactful interventions are used? This study only begins to scratch at the surface of this question. Intervention Exposure and Participation The 2020 Annual Report data indicate the RFSA was reaching 47,000 unique direct participant households, which is roughly 34% of households in the coverage area. However, due to registration challenges from the COVID-19 pandemic, this is likely an underestimate. Looking at the interim survey data, 52% of households in the Round 2 survey reported participation in one or more of the surveyed RFSA interventions. The interim survey shows that the Budikadidi RFSA achieved some (relatively) higher self-reported household (HH) participation rates of some of their interventions. These include water, sanitation, and hygiene (WASH) trainings/events (37%), toilet building (32%), farmers groups (35%), Village Savings and Loan Associations (VSLAs) (31%), and others. These interventions took place across all (or nearly all) the villages included in the interim survey. Other interventions had lower self-reported HH participation rates, including youth leadership training (5%), adolescent live skills training (6%), alternative livelihood interventions (8%), and others despite also taking place in all/nearly all the villages surveyed. IMPEL | Implementer-Led Evaluation and Learning viii Executive Summary Shocks, Resilience, and Food Security Food security, as measured by the Household Dietary Diversity Score (HDDS), showed only a small improvement between survey rounds in the overlapping areas, from an average of 3.6 food groups at baseline, to 3.9 at interim. Considering the volatile food security in the DRC, compounded in the year prior to the survey by the COVID-19 pandemic, stable HH dietary diversity could be interpreted as a positive outcome. Moderate and severe food insecurity, as measured by the Food Insecurity Experience Scale (FIES), increased significantly between baseline and interim in the overlapping areas (87% vs. 97%). However, the FIES data at baseline and interim appear to have limitations in adequately describing food security in the populations surveyed. All three resilience capacity indices (absorptive, adaptive, and transformative) increased significantly in the overlap areas between baseline and interim. However, changes in the indices were largely driven by only one (or a few) of their component indicators. The large increase of the absorptive index was driven by an increase in reported prevalence of humanitarian assistance. Rapid response activities conducted by other actors in some of the areas of implementation may have led to this change. The small increase in the adaptive capacity index was driven mainly by an increase in social safety nets, which was offset by a reduction in livelihood diversity (mainly from a reduction in HHs reporting remittances/gifts). The modest increase in the transformative capacity index was driven mainly but improvements reported in formal safety nets but was offset by small decreases in some of the other component indicators. There is a positive correlation between the HDDS and higher absorptive and adaptive capacity indices. This does not hold true for the transformative capacity index, which is defined mainly but community￾level indicators rather than household. Water, Sanitation, and Hygiene Drinking water quality and access showed neutral or positive changes in the overlapping areas. Households with improved drinking water sources increased significantly, from 47% to 57%. The percentage of HHs that can access drinking water in less than 30 minutes increased from 56% to 72%. Use of recommended water treatment technologies remained low, however. The 2019 mid-term evaluation highlighted that the demand for potable water in the intervention areas was large, to the point of straining the system. However, the RFSA continues to make strides to expand and reinforce the system to meet that demand, which promises further improvements during the extension period of the activity. Improvements in sanitation are best observed in the decrease of the percentage of HHs using open defecation, which decreased in the overlapping areas from 35% to 18% of HHs. Households with an adequate handwashing station only saw very small improvements, however. Participation in WASH trainings/events and toilet building interventions are significantly associated with improvements in some WASH outcomes. WASH trainings/events took place in all but one of the surveyed villages, and 31% of HHs reported participation in these trainings. Toilet building interventions were also implemented in nearly all the villages sampled in the survey, with 32% of HHs at interim reporting participation. Participation in WASH training/events was associated with significantly higher Interim Evaluation of the Budikadidi RFSA in the DRC (Vol. I) Executive Summary ix handwashing and correct water treatment, and participation in toilet building was significantly associated with reduced open defecation. The addition of Kasansa to the interim sampling frame2 contributed to an apparent reduction in the relative proportion of Budikadidi RFSA HHs with access to water compared to the baseline. While 55% of HHs in Cilundu and 60% of HHs in Miabi indicated that they had access to a basic water source, this was true for only 38% of Kasansa HHs. Similarly, 49% of Kasansa HHs required over 30 minutes to obtain water, compared with 29% and 27% of HHs in Cilundu and Miabi, respectively. Activity targets may need to be revised considering that Kasansa likely started out further behind than the other health zones. Agriculture Changes in agriculture indicators in the overlapping areas were of mixed results. Most remained unchanged; some showed some degree of improvement, while others showed worsening trends. One important, large improvement is seen in farmers practicing value-chain activities (19% to 42%). Positive impacts of the activity on agricultural market linkages are also highlighted in the 2020 Annual Report. The aggregate indicators measuring the use of sustainable crop and livestock practices showed negative changes in the overlapping areas between rounds. However, there appears to be a bias in the baseline and/or interim data that may be masking positive change. Some indications of positive change are seen when looking deeper into the data. The percentage of HHs practicing at least one type of sustainable crop practice increased from 62% at baseline to 82% of farmers at interim. The percentage of farmers with livestock practicing at least one sustainable livestock practice increased from 39% at baseline to 45% at interim. There were significant increases in some of the crop practices promoted by the RFSA, including the use of manure, which increased from 11% of farmers at baseline to 33% at interim, and compost, which increased from 12% of farmers at baseline to 24% of farmers at interim. Other promoted crop practices remained relatively unchanged, except for weed control (which decreased from 31% to 15% of farmers). Looking at the individual livestock practices, the use remained relatively unchanged, with only some relatively small fluctuations for certain practices. Household participation in certain interventions was associated with better agriculture outcome indicators in the interim survey. The 32% of farmers living in HHs reporting VSLA participation had significantly higher rates of financial services use. Farmers participating in agricultural input interventions (15% of farmers) were more likely to use value chain activities (41% compared to 35% among non-participants). Women’s Health and Nutrition Indicators of women’s diets experienced some decrease in the overlapping areas between baseline and interim surveys. Consumption by women of a minimum acceptable diet (MAD) decreased from 20% to 16%, though this was not significant. Consumption of targeted nutrient-rich commodities did experience a large drop, however, from 66% at baseline to 30% at interim. The drop in nutrient-rich commodities 2 Only 27% of the sample (about 335 HHs) overlap with the baseline sample. As such, comparisons between baseline and interim are difficult to make. IMPEL | Implementer-Led Evaluation and Learning x Executive Summary was primarily driven by a reduction in the consumption of bio-fortified foods, which fell from 37% at baseline to only 10% at interim. This may be due to changes in other activities implemented in the area. Participation in nutrition trainings is associated with better dietary diversity. Among the 16% of women who participated in nutrition trainings, the prevalence of consumption of a minimum dietary diversity was 21%, compared to only 8% among women that did not participate. The use of contraception among women of reproductive age that were married or partnered increased significantly in the overlapping areas, from 9% at baseline to 16% at interim. This increase was largely driven by an increase in traditional contraceptive use (vs. modern). Self-reported participation in mother’s groups was associated with an increased use of contraception. However, the care group promoted by the RFSA did not organize any sessions to promote any form of contraception. This association may be related to participation mother’s groups organized by other implementers, or it may be that women who chose to participate in members of mother’s groups are also more likely to use contraception. Child Health and Nutrition There was a significant decrease in the percentage of children under 5 years who had diarrhea in the last 2 weeks in the overlapping areas (44% to 35%). Positive changes were also seen in the use of oral rehydration therapy (ORT) to treat diarrhea in children under 5 years, and exclusive breastfeeding of children under 6 months. However, the changes were not sufficient for the small sample sizes to find significant. The prevalence of children 6-23 months receiving a minimum acceptable diet, and the prevalence of children 6–23 months who consume targeted commodities did not experience a significant change. The lack of important changes in diet quality/diversity is also seen among women of reproductive age, and at the household level. Participation in Mothers’ groups, nutrition training, and home health visits were not associated with better outcomes in exclusive breastfeeding, use of ORT, children’s MAD, or consumption of nutrient-rich commodities. The 2018 mid-term evaluation highlighted that many of these activities demand highly effective Lead Mothers to succeed. It may be that the COVID-19 restrictions limited the ability of the Lead Mothers to adequately conduct their work. Gender Significant improvements from baseline to interim were observed in the overlapping areas in many of the gender indicators. However, certain decision-making indicators did not show obvious positive change, particularly among women. As emphasized in the 2018 mid-term evaluation, gender norms in Kasaï are deeply entrenched and will take a long time to change. A significant increase in the percentage of adult women earning cash in the previous year (27% to 35%). Among men, this remained unchanged, indicating that new cash-earning opportunities are being made available to women. Men in unions showed a shift from making decisions about their own self-earned cash alone, to making these decisions jointly with their wife/partner. Interim Evaluation of the Budikadidi RFSA in the DRC (Vol. I) Executive Summary xi There was a significant increase in knowledge of maternal and child health and nutrition (MCHN) practices (for both men and women). There was also a significant shift in men making maternal or child health decisions alone to making them jointly with their spouse/partner. The combined percentage of women reporting making decisions about their own self-earned cash alone plus those making them jointly decreased slightly between survey rounds. This indicates that women did not appear to gain decision-making power related to their cash earnings. Similarly, the combined percentage reporting making maternal health or child health decisions alone plus those making them jointly did not see large changes between survey rounds. This indicates that women did not appear to gain decision-making power of their own or their children’s health between rounds. The percentage of men in the overlapping areas saying that it is ok for a man to batter his wife for any reasons saw a moderate increase, from 62% to 71%. There was no change in the overlapping areas in the percentage of women reporting participation in community decision-making bodies. Methodology and Design This evaluation had some methodological challenges that should be taken into consideration in future evaluations, most importantly those related to the limitations in the pre/post PBS design, and the utility and function of certain outcome indicators. Modifications in areas of implementation after the baseline PBS is common across RFSAs. However, pre/post PBS methodology may not be well suited to adapt to changing areas of implementation. Population-level changes in many of the low-level indicators should not realistically be expected to change with the given intensity of certain interventions. There is a desire to have evaluation data that can show impact of interventions on the various outcomes. PBS do not readily allow this level of analysis. Also related to limitations with the PBS design, it should be highlighted that sampling frame data in DRC often has large inaccuracies. This results in highly variable probability/population weights, and a loss of statistical power. Alternative PBS sampling strategies should be considered that would have less loss of statistical power with similar sample sizes and budgets. Some key outcome indicators may not adequately perform in the context. The utility of certain food security indicators (such as the FIES) should be re-evaluated, including an assessment of their functionality in specific locations/contexts. Additionally, the resilience capacity indices tend be less useful when presented as composite indicators. The sub-components of the indices tend to reveal more useful information. For example, three of the adaptive capacity component indicators are related to agriculture, so households that did not engage in agriculture tend to have lower scores on this index even if they are highly resilient. This also indicates that the adaptive capacity index in the aggregate may be of limited use for households not engaged in agriculture. It may benefit from adaptation to reflect adaptability as a function of livelihood. This page is left intentionally blank. Interim Evaluation of the Budikadidi RFSA in the DRC (Vol. I) Introduction 1 1. INTRODUCTION 1.1 Background and Setting Kasaï Oriental is the smallest and most densely populated province in the Democratic Republic of the Congo (DRC). Poor infrastructure makes some areas in Kasaï Oriental inaccessible during the rainy season. Agricultural and mining constitute the primary economic activities. Poverty is widespread, particularly in rural areas, where agricultural production is inadequate, artisanal mining renders little profit, and unemployment is high. Strong patriarchal systems favor male dominance, which impacts gender relations at all levels of the population causing female discrimination and gender inequality. Recent militia activities have caused unrest and internal displacement within the region, contributing to an already fragile situation and creating additional strain on weak government services and systems. While conflict prompted a humanitarian response, Kasaï Oriental has received relatively little development assistance as compared with other areas in the DRC. 1.2 Resilience Food Security Activity Goals and Activities In 2016, the United States Agency for International Development's (USAID) Bureau for Humanitarian Assistance (BHA) issued an award to the Catholic Relief Services (CRS) to implement a Resilience Food Security Activity (RFSA) in the DRC. 3 The activity, named Budikadidi, which means "self-sufficiency" in Tshiluba, is being carried out in the Kasaï Oriental province. Budikadidi is implemented by a consortium of partners. The partners have expertise in governance, agriculture, youth, gender, and nutrition. Included are the National Cooperative Business Association CLUSA International (NCBA-CLUSA), Caritas, Réseaux Femmes et Développement (REFED), and Réseau des Associations Congolaises de Jeunes (RACOJ). The start date of the RFSA was December 16, 2016. The RFSA uses a multi-sectoral approach to deliver a package of interventions aimed to build local capacity, strengthen service-delivery systems, and increase accountability, as well as reduce structural, cultural, and gender-based barriers to change. Activities work with vulnerable populations to provide technical assistance in agricultural production and livelihoods; management of natural resources; nutrition and health; water, sanitation, and hygiene (WASH); and resilience. Also, there are critical cross￾cutting interventions related to themes such as governance, gender, and youth. Budikadidi is one of the RFSAs piloting a refine and implement (R&I) approach, which involves a preliminary period of formative and applied research and pilot interventions aimed at improving activity design, then followed by full implementation. Activities are designed to assist Congolese to identify opportunities to learn, mitigate, adapt, invest, and thrive. Budikadidi results are intended to contribute to USAID/DRC's Country Development Cooperation Strategy (CDCS), which supports the DRC's long-term transition to more effective and empowering development. The goal of Budikadidi is to ensure that nutrition and food security for households improves to achieve sustained nutrition, food security, and economic well-being outcomes. Participants include approximately 426,420 community members living in 85,300 households located in 474 villages in three 3 The activity was previously awarded as a Food for Peace Development Food Security Activity. The terminology for these awards changed in 2020. IMPEL | Implementer-Led Evaluation and Learning 2 Introduction rural health zones (Miabi, Cilundu, and Kasansa). The activity theory of change (TOC) includes the following purposes (P) and sub-purposes (SP): Table 1. Budikadidi activity purposes Foundational Purpose (FP): Communities empowered to sustain improvements to food security and nutrition. SP F.1: Environment favors multi-sectoral development toward food & nutrition security for all community members SP F.2: Systemic gender barriers to Food and Nutrition Security reduced community members SP F.3: Communities are resilient to common shocks Purpose (P) 1: Chronic malnutrition in children under 5 years sustainably reduced. SP 1.1: Early pregnancies & forced marriages reduced SP 1.2: Households practice optimal Health, Nutrition, and WASH (HNW) behaviors during the 1000-day period SP 1.3: All HH members make use of high quality, accessible health services SP 1.4: Communities & HHs maintain a clean environment conducive to good health & nutrition Purpose (P) 2: Household inclusive social and economic well-being improved. SP 2.1: Men and women share household responsibilities and decision-making equitably. SP 2.2: HH income increased. SP 2.3: HHs have access to diverse, appropriate foods for all members at all times Setting up local governance structures designed to ensure multi-sectoral development toward food and nutrition security for all community members is pivotal to the foundation and long-term sustainability of the approach. The aim is to have participants involved in and benefit from a range of complementary activities that, over time, become independent of activity inputs and self-reliant. The violence that erupted in the greater Kasaï region in 2016 forced Budikadidi to alter RFSA implementation plans, which were initially designed to execute activities in both Kasaï Central and Kasaï Oriental. Ongoing security concerns in Kasaï Central influenced a decision to implement activities only in Kasaï Oriental. This change occurred after the activity had already set up offices in Kasaï Central, causing delays in the start of R&I, recruitment of staff, and the development of working relationships with local and international partners. With this change, the activity added a third health zone in Kasaï Oriental, not initially included in the RFSA. Since the official start of Budikadidi on October 1, 2017, a range of factors both within and outside the control of the implementing partner (IP) has continued to contribute to a slow start. These factors include delays in meeting USAID environmental compliance requirements, procurement challenges, and technical staffing challenges and changes. Contextual factors including the focus on mining for economic revenue and the social consequences associated with mining, inadequate agricultural production, and limited crop diversity, limited economic opportunities, extreme gender inequality, poor communication and road infrastructures, and minimum previous exposure to development projects and approaches, present significant obstacles in implementing effective activities. Also, the assistance provided by humanitarian agencies involving the distribution of food and cash in RFSA health zones, which generally last about 3 months, detract from RFSA activities as community members focus on benefiting from the aid. Monetary and in-kind assistance appear to alter expectations and undermine the development process. Another challenge is that residents living in villages affected by the violence in 2016 are Interim Evaluation of the Budikadidi RFSA in the DRC (Vol. I) Introduction 3 reestablishing living structures and trying to meet basic needs and therefore have different development requirements. The inability of existing government structures to consistently provide mandated services, such as basic curative and preventive health care, has impacted negatively on the efficiency of implementation of Budikadidi activities. The slow starts of key USAID funded collaborators, including USAID Integrated Health Program (IHP) and Integrated Government Activity (IGA), have affected health, nutrition, and governance activities specific sub purposes, such as SP 1.1 (Early pregnancies & forced marriages reduced) and 1.3 (All household members make use of high quality, accessible health services), which are not evaluated in detail. 1.3 Interim Evaluation Purpose and Objectives In 2021, under the USAID BHA Implementer Led Evaluation and Learning (IMPEL) with Save the Children, Tulane University School of Public Health and Tropical Medicine (Tulane) was subcontracted to conduct an interim evaluation, including a population-based survey (PBS) and a resilience assessment, in the Budikadidi RFSA area. As part of the overall R&I approach, the interim evaluation was originally planned to take place in 2020 so that the findings could help inform decisions on RFSA extensions. However, the evaluation was delayed until 2021 because of the COVID-19 pandemic. Prior to the interim evaluation, BHA made the decision to extend the Budikadidi RFSA for 2 years. 4 The extension took many factors into account, but the design of the interim evaluation to track lower-level outcome indicators remained the same. The PBS serves as the second phase of a pre-post survey cycle, with data on the same indicators collected in both survey waves. This pre-post design allows for the determination of statistically significant change in indicators between baseline (Round 1) and the interim (Round 2) data. In addition, statistically detecting changes (if any) for all practice and behavioral change indicators can help inform the performance of the award to date. Change over time of indicators in the following categories will be assessed: • Food security • Water, sanitation, and hygiene • Agriculture • Women’s health and nutrition • Children’s health and nutrition • Gender • Resilience The interim evaluation will not assess higher-level poverty and anthropometry indicators but rather lower-level outcome indicators. BHA may conduct a third (endline) survey in the future which could collect the HH consumption/expenditure (poverty) data and anthropometric measurements. 4 The vision for the R&I approach is generally to issue up to 2-year performance-based extension to BHA resilience activities that demonstrate 'outstanding' performance. The interim evaluation is one component that usually contributes to that determination. IMPEL | Implementer-Led Evaluation and Learning 4 Introduction It should be noted that the evaluation does not include data from areas in which the intervention was not implemented (i.e., a counterfactual). While the evaluation results can be used to help explain differences between the pre- and post-values and may therefore help paint a general picture about overall performance, true attribution is only possible with experimental and quasi-experimental evaluation designs, which are not being used in this evaluation. However, differences in outcomes based on the level of exposure to the activity will be assessed where feasible. 1.4 Study Team and Partners The baseline HH survey was administered in July and August of 2017 by the Mendez England and Associates (ME&A) consortium. Tulane University sub-contracted the Kinshasa School of Public Health (KSPH) to conduct the second HH survey in 2021. Tulane has a long-standing relationship with KSPH, going back to the founding of the school in 1986. Dr. Pierre Akilimali, an associate professor at KSPH, directed the fieldwork. Dr. Janna Wisniewski, a faculty member from Tulane, traveled to Kinshasa to work with Dr. Akilimali on supervisor training and pilot testing. Dr. Wisniewski also oversaw the independent survey monitors, who were not associated with KSPH; they were supervised by Elvis Cidoro, a Congolese national and graduate of the Tulane School of Public Health and Tropical Medicine. The Tulane team who conducted the analysis consisted of Dr. Nancy Mock, Dr. Michelle Lacey, and Peter Horjus. Interim Evaluation of the Budikadidi RFSA in the DRC (Vol. I) Quantitative Evaluation Methods 5 2. QUANTITATIVE EVALUATION METHODS 2.1 Overview The PBS serves as the second phase of a pre-post survey cycle, with data on the same indicators collected in both survey waves. This pre-post design allows for the determination of statistically significant change in indicators between baseline (Round 1) and the interim (Round 2) data; however, it does not allow statements to be made about attribution or causation relating to activity impact. As such, the evaluation results can be used to help explain differences between the pre- and post-values and may therefore help paint a general picture about overall performance. True attribution is only possible with experimental and quasi-experimental evaluation designs, which are not being used in this evaluation. Data were collected in person through a population-based HH survey, conducted in the same months (July/August) to avoid potential bias from seasonality). The baseline survey used a multi-stage cluster sample design, and in development of the scope of work for the interim survey, it was decided that the same clusters (villages) were to be resampled for the interim. However, the Round 2 sample needed to be modified to account for changes made to the RFSA coverage areas after the baseline was conducted (see section on sample design). Results are disaggregated—where the sample allows—to report changes in those areas covered by both the baseline and current survey, as well as the current estimates of the full sample for each RFSA, representative of the entire current RFSA coverage area. Additionally, activity data on the location of the implementation of various components of the activity are used to assess differential associations by exposure. Findings from the PBS are triangulated with findings from the mid-term qualitative evaluation to develop insight into the reasons why changes in key outcomes were or were not observed and to offer recommendations for future activities. The interim PBS was conducted in compliance with both ethical and human rights standards. Survey procedures were designed to protect participants’ privacy, allowing for anonymity and voluntary participation. Ethical was obtained from the Tulane Internal Review Board as well as the Kinshasa School of Public Health (KSPH) Comité d'éthique (Ethics Committee) prior to the start of all fieldwork. 2.2 Sample Design The target population for the PBS was all HHs in the current Budikadidi RFSA implementation area. The sampling frame consisted of a list of all villages in which the Budikadidi RFSA implemented their activities, provided by a consortium led by CRS. At baseline, the implementing partners (IPs) provided a list of villages where they planned to implement their activities along with the estimated HH counts in each village; villages were grouped by health zones and health areas, and HH counts were obtained from the health zone office. This list was used to draw the baseline sample of villages using probability proportional to size systematic random sampling. However, in the second round of quantitative data collection, the evaluation team conferred with BHA and CRS to check whether activities had been implemented in the villages from the baseline sampling IMPEL | Implementer-Led Evaluation and Learning 6 Quantitative Evaluation Methods frame and if other areas not covered by the baseline sampling frame had been added to the implementation area. In the Budikadidi implementation area, baseline data collection was done in only two health zones (Cilundu and Miabi). However, the RFSA later added a third health zone (Kasansa). Additionally, some villages were added to the existing coverage areas in Cilundo and Miabi. After the evaluation team consulted with BHA, BHA decided that it would be preferable to adjust the sampling frames to account for areas that were removed from the coverage area following the baseline, but much more importantly, to include the areas that were added to the coverage area after the baseline. This decision was based on the expressed need for current population estimates of the entire RFSA coverage area, despite the loss of an important degree of comparability between rounds. This resulted in an interim survey sampling frame that had 64% of the estimated number of HHs living in the ‘new’ areas (areas not covered by the baseline survey). Because of the small overlap between the baseline and interim sampling frames, rather than take a sub-set of the baseline villages to re-sample, a decision was made with BHA to draw a fresh sample rather than attempt to re-sample a sub-set of villages in the area covered by both the baseline and interim. The total number of villages and households in the baseline and interim sampling frames are found in Table 2. Table 2. Baseline and interim evaluation population-based survey sampling frame Survey round Villages TOTAL estimated number of HHs Sampling frame coverage areas Interim 474 139,116 • Kasansa health zone: 73,025 HHs (52% of sampling frame). • Miabi and Cilundo health zones: 15,597 (11% of sampling frame). Baseline 229 67,565 • Miabi and Cilundo health zones only. 2.3 Sample Size The baseline survey collected data from approximately 1,300 HHs in 44 villages. The sampling approach/size was designed to detect an eight-percentage point change in stunting prevalence among children under 5 years of age (0–59 months) in the pre-post comparison. However, data to assess stunting was not collected in the interim performance evaluation surveys. The indicators listed in the evaluation protocol (Annex A) are used to assess pre-post improvements in the intervention areas. The RFSA expected ‘substantial improvements’ for each of the indicators, tentatively defined by BHA as an improvement of 25%5 over baseline levels. In early planning discussions with BHA, it was recommended that a maximum sample of approximately 1,500 HHs (4,500 total), with allowance for non-response, would be sufficient to measure changes for many of the desired indicators and would allow for subsequent disaggregation of indicators by various 5 This refers to percentage change (applicable to indicators expressed as totals and as percentages), not a difference in percentage points (which would refer only to differences between indicators expressed as percentages). Interim Evaluation of the Budikadidi RFSA in the DRC (Vol. I) Quantitative Evaluation Methods 7 household characteristics (gender of household head, for example) and more complex multivariate techniques to allow for a deeper analysis of the data. The initial planning also followed the baseline methodology recommendations of sampling the same clusters (villages) that were in the baseline survey (44 village). However, after the survey planning began, it was found that the RFSA had some degree of change to the coverage areas (villages added and removed from the RFSA coverage area, as described in the previous section). BHA requested to have a representative sample of the current coverage despite the loss of a large degree of comparability between the baseline and Round 2 data collection. In the Round 2 sampling frame, only 36% of HHs were also in the baseline sampling frame (see the section above on the sampling frames). This made the re-sampling of baseline villages with additional villages sampled from the areas added to the sampling frame impractical. Instead, it was decided with BHA to draw a new sample of villages (systematic random sample using probability proportional to size, as was done in the baseline). The number of clusters sampled in Round 2 was larger than baseline. In the case of Budikadidi, an additional health zone was also added to the coverage area. This meant that the fieldwork travel time between villages would exceed that initially planned. In order to stay within budget, the number of HHs per cluster was decreased slightly from baseline (30 HHs/cluster) to 27 HHs per cluster. Table 3. Population-based survey sampled villages/clusters and sample size Health zone (All in Kasaï Oriental Province) Clusters sampled in baseline Re-sampled clusters (Round 2) Newly sampled clusters (Round 2) Total clusters sampled (Round 2) Total HHs to be sampled (Round 2) Cilundu 24 2* 12 14 378 Miabi 20 1* 8 9 243 Kasansa 0 0 23 23 621 TOTAL Budikadidi 44 3* 44 46 1,242 Note: The Budikadidi RFSA sample did not aim to re-sample the same villages as baseline. However, three villages sampled in the baseline were by chance alone re-sampled in Round 2. 2.4 Sample Selection The sample selection followed a similar approach as used at baseline. The sample was selected using multi-stage cluster sampling with three stages of sampling: 1) selection of clusters (or villages), 2) selection of households, and 3) selection of individuals. For the purposes of the household survey, a “household” is defined as “a person or group of people who live together and share meals (i.e., eating from the same pot).” Full sample selection procedures are detailed in the evaluation protocol in Annex A. 2.5 Survey Tools The baseline questionnaire was developed through a series of consultations with BHA, the Food and Nutrition Technical Assistance III Project (FANTA), and the IPs. The same modules were used in Round 2, IMPEL | Implementer-Led Evaluation and Learning 8 Quantitative Evaluation Methods except for the poverty and anthropometry modules. Additionally, questions on household-level participation in RFSA interventions were added to the household questionnaire. The Round 2 questionnaire consists of modules covering the following topics: • Household identification and informed consent. • Household roster. • Household food security (HDDS and FIES). • Children’s nutrition and health. • Women’s nutrition and health. • Water, sanitation, and hygiene. • Agriculture. • Gender – Cash. • Gender – Maternal and child health and nutrition. • Resilience. • Intervention participation. Data on village-level exposure to RFSA interventions was collected from the IPs. 2.6 Fieldwork and Data Collection The baseline questionnaires were programmed in Open Data Kit (ODK) by Trestle Research and Consulting. The surveys were pilot tested in a rural area on the outskirts of Kinshasa. Seven experienced supervisors were hired to oversee fieldwork in Round 2. Drs. Akilimali and Wisniewski led a supervisor training in Kinshasa in June 2021 to orient them to the topics covered by the surveys (nutrition, agricultural practices, resiliency measurement, etc.) and fieldwork activities, including sampling procedures, research ethics and informed consent, COVID-19 precautions, and electronic data collection. A representative from one of the IPs also gave an overview of the activity. Tulane and KSPH adapted the supervisor manual, enumerator manual, and question-by-question guide that were used at baseline to foster comparability between survey rounds. The Tulane team customized the manuals only to align with the final questionnaire, the general supervision approach of KSPH, and the protocol for using ODK rather than CSPro, which was the program used at baseline.6 Supervisors practiced administering the questionnaires to each other, and then a full pilot test was held in a rural area on the outskirts of Kinshasa. After the pilot, the two supervisors from Kasaï Oriental worked together to translate technical terminology from French into local languages. They later verified these translations with enumerators in the province. 6 Originally, the plan was to use the same data entry program and software as used in the baseline (CSPro), with only the minor changes made to the tool where needed. This was to save time and expense, as well as to ensure the data collection tools were the same in both rounds. However, when the survey preparation began, and Tulane was provided with the baseline survey program, it was discovered that it was written in an older version of CSPro that was not compatible with the current version of the software, and updating it would require starting the programming from scratch. The evaluation team chose to use ODK to re-create the data entry program to minimize the additional costs and time required, as the team had a stronger knowledge and experience with that software. Interim Evaluation of the Budikadidi RFSA in the DRC (Vol. I) Quantitative Evaluation Methods 9 The supervisors then traveled to the provinces where they recruited enumerators. All field supervisors and enumerators (including team leaders) were identified from the KSPH roster of experienced survey data collectors. These team members were recruited from the region in which they were working and spoke fluent French as well as the relevant local languages. Representatives from the implementing partners attended training and presented the activity overview to the enumerators. Enumerators underwent training similar to that of the supervisors, including practice administering the survey in pairs and a full pilot test. Following training, enumerators deployed in teams of two to selected villages. The ‘teams’ are pairs of enumerators, with one of the enumerators also serving as the ‘team leader’ (but still collecting data). In the KSPH experience, it is less costly to have a team of two stays in a village until all surveys are completed rather than having larger teams working in a single village with more frequent movement. Thirty enumerators were deployed in Kasaï Oriental. To identify the villages selected, the teams used an application called Open Street Map Automated Navigation Directions, which was installed on the tablets. GPS points for each village were provided to the field teams. Once in the villages, enumerators counted and selected HHs to participate in the survey, obtained consent from respondents, and collected data. No villages in the Kasaï Oriental sample needed to be replaced. Data were transmitted electronically from tablets to a cloud-based server. Data collection occurred in July and August 2021. An independent survey monitor observed 20 HH surveys to assess the extent to which proper procedures were followed. The monitor used standardized data collection forms, which they transmitted electronically to a survey monitor supervisor based in Kinshasa. Data from these reports were aggregated and shared with Dr. Akilimali on an ongoing basis so that adjustments could be made quickly. The survey monitor did not report any major issues with data collection. Additionally, Dr. Akilimali ran quality checks on incoming data, including age pyramids and completion rates by each enumerator. 2.7 Data Analysis Data was exported from ODK into SPSS for initial cleaning. Then further cleaning, indicator calculation, and analysis were conducted in STATA, SPSS, and R. Tulane generated estimates for all BHA and activity-specific indicators, along with additional analyses to explore relationships and plausible determinants for key outcome indicators and a select number of resilience indicators. The syntax/do-files from the baseline were used to ensure all indicators were calculated in the same way. The indicator results tabulated in Annexes D and E take into account the populating weights and sampling design to generate estimates and conduct statistical tests, using the entire sample from both rounds (despite differences in areas of estimation). Annex D has information on the key indicator results, including the confidence intervals, design effects, record numbers, standard errors, and weighted population. Annex E shows the indicator values from both rounds, as well the statistical comparisons between rounds. IMPEL | Implementer-Led Evaluation and Learning 10 Quantitative Evaluation Methods When conducting comparative analysis between rounds as presented in the body of the report, the portion of the Round 2 sample drawn from Kansasa health zone is excluded, as this area was not covered by the baseline. The sample size used in this comparative analysis is 1,258 HHs at baseline, and 611 HHs at Round 2 (interim). This analysis also does not make use of the probability weights for the calculation of the indicators in both rounds, or in the tests of significance. This renders the results ‘less’ representative of the greater population. However, this also removes the bias from outlier sampling weights. This analysis is referred to as the “overlapping” areas throughout the report. When conducting analysis that relies only on Round 2 data, such as comparisons of outcomes between HHs reporting exposure and those reporting non-exposure to certain RFSA interventions, the entire sample of 1,224 HHs (including Kasansa) is used, with weights applied. 2.8 Final Sample and Sampling Weights Sampling weights were computed and used in the data analyses (except where noted, see previous section), following the same general approach as the baseline. Weights were computed according to the unique sampling scheme that is relevant to the associated sampled household or individual. This involved computing an overall HH sampling weight for each distinct sampling group by taking the inverse of the product of the probabilities of selection from each stage of sampling (village selection and household selection), also accounting for the correction of sampling probabilities from each sampled village (clusters) estimated number of HHs (from the sampling frames), and the actual number of HHs in the village (collected in the listing exercise during data collection). Weights were calculated for the following distinct sampling groups: • Households (used for indicators derived from Modules B, C, F, and R) • Children under 5 years of age (Module D) • Women 15–49 years of age (Module E) • Farmers (Module G) • Cash-earning adults in a union (Module J) • Parents of children under 2 years of age (Module K) Household-level weights were calculated to adjust for non-response (where households declined to be interviewed). In each of the individual-level sampling groups, weights were calculated to adjust for non￾response based on the number of eligible individuals listed on the HH roster and the number of individuals interviewed. The non-response then accounts for individuals that declined to be interviewed (rare), and individuals that were listed in the HH roster were not located despite the repeat visits to the household. The household and individual-level non-response are shown in Table 4. Interim Evaluation of the Budikadidi RFSA in the DRC (Vol. I) Quantitative Evaluation Methods 11 Table 4. Final sample size and response rates Sampling group Number sampled Number interviewed Response rate Households* 1,242 1,224 98.6% Children 0–59 months** 1,189 1,084 91.2% Women 15–49 years of age** 1,478 1,205 81.5% Farmers** 1,800 1,514 84.1% Cash earning adults in a union** 1,138 949 83.4% Parents of children under 2 years** 953 749 78.6% * For households, the sampled vs. interviewed reflects refusals to participate in the survey. ** For the individual-level data, “number sampled” refers to the total number of eligible household members as recorded on the household rosters, and the “number interviewed” reflects the total number interviewed/collected data on. Non-response is both from refusal to participate by the individual (rare) or absence from the household upon repeated visits. 2.9 Integration of Secondary Data CRS provided data on the timing and location of the implementation of various components of the activity, which is used to assess differential associations by the level of exposure. Findings from the PBS are also triangulated with findings from the mid-term qualitative evaluation as well as subsequent project documents outlining adaptations to the mid-term findings in order to develop insight into the reasons why changes in key outcomes were or were not observed, and to offer recommendations for future activities. 2.10 Limitations and Delimitations This evaluation has several limitations, defined as factors outside the evaluators’ scope of influence. 1. As the evaluation is only based on data from areas that received activity support, the evaluation cannot conclude whether a RFSA caused an observed change in outcomes. 2. The baseline and interim surveys were administered several years apart and by different organizations. While efforts were made to maintain consistency by using the same fieldwork manuals and question-by-question guidance, by incorporating input from IPs, and by rehiring some of the enumerators that worked on the baseline, differences in the administration of the two surveys may have occurred. 3. Although independent survey monitors assessed compliance with fieldwork procedures and data quality checks were run, it is possible that there are quality issues, either in the baseline or interim data sets, which cannot be identified. 4. Household exposure to the activity was measured by asking household members about their participation in various activities. It is possible that participation was under-reported, as heads of households may not have been aware of activity names, IPs, or whether other members of their household had participated. 5. Inaccuracies of the sampling frame required large population weights adjustments in some of the sampled villages in both Rounds 1 and 2. The villages with outlying household weighting IMPEL | Implementer-Led Evaluation and Learning 12 Quantitative Evaluation Methods values, particularly those with large weights, tend to increase the design effects and reduce the statistical power of the analysis. This also means that the population estimates living in the areas of implementation are potentially unreliable. The evaluation also has several delimitations, defined as factors that the evaluation team and BHA agreed to accept, though they were less than ideal. 1. The poverty and anthropometry modules were not administered in Round 2. The poverty module was deemed to be too long and labor-intensive relative to its value, and the physical proximity of enumerators to children while taking anthropometric measurements was considered too risky during the COVID-19 pandemic. 2. Round 2 sample design prioritized the generation of representative results of the current RFSA coverage areas, which had changed/expanded significantly from what was covered in the baseline survey. Approximately two-thirds of the interim sample covered areas that were not covered by the baseline survey, and half was from Kansasa, an area not covered at all by the baseline survey. This limits the comparability of the two rounds. Following comments from the IPs after data collection, additional analyses were run comparing only the overlapping areas, using the sub-set of the Round 2 sample. (See detailed description in the analysis section.) 3. FINDINGS This section begins with an overview and basic analysis of RFSA activity exposure and participation. Then, results are presented by sector. The sector analyses being with a comparative analysis between Rounds 1 and 2. This analysis is unweighted, and the Round 2 sample is limited to only those areas that overlap with Round 1 (see analysis section above). 7 Indicators are also stratified (e.g., by gender) as appropriate (and where the limited Round 2 sample size allow). As applicable, analysis of the components of the indicators is conducted, particularly if that analysis provides additional information that may alter the conclusions drawn from the findings. The comparative analysis for each sector is followed by an analysis of the relationship between RFSA activity exposure and outcomes. This analysis makes use of the full sample with probability weights applied (unless otherwise noted). As such, those are representative of the entire coverage area. 3.1 Intervention Exposure and Participation To assess the levels of RFSA intervention participation and coverage/exposure in the surveyed, data were collected at the community and HH level related to a variety of RFSA interventions. When discussing this data, exposure to an intervention is defined as a household living in a village where the RFSA implemented the intervention (data sourced from the IP). Participation in an intervention is defined (and measured here) as the survey respondent reporting that one or more members of their household having participated in the intervention over the previous 5 years. It is important to note, however, that participation data collected in the Round 2 survey is not meant to be interpreted as providing accurate estimates of coverage by the RFSA, which is better estimated by activity monitoring data. The primary use of the information collected in Round 2 on intervention participation is to allow for analysis of associations (correlations) between lower-level outcome indicators and intervention participation. Overall, 52% of HHs in the Round 2 survey reported participation in one or more of the surveyed RFSA interventions. According to Table 4 in the RFSA’s 2020 Annual Report, the activity was reaching approximately 47,000 unique direct participant HHs. The interim survey sampling frame estimates roughly 140,000 HHs in the area of implementation. This would indicate that roughly 34% of HHs in the implementation area were direct participants in one or more RFSA activity. However, this estimated coverage is likely an underestimate due to registration challenges during the COVID-19 pandemic. 7 Indicators for the full weighted samples from both survey rounds are reported in Annex E, along with tests for the significance of differences between rounds. Interim Evaluation of the Budikadidi RFSA in the DRC (Vol. I) Findings 13 IMPEL | Implementer-Led Evaluation and Learning 14 Findings Activity exposure was high for most interventions (Figure 1), with most interventions being implemented in all the villages sampled in the Round 2 survey. The exception was cash for work activities (28% coverage). Participation ranged from 3% (cash for work) to 37% (WASH training/events). Reporting error is possible, where the respondent (head of household or other decision maker) was simply not aware of the participation of other household members, for example. 61% of the sampled respondents to these questions were male, and 39% were female. 98% were the head of the household or the spouse of the household head. Respondent bias due to gender did not appear to play a role, however. The reported participation in mother’s groups and nutrition trainings, for example, showed no significant difference between male and female respondents. Figure 1. Intervention exposure and participation *Interventions with no reported coverage in the sampled villages are excluded. These include safe space groups and food for work. 0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100% VSLA Employment Training Alterbative Livelihoods Farmers' Groups Resource Management Agricultural Training Improved Agricultural Production Agricultural Inputs Farming Information Market Support WASH Training Toilet Building Mothers' Groups Nutrition Training Home Visits Climate Adaptation Groups Cash For Work Programs Literacy Training Couples' Support Groups Adolescent Life Skills Training Youth Leadership Training Intervention Exposure and Participation % HHs reporting one or more members participated in this intervention in the past 5 years % HHs living in villages where intervention was done by RFSA Interim Evaluation of the Budikadidi RFSA in the DRC (Vol. I) Findings 15 3.2 Water, Sanitation, and Hygiene Table 6 displays a comparison of water, sanitation, and hygiene (WASH) indicators between 2017 and 2021 in the overlapping areas. Looking at drinking water quality and access, indicators remained the same or improved across the board. The percentage of HHs with an improved source of drinking water increases significantly from 47% to 57%, and the percentage of HHs that can obtain drinking water in less than 30 minutes increased from 56% to 72%. Although the percentage of HHs practicing correct use of recommended water treatment technologies increased from 3% to 5%, the change was not significant. The 2019 mid-term evaluation also indicated several weak points in the drinking water interventions at that time, including high demand for potable water leading to long wait times, and concerns about the quality/durability of the borehole foot pumps. The 2020 Annual Report indicates that the RFSA had taken action to address the pump quality/durability, and they plan to continue to expand water interventions to meet the high demand. In terms of sanitation, the percentage of HHs using open defecation shows important, significant changes in the overlapping areas, decreasing from 35% to 18%. While the percentage of HHs with soap and water at a handwashing station commonly used by family members was low in 2021 (5%), it had increased significantly since 2017 (1%). The percentage of HHs using a basic sanitation facility decreased significantly, although prevalence in both Round 1 and Round 2 was low (8% and 3%, respectively). The percentage of HHs using a basic sanitation facility was low in both rounds, but data indicate it decreased significantly (8% to 3%). However, possible discrepancies in latrine type classification may be masking change. The overlapping areas of the interim survey shows an increased percentage of HHs using unimproved pit latrines (pit latrines without a slab base) than at baseline (68% at interim, 31% at baseline). The first generation of latrines built under the RFSA used local materials for slab construction (not cement), which in some cases may have been classified as unimproved pit latrines when observed by enumerators. As such, the prevalence of open defecation is more likely to show the population level￾improvements in sanitation. The addition of Kasansa to the sampling frame contributed to an apparent reduction in the relative proportion of Budikadidi RFSA HHs with access to water in the interim survey as compared to baseline. Looking at the full weighted Round 1 and Round 2 samples, 55% of HHs in Cilundu and 60% of HHs in Miabi indicated that they had access to a basic water source, while this was true for only 38% of Kasansa HHs. Similarly, 49% of Kasansa HHs required over 30 minutes to obtain water, compared with 29% and 27% of HHs in Cilundu and Miabi, respectively. Table 5. Baseline (BL) and interim water, sanitation, and hygiene indicator comparisons (overlapping areas only, unweighted) Indicator Indicator value Significance level Number of observations 2017 Baseline 2021 Interim BL Int. 6. Percentage of households using an improved drinking water source 47.3% 56.6% *** 1258 611 IMPEL | Implementer-Led Evaluation and Learning 16 Findings Indicator Indicator value Significance level Number of observations 2017 Baseline 2021 Interim BL Int. 7. Percentage of households practicing correct use of recommended household water treatment technologies 3.4% 4.6% ns 1258 611 8. Percentage of households that can obtain drinking water in less than 30 minutes (round trip) 55.8% 71.8% *** 1258 611 9. Percentage of households using a basic sanitation facility 8.1% 3.1% *** 1258 611 10. Percentage of households in target areas practicing open defecation 34.5% 17.8% *** 1258 611 11. Percentage of households with soap and water at a handwashing station commonly used by family members 0.6% 4.9% *** 1258 611 Note: ns = not significant, *** p < 0.001 WASH training/events were implemented in all villages except Bakuamba, with 31.2% of HHs reporting participation in these activities. Among those who report HH participation in WASH trainings/events, proper handwashing rates were 7.0%, significantly higher than the 1.9% reported among non￾participants (p < .001), and water treatment rates were also higher, 3.8% in non-participants, and 6.7% among participants (p = 0.045). Looking at toilet building interventions, 96% of HHs lived in villages where the RFSA implemented these, with 32% of HHs reporting having built a toilet. The rate of open defecation among participants was 11.2%, significantly lower than the 16.9% reported from non-participants (p = 0.02). 3.3 Agriculture Changes in agricultural practices between 2017 and 2021 are shown in Table 6. In the overlapping areas, some indicators remained stable or increased significantly. The percentage of farmers who practiced value chain activities promoted by the RFSA in the previous 12 months increased significantly from 19% in 2017 to 37% in 2021. The use of financial services, the use of improved storage practices, and the use of at least 3 sustainable natural resource management (NRM) practices/technologies saw no significant changes between survey rounds. Looking at sustainable crop and livestock) practices/technologies in the overlapping areas, there were unexpected significant decreases observed. 8 The percentage of farmers who used three sustainable crop practices/technologies remained low, but experience a significant decrease from 7% to 3%. Similarly, the 8 Indicator 14, the percentage of farmers using at least 3 sustainable crop/livestock/NRM practices/technologies in the past 12 months, is calculated from the same data as indicators 15, 16 and 17, and so simply reflects the decreases observed in 15 and 16 (and no change in 17). Interim Evaluation of the Budikadidi RFSA in the DRC (Vol. I) Findings 17 percentage of farmers using at least three sustainable livestock practices/technologies remained low, decreasing from 10% to 4%. Table 6. Baseline and interim agriculture indicator comparisons (overlapping areas only, unweighted) Indicator Indicator value Significance level Number of observations 2017 Baseline 2021 Interim BL Int. 12. Percentage of farmers who used financial services (savings, ag credit, &/or ag insurance in the past 12 months 32.2% 35.7% ns 1306 746 13. Percentage of farmers who practiced value chain activities promoted by the RFSA in the past 12 months 18.8% 36.6% *** 722 527 14. Percentage of farmers who used at least 3 sustainable agriculture (crop, livestock, and NRM) practices and/or technologies in the past 12 months 18.1% 14.2% * 1306 746 15. Percentage of farmers who used at least 3 sustainable crop practices and/or technologies in the past 12 months 7.4% 2.9% *** 1306 725 16. Percentage of farmers who used at least 3 sustainable livestock practices and/or technologies in the past 12 months 9.8% 4.2% ** 387 262 17. Percentage of farmers who used at least 3 sustainable NRM practices and/or technologies in the past 12 months 0.2% 0.0% ns 1306 746 18. Percentage of farmers who used improved storage practices in the past 12 months 32.2% 35.2% ns 1290 725 Note: ns = not significant, * p < 0.05, ** p < 0.01, *** p < 0.001 Table 7 below shows the change in reported use of each type of sustainable crop practice/technology in the overlapping areas. Among the promoted practices, the use of manure and compost saw large increases. The use of mulching and crop rotations saw small increases. The use of weed control saw a moderately large decrease, and the others (Integrated Pest Management (IPM)), improved seeds/crop varieties, improved fallow with cover, and planting of perennial forage crops) remained very low, with only minor decreases between rounds. The use of the non-promoted practices all remained low, with little change between rounds. Overall, the percentage of farmers using at least one of the surveyed sustainable crop practices over the past 12 months increased greatly, from 62% to 82%. Despite improvements in the use of many of the sustainable crop practices promoted by the RFSA, the prevalence of the use of three or more decreased (see indicator 14, above). This discrepancy is accounted for primarily by the observed difference between baseline and interim in the percentage of farmers practicing at least one sustainable crop practice, and the number of sustainable crop practices reported on average by farmers. The baseline observed fewer farmers practicing at least one sustainable crop practice (62% of farmers) compared to the interim survey, where 82% of farmers were found to be practicing at least one sustainable crop practice. However, among these farmers who reported using at least one of these practices, those at baseline reported a larger total number of these practices on IMPEL | Implementer-Led Evaluation and Learning 18 Findings average than in the interim survey. So, while the percentage of farmers using several interventions went up, the percentage of farmers using three or more decreased slightly. It’s difficult to assess whether or not these differences between rounds stem from methodology/collection issues (differences in how the questions were administered between rounds, or misunderstanding of the practices by the enumerators in the baseline and/or interim surveys for example). They may also represent actual changes in the type and number of practices used. For the purposes of the evaluation, the evaluation team has chosen to place focus on the changes in the individual sustainable crop practices, which show neutral or positive change (with the exception of weed control). Table 7. Changes in prevalence of sustainable crop practices between baseline (2017) and interim (2021) (overlapping areas only, unweighted) Crop practices* Baseline** Interim Difference Manure 11.4% 32.8% 21.4% Compost 11.6% 23.9% 12.3% Mulching 16.2% 17.9% 1.7% Weed control 31.1% 15.2% -15.9% Crop rotations 9.8% 9.8% 0% Intercropping 6.5% 2.9% -3.6% IPM 0.2% 0.1% -0.1% Improved seeds/crop varieties 0.2% 0% -0.2% Improved fallow with cover 1.0% 0.4% -0.6% Planting of perennial forage crops 0.4% 0.1% -0.3% Dry Planting 0.7% 0.8% 0.1% Ripping into residues 1.1% 0.4% -0.7% Clean ripping 0.0% 0.4% 0.4% Tied ridges 0.2% 0% -0.2% Zai pits 0.0% 0.1% 0.1% Pot-holing 0.0% 0.0% 0.0% Contour planting 0.2% 2.9% 2.7% Terracing 1.4% 3.3% 1.9% Land leveling 1.2% 4.8% 3.6% Maintain indigenous trees to improve soil fertility 0.9% 0.1% -0.8% Used at least ONE of these practices in the past 12 months 62.4% 81.5% 19.1% *Practices with BOLD font are promoted by activity. Only the promoted practices in BOLD contribute to the improved practices indicator and sub-indicators. **Baseline data is weighted. Table 8 shows the breakdown by livestock practice/technology use in the overlapping areas. It is important to note that in the overlapping Round 2 sample, the results are based only 262 farmers, and so should be interpreted with caution. Among the promoted sustainable livestock practices, only a few modest changes were observed. The use of improved animal shelters, use of homemade animal feeds from local products, and pen feeding all fell by a few percentage points. The practices for breeding and keeping of rabbits increased by a few percentage points. The other promoted practices saw smaller changes. The use of the non-promoted Interim Evaluation of the Budikadidi RFSA in the DRC (Vol. I) Findings 19 practices all remained low, with little change between rounds, with the exception of castration, which increased from 6% to 9%. Overall, the percentage of farmers not at least one of the sustainable livestock practices over the past 12 months increased modestly, from 39% to 45%. Similar to the findings in the crop practices, while the number of farmers using at least one of the surveyed livestock practices increased from Round 1 to Round 2, these farmers reported using fewer total numbers of practices, resulting in a decrease in indicator 16 (percentage of farmers who used at least 3 sustainable livestock practices and/or technologies in the past 12 months). As for crop practices, it is difficult to determine whether the differences are from methodological issues, enumerator error, or actual changes in practices. Considering the small sample sizes, there seem to be few large, important changes at the population level in the overlapping areas in terms of use of sustainable livestock practices. Table 8. Changes in prevalence of sustainable livestock practices between baseline (2017) and interim (2021) (overlapping areas only, unweighted) Livestock Practices* Baseline** Interim Difference Improved animal shelters 15.4% 9.2% -6.2% Vaccinations 14.6% 15.6% 1.0% Deworming 8.4% 8.8% 0.4% Homemade animal feeds from local products 14.3% 9.2% -5.1% Animal feed supplied by stockfeed manufacturer 0.6% 0.4% -0.2% Pen Feeding 7.1% 3.4% -3.7% Fodder production and/or veld reinforcement 0.4% 0.8% 0.4% Used the services of community animal health workers/para￾veterinarians 0.1% 0% -0.1% Practices for breeding and keeping of rabbits 8.1% 12.6% 4.5% Feed animals with nutritional supplements during the hunger season 2.5% 0% -2.5% Castration 6.1% 9.2% 3.1% Dehorning 1.4% 0% -1.4% Artificial insemination 1.6% 0% -1.6% Make hay or silage to feed animal during the dry season 1.3% 0% -1.3% Used at least ONE of these practices in the past 12 months 39.4% 45.4% 6.0% * Only the promoted practices in BOLD contribute to the improved practices indicator and sub-indicators. **Baseline data is weighted. Participation in Village Savings and Loan Associations (VSLAs) was reported by 32% of farmers and was associated with a significantly higher rate of access to financial services. Value chain activities were implemented by higher percentages of farmers who received agricultural inputs (41% vs. 35%), although only 15% of farmers reported receiving such inputs. Furthermore, among the 8% of farmers who received market support, rates of utilization of improved storage were significantly higher (41.5% IMPEL | Implementer-Led Evaluation and Learning 20 Findings compared to 67%). Market support was also associated with higher levels of sustainable agriculture, as was participation in agricultural training (which was reported by 28% of farmers). 3.4 Women’s Health and Nutrition Indicators of women’s health and nutrition in the overlapping areas are displayed in Table 9. The prevalence of women of reproductive age consuming a diet of minimum diversity in the overlapping areas decreased slightly (20% to 16%), though this was not a significant change. However, reported consumption of targeted nutrient-rich value chain and non-value chain commodities the previous day dropped significantly, from 66% of women of reproductive age to 30%. This was primarily driven by a large reduction in the consumption of bio-fortified food (cassava, maize, beans), from 37% at baseline to only 10% at interim. This may be due to the distribution of fortified foods by other actors during the baseline. Consumption of other targeted nutrient rich foods generally remained low, with no or negative changes. As such, the change in women’s consumption of targeted nutrient foods is considered generally neutral, in line with the relatively small changes in women’s minimum diet diversity, and household diet diversity (presented in a following section). The use of contraceptives among women of reproductive age in a union increased significantly, driven mainly by increases in the traditional methods. 9 The prevalence of underweight women and births receiving at least four antenatal care visits were not assessed at interim.10 Participation in nutrition training is significantly associated with higher Minimum Dietary Diversity levels. Among the 16% of women in HHs reporting participation, 21% had consumed a diet of minimum diversity, compared to only 13% of women from HHs not reporting participation. Additionally, among women from HHs reporting participation in mother’s groups, contraceptive use (modern or traditional) was 13 percentage points higher than those not from HHs reporting participation. However, the care group promoted by the RFSA did not organize any sessions to promote any form of contraception. This association may be related to participation mother’s groups organized by other implementers, or it may be that women who chose to participate in members of mother’s groups are also more likely to use contraception. Table 9. Baseline and interim women’s health and nutrition indicator comparisons (overlapping areas only, unweighted) Indicator 2017 Baseline 2021 Interim Significance level Number of Observations BL Int. 20. Prevalence of women of reproductive age consuming a diet of minimum diversity 19.7% 16.0% † 1309 592 9 Traditional contraceptive methods include: Standard days method, lactational amenorrhea method, rhythm method, withdrawal, and other traditional method. Modern contraceptive methods include: female sterilization; male sterilization; intrauterine device (IUD); injectables; implants; pills; male and female condoms; other modern methods. 10 An error in the skip patterns in the data collection program resulted in excessive missing data the antenatal visit variable. Anthropometry was not collected due to concerns related COVID-19. Interim Evaluation of the Budikadidi RFSA in the DRC (Vol. I) Findings 21 Indicator 2017 Baseline 2021 Interim Significance level Number of Observations BL Int. 21. Contraceptive Prevalence Rate^ 8.6% 15.5% ** 710 290 23. Prevalence of women of reproductive age who consume targeted nutrient-rich value chain and non-value chain commodities 66.4% 30.4% *** 1309 592 Value Chain (all) 39.7% 10.0% *** 1309 592 Bio-fortified (cassava, maize, beans) (VC) 37.1% 9.6% *** 1309 592 Cabbage (VC) 0.9% 0.0% * 1309 592 Moringa (VC) 0.8% 0.0% ns 1309 592 Okra (VC) 5.0% 0.5% *** 1309 592 Non-value chain (all) 26.7% 20.4% ** 1309 592 Orange-flesh sweet potatoes 5.0% 4.4% ns 1309 592 Soybean 9.2% 3.2% *** 1309 592 Cowpea 6.6% 1.4% *** 1309 592 Carrots 0.8% 0.0% ns 1309 592 Orange 3.1% 0.0% *** 1309 592 Pineapple 1.1% 0.7% ns 1309 592 Passionfruit 0.8% 0.3% ns 1309 592 Mango 10.8% 0.0% *** 1309 592 Sweet green pepper 14.2% 13.7% ns 1309 592 Eggs 3.4% 1.4% * 1309 592 Animal protein 33.6% 16.7% *** 1309 592 Note: ns = not significant, † p < 0.1, * p < 0.05, ** p < 0.01, *** p < 0.001 Bolded Commodities are those promoted by the RFSA ^ The Contraceptive Prevalence Rate indicator is included in the report because it is a standard BHA indicator. However, CRS’ project does not include activities to promote contraceptive uptake. Activities relevant to this indicator are implemented by Abt Associates in the Integrated Health Program, a separate USAID-funded project that operates in the same target area as Budikadidi. 3.5 Children’s Health and Nutrition Indicators of the health and nutrition of children in the overlapping areas are shown in Table 11. There was a significant decrease in the percentage of children under 5 years who had diarrhea in the last 2 weeks (44% to 33%). Treatment of diarrhea with ORT increased from 39% to 45%, and exclusive breastfeeding of children under 6 months of age increased from 60% to 71%. However, these changes were not significant. The small Round 2 overlapping area sample limits power to detect change, particularly for breastfeeding. The prevalence of children 6-23 months receiving a minimum acceptable diet, and the prevalence of children 6–23 months who consume targeted nutrient-rich value chain and/or non-value chain IMPEL | Implementer-Led Evaluation and Learning 22 Findings commodities did not experience a significant change. The lack of movement in diet quality/diversity is also found among women of reproductive age (see previous section), and at the household level (household dietary diversity). The small sample size of children 6-23 months at interim in the overlapping areas is small (152), so further analysis of the food items consumed is not presented here. The information for the full weighted sample is included in Annex D and E. Anthropometric measurements were not taken during interim data collection due to the risk of close contact during the COVID-19 pandemic. If possible, they will be taken at endline. Table 10. Baseline and interim children’s health and nutrition indicator comparisons (overlapping areas only, unweighted) Indicator 2017 Baseline 2021 Interim Significance level Number of observations BL Int. 27. Percentage of children under age 5 who had diarrhea in the last 2 weeks 44.2% 33.1% *** 1379 559 28. Percentage of children under age 5 with diarrhea treated with ORT 38.8% 45.4% ns 609 185 29. Prevalence of exclusive breast-feeding of children under 6 months of age 56.9% 71.4% † 167 42 30. Prevalence of children 6–23 months of age receiving a minimum acceptable diet (MAD) 4.3% 2.5% ns 420 157 31. Prevalence of children 6–23 months who consume targeted nutrient-rich value chain and/or non-value chain commodities 51.9% 49.7% ns 420 157 Note: ns = not significant, † p < 0.1, *** p < 0.001 Mothers’ groups, nutrition training, and home health visits were implemented in all villages where data collection took place. However, the HH self-reported participation rates were low, with between 19% and 20% of children living in HHs that reported engaging in each of these interventions. Furthermore, there were no associations between reported participation in these interventions and any improvements in child MAD, breastfeeding practices, frequencies of diarrhea, or knowledge of MCHN practices. The 2018 mid-term evaluation highlighted that many of these activities demand highly effective Lead Mothers to be successful. It may be that the COVID-19 restrictions limited the ability of the Lead Mothers to adequately conduct their work. 3.6 Gender Changes in indicators related to gender in the overlapping areas are displayed in Table 11. Generally positive changes were observed in many of the indicators. Although some indicators did not show positive change, the fact emphasized in the mid-germ evaluation is that gender norms in Kasaï are deeply entrenched and will take a long time to change. The percentage of adult women earning cash in the past year increased significantly between rounds (27% to 35%). Among men, the figure did not change significantly (44% to 45%). This would indicate that new cash earing opportunities are being made available to women in particular. Interim Evaluation of the Budikadidi RFSA in the DRC (Vol. I) Findings 23 Among women in unions earning cash, there was a shift from women making decisions about their self￾earned cash from alone to jointly with their spouse/partner. These results, however, are difficult to interpret. The combined figures (women making decisions alone plus those making decisions with spouse) may give a clearer idea of the overall degree of control women may have over self-earned cash. This prevalence of cash-earning women in a union making decisions about their cash along or jointly with their spouse decreased from 58% at baseline to 47% at interim. Cash-earning men in unions, however, reported making decisions alone about the cash they earned significantly less often in the interim than at baseline. However, joint decision-making about self-earned cash among men did not change significantly between rounds. There was a statistically significant increase in the percentage of men and women with children under 2 years who have knowledge of maternal and child health and nutrition practices. Looking at decision-making in men and women in unions about maternal health and nutrition (MHN), and about child health/nutrition (among those with a child under 2 years old), there were significant shifts from making decisions alone to making them jointly with their spouse/partner among both men and women. Among men, this shift away from being the sole decision maker in these women and child health decisions is clearly positive. Among women, however, the combined percentage of women reporting making these decisions alone or jointly did not see large changes between survey rounds, indicating that women did not gain decision-making power their own or their children’s health in general between rounds. Table 11. Baseline and interim gender indicator comparisons (overlapping areas only, unweighted) Indicator 2017 Baseline 2021 Interim Significance level Number of observations BL Int. 32. Percentage of adults who earned cash in the past 12 months Male 35.5% 39.8% ** 3518 1795 Male 43.9% 44.5% ns 1774 889 Female 26.9% 35.2% *** 1744 906 Men: Self￾earned cash decisions 33a. Percentage of men in union and earning cash who make decisions alone about the use of self-earned cash 50.5% 38.9% ** 664 285 34a. Percentage of men in union and earning cash who make decisions jointly with spouse/partner about the use of self￾earned cash 22.0% 23.5% ns 664 285 Women: Self￾earned cash decisions 33b. Percentage of women in union and earning cash who make decisions alone about the use of self-earned cash 33.2% 11.7% *** 352 213 34b. Percentage of women in union and earning cash who make decisions jointly with spouse/partner about the use of self￾earned cash 24.4% 35.2% ** 352 213 IMPEL | Implementer-Led Evaluation and Learning 24 Findings Indicator 2017 Baseline 2021 Interim Significance level Number of observations BL Int. Men: MHN Decisions 36a. Percentage of men in union with children under 2 years who make maternal health and nutrition decisions alone 54.2% 25.4% *** 450 181 37a. Percentage of men in union with children under 2 years who make maternal health and nutrition decisions jointly with spouse/partner 13.1% 26.0% *** 450 181 Women: MHN Decisions 36b. Percentage of women in union with children under 2 years who make maternal health and nutrition decisions alone 26.1% 14.6% ** 479 205 37b. Percentage of women in union with children under 2 years who make maternal health and nutrition decisions jointly with spouse/partner 15.7% 26.8% ** 479 205 Men: CHN Decisions 38a. Percentage of men in union with children under 2 years who make child health and nutrition decisions alone 42.0% 18.2% *** 450 181 39a. Percentage of men in union with children under 2 years who make child health and nutrition (CHN) decisions jointly with spouse/partner 22.2% 35.9% *** 450 181 Women: CHN Decisions 38b. Percentage of women in union with children under 2 years who make child health and nutrition decisions alone 23.8% 11.2% *** 479 205 39b. Percentage of women in union with children under 2 years who make child health and nutrition decisions jointly with spouse/partner 24.0% 32.7% * 479 205 35a. Percentage of men with children under 2 years who have knowledge of maternal and child health and nutrition (MCHN) practices 57.1% 69.8% ** 450 182 35b. Percentage of women with children under 2 years who have knowledge of MCHN practices 65.1% 78.9% *** 507 213 Note: ns = not significant, * p < 0.05, ** p < 0.01, *** p < 0.001 3.7 Activity-Specific Indicators custom-made for the interim survey in the overlapping areas are shown in Table 13. The only significant change was in indicator 47, the percentage of men/women who say it is ok for a man to batter his wife for at least one of the reasons surveyed, which went up a moderate but significant amount (64% to 70%). This was driven uniquely by male respondents, which increased significantly from 62% to 71%. Women saw a non-significant decrease from 34% to 31%. Interim Evaluation of the Budikadidi RFSA in the DRC (Vol. I) Findings 25 Data on livestock penning was not collected at interim. 11 Table 12. Baseline and interim activity-specific indicator comparisons (overlapping areas only, unweighted) Indicator 2017 Baseline 2021 Interim Significance level Number of observations BL Int. 46. Percentage of caregivers who properly disposed of child feces 51.2% 52.3% ns 586 197 Male 52.8% 52.7% ns 307 112 Female 49.5% 51.8% ns 279 85 47. Percentage of men/women who say it is ok for a man to batter his wife for any reason 63.8% 70.0% ** 1311 746 Male 61.7% 71.2% ** 681 386 Female 33.8% 31.2% ns 417 260 48. Percentage of women that report participating in community decision-making bodies 18.2% 19.4% ns 1309 592 Note: ns = not significant, ** p < 0.01 3.8 Shock Exposure and Resilience Indices Significant declines in the shock exposure indices (both unweighted and severity weighted) relative to the baseline were predominantly due to sizable drops in the proportion of HHs reporting economic shocks (Table 13). In 2017, 79% of all HHs overall reported rising food prices as a shock, compared with 47% in 2021, and the proportion of HHs reported shocks related to fluctuating exchange rates and currency devaluation declined from 49% to 13%. Declines of over 10% were also observed for excessive rain and/or drought, looting or robbing, unavailability of agricultural inputs, rising or falling prices for agricultural inputs, and forced displacement. Interestingly, there were no shocks that significantly increased in magnitude during this period, while rates of drought/flooding and family illness or death remained above 40% overall. 11 An error in the ODK skip pattern resulted in excessive missing data. IMPEL | Implementer-Led Evaluation and Learning 26 Findings Figure 2. Household-reported shocks Reported Shocks in Budikadidi RFSA area All three resilience capacities (absorptive, adaptive, and transformative) increased significantly in the overlap areas between baseline and interim. Changes in the indices were largely driven by a single component indicators and/or shifts in the symmetry of the score distribution such that the outlying scores in the tails had more of an impact on the change (rather than an overall shift in the population). Similar trends observed when looking at the full, weighted results. Note that the interim evaluation did not collect poverty data, which was used as one of the components in the calculation of the resilience indices. The baseline indicators were re-calculated to match the calculations used in the interim survey to allow for improved comparability between rounds. Interim Evaluation of the Budikadidi RFSA in the DRC (Vol. I) Findings 27 Table 13. Baseline and interim shock exposure and resilience indicator comparisons (overlapping areas only, unweighted) Indicator Indicator value Significance level Number of observations 2017 Baseline 2021 Interim BL Int 40. Shock exposure index 4.9 3.2 *** 1,253 611 41. Cumulative impact of shock exposure index (severity weighted shock exposure) 27.5 18.0 *** 1,253 611 42. Absorptive capacity index 21.1 42.3 *** 1,253 611 43. Adaptive capacity index 23.7 29.4 *** 1,253 611 44. Transformative capacity index 15.8 18.7 *** 1,253 611 Note: *** p < 0.001 The absorptive capacity index increased significantly in the overlapping areas (21 to 42). This change was driven largely by the increased presence of humanitarian assistance in the surveyed communities (3% at baseline to 31% in the interim overlapping areas), which may be due to other organizations conducting rapid responses in some of the RFSA implementation area. This was also driven by smaller increases in informal safety nets and productive assets. Improvements were tempered by a moderate decrease in preparedness/mitigation, and small decreases in a few of the other components (see table 14). The distribution of the absorptive asset scores showed greater symmetry in the interim survey than the baseline, which indicates that the improvement in the mean scores between survey rounds does not necessarily imply an overall population improvement. Additional figures showing the distributions of the index are found in Annex C. Table 14. Absorptive Capacity Index: Changes in component indicators (overlapping areas only, unweighted) Indicator Direction of change (BL to Interim) Significance, scale 1. Informal Safety Nets Increase Significant, moderate 2. Bonding Social Capital Decrease Significant, small 3. Cash Savings Decrease Significant, small 4. Productive Assets Increase Significant, small 5. Livestock Assets Decrease Non-significant 6. Preparedness/Mitigation Decrease Significant, moderate 7. Agricultural Insurance Increase Non-significant 8. Humanitarian Assistance Increase Significant, large 9. Access to Remittances Decrease Significant, large The change in the adaptive capacity index between rounds in the overlapping areas was moderate, though significant (24 to 29). The improvement was driven primarily by the increase in social safety nets, 28 Findings IMPEL | Implementer-Led Evaluation and Learning and to a lesser degree by increases in productive asset ownership. This was offset to some degree by decreases in other indicators, such as livelihood diversity (see Table 15). This decrease in livelihood diversity was mainly due to the reduction in the number of HHs reporting remittances/gifts as one of their livelihood sources, which could be a result of the global economic downturn due to the COVID-19 pandemic. The slight increase in the mean value of the index reflects a reduction in skewness of the distribution of the index values. Between baseline and interim, there was a shift toward the middle of the distribution, with fewer HHs at the upper end of the aggregate scale and little improvement for HHs at the lower end. Additional figures showing the distributions of the index are found in Annex C. Table 15. Adaptive Capacity Index: Changes in component indicators (overlapping areas only, unweighted) Indicator Direction of change (BL to Interim) Significance, scale 1. Bridging Social Capital Decrease Significant, small 2. Social Safety Nets Increase Significant, large 3. Education/Training Decrease Significant, moderate 4. Livelihood Diversity Decrease Significant, large 5. Productive Assets Increase Significant, moderate 6. Livestock Assets Decrease Non-significant 7. Improved Agric. Practices Decrease Non-significant The transformative capacity index showed only a modest (though statistically significant) shift in the overlapping areas (from 16 to 19) and remained low overall. The positive shift was mainly driven by a significant improvement of formal safety nets and was offset by decreases in indicators of collective action,12 bridging social capital, and local government responsiveness, with no changes observed for the remaining indicators included in the calculations (see table 16). The low value of this index demonstrates a high degree of skewness, such that only a few HHs were assigned high scores based on these criteria. Additional figures showing the distributions of the index are found in Annex C. 12 The collective action indicator is defined as the sum of the number of the following seven activities that the HH engaged in over the past 12 months: Soil conservation (terracing, bunds, half-moons, gabions, etc.), Flood diversion activities, Repaired/built schools, Repaired/built health posts or centers, Road maintenance/construction, Planted trees on communal land, Formed a cooperative. In the baseline, 33% of HHs reported participation in at least one activity, most commonly road maintenance (17%), flood diversion (8%), school building/repair (8%) and soil conservation (7%). In the interim, the overall participation levels in the overlap areas declined to 23%, with participation rates for all activities below 5% except for road maintenance (18%). Interim Evaluation of the Budikadidi RFSA in the DRC (Vol. I) Findings 29 Table 16. Transformative Capacity Index: Changes in component indicators (overlapping areas only, unweighted) Indicator Direction of change (BL to Interim) Significance, scale 1. Access to Natural Resources Increase Non-significant 2. Bridging Social Capital Decrease Significant, small 3. Collective Action Decrease Significant, moderate 4. Local Gov’t Responsiveness Decrease Significant, small 5. Participation in Local Decisions Decrease Non-significant 6. Formal Safety Nets Increase Significant, large Looking at the relationship between coping strategies and resilience indicators, only the adaptive capacity index shows a relationship between the number (raw count) of coping activities, with the mean index value increasing as more coping strategies were used (see figure X below). However, three of the adaptive capacity component indicators are related to agriculture, so HHs that did not engage in agriculture tend to have lower scores on this index even if they are highly resilient. This indicates that the adaptive capacity index may be of limited use for HHs not engaged in agriculture. It may benefit from adaptation to reflect adaptability as a function of livelihood. Figure 3. Mean adaptive capacity index by number of coping activities IMPEL | Implementer-Led Evaluation and Learning 30 Findings 3.9 Food Security Indicators related to food security in the overlapping areas are summarized in Table 17 below. The prevalence of moderate and severe food insecurity (as measured by the FIES) was very high at baseline (87%) and increased significantly at the interim (97%) in the overlapping areas, with similar results found in the full sample. However, the FIES questions may not be adequately describing the food security situation in this context. Most HHs answered yes to all (or nearly all) of the eight FIES questions, and there was no observation of the expected delineation between frequency of responses to the less severe questions vs. the most severe questions. Similar patterns are found in the baseline data. Additionally, there is a lower-than-expected correlation between the FIES data and the similar food￾related coping strategy questions found in the resilience module. The two indicators use different recall periods (30 days for the FIES, 7 days for the food security coping), which may indicate that things like missing meals may not happen every week, they may be more likely to occur within the time period of a month. Additional figures showing the relationship between the individual FIES and food security coping questions are found in Annex C. No associations between the FIES and resilience indices were observed in the Round 2 data, likely due to low variation in the FIES variable. Contrary to the worsening food security as measured by the FIES, the household dietary diversity score (HDDS) showed a small but significant increase from 3.6 to 3.9 food groups in the overlapping areas. Table 17. Baseline and interim food security indicator comparisons (overlapping areas only, unweighted) Indicator Indicator value Significance level Number of observations 2017 Baseline 2021 Interim BL Int 1. Average Household Dietary Diversity Score (HDDS) 3.6 3.9 * 1,258 611 2. Prevalence of moderate or severe food insecurity based on 30 day recall (FIES) 87.0% 96.9% *** 1,246 619 Note: * p < 0.05, *** p < 0.001 Looking at the full sample from the interim survey, there is some association between the HDDS and the resilience indices, particularly the absorptive and adaptive indices. The transformative capacity index is largely defined by community-level indicators rather than HH-level indicators, and so the associations between that index and the HDDS are less likely to be observed. The graphs below show the relationship between the HDDS and the absorptive and adaptive indices. It should be noted that the sample size at the very high end of the HDDS is small, those means and ranges at the high end of the scale should be disregarded. Interim Evaluation of the Budikadidi RFSA in the DRC (Vol. I) Findings 31 Figure 4. Absorptive and Adaptive Capacity Index by Household Dietary Diversity Score Absorptive Capacity Index Adaptive Capacity Index 4. CONCLUSIONS Overview • Overall, the interim evaluation, as well as the 2019 mid-term evaluation, show the Budikadidi RFSA in the DRC in a positive light. • The interim evaluation shows promising results. Many indicators appear to be moving in the correct direction, and the analysis tends to show that participation in interventions are associated with improvements in many lower-level indicators. • However, more programmatic intensity and focus may be required to positively impact food security and resilience at the population level. o The important question for BHA and CRS is: Considering the cost of implementation of this activity/these interventions relative to the “saturation” that the activity might be expected to reach, is it worth running an intervention that is “a mile wide and an inch deep”? o How can the diversity of projects be reduced, selecting for highest and most sustainable impact? This study only begins to scratch at the surface of this question. Intervention Exposure and Participation • Overall, 52% of HHs in the Round 2 survey reported participation in one or more of the surveyed RFSA interventions. o The 2020 Annual Report data indicate the RFSA was reaching 47,000 unique direct participant HHs, which is roughly 34% of HHs in the coverage area. However, due to registration challenges from the COVID-19 pandemic, this is likely an underestimate. • The interim survey shows that the Budikadidi RFSA achieved some (relatively) higher self-reported HH participation rates of some of their interventions. o These include WASH trainings/events (37%), toilet building (32%), farmers groups (35%), VSLAs (31%), and others. These interventions took place across all (or nearly all) the villages included in the interim survey. • Other interventions had lower self-reported HH participation rates. o These include youth leadership training (5%), adolescent live skills training (6%), alternative livelihood interventions (8%), and others, despite also taking place in all/nearly all the villages surveyed. WASH • The addition of Kasansa to the interim sampling frame13 contributed to an apparent reduction in the relative proportion of Budikadidi RFSA HHs with access to water in the interim survey as compared to baseline. o While 55% of HHs in Cilundu and 60% of HHs in Miabi indicated that they had access to a basic water source, this was true for only 38% of Kasansa HHs. o Similarly, 49% of Kasansa HHs required over 30 minutes to obtain water, compared with 29% and 27% of HHs in Cilundu and Miabi, respectively. 13 Only 27% of the sample (about 335 HHs) overlap with the baseline sample. As such, comparisons between baseline and interim are difficult to make. IMPEL | Implementer-Led Evaluation and Learning 32 Interim Evaluation of the Budikadidi RFSA in the DRC (Vol. I) Conclusions 33 • Drinking water quality and access showed neutral or positive changes in the overlapping areas. Household with improved drinking water sources increased significantly, from 47% to 57%. The percentage of HHs that can access drinking water in less than 30 minutes increased from 56% to 72%. Use of recommended water treatment technologies remained low, however. o The 2019 mid-term evaluation highlighted that the demand for potable water in the intervention areas was large, to the point of straining the system. However, the RFSA continues to make strides to expand and reinforce the system to meet that demand, which promises further improvements during the extension period of the activity. • Improvements in sanitation are best observed in the decrease of the percentage of HHs using open defecation, which decreased in the overlapping areas from 35% to 18% of HHs. Households with an adequate handwashing station only saw very small improvements, however. • Participation in WASH trainings/events and toilet building interventions are significantly associated with improvements in some WASH outcomes. o Participants in WASH trainings/events reported a significantly higher prevalence of handwashing (7% vs. 2%) and correct water treatment (7% vs. 4%). WASH trainings/events took place in all but one of the surveyed villages, and 31% of HHs reported participation in these trainings. o Households that participated in toilet building interventions reported significantly lower rates of open defecation (17% vs. 11% of non-participants). Toilet building interventions were implemented in villages representing 96% of HHs, with 27% of HHs at interim reporting participation. Agriculture • Changes in agriculture indicators in the overlapping areas were of mixed results. • One important, large improvement is seen in farmers practicing value-chain activities (19% to 42%). Positive impacts of the activity on agricultural market linkages are also highlighted in the 2020 Annual Report. • The aggregate indicators measuring the use of sustainable crop and livestock practices showed negative changes in the overlapping areas between rounds. However, there appears to be a bias in the baseline and/or interim data that may be masking positive change. Some indications of positive change are seen when looking deeper into the data. o The percentage of HHs practicing at least one type of sustainable crop practice increased from 62% at baseline to 82% of farmers at interim. The percentage of farmers with livestock practicing at least one sustainable livestock practice increased from 39% at baseline to 45% at interim. o There were significant increases in some of the crop practices promoted by the RFSA, including the use of manure, which increased from 11% of farmers at baseline to 33% at interim, and compost, which increased from 12% of farmers at baseline to 24% of farmers at interim. Other promoted crop practices remained relatively unchanged, with the exception of weed control (which decreased from 31% to 15% of farmers). o Looking at the individual livestock practices, the use remained relatively unchanged, with only some small fluctuations. IMPEL | Implementer-Led Evaluation and Learning 34 Conclusions • Household participation in certain interventions was associated with better agriculture outcome indicators in the interim survey. o The 32% of farmers living in HHs reporting VSLA participation had significantly higher rates of financial services use. o Farmers participating in agricultural input interventions (15% of farmers) were more likely to use value chain activities (41% compared to 35% among non-participants). Women’s Health and Nutrition • Indicators of women’s diets experienced some decrease in the overlapping areas between baseline and interim surveys. Consumption by women of a MAD decreased from 20% to 16%, though this was not significant. Consumption of targeted nutrient-rich commodities did experience a large drop, however, from 66% at baseline to 30% at interim. o The drop in nutrient-rich commodities was primarily driven by a reduction in the consumption of bio-fortified foods, which fell from 37% at baseline to only 10% at interim. This may be due to changes in other activities implemented in the area. • Participation in nutrition trainings is associated with better dietary diversity. o Among the 16% of women who participated in nutrition trainings, the prevalence of consumption of a minimum dietary diversity was 21%, compared to only 8% among women that did not participate. • The use of contraception among women of reproductive age that were married or partnered increased significantly in the overlapping areas, from 9% at baseline to 16% at interim. This increase was largely driven by an increase in traditional contraceptive use (vs. modern). o Self-reported participation in mother’s groups was associated with an increased use of contraception. However, the care group promoted by the RFSA did not organize any sessions to promote any form of contraception. This association may be related to participation mother’s groups organized by other implementers, or it may be that women who chose to participate in mother’s groups are more likely to use contraception. Children’s Health and Nutrition • There was a significant decrease in the percentage of children under 5 years who had diarrhea in the last 2 weeks in the overlapping areas (44% to 35%). • Positive changes were also seen in the use of ORT to treat diarrhea in children under 5 years of age, and exclusive breastfeeding of children under 6 months. However, the changes were not sufficient for the small sample sizes to find significant. • The prevalence of children 6–23 months receiving a minimum acceptable diet, and the prevalence of children 6–23 months who consume targeted commodities did not experience a significant change. o The lack of important changes in diet quality/diversity is also seen among women of reproductive age, and at the HH level. • Participation in Mothers’ groups, nutrition training, and home health visits were not associated with better outcomes in exclusive breastfeeding, use of ORT, children’s MAD, or consumption of nutrient-rich commodities. Interim Evaluation of the Budikadidi RFSA in the DRC (Vol. I) Conclusions 35 o The 2018 mid-term evaluation highlighted that many of these activities demand highly effective Lead Mothers in order to have success. It may be that the COVID-19 restrictions limited the ability of the Lead Mothers to adequately conduct their work. Gender • Significant improvements from baseline to interim were observed in the overlapping areas in a many of the gender indicators, including: o A significant increase in the percentage of adult women earning cash in the previous year (27% to 35%). Among men, this remained unchanged, indicating that new cash-earning opportunities are being made available to women. o A significant increase in knowledge of MCHN practices (for both men and women). o There was a significant shift in men making maternal or child health decisions alone to making them jointly with their spouse/partner. o Men in unions showed a shift from making decisions about their own self-earned cash alone, to making these decisions jointly with their wife/partner. • However, other decision-making indicators did not show obvious positive change, particularly among women. o The combined percentage of women reporting making decisions about their own self￾earned cash alone plus those making them jointly decreased slightly between survey rounds. This indicates that women did not appear to gain decision-making power related to their cash earnings. o The combined percentage reporting making maternal health or child health decisions alone plus those making them jointly did not see large changes between survey rounds. This indicates that women did not appear to gain decision-making power over their own or their children’s health between rounds. o The percentage of men in the overlapping areas saying that it is ok for a man to batter his wife for any reason saw a moderate increase, from 62% to 71%. o There was no change in the overlapping areas in the percentage of women reporting participation in community decision-making bodies. o As emphasized in the 2018 mid-term evaluation, gender norms in Kasaï are deeply entrenched and will take a long time to change. Food Security, Shock Exposure, and Resilience • All three resilience capacity indices (absorptive, adaptive, and transformative) increased significantly in the overlap areas between baseline and interim. However, changes in the indices were largely driven by only one (or a few) of their component indicators: o The large increase of the absorptive capacity index was driven by an increase in reported prevalence of humanitarian assistance. Rapid response activities conducted by other actors in some of the areas of implementation may have led to this change. o The small increase in the adaptive capacity index was driven mainly by an increase in social safety nets, which was offset by a reduction in livelihood diversity (mainly from a reduction in HHs reporting remittances/gifts). IMPEL | Implementer-Led Evaluation and Learning 36 Conclusions o The modest increase in the transformative capacity index was driven mainly but improvements reported in formal safety nets, but was offset by small decreases in some of the other component indicators. • Food security, as measured by the Household Dietary Diversity Score (HDDS), showed only a small improvement between survey rounds in the overlapping areas, from an average of 3.6 food groups at baseline, to 3.9 at interim. o Considering the volatile food security in the DRC, compounded in the year prior to the survey by the COVID-19 pandemic, stable household dietary diversity could be interpreted as a positive outcome. • Moderate and severe food insecurity, as measured by the Food Insecurity Experience Scale (FIES), increased significantly between baseline and interim in the overlapping areas (87% vs. 97%). o The FIES data at baseline and interim appear to have limitations in adequately describing food security in the populations surveyed. • There is a positive correlation between the HDDS and higher absorptive and adaptive capacity indices. o This does not hold true for the transformative capacity index, which is defined mainly but community-level indicators rather than household. Methodology This evaluation had some methodological challenges that should be taken into consideration in future evaluations. • Pre/post PBS design has some important limitations. o Modifications in areas of implementation after the baseline PBS is common across RFSAs. The pre/post PBS methodology may not be well suited to adapt to changing areas of implementation. o Sampling frame data in DRC often has large inaccuracies. This results in highly variable probability/population weights, and a loss of statistical power. Alternative PBS sampling strategies should be considered that would have less loss of statistical power with similar sample sizes and budgets. o Population-level changes in many of the low-level indicators should not realistically be expected to change with the given intensity of certain interventions. o There is a desire to have evaluation data that can show impact of interventions on the various outcomes. PBS do not readily allow this level of analysis. • Some key outcome indicators may not adequately preform in the context. o The utility of certain food security indicators (such as the FIES) should be re-evaluated, including an assessment of their functionality in specific locations/contexts. o The resilience capacity indices are less useful as composite indicators. The sub-components of the indices tend to reveal more useful information. For example, three of the adaptive capacity component indicators are related to agriculture, so HHs that did not engage in agriculture tend to have lower scores on this index even if they are highly resilient. This indicates that the adaptive capacity index in the aggregate may be of limited use for HHs not engaged in agriculture. It may benefit from adaptation to reflect adaptability as a function of livelihood.