Performance Evaluation of the Tuendelee Pamoja II Resilience Food Security Activity in the Democratic Republic of the Congo May 2022 | Volume I IMPEL | Implementer-Led Evaluation & Learning Associate Award ABOUT IMPEL The Implementer-Led Evaluation & Learning Associate Award (IMPEL) works to improve the design and implementation of Bureau for Humanitarian Assistance (BHA)-funded resilience food security activities (RFSAs) through implementer-led evaluations and knowledge sharing. Funded by the USAID Bureau for Humanitarian Assistance (BHA), IMPEL will gather information and knowledge in order to measure performance of RFSAs, strengthen accountability, and improve guidance and policy. This information will help the food security community of practice and USAID to design projects and modify existing projects in ways that bolster performance, efficiency, and effectiveness. IMPEL is a seven-year activity (2019–2026) implemented by Save the Children (lead), TANGO International, Tulane University, Causal Design, Innovations for Poverty Action, and International Food Policy Research Institute. RECOMMENDED CITATION IMPEL. (2022). Performance Evaluation of the Tuendelee Pamoja II Resilience Food Security Activity in the Democratic Republic of the Congo (Vol. I). Washington, DC: The Implementer-Led Evaluation & Learning Associate Award. PHOTO CREDITS Felly Muambayi, Kinshasa School of Public Health DISCLAIMER This report is made possible by the generous support of the American people through the United States Agency for International Development (USAID). The contents are the responsibility of the Implementer-Led Evaluation & Learning (IMPEL) award and do not necessarily reflect the views of USAID or the United States Government. CONTACT INFORMATION IDEAL Activity c/o Save the Children 899 North Capitol Street NE, Suite #900 Washington, DC 20002 www.fsnnetwork.org IMPEL@savechildren.org PREPARED BY: Performance Evaluation of the Tuendelee Pamoja II RFSA in the DRC (Vol. I) Acknowledgments i ACKNOWLEDGMENTS USAID/BHA: Staff in Washington, DC and at the DRC mission Kinshasa School of Public Health: Pierre Akilimali, Prescillia Visi, John Lhudal, Charles Kasongo, Dynah Kayembe, Steve Mbikayi, Richard Mubikayi, Guy Ngindu, Marc Mchanga Useni, Maurice Kabangu, and Michel Kayembe IMPEL: Tom Spangler, Mohamoud Ahmed, and Corena Sharp Tulane University: Nancy Mock, Peter Horjus, Michelle Lacey, Sarah Truxillo, and Janna Wisniewski Trestle Research and Consulting: Maulik Chauhan and Vikas Arora Independent survey monitors: Elvis Cidoro (supervisor), Innocent Nshombo (South Kivu), Samuel Kataya (Tanganyika), and Roussel Kabeya (Kasai Oriental) IMPEL | Implementer-Led Evaluation and Learning ii Table of Contents TABLE OF CONTENTS Acknowledgments............................................................................................... i List of Tables......................................................................................................iii List of Figures.....................................................................................................iii Acronyms...........................................................................................................iv Executive Summary ............................................................................................ v 1. Introduction................................................................................................... 1 1.1 Background and Setting ................................................................................................................. 1 1.2 Resilience Food Security Activity Goals and Activities................................................................... 2 1.3 Evaluation Purpose and Objectives................................................................................................ 3 1.4 Study Team and Partners............................................................................................................... 4 2. Quantitative Evaluation Methods.................................................................. 5 2.1 Overview ........................................................................................................................................ 5 2.2 Sample Design................................................................................................................................ 5 2.3 Sample Size..................................................................................................................................... 6 2.4 Sample Selection............................................................................................................................ 8 2.5 Survey Tools................................................................................................................................... 8 2.6 Fieldwork and Data Collection ....................................................................................................... 8 2.7 Data Analysis................................................................................................................................10 2.8 Final Sample and Sample Weights...............................................................................................10 2.9 Integration of Secondary Data .....................................................................................................11 2.10 Limitations and Delimitations ......................................................................................................11 3. Findings .......................................................................................................13 3.1 Activity Intervention Exposure and Participation ........................................................................13 3.2 Water, Sanitation, and Hygiene ...................................................................................................15 3.3 Agriculture....................................................................................................................................16 3.4 Women’s Health and Nutrition....................................................................................................19 3.5 Children’s Health and Nutrition ...................................................................................................21 3.6 Gender..........................................................................................................................................23 3.7 Activity-Specific ............................................................................................................................24 3.8 Shock Exposure and Resilience Indices........................................................................................25 3.9 Food Security................................................................................................................................28 4. Conclusions..................................................................................................31 Performance Evaluation of the Tuendelee Pamoja II RFSA in the DRC (Vol. I) List of Tables and Figures iii VOLUME II Annex A: Protocol for the Quantitative Evaluations of three Resilience Food Security Activities in the Democratic Republic of the Congo Annex B: Survey Questionnaire Annex C: Additional Tables and Figures Annex D: Descriptive Analysis Tables Annex E: Indicator Change Tables LIST OF TABLES Table 1. Round 2 performance evaluation population-based survey sampling frame ................................ 6 Table 2. Population-based survey sampled villages/clusters and sample size............................................. 7 Table 3. Final sample size and response rates............................................................................................11 Table 4. Baseline (BL) and Round 2 (R2) water, sanitation, and hygiene indicator comparisons..............15 Table 5. Baseline and Round 2 agriculture indicator comparisons.............................................................16 Table 6. Changes in prevalence of sustainable crop practices between baseline (2017) and Round 2 (2021)..........................................................................................................................................................18 Table 7. Changes in prevalence of livestock practices and technologies between baseline (2017) and Round 2 (2021)............................................................................................................................................19 Table 8. Baseline and Round 2 women’s health and nutrition indicator comparisons..............................20 Table 9. Baseline and Round 2 children’s health and nutrition indicator comparisons.............................21 Table 10. Baseline and Round 2 children’s health and nutrition indicator comparisons...........................23 Table 11. Baseline and Round 2 activity-specific indicator comparisons...................................................25 Table 12. Baseline and Round 2 shock exposure and resilience indicator comparisons............................26 Table 13. Absorptive Capacity Index: Changes in component indicators...................................................26 Table 14. Adaptive Capacity Index: Changes in component Indicators......................................................27 Table 15. Transformative Capacity Index: Changes in component indicators............................................27 Table 16. Baseline and Round 2 food security indicator comparisons.......................................................28 LIST OF FIGURES Figure 1. TPII coverage area map.................................................................................................................. 2 Figure 2. Household exposure and participation in RFSA interventions (Round 2 data) ...........................14 Figure 3. Resilience capacity indices by Household Dietary Diversity Score (Round 2 data) .....................29 IMPEL | Implementer-Led Evaluation and Learning iv Acronyms ACRONYMS ANC Antenatal Care BHA Bureau for Humanitarian Assistance BL Baseline CMC Community Marketing Center DHS Demographic and Health Survey DRC Democratic Republic of the Congo ET Evaluation Team FANTA Food and Nutrition Technical Assistance III Project FIES Food Insecurity Experience Scale FY Fiscal Year HDDS Household Dietary Diversity Score HH Household IMPEL Implementer-Led Evaluation & Learning Associate Award IP Implementing Partner IPC Integrated Food Security Phase Classification KSPH Kinshasa School of Public Health MAD Minimum Acceptable Diet MCHN Maternal and Child Health and Nutrition MDD Minimum Dietary Diversity MYAP Multi-Year Assistance Project NGO Non-Governmental Organization NRM Natural Resource Management ODK Open Data Kit ORT Oral Rehydration Therapy PBS Population-Based Survey R&I Refine and Implement RFSA Resilience Food Security Activity SD Standard Deviation TPI Tuendelee Pamoja I TPII Tuendelee Pamoja II USAID United States Agency for International Development VC Value Chain VSLA Village Savings and Loan Association WASH Water, Sanitation, and Hygiene Performance Evaluation of the Tuendelee Pamoja II RFSA in the DRC (Vol. I) Executive Summary v EXECUTIVE SUMMARY Overview To assist vulnerable populations of the Eastern Democratic Republic of the Congo (DRC), the United States Agency for International Development’s (USAID) Bureau for Humanitarian Assistance (BHA) awarded Food for the Hungry a 5-year Resilience Food Security Activity (RFSA), Tuendelee Pamoja II (TPII), 1 to be implemented from October 2016–September 2021. TPII was designed to reach 214,000 households (HHs) in the two territories of Tanganyika (Kalemie territory: Kalemie and Nyemba health zones and in Moba territory: Moba and Kansimba health zones) and one territory in South Kivu (Walungu territory: Walungu, Mubumbano, Kaziba health zones), with interventions in agriculture, health, nutrition, water and sanitation, literacy, and conflict transformation. The central goal of TPII was “All members of households from all tribes in South Kivu and Tanganyika provinces of the DRC live with social and economic well-being.” In 2021, under the USAID BHA Implementer-Led Evaluation and Learning (IMPEL) Associate Award with Save the Children, Tulane University School of Public Health and Tropical Medicine (Tulane) was subcontracted to conduct a performance evaluation, including a population-based survey (PBS) and a resilience assessment of TPII. Study Purpose and Design The PBS serves as the second phase of a pre-post survey cycle, with data on the same indicators collected in both survey rounds. Statistically detecting changes (if any) for all practice and behavioral change indicators at the population level (the TPII coverage area) can help inform the performance of the award. It should be noted that the evaluation does not include data from areas in which the intervention was not implemented (i.e., a counterfactual). While the evaluation results can be used to help explain differences between the pre- and post-values and may therefore help paint a general picture about overall performance, true attribution is only possible with experimental and quasi￾experimental evaluation designs, which are not being used in this evaluation. This PBS collected representative data on a number of lower-level outcomes from 1,231 HHs and/or individuals in July/August 2021. Key Findings Overall Assessment Beyond the impacts felt across the DRC and globally from the COVID-19 pandemic, TPII was also challenged by interventions in conflict zones, which experienced an exodus in 2016–18, and then in 2018 and 2019, received a large wave of arriving internally displaced persons. Furthermore, most of the TPII’s high-impact activities (such as irrigation, drainage, Community Marketing Centers (CMCs), and rehabilitation of farm-to-market roads) were only completed in 2021, and so their impacts may not have been fully realized at the time of the evaluation data collection. The evaluation shows some promising results. Certain indicators appear to be moving in the correct direction, and the analysis tends to show that direct participation in some interventions is associated 1 This means “moving forward together” in Swahili. IMPEL | Implementer-Led Evaluation and Learning vi Executive Summary with improvements in lower-level indicators. However, results at the population level do not show consistent, significant improvement. The TPII RFSA achieved wide coverage of its interventions, but this did not result in high levels of reported HH participation. Additionally, spillover of the impacts from the RFSA interventions to indirect participants may be less than hoped. As such, impacts may be diluted at the population level. The small number of participants made it difficult to assess the associations between intervention participation and differences in the outcome indicators. While some associations were made between participation and improvements in outcomes, HH participation was generally too low to be expected to have any impact at the population level. It is important for BHA and Food for the Hungry to consider, in general, the cost of implementation of an activity and its interventions relative to the saturation that the activity may be expected to reach, in order to determine if it is worth running an intervention that is “a mile wide and an inch deep.” The diversity of projects may need to be reduced, selecting for the highest and most sustainable impact. This study only begins to scratch at the surface of these issues. Intervention Exposure and Participation The Round 2 survey shows that self-reported HH participation was (relatively) low for most interventions, even though most were implemented in all the villages included in the Round 2 survey. Overall, 34% of HHs in the Round 2 survey reported participation in one or more of the surveyed RFSA interventions. The highest reported prevalence of HH participation include agriculture trainings (16% of HHs), WASH trainings (14% of HHs), farmers groups (13% of HHs), and activities focused on improved agricultural production (13%). Others had very low coverage. For example, youth-related interventions had very low coverage, despite being implemented in all surveyed villages. Youth leadership training participation was reported by 2% of HHs, adolescent life skill trainings by 3%. The TPII mid-term evaluation conducted in 2019 found similar concerns. It highlighted that TPII had undertaken too many interventions and was behind on implementing several activities at that time. It also noted that there was minimal direct targeting of youth for health and nutrition-related messages and indicated they were reaching only a very small segment of the adolescent population. Food Security, Shocks, and Resilience Food security, as measured by the Household Dietary Diversity Score (HDDS) and the Food Insecurity Experience Scale (FIES), showed no significant change. The prevalence of moderate and severe food insecurity was very high at baseline (93%) and at Round 2 (90%). Considering the consistently evolving food security situation in the DRC, potentially compounded in the year prior to the survey by the COVID￾19 pandemic, stable HH dietary diversity could be interpreted as a positive outcome. All three resilience capacity indices (absorptive, adaptive, and transformative) increased significantly between survey rounds. However, changes in the indices were largely driven by only one (or a few) of their component indicators. Performance Evaluation of the Tuendelee Pamoja II RFSA in the DRC (Vol. I) Executive Summary vii The increase in absorptive capacity was primarily due to a significantly increased presence of humanitarian assistance. 2 This increase in reported humanitarian assistance may be due to additional humanitarian and development assistance, which may include certain types of services or other assistance provided through TPII, although it is important to note that this cannot be confirmed by available data. There was also a small but significant improvement in the availability of informal safety nets, cash savings, and productive assets, offset by a sizable decline in access to remittances. The increase in the absorptive capacity index was primarily due to a significantly increased presence of the humanitarian assistance component of the index, as well as smaller improvements in the availability of informal safety nets, cash savings, and productive assets. The improvement was offset by a sizable decline in access to remittances. The adaptive capacity index increased significantly, but the change was small. The improvement was driven by positive changes in social safety nets and productive assets, but lessened by decreases in education/training, livelihood diversity (primarily due to the reduction of “livelihood sources” such as remittances and gifts), and improved agricultural practices (all sub-components of the index). Transformative capacity experienced a small but significant increase. This improvement was driven by improvements in formal safety nets, access to natural resources, and collective action (all sub￾components of the index). There is a positive association between the HDDS and the resilience indices in the Round 2 data, particularly the absorptive and adaptive indices. The transformative capacity index is largely defined by community-level indicators rather than household-level indicators, and so the associations between that index and the HDDS are less likely to be observed. Water, Sanitation, and Hygiene Drinking water access improved significantly between survey rounds. The percentage of HHs that can obtain drinking water in under 30 minutes increased from 57% to 78%. Additionally, access to improved drinking water improved from 38% at baseline to 51% at Round 2, although the change was not significant. However, no significant changes in the use of water treatment technologies were observed. Although the use of basic (improved) sanitation did not change significantly between surveys, 3 the percentage of HHs practicing open defecation decreased from 8% to 3%. Toilet-building interventions were only implemented in villages representing 6% of HHs and did not show a significant relationship to sanitation. However, HHs that had participated in WASH training interventions had significantly higher use of basic (improved) sanitation facilities (15%) than non-participation HHs (5%). However, WASH training participation was reported by only 13% of HHs. 2 This variable is defined as the respondent indicating that government or non-governmental organization (NGO) emergency food or cash assistance is available in the respondent’s village OR the household reported receiving emergency food or cash assistance from the government or NGO during the 12 months prior to the survey. 3 It’s difficult to assess the change in toilet type compared to baseline. The baseline indicates that 82% of households had a “water flush to pit latrine” type toilet, which is likely a mistake in data collection (although still classified as unsafe/unimproved sanitation). The Round 2 survey reports that <1% of households have flush-to-pit-latrine toilets- similar to the most recent DHS survey, which indicates that 0.5% of urban and 0% of rural have flush-to-pit-latrine toilets. IMPEL | Implementer-Led Evaluation and Learning viii Executive Summary Agriculture Among the agriculture indicators, only the percentage of farmers using improved storage practices increased from 27% to 38% between rounds. No significant change was seen in the percentage of farmers using financial services, nor in the practice of value chain activities promoted by the activity. Significant decreases in the use of sustainable crop practices and sustainable livestock practices were observed. However, it is challenging to assess whether these negative changes between rounds are methodology/collection issues or if they accurately reflect changes. There was little to no association between HH participation in any of the surveyed interventions and the agricultural outcomes measured. Village Savings and Loan Association (VSLA) participation was the one exception, associated with a significantly higher rate of access to financial services. However, only 7% of HHs reported participating in VSLAs despite 100% of the villages surveyed having had VSLA interventions conducted. Women’s Health and Nutrition A small but significant improvement was observed in women’s consumption of a diet of minimum diversity (18% at baseline to 20% at Round 2). However, women’s consumption of targeted nutrient-rich commodities did not change significantly, and only small changes were seen in the various specific commodities assessed. No significant change was observed in contraceptive use. Household participation in nutrition training was associated with a better Minimum Dietary Diversity (MDD) for women from (30% among participants vs. 18% among non-participants). However, only 7% of HHs reported participating in this type of training. The percentage of women who reported contraceptive use was also positively associated with participation in mothers’ groups for the small numbers of HHs who engaged in these activities. Children’s Health and Nutrition Significant improvements were observed in diarrhea prevalence (23% at baseline to 17% at Round 2) and in the prevalence of exclusive breastfeeding of children under 6 months (44% to 72%). However, no significant change was seen in oral rehydration therapy (ORT) treatment, children consuming a minimum acceptable diet, or children consuming targeted nutrient-rich foods. Although mothers’ groups, nutrition training, and home health visits were implemented in all communities surveyed at Round 2, the participation rates were low. Only about 7% of HHs reported participation in each of these interventions. Among the few (26) children in the surveyed households who benefited from HH participation in nutrition training, the prevalence of a minimum acceptable diet was 23%, compared to 7% for the non-participating. Gender Little change was noted across the gender indicators. Gender norms are deeply entrenched and will likely take years or even decades to change. The percentage of women who earned cash in the previous year decreased significantly, from 36% to 20% at Round 2. Other indicators showed no significant change, including the percentage of men and women who say it’s ok for a man to batter his wife for any reason, the percentage of women participating in decision-making bodies, and the percentage of men or women that have knowledge of Performance Evaluation of the Tuendelee Pamoja II RFSA in the DRC (Vol. I) Executive Summary ix maternal and child health practices. However, HHs receiving home visits (8% of all HHs) were significantly associated with better knowledge of Maternal and child health and nutrition (MCHN) practices. Methodological Challenges This evaluation had some methodological challenges that should be taken into consideration in future evaluations, revolving primarily around the limitations of the pre/post-PBS design, as well as some key outcome indicators that may not adequately perform in the context. Modifications in areas of implementation after the baseline PBS is common across RFSAs. The pre/post￾PBS methodology may not be well suited to adapt to changing areas of implementation. Sampling frame data in the DRC often has large inaccuracies. This results in highly variable probability/population weights and a loss of statistical power. Alternative PBS sampling strategies should be considered that would have less loss of statistical power with similar sample sizes and budgets. Population-level changes in many of the lower-level indicators have a low likelihood of occurring with the given intensity of certain interventions. Furthermore, there is a desire to have evaluation data that can show the impact of interventions on the various outcomes. A PBS does not readily allow this level of analysis. The utility of certain food security indicators (such as the FIES) should be re-evaluated, including an assessment of their functionality in specific locations/contexts. The prevalence of moderate and severe food insecurity (as measured by the FIES) was very high at baseline (95%) and at Round 2 (96%). This homogeneity of the food security status as measured by this indicator renders a more detailed analysis less useful in assessing change. The FIES has other statistical limitations in the populations surveyed. It may not adequately describe the food security situation in the populations surveyed. The resilience capacity indices are less useful as composite indicators. The sub-components of the indices tend to reveal more useful information. For example, three of the adaptive capacity component indicators are related to agriculture, so households that did not engage in agriculture tend to have lower scores on this index even if they are highly resilient. This indicates that the adaptive capacity index in the aggregate may be of limited use for households not engaged in agriculture. It may benefit from adaptation to reflect adaptability as a function of livelihood. This page is left intentionally blank. Performance Evaluation of the Tuendelee Pamoja II RFSA in the DRC (Vol. I) Introduction 1 1. INTRODUCTION 1.1 Background and Setting Eastern Democratic Republic of Congo (DRC) has experienced decades of conflict, poor governance, and, as a result, widespread extreme poverty. The genocide that took place in neighboring Rwanda unfortunately spilled over into eastern DRC, as an easy and convenient place for Hutus to flee to. The region suffered tremendously during the aftermath of that conflict which then evolved into the war in eastern DRC between 1998 and 2007, during which an estimated 5.4 million people died as a result of the conflict and protracted humanitarian crisis. Mortality rates were higher in eastern DRC, demonstrating the effect of insecurity, with most deaths from easily preventable and treatable illnesses rather than violence. Millions of more citizens were pushed into poverty due to displacement and loss of economic livelihoods. By 2012, most of the country was relatively stable, although armed factions have persisted. In 2017, 52 armed groups were active in South Kivu alone, committing thefts, attacks, collecting illegal taxes, and exploiting natural resources (forests and mines). Coupled with the longstanding conflict, the DRC experiences significant political tensions: both the 2011 and 2018 presidential and legislative elections were marred by violence and disputes leading up to the elections and regarding the results across the country. Gender inequalities are prevalent; in 2018 the DRC ranked 147 out of 166 countries on the Gender Development Index. The Demographic and Health Survey (DHS) of 2013–2014 found that 27% of Congolese women aged 15–49 have experienced sexual violence in their lifetime, and more than half had experienced some form of physical violence. The 2013–2014 DHS reported that over 40% of Congolese children suffer from chronic malnutrition indicated by stunting (-2 Standard Deviation (SD) height-for-age), with more than half of these children falling into the severe range (-3 SD height-for-age), signifying a very serious public health problem according to World Health Organization classifications. Eastern DRC is also grappling with the world’s second largest Ebola epidemic on record, with more than 2,000 lives lost and 3,000 confirmed infections since the outbreak was declared on August 1, 2018. Other issues in the region are high rates of alcoholism and difficult land access. The combination of the rapidly growing population, traditional practices related to land inheritance, which causes division of family properties, and land degradation is decreasing the availability of arable land and the size of agricultural fields. A Comprehensive Food Security and Vulnerability Analysis conducted in 2011–2012 by the World Food Programme found 64% of the rural population in South Kivu to be food insecure, with Kalehe ranking third (72.2% of the population food insecure) and Kabare ranking fourth (70.8%) of the 12 South Kivu territories. A recent Integrated Food Security Phase Classification (IPC) analysis from 20214 indicated that in South Kivu, 53% of the population was in acute food insecurity phase 2, 19% in phase 3, and 3% in phase 4 (none in phase 5). To address these challenges, in Fiscal Year 2016, the United States Agency for International Development’s (USAID) Bureau for Humanitarian Assistance (BHA) awarded funding for a multi-year 4 https://www.ipcinfo.org/fileadmin/user_upload/ipcinfo/docs/IPC_DRC_Acute_Food_Insecurity_2021FebDec_Report_ French.Updated.pdf IMPEL | Implementer-Led Evaluation and Learning 2 Introduction Resilience Food Security Activity (RFSA) in the Democratic Republic of Congo, called Tuendelee Pamoja II (TPII). 5 TPII was implemented. TPII was implemented by a consortium led by Food for the Hungry in the two territories of Tanganyika (Kalemie territory: Kalemie and Nyemba health zones; and in Moba territory: Moba and Kansimba health zones) and one territory in South Kivu (Walungu territory: Walungu, Mubumbano, Kaziba health zones). TPII is a follow-on activity to the Development Food Assistance Program Tuendelee Pamoja (TPI), which operated in the same geographic zones between 2011 and 2016. Prior to TPI, Food for the Hungry implemented the Multi-Year Assistance Project (MYAP) in Kalemie and Moba between 2008 and 2011. 6 Figure 1. TPII coverage area map 1.2 Resilience Food Security Activity Goals and Activities To assist vulnerable populations of eastern DRC given the challenges described above, BHA awarded Food for the Hungry a 5-year RFSA, Tuendelee Pamoja II, 7 to be implemented from October 2016– September 2021. TPII was designed to reach 214,000 households (HHs) in the two territories of Tanganyika (Kalemie territory: Kalemie and Nyemba health zones and in Moba territory: Moba and Kansimba health zones) and one territory in South Kivu (Walungu territory: Walungu, Mubumbano, Kaziba health zones) to directly benefit up to 1,427,487 individuals through interventions in agriculture, 5 This means “moving forward together” in Swahili. 6 FY 2019 Pipeline and Resource Estimate Proposal, resubmission version March 14, 2019. 7 The activity was previously awarded as a Food for Peace Development Food Security Activity. The terminology for these awards changed in 2020. Performance Evaluation of the Tuendelee Pamoja II RFSA in the DRC (Vol. I) Introduction 3 health, nutrition, water and sanitation, literacy, and conflict transformation. TPII had a budget of $71 million for the 5 years and was a follow-on activity to the Development Food Assistance Program TPI, which operated in the same geographic zones between 2011 and 2016. Prior to TPI, Food for the Hungry implemented the Multi-Year Assistance Project in Kalemie and Moba between 2008 and 2011.8 TPII was implemented by Food for the Hungry as the prime and consortium lead, with Search for Common Ground as the partner responsible for conflict transformation; CGIAR (International Institute of Tropical Agriculture, International Livestock Research Institute, World Agroforestry) and Institut de l'Environnement et de Recherches Agricoles are introducing climate smart agriculture, livestock health and agroforestry techniques; Inspection Provinciale l’Agriculture, Pêche et Élevage is leading livestock training; National Seed Service provided seed certification technical support; HarvestPlus supplied bio￾fortified seeds; TearFund International was the partner responsible for water and sanitation; Union for the Emancipation of the Indigenous Woman was the partner responsible for conflict mediation between Twa and Bantu; and LIPEDEM was responsible for literacy classes. Tillers International introduced ox￾powered agriculture and Johns Hopkins University conducted the BabyWASH research study. The central goal of TPII was “All members of households from all tribes in South Kivu and Tanganyika provinces of the DRC live with social and economic well-being.” The activity purposes supporting this goal were: • Purpose 1: Households Have Food and Income Security. • Purpose 2: Improved Nutrition and Health Status of Women of Reproductive Age, Pregnant and Lactating Women, Adolescent Girls, and Children under 5 Years Old. • Purpose 3: Women, Men and Youth of All Tribes are Social Equals and Feel Safe in their Homes and Communities. o Reduce conflict for community cohesion and development. o Reduce incidence of Sexual Gender Based Violence for community cohesion and development. This RFSA relied on staff health and agriculture promoters and their supervisors who lived in local communities. The supervisor and promoters worked directly with the many volunteers who were the backbone of TPII. It is important to note that most of the TPII RFSA’s high-impact activities (such as irrigation, drainage, CMCs, and rehabilitation of farm-to-market roads) were only completed in 2021, and so their impacts may not have been fully realized at the time of the second round of survey data collection. 1.3 Evaluation Purpose and Objectives In 2021, under the USAID BHA Implementer-Led Evaluation and Learning (IMPEL) Associate Award with Save the Children, Tulane University School of Public Health and Tropical Medicine (Tulane) was subcontracted to conduct a performance evaluation, including a population-based survey (PBS) and a resilience assessment, in the TPII RFSA area. 8 FY 2019 Pipeline and Resource Estimate Proposal, resubmission version March 14, 2019. IMPEL | Implementer-Led Evaluation and Learning 4 Introduction As part of the overall Refine and Implement (R&I) approach, the performance evaluation was originally planned to take place in 2020 so that the findings could help inform decisions on activity extensions. However, the evaluation was delayed until 2021 because of the COVID-19 pandemic. The design of the evaluation to track lower-level outcome indicators remained the same. The PBS serves as the second phase of a pre-post survey cycle, with data on the same indicators collected in both survey waves. This pre-post design allows for the determination of statistically significant change in indicators between baseline (Round 1) and the performance evaluation (Round 2) data. In addition, statistically detecting changes (if any) for all practice and behavioral change indicators can help inform the performance of each award to date. Change over time of indicators in the following categories will be assessed: • Food security • Water, sanitation, and hygiene • Agriculture • Women’s health and nutrition • Children’s health and nutrition • Gender • Resilience It should be noted that the evaluation does not include data from areas in which the intervention was not implemented, (i.e., a counterfactual). While the evaluation results can be used to help explain differences between the pre and post values and may therefore help paint a general picture about overall performance, true attribution is only possible with experimental and quasi-experimental evaluation designs, which are not being used in this evaluation. However, differences in outcomes based on level of exposure to the program will be assessed where feasible. 1.4 Study Team and Partners The baseline HH survey was administered in 2017 by the Mendez England and Associates consortium. Details of that survey administration can be found in the baseline survey report. 9 Tulane University sub￾contracted the Kinshasa School of Public Health (KSPH) to conduct the Round 2 HH survey in 2021. Tulane has a long-standing relationship with KSPH, going back to the founding of the school in 1986. Dr. Pierre Akilimali, an associate professor at KSPH, directed the fieldwork. Dr. Janna Wisniewski, a faculty member from Tulane, traveled to Kinshasa to work with Dr. Akilimali on supervisor training and pilot testing. Dr. Wisniewski also oversaw the independent survey monitors, who were not associated with KSPH; they were supervised by Elvis Cidoro, a Congolese national and graduate of the Tulane School of Public Health and Tropical Medicine. The Tulane team who conducted the analysis consisted of Dr. Nancy Mock, Dr. Michelle Lacey, and Peter Horjus. 9 https://pdf.usaid.gov/pdf_docs/PA00THMS.pdf Performance Evaluation of the Tuendelee Pamoja II RFSA in the DRC (Vol. I) Quantitative Evaluation Methods 5 2. QUANTITATIVE EVALUATION METHODS 2.1 Overview The PBS serves as the second phase of a pre-post survey cycle, with data on the same indicators collected in both survey waves. This pre-post design allows for the determination of statistically significant change in indicators between baseline (Round 1) and the performance evaluation (Round 2) data; however, it does not allow statements to be made about attribution or causation relating to activity impact. As such, the evaluation results can be used to help explain differences between the pre and post values and may therefore help paint a general picture about overall performance. True attribution is only possible with experimental and quasi-experimental evaluation designs, which are not being used in this evaluation. Data was collected in-person through a population-based HH survey. Baseline data collection took place from June–August, and the Round 2 survey was planned to take place during the same months to avoid potential bias from seasonality. The baseline survey used a multi-stage cluster sample design. In the early planning stages of the Round 2 survey, it was planned to re-sample the same clusters (villages) resampled in the Round 2 survey. However, the Round 2 sample needed to be slightly modified to account for changes made to the RFSA coverage areas after the baseline was conducted. Results are disaggregated where the sample allows, to report changes in those areas covered by both the baseline and Round 2 survey, as well as the current estimates of the full sample, representative of the entire current activity coverage area. Additionally, program data on the timing and location of implementation of various components of the activity are used to assess differential impact by level of exposure. Findings from the PBS are triangulated with findings from the mid-term qualitative evaluation to develop insight into the reasons why changes in key outcomes were or where not observed, and to offer recommendations for future activities. The Round 2 PBS was conducted in compliance with both ethical and human rights standards. Survey procedures were designed to protect participants’ privacy, allowing for anonymity and voluntary participation. Ethical approval was obtained from the Tulane Internal Review Board as well as the Kinshasa School of Public Health Comité d'éthique (Ethics Committee) prior to the start of all fieldwork. 2.2 Sample Design The target population for the PBS was all HHs in the current RFSA implementation area. The sampling frame consisted of a list of all villages in which the TPII RFSA implemented their activities, provided by Food for the Hungry. At baseline, Food for the Hungry provided a list of villages where they planned to implement their activities along with the estimated HH counts in each village; villages were grouped by health zones and health areas, and HH counts were obtained from the health zone office. This list was used to draw the baseline sample of villages using probability proportional to size systematic random sampling. The plan during early stages of the Round 2 survey was to re-sample the same villages in the Round 2 survey. IMPEL | Implementer-Led Evaluation and Learning 6 Quantitative Evaluation Methods However, in the second round of quantitative data collection, the evaluation team (ET) conferred with BHA and Food for the Hungry to check whether activities had been implemented in the villages from the baseline sampling frame, and if other areas that were not covered by the baseline sampling frame had been added to the implementation area. In the TPII RFSA implementation area, some of the originally planned coverage areas had been dropped, while others had been added. After the ET consulted with BHA, BHA decided that it would be preferable to adjust the sampling frames to adjust not only for areas that were removed from the coverage area following the baseline, but also to add the areas that were added to the coverage area after the baseline. This decision was based on the expressed need for current population estimates of the entire RFSA coverage area, and worth the slightly lessened degree of comparability between rounds. The data provided by Food for the Hungry for the updated sampling frame of their current implementation area consisted of 715 villages10 where RFSA interventions were being implemented (area of implementation). The total number of villages, households, and population in the TPII RFSA implementation area sampling frame is found in Table 1, below. Table 1. Round 2 performance evaluation population-based survey sampling frame Survey round Villages TOTAL estimated number of households Households added to sampling frame (new coverage areas) Round 2 (2021) 715* 101,916 14,437 households (14% of sampling frame) Round 1 (2017) 824 110,940 * These counts exclude villages with less than 30 households. The Round 2 sampling frame differed from the baseline sampling frame in two ways: • 219 villages in the baseline sampling frame were not covered by the TPII RFSA interventions and were excluded from the Round 2 sampling frame. • 110 villages were added to the coverage area after the baseline was conducted. These are included in the Round 2 sampling frame. 2.3 Sample Size The baseline survey collected data from approximately 1,300 HHs in the implementation area (from 44 villages). The sampling approach/size was designed to detect an eight-percentage point change in stunting prevalence among children under 5 years of age (0–59 months) in the pre-post comparison. This approach was advised by BHA at baseline as standard practice. However, data to assess stunting was not collected in the Round 2 performance evaluation survey. 10 As done in the baseline, the villages with fewer than 30 HHs were removed from the sampling frame (225 villages total for the Round 2 sampling frame, not counted here in the 716 villages cited). These villages accounted for 3.5% of the total estimated number of households in the RFSA coverage area. Performance Evaluation of the Tuendelee Pamoja II RFSA in the DRC (Vol. I) Quantitative Evaluation Methods 7 The indicators listed the evaluation protocol (Annex A) will be used to assess pre-post improvements in the intervention areas. The RFSAs expect “substantial improvements” for each of the indicators, tentatively defined by BHA as an improvement of 25%11 over baseline levels. In early planning discussions with BHA, it was recommended that a maximum sample of approximately 1,500 HHs, with allowance for non-response, would be sufficient to measure changes for many of the desired indicators, and would allow for subsequent disaggregation of indicators by various HH characteristics (gender of household head, for example) and more complex multivariate techniques to allow for a deeper analysis of the data. The initial planning also followed the baseline methodology recommendations of sampling the same clusters (villages) that were in the baseline survey (44 villages per RFSA, or 132 clusters/villages total). However, after the survey planning began, it was found that the RFSA had some degree of change to their coverage areas (villages added and removed from the RFSA coverage areas, as described in the previous section). BHA requested to have a representative sample of the current coverage while also trying to maintain comparability between the baseline and Round 2 data collection. Of the 44 villages sampled in baseline, four were in areas not covered by the RFSA interventions. These four villages were not re-sampled in the Round 2 survey. The remaining 40 villages sampled at baseline were also found in the current Round 2 sampling frame (coverage area) were re-sampled in the Round 2 survey. Additionally, seven villages were sampled from the portion of the sampling frame that was added after the baseline (using random systematic Probability Proportional to Size selection). This resulted in a sample size roughly proportional to the estimated number of HHs in the original and the newly added strata of the sampling frame. This gave a total sample of 47 villages. The number of clusters sampled in Round 2 was larger than originally planned, as was the geographic area being covered, which meant that the field work travel time between villages would exceed that initially planned. To stay within budget, the number of HHs per cluster was decreased slightly from baseline (30 HHs/cluster) to 27 households per cluster. Table 2. Population-based survey sampled villages/clusters and sample size Province/health zone Clusters sampled in BL Re-sampled clusters (R2) Newly sampled clusters (R2) Total clusters sampled (R2) Total HHs to be sampled (R2) South Kivu Province 20 17 3 20 540 Kaziba 3 2 0 2 54 Mubambamo 7 6 2 8 216 Walungu 10 9 1 10 270 Tanganyika Province 24 23 4 27 729 Kalemie 6 6 1 7 189 Kansimba 4 4 0 4 108 Moba 9 8 3 11 297 11 This refers to percentage change (applicable to indicators expressed as totals and as percentages), not difference in percentage points (which would refer only to differences between indicators expressed as percentages). IMPEL | Implementer-Led Evaluation and Learning 8 Quantitative Evaluation Methods Province/health zone Clusters sampled in BL Re-sampled clusters (R2) Newly sampled clusters (R2) Total clusters sampled (R2) Total HHs to be sampled (R2) Nyemba 5 5 0 5 135 TOTAL TPII RFSA 44 40 7 47 1,269 Note: Baseline (BL) and Round 2 (R2). 2.4 Sample Selection The sample selection followed a similar approach as used at baseline. The sample for the RFSA was selected using multi-stage cluster sampling with three stages of sampling: 1) selection of clusters (or villages), 2) selection of households, and 3) selection of individuals. For the purposes of the household survey, a “household” is defined as “a person or group of people who live together and share meals (i.e., eating from the same pot).” Full sample selection procedures are detailed in the evaluation protocol in Annex A. 2.5 Survey Tools The baseline questionnaire was developed through a series of consultations with BHA, the Food and Nutrition Technical Assistance III Project (FANTA), and the implementing partners (IPs). The same modules were used in Round 2, with the exception of the poverty and anthropometry modules. Additionally, questions on household-level participation in RFSA interventions were added to the HH questionnaire. The Round 2 questionnaire consists of modules covering the following topics: • Household identification and informed consent • Household roster • Household food security – Household Dietary Diversity Score (HDDS) and the Food Insecurity Experience Scale (FIES) • Children’s nutrition and health • Women’s nutrition and health • Water, sanitation, and hygiene • Agriculture • Gender – Cash • Gender – Maternal and child health and nutrition • Resilience • Intervention participation Data on village-level exposure to RFSA interventions was collected from the IPs. 2.6 Fieldwork and Data Collection The baseline questionnaires were programmed in Open Data Kit (ODK) by Trestle Research and Consulting. The surveys were pilot tested in a rural area on the outskirts of Kinshasa. Seven experienced Performance Evaluation of the Tuendelee Pamoja II RFSA in the DRC (Vol. I) Quantitative Evaluation Methods 9 supervisors were hired to oversee fieldwork in Round 2. Dr. Akilimali and Dr. Wisniewski led a supervisor training in Kinshasa in June 2021 to orient them to the topics covered by the surveys (nutrition, agricultural practices, resiliency measurement, etc.), and fieldwork activities including sampling procedures, research ethics and informed consent, COVID-19 precautions, and electronic data collection. A representative from one of the IP’s also gave an overview of the program. Tulane and KSPH adapted the supervisor manual, enumerator manual, and question-by-question guide that were used at baseline to foster comparability between survey rounds. The Tulane team customized the manuals only to align with the final questionnaire, the general supervision approach of KSPH, and the protocol for using ODK rather than CSPro, which was the program used at baseline. Supervisors practiced administering the questionnaires to each other, and then a full pilot test was held in a rural area on the outskirts of Kinshasa. The supervisors then traveled to the provinces (three to South Kivu and two to Tanganyika) where they recruited enumerators. All field supervisors and enumerators (including team leaders) were identified from the KSPH roster of experienced survey data collectors. These team members were recruited from the region in which they were working and spoke fluent French as well as the relevant local languages. The data collectors were trained in Bukavu and Mbuji-Mayi. In South Kivu, representatives from the implementing partners attended training and presented the activity overview to the enumerators. Enumerators underwent training similar to that of the supervisors, including practice administering the survey in pairs and a full pilot test. Following training, enumerators deployed in teams of two to selected villages. The “teams” are pairs of enumerators, with one of the enumerators also serving as the “team leader” (but still collecting data). In the KSPH experience, it is less costly to have a team of two stay in a village until all surveys are completed rather than having larger teams working in a single village with more frequent movement. The field teams and field supervisors were divided geographically as follows: Sud Kivu (42 enumerators), and Tanganyika (16 enumerators). To identify the villages selected, the teams used an application called Open Street Map Automated Navigation Directions which was installed on the tablets. GPS points for each village were provided to the field teams. Once in the villages, enumerators counted and selected HHs to participate in the survey, consented respondents, and collected data. In the case that enumerators needed to replace a village due to insecurity or road conditions, Dr. Akilimali and Dr. Wisniewski worked together to select a new village that was geographically near and of a similar population size to the replaced village. Data were transmitted electronically from tablets to a cloud-based server. Data collection occurred in July and August 2021. In each province, an independent survey monitor observed 20 HH surveys to assess the extent to which proper procedures were followed. The monitors used standardized data collection forms, which they transmitted electronically to a survey monitor supervisor based in Kinshasa. Data from these reports were aggregated and shared with Dr. Akilimali on an ongoing basis so that adjustments could be made quickly. The survey monitors did not report any major issues with data collection. Additionally, Dr. Akilimali ran quality checks on incoming data, including age pyramids and completion rates by enumerator. IMPEL | Implementer-Led Evaluation and Learning 10 Quantitative Evaluation Methods 2.7 Data Analysis Tulane generated estimates for all BHA and activity-specific indicators, along with additional analyses to explore relationships and plausible determinants for key outcome indicators and a select number of resilience indicators. The syntax/do-files from the baseline were used to ensure all indicators were calculated in the same way. Data analysis was conducted in STATA, SPSS, and R. Unless otherwise noted, analysis takes into account the populating weights and sampling design to generate estimates and conduct statistical tests. Annex C has information on the key indicator results, including the confidence intervals, design effects, record numbers, standard errors, and weighted population. 2.8 Final Sample and Sample Weights Sampling weights were computed and used in the data analyses, following the same general approach as the baseline. Weights were computed according to the unique sampling scheme that is relevant to the associated sampled household or individual. This involved computing an overall HH sampling weight for each distinct sampling group by taking the inverse of the product of the probabilities of selection from each stage of sampling (village selection and household selection), also accounting for the correction of sampling probabilities from each sampled villages (clusters) estimated number of HHs (from the sampling frames), and the actual number of HHs in the village (collected in the listing exercise during data collection). Weights were calculated for the following distinct sampling groups: • Households (used for indicators derived from Modules B, C, F, and R) • Children under 5 years of age (Module D) • Women 15–49 years of age (Module E) • Farmers (Module G) • Cash-earning adults in a union (Module J) • Parents of children under 2 years of age (Module K) Household level-weights were calculated to adjust for non-response (where HHs declined to be interviewed). In each of the individual level sampling groups, weights were calculated to adjust for non￾response based on the number of eligible individuals listed on the HH roster, and the number of individuals interviewed. The non-response then accounts for individuals that declined to be interviewed (rare), and individuals that were listed in the HH roster were not located despite the repeat visits to the HH. The household and individual-level non-response are shown in Table 3. Performance Evaluation of the Tuendelee Pamoja II RFSA in the DRC (Vol. I) Quantitative Evaluation Methods 11 Table 3. Final sample size and response rates Sampling group Number sampled Number interviewed Response rate Households* 1,269 1,261 99.4% Children 0–59 months** 1,135 1,114 98.1% Women 15–49 years** 1,417 1,314 92.7% Farmers** 1,336 1,194 89.4% Cash earning adults in a union** 791 707 89.4% Parents of children under 2 years** 840 715 85.1% * For households, the sampled vs. interviewed reflects refusals to participate in the survey. ** For the individual-level data, “number sampled” refers to the total number of eligible household members as recorded on the household rosters, and the ‘number interviewed’ reflects the total number interviewed/collected data on. Non-response is both from refusal to participate by the individual (rare) or absence from the household upon repeated visits. 2.9 Integration of Secondary Data IPs provided data on the timing and location of implementation of various components of the activity, which will be used to assess differential impact by level of exposure. Findings from the PBS will be triangulated with findings from the mid-term qualitative evaluation to develop insight into the reasons why changes in key outcomes were or where not observed, and to offer recommendations for future activities. 2.10 Limitations and Delimitations This evaluation has several limitations, defined as factors outside the evaluators’ scope of influence. 1. As the evaluation is only based on data from areas that received program support, the evaluation cannot conclude whether a RFSA caused an observed change in outcomes. 2. The baseline and Round 2 surveys were administered several years apart and by different organizations. While efforts were made to maintain consistency by using the same fieldwork manuals and question-by-question guidance, by incorporating input from IPs, and by rehiring some of the enumerators that worked on the baseline, differences in the administration of the two surveys may have occurred. 3. Although independent survey monitors assessed compliance with fieldwork procedures, and data quality checks were run, it is possible that there are quality issues either in the baseline or Round 2 data sets that cannot be identified. 4. Household exposure to the program was measured by asking HH members about their participation in various activities. It is possible that participation was under-reported, as heads of households may not have been aware of activity names or IPs, or whether other members of their household had participated. IMPEL | Implementer-Led Evaluation and Learning 12 Quantitative Evaluation Methods 5. Inaccuracies of the sampling frame required large population weights adjustments in some of the sampled villages in both Rounds 1 and 2. The villages with outlying HH weighting values, particularly those with large weights, tend to increase the design effects and reduce the statistical power of the analysis. This also means that the population estimates living in the areas of implementation are potentially unreliable. The evaluation also has several delimitations, defined as factors that the evaluation team and BHA agreed to accept, though they were less than ideal. 1. The poverty and anthropometry modules were not administered in Round 2. The poverty module was deemed to be too long and labor-intensive relative to its value, and the physical proximity of enumerators to children while taking anthropometric measurements was considered too risky during the COVID-19 pandemic. 2. Where the baseline and Round 2 sampling frames overlapped, villages surveyed in the baseline were re-sampled. However, additional villages in areas where the activity coverage had expanded to, were also included in the sample, and villages in the baseline that were later dropped from the coverage areas of the RFSA were not re-sampled in the baseline. This may have some impact on the comparability of the two rounds. 3. Most of the TPII’s high-impact activities (such as irrigation, drainage, CMCs, and rehabilitation of farm-to-market roads) were only completed in 2021, and so their impacts may not have been fully realized at the time of the Round 2 evaluation data collection. Performance Evaluation of the Tuendelee Pamoja II RFSA in the DRC (Vol. I) Findings 13 3. FINDINGS This section begins with an overview and basic analysis of activity intervention exposure and participation. Then, results are presented by sector. Indicators are calculated at baseline and Round 2 for the entire sample, and the statistical significance of the difference between the two rounds is calculated. Indicators are also stratified (e.g., by gender) as appropriate. As applicable, analysis of the components of the indicators is conducted, particularly if that analysis provides additional information that may alter the conclusions drawn from the findings. We also conduct analysis of the impacts of using different sampling frames, the relationship between activity intervention exposure and outcomes, and relationships between HH characteristics and changes in outcomes, as applicable. 3.1 Activity Intervention Exposure and Participation To assess the levels of RFSA intervention participation and coverage/exposure in the surveyed, data were collected at the community and household-level related to a variety of RFSA interventions. When presenting these data, exposure to an intervention is defined as a household living in a village where the RFSA implemented the intervention (data sourced from the IP). Participation in an intervention is defined (and measured here) as the survey respondent reporting that one or more members of their HH had participated in the intervention over the previous 5 years. It is important to note, however, that participation data collected in the Round 2 survey is not meant to be interpreted as providing accurate estimates of coverage by the RFSA, which is better estimated by activity monitoring data. The primary use of the information collected in Round 2 on intervention participation is to allow for analysis of associations (correlations) between lower-level outcome indicators and intervention participation. Intervention exposure was high for most interventions (Figure 1 below). The exceptions were employment training, alternative livelihoods, and literacy training (53% coverage each), toilet building (7% coverage) and market support (5% coverage). When looking at HH reported intervention participation, 34% of HHs reported participation in at least one of the RFSA surveyed interventions. Self-reported household participation for all interventions was generally low, with the highest coverage of any individual intervention surveyed only reported by 16% of HHs (agriculture training). Youth-related interventions had very low coverage, despite being implemented in all surveyed villages. Youth leadership training participation was reported by 2% of HHs, adolescent life skill trainings by 3%. The mid-term evaluation in 2019 noted that there was minimal direct targeting of youth on health and nutrition related messages, and indicated they were reaching only a very small segment of the adolescent population. IMPEL | Implementer-Led Evaluation and Learning 14 Findings Figure 2. Household exposure and participation in RFSA interventions (Round 2 data) Note: Interventions with no coverage in the sampled villages are excluded. These include: Safe space groups, Climate change adaptation groups, and Cash for work. These self-reported household participation survey findings are difficult to triangulate against the activity-reported figures, however. The Round 2 sampling frame data (provided by Food for the Hungry) suggest that the implementation area has a population of about 100,000 HHs (which, accounting for changes in the sampling frame, corresponds to the baseline reported estimate of approximately 110,000 HHs in the implementation area). However, the 2020 annual report indicates there were 155,000 unique direct-participant households (over 150% of the total HHs in the area of implementation). The same 0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100% VSLA Employment Training Alterbative Livelihoods Farmers' Groups Resource Management Agricultural Training Improved Agricultural Production Agricultural Inputs Farming Information Market Support WASH Training Toilet Building Mothers' Groups Nutrition Training Home Visits Food For Work Programs Literacy Training Couples' Support Groups Adolescent Life Skills Training Youth Leadership Training % HHs reporting one or more members participated in this intervention in the past year (ALL villages) % HHs living in villages where intervention was done by RFSA Performance Evaluation of the Tuendelee Pamoja II RFSA in the DRC (Vol. I) Findings 15 report indicates that the RFSA target is to reach 210,000 unique direct-participant households, which is roughly the entire population of the territories where the RFSA was implemented. Survey reporting error is possible, where the respondent (head of household or other decision maker) was simply not aware of the participation of other household members, for example. Respondent bias due to gender did not appear to play a role, however. The reported participation in mother’s groups and nutrition trainings, for example, showed no significant difference between male and female respondents. 3.2 Water, Sanitation, and Hygiene The table below displays a comparison of Water, Sanitation, and Hygiene (WASH) indicators between 2017 and 2021. The percentage of HH with an improved source of drinking water available within 30 minutes increased significantly, with nearly 51% of HHs having such access in 2021. Similarly, the percentage of HHs that could obtain drinking water in less than 30 minutes (round trip) increased by 18.7 percentage points. 12 The percentage of HHs disinfecting water through various methods did not change significantly, with the exception of a marginally significant decrease in the percentage of HHs boiling their water. In terms of sanitation, the percentage of HHs practicing open defecation decreased significantly to only 3.2% in 2021. There were no other significant changes in WASH indicators detected. Although WASH training was implemented in all villages, only 13.4% of HHs reported participating in these activities. However, among these participants, there were significantly higher rates of use of sanitation facilities (15.4% of those who attended the WASH training vs 5% of those who did not (p < .001) and proper handwashing (4.7% vs. 1.6%, p = 0.014). Toilet building interventions were only implemented in villages representing 6.4% of HHs and did not have a significant relation to sanitation. Table 4. Baseline (BL) and Round 2 (R2) water, sanitation, and hygiene indicator comparisons Indicator Indicator value Raw difference (R2– BL) Significance level* Number of observations 2017 BL 2021 R2 BL Int. 6. Percentage of households using an improved drinking water source 38.3% 51.0% 12.7% ns 1,216 1,261 Available on premises 0.7% 0.3% -0.4% ns 1,216 1,261 Available in 30 minutes or less 27.1% 38.1% 11.0% ** 1,216 1,261 Available in more than 30 minutes 10.5% 11.9% 1.4% ns 1,216 1,261 7. Percentage of households practicing correct use of recommended household water treatment technologies 8.7% 6.5% -2.2% ns 1,216 1,261 Chlorination 4.3% 2.9% -1.4% ns 1,216 1,261 Flocculent/Disinfectant 0.2% 0.0% -0.2% N/A 1,216 1,261 12 Due to a very high design effect of the drinking water source variable, this difference is not significant when accounting for design effect. IMPEL | Implementer-Led Evaluation and Learning 16 Findings Indicator Indicator value Raw difference (R2– BL) Significance level* Number of observations 2017 BL 2021 R2 BL Int. Filtration 0.6% 0.3% -0.3% ns 1,216 1,261 Solar 0.0% 0.1% 0.1% ns 1,216 1,261 Boiling 3.9% 3.4% -0.5% † 1,216 1,261 8. Percentage of households that can obtain drinking water in less than 30 minutes (round trip) 42.8% (GS) 75.9% 33.1% *** 1,216 1,252 9. Percentage of households using a basic sanitation facility 8.7% 6.6% -2.1% ns 1,216 1,261 10. Percentage of households in target areas practicing open defecation 8.0% 3.2% -4.8% ** 1,216 1,261 11. Percentage of households with soap and water at a handwashing station commonly used by family members 2.9% 1.9% -1.0% ns 1,216 1,261 Note: ns = not significant, N/A = not available, † p < 0.1, ** p < 0.01, *** p < 0.001 Green shading (GS): Indicator updated from baseline reported value. 3.3 Agriculture Changes in agricultural practices between 2017 and 2021 are shown in Table 5, below. Significant decreases were observed in the percentage of farmers who used at least two sustainable crop practices in the past 12 months, as well as those using at least two sustainable livestock practices, and the percentage of farmers practicing at least one of the value chain activities promoted by the project. However, it is challenging to assess whether these differences between rounds are methodology/collection issues, or if they accurately reflect changes. Table 5. Baseline and Round 2 agriculture indicator comparisons Indicator Indicator value Raw difference (R2–BL) Significance level* Number of observations 2017 BL 2021 R2 BL Int. 12. Percentage of farmers who used financial services (savings, ag credit, &/or ag insurance in the past 12 months 30.6% 30.0% -0.6% ns 1,706 1,194 Male 32.8% 32.8% 0.0% ns 807 746 Female 28.6% 25.8% -2.8% ns 899 448 13. Percentage of farmers who practiced value chain activities promoted by the activity in the past 12 months 29.4% (GS) 15.0% -14.4% * 874 568 Male 30.0% (GS) 17.2% -12.8% * 467 385 Performance Evaluation of the Tuendelee Pamoja II RFSA in the DRC (Vol. I) Findings 17 Indicator Indicator value Raw difference (R2–BL) Significance level* Number of observations 2017 BL 2021 R2 BL Int. Female 28.7% (GS) 11.1% -17.6% * 407 183 14. Percentage of farmers who used at least 2 sustainable agriculture (crop, livestock, and NRM) practices and/or technologies in the past 12 months 25.6% (GS) 9.7% -15.9% *** 1,706 1,194 Male 29.2% (GS) 9.7% -19.5% *** 807 746 Female 22.5% (GS) 9.6% -12.9% *** 899 448 15. Percentage of farmers who used at least 2 sustainable crop practices and/or technologies in the past 12 months 16.5% (GS) 4.8% -11.7% ** 1,559 (GS) 1,177 Male 17.6% (GS) 4.0% -13.6% *** 730 (GS) 735 Female 15.6% (GS) 6.0% -9.6% * 829 (GS) 442 16. Percentage of farmers who used at least 2 sustainable livestock practices and/or technologies in the past 12 months 19.9% 4.9% -15.0% *** 567 479 Male 21.8% 6.0% -15.8% ** 306 322 Female 17.7% 2.7% -15.0% *** 261 157 17. Percentage of farmers who used at least 2 sustainable NRM practices and/or technologies in the past 12 months 1.9% 1.0% -0.1% ns 1,706 1,194 Male 1.% 1.7% 0.7% ns 807 746 Female 1.2% 0.0% -1.2% ns 899 448 18. Percentage of farmers who used improved storage practices in the past 12 months 26.6% 37.7% 11.1% * 1,696 1,177 Male 27.5% 40.8% 13.3% ** 799 735 Female 25.8% 33.1% 7.3% ns 897 442 Note: ns = not significant, † p < 0.1, * p < 0.05, ** p < 0.01, *** p < 0.001, Green shading (GS): Indicator updated from baseline reported value. The table further below shows the change by crop practice/technology. There were decreases in the majority of practices. The largest decreases were observed in crop rotations and intercropping, which were promoted by the activity, and weed control, which was not. IMPEL | Implementer-Led Evaluation and Learning 18 Findings Table 6. Changes in prevalence of sustainable crop practices between baseline (2017) and Round 2 (2021) Crop Practices Baseline Round 2 Difference Manure 30.3% 24.2% -5.5% Compost 23.6% 19.1% -4.7% Mulching 6.7% 12.0% 5.3% Weed control 52.3% 38.3% -14.0% Dry Planting 10.1% 9.1% -1.0% Ripping into residues 11.6% 1.9% -9.7% Clean ripping 2.6% 1.6% -1.0% Tied ridges 3.4% 0.2% -3.2% Zai pits 0.9% 1.0% 0.1% Potholing 0.3% 0.0% -0.3% Crop rotations 14.6% 3.9% -10.7% Intercropping 29.3% 16.7% -12.6 Contour planting 1.6% 0.7% -0.9% Terracing 4.3% 1.3% -3.0% Land leveling 6.5% 3.0% -3.5% Integrated Pest Management 0.7% 0.5% -0.2% Improved seeds/crop varieties 0.1% 1.3% 1.2% Improved fallow with cover 1.1% 0.3% -0.8% Maintain indigenous trees to improve soil fertility 0.6% 0.4% -0.2% Planting of perennial forage crops 0.6% 1.4% 00.08% Did not use ANY of these practices in the past 12 months 15.6% 37.8% 22.2% Note: Practices with BOLD font are promoted by activity The table further below shows the breakdown by livestock practice/technology. The percentage of farmers who did not use ANY of the practices in the past 12 months increased by 20.5 percentage points. The largest decreases were observed in homemade animal feeds from local product, vaccinations, and deworming, which were promoted by the activity, and improved animal shelters, which was not. Performance Evaluation of the Tuendelee Pamoja II RFSA in the DRC (Vol. I) Findings 19 Table 7. Changes in prevalence of livestock practices and technologies between baseline (2017) and Round 2 (2021) Livestock Practices Baseline Round 2 Difference Improved animal shelters 22.7% 8.4% -14.3% Vaccinations 19.3% 5.2% -14.1% Deworming 18.0% 4.9% -13.1% Castration 0.4% 0.0% -0.4% Dehorning 1.9% 1.1% -0.8% Homemade animal feeds from local products 21.1% 3.8% -17.3% Animal feed supplied by stockfeed manufacturer 1.5% 1.4% -0.1% Artificial insemination 0.3% 0.0% -0.3% Pen Feeding 16.3% 14.9% -1.4% Fodder production and/or veld reinforcement 5.6% 3.0% -2.6% Used the services of community animal health workers/para￾veterinarians 1.9% 0.0% -1.9% Make hay or silage to feed animal during the dry season 0.3% 1.8% 1.5% practices for breeding and keeping of rabbits 5.6% 1.6% -4.0% Feed animals with nutritional supplements during the hunger season 0.5% 2.9% 2.4% Did not use ANY of these practices in the past 12 months 44.0% 64.5% 20.5% Note: Practices with bold font are promoted by activity The 2019 mid-term evaluation indicated that TPII was undertaking too broad of a set of agricultural interventions, resulting in partial implementation and many noted delays and poor success at that point. For example, it stated that rabbits or poultry distributed for animal husbandry had not been the preferred choice, and were often sick, introducing disease to household animals, with many dying. Seeds for home gardening were distributed too late in the season. Participation in VSLAs was associated with a significantly higher rate of access to financial services, although only 6.9% of farmers reported participation. There were no significant associations between participation in interventions and use of value chain activities, sustainable agriculture, or improved storage. 3.4 Women’s Health and Nutrition Indicators of women’s health and nutrition are displayed in Table 8 below. 13 The prevalence of women of reproductive age consuming a diet of minimum diversity increased slightly and (marginally) significantly between 2017 and 2021. There were statistically significant increases in the percentage of women who consumed orange-flesh sweet potatoes, mangoes, and sweet green peppers in the past 24 hours, and a significant decrease in consumption of cabbage. The mid-term evaluation found that the dissemination of lessons to women through care group meetings is significantly behind schedule. It also 13 Anthropometric measurements were not taken during interim data collection due to the risk of close contact during the COVID-19 pandemic. IMPEL | Implementer-Led Evaluation and Learning 20 Findings indicated that agriculture and livestock interventions designed to improve women’s diet quality had not performed well to date. The contraceptive prevalence rate and the prevalence of women of reproductive age who consume targeted nutrient-rich value chain and non-value chain commodities did not change significantly. The prevalence of underweight women and births receiving at least four ANC visits were not assessed at Round 2. 14 Household participation in nutrition training was associated with a better MDD for women from (30% among participants vs. 18% among non-participants). The percentage of women who reported contraceptive use was also positively associated with participation in mothers’ groups for the small numbers of HHs who engaged in these activities. Table 8. Baseline and Round 2 women’s health and nutrition indicator comparisons Indicator Indicator value Raw difference (R2– BL) Significance level* Number of observations 2017 BL 2021 R2 BL Int. 20. Prevalence of women of reproductive age consuming a diet of minimum diversity 18.4% 19.6% 1.2% † 1,273 1,341 21. Contraceptive Prevalence Rate 10.7% 11.6% 0.9% ns 602 647 Modern methods 2.9% 4.7% 1.8% n/a 602 647 Traditional methods 9.0% 7.0% -2.0% ns 602 647 23. Prevalence of women of reproductive age who consume targeted nutrient-rich value chain and non-value chain commodities 66.6% 61.6% -5.0% ns 1,273 1,341 Value chain (VC) commodities 29.1% 30.6% 1.5% ns 1,273 1,341 Bio-fortified (cassava, maize, beans) (VC) 12.7% 19.5% 6.8% † 1,273 1,341 Orange-flesh sweet potatoes (VC) 3.2% 7.9% 4.7% * 1,273 1,341 Soybean (VC) 14.9% 8.8% -6.0% † 1,273 1,341 Passionfruit (VC) 1.2% 2.3% 1.1% ns 1,273 1,341 Mango (VC) 0.6% 3.3% 2.6% ** 1,273 1,341 Non-value chain commodities 37.5% 31.0% -6.5% ns 1,273 1,341 Cabbage (GS) 5.3% 2.9% -2.4% * 1,273 1,341 Cowpea (GS) 0.1% 0.2% 0.1% ns 1,273 1,341 Carrots 0.1% 0.6% 0.5% ns 1,273 1,341 Moringa (GS) 0.0% 0.2% 0.2% ns 1,273 1,341 Orange 1.2% 2.4% 1.2% † 1,273 1,341 Pineapple 0.8% 1.3% 0.5% ns 1,273 1,341 14 ANC visit data had excessive missing data from a skip pattern error in the data collection tool, and so was not reported. Performance Evaluation of the Tuendelee Pamoja II RFSA in the DRC (Vol. I) Findings 21 Indicator Indicator value Raw difference (R2– BL) Significance level* Number of observations 2017 BL 2021 R2 BL Int. Okra 0.9% 0.4% -0.5% ns 1,273 1,341 Sweet green pepper 3.1% 6.2% 3.0% * 1,273 1,341 Eggs 2.7% 3.1% 0.4% ns 1,273 1,341 Animal protein 56.5% 45.6% -10.9% † 1,273 1,341 Note: ns = not significant, † p < 0.1,* p < 0.05, ** p < 0.01, Green shading: Indicator updated from baseline reported value 3.5 Children’s Health and Nutrition Indicators of the health and nutrition of children are shown in the table below. 15 There was a significant decrease in the percentage of children under 5 years who had diarrhea in the last 2 weeks. This decrease was slightly larger for female children (7.2%) than for male children (6.1%). Rates of exclusive breastfeeding of children younger than 6 months increased by 27.5 percentage points, a statistically significant change. While female children were less likely to be exclusively breastfed in 2017, this had reversed in 2021. There were no significant changes observed for diarrhea treated with ORT, prevalence of a minimum acceptable diet, or prevalence of children 6–23 months who consume targeted nutrient-rich value chain and/or non-value chain commodities. Anthropometric measurements were only taken at baseline due to the COVID-19 pandemic which was underway at Round 2. Table 9. Baseline and Round 2 children’s health and nutrition indicator comparisons Indicator Indicator value Raw difference (R2–BL) Significance level* Number of observations 2017 BL 2021 R2 BL Int. 27. Percentage of children under age 5 who had diarrhea in the last two weeks 23.4% 16.7% -6.7% ** 1,284 1,114 Male 25.3% 19.2% -6.1% † 612 528 Female 21.7% 14.5% -7.2% * 672 586 28. Percentage of children under age 5 with diarrhea treated with ORT 42.2% 36.6% -5.6% ns 297 182 Male 37.3% 40.9% 3.6% ns 156 99 Female 47.4% 31.7% -15.7% ns 141 83 29. Prevalence of exclusive breast-feeding of children under six months of age 44.3% 71.8% 27.5% *** 130 134 Male 47.6% 69.5% 21.9% * 69 64 15 Anthropometric measurements were not taken during interim data collection due to the risk of close contact during the COVID-19 pandemic. IMPEL | Implementer-Led Evaluation and Learning 22 Findings Indicator Indicator value Raw difference (R2–BL) Significance level* Number of observations 2017 BL 2021 R2 BL Int. Female 40.7% 73.8% 33.0% ** 61 70 30. Prevalence of children 6–23 months of age receiving a minimum acceptable diet (MAD) 8.5% 8.1% -0.4% ns 392 314 Male 9.0% 9.8% 0.9% ns 190 144 Female 8.0% 6.6% -1.4% ns 202 170 31. Prevalence of children 6–23 months who consume targeted nutrient-rich value chain and/or non-value chain commodities 63.7% 63.4% -0.3% ns 392 314 Male 67.4% 66.9% -0.6% ns 190 144 Female 60.1% 60.4% 0.3% ns 202 170 Value chain (VC) commodities 36.8% 46.0% 9.2% ns 392 314 Male 40.0% 52.7% 12.7% ns 190 144 Female 33.7% 40.3% 6.5% ns 202 170 Bio-fortified (cassava, maize, beans) (VC) 16.3% 25.2% 8.9% † 392 314 Orange-flesh sweet potatoes (VC) 6.7% 15.9% 9.2% * 392 314 Soybean (VC) 16.9% 11.1% -5.8% ns 392 314 Passionfruit (VC) 2.0% 3.8% 1.8% ns 392 314 Mango (VC) 1.7% 3.6% 1.9% ns 392 314 Non-value chain commodities 26.9% 17.3% -9.5% * 392 314 Male 27.4% 14.1% -13.3% * 190 144 Female 26.3% 20.1% -6.2% ns 202 170 Cabbage 5.7% 3.9% -1.9% ns 392 314 Cowpea 0.7% (GS) 0.6% -0.1% ns 392 314 Carrots 0.7% (GS) 0.0% -0.7% na 392 314 Moringa 0.7% (GS) 0.0% -0.7% na 392 314 Orange 1.8% (GS) 2.8% 1.0% ns 392 314 Pineapple 3.1% 1.9% -1.2% ns 392 314 Okra 1.5% 0.3% -1.2% ns 392 314 Sweet green pepper 1.0% 0.4% -0.6% ns 392 314 Eggs 1.9% 2.3% 0.4% ns 392 314 Animal protein 43.2% 39.0% -4.2% ns 392 314 Note: ns = not significant, na = not available, † p < 0.1, * p < 0.05, ** p < 0.01, *** p < 0.001, Green shading (GS): Indicator value not reported in baseline report Performance Evaluation of the Tuendelee Pamoja II RFSA in the DRC (Vol. I) Findings 23 Mothers’ groups, nutrition training, and home health visits were implemented in all communities. However, participation rates were low, with only approximately 7% of children 6–23 months live in HHs that reported engaging in each of these activities. This is unfortunate, as rates of children receiving a minimally adequate diet was 23% among the 26 children living in HHs that participated in nutrition trainings, compared to 6.6% for the children from non-participating HHs. 3.6 Gender Changes in indicators related to gender are displayed in the table below. There was a statistically significant decrease in the percentage of men and women who earned cash in the past 12 months; this decrease was larger for women than for men. The percentage of women in union with children under two who make maternal health and nutrition decisions alone also decreased significantly, while the percentage of men who make such decisions jointly with a partner increased significantly. Similarly, the percentage of men and the percentage of women in union with children under 2 years who make child health and nutrition decisions alone both decreased significantly, while the percentages of men and women who make such decisions jointly with a partner increased significantly. Households having received home visits (8% of all HHs) was significantly associated with better knowledge of MCHN practices. Table 10. Baseline and Round 2 children’s health and nutrition indicator comparisons Indicator Indicator value Raw difference (R2–BL) Significance level* Number of observations 2017 BL 2021 R2 BL Int. 32. Percentage of men and women who earned cash in the past 12 months 42.2% 29.9% -12.34% * 3,510 3,437 Male 49.0% 39.7% -9.34% † 1,705 1,697 Female 35.9% 20.3% -15.55% ** 1,805 1,740 33a. Percentage of men in union and earning cash who make decisions alone about the use of self-earned cash 34.1% 25.9% -8.22% † 600 488 33b. Percentage of women in union and earning cash who make decisions alone about the use of self-earned cash 18.8% 16.0% -2.83% ns 420 219 34a. Percentage of men in union and earning cash who make decisions jointly with spouse/partner about the use of self-earned cash 35.2% 45.7% 10.51% † 600 488 34b. Percentage of women in union and earning cash who make decisions jointly with spouse/partner about the use of self-earned cash 34.6% 43.9% 9.32% ns 420 219 IMPEL | Implementer-Led Evaluation and Learning 24 Findings Indicator Indicator value Raw difference (R2–BL) Significance level* Number of observations 2017 BL 2021 R2 BL Int. 35. Percentage of men and women with children under two who have knowledge of maternal and child health and nutrition (MCHN) practices 59.1% 66.1% 6.98% ns 821 737 Male 51.4% 57.7% 6.32% ns 356 318 Female 65.4% 72.2% 6.83% ns 465 419 36a. Percentage of men in union with children under two who make maternal health and nutrition decisions alone 29.1% 20.6% -8.50% † 353 318 36b. Percentage of women in union with children under two who make maternal health and nutrition decisions alone 25.2% 15.1% -10.13% ** 404 397 37a. Percentage of men in union with children under two who make maternal health and nutrition decisions jointly with spouse/partner 27.6% 39.9% 12.34% * 353 318 37b. Percentage of women in union with children under two who make maternal health and nutrition decisions jointly with spouse/partner 29.3% 40.1% 10.80% † 404 397 38a. Percentage of men in union with children under two who make child health and nutrition decisions alone 18.5% 10.9% -7.58% * 353 318 38b. Percentage of women in union with children under two who make child health and nutrition decisions alone 24.0% 14.7% -9.26% ** 404 397 39a. Percentage of men in union with children under two who make child health and nutrition decisions jointly with spouse/partner 32.2% 47.2% 14.97% * 353 318 39b. Percentage of women in union with children under two who make child health and nutrition decisions jointly with spouse/partner 30.8% 48.4% 17.63% ** 404 397 Note: ns = not significant, † p < 0.1, * p < 0.05, ** p < 0.01 3.7 Activity-Specific Indicators specific to the TPII RFSA are shown in Table 11. A small, non-significant decrease in the percentage of children whose caretakers properly disposed of their feces was observed between Performance Evaluation of the Tuendelee Pamoja II RFSA in the DRC (Vol. I) Findings 25 rounds. The difference was slightly more pronounced among male children than female. Data on livestock penning was not collected at Round 2.16 Table 11. Baseline and Round 2 activity-specific indicator comparisons Indicator Indicator value Raw difference (R2–BL) Significance level* Number of observations 2017 BL 2021 R2 BL Int. 46. Percentage of children whose caregivers who properly disposed of child feces 67.7% 62.6% -5.10% ns 519 447 Male 69.9% 59.6% -10.30% * 257 208 Female 65.6% 65.2% -0.40% ns 262 239 47. Percentage of men/women who say it is ok for a man to batter his wife for any reason 40.5% 50.6% 10.1% † 1,744 1,194 Male 40.1% 50.5% 10.4% † 822 746 Female 40.8% 50.7% 9.8% ns 922 448 48. Percentage of women that report participating in community decision￾making bodies 24.3% 20.6% -3.7% ns 1,273 1,341 Note: ns = not significant, † p < 0.1, * p < 0.05 3.8 Shock Exposure and Resilience Indices Significant declines in the shock exposure indices (both unweighted and severity weighted) relative to the baseline were predominantly due to sizable drops in the proportion of HHs reporting economic shocks (Table 12). In 2017, 85% of all HHs overall reported rising food prices as a shock, compared with 45% in 2021, and the proportion of HHs reported shocks related to fluctuating exchange rates and currency devaluation declined from 61% to 6%. Declines of over 10% were also observed for agricultural pests or diseases, inter-tribal conflict, looting or robbing, unavailability of agricultural inputs, and rising or falling prices for agricultural inputs. Interestingly, there were no shocks that significantly increased in magnitude during this period, while rates of drought/flooding and family illness or death remained above 40% overall. All three resilience capacities (absorptive, adaptive, and transformative) increased significantly between baseline and Round 2. Changes in the indices were largely driven by a single component indicator and/or shifts in the symmetry of the score distribution such that the outlying scores in the tails had more of an impact on the change (rather than an overall shift in the population). Note that the Round 2 evaluation did not collect poverty data, which was used as one of the components in the calculation of the resilience indices. The baseline indices were re-calculated to match the calculations used in the Round 2 survey to allow for improved comparability between rounds. 16 An error in the ODK skip pattern resulted in excessive missing data. IMPEL | Implementer-Led Evaluation and Learning 26 Findings Table 12. Baseline and Round 2 shock exposure and resilience indicator comparisons Indicator Indicator value Raw difference (R2–BL) Significance level* Number of observations 2017 BL 2021 R2 BL Int. 40 Shock exposure index (GS) 5.5 2.8 -2,7 *** 1,212 1,263 41. Cumulative impact of shock exposure index (severity weighted shock exposure) (GS) 32.3 14.5 -17.8 *** 1,212 1,263 42. Absorptive capacity index (GS) 19.4 41.6 22.2 *** 1,212 1,263 43. Adaptive capacity index (GS) 24.9 27.0 2.1 *** 1,212 1,263 44. Transformative capacity index (GS) 12.8 16.2 3.4 *** 1,212 1,263 Note: *** p < 0.001, Green shading (GS): indicator value updated from baseline report The significant increase in the absorptive capacity index between rounds was primarily due to a significantly increased presence of humanitarian assistance. 17 This increase in reported humanitarian assistance may be due to additional humanitarian and development assistance, which may include certain types of services or other assistance provided through TPII, although it is important to note that this cannot be confirmed by available data. Small but significant improvement in the availability of informal safety nets, cash savings, and productive assets were also observed, offset by a sizable decline in access to remittances. Additional figures showing the distributions of the index are found in Annex C. Table 13. Absorptive Capacity Index: Changes in component indicators Indicator Direction of change (BL to R2) Significance, scale 1. Informal Safety Nets Increase Significant, small 2. Bonding Social Capital Increase Non-significant 3. Cash Savings Increase Significant, small 4. Productive Assets Increase Significant, small 5. Livestock Assets Unchanged Non-significant 6. Preparedness/Mitigation Increase Significant, small 7. Agricultural Insurance Decrease Non-significant 8. Humanitarian Assistance Increase Significant, large 9. Access to Remittances Decrease Significant, moderate 17 This variable is defined as the respondent indicating that government or NGO emergency food or cash assistance is available in the respondent’s village OR the household reported receiving emergency food or cash assistance from the government or NGO during the 12 months prior to the survey. Performance Evaluation of the Tuendelee Pamoja II RFSA in the DRC (Vol. I) Findings 27 The adaptive capacity index increased significantly between rounds, but the improvement was small. The only indicators which showed significant positive change were social safety nets and productive assets, while significant declines were observed for education/training, livelihood diversity (primarily due to the reduction of “livelihood sources” such as remittances and gifts) and improved agricultural practices. This suggests that the increase in the index likely reflects a reduction in skewness due to a shift toward the middle, with fewer HHs at the upper end of the aggregate scale and little improvement for HHs at the lower end. Additional figures showing the distributions of the index are found in Annex C. Table 14. Adaptive Capacity Index: Changes in component Indicators Indicator Direction of change (BL to R2) Significance, scale 1. Bridging Social Capital Increase Non-significant 2. Social Safety Nets Increase Significant, small 3. Education/Training Decrease Significant, large 4. Livelihood Diversity Decrease Significant, large 5. Productive Assets Increase Significant, small 6. Livestock Assets Unchanged Non-significant 7. Improved Agric. Practices Decrease Significant, moderate The transformative capacity index also increased, driven by small but significant improvements in formal safety nets, 18 access to natural resources, 19 and collective action. A significant decrease was observed for local government responsiveness, with no changes observed for bridging social capital or participation in local decisions. The low value of this index demonstrates a high degree of skewness, such that only a few HHs were assigned high scores based on these criteria. Additional figures showing the distributions of the index are found in Annex C. Table 15. Transformative Capacity Index: Changes in component indicators Indicator Direction of change (BL to R2) Significance, scale 1. Access to Natural Resources Increase Significant, small 2. Bridging Social Capital Unchanged Non-significant 3. Collective Action Increase Significant, small 4. Local Gov’t Responsiveness Decrease Significant, moderate 18 Community-level variable indicating the number of formal safety nets a household reports to be present in their village. These safety nets include: Food assistance, non-food items, assistance due to losses in livestock, NGO disaster response program, NGO help when faced with a shock, if the HH received assistance from the gov’t or NGO. 19 community-level variable, calculated as the sum of the number of communal natural resources that are available in a village. These resources include: communal grazing land, communal water source for livestock, communal source of firewood, communal source of irrigation water. IMPEL | Implementer-Led Evaluation and Learning 28 Findings Indicator Direction of change (BL to R2) Significance, scale 5. Participation In Local Decisions Unchanged Non-significant 6. Formal Safety Nets Increase Significant, large Looking at the relationship between coping strategies and resilience indicators, no associations were observed. However, three of the adaptive capacity component indicators are related to agriculture, so HHs that did not engage in agriculture tend to have lower scores on this index even if they are highly resilient. This indicates that the adaptive capacity index may be of limited use for HHs not engaged in agriculture. It may benefit from adaptation to reflect adaptability as a function of livelihood. 3.9 Food Security Indicators related to food security are summarized in Table 13 below. There was a marginally significant decrease in the prevalence of moderate or severe food insecurity (FIES) based on a 30-day recall. This was driven by HHs with male and female adults, which showed a 3.9 percentage point decrease in FIES. Table 16. Baseline and Round 2 food security indicator comparisons Indicator Indicator value Raw difference (R2–BL) Significance level* Number of observations 2017 BL 2021 R2 BL Int. 1. Average Household Dietary Diversity Score (HDDS) 3.9 3.9 0.0 ns 1,116 1,120 2. Prevalence of moderate or severe food insecurity based on 30-day recall (FIES) 93.0% 89.6% -3.4% † 1,208 1,231 Male and female adults 93.1% 89.2% -3.9% * 1,000 976 Adult female, no adult male 92.0% 91.1% -0.9% ns 152 212 Adult male, no adult female 90.7% 94.4% 3.7% ns 42 43 Child, no adults na na na na 14 0 Note: ns = not significant, na = not available, † p < 0.1, * p <0.05 The prevalence of moderate and severe food insecurity was very high at baseline (93%) and in Round 2 (90%). However, the FIES questions may not be adequately describing the food security situation in this context. Most households answered yes to all (or nearly all) of the eight FIES questions, and there was no observation of the expected delineation between frequency of responses to the less severe questions vs. the most severe questions. Similar patterns are found in the baseline data. Additionally, there is a lower-than-expected correlation between the FIES data and the similar food-related coping strategy questions found in the resilience module. The two indicators use different recall periods (30 days for the FIES, 7 days for the food security coping), which may indicate that things like missing meals may not happen every week, they may be more likely to occur within the recall period of a month. Additional figures showing the relationship between the individual FIES and food security coping questions are found in Annex C. Performance Evaluation of the Tuendelee Pamoja II RFSA in the DRC (Vol. I) Findings 29 No associations between the FIES and resilience indices were observed in the Round 2 data, likely due to low variation in the FIES variable. Looking at the Round 2 survey data, there is some association between the HDDS and the resilience indices, particularly the absorptive and adaptive indices. The transformative capacity index is largely defined by community-level indicators rather than household-level indicators, and so the associations between that index and the HDDS are less likely to be observed. The graphs below show the relationship between the HDDS and the absorptive and adaptive indices. It should be noted that the sample size at the very high end of the HDDS is small, those means and ranges at the high end of the scale should be disregarded. Figure 3. Resilience capacity indices by Household Dietary Diversity Score (Round 2 data) Absorptive capacity index Adaptive capacity index IMPEL | Implementer-Led Evaluation and Learning 30 Findings Transformative capacity index Performance Evaluation of the Tuendelee Pamoja II RFSA in the DRC (Vol. I) Conclusions 31 4. CONCLUSIONS Overview • The evaluation shows some promising results. Certain indicators appear to be moving in the correct direction, and the analysis tends to show that direct participation in some interventions is associated with improvements in lower-level indicators. o However, most of the TPII RFSA’s high-impact activities (such as irrigation, drainage, CMCs, and rehabilitation of farm-to-market roads) were only completed in 2021, and so their impacts may not have been fully realized at the time of the Round 2 evaluation data collection. • However, results at the population level do not show consistent, significant improvement. o Coverage rates of direct participants of the interventions are often low (a small portion of the total population), and spillover of the impacts from the RFSA interventions to indirect participants may be less than hoped. As such, impacts may be diluted at the population level. o More programmatic intensity and focus may be required to positively impact food security and resilience at the population level. • The important questions for BHA and Food for the Hungry are: o Considering the cost of implementation of this activity and its interventions relative to the “saturation” that the activity might be expected to reach, is it worth running an intervention that is “a mile wide and an inch deep”? o How can the diversity of projects be reduced, selecting for the highest and most sustainable impact? o This study only begins to scratch at the surface of these questions. Intervention exposure and participation • The self-reported HH participation was generally low for the individual interventions, despite most having been implemented in all the villages included in the Round 2 survey. o Overall, 34% of HHs in the Round 2 survey reported participation in one or more of the surveyed RFSA interventions. o The highest reported prevalence of HH participation include agriculture trainings (16% of HHs), WASH trainings (14% of HHs), farmers groups (13% of HHs), and activities focused on improved agricultural production (13%). o Others had very low coverage. For example, youth-related interventions had very low coverage, despite being implemented in all surveyed villages. (Youth leadership training participation was reported by 2% of HHs, adolescent life skill trainings by 3%. • The direct participation findings, however, are difficult to triangulate against the activity￾reported figures. o The Round 2 sampling frame data (provided by Food for the Hungry) suggest that the implementation area has a population of about 100,000 HHs. However, the 2020 annual report indicates there were 155,000 unique direct-participant HHs (over 150% of the total HHs in the area of implementation). IMPEL | Implementer-Led Evaluation and Learning 32 Conclusions o The 2020 annual report also indicates that the RFSA target is to reach 210,000 unique direct-participant HHs, which is roughly the entire population of the territories where the RFSA was implemented. WASH • Positive improvements were observed in access to safe drinking water between rounds. o A significant improvement was observed in the percentage of HHs that can obtain drinking water in under 30 minutes (increased from 43% to 76%). o Access to improved drinking water also improved (38% at baseline to 51% at Round 2), although the change was not significant. o No significant changes in the use of water treatment technologies were observed. • Use of basic (improved) sanitation did not change significantly between surveys. 20 However, the percentage of HHs practicing open defecation decreased significantly from 8% to 3%. o Toilet building interventions were only implemented in villages representing 6% of HHs and did not show a significant relationship to sanitation. o However, WASH training participant HHs (13% of HHs surveyed) reported a significantly higher prevalence of use of basic (improved) sanitation facilities (15%) than HHs that did not participate (5%), as well as a significantly (though small) higher use of handwashing stations (5% among participants, 2% among non-participants). Agriculture • Few of the agriculture indicators reported improvements: o Use of improved storage practices increased significantly (27% to 38%) • Others showed only small/non-significant changes: o Practice of value chain activities promoted by project decreased, but not significantly. o Use of financial services remained unchanged. o Use of Natural Resource Management practices/technologies remained very low. • Some outcome indicators showed negative change. However, it is challenging to assess whether these differences between rounds are methodology/collection issues or if they accurately reflect changes. o Use of at least two sustainable crop practices (17% to 4%) o Use of at least two sustainable livestock practices/technologies (20% to 5%) o Practicing at least one of the value chain activities promoted by the project (29% to 15%) • In the Round 2 survey, HH reported participation in certain agriculture-related interventions was significantly associated with better outcomes in only one instance: o Participation in VSLAs is associated with higher rates of access to financial services (but only 7% of farmers had participation reported at the HH level). o No significant associations between participation in interventions and the use of value chain activities, sustainable agriculture, or improved storage. 20 It’s difficult to assess the change in toilet type compared to baseline. The baseline indicates that 82% of households had a “water flush to pit latrine” type toilet, which is likely a mistake in data collection (although still classified as unsafe/unimproved sanitation). The interim survey reports that <1% of households have flush-to-pit-latrine toilets- similar to the most recent DHS survey, which indicates that 0.5% of urban and 0% of rural have flush-to-pit-latrine toilets. Performance Evaluation of the Tuendelee Pamoja II RFSA in the DRC (Vol. I) Conclusions 33 Women’s Health and Nutrition • A small, marginally significant improvement was observed in women’s consumption of a diet of minimum diversity (18% at baseline to 20% at Round 2). However, women’s consumption of targeted nutrient-rich commodities did not change significantly, and only small changes were seen in the various specific commodities assessed. o However, Household participation in nutrition training was associated with a better MDD for women from (30% among participants vs. 18% among non-participants). • No significant change was observed in contraceptive use. o However, the percentage of women who reported contraceptive use was positively associated with participation in mothers’ groups for the small numbers of HHs who engaged in these activities. Children’s Health and Nutrition • Significant improvements were observed in some of the key outcomes of children’s health and nutrition between survey rounds. o Diarrhea prevalence decreased significantly, from 23% to 17%. o The prevalence of exclusive breastfeeding of children under 6 months experienced a large, significant increase, from 44% to 72%). • No significant change was seen in ORT treatment, children consuming a minimum acceptable diet, or children consuming targeted nutrient-rich foods. • Although Mothers’ groups, nutrition training, and home health visits were implemented in all communities surveyed at Round 2, the participation rates were low. Only about 7% of HHs reported participation in each of these interventions. o Among the few (26) children benefiting from HH participation in nutrition training in the survey, the prevalence of a minimum acceptable diet was 23%, compared to 7% for the non-participating. Gender • Little change was noted across the gender indicators. o Gender norms are deeply entrenched and will likely take years or even decades to change. • The percentage of women who earned cash in the previous year decreased significantly, from 36% to 20% at Round 2. • Other indicators showed no significant change, including the percentage of men and women who say it’s ok for a man to batter his wife for any reason, the percentage of women participating in decision-making bodies, and the percentage of men or women that have knowledge of maternal and child health practices. o However, HHs having received home visits (8% of all HHs) were significantly associated with better knowledge of MCHN practices. IMPEL | Implementer-Led Evaluation and Learning 34 Conclusions Food Security, Shock Exposure, and Resilience • All three resilience capacity indices (absorptive, adaptive, and transformative) increased significantly between baseline and Round 2. However, changes in the indices were largely driven by only one (or a few) of their component indicators. o The increase in the absorptive capacity index was primarily due to a significantly increased presence of the humanitarian assistance component of the index, as well as smaller improvements in the availability of informal safety nets, cash savings, and productive assets. The improvement was offset by a sizable decline in access to remittances. o The adaptive capacity index increased significantly, but the change was small. The improvement was driven by positive changes in social safety nets and productive assets, but lessened by decreases in education/training, livelihood diversity (primarily due to the reduction of “livelihood sources” such as remittances and gifts), and improved agricultural practices (all sub-components of the index). o Transformative capacity experienced a small but significant increase. This improvement was driven by improvements in formal safety nets, access to natural resources, and collective action (all sub-components of the index). • Food security, as measured by the HDDS and the FIES, showed no significant change. o The prevalence of moderate and severe food insecurity was very high at baseline (93%) and at Round 2 (90%). This homogeneity of the food security status as measured by this indicator renders a more detailed analysis less useful in assessing change. The FIES has other statistical limitations in the populations surveyed. It may not adequately describe the food security situation. o Considering the rapidly evolving food security in the DRC, compounded in the year prior to the survey by the COVID-19 pandemic, stable food security measures could be interpreted as a positive outcome. • There is a positive association between the HDDS and the resilience indices in the Round 2 data, particularly the absorptive and adaptive indices. o The transformative capacity index is largely defined by community-level indicators rather than household-level indicators, and so the associations between that index and the HDDS are less likely to be observed. Methodology • The pre/post-PBS design has some important limitations. o Modifications in areas of implementation after the baseline PBS is common across RFSAs. The pre/post-PBS methodology may not be well suited to adapt to changing areas of implementation. o Sampling frame data in the DRC often has large inaccuracies. This results in highly variable probability/population weights and a loss of statistical power. Alternative PBS sampling strategies should be considered that would have less loss of statistical power with similar sample sizes and budgets. o Population-level changes in many of the lower-level indicators have a low likelihood of occurring with the given intensity of certain interventions. Performance Evaluation of the Tuendelee Pamoja II RFSA in the DRC (Vol. I) Conclusions 35 o There is a desire to have evaluation data that can show the impact of interventions on the various outcomes. A PBS does not readily allow this level of analysis. • Some key outcome indicators may not adequately perform in the context. o The utility of certain food security indicators (such as the FIES) should be re-evaluated, including an assessment of their functionality in specific locations/contexts. o The resilience capacity indices are less useful as composite indicators. The sub￾components of the indices tend to reveal more useful information. For example, three of the adaptive capacity component indicators are related to agriculture, so HHs that did not engage in agriculture tend to have lower scores on this index even if they are highly resilient. This indicates that the adaptive capacity index in the aggregate may be of limited use for HHs not engaged in agriculture. It may benefit from adaptation to reflect adaptability as a function of livelihood.