July 24, 2023 This publication was produced by the Mission Strategic Support Program (MSSP) at the request of the United States Agency for International Development (USAID) Mission to the Democratic Republic of the Congo (DRC). INTEGRATED PROGRAMMING ASSESSMENT Final Report INTEGRATED PROGRAMMING ASSESSMENT Submission date: July 24, 2023 Contract Number: 7200AA20D00023 COR Name: James Mashikano Submitted by: SoCha LLC 1201 N Orange Street, Suite 600 Wilmington, DE 19801 Email: wstuppert@socha.net (corresponding author) DISCLAIMER The authors’ views expressed in this publication do not necessarily reflect the views of the United States Agency for International Development or the United States Government. iii ABSTRACT USAID/DRC commissioned the Integrated Programming Assessment (IPA) to support the Mission in realizing the full potential of cross-sectoral integration by investigating current integrative practices and distilling recommendations on how to facilitate integration. Cross-sectoral integration is the intentional sequencing and layering of interventions from different development sectors to create synergies that benefit target populations (USAID 2022). The IPA investigated three forms of cross-sectoral integration: “Integration by design” through singular activities designed to encompass interventions from multiple sectors (governance, health, peace and security, economic growth, and humanitarian assistance) managed by, 1) a single office in the Mission or 2) multiple offices in the Mission, and 3) “integration post design” through collaboration between single-sector activities. To answer the assessment questions, the IPA methodology used Participatory Adaptive Modeling (PAM) to produce Bayesian change models for two integrated, multi-sectoral activities along with in￾depth interviews with activity managers and implementing partners. The findings highlight the complex factors that contribute to the success of integration. The assessment found that “integration by design” when a multi-sectoral activity is managed by a single technical office is a more efficient and effective approach to cross-sectoral integration than “integration post design” when single-sector activities attempt to collaborate during implementation. Management of a multi-sectoral activity by multiple offices can make it more difficult to realize the benefits of “integration by design.” By using the Bayesian change models for scenario analysis, the assessment team quantified the “integration dividend,” i.e., the difference in the probability that an activity will achieve its intended results through interventions in multiple sectors compared to interventions in a single sector. iv ACKNOWLEDGMENTS This work was commissioned by USAID/DRC. The assessment was led by Veronica Mulligan, CLA and Communications Advisor, and Wolfgang Stuppert, SoCha Global Evaluation Manager, on behalf of the USAID/DRC Mission Strategic Support Program (MSSP). The assessment team included Julie Bela, CLA and Communications Specialist. The assessment team highly appreciates the excellent collaboration with the activity managers and implementing partners of the USAID/DRC Bureau for Humanitarian Assistance (BHA) Resilience Food Security Activities (RFSAs) “Budikadidi” and “FSP Enyanya.” Their contributions to the assessment team's understanding of the activities’ program logic and expected outcomes provided in online workshops and interviews were crucial in making this assessment exercise successful. The assessment team would also like to extend their sincere thanks to the USAID/DRC Agreement Officer's Representatives (AORs) and Contracting Officer's Representatives (CORs), who openly discussed their successes and challenges in cross-sectoral integration. Finally, the assessment team would like to thank the USAID/DRC Integrated Health Program (IHP) and Fall Army Worm (FAW) Activity representatives, who agreed to share their insights and experiences with the assessment team. v CONTENTS Executive Summary..................................................................................................................I 1. Background......................................................................................................................1 2. Assessment Design..........................................................................................................5 2.1 Purpose and Questions........................................................................................................................... 5 2.2 Approach.................................................................................................................................................... 6 2.3 Data Collection and Analysis...............................................................................................................11 2.4 Limitations................................................................................................................................................13 3. Findings ..........................................................................................................................14 3.1 Strategic Question: The Integration Dividend.................................................................................14 3.2 Assessment Question 1: Mission Collaboration Culture and Leadership................................20 3.3 Assessment Question 2: Roles and Responsibilities in Cross-Office Activities......................23 3.4 Assessment Question 3: Collaboration between Mission & Implementing Partners............24 3.5 Assessment Question 4: Cross-Sectoral Integration and Multi-Sectoral Activities...............25 3.6 Assessment Question 5: Cross-Sectoral Integration through Collaboration..........................29 4. Conclusions ....................................................................................................................33 4.1 Overarching Conclusions.....................................................................................................................33 4.2 Strategic Question: The Integration Dividend.................................................................................33 4.3 Assessment Question 1: Mission Collaboration Culture and Leadership................................34 4.4 Assessment Question 2: Roles and Responsibilities in Cross-Office Activities......................34 4.5 Assessment Question 3: Collaboration between Mission & Implementing Partners............35 4.6 Assessment Question 4: Cross-Sectoral Integration and Multi-Sectoral Activities...............35 4.7 Assessment Question 5: Cross-Sectoral Integration through Collaboration..........................36 5. Assessment Question 6: Recommendations ..............................................................37 Annexes...................................................................................................................................39 Annex I Scope of Work .....................................................................................................................................40 Annex II Data Collection Tools........................................................................................................................47 Annex III List of Informants...............................................................................................................................55 Annex IV Core Behavior Changes...................................................................................................................57 Annex V Scenario Analyses...............................................................................................................................59 Annex VI References...........................................................................................................................................62 vi FIGURES Figure 1 - Results Achievement in % of Multi-Sectoral Approach (FSP-Enyanya).......................................II Figure 3 – Eliciting Contribution Estimates..........................................................................................................8 Figure 4 – Chaining Contribution Estimates........................................................................................................9 Figure 2 – Comprehensive Bayesian Change Model for Budikadidi .............................................................10 Figure 6 - Success Probabilities for "Community Members Access Health Care" for Different Implementation Levels.............................................................................................................................................20 TABLES Table 1 – Budikadidi Core Behavior Changes...................................................................................................16 Table 2 – FSP-Enyanya Core Behavior Changes...............................................................................................16 Table 3 – USAID/DRC Staff (KIIs)........................................................................................................................55 Table 4 – BHA RFSA Implementing Partner Staff (Workshops)...................................................................55 Table 5 – Other Activities (KIIs)...........................................................................................................................56 Table 6 – Budikadidi Core Behavior Changes...................................................................................................57 Table 7 – FSP-Enyanya Core Behavior Changes...............................................................................................58 Table 8 – Budikadidi: Percentage Changes in Success Probabilities for Different Levels of Implementation, Comparison to Extrapolated Baseline..................................................................................59 Table 9 – FSP-Enyanya: Percentage Changes in Success Probabilities for Different Levels of Implementation, Comparison to Extrapolated Baseline..................................................................................60 Table 10 – Budikadidi: Integration Dividend for Different Sectors and Different Levels of Implementation .........................................................................................................................................................60 Table 11 – FSP-Enyanya: Integration Dividend for Different Sectors and Different Levels of Implementation .........................................................................................................................................................61 vii ACRONYMS AMELP Activity Monitoring, Evaluation, and Learning Plan AOR Agreement Officer's Representative BHA Bureau for Humanitarian Assistance CAC Community Action Committee CARPE Central Africa Regional Program for the Environment CDCS Country Development Cooperation Strategy CLA Collaborating, Learning, and Adapting COP Chief of Party COR Contracting Officer's Representative CVD Comité Villageois de Développement DO Development Objective DRC Democratic Republic of the Congo DRG Democracy, Human Rights, and Governance Office EG Economic Growth office FY Fiscal Year IP Implementing Partner IPA Integrated Programming Assessment KII Key Informant Interview MCST Mid-Course Stock Taking MSSP Mission Strategic Support Program PAM Participatory Adaptive Modeling P-DEC Partnership for the Development of Eastern Congo PSO Peace and Security Office RFSA Resilience Food Security Activity SILC Savings and Internal Leading Community SOW Scope of Work TOC Theory of Change VSLA Village Savings and Loan Association WASH Water, Sanitation, and Hygiene I EXECUTIVE SUMMARY Background Cross-sectoral Integration is one of the three strategic pillars of the 2020-2025 Country Development Cooperation Strategy (CDCS) of the USAID Mission to the Democratic Republic of the Congo (DRC). Cross-sectoral integration is the intentional sequencing and layering of interventions from different development sectors to create synergies that benefit target populations (USAID 2022). 1 Integrative practices include the collaboration between technical teams at the Mission in the design, procurement, and monitoring of activities (Mission level), and the collaboration between implementing partners in delivering integrated interventions in target communities (activity level). A particular focus of the USAID/DRC CDCS is the integration of development and humanitarian interventions to simultaneously “address emergency situations and the factors that precipitate them” (USAID/DRC 2020). This integration is sought particularly in key provinces in Eastern DRC where the Mission strives to "mitigate the effect of a broad range of shocks by building more resilient systems and communities and stimulating more reliable and higher-quality basic services” (USAID/DRC 2020). USAID/DRC engages in three types of cross-sectoral integration: Implementation of activities with multi-sectoral approaches managed by a single technical office, implementation of activities with multi-sectoral approaches jointly financed and managed by several offices, and collaboration between activities from different sectors. The first two types of cross-sectoral integration are forms of “integration by design,” the latter is a form of “integration post design.” Previous attempts at studying the contributions of cross-sectoral integration to achieving USAID/DRC’s Development Objectives (DOs) fell short of providing reliable information on the benefits of integration as compared to single-sector activities and lacked actionable recommendations for improving the effectiveness and efficiency of the CDCS. The present assessment focuses on in-depth case studies to systematically compare the Mission’s experiences with single-sector and multi-sector activities that involve both “integration by design” and “integration post design.” The systematic comparison of concrete achievements and challenges allowed the assessment team to produce new knowledge about how integration works to strengthen activities’ contributions to achieving the Mission’s development objectives (DOs). Assessment Design The Integrated Programming Assessment (IPA) was commissioned by USAID/DRC. The IPA’s purpose is to support the Mission in realizing the full potential of integration. The assessment aims to achieve this purpose by distilling concrete and actionable recommendations on how to facilitate integration that can inform CDCS strategy adaptation and collaborative practice. The current USAID/DRC CDCS and the assessment purpose both assume that integration is beneficial. Consequently, most assessment questions focus on how integration benefits come about and how integration can be facilitated. Additionally, the Mission leadership emphasized fundamental strategic questions: Should the Mission invest efforts in increased cross-sectoral integration? Is integrated programming more effective than single-sector programming? To what extent does multi￾sectoral integration improve the effectiveness of activities? To answer these strategic questions (c.f., Question 6 in the assessment scope of work annexed to this report), the assessment team developed a new metric, the “Integration Dividend,” based on scenario analyses that used Bayesian statistics to compare integrated programming to single-sector 1 The report uses the author-date method for in-text citations. Full citations can be found in Annex VI. II programming. The other six assessment questions are as follows: 1. How do Mission work culture and leadership influence collaboration across offices? 2. How do technical offices distribute roles and responsibilities in cross-office activities? 3. How does the Mission collaborate with implementing partners to further integration? 4. How do multi-sectoral activities achieve effective integration? 5. How does collaboration between activities contribute to effective integration? 6. How could integrated programming better contribute to the effectiveness and efficiency of integrated activities and strengthen their contributions to CDCS Objectives? (Recommendations) The assessment’s primary approach is Participatory Adaptive Modeling (PAM). PAM is a process that uses monitoring data and the experiential knowledge of implementing partner (IP) staff at all levels and across sector teams to produce a comprehensive model of an activity’s approach to generating change among target populations. Workshops are used to reconstruct an activity’s change processes and determine the contributions of each intervention to activity objectives. The resulting model is known as a Bayesian change model. The model contains an activity’s outputs, outcomes, and impacts and shows to what extent each intervention is effecting change across the full results framework. The models can be used for scenario analyses that show how an activity is expected to perform in changing contexts, and how adaptive measures would affect activity effectiveness. PAM focused on two Resilience Food Security Activities (RFSAs) by the Bureau for Humanitarian Assistance (BHA): 1) Budikadidi, and 2) FSP-Enyanya. The RFSAs combine interventions in agriculture, nutrition, WASH, local governance, and gender to improve food and nutrition security and the economic well-being of vulnerable households. The Bayesian change models for the two RFSAs were built in 16 online workshops with IP staff in two steps. First, the assessment team and IP staff jointly produced an action-focused Theory of Change (TOC) for each activity that visualizes an activity’s change processes as stepwise causal chains. Here, each step is formulated as an action taken by a stakeholder of the activity. Second, the assessment team elicited information from IP staff on how strongly one step in each change process is expected to contribute to the next and included this information in the TOC model structure. Figure 1 - Results Achievement in % of Multi-Sectoral Approach (FSP-Enyanya) Notes: The figure shows average results achievement under four scenarios: when FSP-Enyanya’s complete multi-sectoral approach is implemented (first column) and when only the interventions of one sector are implemented (columns 2 to 4). Average results achievement is measured based on core behaviors changes (see Annex IV). Calculations use elicited effect estimates and scenario analyses (see textboxes on page 9 and 11 for more details). 60% 65% 70% 75% 80% 85% 90% 95% 100% Multi-Sectoral Approach Only Governance Interventions Only Health Interventions Only Livelihoods Interventions Integration Dividend III The resulting Bayesian change models were used to calculate the expected results of each activity for different scenarios, including scenarios where the activities are only partially implemented. To quantify the contributions of intra-activity integration to overall activity effectiveness, the assessment compared the results that implementers expected a RFSA to achieve after implementing the complete multi-sectoral approach, versus the expected results from implementing interventions in only one sector. The differences in success probabilities provide an estimate of the “Integration Dividend” realized by multi-sectoral integration (see Figure 1). In addition to PAM, the assessment team conducted interviews with 19 key informants, including representatives of the Integrated Health Program (IHP), representatives of the Fall Army Worm (FAW) activity, and USAID/DRC activity managers.2 The interviews had different purposes. KIIs with IHP and the FAW activity provided additional context for the analysis of the RFSAs. IHP was interviewed as a collaboration partner of the RFSAs, while FAW was interviewed as a single-sector activity with an approach toward working with farmers comparable to that of the RFSAs. Interviews with AORs and CORs were conducted to broaden the assessment’s perspective on integration beyond the four case studies and the activity level. Overarching Conclusions Integration substantially improves the effectiveness of activities. Systematic scenario analyses of Bayesian change models allowed the IPA to quantify the contributions of cross-sectoral integration to activity effectiveness. By comparing extended baseline scenarios with scenarios in which the RFSAs’ multi-sectoral approaches are implemented fully or partially, the assessment team calculated the “Integration Dividend,” i.e., the increased average results achievement of multi-sector programming over single-sector programming. In the case of the two RFSAs, the integration dividend varies between 3% (Budikadidi, when comparing health-only approach to full approach) and 31.5% (FSP-Enyanya, when comparing livelihoods-only approach to full approach, see Figure 1). “Integration by design” is a more efficient means of implementing integrated activities than “Integration post design.” The assessment’s findings suggest that “Integration by Design,” i.e., through the integration of interventions from multiple sectors in a single activity, is a more efficient means to achieve integration and strengthen activities’ contributions to DOs than “Integration post design,” i.e., through collaboration between activities. The assessment described several preconditions for the successful collaboration of activities that are complicated to achieve in practice (see AQ5 findings). At the same time the IPA identified several unique advantages of multi￾sectoral activity designs (see AQ4 findings). “Integration by Design” requires multi-lateral communication to work. Our analysis of the RFSAs shows that being able to flexibly layer interventions and temporarily shift the focus of community-level interventions is a major advantage of multi-sectoral activities. The Mission’s experiences with the management of P-DEC suggest that joint activity management by several offices poses a risk to realizing these strengths of “Integration by Design.” A lack of multi-lateral communication between technical offices at the Mission hampered the development of a joint vision for the success of P-DEC. Instead, offices insisted on sectoral progress indicators in bilateral communication with the implementing partner. In these conditions, the ability of an implementing partner to realize synergies among interventions is limited. Additional Findings and Conclusions Strategic Question: To what extent does multi-sectoral integration improve the effectiveness of activities? Systematic scenario analyses of the Bayesian change models for the BHA RFSAs show that integrated programming boosts the effectiveness and reach of sector-specific interventions and can act as an insurance policy in challenging contexts. The strongest benefits from 2 Throughout the report, „activity manager” is used as a general team for all Mission staff who are involved in the management of activities, including AORs, CORs, and their alternates. The umbrella term is also used to further protect the anonymity of interviewees. IV multi-sectoral integration are achieved for livelihoods programming, followed by governance and health programming. Bayesian change models allow us to systematically vary the assumed output achievement and compare expected results. Such comparative scenario analyses show that the added benefit of multi-sectoral programming often increases, the lower the average output achievement across the activity is. This means that in challenging contexts, where output achievement is suboptimal, interventions can substitute for each other across sectors in achieving core behavior change. • Integrated programming boosts the efficiency and reach of single-sector programming. • Integrated programming can act as an insurance policy in challenging contexts where interventions are able to substitute for each other in achieving core behavior change. Assessment Question (AQ) 1: How do Mission work culture and leadership influence collaboration across offices? There is a strong awareness of the strategic importance of integration among USAID/DRC activity managers. Activity managers are also aware of the Mission leadership’s high expectations regarding collaboration. Regular information exchange between offices at the activity manager level is understood as a cornerstone for the development of a collaborative culture. However, our key informants believe that today, regular information exchange between activity managers is still insufficient. This results in missed opportunities to collaborate. Institutional barriers hamper the development of a collaborative culture at the Mission. They are: (1) Differences in project cycles that lead to misaligned activity designs, (2) inconsistent use of award and procurement language that would incentivize collaboration, (3) delays in getting approval for design changes that enable collaboration, and (4) high workload that prevents activity managers from identifying opportunities to collaborate. • Integration occupies a prominent place in the Mission’s organizational culture. • While improving, collaboration culture is still marked by insufficient regular information exchange between activity managers. AQ 2: How do technical offices distribute roles and responsibilities in cross-office activities? Currently, the Mission’s major multi-sectoral activity, the Partnership for the Development of Eastern Congo (P-DEC), is jointly funded by several offices. The AOR acts as an information broker and convenor for actors inside the Mission and the prime implementing partner. Communication between offices was rarely multi-lateral. Instead, each office communicated with the implementing partner about indicators and expected progress in separate discussions. The lack of direct communication between offices seems to have hampered the development of a shared vision for the success of P-DEC and contributed to protracted negotiations around the AMELP and first￾year work plan. The lengthy and controversial discussions seem to have left the collaborating offices with the impression that continued support of the joint activity is in question. • The Mission’s experiences with P-DEC highlight the challenges of inter-office collaboration. • Direct, multi-lateral communication between offices appears to be essential for success. AQ3: How does the Mission collaborate with implementing partners to further integration? When it comes to furthering integration, some activity managers clearly see themselves in the driver’s seat, while others suggest rather than insist on collaboration. The different interpretations of what an activity manager can do to further collaboration contribute to frustrations between activity managers. Activity managers take three different roles in supporting collaboration: They initiate, facilitate, and monitor. Activity managers facilitate ongoing collaborations by resolving issues around communication and information sharing among implementing partners and finding solutions around results attribution. • Activity managers interpret their latitude in promoting integration differently. • Activity managers play essential roles in supporting collaboration, including initiation, facilitation, and monitoring. AQ4: How do multi-sectoral activities achieve effective integration? The IPA identified five advantages of “integration by design,” i.e., through the implementation of multi-sectoral approaches: V (1) A “thick” local presence that allows an activity to deeply engage with the social fabric of a target community, build trust, and identify local champions; (2) detailed alignment of interventions across sectors so that interventions with inherent trade-offs can be carefully balanced; (3) systemic engagement with local communities to achieve lasting improvements in gender equality; (4) using economic interventions as an incentive and platform to engage community members in non￾economic activities; (5) ability to use governance interventions to systematically work on the sustainability of local results. • “Integration by design,” i.e., through the implementation of multi-sectoral activities, has distinct advantages over “integration post design,” i.e., through the collaboration of activities. AQ5: How does collaboration between activities contribute to effective integration? The IPA identified six types of cross-sectoral effects sought in inter-activity collaboration: Increasing reach, gaining access to communities, building on relationships of trust, complementing interventions, joint monitoring, and covering emerging topics. The first four effects increase the efficiency of activities by taking advantage of the fact that an activity is already working in a community that another activity seeks to enter. The last two effects profit from the complementarity of activity objectives. Among the preconditions for achieving cross-sectoral collaboration are flexibility in design and implementation, substantial overlap in the target area and implementation period, complementary results attribution, and alleviating concerns regarding competitive disadvantages. These preconditions are complicated to achieve in practice, due to misaligned project cycles, incompatible reporting requirements, and concerns of implementing partners. • By collaborating, activities increase the efficiency of interventions by building on previous community work or seeking synergies with complementary interventions. • Preconditions for successful collaboration are complicated to achieve in practice. Recommendations (AQ 6) USAID/DRC Should Increase the Use of “Integration by Design:” The assessment focused on two forms of integration: “integration by design,” i.e., through the implementation of multi￾sectoral activities, and “integration post design,” through collaboration between single-sector activities. The assessment’s findings and conclusions suggest that “integration by design” is a more effective and efficient option to achieve integration. USAID/DRC Should Strengthen Cross-Office Information Exchange in Geographic Clusters: Activity managers underlined the importance of learning about the work of other technical offices in sufficient detail so that opportunities for collaboration and potential synergies between activities can be identified. Given the importance of geographic overlap for successful inter￾activity collaboration, we believe that meeting in geographic clusters, e.g., provincial working groups, is the most efficient way to facilitate cross-office information exchange for increased integration. USAID/DRC Should Standardize Practices in Support of Integration: The assessment’s findings and conclusions suggest that collaboration between activities could be supported by standardizing Mission practices around integration. The Mission should promote the use of language in award and procurement that incentivizes collaboration between activities. Activity managers should be informed about best practices in facilitating collaboration and communication with implementing partners. The Mission should also ensure that information supporting collaboration is more readily available to activity managers by revising handover practices and informing activity managers how to access information about activities in relevant target zones. USAID/DRC Should Make Use of Participatory Adaptive Modeling to Facilitate the Adaptive Management of Multi-Sectoral Approaches: Activities with integrated, multi￾sectoral approaches are challenging to manage. The RFSAs currently lack an effective means to systematically evaluate context changes, interventions, and adaptive measures on an ongoing basis, so that they could streamline their approach over time. The current assessment used PAM for an end￾of-project review, but the tool may also be used on an ongoing basis during implementation. PAM VI would provide integrated, multi-sectoral activities with the possibility to manage the complexity of their approaches and home in on effective interventions and adaptive measures. Using PAM to identify levers, bridges, and ultimate outcomes helps activity leadership to adapt programming to focus on key interventions and changes in context when managing complex activities. Scenario analyses of Bayesian change models would allow activity leadership to systematically compare the likely effects of alternative courses of action before making decisions based on the collective knowledge and experiences of their staff and partners. 1 1. BACKGROUND Cross-sectoral Integration is one of the three strategic pillars of the current Country Development Cooperation Strategy (CDCS) of the USAID Mission to the Democratic Republic of the Congo (DRC). The integration of development and humanitarian interventions to simultaneously “address emergency situations and the factors that precipitate them” (USAID/DRC 2020) is a particular focus of the CDCS. This integration is sought particularly in key provinces in Eastern DRC where the Mission strives to "mitigate the effect of a broad range of shocks by building more resilient systems and communities and stimulating more reliable and higher-quality basic services” (USAID/DRC 2020). Three Types of Integration Cross-sectoral integration is understood as the intentional sequencing and layering of interventions from different development sectors (USAID 2022). USAID/DRC engages in three types of cross￾sectoral integration. The first two types are “integration by design” where the Mission designs and implements activities with a multi-sectoral approach. A multi-sectoral approach combines interventions that are typically assigned to different development sectors. For example, resilience activities often combine interventions designed to improve household income with Water, Sanitation, and Hygiene (WASH) or nutrition interventions. Among activities with a multi-sectoral approach are activities implemented by a single office (Type 1) and activities jointly financed and managed by several offices (Type II). Examples of the first type of “integration by design” are the Resilience Food Security Activities (RFSAs) of the Bureau for Humanitarian Assistance (BHA). BHA is currently implementing two RFSAs in the DRC: the Resilience Food Security Activity in Kasaï Oriental (Budikadidi) and the South Kivu Food Security Project (FSP-Enyanya). Both RFSAs include agricultural interventions to increase household incomes, governance interventions, and interventions to improve health, nutrition, and WASH. An example of the second type of “integration by design” is the Partnership for the Development of Eastern Congo (P-DEC). P-DEC takes a resilience and systems-strengthening approach to peacebuilding. The activity combines interventions to improve local governance, especially through the strengthening of community-level institutions, with economic interventions developing agricultural value chains and interventions promoting environmental conservation. P-DEC’s Agreement Officer's Representative (AOR) is attached to the Peace and Security Office (PSO). Central Africa Regional Program for the Environment (CARPE), the Democracy, Human Rights, and Governance Office (DRG), the Economic Growth Office (EG), and the Education office contributed to the design, provide funding, and are involved in the monitoring of P-DEC. Type III cross-sectoral integration takes place post-design when technical offices incentivize activities from different sectors to coordinate and collaborate. An example of this type of cross-sectoral integration is the collaboration between the Integrated Health Program (IHP) and BHA, DRG, and EG activities. Types I and II provide similar opportunities for a deep integration of interventions under the leadership of the same implementing partner. In line with its focus on the community-level benefits of integration, the IPA concentrated its efforts on investigating types I and III where the operational conditions for a successful integration at household level are very different. Previous Studies of Integration In 2020, the Monitoring, Evaluation, and Coordination Contract (MECC) conducted a “Development Objective 2 (DO2) Assessment” that investigated among other things whether “geographic synergy and a coordinated approach to programming yield better development results.” To answer the question, the study team compared the effectiveness of integrated versus non-integrated activities 2 based on average indicator target performance. It found that average target achievement was lower for integrated activities and concluded that non-integrated activities perform better. However, there are important questions regarding the meaningfulness of these numbers. First, integrated and non-integrated activities differ in more than just their level of integration. They are implemented in different target regions with different objectives by different implementing partners. Indicator target performance is hardly comparable across such different contexts. But even if they were, differences in average performance could be a result of any design differences. Second, the study authors argue that integrated activities are larger in scope and have more indicators, so target setting and monitoring are more complex. They conclude that differences in average target achievement might therefore reflect the fact that setting realistic targets is more challenging for integrated activities than for non-integrated activities rather than reflecting actual differences in performance. More importantly, the study’s approach leaves open how integration influenced differences in performance. It remains unclear how the Mission could prevent negative consequences or effectively promote any positive effects of integration if integration was the cause for differences in performance. The IPA design ensures that findings on the link between integration and performance are plausible and robust and can support recommendations on how to improve the effectiveness of integration that are concrete and actionable. To achieve that, the IPA follows two design principles. First, the IPA studies intra- and inter-activity integration based on the experiences of four representative activities: Resilience Food Security Activity in Kasaï Oriental (Budikadidi), South Kivu Food Security Project (FSP-Enyanya), Integrated Health Program (IHP), and Management of Fall Armyworm in Maize for Smallholder Farmers in the Democratic Republic of Congo (Fall Armyworm Activity). Case studies of these activities provide an in-depth understanding of the implementation processes to show how integration can increase the effectiveness of activities. The first two activities are Resilience Food Security Activities (RFSAs) by the Bureau for Humanitarian Action (BHA). They are examples of cross-sectoral, intra-activity integration “by design.” The latter activities are single sector. They were selected because they collaborate with the RFSAs (IHP) or use comparable approaches without multi-sectoral support (FAW activity). The focus of the assessment is case studies of Budikadidi and FSP-Enyanya using Participatory Adaptive Modeling (PAM). Studying these complex, integrated BHA activities using PAM provides IPA with the ability to focus on integration at the nexus of humanitarian-to-development contexts and to focus on “on-the-ground” integration, without the added complexity of inter-office coordination and collaboration. PAM is an analytical approach that ensures that comparisons between integrated and non-integrated programming are meaningful and systematic. It enables the assessment team to disassemble the multi-sectoral approaches of the two BHA activities into their sectoral components. This allows the team to compare outcome achievement under the assumption that only interventions of one sector were implemented with outcome achievement when the full, multi-sectoral approach is used. For these scenario-based comparisons, target regions, objectives, and implementing partners stay the same; only the level of integration varies. This enabled the assessment team to determine the exact extent of the “integration dividend.” Both activities include collaboration with IHP in their implementation approach. Studying the successes and challenges of the collaboration between Budikadidi, FSP-Enyanya, and IHP from the 3 perspectives of each implementing partner provided the assessment team with an opportunity to understand the drivers of successful inter-activity integration in concrete implementation contexts. As part of their multi-sectoral approaches, the BHA activities train farmers in agricultural techniques to improve yields and increase household incomes. The Fall Armyworm Activity trains smallholder maize producers in good agricultural practices and integrated pest management to increase maize yields. Comparing the experiences of the three activities regarding these shared approaches and objectives allowed the assessment team to better understand the differences in efficiency and effectiveness between integrated and non-integrated programming. Second, the IPA incorporates the perspectives of the Mission’s technical offices and implementing partners involved in cross-sectoral integration. While the IPA’s focus is on the integration of activities in the field, the inclusion of the Mission’s perspective is vital because successful integration begins with the collaboration between technical teams at the Mission in the design, procurement, and monitoring of activities. This collaboration determines the incentives and constraints for cross￾sectoral integration on the ground. The Two BHA RFSAs The experiences of Budikadidi and FSP-Enyanya in implementing complex multi-sectoral approaches on the local level provides an important context for understanding the advantages and challenges of “integration by design” that are discussed in this report. The following paragraphs provide a short overview of the activities’ program logic. The activities’ overall goal is to improve food and nutrition security and the economic well-being of vulnerable households. The multi-sectoral approach of these activities includes agricultural interventions to increase household incomes, interventions in health, nutrition, and WASH, and interventions to improve local governance and gender equality. The internal breakdown of the RFSA’s multi-sectoral approaches into purposes and sub-purposes differs, but follows the familiar outlines of governance, livelihoods, and health programming. Gender is treated as a cross-cutting topic. In terms of community-level collaboration partners and change processes, gender is most closely associated with governance activities. This is why the assessment treats gender interventions as part of the activities’ governance programming. The design of individual interventions, community groups, and training content under each sector differs as well, but follows similar causal logics. Important differences will be mentioned for each sector separately. Under governance programming, the RFSAs work with local institutions such as Community Action Committees (CAC) on the coordination of community development activities and engage in a number of gender-related interventions. FSP-Enyanya also works with local institutions to support small-holder farmers in acquiring land titles. Gender-related interventions involve activities with community leaders, couples, fathers, and youth. Gender-related training is also delivered to savings and farmer groups. Gender programming has the goal to include women in decision-making on the community, group, and household level and reduce gender-based violence. Livelihoods programming revolves around group savings, agricultural production, and income￾generating activities. Both RFSAs also supported the establishment of tree nurseries. Savings groups aim to provide households with access to credit for investments in agricultural inputs and income￾generating activities. Activities to improve agricultural production include the provision of training on good agricultural practices to farmer groups and supporting their access to input and output markets. Health programming focuses on nutrition and water, sanitation, and hygiene (WASH). To improve a household’s nutritional status, the RFSAs promote permagardening and the raising of small animals such as goats and rabbits. They also support the organization of mothers in Care Groups to train 4 them on maternal, infant, young child, and adolescent nutrition (MIYCAN). Topics such as birth￾spacing are also covered. Community agents are trained to detect and refer cases of child malnutrition. WASH is promoted on a communal and household level. WASH-related interventions include supporting water committees in maintaining functioning water points and supporting households in improving hygiene practices, with a final goal to achieve open defecation free status for target communities. Detailed Theories of Change for the two RFSAs can be assessed through the following links: • Budikadidi final action-focused TOC: https://kumu.io/embed/039cc03da9d294399ac2c522f2e2568d (password: #IPA2023) • FSP-Enyanya final action-focused TOC: https://kumu.io/embed/83c5dd8fd9e7a3b77c0a4c2b8b8e94e8 (password: #IPA2023) 5 2. ASSESSMENT DESIGN 2.1 PURPOSE AND QUESTIONS The IPA's purpose is threefold. First, it was designed to support the Mission in reviewing the current Country Development Cooperation Strategy (CDCS). Integration is one of the three strategic pillars of the current CDCS. A particular focus is the integration of development and humanitarian interventions to simultaneously “address emergency situations and the factors that precipitate them” (USAID/DRC 2020). This integration is sought particularly in key provinces in Eastern DRC where the Mission strives to "mitigate the effect of a broad range of shocks by building more resilient systems and communities and stimulating more reliable and higher-quality basic services” (USAID/DRC 2020). To support the Mission in reviewing the current CDCS, a presentation of preliminary findings was held at the Mid-Course Stock Taking (MCST) that addressed the question to what extent the implementation of the strategy’s integration pillar contributed to progress on Development Objectives (DOs). Second, the IPA contributes to the Mission’s CDCS learning agenda by furthering the Mission’s understanding of how integration improves USAID’s work within and across offices. Third, the IPA supports the Mission in realizing the full potential of integration by distilling recommendations on how to facilitate integration that are concrete and actionable, so that they can inform CDCS strategy adaptation and collaborative practice. What is Cross-Sectoral Integration? Cross-sectoral integration is understood as the intentional sequencing and layering of interventions from different development sectors (USAID 2022). The IPA differentiates between three types of cross-sectoral integration. Intra-activity cross-sectoral integration is cross-sectoral integration “by design.” It occurs when multi-sectoral activities are designed and implemented that combine interventions that are typically assigned to different sectors. For example, a resilience activity might combine interventions that aim at improving household income with WASH or nutrition interventions. Among activities with a multi-sectoral approach are activities implemented by a single office (type 1) and activities jointly financed and managed by several offices (type II). Inter-activity cross-sectoral integration (type III) takes place post-design when activities of different offices successfully collaborate with each other. The IPA addresses both inter-activity and intra-activity cross-sectoral integration and incorporates the perspectives of the Mission’s technical offices and implementing partners. Assessment Questions The IPA scope of work (SOW) contained six clusters of research questions and a question focusing on actionable recommendations that further the effectiveness of integration. For each cluster of research questions, the assessment team formulated a main assessment question that covers the underlying theme of the cluster (set in bold print in the list below). The cluster questions are understood as additional indications of how to address the main theme and are addressed in the course of answering the main assessment question. To answer the last question listed in the SOW, the assessment team provides a list of actionable recommendations in a separate chapter. The current CDCS and the assessment purpose both assume that integration is beneficial. Consequently, most assessment questions focus on how integration can be best facilitated. The Mission leadership emphasized a fundamental strategic question: Should the Mission invest efforts in increased cross-sectoral integration? Is integrated programming more effective than single-sector programming? To better meet this interest, the assessment team decided to take a more comparative approach to answering question 6 of the original SOW: To what extent does integration improve the effectiveness of activities? This strategic question will be discussed first throughout the report. 6 • Strategic Question: To what extent does multi-sectoral integration improve the effectiveness of activities? How do intra- and inter-activity integration influence the effectiveness of activities and their contributions to CDCS Objectives? How much does cross-sectoral integration add to the effectiveness of interventions? Which sectoral interventions are understood to profit most strongly from cross-sectoral integration, which less so? • Assessment Question 1 (AQ1): How do Mission work culture and leadership influence collaboration across offices? How is the Mission fostering and sustaining a culture internally across teams to operationalize and sustain the integrated approach? Have old organizational behaviors been modified or new behaviors initiated to reinforce the integrated approach? What role has leadership at the Mission or office level played in shaping that internal culture and any changes generated as a result? • AQ2: How do technical offices distribute roles and responsibilities in cross-office activities? How are the responsibilities and coordination of management and decision-making authority for activity design, procurement, and monitoring shared across teams? How and when do those teams work together, and how are concerns or disputes resolved? How do they identify and determine the attribution of results achieved through joint efforts contributing to an integrated DO? • AQ3: How does the Mission collaborate with implementing partners to further integration? How have relationships and roles between the Mission's internal teams and their implementing partners been defined and operationalized for any cross-cutting activities? Have any new behavioral practices been adopted across these organizational relationships to facilitate implementation? • AQ4: How do multi-sectoral activities achieve effective integration? At which points in their program logic do integrated activities seek cross-sectoral integration? Which cross-sectoral effects do they hope to achieve? How do actors at such points of integration (staff, community volunteers, community members) collaborate? Which challenges do they face and how do they overcome them? • AQ5: How does collaboration between activities contribute to effective integration? How do activities from different sectors collaborate? How do they share information, sequence or layer interventions, or jointly plan interventions? Which cross-sectoral effects do they hope to achieve? How do actors from different activities collaborate? Which challenges do they face and how do they overcome them? • AQ6 (Recommendations): How could integrated programming better contribute to the effectiveness and efficiency of integrated activities and strengthen their contributions to CDCS Objectives? 2.2 APPROACH The IPA employs a mixed-method approach including Participatory Adaptive Modeling (PAM) and additional qualitative data collection and analysis. PAM is used to conduct in-depth studies of the complex, integrated approaches of the BHA RFSAs Budikadidi and FSP-Enyanya. Additional qualitative data collection and analysis target the experiences of IHP and the FAW activity, as well as the perspectives of Mission AORs and CORs. Participatory Adaptive Modeling PAM is a participatory process that uses the experiential knowledge of implementing partner (IP) staff at all levels and across sector teams to produce a comprehensive model of an activity’s approach to generating change among its target populations. The model contains an activity’s full results framework. It shows how strongly each intervention is expected to contribute to the activity’s objectives across intermediate results and final outcomes. A comprehensive change model that includes information on the strength of contributions is called a Bayesian change model (see textbox “What is a Bayesian change model?”). The change models allow the assessment team to 7 disassemble complex, multi-sectoral approaches into sectoral components and to determine the exact extent of the “integration dividend.” The Bayesian change models for the two BHA RFSAs were built in three steps. First, for each activity, the assessment team and IP staff jointly produced a detailed, action-focused Theory of Change that contains all interventions, outputs, outcomes, and impacts as individual elements. The TOC shows which of these elements are connected. For example, it shows which outputs an individual intervention is expected to produce and to which outcomes and impacts the resulting outputs are expected to contribute. The TOCs that result from our process encompass a far greater number of interlinked cause-and-effect relationships than traditional logical frameworks and lengthy “if, then” statements. What is an action-focused Theory of Change? An action-focused Theory of Change visualizes an activity’s change processes as causal chains. Each step is formulated as an action taken by a stakeholder of the activity. For example, the outcome “Agricultural production increased” is included in an action-focused TOC as “farmers increase agricultural production.” Formulating outputs and outcomes in that way ensures that responsible actors are clearly named. Outputs and outcomes that do not represent observable behavior are omitted. For example, “information acquired” or “awareness increased” is not included in an action-focused TOC. This is done for two reasons: First, stakeholders cannot meaningfully discuss their observations regarding changes in information or norms without referring to observable behavior. Second, changes in information or norms are always a means to an end. Every tangible improvement in living conditions requires a change in behavior. Including non-observable change in the TOC would therefore complicate the model without adding to the model’s usefulness for discussing and understanding change. The detailed, action-focused TOCs for Budikadidi and FSP-Enyanya are hosted on kumu.io as interactive flowcharts. They can be assessed through the following links: • Budikadidi final action-focused TOC: https://kumu.io/embed/039cc03da9d294399ac2c522f2e2568d (password: #IPA2023) • FSP-Enyanya final action-focused TOC: https://kumu.io/embed/83c5dd8fd9e7a3b77c0a4c2b8b8e94e8 (password: #IPA2023) The first TOC draft was based on a thorough review of the available activity documentation. The draft TOC was visualized as an interactive logic flowchart. The assessment team validated the TOC in workshops with IP staff. In sector-specific workshops, the team invited IP staff to “take walks” through the TOC visualization along change processes in their area of expertise and identify missing steps and missing links. Whenever a walk came across a result with several contributing factors, participants were asked to rank the factors according to their importance. If these factors and their contributions resulted from interventions under different sectors, participants were asked to elaborate on how collaboration between different sector teams is taking place to achieve the joint result. The team organized a cross-sectoral TOC validation workshop to discuss how interventions are expected to produce joint effects at cross-sectoral junctures in the intervention logic. The validated, action-oriented TOCs were developed and shared as interactive logic flowcharts on kumu.io. The assessment team analyzed the causal structure the action-focused TOCs to identify interventions and outcomes that tie the overall integrative approach together. These are interventions with a high number of immediate contributions (“levers of integrated change”), intermediate outcomes with a high betweenness centrality (“bridges of integrated change”), and ultimate outcomes that make no further contributions to other behavior changes (see textbox “What are core behavior changes?”). Annex IV provides a list of bridges and ultimate outcomes for both activities. What are core behavior changes? There are three types of core behavior changes in the action-focused TOC of an integrated activity: Levers, bridges, and ultimate outcomes. Annex IV 8 provides a list of bridges and ultimate outcomes for each RFSA. • Levers of integrated change are interventions with a high number of immediate contributions. For example, under FSP-Enyanya, Lead Mothers providing training to Care Group members contributes to households maintaining permagardens, households adopting improved WASH practices, and four other target behavior changes. • Bridges of integrated change are behavior changes with a high betweenness centrality. Betweenness centrality measures how many times an element lies on the shortest path between two other elements in a causal network. In a radial network graph, such elements would tend to be found in the middle, on the shortest path between different “arms” of the network. In an action-focused Theory of Change, intermediate outcomes with a high betweenness centrality are behavior changes that connect a high number of interventions with a high number of desired outcomes. In the case of Budikadidi, couples making joint decisions on family planning, nutrition, and hygiene is such a bridging behavior change. There are six interventions that Budikadidi expects to contribute to this change, and there are six outcomes that joint decision-making is expected to positively affect, ranging from peaceful conflict resolution in the household to practicing birth spacing. • Ultimate outcomes are behavior changes that make no further contributions to other behavior changes. An activity’s goal-level results typically include all ultimate outcomes. However, there are often goal-level results that make contributions to other goals. For example, FSP-Enyanya’s poverty reduction goals are a means to improving food security. Studying ultimate behavior change outcomes provides a more focused view on what success ultimately means for an activity. Focusing on levers, bridges, and ultimate outcomes provides a way to embrace the complexity of the RFSA multi-sectoral approaches and at the same time separate the wheat from the chaff. This allowed the assessment team to focus analyses on a smaller set of core behavior changes to track overall results achievement. Figure 2 – Eliciting Contribution Estimates In the second step, the assessment team infused the TOC model structure with information on how strongly a step in a change process is expected to contribute to the next. The assessment team convened sector-specific pause and reflect workshops to understand to which degree interventions produce the desired effects, and what role cross-sectoral integration plays in results achievement. In these workshops, IP staff estimated the strength of contributions based on their experiences in implementing the activity. Figure 2 shows how straightforward questions asked in a workshop – Out of 10 VSLA members, how many take out loans to start small businesses?” – leads to numerical estimates that are included in the model. The textbox “How are effects elicited?” provides additional detail on the process. By providing their own numerical estimates of cause and effect in different 9 intervention scenarios, IP staff were able to communicate expectations across sectoral teams. How are effects elicited? The goal of effect elicitation is to provide a numerical estimate for the contribution of one element in an action-focused TOC to achieving another. These estimates are based on the experiences and knowledge of IP staff. For example, to estimate the contribution of training Village Savings and Loan Associations (VSLAs) to investments in small business activities, VSLA trainers and other field-level staff are asked in workshops: “Based on your experience, out of 10 VSLA members, how many take out loans to start small businesses if you support their VSLA with training? And how many take out loans if you do not support their VSLA with training?” The estimates are interpreted as percentage likelihoods, i.e., if 6 out of 10 VSLA members are estimated to take out loans to start a small business if their VSLA received training, then this is understood as a 60% likelihood that an average VSLA member would take out such a loan in this scenario. The difference between the estimates for the scenarios “training is provided” and “training is not provided” is understood as the strength of the contribution of VSLA training on investments in small business activities. For example, the contribution of VSLA training to new business investments could be 30%, meaning that the likelihood that an average VSLA member takes out a loan to start a business is expected to be 30% higher if the VSLA training was provided compared to if it was not. Contribution estimates were elicited for each causal link and then chained together to build a comprehensive change model. For example, IP staff from Budikadidi provided estimates for the contributions of VSLA training to investments in small business activities and for the contributions of these investments to VSLA members engaging in income-generating activities. These estimates were then linked, so that the ripple effect of VSLA training on income-generating activities could be calculated (see Figure 3). The assessment team asked IP staff to provide estimates for all causal steps in the action-focused TOC. The resulting Bayesian change models provide a comprehensive picture of the BHA RFSA's intervention logic and expected effects, based on the best available knowledge. Figure 3 – Chaining Contribution Estimates In the example provided in Figure 3, the probabilities can be multiplied: If 100% of VSLA members received training, we expect 60% of VSLA members to take out loans to start small businesses. Of the members taking out loans, 50% are expected to engage in income-generating activities, or 60% x 10 50% = 30% overall. If the training was not implemented, 20% of VSLA members would be expected to take out a loan to start a small business, and 10% of those members would be expected to engage in income-generating activities, so 20% x 10% = 2% overall. (The second calculation is not pictured in Figure 3.) The Bayesian change model for Budikadidi contains 79 elements and 134 connections, while the model for FSP-Enyanya contains 77 elements and 107 connections. For each activity, more than 40 elements are separate interventions by staff and community volunteers that aim to contribute to 36 (FSP-Enyanya) and 42 (Budikadidi) behavior changes among local community members. Figure 4 shows the Bayesian change model for Budikadidi. The cut-out on the right side highlights how the information on contributions is stored in a Bayesian change model. The textbox “What is a Bayesian change model?” provides more detail. Figure 4 – Comprehensive Bayesian Change Model for Budikadidi What is a Bayesian change model? A Bayesian change model builds on a detailed action￾focused Theory of Change (see textbox “What is an action-focused Theory of Change?”). An action-focused TOC presents an activity’s change processes as distinct causal chains, with each link representing an action by a stakeholder of the activity. In the model, colors represent types of activity stakeholders. Grey stands for activities by field-level staff, red indicates activities by community volunteers, and blue is used for actions taken by target groups. The percentage for “Present” reflects the likelihood that a given activity or action is carried out. The degree to which changes in a contributing activity cause changes in a dependent action reflects the strength of the contribution the activity makes to the dependent action. The example highlighted by the cut-out in Figure 4 shows that if field agents provide training to Savings and Internal Lending Communities (SILCs) to 25% of the target value, then IP staff expect 27% of SILC members to take out loans to start small businesses. If a higher value for training output is chosen, then the percentage of SILC members taking out loans increases accordingly. This is how the model estimates the strength of the contribution of SILC training to small business investments. In a third step, the resulting Bayesian change model was used to calculate the expected results achievement of an activity for different scenarios, including scenarios where an activity is only partially implemented. For each BHA RFSA, the assessment team calculated the expected results achievement under the assumption that the RFSA’s complex, multi-sectoral approach was fully implemented and compared the calculated results to the expected results achievement under the assumption that only the interventions of one of its sectors were implemented. By comparing these 11 scenarios, precise estimates of the added benefits of integration were determined. Why conduct scenario analyses? To identify the added contribution of a multi-sectoral approach to joint outcomes, it is important to understand what would have happened if the activity hadn’t been implemented as a multi-sector activity. If the assessment compared a single￾sector activity to a multi-sector activity, a lot of things would vary: Target region, implementation period, implementing partner, target beneficiaries, budgets, etc. If differences in outcome achievement between a single-sector and a multi-sector activity were detected, it would be impossible to single out the causes of these differences. By building a comprehensive change model of a multi-sector activity and running scenarios where different parts of the activity are assumed to be implemented while others are not, the assessment can conduct a systematic comparison of single-sector and multi-sector programming using the experiences of the same multi-sector activity and holding constant time, place, and stakeholders. Thus, the assessment team can meaningfully and systematically compare single-sector programming with integrated programming. Through running different scenarios, the multi-sectoral approaches of the two BHA activities are disassembled into their component parts and their effects are measured separately. The assessment team can then compare results achievement under the assumption that only interventions of one sector were implemented with outcome achievement when the full, multi￾sectoral approach is used. This enables the assessment team to determine the exact extent of the “integration dividend.” 2.3 DATA COLLECTION AND ANALYSIS The assessment engaged in two types of data collection and two types of analyses. The types of data collection are workshops (TOC validation workshops and pause and reflect workshops) and key informant interviews with IHP, FAW, and AORs/CORs. The types of analyses are content analysis of workshops and interviews, and scenario analysis of the Bayesian change models for the BHA RFSAs. Data Collection Workshops The assessment team held a total of 16 workshops with BHA RFSA IP staff at different levels, including purpose leads, thematic specialists, and field-level staff. The workshops took about 2.5 hours each, were held online, and were attended by four to ten participants. Introductory presentations were held in English and French. The discussions were held in French. Annex II provides detailed workshop designs. Annex III provides a list of participants across workshops. For each BHA RFSA, eight workshops were held: Three sectoral TOC validation workshops, one cross-sectoral TOC validation workshop, three sectoral pause and reflect workshops, and one cross-sectoral pause and reflect workshop. For the sectoral workshops, governance, livelihoods, and health teams of the activities met separately. The main purpose of the sectoral TOC validation workshops was to make sure that the draft TOC is complete, remaining questions on actors, the nature of actions, and their purpose are clarified, it is understood how teams collaborate across sectors whenever the TOC foresees such collaboration, and contributing factors to joint effects are ranked from most to least influential. For the sectoral TOC workshops, the assessment team reviewed activity documentation and prepared a visualization of the draft TOC as an interactive flowchart on kumu.io. The main purpose of the cross-sectoral TOC validation workshop was to focus on the experiences with cross-sectoral collaboration at major points of integration. Such points of integration are results (outputs, outcomes, and impacts) to which different sector teams contribute with separate interventions. Participants were asked to discuss each sector team’s contributions, describe challenges in the collaboration, suggest improvements to their collaborative process, and identify additional opportunities for collaboration in achieving joint results. 12 The main purpose of the sectoral pause and reflect workshops was to produce numerical estimates for the strengths of contributions along change processes. The assessment team prepared Excel files that provided a tab for each result in the TOC. These results are action-focused reformulations of outputs, outcomes, and impacts (see textbox “What is an action-focused TOC?”). The tabs contained the effect elicitation questions and tables to record numerical estimates for the strengths of contributions to each result (see textbox “How are effects elicited?”). Contribution strengths were discussed one by one. The sectoral workshops focused on results that did not require cross￾sectoral collaboration. The cross-sectoral pause and reflect workshops were held to discuss the contributions of sector teams to joint, cross-sectoral results and produce estimates for the strength of their contributions. Contributions to each cross-sectoral result were discussed in breakout groups that included representatives of each sector contributing to the result. Key Informant Interviews The assessment team conducted interviews with 19 key informants: Six IHP representatives, four FAW activity representatives, and nine USAID/DRC activity managers. The interviews lasted about 90 min each and took place online. The goal of the interviews with IHP and the FAW activity was to complement the perspectives of the BHA RFSAs on integration. IHP interviews focused on the experiences of IHP in collaborating with Budikadidi and FSP Enyanya. In the PAM process, the BHA RFSA implementing partner staff identified several points of collaboration with IHP. IHP staff were asked about their perspectives on collaboration with the BHA RFSAs around these points of integration. The KIIs with the FAW activity focused on aspects of the FAW activity’s approach similar to that of the BHA RFSAs. These similarities relate mainly to the activities’ approach toward working with farmers on the community level. Both the BHA RFSAs and the FAW activity cascade agricultural training down to the community level through field agents and local farmer groups. The main analytical focus was on the differences it makes to the implementation and effectiveness of agricultural training if an activity can focus entirely on such training or potentially benefit from operational synergies with interventions in the broad portfolio of a multi-sectoral approach. Interviews with AORs and CORs were conducted to broaden the perspective on integration gained through the BHA RFSAs. The main goal of the interviews was to capture the variety of activity managers’ experiences with cross-sectoral integration.3 This included their experiences with internal collaboration, i.e., between TOs, and external collaboration, i.e., with implementing partners involved in cross-sectoral activities. Interviews with activity representatives were held in French, while interviews with activity managers were held in English. Annex II provides the semi-structured interview guides. Annex III provides the list of interviewees. Data Analysis Content Analysis Key-informant interviews (KIIs) were transcribed, translated to English where necessary, and subjected to content analysis. Themes were identified, defined, and applied to the transcripts in an iterative process with the help of Freeplane, a mind-mapping software. 3 Throughout the report, „activity manager” is used as a general team for all Mission staff who are involved in the management of activities, including AORs, CORs, and their alternates. The umbrella term is also used to further protect the anonymity of interviewees. 13 In addition to the KIIs, all workshops held in the process of building and jointly discussing the Bayesian change models for the BHA RFSAs were recorded, transcribed, translated to English, and subjected to content analysis. The analysis of the workshops focused on the following topics: • Advantages and disadvantages of cross-sectoral integration. • How sectoral teams collaborate internally to achieve joint effects. • How the BHA RFSAs collaborate with external partners. • Challenges in internal and external collaboration and solutions to these challenges. • Opportunities to improve internal and external collaboration. Scenario Analysis The Bayesian change models were used to calculate the expected results of the activities for different scenarios. The assessment team ran more than 30 different scenarios of four types: • Extrapolated Baseline Scenario. Scenario assuming that no intervention produced any outputs throughout the implementation period of the activity. • Multi-Sector Scenarios. Scenarios assuming the full implementation of the RFSA’s complex, multi-sectoral approaches, with varying levels of output achievement (25%, 50%, 100%). • Single-Sector Scenarios. Scenarios assuming only the interventions of one sector (governance, health, livelihoods) are implemented, with varying levels of output achievement (25%, 50%, 100%). By comparing multi-sector with single-sector scenarios at the same level of output achievement, precise estimates of the added benefits of integration were determined. • Individual Core Behavior Scenarios. Scenarios assuming only one core behavior change is achieved. These scenarios demonstrate the ripple effects of a single change throughout the results framework. Annex V provides more details on the results of the first three types of scenarios. Results for the fourth type of scenario can be provided upon request. 2.4 LIMITATIONS • The IPA addresses both inter- and intra-activity cross-sectoral integration. The focus is on Type I and Type III integration, i.e., integration through multi-sectoral activities implemented by the same office and integration through the collaboration of activities managed by different technical offices. Regarding Type II integration (multi-sectoral activities jointly managed by several technical offices), the analysis rests on an interview covering the AOR’s experiences with managing P-DEC. Findings on Type II integration are discussed in section 3.3. • The IPA incorporates the perspectives of the Mission’s technical offices and implementing partners involved in cross-sectoral integration. Mission collaboration culture and practices are examined primarily from the perspective of activity managers. Mission and technical office leadership were not among the key informants. The IPA’s answers to the Mission￾focused assessment questions do therefore not provide a comprehensive assessment of leadership initiatives or recent changes in organizational practices. Instead, they reflect the awareness and current practices of activity managers that are directly involved in facilitating integration. • Due to time and resource constraints, the assessment team was unable to make full use of monitoring data in triangulating effect estimates by implementing partner staff that entered the Bayesian change models for the BHA RFSAs. While the assessment team is confident that effect estimates provide meaningful insights into the relative strength of contributions to results, social desirability biases may have increased absolute values of achievement probabilities across the board. This means that scenario analyses can be expected to provide precise estimates regarding the added benefits of integration but the scenario’s projected results achievement probabilities should not be interpreted in absolute terms. 14 3. FINDINGS 3.1 STRATEGIC QUESTION: THE INTEGRATION DIVIDEND Strategic Question: To what extent does multi-sectoral integration improve the effectiveness of activities? How do intra- and inter-activity integration influence the effectiveness of activities and their contributions to CDCS Objectives? How much does cross￾sectoral integration add to the effectiveness of interventions? Which sectoral interventions are understood to profit most strongly from cross-sectoral integration, which less so? The current USAID/DRC CDCS assumes that cross-sectoral integration provides key advantages in tackling the DRC’s complex development problems. Each development objective (DO) is designed around a multi-sectoral approach to a key development challenge, and the CDCS calls for using “joint results frameworks, common indicators, collaborative work planning, cross-sectoral financing, and/or multi-sectoral project and activity designs” (USAID 2020) to operationalize them. In line with this thinking, the IPA purpose and most assessment questions focus on how integration can be best facilitated. However, in our preliminary discussions of the SOW, Mission leadership emphasized a fundamental strategic question: Do the benefits of integration justify the Mission’s efforts in facilitating cross￾sectoral integration? Is integrated programming more effective than non-integrated programming? Measuring the benefits of integration is not easy. To identify the added contribution of a multi￾sectoral approach to activity effectiveness, it is important to understand what would have happened if the activity had been implemented as a single-sector activity. If the assessment compared a multi￾sector activity to a single-sector activity, many things would vary, e.g., target region, implementation period, implementing partner, target beneficiaries, budgets, etc. If differences in outcome achievement between a single-sector and a multi-sector activity were detected, it would be impossible to determine whether integration made the difference. By building comprehensive Bayesian change models, the assessment team was able to dissemble the RFSAs’ multi-sectoral approaches into their component parts and measure their effects separately. Bayesian change models are based on the structure of action-oriented Theories of Change (see the textbox on page 7, “What is an action-focused Theory of Change?”). An action-focused Theory of Change visualizes an activity’s change processes as stepwise causal chains. Each step is formulated as an action taken by a stakeholder of the activity. To produce a Bayesian change model, the TOC model structure is infused with information on the strength of each causal connection (see the textbox on page 10, “What is a Bayesian change model?”). The strengths of causal connections are estimated by activity staff based on their experiences in implementing activity interventions in their field of expertise (see the textbox on page 9, “How are effects elicited?“). The Bayesian change models allowed the assessment team to estimate the contributions of integration by comparing different implementation scenarios for the RFSAs. The assessment team compared the expected results that are achieved when the implementation of the RFSAs’ full multi￾sectoral approach is assumed with the expected results achieved when the interventions of only one sector are implemented. For example, if multi-sectoral integration made a positive contribution to the achievement of livelihoods results, then the comparison would find that livelihood outcomes are achieved with a higher probability when governance and health interventions are implemented in addition to livelihoods programming. Thus, the assessment was able to conduct a systematic comparison of single-sector and multi-sector programming using the experiences of the same multi-sector activity and holding constant time, 15 place, and stakeholders. This enabled the assessment team to determine the exact extent of the “integration dividend,” i.e., the differences in the results achieved by integrated cross-sectoral programming versus single-sector programming. The assessment team planned to also use the model for estimating the contributions of inter-activity integration to the RFSAs’ effectiveness. This would have been based on a calculation of the changes in results achievement that collaboration with IHP causes across the RFSAs’ results frameworks. However, at the time of the assessment, neither Budikadidi, nor FSP-Enyanya actively collaborated with IHP. The estimated contributions would therefore not provide meaningful insights into the effects of inter-activity collaboration. The chapter proceeds in three steps. First, we will chart the territory of the complex, multi-sectoral RFSA approaches by identifying core behavior changes that tie the sectoral components together. An analysis of these behaviors sheds light on the roles different sectors play in the concert of the RFSAs. The set of core behavior changes also provides the basis for the comparison of average results achievement in comparative scenario analyses. Second, we report the results of a first set of scenario analyses. These scenarios show how multi-sector programming boosts the effectiveness and reach of single-sector activities. Third, we will discuss the results of scenarios with varying levels of output achievement. The comparative analysis shows that the added benefit of multi-sectoral programming often increases when the average output achievement across the activity is lower. Charting the Territory: Levers, Bridges, and Ultimate Outcomes Budikadidi’s and FSP-Enyanya’s multi-sectoral designs are complex, interdependent approaches to generating change among target populations. The assessment team identified more than 40 separate interventions by staff and community volunteers for each activity that aim to contribute to 36 (FSP￾Enyanya) and 42 (Budikadidi) behavior changes among local community members. However, not every element and every contribution are equally important. There are interventions and outcomes that tie the overall integrative approach together. These are interventions with a high number of immediate contributions (“levers of integrated change”), intermediate outcomes with a high betweenness centrality (“bridges of integrated change”), and ultimate outcomes that make no further contributions to other behavior changes (see textbox “What are core behavior changes?” on page 7). Focusing on these core behavior changes provides a way to embrace the complexity of RFSAs’ multi-sectoral approaches without getting lost in the weeds. For both RFSAs, most levers of integrated change are health interventions. Among interventions with more than three immediate outcomes, 5 out of 11 (Budikadidi) and 4 out of 6 (FSP-Enyanya) are health-related. And in both cases, Lead Mothers running Care Groups has the highest number of immediate outcomes among all interventions. Table 1 and Table 2 show the bridges and ultimate outcomes among Budikadidi’s and FSP-Enyanya’s target behavior changes. Positive changes in these core behaviors can be understood as key indicators for the activity’s achievement across their results framework. The tables show that regarding bridges of integrated change, the governance and livelihoods sectors play a much more important role than the health sector. Among stakeholder actions with the highest betweenness centrality, most are governance- or livelihoods-related. In the case of Budikadidi, the most central stakeholder action is Community Action Committees (CACs) providing action planning, by gathering information from community volunteers, organizing meetings, and producing village development plans. This is followed by the CACs role in monitoring community action and mobilizing the community. For FSP-Enyanya, the most central stakeholder action is households building up their financial reserves. The next most central behavior changes are farmers increasing agricultural production and farmers meeting legal requirements to become and maintain rightful landowners. 16 Table 1 – Budikadidi Core Behavior Changes Core Behavior Change Sector Type Communities peacefully resolve community conflicts Governance Ultimate Men do not engage in gender-based violence Governance Ultimate Mothers engage in exclusive breastfeeding Health Ultimate Households use high quality health and nutrition services Health Ultimate Households consume diversified diet Health Ultimate Communities do not practice open defecation Health Ultimate Women get pregnant later and practice birth spacing methods Health Ultimate CACs gather information, organize meetings, develop plans Governance Bridge CACs monitor plan implementation, mobilize community Governance Bridge Couples make joint decisions Governance Bridge Women occupy positions of authority Governance Bridge Community members learn how to read and write Governance Bridge Water committees maintain functioning water points Health Bridge Households follow recommended hygiene behaviors Health Bridge Households maintain permagardens Livelihoods Bridge Households maintain market gardens that produce substantial yields Livelihoods Bridge Households build up financial reserves Livelihoods Bridge Households sell agricultural products on local markets Livelihoods Bridge Women and girls work less time on household chores Livelihoods Bridge Table 2 – FSP-Enyanya Core Behavior Changes Core Behavior Change Sector Type Communities resolve conflicts peacefully through community platforms Governance Ultimate PLW adopt practices for better nutrition during pregnancy, exclusive breastfeeding, child nutrition, prenatal consultations Health Ultimate Women, men, children increase consumption of nutrient-rich foods Health Ultimate Households adopt improved WASH practices and abandon open defecation Health Ultimate Community members access preventive, curative health care at health facilities Health Ultimate Women practice healthy timing and spacing of pregnancy Health Ultimate Farmers meet legal requirements as landowners to hold titles and don't pay illegal taxes Governance Bridge Small-holder farmers acquire land titles Governance Bridge Women and youth hold leadership roles in their communities Governance Bridge Men and women construct HH latrines and handwashing stations and purchase essential WASH supplies Health Bridge Couples make joint decisions about family healthcare Health Bridge 17 Core Behavior Change Sector Type Households build up their financial reserves Livelihoods Bridge Farmers increase agricultural production, especially of nutritious, lower-risk food crops Livelihoods Bridge Farmers sell products on local markets Livelihoods Bridge Farmers adopt control techniques to improve soil quality and erosion conditions Livelihoods Bridge Through PO/YBGs, farmers sell produce to wholesalers Livelihoods Bridge Men, women, youth effectively manage VSLAs Livelihoods Bridge A look at the activities’ ultimate outcomes underlines the clear health focus of the RFSAs. Among these outcomes, 5 out of 8 (Budikadidi) and 4 out of 6 (FSP-Enyanya) are health-related. At the same time, none of the ultimate outcomes are related to livelihoods. The core behavior change analysis shows that the three sectors have distinct roles to play in the concert of the RFSAs. Health programming has a central position, both in terms of providing levers for integrated change and by providing most of the RFSA’s ultimate outcomes. Livelihoods programming, in turn, is the motor behind results achievement across health programming, by providing important bridges between interventions and ultimate outcomes. This happens, for example, by providing households with the financial means to purchase items such as soap that are important to institutionalize health-related behavior changes such as daily hygiene practices. Governance programming plays a similar role in bridging interventions and target behavior changes, but with a different contribution logic. A look at individual change processes in the action-focused TOCs for Budikadidi and FSP-Enyanya shows that governance interventions support livelihoods and health programming by strengthening community institutions and community leadership. In addition to that, it also contributes to separate ultimate outcomes such as “Men do not engage in gender￾based violence” and “Communities resolve conflicts peacefully.” Boosting the Effectiveness and Reach of Sector-Specific Programming Scenario analyses of the RFSAs’ Bayesian change models focus on changes in the probability that core behavior changes are achieved among target populations. The core behavior changes include “bridging” actions and ultimate outcomes. Table 1 and Table 2 in the previous section provide a complete list of bridges and ultimate outcomes for both activities. The scenario analyses show that regarding core results, RFSA staff expect sector-specific programming to deliver most of the desired sector-related behavior change. This means they expect governance interventions to deliver most of the behavior change related to governance, health interventions to drive health outcomes, and livelihoods interventions to bring about change in livelihoods. If sector-specific interventions are implemented to 100% of the target output, sector￾specific results are expected to vary between 95.9% and 100% of what staff would expect to be achieved if the full, multi-sectoral approach was implemented. Put differently, if 100% implementation success is assumed, the integration dividend for sector-specific outcomes varies between 0% and 4.1% of overall results achievement. Annex V provides the detailed results of the comparative scenario analyses. Under the assumption of full implementation success, the RFSAs’ governance programming is expected to benefit the least from multi-sectoral integration. Neither Budikadidi, nor FSP-Enyanya staff would expect the achievement of ultimate governance outcomes to improve if livelihoods and health interventions were implemented alongside governance interventions. The highest integration dividend, on the other hand, is expected for livelihoods programming. Here, governance and health interventions are expected to contribute an extra 3.4% and 4.1% of success probability to core livelihoods behavior changes such as “farmers selling products on local markets.” 18 The picture changes if the focus is on the activities’ objectives as a whole. Among the ultimate outcomes of Budikadidi and FSP-Enyanya, a majority are health-related, followed by governance￾related behavior changes. None are livelihoods-related, which led to the characterization of livelihoods programming as a motor behind the RFSAs’ ultimate outcome achievement. If only livelihoods interventions were implemented, FSP-Enyanya and Budikadidi staff expect the achievement of ultimate outcomes to be at 69.5% and 89.5%, respectively, of what would be achieved if all interventions were fully implemented. For governance interventions, these numbers increase to 78.2% (FSP-Enyanya) and 93.6% (Budikadidi). For health interventions, the numbers are 95.1% (Enyanya) and 97.0% (Budikadidi). That is, regarding the overall ultimate outcomes of an activity, integration “boosts” the effectiveness of sectoral programming in terms of improved confidence in results achievement by between 3% (Budikadidi, if full approach is compared to health￾only) and 31.5% (FSP-Enyanya, if full approach is compared to livelihoods-only). In addition to “boosting” effectiveness, the scenario analyses of the Bayesian change models show how multi-sector integration is adding reach to sector-specific interventions. For example, compared to the extrapolated baseline scenario, governance interventions under Budikadidi are estimated to improve the probabilities that core health- and livelihoods-related behavior changes will be achieved by 12.5% and 36.2%, respectively. Health and livelihoods interventions show significant less average cross-sectoral contributions. The extrapolated baseline scenario calculates the probabilities of results achievement at the end of the implementation period if the activity hadn’t been implemented. By comparing scenarios with the extrapolated baseline, analyses of the Bayesian change model can show the effects of individual interventions or target group behavior changes across the activity’s results framework. An important part of governance programming are gender-related behavior changes. Analyses of the effects of individual results in the RFSA change models underline the importance of gender-related behavior change for achieving the activities’ objectives. Both activities invest resources in improving the likelihood that couples in target communities make joint decisions on family planning, finances, hygiene, and nutrition. If full success with regard to this result is assumed, probabilities for achieving core behavior change under FSP-Enyanya’s livelihoods and health programming improve on average by 25.5% and 34.0%, respectively. For example, the likelihood that households adopt improved WASH practices and abandon open defecation is estimated to increase from 42% to 49% if couples consistently practice joint decision-making. This corresponds to a change in probabilities by 16.7%. For Budikadidi’s livelihoods and health programming, the average probability change regarding core behaviors is 27.4% and 62.7%, respectively. Multi-Sector Programming As Insurance Policy In Challenging Contexts Chapter 3.5 talks about key advantages of cross-sectoral integration. It introduces the broadening of the permagardening approach under Budikadidi as an example for the advantages of multi-sectoral activities in achieving lasting improvements in gender equality. Budikadidi decided to change the approach to promoting permagardens, so that men and youth received permagardening training in addition to women through their separate community groups. This helped increase the participation of other household members in the maintenance of the gardens, so that it “does not rest on the woman only” (B3). However, the “mainstreaming” of the permagardening approach also serves as an example for another advantage of multi-sectoral programming: With regard to key intermediate results, it affords activities multiple shots at the same target. If individual interventions do not lead to the desired effect, other interventions in the activity’s portfolio can be adapted to step in and fill the gap. This level of redundancy or substitutability of interventions improves the chances that key intermediate results are achieved, and their valuable knock-on effects can be preserved. For example, households maintaining permagardens is a key intermediate behavior change in Budikadidi’s and FSP-Enyanya’s 19 Theories of Change that “bridges” several major change processes such as improving female participation, increasing households’ financial reserves, and improving their nutritional status. Mainstreaming the permagardening approach ensured that households maintained permagardens where women alone could not. This suggests that multi-sectoral approaches can act as an insurance policy in challenging contexts. If interventions are assumed to achieve desired outputs to a 100%, a multi-sectoral approach cannot play out its redundancy or substitutability advantage. However, if there is a significant risk that individual interventions could fail, a multi-sectoral approach can provide added assurance that key outcomes are achieved. In scenario analyses, the level of output achievement can be changed for each intervention individually. For example, a scenario might be assumed where 75% of households in target communities join a Village Loan and Savings Association, but only 50% of households are reached by cooking demonstrations. When achievement levels are set, the Bayesian change models calculate the likelihoods of results achievement for all dependent behavior changes. By varying output achievement, different implementation context can be simulated. For example, it might be assumed that external conditions such as the COVID-19 pandemic curtail movement by community volunteers and only 25% of all planned training sessions can be held. The Bayesian change model can be set to reflect these assumptions and provide estimates for results achievement under such challenging conditions. For the IPA, output achievement levels for both RFSAs were systematically varied in 25% steps from 0% to 100% for each sector individually and for the overall multi-sectoral approach. This was done to calculate results achievement under different levels of implementation success and to determine the added benefit of cross-sectoral integration at each of these levels. The latter was achieved by comparing results achievement in a single-sector scenario with results achievement in a multi￾sectoral scenario, assuming the same level of output achievement across all interventions. The comparative analysis of single- and multi-sector scenarios shows that the added benefit of multi￾sectoral programming often increases, the lower the assumed level of output achievement is. For example, if it is assumed that output achievement is at 100% for all interventions, Budikadidi’s governance programming alone is expected to achieve governance-related core behavior change with the same probability as if the whole multi-sectoral approach was implemented. If, however, only 25% output achievement for all interventions is assumed, there is an average difference in success probabilities for governance-related core behavior changes of 3.8% between the implementation of Budikadidi as a single-sector governance activity and the comprehensive implementation of Budikadidi’s multi-sectoral approach. For FSP-Enyanya, the integration dividend for governance programming is at 2.6% for the 25% output scenario, and at 0% for the 100% output scenario. Annex V provides the detailed results of the comparative scenario analyses. What does this mean in concrete terms? For example, in lower output achievement scenarios, governance- and livelihoods-related interventions under FSP-Enyanya make important contributions to community members accessing public health care. If health interventions are assumed to achieve 25% of the targeted output, they alone achieve only 97.3% of the success probability for this result that would be achieved if governance and livelihoods interventions were also implemented to 25% of the target output. In concrete terms, this is a difference of two percentage points in the likelihood that community members access public health care (see Figure 5, difference between the first set of columns). If implementation is assumed to achieve 100% of the target outputs, governance- and livelihoods-related interventions are not expected to make any additional contributions to healthcare access. 20 Figure 5 - Success Probabilities for "Community Members Access Health Care" for Different Output Levels 3.2 ASSESSMENT QUESTION 1: MISSION COLLABORATION CULTURE AND LEADERSHIP Assessment Question 1: How do Mission work culture and leadership influence collaboration across offices? How is the Mission fostering and sustaining a culture internally across teams to operationalize and sustain the integrated approach? Have old organizational behaviors been modified or new behaviors initiated to reinforce the integrated approach? What role has leadership at the Mission or office level played in shaping that internal culture and any changes generated as a result? The findings in this section are based on interviews with activity managers.4 They reflect activity managers’ awareness of collaboration culture, changes in organizational behaviors, and leadership roles. The findings are by no means a comprehensive account of leadership initiatives and positive or negative organizational changes around integration. Gaps or omissions may indicate limitations in the communication around integration within the Mission or deficiencies in activity managers’ involvement in related initiatives. Strong Awareness of the Strategic Importance of Integration There is a strong awareness among activity managers interviewed by the assessment team that cross-sectoral integration is a major strategic pillar of the current CDCS. There is also universal agreement among interviewees that greater integration is needed to achieve the Mission’s development objectives. Activity managers who call for increased integration often do this in connection with describing the immense complexity of the challenges faced by DRC and expressing their frustrations with the development community’s response to these challenges. The current approach to delivering humanitarian aid and development programs is often described as “fractured” with “too many players and actors and agendas and funding levels and start 4 Throughout the report, „activity manager” is used as a general team for all Mission staff who are involved in the management of activities, including AORs, CORs, and their alternates. The umbrella term is also used to further protect the anonymity of interviewees. 66% 68% 70% 72% 74% 76% 78% 80% 82% Very Low (25%) Low (50%) High (100%) Output Level Only Health Interventions Full Multi-Sectoral Approach "Integration is for me something excellent, that should continue. When we put our resources together, we can provide tangible assistance. By ourselves, we are not able to do that. Because when we start with one problem, there is always another that we cannot address." (Activity Manager T2) 21 and exit dates” (T3)5. Integration is seen as a way to improve coordination, align approaches, join forces, and increase impact. Institutional Barriers to Improving Integration Activity managers describe several institutional barriers to improving integration, especially in the form of collaboration between activities. First, they mention differences in the timing of project cycles. Most forms of collaboration between activities require substantial overlap in implementation periods and geographies. If project cycles are not aligned, it is often not yet clear which activities should be collaborating when an activity is designed and tendered. This lack of information means that collaboration cannot be written into grants, cooperative agreements, or contracts in sufficiently concrete terms. If other offices have no concrete plans yet regarding follow-on activities, “you just don’t know exactly who you want [your activity] to collaborate with or coordinate with” (T1). In such a situation “unfortunately, you have to be really vague” when describing collaboration requirements (T1). This lack of concreteness, in turn, limits the ability of activity managers to insist on collaboration during the implementation phase of an activity. According to an interviewee, “if their contract doesn't say they have to [collaborate], we really can't make them” (T9). However, second, activity managers believe that options to facilitate collaboration by including appropriate text in award or procurement documentation are currently not fully used. They suggest that including suitable formulations could support collaboration in two ways. On one hand, awards and contracts could include language that increases the flexibility of implementation partners to change their approaches in ways that ensure opportunities for collaboration are fully exploited. Changes in activity approaches, target areas, and performance indicators are often necessary to allow activities to collaborate successfully. An activity manager suggests that for example, including descriptions of “situations that can modify the design” could enable the Mission to pursue additional integration opportunities. On the other hand, activity managers maintain that more awards and contracts should include “some expectations, some requirement for them to collaborate and coordinate” (T9). Without such requirements, collaboration across the portfolio is negatively affected because “if only one of the two partners has to work together, then that doesn’t really work” (T9). Third, activity managers express their frustrations with the time it takes to get approval for changes in activity approaches, target areas, or performance indicators. As explained above, such changes are necessary to allow activities to collaborate successfully. Given the challenges in getting changes approved, the remaining overlap in implementation periods is sometimes considered too short to pursue a collaborative strategy. When changes are “taking too long to be approved, we don’t have enough time, and sometimes we just drop the idea” (T2). Fourth, many interviewees described workload as a major barrier to integration. Integration requires time to learn about the work of other technical offices in sufficient detail so that opportunities for collaboration and potential synergies between programs can be identified. Short introductions are not enough; interviewees maintain that you have “to go in deep” and know “exactly what others are doing” to understand how activities could integrate (T2). Interviewees mentioned that they currently learn about the work of other technical offices in all-staff meetings, through their participation in design processes, and on joint monitoring trips. Once opportunities for increased integration have been identified, offices must coordinate timelines to discuss joint initiatives. Some activity managers believe that “there is an intention to do more,” but “it is just a question of bandwidth” (T3). Challenges to Collaboration Culture The institutional barriers to integration create a backdrop that sets limits for the development of a collaborative culture at the Mission. For example, interviewees describe a lack of communication across offices as a major issue in facilitating integration. This lack of regular communication leads to 5 The report uses codes for each interviewee that consist of a letter and a number. The code guarantees that each quote can be attributed to an interview transcript. The transcripts will not be shared to protect the anonymity of interviewees. 22 an information deficit regarding the activities other technical offices are undertaking. This, in turn, means that activity managers are not always aware of opportunities to collaborate. Often, the lack of communication is seen as a direct consequence of the high workload activity managers are facing (see above). However, sometimes, activity managers also mention a lack of collaborative spirit that hinders the exchange of information and thus gets in the way of collaboration. An activity manager (T1) says: “I would say, you know, big picture, at the Mission there are some people that are interested and motivated to do integration across offices, and there’s some people that are frankly not. I think that’s just the cold hard truth." For example, one activity manager described a situation where they were asked by another office not to share information with their implementing partners about additional support opportunities. The activity manager (T1) says: “Sometimes other offices are cagey. They are like ‘no, no, don’t say that, don’t provide that information.’ But if you have this and it’s already there, why can’t we tell our partners?” Another activity manager maintained that, due to prevailing misinformation regarding the incompatible nature of approaches, an activity from another office was designed not to share the same geographic space as the activity they manage. Instead of reaching out and discussing the issue, however, the office went forward with the activity design without additional consultation. Such incidences indicate that deficiencies in collaboration across offices may also sometimes result from a lack of interest in cross-office communication. Mission Leadership and Improvements in Integration Activity managers agree that Mission leadership has high expectations regarding collaboration. Few recollect instances in which this has been directly communicated to them by Mission leadership, but most maintain that there is a culture of collaboration among senior management that “has trickled down to a lot of the rank and file,” as one activity manager puts it (T1). Awareness of concrete initiatives that support collaboration is uneven. Most activity managers maintain that positive changes took place and can often describe initiatives and their impact, but some fail to identify any changes in institutional practices or organizational behaviors that would have furthered collaboration. Most activity managers interviewed by the assessment team agree that the exchange of information across offices about current activities, partners, and plans has improved. As mentioned earlier, activity managers learn about other offices’ activities mainly in three ways: Regular cross-office meetings, personal involvement in design processes, and joint field trips. Interviewees reported improvements regarding all three means of acquiring information about other offices’ activities. First, an increase in regular meetings was reported by interviewees on both sides of the humanitarian-to￾development nexus. The meetings they referred to involve BHA and EG. Second, some activity managers reported a stronger involvement in the design processes of other offices. Third, some activity managers maintain that more joint monitoring trips are planned between offices. The latter might also be an impression resulting from the fact that the recent COVID-19 pandemic did not allow for any monitoring trips. The perceived increase in joint trips might represent a return to the status quo ante. In addition to improvements in inter-office communication, one activity manager mentioned the development of joint programmatic documents as an important initiative that facilitates integration. Reference was made to documents around climate change, gender, and nutrition. The documents were described as “laying out the opportunities for collaboration, explaining the mission, vision, and how all the offices will be working around the topic so that you can be complementary and very synergistic and have a common understanding of the way we communicate on this” (T4). "From what I see, things have gotten a lot better with coordination at the Mission. I think in the past, the coordination wasn’t that great." (Activity Manager T1) 23 “Pivot to the East” A Mixed Blessing for Integration The Mission’s “Pivot to the East” was seen as a mixed blessing for integration. Some activity managers shared the view that given how vast the DRC is, a concentration on specific regions might increase the geographic overlap of activities, thereby facilitating collaboration, and increasing overall impact. However, others pointed out that “just because you’re in the East doesn’t mean you’re in the same place” (T3) because “there’s still a lack of geographic specificity when you say ‘Eastern Congo’” (T1). At the same time, it was feared that the concentration on Eastern Congo has led to a situation where opportunities for collaboration in other parts of the country moved out of focus: “There are other parts of the country where there are opportunities [for collaboration], but we're just not looking at them.” (T1). 3.3 ASSESSMENT QUESTION 2: ROLES AND RESPONSIBILITIES IN CROSS￾OFFICE ACTIVITIES Assessment Question 2: How do technical offices distribute roles and responsibilities in cross-office activities? How are the responsibilities and coordination of management and decision-making authority for activity design, procurement, and monitoring shared across teams? How and when do those teams work together, and how are concerns or disputes resolved? How do they identify and determine the attribution of results achieved through joint efforts contributing to an integrated DO? Currently, the Mission’s major multi-sectoral activity that is jointly funded by several offices is the Partnership for the Development of Eastern Congo (P-DEC). The AOR is attached to the Peace and Security Office (PSO). Funding is provided by CARPE, DRG, EG, and the Education office. Initially, the Health office also provided funding, but later decided to pull out. AOR as Sole Information Broker and Convenor The AOR acts as an information broker and convenor for actors inside the Mission and the prime implementing partner. 6 If technical offices require information from the implementing partner, the activity manager forwards their requests, receives the information, and provides it to the requesting office. If technical offices want to meet and discuss with the implementing partner, the activity manager schedules a meeting and sends out invitations. This also happens if the implementing partner requests technical advice or guidance from a specific technical office. Communication between the participating offices was described as frequent and fruitful during the design and procurement stages. However, even at these stages, communication between offices was rarely direct. The activity manager recalls that mostly, “it was just sharing documents.” For example, he explains: “When the applicant sent their proposal, our office shared them with the technical offices to receive their feedback, then they were reviewing the documents and sent them back with their comments, we consolidated them, and shared them with the applicant.“ Sometimes, this process was accompanied by one-on-one phone conversations between the PSO office lead and other office leads. Lack of Inter-Office Communication In the first year of activity implementation, the lack of direct communication between offices likely contributed to delays. Finalizing the AMELP and the first-year work plan took the better part of the first year of activity implementation. AMELP-related issues revolved around results attribution. Participating offices wanted to see specific indicators reflected in the P-DEC results framework. With regard to the work plan, discussions focused on the prioritization of activities. Solutions were sought bilaterally, between the implementing partner and one technical office at a time, facilitated by the activity manager. Thus, individual office requests were pulling the activity in different directions 6 In this chapter, no interviewee identifier is provided for the P-DEC activity manager as this would de-anonymize quoted statements in other chapters. 24 that were complicated to reconcile. This hampered the development of a joint vision for the success of P-DEC. The activity manager says: “It was difficult to come up with a joint understanding. Issues were not discussed between technical offices, but they were discussed between technical offices and the implementing partner, and me as AOR.” He recollects one inter-office meeting where the implementing partner presented their approach. This meeting “helped us to find solutions to avoid all the back and forth over emails.” However, no additional meetings were organized. Continued Collaboration? The lengthy and controversial discussions around the AMELP and the work plan seem to have left all sides with the impression that continued support of the joint activity is in question. An activity manager (T2) from a co-funding office says: “It looks like the results we are getting from P-DEC are not satisfying and people think that at this point it’s better to withdraw ourselves from the project.” P-DEC’s activity manager is left with the feeling that he must continually convince participating offices to stay on board and “make their funds available to P-DEC.” He says: “What I know is that each year, they need to confirm if those funds are still available for P-DEC or if they will use them for other priorities. So, I have discussions with them and these are hard discussions for me as AOR.” 3.4 ASSESSMENT QUESTION 3: COLLABORATION BETWEEN MISSION & IMPLEMENTING PARTNERS Assessment Question 3: How does the Mission collaborate with implementing partners to further integration? How have relationships and roles between the Mission's internal teams and their implementing partners been defined and operationalized for any cross￾cutting activities? Have any new behavioral practices been adopted across these organizational relationships to facilitate implementation? Different types of awards provide USAID and activity managers with different levels of involvement and control over the award administration and performance. This includes the activity managers’ ability to bring about collaboration between activities. However, based on the interviews with activity managers undertaken by the assessment team, there are differences in how activity managers perceive their roles that are not fully corresponding to differences in the type of awards they manage. As one activity manager (T4) puts it, having a contract instead of a cooperative agreement or grant is “something that helps. But for me, I think the most important is to make sure that your implementing partners understand why you are very supportive of an idea to collaborate.” The Difference Between Suggesting and Insisting on Collaboration When it comes to furthering integration, some activity managers clearly see themselves in the driver’s seat, while others suggest rather than insist on collaboration. One interviewee (T4) with a proactive understanding of their role says: “[The other activity manager] and myself were telling [the implementing partners] the same thing. You need to collaborate. We really want to see these things happening. So, they clearly identified in their own work plan what each one will be bringing to the table.” In contrast to that, another activity manager maintains that he, “can just recommend them to collaborate and to facilitate that collaboration” but if emails to implementing partners (IPs) do not lead to the desired result, then collaboration is not further pursued (T5). The different interpretations of what an activity manager can do to further collaboration also lead to frustrations between activity managers. For example, one activity manager (T3), recalling a failed collaboration with activities under another technical office, says that “maybe because they’re cooperative agreements, we just can’t force them to do it, but there was a feeling they also didn’t really try whatever they could.” 25 Initiate, Facilitate, and Monitor Activity managers play three distinct roles in supporting collaboration: They initiate, facilitate, and monitor. Most interviewees agreed that successful initiation of collaboration between activities requires persistence to overcome initial reluctance on the part of implementing partners and meetings where activity managers jointly discuss their plans for collaboration with IPs. Unilateral initiatives that do not involve the activity manager of the intended collaboration partner leave IPs in a situation with limited options to bring about the desired collaboration. An activity manager (T5) explains: “We can put the collaboration in the work plan, and we approve the work plan, but then our implementing partner doesn’t have enough power to involve the other partner.” Initiating collaborations can also involve providing implementing partners with a platform to present their activity to other activity managers so that opportunities for collaboration can be identified. Activity managers described two ways in which they facilitate ongoing collaborations. First, they resolve issues around communication and information sharing among implementing partners. For example, IPs might request contact information, maps, or other data from an implementing partner that allow them to layer their interventions onto another activities’. Sometimes, partners are unwilling to share such information with their potential competitors. Activity managers then step in to facilitate the information exchange. This can mean making an information request on behalf of the implementing partner to “amplify their message” because the collaboration partner “will react differently than if [another COP] writes to them” (T8). Second, activity managers get involved to solve issues around results attribution. According to interviewees, this can involve discussions at three levels: between technical offices, between activity managers, and between implementing partners. An activity manager (T4) explains: “It is possible not to fight over this, but you have to see exactly what each one could be reporting. This requires dialogue on our level.” Several activity managers underlined the importance of joint monitoring visits to understand how collaborations played out on the ground. As one activity manager (T9) explained: “It can't just be, here's the work plan, let's go. But having that check-in to say, how is this working?” 3.5 ASSESSMENT QUESTION 4: CROSS-SECTORAL INTEGRATION AND MULTI-SECTORAL ACTIVITIES Assessment Question 4: How do multi-sectoral activities achieve effective integration? At which points in their program logic do integrated activities seek cross-sectoral integration? Which cross-sectoral effects do they hope to achieve? How do actors at such points of integration (staff, community volunteers, community members) collaborate? Which challenges do they face and how do they overcome them? To understand how multi-sectoral activities achieve effective integration, the assessment team conducted in-depth case studies of two BHA activities: The Resilience Food Security Activity (RFSA) in Kasaï Oriental (“Budikadidi”) and the South Kivu Food Security Project (“FSP-Enyanya”). Despite only one of the activities including “Resilience Food Security Activity” in their name, both activities are RFSAs and follow very similar multi-sectoral approaches to increasing the resilience of communities and households. The approaches of Budikadidi and FSP-Enyanya are multi-sectoral because they combine interventions in governance, health, and livelihoods. Gender is understood as a cross-cutting topic by "We have the power to bring [implementing partners] together and speak to each other and come up with some solution and reach consensus on different aspects." (Activity Manager T8) 26 both activities. In terms of community-level collaboration partners and change processes, however, gender is most closely interlinked with governance activities. This is why the study treats gender interventions as part of the activities’ governance interventions when cross-sectoral analyses are carried out. The RFSAs’ health interventions include nutrition and WASH-related interventions. Advantages of Cross-Sectoral Integration Workshops and interviews with Budikadidi, FSP Enyanya, and the FAW activity provided insights into the advantages and disadvantages of integrated programming on the community level. Workshops with Budikadidi and FSP-Enyanya provided detailed information on the RFSA’s experiences with integrated programming. In our discussions, staff often contrasted their experiences with that of single-sector activities or activities trying to achieve the same degree of integrated implementation through inter-activity collaboration. In interviews with FAW activity representatives, the assessment team was able to inquire directly how cross-sectoral integration might have benefitted the FAW activity, and where the advantages of focusing on a small set of agricultural interventions may lie. Five advantages of cross-sectoral “integration by design,” i.e., through the implementation of multi-sectoral approaches, were identified. First, activities with a multi￾sectoral approach have a “thick” local presence. They support local institutions and create and work with several local groups that target different stakeholder groups simultaneously. For example, in each target community, FSP-Enyanya works with dialogue and mediation committees, faith leaders, farmer organizations, hillside development committees, mother Care Groups, village development committees, water committees, and many others. This “thick” local presence allows multi-sectoral activities to deeply engage with the social fabric of the target community, build trust, and identify local champions. It also allows such activities to positively respond to the expectations of a community for comprehensive partnership and support. Single-sector activities are faced with the same challenges of building trust and identifying champions in target communities but have less entry points into a community and have less support to offer. Providing limited support often leads to disappointment. As a FAW activity representative (F3) explains: “Every time we go to the field, the local authorities present us with their development plan. They present us with all their needs, and they ask: Can you support us with these issues as well?” Second, implementation of a cross-sectoral approach under the coordination of the same implementing partner provides opportunities for a detailed alignment of sectoral activities that would be challenging to achieve through activity collaboration. For example, Budikadidi was able to tailor agricultural training to the production of foods that fulfill the nutritional needs of lactating mothers. Through their governance activities, they were also able to facilitate the inclusion of women in household decision-making, so that it was possible for more women to establish a permagarden on a sufficiently large piece of land. Complex challenges with an inherent trade-off, such as improving food security while also promoting exclusive breastfeeding, require a careful balancing of interventions. The trade-off exists because activities often try to improve households’ nutritional status by increasing agricultural production. However, women often do most of the fieldwork and fields are often at a distance from the families’ houses. This means that in order to achieve sustainable progress on both objectives, an activity must provide means for a lactating woman to contribute to household nutrition in a way that is compatible with the breastfeeding of a newborn child at home or must support households in carefully prioritizing objectives. This requires close collaboration between agricultural and health interventions that separate activities with competing objectives may find challenging to achieve. "When there is no other USAID project active in the village, things are not easy. Because you need to present yourself, there is no person to receive you, there is no person to introduce you to anyone there." (FAW Activity Representative F2) 27 Third, multi-sectoral activities seem better positioned to achieve lasting improvements in gender equality. Improving gender equality requires systemic change (USAID 2023). To achieve systemic change, building the capacity of a single actor or changing an individual relationship is insufficient; the focus must be on the system in its entirety (USAID 2014). A multi-sectoral approach provides an activity with multiple entry points into a local community. If well aligned, governance, livelihoods, and health interventions can change the behavioral expectations of key actors towards women while providing women with the local spaces to grow their skills and improve their economic situation. In Budikadidi’s multi-sectoral approach, there are elements designed to sensitize community leaders on gender-based violence and gender equality, build women’s skills and self-esteem, provide women with opportunities to improve their economic situation, and support their participation in community decision-making. Coordination under one roof allowed Budikadidi to complement and adapt their activities in support of systemic change. For example, recognizing that economic empowerment and inclusion increase the demands on women’s time, the activity intervened on the household level to facilitate a more equal distribution of household chores and introduce time-saving household devices, such as improved cooking stoves. Budikadidi also adapted some of their approaches to free women from tasks and allow them to participate in empowering activities. For example, the activity shifted the approach to promoting permagardening to increase the participation of other household members in the maintenance of the gardens. While training on permagardening originally focused on Care Groups that are women-only and facilitated by Lead Mothers, the approach was later changed to include community groups that other household members participate in. A staff member explains (B3): “We used to do the training with Lead Mothers only, but that led to a situation where permagardens were seen by the community as the responsibility of the woman. So instead of lightening her workload, we increased it. Now that everyone has the same skills, the maintenance of the garden does not rest on the woman only.” Fourth, in an integrated, multi-sectoral approach, economic interventions can be used as an incentive and platform to engage community members in activities whose benefits are felt less immediately. Both RFSAs establish and train savings and farmer groups. Such groups bring an immediate economic benefit to households, either by increasing access to credit or by improving income through increased agricultural production and improved access to local markets. These groups provide Budikadidi and FSP-Enyanya with an entry point for gender and health programming. The lack of such an entry point is felt by single-sector activities that do not provide direct economic benefits to target groups such as the FAW activity. A representative (F3) describes the challenges in mobilizing participants for pest-control trainings: “Farmers say to us: ‘It’s good that you help us against fall armyworm, but please also support us with economic development activities that will allow us to increase our income.’” Another staff member (F2) explains: “What’s the motivation for a smallholder farmer to leave his daily activities and come to attend a training? People are looking for direct impact. So it’s important to integrate microfinance activities such as VSLAs.” In addition to the role of motivating community members to attend training sessions, livelihoods programming also plays a systemic role in promoting health outcomes. Many WASH- and nutrition￾related behavior changes require regular investments in products and tools to be sustainable. For example, to engage in regular handwashing, soap must be available. Increased incomes through, e.g., income-generating activities or improved agricultural production, allow households to make such purchases. Fifth, activities with an integrated, multi-sectoral approach can systematically work on improving the sustainability of local interventions. The RFSAs can leverage their governance programming to build the capacity of local institutions in planning and coordinating development activities and involve leaders in institutionalizing newly established processes and methods. The importance of governance 28 work for sustainability was confirmed by FAW activity representatives. When asked about what additional programming would support the long-term success of their pest-control trainings, FAW activity representatives agreed that local governance interventions would be a vital addition (see quote on the right). Challenges in Integrating Multi-Sectoral Activities While multi-sectoral approaches have unique advantages in delivering comprehensive community support as described in the previous section, managing such complex activities comes with its own challenges. In our discussions with Budikadidi and FSP-Enyanya staff, two challenges stood out: • How do you facilitate the coordination of activities on the local level and ensure coherent messaging across interventions? • How do you support communities and households in prioritizing time and resource investments? The BHA RFSAs implement their multi-sectoral approaches through different community volunteers and local groups. Sector teams design training materials and provide training of community volunteers that implement interventions in their sector. Community volunteers manage their own local groups with overlapping membership. This approach leads to parallel training cascades with no immediate integration of content as well as scheduling conflicts on the local level. This led to a situation where “at the beginning, everyone just came with their approach: I come with my care group approach, you work with your approach. Everyone was just interested in the activities of their component” (B5). The lack of coordination made it difficult to achieve coherent messaging across activities and fully exploit the potential for synergies. A Budikadidi staff member explains: “The consequence was, for example, that the agriculture team that should promote permagardening did not know exactly what the key nutrition messages are that they need to put forward and convince the communities. The team was not able to provide good information and to convince the households to set up the gardens. Instead, they were much more focused on technical approaches without really being able to pass on key messages about the importance of the vegetables and what could be done with them.” On the other hand, the Budikadidi mid-term evaluation confirmed that “the health and nutrition team (especially below senior management level) does not have enough familiarity with the targeting, strategies, and timing of the livelihoods work to understand how it should be supporting health and nutrition at the household level” (IMPEL 2020a). The RFSAs’ solution for improving messaging and coordination was to identify local institutions and community agents that reflect the multi-sectoral nature of their approach and empower them to meet, plan, and coordinate implementation on the community level. Local institutions, such as the Community Action Committee (CAC) or Comité Villageois de Développement (CVD), were positioned as central nodes in the multi-sectoral local networks of community groups and local stakeholders. The committees cover the multi-sectoral breadth of local programming through different members with sectoral expertise. Their ties with individual community volunteers and groups were strengthened, so that information on activities, issues, and priorities from across the community became available to inform the committees’ community action planning, mobilization, and monitoring. Besides the committees, Budikadidi and FSP-Enyanya also installed community agents to support and coordinate local activities. The community agent is thought of as a generalist that facilitates joint planning processes. They are “expected to speak every language in development. They are expected "When we are looking for sustainability, it is important to ensure that we work with government actors. When we talk about agriculture, we cannot work with a farmer alone. We need to engage their social environment, local governance, local rules." (FAW Activity Representative F2) 29 to speak food, speak agriculture, speak gender, speak health, so that they are interfacing with the committee and the community members in an integrated way” (B1). To build up and retain a person in this role, however, is a challenge in its own right. Budikadidi is currently training volunteer replacements for paid field agent positions. Both RFSAs provide households in their target communities with many different opportunities to improve their socioeconomic well-being and health. With limited time and resources, households cannot engage in all these opportunities at the same time. In addition to that, objectives of some activities may be contradictory at times. As discussed earlier, a household might not be able to fully engage in improving agricultural production while also following recommendations regarding exclusive breastfeeding. Either the lactating mother joins the household’s efforts in farming or stays back at the house to breastfeed. A Budikadidi representative (B2) explains: “It’s the household decision at the end of the day. We kind of present them a menu of things and it’s their decision. It's difficult. It’s impossible for them to do everything.” This is a known issue with complex, integrated activities. For example, a review of USAID-funded food security program evaluations concludes that “community members are overwhelmed by the number of committees they may have to be part of, and meetings and training courses they are expected to participate in” (IMPEL 2020b). The RFSA’s solution lies in supporting households in planning and prioritization. First, households are presented with a “minimum package” (B2), i.e., a shorter list of priority behaviors across sectors. Second, group meetings are followed up with home visits. When community volunteers are visiting households, they can provide advice that is tailored to the household’s needs. Third, community volunteers coordinate activities across sectors to support household in long-term planning. For example, Budikadidi noticed that increased market access sometimes led to a situation where households were unable to cover their nutritional needs because they sold off too much produce. The solution was to involve the health team (B5): “They had to work in the households with the Lead Mothers, through the Care Groups, so that before any sale, the household would first ensure that enough food would remain available over the season. So, there is a need to improve planning mechanisms at the household regarding food needs while also ensuring production and sale.” 3.6 ASSESSMENT QUESTION 5: CROSS-SECTORAL INTEGRATION THROUGH COLLABORATION Assessment Question 5: How does collaboration between activities contribute to effective integration? How do activities from different sectors collaborate? How do they share information, sequence or layer interventions, or jointly plan interventions? Which cross-sectoral effects do they hope to achieve? How do actors from different activities collaborate? Which challenges do they face and how do they overcome them? The question on how collaboration between activities contributes to effective integration will be answered in two parts. In the first part, the analysis focuses on the BHA RFSAs and their collaborations with IHP and the FAW activity. In workshops and interviews, activity representatives were asked to identify and discuss points in the activities’ intervention logics where they expected collaborations to make major contributions to results achievement on both sides. Transcripts of these discussions were analyzed to identify the types of cross-sectoral effects the activities hoped to achieve and to understand how collaborations unfolded. The second part goes beyond the experiences of the four activities and includes information from our interviews with activity managers to gain a broader perspective on inter-activity collaboration. With this broader perspective, the analysis was able to identify recurring challenges in achieving inter-activity collaboration. Effects of Inter-Activity Collaboration Which cross-sectoral effects do activities hope to achieve when they enter collaborations? Since no 30 two activities are alike, one could argue that every cross-sectoral effect sought in collaboration is necessarily unique, and a meaningful answer to this question impossible. However, many collaborations follow a similar logic. For example, they are initiated when activities interface with the same local actors in a complimentary way or assets and immaterial resources can be jointly used. From this perspective, types of cross-sectoral effects can be defined. The analysis of the RFSAs’ collaboration with IHP and the FAW activity yielded six types of cross￾sectoral effects: Increasing reach, gaining access to communities, building on relationships of trust, complementing interventions, joint monitoring, and covering emerging topics. Increasing reach: The FAW activity collaborated with Budikadidi in Kasaï-Oriental. Budikadidi’s community agents were invited to FAW activity trainings on pest control. The FAW activity provided Budikadidi with communication materials. In turn, the FAW activity was able to benefit from existing community structures in Budikadidi’s target communities when delivering training to farmers. By training Budikadidi’s community agents and providing them with communication materials, the FAW activity was able to cascade training in places where they didn’t operate, thus increasing their reach. Gaining access to communities: IHP’s interventions on the community level are focused on health centers. The RFSAs work with local institutions and community actors. When IHP wanted to involve the community in their activities, Budikadidi was able to activate their local structures and support community mobilization. An IHP representative (H1) says: “This meant that whenever we had an activity in a [Budikadidi community], we always had a correspondent at the community level, for almost every topic.” The FAW activity also profited from the RFSA’s access to community structures. A FAW activity representative (F4) explains: “They are more established in a community. They have a system of farmer leaders in place. When we came, we already found this basis. We then just had to train and ensure follow-up regarding the application of good agricultural practices for the management of the Fall Army Worm.” Building on relationships of trust: Before starting implementation in a community, an activity must introduce itself to the community leadership, gain their trust, and seek out collaborators that support their interventions. This relationship work is resource-intensive. This is why a USAID activity already operating in a community can be a huge asset to an activity, especially if it is a single￾sector activity such as FAW which delivers a limited local intervention and has a large target area to cover. A FAW activity representative (F2) explains: “To be honest, when there is no USAID project, things are not easy. Because there is no person to introduce you to anyone there. You have to quickly identify good people that can help you, that can influence local opinion.” If a previous development actor has disappointed local expectations or broken trust, local implementation is in jeopardy: “I know one or two examples of villages that refused our project because another project arrived there previously that didn’t respect deadlines, wasn’t accountable, and had bad leadership.” Complementing interventions: Some activities work with the same type of local institutions or community actors. Precise geographic targeting and careful coordination of activities can help create synergies around these interventions. Budikadidi and IHP both interact with local health centers. IHP provides training and resources. Budikadidi improves outreach and increases local demand for health services. For example, Budikadidi works with Lead Mothers and community agents to identify cases of child malnutrition and refer them to local health centers. However, “if there is no therapeutic products at the health center, and the lead mother knows there is nothing there, they are not going to come” (B2). This means that IHP support to health centers “is a key step in this pathway of [reducing child malnutrition]” (B2). Joint monitoring: When interventions are targeting the same communities, activities can collaborate in collecting and analyzing monitoring data and thus save resources. FAW activity representatives describe the potential for this type of collaboration around collecting data on agricultural yields and 31 post-harvest loss. Budikadidi mentions joint analyses of context data as an example (B2): “We looked at DHIS2 data every quarter with IHP, to see the utilization of [health] services, and it allowed us to see where men and women have health problems.” Covering emerging topics: During the life of an activity, development issues can emerge that were not foreseen at the design stage. In such cases, collaboration with another activity can be the most efficient way to address the issue. A FAW activity representative (F2) explains: “In a project, it is always tricky to include a new intervention. You need specialists, which require extra funds, new tools, and so on. And many projects implemented by USAID did not consider Fall Army Worm as a problem. So we put in place a strategy so that each USAID project working in agriculture can address the Fally Army Worm problem.” How does collaboration around these types of cross-sectoral effects look in practice? Activities must invest considerable time in communication and coordination. Successful communication entails regular calls and meetings and clear points of contact. Coordination must ensure that activities address the same community members in the same capacity and there are no scheduling conflicts preventing community members from participating in trainings and other local events. A FAW activity representative (F4) explains: “When we installed farmer field schools, we ensured that we worked with the same producers that Budikadidi provides seeds to, so that we do not create conflicts among them.” To ease scheduling, Budikadidi and the FAW activity chose different weekdays for their engagement with the local community. Challenges in Achieving Cross-Sectoral Collaboration Interviews with USAID/DRC activity managers and implementing partners revealed that there are important preconditions that must be met so that activities can successfully engage in cross-sectoral collaboration. Among these are the following: Flexibility in design and implementation, substantial overlap in target area and implementation period, complementary results attribution, and alleviated concerns regarding competitive disadvantages. First, any collaboration between activities requires flexibility to align approaches. This can involve geographies, timing, target groups, and training contents. Flexibility varies with the types of awards. While a grant leaves activity managers with less room to enforce collaboration (see section 3.3), it provides an implementing partner with more flexibility to adapt implementation processes, so that collaboration can take place. According to a FAW activity representative (F1), contracts can be a real hindrance for collaboration: “Because they had a contract, they did not have the flexibility to collaborate. They have not budgeted for some of the activities where collaboration would be really a win-win, so they couldn’t do it.” Second, collaboration between activities requires substantial overlap in terms of target areas and implementation period. Both are complicated to achieve. Project cycles across funding streams and technical offices are not aligned. When an activity is designed, it is often unclear which other activities could provide synergies during the activity’s implementation period. As a result, geographic alignment is not often intentional. Regarding timing, activity representatives report that differences in start dates quickly constitute major issues for a successful collaboration. If an activity in a planned collaboration has a delayed start, then the gap has to be filled by the collaboration partner until the activity starts implementation. If an activity seeks collaboration with a partner one year before close￾out, the partner may be reluctant to collaborate. This leaves little tolerance for differences in project cycles. " When we design an activity, we just don’t know enough about what is out there yet. So, we said: Please collaborate when possible. I guess, what I’m saying, it's very hard to make this work." (Activity Manager T1) 32 Third, activities that are willing to enter a collaboration must find a way to attribute joint results in a complementary manner. Activities must justify their allocation of resources to USAID/DRC in reference to their own results frameworks, while avoiding double counting. This means that collaborating partners can only fully engage in joint activities if they can allocate results to complementary performance indicators. This is not always the case, especially when activities are working with the same local groups towards similar goals. 33 4. CONCLUSIONS 4.1 OVERARCHING CONCLUSIONS • Integration substantially improves the effectiveness of activities. Systematic scenario analyses of Bayesian change models allowed the IPA to quantify the contributions of cross￾sectoral integration to activity effectiveness. By comparing extended baseline scenarios with scenarios in which the RFSAs’ multi-sectoral approaches are implemented fully or partially, the assessment team calculated the “Integration Dividend,” i.e., the increased average results achievement of multi-sector programming over single-sector programming. In the case of the two RFSAs, the integration dividend varies between 3% (Budikadidi, when comparing health-only approach to full approach) and 31.5% (FSP-Enyanya, when comparing livelihoods-only approach to full approach). • “Integration by Design” is a more efficient means to achieve integration than “Integration post Design.” The assessment’s findings suggest that “Integration by Design,” i.e., through the design and implementation of integrated activities, is a more efficient means to achieve integration and strengthen activities’ contributions to DOs than “Integration post Design,” i.e., through the collaboration of activities. The assessment described several preconditions for the successful collaboration of activities that are complicated to achieve in practice. Among these preconditions are flexibility in design and implementation, substantial overlap in target area and implementation period, complementary results attribution, and alleviating concerns regarding competitive disadvantages by implementing partners involved in the collaboration. At the same time the assessment described several unique advantages of multi-sectoral activity designs. Among these are a “thick” local presence, the possibility of a detailed alignment of interventions across sectors, a systemic engagement with local communities to achieve lasting improvements in gender equality, using economic interventions as an incentive and platform to engage community members in non-economic activities, and the ability to use governance interventions to systematically work on the sustainability of local results. • “Integration by Design” requires multi-lateral communication to work. Our analysis of the RFSAs shows that being able to flexibly layer interventions and temporarily shift the focus of community-level activities is a major advantage of multi-sectoral activities. The Mission’s experiences with the management of P-DEC suggest that joint activity management by several offices poses a risk to realizing these strengths of “Integration by Design.” A lack of multi-lateral communication between technical offices at the Mission hampered the development of a joint vision for the success of P-DEC. Instead, offices insisted on sectoral progress indicators in bilateral communication with the implementing partner. In these conditions, the ability of an implementing partner to realize synergies among interventions is limited. 4.2 STRATEGIC QUESTION: THE INTEGRATION DIVIDEND • Identifying levers, bridges, and ultimate outcomes in an activity’s intervention logic provides a way to embrace the complexity of a multi-sectoral approach while focusing on its key elements. This allows the analysis to gain a clear picture of the roles that different sectors play in achieving integrated change. Levers of integration are interventions with a high number of immediate contributions, bridges are intermediate outcomes with a high betweenness centrality, and ultimate outcomes are key results that make no further contributions to other activity objectives. • Scenario analyses that track changes in bridges and ultimate outcomes show that integrated programming boosts the effectiveness and reach of sector-specific interventions. The strongest benefits from multi-sectoral integration are achieved for livelihoods programming, followed by governance and health programming. For example, the Integration Dividend is 31.5% for livelihoods programming under FSP-Enyanya, meaning that if only livelihoods interventions are implemented, the expected results are 31.5% lower than if all interventions were implemented. 34 With regard to added reach, the scenario analyses show how important gender-related behavior change is for overall results achievement. • The systematic variation of output achievement in comparative scenario analyses shows that multi-sectoral approaches can act as an insurance policy in challenging contexts. In multiple scenarios, output achievement levels for both RFSAs were systematically varied in 25% steps from 0% to 100% for each sector individually and for the overall multi-sectoral approach. • The results show that the added benefit of multi-sectoral programming often increases, the lower the assumed level of output achievement is. This means that in challenging contexts, where output achievement is suboptimal, interventions can substitute for each other across sectors in achieving core behavior change. If 100% success in implementation is assumed, the Integration Dividend is often smaller or non-existent. 4.3 ASSESSMENT QUESTION 1: MISSION COLLABORATION CULTURE AND LEADERSHIP • There is a strong awareness of the strategic importance of integration among USAID/DRC activity managers. Integration is seen as a way to join forces in an environment where “fractured” responses by the development community have proven inadequate to solve DRC’s multi-faceted development challenges. • Activity managers are also aware of the high expectations the Mission leadership has regarding collaboration. However, awareness of concrete initiatives that support collaboration is uneven. • Regular information exchange between offices at the activity manager level is understood as a cornerstone for the development of a collaborative culture. Most activity managers agree that the exchange of information across offices has improved. They report three changes that contributed to the improvement: More regular cross-office meetings, stronger involvement in activity design processes, and more joint monitoring trips. The perception of positive changes and improvements in communication in recent times may also be influenced by the fact that the COVID-19 pandemic disrupted all three of these practices. • Activity managers believe that today, collaboration culture at the Mission is still marked by insufficient regular information exchange between offices at the activity manager level, resulting in missed opportunities to collaborate. While a high workload is seen as a major cause of the situation, some activity managers report incidents that speak to a lack of interest in cross-office collaboration among some colleagues. • Institutional barriers hamper the development of a collaborative culture at the Mission. They are: (1) Differences in project cycles that lead to misaligned activity designs, (2) inconsistent use of award and procurement language that would incentivize collaboration, (3) delays in getting approval for design changes that enable collaboration, and (4) high workload that prevents activity managers from identifying opportunities to collaborate. • The Mission’s “Pivot to the East” is seen as a mixed blessing for integration. On one hand, some activity managers shared the view that given how vast the DRC is, a concentration on specific regions might increase the geographic overlap of activities, thereby facilitating collaboration. On the other hand, some activity managers think that the concentration on Eastern Congo has led to a situation where opportunities for collaboration in other parts of the country moved out of focus. 4.4 ASSESSMENT QUESTION 2: ROLES AND RESPONSIBILITIES IN CROSS￾OFFICE ACTIVITIES • Currently, the Mission’s major multi-sectoral activity that is jointly funded by several offices is the Partnership for the Development of Eastern Congo (P-DEC). • The AOR acts as an information broker and convenor for actors inside the Mission and the prime implementing partner. • Communication between offices is rarely multi-lateral. Instead, solutions are sought in bilateral communication between an office and the implementing partner, facilitated by the AOR. The lack of direct communication between offices likely contributed to delays in activity 35 implementation. Individual office requests pull the activity in different directions that are complicated to reconcile. • The lengthy and controversial discussions around the activity AMELP and first year work plan seem to have left the collaborating offices with the impression that continued support of the joint activity is in question. 4.5 ASSESSMENT QUESTION 3: COLLABORATION BETWEEN MISSION & IMPLEMENTING PARTNERS • Different types of awards provide USAID and activity managers with different levels of involvement and control over the award administration and performance. This includes the activity managers’ ability to bring about collaboration between activities. • When it comes to furthering integration, some activity managers clearly see themselves in the driver’s seat, while others suggest collaborations to implementing partners rather than insist on them. The different interpretations of what an activity manager can do to further collaboration contribute to frustrations between activity managers. • Activity managers take three different roles in supporting collaboration: They initiate, facilitate, and monitor. Activity managers described two ways in which they facilitate ongoing collaborations: They resolve issues around communication and information sharing among implementing partners and they get involved in finding solutions around results attribution. 4.6 ASSESSMENT QUESTION 4: CROSS-SECTORAL INTEGRATION AND MULTI-SECTORAL ACTIVITIES • The three sectors governance, livelihoods, and health have distinct roles to play in the concert of the two BHA RFSAs Budikadidi and FSP-Enyanya. Health programming has a central position, both in terms of providing levers for integrated change and by providing most of the RFSA’s ultimate outcomes. Livelihoods programming, in turn, is the motor behind results achievement across health programming, by providing important bridges between interventions and ultimate outcomes. Governance interventions support livelihoods and health programming by strengthening community institutions and community leadership and contribute to separate ultimate outcomes. • Five advantages of cross-sectoral “integration by design,” i.e., through the implementation of multi-sectoral approaches, were identified in workshops and interviews with activity representatives. First, multi-sectoral activities, through their “thick” local presence that involves working with many community stakeholders simultaneously, are able to deeply engage with the social fabric of the target community, build trust, and identify local champions. Single-sector activities are faced with the same challenges of building trust and identifying champions but have fewer entry points into a community and have less support to offer. • Second, implementation of a cross-sectoral approach under the coordination of the same implementing partner provides opportunities for a detailed alignment of sectoral activities that would be challenging to achieve through activity collaboration. This is especially important when addressing complex challenges with an inherent trade-off. Separate activities may find it challenging to achieve a careful balancing of interventions with competing objectives. • Third, multi-sectoral activities seem better positioned to achieve lasting improvements in gender equality. Improving gender equality requires systemic change, which can only be achieved if the intervention focuses on a local system in its entirety. A multi-sectoral approach provides an activity with multiple entry points into a local community. • Fourth, in an integrated, multi-sectoral approach, economic interventions can be used as an incentive and platform to engage community members in activities whose benefits are felt less immediately. The lack of such an entry point can hamper the mobilization efforts by single-sector activities that do not provide direct economic benefits to target groups. • Fifth, activities with an integrated, multi-sectoral approach can systematically work on improving the sustainability of livelihoods and health interventions through their governance programming. • Multi-sectoral activities face two major challenges with regard to their management: The local 36 layering of activities, including a coherent messaging across interventions, and the risk of overwhelming community members by the number of simultaneous interventions that require their participation. • The BHA RFSAs have found promising solutions to these challenges. To ensure local layering and coherent messaging, they installed local institutions and community agents that reflect the multi-sectoral nature of their approach and empowered them to meet, plan, and coordinate implementation on the community level. To support households in setting priorities, the RFSAs defined a “minimum package” of key behavior change, had community volunteers conduct home visits to support individual decision-making, and increased the households’ capacity to plan for the long-term. 4.7 ASSESSMENT QUESTION 5: CROSS-SECTORAL INTEGRATION THROUGH COLLABORATION • The analysis of the RFSAs’ collaboration with IHP and the FAW activity yields six types of cross￾sectoral effects: Increasing reach, gaining access to communities, building on relationships of trust, complementing interventions, joint monitoring, and covering emerging topics. • Some of these effects (increasing reach, gaining access to communities, building on relationships of trust, joint monitoring) increase the efficiency of activities by taking advantage of the fact that an activity is already working in a community that another activity seeks to enter. The benefits of collaboration are on the side of the new activity that uses existing community structures, builds on existing relationships with community leaders, and saves resources by tapping into ongoing data collection efforts. • Complementing interventions and covering emerging topics are cross-sectoral effects that profit from the complementarity of activity objectives. In the case of complementing interventions, activities work with the same actor or institution but provide different kinds of support which, if provided at the same time, remove bottlenecks for results achievement on both sides. For example, this can be the case if one activity increases demand for a service, and the other activity improves supply in terms of quality or quantity. In the case of covering emerging topics, gaps in the approach of an existing activity are filled by a new activity that provides expertise and tools. • The RFSAs’ collaborative practices show that activities have to invest considerable time in communication and coordination to achieve the desired effects of inter-activity collaboration. Coordination must ensure that activities address the same community members in the same capacity and there are no scheduling conflicts preventing community members from participating in trainings and other local events. • Among the preconditions for achieving cross-sectoral collaboration are flexibility in design and implementation, substantial overlap in the target area and implementation period, complementary results attribution, and alleviating concerns regarding competitive disadvantages. The experiences of USAID/DRC activity managers show that these preconditions are complicated to achieve in practice, due to misaligned project cycles, incompatible reporting requirements, and concerns of implementing partners. 37 5. ASSESSMENT QUESTION 6: RECOMMENDATIONS Assessment Question 6: How could integrated programming better contribute to the effectiveness and efficiency of integrated activities and strengthen their contributions to CDCS Objectives? Activities contribute to the USAID/DRC CDCS Development Objectives (DOs) by achieving the activity objectives set out in their agreements and contracts. This means that whenever integrated programming contributes to the effectiveness and efficiency of integrated activities, it immediately also strengthens the activities’ contributions to DOs. The following recommendations address USAID/DRC and suggest changes to internal practices that facilitate intra- and inter-activity integration. USAID/DRC should increase the use of “Integration by Design.” The assessment focused on two forms of integration: “Integration by Design,” i.e., through the implementation of multi￾sectoral activities, and “Integration post Design,” i.e., through the collaboration of activities. The assessment recognizes the challenges in managing multi-sectoral approaches, especially if it requires the coordination of funding streams and monitoring requirements by several technical offices. However, the BHA RFSAs’ experiences show promising solutions to challenges in the local sequencing and layering of interventions. USAID/DRC’s experiences with the management of P-DEC would require further study to yield lessons learned for more effective coordination among technical offices. USAID/DRC should strengthen cross-office information exchange in geographic clusters. Activity managers underlined the importance of learning about the work of other technical offices in sufficient detail so that opportunities for collaboration and potential synergies between activities can be identified. Given the importance of geographic overlap for successful inter-activity collaboration, we believe that meeting in geographic clusters is the most efficient way to facilitate cross-office information exchange for increased integration. A good practice example for these types of meetings is USAID/Uganda’s Karamoja Livelihoods Cluster (USAID/ULA 2021) where activity managers and implementing partners working in Uganda’s Karamoja region regular meet, learn about topics of joint interest, discuss potential collaborations, and jointly monitor their progress. USAID/DRC should standardize practices in support of integration. The assessment’s findings and conclusions suggest several initiatives that could support inter-activity collaboration by standardizing Mission practices. First, the Mission should share best practice in formulating solicitations, awards, and contracts that strongly incentivize collaboration across the Mission. Second, some activity managers shared experiences that indicate uncertainty about the roles they can play in promoting collaboration between activities. Providing information to activity managers about best practices in facilitating collaboration and communication with implementation partners could help maximize the potential for integration among existing activities. Third, the Mission should ensure that information supporting collaboration is more readily available to activity managers. This could include making a summary on collaboration and coordination part of AOR/COR handover documents, as some activity managers suggested. Also, not all activity managers interviewed by the assessment team knew where to obtain information on other USAID/DRC activities in relevant target zones. USAID/DRC should make use of Participate Adaptive Modeling to facilitate the adaptative management of multi-sectoral approaches. Activities with integrated, multi- 38 sectoral approaches are challenging to manage. An activity that combines dozens of individual interventions from several sectors comes up against two bottlenecks. First, regional and community￾level coordinators often face an overwhelming workload in coordinating local interventions and community groups across sectors. Second, community members can feel disoriented by the number of requests to participate in committees, communal activities, and training. A review of USAID￾funded food security program evaluations reports that “without exception, all [reviewed evaluations] had recommendations to reduce the number of interventions” in response to these bottlenecks and focus on interventions with a high impact and the best chances of achieving sustainability (IMPEL 2020b). While the RFSAs have found ways to mitigate the effect of these challenges, they currently lack an effective means to systematically evaluate adaptive measures on an ongoing basis, so that they could streamline their approach over time. The complex Theories of Change that the activities have produced do “not have a clear way of identifying the relative weight/importance of each of the pathways in addressing the problem” (IMPEL 2020a). In selecting approaches, “the project worked to cover pathways and tried to ensure some balance” (IMPEL 2020a). However, “this approach does not necessarily help prioritize the pathways that are likely to have the most significant impact” (IMPEL 2020a). Participatory Adaptive Modeling (PAM) would provide integrated, multi-sectoral activities with a possibility to manage the complexity of their approaches and home in on effective interventions and adaptive measures. PAM is a nimble and efficient approach that can be deployed quickly and regularly. The Bayesian change models that PAM produces provide a comprehensive model of an activity’s approach to generating change among its target populations. They show which context factors, implementation processes, and results are connected and how strong these connections are. The focus on levers, bridges, and ultimate outcomes helps to separate the wheat from the chaff when managing complex activities. Scenario analyses of the models can be used to identify change processes with strong expected effects and simulate the effects of context changes and adaptive measures. This enables activity managers and leadership to systematically compare the likely effects of alternative courses of action before making a decision, based on the collective knowledge and experiences of their staff and partners. 39 ANNEXES Annex I Scope of Work..............................................................................................40 Annex II Data Collection Tools.................................................................................47 Annex III List of Informants ......................................................................................55 Annex IV Core Behavior Changes ............................................................................57 Annex V Scenario Analyses .......................................................................................59 Annex VI References ..................................................................................................62 40 ANNEX I SCOPE OF WORK Statement of Work / Description of Requested Services 1. Background The USAID Mission to the Democratic Republic of the Congo (DRC) is undertaking a Mid-Course Stocktaking (MCST) to review the current Country Development Cooperation Strategy (CDCS). One important objective of the MCST is to examine how changes in context and results thus far confirm or challenge the core assumptions of the strategy. Four learning questions were formulated to guide the review process: 1. What progress have we made to date towards the achievement of the CDCS goal? 2. What is the status of critical assumptions and changes in context, along with related implications for performance? 3. To what extent has the implementation of the three pillars of the strategy (Integration, Partnership, and Adaptation and Risk Management) and the integration of the cross-cutting themes (Youth and Gender) contributed to progress on the DOs? 4. How do we adapt or modify our strategy to address changes in the operating environment and achieve our results? What progress can we expect by considering the adaptation measures selected? Learning Question 3 highlights the importance of cross-sectoral integration as one of the three pillars of the current CDCS. A similar question features in the CDCS Learning Agenda on how integration improved USAID’s work within and across activities. 2. USAID/DRC Integrated Programming Cross-sectoral integration includes coordination and cooperation between activities from different development sectors (inter-activity integration) as well as the integration of interventions in multi￾sector activity designs (intra-activity integration). The integration of development and humanitarian interventions to simultaneously “address emergency situations and the factors that precipitate them” is a particular focus of the CDCS. The MCST includes an integrated programming assessment to examine “whether and how integrated programming contributes to greater development outcomes, particularly in regard to the transition from humanitarian to development programming.” Cross-sectoral integration starts with the collaboration between technical teams at the Mission in the design, procurement, and monitoring of activities. This collaboration determines the incentives and constraints for inter-activity and intra-activity cross-sectoral integration. The Mission is currently implementing several activities that combine interventions from different development sectors through sequencing, layering, and integration. Prominent examples are the Resilience Food Security Activity in Kasaï Oriental (Budikadidi) and the South Kivu Food Security Project (FSP-Enyanya), both of which operate in the nexus of humanitarian-to-development contexts. Their intra-activity integration includes agricultural interventions to increase household incomes, governance interventions, and interventions to improve health, nutrition, and WASH. The Integrated Health Program (IHP) is an example of inter-activity integration through collaboration with BHA, DRG and EG activities. 3. Assessment Purpose The purpose of the Integrated Programming Assessment (IPA) is to contribute to the Mid-Course Stocktaking. The assessment will address Learning Question 3 most directly on how implementing the integration pillar of the strategy has contributed to progress on the DOs. The IPA will also offer evidence on how integrated programming has contributed to the CDCS goal (Question 1), uncover key performance implications for how integrated programming has responded to changes in context 41 (Question 2), and provide insights for adaptive management at the activity level that have implications for strategic decision making (Question 4). The IPA will address both inter-activity and intra-activity integration and incorporate the perspectives of the Mission’s technical offices and implementing partners involved in cross-sectoral integration. The IPA's purpose is to distill lessons learned that are concrete and actionable, so that they can inform CDCS strategy adaptation. 4. Assessment Questions • How is the Mission fostering and sustaining a culture internally across teams to operationalize and sustain the integrated approach? Have old organizational behaviors been modified or new behaviors initiated to reinforce the integrated approach? What role has leadership at the Mission or office level played in shaping that internal culture and any changes generated as a result? • How are the responsibilities and coordination of management and decision-making authority for activity design, procurement, and monitoring shared across teams? How and when do those teams work together, and how are concerns or disputes resolved? How do they identify and determine attribution of results achieved through joint efforts contributing to an integrated DO? • How have relationships and roles between the Mission's internal teams and their implementing partners been defined and operationalized for any cross-cutting activities? Have any new behavioral practices been adopted across these organizational relationships to facilitate implementation? • At which points in their program logic do integrated activities seek cross-sectoral integration? Which cross-sectoral effects do they hope to achieve? How do actors at such points of integration (staff, community volunteers, community members) collaborate? Which challenges do they face and how do they overcome them? • How do activities from different sectors collaborate? How do they share information, sequence or layer interventions, or jointly plan interventions? Which cross-sectoral effects do they hope to achieve? How do actors from different activities collaborate? Which challenges do they face and how do they overcome them? • How do intra- and inter-activity integration influence the effectiveness of activities and their contributions to CDCS Objectives? How much does cross-sectoral integration add to the effectiveness of interventions? Which sectoral interventions are understood to profit most strongly from cross-sectoral integration, which less so? • How could integrated programming better contribute to the effectiveness and efficiency of integrated activities and strengthen their contributions to CDCS Objectives? 5. Methodology Approach To ensure that the lessons learned are concrete and actionable, the IPA will follow two design principles: 1) Pause and reflect for adaptive management, and 2) in-depth case studies of multiple activities for comparison and generalization of lessons learned at the strategy level. The adaptive management approach will allow the Mission to learn about integration where it effectively supports activities in strengthening their contributions to DOs. The case studies of carefully selected integrated and non-integrated activities will provide an in-depth understanding of implementation processes to reveal how integration can increase the effectiveness of activities. 42 Integrated Programming Assessment in Three Steps The IPA will look at both inter-activity and intra-activity integration and incorporate the experiences of technical offices and implementing partners with cross-sectoral integration. The assessment will proceed in three steps. First, it will provide in-depth case studies that look at the implementation processes of two integrated multi-sectoral activities, Budikadidi and FSP-Enyanya. Second, it will include focused assessments of the Integrated Health Program and the Fall Armyworm Activity to understand inter-activity integration and to further explore differences between integrated and non-integrated programming. Third, it will involve technical offices in inter￾office exchanges to look at the culture and practice of collaboration among technical offices and its effect on activity design, procurement, and monitoring. Case Selection The IPA will study intra- and inter-activity integration based on the experiences of four representative activities: Resilience Food Security Activity in Kasaï Oriental (Budikadidi), South Kivu Food Security Project (FSP-Enyanya), Integrated Health Program (IHP), and Management of Fall Armyworm in Maize for Smallholder Farmers in the Democratic Republic of Congo (Fall Armyworm Activity). The study team will engage technical offices, prime implementing partners, and consortium members of the four activities. The focus of the assessment will be Budikadidi and FSP-Enyanya, two integrated activities currently implemented by the Bureau for Humanitarian Assistance (BHA). The activities’ overall goal is to improve food and nutrition security and the economic well-being of vulnerable households. The multi-sectoral approach of these activities includes agricultural interventions to increase household incomes, interventions in health, nutrition, and WASH, and well as interventions to improve local governance. This combination of interventions places the activities squarely in the nexus of humanitarian-to-development contexts. Selecting these activities for further study will allow the IPA to contribute to the Mission’s ability to realize the benefits of cross-sectoral integration at the intersection of humanitarian and development programming. Both activities were evaluated at baseline and mid-term by the Tulane University School of Public Health and Tropical Medicine under the BHA Implementer-Led Evaluation and Learning (IMPEL) Associate Award. Data collection included population-based surveys at both time points. To date, the survey data has been mainly used to describe changes in performance indicators. Reports indicate that the data would provide a sound basis for a detailed analysis of cross-sectoral integration. Both activities include collaborations with IHP in their implementation approach. Studying the successes and challenges of collaboration between Budikadidi, FSP-Enyanya, and IHP from the perspectives of each implementing partner will provide the IPA with an opportunity to understand the drivers of successful inter-activity integration in concrete implementation contexts. Step I: In-Depth Case Studies •Budikadidi •FSP-Enyanya Step II: Focused Assessments •Integrated Health Program •Fall Armyworm Activity Step III: Inter-Office Exchange •A/CORs of integrated activities 43 As part of their multi-sectoral approaches, the BHA activities train farmers in agricultural techniques to improve yields and increase household incomes. The Fall Armyworm Activity trains smallholder maize producers in good agricultural practices and integrated pest management to increase maize yields. Comparing the experiences of the three activities regarding these shared approaches and objectives will allow the IPA to better understand differences in efficiency and effectiveness between integrated and non-integrated programming. Workshop Methods The IPA will employ Participatory Adaptive Modeling in pause and reflect workshops with IPs. The assessment will use Participatory Adaptive Modeling (PAM) to examine how the theories of change of the BHA activities are operationalized and analyze how integration contributes to activity results. Participatory Adaptive Modeling The PAM process includes validating how each activity’s theories of change is being implemented to ensure they represent the current on-the-ground approach of the activities. The study team will ensure that the TOCs break down high-level development hypotheses to concrete step-by-step change processes and identify actors and actions at each step. That way, activity management can read change processes from the chart, identify critical junctures where interventions from different sectors influence each other, and address actors at those junctures to better understand the effects of integration. This allows implementers to identify immediate actions to improve the effectiveness and efficiency of interventions. The TOC analysis will take place in two steps. First, the study team will analyze the available activity documentation, including the current TOC, and draft an updated (if necessary), action-oriented TOC. Second, the study team will validate the TOC in workshops with IP staff. In sector-specific workshops, the study team will invite IP staff (sector leads and frontline staff) to “take walks” along change processes in their area of expertise and identify missing steps and missing links. By conducting several workshops with fewer participants, we minimize IP staff time in validating the TOC. The study team will organize a cross-sectoral TOC validation workshop to discuss how interventions are expected to produce joint effects at cross-sectoral junctures in the intervention logic. The validated, action-oriented TOC will be developed and shared as an interactive logic flowchart on kumu.io. The TOC models will be handed over for further use and adaptation to activity accounts. Bayesian decision support systems (DSS) are a tool for facilitating pause and reflect workshops. They help activities to assess issues in cross-sectoral integration and identify adaptive measures to improve effectiveness. DSS are built on the familiar structure of Theories of Change. Through point and click, users can switch individual context factors and interventions on and off and witness the effects of these changes propagate through the system. By visualizing the ripple effects of context factors interventions across sector-specific and cross-sectoral results, DSS allow activities to estimate goal achievement under different scenarios and to systematically compare the likely effects of adaptive measures in changing circumstances. To turn a TOC into a DSS, the model structure is infused with information on how strongly a step in a change process is expected to contribute to the next. For the IPA, we will use available monitoring and evaluation (M&E) data and the experiential knowledge of IP staff to estimate the strength of expected contributions along major results chains. DSS allow the study team to deal with complex processes step-by-step and find the best data for each causal step. For the BHA activities, high-quality survey data is available. Information on expected contributions will be included in the decision support system in two steps. First, wherever possible, the study team will use M&E data to estimate the strength of contributions. 44 Second, the study team will convene sector-specific pause and reflect workshops to understand which interventions produce the desired effects and to which degree, and what role cross-sectoral integration plays in results achievement. The study team will ask straightforward questions to elicit effect estimates. For example, the team may ask: "Out of 10 mothers, how many regularly use soap and water when participating in Care Groups? How many do so without participating in Care Groups?" This elicitation will be done for all causal steps where M&E data is not available. The resulting decision support system will provide a comprehensive picture of an activity's intervention logic and expected effects, based on the best available knowledge. In cross-sectoral pause and reflect workshops (one per activity), the study team will use the DSS to jointly analyze the causes and effects of current cross-sectoral collaboration, identify recurring issues, and identify ways to improve intra-activity integration. Adaptive measures to strengthen the positive effects of cross-sectoral integration identified and discussed during the PAM process will provide the basis for scenario analyses. The study team will conduct these scenario analyses with the help of the DSS. The scenarios will highlight the likely effects of adaptive measures under varying contexts. 6. Expected Output The study will produce the following deliverables: ● Validated Theories of Change (one per BHA activity) that document the activities’ current on-the-ground approaches to integrated programming, visualize intervention logics, and identify responsible actors at critical steps in the creation of synergies between humanitarian and development programming. ● Decision support systems (one per BHA activity) in the form of Bayesian Belief Networks that allow activity management to systematically compare the likely effects of alternative courses of action in improving cross-sectoral integration and furthering the effectiveness and efficiency of integrated activities. Context changes and adaptive management decisions will be included in these systems to understand their effects across activity result areas. ● Guide for CORs and IP management on how to make the best use of decision support systems in the adaptive management of their activities (in the form of a video demonstration or how-to note). ● Final report that documents findings on the study questions with regard to the selected activities and provides conclusions and recommendations on how the effectiveness and efficiency of integrated programming can be improved so that it contributes to greater development outcomes. Potential uses of the assessment findings by USAID/DRC and implementing partners ● Understand how cross-sectoral integration of interventions in multi-sector activity designs has contributed to greater development outcomes. ● Improve implementation of integration as one of the three strategic pillars of the CDCS. ● Design future integrated activities in humanitarian-to-development contexts. ● Adaptive management of selected activities. Intended audience ● Participants in Mid-Course Stocktaking events. ● USAID/DRC Offices designing new integrated activities, especially at the intersection of humanitarian to development programming. ● Implementing partners currently implementing integrated activities. 7. Type of MEL support to be delivered and targeted participants MSSP Objective # and type of MEL service: ● MSSP Objective 3: Evaluation, Assessment and Studies. 45 o Assessment ● MSSP Objective 4: Collaborating, Learning and Adapting. o Activity-specific workshops on Theory of Change development and adaptive management Selected Geographic Administrative unit(s): ● No field work, no sampling. Data collection in online and in-person workshops in Kinshasa with headquarter and front-line staff of activity IPs. Workshops with staff residing in Goma will be held online. Analytical cross-sectoral and scenario workshops will include USAID staff. Participants and #: ● Workshop participants: o Sector-specific TOC Validation Workshops (3 workshops per BHA activity, 3-5 participants each). Participants: IP activity sector leads and area coordinators. Data collected: Detailed information on intervention logic, including the role of context factors and inter-activity collaboration. Expected outcome: Validated, action-oriented Theories of Change for each activity that adequately represent the activity’s current on-the-ground implementation approach. o Cross-sectoral TOC Validation Workshop (1 workshop per BHA activity, 5-8 participants). Participants: A/COR, USAID/DRC and IP sector leads. Data collected: Information on intervention logic at cross-sectoral junctures of the activity’s integrated approach. Expected outcome: Deepened understanding of synergies and challenges in the cross-sectoral integration of the activities. o Sector-specific Pause and Reflect Workshops (3 workshops per BHA activity, 3-5 participants each). Participants: IP activity sector leads and front-line staff. Data collected: Conditional probabilities for steps in activity result chains not covered by M&E data. Expected outcome: Estimated effects of interventions across activity result areas. o Cross-sectoral Pause and Reflect Workshop (1 workshop per BHA activity, 5-8 participants). Participants: IP activity sector leads and front-line staff. Data collected: Cross-sectoral conditional probabilities for steps in activity result chains not covered by M&E data. Expected outcome: Improved point estimates for the added effects of integrated programming on outputs, outcomes, and impacts. ● Interviewees: o Three interviews with relevant IHP sector leads. Data collected: Information on collaboration with BHA activities. Expected outcome: Understand IHP’s perspective on collaboration with BHA activities. o Three interviews with relevant FAW activity sector leads. Data collected: Information on implementation processes. Expected outcome: Understand differences in efficiency and effectiveness between integrated and non-integrated programming. o Eight interviews with A/CORs of integrated activities. Data collected: Information on the causes of recurring issues in cross-sectoral integration, information on the causes and effects of collaboration culture and technical office collaboration. Expected outcome: Understand Mission perspective on the causes of successful cross-sectoral collaboration, identification of measures to improve technical office collaboration. Type of tool, or of capacity building topic, or of outreach topic to be developed: ● Participatory Adaptive Management (PAM) process document (incl. workshop designs). The methodology section provides more detail on PAM. ● Guide for CORs and IP management on how to make the best use of decision support systems in the adaptive management of their activities. The guide will be presented at the Scenario Analysis Workshop. 46 8. Specific Tasks to be completed Phase 1: Inception ● Visualize BHA activities’ Theories of Change (TOCs) as interactive logic flowcharts in Kumu (https://kumu.io/) based on the analysis of activity documentation (contract, AMELP, annual and quarterly reports, activity TOCs). ● Introduce USAID and IPs to the Participatory Adaptive Modeling (PAM) process in online meetings. The methodology section provides more detail on the PAM process. Phase 2: Data Collection and Analysis • Step 1: In-Depth Case Studies of Budikadidi and FSP-Enyanya o Hold sector-specific workshops with IP sector leads and field-level staff (3 workshops per activity) to “take walks” along change processes in their area of expertise, identify missing steps and missing links, and assess sectoral intervention logics and cross￾sectoral linkages. o Hold cross-sectoral workshops with A/COR, USAID/DRC and IP sector leads and field-level staff (1 workshop per activity) to discuss how interventions are expected to create cross-sectoral synergies and to understand challenges in cross-sectoral collaboration. o Ground-truth TOCs with activity monitoring and evaluation data. Where available, data on outputs and outcomes will be used to estimate the joint effects of interventions at the intersection of sectoral intervention logics. o Hold sector-specific pause and reflect workshops with IP sector leads and field-level staff (3 workshops per activity) to understand which interventions produce the desired effects and to which degree, and what role cross-sectoral integration plays in results achievement. The study team will ask straightforward questions to elicit effect estimates (see section on assessment methodology). Elicitation will be done for all causal steps where M&E data is not available. These workshops will compile the knowledge of IP sector leads and field-level staff gained over several years of activity implementation and make this knowledge available to activity management and the Mission in an interactive visual decision support system. o Hold cross-sectoral pause and reflect workshops with IP sector leads and field-level staff (1 workshop per activity) to home in on expected cross-sectoral interactions, identify recurring issues, and identify ways to improve intra-activity integration. • Step 2: Focused Assessments of IHP and FAW Activity o Conduct interviews with relevant IHP sector leads to understand the IHP perspective on collaborations with BHA activities. o Conduct interviews with relevant FAW activity sector leads to understand differences in efficiency and effectiveness between integrated and non-integrated programming. • Step 3: Inter-Office Exchange o Conduct interviews with A/CORs of integrated activities to understand the Mission’s perspective on the causes of successful cross-sectoral collaboration and identify measures to improve technical office collaboration. Phase 3: Reporting and Dissemination ● Create guide for A/CORs and IP activity management on how to make the best use of Bayesian decision support systems in the adaptive management of activities with multi-sectoral approaches. ● Produce final report that documents findings on synergies and challenges in the cross-sectoral approaches of the selected activities and provides conclusions and recommendations on how the effectiveness and efficiency of integrated programming can be improved. 47 ANNEX II DATA COLLECTION TOOLS The following data collection tools are provided: 1. Sectoral Theory of Change Workshop Design 2. Cross-Sectoral Theory of Change Workshop Design 3. Sectoral Pause and Reflect Workshop Design 4. Cross-Sectoral Pause and Reflect Workshop Design 5. IHP Key-Informant Interview Guide 6. FAW Activity Key-Informant Interview Guide 7. AOR/COR Key-Informant Interview Guide 1. Sectoral Theory of Change Workshop Design Duration: 150 min. Purpose The purpose of the sectoral workshops is to harvest IP staff’s sector expertise and experiential knowledge to • make sure that the draft TOC is complete (i.e., all major context factors, external collaborations, actions, and connections are represented), • clarify remaining questions on actors, the nature of actions, or their purpose that may have arisen during the review of activity documentation, • rank influencing factors from most to least influential at each causal step with more than two influencing factors and identify major reasons for this ranking, and • understand how cross-sectoral joint effects are expected to be achieved and what the expected added benefits of cross-sectoral integration are. One sectoral workshop will be organized for each sector included in the activity’s multi-sectoral approach. The workshop will focus on those parts of the whole-of-activity TOC that belong to the respective sector. Participants (3-5) • At least one staff member of the activity consortium lead who acts as a sectoral expert and has knowledge of the activity’s design logic including the design of the overall approach, the role of external collaboration partners, and the design of individual sectoral interventions. • At least one staff member of the activity consortium lead who acts as a coordinator of local activity implementation (regional or lower) and has detailed knowledge of the activity’s implementation processes in several geographic units (e.g., several communities). • At least one staff member of an activity consortium member other than the lead who acts as a coordinator of local activity implementation across several geographic units (e.g., several communities), if possible, with a sector focus. • If such positions exist, an expert for a cross-cutting theme or objective such as social behavior change communication, gender, or youth, from either the consortium lead or another consortium member. If possible, participants with detailed knowledge of implementation processes should predominate. Preparation • Define “walks” through the TOC. A walk starts with an intervention or a cluster of interventions and ends with one or several ultimate outcomes. A “walk” traces a change process. It tells the story of how interventions are connected to necessary intermediary steps and expected ultimate outcomes. A “walk” includes context factors, external collaborations, and influences from other actions that are connected to the change process. 48 The walks should jointly cover all parts of the TOC that belong to the sector. Not more than three walks should be defined. • Identify and mark all remaining questions regarding actors, the nature of actions, or their purpose along the walks. • Send a PDF of the draft map to participants not more than 2 hours before the meeting. • Prepare an introductory presentation of the assessment purpose, assessment timeline, the definition of “action-focused” TOCs (what they cover, what they exclude, how they will be used in the assessment, how they are “read”), and the workshop agenda. Agenda Time Activity 00:00 – 00:25 Introduction. • Introductory presentation. • Personal introductions: Name, function, one additional piece of information about the assessment or workshop they would like to hear before we start. • Answer questions. 00:25 – 01:00 First walk. Tell the change story including external collaborations and influences from other actions. • When coming across a question mark, ask: o “Did we identify the correct actor of this action?” o “Did we understand the nature of the action correctly?” o “Did we understand the purpose of this action correctly?” • When coming across a point of cross-sectoral integration, ask: o “How is the cross-sectoral collaboration supposed to look?” o “What do you think is the added benefit of integration here? What would you expect to be different if each action contributing to this step was implemented separately?” • When coming across a causal step with more than two influencing factors, ask: o “Which of these contributing factors, do you believe, has the strongest influence on this causal step and why?” o “Which of these contributing factors, do you believe, has the least influence on this causal step and why?” o “Which comes second/third… in terms of influence?” • After finishing the walk, ask: o “Are actions in this change process expected to make substantial contributions to other change processes in the TOC?” o “Are there other actions by the activity that are expected to contribute to causal steps in this change process?” o “How about support by IHP? Where is this support expected to play an important role in this change process, if at all?” 01:00 – 01:35 Second walk. 01:35 – 02:10 Third walk. 02:10 – 02:20 Next steps. Explain how we will use the results of this workshop. Announce that updated TOC will be sent for final review. 49 1. Cross-Sectoral Theory of Change Workshop Design Duration: 150 min. Purpose The purpose of the cross-sectoral workshops is to harvest IP staff’s sector expertise and experiential knowledge to • discuss the added benefits of intra-activity integration in achieving cross-sectoral effects, • discuss challenges in collaboration across sectors and identify solutions to these challenges, • identify additional potential for cross-sectoral integration. Participants (6-10) • For each sector, at least one staff member of the activity consortium lead who acts as a sectoral expert and has knowledge of the activity’s design logic including the design of the overall approach, the role of external collaboration partners, and the design of individual sectoral interventions. • At least one staff member of the activity consortium lead who acts as a coordinator of local activity implementation (regional or lower) and has detailed knowledge of the activity’s implementation processes in several geographic units (e.g., several communities). • Staff members of activity consortium members other than the lead that act as coordinators of local activity implementation across several geographic units (e.g., several communities) and jointly cover implementation across all sectors involved in the activity design. • If such positions exist, an expert for a cross-cutting theme or objective such as social behavior change communication, gender, or youth, from either the consortium lead or another consortium member. If possible, participants with detailed knowledge of implementation processes should predominate. Preparation • Share PDF of updated TOC and request feedback until one day before cross-sectoral workshop. • List out all joint (inter- and/or intra-activity) cross-sectoral effects and select up to six for discussion in this workshop. Cross-sectoral effects are joint effects of actions from different sectors of the same activity and effects to which other activities contribute. • Prepare a presentation of the selected six cross-sectoral effects. • Prepare online documents with questions for each breakout group. Agenda Time Activity 00:00 – 00:30 Introduction. • Introductory presentation. • Personal introductions: Name, function, one additional piece of information about the assessment or workshop they would like to hear before we start. • Answer questions. 00:30 – 00:40 Present joint effects of cross-sectoral action 00:40 – 01:20 First cross-sectoral breakout session. Each group discusses one joint effect. Groups are formed so that each sector is represented that contributes to the joint effect discussed by the group. Each group is asked to discuss the following questions and note down results in the shared online document: • What would constitute the best approach in working together to achieve this joint effect? How do you currently engage with each other? 50 • What would be different with regard to this joint effect if you worked independently of each other? How would it influence the efficiency or effectiveness of your work if the other sector didn't contribute to the joint effect? • What are challenges in working together to achieve this joint effect? How could these challenges be overcome? • How could other sectors of this activity help increase the efficiency or effectiveness of your actions with regard to this joint effect? • Are there actions undertaken by other sectors of this activity that constrain your ability to achieve this joint effect? What could they do differently to reduce this constraint? 01:20 – 01:35 Plenary share-out. Please share one aspect of your collaboration that is working well and one thing that needs improvement. 01:35 – 02:15 Second cross-sectoral breakout session. 02:15 – 02:30 Plenary share-out. Please share one action that you could take tomorrow to improve collaboration. 2. Sectoral Pause and Reflect Workshop Design Duration: 150 min. Purpose The purpose of the sectoral workshops is to harvest IP staff’s sector expertise and experiential knowledge to provide numerical estimates for the contributions of interventions along change processes in the activity’s action-focused Theory of Change. One sectoral workshop will be organized for each sector included in the activity’s multi-sectoral approach. The workshop will focus on those parts of the whole-of-activity TOC that belong to the respective sector. Participants (3-5) • At least one staff member of the activity consortium lead who acts as a sectoral expert and has knowledge of the activity’s design logic including the design of the overall approach, the role of external collaboration partners, and the design of individual sectoral interventions. • At least one staff member of the activity consortium lead who acts as a coordinator of local activity implementation (regional or lower) and has detailed knowledge of the activity’s implementation processes in several geographic units (e.g., several communities). • At least one staff member of an activity consortium member other than the lead who acts as a coordinator of local activity implementation across several geographic units (e.g., several communities), if possible, with a sector focus. • If such positions exist, an expert for a cross-cutting theme or objective such as social behavior change communication, gender, or youth, from either the consortium lead or another consortium member. If possible, participants with detailed knowledge of implementation processes should predominate. Preparation • Identify actions where the threshold for presence needs to be deliberated. This is the case whenever success and failure do not have a clear dichotomous nature. For example, if the action in question is “Communities maintain literacy centers in villages,” the threshold for the presence of the action is relatively easy to determine: Is the literacy center still functional at the end of the project? If it is functional, then the action is successfully implemented. In some cases, however, it might be less clear when an action is taking place as intended. For example, “SILC members successfully raise goats.” When do we say that this is true, i.e., that a member is successfully raising goats? Is that the case when the member is 51 raising enough goats so that he or she can regularly slaughter some of them for consumption and remain with a herd large enough to maintain itself? Or should “success” also include the ability to sell some of these goats? In these cases, it is important to first define a threshold jointly with workshop participants so that they can discuss estimates of the number of, e.g., SILC members successfully raising goats based on the same standards for success. • Prepare an Excel file with one tab for each conditional probability table that must be elicited during the workshop. Three scenarios must be differentiated: (1) One contributing element, (2) two contributing elements, (3) three or more contributing elements. Agenda Time Activity 00:00 – 00:30 Introduction. • Introductory presentation: Where we are in the process, expert elicitation of effects, why participants’ expertise matters. • Personal introductions: Name, function, and one additional piece of information about the assessment or workshop they would like to hear before we start. • Answer questions. 00:30 – 00:45 Joint test discussion • Effect estimates for a result with two contributing elements. 00:30 – 02:30 Elicitation of sectoral effect estimates. • Start with interventions and move along change processes towards activity objectives. • Elicit effect estimates at each causal step, noting down the results in the correct tab of the Excel file. 3. Cross-Sectoral Pause and Reflect Workshop Design Duration: 150 min. Purpose The purpose of the sectoral workshops is to harvest IP staff’s sector expertise and experiential knowledge to provide numerical estimates for the contributions of interventions to cross-sectoral effects in the activity’s action-focused Theory of Change. Participants (6-10) • For each sector, at least one staff member of the activity consortium lead who acts as a sectoral expert and has knowledge of the activity’s design logic including the design of the overall approach, the role of external collaboration partners, and the design of individual sectoral interventions. • At least one staff member of the activity consortium lead who acts as a coordinator of local activity implementation (regional or lower) and has detailed knowledge of the activity’s implementation processes in several geographic units (e.g., several communities). • Staff members of activity consortium members other than the lead that act as coordinators of local activity implementation across several geographic units (e.g., several communities) and jointly cover implementation across all sectors involved in the activity design. • If such positions exist, an expert for a cross-cutting theme or objective such as social behavior change communication, gender, or youth, from either the consortium lead or another consortium member. If possible, participants with detailed knowledge of implementation processes should predominate. 52 Preparation • Prepare an Excel file with one tab for each conditional probability table that must be elicited during the workshop. Three scenarios must be differentiated: (1) One contributing element, (2) two contributing elements, (3) three or more contributing elements. • Identify 9 to 12 cross-sectoral effects where collaboration across sector teams likely plays an important role. Agenda Time Activity 00:00 – 00:30 Introduction. • Introductory presentation: Effect estimates and integration, elicitation in cross-sectoral groups, cross-sectoral effects. • Personal introductions: Name, function, and one additional piece of information about the assessment or workshop they would like to hear before we start. • Answer questions. • Formation of cross-sectoral breakout groups. 00:30 – 02:00 Discussion of contributions to cross-sectoral effects in breakout groups. • Estimates for contributions to 3-4 cross-sectoral effects per group. 02:00 – 02:30 Sharing of experiences with elicitation process in plenary. • What were your experiences with the elicitation process? • What was helpful? What was challenging? 4. IHP Key-Informant Interview Guide Duration: 90 min. Purpose The purpose of the interviews with IHP representatives is to get their perspective on collaborations with the two BHA activities “Budikadidi” and “FSP-Enyanya.” Preparation Compile a list of Budikadidi and FSP-Enyanya actions that, according to these activities, IHP actions contributed to or were expected to contribute to. The Budikadidi and FSP-Enyanya actions on this list are potential joint effects of the collaboration with IHP. Interview Questions 1. Do you think that your work affected the work of Budikadidi and FSP-Enyanya? If so, in which ways? o Which aspects of your work have positively affected the work of Budikadidi and FSP-Enyanya? o Which aspects of their work have been positively affected? 2. Are there any ways in which your work has profited from the work of Budikadidi and FSP￾Enyanya? 3. How did you interact with Budikadidi and FSP-Enyanya, if at all? o Which people were involved in these interactions on both sides? Which means of communication or collaboration did you use? o What were the immediate outputs of these interactions? How did these outputs contribute to further effects, if any? o What challenges did you face in your interactions with Budikadidi and FSP-Enyanya? What solutions did you find to these challenges? 53 4. In your understanding, which factors were decisive for the successes you achieved in your collaboration with Budikadidi and FSP-Enyanya? Which factors, if present, would have allowed additional successes? 5. From our conversations with Budikadidi and FSP-Enyanya, we understand that your work did or was expected to support their work in several ways. I would like to discuss these actual or potential synergies with you one by one. Describe actual/potential joint effect and IHP’s actual/expected contribution. For each effect on your list, ask the following questions: o Do you think that your work contributed to this potential joint effect? If so, do you believe that jointly working towards this goal has created any synergies between your activities, for example by increasing your efficiency or effectiveness? o What would have been different for IHP if Budikadidi and FSP-Enyanya hadn’t been working towards this goal? o Have there been any direct interactions with Budikadidi and FSP-Enyanya around working towards this goal? How did these interactions unfold? Who was involved and what forms of communication/collaboration did you engage in? o What challenges did you encounter and what solutions did you find, if any? 5. FAW Activity Key-Informant Interview Guide Duration: 60 min. Purpose The purpose of the interviews is to provide the basis for a comparison of Budikadidi’s and Enyanya’s multi-sector approach with FAW activity’s single-sector approach. The comparison will focus on the advantages and disadvantages regarding community-level implementation. The interview identifies community-level activities as points of comparison, asks about enabling and hindering factors, and tries to identify additional activities that would support success. Interview Questions 1. If I was living in a community that you target, how would I notice your presence? What activities are you performing on the community level? Collaboration with other activities 2. I understand that you are sometimes working in target areas where you collaborate with other activities and sometimes you are going alone. Why are you collaborating with other USAID activities such as Budikadidi? What benefits do you expect? a. For each of these benefits, to what degree are you able to realize them? b. What challenges are you facing in realizing them? 3. How do you work together with other USAID activities? a. How do you coordinate your work? b. How do you implement community-level activities together? 4. What do you do when there is no other USAID activity to collaborate with? How does your approach to a target community differ if you are on your own? a. What are the advantages of going alone? b. What are the disadvantages of going alone? Activities on community level For each activity mentioned in response to Q1: 5. When you are implementing [this activity] in a target community, what usually makes it easier for you to achieve success? What makes it harder? 6. Imagine you could broaden your approach and engage in additional community-level activities, what additional activities would help you succeed with [this activity]? 54 6. AOR/COR Key-Informant Interview Guide Duration: 60 min. Purpose The purpose of the interviews with activity managers is to get their perspective on cross-sectoral integration. This includes their perspective on internal collaboration, i.e., between TOs, and external collaboration, i.e., with implementing partners involved in cross-sectoral activities. Interview Questions We are interested in learning more about how the Mission is putting its integration strategy into practice. Integration can mean many things, so please give me a minute to explain how we understand integration. We are especially interested in integration across offices, or what we call cross-sectoral integration. This includes the collaboration between activities managed by different offices. Such integration would, for example, take place when the Integrated Health Program collaborates with an EG activity. Cross-sectoral integration also includes the collaboration of different offices in jointly managing a multi-sectoral activity. A. Collaboration in general 1. [Individual level:] What are your experiences with collaboration between offices? How does that benefit your work? 2. [TO/mission level:] What would allow you to engage in more collaboration between offices? 3. [Mission level:] Are you aware of any initiatives or measures at the Mission that would allow teams to engage in more collaboration between offices? 4. [Mission level:] What role has leadership at the Mission or office level played in shaping the internal collaboration culture? B. Collaboration between activities managed by different offices 5. What role do you play as an activity manager in supporting collaboration between activities? 6. When you think back to your experiences in managing activities, what do you think, under which circumstances can activities of different offices successfully collaborate with each other? 7. Since 2020, have there been any changes with regard to collaboration between activities managed by different offices? C. Collaboration between offices in managing a joint activity 8. [Internal:] How do you share roles and responsibilities across TOs when it comes to the (a) design and procurement and (b) the monitoring of joint activities? How and when do teams from different TOs work together, and how are concerns or disputes resolved? 9. [Internal:] How do you currently identify and determine the attribution of results achieved through joint efforts? 10. [External:] For joint activities, how are relationships and roles between internal teams and implementing partners defined and operationalized? Is there any difference to the practice around activities managed by a single office? 11. Since 2020, have there been any changes with regard to collaboration between offices around jointly managing activities? 55 ANNEX III LIST OF INFORMANTS Table 3 – USAID/DRC Staff (KIIs) Name Position Date Monzambe, Charly Partnership for the Development of Eastern Congo (P-DEC) AOR April 21, 2023 Mujinga, Monique Education is the Response Activity AOR April 20, 2023 Mukuna, Anne-Marie Media Sector Development Activity AOR April 26, 2023 Mutendo, Richard Integrated Health Program (IHP) COR April 28, 2023 Nsamba, Elodie Shade-Grown Cocoa Activity AOR April 19, 2023 Ntoya, Ferdinand End Malaria Project COR April 21, 2023 Obolensky, Dimitri Resilience Food Security Activity in Kasaï Oriental (“Budikadidi”) Activity Manager April 21, 2023 Schweisguth, Melissa Strengthening Livelihoods and Resilience (SLR) Activity COR April 20, 2023 Vollersten, John South Kivu Food Security Project (“FSP￾Enyanya”) Activity Manager April 19, 2023 Table 4 – BHA RFSA Implementing Partner Staff (Workshops) Name Position Date Amani, Edmond FSP-Enyanya Purpose 1 Manager March 2023 Bahati, Theodose FSP-Enyanya Purpose 2 Manager March 2023 Busaka, Augustin Budikadidi Value Chain Specialist Feb – Mar 2023 Coulibaly, Aly Budikadidi Lead Governance Feb – Mar 2023 Habamungu, Jean-Marie FSP-Enyanya Purpose 2 Team Leader March 2023 Itambwe, Itoskit FSP-Enyanya Purpose 1 Team Leader March 2023 Kabongo, Jean Budikadidi Literacy Supervisor Feb – Mar 2023 Kahia, Alex Budikadidi Health Coordinator Feb – Mar 2023 Kambale, Jeadot Nzilamba FSP-Enyanya Purpose 1 Team Leader March 2023 Kitumaini, Claude FSP-Enyanya Purpose 3 Team Leader March 2023 Kouazounde, Benjamin Budikadidi P3 Purpose Lead Feb – Mar 2023 Lubangi, Louis Budikadidi Technical Lead Nutrition Feb – Mar 2023 Lufungulo, Emmanuel FSP-Enyanya Manager Bonne Gouvernance March 2023 Mpiana, Joseph Budikadidi SILC Manager Feb – Mar 2023 Mpoyi, Cedrick Budikadidi Facilitator Coordinator Feb – Mar 2023 Muepu, Joseph Budikadidi Agriculture and Livelihood Specialist Feb – Mar 2023 Muhimuzi, Justin FSP-Enyanya Purpose 2 Manager March 2023 Muliri, Daniel FSP-Enyanya Purpose 3 Manager March 2023 Ngoyi, Philippe Budikadidi Facilitator Coordinator Feb – Mar 2023 Nibasumba, Marcel FSP-Enyanya Purpose 1 Manager March 2023 Nono, Mahili Budikadidi Technical Lead WASH Feb – Mar 2023 Ntumba, Baudouin Budikadidi Facilitator Coordinator Feb – Mar 2023 Quarshie, James Budikadidi Chief of Party Feb – Mar 2023 Semakoma, Viateur Ndajimana FSP-Enyanya Purpose 2 Team Leader March 2023 Volanarisoa, Rafaramalala Budikadidi Purpose Lead Feb – Mar 2023 Wabenga, Marie France FSP-Enyanya Responsable Genre et Jeunesse March 2023 Walelu, Medard Budikadidi Marketgarden Specialist Feb – Mar 2023 56 Table 5 – Other Activities (KIIs) Name Position Date Nshombo, Jean-Paul FAW Activity Deputy Chief of Party April 26, 2023 Cimanga, Raphael FAW Regional Coordinator Kasaï Oriental April 27, 2023 Mushitu, Jean FAW Regional Coordinator Haut Katanga April 27, 2023 Gaye, Moustapha FAW Activity Chief of Party April 13, 2023 Mutombo, Jean Michel IHP Health Systems Strengthening Specialist March 30, 2023 Mutamba, Isabelle IHP Health Specialist Kasaï Oriental March 30, 2023 Demba, Didace IHP Provincial Director Sud Kivu March 30, 2023 Kamango, Cosmas IHP Health Systems Strengthening Specialist March 30, 2023 Luendo, Jimmy IHP Health Specialist Sud Kivu March 30, 2023 Mingashanga, Constatin IHP Infectious Diseases Advisor March 30, 2023 57 ANNEX IV CORE BEHAVIOR CHANGES Core behavior changes include “bridges of integrated change” and ultimate outcomes. Bridges of integrated change are intermediate outcomes with a high betweenness centrality. Betweenness centrality measures how many times an element lies on the shortest path between two other elements in a causal network. In a radial network graph, such elements would tend to be found in the middle, on the shortest path between different “arms” of the network. In an action-focused Theory of Change, intermediate outcomes with a high betweenness centrality are behavior changes that connect a high number of interventions with a high number of desired outcomes. This means that these behavior changes are central to achieving integrated change. In the case of Budikadidi, couples making joint decisions on family planning, nutrition, and hygiene is such a bridging behavior change. There are six interventions that Budikadidi expects to contribute to this change, and there are six outcomes that joint decision-making is expected to positively affect, ranging from peaceful conflict resolution in the household to practicing birth spacing. Ultimate outcomes are behavior changes that make no further contributions to other behavior changes. An activity’s goal-level results typically include all ultimate outcomes. However, there are often goal-level results that make contributions to other goals. Studying ultimate behavior change outcomes provides, therefore, a more focused view on what success ultimately means for an activity. For the comparative scenario analyses, intermediate behavior changes with the ten highest betweenness centrality scores and all ultimate behavior changes were selected for each activity. These behavior changes are listed below. Table 6 – Budikadidi Core Behavior Changes Core Behavior Change Sector Type Communities peacefully resolve community conflicts Governance Ultimate Men do not engage in gender-based violence Governance Ultimate Mothers engage in exclusive breastfeeding Health Ultimate Households use high quality health and nutrition services Health Ultimate Households consume diversified diet Health Ultimate Communities do not practice open defecation Health Ultimate Women get pregnant later and practice birth spacing methods Health Ultimate CACs gather information, organize meetings, develop plans Governance Bridge CACs monitor plan implementation, mobilize community Governance Bridge Couples make joint decisions Governance Bridge Women occupy positions of authority Governance Bridge Community members learn how to read and write Governance Bridge Water committees maintain functioning water points Health Bridge Households follow recommended hygiene behaviors Health Bridge Households maintain permagardens Livelihoods Bridge Households maintain market gardens that produce substantial yields Livelihoods Bridge Households build up financial reserves Livelihoods Bridge Households sell agricultural products on local markets Livelihoods Bridge Women and girls work less time on household chores Livelihoods Bridge 58 Table 7 – FSP-Enyanya Core Behavior Changes Core Behavior Change Sector Type Communities resolve conflicts peacefully through community platforms Governance Ultimate PLW adopt practices for better nutrition during pregnancy, exclusive breastfeeding, child nutrition, prenatal consultations Health Ultimate Women, men, children increase consumption of nutrient-rich foods Health Ultimate Households adopt improved WASH practices and abandon open defecation Health Ultimate Community members access preventive, curative health care at health facilities Health Ultimate Women practice healthy timing and spacing of pregnancy Health Ultimate Farmers meet legal requirements as landowners to hold titles and don't pay illegal taxes Governance Bridge Small-holder farmers acquire land titles Governance Bridge Women and youth hold leadership roles in their communities Governance Bridge Men and women construct HH latrines and handwashing stations and purchase essential WASH supplies Health Bridge Couples make joint decisions about family healthcare Health Bridge Households build up their financial reserves Livelihoods Bridge Farmers increase agricultural production, especially of nutritious, lower-risk food crops Livelihoods Bridge Farmers sell products on local markets Livelihoods Bridge Farmers adopt control techniques to improve soil quality and erosion conditions Livelihoods Bridge Through PO/YBGs, farmers sell produce to wholesalers Livelihoods Bridge Men, women, youth effectively manage VSLAs Livelihoods Bridge 59 ANNEX V SCENARIO ANALYSES The following tables summarize core results of the comparative scenario analyses of the Bayesian change models for the BHA RFSAs Budikadidi and FSP￾Enyanya. Table 8 and Table 9 provide changes in success probabilities for three different levels of implementation: Very low (25%), low (50%), and high (100%). The percentages in parentheses refer to the assumed target output of interventions. For example, in the very low implementation scenario, it is assumed that interventions achieved 25% of the target output. For Table 8 and Table 9, comparisons are made to the extrapolated baseline. The extrapolated baseline assumes that no sector produced any outputs throughout the implementation period of the activity. So, what do the percentages in the table mean? For example, the 25.2% in Table 8, row “All Core Behaviors,” column “Very Low (25%)” under “Multi-Sectoral Approach,” means that on average, the success probabilities for all core behavior changes improved by 25.2% when Budikadidi’s full multi-sectoral approach was implemented to 25% of target outputs, compared to the success probabilities achieved in the extrapolated baseline scenario. Table 10 and Table 11 provide changes integration dividends for the same sectoral components and implementation levels as in Table 8 and Table 9. To calculate the integration dividends, a comparison is made to a scenario where the activities’ full multi-sectoral approach is implemented to the same output level. For example, the first column under “Only Governance Interventions” in Table 10 compares results achievement for a very low implementation scenario where only governance interventions are implemented to the results achievement for a very low implementation scenario where the full multi￾sectoral approach is implemented. To calculate the integration dividend, we divide the average results achievement for the single-sector scenario by the average results achievement for the multi-sector scenario and deduct the resulting number from 100%. Thus, the integration dividend represents the average shortfall in success probabilities of the single-sector scenario compared to the multi-sectoral scenario. The lower the integration dividend, the smaller the added benefit of multi-sectoral programming. For example, the 9.0% in Table 10, row “All Core Behaviors,” column “Very Low (25%)” under “Only Governance Interventions,” means that on average, the success probabilities for all core behavior changes worsened by 9% when only governance interventions were implemented to 25% of target outputs, compared to the success probabilities achieved when the full multi-sectoral approach is implemented to 25% of target outputs. The 9% represents the “Integration Dividend,” i.e., the boost added by a multi-sectoral approach compared to a governance-only approach, when implementation levels are very low. Table 8 – Budikadidi: Percentage Changes in Success Probabilities for Different Levels of Implementation, Comparison to Extrapolated Baseline Behavior Changes Multi-Sectoral Approach Only Governance Interventions Only Health Interventions Only Livelihoods Interventions Very Low (25%) Low (50%) High (100%) Very Low (25%) Low (50%) High (100%) Very Low (25%) Low (50%) High (100%) Very Low (25%) Low (50%) High (100%) All Core Behaviors 25.2% 37.6% 48.5% 12.4% 21.0% 30.1% 10.7% 19.3% 29.8% 7.9% 14.2% 24.8% Ultimate Outcomes 13.2% 18.2% 22.4% 7.5% 11.6% 13.9% 7.2% 12.4% 18.2% 3.0% 5.4% 9.5% Bridges 32.2% 48.9% 63.8% 15.2% 26.4% 39.5% 12.8% 23.4% 36.5% 10.7% 19.3% 33.8% 60 Governance￾related 22.8% 33.2% 43.3% 17.9% 30.4% 43.3% 7.9% 14.4% 20.3% 2.5% 4.2% 8.2% Health-related 17.1% 25.4% 34.8% 6.5% 9.9% 12.5% 12.2% 20.8% 30.4% 2.8% 5.4% 9.5% Livelihoods￾related 39.9% 60.7% 75.0% 12.8% 23.1% 36.2% 12.5% 24.2% 42.1% 22.6% 40.5% 69.6% Table 9 – FSP-Enyanya: Percentage Changes in Success Probabilities for Different Levels of Implementation, Comparison to Extrapolated Baseline Behavior Changes Multi-Sectoral Approach Only Governance Interventions Only Health Interventions Only Livelihoods Interventions Very Low (25%) Low (50%) High (100%) Very Low (25%) Low (50%) High (100%) Very Low (25%) Low (50%) High (100%) Very Low (25%) Low (50%) High (100%) All Core Behaviors 25.0% 39.1% 54.2% 10.3% 18.0% 28.3% 20.9% 19.5% 28.3% 6.1% 10.3% 15.6% Ultimate Outcomes 21.1% 34.6% 54.4% 3.5% 7.1% 13.3% 18.9% 31.6% 47.5% 0.2% 0.2% 0.8% Bridges 27.2% 41.6% 54.1% 13.9% 24.0% 36.5% 22.0% 13.0% 17.8% 9.3% 15.8% 23.7% Governance￾related 28.2% 44.1% 61.9% 23.9% 41.2% 61.9% 8.6% 15.0% 25.0% 0.0% 0.0% 0.0% Health-related 25.5% 38.1% 52.5% 4.7% 8.0% 10.3% 23.8% 37.6% 52.5% 0.2% 0.2% 0.7% Livelihoods￾related 22.4% 37.1% 51.2% 7.7% 14.2% 26.9% 25.7% 1.4% 2.2% 17.1% 28.9% 43.4% Table 10 – Budikadidi: Integration Dividend for Different Sectors and Different Levels of Implementation Behavior Changes Only Governance Interventions Only Health Interventions Only Livelihoods Interventions Very Low (25%) Low (50%) High (100%) Very Low (25%) Low (50%) High (100%) Very Low (25%) Low (50%) High (100%) All Core Behaviors 9.0% 9.9% 9.9% 10.3% 11.1% 9.6% 12.7% 15.4% 14.1% Ultimate Outcomes 4.9% 5.3% 6.4% 5.2% 4.6% 3.0% 8.8% 10.7% 10.5% Bridges 11.3% 12.6% 12.0% 3.3% 14.8% 13.5% 15.0% 18.1% 16.1% 61 Governance￾related 3.8% 2.1% 0.0% 11.5% 12.5% 11.1% 15.7% 20.0% 20.2% Health-related 8.3% 10.5% 13.2% 4.1% 3.6% 3.1% 11.4% 14.1% 15.6% Livelihoods￾related 7.1% 20.2% 19.3% 17.3% 19.6% 16.5% 10.5% 10.6% 3.4% Table 11 – FSP-Enyanya: Integration Dividend for Different Sectors and Different Levels of Implementation Behavior Changes Only Governance Interventions Only Health Interventions Only Livelihoods Interventions Very Low (25%) Low (50%) High (100%) Very Low (25%) Low (50%) High (100%) Very Low (25%) Low (50%) High (100%) All Core Behaviors 11.0% 13.7% 14.3% 9.6% 12.9% 15.0% 13.9% 18.9% 22.3% Ultimate Outcomes 13.4% 18.0% 21.9% 1.9% 2.6% 4.9% 16.2% 23.3% 30.5% Bridges 9.6% 11.4% 10.1% 13.8% 18.5% 20.5% 12.7% 16.4% 17.8% Governance￾related 2.6% 1.5% 0.0% 14.7% 20.2% 22.9% 19.8% 28.1% 36.4% Health-related 15.7% 19.8% 23.4% 1.4% 0.4% 0.0% 19.0% 25.2% 29.8% Livelihoods￾related 11.1% 14.7% 13.1% 15.9% 22.6% 26.5% 4.0% 5.3% 4.1% 62 ANNEX VI REFERENCES IMPEL (2020a). Mid-Term Evaluation of the Budikadidi Development Food Security Activity in the Democratic Republic of the Congo (DRC). IMPEL (2020b). Learning from Evaluations: A Review of 16 Mid-Term Evaluations of USAID-funded Food Security Development Programs from 2015-2020. Washington, DC: The Implementer-Led Evaluation & Learning Associate Award. USAID (2014). Local Systems: A Framework for Supporting Sustained Development. https://www.usaid.gov/sites/default/files/2022-05/LocalSystemsFramework.pdf (last accessed June 28, 2023). USAID (2022). 2022 Resilience Policy Revision. Draft as of December 2022. USAID (2023). Gender Equality and Women’s Empowerment Policy. https://www.usaid.gov/sites/default/files/2023-03/2023_Gender%20Policy_508.pdf (last accessed June 28, 2023). USAID/DRC (2020). Country Development Cooperation Strategy December 10, 2020 – December 09, 2025. USAID/Uganda Learning Activity (2021). Karamoja Livelihoods Cluster: How Did It Change Collaboration? https://pdf.usaid.gov/pdf_docs/PA00XS55.pdf (last accessed July 20, 2023).