USAID/GLOBAL HEALTH EVALUATION AND LEARNING SUPPORT ACTIVITY (GH EvaLS) PERFORMANCE EVALUATION OF BREAKTHROUGH ACTION: MOVING BEYOND COMMUNICATION July 2021 This publication was produced at the request of the United States Agency for International Development. It was prepared independently by ME&A Inc., its subcontractor Dexis Consulting Group and the evaluation team comprised of Julie Solo, Julianne Weis, Lynda Bardfield, Alexandria Schmall, Willow Gerber, Opeyemi Adeosun, Emmanuel Ogbudu, Souley Aboubacar, Mahamane Tahirou Ali Bako, Kacou Armand Tokou, and Djedou Martin Amalaman. PERFORMANCE EVALUATION OF BREAKTHROUGH ACTION: MOVING BEYOND COMMUNICATION July 2021 USAID Contract No. GS-10F-154BA/7200AA20M00003, GH EvaLS Assignment Number: 002 Period of Performance: November 6, 2020- May 15, 2021 Contracting Officer Representative (COR): Carl Hawkins Submitted by: Andrea Camoens, Chief of Party GH EvaLS 4350 East-West Highway, Suite 210 Bethesda, MD 20814-4410 Tel: (301) 652-4334 Email: acamoens@engl.com Cover Photo: Courtesy of Jonathan Torgovnik/Reportage by Getty Images. Courtesy of Paula Bronstein/The Verbatim Agency/Getty Images. Courtesy of Caitlin Loehr. DISCLAIMER The views expressed in this publication do not necessarily reflect the views of the United States Agency for International Development or the United States Government. This document is available online. Online documents can be located in the GH EvaLS website at https://ghevals.meandahq.com, Documents are also made available through the Development Experience Clearinghouse (http://dec.usaid.gov). Additional information can be obtained from: Global Health Evaluation and Learning Support Activity (EvaLS) 4350 East-West Highway, Suite 210 Bethesda, MD 20814-4410 Tel: (301) 652-4334 Website: https://ghevals.meandahq.com MID-TERM PERFORMANCE EVALUATION OF BA: LOCAL LESSONS, GLOBAL LEARNING iv ABSTRACT USAID invested in two separate but linked mechanisms as its flagship social and behavior change (SBC) programs: Breakthrough RESEARCH (BR) and Breakthrough ACTION (BA), with the shared strategic objective of increased integration of proven SBC interventions in health and development programs. Because BR and BA are closely related but have experienced distinct successes and challenges, USAID contracted a single evaluation team to conduct separate evaluations of the projects, thereby measuring individual performance while also exploring relational factors. This report focuses on the successes, challenges, and recommendations for BA. An 11-person evaluation team, including members based in the United States, Côte d’Ivoire, Niger, and Nigeria, conducted the mid-term evaluation between November 2020 and May 2021. The evaluation team interviewed 152 individuals using a semi-structured interview guide, reviewed program documents, and synthesized the data to answer the four evaluation questions. Working in more than 35 countries, BA has expanded the reach and breadth of SBC work, both in terms of the approaches used and the health issues addressed. BA has helped USAID missions achieve their desired objectives in SBC across a wide range of health areas, including a rapid response to COVID-19, and implementation of integrated, cross-sector activities. Key informants spoke highly of human-centered design and behavioral economics as innovative approaches that lead to stronger program design as well as capacity building of partners, and that are worth the time they require for the results they achieve. The success of capacity strengthening efforts has varied, with key informants seeing the need for a more strategic approach. BA has greatly elevated the understanding and support for SBC programming at the USAID mission, host country government, and regional levels. Looking forward, it will be important for BA to share key lessons, work with BR to synthesize research findings and lessons into practical programmatic guidance, and support USAID in sharing these lessons among missions. MID-TERM PERFORMANCE EVALUATION OF BA: LOCAL LESSONS, GLOBAL LEARNING v ACKNOWLEDGMENTS The evaluation team would like to thank all the people who helped in conducting this evaluation. First of all, we want to thank the 152 individuals who generously gave their time and shared their perspectives on the work of Breakthrough ACTION (BA) and Breakthrough RESEARCH (BR). In particular, we appreciate the willingness of the teams at BR and BA to answer our questions and respond to all of our follow-up data requests. The evaluation team enjoyed these conversations, and we hope that we have effectively synthesized the ideas we heard to tell a clear story with useful recommendations going forward. We would also like to thank the team at Global Health Evaluation and Learning Support Activity (GH EvaLS) who facilitated the process, including Randi Rumbold, who was always responsive and good-natured about our many requests, and Zhuzhi Moore and Andrea Camoens, who provided useful guidance throughout. The Agreement Officer's Representative team at USAID—particularly Angie Brasington—was always available to respond to our questions and provided very valuable feedback on our preliminary findings to help strengthen the report. The staff at the USAID missions in Côte d’Ivoire, Niger, and Nigeria were also essential in helping the team conduct this evaluation and we thank them for their time and support. In this time of the COVID-19 pandemic, we utilized Zoom, Google Meet, Skype, and WhatsApp that allowed the evaluation team to communicate throughout the process. Although we were never able to meet in person, we could still function like a team. We look forward to the time when we can all safely meet in person again. MID-TERM PERFORMANCE EVALUATION OF BA: LOCAL LESSONS, GLOBAL LEARNING vi TABLE OF CONTENTS Acronyms ..................................................................................................................................................................... vii Executive Summary ........................................................................................................................................................... x 1. Evaluation Purpose and Evaluation Questions............................................................................................. 1 1.1 Evaluation Purpose...................................................................................................................................... 1 1.2 Evaluation Questions.................................................................................................................................. 1 1.3 Evaluation Audiences.................................................................................................................................. 1 2. Background........................................................................................................................................................... 2 3. Evaluation Methods and Limitations............................................................................................................... 5 3.1 Evaluation Methodology ............................................................................................................................ 5 3.2 Data Sources................................................................................................................................................ 7 3.3 Data Management and Analysis ............................................................................................................... 6 3.4 Ethical Considerations............................................................................................................................... 7 3.5 Limitations ................................................................................................................................................... 7 4. Findings.................................................................................................................................................................. 7 4.1 EQ 1: To what extent has BA, through its country buy-ins, achieved the Missions’ desired objectives in behavior change and capacity strengthening for SBC?....................................... 8 4.2 EQ 2: How have BA’s country buy-ins reflected Missions’ expected timelines, scale, and quality of design and implementation? ....................................................................................................... 16 4.3 EQ 3: How and to what extent has BA advanced the practice of SBC globally and in the OP countries?............................................................................................................................................. 22 4.4 EQ 4: How and to what extent has each project leveraged its relationship with the other to improve the scale, quality, and impact of SBC at the country, regional, and global levels?........................................................................................................................................ 26 5. Conclusions ................................................................................................................................................ 33 6. Recommendations..................................................................................................................................... 35 Annexes ..................................................................................................................................................................... 36 Annex 1: Scope Of Work ................................................................................................................................... 37 Annex 2: Data Collection Instrument.............................................................................................................. 72 Annex 3: Sources Of Information ..................................................................................................................... 71 Annex 4: Nigeria Country Summary ................................................................................................................ 91 Annex 5: Niger Country Summary ................................................................................................................ 103 Annex 6: Cote d'Ivoire Country Summary .................................................................................................. 110 Annex 7: Disclosure of any Conflicts of Interest........................................................................................ 120 Annex 8: Evaluation Team Members............................................................................................................. 131 MID-TERM PERFORMANCE EVALUATION OF BA: LOCAL LESSONS, GLOBAL LEARNING vii ACRONYMS Acronym Description ACSM Advocacy, Communication and Social Mobilization ANC Antenatal care ARSIP Alliance des Religieux contre le Sida et les autres pandémies ARVs Antiretrovirals BA Breakthrough ACTION BE Behavioral Economics BR Breakthrough RESEARCH BSS Behavioral Sentinel Surveillance CA Cooperative Agreement CAC Community Action Cycle CCP Center for Communication Programs CIP Costed Implementation Plan CLA Collaborating, Learning, and Adapting COR Contracting Officer Representative COVID-19 Coronavirus Disease 2019 CVs Community Volunteers DEXIS DEXIS Consulting Group DFSA Development Food Security Activities DHIS-2 District Health Information Software 2 EQ Evaluation Question FCR Findings-Conclusions-Recommendations FMOH Federal Ministry of Health FP Family Planning FY Fiscal Year GH EvaLS Global Health Evaluation and Learning Support Activity GH Bureau for Global Health GHSA Global Health Security Agenda HC3 Health Communication Capacity Collaborative HCD Human-Centered Design ICRW International Center for Research on Women IEC Information, Education and Communication IHP USAID’s Integrated Health Program IP Implementing Partner MID-TERM PERFORMANCE EVALUATION OF BA: LOCAL LESSONS, GLOBAL LEARNING viii Acronym Description IPC Interpersonal Communication IR Intermediate Result IRB Institutional Review Board KAP Knowledge, Attitudes, and Practices KII Key Informant Interview LLINs Long-Lasting Insecticide-Treated Nets M&E Monitoring and Evaluation MBS Malaria Behavior Survey ME&A formerly Mendez England & Associates MEL Monitoring, Evaluation, and Learning MMH Merci Mon Héros MNCH Maternal Neonatal and Child Health MNCH+N MNCH+Nutrition MOH Ministry of Health NMEP Nigeria Malaria Elimination Program NTBLCP Nigeria TB & Leprosy Control Program of the Ministry of Health and Hygiene) OP Ouagadougou Partnership OPCU Ouagadougou Partnership Coordinating Unit PEC (USAID) Policy, Evaluation and Communication PEPFAR The President’s Emergency Plan for AIDS Relief PMI U.S. President’s Malaria Initiative PMP Performance Management Plan PNLP Programme National de Lutte Contre le Paludisme (National Malaria Program) PNLS Programme National de Lutte contre le SIDA PRH (USAID) Population and Reproductive Health RFA Request for Application RFSA Resilience and Food Security RH Reproductive Health RISE Resilience in the Sahel Enhanced SBC Social and Behavior Change SBCC Social and Behavior Change Communication SIR Sub-Intermediate Result SMC Seasonal Malaria Chemoprevention SOW Scope of Work SP Sulfadoxine-pyrimethamine MID-TERM PERFORMANCE EVALUATION OF BA: LOCAL LESSONS, GLOBAL LEARNING ix Acronym Description SRO (USAID’s) Sahel Regional Office TB Tuberculosis TL Team Lead TWG Technical Working Group USAID United States Agency for International Development USG United States Government WABA West Africa Breakthrough Action WDC Ward Development Committee YTD Year-to-Date MID-TERM PERFORMANCE EVALUATION OF BA: LOCAL LESSONS, GLOBAL LEARNING x EXECUTIVE SUMMARY INTRODUCTION The social and behavior change (SBC) field has evolved to move beyond communication and encompass new approaches like human-centered design (HCD), marketing science, and behavioral economics (BE), while also addressing a wide range of health issues as well as integrated programming. To meet country needs and advance the practice of SBC globally, USAID invested in two separate but linked mechanisms as its flagship SBC programs: Breakthrough ACTION (BA) and Breakthrough RESEARCH (BR). The shared strategic objective of the two Breakthrough projects is increased integration of proven SBC interventions in health and development programs. PROJECT BACKGROUND BA is a five-year United States Agency for International Development (USAID)-funded global project that accelerates the use of SBC through state-of-the-art, evidence-based tools and processes that encourage the adoption of healthy behaviors, while addressing structural barriers and underlying social and gender norms that prevent uptake of services and positive health practices. BA stands firmly on decades of experience and evidence that strategic communication facilitates shifts in behavior, and also embraces newer SBC approaches that are rapidly moving into the mainstream. Under the leadership of Johns Hopkins Center for Communication Programs (CCP), BA is implemented together with principal partner, Save the Children, and core partners ideas42, ThinkPlace, and Camber Collective. EVALUATION PURPOSE AND KEY QUESTIONS The mid-term evaluation of BA and BR cooperative agreements was commissioned by USAID’s Population and Reproductive Health Office, Policy, Evaluation and Communication Division. The purposes of this mid-term evaluation are to: (1) Assess BA’s and BR’s performance thus far in their tenure as benchmarked by the intermediate and sub-intermediate results dictated in their awards (2) Garner evidence for BA’s and BR’s underlying theory of change (3) Capture emerging results to inform decisions about current and future SBC programming. Because BR and BA are closely related but have experienced distinct successes and challenges since their inception, USAID has contracted a single evaluation team to conduct separate evaluations of the projects, thereby measuring individual performance while exploring relational factors as well. The following four evaluation questions (EQs) guided the evaluation of BA. The final EQ on BA/BR collaboration was included in both the BA and BR evaluations: (1) To what extent has BA, through its country buy-ins, achieved the missions’ desired objectives in behavior change and capacity strengthening for SBC? (2) How have BA’s country buy-ins reflected missions’ expected timelines, scale, and quality of design and implementation? (3) How and to what extent has BA advanced the practice of SBC globally and in the Ouagadougou Partnership (OP) countries? (4) (For BA/BR collaboration) How and to what extent has each project leveraged its relationship with the other to improve the scale, quality, and impact of SBC at the country, regional, and global levels? MID-TERM PERFORMANCE EVALUATION OF BA: LOCAL LESSONS, GLOBAL LEARNING xi EVALUATION METHODS AND LIMITATIONS An 11-person evaluation team, including members based in the United States, Côte d’Ivoire, Niger, and Nigeria, conducted the evaluation between November 2020 and May 2021. The evaluation team interviewed 152 individuals using a semi-structured interview guide and reviewed a wide range of program documents. The evaluation team minimized the potential bias inherent in qualitative data by interviewing a large number of key informants, having all team members collaborate on analysis and interpretation of the data and findings, and triangulating with project data and reports. Using this process, the team was able to identify clear and consistent themes. KEY FINDINGS AND CONCLUSIONS EQ 1: To what extent has BA, through its country buy-ins, achieved the Missions’ desired objectives in behavior change and capacity strengthening for SBC? BA is currently working in 35 countries and has helped achieve missions’ desired objectives in SBC and capacity strengthening across a wide range of health areas, including in responding to COVID-19, and in innovative integrated, cross-sector activities. There is widespread interest in integrated SBC, but there are challenges in implementation, in part due to siloed funding. Most capacity strengthening of in-country implementing partners (IPs) and governments occurred through collaborative work, co-creation design sessions, programmatic workshops, and direct mentorship, using a “learning by doing” approach rather than formal trainings. These efforts enabled host government and IPs to learn to “think differently” about SBC. The success of these efforts has varied, with continued capacity gaps among local partners to finance and implement SBC activities without the assistance of BA. Key informants see the need for a more strategic approach and theory of change around capacity strengthening. While COVID-19 diverted attention and resources, it also highlighted the importance of effective risk communication. BA was able to mobilize quickly to meet these urgent needs, demonstrating the importance of having a global mechanism in place, along with established relationships and proven technical and management expertise. EQ 2: How have BA’s country buy-ins reflected mission’s expected timelines, scale, and quality of design and implementation? Key informants spoke highly of HCD and BE as important and innovative approaches that lead to stronger program design as well as capacity building of partners. While acknowledging that these approaches can take time, most saw them as worth it for the results. In terms of new SBC tools, while many BA staff see the SBC Flow Chart as helpful, there are also issues around it being too complicated for common practical use, making it most appropriate for internal use by BA staff and project partners. Key informants were universally impressed with BA’s speed in getting COVID-19 work going in 22 countries. BA made use of innovative monitoring and evaluation (M&E) techniques, generating unique data, when possible, including helping create a COVID-19 dashboard through a productive partnership with Facebook. While large, multi-country programs require strong and comprehensive M&E systems, many key informants felt that the large number of indicators required by the BA project were burdensome. A streamlined set of indicators could facilitate an increased use of data and an emphasis on learning that many recommended. EQ 3: How and to what extent has BA advanced the practice of SBC globally and in the OP countries? BA has greatly elevated the understanding and support for SBC programming at the levels of USAID missions, host country governments, and in regional bodies like the OP. Key informants noted the increased funding for SBC by missions, the increased dialogue about SBC among donors and partners, and how the COVID-19 pandemic has increased recognition of the importance of SBC. BA has helped advance the practice through the targeted sharing of evidence, demonstrating both the cost-effectiveness of SBC, and also the potential for SBC to shift outcomes through approaches that go beyond messaging, including MID-TERM PERFORMANCE EVALUATION OF BA: LOCAL LESSONS, GLOBAL LEARNING xii HCD. Ongoing challenges remain however, including gaps in financing and technical capacity among partners to sustain the design and implementation of SBC activities. EQ 4: (BA/BR collaboration) How and to what extent has each project leveraged its relationship with the other to improve the scale, quality, and impact of SBC at the country, regional, and global levels? Since their respective formal awards, BA and BR have worked together as mandated through the assistance mechanism set up by USAID. This funder-initiated partnership was strategically designed to deliver efficient and successful project results. The intentional efforts in coordination led to multiple successes, working particularly well when roles were clearly defined and each side needed the other—a "codependence" that key informants noted worked well, for example, in the work on Zika prevention behaviors. While BA and BR worked well together overall, challenges in the partnership seemed to arise when expectations and work orders were less defined (including timeline differences), when frequent staffing changes disrupted workflows and team dynamics, and when perceived or actual power imbalances and different corporate cultures hampered staff members’ comfort levels. RECOMMENDATIONS 1. BA’s success across a wide range of health areas provides an opportunity to continue to share SBC lessons broadly within USAID and with other donors and IPs, reaching out to groups working on malaria, TB, and other health areas. 2. BA should collaborate with BR to synthesize findings on integrated SBC and develop clear guidance for donors and implementers, including possible ways to reduce the barriers due to siloed funding streams and systems. 3. BA should use existing platforms and relationships to maximize the impact of BA and BR’s research and lessons by crafting practical, programmatic guidance and conducting joint dissemination activities, for example on provider behavior change and other topics. 4. BA should work closely with USAID to support sharing of lessons among missions, including preparing briefs and other supporting materials. 5. The SBC field should continue to use newer participatory approaches, such as HCD and behavioral economics, with appropriate expectations on timeframes. While HCD can be a slow process, many partners speak highly of it, and both HCD and behavioral economics appear to have a positive impact on capacity building and local ownership. 6. BA should share lessons learned on the life stages segmentation approach used in Nigeria to ensure that integrated programs are client-centered and relevant. 7. Given its complexity and based on key informant feedback, the SBC Flow Chart should be used mostly among BA coalition members for design and strategic planning purposes. As BA continues to streamline the tool, they should also continue to obtain feedback about its suitability for wider use. 8. BA should develop a clear articulation and theory of change of how work with IPs is strategically building SBC capacity in a way that leads to greater sustainability. 9. In addition to its work with local governments, BA should focus capacity strengthening efforts on civil society and private sector groups to ensure that the responsibility and skills to design and implement SBC activities are distributed outside government. MID-TERM PERFORMANCE EVALUATION OF BA: LOCAL LESSONS, GLOBAL LEARNING xiii 10. In support of USAID’s commitment to being a learning organization, BA should continue integrating routine knowledge management and Collaborating, Learning, and Adapting (CLA) practices into the project and encourage more continuous learning and engagement with partners to stimulate innovation, foster better decision-making, and build more systematic exchanges. 11. Future SBC projects should try to improve the balance between the desire for comprehensive M&E systems with a condensed, streamlined indicator list in order to make better use of the data that is collected. 12. BA and USAID should leverage current elevated global interest in SBC, due to COVID-19, to advocate and increase support for SBC globally, including through use of the business cases developed by BR. MID-TERM PERFORMANCE EVALUATION OF BA: LOCAL LESSONS, GLOBAL LEARNING 1 1. EVALUATION PURPOSE AND EVALUATION QUESTIONS 1.1 EVALUATION PURPOSE United States Agency for International Development (USAID)’s Population and Reproductive Health (PRH) Office, Policy, Evaluation and Communication (PEC) Division contracted the Global Health Evaluation and Learning Support Activity (GH EvaLS) to conduct the mid-term evaluation of Breakthrough ACTION (BA) and Breakthrough RESEARCH (BR), two five-year cooperative agreements (CAs). The purposes of this mid-term evaluation are to: 1) Assess BA’s and BR’s performance thus far in their tenure as benchmarked by the intermediate and sub-intermediate results dictated in their awards 2) Garner evidence for BA’s and BR’s underlying theory of change 3) Capture emerging results to inform decisions about current and future social and behavior change (SBC) programming BA and BR are closely related but have experienced distinct successes and challenges since their inception. USAID contracted a single evaluation team to conduct separate evaluations of the two projects, thereby measuring individual performance while also exploring relational factors. The results of this mid-term evaluation will inform the implementation during the remaining time on each of the projects, as well as future funding and design considerations for USAID. This report presents BA mid-term evaluation findings, conclusions, and recommendations. 1.2 EVALUATION QUESTIONS The following four evaluation questions (EQs) guided the evaluation of BA. The final EQ on BA/BR collaboration was included in both the BA and BR evaluations: 1) To what extent has BA, through its country buy-ins, achieved the missions’ desired objectives in behavior change and capacity strengthening for SBC? 2) How have BA’s country buy-ins reflected missions’ expected timelines, scale, and quality of design and implementation? 3) How and to what extent has BA advanced the practice of SBC globally and in the Ouagadougou Partnership (OP) countries? 4) (for BA/BR collaboration) How and to what extent has each project leveraged its relationship with the other to improve the scale, quality, and impact of SBC at the country, regional, and global levels? 1.3 EVALUATION AUDIENCES The primary audiences for this evaluation are the USAID BA and BR management teams, which include USAID SBC advisors across health areas. Secondary audiences include BA and BR implementing partners (IPs). For BA, these include: the Johns Hopkins Center for Communication Programs (CCP) (Prime), and Save the Children, ThinkPlace, Camber Collective, ideas42, Viamo, and International Center for Research on Women (ICRW) (Sub-primes). For BR, the IPs include: the Population Council (Prime), and Avenir Health, ideas42, Institute for Reproductive Health, Population Reference Bureau, and Tulane University (Sub-primes). Other secondary audiences for the evaluation include USAID mission staff implementing BA and BR projects in-country and other SBC-related projects and their management teams. Sensitive components of these evaluations will be delivered in an internal memo to USAID. MID-TERM PERFORMANCE EVALUATION OF BA: LOCAL LESSONS, GLOBAL LEARNING 2 2. BACKGROUND USAID invested in the two five-year CAs—BA and BR—to support countries in achieving desired improvements in health and development outcomes, including increasing the demand for family planning (FP) that is satisfied with modern contraception, ending preventable child and maternal deaths, achieving and maintaining an AIDS-free generation, and achieving a malaria-free world. BA and BR are closely linked and coordinate with one another. The sister projects comprise USAID’s flagship investment in SBC, providing global and country-level technical leadership in SBC advocacy, design, implementation, research, and evaluation. Both projects contribute to the shared purpose of increasing the practice of priority health behaviors and enabling social norms for improved health and development outcomes. Specifically, BA works to increase coverage of, and innovate based on, investments in SBC programming that already have significant evidence, while BR works to disseminate and advance research around SBC technical areas and interventions in which existing evidence is insufficient. The two projects build upon USAID’s previous investments in SBC research and programming, including both global and country-level projects, to simultaneously guide new learning and drive broader application of proven SBC practices and tools. BA aims to fulfill a global leadership function that is desperately needed within SBC, working through a number of new and existing platforms to create opportunities for technical agenda-setting, learning and collaboration, designing and implementing innovative and strategic SBC programs, and promoting agreed-upon priorities through its own programs and knowledge management efforts. Meanwhile, BR strives to convene and engage a broad range of health and development stakeholders, supporting them in developing, promoting, and operationalizing visionary, consensus-driven agendas for SBC research that contribute to measurable global health impact. BR also builds on the existing SBC research agendas to generate new evidence around the priority themes. The shared strategic objective of the two Breakthrough projects is increased integration of proven SBC interventions in health and development programs, particularly health service delivery platforms. While focused primarily on health, the projects also occasionally address SBC needs in other sectors, with particular attention to areas of potential complementarity such as environmental conservation, agriculture, and food security. Within the health sector, the projects maintain a substantive focus on FP and reproductive health (RH), HIV/AIDS, malaria, and maternal, neonatal, and child health (MNCH), with attention to emerging pandemic threats and other infectious diseases. Figure 1 shows BA and BR’s shared intermediate results (IRs) detailing the expected results of their work. BA was designed to directly contribute to IRs 1 and 2, as well as their eight sub-intermediate results (SIRs), while BR was designed to contribute to IR 3 and its associated SIRs. MID-TERM PERFORMANCE EVALUATION OF BA: LOCAL LESSONS, GLOBAL LEARNING 3 Figure 1. BA and BR Shared Results Framework Goal: Improved health and development outcomes Purpose: Increased practice of health behaviors and enabling social norms Strategic objective: Increased integration of proven SBC interventions in health and development programs IR 1: Country-driven, high quality SBC interventions implemented IR 2: Coordinated global and country leadership mobilized to address priority SBC challenges IR 3: Evidence for the impact, feasibility, and cost-benefit of SBC interventions applied SIR 1: Innovative and effective solutions to high-priority SBC challenges designed and implemented SIR 1: Multi-institutional platforms at global and regional levels leveraged to share and coordinate around SBC challenges SIR 1: Research agendas developed and disseminated within health and development communities of practice SIR 2: Improved SBC capacity demonstrated by host-country governments and local health and development program implementers SIR 2: Global, regional, and country SBC programming and investment agendas created to address priority gaps and opportunities SIR 2: Evidence of SBC impact, feasibility, and cost-benefit generated SIR 3: Strategies applied for improved coordination and integration among SBC service delivery and development program implementers SIR 3: Investments in SBC programming leveraged SIR 3: Evidence of SBC impact, feasibility, and cost-benefit packaged and shared with priority audiences including implementers at global, regional, and country levels SIR 4: Rigorous monitoring and quality assurance tools and approaches applied to SBC SIR 4: SBC priorities and agendas diffused through new and existing platforms at the global, regional, and country levels SIR 4: SBC evidence and learning used in targeted USG-supported country programs In addition to addressing a range of health areas in its work, the BA project also has broad geographical coverage (Figure 2), working in Africa, Asia, and Latin America and the Caribbean. MID-TERM PERFORMANCE EVALUATION OF BA: LOCAL LESSONS, GLOBAL LEARNING 4 Figure 2: BA’s Regional and Country Presence MID-TERM PERFORMANCE EVALUATION OF BA: LOCAL LESSONS, GLOBAL LEARNING 5 3. EVALUATION METHODS AND LIMITATIONS 3.1 EVALUATION METHODOLOGY An evaluation team composed of 11 persons conducted the mid-term evaluation of BA and BR between November 2020 and May 2021. As previously noted, USAID decided to have one evaluation team evaluate both BA and BR since the two projects are closely related but have experienced distinct successes and challenges since their inception. This allowed the evaluation team to measure individual project performance as well as explore relational factors. It became clear while conducting the evaluation that the integrated evaluation approach made sense and strengthened the evaluation of both projects. For example, whether a key informant interview (KII) focused on BA or BR, many key informants shed light on the activities of the other sister project. The evaluation team conducted the entire evaluation virtually due to the COVID-19 pandemic restrictions, including numerous virtual team meetings and data collection. The evaluation team included four external consultants based in the United States and Canada, one USAID member based in the United States, and six local consultants based in West Africa: two in Côte d’Ivoire, two in Niger, and two in Nigeria. USAID identified Côte d’Ivoire, Niger, and Nigeria as countries for special consideration in the evaluation as they exemplify different aspects of BA and BR’s portfolios. For example, the Nigeria Mission is the single largest funder of both BA and BR outside USAID’s Bureau for Global Health. The BA and BR activities in that country, which are closely linked, reflect the mandate of the two sister projects working in close collaboration, as envisioned by USAID. As some members of the evaluation team were more comfortable working in French and some in English, the team frequently divided into smaller groups during the virtual meetings. This evolved into grouping two United States/Canada-based members with the Nigeria team, one with the Niger team, and two with the Côte d’Ivoire team, to facilitate communication, standardize data collection, and streamline data analysis. During the data collection, the country teams held virtual meetings to share preliminary findings and discuss any challenges. 3.2 DATA SOURCES 3.2.1 Document Review and Project Overviews The evaluation team reviewed key project documents, including Requests for Application (RFAs), project descriptions, annual reports, and performance reviews. In addition, the BA and BR staff provided virtual project overviews to the evaluation team, allowing for a period for questions at the end. After the global project overviews, the BR staff provided Nigeria-specific and Niger- and Côte d’Ivoire-specific overviews and the BA staff provided overviews of the Niger and Côte d’Ivoire projects in French to the respective country teams. 3.2.2 Key Informant and Group Interviews The evaluation team collected qualitative data through in-depth KIIs. They developed a semi-structured interview guide organized around the EQs to be used during the KIIs (see Annex 2). During the interviews, the evaluators adapted the key informant interview (KII) questions based on key informants’ particular interaction with the projects. The evaluation team held multiple data collection preparation meetings to ensure a uniform approach to data collection. KIIs included both individual and group interviews. The majority of the KIIS were individual. However, in some cases two or three key informants from the same organization opted to be interviewed together as a group. MID-TERM PERFORMANCE EVALUATION OF BA: LOCAL LESSONS, GLOBAL LEARNING 6 In view of COVID-19 safety precautions, the evaluation team conducted most KIIs by phone or virtually via Zoom, Google Meet, Skype, and WhatsApp. The Niger and Côte d’Ivoire teams were able to conduct some interviews in person (see country summaries in Annexes 4, 5, and 6 for details). Interviews were conducted in English and French, depending on the preferred language of the respondent. 3.2.3 Site and Sample Selection The evaluation team worked closely with USAID and the BA and BR staff to determine the main stakeholder groups to sample for the evaluation. The five stakeholder groups were: (1) USAID (Washington, D.C. and missions); (2) BA Prime; (3) BR Prime; (4) IPs, including BA and BR Sub-primes and other IPs who have worked alongside BA and BR, but not as sub-primes; and (5) national governments (see Table 1). KIIs at the global level: USAID developed a preliminary list of key informants for the evaluation team, who then discussed it with the BA and BR staff and modified it as necessary. KIIs in Nigeria, Niger, and Côte d’Ivoire: USAID and the BA and BR staff provided a preliminary list of key informants in each of the three focus countries. The evaluation team worked with USAID and the BA and BR staff to further refine the list. Table 1 shows the breakdown of the 152 key informants interviewed through individual or small group (2-3 respondents) KIIs. It should be noted that some of the key informants interviewed in Niger and Côte d’Ivoire were also relevant to the West Africa Breakthrough Action (WABA) work. A full list of the key informants is included in Annex 3. Table 1: Number of Completed Key Informant Interviews by Country/Region and Stakeholder Group US-based (Washington) Nigeria Niger Côte d’Ivoire WABA Total USAID 14 3 4 4 0 25 BA Prime 12 8 2 5 2 27 BR Prime 7 3 0 0 0 10 IPs 21 11 24 0 4 62 National government n/a 9 10 9 n/a 28 Total 54 34 40 18 6 152 n/a = not applicable 3.3 DATA MANAGEMENT AND ANALYSIS The evaluation employed primarily qualitative methodology, supplemented by quantitative project data, where possible. The evaluation team used a Findings-Conclusions-Recommendations (FCR) matrix to triangulate the findings and derive conclusions and recommendations. Because of the size of the projects and the diversity of the key informants in terms of their interaction with or knowledge about the projects, it was not appropriate to quantify the responses (most denominators would change for every piece of information obtained by the KIIs). Key informants interacted with BA and BR in very different ways, and even those not fully involved with one or the other had interesting things to say about both projects. While the report does not quantify the responses, it should be noted that any quote, statement, or finding in this report is based on information from multiple sources. The evaluation team distinguishes frequency of responses by stating "many" or "most" when it was mentioned by a large number of key informants and "some" or "a few" when it was mentioned less frequently, but mentioned enough to be considered an important theme. MID-TERM PERFORMANCE EVALUATION OF BA: LOCAL LESSONS, GLOBAL LEARNING 7 Each evaluation team member took detailed notes for each interview, either manually, by writing them down, or recording and transcribing the interviews. Evaluation team members then organized the KII notes using the structure of the KII question guide, which was organized by EQ. This facilitated data analysis by making the process straightforward to identify important points and themes for each EQ. The U.S./Canada-based evaluation team members read the transcripts from all the interviews to get multiple perspectives and reduce personal bias in identifying findings. In addition, each country team prepared a summary document of the key findings for their respective country (see Annexes 4, 5, and 6). These findings were incorporated into the overall BA and BR reports, and the summaries are included as annexes to provide a fuller picture of the specifics for each country and provide a document for in-country stakeholders. The five United States/Canada-based evaluation team members prepared an FCR matrix organized by EQ to facilitate thematic analysis and help derive conclusions and recommendations that were agreed upon by the whole team. Each evaluation team member entered information into the FCR matrix. After organizing the findings points under each EQ, the evaluation team arranged them into specific themes and key findings, which were then discussed during virtual team meetings. When possible, the evaluation team triangulated the qualitative findings with the quantitative data from project reports and project indicators. The evaluation team then developed conclusions and recommendations based on the findings and through team discussions. 3.4 ETHICAL CONSIDERATIONS The evaluation team ensured privacy and confidentiality in all data collection. All interviews began with an informed consent process that included the purpose of the evaluation and of the interview, the right to refuse to answer any question, and the right to refuse participation in the evaluation at any time without consequences. Data were analyzed without any identifying information. The citations in the report do not include any names of the person who was quoted. Instead, they refer to "key informants," "respondents" or "participants" and only refer to the stakeholder group they belong to (USAID, BA Prime, BR Prime, IPs, or the national government). 3.5 LIMITATIONS There are limitations to any qualitative evaluation due to potential bias in the collection and interpretation of data. The evaluation team minimized this bias by interviewing a large number of respondents and having all evaluation team members collaborate on analysis and interpretation of the data and findings and triangulating with project data and reports. Using this process, the evaluation team was able to identify clear and consistent themes. As with any endeavor in 2020-21, the evaluation faced limitations due to the COVID-19 pandemic. The evaluation team was not able to meet in person during the evaluation process. Although this has become the norm during the pandemic, it remains a more challenging way to work. However, the evaluation team was able to overcome this challenge by having more meetings of shorter duration rather than long meetings that can become tiring when organized virtually, particularly when translation between French and English is necessary. The evaluation team was fortunate to have multiple bilingual team members who could help translate between English and French, which enabled us to hold full team meetings frequently for the team members to touch base, share thoughts, and plan next steps. It is important to note that as this is a mid-term evaluation, much of the work of BR is still in progress— this is particularly true due to COVID-related delays—with more findings and results still to come, particularly around dissemination and use of results. While this limits the ability to present firm conclusions on certain aspects of the project, it also means that our findings can help inform the remainder of the project. MID-TERM PERFORMANCE EVALUATION OF BA: LOCAL LESSONS, GLOBAL LEARNING 8 4. FINDINGS 4.1 EQ 1: TO WHAT EXTENT HAS BA, THROUGH ITS COUNTRY BUY-INS, ACHIEVED THE MISSIONS’ DESIRED OBJECTIVES IN BEHAVIOR CHANGE AND CAPACITY STRENGTHENING FOR SBC? BA is currently working in 35 countries and has helped achieve missions’ desired objectives in behavior change and capacity strengthening across a wide range of health areas, including in addressing COVID-19, and in innovative integrated, cross-sector activities. BA has included capacity strengthening of local actors and partners to design and implement SBC in activities across the breadth of the project. As one BA Prime key informant explained, the project does not operate in countries without the involvement of the Ministry of Health (MOH) and local civil society partners. BA has faced challenges in achieving mission outcomes, however, particularly in the need for more cross-sectoral funding to meet the demand for integrated activities, the continued capacity gaps among local partners to finance and implement SBC activities without BA assistance, and significant added demands on project staff related to COVID-19. THEME 1: ACHIEVING OBJECTIVES IN BEHAVIOR CHANGE KEY FINDING 1: Important achievements in a range of health areas BA has worked across multiple health areas through numerous mission buy-ins and support to technical offices in the Bureau for Global Health (GH). BA has provided targeted technical assistance in the design, implementation, and monitoring of SBC activities in a number of health areas, as described below. Family Planning There have been numerous BA activities related to FP, particularly in the West Africa buy-in, WABA. BA has provided critical technical support to the Ouagadougou Partnership Coordinating Unit (OPCU). Key informants from AmplifyPF, USAID West Africa’s FP service delivery project, noted that BA has worked with them closely to generate demand for FP alongside their clinic-level activities. BA has also directed two successful media campaigns in West Africa to improve FP uptake, Confiance Totale and Merci Mon Héros (MMH). MMH was highlighted by key informants across USAID, BA Prime, and partners as a true innovation of co-created programming. First conceived during the Francophone Africa SBC Summit in Abidjan, Côte d'Ivoire in 2019, this youth-led campaign helped to break down challenges in communication between parents and adolescents around sexual and reproductive health. As one BA staff explained, “I followed some of the youth involved in this campaign and saw the complications in their lives. In multiple countries, I’ve seen groups of parents and youth, and seen how this campaign helped change their lives. They can talk to their children now.” A USAID/Washington (USAID/W) staff member also stated, “[S]triking improvements have been seen in intergenerational communication. Particularly with girls/adolescents in conservative countries in West Africa where they’ve been able to have real breakthroughs in communication with parents and elders.” MID-TERM PERFORMANCE EVALUATION OF BA: LOCAL LESSONS, GLOBAL LEARNING 9 Malaria Both USAID and BA key informants noted the importance of having a global SBC mechanism to work on malaria-specific activities, as SBC is often the first area of investment dropped by the Global Fund to Fight AIDS, Tuberculosis and Malaria and other multi-national malaria initiatives. This is especially evident in the Malaria Behavior Survey (MBS), an innovative cross-sectional household survey created and implemented by BA that captures the determinants of malaria-related behavior, going beyond knowledge and attitudes alone. Initially piloted in Côte d’Ivoire, plans for country-level MBSs are underway in seven more African countries. There has been broad support for the MBS among country and global actors, although implementation has been delayed due to the COVID-19 pandemic. The survey has become a critical advocacy tool on the importance of identifying behavioral determinants to achieve malaria outcomes. Another achievement mentioned by key informants from USAID and BA Prime was BA’s malaria work in Guyana, targeting a remote population of gold miners with a need to get tested for malaria and adhere to the medication. In addition to providing technical support and materials to volunteer testers, BA launched the “Little Mosquito/Big Problem” campaign that includes a Facebook page featuring the “Jungle Feevah” mini￾series and mining personalities who share their personal experiences with malaria and offer advice. Given the unique problems facing this transient population, including connectivity challenges, BA developed a separate baseline survey for Guyana and worked with the local MOH Rapid Diagnostic Testing Unit to capture behaviors and malaria outcomes through improved monitoring forms and indicators. An endline survey will be conducted in 2022. Tuberculosis (TB) BA was able to make significant strides in SBC work targeted at TB through their Nigeria buy-in. USAID Nigeria Mission key informants noted that previous TB strategies did not draw from evidence on behavioral determinants of health outcomes, and TB projects had little investment in SBC. In Nigeria, a TB prevalence survey showed that TB programs were detecting approximately 16 percent of TB cases and not the 45 percent as expected, which was surprising to partners working to address TB. To identify the gaps, a Knowledge, Attitudes, and Practices (KAP) survey was conducted that showed a low percentage of awareness of signs and symptoms among respondents—around 27 percent. USAID Nigeria Mission key informants described this as a much bigger problem than they thought and recognized the need for more investment in TB-specific SBC interventions. BA then worked with Nigeria’s National TB & Leprosy Control Program (NTBLCP) to improve TB detection, leading multiple stakeholders through a human-centered design (HCD) process from mapping out the client journey to participating in a design workshop. Several key informants reported not only an increase in the level of interest in SBC, but also ownership of the process and the resulting “Brother’s Keeper” campaign promoting shared responsibility. Small medicine stores (patent and proprietary medical vendors), religious leaders, and people outside of the medical community were engaged to improve TB case detection. The process was a true marriage between participatory research/co-design and capacity strengthening. USAID Mission staff in Nigeria noted that their initial buy-in for the TB work was small and “was a hard sell,” but having seen the results they now “wish that our buy-in was earlier.” MNCH and Nutrition (MCHN+N) The Nigeria buy-in to BA has had clear successes in elevating SBC approaches to MNCH, particularly in the Northern region of the country where health outcomes for women and children remain poor. BA focused their approach on community capacity strengthening with ward development committees (WDCs) to identify and support critical behaviors for maternal and child survival. Community volunteers (CVs) supported BA’s MCHN+N activities, working closely with WDCs who helped organize venues for dialogues and compound meetings, identify households, and participate in volunteer review meetings to address challenges encountered in activities implementation. The WDCs also played a key role during the MID-TERM PERFORMANCE EVALUATION OF BA: LOCAL LESSONS, GLOBAL LEARNING 10 recruitment of CVs. In Northern Nigeria, BA engaged religious leaders as well, in order to disseminate critical messages on maternal and child health behaviors through sermons and community meetings. In these interventions, the ownership, buy-in, and leadership at the community level has been significant. BA staff described how deliberate efforts were made to foster leadership among local government area teams and WDCs by encouraging the latter to coordinate all community structures within the health eco￾system and provide support to CVs. Right from the onset, the WDCs were part of the CV training, aimed at fostering linkages and support to the CVs. Through the application of the Community Action Cycle (CAC),1 BA ensured that all local health-related activities were planned for and coordinated by the WDCs. BA directed a similar community-driven activity for improved complementary feeding of infants and children under five in Niger and Burkina Faso, hosting a workshop with local stakeholders to identify innovative solutions to local challenges of inadequate infant feeding behaviors. The importance of this approach was noted by BA Prime staff as key for ensuring the sustainability of SBC interventions to improve MNCH outcomes. Zika Multiple key informants from BA, BR, and USAID mentioned BA’s response to Zika in Latin America and the Caribbean as a significant achievement. BA and BR key informants described the work on Zika as an example of optimum collaboration between the two projects. BR created a critical synthesis of evidence of various prevention behaviors for Zika, then worked with BA to prioritize the most effective behaviors. The two projects produced the “Zika Prevention Behavior Matrix” and prioritized seven key behaviors from the more than 30 that had initially been identified. The projects used this tool with the MOH in the respective countries to advocate for more targeted Zika behavioral interventions. In addition, the two projects developed an interactive online platform outlining evidence-based behaviors with the greatest potential to prevent Zika, and provided the IPs with the knowledge, tools, and resources to promote these behaviors. This evidence-synthesis and advocacy work helped support continued investment in behavior change interventions for Zika at the level of host country governments and the IPs. Integrated SBC There is a high level of interest in integrated SBC programming, particularly in Nigeria. As one BA staff member explained, and other key informants echoed, “most countries have vertical health funding that is siloed, people don’t live siloed lives!” Even in a targeted health intervention, BA activities often touch upon multiple health behaviors, understanding that health-seeking behaviors intersect with one another to achieve outcomes in areas of FP, maternal health, and infectious disease at the same time. For example, one BA key informant explained that “the behavior that we focus on for malaria, like early antenatal care attendance, is not specific to malaria.” BA’s Nigeria buy-in has prioritized integrated SBC activities, most notably the integrated umbrella campaign, Albishirin Ku! (Good tidings! in Hausa), in Northern Nigeria that is promoting 17 behaviors (see Nigeria Country Summary, Annex 4). Key informants, from the Nigerian government to IPs to USAID, all praised the campaign, calling it a “household name” (USAID Nigeria Mission), “very successful and I am proud to be part of it” (Federal MOH), “a masterpiece” (IP), and saying that it has “really touched lives” (BA). Initial data from the project has shown that the campaign recall has been extremely high. Using both a life-stages approach and audience segmentation strategy to target interventions to the right people at the right time, the campaign has disseminated life-saving information through community engagement, including household visits and dialogues, religious and traditional leader advocacy groups, and an entertaining weekly radio drama with mobile phone strategies to increase audience engagement. 1 The CAC is a proven community mobilization approach which fosters individual and collective action to address key health program goals and related outcomes. MID-TERM PERFORMANCE EVALUATION OF BA: LOCAL LESSONS, GLOBAL LEARNING 11 While BA has made strides in improving integrated SBC interventions, challenges of vertical funding streams remain. USAID funding is uneven, and GH funds must report on specific technical outcomes. As one USAID/W respondent explained, “Different levels of funding for different health areas makes output uneven. When you have $16 million for malaria and $2 million for everything else, how can that really be integrated?” Cross-cutting themes Cross-cutting themes like gender and youth are clear opportunities to work across health areas and in sectors beyond health. However, BA staff remarked that it has been a challenge to secure funds for cross￾cutting issues as discreet activities. Some buy-ins, like Guatemala and South Sudan, have included gender more deliberately in their programming, understanding that addressing gender is a way to address multiple health outcomes. Other missions have struggled to think about interventions not explicitly addressing a specific health outcome, although USAID regularly invests PRH resources in gender transformative interventions, recognizing that inequities and restrictive norms are a key barrier to improved health outcomes, including for FP/RH. BA is developing case studies on gender integration across the project, digging into lessons learned and recommendations to apply a gender analysis more rigorously to other health activities. Some initial success stories noted by the project include BA’s gender training in Guinea with the MOH and civil society organizations. In Zambia, BA had a deliberate male engagement strategy leading to a successful male health day campaign. In Botswana, a core-funded BA project used an HCD approach to improve uptake of health services among men. BA staff explained that it was “fascinating in disentangling things that were keeping men from seeking care for HIV.” THEME 2: CAPACITY STRENGTHENING BA has a cross-cutting goal (SIR 1.2) to improve capacity in SBC “among host country governments and local health and development program implementers.” It was evident throughout the BA mid-term evaluation that capacity strengthening activities were well incorporated across BA activities, with both other IPs, such as Food for Peace’s Development Food Security Activities (DFSAs) in Niger and Burkina Faso, and host country governments. Most capacity strengthening of in-country partners and governments has occurred through collaborative work, co-creation design sessions, programmatic workshops, and direct mentorship. These activities utilized a “learning by doing” approach, providing government and IPs with experiential learning opportunities, rather than formal training. A common theme across the KIIs was that BA capacity strengthening activities, including collaborative work and exposure of experts in other fields of public health to SBC, enabled host governments and implementing partners to learn to “think differently” about SBC. KEY FINDING 1: Local ownership BA’s capacity strengthening work aims to foster local ownership of SBC activities among host governments and local implementing partner staff, contributing to long-term programmatic sustainability. In many instances, BA staff worked directly with host government ministry staff in the design of activities and adoption of new SBC techniques. Local staff among BA partners were highly engaged throughout the activity design, implementation, and evaluation process. While many of the capacity building opportunities took place organically, the “learning by doing” approach helped build capacity and increase local buy-in among both host government officials and the IP staff. A key informant from the BA global team highlighted the intentionality of the global team to build local ownership through capacity strengthening activities. The respondent noted the strong collaboration with local staff, and mentioned that while the global BA team analyzed the survey data, they also provided support to enable the local staff to interpret and apply the results in their SBC programs: MID-TERM PERFORMANCE EVALUATION OF BA: LOCAL LESSONS, GLOBAL LEARNING 12 “The [U.S. President’s Malaria Initiative] PMI HQ team has been pushing us to push the envelope more on local ownership and capacity strengthening—we’ve been taking a ‘learning by doing’ approach, and that involves people fully comprehending the purpose of the malaria survey—we involve them in the training of the field workers, supervision of field visits, then we engage them in the results interpretation workshop. We [HQ team] bring the results in raw tables, split them into groups and ask them how to interpret the table, know what each column/row means, then do group work on how to interpret what the findings mean—what does this mean for promoting behaviors in your country —so that’s a three￾four day workshop, and then we work with them to make a presentation for dissemination.” Several key informants from BA and USAID mentioned the importance of collaborative work for capacity building and the usefulness of programming workshops, including co-design of activities, where IPs, both BA Sub-primes and from other USAID activities, would work alongside technical experts from the BA core team. One key informant from WABA said, “Capacity building is the backbone that ties together all of our work. One example is ‘Confiance Totale’ a FP intervention that was developed for WABA, which included co-design, capacity strengthening, and a one-week long process where participants were empowered with the HCD methodology. For us, it was an opportunity to think differently. We saw mindset shifts.” In Nigeria, BA staff noted that when Ideas42, a partner on the BA project, came to the country to apply a behavioral economics (BE) approach and provide behavior change to the malaria work, all IP staff involved were trained on BE, what it entails, and how to apply it within project activities. One BA staff noted that Nigerian stakeholders were “always absolutely integrated throughout the process” in a “learning by doing approach.” Key informants from IPs across multiple countries expressed sincere appreciation for BA’s capacity strengthening activities. Capacity strengthening work was especially critical at the level of government, to increase the interest in SBC programming, and ensure that MOHs understood the value of SBC activities alongside service delivery. A government official in Niger who had worked with the BA core team explained: “Really, to be frank, they have helped us in capacity building. They really helped us to upgrade our communication strategy because they were a bit outdated.” BA staff noted that involving government actors at each stage of the formative research, design, and implementation process enabled them to experience “ah-ha moments,” connecting behaviors with health outcomes, and understanding how to intervene with SBC activities to improve critical health-seeking behaviors. For example, in Côte d’Ivoire, government officials conducted field visits to BA-implemented SBC activities and gained an increased appreciation of SBC’s programming role in improving health, which resulted in increased commitment to SBC within government-led malaria prevention programming. BA has involved government stakeholders in every country they have worked in, and across every technical health area. In Botswana, BA led the HCD approach to designing an HIV activity, and in Nigeria, government staff were involved in the design of TB activities, including interviewing the patent medicine vendors. One BA staff member explained that the government counterparts “are really bright people but because they don’t have the time to get out, these [field] insights were really important because these people are sitting centrally making these big decisions. And sometimes, without taking the time to gather some of these insights, which we help facilitate,” decisions are made that are not rooted in evidence or realities at the community level. BA has also tried to involve host government partners in the implementation of SBC activities to ensure long-term sustainability and uptake of SBC approaches in routine government activities. A key informant from a BA partner noted that when they have done training-of-trainer activities in SBC, they have ensured local trainers are drawn from teams not only at the national and regional health levels, but also within health districts. They explained that then, “the actual implementation is carried out by existing stakeholders and we just make sure that they have the ability to implement the SBC approaches that we are suggesting.” BA works within the decentralized levels of the government to ensure that SBC is integrated across the entire MID-TERM PERFORMANCE EVALUATION OF BA: LOCAL LESSONS, GLOBAL LEARNING 13 health system, that one level of government does not have to wait for another to do SBC work, so that all levels have a sense of ownership in the design and implementation of technical approaches for SBC. KEY FINDING 2: Use of existing tools Several key informants, including project, partner, and USAID respondents, described the benefit of existing SBC tools, such as those available on Springboard and Compass, to support capacity building. The two platforms were developed by the Health Communication Capacity Collaborative (HC3), the previous SBCC project led by CCP, and are now supported by BA. Springboard and Compass are large, online platforms that contain hundreds of free SBC resources, ranging from tools to design formative research, to creating an SBC strategy, to developing a custom monitoring and evaluation (M&E) plan. They also host case studies and lessons learned from diverse SBC programs across Africa, Asia, and Latin America, providing useful insights for programs in similar contexts. A few key informants mentioned that these online tools supported local implementing partners and government staff to learn by reviewing previous examples and applying the tools in their own work. A member of the OPCU noted that Springboard was an “amazing platform” because of its ability to host interactive, multi-directional conversations for shared learning. They contrasted Springboard to the typical webinar format that uses a unidirectional presentation and a short question and answer session. Springboard instead allows for a more interactive conversation and fosters collaborative learning, which they found critical in capacity strengthening work around SBC in Francophone West Africa. Data analytics provided by BA staff show significant increases in use of both Springboard and Compass between 2017 and 2021 (see Table 2), with increases in number of users and page views for Springboard and increases in number of users, materials, and downloads for Compass. The data show that in some categories the numbers for the year-to-date (YTD) for 2020-21 are high indicating that the increasing trend will continue in 2021. Table 2: Use of Springboard and Compass 2017-18 2018-19 2019-20 2020-21 (YTD) Springboard Users Page views 650 139,758 2,215 81,528 3,236 153,303 3,656 52,741 Compass Users Materials Downloads 288,951 1,932 66,765 219,711 2,565 54,499 424,455 3,162 110,383 375,615 3,682 124,961 As shown in Table 2, the number of page views for Springboard is not increasing to the same degree as the number of users. This could be due to several factors. There were some reservations about the online platforms. For example, a key informant from a BA partner noted that using Springboard for capacity strengthening activities with public sector partners was a mixed experience—there was little engagement and the discussion was very brief. The language barrier continues to be a problem for both platforms. While some of the online tools are in French, there are many resources and interactive sessions that are not translated from English. Accessing the platforms themselves requires a basic level of English comprehension, which can be a barrier for local partners in Francophone regions. It is encouraging that the percentage of Francophone Springboard users has increased from 3.9 percent in September 2018 to 11.7 percent in March 2021, but the evaluation team still heard concerns about language issues. Other key informants from BA and IPs felt that the platform’s tools were underutilized or not well-suited for the needs, language preferences, and technical interests of field-based practitioners. For example, tools might be outdated, not adapted to a particular context, or not available in the local languages. One key MID-TERM PERFORMANCE EVALUATION OF BA: LOCAL LESSONS, GLOBAL LEARNING 14 informant noted that “Springboard is relied on heavily for dialogue, but it falls flat [in practice].” While the platforms are seen as opportunities to “socialize” achievements and resources for SBC, and a tool for indirect capacity strengthening, accessibility and engagement on the platform could be improved. KEY FINDING 3: Varied success of capacity strengthening While capacity strengthening activities were well incorporated into the BA activities, the success of capacity strengthening efforts varied. BA experienced similar challenges as other global technical assistance mechanisms in ensuring that short-term workshops and technical assistance actually led to sustainable capacity to design and implement SBC activities. USAID staff in missions and in Washington expressed their concern on whether capacity strengthening activities have actually transferred a sufficient level of knowledge on SBC design and implementation to local partners. For example, in Côte d’Ivoire, a key informant from the USAID Mission highlighted the need for better coordination and organization of capacity strengthening efforts to support partners to ensure that “we do not always need an [international] partner” to lead SBC activities. There needs to be a robust plan at the outset of capacity building activities to improve the “transfer of skills.” Another USAID mission key informant noted that donor agencies need a more intentional, long-term strategy for capacity building to ensure efforts like those from BA fit into a larger theory of change for development programming: “I think part of it is making sure whatever is being proposed from a capacity building framework is understood as part of a larger theory of change—rather than presented as one-off things in one work plan after the next. That theory of change [approach] is typically done more in relation to our behavior change activities as opposed to our capacity building activities.” The success of BA’s capacity strengthening activities was also highly determined by relationships with local actors across different country contexts, including civil society organizations. BA conducted a mapping of local actors to determine existing capacity for SBC across several West African countries, which showed that the SBC experience varied tremendously among the local actors. Differences in country capacity were in part related to previous relationships with USAID-funded SBC programming. In Côte d’Ivoire for example, CCP had worked for many years with local actors under the HC3 project. Capacity strengthening activities under BA built on these existing relationships. This contrasted heavily to a country like Niger. Before BA, Niger had never had a large USAID-funded SBC award active in the country, and therefore, BA had to first work to build relationships and connections in Niger before engaging in capacity strengthening activities. THEME 3: MEETING THE EMERGENCY NEEDS OF COVID-19 The COVID-19 pandemic has presented multiple challenges as well as opportunities for BA. These include decreased access to country programming, with less opportunity for data collection, travel, and interpersonal communication. One BA staff member explained that “COVID is top of mind in both successes and challenges,” while another stated that “COVID has overshadowed the whole project in both successes and challenges at the moment.” While COVID-19 forced BA’s activities to slow down, the project has also met the rapidly developing—and urgent—needs in assisting the pandemic response, including launching new COVID-19-related activities and integrating COVID-19 messaging within other health activities and interventions. As a key informant from BA concluded, “I’m proud of the project for how it’s responded to the massive demand for COVID-19 prevention scopes of work, in a time of personal upheaval for many people—but that has been done really successfully, [with] good insights and lessons learned.” KEY FINDING 1: BA was able to step in quickly to meet urgent needs BA was one of the earliest USAID mechanisms to respond to the COVID-19 pandemic (Figure 3). Given the project’s size as a global mechanism with numerous country buy-ins, and its proven technical capacity MID-TERM PERFORMANCE EVALUATION OF BA: LOCAL LESSONS, GLOBAL LEARNING 15 in risk communication and community engagement, BA was well positioned to respond rapidly to the pandemic in early 2020. CCP has long historical working relationships with community and government actors across USAID focus countries. As a result, BA was able to build on the trust and credibility CCP had generated to rapidly introduce COVID-19-related activities. As one BA staff member explained, “there were disruptions but also opportunities. We were able to respond because of our relationships.” Even in countries where CCP was not present, other BA coalition members like Save the Children were able to respond. As BA Prime staff explained, “Having those investments in community engagement early on and in place meant we could leverage them. Having that footprint meant we could respond in an agile way.” In addition, BA’s coordinating role in the Global Health Security Agenda (GHSA) facilitated a rapid response on COVID-19-related activities. In countries where GHSA was present, those countries pivoted to COVID-19 response work quickly and early. A USAID/Washington key informant explained that because “GHSA activities must be country-led,” BA had already been “building government capacity in risk communication, strategy development, and coordination.” They noted that in relation to COVID-19, BA “did a remarkable job—turned around work plans in one week or less—just working in that many countries (22) accomplishing a significant number of activities in only three months. They came through.” Another significant achievement of the BA project was their work with Facebook to disseminate critical messaging on behaviors to mitigate COVID-19 spread. BA worked with Facebook to package messages that were labelled safe and scientific. Facebook then boosted these messages within the algorithm in several key countries, including Bangladesh and Nigeria. BA has captured user engagement with these messages across countries, demonstrating that by collaborating directly with social media platforms, millions of users can be reached rapidly with critical behavioral communication. Viamo is an organization that uses simple, low-cost technology for data collection and the provision of public service information, and their work, “Mobile Solutions for the COVID-19 Response,” has also been a critical activity in BA’s COVID-19 response. Viamo has worked in several countries, including Nigeria and Niger, to provide targeted messaging, edutainment, and health systems strengthening through remote training and education. Viamo’s work has been crucial in filling information gaps on the rapidly evolving pandemic, targeting messages to both the general population and specifically with healthcare providers. Figure 3. Breakthrough ACTION COVID-19 Response MID-TERM PERFORMANCE EVALUATION OF BA: LOCAL LESSONS, GLOBAL LEARNING 16 KEY FINDING 2: COVID-19 highlighted the importance of SBC The rapid mobilization of resources into BA as part of the COVID-19 response was itself a testament to the increased recognition of SBC’s utility in achieving health outcomes alongside health service delivery, and particularly the importance of the Risk Communication and Community Engagement pillar when responding to health emergencies. One USAID/W respondent said, “So much COVID funding is going to BA and it shows how much people are paying attention to SBC. They wouldn’t put that kind of funding in if they didn’t believe in the practice. It has convinced people that it’s an effective method.” Because preventing and mitigating COVID-19 requires the adoption of key individual and communal behaviors like mask wearing, social distancing, and hand washing, the pandemic response demonstrated both the need and the value of effective SBC strategies and interventions. BA staff noted that “COVID-19 has elevated SBC.” They mentioned that at the government level across the world, policymakers are recognizing the “usefulness of talking to your audience in ways that they understand.” Another USAID respondent mentioned how “COVID has put a spotlight on the importance of really strong well-designed SBC for risk communication. Not much else could have demonstrated how important it is to get your messaging right.” Not only was BA well positioned to respond rapidly to COVID-19 as a global mechanism, but there was a clear recognition of the value of investing in high quality risk communication and community engagement work at scale as part of the pandemic response. KEY FINDING 3: Integrating other health areas into the COVID-19 response While COVID-19 led to delays in BA’s ongoing project activities, the project also rapidly mobilized to integrate COVID-19 messaging and behavior change communication with other health activities. In partnership with AmplifyPF, West Africa, BA developed the Confiance Totale radio spots in French and eight local languages from West African countries. While both AmplifyPF and Confiance Totale’s primary health focus is FP, the radio spots promoted a range of health behaviors related to FP and COVID-19, particularly those that individuals could do at home during lockdown, including improving couples’ communication around RH, highlighting health information hotlines, and wearing masks when venturing outside. In Nigeria, BA repositioned a TB campaign to also address COVID-19: “Since COVID had stigmatized coughing, the campaign aimed at reducing stigma and driving people to get tested.” In Côte d’Ivoire, COVID￾19 preventative health and safety messages were integrated with malaria activities. In other countries, BA has created targeted communication tools like counseling cards on myths and misconceptions for breastfeeding while COVID-positive. There was interest in expanding this work to nutrition and other areas. Activities are ongoing, so data on efficacy of these initiatives are not yet available. 4.2 EQ 2: HOW HAVE BA’S COUNTRY BUY-INS REFLECTED MISSIONS’ EXPECTED TIMELINES, SCALE, AND QUALITY OF DESIGN AND IMPLEMENTATION? THEME 1: DESIGN APPROACHES KEY FINDING 1: Praise for HCD When asked about project achievements, one of the most frequently mentioned was HCD. Although SBCC and behavioral interventions have always been client/beneficiary-focused, HCD brought the focus on the “end user” to a new level. As a BA key informant explained, “HCD was a way to try to swing the pendulum back to make sure people are at the center.” Another BA key informant stated that the HCD approach ensures “that person is equal to the other voices around the table when designing any kind of SBC MID-TERM PERFORMANCE EVALUATION OF BA: LOCAL LESSONS, GLOBAL LEARNING 17 intervention.” They explained that while this has been a feature of the SBC work throughout, it was not done as “explicitly as we’re doing it now.” When HCD was implemented in Botswana, BA staff noted that “nobody had asked people what they thought in that way before.” Other BA staff noted that “HCD and co￾design have enabled us to gain insights that we never had before. It’s opened up a space for us to think differently. The emphasis on HCD has been a welcome opportunity to re-center ourselves.” Many respondents brought up the MMH campaign in West Africa as an example of the HCD process and co-design at its best. BA staff explained that the campaign “brought youth into the conversation: youth consultants developing, creating, monitoring the campaign.” Another key informant from a partner said that in Merci Mon Héros, “youth were actually heard. Their needs and perspectives were really listened to.” Another example of HCD’s popularity was BA’s collaboration with the Nigeria National Malaria Elimination Program (NMEP). Service delivery partners from USAID’s Integrated Health Program (IHP) and PMI reported an appreciation of the BE co-design process to address malaria case management, noting that they were involved in the design process from beginning to end. Another example of the HCD process leading to results and success was Male Engagement Days in Zambia. USAID Zambia Mission’s Gender Integration Specialist noted that “…the popularity of these days was overwhelming—men lined up around the block. At the outset there were good outcomes on the number of HIV and blood pressure tests, but there were other outcomes underlying conceptions of masculinity and what it means to be a man that were really useful.” BA staff acknowledged that one result of participating in the HCD process is a trust in interventions that are co-designed, representing a partnership with community and service providers instead of a preconceived notion of what SBC is. And USAID observed a more systematic use of HCD in the BA project—an awareness of what has failed and how interventions can be tweaked and adjusted. A notable compliment came from a donor who had previously worked with HC3 for PMI, highlighting the shift in BA from earlier SBCC efforts: “Given the design of BA, the project was forced to move into other areas, like HCD and BE. They’ve brought SBC programming up to the state-of-the-art.” KEY FINDING 2: Mixed feelings about the SBC flow chart BA synthesized multiple behavior change models into the SBC Flow Chart (see Figure 4), a tool which is meant to help stakeholders design relevant SBC interventions for local contexts and problems and to unify all new SBC approaches into one discipline, setting the standard for “modern SBC.” Key informants from USAID and BA partners had mixed feelings on the utility of the SBC Flow Chart, however. Among some, there was an appreciation that the SBC Flow Chart was able to loop academic concepts of behavior change into the activity design. One BA staff explained that “the model hasn’t answered all the questions, but it has helped people understand how to best apply the most effective SBC interventions at different points.” Another key informant from a BA partner explained that the SBC Flow Chart allowed stakeholders to “approach a problem [and] design a solution without a foregone conclusion.” MID-TERM PERFORMANCE EVALUATION OF BA: LOCAL LESSONS, GLOBAL LEARNING 18 Figure 4: SBC Flow Chart2 The NTBLCP was one of the first stakeholders to use the SBC Flow Chart to design and conduct activities. BA Nigeria staff described the process as follows: “We worked hand-in-hand with the national and state teams, both government and IPs (like SHOPS+, Stop TB, KNCV). We took a strong capacity strengthening approach. The implementers were the foot soldiers, the head and the hands. We also trained them to do qualitative data collection, analysis and interpretation---how to improve TB detection in Nigeria.” In Guyana, the BA Malaria SBC Advisor reported that they used the SBC Flow Chart from the inception of the project. This, along with use of HCD, led to the generation of a number of approaches, including repackaging malaria medication to facilitate use. Although a catalyst for collaboration and bringing BA coalition members together, as mentioned earlier, the SBC Flow Chart was not universally appreciated. BA staff and coalition partners described it as an “agnostic approach,” a “repackaging of best practices,” and said that “it can be confusing for non-SBC stakeholders.” Both BA and USAID key informants gave mixed feedback on the time it has taken to develop the SBC Flow Chart and its usefulness. One BA coalition member explained that while they appreciated the collaboration, “sometimes it’s overly consensual,” taking “too long to develop.” Others noted that “it’s a bit vague and doesn’t really help to select an approach,” and that the SBC Flow Chart was “so big and intimidating.” USAID expressed concern that the lengthy time taken to develop and achieve consensus on the SBC Flow Chart slowed down project implementation in some countries. The Nigeria Mission remarked that the SBC Flow Chart “needs to be adapted to specific situations with the ability to shorten the timeline (of the process).” 2 BA has continued to streamline the SBC Flow Chart (see webpage: https://breakthroughactionandresearch.org/sbc￾flow-chart/), but the comments in this evaluation report are based on key informants’ experiences with earlier versions. MID-TERM PERFORMANCE EVALUATION OF BA: LOCAL LESSONS, GLOBAL LEARNING 19 KEY FINDING 3: Mixed reactions from the missions about new approaches Along with a more expansive view of behavior change and appreciation of new approaches like HCD, BE, and marketing science,3 select mission staff and partners expressed reservations about BA’s activities. In some instances, BA and USAID key informants mentioned that host country counterparts “still just want posters,” not recognizing the potential in other participatory approaches. There was also concern, for example from some USAID missions, that BA’s approaches were not relevant to specific country needs, that the focus was more on innovative approaches than filling gaps in SBC implementation. Key informants from USAID missions expressed concern that activities in Niger and Burkina Faso were overly theoretical and not grounded in the actual needs of the respective countries. As one USAID mission respondent explained, “BA should make their work much more practical. It is too abstract, theoretical, especially in Niger. Instead of testing something innovative, why don’t we do a clear scoping of what we already know works, doesn’t work, what are the gaps? [BA] should have been working hand in hand with [Food for Peace partners] to deliver different types of messages to actually change behavior. Capacity strengthening has been more like ‘teaching SBC theory and process’ but not so much grounded in the ground realities, barriers, practices.” Several BA and USAID staff noted that some missions still want an information, education and communication (IEC) type of intervention only, rather than interpersonal communication (IPC), HCD, the community action cycle, etc. In the Niger Mission and the Sahel Regional Office (SRO), the USAID staff described struggles with BA's approach with Food for Peace partners. Those implementing partners working in Niger also explained that it is not necessarily theoretical “capacity building” or “knowledge sharing” that they need, but rather resources to do community engagement activities. Although there was a clear appreciation among partners for the community facilitator funding that BA’s presence brought to Niger, some felt that the technical assistance that was provided was not relevant to the existing gaps in implementation capacity. Clearly, the above reactions do not reflect all USAID missions or all country experiences, but there are still struggles, including moving beyond traditional IEC approaches. BA leadership acknowledged the challenges faced in Niger. One key informant mentioned that “the slope was steeper, the challenges were greater, the obstacles were more entrenched in Niger.” While CCP has had a long presence in other BA focus countries, they were building new relationships with partners in Niger. CCP’s in-person country presence was limited, and communication among stakeholders in identifying the type of technical assistance most relevant to the country context and capacity gaps was a challenge. This limited the efficacy of BA’s work in Niger, and the level of buy-in to activities among local partners. THEME 2: TIMELINESS OF DELIVERABLES KEY FINDING 1: Challenges in quick turnaround of activities The implementation of BA’s activities has centered on participatory approaches, capacity strengthening, and evidence-based approaches. BA project staff noted that at times this has made the quick turnaround of deliverables to the missions a challenge. As one BA key informant explained, “[P]articipatory designs can be slower than the previous status quo of doing a [community] survey or focus groups. It is more time-intensive, but the outcomes have been better because [it has been] more transparent and more involved with the stakeholder groups.” 3 Marketing science uses a number of techniques to understand demand and distinguish between different types of potential users of a product or service. MID-TERM PERFORMANCE EVALUATION OF BA: LOCAL LESSONS, GLOBAL LEARNING 20 Given BA’s emphasis on both capacity strengthening and HCD as a behavior change approach, most BA staff felt this was a tradeoff well worth making, as community engagement and co-design were critical for the long-term uptake and success of behavior change activities. At the same time, some key informants from USAID missions noted their frustration with BA’s process being overly theoretical in the technical assistance given. While BA saw their role in strengthening local capacity to help think through key problems in SBC programming, some key informants in the USAID missions felt that this resulted in undue delays to programming and was not relevant to local needs. This was particularly the case in the Resilience in the Sahel Enhanced (RISE) II zones of Burkina Faso and Niger, where USAID mission staff expressed real concern about continued project delays in establishing working relationships with local stakeholders. BA has been tasked with building the technical capacity among Food for Peace IPs to design and implement SBC activities. However, this activity has been met with many obstacles and delays and USAID mission staff expressed concern about BA meeting their objectives in this buy-in. USAID mission key informants said that BA’s approach has been overly theoretical, rooted in teaching SBC process to Food for Peace’s DFSA partners, and that they would prefer a more operational, hands-on approach. BA has attempted to work collaboratively throughout, rooting their technical assistance in the principles of community dialogue and participation. However, getting on the same page with DFSA partners through implementation of this approach has taken a great deal of time. Some DFSA partners have asked for more hands-on support from BA, leading the project to hire more local staff to be embedded with the DFSA partners to provide more direct coaching. There was a protracted learning process for BA and the DFSAs to agree on approaches to both technical assistance and SBC strategies that made the most sense within existing SBC work plans and project implementation under the RISE II project. BA staff also explained that in the few countries where both BA and BR are operational, utilizing research findings and evidence within project implementation presented a challenge in timing. Utilizing research findings in real-time project implementation is a challenge across health and development programming. BA and BR projects were able to overcome this in some instances, most notably in the MMH campaign in Francophone West Africa. The MMH digital campaign incorporated findings from BR’s live social media listening into the project implementation to improve the efficacy of the project. Utilizing research and evidence in other BA activities was not as straightforward, however. A USAID/W key informant explained: “The research has to answer questions that an implementor was asking or a gap that they identified and then the researcher and the implementor have to agree from the beginning that they would take the results and use them. I think we underestimated the difficulty in that. Not just from the standpoint of BR answering things that BA needed to improve their work, but there's also just issues of timing and when results are available for application, and how long some research takes.” It was clear that BA would have liked to utilize research from BR more proactively, but research timelines often took longer to see results, and studies did not match up well with BA’s implementation work plans. BR’s development of innovative, rapid research tools like social media listening addressed this issue and allowed researchers to feed information rapidly on project implementation to BA, resulting in improvements and alterations to the rollout of MMH videos and messages. KEY FINDING 2: Quick adaptation to COVID-19 response While COVID-19 has delayed many BA program activities, overall, there was a recognition that BA has managed to quickly adapt to challenges faced in the pandemic. In Côte d’Ivoire, a key informant from the USAID mission remarked how well BA staff adapted to the COVID-19 context, creating new, relevant SBC communications materials, while also responding to reporting requirements: “For BA in particular, I'll say in the COVID-19 context they've done a tremendous job, really. Meeting expectations from USAID on the reporting side of things, as they pivoted really quickly to take up so much new work… For example, when the COVID-19 pandemic started, there was obviously a shift of MID-TERM PERFORMANCE EVALUATION OF BA: LOCAL LESSONS, GLOBAL LEARNING 21 focus on SBC during the global health crisis, with risk communication needing to be a key component of the response. Breakthrough ACTION has become the primary risk communication partner in the COVID￾19 response. As a result, within the USAID circle, BA has developed a really positive reputation for the work that they're doing during COVID-19. In other health areas, when COVID-19 hit, the nutrition team, the PMI team, the FP team, the MCH team, all came to BA and said, ‘Can you put together a technical brief on how to do SBC for these programs during COVID?’ The ability to rapidly use their expertise to put out relevant materials during COVID-19 that help people recognize the importance of SBC has been a really big benefit of their project.” THEME 3: MONITORING AND LEARNING KEY FINDING 1: Innovative use of data BA has leveraged new digital tools and strategies to collect innovative data for routine project monitoring. All country projects and activities under BA report their data up to a central District Health Information Software 2 (DHIS-2) repository of the project. This can prove challenging given the size of BA and numerous buy-ins, particularly during the last year as COVID-19 activities have required an increased pace of data turnaround. USAID has requested BA to capture data on COVID-19 activities on a monthly basis, much faster than in typical project implementation cycles, forcing innovation for data collection and analysis tools. To assist in the data collection and analysis, BA created a separate “DHIS-2 COVID-19 Dashboard” to aggregate and compare individual country and project data on standard indicators related to the pandemic. These include the number of users hearing COVID-19 messages and number of trainings conducted with healthcare providers and journalists. BA’s direct partnership with Facebook allowed them to not only disseminate messages across hundreds of thousands of users, but also monitor global level engagement in real-time through data analytics. Use of data analytics for rapid data collection and analysis has also been used in other activities beyond COVID-19. MMH’s digital campaign has successfully reached young people in 10 countries across Francophone West Africa through social media, but also through more traditional media on TV and radio broadcasts. BA has tracked engagement with the campaign over social media platforms, in addition to WhatsApp. In Mali alone, the campaign messages have been promoted through more than 21,000 text messages and a WhatsApp group for 250 youth. Ensuring that many of BA’s activities are disseminated digitally has allowed the project to use data analytics and capture innovative data on user engagement and reach of messages in real time. KEY FINDING 2: Challenges in data utilization and learning The global reach of BA and the high number of USAID mission buy-ins has meant that the project has collected a huge amount of monitoring data. Every mission buy-in has their own performance management plan (PMP). BA headquarters asks each activity to feed indicators back upwards to report against global project indicators. All the data are collected within a DHIS-2 database specific to BA. However, reconciling the global indicators with each mission buy-in can be challenging. Some BA headquarters staff mentioned feeling overwhelmed by the number of indicators being tracked in the project. As one key informant explained: “There are a lot of indicators—maybe too many. It can be onerous. I am not even sure if we use the results from all those [collected]. Maybe we are collecting stuff we want to know about, but we have no time to feed that [information] back systematically into the projects.” BA staff stressed that each mission buy-in has their own internal PMP and accompanying M&E plan but tracking these across the buy-ins globally is challenging. Each mission may have their own area of health focus and works with BA to track multiple indicators for those activities. These indicators and their timing for collection may not match with the global project indicators, making cross-project analysis at the global level a challenge. The responsibility to monitor and evaluate projects for adaptive management falls within each mission: the mission activity manager and the BA project staff develop the PMP and monitor activities MID-TERM PERFORMANCE EVALUATION OF BA: LOCAL LESSONS, GLOBAL LEARNING 22 directly, but it can be a struggle to report this information back to the BA headquarters. Given the number of buy-ins and different activities, the number of indicators being collected is also enormous, posing a challenge for analysis and adaptive management on the global scale. One BA key informant said: “I know where that information sits, but the volume of the data sometimes makes it hard to identify the nuggets of [useful] information. The indicators’ pages run 80 or 90 rows down in a spreadsheet. We have massive amounts of data. If there could be a better synthesis of that—maybe have the [monitoring, evaluation, and learning] MEL teams synthesize. That would be helpful.” Given that this is a mid-term evaluation, the evaluation team was unable to fully assess performance in terms of impact, as few projects have endline data at this point. Both USAID and BA staff described a desire for project data to be more efficiently utilized at the global level to document key lessons learned and case studies across projects, to share with the wider SBC community. Further, respondents highlighted the importance of rigorous evaluation of project outcomes to make the business case for SBC in health and development programming. BA staff were cognizant of the need for concrete evidence—beyond anecdotes—that various SBC strategies can work to achieve health outcomes. Several key informants from a range of key informants noted the importance of rigorous evaluation data for donors to ensure “they are getting the most bang for their buck.” Ensuring that monitoring data collected from BA is not only rigorous but can also be used to tell a story of the effectiveness of SBC activities, remains a crucial challenge for BA. One USAID staff member noted, “I want to see the BA project place more emphasis on learning activities and pay more attention to the numbers they’re seeing and what should change accordingly. I want the project to move SBC forward and disseminate information.” 4.3 EQ 3: HOW AND TO WHAT EXTENT HAS BA ADVANCED THE PRACTICE OF SBC GLOBALLY AND IN THE OP COUNTRIES? BA has greatly elevated the understanding and support for SBC programming at the levels of USAID missions, host country governments, and in regional bodies like the OP. BA has done this through the targeted sharing of evidence, demonstrating both the cost-effectiveness of SBC and SBC’s potential to shift outcomes through tools that go beyond messaging, including HCD. Ongoing challenges remain however, including gaps in financing and technical capacity among partners to sustain the design and implementation of SBC activities. THEME 1: EVIDENCE OF INCREASED APPRECIATION OF AND SUPPORT FOR SBC GLOBALLY AND IN COUNTRIES Increased support for SBC programming is notable in several West African countries where BA has been active. In Nigeria, a range of key informants spoke of their increased appreciation for SBC. One government key informant stated that “BA has converted us and now I am an ambassador for SBC.” BA noted that even state governments in Nigeria are now investing their own funds in SBC activities. The USAID Ghana Mission described a similar movement of local district assemblies releasing funds in their SBC activity budgets for malaria prevention. USAID acknowledged that this would not have been possible without BA’s support and engagement in the country. Communicate for Health, an SBCC project in Ghana led by FHI360 from 2014-2019, had begun work in Ghana with district assemblies, and BA helped the Ghana Mission follow through on previous activities and their initial capacity investments. BA was brought into Ghana while the project waited to launch the national SBC award, demonstrating the necessity of having a global mechanism that missions can rely on for high quality SBC interventions. BA supported community action plans at the district level in Ghana, helping stakeholders identify key barriers to health behaviors, and creating action plans to address behaviors through prevention activities. As a key informant from the USAID Ghana Mission explained, MID-TERM PERFORMANCE EVALUATION OF BA: LOCAL LESSONS, GLOBAL LEARNING 23 “[Before BA’s work in this area] district assemblies hadn’t even been aware of this [prevention] funding [for malaria], but they were then able to release those funds and use them in community prevention activities. I saw the kind of engagement and willingness from district assemblies themselves on how we can work on regularization of the funds.” USAID highlighted the sustainability of this approach to SBC because of the community engagement and ownership: “even if [the funds released] are only for malaria, we see them as a pathfinder for community engagement and other [health] outcomes.” At the USAID mission level, key informants noted that many country operational plans have shown an increase in funding for SBC activities. Increased budgets for SBC have also been seen in areas of nutrition, HIV, MCH, and cross-Global Health funding. There has been a clear demonstration in BA on how SBC investments “generate results,” and therefore merit more investment. The significant level of funding for COVID-19 activities under BA, both new buy-ins and adaptations of existing activities to disseminate COVID-19 messaging and information, also demonstrated the increased support of SBC. Stakeholders understood how critical SBC work is for the prevention and slowing of infectious disease spread and were ready to use BA as a global mechanism available to provide nimble and high quality programming globally in a time of crisis. One USAID staff member explained that “USAID wouldn’t put that kind of funding in if they didn’t believe in the practice of [SBC], and it has convinced people it’s an effective method.” KEY FINDING 1: Critical focus in Francophone West African countries BA’s targeted focus on Francophone West African countries has been especially critical for increasing the uptake of SBC activities in a region where health outcomes have lagged behind other African countries over the last decade. USAID staff explained that “…the regional and country Missions feel that BA, in particular, has really been able to leverage expertise that they have and truly tailored things for the countries rather than just bring things in ad hoc. A real attention to tailoring activities for Francophone countries and the Francophone expertise within BA has been a crucial lever.” BA was the main organizer of the highly successful Francophone Africa SBC Summit in Abidjan in 2019, a catalyzing event that brought together targeted specialists on SBC from across the West African region. They have also collaborated with the Francophone Community of Practice on social norms and been active on Springboard to facilitate conversations in French among partners. A key informant from BA noted that the “clearest example” of SBC advancement in the region is with the OPCU, which now has a dedicated “strategy for SBC,” whereas before, SBC was folded under a broad category of “demand activities.” Capitalizing on the 2019 Francophone Africa SBC Summit, BA has strengthened its relationship with the OPCU, “taking them from a place of skepticism in SBC to a real believer.” SBC was a focus of the OPCU’s annual meeting in 2019 after the Francophone Africa SBC Summit. The OPCU members have become crucial advocates for countries in West Africa to advance SBC within national Costed Implementation Plans (CIPs). A member of the OPCU mentioned their appreciation for BA’s active commitment to the Francophone West Africa region, and their strategic work with the OPCU and local stakeholders. BA made critical connections with local governing members and civil society organizations in OP countries, ensuring networks were engaged and collaborative in strategizing around SBC. The OPCU has fed directly into the planning of in-person and digital workshops on SBC with BA. OPCU staff noted with appreciation the collaborative and inclusive nature of BA’s work on SBC in the region. KEY FINDING 2: Strategic advocacy has led to increased interest in SBC BA has conducted careful, targeted, and evidence-based advocacy work at the USAID mission-, regional-, and national government-levels, creating increased interest and acknowledgment of the importance of SBC in health and development programming. BA was able to use key evidence generated both internally and from BR to “make the case” for SBC as a cost-effective and essential aspect of development programming, though actual increases in funding for SBC activities have not been seen across all MID-TERM PERFORMANCE EVALUATION OF BA: LOCAL LESSONS, GLOBAL LEARNING 24 stakeholders, particularly national governments. Several USAID/W key informants noted seeing increases in SBC funding at the USAID mission-level however, explaining that BA was able to demonstrate to stakeholders that SBC goes beyond communication activities. BA’s SBC activities and advocacy work have shown how health and development outcomes can be achieved through investments in HCD, IPC, community engagement, integrated SBC programming, and provider behavior change. Business cases for investing in SBC BR has conducted costing analysis and generated two business cases for investing in SBC, one for FP and another for malaria. BA staff noted how useful these tools have been in their advocacy work with national governments, regional bodies, and global secretariats. As one BA key informant explained, “we have so treasured that business case [for FP], it has helped us enormously in making the business case for SBC, which is what we’ve seen as BA’s mandate—but having that data and tool was enormously helpful.” Another BA staff also mentioned that this has been the “strongest and most successful area of collaboration” with BR: “[BR] did the data analysis, we’ve used it, shared it, and disseminated it.” Malaria technical advisors also noted that the malaria business case will be extremely helpful in their work to try and leverage more funding for SBC. BA plans to use the business case in a presentation to Roll-Back Malaria Partnership’s SBC Working Group. As a BA staff member explained, “from where I sit, global advocacy and tools for SBC, they’ve been really helpful.” Demonstrating that SBC goes beyond SBCC A key informant from a BA partner commented that this is the first global USAID-funded SBC project whose name does not include “communication” in its title. USAID deliberately mandated the award with this title to demonstrate how SBC can move beyond communication, to use and scale-up other tools for SBC, leading BA to focus on increasing partner capacity for HCD, BE, and other SBC approaches that go beyond messaging. A BA staff member explained that in the area of FP, their activities have gone “beyond demand creation,” to areas like “provider behavior, maintenance [of behaviors], how do we slow [FP] method discontinuation.” BA’s activities have generated evidence on what SBC can achieve beyond initial demand creation. Multiple key informants from BA, USAID, and IPs noted how BA’s work in Francophone West Africa has been especially successful in demonstrating what is possible with increased SBC investments alongside health service delivery. USAID’s West Africa Regional Office’s buy-in to BA, WABA, has created multiple digital campaigns to improve IPC and community dialogue around sexual and reproductive health behaviors. WABA has also held community workshops and site walk-throughs alongside AmplifyFP, the West Africa Regional Office’s FP service delivery mechanism. A key informant from a WABA partner explained that they have seen a “big change over the last years” because of this targeted work with local partners in the Francophone region, demonstrating the potential of SBC beyond messaging and demand creation. BA’s “Taxonomy for Social Norms that Influence Family Planning in OP Countries” is another critical piece of evidence to demonstrate how targeted SBC interventions can influence key health behaviors. A key informant from a WABA partner explained that this work was “especially important in Francophone West Africa, which are very strong, patriarchal societies, so the social norms influence decisions for FP a lot. The social norms mapping helped us clarify where we should invest, where we should work, and what we should do specifically to improve FP uptake. This has helped us to better define decision-making processes, including in CIPs, the principal strategic documents to help OP countries make FP decisions.” BA has used the taxonomy in advocacy work across the region, but it is not yet clear if this tool has increased the level of investment in norms-shifting programming among USAID missions or other stakeholders in the region. Moving forward, it will be important for BA to document examples when the MID-TERM PERFORMANCE EVALUATION OF BA: LOCAL LESSONS, GLOBAL LEARNING 25 tool has helped stakeholders measure norms at the outset of SBC activities and/or develop and implement norms-shifting interventions in their portfolio of health and development programming. THEME 2: ONGOING BARRIERS A range of key informants noted that barriers to increased SBC investments, particularly at the country level, are ongoing. These barriers are related to both financial constraints and continued capacity gaps in understanding how to design and implement SBC activities, particularly those that go beyond communication strategies. Financing continues to be prioritized for the supply of health services; however, gaps in SBC technical capacities still remain among service delivery partners, limiting the sustainability and scaling-up of SBC investments at the country level. KEY FINDING 1: Financial constraints in investing in SBC over the long-term There is a recognition that USAID-focus countries struggle to adequately fund their national health systems, including SBC activities. Even if there is more interest in SBC among stakeholders, a key informant from a BA partner noted that “ministries of health are still waiting for the donor to fund SBC activities.” There is a recognition that sustaining community-based activities, like norms-shifting interventions, can be expensive. One USAID/W staff member noted that this expense is still a barrier, particularly in Francophone West African countries: “We still, I don’t think, have the resources or willingness to put the resources in really to do [SBC] in a comprehensive way. When working with the government, they want to see things that are much more tangible.” Key informants noted that in resource-scarce environments, demand-side activities are often the first to be defunded. A USAID/W respondent mentioned that the Global Fund, for example, could “be more intentional in SBC. If something has to be cut, it will be there [with SBC].” BA has been trying to counter this with the Global Fund, specifically by highlighting results from the MBS, trying to ensure SBC investments continue to be funded and prioritized alongside service delivery. There is also concern about the longevity of SBC investments. MMH has been an enormously successful intervention for youth and FP/RH outcomes in Francophone West Africa, but IPs questioned how long the activity will be funded and supported, and “what will come next.” There was concern that the donor￾driven investment cycle leads to short-term activities, and does not allow activities to reach their potential scale and efficacy. One BA staff member noted that budget timelines remain a challenge; it is “intractable in many cases.” While investments can move the needle on certain behaviors, longer timelines with longer term financing are critical to see widespread social and behavioral change. These budgets have to come in part from host country governments. KEY FINDING 2: Limitations in technical capacity to do SBC work without BA remain, especially with service delivery partners In addition to financial constraints, limitations in technical capacity to design and conduct SBC activities among health and development partners continue to limit the longer-term sustainability of SBC implementation in USAID-focus countries. While BA has worked to strengthen the capacity of local partners to design and implement SBC activities themselves, multiple stakeholders remarked that service delivery partners continue to focus on simple communication measures like posters, rather than a fully￾developed demand-side strategy. One BA partner said that “the challenge is to make sure SBC in CIPs goes beyond posters, brochures, and a radio program, challenging them to think on the whole process: formative research, audience segmentation, discreet demand creation activities.” BA staff noted that the West Africa buy-in was expanded so that BA themselves would do implementation, when the initial intention was that BA would only provide technical assistance with local stakeholders. However, the project found that the technical capacity to design and implement robust SBC activities in countries was not sufficient among local actors. BA key informants explained that they always work very closely with the MOHs and local organizations in host countries, but there is still a need to transfer the MID-TERM PERFORMANCE EVALUATION OF BA: LOCAL LESSONS, GLOBAL LEARNING 26 skill set of both conducting formative research and using research findings to design and sustain effective interventions. This transfer of skills requires a long-term project and continued relationship building, a challenge with short donor funding cycles. This was evident in BA’s work in the Democratic Republic of the Congo (DRC), where BA staff explained the struggles to hand off activities to local implementing groups: “USAID timelines are so short, we can’t pilot things for six months, so we apply activities straight away. We do close monitoring of the first implementation before we scale up, and work hard to identify good prototypes for rapid implementation. BA tests, makes sure it’s good, and then hands over to service delivery to scale-up. BA’s involvement beyond that part will vary country by country, partner to partner. BA has been implementing a set of interventions through this process in DRC—service delivery partner said we’re not ready yet, we need more tools and guidance, but does BA have the resources to do it? So in DRC, they took one geographic area and have hand held the implementation for others to replicate in other provinces—BA is in three provinces full-time, in three doing training and introduction, in the last three provinces, the service delivery partner has to do it fully. But interventions are designed simply enough that the service delivery partner can run with them—they develop a monitoring plan with the service delivery partner beforehand.” 4.4 EQ 4: HOW AND TO WHAT EXTENT HAS EACH PROJECT LEVERAGED ITS RELATIONSHIP WITH THE OTHER TO IMPROVE THE SCALE, QUALITY, AND IMPACT OF SBC AT THE COUNTRY, REGIONAL, AND GLOBAL LEVELS? “Inter-organizational collaboration is no longer entirely a free choice, but is close to a necessity imposed by economic, technical, and knowledge-related concerns.” (Jastroch et al. 2011)4 THEME 1: WHAT HAS WORKED WELL? Since their respective awards, BA and BR have worked together, as suggested in their respective CAs set up by USAID. This funder-initiated partnership (referred to frequently during the KIIs as an “arranged marriage”) was strategically designed to deliver efficient research and evaluation outcomes. In favorable terms, the “sister projects”5 were set up to provide complementary support and are expected to “collaborate closely across the life of both projects” (BR RFA, 2017). According to BA’s RFA, BR works to advance and disseminate research around SBC technical areas and interventions in which existing evidence is considered insufficient, while BA works to increase coverage of, and innovate based on investments in SBC programming that already have significant evidence. Common Structures Structurally, BA and BR have a common strategic objective and share IRs (see Figure 1 in Section 2, Background). Overall, to achieve these results, BA carries out rapid, programmatically useful monitoring, concept- and pre-testing, and formative research, while BR takes on primarily operations research and evaluation. Though to some key informants in BR and IPs, BR is primarily viewed as evaluating BA’s work, that does not give either of the projects the appropriate recognition for their independent and mutually supportive technical capabilities, nor for their broader program mandates. 4 Jastroch N, Ku C, Marlowe T, Kirova V, Mohtashami M, Nousala S. (2011) Inter-organizational Collaboration: Product, Knowledge and Risk. JOUR Vol 9. https://www.researchgate.net/publication/228216091_Inter￾Organizational_Collaboration_Product_Knowledge_and_Risk 5 Both BA's and BR’s RFAs assign this term to the projects. MID-TERM PERFORMANCE EVALUATION OF BA: LOCAL LESSONS, GLOBAL LEARNING 27 The partners have a joint logo and a joint webpage at https://breakthroughactionandresearch.org, with fine print indicating that “The contents of this website are the sole responsibility of Breakthrough ACTION and Breakthrough RESEARCH.” Though the website indicates shared ownership, BA is responsible for management of the platform. However, BR did contribute LOE of several staff to support BA as they reorganized and updated the website. The significant difference in project funding likely accounts for some of the uneven distribution of labor for this key communication platform. The two projects also have shared social media handles. While much of the content is developed and delivered by the BA communication staff, BR staff, through its partner PRB, develops and posts all BR content on the shared social media handles. BA and BR have attended key conferences together, setting up a single “Breakthrough Action & Research” booth, such that their outward-facing image is as a seamless partnership. On all accounts, that type of integrated presentation and effort has worked very nicely, and staff have enjoyed it. As described by a BA key informant, “presenting that kind of united front has worked well. It shows we’re two arms of one body. Presenting our work as separate but combined.” According to BA’s SIR 2.4.2, in their last semi-annual count, there was a cumulative total of 119 events for outside audiences with BA and BR coordinated participation since the projects launched, including jointly-organized workshops or presentations. That translates to substantial shared time and coordination between staff from the two projects, which can help foster and sustain an important culture of collaboration between colleagues. BA’s Prime, CCP, was also the Prime for USAID’s predecessor flagship SBC project, Health Communication Capacity Collaborative (HC3). HC3 developed two important KM resources, which USAID continues to fund under BA: the Springboard platform—an online networking and sharing resource that supports and nurtures global-, regional-, and country-level SBCC COPs and the Health COMpass platform—a curated, interactive online materials repository. In 2018, BA revamped the Springboard platform and included COMpass in its banner with the tagline, “Learn from the Experts. Find the Right Tools.” Springboard can be found at www.springboardforsbc.org. COMpass can be found at www.thecompassforsbc.org. Though the websites attribute BA as their developer, both BA and BR projects use the platforms as key KM tools and provide jointly-developed as well as project-specific content. Such a robust undertaking speaks to a productive level of coordination and collaboration. Joint activities Other positive examples of the two projects’ collaborative partnership are the numerous successful activities that have resulted from their combined work. According to key informants, some notable BA/BR collaborations included activities around Zika, malaria, social media monitoring and listening, and research on cost effectiveness of SBC activities. For Zika, BA and BR compiled lessons learned, best practices, and resources from their shared work on SBC programming for USAID’s response to the disease. An overview of their collaborative efforts can be found at https://breakthroughactionandresearch.org/breakthrough-action-research-resources-from-the￾usaid-zika-response/ and as a “Trending Topic” on the COMpass site, with details of their success stories, at: https://www.thecompassforsbc.org/trending-topics/promoting-social-behavior-change-sbc-during￾usaid-zika-response. The cornerstone work of the Zika response was identifying the most important evidence-based prevention behaviors and harmonizing messages on disease prevention. The teams MID-TERM PERFORMANCE EVALUATION OF BA: LOCAL LESSONS, GLOBAL LEARNING 28 identified seven out of 30 behaviors that had the greatest potential to prevent Zika, and then provided IPs with the necessary knowledge, tools, and resources to promote them.6 In Nigeria, BR is evaluating the effectiveness of BA’s malaria program (integrated versus malaria-only SBC programming) on priority outcomes for malaria, FP, and MNCH+N behaviors and ideations among pregnant women and women with a child under two years old. BR carried out Behavioral Sentinel Surveillance (BSS) Surveys, in three of the 11 states where BA has a presence. Their results informed BA’s successful SBC programming and scale-up. The BA and BR teams worked closely together on the study design and the questionnaires. BR was able to leverage BA’s state presence to help disseminate results to state coordinators, state teams, and local government officials. In the end, they also developed and presented a webinar together about this successful joint effort. In Francophone West Africa, the BA and BR projects worked together on the hugely successful “Merci Mon Héros” SBC campaign. BR’s groundbreaking "social listening" data mining research provided critical results that informed BA’s adaptive management for SBC activities, allowing BA to implement real-time feedback for the FP/RH social media campaign in Burkina Faso, Côte d’Ivoire, Niger, and Togo. In addition to these examples, key informants cited other very good works that BA and BR collaborated on. Several US-based and country-based key informants from USAID, BA, BR, and IPs mentioned in particular that BA and BR, together, had managed COVID-19 adaptations expertly. They cited multiple examples of how BA and BR worked well together to modify activities during the pandemic, including how BR evidence had helped BA redesign the contents of some of its radio health drama family planning programming to factor in COVID-related issues. In some cases, at the country-level, key informants from USAID Missions reported viewing BA and BR as a single project, which is frequently considered an indicator of successful partnership in organizational science. A counterpoint to that, however, is that the BA Prime’s predecessor SBC project (HC3) and the Prime (CCP) overall, have such a significant field presence that mission and field staff may default to recognizing BA as the sole implementer for what is BA/BR shared work. In Nigeria, there was a three-month overlap between HC3 and BA projects. This enabled some staff to simply move over from HC3 to BA with all the expected benefits of such a transition, including name recognition, process continuity, and familiarity with organizational culture. A key informant from an IP in Nigeria explained that one of BR’s challenges was in community engagement because, “it’s mostly BA on the ground. BA is like a household name” in the states where they are working. Positive codependency What seems to work well in terms of collaboration between BA and BR is when they have a “codependent relationship” (as described by some key informants), where each must count on the other to get their assigned work done, and where the value of each group’s respective contribution can be easily understood and acknowledged (as happened with the Zika work). When terms of work and engagement are clear and where the results of one team’s assignment is needed for the other to carry out their responsibilities, 6 For more information on Zika activities, see page 10 “Collaboration works best when objectives are clear from the outset, and each partner has a clear scope of work and responsibilities.” (BR Prime Key informant) MID-TERM PERFORMANCE EVALUATION OF BA: LOCAL LESSONS, GLOBAL LEARNING 29 then the mutually reinforcing nature of the partnership operates well. It takes on a transactional quality that is easy to see and comprehend, and that which has consequential value for both partners. In an effort to codify the importance of the partners’ collaboration, USAID provided a results framework with specific sub-indicators that acknowledge the valuable outputs and outcomes of BA and BR’s tandem and complementary tasks. For example, BA’s SIR 1.1, Innovative and effective solutions to high-priority SBC challenges designed and implemented, and SIR 1.4, Rigorous monitoring and quality assurance tools and approaches applied to SBC interventions, include indicators that are quantitative measures of the projects’ collaboration (see Table 2 below). Table 2: Indicators of Collaboration Between BA and BR Intermediate Result 1: Country-driven, high-quality SBC interventions implemented Sub-IR 1.1: Innovative and effective solutions to high-priority SBC challenges designed and implemented ID Indicator Type Data Source Frequency Calculation Project Total To Date 1.1.1 # of tools jointly developed by BA and BR pertaining to innovative topics, approaches, or dissemination formats Output Program documentation or tacit knowledge of program staff Semiannual Simple count of research- or program-related tools pertaining to innovative topics, approaches, or dissemination formats that BA has jointly developed with BR 30 1.1.7 # of research￾related materials developed jointly by BA and BR in the last 6 months Output Program documentation or tacit knowledge of program staff Semiannual Simple count from program documentation or the tacit knowledge of program staff or research-related materials that are jointly developed. These materials may include data collection instruments, data collection plans, infographics, research briefs, technical reports, blogs, or peer-reviewed publications 42 Sub-IR 1.4: Rigorous monitoring and quality assurance tools and approaches applied to SBC interventions 1.4.1 # of countries from which BA shared data with BR in last 6 months Output Program documentation or tacit knowledge of program staff Semiannual Simple count of the # of countries from which data was shared. By “countries” this indicator refers to where BA collected data and not where data sharing occurred. These could either be qualitative or quantitative data sets 10 MID-TERM PERFORMANCE EVALUATION OF BA: LOCAL LESSONS, GLOBAL LEARNING 30 ID Indicator Type Data Source Frequency Calculation Project Total To Date 1.4.5 # of BA interventions using BR data/findings to develop or modify their design implementation, or evaluation in the last 6 months Outcome Program documentation or tacit knowledge of program staff Semiannual Simple count of interventions that use BR data/findings. May include quantitative or qualitative data sets, findings from primary or secondary analyses (e.g., desk reviews, business cases). Use refers to whether data/findings informed design, implementation or evaluation of SBC interventions 30 THEME 2: MAIN CHALLENGES While BA and BR worked well together overall, challenges in the partnership seemed to arise when expectations and work orders were less defined (including timeline differences), when frequent staffing changes disrupted workflows and team dynamics, and when perceived or actual power imbalances and different corporate cultures hampered staff members’ comfort levels. Unclear assignments and short timelines When things are less defined in partnerships, there is more room for confusion, which can lead to missed opportunities and failed expectations. In the BA/BR partnership, for example, key project staff did not know when to use the joint BA/BR logo—the very emblem of their formal collaboration. This created confusion and tension about how knowledge products should be branded. The solution was to widely distribute explicit marking guidelines, which included when and where to use the shared logo. Marking and branding is generally covered at the beginning of projects but the ramp up for both partners was seemingly rushed. Partners in both BA and BR mentioned they were “building the plane as [they] were flying it,” feeling as though there had not been enough time at the start of the projects to adequately cover overarching general issues, like marking and branding. They also said there was little time for BA and BR staff to get to know one another or to fully understand and appreciate how the different teams worked, and how their institutional strengths could complement each other rather than pose an omnipresent threat to future funding. The following simple comment summed up what many key informants also expressed, “It’s so hard when the roles aren’t clear.” Key staff turnover and different corporate cultures Though staff turnover is always to be expected, high staff turnover can pose significant disruptions and may indicate a more serious management problem. The Knowledge Management & Research Application Team Lead, a key position on BR, changed multiple times. During this evaluation, several key informants from BA, BR, and USAID/W mentioned this specific staffing issue. A BA Prime key informant explained that colleagues had to “start all over” every time a new person was hired. Key informants from BA and BR Primes often mentioned that BA and BR had different corporate cultures that influenced the way they collaborated, particularly in their management and communication styles. “Organizations that depend heavily on similar funding sources or on the same donor may find it difficult to work together because of the propensity to compete with each other for scarce resources and also because of an underlying awareness that their association may not yield a net increase in income.” —Designing and Managing Partnerships, FHI360 MID-TERM PERFORMANCE EVALUATION OF BA: LOCAL LESSONS, GLOBAL LEARNING 31 Some key informants from USAID/W and BA felt that BR’s limited experience in SBC as compared with CCP also created challenges. Their perceived power imbalance was also problematic at times. The imbalance reportedly derives from the considerable difference in the projects’ funding, the historical depth of BA’s presence and overwhelming reputation in the field, and BR’s assumed role as BA’s evaluator. Some level of power imbalance is inevitable between groups and can generally be managed by strategic project planning and established goodwill between professional colleagues. A difference in corporate cultures between working partners may be difficult to negotiate but is not necessarily negative. Most importantly, projects need a sufficient amount of time and dedicated resources to explore areas of agreement and potential disagreement that might arise during the partnership. It should not be left up to individual staff to negotiate corporate relations alone, but rather, the leadership and governing members of the partnership should consider these variables during the start-up phase of the collaboration and throughout the groups’ periods of performance. As expressed by one BR key informant, “It was up to us to build the relationship and the corporate culture made it hard. Intent and desire around the collaboration and a framework or guidance in how to work together from USAID would have been good.” THEME 3: RECOMMENDATIONS FOR IMPROVING BA/BR COLLABORATION It is evident that USAID expects BA and BR to align and coordinate their efforts and designed the projects to be mutually reinforcing. In organizational terms, BA and BR are expected to carry out program leadership and management roles for their respective and joint work, while USAID plays a governing role for the collaborative partnership.7 The time and resources that the Bureau for Global Health has invested in SBC for well over three decades, as well as USAID’s more recent focus on building effective partnerships, are the backdrop for this distinct opportunity to inform future SBC programming partnerships. The excerpt below is from the BR’s CA: “BA and BR will work together to achieve their shared purpose and strategic objective. The USAID Washington management team will closely facilitate and oversee this integral partnership. The sister projects may be funded to jointly design and/or implement certain activities and are expected to establish a strong, collaborative model of engagement, including regular meetings to advance their shared work… To achieve maximum coordination and synergies across the two projects, it is suggested that each project dedicate funded staff time to coordination activities.” The above excerpt suggests that BA and BR coordinate and collaborate but leaves the decision on how to do so up to the projects. Intentional partnerships benefit from structural pillars and entry points that systematically facilitate proven practices for effective partnerships and collaboration. USAID has supported evidence-based research about what makes partnerships effective, and they also have a Collaborating, Learning, and Adapting (CLA)8 set of practices and toolkits that can provide valuable information on ways to improve BA/BR collaboration. While the current collaboration may yield some positive transactional results, to truly transform SBC for health results, the implementing teams and USAID would benefit from intentionally using more organizational and KM best practices, such as after action reviews, peer assists, pause-and-reflect, digital whiteboarding, and culturally appropriate fail fairs, among others. 7 Management Sciences for Health. (2017) Leadership, Management and Governance Evidence Compendium: From Intuition to Evidence: Why Leadership, Management, and Governance Matters for Health System Strengthening. Washington, DC: USAID. https://www.usaid.gov/sites/default/files/documents/1864/LMG_Evidence_Compendium_Introduction_and_Pharm_ chapters-508.pdf 8 USAID (ND) CLA at USAID – Making Collaboration more Effective. Website. https://usaidlearninglab.org/lab￾notes/were-always-collaborating-how-can-we-make-it-more-effective MID-TERM PERFORMANCE EVALUATION OF BA: LOCAL LESSONS, GLOBAL LEARNING 32 Need for improved collaboration and coordination Overall, evaluation key informants felt that the partnership was collaborative but uneven. Many found the mechanism awkward, but also considered it an interesting experience. According to a USAID/W key informant, “I think it was a good idea to find out it’s not a good idea.” Recommendations from key informants at USAID, BA, BR, and IPs on how to improve BA/BR collaboration include, but are not limited to: having a researcher as a part of the AOR team, clarifying and underscoring their shared objectives, co-managing the projects, having an SBC research project that is not tied to the implementation work, having collaboration objectives set up from the beginning that recognize the partners’ respective institutional norms and organizational cultures, building in more time for staff interaction between partners, mandating a key staff communication position on BR, earmarking a percentage of every BA buy-in for BR, and routinely using KM practices for exchanges between BA and BR. In addition to these recommendations, organizational science would also suggest that the groups (including USAID) have clear ways to address gaps in expectations and understanding, that the partners’ mutual accountability be reflected in their formal agreements, and that a trained facilitator help the pair explore areas of agreement and potential disagreement.9,10,11,12 Collaboration and partnership behaviors can vary widely and require interpersonal agreement and support. In many ways, the complex interplay of this collaborative partnership is its own SBC effort. BA, BR, and USAID have a great opportunity to apply some of their existing expertise in behavioral and organizational science to better implement and sustain this effort. Interestingly, key informants from BR, BA, and USAID independently shared the same sentiment, aptly put by a BA key informant who remarked, “It’s always a surprise to me that we aren’t practicing what we preach.” As noted earlier, for partnerships to be effective, they do require adequate time, trust, and transparency. 9 Academy for Educational Development. (2001) Designing and Managing Partnerships Between U.S. and Host Country Entities. Washington, DC: USAID. https://www.fhi360.org/sites/default/files/media/documents/Designing_and_managing_partnerships.pdf 10 KPMG International. (2016) Unlocking the Power of Partnership: KPMG Framework for Collaboration. Geneva: KPMG. https://assets.kpmg/content/dam/kpmg/pdf/2016/01/unlocking-power-of-partnership.pdf 11 Branstetter R. et al (2006) Successful Partnerships: a Guide. Paris, France: OECD. https://www.oecd.org/cfe/leed/36279186.pdf 12 Gulati, R. et al. (2012) "The Two Facets of Collaboration: Cooperation and Coordination in Strategic Alliances." Academy of Management Annals 6 (2012): 531–583. https://dash.harvard.edu/handle/1/10996795 MID-TERM PERFORMANCE EVALUATION OF BA: LOCAL LESSONS, GLOBAL LEARNING 33 5. CONCLUSIONS EQ 1: TO WHAT EXTENT HAS BA, THROUGH ITS COUNTRY BUY-INS, ACHIEVED THE MISSIONS’ DESIRED OBJECTIVES IN BEHAVIOR CHANGE AND CAPACITY STRENGTHENING FOR SBC? BA is currently working in 35 countries and has managed to help achieve missions’ desired objectives in SBC and capacity strengthening across a wide range of health areas, including in responding to COVID￾19, and in innovative integrated, cross-sector activities. There is widespread interest in integrated SBC, but there are challenges in implementation, in part due to siloed funding. Most capacity strengthening of in-country IPs and governments occurred through collaborative work, co-creation design sessions, programmatic workshops, and direct mentorship, using a “learning by doing” approach rather than formal training. These efforts enabled host governments and IPs to learn to “think differently” about SBC. The success of these efforts has varied, with continued capacity gaps among local partners to finance and implement SBC activities without the assistance of BA. Key informants see the need for a more strategic approach and theory of change around capacity strengthening. While COVID-19 diverted attention and resources, it also highlighted the importance of effective risk communication. BA was able to mobilize quickly to meet these urgent needs, demonstrating the importance of having a global mechanism in place, along with established relationships and proven technical and management expertise. EQ 2: HOW HAVE BA’S COUNTRY BUY-INS REFLECTED MISSIONS’ EXPECTED TIMELINES, SCALE, AND QUALITY OF DESIGN AND IMPLEMENTATION? Key informants spoke highly of HCD and BE as important and innovative approaches that lead to stronger program design as well as capacity building of partners. While acknowledging that these approaches can take time, most saw them as worth it for the results. In terms of new SBC tools, while many BA staff see the SBC Flow Chart as helpful, there are also issues around it being too complicated for common practical use, making it most appropriate for internal use by BA staff and project partners. Key informants were universally impressed with BA’s speed in getting COVID-19 work going in 22 countries. BA made use of innovative M&E techniques, generating unique data, when possible, including helping create a COVID-19 dashboard with a productive partnership with Facebook. While large, multi-country programs require strong and comprehensive M&E systems, many key informants felt that the large number of indicators required by the BA project were burdensome. A streamlined set of indicators could facilitate an increased use of data and an emphasis on learning that many recommended. EQ 3: HOW AND TO WHAT EXTENT HAS BA ADVANCED THE PRACTICE OF SBC GLOBALLY AND IN THE OP COUNTRIES? BA has greatly elevated the understanding and support for SBC programming at the levels of USAID missions, host country governments, and in regional bodies like the OP. Key informants noted the increased funding for SBC by missions, the increased dialogue about SBC among donors and partners, and how the COVID-19 pandemic has increased recognition of the importance of SBC. BA has helped advance the practice through the targeted sharing of evidence, demonstrating both the cost-effectiveness of SBC and the potential for SBC to shift outcomes through approaches that go beyond messaging, including HCD. Ongoing challenges remain however, including gaps in financing and technical capacity among partners to sustain the design and implementation of SBC activities. MID-TERM PERFORMANCE EVALUATION OF BA: LOCAL LESSONS, GLOBAL LEARNING 34 EQ 4: (BA/BR COLLABORATION) HOW AND TO WHAT EXTENT HAS EACH PROJECT LEVERAGED ITS RELATIONSHIP WITH THE OTHER TO IMPROVE THE SCALE, QUALITY, AND IMPACT OF SBC AT THE COUNTRY, REGIONAL, AND GLOBAL LEVELS? Since their respective formal awards, BA and BR have worked together as mandated through the assistance mechanism set up by USAID. This funder-initiated partnership was strategically designed to deliver efficient and successful project results. The intentional efforts in coordination led to multiple successes, working particularly well when roles were clearly defined and each side needed the other—a "codependence" that key informants noted worked well, for example, in the work on Zika prevention behaviors. While BA and BR worked well together overall, challenges in the partnership seemed to arise when expectations and work orders were less defined (including timeline differences), when frequent staffing changes disrupted workflows and team dynamics, and when perceived or actual power imbalances and different corporate cultures hampered staff members’ comfort levels. MID-TERM PERFORMANCE EVALUATION OF BA: LOCAL LESSONS, GLOBAL LEARNING 35 6. RECOMMENDATIONS 1. BA’s success across a wide range of health areas provides an opportunity to continue to share SBC lessons broadly within USAID and with other donors and IPs, reaching out to groups working on malaria, TB, and other health areas. 2. BA should collaborate with BR to synthesize findings on integrated SBC and develop clear guidance for donors and implementers, including possible ways to reduce the barriers due to siloed funding streams and systems. 3. BA should use existing platforms and relationships to maximize the impact of BA and BR’s research and lessons by crafting practical, programmatic guidance and conducting joint dissemination activities, for example, on provider behavior change and other topics. 4. BA should work closely with USAID to support sharing of lessons among missions, including preparing briefs and other supporting materials. 5. The SBC field should continue to use newer participatory approaches, such as HCD and BE, with appropriate expectations on timeframes. While HCD can be a slow process, many partners speak highly of it, and both HCD and BE appear to have a positive impact on capacity building and local ownership. 6. BA should share lessons learned on the life stages segmentation approach used in Nigeria to ensure that integrated programs are client-centered and relevant. 7. Given its complexity and based on key informant feedback, the SBC Flow Chart should be used mostly among BA coalition members for design and strategic planning purposes. As BA continues to streamline the tool, they should also continue to obtain feedback about its suitability for wider use. 8. BA should develop a clear articulation and theory of change of how work with IPs is strategically building SBC capacity in a way that leads to greater sustainability. 9. In addition to its work with local governments, BA should focus capacity strengthening efforts on civil society and private sector groups to ensure that the responsibility and skills to design and implement SBC activities are distributed outside government. 10. In support of USAID’s commitment to being a learning organization, BA should continue integrating routine knowledge management and CLA practices into the project and encourage more continuous learning and engagement with partners to stimulate innovation, foster better decision-making, and build more systematic exchanges. 11. Future SBC projects should try to improve the balance between the desire for comprehensive monitoring and evaluation systems with a condensed, streamlined indicator list in order to make better use of the data that is collected. 12. BA and USAID should leverage current elevated global interest in SBC, due to COVID￾19, to advocate and increase support for SBC globally, including through use of the business cases developed by BR. MID-TERM PERFORMANCE EVALUATION OF BA: LOCAL LESSONS, GLOBAL LEARNING 36 ANNEXES MID-TERM PERFORMANCE EVALUATION OF BA: LOCAL LESSONS, GLOBAL LEARNING 37 ANNEX 1: SCOPE OF WORK Assignment #: 002 [assigned by GH EvaLS] Global Health Evaluation and Learning Support Project (GH EvaLS) Contract No. GS-10F-154BA EVALUATION OR ANALYTIC ACTIVITY STATEMENT OF WORK (SOW) Date of Submission: July 6, 2020 Last update: October 7, 2020 INSTRUCTIONS: Complete this template in MS Word to develop a SOW for an evaluation, assessment, or other analytic activity. Please be as thorough as possible in completing this SOW. Some of the sections below have been pre-populated with information that is common to most analytic activities. Please review these details and edit as needed to fit the needs of your specific analytic activity. Refer to the USAID How-To Note: Evaluation SOW and the Evaluation SOW: Good Practice Examples when developing your SOW. I. TITLE: Performance Evaluations of Breakthrough ACTION and Breakthrough RESEARCH II. Funder/Requester/Client: USAID/Washington Office/Division: PRH/PEC III. Funding Account Source(s): (Click on box(es) to indicate source of payment for this assignment) c 3.1.1 HIV c 3.1.2 TB c 3.1.3 Malaria c 3.1.4 PIOET c 3.1.5 Other public health threats c 3.1.6 MCH g 3.1.7 FP/RH c 3.1.8 WSSH c 3.1.9 Nutrition c 3.2.0 Other (specify) IV. Budget Ceiling: $431,008.18 (Note: GH EvaLS will provide a cost estimate based on this SOW) V. Performance Period Expected Start Date (on or about): o/a October 19, 2020 Anticipated End Date (on or about): o/a May 15, 2021 VI. Location(s) of Assignment: (Indicate where work will be performed) Work will be remote from consultants’ homes of record. The possibility of travel will be revisited according to the COVID-19 situation. VII. Type of Analytic Activity (Check the box to indicate the type of analytic activity) EVALUATION: g Performance Evaluation (Check timing of data collection) g Midterm c Endline c Other (specify): MID-TERM PERFORMANCE EVALUATION OF BA: LOCAL LESSONS, GLOBAL LEARNING 38 Performance evaluations encompass a broad range of evaluation methods. They often incorporate before– after comparisons but generally lack a rigorously defined counterfactual. Performance evaluations may address descriptive, normative, and/or cause-and-effect questions. They may focus on what a particular project or program has achieved (at any point during or after implementation); how it was implemented; how it was perceived and valued; and other questions that are pertinent to design, management, and operational decision making. VIII. BACKGROUND If an evaluation, Project/Program being evaluated: Project/Activity Title: Breakthrough ACTION (BA) and Breakthrough RESEARCH (BR) Award/Contract Number: BA: AID-OAA-A-17-00017 BR: AID-OAA-A-17-00018 Award/Contract Dates: BA: July 21, 2017 – July 20, 2022 BR: August 1, 2017 - July 31, 2022 Project/Activity Funding: BA: $300,000,000.00 BR: $53,979,420.00 Implementing Organization(s): BA Prime: Johns Hopkins Center for Communication Programs; BA Sub-primes: Save the Children, ThinkPlace, Camber Collective, ideas42, Viamo, International Center for Research on Women (ICRW) BR Prime: Population Council; BR Sub-primes: Avenir Health, ideas 42, Institute for Reproductive Health, Population Reference Bureau, Tulane University Project/Activity AOR/COR: Lindsay Swisher Background of project/program/intervention (Provide a brief background on the country and/or sector context; specific problem or opportunity the intervention addresses; and the development hypothesis) The United States Agency for International Development (USAID) invested in five-year cooperative agreements titled Breakthrough ACTION (BA) and Breakthrough RESEARCH (BR) to support countries in achieving desired improvements in health and development outcomes, including increasing the demand for family planning that is satisfied with modern contraception; ending preventable child and maternal deaths; achieving and maintaining an AIDS-free generation; and achieving a malaria-free world. BA and BR are closely linked and coordinate with one another. The sister projects comprise USAID’s flagship investment in social and behavior change (SBC), providing global and country-level technical leadership in SBC advocacy, design, implementation, research, and evaluation. Both projects contribute to the shared purpose of increasing the practice of priority health behaviors and enabling social norms for improved health and development outcomes. Specifically, BA works to increase coverage of, and innovate based on, investments in SBC programming that already have significant evidence, while BR works to disseminate and advance research around SBC technical areas and interventions in which existing evidence is considered insufficient. The projects build upon USAID’s previous investments in SBC research and programming, including both global and bilateral projects, to simultaneously guide new learning and drive broader application of proven practices and tools in SBC. BA aims to fulfill a global leadership function that is desperately needed within SBC, working through a number of new and existing platforms to create opportunities for technical agenda-setting, learning, and collaboration; designing and implementing innovative and MID-TERM PERFORMANCE EVALUATION OF BA: LOCAL LESSONS, GLOBAL LEARNING 39 strategic SBC programs; and promoting agreed-upon priorities through its own programs and knowledge management efforts. Meanwhile, BR strives to convene and engage a broad range of health and development stakeholders, supporting them in developing, promoting, and operationalizing visionary, consensus-driven agendas for SBC research that contribute to measurable global health impact. The shared strategic objective of the two Breakthrough projects is increased integration of proven SBC interventions in health and development programs, particularly health service delivery platforms. While focused primarily on health, the projects also occasionally address SBC needs in other sectors, with particular attention to areas of potential complementarity such as environmental conservation, agriculture, food security, and nutrition. Within the health sector, the projects maintain a substantive focus on family planning and reproductive health (FP/RH); HIV/AIDS; malaria; and maternal, neonatal, and child health (MNCH); with attention to emerging pandemic threats and other infectious diseases as needed. Theory of Change of target project/program/intervention Breakthrough ACTION (BA) BA was designed to maintain a clear focus on achieving measurable change in priority health and development behaviors. In health, these behaviors include those posited to offer the greatest impact upon health outcomes. BA is expected to contribute to shifts in several priority behaviors and enabling social norms (with an emphasis on FP/RH, HIV/AIDS, MNCH, and malaria health outcomes), as well as the generation and synthesis of evidence on “gateway behaviors” and social norms. These include, but are not limited to, gender norms that offer potential to impact outcomes in one or more health and development area, including couples’ communication and shared decision-making, health-seeking behaviors, etc. BA strives to engage a broad range of health and development stakeholders, supporting them in developing, promoting, and operationalizing a visionary, consensus-driven agenda for SBC that contributes to measurable global health impact through the use of structural and environmental interventions, drawing upon concepts in behavioral economics, human-centered design, social capital, and social psychology in designing effective solutions to social and behavioral challenges. BA’s country-level behavior change programs are premised on the Social-Ecological Model. Thus, its interventions are implemented across social-ecological levels, as appropriate: at the individual and household levels to address ideational factors; at the community level to respond to access factors and social norms, including gender norms; at the institutional level to enhance provider skills; and at the structural level to advocate for policies and resources that support and enable health practices. In many countries, BA’s programs are also informed by the Circle of Care, a model developed under the HC3 project that articulates the role of SBC before, during, and after the time that a client accesses health services. In addition to these broad frameworks, some BA country programs have developed theories of change specific to their program portfolios. Breakthrough RESEARCH (BR) BR addresses a clear need for increased study of SBC interventions. Few existing studies examine mechanisms of effect in large, multi-component interventions, raise questions of cost-benefit or cost￾effectiveness, or explore challenges in implementation that may negatively affect behavioral impact. There is also limited understanding of the sustained behavioral or health impact of SBC due in part to donor priorities and funding cycles as well as a failure of donors and governments to systematically MID-TERM PERFORMANCE EVALUATION OF BA: LOCAL LESSONS, GLOBAL LEARNING 40 coordinate their evaluation efforts to allow for longer-term review of intervention impact. Lack of systematic exchange across health areas and development sectors has contributed to all of these challenges. BR was designed to meet this need with a focus on producing, packaging, and disseminating research that may be utilized by SBC implementers in their work. In addition to conducting and disseminating the social science research that has traditionally been the mainstay of USAID’s investments in SBC, BR was also designed to develop, test, or disseminate innovative or under-utilized research, monitoring, and evaluation approaches for SBC, with an eye to supporting tactical, real-time application of data in programmatic decision-making. BR’s work is premised on the theory that coordination, collaboration, and consensus-building around SBC research needs would enable the development of effective shared agendas for research, which would in turn support design, implementation, and utilization of relevant evidence that was accessible and useful to SBC practitioners. BR envisioned that the evidence generated by the project would demonstrate “what works” for SBC programs; refine proven interventions to identify “how it can work best” in a given context; and determine “how it can be scaled and sustained,” with particular attention to cost efficiencies. BR acknowledged that operationalizing this theory of change requires recognizing relationship dynamics and flows of funding, ideas, and accountability among actors, and addressing interlinkages, feedback loops, and hidden assumptions to expose elements that constrain or highlight opportunities for more effective and efficient policy and programming.” Strategic or Results Framework for the project/program/intervention (paste framework below) Shared Results Framework The following are BA and BR’s shared intermediate results (IRs) detailing the expected results of their work. BA was designed to directly contribute to IRs 1 and 2, as well as their eight sub-intermediate results (SIRs); while BR was designed to contribute to IR 3 and its associated SIRs: ● IR 1: Country-driven, high-quality SBC interventions implemented ○ SIR 1.1: Innovative and effective solutions to high-priority SBC challenges designed and implemented ○ SIR 1.2: Improved SBC capacity demonstrated by host-country governments and local health and development program implementers ○ SIR 1.3: Strategies applied for improved coordination and integration among SBC, service delivery, and development program implementers ○ SIR 1.4: Rigorous monitoring and quality assurance tools and approaches applied to SBC interventions ● IR 2: Coordinated global and country leadership mobilized to address priority SBC challenges ○ SIR 2.1: Multi-institutional platforms at global and regional levels leveraged to share and coordinate around SBC challenges ○ SIR 2.2: Global, regional, and country SBC programming and investment agendas created to address priority gaps and opportunities ○ SIR 2.3: Investment in SBC programming leveraged ○ SIR 2.4: SBC priorities and agendas diffused through new and existing platforms at the global, regional, and country levels ● IR 3: Evidence for impact, feasibility, and cost-benefit of SBC intervention applied ○ SIR 3.1: Research agendas developed and disseminated within health and development communities of practice ○ SIR 3.2: Evidence of SBC impact, feasibility, and cost-benefit generated MID-TERM PERFORMANCE EVALUATION OF BA: LOCAL LESSONS, GLOBAL LEARNING 41 ○ SIR 3.3: Evidence of SBC impact, feasibility, and cost-benefit packaged and shared with priority audiences, including implementers, at global, regional, and country levels ○ SIR 3.4: SBC evidence and learning used in targeted United States Government (USG)- supported country programs As evidenced by their shared Results Framework, BA and BR are closely related and coordinate with one another. However, because they have experienced distinct successes and challenges since their inception, USAID proposes contracting a single evaluation team to conduct separate evaluations of the projects, thereby measuring their individual performance while exploring relational factors as well. The Global Health Bureau (GH)’s previous social and behavior change flagship, the Health Communication Capacity Collaborative (HC3), had a mandate to provide technical assistance for design, implementation, and evaluation of social and behavior change communication (SBCC) programs. During the re-design phase, GH staff saw potential advantages to the procurement of two separate SBC mechanisms, one focused on supporting SBC research and evaluation needs, and one focused on providing technical assistance to SBC program delivery. The possible benefits of two mechanisms included: access to specialized skills in SBC research and evaluation, particularly for external performance and impact evaluations of USAID’s Mission-level SBC investments; attention to evidence generation, synthesis, and learning within the broader SBC community; and expansion of USAID’s SBC partnership base to include less established organizations with potential to support the Agency’s steadily growing demand for SBC technical expertise, while continuing to meet an ongoing need for SBC technical assistance from organizations with global leadership experience and deep program design and delivery expertise. Please refer to the Breakthrough Log Frame and BA and BR’s Performance Monitoring Plans (PMPs) for more information on anticipated and actual project outputs. What is the geographic coverage and/or the target groups for the project or program that is the subject of analysis? Both projects are global in nature. BA currently works in 35 countries with a range of funding and programming scenarios. BR works in significantly fewer countries. We have proposed a subset of countries for closer examination: Nigeria, West Africa region (potentially Niger), and Côte d’Ivoire. IX. Purpose, Audience & Application A. Purpose: Why is this evaluation/assessment being conducted (purpose of analytic activity)? Provide the specific reason for this activity, linking it to future decisions to be made by USAID leadership, partner governments, and/or other key stakeholders. Given that several Breakthrough ACTION (BA) and Breakthrough RESEARCH (BR) activities are still in early stages of implementation, the purpose of these mid-term evaluations is to (1) assess BA and BR’s performance thus far in their tenure as benchmarked by the intermediate and sub-intermediate results dictated in their scopes of work; (2) garner evidence for BA and BR’s underlying theory of change; and (3) capture emerging results to inform decisions about current and future social and behavior change (SBC) programming. Because BR and BA are closely related but have experienced distinct successes and challenges since their inception, USAID proposes contracting a single evaluation team to conduct separate evaluations of the projects, thereby measuring individual performance while exploring relational factors as well. MID-TERM PERFORMANCE EVALUATION OF BA: LOCAL LESSONS, GLOBAL LEARNING 42 B. Audience: Who is the intended audience for this analysis? Who will use the results? If listing multiple audiences, indicate which are most important. The primary audiences for these evaluations are the USAID BA and BR management teams, which include USAID SBC advisors across health areas. Secondary audiences include BA; BR; and their sub￾partners (i.e. Johns Hopkins Center for Communication Programs, Population Council, Save the Children, ThinkPlace, Camber Collective, ideas42, Avenir Health, Institute for Reproductive Health, Population Reference Bureau, Tulane University); USAID Missions staff implementing BA and BR activities in-country; and other SBC-related projects and their management teams. Sensitive components of these evaluations will be delivered in an internal memo to USAID. C. Applications and use: How will the findings be used? What future decisions will be made based on these findings? Findings of the evaluations will be used to improve ongoing BA and BR interactions (with each other, with USAID), work processes, and activities for the remaining two years of these projects. Additionally, findings will be used to inform future programming in relevant technical areas, including addressing gaps in capacity related to SBC interventions and the benefits and challenges of separating research and implementation mechanisms into separate but related projects. The findings will also contribute to a larger body of evidence regarding the success of SBC interventions across health areas. X. Evaluation/Analytic Questions & Matrix: ● Questions should be: a) aligned with the evaluation/assessment purpose and the expected use of findings; b) clearly defined to produce needed evidence and results; and c) answerable given the time and budget constraints. Include any disaggregation (e.g., sex, geographic locale, age, etc.), they must be incorporated into the evaluation/assessment questions. USAID Evaluation Policy recommends 1to 5 evaluation questions. ● State the method and/or data source and describe the data elements needed to answer the evaluation questions MID-TERM PERFORMANCE EVALUATION OF BA: LOCAL LESSONS, GLOBAL LEARNING 43 Breakthrough ACTION Evaluation Question* Method & Data Source Relevant IRs/SIRs 1 To what extent has BA, through its country buy-ins, achieved the Mission’s desired objectives in behavior change and capacity strengthening for SBC? Areas to consider: What factors have influenced differences in results among health areas? To what extent was the project able to design and implement strategies to influence social and gender norms that constrain or facilitate practice of targeted behaviors? How did integration across health areas influence or hinder achievement of project objectives? For buy￾ins that explicitly name capacity strengthening in their SOWs, has BA affected improvements in country-level capacity to design, implement, and monitor SBC activities? How and to what extent has BA affected improvements in the ability of other USAID partners (e.g., service delivery partners, vector control partners, Food for Peace DFSAs) to design and implement SBC activities? What data are available, (e.g., monitoring & tracking data) to support these claims? − Key informant interviews − Observation of BA activities − In-depth interviews or focus group discussions with stakeholders involved in the implementation of BA activities, such as state- or district health officials, religious authorities, or community leaders/targeted beneficiaries. − BA activity reports, materials/products developed and monitoring data IR 1 MID-TERM PERFORMANCE EVALUATION OF BA: LOCAL LESSONS, GLOBAL LEARNING 44 Evaluation Question* Method & Data Source Relevant IRs/SIRs 2 How have BA’s country buy-ins reflected Missions’ expected timelines, scale, and quality of design and implementation? Areas to consider: How did BA’s design process (The Flow Chart) contribute to quality of program designs? To what extent did BA’s programming identify and clearly link to key behavioral determinants of targeted behaviors? How well did BA’s monitoring systems produce data for program refinement and adaptive management? Were stakeholders satisfied with their involvement in design, implementation, and monitoring of programs? What were some gaps or areas where there could be improvement? − BA Management Reviews − Key informant interviews − BA Work Plans − BA Annual and Quarterly Reports (for specific buy-ins) IR 1 MID-TERM PERFORMANCE EVALUATION OF BA: LOCAL LESSONS, GLOBAL LEARNING 45 Evaluation Question* Method & Data Source Relevant IRs/SIRs 3 How and to what extent has BA advanced the practice of SBC globally and in Ouagadougou Partnership countries? Areas to consider: How and to what extent has BA advanced the practice of new approaches to social and behavior change (including those not grounded in communication) globally? How and to what extent has BA advanced the practice of integrated (multi-health element or cross sectoral) SBC globally? How and to what extent have BA’s knowledge management and community building activities contributed to improved understanding of and capacity for SBC among practitioners (including USAID service delivery partners), researchers, governments, and funders? To what extent have shared agendas developed by BA enabled improved coordination or impact in programming? To what extent has BA’s work informed investment decisions within and beyond USAID? − Key informant interviews − BA activity reports and monitoring data − Data analytics: Springboard/Compass https://springboardforsbc.org/ https://www.thecompassforsbc.org − Relevant publications (peer￾reviewed journals, white papers, policy briefs) IR 2 * The evaluation may not answer all sub-questions; sub-questions will be used for probing. MID-TERM PERFORMANCE EVALUATION OF BA: LOCAL LESSONS, GLOBAL LEARNING 46 Breakthrough RESEARCH Evaluation Question* Method & Data Source Relevant IRs/SIRs 1 How and to what extent has BR generated evidence to inform SBC programming in USAID priority countries? Areas to consider: What types of evidence have been or are on track to be produced by the end of the project? How does the evidence complement or expand upon the evidence generated by other mechanisms, including Breakthrough Action, USAID mission bilaterals, and the investments of other donors? To what extent does this evidence address gaps identified by SBC funders, normative bodies, implementers, and communities of practice? What remains that has not been done in this space? − BR activity reports and monitoring data − Journal articles and other publications from prime and sub-primes − Key informant interviews SIR 3.2 2 To what extent has the evidence produced by the project been timely, programmatically relevant, and rigorous to inform programming and investment? Areas to consider: How effectively did BR engage stakeholders in the design and implementation of research? What systems and platforms did BR use to engage stakeholders in determining research needs, priorities, and gaps? To what extent have designs been able to produce generalizable findings, including findings to answer questions around implementation and questions around impact (e.g., behavior change)? − BR Management Reviews − Key informant interviews − BR products, such as briefers and reports IRs 1, 2, 3 MID-TERM PERFORMANCE EVALUATION OF BA: LOCAL LESSONS, GLOBAL LEARNING 47 Evaluation Question* Method & Data Source Relevant IRs/SIRs 3 How has BR advanced the practice of SBC globally and in priority regions? Areas to consider: To what extent have shared agendas developed by BR enabled improved coordination or impact in programming? To what extent has BR’s work informed investment decisions within USAID, within USAID implementing partners, local governments, and beyond? To what extent has BR advanced the practice of integrated (multi-health element or cross sectoral) SBC globally or at the country level? Has BR influenced the uptake of provider behavior change activities by SBC or health service delivery partners globally or at country level? To what extent has BR influenced the application of SBC costing guidelines? Has BR produced resources that are accessible, comprehensible, relevant, and actionable for its primary audiences, including SBC and service delivery implementers, funders, and government decision-makers? To what extent have they reached and been used by the intended audiences? − BR activity reports and monitoring data − Journal articles and other publications from prime, sub￾primes, and other donors that address SBC research − Key informant interviews − BR research utilization plan and analytics from dissemination and utilization activities, including those conducted via Springboard/Compass. SIRs 3.1 & 3.2 MID-TERM PERFORMANCE EVALUATION OF BA: LOCAL LESSONS, GLOBAL LEARNING 48 Evaluation Question* Method & Data Source Relevant IRs/SIRs 4 To what extent has BR situated its work within the larger context of SBC and technical area-specific programming? Areas to consider: How effective has BR been in positioning itself as a global leader in SBC research and evaluation? How has this impacted its ability to achieve project objectives? How has BR engaged with existing communities of practice and thought leaders/experts in SBC, priority health technical areas, service delivery, and health systems strengthening? How and to what extent has BR adapted its priorities and programming based on evolving thinking within these communities of practice, governments, and donors? − Key informant interviews − BR Management Reviews SIR 3.3 * The evaluation may not answer all sub-question; sub-questions will be used for probing. Both Breakthrough ACTION and Breakthrough RESEARCH: The purpose of this question is to inform future project design by assessing the costs and benefits of having BA and BR function as separate SBC mechanisms, as opposed to having only one comprehensive SBC mechanism. This question need not be included in the external evaluation report. Evaluation Question* Method & Data Source Relevant IRs/SIRs 1 How and to what extent has each project leveraged its relationship with the other to improve the scale, quality, and impact of SBC at the country, regional, and global levels? Areas to consider: How does this collaboration facilitate or impede achievement of project objectives? How has evidence and tools generated by BR been used by BA? What opportunities for collaboration have been missed? What has been the strategic advantage and value-add of separate implementation and research/evaluation mechanisms? What were the unintended consequences of having two mechanisms? − Key informant interviews − BA and BR Management Reviews − BA and BR activity reports and monitoring data IRs 1, 2, 3 * The evaluation may not answer all sub-questions; sub-questions will be used for probing. MID-TERM PERFORMANCE EVALUATION OF BA: LOCAL LESSONS, GLOBAL LEARNING 49 XI. Methods: Check and describe the recommended methods for this analytic activity. Selection of methods should be aligned with the evaluation/assessment questions and fit within the time and resources allotted for this analytic activity. Also, include the sample or sampling frame in the description of each method selected. General Comments related to Methods: This scope of work accounts for two separate performance evaluations of BA and BR. Although the evaluation teams will overlap, the teams will be expected to submit two separate sets of deliverables. USAID requests that the contractor propose the appropriate methodologies and instruments in the evaluation design phase to carry out this requirement. For purposes of review of BA’s country activities, the evaluation team will explore Mission buy-ins from 1) USAID/Nigeria, 2) USAID/West Africa, and 3) USAID/Côte d’Ivoire. It should be noted that USAID/West Africa’s buy-in, which focuses exclusively on family planning and reproductive health, includes activities in urban and peri-urban areas of Niger, Burkina Faso, Togo, and Cote d’Ivoire. USAID/Côte d’Ivoire’s bilateral buy-in focuses on malaria, HIV/AIDS, and emerging pandemic threats/global health security, and is distinct from the family planning activities funded by USAID/West Africa in Abidjan. The USAID/West Africa-funded work in CDI is located in three intervention areas: Abidjan, Bouake, and Daloa. USAID expects that the evaluation team will conduct site visits in Nigeria, Côte d’Ivoire, and some or all of the additional countries covered by USAID/West Africa’s regional buy-in (Niger, Burkina Faso, and Togo). For purposes of review of BR’s country activities, the evaluation team will explore a Mission buy-in from USAID/Nigeria, as well as a large portfolio of USAID/PRH-funded work in urban and peri-urban areas of Niger, Burkina Faso, Togo, and Côte d’Ivoire, which is intended to complement USAID/West Africa’s regional investment in BA (described above). USAID expects that local consultant members of the evaluation team will conduct one site visit each in Nigeria, Côte d’Ivoire, and at least one additional country covered by USAID/PRH’s regional investments (Niger, Burkina Faso, and Togo), when local travel regulations and security situations permit. Key informant interviews with country-level stakeholders (USAID and BA/BR to provide) will be conducted on a locally accessible and convenient digital platform, or by phone. USAID identified the countries above as areas for special consideration in the evaluation as they exemplify different aspects of BA and BR’s portfolios. USAID/Nigeria is the single largest funder of both BA and BR outside USAID/Global Health, and the projects’ activities there, which are closely linked, reflect the mandate of the projects as envisioned by USAID. The projects’ family planning-related work in West Africa, which is jointly funded by USAID/West Africa and USAID/PRH, is of critical strategic importance to both funding units, and exemplifies the projects’ mandate to advance health outcomes through a combination of direct implementation, advocacy and agenda-setting, monitoring and evaluation, and research utilization. BA’s bilateral program in Côte d’Ivoire illustrates the project’s vertical work in HIV/AIDS and emerging pandemic threats/global health security and offers a counterpoint to the Nigeria program, in which interventions across health areas are integrated. The matrices below detail the current (March 2020) BA and BR buy-ins to provide greater context for the recommended field sites for data collection. (Brand new buy-ins that include only activities related to the COVID-19 pandemic are excluded from these matrices.) MID-TERM PERFORMANCE EVALUATION OF BA: LOCAL LESSONS, GLOBAL LEARNING 50 Breakthrough ACTION: Country/ Region Health Elements Approx. Investment Size ($/year) Age/Duration of Buy-In Current Status Extent of Collaboration with USAID/W (Low/Med/High) Botswana HIV $1,078,000 1 year/1year Closed N/A Cameroon PMI $750,000 2 years/ 3.5 years Active Low Côte d'Ivoire PEPFAR, PMI, FP, MCH, Nut, WASH $6,200,000 2.5 years/ 5 years Active High DRC MCH, TB, FP, TB, PMI, WASH, Nut, Ed, Media $5,400,000 2.5years/ 5 years Active High eSwatini HIV, DREAMS 2,540,000/+ $166,667 (Y3- Y5) 2.5years/ 5 years Active High Ghana MCH, FP, Nut, PMI, GHSA $1,443,050 2 years/3 years Active High Guatemala Ag $500,000/7 months 7 months/ 7months Closed Low Guinea GHSA $1,250,000 $800,000 1.5 years/ 2 years; GHSA: 2 years/2 years Active High Guyana LAC $736,770 2.5-3/3 years Active Low Liberia PMI, MCH, FP, WASH, GHSA $4,685,000 0 years/1 year New Medium Malawi DRG, PMI, Ed (CEFM, upcoming) $350,000 (PMI) 0.5 year/ 1 year Active Medium Mali GHSA $1,326,667 1.5 years/ 1.5 years Active Low Nepal FP, MCH, WASH, Ed, PMI $1,000,000; 700,000 (CEFM) 1.25 years/ 1.5 years; NEW/3 years Active Medium Niger PMI $444,444 2 years/ 4.5 years Active Medium Nigeria* FP, MCH, Nut, PMI, TB, PIOET $11,800,000 2.5 years/ 5 years Active High Rwanda PMI, MCH, FP $700,000 2 years/2years Active N/A Sahel/Rise MCH, FP, Nut, TB, WASH $4,000,000 1.75 years/ 4 years Active High MID-TERM PERFORMANCE EVALUATION OF BA: LOCAL LESSONS, GLOBAL LEARNING 51 Senegal GHSA $750,000 1.5/2 years Active High Sierra Leone PMI, GHSA $1,450,000 2.5/4 years (PMI); 2.5/3 years (GHSA) Active High Tanzania PMI $533,333 0.25 years/ 1.25 years Active Medium WABA FP $1,000,000 2 years/3 years Active Medium Zambia PMI, MCH, FP, Nut, HIV $3,897,717 2.5 years/ 3 years Pending Closure Medium * Nigeria has an additional COVID-19 related buy-in not included here. Breakthrough RESEARCH: Country/ Region Health Elements Investment Size ($/year) Age/Duration of Buy-In Current Status (Active/ Closing) Extent of Collaboration with USAID/W Philippines TB, FP $370,000 1 year/1year Closed Low Nigeria MCH, FP/RH, PMI $1,500,000 1.5 years/4 years Active High Mali PD&L $870,544 1 yr/1yr Closing Medium Tanzania HIV/AIDS, TB, MCH, FP/RH $400,000 1 yr/X yrs Active High Sahel WASH, MCH, Nut, FP/RH $400,000 1yr/X yrs Active Medium In addition to the site visits described above, all other data collection will be performed by other means including, but not limited to, phone, Skype, and email, as well as document retrieval/review. A preliminary list of interviewees, survey respondents, and/or focus group participants (including Mission staff, USAID Washington staff, and others) will be provided. Document and Data Review (list of documents and data recommended for review) This desk review will be used to provide background information on the project/program and will also provide data for analysis for these evaluations. Documents and data to be reviewed include: ● RFAs; ● Cooperative Agreements; ● Financial tracking documents and financial reports; ● Annual and Quarterly Reports; ● Annual Work Plans; ● MEL Plans; ● Trip reports; ● Performance reports; ● Gender analyses; ● Relevant sections of the Project Appraisal Document; ● Management Reviews; MID-TERM PERFORMANCE EVALUATION OF BA: LOCAL LESSONS, GLOBAL LEARNING 52 ● Miscellaneous thematic reports from other sources; ● Journal articles and other publications from prime, sub-primes, and other donors that address SBC research ● Field support SOWs, work plans, and performance reports relevant to the selected field sites; ● Project-developed deliverables, including communication products, tools, presentations, reports and publications; ● Other relevant publications and reporting from USAID or other donor projects that address SBC Key Informant Interviews (list categories of key informants, and purpose of inquiry) We anticipate interviewing the following groups of Key Informants: ● USAID/Washington-based staff; ● Global Health SBC Technical Advisors involved in those Missions selected for site visits; ● The AOR team managing BR and BA in Washington (in order to gauge BR and BA’s performance from USAID’s standpoint); ● West Africa, Nigeria, and Côte d’Ivoire BR/BA Mission staff (in order to gauge BR and BA’s performance from the projects’ standpoints); ● US-based and field-based actors familiar with BR and BA’s performance with other partners and their roles in the SBC global landscape. Focus Group Discussions (list categories of groups, and purpose of inquiry) To be determined. Group Interviews (list categories of groups, and purpose of inquiry) Key informants may be interviewed in small groups of similar respondents, as long as all participants feel free to express their own opinions. XII. HUMAN SUBJECT PROTECTION The Analytic Team must develop protocols to insure privacy and confidentiality prior to any data collection. Primary data collection must include a consent process that contains the purpose of the evaluation, the risk and benefits to the respondents and community, the right to refuse to answer any question, and the right to refuse participation in the evaluation at any time without consequences. Only adults can consent as part of this evaluation. Minors cannot be respondents to any interview or survey and cannot participate in a focus group discussion without going through an IRB. The only time minors can be observed as part of this evaluation is as part of a large community-wide public event, when they are part of family and community in the public setting. During the process of this evaluation, if data are abstracted from existing documents that include unique identifiers, data can only be abstracted without this identifying information. An Informed Consent statement included in all data collection interactions must contain: ● Introduction of facilitator/note-taker ● Purpose of the evaluation/assessment ● Purpose of interview/discussion/survey ● Statement that all information provided is confidential and information provided will not be connected to the individual ● Right to refuse to answer questions or participate in interview/discussion/survey ● Request consent prior to initiating data collection (i.e., interview/discussion/survey) MID-TERM PERFORMANCE EVALUATION OF BA: LOCAL LESSONS, GLOBAL LEARNING 53 XIII. ANALYTIC PLAN Describe how the quantitative and qualitative data will be analyzed. Include method or type of analyses, statistical tests, and what data it to be triangulated (if appropriate). For example, a thematic analysis of qualitative interview data, or a descriptive analysis of quantitative survey data. All analyses will be geared to answer the evaluation questions. Additionally, the evaluation will review both qualitative and quantitative data related to the project/program’s achievements against its objectives and/or targets. Thematic review of qualitative data will be performed, connecting the data to the evaluation questions, seeking relationships, context, interpretation, nuances, and homogeneity and outliers to better explain what is happening and the perception of those involved. If relevant, qualitative data will be used to substantiate quantitative findings, provide more insights than quantitative data can provide, and answer questions where other data do not exist. Further, the contractor is requested to provide disaggregated data (e.g., sex, age, geography, or other relevant aspects of beneficiaries) whenever possible. The Evaluation Reports will describe analytic methods employed in these evaluations, including the methods used to ensure reliability of coding and identifying themes in qualitative data. The evaluation team, in collaboration with USAID, will finalize the evaluation methods before fieldwork begins. USAID expects that, at a minimum, the evaluation team will: ● Upon award, familiarize themselves with documentation about the project and USAID’s current assistance for SBC in the health sector as well as other relevant USG health investments in the 3 BA/BR buy-in countries/regions of interest. USAID will ensure that relevant documentation is available to the team. ● Review and assess the existing performance and effectiveness information or data. ● Meet and interview USAID project beneficiaries (as feasible), partners, and host government counterparts at appropriate levels. ● Interview USAID staff and a representative number of experts working in the sector. ● Spend approximately 5-6 weeks total in the specified countries carrying out the field work for evaluation of field support buy-ins identified in this SOW. XIV. ACTIVITIES List the expected activities, such as Team Planning Meeting (TPM), briefings, verification workshop with IPs and stakeholders, etc. Activities and Deliverables may overlap. Give as much detail as possible. A. As suggested by GH EvaLS: Background reading – Several documents are available for review for this analytic activity. These include Breakthrough ACTION and Breakthrough RESEARCH proposals, annual work plans, M&E plans, quarterly progress reports, and routine reports of project performance indicator data as well as survey data reports (i.e., DHS and MICS). This desk review will provide background information for the Evaluation Team and will also be used as data input and evidence for the evaluation. Team Planning Meeting (TPM) – A three to four-day team planning meeting (TPM) will be held at the initiation of this assignment and before the data collection begins. The TPM will: ● Review and clarify any questions on the evaluation SOW ● Clarify team members’ roles and responsibilities ● Establish a team atmosphere, share individual working styles, and agree on procedures for resolving differences of opinion ● Review and finalize evaluation questions MID-TERM PERFORMANCE EVALUATION OF BA: LOCAL LESSONS, GLOBAL LEARNING 54 ● Review and finalize the assignment timeline ● Develop data collection methods, instruments, tools, and guidelines ● Review and clarify any logistical and administrative procedures for the assignment ● Develop a data collection plan ● Draft the evaluation work plan for USAID’s approval ● Develop a preliminary draft outline of the team’s report ● Assign drafting/writing responsibilities for the final report Briefing and Debriefing Meetings – Throughout the evaluation the Team Lead will provide briefings to USAID. The In-Brief and Debrief are likely to include the all Evaluation Team experts, but will be determined in consultation with the Mission. These briefings are: ● Evaluation Launch Call, a call among the USAID, GH EvaLS and the Evaluation Team (ET) to initiate the evaluation activity and review expectations. USAID will review the purpose, expectations, and agenda of the assignment. GH EvaLS will introduce the ET and review the initial schedule and other management issues. ● In-briefs with USAID, as part of the Launch Call and the TPM. At the beginning of the Launch Call and the TPM, the Evaluation Team will meet with USAID to discuss expectations, review evaluation questions, and intended plans. The Team will also raise questions that they may have about the project/program and SOW resulting from their background document review. The time for the in-briefs will be determined among USAID, the Team Lead (TL) and EvaLS prior to the Launch Call and the TPM. ● Work plan and methodology review briefing. Following the TPM, the Evaluation Team will meet with USAID to present an outline of the methods/protocols, timeline and data collection tools. Also, the format and content of the Evaluation report(s) will be discussed. ● In-brief with project to review the evaluation plans and timeline, and for the project to give an overview of the project to the Evaluation Team. ● The Team Lead (TL) will brief the USAID POC weekly to discuss and document in a brief, bulleted email progress on the evaluation. As preliminary findings arise, the TL will share these during the routine briefing, and in an email. ● A final debrief between the Evaluation Team and USAID will be held at the end of the evaluation to present preliminary findings to USAID. During this meeting a summary of the data will be presented, along with high level findings and draft recommendations. For the debrief, the Evaluation Team will prepare a PowerPoint Presentation of the key findings, issues, and recommendations. The evaluation team shall incorporate comments received from USAID during the debrief in the evaluation report. (Note: preliminary findings are not final and as more data sources are developed and analyzed these findings may change.) ● IPs and Stakeholders will be shown a separate less detailed PowerPoint Presentation (no more than 20 slides, based on key findings, conclusions, and recommendations). Fieldwork, Site Visits and Data Collection – The local consultant evaluation team members will conduct site visits for data collection. Selection of sites to be visited will be finalized during TPM in consultation with USAID. The evaluation team will outline and schedule key meetings and site visits prior to departing to the field. Consultations with local stakeholders (KIs) in the Mission/project headquarters (USAID staff, MOH staff, etc.) normally will happen virtually. The exception to this norm is for travel to an implementation site(s) to speak with site-based government staff, project staff and/or individuals who are participating in project activities. Evaluation/Analytic Report – The Evaluation/Analytic Team under the leadership of the Team Lead will develop a report with findings and recommendations (see Analytic Report below). Report writing and submission will include the following steps: MID-TERM PERFORMANCE EVALUATION OF BA: LOCAL LESSONS, GLOBAL LEARNING 55 1. Team Lead will submit draft evaluation report to GH EvaLS for review and formatting 2. GH EvaLS will submit the draft report to USAID 3. USAID will review the draft report in a timely manner, and send their comments and edits back to GH EvaLS 4. USAID will manage implementing partner(s)’s (IP) review of the report and compile and send their comments and edits to GH EvaLS. (Note: USAID will decide what draft they want the IP to review.) 5. GH EvaLS will share USAID’s comments and edits with the Team Lead, who will then do final edits, as needed, and resubmit to GH EvaLS 6. GH EvaLS will review and reformat the final Evaluation/Analytic Report, as needed, and resubmit to USAID for approval. 7. Once Evaluation Report is approved, GH EvaLS will re-format it for 508 compliance and post it to the DEC. The Evaluation Report excludes any procurement-sensitive and other sensitive but unclassified (SBU) information. This information will be submitted in a memo to USAID separate from the Evaluation Report. Data Submission – All quantitative data will be submitted to GH EvaLS in a machine-readable format (CSV or XML). The datasets created as part of this evaluation must be accompanied by a data dictionary that includes a codebook and any other information needed for others to use these data. It is essential that the datasets are stripped of all identifying information, as the data will be public once posted on USAID Development Data Library (DDL). Where feasible, qualitative data that do not contain identifying information should also be submitted to GH EvaLS. B. As suggested by GH/PRH Background Reading: Documents are available for review for this evaluation. These include Breakthrough Project-specific documents such as proposals, annual work plans, M&E plans, quarterly progress reports, and routine reports of project performance indicator data, communication materials, reports, publications, and other deliverable documentation. Documents are also available that provide relevant background information on topics related to the work conducted by BA and BR. This desk review will provide background information for the evaluation team and will also be used as data input and evidence for the evaluation and for the assessment. Team Planning Meeting (TPM): Team planning will be held at the initiation of this assignment and before the data collection begins. Ideally, this will consist of 4 days of in-person meetings, but this will not be a requirement. During the initial planning period, team members will: ● Review and clarify any questions on the evaluation SOW ● Clarify team members’ roles and responsibilities ● Establish a team atmosphere, share individual working styles, and agree on procedures for resolving differences of opinion ● Review and finalize evaluation questions ● Review and finalize the timeline for deliverables/products detailed below ● Develop data collection methods, instruments, tools and guidelines ● Review and clarify any logistical and administrative procedures for the assignment ● Develop a data collection plan ● Draft the evaluation work plan for USAID’s approval ● Develop a preliminary draft outline of the team’s report MID-TERM PERFORMANCE EVALUATION OF BA: LOCAL LESSONS, GLOBAL LEARNING 56 ● Assign drafting/writing responsibilities for the final report Briefing and Debriefing Meetings: Throughout the evaluation, the Team Lead will provide briefings to USAID. The In-Brief and Debrief are likely to include all Evaluation Team experts. These briefings are as follows. ● Evaluation Launch: This will be a call/meeting among USAID and the evaluation team to initiate the evaluation activity and review expectations. USAID will review the purpose, expectations, and agenda of the assignment. The Team Lead will review the initial schedule and review other management issues. ● In-brief with USAID: As part of the Launch Call and at the beginning of the Team Planning period, the Evaluation Team (ET) will meet with USAID to discuss expectations, review evaluation questions, and review intended plans. The evaluation team will also raise questions that they may have about the projects and SOW resulting from their background document review. The time and place for this in-brief will be determined between the Team Lead and USAID prior to the Launch Call and the Team Planning period. ● Work plan and methodology review briefing: At the end of the Team Planning period, the Evaluation Team will meet with USAID to present the evaluation work plan, including the draft schedule and logistical arrangements, delineated role and responsibilities of the team members, evaluation milestones, anticipated schedule of data collection efforts, locations and dates for data collection efforts, and proposed evaluation methodology. Also, the format and content of the evaluation report(s) will be discussed. The contractor will update the evaluation work plan (the lists of interviewees, survey participants, the schedule, etc.) and submit the updated version to the AOR on a weekly basis. ● Project In-brief: The evaluation team will conduct an in-brief with projects to review the evaluation plans and timeline, and for the projects to give an overview of BA and BR to the evaluation team. ● Biweekly USAID briefings: The evaluation team will brief USAID biweekly to discuss progress on the evaluation. ● In-Brief with Mission Staff: An in-brief will be conducted in each country upon the arrival of the evaluation team. The in-briefs will be used to provide the evaluation team with on-the￾ground knowledge about local activities and relevant context prior to conducting data collection. ● Preliminary Debrief to Mission Staff: A preliminary debrief of data collected in each country will be presented to Mission staff and other stakeholders in-country at the end of each site visit. These PowerPoint presentations of key findings from each site visit will also be provided to USAID Washington for review. ● Preliminary Debrief to USAID Washington: A preliminary debrief with USAID BA and BR Management Team and other USAID colleagues, as determined by the management team, will be held at the end of the evaluation to present preliminary findings to USAID. During this meeting, a summary of the data will be presented, along with high-level findings and draft recommendations. For the preliminary debrief, the evaluation team will prepare a PowerPoint Presentation of the key findings, issues, and recommendations. The evaluation team shall incorporate comments received from USAID during the preliminary debrief into the final debrief and evaluation report, as appropriate. (Note: Preliminary findings are not final, and as more data sources are developed and analyzed, these findings may change.) ● Final Debrief to USAID Management Teams: Following submission of the first draft of the report, the evaluation team will hold a final debrief with the USAID BA and BR management teams, with a PowerPoint presentation, prior to the Final Debrief with USAID/PRH. MID-TERM PERFORMANCE EVALUATION OF BA: LOCAL LESSONS, GLOBAL LEARNING 57 ● Final Debrief to USAID/PRH: A final debrief with USAID/PRH will be held at the end of the evaluation following the debrief with the USAID BA and BR management teams. During this meeting, a summary of the evaluation/assessment results will be presented, along with high￾level findings and draft recommendations. For the USAID/PRH final debrief, the evaluation team will prepare a PowerPoint Presentation of the key findings, issues, and recommendations. The evaluation team shall incorporate comments received from USAID during the debrief in the evaluation report. ● Final Debriefs to IPs: A debrief/workshop will be held with each project team. These will occur following the final debrief with USAID BA and BR management teams and will not include any information that may be deemed procurement-sensitive or not suitable by USAID. Evaluation Designs: The evaluation team must submit evaluation designs to the BA/BR AOR. The design will become an annex to the evaluation report and will include: Detailed evaluation design matrix that links the Evaluation Questions from the SOW (in their finalized form) to data sources, methods, and the data analysis plan; draft questionnaires and other data collection instruments or their main features; list of potential interviewees and sites to be visited and proposed selection criteria and/or sampling plan (must include sampling methodology and methods, including a justification of sample size and any applicable calculations); limitations to the evaluation design; and dissemination plan (designed in collaboration with USAID). The data analysis plan should clearly describe the evaluation team’s approach for analyzing quantitative and qualitative data (as applicable), including proposed sample sizes, specific data analysis tools, and any software proposed to be used, with an explanation of how/why these selections will be useful in answering the evaluation questions for this task. Qualitative data should be coded as part of the analysis approach, and the coding used should be included in the appendix of the final report. Sex, geographic, and role (beneficiary, implementer, government official, NGO, etc.) disaggregation must be included in the data analysis where applicable. Once the evaluation team receives the consolidated comments on the initial evaluation design and work plan from USAID, they will be expected to return with a revised evaluation design and work plan. Fieldwork, Site Visits, and Data Collection: The US-based evaluation team members will conduct data collection in the US. Local consultant members of the evaluation team will conduct data collection in several in several international sites (specified above), with calls, emails, and visits to the field as needed/possible. Selection of key informants will be finalized during the initial planning period in consultation with USAID. The evaluation team will outline and schedule key meetings during the Team Planning period. Draft Evaluation Reports: The two draft evaluation reports will address each of the questions identified in the SOW and any other issues the team considers having a bearing on the objectives of the evaluation. Any such issues can be included in the report only after consultation with USAID. The submission date for the draft evaluation reports will be determined in the evaluation work plan. Once the initial draft evaluation reports are submitted, USAID will review and comment on the draft, after which point the AOR will submit the consolidated comments to the evaluation team. The evaluation team will then be asked to submit revised final draft reports, and again USAID will review and send comments on these final draft reports. Final Evaluation Reports: Under the leadership of the Team Lead, the evaluation team will be asked to respond to and incorporate USAID’s draft evaluation report comments into the final evaluation MID-TERM PERFORMANCE EVALUATION OF BA: LOCAL LESSONS, GLOBAL LEARNING 58 report, which will include findings and recommendations. The evaluation team lead will then submit the final reports to the AOR. The Evaluation Reports exclude any procurement-sensitive and other sensitive but unclassified (SBU) information. Please see the Final Report Format Section below for more details. Submission of Datasets to the Development Data Library Per USAID’s Open Data policy (see ADS 579, USAID Development Data) the contractor must also submit to the AOR and the Development Data Library (DDL), at www.usaid.gov/data, in a machine￾readable, non-proprietary format, a copy of any dataset created or obtained in performance of this award, if applicable. The dataset should be organized and documented for use by those not fully familiar with the intervention or evaluation. The datasets created as part of this evaluation must be accompanied by a data dictionary that includes a codebook and any other information needed for others to use these data. It is essential that the datasets are stripped of all identifying information, as the data will be public once posted on USAID Development Data Library (DDL). Where feasible, qualitative data that do not contain identifying information should also be submitted to USAID. Please review ADS 579.3.2.2 Types of Data To Be Submitted to the DDL to determine applicability. Submission of Final Evaluation Report to the Development Experience Clearinghouse Per USAID policy (ADS 201.3.5.18) the contractor must submit the evaluation final report and its summary or summaries to the Development Experience Clearinghouse (DEC) within three months of final approval by USAID. XV. DELIVERABLES AND PRODUCTS Select all deliverables and products required on this analytic activity. For those not listed, add rows as needed or enter them under “Other” in the table below. Provide timelines and deliverable deadlines for each. MID-TERM PERFORMANCE EVALUATION OF BA: LOCAL LESSONS, GLOBAL LEARNING 59 Deliverable/Product Timelines & Deadlines (Estimated) Evaluation Launch/In-brief with USAID Week 1 (week of October 19, 2020) Team Planning Meeting (TPM)/In-depth discussion with USAID on workplan and methodology Week 2 (week of October 26, 2020) Workplan and methodology review briefing with USAID Week 3 (week of November 2, 2020) Submission of workplan to USAID (with methodology, protocol, and data collection instruments, timeline, and evaluation report outlines) Week 4 (week of November 9, 2020) In-brief with Breakthrough Projects Week 4 (week of November 9, 2020) In-Brief with Mission staff To be scheduled before in-country interviews begin Data collection Weeks 5-12 (November 16-January 8, 2021) Routine USAID briefings Biweekly Preliminary Debrief to Mission staff To be scheduled soon after in-county interviews/data collection is completed Data analysis Weeks 13-14 (January 11-22, 2021) Preliminary Debrief to USAID Management Teams and select PRH Staff Week 15 (week of January 25, 2021) Draft Evaluation Reports By February 19, 2021 (end of week 18) Draft Internal USAID Memo By February 19, 2021 (end of week 18) Final Debrief to USAID Management Teams Week 19 (week of February 22, 2021) Final Debriefs to Breakthrough Projects Week 19 (week of February 22, 2021) Final Debrief to USAID/PRH (focusing on results of memo, after having incorporated feedback from various debriefs) Week 21 (week of March 8, 2021) Final Evaluation Reports (including USAID review) Week 25 (by April 9, 2021) Internal USAID Memo Week 26 (by April 16, 2021) Datasets submitted to DDL May 15, 2021 Reports posted to the DEC May 15, 2021 Other (specify): MID-TERM PERFORMANCE EVALUATION OF BA: LOCAL LESSONS, GLOBAL LEARNING 60 Holidays: USAID holidays that fall within this timeline include: - October 12, 2020 (Columbus Day, US); - November 11, 2020 (Veterans Day, US); - November 26, 2020 (Thanksgiving Day, US); - December 25, 2020 (Christmas Day, US); - January 1, 2021 (New Year’s Day, US); - January 18, 2021 (Birthday of Martin Luther King, Jr., US); and - February 15, 2021 (Washington’s Birthday, US) Estimated USAID review time Average number of business days USAID will need to review the Report? 10 business days XVI. TEAM COMPOSITION, SKILLS AND LEVEL OF EFFORT (LOE) Overall Team requirements: Given that BA and BR are closely related but require separate evaluations to assess their performance, separate but overlapping evaluation teams will allow for team members to benefit from improved collective knowledge across evaluations and will ensure cohesive evaluation deliverables. The intended combined size of both evaluation teams is 4-5 team members: Team Members 1-3, and 5 will work across both evaluations and Team Member 4 (Eval Specialist - Research focused) will work exclusively on BR’s evaluation. Given the size and complexity of this activity, the evaluation contractor may specify sub-team leads and/or make other decisions regarding the team composition and distribution of LOE to best carry out this SOW. French language skills are preferred across all team members. The contractor must provide information about evaluation team members, including their curricula vitae, and explain how they meet the requirements in the evaluation SOW. Submissions of writing samples or links to past evaluation reports and related deliverables composed by proposed team members are highly desirable. Per ADS 201.3.5.14, all team members must provide to USAID a signed statement attesting to a lack of conflict of interest or describing an existing conflict of interest relative to the project or activity being evaluated (i.e., a conflict of interest form). Proposed evaluation team members are expected to be the people who execute the work of this assignment. Any substitutes to the proposed evaluation team members must be approved by the AOR before they begin work. USAID may request an interview with any of the proposed evaluation team members via conference call, Skype, or other means. Team Lead (BA & BR): This consultant will be selected from among the key staff and will meet the requirements of both this and the second position. The team lead should have significant experience conducting and leading project evaluations and/or assessments. Roles & Responsibilities: The team leader will be responsible for (1) providing team leadership; (2) managing the team’s activities, (3) leading the evaluation team planning meeting, including the development of the evaluation work plan and data collection tools, (4) ensuring that all deliverables are met in a timely manner, (5) leading data analysis and drafting of the evaluation report, and (6) leading briefings and presentations. MID-TERM PERFORMANCE EVALUATION OF BA: LOCAL LESSONS, GLOBAL LEARNING 61 Qualifications: ● An advance degree in public health, evaluation, or research or related field ● Significant experience leading evaluations of large scale, integrated (multi-health element or cross sectoral) and/or complex USAID-supported projects using both quantitative and qualitative methods required ● Proficiency in English required ● Proficiency in French preferred ● Experience in social and behavior change design, implementation and/or research and evaluation of SBC interventions/programs preferred. ● Familiarity with USAID and other donor agencies preferred ● Familiarity with USAID policies and practices preferred ● Excellent interpersonal skills, including experience successfully interacting with host government officials, civil society partners, and other stakeholders ● Excellent organizational skills and ability to keep to a timeline ● Excellent writing skills, with extensive report writing experience ● Excellent skills in planning, facilitation, and consensus building Key Staff 1 Title: SBC Technical Specialist (BA & BR) Roles & Responsibilities: The SBC Technical Specialist will provide technical input and support the Team Lead in managing the evaluation and key outputs. They will also assist the Team Lead in designing the evaluation plan, conducting the desk review, leading the stakeholder mapping, participating in key informant interviews, conducting data analysis, drafting key sections of the final evaluation report, and in presenting and disseminating findings. Qualifications: ● An advanced degree in public health, evaluation, or research or related field ● Significant experience designing and implementing national-level behavior change programs in LMICs required ● Experience designing and implementing interventions to address social and gender norms strongly preferred ● Knowledge, skills, and experience in qualitative evaluation tools preferred ● Proficiency in English required ● Proficiency in French preferred ● Familiarity with USAID and other donor agencies preferred ● Familiarity with USAID policies and practices preferred ● Excellent interpersonal skills, including experience successfully interacting with host government officials, civil society partners, and other stakeholders ● Excellent organizational skills and ability to keep to a timeline ● Excellent writing skills, with extensive report writing experience Key Staff 2 Title: SBC Technical Specialist with capacity strengthening expertise (BA & BR) Roles & Responsibilities: The SBC Technical Specialist with Capacity Strengthening Expertise will provide technical input and support the Team Lead in managing the evaluation and key outputs. They will help the Team Leads in designing the evaluation plan, interpreting key findings and draft recommendations for the evaluation report, and assisting in the final presentation and dissemination of findings. MID-TERM PERFORMANCE EVALUATION OF BA: LOCAL LESSONS, GLOBAL LEARNING 62 Qualifications: ● An advanced degree in public health, evaluation, or research or related field ● Significant experience designing and implementing interventions to improve capacity at the individual, institutional, and systems level required ● Expertise in social and behavior change programming in LMIC required ● Experience measuring capacity strengthening strongly preferred ● Experience in design and implementation of evaluations preferred ● Proficiency in English required ● Proficiency in French preferred ● Familiarity with USAID and other donor agencies preferred ● Familiarity with USAID policies and practices preferred ● Excellent organizational skills and ability to keep to a timeline ● Excellent writing skills, with extensive report writing experience Key Staff 3 Title: Evaluation Specialist (Research-focused) (BR) Roles & Responsibilities: The Evaluation Specialist will provide quality assurance on evaluation issues, including methods, development of data collection instruments, protocols for data collection, data management, and data analysis. They will participate in all aspects of the evaluation, from planning, data collection, data analysis to drafting sections of the report. Qualifications: ● An advanced degree in public health, evaluation, or research or related field ● 10+ years of experience in M&E procedures and implementation ● 5+ years managing M&E, including evaluations ● Experience in design and implementation of evaluations, data analysis, and data management ● Experience implementing and coordinating surveys, key informant interviews, focus groups, observations and other evaluation methods that assure reliability and validity of the data. ● Experience conducting mixed-method performance evaluations of complex USAID-supported research projects ● Strong knowledge, skills, and experience in qualitative and quantitative evaluation tools ● Proficiency in English required ● Proficiency in French preferred ● Excellent writing skills with extensive report writing experience required ● Expertise in social and behavior change preferred ● Familiarity with USAID and other donor agencies preferred ● Familiarity with USAID policies and practices preferred Key Staff 4 Title: Research Utilization and Knowledge Management Specialist (BA & BR) Roles & Responsibilities: The Research Utilization and Knowledge Management Specialist will provide expertise in assessing communities of practice, partnerships, etc. as a member of the evaluation team. Qualifications: ● An advanced degree in public health, evaluation or research or related field ● Experience assessing communities of practice, partnerships, etc. ● Expertise in social and behavior change programming and research in LMICs preferred ● Proficiency in English required ● Proficiency in French preferred ● Familiarity with USAID and other donor agencies preferred MID-TERM PERFORMANCE EVALUATION OF BA: LOCAL LESSONS, GLOBAL LEARNING 63 ● Familiarity with USAID policies and practices preferred Other Staff Titles with Roles & Responsibilities (include number of individuals needed): Local Evaluators (one per country) will conduct data collection at the country level, as well as contribute to the data analysis and interpretation. They will have familiarity with health topics, as well as experience conducting surveys, key informant interviews, and focus group discussions, both facilitating and note taking during interviews and group discussions. The Local Evaluators will assist with translation of data collection instruments and transcripts, as needed. They will have a good command of English and the local languages, as necessary. They will report to the Team Lead. Local Logistics Coordinators (one per country) will support the Evaluation Team with all logistics and administration to allow them to carry out this evaluation. They will have knowledge of key actors in the health sector and their locations including MOH, donors, and other stakeholders. To support the Team, they will be able to efficiently liaise with hotel staff, arrange in-country transportation (ground and air), arrange meeting and workspace as needed, and ensure business center support, e.g. copying, internet, and printing. The Local Logistics Coordinators will work under the guidance of the Team Leader and Local Evaluators to make preparations, arrange meetings, and appointments. They will conduct programmatic administrative and support tasks as assigned and ensure the processes move forward smoothly. They may also be asked to assist in translation of data collection tools and transcripts, if needed. Will USAID participate as an active team member or designate other key stakeholders to as an active team member? This will require full time commitment during the evaluation or assessment activity. X Full member of the Evaluation Team (including planning, data collection, analysis and report development): Julianne Weis, Family Planning and Reproductive Health Research Advisor, GH/PRH/RTU, will assist in various aspects of the BR and BA evaluations, including the development of data collection tools, methodology discussions, field work and data collection, and data analysis. Some Involvement anticipated – If yes, specify who: [A USAID staff member may join the evaluation team on a part-time basis to perform administrative, logistical, and other tasks as needed. The USAID staff member will not be responsible for flight arrangements and basic logistics but will be responsible for communicating with missions, assisting with administering online surveys and analysis of survey data. The evaluation mechanism is responsible for all travel and other logistics.] Staffing Level of Effort (LOE) Matrix: The LOE Matrix will help you estimate the LOE needed to implement this analytic activity. If you are unsure, GH Pro can assist you to complete this table. a) For each column, replace the label "Position Title" with the actual position title of staff needed for this analytic activity. b) Immediately below each staff title enter the anticipated number of people for each titled position. c) Enter Row labels for each activity, task and deliverable needed to implement this analytic activity. d) Then enter the LOE (estimated number of days) for each activity/task/deliverable corresponding to each titled position. e) At the bottom of the table total the LOE days for each consultant title in the ‘Sub-Total’ cell, then multiply the subtotals in each column by the number of individuals that will hold this title. MID-TERM PERFORMANCE EVALUATION OF BA: LOCAL LESSONS, GLOBAL LEARNING 64 Level of Effort in days for each Evaluation/Analytic Team member (The following is an Illustrative LOE Chart. Please edit to meet the requirements of this activity.) Activity/Deliverable Evaluation Team Team Lead SBC Technical Specialist RUKM Specialist Local Evaluator Local Logistics Coordinator Number of persons → 1 2 1 3 3 1 Launch Briefing 0.5 0.5 0.5 0.5 2 Desk review 5 5 4 1 3 Preparation for Team convening in-country 1 1 4 In-brief with Missions (half a day/Mission) 1.5 1.5 1.5 0.5 5 Team Planning Meeting 4 4 4 1 6 Workplan and methodology briefing with USAID 0.5 0.5 0.5 0.5 7 Evaluation workplan (includes evaluation design matrix and questions, methodology, sampling, data analysis plan, data collection instruments, timeline, and the outline for the final evaluation reports) 2 1.5 1 2 8 US-based data collection 20 20 10 9 Prep/Logistics for in-country data collection and site visits 0.5 0.5 10 In-country data collection and site visits (including travel to sites) 10 10 11 Data analysis 3 3 3 3 12 Out-brief/Debrief with Missions, including preparation time (half a day/Mission) 1.5 1.5 1.5 0.5 13 Project, Project Management, and PRH final debriefs – workshop, including preparation time 2 2 2 14 Draft report(s) 4 4 2 1 15 Revise report(s) per USAID comments, finalize and submit to USAID 8 5 5 Total LOE per person 52 48.5 35 21.5 11.5 Total LOE 52 97 35 64.5 34.5 A 6-day workweek permitted X Yes No 6-day workweek approved for travel to/from work locations X Yes No MID-TERM PERFORMANCE EVALUATION OF BA: LOCAL LESSONS, GLOBAL LEARNING 65 Travel anticipated: List international and local travel anticipated by what team members. International travel to each field site (Nigeria, Côte d’Ivoire, and Niger) by US-based members of the evaluation team is not expected. Travel may be required in part for USAID staff members participating on the evaluation team as needed. Domestic travel to Washington D.C. may be expected for the Team Lead and/or Technical Specialists residing outside the Washington D.C. area depending on the COVID-19 pandemic. Other domestic travel may be approved for team members if deemed necessary, depending on the location of key informants. Travel is permitted on weekends as needed. This SOW assumes that, to mitigate the impacts of the current COVID-19 pandemic on the evaluation, local evaluators will be used so as not to delay the evaluations due to travel restrictions or other safety concerns. Travel restrictions or significant delays may merit a change in scope or timeline. The current COVID-19 pandemic may restrict travel at certain points during the evaluation. If travel is prohibited, USAID and the evaluation team will explore alternative means of data collection and communication. XVII. LOGISTICS Billing up to seven (7) days in any consecutive seven (7)-day period is approved when traveling to or from the Consultant’s home of record X Yes No Visa Requirements List any specific Visa requirements or considerations for entry to countries that will be visited by consultant(s): N/A List recommended/required type of Visa for entry into counties where consultant(s) will work Name of Country Type of Visa N/A Tourist Business No preference Tourist Business No preference Tourist Business No preference Tourist Business No preference Clearances & Other Requirements Note: Most Evaluation/Analytic Teams arrange their own work space, often in conference rooms at their hotels. However, if a Security Clearance or Facility Access is preferred, GH Pro can submit an application for it on the consultant’s behalf. GH EvaLS can obtain Facility Access (FA) and transfer existing Secret Security Clearance for our consultants, but please note these requests, processed through AMS at USAID/GH (Washington, DC), can take 4-6 months to be granted. If you are in a Mission and the RSO is able to grant a temporary FA locally, this can expedite the process. FAs for non-US citizens or Green Card holders must be obtained through the RSO. If FA or Security Clearance is granted through Washington, DC, the consultant must pick up his/her badge in person at the Office of Security in Washington, DC, regardless of where the consultant resides or will work. If Electronic Country Clearance (eCC) is required prior to the consultant’s travel, the consultant is also required to complete the High Threat Security Overseas Seminar (HTSOS). HTSOS is an interactive e-Learning (online) course designed to provide participants with threat and situational MID-TERM PERFORMANCE EVALUATION OF BA: LOCAL LESSONS, GLOBAL LEARNING 66 awareness training against criminal and terrorist attacks while working in high threat regions. There is a small fee required to register for this course. [Note: The course is not required for employees who have taken FACT training within the past five years or have taken HTSOS within the same calendar year.] If eCC is required, and the consultant is expected to work in country more than 45 consecutive days, the consultant may be required complete the one-week Foreign Affairs Counter Threat (FACT) course offered by FSI in West Virginia. This course provides participants with the knowledge and skills to better prepare themselves for living and working in critical and high threat overseas environments. Registration for this course is complicated by high demand (consultants must register approximately 3- 4 months in advance). Additionally, there will be the cost for additional lodging and M&IE to take this course. Check all that the consultant will need to perform this assignment, including USAID Facility Access, GH Pro workspace and travel (other than to and from post). USAID Facility Access (FA) Specify who will require Facility Access: Electronic County Clearance (ECC) (International travelers only) High Threat Security Overseas Seminar (HTSOS) (required in most countries with ECC) Foreign Affairs Counter Threat (FACT) (for consultants working on country more than 45 consecutive days) GH Pro workspace Specify who will require workspace at GH Pro: Travel -other than posting (specify): Other (specify): Specify any country-specific security concerns and/or requirements XVIII. GH EvaLS ROLES AND RESPONSIBILITIES GH EvaLS will coordinate and manage the evaluation/assessment team and provide quality assurance oversight, including: ● Review SOW and recommend revisions as needed ● Provide technical assistance on methodology, as needed ● Develop budget for analytic activity ● Recruit and hire the evaluation/assessment team, with USAID POC approval ● Arrange international travel and lodging for international consultants ● Request for country clearance and/or facility access (if needed) ● Review and assist with development of methods, workplan, analytic instruments, reports, and other deliverables as part of the quality assurance oversight, as appropriate ● Report production - If the report is public, then coordination of draft and finalization steps, editing/formatting, 508ing required in addition to and submission to the DEC and posting on GH Pro website. If the report is internal, then copy editing/formatting for internal distribution. XIX. USAID ROLES AND RESPONSIBILITIES Below is the standard list of USAID’s roles and responsibilities. Add other roles and responsibilities as appropriate. USAID Roles and Responsibilities USAID will provide overall technical leadership and direction for the analytic team throughout the assignment and will provide assistance with the following tasks: MID-TERM PERFORMANCE EVALUATION OF BA: LOCAL LESSONS, GLOBAL LEARNING 67 Before Field Work ● SOW. o Develop SOW. o Peer Review SOW o Respond to queries about the SOW and/or the assignment at large. Consultant Conflict of Interest (COI). To avoid conflicts of interest or the appearance of a COI, review previous employers listed on the CV’s for proposed consultants and provide additional information regarding potential COI with the project contractors evaluated/assessed and information regarding their affiliates. Documents. Identify and prioritize background materials for the consultants and provide them to GH Pro, preferably in electronic form, at least one week prior to the inception of the assignment. Local Consultants. Assist with identification of potential local consultants, including contact information. Site Visit Preparations. Provide a list of site visit locations, key contacts, and suggested length of visit for use in planning in-country travel and accurate estimation of country travel line items costs. Lodgings and Travel. Provide guidance on recommended secure hotels and methods of in￾country travel (i.e., car rental companies and other means of transportation). During Field Work ● Mission Point of Contact. Throughout the in-country work, ensure constant availability of the Point of Contact person and provide technical leadership and direction for the team’s work. ● Meeting Space. Provide guidance on the team’s selection of a meeting space for interviews and/or focus group discussions (i.e. USAID space if available, or other known office/hotel meeting space). ● Meeting Arrangements. Assist the team in arranging and coordinating meetings with stakeholders. ● Facilitate Contact with Implementing Partners. Introduce the analytic team to implementing partners and other stakeholders, and where applicable and appropriate prepare and send out an introduction letter for team’s arrival and/or anticipated meetings. After Field Work ● Timely Reviews. Provide timely review of draft/final reports and approval of deliverables. XX. ANALYTIC REPORT Provide any desired guidance or specifications for Final Report. (See How-To Note: Preparing Evaluation Reports) The Evaluation/Analytic Final Report must follow USAID’s Criteria to Ensure the Quality of the Evaluation Report (found in Appendix I of the USAID Evaluation Policy). ● Each evaluation report must not exceed 25 pages (excluding executive summary, table of contents, acronym list, and annexes). ● The structure of the reports should follow the Evaluation Report template, including branding found here or here. ● Draft reports must be provided electronically, in English, to GH EvaLS who will then submit it to USAID. ● For additional Guidance, please see the Evaluation Reports to the How-To Note on preparing Evaluation Draft Reports found here. MID-TERM PERFORMANCE EVALUATION OF BA: LOCAL LESSONS, GLOBAL LEARNING 68 The Evaluation Reports should exclude any potentially procurement-sensitive information. As needed, any procurement-sensitive information or other sensitive but unclassified (SBU) information will be submitted in a memo to USAID separate from the Evaluation Reports. USAID Criteria to Ensure the Quality of the Evaluation Report (USAID ADS 201): ● Evaluation reports should be readily understood and should identify key points clearly, distinctly, and succinctly. ● The Executive Summary of an evaluation report should present a concise and accurate statement of the most critical elements of the report. ● Evaluation reports should adequately address all evaluation questions included in the SOW, or the evaluation questions subsequently revised and documented in consultation and agreement with USAID. ● Evaluation methodology should be explained in detail and sources of information properly identified. ● Limitations to the evaluation should be adequately disclosed in the report, with particular attention to the limitations associated with the evaluation methodology (selection bias, recall bias, unobservable differences between comparator groups, etc.). ● Evaluation findings should be presented as analyzed facts, evidence, and data and not based on anecdotes, hearsay, or simply the compilation of people’s opinions. ● Findings and conclusions should be specific, concise, and supported by strong quantitative or qualitative evidence. ● If evaluation findings assess person-level outcomes or impact, they should also be separately assessed for both males and females. ● If recommendations are included, they should be supported by a specific set of findings and should be action-oriented, practical, and specific. Reporting Guidelines: The draft report should be a comprehensive analytical evidence-based evaluation report. It should detail and describe results, effects, constraints, and lessons learned, and provide recommendations and identify key questions for future consideration. The report shall follow USAID branding procedures. The report will be edited/formatted and made 508 compliant as required by USAID for public reports and will be posted to the USAID/DEC. The findings from the evaluation will be presented in a draft report at a full briefing with USAID and at a follow-up meeting with key stakeholders. The report should use the following format: ● Abstract: briefly describing what was evaluated, evaluation questions, methods, and key findings or conclusions (not more than 250 words) ● Executive Summary: summarizes key points, including the purpose, background, evaluation questions, methods, limitations, findings, conclusions, and most salient recommendations (2-5 pages) ● Table of Contents (1 page) ● Acronyms ● Evaluation/Analytic Purpose and Evaluation/Analytic Questions: state purpose of, audience for, and anticipated use(s) of the evaluation/assessment (1-2 pages) ● Project [or Program] Background: describe the project/program and the background, including country and sector context, and how the project/program addresses a problem or opportunity (1-3 pages) ● Evaluation/Analytic Methods and Limitations: data collection, sampling, data analysis and limitations (1-3 pages) ● Findings (organized by Evaluation/Analytic Questions): substantiate findings with evidence/data MID-TERM PERFORMANCE EVALUATION OF BA: LOCAL LESSONS, GLOBAL LEARNING 69 ● Conclusions ● Recommendations ● Annexes o Annex I: Evaluation/Analytic Statement of Work o Annex II: Evaluation/Analytic Methods and Limitations ((if not described in full in the main body of the evaluation report) o Annex III: Data Collection Instruments used in conducting the evaluation, such as questionnaires, checklists, and discussion guides o Annex IV: Sources of Information and data identified and listed ▪ List of Persons Interviews ▪ Bibliography of Documents Reviewed ▪ Databases ▪ [etc.] o Annex V: Statement of Differences (if applicable) regarding significant unresolved differences of opinion by funders, implementers, and/or members of the evaluation team o Annex VI: Disclosure of Any Conflicts of Interest o Annex VII: Summary information about evaluation team members, including qualifications, experience, and role on the team. The evaluation methodology and report will be compliant with the USAID Evaluation Policy and Checklist for Assessing USAID Evaluation Reports -------------------------------- The Evaluation Report should exclude any potentially procurement-sensitive information. As needed, any procurement sensitive information or other sensitive but unclassified (SBU) information will be submitted in a memo to USIAD separate from the Evaluation Report. -------------------------------- All data instruments, data sets (if appropriate), presentations, meeting notes and report for this evaluation/analysis will be submitted electronically to the GH EvaLS Program Manager. All datasets developed as part of this evaluation will be submitted to GH EvaLS in an unlocked machine-readable format (CSV or XML). The datasets must not include any identifying or confidential information. The datasets must also be accompanied by a data dictionary that includes a codebook and any other information needed for others to use these data. Qualitative data included in this submission should not contain identifying or confidential information. Category of respondent is acceptable, but names, addresses and other confidential information that can easily lead to identifying the respondent should not be included in any quantitative or qualitative data submitted. MID-TERM PERFORMANCE EVALUATION OF BA: LOCAL LESSONS, GLOBAL LEARNING 70 XXI. USAID CONTACTS Primary Contact Alternate Contact 1 Alternate Contact 2 Name: Lindsay Swisher Angela Brasington Sirisha Bhadriraju Title: Public Health Specialist Senior Social and Behavior Change Advisor Social and Behavior Change Intern USAID Office/Mission PRH PRH PRH Email: Lswisher@usaid.gov abrasington@usaid.gov sbhadriraju@usaid.gov Telephone: (202) 916-2154 (202) 915-2142 N/A Cell Phone: (202) 550-7748 (571) 242-0717 (972) 352-8421 List other contacts who will be supporting the Requesting Team with technical support, such as reviewing SOW and Report (such as USAID/W GH EvaLS management team staff) Technical Support Contact 1 Technical Support Contact 2 Name: Amani Selim Title: Evaluation Technical Advisor USAID Office/Mission PRH Email: aselim@usaid.gov Telephone: (202) 916-2146 Cell Phone: (571) 721-9577 XXII. OTHER REFERENCE MATERIALS Documents and materials needed and/or useful for consultant assignment, that are not listed above LIST OF ANNEXES 1. Key Terms 2. USAID Priority Regions/Countries 3. Breakthrough Website: https://breakthroughactionandresearch.org/ a. “Where We Work” map of geographic areas of operation: https://breakthroughactionandresearch.org/where-we-work/ 4. BA Theory of Change: Social-Ecological Model (Suggs, S.L., & Ratzan, S.C. (2012). Global E￾Health Communication. In R. Obregon & S. Waisboard (Eds.), The Handbook of Global Communication (pp. 251-273). West Sussex: John Wiley and Sons.) and Circle of Care Model (https://breakthroughactionandresearch.org/resources/circle-of-care-model/) 5. BA Design Process: “The Flow Chart” Diagram 6. Breakthrough 2016 Concept Presentation 7. Breakthrough Log Frame 8. BA and BR years 1, 2, and 3 Work Plans of Core and Relevant Buy-ins 9. BA and BR Monitoring, Evaluation, and Learning (MEL) Plans 10. BA and BR years 1 and 2 country Performance Reports (Semi-annual Reports, Annual Reports, and Performance Monitoring Plans) 11. BA and BR years 1 and 2 Management Reviews 12. BA and BR examples of studies conducted and reports disseminated, including types of BR studied conducted (implementation science studies, impact evaluation studies, and descriptive studies) a. BR Implementation Science Studies (12) b. BR Impact Evaluation Studies (2) 13. Relevant BA and BR financial tracking and reporting documents MID-TERM PERFORMANCE EVALUATION OF BA: LOCAL LESSONS, GLOBAL LEARNING 71 14. List of Suggested Evaluation Interviewees XXIII. ADJUSTMENTS MADE IN CARRYING OUT THIS SOW AFTER APPROVAL OF THE SOW (To be completed after Assignment Implementation by GH EvaLS) 1. LOE was adjusted to allow for additional management required by the Team Lead, additional input from the country teams, and additional input from the Research Utilization and Knowledge Management Specialist. 2. The timeline was adjusted both due to delays around the holidays as well as to allow for complete and thorough revision of the report. MID-TERM PERFORMANCE EVALUATION OF BA: LOCAL LESSONS, GLOBAL LEARNING 72 ANNEX 2: DATA COLLECTION INSTRUMENT Breakthrough Action and Breakthrough Research Mid-term Evaluation Combined Question Guide Respondent type: Respondent country: Intro paragraph: My name is . I am part of USAID’s Global Health Evaluation and Learning Support project team conducting an evaluation of two USAID-funded Social and Behavior Change (SBC) projects, Breakthrough Action and Breakthrough Research. We are conducting interviews with a wide range of stakeholders to understand the accomplishments of the projects, as well as the challenges the projects have faced. Your comments and experience will help inform this process and the design of future SBC projects. The interview will take about one hour. All information is confidential. You can refuse to answer any question, and you can decide to stop the interview at any time at no consequence to you. (I would like to record the interview, if you agree to that). Do I have your permission to continue? YES NO Intro question: Please tell me about your experience with Breakthrough Action and Breakthrough Research. Have you interacted directly with the projects and if so, how? PART I. Achievements of BA and BR in SBC programming, research and capacity building 1. What do you see as the main achievements of BA/BR? [for USAID staff also ask] How well has BA helped USAID Missions achieve their objectives? Please give some specific examples. 2. Do you know about/were you involved with any BA and/or BR activities that cut across multiple health areas or across sectors (e.g. nutrition, agriculture)? a. If yes, what were the enablers to the success of these activities? b. What, if any, were the barriers? 3. To what degree have BA and/or BR collaborations with country partners led to improved local capacity for SBC programming/research? Why/Why not? a. How could efforts to build local capacity be improved? 4. What do you think have been BR’s primary contributions to the global evidence-base for SBC programming? 5. How has evidence generated and shared by BR been applied in BA and other USAID SBC programming? a. How can use of BR evidence be increased in BA and other USAID programs? b. Has SBC evidence generated from BR been applied in other SBC programming? 6. Do you know about/Were you involved in BR’s process of identifying gaps in evidence? a. If yes, what worked well in this process? b. What were the challenges in this process? c. What, if anything, would you do differently next time? MID-TERM PERFORMANCE EVALUATION OF BA: LOCAL LESSONS, GLOBAL LEARNING 73 Part II. Design and implementation of activities 7. Were you involved with/Do you know about the design process used for BA activities/BR research? a. If yes, what did you think of this process? b. Do you have suggestions for how it could be improved? 8. Were you involved with/Do you know about the implementation of BA and/or BR activities? a. If yes, how do you think the implementation is going? b. What has contributed to progress or lack of progress in implementation? c. Do you have suggestions for how it could be improved? 9. Do you know about BA and BR’s monitoring and evaluation activities? If yes, a. Are the systems/indicators used for monitoring/evaluation adequate for your needs? b. If not, what could be done to improve monitoring/evaluation processes for BA and BR and for SBC more broadly? Part III. Advancing the practice of SBC 10. Which, if any, BA or BR resources or tools have you used in your SBC activities? a. How useful did you find those resources/tools? 11. From your perspective, has BA programming or BR research increased the level of interest in SBC as an essential component of effective health and development programming? a. If yes, how and among which groups? b. Has it influenced investment in SBC activities within USAID programs and among partner programs (e.g. governments, Ouagadougou Partnership countries, NGO project partners)? If yes, how? 12. On a more personal note, can you tell me about an activity that really seems to be making a difference in people's lives? Or a specific insight that you've had around SBC since working with BA/BR? Part IV. Collaboration between BA and BR 13. In what ways have BA and BR worked together? Please provide specific examples a. What has worked well in terms of collaboration between BR and BA? b. What have been the main challenges of collaboration between BR and BA? 14. What recommendations do you have for improving collaboration between BA and BR? Closing: Thank you so much for taking the time to talk with me today. Your insights and contributions will be very valuable to the evaluation and will help inform future SBC programming. If you have any additional questions or comments, please feel free to contact me. MID-TERM PERFORMANCE EVALUATION OF BA: LOCAL LESSONS, GLOBAL LEARNING 74 ANNEX 3: SOURCES OF INFORMATION KEY INFORMANTS FOR BREAKTHROUGH ACTION AND BREAKTHROUGH RESEARCH MID-TERM EVALUATION Name Organization and Title US-BASED 1. Ellen Starbird USAID 2. Karla Fossand USAID 3. Angie Brasington USAID 4. Lindsay Swisher USAID 5. Kate Howell USAID 6. Avery Avrakotos USAID 7. Jacqueline Devine USAID 8. Kama Garrison USAID 9. Afeefa Abdur-Rahman USAID 10. Joan Kraft USAID 11. Don Dickerson USAID 12. Anton Schneider USAID 13. Alex Todd USAID 14. Laura Itzkowitz USAID 15. Juanita Rodriguez ThinkPlace 16. Brooke Cretz ThinkPlace 17. Janine Kuehlick Think Place 18. Sarah Burgess Camber Collective 19. Jessica Vandermark Camber Collective 20. Stephanie Levy Ideas42 21. Jana Smith Ideas42 22. Elizabeth Serlemitsos BA/CCP 23. Jane Brown BA/CCP 24. Dominick Shattuck BA/CCP 25. Mwikali Kioko BA/CCP 26. Marcela Aguilar BA/CCP 27. Jarret Cassaniti BA/CCP 28. Marta Levitt IHP 29. Beth Schlacter FP2020 30. Anna Helland BA/CCP 31. Lynn Van Lith BA/CCP 32. Joanna Skinner BA MID-TERM PERFORMANCE EVALUATION OF BA: LOCAL LESSONS, GLOBAL LEARNING 75 Name Organization and Title 33. Claudia Vondrasek BA 34. Gabrielle Hunter BA 35. Kirsten Bose BA 36. Telesphore Kabore Save the Children 37. Antje Becker-Benton Save the Children 38. Renuka Bery Save the Children 39. Laura Reichenbach BR 40. Paul Hewett BR 41. Adaku Ejiogu BR 42. Leanne Dougherty BR 43. Martha Silva BR 44. Sanyukta Mathur BR 45. Rachel Yavinsky BR 46. Paul Hutchinson Tulane University 47. Dominique Meekers Tulane University 48. Lori Bollinger Avenir 49. Nicole Bellows Avenir 50. Michelle Hindin Population Council 51. Kara Tureski FHI360 52. Sohail Agha Bill and Melinda Gates Foundation 53. Djenebou Diallo Ouagadougou Partnership 54. Rebecka Lundgren University of California/San Diego (UCSD) WABA (other WABA-related KIs in Niger or Côte D’Ivoire list) 55. Mohamed Sangare BA, SBC Advisor for Francophone West Africa 56. Denise Adou BA, Côte d’Ivoire Program Officer 57. Tenin Traore Communication Manager, NGO – Mission des Jeunes pour l'Education, la Santé, la Solidarité et l'Inclusion (MESSI), Côte d’Ivoire 58. Rebecca Ezouatchi Research Consultant, Côte d’Ivoire 59. Anoh Georges Amplify-FP, Côte d’Ivoire 60. Binta Soumana Communication Manager, Niger NIGERIA 61. Foyeke Adebagbo USAID 62. Emma Mtiro USAID 63. Debby Nongo USAID 64. Ian Tweedie Chief of Party, BA 65. Shittu Abdu-Aguye Deputy Project Director, BA MID-TERM PERFORMANCE EVALUATION OF BA: LOCAL LESSONS, GLOBAL LEARNING 76 Name Organization and Title 66. Bolatito Aiyenigba Deputy Project Director (Malaria and TB), BA 67. Usman Sabo Capacity and Coordination advisor, BA 68. Olayinka Umar-Farouk Senior Technical Advisor RMNCH+N, BA 69. Justin De Normandie BA 70. Mathew Okoh Director of Research, Monitoring & Quality Assurance, BA 71. Bolade Oladejo Documentation & Knowledge Management Officer, BA 72. Dr. Dele Abegunde Country Director, BR 73. Emily White Johansson Tulane/BR 74. Udo Anaba Research Coordinator, BR 75. Chukwu Okoronkwo Head, ACSM Branch NMEP 76. Dr. Nnena Ogbulafor Head, Case Management Branch NMEP 77. Dr. Kayode Afolabi Director/Head Reproductive Health Division, FHD – FMOH 78. Dr. Chris Elemuwa HOD Health Services Department of Community Health 79. Dr Yahaya Disu Head Risk Communication Pillar (COVID-19 response) 80. Ladi Bako-Aiyegbusi Director/Head Health Promotion Division FHD-FMOH 81. Cecilia Abimaje TB Network 82. Dr. Paul Charles Waibale Country Director, PMI for States 83. Dr. Mariya Saleh Technical Director RMNCH, GHSC/PSM 84. Dr. Ifeanyi Okekearu Chief of Party, SHOPS+ 85. Dr. Chijioke Kaduru Program Manager, Corona MS 86. Paulina Akanet Quality Improvement Specialist, SHOPS+ 87. Uzoma Uwafor Communication Specialist, IHVN 88. Dr. Lawanson National Coordinator, National Tuberculosis and Leprosy Control Program (NTBLCP) 89. Dr. Salma Anas-Kolo Director/Head Family Health Department FMOH 90. Dr. Adebimpe Adebiyi Director/Head Hospital Services Department FMOH 91. Dr. Usman Gebi Deputy Chief of Party, IHP 92. Dr. Mike Egboh Chief of Party, GHSC/PSM 93. Itohow Uko Head, ACSM NTBLCP 94. Kenny Otto Deputy Chief of Party, GHSC/PSM NIGER 95. Safiatou Abdoulwabi Louis USAID/Niger Activity Manager/Development program specialist (Health) MID-TERM PERFORMANCE EVALUATION OF BA: LOCAL LESSONS, GLOBAL LEARNING 77 Name Organization and Title 96. Christina Chappell USAID/Niger Health Team Leader 97. Zilahatou Bahari-Tohon USAID/Niger Activity Manager BA / PMI 98. Eric Coulibaly USAID/Niger PMI Resident Advisor 99. Chaibou Dadi CAB/ONG/OSC Manager CESAF (Répondant national BR) 100. Lansena Zeinabou MSP Niger Regional Director & District Medical Officer 101. Dr. Asma Gali MSP Niger Amplify PF Country Coordinator. 102. Dr. Assoumane Guero Issoufou Program Office adaptation, Amplify 103. Dr. Basso Omar Program Officer. Chargé des pratiques à haut impact. Amplify 104. Cheik Bachir CAB/ONG/OSC Religious Leaders Association 105. Badara Seye CAB/ONG/OSC Save the Children (Niger COP) 106. M. Mato Saidou Communicateur district sanitaire commune N°1. 107. Mme Binta Tchiambiano Communicatrice DPF 108. Dr. Harou Issoufou WABA Director (DPF) 109. Mme Amadou Halima Hamidou Communicatrice district sanitaire commune N°3 110. Mme Riba Aïssa Communicatrice district sanitaire commune N°5 111. Mme Karimou Mariama Mamadou NMCP IEC unit. PNLP (NMCP)/MOH 112. Ntji Coulibaly Sr. SBC Advisor. Niamey. Save the Children 113. Rabi Hassane Bako M&E Officer. Niamey. Save the Children 114. Martha Populin CAB/ONG/OSC Girma CRS/Strategic Learning Lead 115. Nouri Dini Issoufou CAB/ONG/OSC Girma CRS/Strategic Learning Lead 116. Koko Daniel CAB/ONG/OSC COP Impact Malaria project 117. Abdoulaye Seydou Souleymane WABA Consultant 118. Dr. Mahamidou Illo MSP Niger Sahel RISE II MOH 119. Dr. Dare Rabiou DRSP Maradi. MSP Niger 120. Saadou Wonkoye MSP Niger Communicateur Guidan Roumdji, Maradi 121. Maire Guidan Roumdji Maire de Guidan Roumdji, Maradi 122. Mme Lantana Oumarou SG Préfecture Guidan Roumdji, Maradi 123. Oudou Souley DRSP Maradi. PF mobilisation Communautaire, Maradi 124. Chafatou Issa DRSP Maradi. Communicatrice, Maradi 125. Tsayabou M. Laouali DRSP Maradi Communicateur 126. Abdoulkader Boubacar Save the Children – Maradi 127. Mariam Dodo Boukari CARE – Hamzari – Maradi 128. Dr. Abdou Daoura District Medical Officer, Matameye. DRSP Zinder MID-TERM PERFORMANCE EVALUATION OF BA: LOCAL LESSONS, GLOBAL LEARNING 78 Name Organization and Title 129. Moussa Abdou Communicateur District sanitaire de Matamèye. DRSP Zinder 130. Boukari Idi Communicateur DRSP Zinder. 131. Amirou Elh Habou CRS – GIRMA. Resp Cycle actions communautaires, Zinder 132. Aboubacar Sale Save The Children Zinder. WADATA 133. Aboubacar Tahirou MSP Niger Communicateur District sanitaire de Mirriah, Zinder 134. Chiwa Ousmane Field Coordinator, BA, Zinder CÔTE D’IVOIRE 135. Olivier Tondoh N’Guetta USAID 136. Blaise Kouadio USAID 137. Willibrord Shasha USAID 138. Mme Thiam USAID 139. Diarra Kamara BA, Chief of Party 140. Antoine Kouame BA, Malaria Technical Advisor 141. Dr. Bombo Pacome BA, HIV Technical Advisor 142. William Benie BA, GHSA/COVID Technical Advisor 143. Abdoul Dosso BA, M&E Manager 144. Dr. Kallo Dep. Director, Veterinary Health Directorate, Ministry of Animal and Fisheries Resources 145. Mme Nebre Jocelyn Chief of Communication Unit, National Institute of Public Hygeine 146. Pr. Guessend Natalie Secretariat GTT RAM, Technical Working Group Antimicrobial Resistance (TWG-AMR) 147. Silue Alia Chief of Communication Unit Programme National de Lutte contre le SIDA (PNLS)/ Ministry of Health and Hygiene 148. Dr. Kouame Blaise Chief of Testing Unit PNLS/ Ministry of Health and Hygiene 149. Dr. Memain Helene Chief of Care and Support Unit PNLS/ Ministry of Health and Hygiene 150. Dr. Tanoh Mea Antoine Director Coordinator PNLP / Ministry of Health and Hygiene 151. Dr. Bleu Bomin Monne Thérèse Chief of Communication Unit PNLP / Ministry of Health and Hygiene 152. Dr. Kokrasset Yah Colette Deputy Director PNLP / Ministry of Health and Hygiene MID-TERM PERFORMANCE EVALUATION OF BA: LOCAL LESSONS, GLOBAL LEARNING 79 DOCUMENTS REVIEWED Abdu-Aguye, S.; Johns Hopkins Center for Communication Programs, Religious Leaders Addressing Social and Gender Norms in Northern Nigeria, 10.05-10.08.20 Aiyenigba, B.; Haqqi, F.; Acosta, A.; Tchofa, J.; Tweedie, I.; Using a Behavioral Economics approach to improve fever case management in Nigeria Breakthrough ACTION – Nigeria, Learnings on Religious Leaders Intervention on MNCH + N, 2019 Breakthrough ACTION and RESEARCH, ITN Access and Use Report, Web Page, 2021 Breakthrough ACTION and RESEARCH, Malaria, You Tube Playlist, 2021 Breakthrough ACTION and RESEARCH, Provider Behavior, Ecosystem Map, Web Page, 2018 Breakthrough ACTION and RESEARCH, Social and Behavior Change Indicator Bank for Family Planning and Service Delivery, Web Page, 2020 Breakthrough ACTION Côte d'Ivoire, Annual Activity Report FY18, 03.18 Breakthrough ACTION Côte d'Ivoire, Annual Activity Report FY18Q4, 10.18 Breakthrough ACTION Côte d'Ivoire, Annual Activity Report FY19Q2, 04.15.19 Breakthrough ACTION Côte d'Ivoire, Breakthrough ACTION PEPFAR - Rapport Trimestriel, October 1-December 31, 2019, 01.15.20 Breakthrough ACTION Côte d'Ivoire, FY18 Activity Report October 1-December 31, 2018 Breakthrough ACTION Côte d'Ivoire, FY18Q2 GHSA Phase I Semi-Annual Report 04.15.18 Breakthrough ACTION Côte d'Ivoire, FY18Q4 GHSA Phase I Semi-Annual Report 09.28.18 Breakthrough ACTION Côte d'Ivoire, FY19 GHSA Workplan, 09.30.18 Breakthrough ACTION Côte d'Ivoire, FY19Q2 GHSA Phase I Semi-Annual Report 04.23.19 Breakthrough ACTION Côte d'Ivoire, FY19Q4 GHSA Phase I Semi-Annual Report 09.23.19 Breakthrough ACTION Côte d'Ivoire, FY20 GHSA Workplan, 10.10.19 Breakthrough ACTION Côte d'Ivoire, PMI, Quarterly Report, October 1-December 31, 2019, 01.15.20 Breakthrough ACTION Côte d'Ivoire, MER 2.2 Q4, 09.15.18 Breakthrough ACTION Côte d'Ivoire, MER 2.3 FY19 Q4, 04.14.19 Breakthrough ACTION Côte d'Ivoire, Partner Financial Status as of 30 June 2018, 07.06.18 Breakthrough ACTION Côte d'Ivoire, Partner Financial Status as of 30 September 2018, 10.16.18 Breakthrough ACTION Côte d'Ivoire, Partner Financial Status as of 31 December 2018, 1.30.19 Breakthrough ACTION Côte d'Ivoire, Partner Financial Status as of 31 March 2018, 04.15.18 Breakthrough ACTION Côte d'Ivoire, PEPFAR - Rapport Trimestriel: October 1-December 31, 2019, 01.15.20 Breakthrough ACTION Côte d'Ivoire, PEPFAR FY18 January 1-March 31, 2018 Breakthrough ACTION Côte d'Ivoire, PEPFAR MER 2.2, 03.18 Breakthrough ACTION Côte d'Ivoire, PEPFAR MER 2.2, 12.17 Breakthrough ACTION Côte d'Ivoire, Q1 FY20 Quarterly Financial Management Report, 01.20 MID-TERM PERFORMANCE EVALUATION OF BA: LOCAL LESSONS, GLOBAL LEARNING 80 Breakthrough ACTION Côte d'Ivoire, Q2 FY19 Quarterly Financial Management Report, 04.19 Breakthrough ACTION Côte d'Ivoire, Q3 FY19 Quarterly Financial Management Report, 07.19 Breakthrough ACTION Côte d'Ivoire, Q4 FY19 Quarterly Financial Management Report, 10.19 Breakthrough ACTION Côte d'Ivoire, Travel Plan PY 2 GHSA Breakthrough ACTION Côte d'Ivoire, Work Plan Year 1, 01.04.18 Breakthrough ACTION Niger, Q1 FY20 Quarterly Financial Management Report, 01.20 Breakthrough ACTION Niger, Q2 FY19 Quarterly Financial Management Report, 04.19 Breakthrough ACTION Niger, Q2 FY19 Quarterly Financial Management Report, 04.19 Breakthrough ACTION Niger, Q2 FY20 Quarterly Financial Management Report, 04.20 Breakthrough ACTION Niger, Q3 FY19 Quarterly Financial Management Report, 07.19 Breakthrough ACTION Niger, Q3 FY19 Quarterly Financial Management Report, 07.19 Breakthrough ACTION Niger, Q3 FY20 Quarterly Financial Management Report, 07.20 Breakthrough ACTION Niger, Q4 FY19 Quarterly Financial Management Report, 10.19 Breakthrough ACTION Niger, Q4 FY19 Quarterly Financial Management Report, 10.19 Breakthrough ACTION Niger, Rapport de l'atelier Net-Map Breakthrough ACTION Nigeria, Annex A-BA Nigeria F-USAID IP Work, 02.27.18 Breakthrough ACTION Nigeria, Annex A-BA-N FY2020 Detailed Implementation Plan Breakthrough ACTION Nigeria, Annex B: BA-N Y2 Budget_USAID Draft3 09.18.18 Breakthrough ACTION Nigeria, Annex B: Detailed Implementation Plan, 08.24.18 Breakthrough ACTION Nigeria, Annex C: Annual Sustainability Action Plan, 08.24.18 Breakthrough ACTION Nigeria, Annex D: Breakthrough ACTION Nigeria Results Framework and Behavioral Objectives, 07.27.18 Breakthrough ACTION Nigeria, Annex D-Theory, Innovation and Illustrative Intermediate Outcomes by Health Area Breakthrough ACTION Nigeria, Annex F-BA Nigeria Year 1 Short Term Technical Assistance Plan (March-September 2018) Breakthrough ACTION Nigeria, Assessment of Digital and Mobile Technology in Nigeria, 09.18 Breakthrough ACTION Nigeria, Compound Meeting Dialogue Framework, South Yoruba Breakthrough ACTION Nigeria, Family Planning Benefits Handbill Breakthrough ACTION Nigeria, Family Planning Methods Flip Chart Breakthrough ACTION Nigeria, Family Planning Methods Handbill Breakthrough ACTION Nigeria, Family Planning Posters and Jacket Breakthrough ACTION Nigeria, Funding Table - Projections Breakthrough ACTION Nigeria, Human-Centered Design for Tuberculosis, Concept Note: Define Phase, 06.05.18 MID-TERM PERFORMANCE EVALUATION OF BA: LOCAL LESSONS, GLOBAL LEARNING 81 Breakthrough ACTION Nigeria, Increasing Case Detection of Tuberculosis in Nigeria, Summary of key findings from the SBC Flowchart activities undertaken to date, 03.02.20 Breakthrough ACTION Nigeria, Integrated Community Dialogue, Lifestage 1-2, English, 08.21.19 Breakthrough ACTION Nigeria, Integrated Community Dialogue, Lifestage 1-2, Hausa, 08.21.19 Breakthrough ACTION Nigeria, Integrated Community Dialogue, Lifestage 3, Hausa, 08.21.19 Breakthrough ACTION Nigeria, Key Hypothesis Identified as Affecting Provider Behavior in Malaria Case Management Breakthrough ACTION Nigeria, Key Messages Booklet Breakthrough ACTION Nigeria, LLIN Distribution Point Banner, Kebbi Breakthrough ACTION Nigeria, LLIN Distribution Point Banner, Kogi Breakthrough ACTION Nigeria, LLIN Distribution Point Banner, Nasarawa, English Breakthrough ACTION Nigeria, LLIN Distribution Point Banner, Nasarawa, Hausa Breakthrough ACTION Nigeria, LLIN Distribution Point Banner, Sokoto Breakthrough ACTION Nigeria, Malaria Leaflet Breakthrough ACTION Nigeria, MNCH+N Design Report: How might we radically improve MNCH+N in northern Nigeria? 01.18.19 Breakthrough ACTION Nigeria, Monthly Progress Update, 03.20 Breakthrough ACTION Nigeria, Monthly Progress Update, 04.20 Breakthrough ACTION Nigeria, Monthly Progress Update, 05.20 Breakthrough ACTION Nigeria, Monthly Progress Update, 06.20 Breakthrough ACTION Nigeria, Pipeline Decemer 31, 2017 Breakthrough ACTION Nigeria, Pipeline Q1 FY19, 01.31.19 Breakthrough ACTION Nigeria, Pipeline Q1 FY19, 01.31.19 Breakthrough ACTION Nigeria, Pipeline Q2 FY18, 03.30.18 Breakthrough ACTION Nigeria, Pipeline Q2 FY19, 04.26.19 Breakthrough ACTION Nigeria, Pipeline Q3 FY19, 07.23.19 Breakthrough ACTION Nigeria, Pipeline Q4 FY19, 10.14.19 Breakthrough ACTION Nigeria, Pipeline through Aug 2018, 09.12.18 Breakthrough ACTION Nigeria, Pipeline through June 30 2018, 08.14.18 Breakthrough ACTION Nigeria, Pipeline through November 30 2019, 01.31.20 Breakthrough ACTION Nigeria, Pipeline through November 30 2019, 02.04.20 Breakthrough ACTION Nigeria, Project Brief One Pager Breakthrough ACTION Nigeria, Provider Behavior Change Activity Formative Assessment Update Report, 11.18 Breakthrough ACTION Nigeria, Provider Behavior Change in Feber Case Management, USAID/PMI Outbrief 02.01.19 MID-TERM PERFORMANCE EVALUATION OF BA: LOCAL LESSONS, GLOBAL LEARNING 82 Breakthrough ACTION Nigeria, Report of 'Women's Group Study Tour & Facilitators' Training' Breakthrough ACTION Nigeria, Response to USAD's Recommendations from September/October 2019 Field Visit to Kebbi, Sokoto and Bauchi, 01.14.20 Breakthrough ACTION Nigeria, TB Pilot Campaign Brothers Keeper Poster, English, 03.08.19 Breakthrough ACTION Nigeria, TB Pilot Campaign Brothers Keeper Poster, Hausa, 03.08.19 Breakthrough ACTION Nigeria, TB Pilot Campaign Brothers Keeper Poster, Pidgin, 03.08.19 Breakthrough ACTION Nigeria, TB Pilot Campaign Brothers Keeper Sticker, All Languages, 03.08.19 Breakthrough ACTION Nigeria, TB Umbrella Campaign Development After Action Review: Synthesis of key reflection points from the After Action Review meeting with the National Tuberculosis and Leprosy Control Programme, 03.15.20 Breakthrough ACTION Nigeria, TB, HCD, PPMV Posters Breakthrough ACTION Nigeria, TB, HCD, Stories from the Field Breakthrough ACTION Nigeria, Total Estimated Cost by Year: Breakthrough ACTION-Nigeria, With Comments, 06.28.19 Breakthrough ACTION Nigeria, USAID's Tri-Projects Brief: Integrated Health, 2018 Breakthrough ACTION Nigeria, Work Plan, Revised, 03.17.20 Breakthrough ACTION Nigeria, Year One Work Plan, 02.27.18, 03.16.18 Breakthrough ACTION PMI/Niger, COVID-19 Adaptations, Memo, 03.31.20, Rev 05.18.20 Breakthrough ACTION Sahel, Q1 FY20 Quarterly Financial Management Report, 01.20 Breakthrough ACTION Sahel, Q2 FY19 Quarterly Financial Management Report, 04.19 Breakthrough ACTION Sahel, Q2 FY20 Quarterly Financial Management Report, 04.20 Breakthrough ACTION Sahel, Q3 FY19 Quarterly Financial Management Report, 07.19 Breakthrough ACTION Sahel, Q4 FY19 Quarterly Financial Management Report, 10.19 Breakthrough ACTION WABA, Q1 FY20 Quarterly Financial Management Report, 01.20 Breakthrough ACTION WABA, Q2 FY19 Quarterly Financial Management Report, 04.19 Breakthrough ACTION WABA, Q3 FY19 Quarterly Financial Management Report, 07.19 Breakthrough ACTION WABA, Q4 FY19 Quarterly Financial Management Report, 10.19 Breakthrough ACTION, A Blueprint for Applying Behavioral Insights to Malaria Service Delivery, Methods and Frameworks for Improving Provider Behavior Breakthrough ACTION, A Shared Agenda for Social and Behavior Change in Family Planning in the Ouagadougou Partnership Region: A Summary Breakthrough ACTION, A Taxonomy for Social Norms that Influence Family Planning in Ouagadougou Partnership Countries, Presentation, 09.20 Breakthrough ACTION, Accelerating Progress in Family Planning, The Impact of Social and Behavior Change, Animation Breakthrough ACTION, Accelerer les Progres en Planification Familiale, L'impact du changement social et de comportement, Animation MID-TERM PERFORMANCE EVALUATION OF BA: LOCAL LESSONS, GLOBAL LEARNING 83 Breakthrough ACTION, Advocating for Social and Behavior Change in Family Planning Programs, A Message Framework, 05.2020 Breakthrough ACTION, Annual Report: October 1, 2018-September 30,2019, 11.20.19 Breakthrough ACTION, Annual Report: October 1, 2018-September 30,2019, Responses to USAID, 11.20.19 Breakthrough ACTION, Behavior Change Impact, Evidence in Action, Social and Behavior Change for Family Planning Evidence Database, An Overview Breakthrough ACTION, Behavior Change Impact, Evidence in Action, Welcome to Behavior Change Impact, Evidence for Social and Behavior Change Impact in Family Planning, HIV, Malaria, Reproductive Empowerment, and Urban Youth, Web Page, 2016 Breakthrough ACTION, Breakthrough ACTION Project Director with HC4 Activity Management Team at USAID, Meeting Notes, 08.16.19 Breakthrough ACTION, Confiance Totale Dissemination tracking, WABA, May-Sept 2020 Breakthrough ACTION, Core Work Plan Modification to Zika Activities Year 1, 10.30.17 Breakthrough ACTION, Core Work Plan Modification to Zika Activities Year 1, Zika Comments, 10.30.17 Breakthrough ACTION, Core Work Plan Modification to Zika Activities Year 1-2, 05.18.18, Rev. 06.01.18 Breakthrough ACTION, Côte d'Ivoire FY18Q2 GHSA Phase I Semi-Annual Report, 04.15.18 Breakthrough ACTION, Côte d'Ivoire FY18Q2 GHSA, Phase I Semi-Annual Report 03.18 Breakthrough ACTION, Côte d'Ivoire FY18Q4 GHSA Phase I Semi-Annual Report, 10.01.18 Breakthrough ACTION, Côte d'Ivoire FY18Q4 GHSA Phase I Semi-Annual Report, 28.09.18 Breakthrough ACTION, Côte d'Ivoire FY19Q2 GHSA, Phase I Semi-Annual Report 04.19 Breakthrough ACTION, Côte d'Ivoire FY19Q4 GHSA Phase I Semi-Annual Report, 09.23.19 Breakthrough ACTION, Côte d'Ivoire MBS Brief, What do the Data Mean? Breakthrough ACTION, Country Operational Plan 2018 Minimum Required Elements and Optional Elements, 11.05.18 Breakthrough ACTION, Encouraging Counseling that Promotes Meaningful Choice, Behavioral Design for Provider Behavior Change in Family Planning Services in Malawi, 03.20 Breakthrough ACTION, Evidence in Action, Behavior Change Impact, About the Malaria Database, Web Page, 2016 Breakthrough ACTION, Factors Associated with Care-Seeking for Fever for Children under Five Years of Age in Côte d'Ivoire, Poster, 2019 Breakthrough ACTION, Getting Practical, Integrating Social Norms into Social and Behavior Change Programs Breakthrough ACTION, Guidance on Social and Behavior Change for Nutrition During COVID-19, 07.20.20 Breakthrough ACTION, Illustrative timeline for implementing the Malaria Behavior Study Breakthrough ACTION, Improving Provider Behaviors Related to Malaria: A Blueprint, Presentation, 06.24.20 MID-TERM PERFORMANCE EVALUATION OF BA: LOCAL LESSONS, GLOBAL LEARNING 84 Breakthrough ACTION, Improving Provider Behaviors Related to Malaria: A Blueprint, YouTube video, 06.24.20 Breakthrough ACTION, Indicator Summary Table, 11.20.19 Breakthrough ACTION, Individual, household and community factors associated with the uptake of three doses of intermittent preventive treatment of malaria in pregnancy (IPTp3) in Côte d'Ivoire: A multilevel analysis, Poster, 2019 Breakthrough ACTION, It Takes A Village, A Shared Agenda for Social and Behavior Change in Family Planning in the Ouagadougou Partnership Region, 08.19 Breakthrough ACTION, It Takes A Village, A Shared Agenda for Social and Behavior Change in Family Planning in the Ouagadougou Partnership Region, 11.19 Breakthrough ACTION, It Takes A Village, A Shared Agenda for Social and Behavior Change in Family Planning, Summary Breakthrough ACTION, Leveraging Connections Between Client and Provider Behavior, Behavioral Design for Provider Behavior Change in Care-Seeking for Children in Zambia Breakthrough ACTION, Literature review on barriers and enablers of FP service utilization, Niger, Togo, Côte d'Ivoire, and Burkina Faso, 05.10.19 Breakthrough ACTION, Malaria Behavior Survey, Web Page, 2021 Breakthrough ACTION, Management and Performance Review SOW Draft, 10.19 Breakthrough ACTION, Management and Performance Review SOW Final, 10.19 Breakthrough ACTION, Management and Performance Review SOW, 07.30.18 Breakthrough ACTION, Management and Performance Review, 11.18.19 Breakthrough ACTION, Management and Performance Review, Written Responses, 09.04.18 Breakthrough ACTION, Management and Performance Review, Written Responses, Final, 09.04.18 Breakthrough ACTION, Management Review SOW, 04.19.18 Breakthrough ACTION, Managing Nutrition Myths and Misconceptions During COVID-19, 07.24.20 Breakthrough ACTION, Monthly Update, 03.02.20 Breakthrough ACTION, MPR, Annex 1, Staffing Matrix 09.03.18 Breakthrough ACTION, MPR, Annex 2, Financial Report 09.03.18 Breakthrough ACTION, Normes sociales et recours a la planification familiale dans les pays du Partenariat de Ouagadougou, Presentation, 12.02.19 Breakthrough ACTION, OP Influence Landscape & Strategy, Presentation, 01.21.19 Breakthrough ACTION, PMI Core Deliverables List Breakthrough ACTION, Project Strategy, Memo, 04.05.19 Breakthrough ACTION, Q1 FY19 Quarterly Financial Management Report, 02.20.19 Breakthrough ACTION, Q1 FY19 Supplemental SF 425 Information Request from USAID's Partners Breakthrough ACTION, Q1 FY20 Quarterly Financial Management Report, 01.20 Breakthrough ACTION, Q1 Quarterly Financial Management Report, Final, 03.14.18 MID-TERM PERFORMANCE EVALUATION OF BA: LOCAL LESSONS, GLOBAL LEARNING 85 Breakthrough ACTION, Q2 FY19 Quarterly Financial Management Report, 04.19 Breakthrough ACTION, Q2 Quarterly Report Financial Management Final, 04.18 Breakthrough ACTION, Q3 FY19 Quarterly Financial Management Report, 07.19 Breakthrough ACTION, Q3 Quarterly Financial Management Report Final, 07.18 Breakthrough ACTION, Q4 FY19 Quarterly Financial Management Report, 10.19 Breakthrough ACTION, Q4 Quarterly Financial Management Report, 10.18 Breakthrough ACTION, Qualitative Research to Complement the Malaria Behavior Survey, 06.24.20 Breakthrough ACTION, Quarterly Report: April 1-June 30, 2018, 07.30.18 Breakthrough ACTION, Quarterly Report: April 1-June 30, 2018, Final, 07.30.18 Breakthrough ACTION, Quarterly Report: April 1-June 30, 2018, Responses to USAID Comments, 07.30.18 Breakthrough ACTION, Quarterly Report: April 1-June 30, 2018, USAID Comments, 07.30.18 Breakthrough ACTION, Quarterly Report: January 1-March 31, 2018, 04.30.18, Rev. 05.25.18 Breakthrough ACTION, Quarterly Report: July 1-September 30, 2018, 10.30.18, Rev 12.18.18 Breakthrough ACTION, Quarterly Report: July 1-September 30, 2018, Responses to USAID, 10.30.18, Rev 12.18.18 Breakthrough ACTION, Quarterly Report: July 21, 2017-December 31, 2017, 01.30.18 Breakthrough ACTION, Quarterly Report: July 21, 2017-December 31, 2017, Responses to USAID Feedback, 01.30.18 Breakthrough ACTION, Rapport de la Reunion de Restitution du Sommet Francophone de Changement Social et Comportemental (CSC) Tenu a Abidjan, August 25 - February 27, 2019, 03.29.19 Breakthrough ACTION, Responses to USAID's Comments on the Indicator Summary Table Breakthrough ACTION, Sahel RISE II Work Plan- PY2 Activities Breakthrough ACTION, SBC Checklist for Mock TRP, 02.07.20 Breakthrough ACTION, Semiannual Report, 04.19 Breakthrough ACTION, Social and Behavior Change for Family Planning: Global Influence Landscape and Strategy, 08.18 Breakthrough ACTION, Synthese des champs d'influence et strategies ressortantes - retion PO, 01.19 Breakthrough ACTION, The SBC Flow Chart Introduction, Presentation, 10.20 Breakthrough ACTION, The SBC Flow Chart: Breakthrough ACTION's Method Convergence Process Model, 08.10.18 Breakthrough ACTION, Trip Report: Breakthrough Action Guyana Technical Meeting 07.18 Breakthrough ACTION, Using Social and Behavior Change to Increase Family Planning Uptake in Zambia: Making the Case, 10.16.20 Breakthrough ACTION, Utilisation du Changement Social et Comportemental pour Ameliorer le Bilan du Planning Familial en Afrique de l'Ouest Breakthrough ACTION, WABA Community Even Referral Tracking Vouchers MID-TERM PERFORMANCE EVALUATION OF BA: LOCAL LESSONS, GLOBAL LEARNING 86 Breakthrough ACTION, WABA Literature Review, Enablers and barriers of FP use in Togo, Burkina Faso, Niger, and Côte d'Ivoire, 07.19 Breakthrough ACTION, WABA Performance Indicator Table Q3, 08.26.20 Breakthrough ACTION, WABA Theory of Change Breakthrough ACTION, West Africa Breakthrough ACTION (WABA): Qualitative research on perceptions of Family Planning service quality, Presentation, 07.08.19 Breakthrough ACTION, What Can Descriptive Norms Tell Us About Care-seeking for Children with Fever in Africa?: A Multi-Country Study, 2019 Breakthrough ACTION, When Knowing the Protocol is Not Enough, Behavioral Design for Provider Change in Malaria Diagnosis and Treatment in Nigeria Breakthrough ACTION, Year 1 Management and Performance Review, Executive Summary, 12.18 Breakthrough ACTION, Year 2 Management and Performance Review, Executive Summary, Final, 12.19 Breakthrough ACTON, Guidance on Social and Behavior Change for Family Planning During COVID-19, 04.14.20 Breakthrough ACTION Niger, COVID-19 Delays, Memo, 04.13.20 CCP Nigeria Breakthrouth ACTION Office, Field Operations Manual, 1.26.20 Chizoba, O,; Leslie, L.; Nasir, F.; Abdu-Aguye, S.; Shifting Gender Norms through Women Empowerment Group for Improved Health-Seeking Behaviors in Northern Nigeria, 2020 Edor, J.; Edoziem, B.; Aiyenigba, B.; Shofowora, O.; Nongo, D.; Using Social Media as a Participatory Tool for Tuberculosis Awareness: A Nigerian Case Study, 2020 Feedback on MBS Tables, 04.14.20 Hempstone, H.; Fouladi, Z.; Brasington, A.; Marcus, R.; Ahanda, K.; Carrasco, M.; Garrison, K.; Levy, S.; Tompsett, A.; USAID, Breakthrough: Social and Behavior Change (SBC) Project, Concept Presentation, 04.16 Hendrickson, Z.; Johns Hopkins School of Public Health, IRB Research Plan for New Data Collection, Rapid assessments of exposure to West Africa Breakthrough ACTION project activities in Côte d'Ivoire, 07.15.20 Integrated SBC Evaluation-Practice Panel Johns Hopkins Center for Communication Programs, Achieving Gender Equality and Female Empowerment through Social and Behavior Change, Project Strategy, 03.14.18 Johns Hopkins Center for Communication Programs, Achieving Gender Equality and Female Empowerment through Social and Behavior Change, Project Strategy, USAID Feedback, 03.14.18 Johns Hopkins Center for Communication Programs, Addressing Behavioral Evidence Gaps to Improve Maternal, Newborn, Child Health, and Nutrition Social and Behavior Change Programs in Nigeria, Formative Research Concept Note, 08.09.18 Johns Hopkins Center for Communication Programs, Atelier sur le Leadership et Approches Innovatrices en Communication Strategique Pour la Sante Publique, June 10-21, 2019, 08.10.19 Johns Hopkins Center for Communication Programs, Breakthrough ACTION - Nigeria Annual Report October 1, 2018 to September 30, 2019, 11.15.19 MID-TERM PERFORMANCE EVALUATION OF BA: LOCAL LESSONS, GLOBAL LEARNING 87 Johns Hopkins Center for Communication Programs, Breakthrough ACTION - Nigeria Quarterly Report April 1 to June 30, 2018, 07.31.18 Johns Hopkins Center for Communication Programs, Breakthrough ACTION - Nigeria Quarterly Report July 1 to September 30, 2019, 01.07.20 Johns Hopkins Center for Communication Programs, Breakthrough ACTION - Nigeria TB Quarterly Report April 1 to June 30, 2019, 07.31.19 Johns Hopkins Center for Communication Programs, Breakthrough ACTION - Sahel - Niger and Burkina Faso, Budget October 1-December 31, 2018 Johns Hopkins Center for Communication Programs, Breakthrough ACTION Côte d'Ivoire US President's Malaria Initiative, Work Plan FY19, 11.01.19, Rev. 11.29.19 Johns Hopkins Center for Communication Programs, Breakthrough ACTION Côte d'Ivoire, FY18 Activity Report, July 1-September 30, 2018, 10.15.18 Johns Hopkins Center for Communication Programs, Breakthrough ACTION Côte d'Ivoire, FY19 Activity Report, January 1-March 31, 2019, 04.04.19 Johns Hopkins Center for Communication Programs, Breakthrough ACTION Côte d'Ivoire, US President's Malaria Initiative, Work Plan FY18, 03.18.18 Johns Hopkins Center for Communication Programs, Breakthrough ACTION MER2.2 FY18Q4 09.15.18 Johns Hopkins Center for Communication Programs, Breakthrough ACTION Monitoring, Evaluation, and Learning Plan, 02.01.18 Johns Hopkins Center for Communication Programs, Breakthrough ACTION Nigeria Field Support, Budget Notes 09.18.18 Johns Hopkins Center for Communication Programs, Breakthrough ACTION Nigeria Field Support, Budget Notes FY2020, 10.05.19 Johns Hopkins Center for Communication Programs, Breakthrough ACTION- Nigeria Quarterly Report January 1 to March 31, 2018, 04.30.18 Johns Hopkins Center for Communication Programs, Breakthrough ACTION Nigeria Quarterly Report October 1 to December 31, 2019, 01.31.20 Johns Hopkins Center for Communication Programs, Breakthrough ACTION Nigeria Quarterly Report￾Revised, January 1 to March 31, 2019, 04.30.20 Johns Hopkins Center for Communication Programs, Breakthrough ACTION Nigeria, Annual Work Plan, 08.24.18 Johns Hopkins Center for Communication Programs, Breakthrough ACTION Nigeria, Annual Work Plan, 10.14.19 Johns Hopkins Center for Communication Programs, Breakthrough ACTION Nigeria, Annual Work Plan, With Comments, 06.28.18 Johns Hopkins Center for Communication Programs, Breakthrough ACTION Nigeria, Quarterly Report April 1 to June 30, 2018, 07.31.18 Johns Hopkins Center for Communication Programs, Breakthrough ACTION Nigeria, TB Quarterly Activity Report, Draft, January 1 to March 31, 2019 Johns Hopkins Center for Communication Programs, Breakthrough ACTION Nigeria, TB Quarterly Activity Report, Draft, January 1 to March 31, 2020, 04.30.20 MID-TERM PERFORMANCE EVALUATION OF BA: LOCAL LESSONS, GLOBAL LEARNING 88 Johns Hopkins Center for Communication Programs, Breakthrough ACTION Nigeria, TB Quarterly Report July 1 to September 30, 2019, 10.31.19 Johns Hopkins Center for Communication Programs, Breakthrough ACTION Nigeria, TB Quarterly Report July 1 to September 30, 2019, 10.31.19 Johns Hopkins Center for Communication Programs, Breakthrough ACTION Nigeria, TB Quarterly Report October 1 to December 31, 2019, 01.31.19 Johns Hopkins Center for Communication Programs, Breakthrough ACTION PMI Niger Quarterly Report: Q3/Y1, April-June 2019, 08.23.19 Johns Hopkins Center for Communication Programs, Breakthrough ACTION PMI Niger Semi-Annual Report, October 1, 2018-March 31, 2019, 04.25.19 Johns Hopkins Center for Communication Programs, Breakthrough ACTION PMI Niger, Quarterly Report March 1-September 30, 2018, 11.05.18 Johns Hopkins Center for Communication Programs, Breakthrough ACTION President's Malaria Initiative (PMI) Annual Report FY18, 10.15.18 Johns Hopkins Center for Communication Programs, Breakthrough ACTION Sahel RISE II Semi-Annual Report, October 1, 2018 - March 31, 2019, 04.30.19 Johns Hopkins Center for Communication Programs, Breakthrough ACTION Sahel RISE II Work Plan, 01.16.19 Johns Hopkins Center for Communication Programs, Breakthrough ACTION Sahel RISE II Work Plan, Revised, 01.16.19, Rev. 03.22.19 Johns Hopkins Center for Communication Programs, Breakthrough ACTION Sahel RISE II Y2Q1 Report, October 1 - December 31, 2019, 01.31.20 Johns Hopkins Center for Communication Programs, Breakthrough ACTION Sahel RISE II Year 2 Work Plan, 10.31.19, Rev. 11.27.19 Johns Hopkins Center for Communication Programs, Breakthrough ACTION Sahel RISE II, Quarterly Report Q3/Y1 April 1-June 30, 2019, 08.02.19 Johns Hopkins Center for Communication Programs, Breakthrough ACTION Sahel RISE II, Y2Q1 Report, October 1 - December 31, 2019, Revised, 01.31.19 Johns Hopkins Center for Communication Programs, Breakthrough ACTION Year 2 Core Work Plan, 08.24.18, Rev. 12.18.18 Johns Hopkins Center for Communication Programs, Breakthrough ACTION Year 2 Core Work Plan, Modification, 08.19 Johns Hopkins Center for Communication Programs, Breakthrough ACTION Year 2 GHSA Work Plan, 10.01.18 Johns Hopkins Center for Communication Programs, Breakthrough ACTION Year 3 Core Work Plan, 10.01.19, Rev. 11.20.19 Johns Hopkins Center for Communication Programs, Breakthrough ACTION Year 3 GHSA Work Plan, 10.10.19 Johns Hopkins Center for Communication Programs, Breakthrough ACTION Year 3 Work Plan, 06.28.19 Johns Hopkins Center for Communication Programs, Breakthrough ACTION, Advancing Male Engagement in Family Planning + Reproductive Health, An Advocacy Tool, 2018 MID-TERM PERFORMANCE EVALUATION OF BA: LOCAL LESSONS, GLOBAL LEARNING 89 Johns Hopkins Center for Communication Programs, Breakthrough ACTION-Nigeria, FY19 Annual Report - October 1, 2018 to September 30, 3019, 11.15.19 Johns Hopkins Center for Communication Programs, Concept Note on Social and Behavior Change Communication Considerations in Global Fund Funding Requests, 07.02.18 Johns Hopkins Center for Communication Programs, Essential Elements for Costed Implementation Plans, Social and Behavior Change Checklist,03.25.19 Johns Hopkins Center for Communication Programs, HC3 FP Nigeria Quarterly Report, October 1 - December 31, 2017 Johns Hopkins Center for Communication Programs, Malaria Behavior Survey, Analysis Plan, 09.19, Rev. 02.20 Johns Hopkins Center for Communication Programs, Malaria Behavior Survey, Guidelines for Design and Implementation, 10.03.18, Rev. 05.09.19 Johns Hopkins Center for Communication Programs, Male engagement in family planning programs in West Africa, Literature Review, 09.30.19 Johns Hopkins Center for Communication Programs, Maternal, Newborn, and Child Health plus Nutrition in Northern Nigeria, Literature Review, 08.18 Johns Hopkins Center for Communication Programs, MBS Standard Questionnaire Men, 09.19.19 Johns Hopkins Center for Communication Programs, MBS Standard Questionnaire Women, 09.19.19 Johns Hopkins Center for Communication Programs, MBS Household Standard Questionnaire, 09.19.19 Johns Hopkins Center for Communication Programs, MBS Tabulation Plan, Section 1, Study Population Description, 01.20 Johns Hopkins Center for Communication Programs, MBS Tabulation Plan, Section 2, General Malaria, 01.20 Johns Hopkins Center for Communication Programs, MBS Tabulation Plan, Section 3, Insecticide Treated Nets, 01.20 Johns Hopkins Center for Communication Programs, MBS Tabulation Plan, Section 4, Case Management, 01.20 Johns Hopkins Center for Communication Programs, MBS Tabulation Plan, Section 5, Malaria in Pregnancy, 01.20 Johns Hopkins Center for Communication Programs, MBS Tabulation Plan, Section 6, Indoor Residual Spraying, 01.20 Johns Hopkins Center for Communication Programs, MBS Tabulation Plan, Section 7, Seasonal Malaria Chemoprevention, 01.20 Johns Hopkins Center for Communication Programs, Niger Malaria Social and Behavior Change Network Analysis, Summary of Net-Map Findings, 09.25.19 Johns Hopkins Center for Communication Programs, Niger US President's Malaria Initiative Work Plan Y2, 11.18 Johns Hopkins Center for Communication Programs, Niger US President's Malaria Initiative Work Plan, 06.20.18 MID-TERM PERFORMANCE EVALUATION OF BA: LOCAL LESSONS, GLOBAL LEARNING 90 Johns Hopkins Center for Communication Programs, President's Malaria Initiative (PMI) Annual Report FY18, March 1-September 30, 2018, 10.15.18 Johns Hopkins Center for Communication Programs, RBM Abstract "Net-Map," Activite: 3.2.1.3-B, 09.30.19 Johns Hopkins Center for Communication Programs, Shifting Behaviors, Sharing Experiences, Identifying What Works, Advocating for SBC, 2019 Francophone Social Behavior Change Summit, Summit Report Johns Hopkins Center for Communication Programs, Social and Behavior Change for Service Delivery Community of Practice, Spearheading Progress Through a Shared Agenda, 11.08.18 Johns Hopkins Center for Communication Programs, West Africa Breakthrough ACTION Quarterly Report, April 1-June 30, 2019, 07.31.19 Johns Hopkins Center for Communication Programs, West Africa Breakthrough ACTION Quarterly Report, October 1-December 31, 2019, 02.03.21 Johns Hopkins Center for Communication Programs, West Africa Breakthrough ACTION Quarterly Report, Q1 and Q2: October 1, 2018-March 31, 2019, 04.19.19 Johns Hopkins Center for Communication Programs, West Africa Breakthrough ACTION Year 1 Work Plan, 06.29.18, Rev. 08.23.18 Johns Hopkins Center for Communication Programs, West Africa Breakthrough ACTION Year 1 Work Plan, Revised, 06.29.18, Rev. 04.15.19 Johns Hopkins Center for Communication Programs, West Africa Breakthrough ACTION Year Two Work Plan, 09.05.19, Rev. 11.25.19 Johns Hopkins Center for Communication Programs, Youth Reproductive Health and Family Planning in West Africa, Literature Review, 09.30.19 Nasiru, I.; Onyechi, C.; DeNormandie, J.; Abdu-Aguye, S.; Heward-Mills, N.L.; Tweedie, I., Leveraging on Mobile Digital Platform to Increase the Reach of Mass Media Intervention – The Albishirin Ku! Story, 2020 Naugle, D. and Rajan, R.; Johns Hopkins School of Public Health, Survey Instrument, Rapid assessments of exposure to West Africa Breakthrough ACTION project activities in Côte d'Ivoire, 05.18.20 Naugle, D.; Johns Hopkins Bloomberg School of Public Health, Survey Interview: Informed Assent Form, Rapid assessments of exposure to West Africa Breakthrough ACTION project activities in Côte d'Ivoire, 05.12.20 Naugle, D.; Johns Hopkins Bloomberg School of Public Health, Survey Interview: Informed Consent, Rapid assessments of exposure to West Africa Breakthrough ACTION project activities in Côte d'Ivoire, 05.12.20 Naugle, D.; Johns Hopkins Bloomberg School of Public Health, Survey Interview: Informed Parental Consent, Rapid assessments of exposure to West Africa Breakthrough ACTION project activities in Côte d'Ivoire, 05.12.20 Oladejo, A; Co-creating solutions to improve MNCH+N outcomes in Northern Nigeria PRH Core MTE Deliverables List Rapport de fin de Mission de formation du personnel du PNLP – Niger sur, Le CMS Wordpress, February 4-8, 2019 RBM Partnership to End Malaria, Malaria SBC Strategy Development Toolkit, Malaria SBC strategy stakeholder workshop overview, Presentation MID-TERM PERFORMANCE EVALUATION OF BA: LOCAL LESSONS, GLOBAL LEARNING 91 RBM Partnership to End Malaria, Malaria SBC Strategy Development Toolkit, Malaria SBC strategy stakeholder workshop, Day 1 Agenda RBM Partnership to End Malaria, Malaria SBC Strategy Development Toolkit, Malaria SBC strategy stakeholder workshop, Day 2 Agenda RBM Partnership to End Malaria, Malaria SBC Strategy Development Toolkit, Malaria SBC strategy stakeholder workshop, Day 3 Agenda RBM Partnership to End Malaria, Malaria SBC Strategy Development Toolkit, Malaria SBC strategy validation meeting, Agenda RBM Partnership to End Malaria, Malaria SBC Strategy Development Toolkit, National malaria social and behavior change strategy development residential writing retreat, Terms of reference RBM Partnership to End Malaria, Malaria SBC Strategy Development Toolkit, National Malaria Social and Behavior Change strategy stakeholders' workshop, Terms of Reference RBM Partnership to End Malaria, Malaria SBC Strategy Development Toolkit, Writing Retreat, Day 1 Agenda RBM Partnership to End Malaria, Malaria SBC Strategy Development Toolkit, Writing Retreat, Day 2 Agenda RBM Partnership to End Malaria, Malaria SBC Strategy Development Toolkit, Writing Retreat, Day 3 Agenda RBM Partnership to End Malaria, Malaria SBC Strategy Development Toolkit, Writing Retreat, Day 4 Agenda RBM Partnership to End Malaria, Malaria Social and Behaviour Change Program Guidance in the Context of COVID-19 Pandemic, 04.16.20 RBM Partnership to End Malaria, National Malaria Social and Behavior (SBC) Strategy Development Package RBM Partnership to End Malaria, National Malaria Social and Behavior Change Strategy, Template, 2020 RBM Partnership to End Malaria, National Malaria Social and Behavior Change Strategy: How to Use this Template, 2020 RBM Partnership to End Malaria, RBM SBC Working Group, Springboard, Web Page RBM Partnership to End Malaria, RBM SBCC WG 2018 Annual Meeting Report RBM Partnership to End Malaria, RBM SBCC WG 2019 Annual Meeting Report RBM Partnership to End Malaria, RBM Social and Behavior Change Working Group Workplan, Oct 2019- Sept 2021, 07.16.20 RBM Partnership to End Malaria, RBM Social and Behavior Change Working Group, Resources RBM Partnership to End Malaria, Social and Behavior Change (SBC) Data Brief, ITN Use RBM Partnership to End Malaria, Social and Behavior Change (SBC) Data Brief, Malaria Case Management RBM Partnership to End Malaria, Social and Behavior Change (SBC) Data Brief, Malaria Case Management (example) RBM Partnership to End Malaria, Social and Behavior Change (SBC) Data Brief, Malaria in Pregnancy MID-TERM PERFORMANCE EVALUATION OF BA: LOCAL LESSONS, GLOBAL LEARNING 92 RBM Partnership to End Malaria, Social and Behavior Change Communication Working Group, Orientation, Presentation RBM Partnership to End Malaria, Social and Behavior Change, Working Group, Web Page, 2020 RBM Partnership to End Malaria, Tll-a-story with data worksheet, ITN Use RBM Partnership to End Malaria, Tll-a-story with data worksheet, Malaria Case Management RBM Partnership to End Malaria, Tll-a-story with data worksheet, Malaria in Pregnancy USAID Mail, Breakthrough ACTION + PMI Niger Meeting, 07.30.18 USAID Mail, Breakthrough ACTION Final Year One Core Workplan, Approval, 11.02.17 USAID Mail, Breakthrough ACTION Quarterly Report- Q2 2018, Approval, 07.30.18 USAID Mail, Breakthrough ACTION Quarterly Report- Y1Q3, Approval, 08.20.18 USAID Mail, Breakthrough ACTION Year 3 Core Work Plan, 10.22.19 USAID Mail, Fwd: USAID feedback Re: Breakthrough ACTION Year 2 Core Workplan Draft, 09.28.18 USAID Mail, Modification to Zika Work Plan - Breakthrough ACTION, Approval, 07.30.18 USAID Mail, Re: Breakthrough ACTION + PMI Niger Meeting, Approval, 07.30.18 USAID Mail, Re: Breakthrough ACTION PMI Workplan clarification, Approval, 02.02.20 USAID Mail, Re: Breakthrough ACTION Travel Approval Request: Niger PMI- [Subject change to: Updated Work Plan PY2 Niger PMI], Approval, 07.01.19 USAID Mail, Re: For APPROVAL: Sahel RISE II Work Plan [Updated version], Approval, 07.01.19 USAID Mail, Re: Proposed activities for reallocating PRH2.3 and 3.4 funding USAID Mail, Revised Breakthrough ACTION Sahel RISE II Workplan, Y2 Oct 2019-Sept 2020 01.27.20 USAID Mail, Revised WABA Year 2 Workplan, 01.22.20 USAID Mail, Year 2 Revised Core Work Plan, 01.11.19 USAID Mail, Year 2 Revised Core Work Plan, Approval, 01.11.19 USAID PMI, Fact Sheet – Akwa Ibom State, 06.20 USAID PMI, Fact Sheet – Benue State, 06.20 USAID PMI, Fact Sheet – Cross River State, 06.20 USAID PMI, Fact Sheet – Ebonyi State, 06.20 USAID PMI, Fact Sheet – Nasarawa State, 06.20 USAID PMI, Fact Sheet – Oyo State, 06.20 USAID PMI, Fact Sheet – Zamfara State, 06.20 USAID, Action Memo for the Acting Assistant Administrator, Bureau for Global Health, 02/07/20 USAID, Action Memo for the Assistant Administrator for Global Health, Project Breakthrough, 06.30.16 USAID, Action Memo for the Assistant Administrator, Bureau for Global Health, Approval of Amendment to the Project Breakthrough Project Appraisal Document (PAD), 01.29.20 USAID, Breakthrough ACTION Niger, Scope of Work, 03.01.18 MID-TERM PERFORMANCE EVALUATION OF BA: LOCAL LESSONS, GLOBAL LEARNING 93 USAID, Breakthrough ACTION RISE, Scope of Work 03.19.18 USAID, Breakthrough ACTION, Mid-Term Evaluation Country/Project Contact List USAID, Breakthrough ACTION/West Africa Regional SOW, 02.26.18 USAID, Demographic and Health Survey, MIS Questionnaires and Manuals, 05.20 USAID, HIV Success Stories USAID, Nigeria Breakthrough ACTION, Scope of Work, 11.20 USAID, Request for Applications (RFA) Number: RFA-OAA-17-000002, Breakthrough ACTION, 01.03.17 USAID, Request for Applications (RFA) Number: RFA-OAA-17-000002, Breakthrough ACTION, Revised, 01.13.17 USAID, RISE Comments from USAID USAID, SBC PAD, Annex A: Logical Framework for Project Design USAID/PRH, Outcomes-High Level Indicators and Priority Concepts, SBC Various Authors, Breakthrough ACTION – Nigeria, Abstracts Submitted for ICT4D, 2020 West Africa Breakthrough ACTION, Workplan Summary, 09.23.19 MID-TERM PERFORMANCE EVALUATION OF BA: LOCAL LESSONS, GLOBAL LEARNING 94 ANNEX 4: NIGERIA COUNTRY SUMMARY INTRODUCTION AND BACKGROUND The work of USAID’s flagship SBC projects—BA and BR—has “raised up SBC as a mantra in Nigeria,” according to a key informant from BR. The work has brought home the message that SBC goes beyond communication, with particular praise for innovations like the use of HCD. The projects have also increased their impact by addressing a wider range of health areas, particularly malaria and TB, as well as important work in exploring integrated SBC. BA’s established relationships in the country—building off the previous Health Communications Capacity Collaborative (HC3) project— also meant it was well￾placed to start up quickly and to effectively help address the COVID-19 emergency. While COVID-19 created challenges in implementation, it also highlighted the importance of SBC. BA is significantly larger than BR, both globally and in Nigeria, with BR providing support in evaluation, including costing and program and cost-effectiveness. The mid-term evaluation team—including two researchers based in Nigeria—reviewed BA and BR project documents and interviewed 34 key informants in Nigeria between January and February 2021 to identify key activities, achievements, and challenges of BA and BR, as well as recommendations moving forward. This section summarizes key findings and recommendations specific to Nigeria. EVALUATION METHODOLOGY The global mid-term evaluations of BA and BR include three focus countries: Nigeria, Niger, and Côte d’Ivoire. A question guide (Annex 2) was developed for the global evaluation and was used in Nigeria. The evaluation team conducted 34 interviews with key informants from a wide range of stakeholders in the government, the USAID/Nigeria Mission, and among IPs at the national level. In total, the evaluation team interviewed three USAID staff members, eight key informants from BA, three from BR, 11 from other IPs, and nine key informants from the government. Prior to the interviews, the evaluation team reviewed the BA and BR project documents to better understand the two projects. In addition, the evaluation team had global briefings from BA and BR as well as a Nigeria-specific briefing from BR. The evaluation team had an initial meeting with the USAID/Nigeria Mission and a debrief to obtain feedback on the findings for Nigeria, which has been incorporated into this summary. It is important to note that this is a mid-term evaluation, so much of the work is still in-progress with more findings and results still to come in the next couple of years. FINDINGS 1. Achievements in SBC Programming, Research, and Capacity Building A. DEMONSTRATING THAT SBC GOES BEYOND COMMUNICATION Several key informants noted BA’s paradigm shift from SBCC to SBC, going beyond just communications to include approaches in behavioral economics, marketing, and more intensive and strategic community engagement. BA respondents described implementing the approaches and noted how many staff and partners were not conversant with the idea that SBC was not just about IEC materials. “It took a bit of struggle to cope with the new approach and re-orient staff’s thinking using the SBC flow-chart. There was lots of capacity building to bring partners and stakeholders onboard with the new vision of SBC.” A key informant from the Federal Ministry of Health (FMOH) explained how this shift from a communication-only approach played out in terms of partnership: “The Health Promotion division of the FMOH erroneously believed that BA should only work with [that] department since they are concerned with the communication. BA having realized this later helped the department understand that SBC is not only about communication, but that technical officers from other MID-TERM PERFORMANCE EVALUATION OF BA: LOCAL LESSONS, GLOBAL LEARNING 95 divisions need to contribute. For example, BA cannot successfully work on RH/FP without collaborating with us.” B. KEY INFORMANTS HIGHLIGHT SUCCESSES ACROSS A RANGE OF HEALTH AREAS An important upside to having BA and BR address a wide range of health issues is the opportunity for learning and sharing across health areas, as well as facilitating the testing of integrated approaches. The map below shows areas where BA is working in Nigeria, highlighting the types of programs being carried out in each state. BR’s Behavioral Surveillance Survey and Cost Effectiveness Study are focused in Kebbi, Sokoto, and Zamfara with other qualitative studies in selected states. Below are a number of the successes from the BA and BR work in Nigeria. A great response to the “Good Tidings!” campaign “Albishirin Ku!” (“Good Tidings!” in Hausa) is an integrated umbrella campaign in Northern Nigeria promoting 17 behaviors (and counting). The life stages segmentation has helped target interventions to the right people at the right time, with a BA project partner noting: “The life-stage approach has been the game changer having the most impact among various groups. The referral that emerges from the life- stage approach and the multiplier behavior generated is an achievement.” In terms of challenges, as one BA staff member shared: “Promoting all these behaviors is a challenge; looking at them through the life stage approach helps.” Key informants from the government to IPs to the donors all praised the campaign: “Albishirin Ku! has become a household name.” (USAID key informant) “The Albishinrin Ku! project has been very successful and I am proud to be part of it.” (FMOH key informant) MID-TERM PERFORMANCE EVALUATION OF BA: LOCAL LESSONS, GLOBAL LEARNING 96 “Albishirin Ku! is a masterpiece. It has really helped IHP and other projects working with BA as they relate with people. It has helped other strategies like the SMS messaging which makes reference to Albishirin Ku.” (IP key informant) “The radio drama has really touched lives.” (BA key informant) The Albishirin Ku! campaign is also linked to an educational mobile health game, which is played on mobile phones (by dialing 3-2-1 on the Airtel network) by answering health-related questions. The use of interactive voice response has allowed beneficiaries to listen to the radio drama series when they miss the live broadcast. A key informant from BA described the popularity of this approach: “The way and manner [in which] people have been accessing it on the 321 platform is quite interesting and I have had direct interviews with some of the radio stations that are airing it and have heard some specific statements, like the radio station manager said that people call them to ask question about the program and at some point, when they were short of funds, Albishirin Ku! the radio drama series, provided an opportunity for advertisers to seek for prime time to place advert before the start of the radio program or end of it. The radio station had to buy fuel to power the generator on one occasion in order to air the Albishirin Ku! radio program.” The Albishirin Ku! program is broadcast on radio in Hausa-speaking integrated SBC states. According to survey data, exposure has increased between 2019 and 2020 from 50 to 78 percent in these states. This is higher than exposure to the malaria-only radio spots broadcast in the malaria focal states which increased from 33 percent in 2019 to 50 percent in 2020, though both met or exceeded the 2020 targets. Malaria: addressing both patient and provider behavior Key informants from various stakeholder groups spoke enthusiastically about BA’s contributions to improving malaria programs in Nigeria. Highlights include the following: ● Campaigns of Seasonal Malaria Chemoprevention (SMC) and Long-Lasting Insecticide￾Treated Nets (LLINs) have been more successful due to the support provided in BA in its focal states. In Zamfara, SBC activities in 2020 also contributed to high demand for nets, resulting in 2,957,848 LLINs distributed, which represents about 96 percent redemption rate. Demand for SMC medicines was also high with 1,102,226 eligible children receiving the preventative dose of SMC medicine during the first cycle. Redemption rate of LLIN in the October-December 2020 campaign in the Benue State was 99.4 percent. ● The strengthening of the Advocacy, Communication and Social Mobilization (ACSM) unit of the NMEP. BA provided TA/capacity strengthening to ACSM on “Guidelines for Malaria ACSM” resulting in this unit’s increased capacity and transition from a low to one of the top performers. ● BR presented malaria results to the NMEP to help inform its revised national strategic plan as well as the 2021 work plan and activities. ● The important focus on behavior change among both patients and health providers, linked to BA/BR’s global focus on provider behavior change, has helped address major challenges in malaria programs. All respondents involved in malaria work in Nigeria mentioned the success in working to change provider behavior: o “The biggest achievement of BA is the aspect of the design of behavioral economics prototypes. The major challenge we had with malaria is the provider’s behaviors which were not in sync with national guidelines. If not for BA, we [would] not be where we are in terms of level of success. The main problem is that the health care provider did not trust the Rapid Diagnostic Test and if tests are conducted, the health provider did not want to use the results especially if it was negative.” (PMI key informant) o “Despite BA not covering all states and local government areas in focal states, there are some significant changes in national data meaning there are reflections of BA results outside its focal areas. For example, the Provider’s behavior tool, which comes under behavioral economics and [was] piloted, is one that NMEP has adapted well.” (NMEP key informant) MID-TERM PERFORMANCE EVALUATION OF BA: LOCAL LESSONS, GLOBAL LEARNING 97 ● Strong partnerships are essential, by linking with service delivery and logistics management interventions, spreading the impact beyond BA focus states: o “BA works jointly with other stakeholders, for example, PMI trained the health care provider in service delivery aspect while BA handled the SBC aspect including mass media.” (PMI key informant) o “Until BA came, there was a serious knowledge gap on management of fever. There has been success in this area from the angle of healthcare providers especially… The BA activities are harvested and replicated in non-BA focal states. This is done through experience-sharing in the area of malaria. This is where government agencies come in. NMEP ensures that SBC activities and messages are spread beyond areas where BA has physical presence by inviting other relevant partners, e.g., Global Fund, to make use of some of the approaches.” (NMEP key informant). TB: SBC at a time when the need was clear A government respondent noted that “BA came at the time when we needed them most especially with the low case finding/detection.” BA facilitated the HCD process to improve TB detection, including mapping out the client journey, drafting a Creative Brief, and participating in a design workshop. Several respondents reported a visible increase at National Coordinator, National Tuberculosis and Leprosy Control Program in the level of interest in SBC and ownership of the process, and a resulting campaign, “Brother’s Keeper.” This campaign promotes shared responsibility by engaging small medicine stores (patent and proprietary medical vendors), religious leaders, and those in marketplaces and transportation hubs, using people outside of the medical community to improve TB case detection. “BA has helped service providers sell their products, which is service delivery in TB. Before now, we were not finding TB cases, but the social marketing put up by BA has made the SBC messages (on TB) more acceptable.” (Government key informant) The participatory/co-design process leads to strong interventions as well as capacity strengthening and ownership. The HCD process is slow but it is also important, as noted by government, IPs, and donors. “One of the great successes for us has been the ownership by the TB community… I think it’s because of the HCD approach.” (USAID key informant) The slogan developed by BA with the support of others—“Check Am O” is one brand that is making a difference. “BA was like a breath of fresh air in SBC. The fact that the project has HCD on TB is an achievement. The very thorough informative assessment on TB in communities really helped. The Institute of Human Virology Nigeria leveraged on the findings of this assessment.” (IP key informant) “Mobilization for TB testing and treatment is reaching more numbers because BA is working with community￾based organizations.” (IP key informant). “They’ve succeeded in creating a large group of stakeholders drawn from range of partners—not just USG supported—who own the process of demand-creation for TB—and that didn’t exist before. They have a national technical working group, which didn’t exist before—that is due to BA.” (USAID key informant) Ensuring a culturally-relevant approach to addressing FP & MNCH-NUTRITION With the understanding of myths and misconceptions, BA approached engagement with religious and community leaders, and government partners to discuss FP from the perspective of health—as a key intervention for saving lives. For example, child spacing allows the mother to breastfeed more and for a longer time, so her child is healthier, the mother will be better prepared for the next delivery having had more time to recover, the family can better provide for their children, etc. Due to mistrust around FP, BA changed the term FP to “childbirth spacing.” Learning from HC3, BA used the term “Tazaran Haihuwa” —Hausa for “childbirth spacing”—which laid a good foundation for the project’s FP activities. MID-TERM PERFORMANCE EVALUATION OF BA: LOCAL LESSONS, GLOBAL LEARNING 98 Integrated SBC lessons will be important both in Nigeria and for the global SBC community BA and BR are collaborating on a study to assess the effectiveness of integrated versus malaria-only SBC on malaria, FP, and MNCH+N behaviors and ideations among pregnant women and those with a child under two years. There is a great deal of interest in this topic, but little evidence of whether it works, the cost-effectiveness of different approaches, etc. Implementing integrated programs is challenging, in part due to the vertical nature of donor funding and of government structures. For example, the malaria and nutrition agencies may be in different places and not communicating with each other. But, while funding and management can be siloed, as a USAID key informant said, “people don’t live siloed lives.” There is a plan for three rounds of data collection for the study. The first occurred before the COVID-19 pandemic, while the second round has been delayed, due to COVID-19, but all respondents agreed that the findings should be very useful. BA played a major role in the COVID-19 response through coordination and collaboration, supporting the Nigeria Centre for Disease Control (NCDC) and the Presidential Task Force for COVID￾19. NCDC mentioned a number of ways that BA supported their efforts: contributing to the strategy document, jointly developing jingles, upgrading the NCDC website, helping with training, and working together to develop a tool for stakeholder analysis. BA suggested a tool, but NCDC already had one; BA suggested some changes to the tool and they worked together to modify it. BA also supported some states (e.g., the Plateau State) where they are present. They tried to replicate what they are doing at the national level after reviewing their strategy and realizing that there were weaknesses at the sub-national level. Some states needed stakeholder coordination and engagement in order to optimize resources available at the state and the local government level. BA supported that effort by identifying states with critical challenges and assisting them. As an example of the process for developing the COVID-19 intervention, BA worked closely with NCDC and stakeholders to understand and adapt to the context of COVID-19 (i.e., virtual platforms) what is called “Journey Mapping” to get insights on the way worship is conducted among religious groups. This process highlighted that at some point the religious groups were not adequately engaged. BA built the capacity of 20 individuals from NCDC and other stakeholders to understand why user-experience and co-designing with the target audience (end-user) was necessary. This has led to setting up a new committee on how to get the buy-in and contribution of different religious groups on COVID-19 messages before passing them on to the populace. To present a unified front and ensure that COVID-19 materials were usable by all, BA obtained an exemption to remove the branding/visual identity of USAID/BA from all their materials. This enabled other IPs and the government counterparts to use all the materials (radio jingles, IEC materials, etc.) produced by BA. This relates to an idea heard from several key informants about the need to clarify how all USAID￾funded IPs are collectively contributing to good health instead of trying to emphasize individual presence and importance to the donor. Another example of an integrated approach was the combined COVID-TB campaign, which was “brilliant,” according to USAID/Washington GHSA Advisor. The COVID-19 response was immediate in introducing capacity strengthening in risk communication and community engagement. During these times of COVID￾19, BA also worked with the Nigerian government to investigate the relationship between TB and COVID￾19 and how to make the COVID-19 message clearer to the masses. C. RESEARCH TO INFORM PROGRAMS BR has played an important role as the research partner to BA. One BA staff member expressed support for the idea of having strong research to inform implementation: “We should have evidence before implementation, it will be very good and useful rather than trying to change things when implementation has started already.” It is worth noting that there are often challenges in getting USAID missions to buy into global research projects, and so USAID/Nigeria has been bold in making this investment. USAID staff explained MID-TERM PERFORMANCE EVALUATION OF BA: LOCAL LESSONS, GLOBAL LEARNING 99 the multiple reasons why it was worthwhile to invest in BR, lessons that can be shared with other USAID missions: “I would note the value of having both awards running concurrently to facilitate adaptive management and increase the potential that program approaches will better result in the anticipated outcomes/impact…. A ton of rich SBC data is most likely an exciting output from investing in BR, e.g., the BSS [Behavioral Sentinel Surveillance] in Nigeria…. Overall, it’s worth the investment in SBC research as there's just a great potential to contribute to the evidence base on what works (where, when, with whom etc.), cost and cost effectiveness.” While there have been challenges in the BA/BR relationship (addressed in detail later), this has improved over time and there have been a number of successes both in terms of informing BA’s programs and also contributing to the global evidence base for SBC. Some examples are shown below: ● Good collaboration on the BSS Survey with joint development by BA and BR to ensure asking the right questions for programmatic relevance. The 2019 BSS baseline had a very fast turnaround, with the work plan approved in July, the survey designed from July–September, the fieldwork conducted from September–October, and preliminary results presented to BA and USAID in November, with a full technical report available in December 2019. The BSS baseline findings led to some adjustments in the implementation and monitoring by BA. For example, BSS findings clearly showed that men have an overwhelming influence on decisions around FP uptake and health-seeking. Although there were existing strategies targeting men, BA had to strengthen activities in the SBC advocacy group intervention, e.g., using the “Adalchi” (fairness and justice) concept to address social norms that would allow women to also partake in decision-making at home, including income generation. Findings from BR also made it possible to identify wards in some states with specific knowledge gaps, guiding BA to focus interventions in specific areas for these wards. ● The BSS included new ideational metrics, e.g., for pneumonia, which will inform Nigeria programs and global SBC research. BR quickly developed a research manuscript on pneumonia, explaining how: “It is relevant in the sense that Nigeria currently does not have a pediatric or childhood pneumonia plan, so when BR brought out the evidence base from an SBC perspective, it was timely because it was presented at a conference as part of a special issue and it was also included as part of the evidence to generate a childhood pneumonia control plan in Nigeria.” ● BSS baseline analysis led to USAID requesting additional analysis on breastfeeding and adolescence. ● BSS findings were disseminated at national and international level through several well￾attended webinars, adapting to the realities of COVID-19. ● BR provided evidence that BA can learn from and use for its SBC needs and the needs of other IPs and government. For example, BA used findings from the BSS to tailor the topics for the radio drama program Albishirin Ku!, to bridge the knowledge/practice gaps identified. ● Qualitative analysis of some SBC program components, like the ACG, will be useful. BA staff noted that the ACG, made up of key-influencers within communities, is also a key factor in success, but it will be useful to have evidence to better understand and adjust this approach as needed. BR staff explained: “We designed work around ACG in part because they kept saying how valuable it was. There was no basis for that, but such a strong success story. I’m optimistic that they’re right, but we designed the study to assess whether the premises of what they’re talking about actually produce the types of results they’re talking about. Really put it to the test from an external objective point of view.” ● Application of costing guidelines. The costing guidelines for SBC that were developed by BR are seen as an important contribution by the SBC community, but making them truly useful requires MID-TERM PERFORMANCE EVALUATION OF BA: LOCAL LESSONS, GLOBAL LEARNING 100 country application, as is being done in Nigeria. This will also allow for a cost-effectiveness comparison between integrated and vertical SBC approaches. ● While BR is the research/evaluation partner for the Breakthrough projects, BA also does critical monitoring of activities. For example, BA’s monitoring of mass media (e.g., radio spots) led to making adjustments to their media strategy based on the findings. For BA monitoring and evaluation system, there is a plan to move data management from paper-based to electronic/digital tools, which needs more funding and capacity for the field team. D. BA HAS HELPED TO STRENGTHEN BUY-IN, OWNERSHIP, AND CAPACITY, FOR SBC Seeing SBC as important A range of key informants spoke about their increased appreciation for SBC, as noted by a government staff member: “BA has converted us and now I am an ambassador of SBC.” Respondents highlighted the ownership and buy-in developed through involvement and in particular the HCD process (discussed more in the next section). As a USAID respondent stated: “I’ve seen a big shift in recognition of SBC as important. And that’s why in terms of ownership, they’ve done a great job. They’ve become an essential partner to the national TB program. Not seen as Breakthrough. With certain projects it’s not seen as our thing, it’s seen as the program. But everyone owns it, including Global Fund partners. That acceptance is great.” BA engaged local stakeholders to be part of the process, e.g., religious/traditional leaders, thereby stimulating community participation and ownership. BA has encouraged buy-in by the government at all levels (Federal, State and Local Government Area), including engaging WDCs in SBC. Government is now getting involved, as SBC activities have started appearing in the budget line of some states, though there is no confirmation that money was released. As noted earlier, BA worked with NCDC and other stakeholders to promote the behavior change needed to prevent the spread of COVID-19 in focal states and beyond through the support provided in risk communication and intensive community engagement for six months. The concrete nature of this work in an emergency situation helped effectively demonstrate the importance of SBC. Strengthened capacity among stakeholders, including government, IPs, and media BA has strengthened the capacity of government officials and IPs in SBC, especially in the space of TB, malaria, MNCH, nutrition, and child spacing. BA has contributed to ongoing efforts in developing some national strategic documents, like the National FP Communication Plan and its implementation in Nigeria. The project has been using the media space effectively by combining appropriate channels and strategies. In collaboration with others, BA has also built the capacity of some media personnel to engage as active partners in SBC. Important subnational involvement and capacity building, particularly with the WDCs Community capacity strengthening started in 75 wards working with Ward Development Committees (WDC). Based on performance, all WDC’s will “graduate” to Phase 2 after adopting simple and practical steps that supported desired behaviors, and new wards are now starting Phase 1. The capacity of the local people has been improved through the use of SBC, especially during the LLIN campaign. The local people actually do the training for the LLIN campaign themselves, instead of depending on BA, because they now have the capacity to conduct the training by themselves. This capacity strengthening has led to some WDCs being able to raise funds from the community and implement development projects for the community. For example, in FY 2020, all 75 WDCs raised about $8,047 to implement several activities, including transporting over 1,146 pregnant women for antenatal care, 1,127 pregnant women for delivery at a health facility, as well as 132 under-five children and 128 women for various other illnesses. MID-TERM PERFORMANCE EVALUATION OF BA: LOCAL LESSONS, GLOBAL LEARNING 101 Key informants from the Nigerian government spoke of their increased investment and cost-sharing for SBC: “It has influenced investment by the government. Government is supporting certain activities on SBC on 2020 budgetary allocation being implemented till the end of March 2021. There is an activity on capacity building on SBC and support of a health promotion forum coming up this February this year, funded by the federal government through the FMOH.” In addition, the Department of Health Services and the Department of Community Health of the National Primary Health Care Development Agency have engaged in cost-sharing, especially in LLIN campaigns. The government has also shown its increased interest in SBC by creating SBC units in different departments and divisions of the FMOH and by providing personnel that work with IPs to enhance sustainability. The focus, starting in FY 2021, is to see more bottom-up approaches/leadership, being deliberate about implementing states taking more responsibilities and making their own decisions in SBC while BA provides technical assistance. The vision of BA is that after closeout, the FMOH can plan and implement key SBC activities using the SBC flow-chart and other resources to facilitate planning and design and reduce assumptions and misperceptions. Learning goes both ways “Building local capacity requires deliberate approach. BA has to be patient with the locals in terms of setting goals and what they need. Capacity building also requires continuous learning as BA must also be ready to learn from local partners. There must be mutual trust built for capacity to be more effective.” (PMI key informant) While there is increased capacity, there is still a need for support A key informant from the government explained the need for continued support: “Yes, BA has increased the level of interest in SBC. I want to urge BA to continue. If there is anything we need for the Nigerian health system, it is SBC and risk communication. Government cannot do it alone.” In an effort to increase the critical mass of SBC specialists at the national, state and LGA levels, BA has sponsored 20-25 participants per year at the Nigerian Leadership in Strategic Health Communication workshop. Each participant develops an action plan towards applying learnings in their jobs. Nevertheless, another informant discussed the useful “learning by doing” that had happened with BA, but felt that intentional traditional capacity building was lacking: “That could be an area of possible improvement. Like workshops, for example, would be a good way to go.” (USAID key informant) MID-TERM PERFORMANCE EVALUATION OF BA: LOCAL LESSONS, GLOBAL LEARNING 102 2. Design and Implementation of Activities Design and implementation of activities has focused both on creating strong programs and building ownership and capacity along the way. A. WIDESPREAD APPRECIATION OF NEWER APPROACHES, INCLUDING HCD AND BEHAVIORAL ECONOMICS As noted earlier, many key informants mentioned BA’s paradigm shift in going from SBCC to SBC. BA has been able to draw upon the expertise of its coalition members, including Ideas42, ThinkPlace, and Save the Children, to incorporate these newer approaches into community engagement work. Coalition members report a good working relationship with shared understanding and mutual respect. There is appreciation for co-design and collaboration, but recognition that it can be slow BA’s work is characterized by and recognized for its collaboration. A key informant noted that from the SBC flow-chart, one will see phrases like “Co-designing” and “Co-development” as key principles with government and IPs. The principles guide effective collaboration with other players. Experience and established relationships from the previous HC3 project, also implemented by Johns Hopkins University’s CCP, made it easy to transition into BA. Government partners praised BA staff for their: “…teamwork and joint effort. BA is a good team player. We were carried along. The concept and implementation plan were developed jointly at all levels with government officials”; and for their technical expertise “the staff are sound and know what they are doing.” A respondent from an IP appreciated BA’s understanding of local context: “One of the things that I will say I cherish and enjoyed with what they have done is their ability to understand the Nigerian work environment and come up with approaches that make it possible to mobilize people.” This collaborative effort/co-design process, using HCD, for example, has been used across health sectors (malaria, TB, MNCH, etc.) While the process is popular, it is also seen as being slow, but still worthwhile, according to many: “The process of needs assessment was tedious, but it was worth it.” However, some key informants noted that some service delivery partners are not always patient with SBC and the time needed for behavior change to happen. For example, it is easier to ask a pregnant woman to get insecticide-treated bed nets for free at health facilities than to influence her behavior to purchase them with her own money. It was noted that long-term behavior change requires more time than demand creation in order to identify and address the many barriers. While the HCD process stresses involvement in design, some key informants talked about the need to find better ways of doing dissemination so that the community or the direct beneficiaries at the grassroots level can see the results of their efforts—not just from the perspective of their gain but also on how their efforts impact their overall life, development, and growth as a group. In addition, there could be more community member involvement in refining the design of the SBC process and having further discussions about analysis and decisions made after the needs-assessment. Most key informants praised HCD HCD is a design approach that involves all stakeholders with a feedback mechanism, which has been a critical enabler, bringing innovation into the work, described by some as “a new way of thinking.” A BA staff member stated: “I love the HCD because it took us to the beginning and it is very broad.” A BA partner stated that “HCD was an eye opener for me. Before now we have been using questionnaires.” HCD also contributes to capacity development; for example, one key informant highlighted how HCD had really helped them, in part through capacity development of WDCs, leading to the WDCs investing resources rather than just receiving them. MID-TERM PERFORMANCE EVALUATION OF BA: LOCAL LESSONS, GLOBAL LEARNING 103 Another key achievement is the design of behavioral economics prototypes, understanding the science that links behavior to attitudes, and how issues are perceived by the people. BA promotes the capacity to understand how people’s perception of an issue (or the way they think or view a particular issue) is linked to the way they behave. BA has also utilized technological innovations, including partnering with Viamo for optimization that allows mobile phones to display content like computers so that more people can listen to the radio drama on their phones and then call in to participate in the game. Mixed feelings about SBC Flow Chart The SBC flow chart, described by one key informant as “a re-packaging of best practices,” is the main tool used by BA to design programs. It maps out different phases (discovery, designing and testing, application) of the project planning and can facilitate collaboration with government partners and other stakeholders. This tool contains various principles and approaches used to build the capacity of stakeholders. It seemed to work well internally to clarify roles among BA coalition members. That said, both globally and in Nigeria, there were some mixed feelings about the flow chart, as expressed in the following comments, with some concerns about how practical and sustainable it is: “confusing for non-SBC stakeholders,” “overly consensual,” “slowed process down.” B. COLLABORATION WITH SERVICE DELIVERY PARTNERS INCREASES COVERAGE AND IMPACT A key informant from an IP noted that BA brought about better integration of USAID-funded projects (IHP, SHOPS+, GHSC/PSM, PMI for the state, etc.) and successfully carried along other IPs and government. That is, with the presence of the SBC project in a particular location, interaction between service delivery and logistics management IPs is better in that location and subsequently leads to better programming. A key informant from the government explained that “BA put through the projects well to other partners and stakeholders. This makes SBC activities rub off in some non-BA states.” While partners spoke to successes, there were sometimes issues around timing. ● “For the three years, BA took up all activities on the demand side of all training conducted in SHOPS+. BA has the ability to mobilize clients to health facilities for SHOPS+ implementation to the point that this reflects on the number of clients using health facilities for FP. Ability to do this effectively is a great achievement… Since BA cannot cover the entire states and country, there is a need to work with the FMOH and state to see how what BA has started can be expanded to other non-focal states and sustained generally. It would have been better if USAID had ensured both BA and SHOPS+ projects worked using the same timeline and moved at the same pace. The results of the latter were to a large extent dependent on activities of BA.” (SHOPS key informant) ● “GHSC/PSM is responsible for logistic distribution of insecticide-treated bed nets while BA works on the SBC aspect… I will give credit for their contribution in the 11 PMI states where they have been supporting the SBC aspect in LLIN campaigns. Also, for their collaborative effort in designing a human-centered approach that has further informed the design of drug revolving fund in Sokoto and Bauchi by GHSC/PSM… Several stakeholders now have increased interest in SBC. The difference is very clear in the result we are getting from the health facilities since BA started mobilizing.” (GHSC/PSM key informant) In terms of supporting the objective of IHP in strengthening RMNCH and nutrition, BA has done fairly well in raising awareness and improving demand for services. While there has been good collaboration, there is still room for improvement. This could include USAID clearly articulating and communicating its overall vision and how the objectives and expected outcomes of each partner fit together in such a way that IPs are not in competition but rather each has its clear role. As an example, some key informants discussed how the roles and responsibilities for provider behavior change could be better defined since responsibility falls on both sides (clinical-behavioral). The Provider Behavior Change Dashboard has helped to keep track of activities. MID-TERM PERFORMANCE EVALUATION OF BA: LOCAL LESSONS, GLOBAL LEARNING 104 C. BARRIERS TO IMPLEMENTATION Below are some of the barriers faced by BA and BR during their implementation. ● The COVID-19 pandemic and other factors like the #EndSARS protest in year 2020 caused delays in timelines for both BA and BR, has required shifting to virtual interaction rather than face-to￾face, and has led to delays in transitioning some wards to the maintenance phase. ● Unhealthy competition between partners sometimes as each partner tries to promote itself more, forgetting that the successes of the USAID-funded projects rely on interdependent activities. ● There were some initial barriers in the way some of the community members saw/perceived BA and other IPs as wanting to impose foreign culture. ● Delays on deliverables from some stakeholders due to bureaucracy with some partners, including the government. ● Staff turnover, both within the project and with partners, e.g., BA explains how new teams had to come on board as some staff that participated in Bauchi, Kebbi, and Sokoto left the project before it expanded. 3. Advancing the Practice of SBC One of the goals of the Breakthrough projects is to advance the practice of SBC, in part through encouraging increased integration of proven SBC interventions into health and development programs. As noted earlier, the experience in Nigeria shows how SBC approaches are needed and can be applied to many health areas to improve impact. It will be important to fully document this impact and share it broadly. A USAID key informant noted that “better reporting of results would make it easier to convince people that it’s worth investing in.” The work in Nigeria represents the practical application of a number of global priorities. For example, BR led a participatory process to identify a global research and learning agenda for SBC, and the two priority areas—provider behavior change and integrated SBC—are both important parts of the work in Nigeria, again showing how the work will not only benefit Nigeria but will also help advance SBC practice globally. 4. Collaboration between BA and BR The design of the Breakthrough projects—dividing implementation and research into two separate but connected pieces—requires collaboration. Initially, the relationship presented challenges, with BR starting a year later than BA with less funding, far fewer staff, and less experience in SBC. Overall, there is a sense that BR needs BA more than BA needs BR. Both projects report closer collaboration and understanding now with joint planning and dissemination activities. Collaboration between BA and BR has included designing studies (protocols and tools), workshops, planning meetings, and holding coordination meetings. During the COVID-19 lockdown, several webinars were organized on each health domain, where both BA and BR made presentations through national and global webinars. BR discussed how studies were conducted, what was being investigated, and main findings while BA focused on the importance of the research study to the project’s programming. BA and BR work closely together. For example, they collaborated on the protocol of the BSS study, including the questionnaires to confirm that the questions cover all areas of intervention. Collaboration takes time so it is important to understand the benefits, particularly having objectivity in evaluation Some key informants from BA pointed out that while that BR’s needs for data can put pressure on BA, it has also “helped BA to be a bit more specific and categorical in the way BA interventions are described or viewed.” In addition, some respondents saw the important value in having an external evaluation of activities, noting that BR is in a unique position to show evidence of the value added of new SBC approaches. However, sometimes this role created tensions: “there were initial teething problems when sometimes BR was perceived MID-TERM PERFORMANCE EVALUATION OF BA: LOCAL LESSONS, GLOBAL LEARNING 105 to be there as police.” According to a number of key informants, the amount of time necessary to forge an effective relationship with BA was underestimated by BR, but now things have changed. Some key informants from BA and BR noted that USAID should include in the agreement clause that BA should work closely with BR and the funding agency should collect performance indicators on collaboration, something that is being done at the global level. BA/BR RECOMMENDATIONS FOR NIGERIA 1. Continue use of newer participatory approaches, such as HCD and behavioral economics, and create shared expectations on timeframes. While HCD can be a slow process, it is highly appreciated and regarded and appears to have a positive impact on capacity and ownership. 2. Explore streamlining the flow-chart process to be more practical to use. 3. Dependent upon results of the effectiveness study, anecdotal evidence suggests to continue the application of the life stages segmentation approach when implementing integrated programs. 4. Improve measurement and reporting of the impact of SBC. Globally, BR and BA have developed a business case that helps with this, and it will be important to communicate these results that show impact on health outcomes at the country level. 5. Encourage a sense of collective contribution to improving health rather than a sense of competition among implementing partners. A good example of this was the lack of branding on BA COVID-19 messages and materials to encourage broader use. 6. Expand engagement of communities and beneficiaries beyond initial needs assessments to also include sharing and discussion of analysis, decisions, and results. 7. Be more deliberate, intentional, and strategic with capacity building by putting all activities in the work plan, building capacity regularly through more structured processes, and articulating clearly how it fits into the overall SBC strategy. 8. Increase involvement of BR in various aspects of implementation, such as training and field visits, to enhance understanding and strengthen relationships among all partners and better enable BR to track implementation challenges and progress towards greater government ownership. 9. To improve BA/BR collaboration, missions should aim to align the timing of buy-ins. 10. In order to sustain innovative SBC approaches, include local private sector marketing, creative, and academic partners in capacity strengthening activities. MID-TERM PERFORMANCE EVALUATION OF BA: LOCAL LESSONS, GLOBAL LEARNING 106 ANNEX 5: NIGER COUNTRY SUMMARY INTRODUCTION AND BACKGROUND BA has been active in Niger through buy-ins from the President’s Malaria Initiative (PMI) and regional buy￾ins from the Sahel and West Africa Regional Offices. BR has a buy-in through the Sahel Regional Office to conduct an evaluation in the RISE II zones of Niger and Burkina Faso. BA has focused heavily on capacity strengthening activities with both the Niger Ministry of Health (MOH) and resilience and food security (RFSA) partners, while also designing demand-side activities to complement the work of AmplifyFP and other service delivery family planning programming in the country. BR has focused primarily on an evaluation of integrated SBC programming in the RISE II zones of Niger and Burkina Faso, while also conducting monitoring and evaluation of the Merci Mon Héros digital campaign. BA has been very successful in improving SBC capacity and implementation in the areas of malaria and family planning, but there have been numerous challenges in executing activities alongside RFSA partners in the RISE II zones. Misaligned timelines, confusion over project objectives, and initial limitations in BA staff presence in Niger all contributed to delays and limitations in the impact of RISE II activities. BA and BR project staff have pushed through these challenges, however, and come to a place of shared understanding between partners for improved collaboration moving forward. EVALUATION METHODOLOGY The global mid-term evaluations of BA and BR include three focus countries: Nigeria, Niger, and Cote d’Ivoire. A question guide (Annex 2) was developed for the global evaluation and was used in Niger. The evaluation team conducted 40 interviews with key informants representing a wide range of stakeholders in the government, the USAID/Niger Mission, and among Implementing Partners (IPs) at the national and district levels. In total, the evaluation team interviewed four USAID staff members, 26 respondents from IPs, and ten respondents from the government. Prior to the interviews, the evaluation team reviewed relevant BA and BR project documents to better understand the two projects. In addition, the evaluation team had global briefings from BA and BR as well as a Niger-specific briefing from both projects. The evaluation team had an initial meeting with USAID/Niger before beginning fieldwork. It is important to note that this is a mid-term evaluation, so much of the work is still in-progress with more findings and results still to come in the next couple of years. FINDINGS 1. Achievements in SBC Programming, Research and Capacity Building a. Malaria PMI in Niger planned to invest $2 million in BA across five years (2018-2022) to improve priority malaria behaviors in the country. This was aligned with Niger’s recent addition as a new PMI focus country as part of a five-country expansion. BA’s work on malaria has focused on local capacity strengthening, including working with the National Malaria Control Program’s (NMCP) Information Education and Communication (IEC) division to improve SBC interventions led by the government of Niger. BA has also participated in the planning of Malaria Operational Plans and is launching a pilot community engagement activity in two districts in Dosso and Tahoua regions, supporting multisectoral community mobilization teams to design and implement SBC activities through community dialogue. Key informants from Niger’s MOH expressed sincere appreciation for the work of BA, noting that this was “the first time a bed net program has benefited from exceptional communication media that attracts and motivates the target audience.” Partners noted that BA has helped bring a real harmonization of messages on malaria, and an operational framework for action for stakeholders in Niger. MID-TERM PERFORMANCE EVALUATION OF BA: LOCAL LESSONS, GLOBAL LEARNING 107 b. Family planning USAID’s West Africa regional office planned to invest $3 million into BA over a three-year period (2019- 2021) to direct SBC activities “to support increased adoption of positive behaviors pertaining to family planning and reproductive health (FP/RH) among individuals in Burkina Faso, Côte d’Ivoire, Niger, and Togo.” The West Africa buy-in to BA, WABA, has worked in close collaboration with the government of Niger, in addition to regional and local partners to strengthen local capacity to design and implement SBC activities alongside health service delivery. USAID/Niger expressed an appreciation for WABA’s work, noting that they’ve completed really “interesting activities” that “can be shared globally.” AmplifyFP, the West Africa regional office’s four-country service delivery program for FP, has been a crucial partner to WABA throughout this process. Staff from AmplifyFP noted that WABA’s work has been critical in driving demand for FP services, while AmplifyFP focuses on supply. WABA has been conducting community dialogues and site walk-throughs which promote FP use, in addition to mounting the digital campaigns Merci Mon Héros and Confiance Totale. There was a widespread appreciation for the community dialogues from service delivery partners, who noted that “working with community members this dialogue then improves communication with patients and providers/community and clinics – improves service quality, service delivery.” Other partners in Niger noted their appreciation for WABA’s community dialogues, stating that they “really allow for discussion of all the community’s problems.” Further, “based on the problems identified, the community goes to the service delivery points, and it’s a dialogue between providers and communities to see what the problems are, how to solve them.” Initial data from BA indicates that a high percentage of participants in community dialogues report being willing to recommend using FP to their communities following participating in the events (median 89.5 percent), with a similar percentage noting that they themselves report an intention to use FP services in the future (median 88 percent). In addition to community activities, WABA has mounted two digital campaigns to improve knowledge and increase demand for FP methods and services, Merci Mon Héros, which is aimed at youth, and Confiance Totale, which has seamlessly integrated COVID-19 messages within its radio programming. Merci Mon Héros has created videos and content to be shared on social media, radio, and television, targeting youth and their caregivers for improved dialogue around sexual and reproductive health. Confiance Totale produced radio spots, communication leaflets, and counseling cue cards with detailed information about FP methods, their effectiveness, and side effects. Both digital campaigns were widely praised in country and global interviews for their efficacy in communicating key FP/RH messages and information, particularly to youth and their caregivers in the Merci Mon Héros campaign. c. RISE II activities BA’s biggest challenges in Niger have been in working with RFSA partners in the RISE II zones. USAID’s Sahel Regional Office (SRO) has invested $3 million annually into BA for a four-year activity (2019-2022) to conduct 1) capacity strengthening activities on SBC with RFSA IPs in the RISE II zones of Niger and Burkina Faso and 2) capacity strengthening activities with the two countries’ MOHs. Key informants from the Sahel Regional Office noted that while SBC programming aimed at improving behaviors linked to persistent development challenges in the region has been ongoing for over ten years, these efforts were overly fragmented and ineffectual, and it was hoped that BA could work as a “single, specialized mechanism” to “shape and streamline SBC activities across RISE II, reducing redundancy and enhancing impact.” In practice however, it took considerable time for the RFSA partners and BA staff to understand their complementary roles in SBC design and implementation. USAID/SRO hosted multiple meetings between the partners to get to a place of shared understanding, but BA’s capacity to perform their scope from the outset has been limited, particularly in having relevant staff in place in Niger and Burkina Faso to work directly with RFSA partners on project activities. This has delayed project activities from BA, potentially blunting the impact of BA’s activities in the RISE II zones. USAID/Niger and SRO expressed sincere worry MID-TERM PERFORMANCE EVALUATION OF BA: LOCAL LESSONS, GLOBAL LEARNING 108 that BA would not achieve its objectives in the Sahel buy-in, due to protracted miscommunication between partners and continued delays in implementation. From the outset, RFSA partners seemed unclear on what BA’s role was in supporting RISE II activities. BA conducted multiple individual and group workshops to explain their role as a “support” to RFSA partners, working with each partner in a “learning by doing” capacity strengthening approach, jointly conducting small scale activities to pilot innovative approaches the RFSA partners would then use moving forward in routine project implementation. BA was there to provide technical assistance and guidance to improve RFSA partners’ SBC activities, working on formative research protocols, strategic design process, refinement of strategies, and sharing new concepts like behavioral ideation and human-centered design (HCD). Some RFSA partners expressed frustration with this approach. Confusion on BA’s role was ongoing, despite multiple workshops and presentations. It was eventually decided that BA would send project staff to the field to work together with RFSAs in a “coaching” role. This has also proved challenging however, as hiring and retaining staff from BA in the RISE II zones has not been straightforward. BA staff have also continued to emphasize that many of the RFSA partners’ SBC plans are actually IEC plans, and they do not go further than IEC to address social and normative barriers to behavior change. USAID’s Bureau for Humanitarian Assistance (BHA) has been supportive of BA and RFSA’s collaboration throughout, and appreciates that RFSA staff have attended all of BA’s trainings and meetings. However, key informants from BA and RFSA partners explained that the initial level of miscommunication and confusion between partners in the outset of collaboration has sincerely limited both the impact and timeliness of outputs from BA in the RISE II zones. d. Capacity strengthening In all of BA’s work in Niger, capacity strengthening has been at the forefront of the design of activities to ensure long term sustainability of efforts. BA has engaged the following partners in capacity strengthening workshops and activities: • MOH (central and department levels) • Ministry of Water and Sanitation (central and department levels) • Ministry of Community Development • Districts and Mayoral offices • DFSA partners (Girma-Catholic Relief Services, Hamzari-CARE, Wadata-Save The Children) BA held multiple workshops and trainings in 2019, in both Niamey and outer provinces of the country, including a Workshop on Leadership and Innovative Approaches to Strategic Public Health Communication in Dosso, training of MOH staff on SBC for malaria, and an orientation workshop for actors on the NetMap process in Maradi. BA also led the establishment of a Technical Working Group for the Promotion of Social and Behavioral Change Interventions which was then merged with the National Communication Committee. A key informant from a partner involved in this work noted that the “two groups with their strengthened capacities now serve as bodies for validation of strategies, approaches, messages and SBC tools at the national level.” BA has also created a National Innovation Team to design prototypes on SBC interventions, which can then be tested and validated. BA staff noted that these prototypes are then integrated into ongoing activities conducted by RFSA partners, but some RFSA partners expressed hesitation about the relevance and feasibility of the tools being developed. BA staff noted that capacity strengthening efforts have focused primarily on moving beyond IEC approaches: MID-TERM PERFORMANCE EVALUATION OF BA: LOCAL LESSONS, GLOBAL LEARNING 109 “The first challenge we faced was to get people to accept and understand the concept of SBC. They just talked about information, education, and communication, which really had no impact on behavior. So we introduced this notion and this concept of SBC, and through a series of workshops, and also hands-on behavior change strategy sessions, we were able to get people to adopt this notion, this concept of SBC [that goes beyond IEC].” While some BA staff noted their satisfaction with this approach, saying it’s been “a major success,” USAID/Niger has expressed reservations that some of the working groups formed have been very limited in participation and engagement. One USAID/Niger staff noted that the technical working group “does not take into account the other partners who are not USAID.” e. Research BR has also been active in Niger, primarily through the Sahel Regional Office’s buy-in to support the RISE II projects with an evaluation of the integrated SBC program implemented by RFSA partners. This has involved considerable coordination among RFSA implementing partners to agree on key indicators to answer overarching research questions including: is integrated SBC programming more effective compared to control group areas without RISE II SBC interventions? BR has completed a literature review, developed the study protocol, received IRB approval, and completed qualitative data collection to assess study questions related to gender and social norms and household decision making. Quantitative data collection was delayed because of COVID-19. Local stakeholders have been engaged throughout this process: BA conducted and BR participated in two workshops in Sadore and another in Niamey to share and validate initial findings. In addition to activities in the RISE II zones, BR also conducted a mapping of relevant SBC indicators across Francophone West Africa, including in Niger, and has been leading the evaluation of the Merci Mon Héros digital campaign in four West African countries. Overall, while research activities have been met with some delays due to COVID-19 and the tenuous security situation limiting travel and engagement between BR staff and local stakeholders, BR has still managed to collaborate with a number of local actors to initiate substantial research activities. While getting all RFSA partners to agree on a set of measurement indicators was not straightforward, it was apparent that there were not the same tensions between BR and RFSA IPs as those faced by BA. This could speak to the advantage BR had in completing very specific, discrete research activities to accompany ongoing activities to BA’s more complex task to provide technical assistance and share innovation with partners already engaged in implementation plans. 2. Challenges in the Design and Implementation of Activities a. Misaligned timelines in RFSA support Challenges in supporting the RFSA projects have been rooted in the perception that activities are duplicative, and timelines between BA and RFSAs remain misaligned. A key informant from an RFSA partner noted: “Our projects started at the same time. We already have our strategies, our activities for behavior change. Because, basically, the [RFSA] projects are almost solely behavior change plus a few stand-alone water stations. But other than that, it’s behavior change. So, we already have a whole battery of activities and all that. They’re starting at the same time. Their mandate is to innovate and inspire us with their innovations. The problem is that if we start at the same time, it’s extremely difficult because we already have our action plans, we have planned things, we have our internal experts. In all honesty, those who are at BA don’t necessarily know much more than we do.” BA’s engagement with RFSA partners after they had already been active in implementation, with set work plans and objectives, frustrated RFSA IPs. RFSAs felt that BA was merely duplicating efforts and coming too late into the process of implementation. RFSA partners then questioned the value-added of BA and felt that BA’s request for information and engagement interfered with their ongoing work plans. MID-TERM PERFORMANCE EVALUATION OF BA: LOCAL LESSONS, GLOBAL LEARNING 110 This has been mitigated in developing a more common roadmap between partners, noting when BA can provide special support for a specified activity at a specified period. Creating this roadmap took considerable time and negotiation, however, as one BA partner noted. They stated that getting all partners to “eliminate any idea of suspicion” of each other was achieved by grounding everyone in a shared understanding that “it is one and the same team, promoting social and behavioral change.” b. Delayed implementation Other activities in Niger have also been delayed by the COVID-19 pandemic. Because of physical distancing guidelines, community engagement activities were delayed and not implemented according to initial work plans. This is especially relevant for implementation of the “community action cycle,” an innovative approach BA is introducing in Niger where communities identify barriers to healthy behaviors and collectively work on strategies to overcome them. While the community action cycle has been much discussed in workshops and trainings, the actual implementation at scale across RFSA IPs has been much slower than anticipated, first because RFSA partners were not ready to implement, and then due to COVID. One partner key informant explained frustrations with the delays: “…our field teams have gone around and around and around without being able to do it. Because the key steps in this community action cycle are to identify the problems at the community level, prioritize the problems, and then have a plan for the priority problems, a plan for solving these problems at the community level, so it’s been a long time coming, it’s only last week that we were able to have the community groups, the teams with whom we will discuss, identify these problems.” The delays have also frustrated some USAID staff in the Niger Mission, who note skepticism with the community action cycle (CAC) approach and activity. One USAID/Niger staff lamented that it “involves a lot of stakeholders, but for the moment I can’t speak about what I’ve seen – I don’t have any information, I don’t have a report that tells me here is what they did, here are the results.” c. Need for a more operational BA Multiple key informants from the partners in Niger and USAID/SRO noted that BA’s approach in the country has been overly top-down, and not responsive to actual gaps in resources and capacity among stakeholders. USAID/SRO noted that BA has been overly “theoretical” in Niger, and they would prefer if they had more staff on the ground to assist in actual community-level implementation. RFSA partners also expressed their need for a more operational BA. One noted, “…we want support and capacity building. One does not exclude the other. If we have the capacity building, it is not only to give knowledge to someone but also to follow him at the operational level to see if he is using what you have taught him.” Another RFSA partner stated, “we want to have BA here with us in terms of planning activities, in terms of documentation: We need information on how to evaluate behavior change.” Partners also expressed frustration that many of the approaches taught by BA are not relevant to the country context. One RFSA partner noted that “Especially in the beginning, we didn’t like that kind of relationship [with BA], quite honestly. Their suggestions could be interesting in some cases and in that case we adopted them, but if they didn’t seem relevant to us, then we could have the freedom to drop them.” Another RFSA partner key informant explained that BA would develop SBC approaches for dissemination without sufficient input from implementing partners on the ground in Niger. The partner felt that they had little flexibility to push-back against the project: “They said, ‘after two years, you’ll take over the pilot phase for the community action cycle.’ We accepted it because the donor required it, but we were really out of step with the planning and even with the approaches. And then they decided to conduct this approach on the community action cycle, they did not involve us. It’s a concept that came from BA, which was not discussed with the programs. Because if they had discussed it with MID-TERM PERFORMANCE EVALUATION OF BA: LOCAL LESSONS, GLOBAL LEARNING 111 us, we were going to explain to them that this is something that we are doing except that it’s the terminology that’s changing. So, it came out of the blue, we took it that way, we’re under pressure, the donor wants us to work with them.” 3. Advancing the Practice of SBC: Appreciation among Stakeholders Among MOH partners, there has been a true recognition of BA’s work and achievement in improving the understanding and practice of SBC in Niger. MOH staff especially noted BA’s establishment of the digital SBC catalog as a “major achievement.” The repository of SBC tools is held virtually and housed on the MOH’s website as a reference tool for stakeholders to design and implement SBC activities in Niger moving forward. Ministry of Water and Sanitation partners did note that while they appreciated BA’s involvement with the Ministry, they would like BA to be more involved in existing networks and policies, including providing input to the national strategy on access to clean water and hygiene, and take part in the common fund for water and sanitation that was created in 2018. The fund is a source for collaboration between UNICEF, Water Aid and other NGOs, and representatives from the Ministry expressed interest in BA becoming more engaged at these levels. Both malaria and FP partners noted appreciation for how BA has advanced their work in Niger in terms of capacity strengthening and centering HCD in implementation strategies. One partner key informant noted that “they have helped us in capacity building: they really helped us upgrade our communication strategy, really helped us to be up to date, to innovate in our communication strategies through training and capacity building. They also helped us to have harmonized messages.” Another noted the novelty of not just the “terminology” BA has used, but also “this way of organizing, of fully involving the community, informing them, letting them do it themselves and following them. And that was what BA was doing.” BA staff also remarked seeing this shift among stakeholders in Niger: “they know that SBC requires a much more rigorous approach, focused on the analysis of behavior and how to remove barriers to the adoption of these behaviors.” They noted how capacity strengthening workshops have “motivated” stakeholders to learn more and refine traditional IEC approaches to include more social determinants of behavior in the design and implementation of SBC activities. 4. Collaboration between BA and BR There has been less direct collaboration between BA and BR in Niger as in other countries with more significant buy-ins and joint activities, but notable areas of collaboration include the RISE II evaluation of integrated SBC and monitoring of the Merci Mon Héros campaign. In both areas, projects have worked together to ensure evaluation activities reflect implementation, and monitoring indicators are agreed on across a wide range of partners. BR’s work in monitoring Merci Mon Héros has helped refine the implementation, and the forthcoming results of the integrated evaluation in RISE II zones will help improve the understanding of how to execute integrated SBC in an effective manner. BA/BR Recommendations for Niger ● Continue to work on a common roadmap between BA and RFSA IPs to ensure work between partners is most collaborative and effective: local BA and RFSA staff in RISE II zones should collectively identify what the actual gaps in RFSA capacity are, and how BA can help fill them. ● USAID should continue regular check in meetings to coordinate BA and RFSAs to improve collaboration among partners. ● Improve and reinforce BA staff presence in Niger, especially in Maradi and Zinder. ● Improve coordination between buy-ins, share successes from WABA and malaria work with staff working with RISE II partners: the SRO/PMI COP and WABA COP should have regular MID-TERM PERFORMANCE EVALUATION OF BA: LOCAL LESSONS, GLOBAL LEARNING 112 communication to share lessons learnt, particularly on overcoming challenges in the RISE II zones (WABA has also provided TA on FP to DFSA partners). ● Expand Technical Working Group engagement to include BR, and with other partners outside USAID. ● Establish a more formal collaboration between BA and the Ministry of Water and Sanitation (e.g., signing a Memorandum of Understanding), ensuring BA’s actions are included in the national strategy for access to drinking water and sanitation. ● Formalize and structure the collaboration with the central level (design) for a better implementation of actions in the field (operational level in communities). ● Have a clear dissemination and utilization plan for the results of BR’s evaluation of integrated SBC in the RISE II zones. ● Improve the documentation around implementation and impact of the WABA work- including community dialogues, site walk-throughs, and digital campaigns- on FP behaviors. ● Improve the documentation on social and behavioral outcomes achieved through the implementation of innovative SBC approaches implemented in Niger, including HCD and the CAC, and the benefits of these approaches beyond traditional IEC. ● Disseminate key lessons learned and outcomes of capacity strengthening successes with malaria partners in Niger. MID-TERM PERFORMANCE EVALUATION OF BA: LOCAL LESSONS, GLOBAL LEARNING 113 ANNEX 6: CÔTE D'IVOIRE COUNTRY SUMMARY INTRODUCTION AND BACKGROUND BA started its activities in Côte d’Ivoire in July 2017 and is currently implementing four programs in the country, addressing a range of health areas. Total funding for these activities is $6.2 million/year and the funding sources are: PEPFAR, PMI, FP, MCH, Nutrition, and WASH. The activity areas are: a. Malaria • Awareness raising/behavior change promotion through the development and production of materials (posters, kakemono, audio, video) • Implementation of a BCC strategy through the development of a National Strategic Plan for SBCC and BCC, the development of an SBC message guide based on the Malaria Behavior Survey study • Capacity building of national partners through SBC training of 25 staff members from different state government structures b. HIV/AIDS • SBC sensitization, especially of vulnerable and hard-to-reach segments of the population, in order to improve their knowledge of HIV and promote adherence to HIV testing • Promotion of self-testing as a means of screening this segment of the population • Care and support for people living with HIV c. The Global Health Security Agenda (GHSA)13/COVID-19—BA interventions addressing diseases with epidemic risk and the response to COVID-19 were made at the institutional level and revolved around five pillars: • Systems and planning • Coordination • Public communication campaigns • Community engagement • Perception and misconceptions management d. FP with WABA to support the Ouagadougou Partnership through three areas of intervention: • Coordinating and building local capacity for SBC • Improving communication and fostering community involvement • Reducing social barriers and encouraging parent/child communication on FP BR's interventions aimed to support BA in certain studies (e.g., MBS, BE Survey on healthcare providers, baseline study for the HIV self-testing pilot project) and also to ensure the M&E of certain activities, such as the Confiance Totale campaign. EVALUATION METHODOLOGY The global mid-term evaluations of BA and BR include three focus countries: Nigeria, Niger, and Côte d’Ivoire. The evaluation team reviewed project documents and developed a question guide (Annex 2) for the global evaluation that was also used in Côte d’Ivoire. Two local evaluation consultants familiar with in￾country USAID programs and processes carried out the data collection in Côte d’Ivoire. Prior to the active data collection phase, the evaluation team scheduled a BA briefing meeting on January 6, 2021 via videoconference. The two local evaluation consultants then set up an initial briefing with USAID in Abidjan on January 19, 2021 via videoconference with various USAID/Côte d’Ivoire staff members. The evaluation 13 The GHSA is a group of 60 countries, international organizations, NGOs, and private sector companies working together to achieve the vision of “a world safe and secure from global health threat posed by infectious diseases.” www.ghsagenda.org MID-TERM PERFORMANCE EVALUATION OF BA: LOCAL LESSONS, GLOBAL LEARNING 114 consultants interviewed a total of 19 people out of 21 scheduled key informants: four from USAID/Côte d’Ivoire, six from BA, three from the National Malaria Program of the Ministry of Health and Hygiene (Programme National de Lutte Contre le Paludisme or PNLP), three from the national HIV/AIDS program of the Ministry of Health and Hygiene (PNLS), one from Antimicrobial Resistance Technical Working Group (TWG), one from the Ministry of Animal and Fisheries Resources, and one from the National Institute of Public Hygiene. US-based evaluators also interviewed WABA staff who were familiar with BA/BR’s work in Côte d’Ivoire (see Annex 3). This annex presents a summary analysis of the information from the key informant interviews conducted in Côte d'Ivoire and some relevant data from interviews conducted outside the country. FINDINGS 1. Achievements in SBC Programming, Research, and Capacity Building BA and BR’s achievements in programming, research, and capacity building in Côte d'Ivoire fall within the following four sectors or main intervention areas: a. Malaria—through PMI and PNLP b. HIV/AIDS—through PEPFAR and PNLS c. Global Health—through GHSA, focused on global health security across the intersection of human and animal health to prevent infectious disease outbreaks and combat the growing threat of antimicrobial resistance d. FP with WABA—to support the Ouagadougou Partnership a. Malaria BA’s work in malaria was reported as one of its strongest successes. The project developed numerous communication tools and media products for a mass media campaign on malaria that reached almost 650,000 people in the listening area of 22 community radio stations (approximately 22 percent of overall audience of nearly 3 million listeners.) The project’s three major achievements on using SBC approaches to address the high burden of malaria in Côte d’Ivoire were: (1) Carrying out the Malaria Behavior Survey (MBS), (2) Using Behavioral Economics Strategies for Provider Behavior Change, and (3) Supporting the CAC and Women's Groups. MBS. BA implemented a benchmark MBS on malaria-related behaviors in Côte d'Ivoire. The project fielded the survey during the 2018 rainy season (September through November) in four zones across the country (North, South/Forest, Abidjan, and Central; see map). The project analyzed the results in coordination with the National Malaria Control Program. The survey goal was two-fold: to better understand the sociodemographic characteristics associated with malaria￾related behavioral outcomes, and to determine what the appropriate focus on SBC program activities should be. The MBS survey made it possible to understand the determinants of malaria-related behaviors, what drives or inhibits them, and the non-use of LLINs and intermittent preventive treatment for pregnant women. The effort was a significant success for BA in-country. After the successful pilot in Côte d’Ivoire, BA is now implementing the MBS in additional countries. The MBS allowed the PMI to share experiences with other countries and supported Côte d’Ivoire’s stakeholders to make improved, evidence-based decisions for SBC activities to combat this endemic disease across the country. The BA-piloted survey proved to be a useful decision-support tool helping the Ivorian government’s fight against malaria, which infects over MID-TERM PERFORMANCE EVALUATION OF BA: LOCAL LESSONS, GLOBAL LEARNING 115 3.5 million Ivorians every year.14 The data from the MBS also made it possible for BA to design the SBC key messages guides for malaria prevention in Côte d'Ivoire. These guides helped partners produce targeted, evidence-based communication media (e.g., posters, kakemono, video, radio messages, etc.) on malaria prevention and recommended behaviors. A BA key informant described how useful their MBS capacity development and collaboration with the government has been as follows: “Within BA, we’ve conducted studies and we use the findings to better orient our [SBC] strategies to improve our interventions. BA also organizes training workshops for government staff within the Ministry of Health and Public Hygiene. In particular, the MBS was very impactful in Côte d’Ivoire. This study has enabled us to provide the government with several SBC tools.” Overall, the MBS enabled the development of communication materials for the mass LLIN distribution campaign in Côte d'Ivoire. Finally, it is noteworthy to mention the success of BA’s malaria support to PNLP by the National Strategic Communication Plan for Social and Behavioral Change 2021–2025. For additional information, see: https://malariabehaviorsurvey.org/countries/Côte-divoire/. Using Behavioral Economics Strategies for Provider Behavior Change. The objective of this activity was to use a behavioral economics approach to design strategies that would help providers systematically counsel and administer three doses of preventative malaria treatment (sulfadoxine￾pyrimethamine or SP) during antenatal care (ANC).15 As the following illustration suggests, the BA team started by examining the quality of services by providers to pregnant women in five health districts in Côte d'Ivoire. With PMI, the project identified the following priority behaviors at five health centers in varied settings: 14 USAID (2017) U.S. President’s Malaria Initiative Cote d’Ivoire Malaria Operational Plan FY2018 and FY2019. Washington, DC: USADI. https://www.pmi.gov/docs/default-source/default-document-library/malaria-operational￾plans/fy19/fy-2019-cote-d'ivoire-malaria-operational-plan.pdf?sfvrsn=5 15 The WHO and the MOH require all pregnant clients receive at least one mosquito net to prevent malaria, three SP doses to prevent malaria infection and transmission from mother to baby, two HIV tests, as HIV exacerbates malaria mortality, and four touchpoints with a facility-based health provider, who can prescribe folic acid and iron supplements. (BARCI Design Strategy Deck, JHU CCP & USAID, 2019) MID-TERM PERFORMANCE EVALUATION OF BA: LOCAL LESSONS, GLOBAL LEARNING 116 Note: a. Midwives do not systematically initiate preventative malaria treatment with pregnant women during ANC visits b. Midwives do not systematically offer counselling on the importance of multiple preventative malaria treatments to pregnant women during ANC visits BA then looked at the underlying drivers of these behaviors, clustering them into five categories (time scarcity, responsibility and identity, lack of feedback, cognitive scarcity, and choice architecture) that were cross-validated by extensive research in similar health settings. The drivers were then subdivided into intention barriers (providers do not intend to offer testing, treatment, or counseling in the first place) and action barriers (providers intend to offer testing, treatment, or counseling but do not follow through on that intention). The project developed a theory of change (see figure) for the intervention and subsequently produced provider-client carnet inserts and certificates to promote adherence to WHO and MOH guidelines for prevention of malaria in pregnancy during ANC visits. A BA key informant noted how significant it was that this activity helped identify 223 women (out of a total of 263) that had not gotten the needed SP because they had missed their follow-up ANC visits. According to several key informants across BA and USAID, the project’s use of behavioral economics strategies provided a better understanding of the significant behavioral and social factors associated with case management and the use of SP. The research has made it possible to develop communication strategies that allow health workers to improve and administer high quality care for their patients, and for pregnant women to understand their own client rights and the quality of care they should benefit from at health facilities/service centers, as well as how they can adhere as closely as possible to the correct follow￾up. An important component of the activity involved participatory design, which contributes to local capacity development. The project included key stakeholders in analyzing the study data as well as in developing the strategies and messages and building capacity for coordinated, evidence-based decision- MID-TERM PERFORMANCE EVALUATION OF BA: LOCAL LESSONS, GLOBAL LEARNING 117 making. Key informants from all stakeholder groups noted the importance of the involvement and participation of community members around malaria issues and the partnerships with Impact Malaria and the PNLP that were strengthened through this process. A key informant from PMI who had praised BA for its “excellent collaboration experience” stressed how important it is that these excellent advances be sustained, “It is important that we talk about the sustainability of these actions and to do this, we must strengthen the capacities of national actors to ensure sustainability. Therefore, we need to make a capacity building plan with the PNLP agents, so that if the consultant is not there, we can do the work as it should be done.” Supporting the CAC and the establishment of community action groups and women's groups. The CAC is an innovative approach developed by the BA implementing partner, Save the Children. Using CAC, BA supported communities in Côte d’Ivoire to develop culturally-relevant action plans to improve health-seeking behavior. BA facilitated the creation and expansion of various community action committees and women’s groups to ensure local uptake of the behaviors of interest. The CAC approach (illustrated below) is based on the recognition that people do not change their behavior on the basis of information alone. Behavior change requires a combination of having relevant information as well as the confidence and an enabling environment for making positive choices, collectively and individually, “while addressing underlying social norms that ultimately leads to changed behaviors.”16 16 The Communication Initiative Network, March 2, 2020: https://www.comminit.com/global/content/save-childrens￾community-action-cycle-cac MID-TERM PERFORMANCE EVALUATION OF BA: LOCAL LESSONS, GLOBAL LEARNING 118 b. HIV/AIDS BA’s HIV/AIDS prevention work using various SBC approaches began under its predecessor project, HC3, which aimed to strengthen the capacity of local groups to implement SBCC programs supporting HIV/AIDS prevention, care, and treatment. HC3 improved the effectiveness of SBCC in promoting HIV service uptake, ART adherence, community-based care and support, as well as healthy social norms. Building on HC3’s work preventing HIV/AIDS in Côte d’Ivoire, BA has developed various learning platforms and targeted BCC strategies. They have also expanded support for HIV/AIDS prevention and awareness programs started under HC3, including "Super Go"—an SBC campaign targeting young females 15-29 years old and "Super TaTa"—a community-based SBC campaign designed to engage Ivoirian adult women in community-based prevention and healthcare-seeking activities. BA worked in partnership with USAID and the Government of Côte d’Ivoire to prepare a community engagement guide to support the implementation of both programs. BA also supported the successful expansion of the “Brothers for Life” HIV/AIDS behavior change campaign for the population segment of males 25 years and older. A BA key informant noted the importance of the Super Go and Super TaTa SBC campaigns to effectively reach young people and adult women: “We used segmentation in those programs. We have to work with youth and 40-year-old women. It must be said that the two populations do not have the same needs, so we used the community action cycle to [prepare relevant SBC approaches.]” Within the current award’s HIV/AIDS prevention programming, BA has been particularly successful at identifying positive cases among the three population segments and increasing community interest in HIV/AIDS screening (to identify positive cases) and utilization of a self-administered test. Identifying positive cases and increasing community interest in HIV/AIDS screening and self￾testing. In 2018, BA developed and implemented its strategy to apply SBC to identify people living with HIV/AIDS. At the time, there were still major challenges in identifying HIV positive cases in Côte d'Ivoire, especially in vulnerable segments of the population (i.e., men aged 25 and older, adolescent girls, and young women). As a result, the BA SBC strategy aimed to “promote HIV prevention behaviors [and foster] positive social norms that encourage healthy behaviors” for the identification of positive cases of HIV/AIDS and for searching for those lost to follow-up. Specifically, the strategy aimed to seek out people living with HIV wherever needed (in camps, prayer camps, large companies, etc.), distribute self-tests, search for those lost to follow-up, and then support community distribution of antiretrovirals (ARVs) by BA-trained peer leaders. The distribution of the HIV/AIDS self-administered tests was based on a pilot study conducted by BA, which provided an understanding of how to better approach the distribution of self-administered tests in the community. By the beginning of 2021, the project had distributed more than 6,000 self-testing kits. Overall, BA carried out various SBC interventions on sensitization and screening for HIV/AIDS, treatment initiation, and treatment retention focused on searching for those lost to follow up. According to data provided by the BA team from January 2021, since the beginning of the project, BA has supported HIV testing for 119,570 people, and helped 5,566 people living with HIV to know their status and get linked to MID-TERM PERFORMANCE EVALUATION OF BA: LOCAL LESSONS, GLOBAL LEARNING 119 treatment, as well as getting 7,730 people lost to follow-up back into treatment. These activities were carried out to help the government of Côte d'Ivoire achieve the country’s 90-90-90 objectives of: • 90 percent of people living with HIV know their status • 90 percent of all people tested for HIV receive sustained ARV therapy • 90 percent of people receiving ARV treatment have a permanently suppressed viral load A USAID key informant cited that “the quality of the services offered, the community distribution of ARVs and awareness-raising are [BA] actions that impact the lives of clients.” In addition to the testing and treatment efforts, the project developed a related mass media campaign that sent out more than 1,200 radio spots/commercials, 40 radio programs, and 128 social media posts. c. Global Health Security Agenda BA achievements in programming, research, and capacity building for global health have yielded positive results in terms of a “One Health” approach (human health, animal health, and the environment) and the creation of the TWG for Communication on Health Risks. BA’s first global health success concerned Côte d'Ivoire’s joining the GHSA and helping the country develop its first National Health Security Plan, in accordance with the WHO’s International Health Regulations. The next phase of GHSA focuses on GHSA 2024, the overarching framework that lays out a strategic approach for addressing gaps and priorities in global health security, including international human and animal health standards. This is particularly important and timely given the current concerns over the COVID-19 pandemic. BA’s technical assistance for GHSA is primarily at the national level. Other successes for BA’s GHSA work in Côte d’Ivoire include: • Facilitating collaboration across all the technical ministries, • Developing a public SBC communication axis and communication guide for journalists, • Carrying out a study on priority zoonoses under the One Health” approach, and • Developing a rumor and misconceptions management/tracking system using DHIS-2. Previously, with support from the US headquarters, the BA Côte d’Ivoire team had developed a training module on strategic communications for journalists on One Health and Risk Communication with focus on Ebola in order to illustrate the larger concepts. This module that aimed to provide support for journalists to cover the Ebola epidemic accurately and without bias, has now been adapted for use in journalistic reporting for COVID-19. An IP key informant involved with BA’s journalistic efforts repeatedly mentioned their appreciation for the project’s support and noted that, “We are one of the TWGs that has really started and has had convincing results and the BA project has contributed to the achievement of these results.” As part of GHSA’s work, BA/Côte d’Ivoire developed a highly successful rumor and misconceptions management tracking system using DHIS-2 software. This was set up prior to COVID-19 as part of the GHSA work, and BA was able to pivot successfully and rapidly to address COVID-19. This novel approach makes it possible to capture all rumors or false narratives across the country and address them Operators answering calls on one of Cote d’Ivoire’s helplines through which officials are collecting and addressing COVID-19-related rumors. MID-TERM PERFORMANCE EVALUATION OF BA: LOCAL LESSONS, GLOBAL LEARNING 120 directly. With this rumor tracking system, the team collects and then uses the misinformation from the general public to adapt and strengthen evidence-based communication during public health crises. For this effort, BA works in collaboration with the 143 Hotline and the risk communication TWG of the National Institute of Public Hygiene. For more information, see https://ccp.jhu.edu/2020/06/29/Côte-divoire-call￾center-tackles-rumors/ d. FP with WABA WABA is implemented in Côte d'Ivoire in five health districts. Its successes in both institutional and community-based areas are included in three lines of intervention. Axis 1 level: Coordinate and empower actors at the government level BA has also established a TWG on FP at this level, which meets quarterly to take stock. According to key informants from all stakeholder groups, BA has strengthened the SBC capacities of the district management teams in the targeted districts and improved the quality of the FP services across the country. Axis 2 level: Improve communication and encourage community involvement BA used community dialogue to identify the problems of low utilization of MCH services in Côte d’Ivoire. This activity is coupled with an engagement tour of health centers for influential members of the community who participated in the community dialogue, to improve understanding of and interest in maternal and child health services. This interaction between providers and communities aims to create local demand for and improve the use of maternal and child health services. A WABA key informant remarked that “Community dialogues have been wonderful activities to really see shifts in community and providers’ attitudes and behaviors—and really seeing what the real needs are in the community, what is important to them—this has been extremely important to me.” According to several key informants, an example of the success of these actions is the establishment of a community mobilization committee at the Bouaké Nord-Ouest health district, specifically in the locality of Abokouamékro, for the rehabilitation of their health center. Axis 3 level: Reduce social barriers and the adoption of FP by young people through the digital campaign Merci Mon Héros (MMH), which makes it possible to break down social barriers (taboos) and encourages communication about RH/FP between parents and children. A BA key informant mentioned the importance of the MMH activity to build community with the young people and also that the tailored MMH activity helped overcome “challenges by really listening and understanding what misconceptions the youth had. We continued to educate them about FP/RH, myths and misconceptions.” A WABA key informant said that MMH seems to really be making a difference, that they have “really seen a real change in behavior, attitudes among youth—seeing the impact immediately with communication between parents and youth has been really remarkable! Facebook really facilitated this, and it’s been incredible to see.” While noting the success of the MMH activity, another IP key informant mentioned the importance of “continuing the financing of MMH, especially a caravan in rural areas, to cover all the zones in areas where digital access isn’t available. There is a true need, and we need to continue this kind of work.” Many key informants noted the challenges of accessing rural youth for these activities and the “immense challenges” and “true need” to continue this kind of intervention. While the MMH campaign received many accolades, key informants were concerned about its sustainability. As one BA participant said, MID-TERM PERFORMANCE EVALUATION OF BA: LOCAL LESSONS, GLOBAL LEARNING 121 “MMH was a great activity, loved the approach and how close it was with the community, but we need a longer term tool that can be sustainable. Sustainability is a huge problem! The dialogue was really, really helpful in getting new attitudes, new questions and ideas around sexual and reproductive health, but what now? We have to think about what comes next. We think about FP/RH work for married women, and not for youth, and really investing in FP/RH for services for youth—this is a problem for them. They already have children! They have many questions. We need to work with them further—we need to make sustainable, long term solutions… perhaps the free HIV hotline… I really see the change in this group and the impact it [MMH] had on their lives.” 2. Design and Implementation of Activities BA conducted project mapping with the staff in three major projects: the malaria component with the PMI, the HIV/AIDS component with PEPFAR, and the Global Health component with the GHSA/COVID￾19. For each project, BA involved the relevant national representatives. For the malaria component, BA worked in conjunction with PNLP. Several key informants from the government and USAID/Côte D’Ivoire mentioned that BA was very inclusive of relevant government counterparts and that there was “very good collaboration” during the development of different BA SBC plans and strategies. One IP participant said, “We participate, so that our activities are in their plan and that they can be funded [by BA].” So, while BA got very high marks across key informants from all stakeholder group and levels for the design and implementation of project activities, several key informants had concerns about funding for sustainability. A national government key informant said, “We need a public health communicator, so that they can transfer their skills to the actors who intervene in the field in order to boost behavior change in the community. In Côte d’Ivoire, there are not enough public health communicators to build the capacity of community health workers, Women’s Groups, and Community Action Groups. And it is necessary to strengthen the financial resources in communication because communication is necessary, and it is expensive.” As for HIV/AIDS, BA's interventions are linked to PNLS, BA’s country institutional partner. At the level of the GHSA project, BA's intervention is primarily strategic. Support is limited to the institutional level and does not extend to the community level. While noting the strength of BA’s GHSA activities, a key informant from an IP reiterated what other participants had expressed about concern for funding, saying, “The problem is that our activities are not programmed in their action plan and this makes BA unable to finance our activities. During the elaboration of BA’s action plan, we have to participate, we have to be together so that our activities are in their plan so that they can be financed.” BA also worked closely with WABA on FP program design and implementation, which many key informants deemed a success. A BA key informant shared the following related to that effort: “…the strategy of really doing deep community work—to really see the real needs and impressions of FP users and healthcare providers—this is really quite new, to work at this very local level and to hear the community—this is a huge success for BA… The point of view of the community was central, and their input did influence service provision for FP—the perceptions as well as the community members changed—about FP/RH—the barriers between clinic/population started to go away! They had so many bad ideas and attitudes about FP/RH, but that really changed and changed the behaviors of care-seeking —even saying ‘no! the clinic is not like that!’…” In short, there is effective stakeholder participation and involvement in the implementation of all BA projects in Côte d'Ivoire. MID-TERM PERFORMANCE EVALUATION OF BA: LOCAL LESSONS, GLOBAL LEARNING 122 3. Advancing the Practice of SBC The appreciation of the practice of SBC is a reality in Côte d'Ivoire in the fight against malaria and in the prevention, screening, and treatment of HIV/AIDS. It allowed the different programs to determine their real communication needs, to adjust communication strategies in the fight against these diseases for a better impact of interventions in the communities. Today, thanks to BA, SBC has generated real interest among national partners (e.g., the FP consortium's request for capacity building in SBC, and more consultants being hired by the MOH to help with the SBC work) and communication resources and/or budgets of the stakeholders involved have increased due to the innovative strategies developed over time. A government key informant noted that “For SBC, BA has increased interest because there was the installation of the Community Action Groups in health districts where there were none before. And this now allows the community itself to deal with its health problems.” According to a few key informants, a “boost in interest” for SBC was also attributed to the work with the Alliance des Religieux contre le Sida et les autres pandémies (the Religious Alliance against AIDS and other pandemics). A WABA key informant noted that the SBC work in Côte d’Ivoire included a TWG on SBC for FP that: “improved the communication and quality of FP services in the country” noting that “the MOH has integrated SBC in its programming and materials and tools—and all Côte d’Ivoire districts have integrated SBC activities in their health action plans…” Such high-level attention and programming are significant factors for advancing the practice of SBC in the region. Unfortunately, while SBC is gaining attention, several key informants felt that the country lacks the technical capacity to do the relative communication component effectively. An IP key informant said, “As far as communication is concerned, we are the last. We don’t know how to do it. We do a lot of things, but we do it badly; this is a weakness that Côte d’Ivoire has. So, let BA support us to get out of our shame.” 4. Collaboration between BA and BR The collaboration between BA and BR is ongoing and strong, but coordination between them could be improved. Key informants noted that many BA activities begin with a study or research initiated in connection with BR. BA’s research team develops effective communication strategies such as the MBS survey for malaria and the pilot study on the self-test for HIV screening. BR does the monitoring of the MMH campaign and WABA through social listening, which provides regular data allowing a good implementation of the digital campaign. A WABA key informant explained: “BR helped with the evaluation and sharing of lessons from MMH—BA developed the campaign and put it out through Facebook, Instagram—and BR monitored how the campaign was received among youth, and helped clarify and improve BA’s implementation strategy. The social media listening has been wonderful, and so helpful—we changed the logo, made improvements to the implementation based on BR’s findings.” In general, key informants were far more familiar with BA than BR, and praised BA for their “flexibility” and “highly responsive” efforts. For example, an MOH key informant said, “When we have needs and call on BA, they respond favorably… We have a great collaboration with BA…. It is in the collaboration that the procedures have been understood.” When asked what they thought contributed to the implementation’s progress, the MOH participant replied, “On both sides, I think it was trust.” MID-TERM PERFORMANCE EVALUATION OF BA: LOCAL LESSONS, GLOBAL LEARNING 123 In the context of COVID-19, BR is helping BA and the government institutions to ensure they have up￾to-date data and information on the pandemic and can effectively address the rumors that have developed around the pandemic. BA/BR RECOMMENDATIONS FOR COTE D'IVOIRE These recommendations for future SBC interventions, like BA, come from direct feedback from key informants. • Continue and increase community-led initiatives for HIV/AIDS self-administered testing • Support the HIV/AIDS hotline in Côte d’Ivoire so youth can continue to have their needs addressed • Support the development of SBC strategies of national partners (from the development of SBC approaches and media to the dissemination) • Continue and strengthen BA's interventions in the area of GHSA • Maintain and strengthen collaboration with the Ivorian national party in the fight against malaria • Develop an advocacy plan so that business leaders are included in the fight against malaria • Maintain and strengthen the interest of national partners for SBC • Focus on capacity building and continue to transfer skills to the national counterparts • Support national BA partners to use of the "Flow Chart" in their interventions • Make sure communication messages are produced in local languages Further, national partners could be better supported to: • Integrate SBC into the different programs • Build their capacity so they feel they “own’ the SBC approaches • Encourage user feedback of the “Flow Chart” for its improvement MID-TERM PERFORMANCE EVALUATION OF BA: LOCAL LESSONS, GLOBAL LEARNING 124 ANNEX 7: DISCLOSURE OF ANY CONFLICTS OF INTEREST MID-TERM PERFORMANCE EVALUATION OF BA: LOCAL LESSONS, GLOBAL LEARNING 125 MID-TERM PERFORMANCE EVALUATION OF BA: LOCAL LESSONS, GLOBAL LEARNING 126 MID-TERM PERFORMANCE EVALUATION OF BA: LOCAL LESSONS, GLOBAL LEARNING 127 MID-TERM PERFORMANCE EVALUATION OF BA: LOCAL LESSONS, GLOBAL LEARNING 128 Local Nigeria Evaluator MID-TERM PERFORMANCE EVALUATION OF BA: LOCAL LESSONS, GLOBAL LEARNING 129 Breakthrough ACTION/Breakthrough Research MID-TERM PERFORMANCE EVALUATION OF BA: LOCAL LESSONS, GLOBAL LEARNING 130 MID-TERM PERFORMANCE EVALUATION OF BA: LOCAL LESSONS, GLOBAL LEARNING 131 MID-TERM PERFORMANCE EVALUATION OF BA: LOCAL LESSONS, GLOBAL LEARNING 132 MID-TERM PERFORMANCE EVALUATION OF BA: LOCAL LESSONS, GLOBAL LEARNING 133 MID-TERM PERFORMANCE EVALUATION OF BA: LOCAL LESSONS, GLOBAL LEARNING 134 ANNEX 8: EVALUATION TEAM MEMBERS Julie Solo, MPH – Team Lead With more than 25 years of experience in global reproductive health, Julie Solo has been working as an independent consultant for 20 years on a wide range of assignments, including conducting evaluations, developing strategies, and writing documents for donors and implementing agencies. She served as the team leader for the Breakthrough ACTION and Breakthrough RESEARCH evaluation team. After getting her MPH in Population Planning and International Health at the University of Michigan School of Public Health, Julie worked with the Population Council in Kenya for several years, conducting operations research on many topics in reproductive health, including family planning services, postabortion care, and integration of STI/HIV services with family planning. As a consultant, Julie has extensive experience as a team leader, including leading evaluations for USAID for global projects, such as: USAID’s flagship implementation science project; a research project on transforming social norms; and an assessment of research utilization efforts. She has also led multiple evaluations for the Bill and Melinda Gates Foundation, including Advance Family Planning, a global family planning advocacy project and the International Conference on Family Planning. Julianne Weis, MSc, PhD – Evaluation Specialist Julianne Weis holds an MSc and PhD from Oxford University, where she studied the history of global reproductive, maternal, and child health policy in the context of local norms and community practices in African countries. She has worked for 15 years in researching and evaluating reproductive, maternal, newborn and child health programs in Africa and South America and is now a Senior Social Science Advisor in the Office of Population and Reproductive Health at USAID. For the evaluation, Julianne worked as the Evaluation Specialist, building on her background in operations research and M&E. As a French speaker, she also assisted with data collection and translation of tools into French and worked closely with the Niger evaluation team to complete analysis for the Niger country summary. Lynda Bardfield – Social and Behavior Change (SBC) Specialist Lynda Bardfield, SBC Technical Specialist on the Breakthrough Action-Breakthrough Research mid-term evaluation team, brings more than two decades of award-winning creative, SBC, social marketing, and international development experience. After leaving a career in the private sector as a senior creative executive for a number of multinational advertising firms, Lynda crossed over from commercial to social marketing and dedicated herself to applying private sector marketing principles to SBC. Lynda has held leadership positions at AED, the American Institutes for Research, and FHI360, where she was Associate Director for SBC, leading a global team of behavioral scientists and communication professionals to address infectious and chronic disease challenges. A native English speaker and fluent in Spanish and Portuguese, this former Peace Corps volunteer’s career has taken her to more than 40 countries. When she is not working with clients to translate audience research into creative strategy, or directing radio, TV, and social media, Lynda is an Adjunct Professor at Tufts University School of Medicine where she teaches MPH students. Team Lead on two previous USAID evaluations—he Health Communication Capacity Collaborative (HC3) Nepal Project and Uganda’s Communication for Healthy Communities (CHC), an integrated Social and Behavior Change Communication (SBCC) Project— she enjoys collaborating with evaluation professionals to take a closer look at SBCC project performance. She continues to keep one foot in domestic social marketing and another in SBC internationally, enabling her to apply lessons learned across projects and understand the big picture and the many tools available to influence social and behavior change. Alexandria Schmall, MDPH – SBC Specialist with Capacity Strengthening Expertise Within the GH-EvaLS Breakthrough Action/Breakthrough Research evaluation team, Alexandria Schmall is the SBC Specialist (Capacity Strengthening). Alexandria is a trained public health nutritionist and MID-TERM PERFORMANCE EVALUATION OF BA: LOCAL LESSONS, GLOBAL LEARNING 135 behavioral scientist with over a decade of program, policy, and mixed-methods research experience in areas including international and maternal-child nutrition; food systems; SBC approaches; food security; capacity strengthening, inclusive development; and diversity, equity, and inclusion. Having worked in more than30 countries across diverse global development and humanitarian contexts, Alexandria focuses her work on multi-sectoral approaches to improve nutrition and health outcomes among vulnerable populations, including women, youth, and young children. She received her dual Bachelor’s degrees in International Agriculture and Rural Development and Development Sociology from Cornell University, her MPH degree in International Health and Human Nutrition from the Johns Hopkins Bloomberg School of Public Health, and holds doctoral candidacy in Nutrition Science and Food Policy at the Gerald J. and Dorothy R. Friedman School of Nutrition Science and Policy at Tufts University. Throughout her career, Alexandria has held various positions with the United Nations World Food Programme, Cultivating New Frontiers in Agriculture, USAID Bureau for Food Security, UNICEF, and other nutrition and food security￾focused organizations. She is fluent in Spanish and French. Willow Gerber, MSc – Research Utilization and Knowledge Management Specialist Willow Gerber has over 25 years of experience in global health and development, including technical and administrative management of reproductive health and family planning programs. She is also an expert trainer who has designed and carried out learning activities and workshops around the world. She has specialized experience in social and behavior change; gender and equity; research utilization; knowledge management (KM) and learning; leadership, management, and governance; and digital health. She has worked extensively in organizational capacity building and proposal development and was the Chair of the USAID-supported Global Health Knowledge Collaborative for many years. Her graduate work in sociology at the University of Wisconsin, Madison, focused on access to reproductive health in rural settings. She has worked for a number of development organizations, as well as the National Academy of Sciences and The Aspen Institute. Willow was born in Papua New Guinea and grew up in Quebec, Canada. For the Breakthrough ACTION and Breakthrough RESEARCH mid-term evaluation, she brought her experience and skill set in research utilization, KM, and leadership and governance to bear. She also provided translation support for the team and helped lead the Côte d’Ivoire data collection and analysis. Opeyemi Adeosun, DVM, MPH – Nigeria Evaluation Specialist Dr. Opeyemi Aanuoluwapo Adeosun brings 11 years of experience in project management, monitoring development programs, impact assessment, and conducting qualitative and quantitative research. His expertise includes design and management of large-scale health, nutrition, MNCH and water, sanitation, and hygiene (WASH) programs, data management, governance, strategic planning, supply chain, and Knowledge Management Systems. He has conducted key informant interviews with top government and civil society officials and focus group discussions with key influencers and community leaders. His expertise includes coordination and analysis of qualitative research. He has worked across the six geopolitical zones of Nigeria interfacing with state actors in development and community members. He is experienced in Health Management Information System, Operations Research, and detailed report-writing. He has worked closely with the Government of Nigeria and other implementing partners, including the Global Alliance for Improved Nutrition and UNICEF. Dr. Adeosun has received a Doctor of Veterinary Medicine degree and MPH in Field Epidemiology from the University of Ibadan in Nigeria. He contributed to this evaluation and report as the Evaluation Specialist for Nigeria. He reviewed project documents and interviewed a wide range of stakeholders—project team, government officials, and other implementing partners. Dr. Adeosun wrote the summary report on the key informant interviews and joined other members of the team to finalize the evaluation report. MID-TERM PERFORMANCE EVALUATION OF BA: LOCAL LESSONS, GLOBAL LEARNING 136 Emmanuel Ogbudu, MSc – Nigeria Evaluation Coordinator Emmanuel Ogbudu is a monitoring & evaluation expert with eight years of experience providing technical expertise to donor-funded projects. He has Bachelor's and Master's degrees in psychology, and certification in project management with APMG. He has experience leading and supporting the design and implementation of program evaluation and Monitoring, Evaluation, Accountability and Learning (MEAL) systems that incorporated multiple sectors such as developing sustainable programs designed to improve health, nutrition, agriculture/livelihood; WASH; gender and protection; maternal and child healthcare service delivery; and governance and peace￾building projects at international, national, state, and local level. He has worked in Nigeria and Ethiopia on a variety of projects funded by the United Kingdom Department for International Development (DFID), USAID, Ford Foundation, Global Fund, European Union, and Global Affairs Canada. Aboubacar Souley, DEA – Niger Evaluation Specialist Aboubacar Souley is an independent consultant and researcher. He is a socio-anthropologist, with a degree in administrative management techniques, and also trained in results-based management, gender, and WASH. His skills cover MEAL processes (including project identification and formulation, baseline, and impact studies), communication, and local development action research. Working for 20 years on development issues, he has a perfect mastery of social dynamics and local development in Niger and in the sub-region. He has a long experience built on the basis of studies and research but also as a development operator. Through his studies and research, he has specialized in two fields of work: resilience in the face of social and environmental change and conflicts and security challenges in a vulnerable economic, political, and social context. Mahamane Tahirou Ali Bako, DEA, PhD – Niger Evaluation Coordinator Kacou Armand Tokou, MD – Côte d’Ivoire Evaluation Specialist Kacou Armand Tokou, Local Evaluator of Breakthrough ACTION and Breakthrough RESEARCH performance evaluation in Côte d’Ivoire, is a medical doctor by training and international public health specialist. Kacou Armand Tokou is Deputy Director of multi-sectoral and health promotion at the Community Health Department in Abidjan. He has 10 years of experience in supervision, coordination, and evaluation in health projects at ASAPSU NGO, FENOSCI, and in the public service. Djedou Martin Amalaman, PhD – Côte d’Ivoire Evaluation Coordinator Dr. Amalaman is the Local Evaluation Coordinator of Breakthrough ACTION and Breakthrough RESEARCH Performance Evaluation in Côte d'Ivoire. A socio-anthropologist by training, Dr. Amalaman is a research professor at the Peleforo GON Coulibaly University of Korhogo. He is also the Director of the Research Unit of the NGO ASAPSU in Côte d'Ivoire. A specialist in cultural and public health issues (HIV/AIDS, Ebola, rabies, etc.), he has nearly 15 years of experience in research, management, and operational and logistical assistance. As part of this performance evaluation study of Breakthrough ACTION and RESEARCH projects in Côte d'Ivoire, he prepared the ground and facilitated the data collection from around 20 participants from the following institutions: USAID-Côte d 'Ivoire, BA- Côte d 'Ivoire, PMI/PNLP, PEPFAR/PNLS, GHSA￾COVID-19/ (DSV, INHP, RAM).