Midterm Evaluation of USAID Tulonge Afya Project AUGUST 2020 TECHNICAL REPORT Breakthrough RESEARCH is made possible by the generous support of the American people through the United States Agency for International Development (USAID) under the terms of cooperative agreement no. AID-OAA-A-17-00018. The contents of this document are the sole responsibility of the Breakthrough RESEARCH and Population Council and do not necessarily reflect the views of USAID or the United States Government. The Population Council confronts critical health and development issues— from stopping the spread of HIV to improving reproductive health and ensuring that young people lead full and productive lives. Through bio￾medical, social science and public health research in about 50 countries, the Council works with our partners to deliver solutions that lead to more effective policies, programs, and technologies to improve lives worldwide. Established in 1952 and headquartered in New York, the Council is a nongovernmental, nonprofit organization with an international board of trustees. Breakthrough RESEARCH catalyzes social and behavior change (SBC) by conducting state-of-the-art research and evaluation and promoting evidence-based solutions to improve health and development programs around the world. Breakthrough RESEARCH is a consortium led by the Population Council in partnership with Avenir Health, ideas42, Institute for Reproductive Health at Georgetown University, Population Reference Bureau, and Tulane University. ©2020 The Population Council. All rights reserved. Cover photo by ©USAID Suggested Citation Kahabuka, Catherine, Salum Mshamu, Nrupa Jani, and Kamden Hoffmann. 2020. "Midterm evaluation of the USAID Tulonge Afya Project," Breakthrough RESEARCH Technical Report. Washington, DC: Population Council. Contact 4301 Connecticut Avenue NW, Suite 280 | Washington, DC 20008 +1 202 237 9400 | BreakthroughResearch@popcouncil.org breakthroughactionandresearch.org Acknowledgments The authors would like to thank Naomi Serbantez, Jane Schueller, Linda Madeleka, Thibaut Williams, Aurora Amoah, Yeronimo Mlawa, Emmanuel Tluway (USAID/Tanzania); Hope Hempstone, Angela Brasington, Joan Kraft, Jacqueline Devine (USAID/W); Waziri Nyoni, Kara Tureski, Prisca Rwezahura, Joseph Msofe, Frank Rweikiza, Jacqueline Moller Larsen (USAID Tulonge Afya/FHI 360); and Laura Reichenbach, Paul C. Hewett, Lou Apicella (Breakthrough RESEARCH), who provided valuable feedback during the development of this report. Midterm Evaluation of USAID Tulonge Afya Project TECHNICAL REPORT AUGUST 2020 Catherine Kahabuka1 Salum Mshamu1 Nrupa Jani2 Kamden Hoffmann2 1 CSK Research Solutions 2 Population Council BREAKTHROUGH RESEARCH | AUGUST 2020 i ADDED Audience-driven demand, design, and delivery AMEP Activity monitoring and evaluation plan ANC Antenatal care ART Antiretroviral therapy CCHP Comprehensive Council Health Plan CHF Community Health Fund CHW Community health worker CHSD Comprehensive health service delivery CSO Civil society organization CV Community volunteer DHIS District Health Information System FGD Focus group discussion FP Family planning FY Fiscal year GOT Government of Tanzania HCD Human-centered design HP Health promotion HPS Health Promotion Section IDI In-depth interview IEC Information, education, and communication IP Implementing partner IPC Interpersonal communication IPTp Intermittent preventive treatment in pregnancy IR Intermediate result ITN Insecticide treated net KAP Knowledge, attitudes, and practices KII Key informant interview KVP Key vulnerable populations MCH Maternal and child health MCM Modern contraceptive method MOHCDGEC Ministry of Health, Community Development, Gender, Elderly, and Children MOH Ministry of Health NGO Nongovernmental organization NIMR National Institute for Medical Research PLHIV People living with HIV PMTCT Prevention of mother-to-child transmission of HIV PNC Postnatal care PORALG President’s Office—Regional Administration and Local Government PPFP Postpartum family planning RH Reproductive health RHPCO Regional Health Promotion Coordinator SBCC Social and behavior change communication SOP Standard operating procedure SRH Sexual and reproductive health TB Tuberculosis TCDC Tanzania Communication and Development Center T-MARC Tanzania Marketing and Communications TFR Total fertility rate TOT Training of trainers TWG Technical working groups UBA USAID Boresha Afya UCD User-centered design USAID United States Agency for International Development USG United States Government VMMC Voluntary medical male circumcision VEO Village executive officer WEO Ward executive officer Acronyms ii MIDTERM EVALUATION OF USAID TULONGE AFYA PROJECT Table of Contents Acronyms . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . ii Executive Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5 Contextual background . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5 Description of the USAID Tulonge Afya Project . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5 USAID Boresha Afya . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8 Methods . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 12 Evaluation purpose, questions, and research design . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 12 Evaluation questions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 12 Research design, study participants, and sample size . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 13 Geographic setting of the evaluation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 13 Data analysis and management . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 14 Data management . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 14 Ethical considerations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 14 Findings . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 16 IR 1: Evaluation question 1a . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 16 IR 1: Evaluation question 1b . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 18 IR 2: Evaluation question 2a . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 19 IR 2: Evaluation question 2b............................................................................19 IR 3: Evaluation question 3 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 21 Cross-cutting: Question 4 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 24 Cross-cutting: Question 5 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 25 Cross-cutting: Question 6 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 25 Cross-cutting: Question 7 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 26 Cross-cutting: Question 8 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 27 Recommendations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 29 IR 1: Improved ability of individuals to practice healthy behaviors . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 29 IR 2: Strengthened community support for healthy behaviors . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 31 IR 3: Improved systems for coordination and implementation of SBCC interventions . . . . . . . . . . . . . . . . . . . . . . . . . . . 32 Cross-cutting recommendations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 33 Recommendations for the follow-on project . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 36 Conclusion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 37 References . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 38 Annexes . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 39 BREAKTHROUGH RESEARCH | AUGUST 2020 iii iv MIDTERM EVALUATION OF USAID TULONGE AFYA PROJECT Executive Summary The USAID Tulonge Afya project (2017–2022) is a cooperative agreement between the U.S. Agency for International Development (USAID) and FHI 360. USAID Tulonge Afya (Swahili for “Let’s Talk About Health”) builds upon USAID’s and the Government of Tanzania’s (GOT) strong legacy of implementing social and behavior change communication (SBCC) programs using an inte￾grated approach wherein efforts are organized around the target group rather than the health area. USAID Tulonge Afya is generating demand for services sup￾ported by its sister projects, USAID Boresha Afya (UBA). This report examines findings from a mid-term evaluation conducted by Breakthrough RESEARCH, a USAID-funded research and evaluation project. The purpose of this evaluation is to provide recommendations to enhance the effectiveness of USAID Tulonge Afya’s strategies and activities as the program enters year three. The eval￾uation identified successes, promising strategies, and interventions that can be sustained throughout the life of the project and scaled-up to other regions and districts. The evaluation also helps to identify facilitating and limit￾ing factors for implementing a complex, integrated SBCC program, with applications for the future of this project as well as future integrated SBCC programming globally. The evaluation used a mixed methods approach and focused on project-supported youth and adult commu￾nity platforms, including SITETEREKI (unmarried, sexually active male and female youth), NAWEZA (pregnant women and their male partners aged 18–49 years and health care workers at the facility and community level), and FURAHA YANGU, an HIV campaign targeting primar￾ily those at greater risk for HIV or tuberculosis (TB) and people living with HIV (PLHIV). Methods included a desk review of relevant documents; key informant interviews (KIIs) among USAID Tulonge Afya staff, implementing partners (IPs), and key govern￾ment stakeholders at the national and subnational level; in-depth interviews (IDIs) among IPs and civil society organization (CSO) staff responsible for implementing and supervising USAID Tulonge Afya SBCC platforms at the regional and district level; focus group discussions (FGDs) among community health workers (CHWs), com￾munity volunteers (CVs), and peer champions delivering the youth and adult platforms and beneficiaries of the platforms; and direct observation of USAID Tulonge Afya’s project activities. The report presents the findings and results by the proj￾ect’s three intermediate result (IR) areas, with key priority questions highlighted under each IR and presentation of findings and results for each of the five cross-cutting questions. Selected key findings and recommendations The selected key findings below are paired with recom￾mendations when appropriate for USAID Tulonge Afya, followed by selected recommendations for the follow-on activity. Detailed findings and recommendations are in the main body of the report. IR 1: Improved ability of individuals to practice healthy behaviors (questions 1a and 1b) • Among beneficiaries, the health topics most fre￾quently discussed as a result of their involvement in the SBCC activities were related to dual protection from condom use, teenage pregnancy, HIV testing, HIV-related stigma, use of modern contraceptive methods, male involvement during antenatal care (ANC) visits and assisting their wives with house chores in the community, and the importance of early ANC visits. • Resonance of similar key messages provided to both adult and youth men and women were reported differently. • Recommendation: Even though similar mes￾sages are provided to male and female youth, knowledge of correct condom use needs to be reinforced among female youth. • USAID Tulonge Afya’s use of audience insights to inform emotional drivers among targeted audi￾ences and use of multiple channels to deliver SBCC messages with an emphasis on interpersonal com￾munication (IPC) demonstrate promise for catalyzing BR E A K THROUGH R ESE A RCH | AUGUST 2020 1 positive change in gender and sociocultural norms within communities across priority health areas. IR 2: Strengthened community support for healthy behaviors (questions 2a and 2b) • USAID Tulonge Afya training and other support to regional/district level Health Promotion (HP) coor￾dinators was key to improving coordination and implementation of SBCC activities. • Recommendation: Include district HP coordi￾nators in national training of trainers (TOT) currently provided to SBCC coordinators work￾ing under the IPs, CSOs, and Regional Health Promotion Coordinators (RHPCOs). • Data are being used more frequently for decision making, but they are currently limited to outputs and do not include ideational indicators. • Recommendation: Integrate outcome SBCC indicators into the national District Health Information System (DHIS)2 and assist the GOT in understanding the rationale for inclusion and uptake nationwide. • While supportive supervision was improved among the GOT and CSOs, challenges remain in reporting and adaptive management. • Recommendation: Establish biannual refresher trainings, including practicums, among session facilitators based on outcomes from support￾ive supervision to improve session facilitator competency. IR 3: Improved systems for coordination and im￾plementation of SBCC interventions (question 3) • USAID Tulonge Afya was successful in building skills of the Health Promotion Section (HPS) and other GOT staff at the national level and trained IPs in develop￾ment and implementation of the SBCC interventions. • CSOs reported that early and meaningful engage￾ment of GOT staff assisted in smooth project implementation. • Co-investment by USAID Tulonge Afya and UBA was facilitated through shared office space and joint funding for supportive supervision and community theaters. Cross-cutting question 4 How has USAID Tulonge Afya defined and measured suc￾cess, and which activities/approaches are demonstrating success across each of the three IRs? • Participatory approaches are appreciated by the GOT, IPs, and beneficiaries (i.e., ADDED [Audience￾driven demand, design, and delivery]/HCD [Human-centered design] approaches). • Project quarterly and annual reports do not report enough detail on knowledge, attitudes, and practices. • Recommendation: Ensure the adaptive man￾agement and learning agenda set forth in the Activity Monitoring and Evaluation Plan (AMEP) is fully utilized for better decision making and program adaptation. • Some recommendations from USAID field visits are yet to be undertaken by the project, e.g., notebooks among facilitators to capture questions, sensitization of GOT on integrated SBCC. • Recommendation: USAID Tulonge Afya should work with USAID to ensure that the recom￾mendations from the USAID team are fully integrated into program implementation. Cross-cutting question 5 What have been internal limitations and/or external constraints in achieving USAID Tulonge Afya objectives for each of the three IRs? • In several districts, it was noted that some health providers’ attitudes and lack of adherence to client confidentiality deterred women and youth from seeking services and commodities. • Recommendation: UBA should work closely with USAID Tulonge Afya to ensure UBA is focus￾ing on health providers’ biases and behaviors to improve youth friendly and client-centered services that respect clients’ privacy both at facilities and community theaters. • Referral and linkage mechanisms for services, and proximity to clinics, were not always available during demand generating activities at the community level, hindering service uptake. • Recommendation: Close the gap between demand creation and service provision and sup￾ply of commodities, e.g., include task shifting, 2 MIDTERM EVALUATION OF USAID TULONGE AFYA PROJECT provision of mobile clinics, review USAID Tulonge Afya and UBA activities in workplans to ensure demand generation and service provision technical assistance are aligned and coordinated. Cross-cutting question 6 What mitigation strategies have been considered to address limitations and constraints, and how effectively have they been adopted? • USAID Tulonge Afya undertook several mitiga￾tion strategies (i.e., improving mentorship and supervision) based on project feedback across all three IRs that improved activities for beneficiaries and strengthened data linkages and coordination frameworks. • Recommendation: Ensure either supportive supervision or another mechanism provides documented confirmation that activities took place. Cross-cutting question 7 What are the facilitators/barriers to shifting from vertical to integrated SBCC programs, particularly within the youth and adult platforms? • An important facilitator was ensuring that the GOT leadership perceived integrated SBCC approaches as cost-effective and efficient. • Recommendation: Maximize and document efficiencies across integrated SBCC approaches for both the health system and clients, as advantages of integrated SBCC. • Limited understanding of what integrated SBCC entails in practice by GOT officials is a barrier. • Recommendation: Assist in a deeper sensiti￾zation among stakeholders and GOT regarding what integrated SBCC involves. Cross-cutting question 8 • How have USAID Tulonge Afya organizational and management structures, systems, processes, and procedures enabled or constrained the success of capacity, coordination, and collaboration, espe￾cially within its integrated SBCC program? • USAID Tulonge Afya ensured GOT leadership and participation of key stakeholders in SBCC design and delivery. • Some approved activities by the GOT and the project did not take place in a timely manner or as planned due to USAID internal processes; differences in fiscal years and activity planning with the GOT also affected this. • Recommendation: Facilitate meetings to address the planning challenges related to differences in fiscal years between USAID and USAID Tulonge Afya. • Sustainability of funds to support TWGs and support￾ive supervision after the project ends is an issue. • Recommendation: Strengthen accountability and sustainability mechanisms for SBCC activ￾ities, i.e., project guidance to CSOs on how to increase local government involvement and/or transparency regarding planned activities and budgets. There was agreement among mid-term evaluation participants that USAID Tulonge Afya activities have improved the ability of individuals to practice healthy behaviors. The most substantial improvements were reported in attitudes and behaviors related to HIV/AIDS, family planning and reproductive health, and maternal and child health (in particular, those related to pregnancy and ANC). USAID Tulonge Afya was successful in empowering and engaging the government and civil society structures at district and regional levels to support and facilitate delivery of quality SBCC, most notably by improving coor￾dination for SBCC activities at the national, regional, and district levels, and strengthening the SBCC data systems. Overall, respondents supported the integrated approach to SBCC and added that it reached beneficiaries with needed information specific to their life stage in a more comprehensive way than vertical interventions, thereby saving time and costs for both the clients and health system. Some barriers limiting access to and/or utilization of services still exist, however, and the evaluation found both programmatic and structural challenges to the implementation of integrated SBCC programming that need to be addressed. While there are areas that the project can continue to improve upon, the project has notable successes that can be documented in having a positive influence on key health behaviors among target audiences throughout the project focal areas. BR E A K THROUGH R ESE A RCH | AUGUST 2020 3 SELECTED RECOMMENDATIONS FOR THE FOLLOW-ON PROJECT IR 1: Improved ability of individuals to practice healthy behaviors • Utilize project surveys of the integrated SBCC activities and related changes in behavior/norms to priori￾tize and bundle health areas in future platforms and campaigns, especially gateway behaviors • Prioritize the sustainability and continued involvement of SITETEREKI volunteers. IR 2: Strengthened community support for healthy behaviors • Consider including documented lessons from any successful outcomes based on proposed USAID Tulonge Afya pilot activities (e.g., “on-call” providers, frequently asked question (FAQ) booklets, health provider collaboratives) in the next mechanism. • Improve DHIS2 data for decision making regarding key ideational factors (e.g., self-efficacy, subjective and social norms). IR 3: Improved systems for coordination and implementation of SBCC interventions • USAID/Tanzania should encourage any IPs working with CSOs to apply lessons learned in GOT engage￾ment through the experience of USAID Tulonge Afya. • Incorporate specific intermediate results pertaining to cross-project linkages (i.e., USAID Tulonge Afya and UBA) in follow-on service delivery and SBCC mechanisms. 4 MIDTERM EVALUATION OF USAID TULONGE AFYA PROJECT Introduction Contextual background Tanzania has a high population growth rate at 32%, ranked 15th in the world,1 which continues to burden its health care system.2 Communicable diseases place further strain on the limited health resources. According to the 2018 Global Burden of Disease report, the leading causes of death in Tanzania include HIV/AIDS, malaria, tuberculosis (TB), other respiratory infections, and diarrhea.3 • In 2016, 1.4 million people were living with HIV (PLHIV) in Tanzania, an estimated HIV prevalence of 4.7 percent.4 HIV prevalence among females aged 15 to 24 years is more than twice as high as that among males in the same age group (2.1% among females and 0.6% among males). • Tanzania also has the third largest population at risk for contracting malaria in Africa, with over 90% of its population living in malaria-endemic areas. Each year, over 12 million people contract malaria and 80,000 die from the disease.5 • According to the 2015–16 Tanzania Demographic and Health Survey, 32% of married women in Tanzania use a modern contraceptive; however, only 9% of young women aged 15 to 19 years and 29% aged 20 to 24 years use a modern method of family planning (FP).6 The total fertility rate (TFR) in Tanzania is 5.2 children per woman, which is among the highest in the world, while the unmet need for FP is 22%. • Moreover, one-third (34%) of Tanzanian children under age five are stunted (short for their age), 5% are wasted (thin for their height), and 14% are under￾weight (thin for their age). Like many other African countries, sociocultural norms and gender inequities (i.e., discriminatory attitudes toward PLHIV, misconceptions about HIV) continue to be the main barriers for improving healthy behaviors and for service uptake among women and youth in Tanzania. Description of the USAID Tulonge Afya Project The USAID Tulonge Afya project (2017–2022) is a cooperative agreement between the U.S. Agency for International Development (USAID) and Family Health International 360 (FHI 360). The project builds upon USAID’s and the Government of Tanzania’s (GOT) strong legacy of implementing social and behavior change com￾munication (SBCC) programs. The three main objectives of USAID Tulonge Afya activity are to: 1. Improve the ability of individuals to practice healthy behaviors 2. Strengthen community support for healthy behaviors 3. Improve systems for coordination and implementa￾tion of SBCC interventions Annex I provides the project’s log frame for achieving improved health status, focused on women and youth. The USAID Tulonge Afya (Swahili for “Let’s Talk About Health”) activity works closely with the GOT’s and the U.S. Government’s (USG) implementing partners (IPs) to develop and support SBCC interventions using participa￾tory, evidence-based, and theory-informed approaches. USAID Tulonge Afya uses an integrated SBCC approach wherein efforts are organized around the target group rather than health area. It respects the complex relation￾ship between health and illness such as HIV/TB and that people have different needs at different times of their lives such as pregnancy. It helps service providers and CHWs engage target groups around multiple relevant health areas when they need them and at opportune times. Figure 1 shows the theory of change used by USAID Tulonge Afya to achieve improved health and gender equality.7 The audience-driven demand, design, and delivery (ADDED) approach illustrates how USAID Tulonge Afya engages audiences throughout the project by ensuring: BR E A K THROUGH R ESE A RCH | AUGUST 2020 5 • Activities focus on what audiences want (demand) • Activities, tools, and materials are co-designed with audiences (design) • Activities will be delivered with and by audiences (delivery), emphasizing participatory and audi￾ence-led activities, such as participatory community radio, peer-led group dialogue, peer champions, etc. The ADDED approach also highlights the use of specific SBCC tactics including empowerment, demand gener￾ation, skills building, behavioral economics, and social action, tailored to where audiences fall along a behavior change continuum. USAID Tulonge Afya uses a participatory, evidence-based, and theory-informed approach to: 1) address norms and inequities that drive poor health and related behaviors; 2) advance health while promoting rights; 3) use data better to support regional and district needs; 4) harmonize messages and media; 5) strengthen institutional capacity to manage and deliver high-quality SBC; and 6) facilitate coordination to maximize SBCC impact and efficiencies. Through an evidence-based integrated SBCC approach, USAID Tulonge Afya aims to improve the health status of Tanzanians in the following focal areas: 1) HIV/AIDS, 2) malaria, 3) family planning and reproductive health (FP/ RH), 4) maternal and child health (MCH), and 5) TB. USAID Tulonge Afya targets pregnant women, caregivers of children under five years of age, adolescents and youth, PLHIV, and men aged 15 to 49 years. USAID Tulonge Afya implements activities with two levels of intensity; in some districts, the project offers a pack￾age of “essential” SBCC activities, while in other districts it provides a more extensive, “enhanced” package of SBCC activities. The key difference between essential and enhanced SBCC programming is that USAID Tulonge Afya conducts district-to-national planning processes for funding and activity prioritization and implements com￾munity-level SBCC activities in enhanced districts through their CSO grants program. In districts where the essential package of activities is offered, other IPs are provided with the project’s SBCC tools and materials (including implementation guides for each tool) to implement community-level SBCC activities in their catchment areas. Table 1 outlines the activities included in the enhanced SBCC package versus those in the essential SBCC package. Only districts offering the enhanced package of services were selected for this evaluation. Additionally, USAID Tulonge Afya works to increase the capacity of local government authorities, non-govern￾mental organizations (NGOs), community or civil society organizations, CHWs, and other health care providers, FIGURE 1 AUDIENCE-DRIVEN DEMAND, DESIGN, AND DELIVERY (ADDED) APPROACH AND THEORY OF CHANGE 6 MIDTERM EVALUATION OF USAID TULONGE AFYA PROJECT community leaders, media, schools, GOT, and imple￾menting partners to design and deliver quality SBCC interventions.8 As illustrated in Table 1, USAID Tulonge Afya uses mul￾tiple channels to implement across health areas among target audiences. This evaluation focused on project-sup￾ported youth and adult platforms, including SITETEREKI (unmarried, sexually active male [JUMA] and female [SUBIRA] youth), NAWEZA (pregnant women and their male partners and health care workers at the facility and community level), and FURAHA YANGU, an HIV campaign targeting primarily those at heightened risk for HIV or TB. Table 2 illustrates a comprehensive list of USAID Tulonge Afya campaigns and platforms, and the key behaviors supported at the time of the evaluation. Platforms are multi-channel, life stage-based SBCC interventions that include branded national-level mass media, communi￾ty-level activities, and interpersonal communication (IPC). Priority behaviors supported by the NAWEZA integrated SBCC platform at the time of this evaluation included: • Early antenatal care (ANC) attendance and at least four visits (with eight preferred) • IPTp-3+ during ANC visits • Use of insecticide-treated nets (ITNs) • Discussion of postpartum FP (PPFP) options with health provider • Use of prevention of mother-to-child transmission of HIV (PMTCT) services • Health facility birth SUBIRA female beneficiaries engaged in participatory strategies and small group discussion around available contraceptive methods and use of condoms as dual protection for preventing pregnancy, HIV, and STIs. Male JUMA beneficiaries were also engaged in participatory strategies and small group discussion on correct and consistent condom use for dual protection. Male bene￾ficiaries received messages related to gender and social norms, including harmful concepts of masculinity that prevent modern contraceptive method (MCM) uptake, parent and youth/intergenerational sexual and reproduc￾tive health (SRH) dialogue, gender-based violence, and discrimination and stigma. The FURAHA YANGU vertical multi-channel campaign emphasized HIV testing among high risk groups and early ART initiation if found HIV-positive. As noted in the table, there were several activities across each of the platforms that had not been rolled out at the time the evaluation was conducted and thus could not be evaluated. The following three priority behaviors were not promoted in the NAWEZA platform at the time of the evaluation: • Initiate breastfeeding within the first hour of birth TABLE 1 USAID TULONGE AFYA PROGRAMMING SUPPORT INTENSITY (ENHANCED VERSUS ESSENTIAL) ACTIVITY ENHANCED SBCC PACKAGE ESSENTIAL SBCC PACKAGE National platforms for adults and youth X X Regionalized radio and messaging X X Community radio and theater X Social media X X mHealth X X SBCC technical assistance and tools X X Opportunistic activities X X Leveraged activities with other implementers X X District mobilization and sensitization campaigns X Joint planning and community small grants X BR E A K THROUGH R ESE A RCH | AUGUST 2020 7 • Attend postnatal care (PNC) visits and seek prompt health facility care for postpartum danger signs • Early infant visit and HIV testing In the SITETEREKI SBCC integrated youth platform, HIV testing and ART adherence materials for female SUBIRA beneficiaries and HIV-related messages including testing, treatment, and voluntary medical male circumcision (VMMC) messaging for male JUMA beneficiaries had not been promoted at the time of the evaluation. Similarly, for the FURAHA YANGU campaign, messages emphasizing ART adherence and viral load monitoring were not yet promoted at the time of this evaluation. USAID Boresha Afya USAID Tulonge Afya is generating demand for services supported by its sister project, USAID Boresha Afya (UBA). The UBA project was designed to support the GOT to increase access to quality comprehensive and integrated health services. UBA has been designed to achieve the following two main results: 1) improved enabling environment for health service provision, and 2) improved availability of quality, integrated health services at the facility level. GOT has made notable efforts in developing its community health programs through the development of policies and guidelines and by support￾ing research studies to deepen the knowledge base. In support of these efforts, the UBA is implemented in three zones of Tanzania by the following implementing partners (IPs):9 • Boresha Afya Southern Zone—Deloitte Consulting Ltd. is prime with subcontractors FHI 360, Management and Development for Health (MDH), and EngenderHealth in six regions: Iringa, Lindi, Morogoro, Mtwara, and Njombe for HIV, TB, malaria, FP, and RMNCH, and Ruvuma for malaria only. • Boresha Afya Lake/West Zones—Jhpiego is prime with sub-recipients PATH and EH in seven regions: Geita, Kagera, Kigoma, Mwanza, Mara, Shinyanga, and Simiyu for FP, sexual/reproductive health (SRH); malaria; maternal, newborn, and child health (MNCH); adolescent and community empowerment; and systems strengthening for health service deliv￾ery. Activities are also carried out in Zanzibar. • Boresha Afya North/Central Zones—The Elizabeth Glaser Pediatric AIDS Foundation (EGPAF) is prime with sub-recipient EngenderHealth in six regions: Arusha, Dodoma, Kilimanjaro, Manyara, Singida, and Tabora for HIV, TB, and FP. UBA is not active in all USAID Tulonge Afya districts but is active in all districts considered for inclusion in this study. Regions where UBA and USAID Tulonge Afya overlap in this evaluation are highlighted in the previ￾ous bullets in bold. UBA promotes integrated services, expands linkages between health units within a facility, and improves referral procedures between facilities and from communities to facilities to increase access to services, responsiveness of service delivery, and quality of care. UBA aims to increase outreach to communities from facilities, using SBCC and sensitization approaches in communities in partnership with CSOs and other USAID-supported programs including USAID Tulonge Afya. Note that UBA has FHI 360 as a sub-partner only in the Southern Zone, but the scope under this project is not SBCC. 8 MIDTERM EVALUATION OF USAID TULONGE AFYA PROJECT TABLE 2 LIST OF USAID TULONGE AFYA CAMPAIGNS AND PLATFORMS WITH PRIORITY BEHAVIORS AND ACTIVITIES SUPPORTED TARGET POPULATION PRIORITY BEHAVIORS & ACTIVITIES SUPPORTED 1) NAWEZA SBC integrated adult platform Targets adults aged 18 years and older during two critical life stages (1) pregnancy and childbirth and 2) caregiving for a child <5 years of age) by providing clear messages and activities that address behavioral determinants and focus on emotional drivers among audiences Pregnancy and Childbirth Package (*not rolled out prior to evaluation) Primary • Pregnant women and their male partners aged 18–49 • Health care workers (facility & community) Secondary • Influential family members • Traditional leaders • Religious leaders NAWEZA’s package is designed to promote priority behaviors (some of which are gateway behaviors, i.e., ANC attendance), in particular: 1. Early ANC attendance (discussed in 1st session) and attending 4 or more ANC visits; 8 con￾tacts are desired, as discussed in 2nd session 2. Take IPTp-3 during ANC visits 3. Sleep under an ITN every night, including pregnant women 4. If HIV+, attend PMTCT services and take ART as prescribed 5. Attend a health facility for delivery (priority behavior) 6. *Initiate breastfeeding within the first hour of birth 7. Talk with your health care provider about post-partum FP (PPFP) options (discussed in sessions 1–4) 8. *Attend postnatal care (PNC) visits and seek prompt and appropriate care at health facility upon the first sight of post-partum danger signs 9. *Bring your infant to the health facility for an early visit at 4–6 weeks and for HIV testing if the mother is positive or status unknown Assumptions: • Facilitators assume that when women practice gateway behaviors, they will be encour￾aged to practice other non-priority behaviors. For example, when women attend ANC they will also be advised to test for HIV, use ITN, be given IPT, etc. • All NAWEZA volunteers are trained in all priority and non-priority behaviors. Non￾priority behaviors are all listed in the cards that are used for the games as part of the sessions. For example, one card used for session discussion reads “advantages of multiple ANC attendance,” which lists the various services women receive if they attend ANC multiple times (priority behavior) • USAID Tulonge Afya expects beneficiaries to receive all the messages, including non-priority messages, from other SBCC channels, e.g., mass media, posters, fliers, banners, etc. Cross-cutting gender and social norms: male support and couple dialogue. BR E A K THROUGH R ESE A RCH | AUGUST 2020 9 *Caregiving for Children Under 5 (*not rolled out prior to evaluation) Primary: • Parents and caregivers ages 18+ years • Health care workers (facility & community) Secondary • Influential family members • Traditional leaders • Religious leaders • Sleep under an ITN every night, including children under five • After a live birth, use an MCM to avoid pregnancy for at least 24 months • Exclusively breastfeed your infant for six months after birth • Seek and receive prompt and appropriate care at first sign of newborn and childhood illness • Seek and receive a full course of timely vaccinations for infants and children under two • For malaria, seek and receive prompt and appropriate care at the health facility for yourself or a child under five with a high fever, including use of a rapid diagnostic test to confirm malaria Cross-cutting gender and social norms: male involvement in care seeking and couple dialogue. 2) SITETEREKI SBCC integrated youth platform Targets sexually active unmarried youth aged 15 to 25 years using multi-media communication approaches to reinforce key messag￾es across all channels Note: At the time of the evaluation both SUBIRA and JUMA sessions were focusing on one priority behavior (for females use MCM, for males use condoms) under each session as an “accelerated roll-out,” as described below • Unmarried, sexually active adolescent girls (ages 15-19 years) (SUBIRA) • Unmarried, sexually active adult women (ages 20–24 years) (Edna) • *Adolescent girls who have begun childbearing (ages 15–19 years) (Maua) • Unmarried, sexually active young men (ages 15–24 years) (JUMA) Secondary • Parents • Sexual partners • School teachers & administration • Health care providers • SUBIRA sessions focus on promoting an MCM to both delay first pregnancy and space future pregnancies, and use of condoms is discussed among FP methods for prevent￾ing both pregnancy and STIs. • *HIV testing and ART adherence materials for SITETEREKI were not yet part of these sessions at the time this evaluation was conducted; they were approved in January 2020. • JUMA sessions focus mainly on correct and consistent condom use to both prevent unwanted pregnancies and STIs, including HIV. • *HIV-related messages including testing, treatment, and VMMC messaging were not yet part of the JUMA sessions at the time this evaluation was conducted; they were approved in early 2020. 10 MIDTERM EVALUATION OF USAID TULONGE AFYA PROJECT 3) FURAHA YANGU—Vertical multi-channel campaign Designed to be rolled out in 3 phases to increase awareness and demand creation momentum for Test and Treat services and 95 95 95 Testing and treatment targets, and to reduce stigma around HIV testing and PLHIV. • Those at greater risk for HIV or TB including: • Men (18 to 45yrs) • Those in higher risk occupations • Adolescent girls and young women ages 15–24 years • Caregivers of children at high risk of HIV • Vulnerable populations (e.g., those having multiple partners, those practicing unsafe sex, survivors of sexual assault) • PLHIV • People with TB ++Messages listed in the FURAHA YANGU community dialogue guide • Phase I: Focuses on HIV testing and early ART initiation if positive • Target beneficiaries for this phase are people at high risk of contracting HIV. • Messages emphasize advantages of HIV testing and early ART initiation. *Note: At the time this evaluation was conducted only Phase I had been implemented. Below are target beneficiaries and messaging for Phases II and III, which had not been imple￾mented at the time of this evaluation. • Phase II: Will focus on ART adherence and viral load monitoring • Target beneficiaries are people living with HIV (PLHIV). • Messages under this phase will focus on importance of adhering to ART, viral load testing, uptake of TB preventive therapy, cervical cancer screening among HIV￾positive women, pre-exposure prophylaxis (PrEP) uptake among key vulnerable populations (KVP), and Tenofovir, Lamivudine, and Dolutegravir (TLD) transitioning. Radio messages on Phase II messages started airing in Feb 2020. • Phase III: Will focus on reinforcing messages from Phase II • Target beneficiaries are PLHIV. • Focus will be on emerging priorities under Phase II, including index testing, retention in care, messages around importance of viral suppression and having undetectable levels of HIV (“treatment as prevention”), and self-testing messages as part of index testing and testing among KVP. • ++Test for HIV if at risk, and receive results • ++If HIV-positive, enroll in care, initiate ART, and follow health care worker guidance • ++Take ART regularly as prescribed and go for routine viral load monitoring • *For HIV+ women go for cervical cancer screening • Ask to be started on TLD when enrolling in ART • For KVP, ask to be initiated on PrEP and take regularly as prescribed • ++Seek care from a qualified TB provider for a cough that persists for more than two weeks • *Ask to be started on TB prevention therapy if HIV positive BR E A K THROUGH R ESE A RCH | AUGUST 2020 11 Methods Evaluation purpose, questions, and research design The purpose of this evaluation is to provide recommen￾dations to enhance the effectiveness of USAID Tulonge Afya’s strategies and activities as the program enters year three. The evaluation identified successes, promis￾ing strategies, and interventions that can be sustained throughout the life of the project and scaled up to other regions and districts. This evaluation is intended to help USAID identify facilitating and limiting factors for implementing a complex, integrated SBCC program, and to inform more effective design, implementation, monitoring, and evaluation of integrated SBCC program￾ming in the future globally. The evaluation findings focus on questions related to the three main objectives of the project: 1) improved ability of individuals to prac￾tice healthy behaviors in program-supported areas; 2) strengthened community support for healthy behaviors; and 3) improved systems for coordination and implemen￾tation of SBCC interventions. Evaluation questions This evaluation aimed to answer the questions outlined in Table 3. There were three main evaluation questions. The first two evaluation questions are further subdivided. TABLE 3 PRIORITY AND CROSS-CUTTING QUESTIONS FOR USAID TULONGE AFYA MIDTERM PERFORMANCE EVALUATION PRIORITY EVALUATION QUESTIONS IR 1: IMPROVED ABILITY OF INDIVIDUALS TO PRACTICE HEALTHY BEHAVIORS IR 2: STRENGTHENED COMMUNITY SUPPORT FOR HEALTHY BEHAVIORS IR 3: IMPROVED SYSTEMS FOR COOR￾DINATION AND IMPLEMENTATION OF SBCC INTERVENTIONS 1a. Have USAID Tulonge Afya activities improved knowledge, attitudes, inten￾tions, and efficacy to practice healthy behaviors across priority health areas (HIV/AIDS, malaria, FP/RH, MCH, and TB) among target populations? 1b. What USAID Tulonge Afya activities/ approaches demonstrate promise for catalyzing positive change in gen￾der and sociocultural norms within communities, enabling the practice of desired behaviors across priority health areas? 2a. Has USAID Tulonge Afya effectively empowered and engaged government and civil society structures at district and regional levels to support and facilitate delivery of quality SBCC? 2b. How have USAID Tulonge Afya ap￾proaches for engagement of audienc￾es enabled or constrained the delivery of quality SBCC at the community level through USAID Tulonge Afya sub-partners and other USAID part￾ners? 3. Have USAID Tulonge Afya activities led to improved capacity, coordination, collaboration, and co-investment for SBCC at national and subnational levels among implementing partners, key stakeholders, and the GOT? CROSS-CUTTING QUESTIONS 4. How has USAID Tulonge Afya defined and measured success, and which activities/approaches are demonstrating success across each of the three IRs? 5. What have been internal limitations and/or external constraints in achieving USAID Tulonge Afya objectives for each of the three IRs? 6. What mitigation strategies have been considered to address limitations and constraints, and how effectively have they been adopted? 7. What are the facilitators/barriers to shifting from vertical to integrated SBCC programs, particularly within the youth and adult platforms? 8. How have USAID Tulonge Afya organizational and management structures, systems, processes, and procedures enabled or con￾strained the success of capacity, coordination, and collaboration, especially within its integrated SBCC program? 12 MIDTERM EVALUATION OF USAID TULONGE AF YA PROJECT Additionally, this evaluation was designed to address five additional cross-cutting questions. Research design, study participants, and sample size The evaluation methodology used a mixed methods approach. USAID chose to focus the evaluation on project-supported youth and adult community platforms, including SITETEREKI (unmarried, sexually active male [aged 15–24, JUMA] and female [aged 15–19, SUBIRA] youth), NAWEZA (pregnant women and their male part￾ners aged 18–49 and health care workers at the facility and community level), and FURAHA YANGU, an HIV cam￾paign targeting primarily those at greater risk for HIV or TB and PLHIV. It did not place emphasis on the platforms’ related national-level activities, such as mass media and the project’s long-and short format radio programming. Methods included a desk review of relevant documents; key informant interviews (KIIs) among USAID Tulonge Afya staff, implementing partners, and key government stakeholders at the national & subnational level; in-depth interviews (IDIs) among IPs and CSO staff responsible for implementing and supervising USAID Tulonge Afya SBCC platforms at the regional and district level; focus group discussions (FGDs) among CHWs, community volunteers (CVs), and peer champions delivering the youth and adult platforms and beneficiaries of the platforms; and direct observations of USAID Tulonge Afya’s project activities. A mixed-methods approach enabled the triangulation of diverse data sets to obtain robust insights on the effectiveness of the project activities and answer the key evaluation questions. Prior to fieldwork, the evaluation team conducted a detailed desk review of the project documents (see Annex VII) and data from sources provided by USAID Tulonge Afya. This information was used to inform the evaluation methodology and questions in the data col￾lection tools (see Annex IX). Despite an initial request for English translation, some documents written in Swahili were not available in English at the time of review (i.e., English guides for group platforms). Team members who were fluent in Swahili translated the session topics for the platforms during data analysis to understand the content related to the interviews. Annex IIa provides a summary of data gathering activities, purpose, and research evalu￾ation questions addressed by each instrument. Annex IIb provides interview type, respondent category, and num￾ber of FGDs, interviews, and observations completed. After field work, data analysis was conducted and the report was drafted; feedback and additional meetings with USAID Tulonge Afya staff in Dar es Salaam were held to clarify specific elements of packages, which helped clarify how elements in each of the packages was rolled out, what deviated from the original plan, and what was maintained. Geographic setting of the evaluation IDIs, FGDs, and semi-structured observations were conducted in six districts in the North/Central, Lake/ Western, and Southern zones of Tanzania. Districts were selected by USAID from the 29 Enhanced Districts where the full suite of USAID Tulonge Afya activities are taking place, in contrast to the Essential Districts that have fewer activities (Table 1). Figure 2 displays a map of all enhanced districts where USAID Tulonge Afya operates. USAID selected two districts from each zone for the evaluation. The selection of districts ensured that half of evaluation activities were conducted in districts where Tanzania Marketing and Communications (T-MARC) is the community-level implementing partner (implementing USAID Tulonge Afya community activities since year one of the project) and half in districts where Tanzania FIGURE 2 29 ENHANCED DISTRICTS WHERE USAID TULONGE AFYA OPERATES BR E A K THROUGH R ESE A RCH | AUGUST 2020 13 Communication and Development Center (TCDC) is the community-level implementing partner (implementing Tulonge Afya activities since year two of the project). The six districts are shown in Annex IIc. Additional details regarding participant and facilitator characteristics, ward selection, participant recruitment, and data collection methods are described in Annexes III–VI. Data analysis and management Data analysis: Researchers transcribed the digitally recorded KIIs, IDIs, and FGDs from Swahili into English. Trained research team members verified all the tran￾scripts against the original audio recordings to ensure that the transcriptions and translations were accurate. After transcripts were validated, researchers imported them into a qualitative software ATLAS.ti (Version 7.0) for data analysis. Data analysts developed a preliminary codebook that comprised pre-set codes derived from questions in the data collection tools for the various respondent cate￾gories. Data analysts piloted the preliminary codebook on a sample of nine transcripts under each respondent category prior to using it for the actual coding of tran￾scripts. For each respondent category, data analysts coded a similar transcript and compared the assigned codes under similar text segments. Next, data analysts resolved any disagreements by refining or merging the preset codes and/or proposing new codes to ensure inter rater reliability. Data analysts repeated the latter process with different transcripts under the same respondent category until there were minimal or no disagreements in the application of codes among all coders. Once data analysts reached intercoder agreement, they used the codebook to code transcripts under the corresponding respondent category. All analysts participated in both the codebook piloting activities and actual coding of transcripts. Data analysts repeated this process for each new respondent category to ensure that the pre-exist￾ing codes were applicable, and where they were not applicable, they made appropriate adjustments to the codebook. Once all data were coded, data analysts began their analysis. Key evaluation questions guided analysis of the data and led to an in-depth exploration of the informa￾tion emerging from the various codes. Additionally, data analysis involved exploration of potential differences in responses among participants of different backgrounds, sexes, and districts. Following data analysis, data analysts merged information under related codes into larger themes and sub-themes that are presented under the subsequent sections of this report. Data management Prior to participating in data collection activities for this evaluation, all interviewers received ethics training and signed a data confidentiality agreement. All interviews were in private areas to ensure participant confidenti￾ality, and interview audios and written transcripts were labeled with participant identifiers rather than the name of the interviewee. The study team kept information received from the study participants confidential and kept files that had identifying information (e.g., recruit￾ment and consent forms) in locked storage. Staff will destroy these forms after 15 years per National Institute for Medical Research (NIMR) Standard Operating Procedures (SOPs). Project staff analyzed the data at an aggregate level; hence, no individual respondent’s information was disclosed. Ethical considerations The protocol for this evaluation was reviewed and approved by the NIMR, Dar es Salaam, Tanzania and by the Institutional Review Board of the Population Council in New York, New York, USA. The team obtained written consent from all participants of the IDIs and KIIs; verbal consent was obtained from participants of KIIs that were conducted by phone. Interviewees were given the option to opt out of ques￾tions or the entire interview and they were assured personal confidentiality for the information provided. Participants in the FGDs (both facilitators and beneficia￾ries) received a small incentive for their participation of 10,000 Tanzanian Shillings, or approximately US$5. The amount was based on the recommendation of the local ethics committee in previous studies conducted by the midterm evaluation team. Limitations Due to self-reporting among participants, there is potential for social desirability bias, i.e., some respon￾dents may have told the interviewers what they felt they needed to hear in favor of the program. Project staff observed significantly fewer activities than planned 14 MIDTERM EVALUATION OF USAID TULONGE AFYA PROJECT due to unforeseeable and unavoidable challenges that caused activities to be cancelled, including weather conditions and changes in field work and implementation that took place after final logistics were agreed upon with Breakthrough RESEARCH and USAID Tulonge Afya. Because some facilitators participated in the FGDs at the same time the community activities were planned, some community activities did not take place. When selecting participants for FGDs, session facilitators may have chosen participants who were more involved and engaged. The evaluation team did not conduct FGDs among 15- to 17-year-olds due to challenges obtaining parental consent; therefore, input from this age group is not included in this report. Similar to other qualitative work, findings from this evaluation cannot be generalized to the larger populations from which participants were drawn but can inform programming related to the areas evaluated in the report. Lastly, the primary emphasis of the evaluation focused at the community level for IR1; however, USAID Tulonge Afya implements wide scale mass media programming, which community level participants may or may not also have been exposed to (alongside community media) and is not fully reflected on in this evaluation. In addition, the full synergistic effect of USAID Tulonge Afya’s technical assistance and support to other USG projects is not reflected in these findings and discussion. BR E A K THROUGH R ESE A RCH | AUGUST 2020 15 Findings The findings from this evaluation are presented under each of the three program intermediate results (IRs) areas. The findings are further broken down by the key priority questions under each IR, and the cross-cutting questions. Additional supporting statements for findings can be found in Annex VIII. The supporting quotations included are illustrative of the broader support for the finding in the qualitative data. IR 1: Evaluation question 1a Have USAID Tulonge Afya activities improved knowl￾edge, attitudes, intentions, and efficacy to practice healthy behaviors across priority health areas (HIV/ AIDS, malaria, FP/RH, MCH, and TB) among target populations? All six districts had similar findings, and there were no notable differences between enhanced T-MARC and TCDC-supported districts. As only specific priority behaviors and activities among health areas across the campaigns and platforms were rolled out at the time of the evaluation (Table 2), the following findings focus on the priority behaviors that were rolled out prior to the evaluation. Among beneficiaries, the health areas most frequently discussed as a result of their involvement in the SBCC activities were related to dual protection from condom use, teenage pregnancy, HIV testing, HIV-related stigma, use of MCMs, male involvement, and ANC. The findings below provide more nuance related to the specific target audiences among the beneficiaries interviewed. Youth (SITETEREKI Platform) findings • Over half of male youth under SITETEREKI reported improved knowledge of correct condom use, and this was the most highly reported change in knowl￾edge across all technical areas and beneficiaries. Male youth noted that, prior to the USAID Tulonge Afya program, they thought condoms only protected against unplanned pregnancies. After reporting being exposed to SBCC activities, they learned that condoms also protect against HIV as well as other STIs. In several districts, GOT respondents also noted an increased demand for condoms by male youth in their facilities. “ When I started attending the class, I didn’t know steps of using a condom and the benefits of using a condom. Therefore, I now know how to wear a condom properly and I do follow all the steps. That’s what I got [after attending the class from USAID Tulonge Afya]. —JUMA Beneficiary, TABORA • Male and female youth participants as well as session facilitators noted a shift in descriptive norms related to reduction in teenage pregnancies in their communities. “ The community [seem to] ha[ve] reduced the rate of unwanted pregnancies. For instance, there ha[ve] been high rates of teen￾age pregnancies at schools in the past, which is not the case nowadays. This is as [sic] the result[s] of [USAID Tulonge Afya] activities. —JUMA beneficiary, NEWALA • Female youth reported perceived increases in knowledge, awareness, and use of modern meth￾ods of contraception among their peers. “ The big change I have observed [after USAID Tulonge Afya support] is that most girls in the society have now decided to use birth control methods. SUBIRA beneficiary—SENGEREMA Adults (NAWEZA platform and FURAHA YANGU campaign) findings The findings in this section combine the adult NAWEZA platform and adult FURAHA YANGU campaign, as some of the findings show exposure from multiple sources among beneficiaries (and highlighted in the recommendations section). 16 MIDTERM EVALUATION OF USAID TULONGE AF YA PROJECT • Lack of SBCC messages on how to conceive safely among HIV discordant couples is a missed opportu￾nity. Although USAID Tulonge Afya’s SBCC messages included how to protect oneself from contracting HIV and PMTCT, messaging on how to prevent HIV transmission among discordant couples who wish to have a child was not embedded in messaging. • Both male and female beneficiaries of FURAHA YANGU perceived a reduction in HIV-related stigma in their communities. Reducing stigma and normal￾izing HIV testing and treatment is a key objective of the campaign. Both male and female beneficiaries noted that discrimination of PLHIV in their commu￾nities was reduced compared to the period before USAID Tulonge Afya. Nearly three times as many males as females perceived that people are now more open about their HIV-positive status and HIV care seeking. Government respondents also noted an increased number of people accessing ART. Almost all UBA partners noted significant increases in people coming for HIV testing at the facilities that they are supporting. “ The challenge has been following up of medicines for these children but the education about the importance of follow￾ing up on (ART) medicines which has been provided in the community [through USAID Tulonge Afya], has been helping us as well in making sure that children are good followers of medicines, because they are told to adhere to the treatments. —UBA, Njombe • Both male and female beneficiaries of FURAHA YANGU sessions reported increased knowledge and practices related to HIV testing in their com￾munities. Four times more males reported this than females. This includes a perceived increase in the number of people undergoing HIV testing, which was mainly perceived among couples and family members. • Some HIV-positive beneficiaries of NAWEZA and FURAHA YANGU sessions noted they benefited from SBCC activities focusing on PMTCT and were grateful to USAID Tulonge Afya for delivering HIV￾negative newborns. “ After receiving this education [from USAID Tulonge Afya partners], I realized why I lost my two children. I stopped it and I thought that I should follow the right path through NAWEZA education. It gave me advan￾tage because a few days ago my wife delivered a child and the child is okay. —NAWEZA Male beneficiary, RORYA • Female and male beneficiaries of NAWEZA sessions equally noted an increase in awareness of the importance of early attendance (before 12 weeks) for ANC. This is a primary objective of the NAWEZA pregnancy and childbirth package, as this behavior serves as a gateway behavior for other maternal and child health objectives. Beneficiaries of FURAHA YANGU sessions, facilitators of NAWEZA and FURAHA YANGU sessions, CSOs, and government officials echoed this finding. “ We used to go to clinics during the last three months. As you go for delivery, it could happen that you are found anemic or the baby is not positioned well. Now with the coming of NAWEZA [from USAID Tulonge Afya], we have been taught to attend every month to the clinic from the time you think you are pregnant. —NAWEZA female beneficiary, SINGIDA). • More male than female NAWEZA beneficiaries perceived an increase in male involvement. This included attending ANC visits and assisting their wives with house chores in the community. CSOs and facilitators of NAWEZA and FURAHA YANGU sessions echoed this finding. “ I used to give a lot of work to my wife, such as going to the farm and do many other things at home. I didn’t know what a pregnant woman should do. But after NAWEZA [facilitators] came [as part of Tulonge Afya] and spoke with us, we realized that a pregnant woman should be given rest, should not be given hard works that can make her tired and even affect the infant in the womb which may end up in losing that child or be born with poor health. —NAWEZA male beneficiary, RORYA BR E A K THROUGH R ESE A RCH | AUGUST 2020 17 • Male and female beneficiaries of NAWEZA sessions perceived an increase in the number of women delivering in health facilities; twice as many males reported this. Facilitators of both NAWEZA and FURAHA YANGU sessions, CSOs, and government respondents echoed the finding on the perceived increase in the number of women delivering in health facilities. • Male and female beneficiaries of NAWEZA sessions did not discuss use of Sulfadoxine Pyrimethamine (SP) during pregnancy nor early breastfeeding initi￾ation—two behaviors that are primarily promoted at the health facility. • Several male beneficiaries of NAWEZA sessions perceived an increased use of ITNs (even during hot seasons) and an increased awareness of malaria in general. NAWEZA session facilitators echoed this finding. CSOs and government officials noted the increase in the number of people sleeping under mosquito nets in their communities. Some facilitators also perceived that misuse of nets has significantly reduced in the communities and many people are now using mosquito nets appropriately. They also perceived a reduction in malaria cases because of the USAID Tulonge Afya activities. “ We also see that the use of mosquito nets has increased as compared to previous times [before USAID Tulonge Afya]. People have got awareness and understand the advantage of using mosquito nets. As such, people have changed unlike before. —GOT official, TABORA The analysis is unclear regarding what effect the expo￾sure to malaria messaging had on other beneficiaries, including pregnant women, who are a vulnerable group. Malaria, while a key component of NAWEZA, is most heavily addressed through mass media, community mobilization and mid media, and vertical activities related to the rainy season in enhanced districts in PMI priority regions, as well as other activities such as school-based distribution campaigns. IR 1: Evaluation question 1b What USAID Tulonge Afya activities/approaches demonstrate promise for catalyzing positive change in gender and sociocultural norms within communities, enabling the practice of desired behaviors across priority health areas? The evaluation team combined document review and interviews with USAID Tulonge Afya, IPs, and UBA staff, to determine key promising approaches. • Audience insights to inform emotional drivers among targeted audiences. USAID Tulonge Afya used an approach linked to behavioral economics focused on audience insight gathering among the targeted audiences for the various SBCC platforms (youth, pregnant women, individuals with HIV/TB, parents/caregivers, and the general population) to identify the emotional drivers (i.e., hopes, dreams, “what’s in it for me to change”) and barriers (i.e., fear of side effects to family planning, limited opportu￾nities among women after marriage) to uptake of priority behaviors. Based on identified emotional drivers, USAID Tulonge Afya developed messages that spoke to what audiences desired, supported with practical actions they can take to have a successful life, a healthy pregnancy, and a healthy family. The identified emotional drivers focused on family values, recognition, status, achievement, independence, control, power, belongingness, and security. • Use of multiple channels to deliver SBCC mes￾sages with an emphasis on IPC. IPs and UBA staff agreed that to create positive changes in gender and sociocultural norms, target populations need multiple exposures to key messages through differ￾ent channels. Therefore, they perceived the use of multiple channels (including interpersonal commu￾nication, community mobilization, mass media, mid media, print media, and social media) to engage audiences as very effective. USAID Tulonge Afya project staff perceived one-on-one sessions (includ￾ing small group sessions) as a particularly important component. Compared to mass media, print media, and social media, audiences preferred one-on-one approaches because this approach gave them an opportunity to ask questions and get responses in real time. Audiences reported that the small group sessions (no more than 10 people) reached tar￾geted sub-groups of the population with necessary information as well as provided opportunities to do a deep dive into barriers to adoption of desired behaviors, and problem-solving to identify potential solutions to overcome these barriers. USAID Tulonge Afya noted that the project works closely with UBA and other IPs to address these types of barriers. 18 MIDTERM EVALUATION OF USAID TULONGE AF YA PROJECT IR 2: Evaluation question 2a Has USAID Tulonge Afya effectively empowered and engaged government and civil society structures at district and regional levels to support and facilitate delivery of quality SBCC? • Revitalization of the regional and district health promotion (HP) coordinators. GOT respondents reported that there were regional and district HP coordinator positions within the government prior to USAID Tulonge Afya, but these positions were largely unfilled and there was little clarity about what duties the positions entailed. USAID Tulonge Afya worked with the GOT to strengthen coordination of integrated SBCC activities by more clearly defining the roles and responsibilities of regional and district HP coordinators and empowering GOT to fill empty positions. Furthermore, HP coordinators (both at the regional and district level) reported that they received training through USAID Tulonge Afya in coordination and supervision of SBCC activities. The CSOs, IPs, and UBA staff reported that the regional and district HP coordinators were instrumental in coordinating SBCC activities at the regional and district levels. For example, regional and district HP coordinators also served as a link between SBCC implementing partners, including the CSOs and the regional and district authorities. Regional and HP coordinators also provided support in securing vari￾ous approvals for implementation of SBCC activities. • Strengthened SBCC data systems at the regional and district level. GOT staff and the CSOs imple￾menting SBCC activities on the ground reported that USAID Tulonge Afya contributed to SBCC data systems strengthening. They noted that prior to USAID Tulonge Afya, systems did not exist to cap￾ture various data on implemented SBCC activities. USAID Tulonge Afya developed output indicators (i.e., number trained, number exposed to messages), implemented an electronic SBCC data system (up to the CSO level), and trained CSOs and CVs on SBCC data reporting. CSOs reported that their CVs collect data (on implemented SBCC activities) on paper and CSO staff enter the paper forms into the electronic system that is directly accessed by USAID Tulonge Afya, IPs, and GOT staff with access to the database. • Strengthened CSOs’ skills in delivering quality SBCC. Almost all the CSOs in the visited districts reported their SBCC skills and capacity were strengthened through USAID Tulonge Afya. CSOs frequently cited audience engagement in SBCC design and delivery as an area in which they attained significant skills through USAID Tulonge Afya. CSOs reported that prior to USAID Tulonge Afya, audiences were not engaged in design and delivery of SBCC activities, and all the CSOs appreciated learning var￾ious approaches for engaging audiences. Similar to other respondents, CSOs felt that audience engage￾ment in SBCC design and delivery is a promising approach for effective SBCC interventions. Various CSOs reported that USAID Tulonge Afya strength￾ened their skills in monitoring the SBCC activities implementation. • Provision of financial support for supportive supervision to both the CSOs and GOT. GOT staff, particularly at the national level, reported that USAID Tulonge Afya significantly aided them in conducting supportive supervision of SBCC activities in the regions and districts. They noted that funding and support provided by USAID Tulonge Afya to strengthen their technical and coordination capacity enabled the Health Promotion Section (HPS) and the President’s Office Regional Administration and Local Government (PORALG) to conduct supportive supervision specifically for SBCC for the first time. The national level GOT officials reported that USAID Tulonge Afya’s provision of funding allowed them to conduct supportive supervision of the newly appointed HP coordinators at the regional and district levels. The regional HP coordinators reported that they were occasionally funded to conduct supportive supervision of the district HP coordina￾tors, who in turn conducted supportive supervision of CSOs and CVs implementing SBCC activities at the community level. While these findings are very encouraging, they raise the important question of the sustainability of funding these activities after the project concludes. IR 2: Evaluation question 2b How have USAID Tulonge Afya approaches for engage￾ment of audiences enabled or constrained the delivery of quality SBCC at the community level through USAID Tulonge Afya sub-partners and other USAID partners? BR E A K THROUGH R ESE A RCH | AUGUST 2020 19 The following elements enabled the delivery of quality SBCC at the community level through USAID Tulonge Afya sub-partners/other USAID partners. • Both facilitators and session participants found the use of participatory approaches (e.g., games) for running small group sessions engaging and enjoyable. Participants liked the games that were used within the small group sessions and found them engaging. In comparison to previously used methods of delivering SBCC sessions, session participants reported that the use of games: 1) facilitated easier/ better understanding of the session contents, 2) allowed participants to participate fully in the discus￾sions, and 3) entertained session participants while they learned. The games attracted new participants to the sessions while some participants attended sessions longer than required so they could continue participating in the games. “ To be honest, there are changes because of the way we teach was different in the past. I mean we use pictures in teaching, we use games, therefore, people understand better. —CSO, SINGIDA • Use of well-trained/skilled facilitators to run small group sessions. Respondents in certain areas, including IPs, CSOs, and GOT officials, mentioned that use of skilled facilitators was a key element in ensuring delivery of quality SBCC at the community level. As noted above, community members found small group sessions enjoyable. However, this was only the case when the small group sessions were run by skilled volunteers who were knowledgeable about the session topics, possessed the ability to properly facilitate the group sessions (using provided SBCC materials and tools), and addressed commonly asked questions by session participants. Engaged and skilled facilitators made the games entertaining and successful among participants. The USAID Tulonge Afya team reported an investment in the training of facilitators and in coaching and mentoring them in strong facilitation skills, including advancement of its ADDED codesign and delivery approach. • Use of health experts in SBCC activities targeting wider audiences. Various respondents, including CSOs, IPs, and GOT officials, noted that within com￾munity radio and theaters these health experts were an essential element to ensuring that communities receive correct health information. These respon￾dents noted that experienced care providers and coordinators of the five health programs are highly trusted and respected by community members, and when invited to provide health education in SBCC sessions, such sessions became highly attractive to the community members. These sessions drew larger numbers of people compared to small group discus￾sions and gave respondents the chance to ask more difficult and technical questions that could not be addressed by small-group facilitators. • Involvement and orientation of the Ward execu￾tive officers (WEOs) and village executive officers (VEOs). This involvement ensured that activities were implemented as planned. Facilitators and CSOs also noted that involvement of the WEOs and VEOs gave them the opportunity to use their routinely organized meetings to sensitize communities and attend ongoing SBCC sessions. The involvement of WEOs and VEOs also helped facilitate attendance, particularly for the SUBIRA sessions. Various parents were reluctant about letting their daughters attend SUBIRA sessions, as they believed that education on FP would lead to promiscuity. However, parents were receptive, and many allowed their daughters to attend the sessions where WEOs and VEOs were organizers. • Tulonge Afya’s use of high quality SBCC materials. Several respondents, including CSOs, IPs, and UBA staff, reported these as a key element for successful audience engagement. They reported that there was a great deal of creativity involved in ensuring messages and pictures resonated with the target audiences. In addition, CSOs mentioned that only quality materials would be approved by the Ministry of Health (MOH), and all USAID Tulonge Afya materials are approved by the MOH. Respondents mentioned they focus on consumer needs and address important concerns among the population (e.g., side effects from FP). Lastly, respondents men￾tioned that the accuracy of the materials helps to reduce disagreements within the GOT over services and commodities available based on the materials. • Use of local language for running the sessions. Various session facilitators noted that this was attrac￾tive to session participants in particular districts, as it allowed for easier comprehension of the session content/topics and provided participants more freedom to contribute their views during the ses￾sions. For example, in Newala district, the facilitator 20 MIDTERM EVALUATION OF USAID TULONGE AFYA PROJECT of a FURAHA YANGU session used local language to ensure that participants understood, and in Sengerema district, the facilitator allowed partici￾pants to contribute in their local language, which was noted to have improved participation. The evaluation team established the following elements as constraints on the delivery of quality SBCC at the com￾munity level through USAID Tulonge Afya sub-partners/ other USAID partners. • Limited number of volunteers for specific sessions. USAID Tulonge Afya staff reported that ideally, there should be one CV for every type of session in each ward (i.e., one for JUMA, one for SUBIRA, and one for NAWEZA/FURAHA YANGU sessions). However, session facilitators and CSOs reported that the ideal number of volunteers was not enough in large wards where households are widely scattered, and one vol￾unteer was not enough to cover all households they were assigned to cover. The evaluation team further determined that the ideal number of volunteers specified above was not present in many wards. Similarly, while there were volunteers present for certain sessions, they were missing from other ses￾sions at times, and youth facilitators often migrated frequently due to jobs and other commitments. • Two-day training among different cadres of facili￾tators is not enough. This led to varying degrees of skill sets across districts. The CSOs and IPs in almost all the visited districts expressed concerns that the two days allocated for training of CVs was not enough time for them to properly grasp the session content, including the games. Additionally, CSOs and IPs noted that the three days allocated for them to supervise session facilitators was also not enough time. CSOs and IPs added that, in addition to the limited time allocated for training the CVs, they had to provide extensive mentorship during supervision visits, which created a challenge in completing all required visits. Observations of the small group discussions noted the following: 1) various volunteers noted that they did not know how to involve all participants in the discussion, specifically quiet participants; 2) one facil￾itator ran the session in a very formal and serious manner which resulted in some participants not feel￾ing comfortable enough to participate; and 3) several facilitators read information that was supposed to be posed as questions, which altered the meaning of the game. Moreover, almost all the facilitators struggled to respond to the questions that were asked by ses￾sion participants, leading a few facilitators to guess answers that were not factual. • Lack of appropriate venues for running small group sessions. The evaluation team observed that most small group sessions were conducted outside under a tree, outside the village ward office, in one of the beneficiary’s houses, a church entrance (on the staircase), or in areas close to the road. Facilitators conducted one session in a local bar, and disrup￾tions occurred. The venues were chosen by the participants, per USAID Tulonge Afya guidance in the training. In one NAWEZA session, pregnant women seemed uncomfortable while sitting on the ground due to a lack of mats for sitting. Several JUMA and SUBIRA sessions required participants to stand for more than one hour due to a lack of seating. • Lack of clear mechanisms for involving WEOs and VEOs. Although WEOs and VEOs were instrumental in facilitating the success of many community based SBCC activities, in locations where WEOs and VEOs were not well-oriented or involved in USAID Tulonge Afya activities, they often hindered the work of CVs. In wards and villages where WEOs and VEOs were well sensitized about USAID Tulonge Afya project activities, such challenges were minimal or absent. IR 3: Evaluation question 3 Have USAID Tulonge Afya activities led to improved capacity, coordination, collaboration, and co-invest￾ment for SBCC at national and subnational levels among implementing partners, key stakeholders, and the GOT? Improved capacity for SBCC • At the national level, USAID Tulonge Afya built skills of the HPS staff in various aspects of quality SBCC design and delivery. USAID Tulonge Afya provided SBCC training and involved HPS staff in the design, delivery, and joint supervision of the inte￾grated SBCC platforms. “ I can say for the people in the health promotion section, whom I deal with. Initially they didn’t have enough skills but as we go along, their involvement has been very high. In every activity which TA conduct[s] in the regions, they involve people from the HPS. BR E A K THROUGH R ESE A RCH | AUGUST 2020 21 We usually have monthly meetings to update each other on how we are faring, [and] you can see the changes. They discuss the activities being implemented, and you can see that people keep developing skills and that they are knowledgeable. So, there is a very big change. —GOT, National level • USAID Tulonge Afya provided innovative training and engaged IPs on implementing the adult and adolescent platforms and other aspects of quality SBCC delivery, which specifically improved their technical skills in interviewing and message design and development. Implementing partners were heavily involved in the design of all USAID Tulonge Afya packages—from prioritization and needs identification to the design and review of produced materials and tools. “ Even the procedure they used when they conducted the interviews during the audience consultation, they used the projective technique that we didn’t know about and one we never used before. So, that one made people to open up so that you know their deep things so that you can know the kind of message you can create. With that, we were happy, and our main problem was to create attractive messages, and which can help us and truthfully Tulonge helped us with that and we see a big difference. —GOT, National level • USAID Tulonge Afya supported SBCC training among GOT staff other than HPS working under the specific health programs, a positive unintended outcome that spread capacity beyond the focus of the activity. Some staff reported developing addi￾tional skills through their close working relationship with USAID Tulonge Afya while supporting various SBCC campaigns under their programs. • USAID Tulonge Afya supported development of the health promotion indicators and SBCC data systems and dashboard, which improved monitor￾ing capacity. GOT officials, both at the national and sub-national level, reported that the availability of SBCC data and the dashboard improved their capac￾ity to monitor implementation of SBCC activities in districts where USAID Tulonge Afya operates. Improved coordination for SBCC • Centralization of SBCC materials production and/ or approval. UBA staff, IPs, and GOT respondents reported that one of the major changes in SBCC programming introduced by USAID Tulonge Afya was the centralization of SBCC materials production. Material design and production for the five health areas supported by USAID Tulonge Afya is conducted under the leadership of USAID Tulonge Afya through the HPS of the Ministry of Health, Community Development, Gender, Elderly, and Children (MOHCDGEC). • USAID Tulonge Afya, along with UNICEF, supported financially and technically the devel￾opment of key national guidelines, policies, and SOPs, which will help coordinate SBCC delivery throughout the country. These include National Strategy for Health Communication, National Policy Guidelines for Health Communication, and National Guide for Development and Approval of Health Communication Materials (Standard Operating Procedures) with the HPS. • Improved recognition of HPS as the coordinating body for SBCC activities in the country. Along with the centralization of SBCC materials production, respondents for this evaluation also reported that USAID Tulonge Afya contributed significantly to improving recognition of the HPS as a coordinating body for SBCC activities in the country at all levels (national, regional, and district). At the national level, USAID Tulonge Afya offered both technical and financial support to the HPS in coordinating SBCC activities with other vertical programs and IPs. The establishment/revival of the various platforms/ meetings (that were financed by USAID Tulonge Afya) that bring together various SBCC partners to identify SBCC needs for their programs and reach agreement on the key messages prompted this coordination. • Continued support to SBCC task forces/working groups improved prioritization of SBCC activities and materials. Several meetings were held relating to coordination of SBCC messages and activities in the country. These meetings included a committee that meets every two months (or is scheduled on an ad hoc basis as needed) to approve SBCC materials that are submitted to the HPS, a biannual health promotion coordinators meeting, and a technical working group meeting that includes all partners dealing with SBCC to decide on their priority SBCC 22 MIDTERM EVALUATION OF USAID TULONGE AFYA PROJECT needs. A good example noted by USAID Tulonge Afya is the project supported FURAHA YANGU Task force, which meets on a needs basis apart from the standing quarterly meeting. Improved collaboration for SBCC • Partners involved to identify SBCC needs for their programs and design of SBCC materials. In the past, IPs and UBA noted that they were offered materials that were already developed by the respective health programs. Now, their involvement in the design of SBCC materials for their programs was reported as a new best practice implemented by USAID Tulonge Afya. Partners are able to include their specific needs, which makes it more applicable to their specific objectives. • Improved engagement in SBCC design and deliv￾ery with the GOT. GOT officials noted that, prior to USAID Tulonge Afya, various partners planned and implemented SBCC activities under their programs without involving the government. USAID Tulonge Afya currently implements all SBCC activities while engaging the government, both at the national level (through the materials approval process by the HPS) and the sub-national level (through HP coordinators). • Improved partner collaboration to link demand creation activities and services among SBCC and service delivery partners. CSOs, IPs, UBA staff, and GOT respondents cited that USAID Tulonge Afya and UBA partners in regions and districts are working closely together while planning their community activities to ensure that demand creation activities (supported by USAID Tulonge Afya) were linked to services (supported by UBA), especially for commu￾nity theater events. In some areas, partners noted that service delivery during community theater events was organized through the public facilities. • Early and meaningful engagement of GOT staff in SBCC design and delivery. CSOs reported that this aided in smooth implementation of commu￾nity activities. They noted that GOT officials were very supportive of their activities when they were involved from the beginning (i.e., at the activities planning stage) and throughout their implemen￾tation. A few CSOs reported that the district GOT officials preferred transparency in terms of the details of planned activities as well as available resources. CSOs that shared budgets and workplans with district officials noted that this practice was much appreciated by the local government, as it increased transparency and led to a better relation￾ship with the GOT. USAID Tulonge Afya staff and one GOT staff reported that having an experienced con￾sultant working with them (placed by USAID Tulonge Afya) was instrumental in facilitating engagement of the government. Improved co-investment for SBCC • USAID Tulonge Afya shared office space/buildings with UBA staff. Shared office space facilitated close collaboration during planning and implementation of USAID Tulonge Afya’s and UBA’s various activities and led to reduced costs for office rent for both projects. • Joint supervision of SBCC and service delivery activities. To achieve meaningful supervision of the integrated platforms, health promotion coordinators are required to work closely with coordinators of specific health programs, which are supported by USAID Tulonge Afya and UBA in the various regions and districts. Where it was done well, it facilitated co-investment among USAID Tulonge Afya and UBA, (e.g., USAID Tulonge Afya and UBA reported sharing vehicles as well as expenses for the government staff). However, it should be noted this practice was not consistently implemented in all regions with similar quality. • Co-investment between USAID Tulonge Afya and UBA during community theaters. A balance was struck so that USAID Tulonge Afya funded commu￾nity mobilization while UBA funded service delivery activities. Where USAID Tulonge Afya was absent, UBA also funded community mobilization activities for their outreach services. IPs, GOT, and UBA staff reported that USAID Tulonge Afya provided support in developing quality SBCC materials for their pro￾grams, and support for SBCC needs specific to their programs. For example, one UBA partner reported that they received support from USAID Tulonge Afya in creating demand for VMMC services. Other UBA partners reported that they received SBCC support from USAID Tulonge Afya to increase male involve￾ment in their activities, and to trace PLHIV who are on ART but were lost to follow-up. BR E A K THROUGH R ESE A RCH | AUGUST 2020 23 Cross-cutting: Question 4 How has USAID Tulonge Afya defined and measured success, and which activities/approaches are demon￾strating success across each of the three IRs? USAID Tulonge Afya described their monitoring and evaluation system as adaptive in nature, and informed by multiple sources, as illustrated in Figure 4. The project’s Activity Monitoring and Evaluation Plan (AMEP) is robust with strong indicators regarding knowledge, attitudes, and practices (KAP) across the packages among key beneficiaries, as well as indicators to measure IR2 and IR3. However, the quarterly and annual reports do not report in depth on KAP information and focus more on the omnibus survey data. The baseline survey, literature review, and qualitative assessments were very helpful in determining the audiences and priority messages across the five health areas. The omnibus surveys were able to show the outputs regarding the number of target bene￾ficiaries reached by specific channels. Media monitoring reports and assessments, program community reports, and sentinel survey data were not discussed during the evaluation, as data from the sentinel survey was not received in time for the evaluation. • Value of ADDED and human-centered design (HCD) approaches. Respondents cited audience engagement in every stage of SBC intervention design and delivery as a highly promising strategy for catalyzing positive change in gender and socio￾cultural norms, enabling the practice of desired behaviors within communities. As discussed, USAID Tulonge Afya uses ADDED and HCD approaches. Through the ADDED approach, USAID Tulonge Afya engaged audiences from the initial stage of activity design in determining key barriers hindering the practice of targeted healthy behaviors in communi￾ties, to the design and delivery of SBCC interventions for addressing the barriers. The ADDED approach illustrates how USAID Tulonge Afya highlights the use of specific social and behavior change tactics including empowerment, demand generation, skills building, behavioral economics, and social action, tailored to where audiences fall along a behavior change continuum. HCD was used by USAID Tulonge Afya during the design stage of SBCC interventions; audiences were engaged in a participatory manner through the proposed SBCC approaches, messages, and materials. Furthermore, USAID Tulonge Afya trained and mobilized members of the target com￾munities, who work as CVs, to deliver the developed SBCC interventions using participatory, facilitative (rather than instructive) approaches. Several respondents for this evaluation, including CSOs, IPs, GOT, and UBA staff perceived that this FIGURE 3 USAID TULONGE AFYA’S AMEP APPROACH ALL PROJECT DATA DQA/supportive supervision Baseline survey Media monitoring reports assessment Program community reports Sentinel and Omnibus Survey National health statistics (DHIS2) Literature review and qualitative assessments 24 MIDTERM EVALUATION OF USAID TULONGE AFYA PROJECT approach resulted in SBCC interventions and mes￾sages that are context-relevant, more acceptable to audiences, and potentially more effective in cata￾lyzing positive change in gender and sociocultural norms, enabling the practice of desired behaviors among the targeted audiences. As part of the ADDED approach, USAID Tulonge Afya collected continuous feedback from SBCC activities implementation, which allowed for real time adjustment of the activities/ messages based on audiences’ needs and priorities. “ With the coming of TA, my capacity and team capacity for education provision [SBCC] has increased. There is a concept of co-design and co-delivery. We used to think that what you understand is what you can enforce to others. With the current approach, we work as facilitators, we involve the target group for opinion in whatever we do. We can see that facilitation skills are increasing. There is another concept of 80/20 role. That means that, the target group is given more chance to discuss the issues, and we believe that they know much more than we do. Therefore, we need to evoke their ideas. —CSO, SENGEREMA Cross-cutting: Question 5 What have been internal limitations and/or external constraints in achieving USAID Tulonge Afya objectives for each of the three IRs? The following findings relate to possible external con￾straints and challenges related to the IRs that USAID Tulonge Afya can work with UBA and other IPs to address. • Lack of confidentiality among providers offering HIV testing services. FURAHA YANGU beneficiaries and facilitators cited a lack of confidentiality that discouraged some community members from seek￾ing an HIV test at facilities in their area. They added that, in such areas, care providers shared clients’ HIV results with other members of their families/commu￾nities without clients’ consent. • Disrespectful care among care providers. Male and female NAWEZA beneficiaries reported negative attitudes among providers as a barrier to service utilization, particularly related to ANC, delivery, and postnatal care (PNC) services. Youth and adults also noted harsh language among care providers when accessing FP services. Particularly, female NAWEZA beneficiaries noted that many care providers were not supportive when they returned to the provider after experiencing side effects from their contracep￾tive method. In some districts, JUMA and SUBIRA beneficiaries reported that they were harassed by care providers, or that they were denied condoms (in the case of JUMA) and MCMs (in the case of SUBIRA) because of their young age. • Referral and linkage mechanisms for services were not always available at the community level after demand was created by USAID Tulonge Afya activities. CSOs and IPs reported that this hindered practice of healthy behaviors among sensitized com￾munity members. This was especially true for those who wanted to receive an HIV test after the session but found it was not available at that location, nor was there a health clinic close enough to provide testing. • Sub-optimal quality of health services linked to community theaters. In several districts, HIV coun￾seling and testing was conducted in locations where other clients could hear and see the procedures. As a result, older people lined up for HIV testing in the morning, while youth appeared to be utilizing this service toward the end of the day when there were few to no other clients queuing for the services. Moreover, the evaluation team reported that the time providers spent with every client for pre-test and post-test counseling was not enough. However, this observation is outside the scope of USAID Tulonge Afya and should be addressed through UBA. Cross-cutting: Question 6 What mitigation strategies have been considered to address limitations and constraints, and how effec￾tively have they been adopted? IR1 mitigation strategies • Facilitators noted while they commonly used pictures to explain commodities, e.g., modern contraceptives and female condoms, participants preferred seeing the actual items during small group sessions. This was addressed and now modern contraceptive prod￾ucts for demonstration are available for the FY20 BR E A K THROUGH R ESE A RCH | AUGUST 2020 25 implementation period. Condoms were available for demonstration previously, but when there was a shortage, this also affected the volunteers’ stocks. IR2 mitigation strategies • As previously noted as a challenge related to linkages of SBCC data systems to the DHIS2, efforts have been made to link the SBCC data with DHIS2 and all the tools (HMIS book 3 and 10) have been revised to include all SBCC indicators approved by MOH through HPS. The indicators and all data collection forms have been developed in the DHIS2 and were to be rolled out after the launch of the system, which was postponed. • To address the challenges of WEO and VEO involve￾ment, USAID Tulonge Afya is supporting monthly meetings, wherein CSOs can identify WEOs’ and VEOs’ challenges and try to engage them better to improve relationships between themselves and volunteers. • To address the limited time allocated among CSOs and IPs to supervise session facilitators and the added burden of extensive mentorship for CVs during supervision visits, USAID Tulonge Afya with the IPs is conducting monthly meetings with all facilitators where common issues observed during supervision and mentorship are addressed. Through monthly meetings and refresher training, USAID Tulonge Afya addresses all common issues identified during supervision. Mentorship is also provided for facilitators that struggle the most after reports from supervisors are reviewed. IR3 mitigation strategies • To address the lack of collaboration between HP coordinators among specific health areas, leading to a lack of comprehensiveness reported by HP coordinators in integrated SBCC, USAID Tulonge Afya supported several SOPs including a coordination framework and coordination SOPs. Cross-cutting: Question 7 What are the facilitators/barriers to shifting from ver￾tical to integrated SBCC programs, particularly within the youth and adult platforms? Facilitators to shifting from vertical to integrat￾ed SBCC programs • Integrated SBCC approaches noted as cost-effec￾tive and efficient. GOT staff and IPs mentioned cost and time savings, for both the health system and clients, as advantages of integrated SBCC. “ I prefer the integrated approach because, it first cuts down costs. For instance, we can merge funds for HIV and fam￾ily planning and work in one setting. Thus, the cost would be less than when everyone could work individually. It also has wide benefit unlike when you target one area. —GOT, National level Barriers to shifting from vertical to integrated SBCC programs • Limited understanding of what integrated SBCC entails in practice: Despite many government officials’ knowledge of what integrated SBCC means as defined by this activity, some still found it hard to understand its implementation in practice. For example, one of the SBCC coordinators for a specific health program thought that the integrated SBCC approach required him to start supporting SBCC activities in other health areas in which he was not competent. • The national health communication strategy is not signed to date. This limited its use to guide integrated SBCC implementation. USAID Tulonge Afya supported the HPS to develop National Health Communication Policy Guidelines, a National Communication Strategy, and SOPs to operationalize the policy and strategy. The project has been follow￾ing up with the HPS to provide the final endorsed documents after submitting them to the MOH authorities in April 2019. It was noted that it has been a lengthy and complex process. The documents are still in the Chief Medical Office awaiting final signatures. 26 MIDTERM EVALUATION OF USAID TULONGE AFYA PROJECT • Competing funding priorities between integrated and vertical SBCC programming. While USAID Tulonge Afya supports only a limited number of integrated messages and health areas under their programs, vertical program staff expressed major concerns regarding the lack of funds to support health messages that are not covered by the inte￾grated SBCC messaging. This concern may be beyond USAID Tulonge Afya’s role as it speaks to broader donor priorities. Cross-cutting: Question 8 How have USAID Tulonge Afya organizational and management structures, systems, processes, and pro￾cedures enabled or constrained the success of capacity, coordination, and collaboration, especially within its integrated SBCC program? Enabling factors to improving capacity, coor￾dination, and collaboration within integrated SBCC program • Existence of national communication strategy and SOPs. This was noted as key in providing guidance to the SBCC partners; however, as noted in cross-cut￾ting question 7, these are still not finalized. However, the HPS has used the content of these guidelines in the coordination of SBCC activities. The current approval processes for SBCC materials are based on these guidelines. The current de-facto coordi￾nation structure of SBCC activities from national to district and community levels is derived from the coordination framework described in the National Strategy for Health Communication that the project supported.10 • The availability of funds to support technical working group (TWG) meetings. These meetings were instrumental in facilitating the coordination of SBCC activities under the integrated approach. Prior to USAID Tulonge Afya, some of the TWGs were pres￾ent, but members rarely met due to lack of funds. With USAID Tulonge Afya’s support, TWG meetings take place more frequently, which facilitates imple￾mentation of the integrated SBCC programming. However, this raises the question of the sustainability of the TWGs and support after the project concludes. • Ensuring government leadership. This was very crit￾ical to switching from the vertical to the integrated approach to SBCC, as it not only facilitated effective coordination of the SBCC activities (both at the national and sub-national levels), but also facilitated government ownership of the SBCC programming. USAID Tulonge Afya is doing a good job in ensuring government ownership through working closely with the HPS of the MOHCDGEC, which, despite the challenges cited above, plays a coordination role for SBCC activities among other vertical programs and IPs. • Ensuring participation of key stakeholders in SBCC design and delivery. Various UBA staff and IPs appreciated that they were meaningfully involved in the SBCC needs identification and design of materi￾als, which facilitated the creation of SBCC materials that met their specific programmatic needs. • USAID Tulonge Afya’s flexibility. UBA staff and IPs expressed their appreciation for USAID Tulonge Afya’s flexibility to support specific SBCC needs for their programs. Additionally, GOT staff from the vertical programs stated that USAID Tulonge Afya’s flexibility in terms of incorporating their feedback made the integrated approach more easily accept￾able to them. USAID Tulonge Afya staff echoed this finding, as they appreciated being able to offer support that was not part of their work plan, e.g., supporting partners based on their SBCC needs. Constraining factors to improving capacity, co￾ordination, and collaboration within integrated SBCC program • A lack of clear guidance on the roles and responsi￾bilities of SBCC coordinators of vertical programs vs. HP coordinators of the integrated SBCC pro￾grams. The evaluation team noted that there is still a lack of clarity on roles and responsibilities of SBCC staff of vertical programs compared to the HP coor￾dinators of the integrated SBCC program who work under the HPS. The evaluation team established through further inquiry that HP coordinators working at the regional and district levels did not collabo￾rate with coordinators of vertical programs at the national level as planned. The lack of collaboration caused confusion regarding their role for coordina￾tors of vertical programs at sub-national levels, who previously worked with sub-national HP coordina￾tors. This concern was also expressed at the national level but was more pronounced at the regional and district level. BR E A K THROUGH R ESE A RCH | AUGUST 2020 27 • At the national level, challenges exist regarding strategies for collaboration between USAID Tulonge Afya, HPS, and vertical programs. The evaluation team noted several challenges related to the ways in which USAID Tulonge Afya collabo￾rated with HPS and the specific health programs during the implementation of the integrated SBCC programming. Some staff of the vertical programs stated that they felt left out and expressed a concern that USAID Tulonge Afya focuses too much on the HP unit. On the other hand, the HPS stated that they felt that USAID Tulonge Afya and vertical programs were implementing various SBCC activities without involving them. Additionally, the HPS expressed a concern that since they have a mandate for coor￾dinating SBCC activities under all the health areas (approximately 12 different health areas), they faced challenges because USAID Tulonge Afya only sup￾ports five different health areas. • As different HP coordinators were not included in certain integrated activities or meetings, this affected the comprehensiveness reported by HP coordinators in integrated SBCC, some of whom were reported as not capturing SBCC activities supported by SBCC coordinators of specific health programs under other partners. “ I am not responsible for only TB or malarial [sic]. My responsibility is like 12 programs. Therefore, if I call a malaria person, I must also call a person for immu￾nization, so it is better either we leave that meeting, or we continue with the meeting with all the people I invited. GOT, National level • Differences in the GOT’s (July–June) and USAID Tulonge Afya’s (October–September) fiscal years. This posed challenges for HPS and USAID Tulonge Afya’s workplan development. HPS staff noted that they included activities already approved by the government in the USAID Tulonge Afya plan, but various activities were removed from the USAID Tulonge Afya plan at a later stage (after disapproval by USAID). This posed a challenge for HPS, as it was difficult for them to get funding to cover omitted activities after their plan was approved by the government. • Informing stakeholders frequently and early for key meetings. Various GOT staff stated that since they have busy schedules, they asked to be informed at least two weeks in advance of planned meeting dates. Without advance notice, they stated that it was difficult for them to attend the meetings. • Limited support for SBCC infrastructure. The HPS noted that, while USAID Tulonge Afya made signifi￾cant capacity building efforts of HPS staff in different areas of SBCC programming, the department still faces a challenge of poor infrastructure to support coordination of SBCC activities nationally. However, this type of support is outside of the mandate of USAID Tulonge Afya as it relates to donor funding restrictions. “ …They helped me have good skills, they helped people under me to have good skills, and they will have helped to produce good messages for 5 years. But when the project ends, I will remain with the skills, but I won’t be able to produce. Helping us like how World Bank has helped us to build the studio, it has helped a lot that’s why we are proud to be able to produce those materials, but we are not good when it comes to printing. The infrastruc￾ture is not good. Therefore, there is time we ask ourselves, whether we are really receiving something, or we should stop working with [USAID] Tulonge [Afya] because of this, there is a big gap. At least they [USAID Tulonge Afya] should have a small component of capacity building in the infrastructure. —GOT, National level 28 MIDTERM EVALUATION OF USAID TULONGE AFYA PROJECT Recommendations Based on the findings of this midterm evaluation, the evaluation team proposed recommendations to enhance the effectiveness of USAID Tulonge Afya’s strategies and activities as the program enters its remaining years and identifies opportunities for the follow-on project. The recommendations are presented under each of the three program intermediate results (IRs). Under each IR, the recommendations are divided into programmatic and structural recommendations, followed by the recommen￾dations for the follow-on project. USAID should work closely with USAID Tulonge Afya to determine which programmatic and structural recommendations are feasible within the timeframe remaining in the life of the project. Recommendations that may not be feasible during this timeframe can be incorporated into the follow-on activity. IR 1: Improved ability of individuals to practice healthy behaviors The evaluation team proposes the following recom￾mendations based on beneficiaries’ and stakeholders’ knowledge, attitudes, and perceptions regarding key health behaviors promoted by the project. Programmatic recommendations Several recommendations cut across the platforms, ses￾sions, campaigns, and other channels, while others are specific to each platform, as illustrated in the following sections. Overall recommendations for IR1 USAID Tulonge Afya should review the questions asked by participants during the sessions (including those observed during the mid-term evaluation) to under￾stand what is resonating most with target audiences, and what needs reinforcement. USAID Tulonge Afya can compare what facilitators are most comfortable answer￾ing and what needs to be reinforced and revised within the session guides and other materials. The evaluation team recommends a deeper emphasis on individual self-efficacy and risk perception, as the FGDs among ben￾eficiaries illustrated more of a shift in descriptive norms rather than inductive norms. This could be emphasized more explicitly during the participatory games and other activities that beneficiaries enjoy. Reinforce the discussion of all health topic areas in the first session of the platforms/campaigns to ensure participants have a clear understanding of the entire health package in sessions. Secondary to beneficiaries’ challenge of committing to multiple sessions, USAID Tulonge Afya should reinforce during the first session IR1: QUESTIONS 1A AND 1B—HIGHLIGHTED FINDINGS • Among beneficiaries, the health areas that were discussed most frequently related to dual protection from condom use, teenage pregnancy, HIV testing, HIV-related stigma, use of MCMs, male involvement, and ANC. • Resonance of similar key messages provided to both adult and youth men and women were reported differently. • Beneficiaries reported perceptions regarding changes in perceived community norms and practices more frequently than actual individual changes in norms and practices. • All six districts reported similar findings, and there were no notable differences between enhanced T-MARC and TCDC-supported districts. • A limited number of health areas were rolled out among the platform and campaign sessions prior to the evaluation. BR E A K THROUGH R ESE A RCH | AUGUST 2020 29 under all platforms a summary of all the key messages that will be discussed in detail in subsequent sessions. FY20 and FY21 will include multiple new messages, which will be critical to balancing message mix and retention of beneficiaries. The activity should also explore what has worked in other programs for additional participant retention strategies. As new messages are introduced in the sessions, USAID Tulonge Afya should work with CSOs and IPs to understand how messages resonate with the beneficiaries, and if initial messages still resonate, or if additional messages are overwhelming. New topics included after the evaluation include getting an HIV test (if at risk), adhering to HIV treatment (if living with HIV), going for voluntary medical male circumcision (VMMC), TB, viral load monitoring among PLWHA, index testing, retention and care for HIV, viral suppression, etc. USAID Tulonge Afya should utilize CSOs to undertake observa￾tions as part of routine monitoring to understand the dynamics within sessions. Post-session discussions with the facilitators and with participants to understand the comprehension and clarity of message content can help improve the balance of the message mix. This is import￾ant because at the time of the evaluation, limited health areas were rolled out among the different packages (detailed in the background section). Ensure service delivery IPs are keeping health pro￾viders up to date with the most relevant technical information to parallel the messaging in USAID Tulonge Afya sessions. USAID Tulonge Afya can collaborate with service delivery IPs in their districts, especially with UBA in the Southern region, as FHI 360 is a sub to UBA, to ensure that re-orientation of service providers on USAID Tulonge Afya community mobilization strategies leads to increased service acceptance and uptake. For example, when the clients from NAWEZA platforms are encour￾aged to start ANC early and aim for eight ANC visits, the service providers should be prepared to receive these clients early with the appropriate information and coun￾seling. Consider updating any current job aides or IEC materials through USAID Tulonge Afya’s UCD approach among health providers. If an endline survey takes place, ensure it has a qual￾itative component to understand the added value, positive and negative unintended consequences, and changes in knowledge, attitudes, behaviors, and norms, among other health areas not included in this evaluation. As USAID Tulonge Afya only rolled out a limited number of integrated health activities prior to the mid-term evaluation among the platforms and other sessions, this is a limitation to the mid-term evaluation. It is important to understand the depth of integration USAID Tulonge Afya reached among its audiences, which can be captured through qualitative research. In addition, the survey should include other channels (e.g., mass media, mid-media, SMS). It will be important to capture which channels have been most effective in affecting knowledge, attitudes, behaviors, and norms among target audiences. Youth platform (SITETEREKI) Even though similar messages are provided to male and female youth, knowledge of correct condom use needs to be reinforced among female youth. Facilita￾tors are now using sample contraceptives among the youth in sessions. For female youth, demonstrations on how to use a condom should be promoted during sessions using locally derived items, like bananas, and included more frequently during games and question and answer sessions to encourage dialogue around the topic. Adult platforms and campaigns (NAWEZA and FURAHA YANGU) Re-evaluate the resonance and balance of HIV-re￾lated issues (awareness of benefits of testing, fear reduction around testing) among female beneficiaries under FURAHA YANGU. As the findings showed that fewer females than males noted specific knowledge, perceptions, and descriptive norms in HIV-related areas, USAID Tulonge Afya should utilize gender disaggregated sessions (e.g., women’s groups) to discuss sensitive topics without male influence.11 Evidence shows that creating safe spaces for women to discuss sensitive issues, such as seeking HIV testing and care, can reduce stigma and increase self-efficacy toward accessing health care.12 Incorporate SBCC messages around “safer conception strategies” in FURAHA YANGU and NAWEZA packages. For example, in the case of a serodiscordant couple who wishes to conceive a pregnancy, SBCC messages could include how to use FP, ART, and peri‐conception pre-exposure prophylaxis (PrEP) to reduce the risk of HIV transmission to the HIV-negative partner and their infant.13 Revisit the focus of the NAWEZA package during sessions to ensure there is a balance in topic discus￾sion that is led by beneficiaries’ needs, corresponding to the ADDED approach. As mentioned earlier, the 30 MIDTERM EVALUATION OF USAID TULONGE AFYA PROJECT NAWEZA package is designed to promote gateway (prior￾ity) behaviors, e.g., early ANC attendance and attending four or more ANC visits. ITN and IPTp use did not reso￾nate with beneficiaries, among other topics included in the package. It should not be assumed that beneficiaries will understand the importance of and attend early ANC and therefore act on additional health behaviors. During facilitator refresher trainings, USAID Tulonge Afya should reinforce the participatory, facilitative approaches to improve resonance among beneficiaries leading to improved uptake of additional health behaviors. Document the “multiplier effect” among FURAHA YANGU and NAWEZA sessions as a promising practice. Both platforms reinforce messages around HIV testing and treatment. The sessions each have specific target audiences; however, some of the FURAHA YANGU primary audiences are also targeted under NAWEZA. This results in the potential for target beneficiaries to have multiple touch points to increase exposure to specific desired behaviors. This is a promising practice that demonstrates the benefits of having multiple reinforcing channels with similar messages that resonate across different target audiences. This “multiplier effect” has the potential to increase effectiveness of programming as well as cost savings across multiple channels, including the activity’s mass media programming. Document the process behind the behavioral science approach, including audience insights, and evolution of multi-channel SBCC approaches. USAID Tulonge Afya used emotional drivers (i.e., hopes, dreams, “what’s in it for me to change?”) to shape the packages and mes￾sages related to the formative work regarding audience insights. The focus of each package (e.g., FURAHA YANGU targeting at-risk men and reducing stigma and normalizing HIV testing and treatment as key objectives of the campaign) came from the audience insights and emotional drivers used in the sessions. The results of these efforts came out strongly in the findings as con￾tributing to men resonating well with the messages and beneficiaries perceiving reduced stigma around HIV in communities. This should be documented as a promising practice for other SBCC IPs. IR 2: Strengthened community support for healthy behaviors The evaluation team proposes the following recommen￾dations based on stakeholder’s perceptions of USAID Tulonge Afya’s ability to support and facilitate delivery of quality SBCC among the GOT, USAID Tulonge Afya sub-partners, and other USAID partners. Programmatic recommendations Include district HP coordinators in national training of trainers (TOT) currently provided to SBCC coordinators working under the IPs, CSOs, and RHPCOs. RHPCOs, SBCC coordinators, and CSOs received TOT from USAID Tulonge Afya; however, DHPCOs received trainings together with CVs, limiting their capacity to supervise. DHPCOs will be more empowered and knowledgeable to conduct supportive supervision to the CSOs and CVs if they are included in the TOT training. Integrate outcome SBCC indicators into the national District Health Information System (DHIS)2 and assist the GOT in understanding the rationale for inclusion and uptake nationwide. USAID Tulonge Afya is working IR2: QUESTIONS 2A AND 2B: HIGHLIGHTED FINDINGS • USAID Tulonge Afya training and other support to regional/district level HP coordinators was key to improve coordination and implementation of SBCC activities. • Data are being used more frequently for decision making, but they are currently limited to outputs and do not include ideational indicators. • While supportive supervision was improved among the GOT and CSOs, challenges remain in reporting and adaptive management. • GOT health experts were helpful in clarifying technical knowledge among sessions. • Coordination among government volunteers and project staff remains a challenge. BR E A K THROUGH R ESE A RCH | AUGUST 2020 31 with HPS to test inputting the SBCC data into the DHIS2 in enhanced districts. HPS wants to confirm the indicators are working as expected before adding to the live DHIS2 module that is used by all districts. The Minister of Health planned to officially launch inclusion of SBCC indicators in the live module in late March; however, due to COVID-19, the launch was postponed until further notice. When the launch does take place, USAID Tulonge Afya should work with national and regional GOT officials to implement SBCC data integration with the DHIS2 in other districts to improve accessibility of SBCC data. Establish biannual refresher training, including practi￾cums, among session facilitators, based on outcomes from supportive supervision to improve session facilitator competency. USAID Tulonge Afya currently conducts annual refresher training, and during monthly meetings, all volunteers are reoriented on common issues identified during supervision, mentorship, or feed￾back collected from community radio frequently asked questions. The frequency of refresher training (including experiential learning and on the job training for facilita￾tors) should be increased to twice a year and planned and conducted in close collaboration with regional/dis￾trict HP coordinators and the coordinators of respective health programs. Establish a mechanism to ensure that no beneficiary questions go unanswered and answers are factually correct. USAID Tulonge Afya can consider launching an “on call” health provider/expert who facilitators can call or flash from their mobile phones during small group sessions when difficult questions arise that they cannot answer themselves. This can be combined with a system for capturing and responding to frequently asked ques￾tions (FAQs) under the various platforms, and session facilitators can be provided with an “FAQ” booklet. The FAQs can serve as monitoring tools for understand￾ing how the audience is responding to the session’s structure, and supervisors can work with facilitators to address these issues during regular check-ins. The out￾comes of the FAQs can also influence slight adaptations in the games and materials used during the sessions to ensure the ADDED approach of audience engagement is prominent. Bolster community support for health behaviors at all levels, and in all stages of SBCC design and delivery. USAID Tulonge Afya should continue to strengthen early and meaningful involvement of local government and focus additional effort on the development of guidance for the CSOs on how to formally engage WEOs and VEOs in SBCC activity orientation and implementation. This proved to be effective in improving coordination and implementation at the district level by USAID Tulonge Afya. Structural recommendations Set up criteria for proper coverage among volunteers, especially youth peer champions under SITETEREKI. Based on discussions about the migration patterns of youth seeking new jobs or relocating, USAID Tulonge Afya should take stock of how currently available volunteers are distributed. USAID Tulonge Afya should consider a “twinning” recruitment and on-the-job training set up for future volunteers, which allows experiential learning while building trust among newly trained volunteers. Develop guidance with minimal requirements for SBCC sessions venues. Beneficiaries had difficulty in certain sessions with confidentiality, comfort, and other issues with venues chosen for the sessions. USAID Tulonge Afya should develop session guidance based on the noted challenges, with specific guidance for venue require￾ments for each community session. For example, youth sessions focusing on sensitive or stigma-related topics should require confidential spaces and spot checks to ensure this is taking place regularly. IR 3: Improved systems for coordination and implementation of SBCC interventions Programmatic recommendations The evaluation team proposes the following recom￾mendations based on the findings regarding how USAID Tulonge Afya activities led to improved capacity, coordi￾nation, collaboration, and co-investment at national and subnational levels among IPs, key stakeholders, and the GOT. Collaborate with HPS at all levels of government to improve coordination of SBCC-related meetings and activities. USAID Tulonge Afya currently works with HPS to ensure all its activities are included in its work plan. USAID Tulonge Afya also holds quarterly progress reviews with the HPS and has a broader memorandum of understanding with the MOH. USAID Tulonge Afya should strengthen regional and district level coordination, as 32 MIDTERM EVALUATION OF USAID TULONGE AFYA PROJECT well as linkages between HPS central and all regional SBC practitioners at the national level. Cross-cutting recommendations The following section highlights key recommendations for the cross-cutting questions that correspond to the findings. Crossing-cutting question 4: How USAID Tu￾longe Afya defined and measured success, and which activities/approaches are demonstrating success across each of the three IRs Ensure the adaptive management and learning agenda set forth in the AMEP is fully utilized for better deci￾sion making and program adaptation. As stated in the findings, the AMEP focuses on key aspects of monitoring and evaluation, adaptive management, and learning agendas to improve outcomes during the project among the three IRs. It has been noted some CSOs are reporting they are using exposure data to improve reach to engage audiences. Feedback from USAID indicated that certain recommendations were not seen in the field (e.g., note￾books among facilitators to capture questions, sensitiza￾tion of GOT on integrated SBCC). When the BR evaluation team conducted follow up meetings with USAID Tulonge Afya, the team learned that TB was inadvertently left out of the FURAHA YANGU manual. USAID Tulonge Afya should work with USAID to ensure that the recommen￾dations from the USAID team are fully integrated into program implementation, just as the recommendation to specify who are the high risk individuals was added to the FURAHA YANGU materials based on USAID feedback. Recommendations regarding supportive supervision for adaptive management and program improvement are cross listed under IR2. Crossing-cutting question 5: Internal limitations and/or external constraints in achieving USAID Tulonge Afya objectives for each of the three IRs UBA should work closely with USAID Tulonge Afya to ensure UBA is focusing on health providers’ biases and behaviors to improve youth friendly and client-cen￾tered services that respect clients’ privacy both at IR 3: QUESTION 3: HIGHLIGHTED FINDINGS • USAID Tulonge Afya was successful in building skills of the HPS and other GOT staff at the national level and trained IPs in development and implementation of the SBCC interventions. • The DHIS2 is piloting output level indicators expected to roll out to all districts in the coming year which has helped prioritize channels based on exposure data. • The project, along with UNICEF, technically and financially supports the development of an SBCC Strategy accompanied by SOPs, and SBCC materials. • CSOs reported that early and meaningful engagement of GOT staff assisted in smooth project implementation. • Co-investment by USAID Tulonge Afya and UBA was facilitated through shared office space and joint funding for supportive supervision and community theaters. CROSS-CUTTING QUESTION 4: HIGHLIGHTED FINDINGS • Participatory approaches are appreciated by the GOT, IPs, and beneficiaries (e.g., ADDED/ HCD approaches). • Project quarterly and annual reports do not report enough detail on knowledge, atti￾tudes, and practices, and it is unclear how frequently the project used the DHIS2 data to validate its interventions. • Some recommendations from USAID field visits are still left to be undertaken by the project (e.g., notebooks among facilitators to capture questions, sensitization of GOT on integrated SBCC). BR E A K THROUGH R ESE A RCH | AUGUST 2020 33 facilities and community theaters. One example is the client-provider promise (NiZii, implemented by Break￾through ACTION Zambiaa ), which worked well to ensure the client and provider agree to conduct confidential, respectful, and transparent dialogues and practices, and are held accountable to this agreement. USAID Tulonge Afya has developed similar approaches and shared them with UBA partners. For example, under its packages USAID Tulonge Afya have provider buttons and promises, provider trainings, and other products; however, these have been rolled out by UBA on a limited basis. USAID Tulonge Afya should align work planning and activities in districts where they overlap to ensure rollout continues. This can help address findings that deter women from seeking ANC and PNC and youth from receiving condoms or MCMs and improve confidentiality during HIV testing services. Close the gap between demand creation and service provision and supply of commodities. As facilitators promote use of services and commodities during ses￾sions, these services and commodities may not be readily available, accessible, or are of poor quality. For FY20, all USAID Tulonge Afya community theater performances were connected with facility providers or others to provide services, and they liaised with UBA and other partners. This needs to be continued and rolled out wherever USAID Tulonge Afya is conducting demand creation activities. In addition, USAID Tulonge Afya, UBA, and other IPs need to map out where gaps exist, espe￾cially in more remote areas to ensure access to not only services, but quality commodities as well (this could also include a strengths, weakness, opportunities, and threats a https://www.facebook.com/LIfeisPreciousTakeCareofit/. analysis among key GOT stakeholders and the projects). This can also reduce long distances to and long waiting times for health facilities reported by beneficiaries, which deters care seeking for healthy behaviors. Both projects can convene a working meeting with the GOT to explore viable solutions based on the types of gaps identified. Illustrative examples may include task shifting, mobile clinics, and review of USAID Tulonge Afya and UBA activities in workplans to ensure demand generation and service provision technical assistance are aligned and coordinated. Crossing-cutting question 6: mitigation strat￾egies have been considered to address limita￾tions and constraints, and how effectively have they been adopted Ensure either supportive supervision or another mechanism provides confirmation that activities took place. One of the challenges in data collection was the ability to do the expected number of observations. While part of this was due to scheduling issues, USAID Tulonge Afya should have a mechanism to review and ensure that activities outside the scope of the evaluation took place as planned. Crossing-cutting question 7: Facilitators and barriers to shifting from vertical to integrated SBCC programs, especially among youth and adult platforms Facilitators to document, strengthen, and scale Maximize and document efficiencies across integrated SBCC approaches. GOT staff and IPs mentioned cost and time savings, for both the health system and clients, as advantages of integrated SBCC. CROSS-CUTTING QUESTION 5: HIGHLIGHTED FINDINGS • In several districts, it was noted that some health providers’ attitudes and lack of adherence to client confidentiality deterred women and youth from seeking services and commodities. • Referral and linkage mechanisms for ser￾vices, or nearby clinics, were not always available during SBCC activities at the com￾munity level, hindering service uptake. CROSS-CUTTING QUESTION 6: HIGHLIGHTED FINDINGS • USAID Tulonge Afya undertook several miti￾gation strategies (e.g., improving mentorship and supervision) based on project feedback among all three IRs that improved activities for beneficiaries, strengthened data link￾ages, and coordination frameworks. 34 MIDTERM EVALUATION OF USAID TULONGE AFYA PROJECT Challenges to address, course correct, and learn from moving forward Assist in a deeper sensitization among stakeholders and GOT regarding what integrated SBCC involves. While the GOT has been involved in the SBCC behavior prioritization and strategy design and implementation process, confusion still exists in these areas. USAID Tulonge Afya should consider appreciative inquiryb tech￾niques among stakeholders at all levels to hold partici￾patory workshops that focus on the key issues identified among the different stakeholders regarding priority strengths and challenges related to integrated SBCC and behavior prioritization. Clarify roles and responsibilities at national and sub-national levels among integrated and vertical SBCC programs. Since uncertainty remains at the national level as to why SBCC is integrated in some areas and not in others, HPCs under HPS need clear guidance on how inte￾grated SBCC staff will harmonize with vertical SBCC staff. Roles and responsibilities should be clear when selecting priority messages, and the respective vertical programs and key stakeholders should collaborate to ensure that all appropriate priority messages are included in the integrated SBCC platforms. This could take place as part of the participatory workshops mentioned above. b See link for example of Encompass LLC’s site for examples of apprecia￾tive inquiry: https://encompassworld.com/tag/appreciative-inquiry/. Accessed 26 March 2020. Crossing-cutting question 8: whether USAID Tulonge Afya’s organizational and management structures, systems, processes, and procedures enabled or constrained the success of capaci￾ty, coordination, and collaboration, especially within its integrated SBCC program Engagement of GOT staff and IPs in SBCC design and delivery is key. This includes national strategies, SOPs, and working groups to assist in SBCC coordination and collaboration. Cost efficiencies and time savings are advantages of these activities and should be continued. Flexibility of the USAID Tulonge Afya is appreciated and necessary with shifting priorities in the GOT and among IPs, as well as CSOs and other stakeholders. Facilitate meetings to address the planning challenges related to differences in fiscal years between USAID and USAID Tulonge Afya. USAID should be included in the GOT work planning meetings with USAID Tulonge Afya, to discuss activities under USAID Tulonge Afya’s forthcoming workplan and reach mutual understanding of what is feasible and not feasible under USAID Tulonge Afya’s mandate. USAID Tulonge Afya should sensitize the GOT that some of the approved activities may not take place as planned based on internal processes. Strengthen accountability and sustainability mecha￾nisms for SBCC activities. As mentioned in the findings, concerns exist about the sustainability of funds to sup￾port TWGs and supportive supervision after the project ends. USAID Tulonge Afya is phasing SBCC activities CROSS-CUTTING QUESTION 7: HIGHLIGHTED FINDINGS • An important facilitator was ensuring that the GOT leadership perceived integrated SBCC approaches as cost-effective and efficient. • Limited understanding of what integrated SBCC entails in practice by GOT officials is a barrier. • Key messages in specific health areas noted by GOT are not featured in the integrated SBCC platforms. CROSS-CUTTING QUESTION 8: HIGHLIGHTED FINDINGS • USAID Tulonge Afya ensured GOT leadership and participation of key stakeholders in SBCC design and delivery. • Some approved activities by the GOT and the project did not take place in a timely manner or as planned due to USAID internal processes; differences in fiscal years and activity planning with the GOT also affected this. • Sustainability of funds to support TWGs and supportive supervision after the project ends is an issue. BR E A K THROUGH R ESE A RCH | AUGUST 2020 35 into the comprehensive council health plans (CCHP) to facilitate successful implementation and supervision of planned activities and ensure sustainability of SBCC activities at the district level in enhanced districts. USAID Tulonge Afya should involve GOT officials at the district and regional levels to identify, prioritize, and plan SBCC activities in non-enhanced districts. USAID Tulonge Afya should provide guidance to CSOs on how to increase local government involvement and/or transparency regarding planned activities and budgets, which led to facilitated local government support for activity implementation where activities were implemented. Recommendations for the follow-on project The evaluation team suggests the following recommen￾dations for the follow-on project based on the findings from the evaluation, the recommendations for the current USAID Tulonge Afya project, and discussions with USAID. Recommendations for the current USAID Tulonge Afya project that cannot be implemented within the remaining life of the project should be considered for the follow-on. IR 1: Improved ability of individuals to practice health behaviors Utilize project surveys of the integrated SBCC activities and related changes in behavior/norms to prioritize and bundle health areas in future platforms and cam￾paigns, especially gateway behaviors. This includes a deeper analysis regarding differences between enhanced and essential districts, if possible. Prioritize the sustainability and continued involvement of SITETEREKI volunteers. A dialogue should take place at all levels within the GOT regarding recruitment and opportunities to provide incentives for youth to continue their volunteer activities (e.g., certificates of service in certain skill areas that can be useful for future employ￾ment, priority access to health services for themselves and partners based on their service, etc.). IR 2: Strengthened community support for health behaviors Consider including documented lessons from any suc￾cessful outcomes based on proposed USAID Tulonge Afya pilot activities (e.g., “on-call” providers, FAQ booklets, health provider collaboratives) in the next mechanism. Documentation can help to ensure facili￾tators and providers provide evidence-based, real time information to inform the follow-on activities. Improve DHIS2 data for decision making regarding key ideational factors (e.g., self-efficacy, subjective and social norms) SBCC indicators in DHIS2 currently track outputs. The follow-on project should focus on how to make the data more useful for decisions. Behavioral determinants (e.g., perceived risk, social norms) can be piloted in several districts, in conjunction with training related to how GOT officials and IPs can use these data for adaptive management. The results of that pilot can inform scale-up as appropriate. IR 3: Improved systems for coordination and implementation of SBCC interventions USAID/Tanzania should encourage any IPs working with CSOs to apply lessons learned in GOT engagement through the experience of USAID Tulonge Afya. Some included the guidance USAID Tulonge Afya provided to CSOs on how to increase local GOT involvement and/ or transparency for planned activities and budgets. This process would help foster scalable and sustained implementation and supervision of SBCC activities at the district level. Incorporate specific intermediate results pertaining to cross-project linkages (i.e., USAID Tulonge Afya and UBA) in follow-on service delivery and SBCC mecha￾nisms. This includes key areas noted under cross-cutting areas, such as supply and demand, and ensuring commu￾nity reach. Include planning meetings to address the challenges regarding differences in fiscal years between USAID and USAID Tulonge Afya and other facilitating fac￾tors recommended under IR3 to continue under the follow-on. 36 MIDTERM EVALUATION OF USAID TULONGE AFYA PROJECT Conclusion There was an agreement across all respondent categories that USAID Tulonge Afya SBCC activities improved the ability of individuals to practice healthy behaviors in the five health areas targeted by the project, with varying degrees of progress. USAID Tulonge Afya reported notable improvements in attitudes and perceptions of behaviors related to HIV/AIDS, FP/RH, and MCH (in partic￾ular, those related to pregnancy and ANC). Some of these results related to USAID Tulonge Afya’s use of audience insights to inform emotional drivers among targeted audiences and use of multiple channels to deliver SBCC messages with an emphasis on IPC, as well as participa￾tory approaches (e.g., ADDED and HCD). Despite the noted gains in individuals’ abilities to practice healthy behaviors, challenges and barriers limiting access and/or utilization of services still exist—some of which fall outside of USAID Tulonge Afya’s direct mandate. In addition, the implementation of integrated SBCC plat￾forms, especially among the small group sessions, still faces some programmatic and structural challenges that need to be addressed. USAID Tulonge Afya carried out activities that were suc￾cessful in empowering and engaging the government and civil society structures at district and regional levels to support and facilitate delivery of quality SBCC. The most notable activity focused on strengthened coordination for SBCC activities at the national, regional, and district levels, and strengthening of the SBCC data systems. Several challenges were faced relating to the SBCC coordination skills and capacity among the regional and district HP coordinators, while CSOs reported improve￾ment in some of their SBCC delivery skills. Many respondents supported the integrated approach to SBCC and added that it reached beneficiaries with needed information specific to their life stage in a more comprehensive way than vertical interventions, thereby saving time and costs for both the clients and health system. Moreover, there was agreement that significant improvements were made in coordination, collabora￾tion, and co-investment in the SBCC programming in Tanzania as a result of the USAID Tulonge Afya project. Various coordinators for vertical programs perceived that integration would require them to start supporting SBCC activities in other health areas. This finding shows that various key stakeholders still lack some understanding of the operationalization of the integrated approach to SBCC, including the rationale for behavior and message prioritization based on formative evidence. USAID Tulonge Afya is a complex, comprehensive, integrated SBCC project. While there are areas that the project can continue to improve upon, the project has notable successes that can be documented in having a positive influence on key health behaviors among target audiences throughout the project focal areas. BR E A K THROUGH R ESE A RCH | AUGUST 2020 37 References 1. UNICEF. United Republic of Tanzania Statistics. 2016. Avail￾able from: www.unicef.org/infobycountry/tanzania_statis￾tics.html. (Date cited 8 August 2017). 2. Kwesigabo, G. et al. 2012. "Tanzania’s health system and workforce crisis," Journal of Public Health Policy 33(S1): S35–44. 3. Institute for Health Metrics and Evaluation (IHME). Tanzania profile. Seattle, WA: IHME, University of Washington, 2018. Available from http://www.healthdata.org/Tanzania. (Date accessed 8 May 2020). 4. Tanzania HIV Impact Survey (THIS), 2016–2017. 5. MalariaSpot. nd. "Malaria in Tanzania." https://malariaspot. org/en/eduspot/malaria-in-tanzania/. (Date accessed 29 July 2019). 6. Ministry of Health, Community Development, Gender, Elderly and Children (MoHCDGEC) [Tanzania Mainland], Ministry of Health (MoH) [Zanzibar], National Bureau of Statistics (NBS), Office of the Chief Government Statistician (OCGS), and ICF. 2016. Tanzania Demographic and Health Survey and Malaria Indicator Survey (TDHS-MIS) 2015–16. Dar es Salaam, Tanzania, and Rockville, Maryland, USA: MoHCDGEC, MoH, NBS, OCGS, and ICF. 7. USAID Tulonge Afya Project Introduction July 2019 Power￾Point presentation. (Date accessed 14 May 2020). 8. USAID Tulonge Afya Year 1 Workplan 9. Mid-Term Performance Evaluation of Boresha Afya, The Com￾prehensive Health Service Delivery Project, USAID/Tanzania. February 2020. ME&A (Mendez England & Associates). 10. Personal communication, USAID Tulonge Afya, FHI 360, 5/29/2020. 11. Treves-Kagan S. et al. 2017. "Gender, HIV testing and stigma: the association of HIV testing behaviors and community-lev￾el and individual-level stigma in rural South Africa differ for men and women," AIDS and Behavior 21(9): 2579–2588. doi:10.1007/s10461-016-1671-8. 12. Population Council. 2019. “Women and HIV: understanding and addressing stigma—evidence from the Population Coun￾cil,” Factsheet. Washington, DC: Population Council. 13. Mason, Jennifer et al. 2017. "The role of family planning in achieving safe pregnancy for serodiscordant couples: com￾mentary from the United States government's interagency task force on family planning and HIV service integration," Journal of the International AIDS Society 20(Suppl 1): 21312. doi.org/10.7448/IAS.20.2.21312 38 MIDTERM EVALUATION OF USAID TULONGE AFYA PROJECT Annexes: Table of Contents Annex I. USAID Tulonge Afya Project Logical Framework...................................................................40 Annex IIa. Methods: Data gathering activities, purpose, and research evaluation questions..............41 Annex IIb. Methods: Interview type, respondents, number of FGDs, interviews, and observations completed...................................................................................................42 Annex IIc. Methods: Final selected districts that participated in the evaluation.................................42 Annex III. Methods: Participant recruitment and data collection procedures.....................................43 Annex IV. List of participants for KIIs/IDIs..........................................................................................44 Annex Va. Number of beneficiaries that participated in the FGDs......................................................46 Annex Vb. Characteristics of beneficiaries that participated in the FGDs............................................46 Annex Vc. Number of wards in each district which beneficiaries that participated in the FGDs came from................................................................................................................47 Annex VIa. Number of facilitators that participated in the FGDs........................................................47 Annex VIb. Characteristics of facilitators that participated in the FGDs..............................................48 Annex VIc. Number of wards in each district that facilitators that participated in the FGDs came from..............................................................................................................48 Annex VII. List of reviewed documents..............................................................................................49 Annex VIII. Supporting statements for key findings............................................................................51 Annex IX. Data collection tools..........................................................................................................57 BR E A K THROUGH R ESE A RCH | AUGUST 2020 39 Annex I. USAID Tulonge Afya Project Logical Frameworkc c USAID Tulonge Afya Project – Overview of M&E Systems. Accessed 14 May 2020. IMPACT Improved health status, especially women and youth ULTIMATE OUTCOME Improved ability of individuals to practice desired health behaviors Strengthened communities to support the practice of desired health Improved health communication systems to coordinate and implement interventions Improved SBCC capacity at all levels for all focus health areas Improved qualtity and harmoniza￾tion of SBCC Shifts toward positive social and gender norms Improved provider and CHW attitudes and SBCC skills IMMEDIATE OUTCOME Improved individual behavioral determinants: KAI OUTPUTS Availability of multimedia platforms Increased linkage to care Increased dialogue Increased localized SBCC material distributed Community-led SBCC and advocacy Trained CHWs and providers on SBCC in malaria, HIV, FP, TB, and MCH Strong community networks to enable provision of SBCC services Co-investment and collaboration SBCC profession strengthened Regional collaboration to enhance provision of SBCC services PROCESSES/ ACTIVITIES Conduct several formative research projects, baseline and endline, and audience consultation by health area and population segment Development of print and radio materials by health area and population segment Develop IPC materials and messages per health area and population segment Run radio spots campaigns Re-purpose different existing campaigns Conduct IPC session through CHW/super fans Conduct community sensitization through theater groups, meetings, and events Develop different tools/ guidelines for SBCC Mobilize district-driven SBCC • Orient regional and district staff • Map SBCC system • Support district to conduct SBCC mobilization Develop district SBCC plans • Participate in CCHP to include SBCC activities in the plan Develop and implement grants to drive demand for health services and transform norms • Work and support GOT to lead grant activities and tie them with national HPS • Orient regional and district on USAID Tulonge Afya SBCC packages • Shujaz • Community radio Conduct systems audit Conduct SBCC capacity assessment Develop and implement SBCC coordination framework Develop capacity and institutional strengthening framework Integrate health promotion indicators into national reporting system to improve reporting and monitoring HPS Strengthen communication within HPS Strengthen Health Promotion TWG and coordination to increase collaboration Develop regional collaborations/TWG 40 MIDTERM EVALUATION OF USAID TULONGE AFYA PROJECT Annex IIa. Methods: Data gathering activities, purpose, and research evaluation questions STUDY PARTICIPANTS PROJECT STAFF AND NATIONAL- AND REGIONAL-LEVEL GOT AND PARTNERS ZONAL IMPLEMENTING PARTNERS AND CSO STAFF FRONT-LINE IMPLE￾MENTERS (CHW/ VOLUNTEERS, PEER CHAMPIONS) AND USAID TULONGE AFYA PRO￾GRAM BENEFICIARIES YOUTH AND ADULT PLAT￾FORMS, SESSIONS, AND CAMPAIGNS (FACILITA￾TORS AND BENEFICIA￾RIES) Evaluation Questions Addressed Question 3, 4, 5, 6, 8 Question 2a, 2b, 3, 7 Question 1a, 1b Fidelity of delivery of inter￾vention, reception among program beneficiaries Purpose Understand perceptions of transition from vertical to integrated service delivery, how integration leads to improved capacity, coordi￾nation, collaboration, and co-investment for SBCC. Gain in-depth knowledge regarding barriers and facilitators for engaging CSO, IPs, and GOT at district and regional levels to support and facilitate delivery of quality SBCC. Understand front-line implementer and program beneficiary perspectives of the project and percep￾tions of individual- and community-level changes in knowledge, attitudes, and norms from activities. Compare against what is written in project workplans and what is observed in field. Method Semi-structured KII Semi-structured IDI Semi-structured FGD Structured observations Location of Activity Organization or health office or other safe and confidential locations Health facility, ward/village office, or other safe and confidential locations Health facility, ward/village office, or other safe and confidential locations At site of session BR E A K THROUGH R ESE A RCH | AUGUST 2020 41 Annex IIb. Methods: Interview type, respondents, number of FGDs, interviews, and observations completed INTERVIEW TYPE RESPONDENT CATEGORY COMPLETED KIIs (national level) GOT 8 TA staff 4 BA 3 IPs 4 USAID 2 Total 21 KIIs (zonal level) TA staff 3 BA 3 Total 6 IDIs (regional/ district level) RHPCOs (Regional HP coordinators) 6 DACC/CHACC 6 IPs 6 CSO 6 Total 24 FGDs Facilitators (NAWEZA, FURAHA YANGU, SUBIRA & JUMA sessions) 24 Beneficiaries (NAWEZA, FURAHA YANGU, SUBIRA & JUMA sessions) 36 Total 60 Direct activities observations NAWEZA sessions 9 FURAHA YANGU sessions 7 JUMA sessions 4 SUBIRA sessions 6 Community theater 3 Total 29 TOTAL 140 Annex IIc. Methods: Final selected districts that participated in the evaluation ZONE REGION ENHANCED DISTRICTS COMMUNITY-LEVEL IMPLEMENTING PARTNER Lake Mara Rorya DC TCDC Lake Mwanza Sengerema DC T-MARC Lake/Western Tabora Tabora MC T-MARC North/Central Singida Singida MC TCDC Southern Njombe Wanging’ombe DC T-MARC Southern Mtwara Newala DC TCDC 42 MIDTERM EVALUATION OF USAID TULONGE AFYA PROJECT Annex III. Methods: Participant recruitment and data collection procedures Participant recruitment Participants for the KIIs and IDIs were selected with assis￾tance from USAID Tulonge Afya based on their technical expertise, role, and/or relationship to the USAID Tulonge Afya at national, zonal, regional, and district levels. The evaluation team selected session facilitators for the FGDs from a list of facilitators shared by USAID Tulonge Afya. The evaluation team aimed to recruit 24–32 facilitators from each district (6–8 per platform); however, due to the limited number of facilitators who were available and/or met the inclusion criteria, fewer were included in some districts. Inclusion criteria required the facilitator to live 20 km or less from the evaluation team’s discussion site at the respective CSO and have at least six months of experiencing working with USAID Tulonge Afya. In some districts, the team had to decrease required time working with USAID Tulonge Afya to a minimum of four months to get an adequate number of participants. Selected names were shared with local CSOs, which assisted with recruit￾ment. The team conducted four FGDs with facilitators in each district, one for each small group session type (NAWEZA, FURAHA YANGU, SUBIRA, and JUMA). Each facilitator participated in only one of the four FGDs. Proj￾ect staff made efforts to equalize ratios of facilitators’ sex (M/F) and position (CHW/CV) for each FGD in NAWEZA and FURAHA YANGU sessions. Session facilitators who participated in the FGDs helped recruit session beneficiaries. Like facilitators’ FGDs, bene￾ficiaries’ FGDs were also stratified by type of session. The NAWEZA and FURAHA YANGU small group session ben￾eficiaries’ FGDs were further stratified by sex, i.e., FGDs were conducted separately for male and female benefi￾ciaries. The evaluation team planned to stratify SUBIRA and JUMA beneficiaries’ FGDs by age (15–17 years and 18–24 years); however, the team could only access youth aged 18–24 years due to the inability to obtain parental consent (noted in the limitations section). The evaluation team instructed (through the local CSOs) each facilitator to invite two beneficiaries from their corresponding FGD category; for example, facilitators who participated in a FGD for a NAWEZA session were each instructed to invite one male and one female from their pool of NAWEZA session participants. This approach enabled the eval￾uation team to get a mix of participants from sessions run by different facilitators and from different wards, which broadened the area covered by the assessment. The inclusion criteria for the beneficiaries, which were shared with the CSOs and facilitators doing the recruit￾ment included that participants: be 18-24 years of age for youth platforms and age 18 and above for the adult platforms, have attended at least one session in the past three months, live 20 km or less away from the discussion site, be a resident in the district for at least six months, and be able to speak Swahili fluently. Data collection Data were collected from 17 November 2019 to 15 January 2020, starting with field work in the six districts, followed by KIIs at the national level in Dodoma and Dar es Salaam. The team conducted KIIs in Dodoma before the December holidays and continued with the KIIs at the national level after the holidays. Staff conducted mainly face-to-face KIIs and IDIs in the offices of the respective respondents; a few KIIs at the national level were con￾ducted over the phone due to conflicts in scheduling in-person interviews. Staff conducted most of the FGDs for beneficiaries and facilitators at the CSOs’ offices, while a few FGDs were conducted at health facilities. When it was difficult for FGD participants to come to the CSOs offices, the evaluation team traveled to the respec￾tive wards to conduct the FGDs. The KIIs and IDIs lasted 30–60 minutes, while FGDs took 45–90 minutes to com￾plete. All interviews were audio-recorded after obtaining written informed consent from each of the participants. BR E A K THROUGH R ESE A RCH | AUGUST 2020 43 Annex IV. List of participants for KIIs/IDIs #. INTERVIEW TYPE RESPONDENT CATEGORY DISTRICT NAME OF ORGANIZATION POSITION WITHIN ORGANIZATION GENDER LENGTH OF TIME AT ORGANIZATION KEY INFORMANT INTERVIEWS (KIIs) 1 KII Boresha Afya DSM JHPIEGO - BA Senior Technical Advisor Community Engagement and Livelihood M 18 months 2 KII Boresha Afya DSM JHPIEGO - BA Acting COP M 18 months 3 KII Boresha Afya DSM EGPAF Senior Technical Director - BA M 4 years 4 KII Boresha Afya DSM JHPIEGO -Sauti project COP - Sauti F 3 months 5 KII Boresha Afya DSM BA- Delloite COP F 4 years 6 KII Boresha Afya DSM BA- Delloite Senior Advisor for PMTCT M 2 years 7 KII Boresha Afya DSM BA- Delloite Communication Coordinator M 30 months 8 KII Boresha Afya Tabora EGPAF Associate Project Manager M 7 months 9 KII Boresha Afya Njombe USAID Boresha Afya Project Manager M 2 years 10 KII Boresha Afya Njombe USAID Boresha Afya M & E Lead F 3 years 11 KII Boresha Afya Nyamagana USAID Boresha Afya Senior Technical Officer M 4 years 12 KII Boresha Afya Nyamagana JHPIEGO Community Engagement Gender Outreach M 4 years 13 KII Boresha Afya Musoma USAID Boresha Afya Manager of BA (JHPIEGO) M 4 years 14 KII Boresha Afya Singida BA Project Coordinator M 2.5 Years 15 KII Boresha Afya Mtwara USAID Boresha Afya Community Service Advisor M 7 months 16 KII Government of Tanzania Dodoma RCHS Head of NCH program M 3 years 17 KII Government of Tanzania Dodoma NACP IEC Officer M 11 years 18 KII Government of Tanzania Dodoma RCHS - GOT Coordinator Gender & Adoles￾cent Health M 1 year 19 KII Government of Tanzania DSM GOT- FHI 360 Capacity Building & Liaison Officer M 1 year 20 KII Government of Tanzania DSM NACP SBCC Program Officer F 13 months 21 KII Government of Tanzania DSM TACAIDS Dir. Advocacy & Information M 9 years 22 KII Government of Tanzania DSM NTLP - GOT SBCC Coordinator F 10 years 23 KII Government of Tanzania Mtwara Mtwara region RRCHCO F 15 months 24 KII Government of Tanzania Mtwara Mtwara region RHPCO M 18 months 25 KII Government of Tanzania Tabora Tabora Regional Offices RHPCO M 2 years 26 KII Government of Tanzania Njombe RHMT RHPCO M 5 years 27 KII Government of Tanzania Nyamagana RHMT RHPCO M I year 28 KII Government of Tanzania Musoma RHMT RHPCO M 7 years 29 KII Government of Tanzania DSM NMCP Head of SBCC Unit F 15 years 30 KII Tulonge Afya DSM FHI 360-TA Dir MERLM Routine Day to Day M 2 years 31 KII Tulonge Afya DSM FHI 360-TA Semi Zonal Manager - TA M 3 years 32 KII Tulonge Afya DSM FHI 360 COP F 33 KII Tulonge Afya DSM FHI 360 Dir. Communication F 2.7 years 34 KII Tulonge Afya Tabora FHI 360 Acting Zonal Manager - TA M 2 years 35 KII Tulonge Afya Njombe T-MARK SBCC Officer M 2 years 36 KII Tulonge Afya Njombe Tulonge Afya Zonal Manager M 2 years 37 KII Tulonge Afya Njombe Tulonge Afya M & E M 2 years 38 KII Tulonge Afya Mtwara Tulonge Afya R-SBCC Officer M 1 year 44 MIDTERM EVALUATION OF USAID TULONGE AFYA PROJECT #. INTERVIEW TYPE RESPONDENT CATEGORY DISTRICT NAME OF ORGANIZATION POSITION WITHIN ORGANIZATION GENDER LENGTH OF TIME AT ORGANIZATION 39 KII Tulonge Afya Nyamagana FHI 360 Zonal Manager M 4 years 40 KII Tulonge Afya Nyamagana FHI 360 Zonal-SBCC - Regional / Zonal M&E M 2 years 41 KII USAID DSM USAID Senior FP Adviser F 5.5 years 42 KII USAID DSM USAID TA Lead/ Manager F 21 months IN-DEPTH INTERVIEWS (IDIs) 1 IDI Government of Tanzania Newala CHMT CHAC M 7 months 2 IDI Government of Tanzania Newala DHMT DHPCo M 6 years 3 IDI Government of Tanzania Wang￾ing`ombe DHMT CHAC F 3 months 4 IDI Government of Tanzania Wang￾ing`ombe DHMT DACC M 4 years 5 IDI Government of Tanzania Wang￾ing`ombe DHMT DHPCo M 3 years 6 IDI Government of Tanzania Sengerema DHMT CHACC M 1.5 years 7 IDI Government of Tanzania Sengerema DHMT DHPC F 1 year 8 IDI Government of Tanzania Singida DHMT DACC M 9 years 9 IDI Government of Tanzania Singida RHMT RHPCo M 2 years 10 IDI Government of Tanzania Singida CHMT MHPC M 18 months 11 IDI Government of Tanzania Rorya DHMT DHPC F 3 years 12 IDI Government of Tanzania Rorya DHMT CHAC, DCDO M 8 years 13 IDI Government of Tanzania Tabora DHMT DHPCO M 20 years 14 IDI Government of Tanzania Tabora DHMT CHAC F 3 years 15 IDI CSO Wang￾ing`ombe ADP - Mbozi SBCC Officer M I year 16 IDI CSO Wang￾ing`ombe ADP - Mbozi SBCC Officer M I year 17 IDI CSO Wang￾ing`ombe ADP - Mbozi TA- Project Manager M 3 months 18 IDI CSO Wang￾ing`ombe ADP - Mbozi M&E F 1 year 19 IDI CSO Tabora AICT Tabora CSO Project Manager M 20 IDI CSO Newala TALIA Program Manager M 3 years 21 IDI CSO Newala TALIA M&E M 3 years 22 IDI CSO Newala TALIA SBCC Coordinator F 5 years 23 IDI CSO Sengerema ELCT-ELVD Program Officer M 24 IDI CSO Sengerema ELCT-ELVD M&E M 25 IDI CSO Sengerema ELCT-ELVD Program Manager F 26 IDI CSO Singida Tanzania Redcross Society M&E F 9 years 27 IDI CSO Singida Tanzania Redcross Society Program Officer F 6 years 28 IDI Implementing partner Singida TCDC SBCC Regional Coordinator M 11 months 29 IDI Implementing partner Rorya TCDC Regional SBCC Officer M 1 year 30 IDI CSO Rorya MVIWANYA Program Coordinator M 9 years 31 IDI CSO Rorya ONE WORLD M&E M 2 years 32 IDI CSO Rorya ONE WORLD SBCC Officer M 7 months 33 IDI CSO Rorya ONE WORLD SBCC Officer M 9 months 34 IDI CSO Musoma One World Sostam Livelihood (OWSL) Org. Coordinator M 3 years 35 IDI Tulonge Afya Tabora T-MARK SBCC Officer F 22 months BR E A K THROUGH R ESE A RCH | AUGUST 2020 45 Annex Va. Number of beneficiaries that participated in the FGDs DISTRICT FREQUENCY (%) TOTAL N=233 (6 FGDS/ DISTRICT) SUBIRA (1 FGD/ DISTRICT) JUMA (1 FGD/ DISTRICT) NAWEZA (2 FGDS/ DISTRICT) FURAHA YANGU (2 FGDS/ DISTRICT) 1 Tabora 8 (16.7%) 8 (16.7%) 16 (33.3%) 16(33.3%) 48 2 Newala 6 (15.8%) 6 (15.8%) 14 (36.8%) 12 (31.6%) 38 3 Wanging`ombe 2 (5.7%) 8 (22.9%) 10 (28.6%) 15 (42.9%) 35 4 Sengerema 7 (15.6%) 8 (17.8% 16 (35.6%) 14 (31.1%) 45 5 Singida 8 (22.2%) 5 (13.9%) 11 (30.6%) 12 (33.3%) 36 6 Rorya 4 (12.9%) 5 (16.1%) 10 (32.3%) 12 (38.7%) 31 Annex Vb. Characteristics of beneficiaries that participated in the FGDs BACKGROUND CHARACTERISTIC FREQUENCY (%) SUBIRA (1 FGD/DISTRICT) JUMA (1 FGD/DISTRICT) NAWEZA (2 FGDS/DISTRICT) FURAHA YANGU (2 FGDS/DISTRICT) (N=233) Age (Total n=233, missing=0) 1 18–24 years 35 (100%) 39 (97.5%) 18 (23.4%) 20 (24.7%) 2 25–34 years 0 (0.0%) 1 (2.5%) 38 (49.4%) 33 (40.7%) 3 35–44 years 0 (0.0%) 0 (0.0%) 13 (16.9%) 19 (23.5%) 4 45 years + 0 (0.0%) 0 (0.0%) 8 (10.4%) 9 (11.1%) Total 35 40 77 81 Gender (n=233, missing=0) 1 Males 0 (0.0%) 40 (100%) 36 (46.8%) 39(48.1%) 2 Females 35 (100%) 0 (0.0%) 41(53.2%) 42 (51.9%) Total 35 40 77 81 Number of sessions (n=202, missing=31) 1 1–2 10 (35.7%) 4 (12.9%) 23 (33.3%) 27 (36.5%) 2 3–4 18 (64.3%) 22 (71.0%) 42 (60.9%) 40 (54.1%) 3 5–6 0 (0.0%) 5 (16.1%) 3 (4.3%) 3 (4.1%) 4 7+ 0 (0.0%) 0 (0.0%) 1 (1.4%) 4 (5.4%) Total 28 31 69 74 Duration lived in district (n=179, missing=54) 1 less than 1 year 0 (0.0%) 0 (0.0%) 0 (0.0%) 1 (1.5%) 2 1–10 years 6 (30.0%) 5 (15.6%) 24 (39.3%) 15 (22.7%) 3 11–20 years 12 (60.0%) 15 (46.9%) 8 (13.1%) 4 (6.1%) 4 21–30 years 2 (10.0%) 12(37.5%) 18 (29.5%) 27 (40.9%) 5 31years+ 0 (0.0%) 0 (0.0%) 11 (18.0%) 19 (28.8%) Total 20 32 61 60 46 MIDTERM EVALUATION OF USAID TULONGE AFYA PROJECT Annex Vc. Number of wards in each district which beneficiaries that participated in the FGDs came from DISTRICTS NUMBER OF WARDS SUBIRA JUMA NAWEZA FURAHA YANGU Tabora (n=18) 6 5 10 8 Newala (n=11) 3 3 5 4 Wanging`ombe (n=9) 1 2 5 3 Sengerema (n=21) 4 8 10 6 Singida (n=10) 1 4 5 2 Rorya (n=9) 3 1 6 4 Annex VIa. Number of facilitators that participated in the FGDs DISTRICT FREQUENCY (%) IN EACH FGD TOTAL (4 FGDS IN EACH DISTRICT) (N=113) SUBIRA JUMA NAWEZA FURAHA YANGU 1 Tabora 8 (27.6%) 5 (17.2%) 8 (27.6%) 8 (27.6%) 29 2 Newala 3 (16.7%) 3 (16.7%) 7 (38.9%) 5 (27.8%) 18 3 Wanging`ombe 2 (18.2%) 2 (18.2%) 3 (27.3%) 4 (36.4%) 11 4 Sengerema 4 (14.3%) 8 (28.6%) 8 (28.6%) 8 (28.6%) 28 5 Singida 1 (8.3%) 3 (25.0%) 4 (33.3%) 4 (33.3%) 12 6 Rorya 4 (26.7%) 1 (6.7%) 4 (26.7%) 6 (40.0%) 15 BR E A K THROUGH R ESE A RCH | AUGUST 2020 47 Annex VIb. Characteristics of facilitators that participated in the FGDs BACKGROUND CHARACTERISTIC FREQUENCY (%) SUBIRA (1 FGD/DISTRICT) JUMA (1 FGD/DISTRICT) NAWEZA (1 FGD/DISTRICT) FURAHA YANGU (1 FGD/DISTRICT) Age (Total n=113, missing=0) 1 18–24 years 15 (68.2%) 9 (40.9%) 0 (0.0%) 1 (2.9%) * 2 25–34 years 7 (31.8%) 13 (59.1%) 8 (23.5%) 7 (20.0%) 3 35–44 years 0 (0.0%) 0 (0.0%) 11 (32.4%) 12 (34.3%) 4 45 years+ 0 (0.0%) 0 (0.0%) 15 (44.1%) 15 (42.9%) Total 22 22 34 35 Gender (Total n=113, missing=0) 1 Males 0 (0.0%) 22 (100.0%) 15 (44.1%) 16 (45.7%) 2 Females 22 (100.0%) 0 (0.0%) 19 (55.9%) 19 (54.3%) Total 22 22 34 35 Position (Total n= 113, missing=0) 1 CHW 8 (36.4%) 0 (0.0%) 23 (67.6%) 23 (65.7%) 2 CV 14 (63.6%) 22 (100.0%) 11 (32.4%) 12 (34.3%) Total 22 22 34 35 Duration worked with TA (Total n=113, missing=0) 1 Less than 6 months 2 (9.1%) 1 (4.5%) 0 (0.0%) 6 (17.1%) 2 6–12 months 8 (36.4%) 16 (72.7%) 22 (64.7%) 14 (40.0%) 3 13–24 months 12 (54.5%) 5 (22.7%) 12 (35.3%) 15 (42.9%) Total 22 22 34 35 *An IDI was conducted with this facilitator because he was alone Annex VIc. Number of wards in each district that facilitators that participated in the FGDs came from DISTRICTS NUMBER OF WARDS SUBIRA JUMA NAWEZA FURAHA YANGU Tabora (n=13) 8 5 8 7 Newala (n=9) 3 3 4 4 Wanging`ombe (n=6) 2 2 3 3 Sengerema (n=18) 4 8 7 7 Singida (n=7) 1 3 3 3 Rorya (n=7) 3 1 4 4 48 MIDTERM EVALUATION OF USAID TULONGE AFYA PROJECT Annex VII. List of reviewed documents #. DOCUMENT TITLE DOCUMENT SOURCE 1 20092018_Outbrief Tanzania SBC TDY_FHI Sept 21 Tulonge Afya 2 AMEP Tulonge Afya_Final Revised 8.13.17 Tulonge Afya 3 AMEP Tulonge Afya_Revised 10-20-2018 submitted Tulonge Afya 4 Discussion Point for Data sources with Evaluation team Tulonge Afya 5 TA DATA FLOW DIAGRAM Tulonge Afya 6 FHI 360 Comprehensive_ IPC Register_Kiswahili_DRAFT • Final Tulonge Afya 7 IPC_Report Form outlined Tulonge Afya 8 Mid-Media Report outlined Tulonge Afya 9 PMP review Tulonge Afya 10 PPT_PEPFAR Partners Meeting-USAID Tulonge Afya-April2019 Tulonge Afya 11 Timeline on recommendations October 16 ns Tulonge Afya 12 Tulonge-Project_ An Overview of ME System updated Tulonge Afya 13 USAID Tulonge Afya Adult Strategy Tulonge Afya 14 USAID Tulonge Afya Baseline Initial Secondary Analysis Tulonge Afya 15 USAID Tulonge Afya Baseline Report Tulonge Afya 16 USAID Tulonge Afya Baseline Report_06 July 2018 Final for USAID Tulonge Afya 17 USAID Tulonge Afya FP insights summary report Tulonge Afya 18 USAID Tulonge Afya FY17 Annual Report submitted 30th November 2017 Tulonge Afya 19 USAID Tulonge Afya FY18_Annual Report Tulonge Afya 20 USAID Tulonge Afya FY18Q1_Quarterly Report Tulonge Afya 21 USAID Tulonge Afya FY18Q2_Quarterly Report Tulonge Afya 22 USAID Tulonge Afya FY18Q3_Quarterly Report Tulonge Afya 23 USAID Tulonge Afya FY19Q1_Quarterly Report Tulonge Afya 24 USAID Tulonge Afya FY19Q2_Quarterly Report Tulonge Afya 25 USAID Tulonge Afya FY19Q3_Quarterly Report Tulonge Afya 26 USAID Tulonge Afya FY19_Annual Report Tulonge Afya 27 USAID Tulonge Afya Malaria insights summary report Tulonge Afya 28 USAID Tulonge Afya MNCH insights summary report Tulonge Afya 29 USAID Tulonge Afya Project Introduction July 2019 Tulonge Afya 30 USAID Tulonge Afya TB insights summary report Tulonge Afya 31 USAID Tulonge Afya Test and Treat insights summary report Tulonge Afya 32 USAID Tulonge Afya Youth SBCC Strategy_FINAL Tulonge Afya 33 USAID Tulonge Afya_AMEP_FINAL DRAFT-Revised 13 12 2018 Tulonge Afya 34 USAID Tulonge Afya-Approach to Integration final Tulonge Afya 35 Work plan status report for the evaluators 30 July 2019-jm Tulonge Afya BR E A K THROUGH R ESE A RCH | AUGUST 2020 49 #. DOCUMENT TITLE DOCUMENT SOURCE 36 Year 1 Work Plan_AID-621-A-17-00002 TULONGE AFYA Final Revised October 13, 2017 nk Tulonge Afya 37 Year 2 Work Plan_AID-621-A-17-00002 USAID Tulonge Afya FY2019 Revised -Nov12- CLEAN Tulonge Afya 38 SOCIO-ECOLOGICAL MODEL_SEM-C4D CDC 39 Midline-Eval-Step-Change-GEC Coffey International Development Ltd 40 Refining Theory of Change JMDE 41 Brief_theory of change_UNICEF_eng UNICEF 42 C-Modules _Evaluating and Replaning USAID 43 Integrated-SBCC-Programs-Review-TOC USAID 44 2014RH_M&E of BCC_TrainingManual USAID 45 SBCC_ppfp_guide_12.11.14 USAID 46 Integrated-SBCC-Brief-FINAL-101617 USAID 47 Integrated-SBCC-Programs-I-Kit_Online-Print-080917-FINAL USAID 48 Mid-Line Evaluation Report USAID 49 Performance Evaluation Report_Tanzania Capacity and Communication Project (TCCP) USAID 50 Uganda Experience of Integrated SBCC USAID 51 WHO - technical_brief_final WHO 52 MnE_Plan_Kagera_FINAL_annexes United Nations 50 MIDTERM EVALUATION OF USAID TULONGE AFYA PROJECT Annex VIII. Supporting statements for key findings This annex contains additional quotes from study partic￾ipants that support the corresponding bulleted findings. These data provide additional context through direct quotations and reinforce the evidence presented in the Findings section of the report. IR 1: Evaluation question 1a—Improved ability of individuals to practice healthy behaviors • Lack of SBCC messages on how to conceive safely among HIV discordant couples is a missed opportunity. “ …he [FURAHA YANGU beneficiary] got tested and was found to be HIV infected and they [care providers] instantly started him on ARV medications. He has used them and currently, he has gained health very well. Surprisingly, he one day came with a beautiful woman, I don’t know where he found her and she was pregnant, they now have two children, twins, we tested them and they are all infected together with their mother and they have already started taking ARV medications. —FURAHA YANGU facilitator, Tabora “ We thank the stakeholders, who came up with this project because I was living with a woman who is HIV-positive, but we got a HIV-negative baby after using ‘Angaza’ medicine [ART]. We were tested then my wife started taking medication to prevent infection from mother to child. We have a healthy baby now, I have taken my baby for testing six times, and he/she is HIV negative. —FURAHA YANGU male beneficiary, Wanging’ombe IR 1: Evaluation question 1b—Catalyzing posi￾tive change in gender and sociocultural norms within communities, enabling the practice of desired behaviors across priority health areas • Use of multiple channels to deliver SBCC messages. “ Our project deals with communication. It makes use of posters, some radio stations and flyers. These channels have helped us to deliver some services, because people have been meeting and sharing some information about health issues. This has been especially with these volunteers. For instance, they are always delivering education by group approaches. These groups provide a platform for the youths to meet and discuss and share some information about health issues. —CSO, Wanging’ombe Cross-cutting question 5: Internal challeng￾es related to IR1 that USAID Tulonge Afya can address • Limited health provider orientation to USAID Tulonge Afya SBCC messages. “ To be honest, there are challenges. Other times, they are told, why are you bringing us water? [Immature pregnancy prior to 12 weeks gestation]. The health provider wants the client to come with mature preg￾nancy. They get discouraged when they are told that. —CSO, Singida Cross-cutting question 5: External constraints and challenges related to IR1 that USAID Tu￾longe Afya can work on with UBA and other IPs • Lack of confidentiality among providers offering HIV testing services. “ Another thing which is hindering them is about the confidentiality of the providers. They are afraid that, after being dis￾covered with [HIV] infection, the providers will expose them. When we move around mobiliz￾ing them, most of them agree to test [for HIV] but they request this project to train us and give us testing tools and do it right there. —FURAHA YANGU facilitators, Newala • Requirement to undergo HIV testing before being offered condoms at health facilities. BR E A K THROUGH R ESE A RCH | AUGUST 2020 51 “ It is hard to get condoms. The places to get condoms are health facilities, and in shops. People are scared to go get condoms because a person believes that she/ he will be told to first test [for HIV]. This is where the problem is. When they go to our leaders [CHWs], he tells them that he only has condoms for teaching. They then decide to go to the shops and if they don’t have the money, then they end up having sex without a condom. —JUMA beneficiary, Rorya IR 2: Evaluation question 2a—Challenges affect￾ing empowerment and engagement • Revitalization of the regional and district health promotion (HP) coordinators. “ In the past, these [HP] coordinators didn’t know their roles because they used to know activities that are in malaria are done by malaria people. Health promotion activities that are in MCH are done with people dealing with MCH. They didn’t know what to do but Tulonge has enabled [them], now they can collaborate with those vertical program coordinators in order for them to come up with comprehensive reports for health promotion activities in the council or regional. This is the main difference that I see. —GOT, National level “ For instance, we currently have health promotion personnel at the district level. This has by great percent been facili￾tated by FHI because one the role for USAID TA is to strengthen health promotion departments in the ministry. So, we have district health pro￾motion officers and we are working with them. They are helping us to talk about integration. —IP, Singida • Strengthened SBCC data systems at the regional and district level. “ For instance, we had no monitoring and evaluation system. It has been introduced by Tulonge Afya. Thus, we are able to make follow up on how the activities are conducted at lower levels up to a diocese level. —CSO, Sengerema • Provision of financial support for supportive supervi￾sion to both the CSOs and GOT. “ As PORALG, their [TA] support have enabled us to at least to be able to go do support supervisions in our council and regions. At least 1 or 2 times per year to be able to go do supervisions. This is the biggest support that we got from Tulonge. We used to do it using the government funds but through different programs not specific for health promotion. But this is one resource that has helped us in supervision specifically in the health promotion area. —GOT, National level “ For instance, we had no monitoring and evaluation system. It has been introduced by Tulonge Afya. Thus, we are able to make follow up on how the activities are conducted at lower levels up to a diocese level. Now looking into a data, you can see that there is low male involvement. Then, as a solution, you can organize a community dialogue. You reach the ward government and explain about the importance of husbands accompanying their wives to the clinic. Then, you may find that there is some improvement on next months. So, you cannot do all these activities if you have no data. Therefore, it is good for decision making. —CSO, Sengerema • Strengthened CSO skills in delivering quality SBCC. “ When we are involved in the design, development, and preview and testing, we also receive some training on how to facil￾itate the issue at the regional level, council level and even to our CSOs. —BA, National level 52 MIDTERM EVALUATION OF USAID TULONGE AFYA PROJECT IR 2: Evaluation question 2b—Enabling elements on the delivery of quality SBCC at the communi￾ty level through USAID Tulonge Afya sub-part￾ners/other USAID partners • Involvement and orientation of the ward executive officers (WEOs) and village executive officers (VEOs). “ It used to be a challenge before because people thought it was a waste of time to come and listen to you. However, we came to coordinate this activity very well. We started involving the street leaders who in turn helped us reach and mobilize these people. So, we were helped by village and ward executive officers and they know about the TA project. —CSO, Newala IR 2: Evaluation question 2b—Constraints on the delivery of quality SBCC at the community level through USAID Tulonge Afya sub-partners/other USAID partners • Both facilitators and session participants found the use of participatory approaches (i.e., games) for run￾ning small group sessions engaging and enjoyable. “ We use games to deliver the message at the community level…We get feedback from the clients that it’s a nice approach and there are some games they like most. —IP, Rorya • Limited number of volunteers for specific sessions. “ Every ward has three volunteers, two for adult platform and one for youth platform. Given the nature of our district, there are wards which are composed of more than five villages. You find that these villages are very scattered and therefore, it Becomes dif￾ficult for two volunteers to get to all villages…. The project can therefore consider adding one volunteer to the wards with many villages, so that all people in the ward may be reached. —CSO, Newala • Two-day training among different cadres of facilita￾tors is not enough. “ Their package is enough and good; the challenge is training time. So, you find that you have to take several hours to teach train them as you go for supervision. They also have slow catch up speed, they are not as experienced as we are. Therefore, they need more time. —CSO, Sengerema “ They once asked this, ‘how long does it take for the virus to be seen in the test?’ I didn’t answer this question because I don’t know. —FURAHA YANGU facilitator, Sengerema • Lack of equipment such as raincoats, umbrellas, and gumboots. “ We would also wish to get umbrellas for our volunteers because we go through different seasons. We are now getting into rainy seasons. It may happen that a group session is conducted in a classroom or office, but the volunteer has to move from his home to that office. So, these umbrellas are highly needed to make sure that the people are get￾ting services regardless of the season. —CSO, Newala IR 3: Evaluation question 3—These findings illustrate how USAID Tulonge Afya activities led to improved capacity, coordination, collabora￾tion, and co-investment for SBCC at national and subnational levels among IPs, key stake￾holders, and the GOT Improved capacity for SBCC • At the national level, USAID Tulonge Afya built skills of the HPS staff in various aspects of quality SBCC design and delivery. “ I can say for the people in the health promotion section, whom I deal with. Initially they didn’t have enough skills but as BR E A K THROUGH R ESE A RCH | AUGUST 2020 53 we go along, their involvement has been very high. In every activity which TA conduct in the regions, they involve people from the HPS. We usually have monthly meetings to update each other on how we are faring, [and] you can see the changes. They discuss the activities being implemented, and you can see that people keep developing skills and that they are knowl￾edgeable. So, there is a very big change. —GOT, National level • USAID Tulonge Afya supported SBCC training among GOT staff other than HPS working under the specific health programs. “ Even the procedure they used when they conducted the interviews during the audience consultation, they used the projective technique that we didn’t know about and one we never used before. So, that one made people to open up so that you know their deep things so that you can know the kind of message you can create. With that, we were happy, and our main problem was to create attractive messages, and which can help us and truthfully Tulonge helped us with that and we see a big difference. —GOT, National level Improved coordination for SBCC • Improved recognition of HPS as the coordinating body for SBCC activities in the country. “ The biggest change I have seen as I told you is that I have been here for a year now, the department that I am in now, the HPS, was not given much priority before. Now, with the support of TA, everybody is aware of it, the MoH, and PORALG, they are aware that they cannot do anything without the HPS. —GOT, National level • Centralization of SBCC materials production and/or approval. “ Everybody now comes to the HPS; In the past, people just used to take adverts directly to the radio stations, for instance Radio Tanzania and other stations. As of today, this is coordinated by the HPS, and this is mainly because of the support from Tulonge Afya. —GOT, National level Improved collaboration for SBCC • Partners involved to establish SBCC needs for their programs and design of SBCC materials. “ ...we are engaged in every step of this [SBCC materials] development, in the past I think we were more just served with the materials like ‘here they are please use them.’ I think that’s the difference. —BA, National level “ …There are meetings called material review workshops and in most cases are done in Dar es Salaam and we are called among the partners, not only us, but, there are other stakeholders and we go and provide inputs before going to pre-testing and when they go to pre-testing, we do it again. —BA, Mwanza • Improved engagement in SBCC design and delivery with the GOT. “ There is good collaboration in a way that, as I told you, we now sit together, plan and implement together effectively. In the past projects, we used to implement together but somehow, somewhere in the planning, there was a problem. Before, we had TCDC, you find that they have initiated something, they have the idea already and they are at a certain level then they suddenly come saying they have 1,2,3 that they want to implement. TA is doing something different. We sit together, we generate idea together then we implement together. —GOT, National level • Early and meaningful engagement of GOT staff in SBCC design and delivery. 54 MIDTERM EVALUATION OF USAID TULONGE AFYA PROJECT “ If you want the government to act, they must be involved from the beginning. When you come to insert them at the middle, it becomes so difficult. —IP, National level “ There is a big involvement [of the GOT] especially by a gentleman called [XX]. He is a very good link between the ministry and stakeholders. He knows a lot about the systems, and I refer this a facilitating agent. —GOT, National level Cross-cutting question 6: What mitigation strat￾egies have been considered to address limita￾tions and constraints, and how effectively have they been adopted? IR1 mitigation strategies • Lack of items for demonstration during group session. “ You are explaining to us about female condom and how to use it while we don’t see it and we don’t know how to use it and you are telling us to go to the health provider while you are the one teaching us? —SUBIRA beneficiaries, Sengerema Cross-cutting question 7: Barriers to shifting from vertical to integrated SBCC programs • Limited understanding of what integrated SBCC entails in practice. “ For instance, for a person dealing with HIV in vertical programs, they are usually involved in everything, so I go for supervision and I create guidelines, and my work here is to support all HIV interventions in SBCC, whether it be HIV testing, or STIs, or VMMC, or care and treatment. It is everything. So, I must know everything. If you tell me now to start doing SBCC for malaria I would have to start to learn so that I can know all I need about malaria. —GOT, National level • Competing funding priorities between integrated and vertical SBCC programming. “ For example, CDC may tell that you have a partner that does SBCC, so we won’t give you funds for these SBCC activities, but when you come back to these ones they tell you that the condom component is not theirs, or that the circumcision component is not theirs, or that they cannot support STIs. So, this is still going on till now, and as a country you find yourselves losing, to the point that you feel like it would be better if you were just given funds directly then we would be setting other priorities. —GOT, National level “ We are doing the vertical programs, and the country is successful, and the people’s understanding of HIV is high, the country is stabilizing HIV. So, what’s wrong? [Interviewer: I feel like you are not supportive of integration.] Not at all. Because, I am a government communications officer, and there are more than 500 of us. We are all over the country, in every region, districts, and every government institution. So, when you come with something to say that you will integrate all these things in one place it means that you are killing something that the government already started like 5 years ago and which has already picked up, and so you are moving it backwards. —GOT, National level • Vertical nature of USAID programming. “ I think at that level, USAID should also be very well integrated. For the program level its fine, but we are responding differently when it comes to USAID. The HIV focal per￾son will talk to you in his own way, the family planning will tell you in their own way, as well as malaria people will tell you in their own way. —BA, National level BR E A K THROUGH R ESE A RCH | AUGUST 2020 55 Cross-cutting question 8: Constraining factors to improving capacity, coordination, and collab￾oration within integrated SBCC program • A lack of clear guidance on the roles and responsibili￾ties of SBCC coordinators of vertical programs vs. HP coordinators of the integrated SBCC programming. “ I think I can say, the problem is prac￾tices. There is a time they [vertical programs] used to do those [SBCC] activities, for their program specifically. And they are used to doing that, they even employed SBCC personnel. So, telling them to remove that role is a bit hard, but with time they will remove. To me, I don’t see any problem even when they continue as long as they report health promo￾tion indicators, we can help each other if they are reporting well. —GOT, National level “ The main thing is to change people’s mind set. Because if you are a coordina￾tor of a specific program and then you are told that, the activity you were coordinating, there is someone else who needs to coordinate it. It is like taking someone’s responsibility, this is not the case because you are not taking the resources, it is just coordinating. You still have the resources; it is you to sit and see ways to merge the interventions so that all can be done but by saving time and other resources. So, if people understand this it will be good. If we remain with the system of saying this is my activity, we won’t get anywhere. —GOT, National level • At the national level, challenges exist regarding strat￾egies for collaboration between USAID Tulonge Afya, HPS, and vertical programs. “ The way I see is that there is bigger communication between TA and Health Promotion compared with TA with RCHS. It has reached a point that RCHS is only looked at as an invitee to these meetings. We feel that RCHS is not being given weight/value in coop￾eration with Tulonge Afya... It has reached a point that RCHS is pulling back, which I do not think will bring success to Tulonge Afya. —GOT, National level “ There is confusion in the government system, if TA is working with HPS, it was supposed to capacitate HPS. So that HPS can work with malaria and TB without TA going there. If TA goes to work there when I don’t know this confuses the system... if you have built capacity to HPS. So, that they can coor￾dinate these activities and not you leaving to implement health promotion again. It is like we have 2 health promotion units, one for Tulonge and one for the Ministry of Health. —GOT, National level • Differences in the GOT’s (July–June) and USAID Tulonge Afya’s (October–September) fiscal years. “ What I would say, although I don’t know if TA has that mandate, is that at least when they begin the year, which is the American year, they have to make sure that come November, then the TA plan should have been approved. This is so that when for exam￾ple we want to do planning next week, then TA should have an approved plan. After they have the approved plan, it will be easy for HPS to fit in all the activities in their plan, because they are approved; rather than putting the activities in unapproved plan, then having to remove them because they are not approved, then the HPS plan will have already been approved.” —GOT, National level • USAID Tulonge Afya’s flexibility. “ Another thing that I have observed is that, in support, even if there are jobs that are out of the plan, for instance, we have our meetings quarterly. And we discuss what we are going to implement if it is 1,2,3. Even if we have activities that are out of the plan, there are considerations that are made so that we can get the support from them [TA]. —GOT, National level 56 MIDTERM EVALUATION OF USAID TULONGE AFYA PROJECT Annex IX. Data collection tools KII guide—BA and other implementing partners Purpose Understand perceptions of transition from vertical to integrated service delivery for [health areas specified]; how integration fits (e.g., leads to improved capacity, coordination, collaboration, and co-investment for SBCC); challenges and opportunities; and vision for long-term transition to integrated programming. Professional background Background characteristics Participant response Age Gender District Name of organization Role/position within organization Length of time working at organization Interview questions 1. What activities does your organization do/support? a. Does your organization conduct/support any SBCC activities? If yes, i. Which SBCC activities does your organization conduct/support? ii. What SBCC tools does your organization use? And where do you obtain them? 2. What has changed in terms of SBCC programming after compared to before the TA project? Probes: a. Capacity [among implementing partners, key stakeholders, and the GOT] for implementing SBCC interven￾tions—can you give me an example? b. Coordination of SBCC interventions coordination of SBCC interventions—can you give me an example? c. Collaboration for SBCC interventions—can you give me an example? d. Co-investments for SBCC interventions—can you give me an example? 3. What specific support has TA provided to your organization to-date? a. What do you think of the support you have received? b. Was support provided by TA enough to meet your program needs? Please explain. 4. To what extent has TA been able to respond/adapt in order to meet the various needs of your program (including by adjusting plans/strategies)? 5. To what extent is TA collaborating with your organization to ensure that community demand creation activities (by TA) are aligned with availability of skilled staff and commodities in your project areas? 6. What are components of TA project that you would recommend improving on? Please explain. BR E A K THROUGH R ESE A RCH | AUGUST 2020 57 KII guide—GOT (national level) Purpose Understand perceptions of transition from vertical to integrated service delivery for [health areas specified]; how integration fits (e.g., leads to improved capacity, coordination, collaboration, and co-investment for SBCC); challenges and opportunities; and vision for long-term transition to integrated programming. Professional background Background characteristics Participant response Interview code Age Gender District Name of organization Role/position within organization Length of time working at organization Interview questions 1. Did your department conduct or support any SBCC activities prior to TA project? If yes, a. Which SBCC activities did you conduct/support? b. Who did you provide the support to and how was the support organized? c. What SBCC tools did you use and where did you obtain them before TA? 2. Can you tell me what has changed in terms of SBCC programming after compared to before the TA project? Probes: a. Skills/Capacity [among implementing partners, key stakeholders, and the GOT] for implementing SBCC interven￾tions—can you give me an example? b. Coordination of SBCC interventions coordination of SBCC interventions – can you give me an example? c. Collaboration for SBCC interventions—can you give me an example? d. Co-investments for SBCC interventions—can you give me an example? 3. What specific support has TA provided to your organization to date? a. What do you think of the support you have received? b. Was support provided by TA enough to meet your program needs? Please explain. c. What additional support do you think you need from TA in order to do your work? 4. Can you describe to me what the TA integrated social and behavior change program entails? a. In your opinion, what are the facilitators in the delivery of quality integrated SBCC? b. In your opinion, what are the barriers in the delivery of quality integrated SBCC? 5. How have TA organizational and management structures, systems, and procedures affected the implementation of quality integrated SBCC? a. At the national level? b. At the sub-national levels? 6. What are the components of the TA program you would recommend improvements upon? How and why? a. Probe for types/kinds of resources needed to strengthen program, i.e., human resources, partnership building, etc. 58 MIDTERM EVALUATION OF USAID TULONGE AFYA PROJECT KII guide—TA staff (national level) Purpose Understand perceptions of transition from vertical to integrated service delivery for [health areas specified]; how integration fits (e.g., leads to improved capacity, coordination, collaboration, and co-investment for SBCC); challenges and opportunities; and vision for long-term transition to integrated programming. Professional background Background characteristics Participant response Interview code Age Gender District Name of organization Role/position within organization Length of time working at organization Interview questions 1. In your views, can you tell me what has changed in terms of SBCC programming after compared to before the TA project? Probes: a. Skills/Capacity [among implementing partners, key stakeholders, and the GOT] for implementing SBCC interven￾tions—can you give me an example? b. Coordination of SBCC interventions coordination of SBCC interventions—can you give me an example? c. Collaboration for SBCC interventions—can you give me an example? d. Co-investments for SBCC interventions—can you give me an example? 2. What tools does TA have in place to measure changes in the three IRs: a. Improving the ability of individuals to practice healthy behaviors? b. Strengthening community support for healthy behaviors? c. Improving systems for coordination and implementation of SBC interventions? 3. In your opinion, which SBCC activities have been most effective in each of the three results areas that you are focusing on? a. Why do you say so? b. Probe for all results areas c. [If not mentioned]: Which SBCC activities have been most effective in catalyzing positive change in gender and socio-cultural norms within communities? Why do you think that is the case? 4. In your opinion, what challenges exist for TA achieving its objectives in promoting integrated SBCC programming? a. What actions or strategies have been used to address these challenges? b. Do you have any ideas on how to address existing challenges? 5. What are some of the key lessons that you have learnt so far from your experience of implementing integrated SBCC that other implementers can benefit from? 6. Can you give me an example of how the TA project has been using collected data for adaptive management (changes in program implementation based on findings on what’s working and not working well)? BR E A K THROUGH R ESE A RCH | AUGUST 2020 59 a. Is there any information/data that you feel could be helpful in informing program implementation but is not currently being collected by the project? b. Is TA collecting any integration-specific performance indicators that will say something about how and how well integration is working? 7. TA is working with many CSOs, in some regions more than one CSO—different CSOs in different districts—how is this affecting TA in delivering quality SBCC at the community level? Probe for: a. Activities reporting/monitoring workload b. Varying technical and implementation ability c. Standardization of procedures 8. What are the components of the TA program you would recommend improvements upon, how and why? a. Probe for types/kinds of resources needed to strengthen program, i.e., human resources, partnership building, etc. 60 MIDTERM EVALUATION OF USAID TULONGE AFYA PROJECT KII guide—USAID Purpose Understand perceptions of transition from vertical to integrated service delivery for [health areas specified]; how integration fits (e.g., leads to improved capacity, coordination, collaboration, and co-investment for SBCC); challenges and opportunities; and vision for long-term transition to integrated programming. Professional background Background characteristics Participant response Name Gender Name of organization Role/position within organization Length of time working at organization Interview questions 1. To start out, at a high level can you tell me about the roles that USAID has played in TA implementation to date? 2. In your perspective, what has changed (positively or negatively) in terms of SBCC programming after compared to before TA started project implementation? Probes: a. Capacity [among implementing partners, key stakeholders, and the GOT] for implementing SBCC interven￾tions—can you give me an example? b. Coordination of SBCC interventions—can you give me an example? c. Collaboration for SBCC interventions—can you give me an example? d. Co-investments for SBCC interventions—can you give me an example? 3. What do you think have been the major successes to date in SBCC programming that can be attributed to the TA project? 4. What do you think of the support that TA has been providing to other organizations and the GOT in the implemen￾tation of quality SBCC? [Probe for national, regional, and district levels] 5. What challenges, that you are aware of, has TA been facing in supporting other organizations and the GOT in the implementation of quality SBCC? [Probe for national, regional, and district levels] a. What has been the role of USAID in tackling the challenges faced by TA supporting other partners and the GOT? 6. How have USAID organizational and management structures, systems, and procedures affected the implementa￾tion of the TA project? a. To what extent has USAID been able to respond/adapt in order to meet the various needs of TA toward sup￾porting other partners and the GOT (including by supporting adjustments of plans/strategies/budgets)? Can you give me some examples? b. What types of challenges exist in being able to respond/adapt to meet these needs, if any? 7. What are the components of the TA program you would recommend improvements upon? a. Probe for types/kinds of resources needed to strengthen program, i.e., human resources, partnership building, etc. b. Probe for at what levels: national, regional, district, etc. BR E A K THROUGH R ESE A RCH | AUGUST 2020 61 KII guide—implementing partners (TCDC & TMARC) Aim Gain in-depth knowledge about the barriers and facilitators for TA engagement of CSO, implementing partners, and GOT at district and regional levels and the engagement of beneficiary audiences at the community level to support and facilitate delivery of quality SBCC. Professional background Background characteristics Participant response Interview code Age Gender District Name of organization Role/position within organization Length of time working at organization Interview questions 1. In a nutshell, can you tell me about the activities that your organization does/supports? 2. Did your organization conduct/support any SBCC activities prior to the TA project? If yes, a. Which SBCC activities? b. What SBCC tools did your organization use and where did you obtain them before TA? 3. Can you tell me what has changed in terms of SBCC programming after compared to before the TA project? Probes: a. Skills/Capacity [among implementing partners, key stakeholders, and the GOT] for implementing SBCC interven￾tions—can you give me an example? b. Coordination of SBCC interventions —can you give me an example? c. Collaboration for SBCC interventions—can you give me an example? d. Co-investments for SBCC interventions—can you give me an example? 4. What specific support has TA provided to your organization to date? a. What do you think of the support you have received? b. Was support provided by TA enough to meet your program needs? Please explain. c. What additional support do you think you need from TA in order to do your work? d. Mobilization alignment to services availability? 5. Can you describe to me what the TA integrated social and behavior change program entails? a. In your opinion, what are the facilitators in the delivery of quality integrated SBCC? b. In your opinion, what are the barriers in the delivery of quality integrated SBCC? 6. How have TA organizational and management structures, systems, and procedures affected the implementation of integrated SBCC at the community level? 7. What are the components of the TA program you would recommend improvements upon? a. Probe for types/kinds of resources needed to strengthen program, i.e., human resources, partnership building, etc. 62 MIDTERM EVALUATION OF USAID TULONGE AFYA PROJECT IDI guide—implementing partners & CSOs Aim Gain in-depth knowledge about the barriers and facilitators for TA engagement of CSO, implementing partners, and GOT at district and regional levels and the engagement of beneficiary audiences at the community level to support and facilitate delivery of quality SBCC. Professional background Background characteristics Participant response Date IDI code (same as audio file) Age Gender District Name of organization Role/position within organization Length of time working at organization All 1. Can you describe what the TA integrated social and behavior change program entails? 2. What activities is your organization implementing? 3. What SBCC materials do you use in your project? a. Where do you obtain the mentioned materials? b. Do you usually have enough materials? c. Are there additional materials that you might need to implement these components? If yes, what materials and why? [Probe for why additional materials needed] 4. What has changed in relation to SBCC after compared to before the TA project? Probes: a. SBCC capacity b. Coordination of SBCC activities c. Collaboration among various partners for SBCC activities implementation 5. What are the components of the TA program you would recommend improvements upon? a. Probe for types/kinds of resources needed to strengthen program, i.e., human resources, partnership building, etc. 6. How has your organization’s capacity been improved through TA support? a. What additional support do you think you need in order to do your work? 7. In your opinion, what are the facilitators in the delivery of quality integrated SBCC? 8. In your opinion, what are the barriers in the delivery of quality integrated SBCC? 9. How do you think TA approaches have enabled your organization to deliver quality SBCC at the community level? 10. How do you think TA approaches have hindered your organization to deliver quality SBCC at the community level? BR E A K THROUGH R ESE A RCH | AUGUST 2020 63 11. How have TA organizational and management structures, systems, and procedures affected your organization in the delivery quality SBCC at the community level? 12. Who provides information or services directly to the target group? a. What training, if any, have they received to carry out the activity? b. Do you think the training received was enough? Please explain. 13. How do you report information on what activities are being conducted, where, with whom, at what frequency, etc.? a. How are the monitoring data helping you track the beneficiaries’ engagement in services/programs? b. Is there anything that could be improved in terms of monitoring data to inform activities planning and imple￾mentation? Please explain. 64 MIDTERM EVALUATION OF USAID TULONGE AFYA PROJECT FGD guide—session facilitators Aim Gain in-depth knowledge about implementation of Tulonge Afya platform sessions, resonance with participants and satisfaction, and perceived changes in their communities. DATE: DISTRICT: SESSION NAME: INTERVIEWER NAME: FGD CODE: NUMBER OF PARTICIPANTS: COMPLETE TABLE OF PARTICIPANTS CHARACTERISTICS #. Initials Gender Age Position (CHW/CV) Ward Duration worked with TA program P1 P2 P3 P4 P5 P6 P7 P8 Interview questions Training Let us now talk about the training you received through the Tulonge Afya Project. 1. What were the top three things you learned from the Tulonge Afya training? 2. How are you putting what you learned from the training into practice? 3. What additional training or information do you think you need to receive to effectively implement what was learned about? TA sessions Let us now talk about the [JUMA, SUBIRA, FURAHA YANGU, OR NAWEZA] sessions 1. Can you tell me what you know about the Tulonge Afya Project? a. What does SBCC mean to you? b. From your perspective, what do you understand by integrated SBCC programming? c. How do you think you think the [insert session name] sessions are affecting health care seeking behaviors among session participants? d. Can you walk me through how you carry out the [insert session name] session? BR E A K THROUGH R ESE A RCH | AUGUST 2020 65 e. What types of questions do [insert session name] session participants ask? 2. Have you ever been asked challenging questions that you couldn’t answer? a. If yes, what do you usually do in such scenarios? 3. Can you tell me about any challenges you face in delivering the [insert session name] sessions? a. Have you done anything to try to overcome these challenges? What? 4. What should TA do to motivate you and make you do your job better? 5. Have you observed any unexpected behaviors/actions (positive or negative) among session participants as a result of participating in the sessions? a. If yes, please explain b. If no, please explain 6. Are you seeing any changes in session participants’ ability to lead healthy lives as a result of their participation in the program? If yes, please explain a. If yes, why? b. If no, why? c. Probe: knowledge, attitudes, intentions, self-efficacy 7. Are you seeing any changes in your communities as a result of the Tulonge Afya Project? If yes, please explain. a. If yes, why? b. If no, why? c. Probe: behaviors, social norms 8. How do community members learn about availability of the [insert session name] sessions? 9. In your opinion, what has worked the best in making beneficiaries attend all [insert session name] sessions? 66 MIDTERM EVALUATION OF USAID TULONGE AFYA PROJECT FGD guide—session beneficiaries Aim Gain in-depth knowledge about implementation of Tulonge Afya platform sessions, resonance with participants and satisfaction, and perceived changes in their communities. DATE: DISTRICT: SESSION NAME: INTERVIEWER NAME: FGD CODE: NUMBER OF PARTICIPANTS: COMPLETE TABLE OF PARTICIPANTS CHARACTERISTICS #. Initials Age Ward # sessions attended Duration lived in the district P1 P2 P3 P4 P5 P6 P7 P8 Interview questions: experience with Tulonge Afya sessions 1. Have you ever heard of TA? a. If yes, how did you first hear about the Tulonge Afya Project? b. Did you hear about TA before participating in the [insert session name] session? 2. We have heard that the Tulonge Afya program has many activities. Can you tell me more about them? a. What activities have you participated in? 3. How did you first learn about availability of the [insert session name] session? 4. Why did you decide to join the [insert session name] session? 5. What was your favorite topic? Why? a. Which topics did you find the most helpful? Why? b. Which topics did you find the least helpful? Why? c. Was there anything that wasn’t discussed that would have been helpful to include? 6. Are you doing anything differently as a result of what you learnt from the [insert session name] session? If yes, please explain. 7. Are you seeing any changes in your communities as a result of the Tulonge Afya Project? If yes, please explain. BR E A K THROUGH R ESE A RCH | AUGUST 2020 67 a. Probe: behaviors, social norms 8. What are your suggestions for improving the [insert session name] sessions? 68 MIDTERM EVALUATION OF USAID TULONGE AFYA PROJECT Structured observations—TA sessions Aim Gain in-depth knowledge about implementation of Tulonge Afya platform sessions, resonance with participants and satisfaction, and perceived changes in their communities. Background characteristics District Session Number of participants Materials used during session Topics covered during session Notes on questions asked by participants or their concerns with messages Notes on perceptions of facilitator’s delivery of session (facilitation skills, knowledge on session content, ability to respond to the question) Notes on engagement of participants and interactive characteristics of session (Allow them ask question, allow participants to give opinions) Notes on post session referrals Notes on any other observations Duration of session BR E A K THROUGH R ESE A RCH | AUGUST 2020 69 Population Council 4301 Connecticut Ave., NW | Suite 280 Washington, DC 20008 +1 202 237 9400 breakthroughactionandresearch.org