MID-TERM PERFORMANCE EVALUATION OF “BORESHA AFYA,” THE COMPREHENSIVE HEALTH SERVICE DELIVERY PROJECT, USAID/TANZANIA February 2020 This publication was produced at the request of the United States Agency for International Development. It was prepared independently by ME&A, Inc. MID-TERM PERFORMANCE EVALUATION OF BORESHA AFYA, THE COMPREHENSIVE HEALTH SERVICE DELIVERY PROJECT, USAID/TANZANIA February 2020 Data for Development, Contract Number: AID-OAA-I-15-00024/AID-621-TO-17- 00005 Period of Performance: August 3, 2017 to February14, 2020 Contracting Officer Representative (CoR): Michael McBroom Submitted by: David Hughes, Chief of Party ME&A (Mendez England & Associates) 4300 Montgomery Ave. Suite 103 Bethesda, MD 20814 Tel: 301-652-4334 Email: dhughes@engl.com Cover Photo: USAID Boresha Afya, Girls Youth Group Mawenzi Regional Hospital DISCLAIMER The author’s views expressed in this publication do not necessarily reflect the views of the United States Agency for International Development or the United States Government. TABLE OF CONTENTS EXECUTIVE SUMMARY.................................................................................................................................. i 1.0 INTRODUCTION.............................................................................................................................. 8 1.1 Evaluation Purpose and Audiences.......................................................................................... 8 1.2 Evaluation Questions.................................................................................................................. 8 2.0 BACKGROUND................................................................................................................................. 9 2.1 Brief Description of Boresha Afya Activities........................................................................ 9 2.1.1 Overview of Boresha Activities in Tanzania ............................................................. 9 2.2 CHSD Results Framework......................................................................................................10 2.2.1 Integrated Approach to Health Programming ....................................................... 11 3.0 METHODOLOGY........................................................................................................................... 13 3.1 Approach.....................................................................................................................................13 3.2 Data Sources..............................................................................................................................13 3.2.1 Document Review........................................................................................................ 13 3.2.2 Key Informant and Group Interviews...................................................................... 13 3.2.3 Focus Group Discussions........................................................................................... 14 3.2.4 Primary Data: Facility Checklist and Client Exit Surveys.................................... 14 3.2.5 Secondary Data............................................................................................................. 15 3.3 Sample Selection........................................................................................................................15 3.3.1 Field Site Selection ....................................................................................................... 15 3.4 Data Analysis..............................................................................................................................16 3.4.1 Quantitative Data Analysis......................................................................................... 16 3.4.2 Qualitative Data Analysis............................................................................................ 16 3.5 Ethical Considerations and Human Subject Protection....................................................16 3.6 Limitations..................................................................................................................................16 4.0 FINDINGS ......................................................................................................................................... 17 4.1 Evaluation Question 1..............................................................................................................17 4.1.1 Organizational Capacity.............................................................................................. 17 4.1.2 External Management and Collaboration ............................................................... 17 4.2 Evaluation Question 2 A-G .....................................................................................................21 4.2.a How Effective and Timely Is Each IP in Making Necessary and Adaptive Changes to Improve Performance?........................................................................... 21 4.2b How effectively has UBA integrated the targeted health areas into a functioning IHC system at the district level and nationwide? ............................. 24 4.2c How has the implementation of UBA’s IHC efforts been influenced by other IHC initiatives and funding sources?.......................................................................... 28 4.2d How effective are the integrated programs under Boresha Afya expanding access and coverage in HIV, TB, MCH, FP, malaria, nutrition, gender, and GBV?................................................................................................................................. 30 4.2.e How cost effective is the integrated approach to implementation to Boresha Afya IPs in coordination with each other, USAID, and the GOT?..................... 38 4.2.f How Has the Developmental Evaluation Program Contributed to the Effectiveness, Learning, and Adaptability of UBA Implementers?....................... 42 4.3 Evaluation Question 3..............................................................................................................45 4.3.1 Background .................................................................................................................... 45 4.3.2 Findings ........................................................................................................................... 46 4.4 Evaluation Question 4..............................................................................................................49 4.4.1 Barriers for Women.................................................................................................... 49 4.4.2 Barriers for Youth........................................................................................................ 50 4.4.3 Barriers for Men........................................................................................................... 51 4.4.4 UBA Impact on Gender and Youth Barriers.......................................................... 52 5.0 SUSTAINABILITY PROSPECTS AND CHALLENGES ........................................................... 56 5.1 Prospects for UBA Sustainability...........................................................................................56 5.2 Challenges for Sustainability ...................................................................................................58 5.3 Sustainability Factors From the National Context............................................................60 6.0 CONCLUSIONS.............................................................................................................................. 62 6.1 Evaluation Question 1..............................................................................................................62 6.1.1 Recommendations for EQ 1 ...................................................................................... 63 6.2 Evaluation Question 2..............................................................................................................63 6.2.1 Recommendations for EQ 2 ...................................................................................... 64 6.3 Evaluation Question 3..............................................................................................................65 6.3.1 Recommendations for EQ 3 ...................................................................................... 65 6.4 Evaluation Question 4..............................................................................................................65 6.4.1 Recommendations for EQ 4 ...................................................................................... 65 6.5 Sustainability ...............................................................................................................................66 6.5.1 Recommendation for Sustainability.......................................................................... 66 ANNEXES ....................................................................................................................................................... 67 Annex 1: Scope of Work ................................................................................................................68 Annex 2: Evaluation Methods and Limitations ...........................................................................94 Annex 3: Persons Interviewed .................................................................................................... 114 Annex 4: Sources of Information/Bibliography of Documents Reviewed......................... 123 Annex 5: Data Collection Instruments..................................................................................... 127 Annex 6: Detailed Data Analysis Tables................................................................................... 149 Annex 7: Disclosure of Any Conflicts Of Interest ................................................................. 162 LIST OF TABLES Table 1: UBA Activity Summary.................................................................................................................... 9 Table 2: Boresha Afya Program Areas by Region .................................................................................. 10 Table 3: Number of KIIs and GIs and Their Affiliations Within the UBA IP Zones....................... 14 Table 5: Health Facilities and Hospitals Selected for Evaluation in the Three UBA Zones.......... 15 Table 6: Summary of each IP’s Approaches and Interventions for delivery of quality IHC.......... 25 Table 7: Comparison of IP Expenditures by Health Area .................................................................... 30 Table 8: IP Cumulative Achievements to Date in FP/RH from IPRS.................................................. 30 Table 9 Malaria Prevalence and Prevention Interventions 2016 to 2019 ......................................... 31 Table 10: Reported Support Received by the HCPs of the 18 Surveyed Facilities........................ 32 Table 11: FP and MNCH Trends 2016 to 2019...................................................................................... 33 Table 12: FP Improvements Attributed to UBA in Surveyed Facilities............................................. 33 Table 13: Reported Integration of FP Services....................................................................................... 33 Table 14: National Trends in UBA Regions for Couple Years Protection....................................... 34 Table 15: HIV Indicator Target Achievement by IP and Year............................................................. 34 Table 16: National DHIS2 Data on HIV Intervention 2017-2019 by UBA Regions....................... 34 Table 17: Burden of Disease for Past Month in Surveyed Facilities................................................... 35 Table 18: Reported Challenges to TB Integration by Facilities........................................................... 36 Table 19: Targets vs Results for TX_TB.................................................................................................. 36 Table 20: Availability of Medical Guidelines in Facilities....................................................................... 36 Table 21: Reported Type of Supervisory Visits in the Last 12 Months............................................. 37 Table 22: Comparison of Perceived Satisfaction with the Quality of Care Received.................... 38 Table 23: IP Budgets, Actuals, and Burn Rates........................................................................................ 38 Table 24: Comparison of Expenditure Patterns in Regions: EGPAF and Deloitte ......................... 38 Table 25: IP Cost Comparisons in Selected Program Areas............................................................... 39 Table 26: Cost and Frequency of Patient Contact for HIV Support Services by Available UBA and Previous Tanzania Service Models..................................................................................................... 42 Table 27: Comparison of HCPs’ Satisfaction Scores............................................................................. 48 Table 28: Percentage of Clients Satisfied with Health Facility Services............................................. 49 LIST OF FIGURES Figure 1: UBA Coverage Areas.................................................................................................................. 11 Figure 2: Performance of RMNCH Composite Indicator in Tanzania, 2015-2018 ......................... 46 ACRONYMS Acronym Description AIDS Acquired Immune Deficiency Syndrome ANC Antenatal Care ANC-PNC Antenatal to Postnatal Care AOR Agreement Officer Representative ART Antiretroviral Therapy ARV Anti-Retroviral BEmONC Basic Emergency Obstetric and Neonatal Care BR Burn Rate CBO Community-Based Organization CDC Centers for Disease Control CDCS Country Development Cooperation Strategy CEmONC Comprehensive Emergency Obstetric and Neonatal Care CH Community Health CHMT Council Health Management Team CHSD Comprehensive Health Service Delivery CHW Community Health Worker CIRCLE Coordinating Implementation Research to Communicate Learning and Evidence CLA Collaborating, Learning, and Adapting CO Clinical Officer COC Continuum of Care COP Chief of Party COP Country Operational Plan (PEPFAR) COR Contracting Officer Representative CSO Civil Society Organization CTC Care and Treatment Clinic CUG Closed User Group DATIM Data for Accountability, Transparency, and Impact Monitoring DC District Council DE Development Evaluation DEC Development Experience Clearinghouse DFF Direct Facility Funding DHIS2 District Health Information System 2 DMO District Medical Officer DO Development Objective Acronym Description DoD Department of Defense DSM Dar es Salaam EGPAF Elizabeth Glaser Pediatric AIDS Foundation EH Engender Health EOP End-of-Project EQ Evaluation Question ET Evaluation Team FBO Faith-Based Organization FGD Focus Group Discussion FHI 360 Family Health International 360 FP Family Planning FY Fiscal Year GBV Gender-Based Violence GI Group Interview GIS Geographic Information System GOT Government of Tanzania HC Health Center HCP Health Care Provider HCT Voluntary Counseling and Testing for HIV HCW Health Care Worker HF Health Facility HIV Human Immunodeficiency Virus HMIS Health Management Information Service HQ Headquarters HR Human Resources HRH Human Resources for Health HSHSP Health Sector HIV and AIDS Strategic Plan HSS Health Systems Strengthening HSSP Health Sector Strategic Plan ICT Information and Communications Technology IHC Integrated Health Care IMCI Integrated Management of Childhood Illness IP Implementing Partner IPRS Implementing Partners Reporting System IR Intermediate Result J2SR Journey to Self-Reliance JPPM Joint Program Planning Meeting Acronym Description KII Key Informant Interview LGA Local Government Authority LIFE Linking Initiatives for the Elimination of Pediatric HIV LOE Level of Effort LOP Life-of-Project LTFU Lost to Follow Up M&E Monitoring and Evaluation MC Municipal Council MCH Maternal and Child Health MD Medical Doctor MDH Management and Development for Health MNCH Maternal, Newborn, and Child Health MoFP Ministry of Finance and Planning MOH Ministry of Health MOHCDGEC Ministry of Health, Community Development, Gender, Elderly, and Children MOU Memorandum of Understanding MTE Mid-Term Evaluation MTR Mid-Term Review NA Not Applicable NAAIA National Accelerated Action and Investment Agenda for Adolescent Health and Wellbeing NACOPHA National Council of People Living with HIV/AIDS NEHCIP-TZ National Essential Health Care Intervention Package Tanzania NGO Non-Governmental Organization NIMAT Nurse-Initiated Management of ART NIMR National Institute for Medical Research OPD Outpatient Department PAD Project Appraisal Document PEPFAR President’s Emergency Plan for AIDS Relief PHC Primary Health Care PMI President’s Malaria Initiative PLHIV People Living with HIV PMTCT Prevention of Mother-to-Child Transmission PNC Post-Natal Care PO-RALG President’s Office for Regional Administration and Local Government Acronym Description PP Percentage Points PS3 Public Sector Systems Strengthening PY Performance Year Q Quarter QI Quality Improvement RBF Results-Based Financing RCH Reproductive and Child Health RH Reproductive Health RHMT Regional Health Management Team RMNCH Reproductive, Maternal, Neonatal, and Child Health RMNCAH Reproductive, Maternal, Neonatal, Child, and Adolescent Health RMO Regional Medical Officer SAA Social Analysis and Action SBCC Social and Behavior Change Communication SRH Sexual and Reproductive Health SOP Standard Operating Procedure SOW Scope of Work STI Sexually Transmitted Infection STTA Short-Term Technical Assistance TA Technical Assistance TB Tuberculosis TWG Technical Working Group U.S. United States UBA USAID Boresha Afya UNAIDS Joint United Nations Program on HIV/AIDS UNFPA United Nations Population Fund USAID United States Agency for International Development USG United States Government WHO World Health Organization ZAMEP Zanzibar Malaria Elimination Program i EXECUTIVE SUMMARY INTRODUCTION: MIDTERM EVALUATION OF BORESHA AFYA PROJECT The United States Agency for International Development (USAID) Boresha Afya (UBA) project was designed to increase access to quality comprehensive and integrated health services with a focus on Human Immunodeficiency Virus (HIV), Tuberculosis (TB), Family Planning (FP), Reproductive, Maternal, Neonatal, and Child Health (RMNCH), malaria, and nutrition outcomes. The evaluation’s findings, conclusions, and recommendations will be used to improve program direction and implementation over its remaining activity period and to assist USAID/Tanzania in shaping the direction of its future health programming. The primary audience for the evaluation is the USAID/Tanzania health team and UBA project staff across all three awards. The period of evaluation is for Years 1-3 of UBA implementation. BACKGROUND The UBA project was designed under the Comprehensive Health Service Delivery (CHSD) to support the Government of Tanzania (GOT) to increase access to quality comprehensive and integrated health services. GOT has made commendable efforts in developing its community health programs through the development of policies, guidelines and supporting research studies to deepen the knowledge base1,2,3 . In support of these efforts, the UBA is implemented in three zones of Tanzania by the following implementing partners (IPs): 1. Boresha Afya Southern Zone – Deloitte Consulting Ltd. is Prime with subcontractors Family Health International 360 (FHI 360), Management and Development for Health (MDH), and Engender Health (EH) in six regions: Iringa, Lindi, Morogoro, Mtwara, and Njombe for HIV, TB, malaria, FP, and RMNCH and Ruvuma, for malaria only. 2. Boresha Afya Lake/West Zones – Jhpiego is Prime with sub-recipients PATH and EH in seven regions: Mwanza, Kagera, Mara, Shinyanga, Geita, Kigoma, and Simiyu for FP, sexual/reproductive health (SRH), malaria, maternal, newborn, and child health (MNCH), adolescent and community empowerment, and systems strengthening for health service delivery. Activities are also carried out in Zanzibar. 3. Boresha Afya North/Central Zones – The Elizabeth Glaser Pediatric AIDS Foundation (EGPAF) is Prime with sub-recipient EH in six regions: Arusha, Dodoma, Kilimanjaro, Manyara, Singida, and Tabora for HIV, TB, and FP. 1 Ministry of Health and Social Welfare (2008). National Guidelines for Implementing Community Based Health Initiatives. Dar Es Salaam Government of Tanzania. 2 Kanté et al The impact of paid community health worker deployment on child survival: the connect randomized cluster trial in rural Tanzania. BMC Health Services Research (2019) 19:492 https://doi.org/10.1186/s12913-019-4203-1 3 Ministry of Health, Community Development, Gender, Equity and Children (MOHCDGEC). 2018a. Improved CHF Design Document. Dar es Salaam: Government of Tanzania ii EVALUATION QUESTIONS The evaluation was guided by the following four main evaluation questions (EQs)4 : 1. To what extent has the UBA program been effectively managed and overseen? 2. How effectively has UBA’s integrated approach5 been implemented and how adaptively have IPs managed performance across program intervention areas? 3. To what extent are service providers at the President’s Office for Regional Administration and Local Government (PO-RALG), local government authority (LGA), and facility levels who are receiving integrated systems strengthening and technical support satisfied with the services and technical assistance (TA) provided and are they applying skills and practices to their work? 4. How effective is UBA at identifying and addressing key gender- and youth-related barriers to service delivery? EVALUATION METHODS • A mixed-methods design was used to collect quantitative and qualitative data in all three Boresha Afya Zones. Two regions and councils from each zone and three facilities from each council were selected for primary source data collection. See Section 4.0 of the body of the report for detailed methods and sampling. • Primary and secondary data were collected to address each EQ and sub-question. • Desk review of approximately 120 documents, related to UBA and integrated health care (IHC). • Approaches included key informant interviews (KIIs), group interviews (GIs), and focus group discussions (FGDs) as well as secondary analysis of the Implementing Partner Reporting System (IPRS), Data for Accountability, Transparency, and Impact Monitoring (DATIM), District Health Information Software (DHIS2) and Program data. • A Facility Checklist was used to capture practitioner perspectives on UBA technical support and capacity-building in all 18 facilities assessed. • Client Exit Interviews were used to capture client satisfaction and perspectives—180 interviews in total, across all 18 facilities.6 • Findings from the above methods were consolidated and analyzed using methodological triangulation for each evaluation question and sub-question. KEY FINDINGS, CONCLUSIONS, AND RECOMMENDATIONS Evaluation Question 1: To what extent has the UBA program been effectively managed and overseen? • The structure of the funding streams makes management of the UBA contracts/cooperative agreements challenging for the USAID Agreement Officer Representatives (AORs) and health office technical staff. They all have the same results framework but have different interventions and measure different indicators. AORs’ roles are focused on individual IPs and 4 Sub-EQs are articulated in the Section 3.0. 5 Integrated approach in this programmatic context has a twofold definition mentioned as mentioned above. 6 This will be identified in the gender and youth action plan and President’s Emergency Plan for AIDS Relief (PEPFAR) guidance. iii contract mechanisms and there is no single central coordination role among them, which makes collaboration and information sharing between IPs/zones and health areas difficult. It also makes the reporting process to Washington challenging in that the reporting is not geared toward integrated activities but non-integrated initiatives and distinct information systems (IPRS and DATIM). • Although there were initial differences in organization approach, intra-consortium collaboration is functioning more effectively. Inter-zonal collaboration between IPs is still inadequate except through informal and technical level sharing. Information is not centralized or shared in real-time among IPs as originally conceived in the design. DE is contributing operations research and information. However, the award is limited in being able to provide a wider scope with actual real time monitoring or being able, or given the mandate, to consolidate M&E data from all three UBA IPs. Although there are hopeful recent advances, each IP has its own Monitoring and Evaluation (M&E) team and is largely invested in reporting on performance on an annual or quarterly basis. While IPs from distinct zones are in some cases (e.g., President’s Emergency Plan for AIDS Relief [PEPFAR] collaboration framework) harmonizing their efforts with community-based partners, there is limited information sharing between prime IPs. • Coordination with the national government could have been stronger. The UBA program was designed with a communication framework for UBA and the national GOT, but this was not operationalized, which resulted in a lack of cohesion and UBA speaking with three separate rather than a single UBA voice on IHC. • Coverage within targeted councils is generally at high-volume facilities. EGPAF and Deloitte have selected facilities in their regions determined under PEPFAR guidance based on volume and have organized their interventions in tiers. Jhpiego offers the same package of interventions in all facilities, irrespective of facility size. Service gaps remain in dispensaries and in Zanzibar (where there is only intervention at high-volume hospitals with some primary health care units added as this evaluation was completing in PY4). A concerted and coordinated expansion strategy and plan for scale up of IHC models has not been developed among the IPs or with national GOT. • At the regional and council levels UBA TA is enhancing coordination and collaboration. Regional health management teams (RHMTs) and Council health management teams (CHMTs) work with UBA staff in supporting health facilities (HF) and performing supportive supervision tasks to reinforce best practices. • Improving synergies between facility-based activities and community-based activities were observed. Community health workers (CHWs) and facility practitioners across zones are progressively strengthening referrals and follow-up for continued treatment and care. The institutionalization of the role of CHWs in facilities and in the community will depend on cascading technical support to all HFs based on new evidence and guidelines on training, incentives, tasks and realistic financings, e.g., evidence based. The success of the community￾to-facility integrated continuum of care will depend on the successful close collaboration between UBA, other USAID community-based projects, GOT, and involved stakeholders. iv Recommendations for EQ 1 • UBA should develop a cross-zonal collaborating, learning, and adapting (CLA) plan with monthly IP meetings similar to the PEPFAR collaboration framework but with in-depth and transparent information and knowledge sharing among IPs and stakeholders. • A common knowledge management system and real-time database is needed that is based on IHC indicators and tools to monitor these activities under the contract mechanisms. UBA should improve use of IPRS for performance reporting, and geographic information system (GIS) and dashboards for improved tracking of integration in service delivery. Cost effectiveness specific to IHC should be tracked on an ongoing basis to better understand the value of activities. • A functional mechanism or structure needs to be instituted whereby UBA plans and coordinates implementation and reporting with USAID and national GOT in a harmonized and a collaborative manner.7 Evaluation Question 2: How effectively has UBA’s integrated approach 8 been implemented and how adaptively have IPs managed performance across program intervention areas? • The aim of the design of the CHSD that guided UBA was to work as three components of a seamless zonal project focused on facilities with community partnerships. However, the submitted technical proposals had significant and unique differences in targets and activities. Consequently, comparing the monitoring of performance and the measures of achievement in each zone, while incorporating their individual approaches, interventions, and results is challenging. • The three UBA IPs have been adaptive and timely in response to numerous changes in United States Government (USG) administration and funding, GOT policy changes, and changing conditions in regions, councils, and facilities. IPs changed standard operating procedures (SOPs), strategies, and procedures to adapt to changes in partnership with GOT partners which enhanced performance. • The structure for UBA funding through PEPFAR, the President’s Malaria Initiative (PMI), and other sources competes with an integrated care approach in that each source has siloed targets. Finances are available only for those activities which align with these targets. While PEPFAR has its own integration agenda including systems strengthening, reproductive health (RH) and TB integration with HIV as its central component with the facility HIV/AIDS care and treatment clinic (CTC) as its focus, it sometimes overshadows other efforts outside of the HIV/AIDs context. PEPFAR’s target councils change year to year based on HIV epidemic control leaving other funding streams’ activities to readjust in the wake of the annual PEPFAR Country Operational Plan (COP). The ET recommends that the HO develop an integration approach through 1) Increased effectiveness and simplified management; 2) Define geographic outcomes in a select number of regions where system strengthening, facility, and community 7 One approach to build upon MoHCDGEC sector wide approach dialogue structure for technical information sharing across partners in the sector. 8 Integrated approach in this programmatic context has a twofold definition as mentioned above. v services are to benefit a defined target population; and 3) Increase cost efficiency and gathering evidence of the most effective IHC models. • This evaluation has seen evidence of efficiencies of an integrated approach at the level of service delivery and of reducing clients’ opportunity costs for receiving multiple services in fewer visits. Health systems strengthening at the national, regional, facility, and community levels plays a crucial role in successful vertical integration. • The IPs did not monitor the cost-effectiveness of various integration approaches or activities and have not developed a common system to share evidence on cost effectiveness. Cost￾effectiveness data from the evaluation of the previous TUNAJALI II project appear not to have been used as a benchmark to make cost-effective decisions. Because the IPs have different sets of activities, it is challenging to compare the overall cost-effectiveness of IPs and their individual IHC models. Such a comparison would have been highly informative to guide IHC decision-making. • While the effectiveness of the development evaluation (DE) as a real-time feedback mechanism is less clear, its role as a tool for learning and adapting has shown more recent gains in collaboration and consortia, particularly during the September reporting of cross￾cutting findings for performance year (PY) 2. While DE may have had difficulties adding value in its first years of implementation, it has the potential to be a productive tool for learning and decision-making. Recommendations for EQ 2 • This evaluation, while not conclusive on the issue of effectiveness of integration, has seen reported evidence of benefits of an integrated approach for clients. On the other hand, health systems strengthening at the national, regional, facility, and community levels should be a central component of any follow-on design. Contract mechanisms should allocate core funding to assist with coordination yet be flexible enough to allow for adaptations and innovation while it induces necessary IP collaboration and cooperation. • Community engagement in strengthening the referral system and treatment and care follow￾up of chronic health conditions should be integrated into a community health care system in partnership with GOT and its stakeholders. • The use of information and communications technology (ICT) technologies and current ICT innovations in the health field need to be integrated into client health care including community outreach, referrals, tracking, and follow-up for successful IHC from both horizontal and vertical perspectives • As stated under EQ 1, support of CHWs including volunteers and their specific roles and responsibilities in community health should be explicit in a manner that will ensure their ability and motivation to participate and meet GOT health care standards, and in a manner without undue opportunity costs or personal financial burden. USAID and UBA should begin to negotiate with LGAs and national GOT as a part of succession planning to ensure budget commitments can support the central role of CHWs in providing community health especially in regard to already initiated IHC activities. Their involvement, as well as costs, need to be informed and aligned with current evidence and debate on the role and conditions of CHW’s contribution to health care and disease prevention in their communities. vi • The DE should be supported to expand its operations research approach at local levels to include higher-level IHC learning and knowledge sharing. IPs could also play a role in supporting an integrated learning agenda across Boresha Afya. Evaluation Question 3: To what extent are service providers at the President’s Office for Regional Administration and Local Government (PO-RALG), local government authority (LGA), and facility levels who are receiving integrated systems strengthening and technical support satisfied with the services and technical assistance (TA) provided and are they applying skills and practices to their work? • Among health management teams in the regions and districts, there was a high level of satisfaction with UBA TA. CHMTs reported that regular supportive supervision and mentoring were being conducted in the facilities. • Facility staff, including supervisors and practitioners, were satisfied with UBA TA and have observed changes in the quality of care in facilities. • Sampled clients reported increased access to more than one service during their visits at facilities. They also reported satisfaction with the quality of integrated services. • While a high level of satisfaction across stakeholders indicates TA was significant, there are some remaining gaps in specialized training and managerial skills for supervisors. Recommendations for EQ 3 • Continuing medical and pre-service training in specialized topics for treatment and care in primary and secondary facilities, e.g., emergency obstetrics should be provided to address the prevention of child and maternal death. • The continued use of district mentors and professional exchange visits from experienced IHC practitioners should be supported in follow-on activities and replicated in other zones to improve the skills of other facility staff. • Assessment approaches in facilities such as those used by Jhpiego could be used for organizational development and rebalancing staff resources in high performing facilities to address Human Resources for Health (HRH) challenges in other facilities. • UBA should continue to support GOT in planning and budgeting to recruit and develop health care workers (HCWs) and CHWs as well as contribute to training costs to ensure ongoing institutionalization of informed IHC practices including pre-service training according to GOT guidelines. Evaluation Question 4: How effective is UBA at identifying and addressing key gender- and youth-related barriers to service delivery? • Respondents reported stigma as the most common barrier for women and youth in accessing health services. Other major barriers faced by women and youth include normative systems that hinder these and other groups from accessing health services. Male partners too often prevent women from accessing and utilizing health services and need to be engaged in resolving this challenge and there are still gaps in gender-based violence (GBV) responses at the community and facility levels. Improved and responsive coordination between HFs, law enforcement, social services, and communities is required for the long term. • UBA interventions such as male engagement and youth groups have been implemented across zones but need to be expanded and intensified with community engagement across activity vii sites. The monitoring of gender and youth outcomes in accessing and using services remains limited. Recommendations for EQ 4 • The gender, GBV, and youth-friendly services components of the UBA program should be more focused and better funded in order to meet the persistent health needs of women and youth. Their participation and that of their communities should be central in this effort. In addition, facilities, police, judicial, social support systems, and communities need to be mobilized to respond quickly to incidences of GBV. • To address the gravity of social norms and institutional stigma challenges, social and behavior change communication (SBCC) needs to be sustained with continued force, if not intensified, to reach more women, youth, and their communities through direct and participatory activities. Community engagement and work through influential community members/peers and religious and traditional leadership is key to expanding awareness and active responses. • Facility HCWs and outreach staff, (e.g. CHWs) can play a central role in addressing gender and youth barriers to service delivery as agents of change in their interactions with clients as well as in communities. Towards this end, UBA should ensure that these staff are actively oriented and engaged in these issues through its mentoring and capacity building activities. Sustainability • Various challenges to institutional and financial sustainability are evident across findings. Sustainability or succession planning has become more urgent in the case of councils where PEPFAR’s targeted funding is discontinuing before the close of the program. To mitigate risks, UBA in the three Zones are starting to actively work with RHMTs and CHMTs to institutionalize practices in integrated health systems strengthening. Formal sustainability planning with LGAs has started in Southern Zone. Recommendation for Sustainability • UBA should begin to negotiate with national GoT regarding succession planning and ensure budget commitments can support the ongoing facility and community-based activities under the program. At this time these efforts will continue to rely on financial support from international donors. 8 1.0 INTRODUCTION 1.1 EVALUATION PURPOSE AND AUDIENCES The United States Agency for International Development (USAID) Boresha Afya (UBA) Project was designed under the Comprehensive Health Service Delivery Project (CHSD) to support the Government of Tanzania (GOT) to increase access to quality, comprehensive, and integrated health services with a focus on reproductive, maternal, neonatal, and child health (RMNCH) and nutrition outcomes. The evaluation’s findings, conclusions, and recommendations will be used to improve project direction and implementation over its remaining activity period and to assist USAID/Tanzania in shaping the direction of its future health programming. The primary audience for the evaluation is the USAID/Tanzania health team and UBA project staff across all three awards. The mid-term evaluation (MTE) will be used by USAID for decision￾making in determining future project design and the upcoming strategy for facility-based health interventions. The evaluation period is Years 1-3 of UBA implementation. 1.2 EVALUATION QUESTIONS Four primary evaluation questions (EQs), and associated sub-EQs, 9 articulated in the UBA performance evaluation Scope of Work (SOW) (see Annex 1), cover the themes of program results and effectiveness, implementation approaches and constraints, lessons learned for future programming, and USAID/Tanzania management and oversight. 1. To what extent has the UBA program been effectively managed and overseen? 2. How effectively has UBA’s integrated approach10 been implemented and how adaptively have implementing partners (IPs) managed performance across program intervention areas, including human immunodeficiency virus (HIV), tuberculosis (TB), maternal, newborn, and child health (MNCH), family planning (FP), malaria, etc.? 3. To what extent are service providers at the President’s Office for Regional Administration and Local Government (PO-RALG), local government authority (LGA), and facility levels who are receiving integrated systems strengthening and technical support satisfied with the services and technical assistance (TA) provided and are they applying skills and practices to their work? 4. How effective is UBA at identifying and addressing key gender - and youth-related barriers11 to service delivery? 9 IPs consist of the prime contractor plus sub-contractors, also referred to as consortium members. 10 Source: SOW Mid-Term Evaluation of the Boresha Afya Awards. 11 The DE is focused on measuring real-time results on the integration of services in the activity across the three implementing mechanisms. A small contract team works with monitoring and evaluation (M&E) staff embedded across all three IP consortia and across regional teams to undertake the DE. Meanwhile M&E staff from each mechanism are responsible for each zone ongoing performance measurement of standard and custom indicators. 9 2.0 BACKGROUND 2.1 BRIEF DESCRIPTION OF BORESHA AFYA ACTIVITIES 2.1.1 Overview of Boresha Activities in Tanzania UBA was designed under the CHSD to support the GOT to increase access to quality comprehensive and integrated health services with a focus on RMNCH and nutrition outcomes. The project is implemented in the three zones by the IPs described below and in Table 1. 1. Boresha Afya Southern Zone – Deloitte Consulting Ltd. is Prime with subcontractors Family Health International 360 (FHI 360), Management and Development for Health (MDH), and Engender Health (EH) in six regions: Iringa, Lindi, Morogoro, Mtwara, Njombe (HIV, TB, malaria, FP), and MNCH with Ruvuma (malaria). 2. Boresha Afya Lake/West Zone – Jhpiego is Prime recipient with sub recipients PATH and EH in seven regions: Mwanza, Kagera, Mara, Shinyanga, Geita, Kigoma, and Simiyu for FP, sexual and reproductive health (SRH), malaria, MNCH, adolescent and community empowerment, and systems strengthening for health service delivery. Activities are also carried out in Zanzibar. 3. Boresha Afya North/Central Zones – The Elizabeth Glaser Pediatric Acquired Immune Deficiency Syndrome (AIDS) Foundation (EGPAF) is Prime recipient with sub recipient EH in six regions: Arusha, Dodoma, Kilimanjaro, Manyara, Singida, and Tabora for HIV, TB, and FP. Table 1: UBA Activity Summary Activity Implementer TEC Southern Zone Deloitte Consulting Ltd. $138,282,996 Lake/Western Zone Jhpiego $56,450,000 North/Central Zone EGPAF $123,029,607 Integrated services to be improved by UBA include: counseling and provision of comprehensive FP services, HIV testing and treatment, TB/HIV integration, TB diagnostics and treatment (including multi-drug resistant TB), malaria diagnostics and treatment, maternal and newborn health, including antenatal care (ANC), normal delivery, basic and comprehensive emergency obstetric and neonatal care (BEmONC and CEmONC), care of the preterm neonatal, postnatal care (PNC), child health and nutrition, and integrated management of childhood illness (IMCI). Table 2 summarizes the UBA program areas by zones and regions. Five Southern Zone regions (Iringa, Lindi, Morogoro, Mtwara, and Njombe) have all UBA components. The Lake and Western Zone regions do not receive funding to improve HIV or TB services, while the North and Central Zone regions do not receive funding to improve MNCH or malaria services. FP/reproductive health (RH) services, nutrition, gender- and youth-focused service delivery, and systems strengthening, and capacity-building are found across zones, but Ruvuma only has malaria interventions. 10 Table 2: Boresha Afya Program Areas by Region Zone/ Prime IP Sub￾Contractor Region HIV TB FP & RH MNCH*** Malaria Nutrition Gender/ GBV* & Youth HSS** Lake and Western￾Jhpiego PATH and EH Mwanza . . X CH X X X X Kagera . . X MNCH X X X X Mara . . X MNCH X X X X Shinyanga . . X CH X X X X Geita . . X MNCH X X X X Kigoma . . X CH X X X X Simiyu . . X CH X X X X Zanzibar . . X MNH X . . X North and Central￾EGPAF EH Arusha X X X . . X X X Dodoma X X X . . X X X Kilimanjaro X X X . .. X X X Manyara X X X . . X X X Singida X X X . . X X X Tabora X X X . . X X X Southern￾Deloitte EH, FHI360, MDH Iringa X X X MNCH . X X X Njombe X X X . . X X X Morogoro X X X . X X X X Lindi X X X . X X X X Mtwara X X X . X X X X Ruvuma . . . . X . . . *GBV = gender-based violence, **HSS = health systems strengthening***MNCH = maternal newborn and child health 2.2 CHSD RESULTS FRAMEWORK The UBA theory of change states the following: “If Tanzania empowers its women and youth, sustains inclusive broad-based growth, and makes governance more effective, its socioeconomic transformation toward middle income status by 2025 will be significantly advanced.”12 Accordingly, UBA has been designed to achieve the following two main results: 1) improved enabling environment for health service provision, and 2) improved availability of quality, integrated health services at the facility level. Building on USAID/Tanzania’s Country Development Cooperation Strategy (CDCS), UBA’s development hypothesis is that Tanzania women and youth will be empowered if they use quality health services. The CHSD Project Appraisal Document (PAD), which informed UBA’s activities, supports the Mission’s Development Objective (DO) 1: Tanzania women and youth empowered and Intermediate Result (IR) 1.2: Health status improved. Based on the hypothesis that FP decisions are critical to improving the economic status of families and communities, the CHSD PAD further supports DO 2: Inclusive broad-based economic growth sustained and IR 2.4: Unmet need for FP reduced. Finally, through its support for systems strengthening and work with key GOT entities, the CHSD PAD also contributes to DO 3: Effective democratic governance improved and IR 3.2: Government delivery of services improved. The UBA theory of change has evolved and become a vehicle to implement the PEPFAR program in Tanzania to achieve the 90-90-90 goals. While the CHSD results remain the main outcomes 12 CHSD NOFO, April 2016. 11 through an IHC delivery model, the target population is largely focused on achieving the PEPFAR outcomes and serving people living with HIV (PLHIV) in the North and Central and Southern Zones. In the Lake and Western Zone, the IHC model is focused on meeting the needs of women and children through MNCH/FP/RH, while meeting the needs of HIV/AIDS patients in collaboration with CDC. Figure 1: UBA Coverage Areas UBA has entered Year 3 of its five-year agreement with USAID Tanzania. IP activities include working horizontally towards integrated health care (IHC) at the facility and community level and vertically by supporting the management capacity of Council Health Management Teams (CHMTs), Regional Health Management Teams (RHMTs), and national GOT entities, e.g., Ministry of Health (MOH) and PO-RALG. They also work in close collaboration with a number of United States Government (USG) and non-USG supported programs as well as non-government and civil society organizations (NGOs, CSOs) at the local, district, regional, national, and international levels. Figure 1 above shows the Tanzania geospatial areas where each of the IPs and their consortia are active. The figure also indicates the districts selected for the MTE. 2.2.1 Integrated Approach to Health Programming The CHSD design conceived of UBA as an integrated approach to health service delivery. In this programmatic context, it has a two-fold definition. At its first level, it refers to collaboration and coordination between zones, consortium members within zones, and GOT structures and 12 facilities. At a second level, it refers to integrated health service delivery where services in one area can create linkages to services in other areas at a facility. This also includes cross-cutting or complimentary services (such as nutrition) to improve different health outcomes. UBA’s design aimed to improve linkages across the full continuum of care (COC) with a special focus on adolescents, women, children, and infants. Improved services for men were included in the third year. Integration included expanded linkages between health units within a facility, improved referral procedures between facilities and from communities to facilities to increase access to services, responsiveness of service delivery, and quality of care. This required increased outreach to communities from facilities and employing social and behavior change communication (SBCC) and sensitization approaches in communities in partnership with CSOs and other USAID￾supported programs (e.g., Tulonge Afya). Improved quality of care included friendlier service provision and a “client centered” approach to service delivery to all clients, especially women and youth. Intensive interventions were carried out to improve “horizontal” integration within and between facilities, and from communities to facilities through partnerships and expanded use of community health workers (CHWs). Vertical integration was addressed through TA and support through all tiers of the GOT health system. In this context, the President’s Emergency Plan for AIDS Relief (PEPFAR) requires a unique IHC service package to test, track, and retain PLHIV. In addition to using anti-retroviral (ARV) medication, PLHIV receive counseling, FP/RH services, and TB, sexually transmitted infection (STI), and cervical cancer screening. Index testing, screenings, and contact tracing of PLHIV’s sexual contacts requires reliable referrals, documentation, and client follow-up, which requires the involvement and engagement of communities. 13 3.0 METHODOLOGY 3.1 APPROACH The evaluation utilized a participatory, mixed-methods approach that included collecting and analyzing quantitative and qualitative data (both primary and secondary) as well as a desk review to address each EQ. The evaluation team (ET) included seven core team members, including a team leader with a background in HIV, TB, and FP/RH evaluation and research; a subject matter expert with a background in medicine and health systems, three Data for Development evaluation specialist staff, the Data for Development evaluation advisor and chief of party (COP), two local medical doctors (MDs) with experience evaluating health programs, and two NORC headquarters (HQ) staff to support the qualitative and quantitative analyses. The ET split into three groups that visited the three UBA zones (Southern, Lake and Western, North and Central) to collect data in the six regions where UBA is active. 3.2 DATA SOURCES 3.2.1 Document Review The ET collected documentation and reports from 120 sources, including USAID/Tanzania, the GOT, online information sources, research databases, and the three UBA IPs. These included quarterly and annual reports, IP work plans, and research studies on IHC and its effectiveness (see Annex 4). 3.2.2 Key Informant and Group Interviews The qualitative component focused on the context, management, and collaboration of UBA implementation, progress, successes, challenges, lessons learned, gender and youth issues, and sustainability. The three IPs based their interventions on different models unique to their zone, and it was necessary to gain an in-depth understanding of each model’s approach, activities, management dynamics, and effects in the field and the overall UBA project. Toward this end, the evaluation framework was used to develop nine targeted key informant interviews (KII) and group interview (GI) guides and these can be found in Annex 5. GIs were utilized when it was more illustrative to gain information from a range of senior staff in an organization rather than only its leadership. Examples include IP senior HQ and regional staff, CHMT and RHMT senior medical and technical staff, senior health unit officers with facility-in￾charge, CSO staff, DE HQ staff and regional advisors, and a GI at USAID Tanzania. The ET conducted a total of 117 KIIs and GIs with IP HQs and field staff, UBA stakeholders and partners, DE, GOT representatives at the national, regional and council levels, health facility supervisors and staff, CSOs, CDC and USAID Tanzania (Table 3). All interviews were recorded if the interviewee consented and notes were taken. If the interviewee didn’t provide consent, the team relied on notes. All recorded interviews were transcribed and translated from Swahili to English when necessary. All interviewees read or were read and signed a confidentiality consent form, and the resulting recordings, transcripts, and notes remained anonymous. 14 Table 3: Number of KIIs and GIs and Their Affiliations Within the UBA IP Zones Stakeholder National Dodoma Kilimanjaro Kagera Mara Morogoro Iringa Total USAID 5 . . . . . . 5 GOT 15 . . . . . . 15 IPs 4 1 1 1 1 1 1 10 DE 2 . . . . . . 2 National Stakeholders 7 . . . . . . 7 RMO/RHMT . 1 1 1 1 1 1 6 DMO/CHMT . 1 1 1 1 1 1 6 CSO . . . 1 1 1 1 4 Facility Supervisors . 2 3 3 3 3 3 17 Facility Staff 13 11 3 3 9 6 45 Total 33 18 17 10 10 16 13 117 3.2.3 Focus Group Discussions The ET conducted 34 focus group discussions (FGDs) at two of the three facilities visited in each district. These groups were composed of individuals who had used the facility and included: 1) women living with HIV or using MNCH services (North West Zone only), 2) youth between 18 and 25 years, 3) men who were involved in male engagement activities or accompanied their wives to services and 4) affiliated CHWs. CHWs affiliated with the examined facilities were assembled at the regional level for the FGDs. The purpose was to gain knowledge on the effects of IP activities among UBA targeted clients (women, youth, males) at the facility level and on outreach into communities (CSWs). Table 4: Number of FGDs Conducted, Affiliation, and UBA Zones North/Central Lake/West Southern Total Dodoma Kilimanjaro Kagera Mara Morogoro Iringa Youth 2 1 0 0 2 1 6 Women (PLHIV & MNCH) 1 2 2 2 2 2 11 Men, (Male Involvement) 0 1 2 2 2 2 9 CHW 1 1 2 2 1 1 8 Total 4 5 6 6 7 6 34 3.2.4 Primary Data: Facility Checklist and Client Exit Surveys The ET administered Facility Checklist and 10 client exit surveys at each health unit visited to capture service quality, for a total of 18 facility checklists and 180 exit surveys (see Annexes 5 and 6). The checklists used quantitative as well as open-ended questions to describe the facility profiles and the IHC approaches used in the three zones. Both surveys were administered using NORC Survey CTO-based software questionnaires on computer tablets. The Facility Checklist was administered to health workers in each of the health units that provided IHC services with support from UBA to capture service quality. The checklists were administered at each facility at a number of individual health units, e.g., CTC, TB, malaria, FP/RH, laboratory. 15 These units were also verbally interviewed to capture the dynamics of IHC, e.g., collaboration and cooperation, knowledge on referral systems as well as staff perceptions and satisfaction with UBA support. The Client Exit Survey was randomly administered to ten female clients at each examined facility as they exited a CTC or MNCH health unit. These were approximately 10 minutes in length and topics covered included subjects such as satisfaction with health services and whether clients were offered additional tests or services beyond the original purpose of their visit. 3.2.5 Secondary Data To analyze IP effectiveness, performance, and cost effectiveness, the ET requested performance and cost data for each of the three IPs from the Data for Accountability, Transparency, and Impact Monitoring (DATIM), the Implementing Partner Reporting System (IPRS), the District Health Information System 2 (DHIS2), and each IP’s own monitoring and evaluation (M&E) and financial data. The ET also requested cost data from the IPs. However, the IPRS lacked baseline data and DATIM lacked targets for some indicators. One IP did not provide financial data. Because the IPs’ accounting systems are different and not designed for cost-effectiveness comparisons, the ET was unable to obtain cost data by interventions, facilities, councils and regions, or sub￾grantees for all three zones. Only total cost information by stream of funding was made available. 3.3 SAMPLE SELECTION 3.3.1 Field Site Selection Two regions were purposively selected by USAID within each of the three UBA IP evaluation zones. Within each region, two health districts were then randomly selected. Initially, two health centers (HCs) in each district were then to be selected, but this was expanded to a district or regional hospital and two primary HCs to capture more data (see Table 5). Some of the randomly selected facilities were logistically unreachable and were thus resampled, and two districts were without hospitals. The latter were purposively replaced by Tarime district council (DC) and Hai DC. This represented an equal number of regions, districts, and sites for each IP zone. The choice of selecting one large hospital and two HCs was based on the inherent differences in the types of IHC services provided and client flow. Dispensaries were not included because their small size and low number of health workers limited an investigation of IHC. Table 5: Health Facilities and Hospitals Selected for Evaluation in the Three UBA Zones North/Central Lake/West Southern Region Dodoma Kilimanjaro Kagera Mara Morogoro Iringa District Dodoma MC* Hai DC Ngara DC Tarime DC Mvomero DC Iringa DC Facility 1 Dodoma General Hospital Hai District Hospital Nyamiaga Hospital Tarime District Hospital Turian Hospital Frelimo Hospital Facility 2 Makole Urban HC Kisiki HC Mabawe HC Nyangoto HC Melela HC Ipogolo HC Facility 3 Mirembe Hospital Massana HC Bukiriro HC Muriba HC Mvomero HC Ngome HC *MC = Municipal Council 16 3.4 DATA ANALYSIS The evaluation utilized a convergent parallel mixed-methods approach for analysis. Each data collection method was carried out in its entirety and analyzed separately. Qualitative data were analyzed using grounded coding techniques, and quantitative data were analyzed using descriptive statistics. The results were then triangulated according to each evaluation question and sub￾question. The results from the different methods were compared, contrasted, and validated. ` 3.4.1 Quantitative Data Analysis Data from the facility checklist and client exit survey were collected from Android-based tablets using NORC’s Survey CTO software. The data were uploaded daily to a server for cloud-based archiving and downloaded to an Excel database for cleaning, validation, and analysis. If raw secondary data was made available, it was collected from the IPs, USAID, and GOT and entered into an Excel database for analysis. The ET computed simple frequencies and descriptive analysis of the data with stratification (where appropriate) by IPs, zones, regions, and councils. 3.4.2 Qualitative Data Analysis The ET analyzed KII, GI, and FGD transcripts and notes using content and thematic analysis techniques in which text was coded according to key themes of interest that aligned with the EQs and sub-EQs. Key themes and their definitions were documented in a code book that was developed collaboratively by ET members. To apply codes, the ET used Dedoose, a qualitative analysis software program. 3.5 ETHICAL CONSIDERATIONS AND HUMAN SUBJECT PROTECTION All persons participating in the evaluation participated voluntarily, were informed of the confidentiality of the interview or survey, and signed a written or electronic consent form prior to data collection. The evaluation protocol was submitted to and approved by the National Institute for Medical Research (NIMR) of Tanzania and the NORC at the University of Chicago Institutional Review Board (consent forms are found in Annex 5). 3.6 LIMITATIONS The evaluation was subject to several limitations. First, it was not possible to conduct the randomized site selection originally planned due to the inaccessibility of some facility sites and the discovery that some districts lacked a district or regional hospital. Second, the number of site visits, interviewees, and FGDs exceeded the number originally planned due to new input from USAID, and the GOT. In response, the ET selected new facilities at the HC level in place of dispensaries to increase the number of observable IP activities in a given facility for administration of the quantitative tools. Third, limited secondary data were made available to the ET by the IPs despite numerous requests by the ET and USAID, while DHIS2 data were not made fully available to the team. Moreover, obtained data often lacked baseline information to track performance over time, and datasets had many unfilled data cells. Only partial financial data were received from IPs for the cost-effectiveness component. This limited the analysis on costs versus outcomes and the ET’s ability to address specific EQs/sub-questions for the evaluation on IP performance and cost-effectiveness (EQ 2). 17 4.0 FINDINGS 4.1 EVALUATION QUESTION 1: TO WHAT EXTENT HAS THE PROJECT BEEN EFFECTIVELY MANAGED AND OVERSEEN? This section is focused on findings regarding organizational capacity, external management, and collaboration and cooperation with other organizations. It also addresses sub-question 1a: Is the organization/management structure of the three prime contractors sufficient in addressing all program areas of integration? 4.1.1 Organizational Capacity The three prime contractors, or IPs, based their original organizational structures on their own specific approaches to meeting established targets and developing an integrated approach to health care. The contractors drew upon their own corporate specialties, lessons learned through the USAID Tunajali and Linking Initiatives for the Elimination of Pediatric HIV (LIFE) projects, and knowledge of the geographical zones they were responsible for including the burden of population health conditions and transmittable diseases. They organized their individual management structures into consortia where expertise from sub-contracted organizations was distributed across health activity areas. Collaboration within consortia is embedded into the structure of the project. Sub awardees are often co allocated in the same office, plan, implement and report on activities through the prime. The consortia are functioning well after an initial adjustment period, according to interview respondents. In KIIs with USAID and IPs, respondents reported that contractors had overcome initial challenges in managing their consortia and were now operating more efficiently (seven of 11 KIIs with IPs, two of five KIIs with USAID). Initial challenges included unclear roles and conflicting human resources (HR)/administrative policies between consortia partners that led to inconsistencies in employee expectations, synchronization of work, and procedures. Respondents noted that these differences did not affect project performance and had been largely resolved following Year 1 of implementation. In one KII, a USAID informant noted the Agency’s role in improving coordination within zones: “USAID helped by facilitating [memoranda of understanding] MOUs that defined the roles of each partner. At first, the subs were on their own and now they work closely together.” None of the interviews with USAID informants raised any concerns about the UBA IPs’ organizational capacity. 4.1.2 External Management and Collaboration Overall, interview respondents reported positive experiences with UBA IPs. When asked about the quality of their interactions, most respondents were very positive, noting that UBA had been collaborative, responsive, and flexible to their needs.13 “Overall, our interaction with UBA and its implementers has been outstanding with no major challenges.” (Male, national GOT KII) 13 Tulonge Afya did not report supporting UBA referrals, linkages, and retention of HIV positives, despite this role being listed in the Collaborative Framework. 18 “To me, the program is just perfect because the sponsors are part of us, we do all of the things together, we are much involved and as an administrator am very comfortable to work with USAID UBA because of the way they involve people.” (Female, regional/district GOT KII, Kilimanjaro) There were no notable differences in these findings according to gender. Positive relationships with UBA IPs were reported by regional/district GOT in equal numbers across the three zones. However, at the facility level, more KIIs reporting positive relationships with UBA IPs came from the Lake/Western (six KIIs) and North/Central Zones (seven KIIs). Inter-Zonal UBA Collaboration Challenges in inter-zonal coordination were evident between UBA IPs. Although USAID originally intended for all three IPs to coordinate with each other through joint program planning meetings (JPPMs) (two of five KIIs with USAID, one of 17 KIIs with national GOT), collaboration emerged mainly between PEPFAR partners Deloitte and EGPAF who in August 2019 began to collaborate with other facility and community IPs through a formal collaboration framework. Two USAID KIIs offered reasons for this shift: Lake/Western Zone activities were said to be too different from the other two IPs and management of all three at large meetings was unwieldy.  Jhpiego, Deloitte, and EGPAF attended the GOT technical working group (TWG) meetings (five of 11 KIIs with IPs, four KIIs with National GOT), which is useful for overall coordination with the national government. Deloitte and EGPAF attended additional meetings together, including national partner meetings with the USG community and facility IPs (one of seven KIIs with national stakeholders). EGPAF also uses the same Community Health Platform created by Deloitte (one of 11 KIIs with IPs). Stakeholders at USAID and National Government would like to see greater alignment, collaboration, and engagement in implementation, coordination and planning between the IPs (two of five KIIs with USAID, one of 17 KIIs with national GOT). “At the national level, IPs don’t agree on who does what. For instance, they were asked to develop a Community Resource System and they ended up competing with each other for the product. It appears they are competing on who gets the credit.” (Female, USAID KII) “I haven’t seen any initiatives where the bidders sit together and speak…I haven’t heard that!! I haven’t seen any forum of knowledge sharing among the bidders!! What I see is vertical programs; here is [either] Jhpiego, EGPAF or Deloitte…All on their own!” (Male, National GOT KII) Collaboration with Other USAID Partners The original Notice of Funding Opportunity (NOFO) stated that UBA IPs were “expected to work in close partnership with the United States (U.S.) Centers for Disease Control (CDC), U.S. Department of Defense (DoD), GOT, other USAID and USG implementing partners, non-governmental organizations (NGOs), including faith-based organizations, the private sector and various other local Tanzanian partners as present in the implementation regions.” This collaboration is defined as: “(1) geographic coordination 19 of activities, where overlap exists; (2) harmonized delivery of services to populations reached by multiple USAID programs; and (3) coordination of CHWs and other community-based workers.”14 The ET interviewed informants from USAID Kizazi Kipya, Sauti, Tulonge Afya, Public Sector Systems Strengthening (PS3), and NACOPHA. All interviewees reported that they were currently collaborating with UBA. The type of collaboration most often reported was in support of referrals, linkages, and retention of HIV positive people (two of four KIIs with CSOs, six of 11 KIIs with IPs, two of seven KIIs with national stakeholders, and one of five KIIs with USAID). USAID Sauti, Kizazi Kipya, and NACOPHA all reported linking clients to CTC units in UBA facilities and conducting follow-ups with clients from UBA facilities to ensure adherence to treatment.15 In July 2018, PEPFAR issued guidance emphasizing the need for more collaboration, this led to a formal revised framework for PEPFAR partners which was agreed upon in August 2019.16 Collaboration with other USG IPs was particularly evident in the area of SBCC. Respondents noted that UBA was collaborating with Tulonge Afya in all three zones (six of 11 KIIs with IPs, three of five KIIs with USAID, one of seven KIIs with national stakeholders), that Tulonge has attended JPPMs for UBA, and the COPs have held coordination meetings. UBA IPs utilize Tulonge Afya SBCC printed materials and distributed Tulonge’s radio, TV, and media campaigns to support their facility and community activities. “I am impressed with the collaboration I have seen in my zone between IPs. For instance, Tulonge Afya works hand-in-hand with UBA on SBCC in communities. This is useful collaboration because of the clarity in their interaction. UBA listens to their input and puts it directly into action. For instance, how and what specific behaviors and behavior change to focus on.” (Female, USAID KII) However, some USG IPs reported ongoing challenges to collaboration with UBA. In three of seven KIIs with national stakeholders, respondents reported that referrals between IPs are occurring, but the lack of M&E systems to track referrals with accurate denotation of who referred the client causes complications for their reporting. Interviewees noted that some providers double counted clients. Others mentioned that there was a sense of competition between PEPFAR partners that should be working together. “I think the problem is within the IPs…It is very important for USAID to keep on tracking systems to ensure that partners adhere to the [collaborative] framework and design and that every partner is contributing to the level that was supposed.” (Male, national stakeholder KII) “… if I really want to measure the effect of the impact for the contribution of a community partners to facility work and facility to community, I think you need to have a built in M&E system that captures these transfers or referrals.” (Male, national stakeholder KII) 14 Adapted definition in the evaluation work plan and design derived from common themes among unique definitions among IPs. Thus IPs do not have a single definition of integration either. 15 Three facilities were skipped in the Lake Zone. 16 Note: pp = percentage points, used for differences between two rates or proportions to describe the change between two numbers, we used percentage changes. For example, the difference between 10 percent and 15 percent is 5 pp, while the change between 100 and 150 is 50 percent. 20 In one KII, a national stakeholder suggested that key process indicators could help to incentivize participation and collaboration between USG IPs: “I think that even though [USAID] thought very carefully about the design of the projects, certain things that they can do were not in place. When you have projects that work together, and are designed to work together, then it’s good to have some key performance indicators that are in process. Built in. Like joint meetings.” (Female, national stakeholder KII) There were no notable differences in these findings according to gender, nor any notable differences in according to zone/region. Collaboration with Tanzania National Government In 11 of 17 KIIs with national GOT stakeholders, respondents shared positive opinions about their collaboration with UBA. The most common positive experiences included joint supportive supervision (five KIIs), TWG meetings (four KIIs), and joint planning sessions (three KIIs). In two KIIs, national GOT stakeholders expressed gratitude to UBA for supporting ad hoc GOT meetings that were not in UBA work plans. They are also collaborating with PO-RALG and the MOH in developing comprehensive council health plans and are engaging GOT in JPPM. However, in 12 of 17 KIIs with national GOT stakeholders respondents shared challenges they have had in collaborating with UBA. A perceived lack of transparency on the part of UBA, especially regarding budgets, was the most cited (five KIIs)17. In two KIIs, national GOT stakeholders remarked that they had little to no contact with UBA IPs despite their own desire to collaborate. The three IPs engage the national program managers for respective programs (FP, child health, HIV, etc.) but do not have a mechanism for regular coordination with PO-RALG, 18 the Director of Policy and Planning, Director of Preventive Health Services at the MOH, etc. Policy, Planning, and Financing decisions are made at this level and the GOT would like to have a centralized UBA support to help the GOT address the MOH and PO-RALG challenges of managing vertical programs and services nationwide. In short, the IPs need to be aligned with the GOT and not, as perceived currently, the other way around. In four of five KIIs with USAID, respondents reported their own challenges faced in collaborating with national GOT. These included a perceived resistance to change, lack of understanding of USAID procedures/policies, differing priorities between MOH and PO-RALG, and a lack of engagement at the highest government levels. All challenges related to collaboration with UBA were reported by national GOT sources in Dodoma (12 KIIs) and not in Zanzibar. 17During peer reviews IPs indicated that they have shared budget and planning information with Regional and Council Level authorities, but this may not have been disseminated at the National Level 18 One IP state that they have meeting annually with PORALG 21 4.2 EVALUATION QUESTION 2 A-G 4.2.a How Effective and Timely Is Each IP in Making Necessary and Adaptive Changes to Improve Performance? 4.2a.1 Overview UBA and the IPs’ consortia were obligated to meet individual targets in specified health areas, e.g., DATIM indicators and also contribute significantly towards the integration of Tanzanian health care in what are, otherwise, siloed health areas, e.g., HIV, RMNCH, and malaria. This required “horizontal integration” at the facility and community level and “vertical integration” from the facility to the national government through CHMTs and RHMTs. There was no detailed plan as to what IHC would look like in Tanzania nor was there a template agreed to by all stakeholders. The IPs needed to be highly adaptive, innovative and begin with improving client referral and tracking systems while building upon integration and referral procedures by PEPFAR. To accomplish this, it was necessary to work in partnership with all layers of the Tanzanian government’s health system including supportive systems such as commodity supply chains, the Health Management Information Service (HMIS), and strategic information (M&E). They needed to cooperate and collaborate with non-USAID programs and USAID supported programs such as Tulonge Afya, PS3, Sauti, and Kizazi Kipya. They needed to educate, mobilize, and train both frontline facilities and communities to improve integration between the two. Embedded within the IHC approach to health care was the need to bring about changes in attitudes among health workers and in communities towards inclusive and improved services to women and youth. The core value in IHC client is “people centered care” and access to and satisfaction with services became benchmarks. IPs also needed to adapt to major health policy and financing changes within the GOT, (e.g., Direct Facility Funding [DFF]) as well as from USAID and PEPFAR. The latter required changes in award administration and measurement and new practices and procedures from the IP down to the service delivery point. 4.2a.2 Timely adaptation In the KIIs with the UBA AORs, they stated overall satisfaction with the performance of the three UBA IPs and had acceptable performance against their targets and, if facing challenges, have adapted effectively and timely to change. In a changing budget climate the UBA IPs have adapted their methodologies and activities through collaboration with other USAID IPs. When asked how effective and timely each IP has been in making changes in response to local needs, new donor strategies, and new government policies to improve performance, respondents remarked positively to the integrated service delivery approach being applied (seven of 11 KIIs with IPs, three of 11 KIIs with regional/district governments). “There is a difference between how we used to implement before and now. The main difference is the approach. It has been unique…the integration approaches. The structuring of the staff and other operational activities...has been different.” (IP regional GI, Kilimanjaro) 22 “I think we are a learning organization and we are allowed to say that, ‘this was our design and we stick to the design.’ Over the years we have made quite a few changes. Some were minor, some were more drastic…” (IP KII, national) IPs adopted changes instituted by USAID and PEPFAR and have successfully cascaded them to the government partners, CSOs, and facilities they support; the three IPs have shown considerable flexibility and adaptability in coordinating these changes. However, the timeliness of the introduction of new procedures at some HFs has differed in practice or has sometimes run behind the desired schedules of PEPFAR and other donors. “Yes, we were disconnected between what the government is saying and what PEPFAR is saying. So, we came back and sat down and looked at where are the gaps and we had a list of issues, areas where we need to collaborate with the government. We have like 16 policy issues or directives that we are looking at from the government national level that can facilitate our implementation at regional, district to the site level.” (National stakeholder KII, national) 4.2a.3 Use of data and information and communications technology (ICT) Increased access and innovation using ICT and achieving results with limited resources requires adaptation and, through USAID, all three IPs have heavily invested in innovative systems and applications to streamline performance (three of 11 KIIs with IPs). Data-driven management has supplemented the projects’ ability to ensure quality and implementation fidelity through their sub-recipients. One of the adaptive changes was the shift from quarterly to monthly and then to weekly data reporting. EGPAF and Deloitte have largely undertaken these changes into DATIM. For example, they have been able to collect the available data, analyze them, and use them in routinely tracking performance against targets on a weekly, monthly, and quarterly basis to assess achievements. Nonetheless, the ET had considerable difficulty in obtaining ICT-based performance and financial information from the IPs, which ended up being incomplete. “…we are doing what we call semi-annual program review, we call all the stakeholders, all members from the Region level but also all Facilities representatives. We do meetings with National Consultants every year, to see if there is any target from the Government, [World Health Organization] WHO, USAID and whatever and how we can actually cooperate in implementing on the target.” (regional IP GI, Lake and Western Zone). An important approach used by all the three IPs in adapting to improve performance is to conduct data review meetings, which aim at sharing project performance to improve data management and data quality as well as to address bottlenecks in implementation across technical areas. 19 A key deliverable of these meetings was the development of district-level action plans to address identified bottlenecks across technical areas. The IP teams and RHMT/CHMTs follow up on the implementation of agreed plans before the next review (four of 11 KIIs with IPs, one of 11 KII with regional/district governments). 19 DE started presenting findings of their operational research during the time of this evaluation. There is no indication how the three IPs had adapted to these findings and recommendations at the time of this evaluation. 23 “…And it is easy to ask someone and get the real causes of good performance or bad performance on the service providers. So, we do those meetings to ensure that we have the real cause of the bad performance so that we can come up with a plan to make sure that we rectify, or we keep up the good performances.” (IP GI, national) Due to demands and local contexts, all UBA IPs were able to introduce different applications and digital tools to support service delivery. USAID supported UBA in the Lake/Western Zone to add tools such as the CommCare Platform for accessible project data collection, reporting, and analytics and system support for the national EDS-MSDQI Mobile Application. JHPIEGO also supported Zanzibar Malaria Elimination Program (ZAMEP) in conducting supervision visits for malaria case management; and the Closed User Group (CUG) Mobile communication platform that enables health care workers (HCWs) to make phone calls free of charge to strengthen the system of referrals between HFs. 4.2a.4 Human Resources for Health (HRH) The end-of-project evaluation of preceding projects identified HRH as an important building block needing strengthening for effective IHC. All three UBA IPs seem to agree that intensive classroom training has not proven to be effective in building the capacity of HCWs based on research literature and their own empirical practices in previous projects20 and hence adapted to on-the￾job training, coaching, and mentorship at the site level, which appears effective especially when combined with supportive supervision for reinforcing health practices in facilities (five of 11 KIIs with IPs). A variety of stakeholders reported their concerns on the shortage of trained HCWs for an IHC approach. National RHR shortages are compounded by staff rotations and new hires with limited skills. The shortage of HRH under the IHC has caused significantly increased client waiting times in some high-volume facilities despite being sure of getting all the services they need in fewer visits (the one-stop shop model). The exit interviews of 180 patients found that 77 percent traveled less than an hour to get to the facility and 71 percent waited less than an hour to be seen at the facility, while 12 percent reported having waited two hours or more. Through USAID support, UBA IPs have responded to the HRH shortage by helping RHMT/CHMTs in recruiting staff such as doctors, nurses, laboratory scientists, data officers, data clerks, and M&E staff to meet weekly performance reports as well as hundreds of CHWs for facilities and community-based health care delivery. This evaluation did not assess the magnitude of the HRH gaps in the three zones other productivity of the current workforce, CHWs have been stationed at facilities to increase linkages, interns have been used to bridge the HRH shortage, and the IP has initiated Nurse-Initiated Management of Antiretroviral Therapy (ART) (NIMAT) (two of 11 KIIs with regional/district governments, six of 11 KIIs with IPs, and one of one KII with DE). Nonetheless, the shortage and efficient use of HCWs remains a serious and acute issue for Tanzania’s health care coverage, access, and effectiveness. The best use of CHWs as either government or CSO caregivers and community mobilizers is being closely examined and rolled out by GOT and stakeholders. 20 Issues for Health Systems Strengthening. Annals of Global Health. 2019; 85(1): 113, 1-15. DOI: https://doi.org/10.5334/aogh.2514 24 “So, as a program, we assisted the council in increasing human resources for health. We have interns employed at high volume sites to support routine services at the facility. So, we identified sites with critical challenges on HRH and with a high volume of clients and supported them with interns.” (regional IP GI, Dodoma) 4.2b How effectively has UBA integrated the targeted health areas into a functioning IHC system at the district level and nationwide? Integration efforts are in progress but vary by zone and are not yet completed. The ET has identified several findings that may be barriers for successful integration. First, the three IPs are part of one program and have the same results framework but have different approaches and strategies to integrate health services and achieve the IRs. This could be a sustainability factor for GOT to be able to carry forward a cohesive set of IHC approaches, interventions, and tools. Second, the integrated approach of the IPs is based on a twofold definition which is not the same as the GOT’s: “In its first sense it refers to integrated programming in terms of the collaboration and coordination between zones, consortium members within zones, GOT structures and facilities. Integration at a second level relates to integrated health service delivery where assistance at the service delivery point for testing or treatment in one area can create linkages to other areas of services at the facility. This also includes cross-cutting or complementary services to improve different health outcomes.”21 Also, GOT would encourage greater alignment and transparency and to see evidence of what works within each UBA IP to “maximize coverage and health outcomes” and learn how to “optimize the use of scarce resources,” as stated in the National Operational Guidelines for Integration. The GOT could better align the more expanded set of HIV indicators under PEPFAR and the 90-90-90 goals; conversely the mission could align its indicators under maternal and child health to achieve a more aligned system for integrated health under National Policy Guidelines. Coverage The number of facilities covered by IHC is limited to selected regions and councils. In the north￾central zone, the RHMT reported that the UBA approach is effective but coverage is not what they would like, especially for TB and pediatric CTC. “We have a lot of health facilities and we have to reach them. The least effective issues, I was thinking that the 66 facilities were not enough for the coverage of the TB services because we have 423 health facilities but also, we have 164 health facilities which provide TB services and from these 164 facilities, the UBA only covers 66 facilities. So, I wish they could have been scaled up to get big coverage to offer the TB services.” (RHMT/ RMO GI, North and Central Zone) “The main focus has been on the adult CTC and we have like forgotten (sic) the pediatric CTC where we are lagging behind.” (GI RHMT/ RMO North and Central Zone) 21 Based on three hospitals only. 25 The integration of the various vertical programs is not comprehensive and has not yet covered all the target regions. Table 6 illustrates the different interventions are used by the IPs to deliver IHC in their zones. The IPs have not yet used this difference in coverage and activities to compare the effectiveness of their own and various other IHC approaches. Operational research to ascertain the most effective interventions has not been conducted yet. Table 6: Summary of each IP’s Approaches and Interventions for delivery of quality IHC Intervention Jhpiego – Lake and Western Zone EGPAF – North and Central Zone Deloitte – Southern Zone Approach to capacity building and system strengthening Learning and Performance HSS model to address the six building blocks for continuous learning among HCWs for Quality Improvement Optimize EGPAF’s district-focused approach to strengthen the systems and institutions necessary to support integrated service delivery. Tailored approach to each geographic area by delivering health services in Scale-Up Saturation Councils Client centered and gender/youth based approaches Client-centered and gender-sensitive health service delivery Promote integrated, client-centered approaches to reach vulnerable populations including children and adolescents. Integrate gender into all aspects of programming to ensure equitable access to health services for women and girls. Optimize the COC, so that the linkages, referrals, and systems connecting HIV, TB, malaria, FP, maternal and child health (MCH), and gender-based violence (GBV) services within and between HFs and communities are seamless. Implement a strategy for women, gender and youth in alignment with the National Adolescent Health and Development Strategy (2018-2022), and the National Accelerated Action & Investment Agenda for Adolescent Health and Wellbeing (NAAIA) Strengthening continuum of care Smart integration to maximize efficiencies by expanding access to an array of high-quality services for every client at each point of care, reducing the missed opportunities Implement proven and enhanced community approaches to improve the COC. Implementing differentiated care models at the sites and the community to address barriers to care Use of monitoring data and information systems Use monitoring data for donor accountability and internal quality improvement (QI). Limited data and information sharing across IPs, consortia, or GOT. Use monitoring data for donor accountability and internal quality improvement (QI). Limited data and information sharing across IPs, consortia, or GOT. Use monitoring data, for donor accountability and internal quality improvement (QI). Limited data and information sharing across IPs, consortia or GOT 26 Intervention Jhpiego – Lake and Western Zone EGPAF – North and Central Zone Deloitte – Southern Zone Digital health interventions Use ICT approaches to digital health interventions including client follow up and continuing care. OpenSRP to strengthen the linkage between Health Facilities, CHWs and the clients; Blended learning to improve performance of health care workers Use ICT approaches to digital health interventions including client follow up and continuing care. Use ICT approaches to digital health interventions including client follow up and continuing care. Social and behavior change communication linkages Engage Community leaders through SBCC to channel messages in child and maternal health and family planning. Sensitization materials are provided at facilities for family planning. Sensitization materials are provided at facilities for family planning. Continue working closely with USAID Tulonge Afya Program for better implementation and measuring change Sensitization materials are provided at facilities for HIV, TB and reproductive health. The integration process has been uneven nationwide, and some areas are still in siloes. The qualitative interviews reveal mostly satisfaction with UBA concerning its effectiveness in integrating the targeted health areas into an IHC system. However, there is no consensus on what an IHC system is. To most it appears to be an improved referral system and provision of more than a single service at a facility’s point of care. There have been challenges with referrals, linkages and client tracking which have been previously examined in a PEPFAR assessment in Tanzania.22 23 Specific challenges are mentioned but the transcripts often reveal greater praise for the steps UBAs has taken up to this point by program managers and regional and council staff. At the national level, some GOT authorities perceive a lack of coordination and cooperation and the need for a formal UBA coordination mechanism. “Let’s work together and measure impact. We cannot afford to have siloes. So far, I do not see the impact on the quality of care and of data, or coverage in the zones supported by UBA. Just training and meetings…There is a need to get engaged and write and an implementation MOU with all parties involved so we are clear what will be achieved and sustained.” (KII – Ministry of Health, Community Development, Gender, Elderly, and Children [MOHCDGEC]) “In my opinion, the key component or process necessary for the successful integration of health care in Tanzania should be comprehensive understanding of the scope of work by both parties UBA and MOHCDGEC.” (KII – MOHCDGEC) 22 WHO Europe. Integrated Health Care Models: An Overview. WHO, Copenhagen 2016. http://www.euro.who.int/en/health￾topics/Health-systems/health-services-delivery/publications/2016/integrated-care-models-an-overview-2016 23 I-Tech Tanzania. Assessment of the National Patient Appointment, Tracking, Referral and Linkage Systems in Health Facilities and Communities in Selected Districts in Tanzania. Dar Es Salaam 2016. https://www.go2itech.org/wp￾content/uploads/2017/07/TZ-Assessment-report-_November-29_2016_Final.pdf 27 Positive feedback on UBA in the interviews often focuses on the IPs’ productive collaboration and cooperation with the GOT at the regional and council levels, stakeholders, partners, and facilities. Some national GOT authorities have noted improvements. “It has added changes by 80 percent and above. Some days back supervision or monitoring was lacking focus but the way we are collaborating now is amazing. We go sit with CHMTs, we go to the facility together we identify challenges they face; we plan together and make a timetable to ensure that action plans are developed so that we can find a solution for the existing problems.” (KII, national GOT, Dodoma) Overall, UBA IPs have created an environment where many players feel that “they are in this together.” The national government is split in its attitudes and this appears to be due to insufficient communication, the lack of an overall effective set of interventions, and a vacant UBA umbrella entity that coordinates and monitors implementation. Each IP has a different approach and set of interventions, which adds to the complexity of UBA integration at the national level. (See Table 6 above for a summary of interventions used to implement IHC.) Other challenges for UBA include tackling the health needs of youth and the persistent GBV problems. UBA has taken the first step by making GBV visible through better reporting. The next step is to strengthen the health system to deliver timely IHC services to GBV victims. “Another challenge is on GBV, the behavior of the people in Iringa. The GBV is big, but they don’t expose the GBV that they get even in facilities GBV services are low, so that is also a challenge. We have 55 facilities for GBV out of 260. So, we are only a quarter of the facilities that get GBV which is a minimal amount to achieve the great results that we want.” (GI with RTMG, Southern Zone) “I can speak about one area of youth-friendly services. We can say this has not yet achieved because as we know, youth need privacy, and we don’t have the supportive infrastructure to enable youth to get friendly services with maximum privacy. For now, what they’ve done is the provision of rooms because of the few infrastructures we have. But these rooms are not performing one activity. Therefore, the problem has been solved but not wholly.” (GI with CHMT, Southern Zone, Iringa) Lack of IHC Indicators There is no list of indicators common to the three IPs that would reflect how well their approaches are addressing IHC. The technical team at USAID Tanzania have their indicators specific to their health programs. On the other hand, the health systems team and HF and community care teams have not yet established IHC output and outcome indicators. “IHC has been a challenge because it’s difficult to say what exactly has been done. People and organizations are continuing to work in silos. Siloed thinking exists down to the sub-partner level. For instance, [an IP] is still using national tools and these do not have an IHC focus…Testing has increased in [outpatient department] OPD and ANC because they are a priority. The labs are doing well and an increasingly better job.” (KII notes with USAID Agreement Officer Representative) The list of IHC indicators included in the MTE work plan had to be adapted to the data available. These data were primarily indicators related to specific vertical programs such as malaria, HIV, or FP. The ET observed that the IPRS lacks client outcomes by facility, such as the numbers of women that completed ANC and HIV procedures, TB patients lost to follow up (LTFU) and the 28 numbers of patients that have achieved viral suppression by facility, council, and region24. This means that the national authorities and IPs cannot determine which facilities, councils, and regions are better performing or which need increased or different support. 4.2c How has the implementation of UBA’s IHC efforts been influenced by other IHC initiatives and funding sources? While there are other multilateral and bilateral organizations that work in the health domain (such as the Global Fund, Joint United Nations Program on HIV/AIDS [UNAIDS]) it is less clear how if at all UBA has been influenced or has coordinated regularly with these entities. While there were limited interviews with international donors, there was some criticism about the limited extent of coordination with their programs. The main influence among all funders, particularly in the North and Central and Southern Zones, has been the USG and PEPFAR. Global Health Supply Chain has helped UBA to integrate supply chain activities and the President’s Malaria Initiative (PMI) has also been an influence on integration of malaria into MNCH in Southern and Lake/Western zones. However, the HIV component through PEPFAR is the largest funding source in UBA and drives the integrated model because other funding for other thematic areas is targeted to specific activity areas and not for IHC. In the North/Central Zone and Southern Zone, HIV/AIDS has been allocated a significant amount of funding of 83 percent and 72 percent of the total amount, respectively. One of the biggest influences on UBA has been the other PEPFAR-funded programs in the collaboration framework with whom it works for community referral system strengthening. Various respondents reported improved referral systems as a result. When asked whether referral systems have improved with the support from UBA, stakeholders reported having better referral systems than before UBA (20 of 35 KIIs with Facility staff, seven of eight FGDs with CHWs and six of 26 with the clients’ FGDs). National Operational Guidelines The main guiding influence at the national level is the National Operational Guidelines for integration of the MOHCDGEC (2012). The review of IP work plans and their interviews showed that the interpretation of these guidelines and the interventions to implement them vary by IP. The ET did not find in the document review the basis for these differences. The MOHCDGEC does not have a mechanism for monitoring or enforcing the application of these guidelines. Policy and Planning Coordination with MOHCDGEC and PO-RALG National-level authorities reported that the three IPs have not had a coordinated role in providing strategic high-level TA at the national level or helped inform health services at lower levels on the integration of vertical health programs. The IPs are perceived to have been very busy supporting facilities to meet service delivery targets through monthly, and currently weekly, facility visits. All three IPs were reported to participate in TWGs and have contributed to the revision and development of technical guidelines. Jhpiego was reported to be more active in representing health program areas such as FP and MNCH with the national government. The 24 In peer review IPs stated that these indicators may be available on DHIS2 29 KIIs reported that the IPs have not worked with the Directors of Policy and Planning, Preventive Services, or M&E, but have worked with the technical program area managers. The need for formal effective coordination is felt at all levels of government, particularly at the national level. Otherwise, it is difficult for GOT authorities to know who does what where, particularly while preparing comprehensive annual plans at the regional and council levels. At the regional level, the issue of accountability and attribution and what works was raised: “…for example, we see some contribution in family planning to these regions where the data we have been receiving from DHIS2 shows an increase in the number of users of FP services. Particularly with Jhpiego in Lake Zone, we see some increment!! But again, it is extremely difficult to say these increments are contributed by Jhpiego because there are other key players in there…AMREF, Engender Health, PATHFINDER, and Marie Stopes…all there!! So, who do you attribute those results? Is it UBA or Marie Stopes…it is extremely difficult!” (KII national GOT, Director of Reproductive and Child Health [RCH], Dodoma) After three years working on integration, the MOHCDGEC and PO-RALG reported they would like to see what works to inform national policy and planning. The authorities reported they need a practical coordinating mechanism with all three IPs (not separate) to show which of their integrated approaches work best and ensure their lessons learned lead to sustainable changes. They reported that other USAID-funded projects have established effective coordination. It is important for the GOT and for USAID’s Journey to Self-Reliance (J2SR) to ensure sustainability by having the GOT actors in the driver’s seat at both the national and regional levels. 4.2c.3 Funding integration is complicated by multiple funding streams with separate global priorities IHC requires a combination of multiple funding sources from PEPFAR, PMI, MNCH, TB, and FP/RH. Each funding source has its own target activities, performance targets, and objectives. This funding has been reported to have been allocated to the three IPs based on the level of epidemiological need in each zone which varies year to year. Because in the UBA context multiple USG funding sources operate in the same space, competing priorities can emerge as target councils for PEPFAR change and affect other epidemiological priorities (such as the case of PMI for malaria). Each IP award is overseen by a single AOR who is responsible for channeling these multiple funding streams and managing the performance of the IP to deliver services for each funding source. The HIV component through PEPFAR is the largest health funding source and its performance targets drive the UBA program. PEPFAR remains a dominant influence in funding health systems nationally and this limits integration in non-HIV related areas (in MNCH, malaria, and nutrition). In the North/Central Zone and Southern Zone, HIV has been allocated a significant amount of funding at 83 percent and 72 percent, respectively. Table 7 shows Performance Year (PY) 2 expenditure data by health area per IP. The expenditure follows the program emphasis of each IP, with MNCH, malaria, and family planning for Jhpiego while HIV PEPFAR funding is the main funder of EGPAF and Deloitte. Funding streams have influenced the focus of integration to be on CTC and TB clinics in the North and Central and Southern Zones. In the Lake and Western Zone, HIV and TB programs are managed through CDC and, although the IP has reported they 30 coordinated integration with CTC services, they have focused on strengthening MNCH, malaria and FP services. Table 7: Comparison of IP Expenditures by Health Area IP Zonal Budget HIV- PEPFAR (HIV) Malaria FP MNCH TB and Other Services EGPAF $19,363,037 83% NA 10% NA 7% Jhpiego $14,538,292 NA* (CDC) 33% 26% 41% NA Deloitte $30,180,751 72% 10% 8% 2% 9% *NA: Not applicable because the IP does not receive funds from that source. 4.2d How effective are the integrated programs under Boresha Afya expanding access and coverage in HIV, TB, MCH, FP, malaria, nutrition, gender, and GBV? Table 8 below provides an example of the UBA IPs’ achievements by selected indicators to date according to FP/RH. The IPs have sometimes exceeded targets and sometimes missed them. The degree of variance in performance versus targets raises the question on the methodology for setting targets, which would ideally be based on expected workplan outputs with the Mission, health system weaknesses, and burden of disease. However, the IPRS had gaps in baseline data to compare and understand progress to date. The IPRS also lacked clear indicators of integrated outcomes such as the “number of HIV positive women who have received FP and been screened for TB.” Only indicator #10 in Table 8 is related to integrated service delivery. Annex 6, Table 8 offers a complete table of 24 IPRS indicators of which six are IHC indicators. Table 8: IP Cumulative Achievements to Date in FP/RH from IPRS25 Indicator IP FY 17 Target FY 17 Actual FY 18 Target FY 18 Actual FY 19 Target FY 19 Actual Cumulative TOTAL End of PY3 Number Of People Trained In Family Planning/Reproductive Health With USG Funds Southern Zone - Deloitte . 102 328 152080 . 65 152247 Lake/Western Zone-Jhpiego 70 71 740 633 325 105 809 North /Central Zone-EGPAF 76 107 . 495 . 309 911 Number Of Clients Accepting A Modern Family Planning Method Southern Zone - Deloitte 134238 . 783385 622373 1002747 570429 1192802 Lake/Western Zone-Jhpiego 81992 . 1079351 1220378 1386003 946444 2166822 North /Central Zone-EGPAF 285321 . . 570270 . 647179 1217449 Number Of Youth Who Have Received Family Planning/Reproductive Health Services Southern Zone - Deloitte . . 23905 352523 181322 250687 603210 Lake/Western Zone-Jhpiego 250996 14951 26637 323805 411782 403604 742360 North /Central Zone-EGPAF . 67999 22336 . 228822 . 67999 25Without baseline and clear accumulative targets IPRS data can only describe achievements to date. 31 Indicator IP FY 17 Target FY 17 Actual FY 18 Target FY 18 Actual FY 19 Target FY 19 Actual Cumulative TOTAL End of PY3 Number Of Usg-Assisted Facilities That Offer Fp Services Immediately Southern Zone - Deloitte . . . 100 298 . 100 Lake/Western Zone-Jhpiego 64 95 85 84 186 200 379 North /Central Zone-EGPAF . 100 61 100 175 . 200 Percent Of Women Receiving Modern Method Of Fp Immediately Southern Zone - Deloitte . . . 90 . . 90 Lake/Western Zone-Jhpiego 7 10 9 30 8 35 75 North /Central Zone-EGPAF NA NA NA NA NA NA . Couple Years Protection In Usg Supported Programs Southern Zone - Deloitte . . 68680 783385 216776 1284522 2067907 Lake/Western Zone-Jhpiego 3000573 264357 260273 1657924 2231298 1788134 3710415 North /Central Zone-EGPAF . 336659 165114 . 1969496 . 336659 Number Of Usg-Assisted Community Health Workers (Chws) Providing Family Planning (Fp) Information, Referrals, And/Or Services During The Year Southern Zone - Deloitte . . 40 1348 . 2524 3872 Lake/Western Zone-Jhpiego 748 748 696 1308 698 1308 3364 North /Central Zone-EGPAF . . . 168 336 208 376 Malaria Program Improvements The malaria disease control program is not an activity area in the UBA North and Central Zone, which brings the total number of malaria facilities sampled by the ET to 12. Eleven of the 12 surveyed facilities reported to have improved malaria treatment and to have seen changes in clients’ preventive behaviors. In the month prior to the survey, 16 malaria deaths were reported in the sampled facilities (shown in Table 17 below). The following table displays the percent prevalence across national and sampled regions for Lake and Western and Southern Zone. National and regional trends show a higher malaria prevalence rate since 2016 compared to 2019. This is contrasts with improvements in the % of children who slept under an ITN and a marked improvement in the % of women who receive IPT for malaria during their last pregnancy. (See Table 9) Table 9 Malaria Prevalence and Prevention Interventions 2016 to 2019 Performance Indicators (DHIS2 indicators defined as reflected in the SOW) National: Tanzania Totals Region: Morogoro Region: Kagera Region: Mara Year 20 16 20 17 20 18 20 19 20 16 20 17 20 18 20 19 20 16 20 17 20 18 20 19 20 16 20 17 20 18 20 19 Malaria prevalence rate (%) 6 10 11 10 . . . . 8 22 17 12 6 8 9 12 % of women who receive IPT for malaria during their last pregnancy 61 66 81 87 . 62 80 92 66 69 77 79 46 55 73 81 % of children who slept under an ITN the night before the survey . 69 84 94 . 16 46 81 0 52 84 90 0 53 79 75 32 Support to Community Health Worker The evaluation also gathered the views of the health workers in these regions. A total of 118 health care providers (HCPs) were interviewed in 18 facilities (average of 6.5 per facility); 64 percent were female, and 36 percent were male. Surprisingly, 14 of 18 facilities reported to have received support from UBA. The IPs have made important investments in training and the facility checklist results confirmed this support. Support was reported to be in the form of training, supervision, mentoring, equipment, and supplies (see Table 10). Not all facilities reported UBA support, but that can be due to the different needs of facilities and recall bias. Any changes to facility organization, health care delivery processes, or patient flow were not reported. Table 10: Reported Support Received by the HCPs of the 18 Surveyed Facilities26 Type of Support Received Number of Facilities (of 18 total) Training 14 Equipment 14 Supportive supervision 13 Mentoring 11 Supplies/Commodities 10 Additional funding 8 Additional staff 5 Application of Training and Quality Improvement In 14 facilities, staff reported to have received training, been able to apply it, and followed quality of care standards. The training topics varied but in seven facilities they reported training mostly in HIV/AIDS-related topics and/or FP. Eleven out of 14 facilities reported this improved the quality of care. Also, staff in 13 facilities reported to have been able to improve the star rating of their facility in the last two years. MNCH Program Improvements Eleven facilities reported that staff received BEmONC and CEmONC training. ANC was reported to be performing well in 16 out of the 18 facilities. In fact, 17 facilities reported to follow all pregnant women in collaboration with CHWs. Given the integration with prevention of mother-to-child transmission (PMTCT), child health, and FP, this area of IHC was the one where HCWs reported having too much to do and not enough time to deliver all the services. This can be due to lack of staff and may also indicate the need to streamline and simplify the healthcare delivery process for more efficient standard operating procedures (SOPs). According to available national health data from DHIS2 there has been significant improvement in the percent of live births attended by skilled health personnel between 2016 (61%) and 2019 (79%). Contraceptive prevalence has decreased. (See table 11) 26 Adapted from “HIV treatment support services in Tanzania: a cost and efficiency Analysis at facility and community levels,” HP+ February 2019. 33 Table 11: FP and MNCH Trends 2016 to 2019 Performance Indicators (DHIS2) National: Tanzania Totals (for IP or the Mission) Region: Kagera Region: Mara Year 20 16 20 17 20 18 20 19 20 16 20 17 20 18 20 19 20 16 20 17 20 18 201 9 Modern contraceptive prevalence rate 40% 35% 38% 33% 34% 33% 37% 30% 36% 32% 33% 31% % of live births attended by skilled health personnel 61% 66% 76% 79% 59% 63% 76% 78% 64% 64% 66% 66% FP Integration and Improvements Out of the 18 examined facilities, 13 reported to have improved FP counseling and nine facilities reported that due to UBA’s support their work is easier. Eleven facilities reported they have improved clients’ choices. Nine facilities reported that FP is now part of every consultation. However, six of the 18 facilities reported to have been unable to provide FP services at least once due to stockouts in the past year. Table 12 below is a summary of the reported support by UBA. Table 12: FP Improvements Attributed to UBA in Surveyed Facilities Reported Improvements Attributed to UBA Number of Facilities FP counseling 13 Implant insertion 12 Improved my general counseling skills 12 Improved client’s choices 11 Integration of FP services 9 Made my work easier 9 FP is now part of every consultation 9 Reduced unmet need 5 In the Exit Interview Surveys in 18 facilities across zones, 23 percent of clients reported to have received other services they had not expected. The ET asked about FP services received on the day of the visit (see Table 13). Only 10 percent reported to have received FP services that day and 58 percent reported not to have been offered one, which might indicate missed opportunities. Table 13: Reported Integration of FP Services Received FP Method During Visit N % No 160 89.9 Yes 18 10.1 Total 178 100 If No, Offered FP Method During Visit N % No 93 58.1 Yes 67 41.9 Total 160 100 34 National data from DHIS2 show that that in both UBA Southern Zone Sampled Regions and North and Central Zone Regions there have been improvements in couple years of protection between 2016 and 2019. (See table 14) Table 14: National Trends in UBA Regions for Couple Years Protection Year 2016 2017 2018 2019 Region: Morogoro - 70914 276558 392039 Region: Iringa - 58566 442215 373830 Region: Kagera 283396 291049 355554 312090 Region: Mara 285635 291346 318691 334107 HIV/AIDS Program Achievements and TB Integration The ET was given access to DATIM and provided with data on the output performance indicators found in Table 15, which presents achievements by fiscal year (FY) in relation to annual targets for the two UBA zones implementing HIV and AIDS activities. This presents the overall achievements since the start of UBA. The table shows that for the most part, IPs have generally met or exceeded targets for HIV testing, PMTCT, ART, and TX_CURR through FY 2018, although performance through Quarter (Q) 3 of FY 2019 reveals that they may fall short of targets in FY 2019. The fact that several targets were exceeded would indicate a need to revise the methodology for setting targets. Table 15: HIV Indicator Target Achievement by IP and Year Target Achievements by Year EGPAF – North Zone Deloitte – Southern Zone Indicator FY 2017 FY 2018 FY 2017 FY 2018 HTS_TST – Number of individuals tested and received results 103% 131% 109% 173% PMTCT_ART – % of HIV positive pregnant women who received ART to reduce the risk of mother to child transmission during pregnancy 92% 123% 103% 122% TX_CURR – Number of HIV positive adults and children receiving ART 357% 98% 380% 100% Source: DATIM, Table 16: National DHIS2 Data on HIV Intervention 2017-2019 by UBA Regions Indicators Southern Zone- Deloitte North and Central Zone- EGPAF Region: Morogoro Region: Iringa Region: Kilimanjaro Region: Dodoma 2017 2018 2019 2017 2018 2019 2017 2018 2019 2017 2018 2019 Number of HIV positive adults and children receiving ART. 66938 80084 49316 98554 109634 60170 23243 26974 29192 20819 20407 28982 35 Number of individuals counseled, tested and received results 292,563 637532 194607 118,874 258601 148879 119906 202197 196834 131262 208702 172661 Number of HIV positive adults and children receiving a minimum of one clinical care. - - - - - - 23243 26974 29192 20819 20407 28982 Number of HIV infected clients attending HIV care and treatment that are receiving treatment for TB disease. 623 557 1208 749 341 1352 - - - 2157 208 608 Health information for August 2019 (the month prior to the survey). Maternal and child health, and related mortality data were available in most facilities and the computerization of DHIS2 in hospitals facilitated data collection. Table 17 compares the performance of the 18 surveyed facilities and shows difference in performance and workload. Naturally the data showed higher number of deaths in the higher volume hospital facilities. Of the 18 facilities, Dodoma Regional Hospital reported the highest total deaths, 48 across categories, in the previous month during the data collection period, followed by Turiani hospital* in Morogoro region with 36 deaths, and Tarime District Hospital in Mara with 22 deaths in the previous month (see Table 17 below). Table 17: Burden of Disease for Past Month in Surveyed Facilities Zone North and Central Zone Lake and Western Zone Southern Zone Facilities 1* 2 3 4 5 6 7* 8 9 10 11 12 13* 14 15 16 17 18 Acute Respiratory Infection (ARI) 37 . 424 148 176 49 . . 17 45 34 342 154 141 407 118 345 Diarrhea 20 . 81 18 20 24 35 13 52 52 16 1284 44 50 221 14 87 Malnutrition cases 13 . 0 21 0 0 6 3 4 2 0 1 93 0 25 12 4 3 Malnutrition treated/referred 0 . 0 0 0 6 4 4 2 0 1 93 0 0 1 4 3 Maternal deaths 2 2 0 0 0 0 0 0 0 0 0 2 0 0 0 0 0 Infant deaths (0- 11 months) 4 0 0 0 0 0 0 0 0 0 0 4 0 0 0 0 0 Neonatal death 23 0 0 0 0 7 1 0 0 1 0 3 0 0 0 0 0 Still births 0 0 0 0 0 4 0 0 4 0 0 10 0 0 1 0 0 Child deaths (12 to 59 months/ under 5) 4 0 0 0 0 2 0 0 0 0 0 0 0 0 0 0 0 HIV deaths 11 0 1 0 1 4 1 0 1 0 0 5 0 0 3 4 1 TB deaths 3 0 1 0 0 2 0 0 0 0 0 0 0 0 0 0 0 Malaria Deaths 1 0 0 0 0 3 0 0 1 0 0 12 0 0 0 0 0 Total Deaths 48 0 2 2 0 1 22 2 0 6 1 0 36 0 0 4 4 1 TB Integration Challenges HCWs’ environment and performance were reported to have improved in all 12 surveyed facilities. However, 11 facilities reported that there are still challenges in integrating TB with HIV services, of which the lack of staff, supplies, and equipment were the challenges most frequently 36 reported (see Table 18). The targets and the results for the proportion of patients screened for TB are nearly 50% above target (for EGPAF) (Table 19). Table 18: Reported Challenges to TB Integration by Facilities Reported Challenges to TB Integration Number of Facilities Lack of staff 7 Lack of lab supplies 4 Lack of time to do everything 4 Lack of X-ray machine 4 Lab reports take too long 2 Lack of space 2 Patients are lost to follow up 2 Lack of medicines 1 Table 19: Targets vs Results for TX_TB27 Cumulative Achievements to Date EGPAF￾North Zone Indicator Targets Results % TX_TB - Proportion of patients screened for TB in the semi-annual reporting period who are receiving TB treatment 1.66% 2.48% 149% Community-Facility Integration There are several USG partners working to improve facility and community-based HIV health care delivery. The “Engagement Framework” of August of this year shows how collaboration between community- and facility-based efforts should be implemented together and their respective reporting responsibilities. To date, there is not a plan to monitor the effectiveness of this framework. Medical Guideline Availability A minimum of seven guidelines were expected to be available to practitioners in each facility in the following areas: HIV, TB, Maternal Health, IMCI, Malaria, QI, and the National Operational Guidelines for IHC. Five of the 18 facilities had less than half of the guidelines according to the Facility Checklist, including most selected facilities in the Southern Zone (see Table 20). Table 20: Availability of Medical Guidelines in Facilities Facilities Per Zone Zone 1 2 3 4 5 6 North and Central 6 3 4 7 6 5 Lake and Western 7 7 6 7 7 7 Southern 5 6 3 2 2 2 27 Only EGPAF provided data for this indicator. Deloitte data could not be retrieved from DATIM because it does not have targets for these indicators. 37 Intensified Supervision and Mentoring There were 287 supervisory visits reported over the last 12 months at the 18 surveyed facilities, of these 47 percent were conducted by UBA staff with most others being conducted by council/district or regional health staff supported by UBA. While there was not a great amount of variation between zones, the Northern and Central Zone had the most supervision visits to the surveyed facilities in the last 12 months (see Table 21). Questions remain on the sustainability of these efforts which are supported by UBA. Table 21: Reported Type of Supervisory Visits in the Last 12 Months Supervisor Type by Origin North & Central Lake & Western Southern Number of Visits District 31% 24% 32% 85 (30%) Regional 13% 16% 14% 40 (14%) National 4% 2% 5% 12 (4%) UBA 48% 52% 40% 134 (47%) Others 4% 5% 9% 16 (6%) Total 100% (112) 99% (82) 100% (93) 287 (101%) Client Satisfaction Based on Exit Interview Surveys The midterm evaluation conducted 180 exit interviews with women, 60 in each zone, that is, 30 in each region. Forty (40) percent of those interviewed were 30 years of age or younger, 44 percent were 31 to 49 years, and 16 percent were 50 to 78 years. Seventy-seven (77) percent traveled less than an hour to get to the facility, 71 percent waited less than one hour to be seen at the facility, and 12 percent waited two hours or more. Forty-five (45) percent came for follow up and prescription refill. Forty (40) percent (n=72) were prescribed a test and all but one patient had the test done that day. Forty-eight (48) percent (n=86) were prescribed a medicine and only eight patients were not able to get the medicine, in six cases because it was not available, and one could not afford it. All those that received a medicine reported the HCP explained well how to take the medicine. Forty (40) percent (n=72) had delivered a baby in the last two years, 66 percent (n=48) in the same facility, three at home, and one on the way to a facility. Ninety-two (92) percent (n=66) of mothers reported being satisfied with the care received. Ninety-one (91) percent (n=61) reported a free delivery and 9 percent (n=11) chose to pay. Responses were also analyzed in a composite score that included items such as favorable perceptions regarding privacy, cleanliness of the facility, and HCW’s attitude and assistance with referrals and care (see Table 22). Results across variables showed greater perceived satisfaction with the quality of care received in the North and Central and Southern Zones. Across all three zones, 70 percent of the patients had suggestions for improvement in areas such as staffing, supply of medicines, equipment, and furniture. 38 Table 22: Comparison of Perceived Satisfaction with the Quality of Care Received At Least Eight of II Variables Scored Good or Very Good (Q1-Q11) Lake and Western Zone Northern Zone Southern Zone Total N % N % N % N % No 22 36.7 6 10.0 6 10.0 34 18.9 Yes 38 63.3 54 90.0 54 90.0 146 81.1b Total 60 100 60 100 60 100 180 100 4.2.e How cost effective is the integrated approach to implementation to Boresha Afya IPs in coordination with each other, USAID, and the GOT? Varied Burn Rates (BR) BRs are usually measurements of the total amount expending in a given project year and on indicative of organizational and executive capacity of an IP to implement its programs. Table 23 shows that the UBA IPs have achieved different BRs. EGPAF and Deloitte both had fairly similar burn rates by PY2 and PY3. For the LoP EGPAF has the highest BR out of the three IPs due to lower burn rates in PY1 for the other two IPs28. Table 23: IP Budgets, Actuals, and Burn Rates29 IP Budget LOP Expenditure PY 1 BR PY 2 BR PY 3 BR LOP BR Jhpiego $51,270,813 $36,154,149 51% 77% 74% 71% EGPAF $61,041,650 $49,019,082 71% 88% 79% 80% Deloitte $87,794,786 $61,607,920 43% 88% 78% 70% Cost estimates are based on data obtained from available documents such as work plans and selected costs provided by the IPs, rather than through cost accounting reviews. The IPs have different accounting systems that are not organized to allocate expenses and make comparisons by region, facilities, and interventions across all three zones. Table 24 below shows the different regional expenditure patterns of EGPAF and Deloitte (Jhpiego not available). EGPAF reported to have spent 44 percent more within the regions than Deloitte. EGPAF reported to have spent almost $54 million while Deloitte spent about $30 million. This is due to a differences in approach and the differences in the extent of activities per region in each zone. Table 24: Comparison of Expenditure Patterns in Regions: EGPAF and Deloitte30 Regions Total Expenditure % of Total Expenditure Regions Total Expenditure % of Total Expenditure Southern Zone North and Central Zone Morogoro $6,669,291 22% Tabora $14,329,565 27% Iringa $6,026,439 20% Arusha $7,213,011 13% Mtwara $5,977,719 20% Kilimanjaro $7,664,561 14% Ruvuma $831,624 3% Dodoma $11,256,284 21% Lindi $4,949,440 16% Singida $7,092,687 13% 28 Jhpiego in peer review stated that they expect to have $6M remaining by the end of PY5 29 PY3 data is quarter 1-3 only. 30 Based on the expenditure data disaggregated by region provided by the IPs 39 Regions Total Expenditure % of Total Expenditure Regions Total Expenditure % of Total Expenditure Njombe $5,726,239 19% Manyara $6,156,950 11% Total $30,180,751 100% Total $53,713,057 100% The ET reviewed available cost information related to the implementation of the integrated approach by each IP. Table 25 presents the number of patients reached, the cost data per client and facility across specific health areas for both EGPAF and Deloitte from the start of the program to date. Note that this comparison is based on data reported by each IP and not the result of an accounting study of each project or a formal HF assessments.31 The costs reported are only UBA costs to improve health care delivery by integrating services and do not include government costs for salaries, building maintenance, and overall regional and council management. Also, the comparison does not include client costs related to out of pocket expenses, travel, and loss of wages. Table 25: IP Cost Comparisons in Selected Program Areas Program Area and Cost Deloitte EGPAF Total HIV clients diagnosed 31,635 37,837 Cost per client $73.23 $146.63 Total clients on FP 824,40832 899,521 Cost per client $4.18 $2.69 Total TB patients diagnosed33 8,723 10,240 Cost per diagnosed client $169.11 $138.19 Total Births34 13,367 NA Cost per birth $26.65 NA Total women on PMTCT 8,380 7,454 Cost per woman35 $112,97 $250.94 Total HIV positive children 10,036 3,006 Cost per child $39.43 $871.38 Total people being tracked36 63,718 958,615 Cost ratio $28.9037 $24.40 Average cost per hospital $113,166 $38,105 38 Average cost per health center $22,564 $22,47439 Average cost per dispensary $6,121 Unavailable 31 Measure Evaluation HFA methods: https://www.measureevaluation.org/resources/tools/health-information-systems/hfa￾methods 32 For Deloitte the number based on the couple of years protection as it was reported by IP 33This number was reported by IP as the total number of people diagnosed and we divided that by the total funding spent to determine the cost per person. 34 The number based on the total facility deliveries of 2 districts of Kilolo and Iringa DC, where IP is supporting MCH services. 35 This includes the testing costs for all pregnant women and PMTCT treatment for HIV positive women. 36 This figure is the result of adding all the people tracked for HIV, MCH, and TB as reported by the IP.. . 37 For cost ration we have used total cost for CSOs so far against total people being tracked for the LoP 38 https://www.hfgproject.org/essential-package-of-health-services-country-snapshot-tanzania/. 39 Handler A., Issel M., and Turnock, B. A Conceptual Framework to Measure Performance of the Public Health System, American Journal of Public Health. 2001 August; 91(8): 1235-1239. 40 Program Area and Cost Deloitte EGPAF Average cost per CSO/Subgrantee $81,578 $127,918 IPs’ External Costs The three IPs have managed several subgrantees including regional and council health teams, faith￾based hospitals and organizations, and CSOs. Jhpiego: Starting in PY 2, Jhpiego has managed grants to 14 local organizations for a total budget of $1,156,756.96, but expenditures reached only 66 percent ($762,198.10) for that year. This gives an average of about $54,433 per CSO per year. The ET did not receive expenditure data for PY 3. Since the engagement of CSOs in the project in PY 3 first quarter (October-December 2018) to last quarter (July-September 2019), the IP reported that the coverage and service utilization in HFs linked with CSO interventions has improved several selected RMNCH indicators. In addition, a total of 33,633 people were reached through CSO demand creation intervention, which is 66 percent of the annual expenditure target of $50,432. Expenditures divided by the 33,633 people reached gives a total cost per person of $23. To determine what approach is most effective, there would be a need to compare the performance achieved by the CSO-supported councils with other councils without support. This comparison might show the value for money of investing in CSOs. EGPAF: This IP has 59 local subgrantees (not CSOs) plus the EH subgrant. EH has a total budget for the three years of implementation of $2,507,677. However, actual expenditures of EH totaled $1,602,330.41 or 64 percent BR. The other subgrantees include 15 regional, national, and faith￾based hospitals and 45 councils for a total LOP budget of $19,346,654. This gives an average of $127,918 per subgrantee in the LOP. This cost is almost $90,000 lower than Deloitte so a study of the effectiveness of the EGPAF approach would be needed to determine whether the cost difference yields the desired sustainable result through the subgrantees and the reason for the low BR. Despite slow burn rates in the first year, expenditures by the EGPAF subgrantees (regions and councils) were reported to have reached 53 percent of the budgeted funds for the LOP by midterm. The KIIs with council staff reported IP delays in disbursements that postponed per diem and salary payments to staff by several months in the first year. Deloitte: Deloitte has managed total of 109 sub awardee agreements which includes 26 local CSOs, 43 councils, 6 regional hospitals, and 12 private hospitals, and 22 faith-based organizations (FBOs) for a total budget of $6,599,003 and total expenditures of $5,524,799 or 84 percent execution. The program is implemented in 273 health facilities (47 hospitals, 97 health centres, 129 dispensaries). The average cost per CSO conducting facility and community-based services is $212,492. The ET was not able to assess the effectiveness of having CSOs complement the IHC model, but the IP reported they have improved healthcare access. Cost of Supporting CHWs The integrated models of both Deloitte and EGPAF rely on CHWs and volunteers, which constitute many IHC providers that are essential to expedite referrals and linkages with the community. The cost of supporting CHWs and other volunteers is only known for the Southern Zone. Deloitte has reported to have invested in 2,713 volunteers at an annual cost of $817,607, that is, $301.37 per volunteer per year. This is a large amount that is not likely to be fundable by 41 the GOT unless additional funds are mobilized. So far, neither EGPAF nor Deloitte have identified financing mechanisms to sustain this. Community health financing and other interventions used in other countries to mobilize local funds to support CHW and other volunteers have not been tested. Several cost-savings practices have been reported by the IPs, such as integrated supervision and on-the-job training. Another useful practice is including budget allocations in the annual work plans. This helps analyze the justification and compare the evidence of effectiveness of various interventions and assess the value for the money. For example, Deloitte’s PY 3 work plan included a budget line for 390 HCWs to provide health services through extended working hours at a cost $595,595, that is $1,527 per HCW per year. In the absence of workflow, workload, productivity, and absenteeism studies, it is difficult to determine how effective and sustainable this intervention is. Potential Benchmarks from Previous Tanzania IHC Evaluations Finally, below is our review of other available health service cost evaluations in Tanzania. These studies include the evidence from the Tunajali II and LIFE end-of-project (EOP) evaluation and the Mid-Term Review (MTR) of the Health Sector Strategic Plan (HSSP) IV. The Tunajali II and LIFE EOP evaluation was designed to measure cost per client of the projects against benchmarks in the literature. It would be useful if UBA had been set up to track costs in the same manner and use the costs of these UBA predecessor projects as benchmarks. Tunajali II Project Costs40 The total financial cost for the Tunajali II program from 2012-2015 was $43,560,991; this is similar to the budget for the Lake and Western Zone and about one-third of what either the Southern or Northern Zones were awarded. Recurrent costs accounted for 81 percent of the total cost, of which personnel wages and fringe benefits consumed 42 percent. Three key services in Tunajali II were identified for cost-effectiveness analysis: voluntary counseling and testing for HIV (HCT), PMTCT, and MNCH. Provision of HCT in Tunajali II-supported areas was estimated to cost an average of $83 per client. The estimated average per capita economic cost of providing a minimum of one care service was $52. As seen in Table 23, UBA has shown a much higher cost per client, due to a much broader set of activities at the facility level and not due to differences in efficiency. To measure the efficiency or cost effectiveness, a more thorough study could be undertaken to better determine more direct comparisons between programs. LIFE Project Costs The total financial cost for the LIFE program from 2012-2015 was $40,553,539. Recurrent costs accounted for 85 percent of the total cost, with personnel wages and fringe benefits consuming 41 percent of the total cost. Two services in the LIFE program were assessed for cost effectiveness, namely, PMTCT and MNCH. The estimated total economic cost for PMTCT was $26,326,227. PMTCT in the LIFE-supported regions is estimated to have averaged $640 per client. Testing a pregnant woman for enrollment in PMTCT was estimated to cost on average $22. The estimated total economic cost for MNHC services was $2,200,469. The economic cost of 40 The information below reflects the period of Tunajali final evaluation and does not reflect total LoP cost. 42 additional funding for MNCH was $9 per woman who delivered at a health HF and the average cost per newborn for postnatal follow-up within two days of birth was $21. Table 26: Cost and Frequency of Patient Contact for HIV Support Services by Available UBA and Previous Tanzania Service Models41 Service Model Defaulter Tracking Peer Support Counseling Peer Support Groups Disclosure Support Opportunistic Infection Screening Outcomes Virally Suppressed Cost/ Client Facility-based programming only Every 1-3 days Weekly Daily Daily Daily 68.0% $108 Community and facility based programming Every other day Every 4 days Every other day Every other day Daily 77.9% $45 Community￾based only Monthly Monthly Monthly Monthly 6-12 months NA $20 Tunajali II - - - - - - $60-115 LIFE project - - - - - - $513-780 UBA HIV/FP integrated care Unavailable Unavailable Unavailable Unavailable Unavailable Unavailable $150-350 4.2.f How Has the Developmental Evaluation Program Contributed to the Effectiveness, Learning, and Adaptability of UBA Implementers? 4.2f.1 Overview A DE is being implemented by the USAID Coordinating Implementation Research to Communicate Learning and Evidence (CIRCLE) project to support real-time collaborative learning and adaptation for UBA with the following aims: 1. Generating evidence in real-time through flexible, situationally tailored evaluation design; 2. Building and supporting adaptive management practices among the program stakeholders; and 3. Catalyzing rapid learning and decision-making toward improving the quality, efficiency, utilization, and scalability of integrated health services in Tanzania. The HQ office is in Dar es Salaam and Zonal Development Evaluators are deployed in each of the three UBA zones. There are six staff. The DE’s task is to design and deploy integration models that encompass each of the HIV health areas addressed by UBA. Its overall approach is that of implementation research and its findings are meant to inform the IPs, USAID, and GOT at the facility, council, regional, and national levels. The integration of FP and reproductive, maternal, newborn, child, and adolescent health platform was the defined focus of Year One. PY 2 DE activities are to focus on the integration of FP, malaria, and nutrition assessment and counseling 41 https://www.integratedcare4people.org/ipchs-framework/. 43 services (NACS) within HIV care and treatment services, and on strengthening the antenatal-to￾postnatal care (ANC-PNC) platform in the three Boresha Afya zones. 4.2f.2 The Developmental Evaluation (DE) Activities DE’s qualitative and quantitative data collection is carried out in a small number of facilities, councils, and regions that are selected in collaboration with the IPs, GOT, and USAID. DE has a presence in 12 districts in three regions. Its agenda is to provide implementation research support to the individual IPs as well as provide a wider overall perspective on IHC nationally. Feedback from their investigations are shared with USAID monthly and at council and facility levels in quarterly meetings, where discussions with IPs, GOT (PO-RALG), are held. Yearly research reports, learning and awareness workshops, an online progress tracker, and other knowledge products are shared. The specific challenges or issues addressed in their investigations are developed in a participatory manner with the IPs and through them the CHMTs and RHMTs. It needs to be noted that the DE is not actually a “real-time” data feedback mechanism, nor does it have a central data repository that combines the results of the three IPs with access to the partners for on demand feedback and results. DE data gathering includes monitoring strategies, outcome harvesting, study of referrals, Rapid Reconnaissance, client shadowing, and client satisfaction surveys; and the findings have been reported as nuanced results for answering emerging questions from the IPs and their collaborative partnerships. Operational learning takes place in specific regional locations where embedded staff supply regular insights. Findings were shared across the IPs and USAID health technical staff and representatives from PO-RALG. IP staff across zones had mixed views on the learning utility of DE. Some staff attested to the utility of results at the regional level for implementation purposes. Others felt that there were limitations in what could be learned from the evaluation efforts thus far because of their limited reach and scope. (three of 11 GIs/KIIs with IPs) “DE has been helpful to us because they move through the facilities and identify some gaps. These gaps are shared here in feedback meetings in the office. We address them immediately. The feedback that we receive from DE has been very useful in terms of improving the services we provide in the facilities.” (GI with regional IP, Southern Zone) The Lake and Western Zone IP found the DE’s input beneficial in terms of evaluation, but otherwise limited in its scope due to the small number of facilities examined and that information supplied was not adapted to their needs. “The scope of DE was very minimal because as it was not implemented in more than four health facilities in Mara and in the whole region…we have more than 200 health facilities. The scope is very minimal, you can’t have robust information to see how we can effectively learn, adapt and improve. Another thing is the focus of DE is very limited, initial they were more on integration, especially looking on how the project will integrate family planning into HIV intervention. As you may know, Boresha Afya is a very comprehensive project with this big thematic area.” (GI with IP, Lake and Western Zone, Mwanza) “So instead of looking at the RMNCH angle, they will come in look from the HIV angle. For instance, we are doing integration in family planning because we want to expand access and coverage of 44 [postpartum family planning] PPFP and not the other way around. Initial, I think the integration they were looking at was the integration of family planning in the RNMCH platform. They were not looking so much at integration on OPD, but then it lacks a systematic framework of what exactly they were looking at. So in between maybe you will see the integration of malaria in CTC. Then looking at the scope of each project, maybe it is understandable because the two are Boresha Afya, and easy to be skewed towards HIV, I think that is what we are currently seeing.” (GI with IP, Lake and Western Zone, Mwanza) Despite a slow start, evidence shows that over time, leaders in some zones/IPs and their regional teams have come to collaborate with the DE regionally and are taking actions in response to evaluation findings. What can be said about these contrasting perspectives is that collaboration with DE in learning varies by zone; the Southern Zone has been particularly involved at the national and regional levels. The opinion of one of the respondents in the USAID GI was critical. “DE was supposed to give answers as we go, so we could document results on what to change to improve integration. It would be a learning agenda…It doesn’t serve as a learning mechanism. Moderator: Why is it that it hasn’t worked? They are having difficulties monitoring the integration. It took so long to see it in their minds, how to measure it, what to do. I don’t think they understand what to do. Don’t know what integration is, maybe.” (KII with USAID AOR taken from notes, Dar es Salaam) 4.2f.3 The IHC Big Picture Role for the DE The definition of “integrated care” used by DE is drawn from minimal standards set by the WHO (2016) and sets a low bar. In this definition, providing more than one type of service in a health unit or referring clients successfully to another facility service point or “vertically” to a hospital is “integrated care.” Embedded in this definition is a client or “people-centered care,” i.e., customer satisfaction, especially for women, youth, and male engagement in the context of UBA. The ET is not aware of how or why this decision was made. However, this “low bar” is, from one perspective, reflective of realities on the ground. DE has uncovered major issues with simple procedures such as referral log keeping, bi-directional referrals, dissatisfaction with services, and so on. In this regard, DE provides valuable independent knowledge on health services that can guide UBA towards its objectives and that might not otherwise be known. DE is beneficial to UBA from an evaluation perspective but is overly focused on operational research issues. It should provide a more overarching approach to serve as a thought leader and as a champion for learning across the three zones. Evidence from USAID (two of five KIIs) and IPs (three of 11 KIIs) showed that DE GOT off to a slow start in its role in knowledge and data generation and information sharing across UBA zones. It is suggested DE could play a more active role in collaborating, learning, and adapting (CLA) to share lessons learned across zones and IPs. A CLA action plan connected to a UBA specific learning agenda would be a useful tool in cementing commitments from each of the zones and in 45 identifying what information gaps or questions for learning are most useful in the second half of the program. 4.3 EVALUATION QUESTION 3: TO WHAT EXTENT ARE SERVICE PROVIDERS AT PO-RALG, LGAS, AND FACILITY LEVELS WHO ARE RECEIVING INTEGRATED SYSTEMS STRENGTHENING AND TECHNICAL SUPPORT SATISFIED WITH THE SERVICES AND TA PROVIDED AND ARE THEY APPLYING SKILLS AND PRACTICES TO THEIR WORK? 4.3.1 Background The current essential health care package in Tanzania was defined in the National Essential Health Care Intervention Package Tanzania (NEHCIP-TZ). An assessment of this package and the tables are included in the Essential Package Assessment report of the USAID-funded Health Financing project.42 The Health Sector Strategic Plan July 2015-June 2020 (HSSP IV) includes a comprehensive framework for measuring the performance of the health system to deliver this package.43 These two documents led to the development of the UBA results framework. It is important to note that the HSSP IV MTR called for a fundamental shift in the way services are funded, managed, and delivered in accordance with the WHO integrated people-centered health service framework.44 USAID/Tanzania and the GOT designed UBA to address the most common primary health care (PHC) problems related to MCH, HIV, TB, and cervical cancer. Each of these conditions or health needs are managed through vertical programs each of which includes evidence-based interventions, tools, and so on to address the problem. The programs have technical experts as managers at the MOHCDGEC and at USAID/Tanzania. For the most part, these vertical programs have the same goals and technical interventions, and USAID-funded programs contribute to their goals. These vertical programs are organized into service delivery packages that are delivered at hospitals, HCs, and dispensaries. The organization and management of their delivery is a “horizontal” process that integrates the correct interventions across the vertical programs to meet the needs of each client. An HCP cannot do the horizontal integration alone but needs a well-organized facility and well designed and tested SOPs. For this to happen, the CHMTs and RHMTs need to efficiently and simultaneously manage the horizontal health care delivery process in the facilities and in each vertical program. 42 Big Results now was a GOT initiative which focused on various areas of service delivery including health infrastructure, economic growth, and education during the previous administration and health strategy. 43 Source: Draft Mid-Term Report of the HSSPIV. 44 Five of eight FGDs with CHWs, five of 11 FGDs with women clients, two of nine FGDs with male involvement groups, two of seven KIIs with national stakeholders, and three of 17 KIIs with National GOT. Text Box 1: HSSP IV 2015- 2020 Priorities “Integrated delivery of a reviewed package of essential healthcare interventions, strengthened Comprehensive Council Health Planning decentralized to the facility level, better management of health facilities at all levels, and health system strengthening in aspects such as Integrated Logistics System, Human Resource, and District Health Information Systems are key features to achieve harmonization and a coordinated approach.” 46 These vertical and horizontal processes set the UBA stage from the GOT’s perspective and are the background to its efforts to integrate health services at each level. The key to quality IHC is consistency so that every patient gets the correct package of PHC that meets their needs. Consistency is achieved through good health facility management and use of tools, such as efficient patient flow patterns, SOPs, supportive supervision, and M&E data to improve productivity and achieve client-centered outcomes. 4.3.2 Findings RMNCH performance data show improvement in all regions since 2016 and in particular in those regions where UBA has focused on its activities, e.g., Geita, Mwanza, Kagera Tabora, Mtwara which are regions with the most significant changes between 2015 and 2018 (see Figure 2). Nonetheless, these improvements cannot be solely attributed to UBA because of other RMNCH initiatives, such as the Health Basket Fund (World Bank, GIZ, USAID) and other bilateral donors and NGOs. The differences between regions still remain big. The yellow bars are regions which received extra inputs under the Big Results Now program.45 Figure 2: Performance of RMNCH Composite Indicator in Tanzania, 2015-201846 National Government Level Authorities at the MOHCDGEC and PO-RALG are appreciative of the UBA support thus far. However, according to interview respondents, they would like to improve the manner in which it is delivered, including in a more coordinated and well-planned manner. They call for UBA to have a formal coordination mechanism at the central level to ensure effective alignment and accountability for the three zonal IPs and reduce the IPs working in silos. “There are lots of them (IPs) and they need a mechanism to ensure what they do is effective in increasing access. The mindset of some IPs is not aligned with what we need. The COP of one of them once told me, ‘you should get what they give you.’ We do not work that way anymore at the GOT. We need accountability and alignment with the GOT priorities. At times, we had to let go 45 Four of 35 KIIs with facilities, eight of 11 FGDs with women clients, and two of seven KIIs with national stakeholders. 46 Three of 11 FGDs with women clients, five of nine FGDs with male involvement groups, four of 35 KIIs with facilities, and three of ten KIIs with IPs. This chart was taken from the report of the midterm evaluation of the HSSPIV. 47 assistance because it was not going to contribute to increased access.” (KII national GOT, PO￾RALG) “They need to involve us with planning at that level and then they can coordinate implementation with the focal staff in the technical areas. We cannot have silos. Cannot manage that way. We need to improve communication and how we work...” (KII national GOT, MOHCDGEC Director) CHMT and RHMT Level When CHMT respondents were asked where UBA has been most successful in improving performance, they mentioned HIV-infected pregnant women receiving a complete HIV package while attending an RCH clinic rather than having to refer them to the CTC (one of five GIs with CHMT). “The provision of integrated services has also been helpful in PMTCT services whereby the HIV pregnant women don’t need to come to the CTC any longer, they can get a whole package of their services at RCH and eventually deliver their children safely without infections.” (KII CHMT, Lake and Western Zone) Respondents reported capacity-building and the supplying of equipment and consumables as the most common strategies used by UBA to improve job performance (CHMT, five of 11; facility staff, six of 35; national GOT stakeholder, one of 17). CHW and Community Level CHWs play a crucial role in facilitating referrals and patient follow-up. CHWs received health care training to support service delivery and demand creation for community health services (two of 11 KIIs with IPs, three of 17 KIIs with facility in-charges, two of five KIIs with CHMTs, two of six GIs with RHMT, two of 17 KIIs with facility staff). “Involving community people has helped us a lot, because we are working at the hospital environment but our fellows [CHWs] from the community know each other better, and they even know our clients more than us. Therefore, by using them, the number of clients lost has decreased, and they come on time and if s/he [client] was lost they must be found because there is a person who knows her/him and makes follow up.” (GI Facility staff, North and Central Zone) “In case of any emergency, CHWs call the health facilities so as to get assistance and service. And if there is any consultation, they can use those phones given for free to CHWs and also staff workers within the facilities, so that they can link with the referral hospitals.” (KII IP, Lake and Western Zone) Facility Level In general, HCPs stated that UBA’s integration efforts have brought benefits (five of 17 KIIs with facility staff, all zones). When asked how UBA strengthened the integrated health capacity in the facilities and communities, respondents observed that on-the-job training, mentorship, and supportive supervision played a key role (three of 11 KIIs with IPs, two of five GIs with CHMTs). When asked about accessibility of IHC service delivery, respondents stated satisfaction with multiple screening tests and treatments being provided in one location (two of 17 KIIs with facility in-charge, one of six FGDs with RHMT). Respondents also reported improved capacity in the 48 organization and functioning of the health care system together with improved referral and data collection, analysis, and utilization as a result of capacity building at the facilities (five of 18 KIIs with facility staff, all zones). “….now healthcare providers can screen HIV, family planning including long-term family planning at once to the same client.” (GI RHMT, North and Central Zone) “UUBA has supported us in capacity building. Before, we did not deliver services according to the standard guidelines, but now we can provide services in accordance with guidelines. Also, we have integrated services; we were not aware that family planning can be provided at maternity after delivery, but now, we are doing that.” (GI Facility staff KII, Lake and Western Zone) UBA has also added hundreds of additional health workers in all three zones to achieve their targets, but staff shortages remain a major problem in some councils. An analysis of the health workforce needs has not yet been conducted in UBA-supported councils and facilities. There are still areas, such as digital health, where UBA has not been able to help facilities: “For now, I think the challenge is a shortage of staff…as you can see! So, the issue is that we alternate from one unit to another—something which causes an overload of activities and lowers clients’ satisfaction to some extent.” (nine of 17 KIIs with Facility staff) “I wish they could do that; they could help us install GOTHOMIS. We hope that was possible because we have also included it in our budget. But our budget depends on the internal collection of revenues…” (GI Facility staff, Southern Zone) Results from the facility checklist revealed a high level of satisfaction among HCPs in the surveyed UBA facilities. The score is a composite of the answers given by six HCPs who were asked to agree or disagree with statements regarding the benefits received from UBA with a maximum possible score of 50 for each statement. As seen in Table 27, the Southern Zone scored the highest with a total score of 264. The statements the providers were asked to agree or disagree included: • My work is easier. • The quality of my work has improved. • I am more satisfied in my job. • More clients are able to access health services in my facility. • Health services have improved in my facility. • Clients receive improved care and are more satisfied. • The needs of my patients/clients are met in an integrated way. Table 27: Comparison of HCPs’ Satisfaction Scores Zone Facilities Total North and Central 47 24 37 31 39 21 199 Lake and Western 41 27 30 29 31 48 206 Southern 48 48 38 48 47 40 264 Client Level In the Client Exit Survey, 91.7 percent of respondents in the examined facilities stated they were satisfied with the HF services (see Table 28). They cited improved quality of services from both 49 primary HFs and hospitals with good reception and/or friendly services to have been the facilitating factors for their answers. Unsatisfied respondents answered mentioned previous challenges encountered receiving services, including harsh language of providers and problems with charges and billing. Table 28: Percentage of Clients Satisfied with Health Facility Services Type of Facility Number Satisfied Percentage Satisfied Hospital 15 83.3% HC 51 94.4% Total 66 91.7% 4.4 EVALUATION QUESTION 4: HOW EFFECTIVE IS THE BORESHA AFYA PROJECT AT IDENTIFYING AND ADDRESSING KEY GENDER AND YOUTH-RELATED BARRIERS TO SERVICE DELIVERY? 4.4.1 Barriers for Women Stigma Respondents reported stigma as the most common gender barrier to women’s care. When asked about barriers, respondents mentioned stigma towards female PLHIV women resulting in them being fearful to expose their status as it could end their marriage or bring shame to their families.47 “I can’t tell anyone about my health status. Once I disclose it will impact to my children, I don’t want to burden them now.” (Dodoma, Women Client FGD) Social and Cultural Norms Other major barriers faced by women include patriarchal systems that allow male partners to prevent them from attending health facilities, force them to introduce food to children under 6 months, or insist they breastfeed until their child is 3 years old. 48 Also, social norms that normalize GBV issues in communities were reported to keep victims from accessing support in a timely manner.49 “Women can’t visit facilities without male consent.” (Zanzibar, Female, national stakeholder KII) “Men are very difficult, because they own us, and their problem is to provide food in the family, they tend to force us not to adhere to 6 months exclusive breastfeeding and introduce porridge to our children instead.” (Kilimanjaro, Women Client FGD) “In Tarime, especially the ‘Kurya’…about 99 percent of male Kurya that you find cannot walk together with their wives to go to any place. Even if he is invited in a certain ceremony, he may tell his wife to just go, he will come later, and women can’t go anywhere without their husbands’ permission.” (Mara, Male Involvement FGD) 47 Three of ten KIIs with IPs, two of eight FGDs with CHWs, two of seven KIIs with national stakeholders, and four of 17 KIIs with National GOT. 48 Three of ten KIIs with IPs, two of seven KIIs with national stakeholders, three of nine FGDs with male involvement groups, and seven of 11 FGDs with women clients. 49 Four of nine FGDs with male involvement groups, five of 11 FGDs with women clients, and two of eight FGDs with CHWs. 50 Health Facility Barriers A variety of respondents mentioned disrespect and mistreatment of female clients by HCPs, especially during the provision of IHC in ANC/MNCH/ RCH.50 In addition, respondents reported that some HF layouts did not allow privacy, nor accommodate PMTCT services. This created difficulties for women who were encouraged to attend facilities with their partners.51 Likewise, respondents noted that facility structures did always accommodate space for pediatric HIV care (three of 35 KIIs with facilities, two of 11 FGDs with women clients). “Now we’re comfortable here in RCH, but after a year we will get shifted in that building and everyone knows that is a CTC, so that is discouraging (Kilimanjaro, Women Client FGD) Transportation Costs and Distances Respondents also identified the long distance to facilities and associated travel costs as barriers for women seeking care.52 Gender-Based Violence Respondents reported that health service providers were not well equipped to provide services to GBV survivors53 and that bi-directional linkages of GBV survivors to and from HFs remains a challenge.54 “The GBV issue is very challenging. You may identify and refer a survivor to local leader, open a case, and the next day you are told the file is missing and if you’re not careful all evidence will be missing.” (Kilimanjaro, Male involvement FGD) 4.4.2 Barriers for Youth Stigma Respondents reported stigma as the major barrier to accessing health services for youth, especially self-stigma among youth.55 Other barriers for youth that were mentioned included parents preventing youth from accessing care, including keeping CHWs away from their homes.56 Social stereotypes that label those who seek health services as lazy have also prevented youth from seeking health services, particularly males (one of ten KIIs with IPs, and two of eight FGDs with CHWs). 50 Four of ten KIIs with IPs, two of nine FGDs with male involvement groups, three of 12 KIIs with district/regional GOT, and seven of 35 KIIs with facilities. 51 Four of 12 KIIs with district/regional GOT, two of eight FGDs with CHWs, and two of nine FGDs with male involvement groups. 52 Two of nine FGDs with CHWs, six of six FGDs with Youth, six of ten KIIs with IPs, one of four KIIs with CSOs, and ten of 35 KIIs with facilities. 53 Three of eight FGDs with CHWs, two of four KIIs with CSOs, one of seven KIIs with national stakeholders, and two of 12 KIIs with district/regional GOT. 54 Two of ten KIIs with IPs, seven of 35 KIIs with facilities, and two of 12 KIIs with district/regional GOT. 55 Three of 11 FGD with women clients, two of six FGD with youth, and two of ten KIIs with IPs. 56 Three of ten KIIs with IPs, nine of nine FGDs with male involvement groups, 11 of 11 FGDs with women clients, four of 15 KIIs with National GOT, and one of five KIIs with USAID. 51 Health Facility Barriers Respondents reported that youth face barriers at facilities due to the lack of youth-friendly infrastructures and environments that allow privacy. 57 They also noted that health service providers resist providing some health services to adolescents and youth, especially RH services (two of ten KIIs with IPs, two of eight FGDs with CHWs). Also, limited facility hours were reportedly in conflict with the school day (three of ten KIIs with IPs, six of 12 KIIs with district/regional GOT). Some youth reported that some youth-friendly services provided in facilities were only medical in nature and failed to meet their non-health needs (three of six FGDs with youth). In addition, many of the UBA-supported youth-friendly corners were in urban areas, providing no access for rural youth (two of 12 KIIs with district/regional GOT, two of six FGDs with youth). “Some providers may deny adolescents and youth a certain type of services, such as contraceptives, believing that they are too young to receive those services. It may encourage them to have sex, maybe unsafe sex. And this is because there is limited knowledge on the part of health care providers on how to respond to youth and adolescent needs” (National, Male, IP KII) “Here we meet, discuss about our problems, eat and go home. Other UBA youth friendly corners have income generating trainings. We also have a lot of talents and we wish to be supported.” (Kilimanjaro, Male, Youth FGD) Gender-Based Violence GBV was also reported as a barrier for youth to attend health services, especially young mothers.58 “Now it is becoming a fashion for our husbands to ask to perform sex in anus, and we do it to keep our marriages, and sometimes it destroys us. I have experience of my friend who didn’t want to go for delivery in the hospital because she was afraid her secret will be exposed.” (Kilimanjaro, Women Client FGD) “The issue of molested young boys is increasing in our communities, and in most cases parents especially mothers, are the first to know about this and keep a secret, as they don’t want to expose their boys in the community, because it will destroy their manhood.” (Morogoro, Male Involvement Group FGD) 4.4.3 Barriers for Men Social and Cultural Norms Respondents reported men’s mindsets as a key barrier that keeps them from visiting health care facilities.59 They reported that health care, especially RH, is often seen by men as a “woman’s issue” because women are the ones who become pregnant and breastfeed. Respondents also noted that some men do not utilize health services because they fear learning they have the same 57 Seven of 11 FGDs with women clients, 12 of 35 KIIs with facilities, and two of nine FGDs with male involvement groups. 58 Three of nine FGDs with male involvement groups and three of 11 FGDs with women clients. 59 Five of nine FGDs with male involvement groups and two of 11 FGDs with women clients. 52 health condition as their partner (e.g., HIV).60 Finding out they are positive can further develop into problems of self-stigma. “The issues of escorting women to the clinic are not in our customs, in short.” (Iringa, Male, Male Involvement FGD) “Advocacy for male involvement is still a challenge because some men believe that it is women things and we do because we are paid to advocate. You will start the conversation in our male corners or provide sensitization in the community meetings and men will start laughing at you. That is how we face challenges.” (Kilimanjaro, Male, Male Involvement FGD) Health Facility Barriers Respondents also noted facility-related barriers that prevented men from using health services. These barriers included the travel time to a facility,61 long waiting times for services,62 and lack of privacy or an environment conducive to males.63 In two of nine FGDs with male involvement groups, men reported waiting long hours in a queue for testing only to be told the reagents had been used up. Some men reported that they no longer visit facilities because services and commodities that were previously available are no longer offered (two of nine FGDs with Male involvement). “Frankly speaking, most of the services are currently not available at facilities; a good example, condoms, which we are used to get for free at health facilities. There was a place where it was written, ‘Take to the maximum!’ Now, condoms are sold in the shops at high costs.” (Morogoro, Male Involvement FGD) 4.4.4 UBA Impact on Gender and Youth Barriers Stigma Respondents reported that UBA has reduced stigma and increased access to health care for women,64 youth,65 and men.66 UBA activities most credited with reducing stigma included the use of male champions,67 creating convenient facility hours for youth,68 and offering hours for PLHIV women to access services privately.69 Also, the availability of receiving more than one service at 60 Five of nine FGDs with male involvement groups, three of ten KIIs with IPs, and ten of 35 KIIs with facilities. 61 Seven of 11 FGDs with women clients, two of nine FGDs with male involvement groups, four of ten KIIs with IPs, 14 of 35 KIIs with facilities, five of 12 KIIs with district/regional GOT, two of four KIIs with CSOs, and two of seven KIIs with national stakeholders. 62 Six of six FGDs with youth, seven of ten KIIs with IPs, three of 15 KIIs with National GOT, and ten of 35 KIIs with facilities. 63 Four of ten KIIs with IPs, five of nine FGDs with male involvement groups, two of 35 KIIs with facilities, and two of four KIIs with CSOs. 64 Five of nine FGDs with male involvement groups, seven of ten KIIs with IPs, three of four KIIs with CSOs, seven of 12 KIIs with district/regional GOT, two of seven KIIs with national stakeholders, and five of 15 KIIs with National GOT. 65 Six of ten KIIs with IPs, six of six FGDs with youth, four of eight FGDs with CHWs, 22 of 35 KIIs with facilities, and eight of 12 KIIs with district/regional GOT. 66 Five of 11 FGDs with women clients, 16 of 35 KIIs with facilities, and four of 12 KIIs with district/regional GOT. 67 SAA is the model for addressing gender and social norm barriers to promote transformative changes in SRH programs. It is a comprehensive tool used by UBA to continue identifying and addressing key barriers to access and utilization of RH and other related health services, including GBV. 68 A handbook that coordinates GBV interventions in an emergency. It maps available resources and is used as a tool for all case management key players in linkages and referrals. 69 Three of ten KIIs with IPs, two of 12 KIIs with district/regional GOT, and two of 15 KIIs with National GOT. 53 a health service point has reportedly improved client privacy and reduced the waiting time to access all required HIV treatment services (six of 11 FGDs, with women PLHIV/RMNCH). “Before we used to line up in the queue to get services in the CTC, and when we needed testing we again lined up in the lab; and with FP you will need to come next time, but with UBA intervention we have our services provided right at one place, CTC, where we meet clients with same conditions like ours. So, I may even feel comfortable to exchange experiences with my fellows because we have similar health status. So, this has helped indeed.” (Morogoro, Women Client FGD) Social and Cultural Norms To address harmful social norms and stereotypes, UBA has implemented gender transformative activities through primary event curriculums and evidence-based social analysis and action (SAA) approaches70 (two of ten KIIs with IPs, two of 12 KIIs with district/regional GOT). During PY 1, UBA’s gender assessment study revealed that male involvement in health service provision to families was an issue that was not well understood and/or addressed in communities because of its strong connection to social-cultural practices. To combat this, UBA has used community leaders and influential people including local government, religious, and tribal leaders to expand the reach of health messages, sensitize communities on the importance of utilizing health services, and advocate for change in harmful norms that oppress women, youth, and other vulnerable groups (six of ten KIIs with IPs, four of 12 KIIs with district/regional GOT). Gender-Based Violence To address GBV issues, UBA developed a GBV screening tool and Gender Service Provider Directory.71 UBA facilitated on the job training to health providers on how to assess victims of gender violence and ensured the distribution of the GBV screening tool to integrated health delivery service points within supported facilities, including OPD, CTC, TB, and RCH units.72 At the community level, UBA trained community leaders and community volunteers on GBV issues, facilitated quarterly meetings with committees at the village level to monitor implementation, provided TA to case management teams, and updated community gender service provider directories (two of ten KIIs with IPs). Community and Outreach Activities Respondents acknowledged UBA’s efforts to ensure key targeted populations were capacitated and participated meaningfully in the UBA project activities. Men, women, and youth respondents reported receiving capacity-building on health issues connected to their needs.73 Respondents also reported that UBA selected ambassadors to work with peers in the community, including male ambassadors to support education on health issues and to advocate the 70 Eight of nine FGDs with male involvement groups, six of six FGDs with youth, and 11 of 11 FGDs with women clients. 71 Nine of nine FGDs with male involvement groups, 14 of 35 KIIs with facilities, three of four KIIs with CSOs, and five of ten KIIs with IPs. 72 Two of nine FGDs with male involvement groups, nine of 12 KIIs with district/regional GOT, eight of 35 KIIs with facilities, and two of ten KIIs with IPs. 73 Two of nine FGDs with male involvement groups, five of six FGDs with youth, nine of 35 KIIs with facilities, six of ten KIIs with IPs, and eight of 12 KIIs with district/regional GOT. 54 importance of male involvement in RH services,74 women champions known as “Mama Kinara,”75 and youth peer educators from youth groups of different ages and sexes. 76 Sensitization conducted by community volunteers through community meetings was also reported to have increased communities’ awareness of health-related and GBV issues.77 This included advising fast health seeking by GBV victims and providing linkages and fast referrals to HFs. Facility and Health Worker Activities Respondents reported that UBA has built the capacity of health providers to respond to the needs of women and youth. UBA provided training and mentorship on essential packages for confidential adolescent and youth-friendly health services according to WHO standards. The essential package for GBV included post-GBV case management, GBV screening and referrals, and linkages within facilities and to key players actor in case management.78 Respondents also noted that UBA supported the implementation of special youth and adolescent mothers’ interventions at facilities, including youth-friendly corners and adolescent mothers’ groups; 79 provided joint supportive supervision to facilities;80 and continues to provide on the job training to facilities on Respectful Maternal Health Care.81 “We have mother support groups at RCH and Ariel clubs, and facilitate their meetings once a month. Meeting with their peers helps them to open up and share their experiences about RH and HIV issues. During meetings they engage in dialogue to understand gender barriers in the uptakes of FP, TB, and HIV including adherence and retention.” (Kilimanjaro, Female, district/regional GOT KII) Advocacy and Policy Support At the national government level, UBA has participated in several gender TWGs to advocate, review, and update gender issues in appropriate national policies and guidelines. The UBA activities most commonly mentioned by respondents include facilitating the mainstreaming of gender issues into the National Strategy for Maternal and Child Health,82 reviewing national guidelines for GBV/violence against children (VAC) and developing SOPs.83 Respondents noted that UBA regularly brought gender-related expertise to TWGs on FP, MNCH, TB/HIV, Pediatric HIV, and adolescent SRH (two of ten KIIs with IPs, four of 17 KIIs with national GOT). “We participate in the national technical working groups and share our experiences on gender issues as a barrier to access and utilization of related health services and advocate for gender 74 Six of ten KIIs with IPs, five of eight FGDs with CHWs, seven of 35 KIIs with facilities, nine of 15 KIIs with National GOT, and seven of 12 district/regional GOT. 75 Four of ten KIIs with IPs, three of four KIIs with CSOs, four of 35 KIIs with facilities, and six of 15 KIIs with National GOT. 76 Eight of 12 KIIs with district/regional GOT, ten of 35 KIIs with facilities, six of six FGDs with youth, and six of ten KIIs with IPs. 77 Twelve of 12 KIIs with district/regional GOT, 16 of 35 KIIs with facilities, six of ten KIIs with IPs, and 12 of 15 KIIs with National GOT. 78 Two of ten KIIs with IPs and five of 12 KIIs with district/regional GOT. 79 Three of ten KIIs with IPs and two of 17 KIIs with National GOT. 80 Two of ten KIIs with IPs, three of 17 KIIs with National GOT, and one of five KIIs with USAID. 81 See Appendix I of the Evaluation Policy and the Evaluation Report Review Checklist from the Evaluation Toolkit for additional guidance. 82 EQs will be answered and used to guide the evaluation’s findings, conclusions and recommendations; sub questions are illustrative and will be used to guide instrument design but may not be directly answered by the evaluation. 83 Program areas are HIV, TB, MCH, FP, Malaria, Nutrition, gender, youth, and GBV. 55 equality in program interventions. Some of the ongoing gender issues that we are advocating for includes the issue of Age of Consent. Current guidelines on pediatric HIV stipulate that youth will continue to be under parent/guardian’s guidance until the age of 18. However, in implementation we are experiencing the issue of mature minors and the guideline is limiting them to access HIV/AIDs services. We therefore advocate for the government to see the need of reviewing Age of Consent.” (National, Female, IP KII) “We are working closely with UBA Lake/Western Zone to incorporating gender issues in nutrition policy and guidelines.” (National, Female, National GOT KII) “In relation to GBV, we are pushing on the agenda of developing a national standardized curriculum for on the job training in post GBV management.” (National, Female, IP KII) There were variances in the ways men and women reported barriers to women’s care. Women more often reported men’s attitudes as a barrier to men seeking care than men. The issue of distance was mainly reported by women, while time spent waiting for services and an unfriendly environment was much more of a concern for men. Some differences in reported barriers to care were also visible across geographic boundaries. In the Southern Zone, female clients reported stigma as a barrier more than women in Lake/Western and North/Central Zone. In Mara and Zanzibar, women reported that men were their biggest barrier. In Lake/Western Zone, social/cultural norms and traditions emerged as a key barrier preventing men from accessing and utilizing health services. In the North/Central Zone, low knowledge of reproductive issues and practices was a strong barrier. Facility infrastructure was reported more often as a barrier by women in the Kilimanjaro region than in any other location. 56 5.0 SUSTAINABILITY PROSPECTS AND CHALLENGES 5.1 PROSPECTS FOR UBA SUSTAINABILITY Institutionalization of the program is assisted by the government facility focused design. The prospects for sustainability in the context of UBA are, by design, furthered by the facility focus of the interventions across the three zones. Following the support for integration at the district level and the facility, there is some confidence that this integrated approach will continue after the program and that the TA will continue to be cascaded and reinforced. This is in part due to ongoing mentoring and supportive supervision from district and council government staff and senior practitioners embedded in facilities. With some reservations due to staff salaries and equipment upkeep, systems strengthening in labs and for data clerks at facilities is also said to be sustainable by facility staff and CHMTs (KIIs/GIs with five facility staff and four CHMTs). Practices at the facility level are guided by and reinforced by national guidance. RHMTs and CHMTs who have been trained to cascade this guidance in the facilities. UBA IPs have contributed to the national guidance, SOPs, and tools for client data gathering and reporting. MNCH guidance and tools for example have been strengthened by Jhpiego in mainland and Zanzibar and are driving practices in facilities. Trained practitioners and mentors were said to be likely to continue services in facilities (four of 11 facility staff in Lake and Western Zone and Southern Zone). TA shows signs of sustainability at the facility level in that the practitioners who are mentors and mentees are part of the formal government facility and are supported by those facilities in their ongoing work (according to 11 of 35 facility staff across the zones). Some IPs (two of 11 IP KIIs) and CHMTs (two of five GIs) felt that government reinforcement of ongoing supportive supervision would continue if USAID assistance were to discontinue. CHMTs in Southern Zone are prepared to continue support for integrated health at the council and facility levels (according to two RHMTs and two CHMTs in the Southern Zone). As a government staff from Iringa states, “…we can proceed even if the project phases out. Because we have been participating in planning. I believe, therefore, because we planned everything together; there are things, we learned from them. When Boresha Afya leaves, we know its budget since we were involved in creating our CCHP. This is something we can do because we have adopted their ways of doing things through our role model during the Boresha Afya activities planning.” This will depend on successful planning and council government’s commitment of budget to continue supportive supervision and mentoring. 57 IPs cited the technical training and systems strengthening for national agencies and LGAs as a strength in support of sustaining practices and use of equipment and supplies following the program. This includes support for labs and data clerks in maintaining data entry into registries and the DHIS2 system. As one Southern Zone IP stated, “The program works very closely with [National AIDS Control Program] NACP who are the ones dealing with HIV…they will be sustainable because they are under the structure of the ministry’s database. In addition on sustainability, every year the program conducts a training package on sustainability for all councils, and CSOs in Morogoro aimed at enhancing capacity on what they will do and how they will sustain when Boresha Afya phases out…even when the project will phase out the government health service providers will continue providing effective services because they have undergone quality training from Boresha Afya.” Boresha Afya has been working with LGAs to institutionalize practices in integrated health systems strengthening and has begun sustainability planning. Planning and working directly with district/council CHMTs has led to strengthening of management and supervision of integrated care in facilities. Some concerted efforts to develop Sustainability Plans with local LGAs have improved GOT buy-in and commitment in councils with UBA-supported facilities. Sustainability training and planning sessions articulate an actionable plan including sustainability strategies to maintain progress on goals, objectives, and performance targets to ensure resources are available to operationalize the sustainability plan and to schedule actions and steps while accounting for risks and issues. Sustainability planning has become more important in cases of councils where PEPFAR’s targeted funding is not likely to continue before the close out of the program. Regional and council government authorities trained by Deloitte noted the value of this training and planning process and that it has helped to inform their actual budgeting for the financial year. As stated from a regional official in Iringa, “On capacity building, Boresha Afya has helped the councils in Iringa to see if they leave if at all there will be sustainability. So we GOT excellent trainers from Deloitte who trained us on the sustainability plan, and the good thing is that in the budget form the council this year that is ending after they came to do the training to check if what they taught is being implemented in our plans. We, later on, came to realize 90 percent planned activities are in the plans, so they helped us a lot in strategic planning mainly on sustainability.” While planning in some zones seems to have improved chances of sustainability, what is less certain in these plans are the shifting of financial responsibility to GOT budgets if PEPFAR, PMI, or USAID support would be discontinued. As PEPFAR reduced funding and changed priorities following the changes proposed in the 2019 Country Operational Plan (COP) meeting to focus more on treatment and care, there are risks to UBA’s efforts in systems strengthening, prevention activities and commodities, community engagement, and referral system support (such as sustaining the role of CHWs). Prospects for sustainability of community efforts to support youth, male engagement, and the client referral system through CHWs are present but are fragile, again due to lack of financial sustainability at the local level. While on a technical level CSOs are optimistic having been capacitated and strengthened to continue efforts after UBA, 58 budget constraints and lack of diversity in funding put continuing services at risk (three of four CSOs). As one CSO partner in the Southern Zone states, “I can say to some extent, the project has aided the whole process of sustainability of services. As we look at all the systems and technical issues because in order for the services to be sustainable systems should be present. The project has enabled us in many ways, from the management level to the technical level meaning that the staff to have the capacity of offering such services to the community. Therefore, I can say that we can carry on but not to the extent or level of support that we used to receive from the project...” 5.2 CHALLENGES FOR SUSTAINABILITY Challenges in Financial Sustainability While BAs efforts supported by donors such as PEPFAR, PMI, and Global Fund have the national support of the GOT for integrated health and have contributed to institutionalization, systems strengthening, and HIV prevention, treatment, and care, gaps remain in readiness for national self￾reliance without the ongoing support of these donors. The share of health sector (all areas of health) budget as a proportion of the total GOT budget has declined from 10.5 percent in 2010/11 to approximately 7 percent in 2017/18. This downward trend—combined with a lack of funding for service delivery, HR, and information systems—could negatively affect the country’s ability to sustain support for integrated care in facilities currently assisted by UBA. PEPFAR, PMI, Global Fund, and USAID core funding for the interim are needed to ensure momentum in addressing the causes of child and maternal death and toward epidemic control to reach the 95/95/95 goals for HIV. This evaluation concurs with PEPFAR’s 2018 assessment that domestic resources and technical and allocative efficiencies for HIV/AIDs are currently unsustainable. Tanzania does not adequately generate the necessary financial resources for HIV and AIDS, to ensure sufficient resource commitments, and use data to strategically allocate funding and maximize investments. (PEPFAR/Tanzania, April 2018). The United Nations (UNAIDS and UNFPA) and World Bank provide a small proportion of additional funding for HIV and MNCH, but there are currently no other bilateral donors and few international NGOs providing HIV support (PEPFAR/Tanzania, April 2018). Funding for integrated health in conjunction with HIV/AIDs will require continued USG/PEPFAR and Global Fund support unless there is a significant shift in the resources allocated from the national government and other donors. Human Resources Challenges and Incentives for Volunteers in GOT Facilities for CSOs HR challenges limit institutional readiness in government facilities and limit long-term sustainability. Nationally in the mainland and Zanzibar, HRH shortages as well as issues with organization and the skills and abilities of HCWs remains a factor of concern and may mean that scale up is needed before succession planning should be considered. Sustainability Challenges – Coverage Gaps In Southern Zone, North and Central Zone, and Zanzibar, participants explained that coverage gaps remain a barrier to self-reliance. Coverage gaps remain evident in mainland Tanzania and Zanzibar, particularly in MNCH and FP. According to the 2019 HSSP MTR, 59 “[t]he availability of emergency obstetric care facilities has improved in hospitals, but not in health centers, by 2017. The increase was mainly in 2017-18. Problems are still inadequate human resources (e.g., anesthetists) and equipment (e.g., blood bank).” (p. 29) In North and Central Zone, stakeholders (one of two IPs and one of 17 national GOT KIIs) discussed limitations in funding to fill gaps in coverage as many facilities are overburdened and cannot meet the demand for services with the available staff and supplies. As an RHMT member in Kilimanjaro stated, “Yes, with our project we always have limitations with our funds. So, you won’t cover all the facilities. We only look at the facilities with higher demands. As I said, you can compare Mawenzi or Mount Meru with our facilities. So, you will find that, with us, efforts go to the facilities which we know have high burden of patients. So, the gap remain to other facilities, because we are looking at the equity of the services accessibility. But that is not the case because the small fund we have we direct to the places with high volume of clients and demand.” National authorities in Dodoma also discussed that certain health areas such as nutrition need further support in mainland. There is also evidence that particular coverage is needed in nutrition to “ensure the availability of nutrition desks in district and regional hospitals.” Coverage gaps, if evident in mainland, are perhaps greater in Zanzibar where the program has had less time and activity. Efforts in Zanzibar while performing well in the facilities where they are present are not yet to scale across the country. The focus has been on the six high-volume facilities compared to the 177 facilities in Unguja and Pemba overall with little to no coverage at the HC or dispensary level for primary care. Gaps remain, particularly in trained personnel and equipment for emergency obstetric care, according to four of five interviews in Zanzibar with national government authorities (two of two), (one of one) IPs, and other national stakeholders (one of one). Differences between UBA models of support to communities and the referral system are a sustainability factor. UBA zonal differences in activities and approach to support CHWs through CSOs, versus directly through LGAs for instance, was questioned by national stakeholders and other donors (two of seven) as well as USAID. One national donor staff discussed that CSOs are often under￾supported to carry on their work after the umbrella NGO with the prime contract goes, however they are the ones who are likely to sustain efforts after the project ends. Comparing North and Central Zonal models and Lake and Western Zonal models she stated, “I was looking at the way they [EGPAF] deliver these integrated services they use sub grants, subcontracts to do some of the interventions according to their cooperative agreements, but I don’t see those CSOs…I don’t know…being capacitated to be able to continue in the longer term. You don’t see these capacitated to be able to do the same or do the integrated kind of services that Jhpiego through their grant were implementing in a certain district or region…maybe they can write another proposal using the same model to be able to do what they were doing under Jhpiego or continuing some of these efforts. It’s like you have all these sub grantees but at the end of the day the prime grantee goes…And you look at the sustainability of such an approach… it’s not working.” (Staff, CDC) 60 5.3 SUSTAINABILITY FACTORS FROM THE NATIONAL CONTEXT Critical factors for the sustainability of UBA’s goals and objectives are the performance and sustainability of the national government’s national health policy and the HSSP IV. While there have been some significant gains nationally in health outcomes, especially in malaria and addressing the causes of under 5 child mortality, according to the HSSP MTR, financial and systemic sustainability concerns remain. There is a lack of sustainability in the Health Financing Strategy and the results-based financing (RBF) system which puts focus on putting money toward results rather than inputs to health facilities. RR suggests that the Health Financing Strategy should be put back on the agenda, and innovative types of funding for health services need to be elaborated and implemented. According to the MTR of the HSSP, the “RBF faces both financial and institutional sustainability issues. There have been consistent delays in payments to health facilities…Part of the reason for delays was insufficient cash at the [Ministry of Finance and Planning] MoFP to advance payment to health facilities before receiving funds from the WB. Also…Non-release of fund from MoFP has impeded the implementation of the program as planned.” (HSSP MTR, p. 53) The MTR sites a need for long-term sustainability of ICT and data systems strengthening. “At this moment there is quite some funding available for development of ICT and start of new systems. However, there is insufficient funding to attract competent ICT personnel (also at lower levels in the health system), procurement and replacement of equipment, servers, etc. The development is very much dependent on external funding. In the National Digital Health Strategy, the long-term sustainability needs to be strengthened.” (HSSP MTR, p. 47) Gaps between urban and rural health outcomes are increasing, which may be a sustainability risk for UBA and other health programs if left unaddressed. For example, under-5 mortality and infant mortality in Tanzania continued its decline, although the gaps between urban and rural children and between the poorest and richest children are increasing. FP gaps remain overall for youth and adolescent girls, but especially for rural adolescents. UBA continues to focus on high disease prevalence regions and high volume facilities (in line with the national health strategy) and addresses the areas of greatest need. However this approach may result in underserving rural wards and villages which also have high prevalence of communicable disease and preventable deaths. National trends suggest that for HIV and TB, epidemic control is a necessary precondition to sustainability and self-reliance. According to the MTR of the HSSP, “the HIV/AIDS national prevalence rate has declined slightly overall and among youth, however, young however young women aged 15-24 still have a considerably higher prevalence than young men (2.2 percent and 0.7 percent respectively). While TB treatment success rates have increased to a high 90 percent, the tuberculosis (TB) case detection rates in 2018 were well below target (50 percent). The TB notification rates declined until 2015 but increased from 128 to 140. This is not necessarily due to an increase in TB cases but could be due to improvements in case detection. (pp. 31-32) 61 Stigma in communities and facilities remains a barrier to seeking testing and treatment services in both the cases of HIV and TB. The risks associated with both viral epidemics spreading are a risk factor for long-term sustainability. National self-reliance in controlling the disease internally may be a far-off goal as mentioned in the discussions on financial sustainability and health financing strategy. 62 6.0 CONCLUSIONS 6.1 EVALUATION QUESTION 1 • The structure of the funding streams makes management of the UBA contracts challenging for the USAID Agreement Officer Representatives (AORs) and health office technical staff. . They all have the same results framework and deliverables but different interventions. AORs’ roles are focused on individual IPs and contract mechanisms and there is no single central coordination role among them, which makes collaboration and information sharing between zones and health areas difficult. It also makes the reporting process to Washington challenging in that the reporting is not geared toward integrated activities but non-integrated initiatives and distinct information systems (IPRS and DATIM). • Although there were initial differences in organization approach, intra-consortium collaboration is functioning more effectively. Inter-zonal collaboration between IPs is still inadequate except through informal and technical level sharing. Information is not centralized or shared in real-time among IPs as originally conceived in the design. DE is contributing significantly through operations research and information. However, the award is limited in being able to provide a wider scope with actual real time monitoring or being able, or given the mandate, to consolidate M&E data from all three UBA IPs. Although there are hopeful recent advances, each IP has its own Monitoring and Evaluation (M&E) team and is largely invested in reporting on performance on an annual or quarterly basis. While IPs are in some cases (e.g., President’s Emergency Plan for AIDS Relief [PEPFAR] collaboration framework) harmonizing their efforts with community-based partners, there remain signs of competition and lack of information sharing between prime IPs. • Coordination with the national government could have been stronger. The UBA program was designed with a communication framework for UBA and the national GOT, but this was not operationalized, which resulted in a lack of cohesion and UBA speaking with three separate rather than a single UBA voice on IHC. • Coverage within targeted councils is generally focused on high-volume facilities. EGPAF and Deloitte have selected facilities in their regions based on volume and have organized their interventions in tiers according to PEPFAR guidance. Jhpiego offers the same package of interventions in all facilities, irrespective of facility size. Service gaps remain in dispensaries and in Zanzibar (where there is only intervention at high-volume hospitals). A concerted and coordinated expansion strategy and plan for scale up of IHC models has not been developed among the IPs or with national GOT. • At the regional and council levels UBA TA is enhancing coordination and collaboration. Regional health management teams (RHMTs) and Council health management teams (CHMTs) work with UBA staff in supporting health facilities (HF) and performing supportive supervision tasks to reinforce best practices. • Improving synergies between facility-based activities and community-based activities were observed. Community health workers (CHWs) and facility practitioners across zones are progressively strengthening referrals and follow-up for continued treatment and care. The institutionalization of the role of CHWs in facilities and in the community will depend on cascading technical support to all HFs based on new evidence and guidelines on training, incentives, tasks and realistic financings, e.g. evidence based. The success of the community- 63 to-facility integrated continuum of care will depend on the successful close collaboration between UBA, other USAID community-based projects, GOT, and involved stakeholders. 6.1.1 Recommendations for EQ 1 • UBA should develop a cross-zonal collaborating, learning, and adapting (CLA) plan with monthly IPs meeting similar to the PEPFAR collaboration framework but with in-depth and transparent information and knowledge sharing among IPs and stakeholders. • A common knowledge management system and real-time database is needed that is based on IHC indicators and tools to monitor these activities under the contract mechanisms. UBA should improve use of IPRS for performance reporting, and geographic information system (GIS) and dashboards for improved tracking of integration in service delivery. Cost effectiveness specific to IHC should be tracked on an ongoing basis to better understand the value of activities. • A functional mechanism or structure needs to be instituted whereby UBA plans and coordinates implementation and reporting with USAID and national GOT in a harmonized and a collaborative manner.84 6.2 EVALUATION QUESTION 2 • The aim of the design of the CHSD that guided UBA was to work as three components of a seamless zonal project focused on facilities with community partnerships. However, the submitted technical proposals had significant and unique differences in targets and activities. Consequently, comparing the monitoring of performance and the measures of achievement in each zone, while incorporating their individual approaches, interventions, and results is challenging. • The three UBA IPs have been adaptive and timely in response to numerous changes in United States Government (USG) administration and funding, GOT policy changes, and changing conditions in regions, councils, and facilities. IPs changed standard operating procedures (SOPs), strategies, and procedures to adapt to changes in partnership with GOT partners which enhanced performance. • The structure for UBA funding through PEPFAR, the President’s Malaria Initiative (PMI), and other sources competes with an integrated care approach in that each source has siloed targets. Finances are available only for those activities which align with these targets. While PEPFAR has its own integration agenda including systems strengthening, reproductive health (RH) and TB integration with HIV as its central component with the facility HIV/AIDS care and treatment clinic (CTC) as its focus, it sometimes overshadows other efforts outside of the HIV/AIDs context. PEPFAR’s target councils change year to year based on HIV epidemic control leaving other funding streams’ activities to readjust in the wake of the annual PEPFAR Country Operational Plan (COP). The ET recommends that the HO develop an integration approach through 1. Increased effectiveness and simplified management; 2. Define geographic outcomes in a select number of regions where system strengthening, facility and community 84 One approach to build upon MoHCDGEC sector wide approach dialogue structure for technical information sharing across partners in the sector. 64 services are to benefit a defined target population and 3. Increase cost efficiency and gathering evidence of the most effective IHC models. • This evaluation has seen evidence of efficiencies of an integrated approach at the level of service delivery and for reducing clients’ opportunity costs for receiving multiple services in fewer visits. Health systems strengthening at the national, regional, facility, and community levels plays a crucial role in successful vertical integration. • The IPs did not monitor the cost-effectiveness of various integration approaches or activities and have not developed a common system to share evidence on cost effectiveness. Cost￾effectiveness data from the previous projects appear not to have been used as a benchmark to make cost-effective decisions. Because the IPs have different sets of activities, it is challenging to compare the overall cost-effectiveness of IPs and their individual IHC models. Such a comparison would have between highly informative to guide IHC decision-making. • While the effectiveness of the development evaluation (DE) as a real-time feedback mechanism is less clear, its role as a tool for learning and adapting has shown more recent gains in collaboration and consortia, particularly during the September reporting of cross￾cutting findings for performance year (PY) 2. While DE may have had difficulties adding value in its first years of implementation, it has the potential to be a productive tool for learning and decision-making. 6.2.1 Recommendations for EQ 2 • This evaluation, while not conclusive on the issue of effectiveness of integration, has seen reported evidence of benefits of an integrated approach for clients. On the other hand, health systems strengthening at the national, regional, facility, and community levels should be a central component of any follow-on design. Contract mechanisms should allocate core funding to assist with coordination yet be flexible enough to allow for adaptations and innovation while it induces necessary IP collaboration and cooperation. • Community engagement in strengthening the referral system and treatment and care follow￾up of chronic health conditions should be integrated into a community health care system in partnership with GOT and its stakeholders. • The use of information and communications technology (ICT) technologies and current ICT innovations in the health field need to be integrated into client health care including community outreach, referrals, tracking and follow-up for successful IHC from both horizontal and vertical perspectives • As stated under EQ 1, support of CHWs including volunteers and their specific roles and responsibilities in community health should be explicit in a manner that will ensure their ability and motivation to participate and meet GOT health care standards, and in a manner without undue opportunity costs or personal financial burden. USAID and UBA should begin to negotiate with LGAs and national GOT as a part of succession planning to ensure budget commitments can support the central role of CHWs in providing community health especially in regard to already initiated IHC activities. Their involvement, as well as costs, need to be informed and aligned with current evidence and debate on the role and conditions of CHW’s contribution to health care and disease prevention in their communities. • The DE should be supported to expand its operations research approach at local levels to include higher-level IHC learning and knowledge sharing. 65 6.3 EVALUATION QUESTION 3 • Among health management teams in the regions and districts, there was a high level of satisfaction with UBA TA. CHMTs reported that regular supportive supervision and mentoring were being conducted in the facilities. • Facility staff, including supervisors and practitioners, were satisfied with UBA TA and have observed changes in the quality of care in facilities. • Sampled clients reported increased access to more than one service during their visits at facilities. They also reported satisfaction with the quality of integrated services. • While a high level of satisfaction across stakeholders indicates TA was significant, there are some remaining gaps in specialized training and managerial skills for supervisors. 6.3.1 Recommendations for EQ 3 • Continuing medical and pre-service training in specialized topics for treatment and care in primary and secondary facilities, e.g., emergency obstetrics should be provided to address the prevention of child and maternal death. • The continued use of district mentors and professional exchange visits from experienced IHC practitioners should be supported in follow-on activities and replicated in other zones to improve the skills of other facility staff. • Assessment approaches in facilities such as those used by Jhpiego could be used for organizational development and rebalancing staff resources in high performing facilities to address Human Resources for Health (HRH) challenges in other facilities. • UBA should continue to support GOT in planning and budgeting to recruit and develop health care workers (HCWs) and CHWs as well as contribute to training costs to ensure ongoing institutionalization of informed IHC practices including pre-service training according to GOT guidelines. 6.4 EVALUATION QUESTION 4 • Respondents reported stigma as the most common barrier for women and youth in accessing health services. Other major barriers faced by women and youth include normative systems that hinder these and other groups from accessing health services. Male partners too often prevent women from accessing and utilizing health services and need to be engaged in resolving this challenge and there are still gaps in gender-based violence (GBV) responses at the community and facility levels. Improved and responsive coordination between HFs, law enforcement, social services, and communities is required for the long term. • UBA interventions such as male engagement and youth groups have been implemented across zones but need to be expanded and intensified with community engagement across activity sites. The monitoring of gender and youth outcomes in accessing and using services remains limited. 6.4.1 Recommendations for EQ 4 • The gender, GBV, and youth-friendly services components of the UBA program should be more focused and better funded in order to meet the persistent health needs of women and youth. Their participation and that of their communities should be central in this effort. In addition, facilities, police, judicial, social support systems, and communities need to be mobilized to respond quickly to incidences of GBV. 66 • To address the gravity of social norms and institutional stigma challenges, social and behavior change communication (SBCC) needs to be sustained with continued force if not intensified to reach more women, youth, and their communities through direct and participatory activities. Community engagement and work through influential community members/peers and religious and traditional leadership is key to expanding awareness and active responses. • Facility HCWs and outreach staff, (e.g., CHWs) can play a central role in addressing gender and youth barriers to service delivery as agents of change in their interactions with clients as well as in communities. Towards this end, UBA should ensure that these staff are actively oriented and engaged in these issues through its mentoring and capacity building activities. 6.5 SUSTAINABILITY • Various challenges to institutional and financial sustainability are evident across findings. Sustainability or succession planning has become more urgent in the case of councils where PEPFAR’s targeted funding is discontinuing before the close of the program. To mitigate risks, UBA in the three Zones are starting to actively work with RHMTs and CHMTs to institutionalize practices in integrated health systems strengthening. Formal sustainability planning with LGAs has started in Southern Zone. 6.5.1 Recommendation for Sustainability • Best practice sustainability training and planning sessions should be investigated and replicated by IPs across regions and councils to better articulate actionable strategies and ensure resources are available in GOT budgets to sustain UBA IHC practices. This will depend on effective capacity-building of GOT and facility managers to embed IHC into the GOT Direct Health Facility Financing Model over the long term. 67 ANNEXES 68 ANNEX 1: SCOPE OF WORK PURPOSE OF THE EVALUATION The purpose of conducting this midterm performance evaluation is to learn from and document: i. Whether the three USAID Boresha Afya awards are on track to achieve their program goals; ii. What are the program service coverage and uptake by region and districts; and iii. How effective is overall oversight and management across the UBA, within and from USAID to the implementing partner then management within the mechanism over field activities in targeted regions This information will be used by the USAID Health Office in the design of future awards. SUMMARY INFORMATION Instructions: Utilize this section to describe the activity/project/program being evaluated. There are two suggested formats. Option 2: For projects/programs with multiple implementing partners, including for a sector or thematic evaluation Activity Name USAID Office Implementer Cooperative Agreement/ Contract # TEC Life of Project/ Activity Active Geographic Regions Mission DO Public USAID Boresha Afya Lake/ Western Zone USAID Health Office Jhpiego AID-621-A-16- 00003 $56,450,000 Five years Geita, Kagera, Kigoma, Mara, Mwanza, Shinyanga, and Simiyu plus Zanzibar DO 1, IR 1.2 No USAID Boresha Afya North/ Central Zone USAID Health Office Elizabeth Glaser Pediatric AIDS Foundation (EGPAF) AID-621-A-16- 00004 $123,029,607 Five years Arusha, Dodoma, Kilimanjaro, Manyara, Singida, and Tabora DO 1, IR 1.2 No USAID Boresha Afya Southern Zone USAID Health Office Deloitte Consulting Ltd. AID-621-A-16- 00002 $138,282,996 Five years Iringa, Lindi, Morogoro, Mtwara, Njombe, and Ruvuma (malaria only) DO 1, IR 1.2 No BACKGROUND Instructions: Provide a detailed description of the context, history, goals, and objectives, current status of the activity/project/program, and other relevant information to help the evaluation team understand the design and implementation plan. Complete the sections noted below. Sections can be consolidated. 69 In 2015, USAID/Tanzania had eight facility-based awards of which several were vertical, disease￾specific, and stove-piped. The drawbacks of this approach included multiple uncoordinated visits, duplication of effort, and inefficient use of resources. In an effort to overcome these challenges, USAID developed and procured an integrated follow-on program, namely USAID Boresha Afya, that is being implemented by three awardees. The USAID Boresha Afya (UBA) program was designed to support the Government of Tanzania (GOT) to increase access to quality comprehensive and integrated health services with a focus on reproductive, maternal, neonatal, and child health (RMNCH) and nutrition outcomes. The program is implemented in three zones of Tanzania by the following lead implementing partners: (1) Lake/Western Zone—Jhpiego; (2) North/Central Zone—EGPAF; and (3) Southern Zone—Deloitte Consulting Ltd. Interventions implemented by USAID Boresha Afya include counseling and provision of comprehensive family planning (FP); HIV testing services; prevention of mother-to-child transmission of HIV (PMTCT); early infant diagnosis (EID); adult and pediatric care and treatment; home-based care; TB/HIV integration; TB diagnostics and treatment (including multi-drug resistant TB); malaria diagnostics and treatment (including intermittent presumptive treatment in pregnancy [IPTp] and distribution of insecticide-treated nets); maternal and newborn health (including antenatal care [ANC], normal delivery, basic and comprehensive emergency obstetric and neonatal care [BEmONC and CEmONC], care of the preterm neonate, and postnatal care [PNC]); child health and nutrition and integrated management of childhood illness (IMCI). All programs have facility and community components to improve service delivery linkages for the client. Description of the Problem, Development Hypothesis(es), and Theory of Change “If Tanzania empowers its women and youth, sustains inclusive broad-based growth, and makes governance more effective, its socioeconomic transformation toward middle income status by 2025 will be significantly advanced.” Instructions: Include details on: --The specific problem or opportunity the activity/project/program to be evaluated was designed to address; --The development hypothesis(es) often expressed as an if/then statement; --The theory of change that underlies the design (including a list of the intended results and critical assumptions). A. Results Frameworks The three USAID Boresha Afya awards are authorized under USAID/Tanzania’s CHSD Project Appraisal Document (PAD). All activities under the CHSD PAD are critical to the achievement of USAID/Tanzania’s Country Development Cooperation Strategy (CDCS). The CDCS emphasizes implementation of an integrated approach across multiple sectors, engagement with a diverse pool of state and non-state actors at local and national levels, and support and resources for key geographic areas, segments of the population, and systems, institutions, and entities to sustain inclusive, broad-based development. All three Development Objectives (DOs) of the 70 CDCS address the needs of women and youth, strengthen local government, build health and education systems, and promote economic growth. In late 2014, USAID/Tanzania began implementing its approved CDCS for the period October 2014-October 2019. The CDCS posits that if Tanzania empowers its women and youth, sustains inclusive broad-based growth, and makes governance more effective, its socio-economic transformation toward middle-income status by 2025 will be significantly advanced. The DOs that support this hypothesis are: DO 1: Tanzanian women and youth empowered; DO 2: Inclusive broad-based economic growth sustained; and DO 3: Effective democratic governance improved. Flowing across the three DOs is a Cross-Cutting Intermediate Result (CCIR): Data-driven decision-making, planning, and implementation improved, which focuses on advancing systems and actions to achieve results across the Mission’s portfolio through greater and more effective data gathering, analysis, and learning. Building on the CDCS development hypothesis that Tanzanian women and youth will be empowered if they use quality health services, the CHSD PAD supports DO 1: Tanzanian women and youth empowered, IR 1.2: Health status improved. In support of the hypothesis that FP decisions are critical to improving the economic status of families and communities, the CHSD PAD supports DO2: Inclusive broad-based economic growth sustained, IR 2.4: Unmet need for FP reduced. Finally, through its support for systems strengthening and work with key GOT entities, the CHSD PAD also contributes to DO3: Effective democratic governance improved, IR 3.2: Government delivery of services improved. See the figure below for the full CDCS results framework. - - CHSD PAD Logical Framework Narrative Summary Objectively Verifiable Indicators Means of Verification Important Assumptions Development Hypothesis: If universal coverage of a minimum Goal Improved health status Malaria prevalence rate TDHS/THMIS, TMIS USG and URT demonstrate commitment to a partnership that supports sustainable programs Under-five mortality rate (per 1,000 live births) TDHS/MIS - range of specific service delivery inputs and processes are established at service Modern contraceptive prevalence rate TDHS/MIS - delivery sites, the primary causes of preventable maternal and child death can be averted Maternal mortality ratio (per 100,000 live births) TDHS/MIS - Objective Increased access to quality, integrated services with a focus on reproductive and child health. % of women who receive IPT for malaria during their last pregnancy TDHS/MIS USG, URT and private partners demonstrate commitment to a partnership that supports sustainable programs and budgeting for health needs. 71 - - Narrative Summary Objectively Verifiable Indicators Means of Verification Important Assumptions % of children who slept under an ITN the night before the survey TDHS/MIS PEPFAR and PMI continue through the duration of the program Couple years of protection in USG-supported programs Program report TDHS/THMIS will be conducted on a routine basis and data will be released in a timely manner % of live births attended by skilled health personnel TDHS/MIS Improved health service delivery system % of infants who receive three doses of DPT MOH service delivery reports (program) Improved health service delivery system % of HIV+ pregnant women who received ART to reduce the risk of mother to child transmission PEPFAR semi/annual program report Improved health service delivery system % of HIV+ pregnant women who received ART for their own health National AIDS Control Program reports Improved health service delivery system Number of HIV positive adults and children receiving ART Program progress report Increased GOT commitment to HIV/AIDS program 72 - - Narrative Summary Objectively Verifiable Indicators Means of Verification Important Assumptions -- Outputs 1.1 Increased treatment with Artemisinin-based combination therapies Number of Artemisinin￾based combination treatments (ACTs) purchased and distributed through USG support. Program reports - 1.2 Increased use of Insecticide treated nets to prevent malaria Number of ITNs distributed that were purchased or subsidized with USG support Program reports/records - 1.3 Increased use of indoor residual spraying (IRS) to prevent malaria Number of houses sprayed with insecticide with USG support. program reports - 1.4 Improved service delivery for modern contraceptives % of women 15-49 yrs. either married or in union who have their need for family planning satisfied Program reports Reduced unmet need for modern contraception 1.5 Improved availability of quality BEmONC services in program region health facilities Number of facilities providing BEmONC services per population Program reports Increase GOT commitment to quality health services 1.6 Improved availability of IMCI services in program region facilities Number of facilities providing IMCI services per population Program reports Increased GOT commitment to quality health services 1.7 Improved HIV preventive behavior and testing for HIV Number of individuals counseled, tested, and received results PEPFAR semi/annual program reports Reduced HIV stigma 73 1.8 Improved care and support for people infected by HIV Number of HIV positive adults and children receiving a minimum of one clinical care PEPFAR semi/annual program reports - 1.9 Increased access to and use of HIV care and treatment clinical services. Number of HIV positive adults and children receiving ART PEPFAR semi/annual program reports - Number of HIV-infected - - 1.10 Improved clients attending HIV care PEPFAR integration of TB and and treatment that are semi/annual - TB/HIV services receiving treatment for TB program reports disease 74 Narrative Summary Objectively Verifiable Indicators Means of Verification Important Assumptions 75 Activity Logical Framework: Instructions: Include here or as an annex the graphic of the Mission’s Results Framework and the Project’s Logical Framework (if applicable) highlighting the elements to be evaluated. If the evaluation is at the Activity level then include the Activity’s Logical Framework (and linkages to the project￾level). B. Summary Activity/Project/Program to Be Evaluated A Summary Description of Boresha Afya North/Central Zone Approach: This activity is focused on the provision of comprehensive health service delivery for preventative and curative services in FP, HIV/AIDS, and TB. This activity currently does not support MCH or Malaria activities. This activity is being implemented in the following geographic areas with the highest disease burden across the following six regions: Arusha, Kilimanjaro, Manyara, Dodoma, Singida, and Tabora. Implementation: Family Planning interventions are provided at all 44 councils in six regions. By the end of the third quarter of PY2, a total of 841,348 clients were served with different FP methods, with injectables followed by implants as the preferred methods. The program emphasized councils with a higher unmet need for FP and a higher volume of patients. Based on 2015/2016 DHS data, the 76 contraceptive prevalence rate was lowest for Tabora (21%) and Manyara (28%) regions in the North/Central Zone. Boresha Afya North/Central Zone supports youth to receive sexual and reproductive (SRH) services, which included FP information and services, HIV testing and treatment, as well as sexually transmitted infections (STI) screening and treatment. These services are provided at Reproductive and Child Health (RCH) clinics. In addition, care and treatment clinics (CTCs) at health facilities have established teen clubs for adolescents living with HIV (ALHIV) who meet one Saturday per month to receive comprehensive HIV and SRH services from trained health providers. Peer educators also conducted health education to fellow peers at the facility and community level to mobilize groups. The program supports HIV services—including HIV testing, PMTCT, EID, and pediatric and adult care and treatment—at 420 health facilities in the above mentioned six regions. By the end of Quarter three of PY2, a total of 196,062 pregnant women and breastfeeding mothers were HIV counselled and tested, 4948 HIV exposed infants were HIV tested and 136, 812 HIV clients were current on ART services The project ensures access to TB diagnosis and treatment and supports diagnostic centers with LED microscopes, conventional microscopes, and GeneXpert machines in four regions except Kilimanjaro and Arusha. During PY2Q3, 2,798 TB patients were notified and put on treatment. In addition, the project supported 1921 sputum samples transportation for GeneXpert examination in all four regions. To strengthen TB/HIV collaborative activities, the project supports IPT services to prevent TB in PLHIV in 255 sites: 40 in Dodoma, 33 in Singida, 61 in Kilimanjaro, 24 in Manyara, 35 in Arusha, and 62 in Tabora. A Summary Description of Boresha Afya Lake/Western Zone Approach This project is aimed to improve the health status—with a focus on women, youth and children— by improving the availability of, and access to, quality, respectful, and integrated health services. The project is implemented in seven regions of the Lake and Western Zones namely Mara, Kagera, Kigoma, Shinyanga, Geita, Simiyu, and Mwanza; and six districts in Zanzibar. The project supported 1,817 health facilities (HFs) across the seven regions and 5HFs in Zanzibar: 809 of these were expanded sites receiving the high intensity intervention package, and 1,014 were essential sites receiving the minimum package Implementation Maternal and Newborn Health, comprehensive MNH programming is implemented in three regions: Kagera, Mara and Geita. The project recorded a 19% increase in first ANC visit before 12 weeks of gestation, from 20% to 39% in—closer to the national target of 40%. Coverage of 4th ANC visit increased to 59% from 49%. Meanwhile, the quality of ANC services has improved; there is a notable improvement in the overall performance of EmONC standards from 54% to 70% in Kagera, and from 55% to 66% in Mara, as a result of regular QI assessments and mentorship. Project has increased initiated CEmONC services, from 31 to 45 sites. The Project has also scaled up KMC (Kangaroo mother care) services to 35 health facilities (HFs). Institutional deliveries have also increased by 16% on average in the three regions. There is an overall 77 decreasing trend in the number of reported maternal deaths in these HFs, and maternal and perinatal deaths are being reviewed at significantly higher rates in the 45 CEmONC facilities, from 24% in 2015 to 80% 2018 in Kagera and from 26% in 2015 to 75% in 2018 in Mara. For Child Health, 98.6% of the 1,817 HFs were visited by the project, and 14,330 cases of under￾five were assessed in Geita, Shinyanga, Kagera, Simiyu, Mara and Kigoma. 1,999 HCWs in 778 health facilities in those regions were equipped with IMCI knowledge and skills through D-IMCI training. This training has led to direct improvements on key IMCI indicators; providers’ performance on correct treatment improved from 47% to 60%, correct diagnosis improved from 25% to 40%, and in recording of danger signs and weight from 21% to 58%. The project assessed 4,384 children aged 0-59 months for their nutrition status through integrated assessments and a total of 31,172 were screened for HIV. Those found to be HIV+ or malnourished were linked to appropriate care and treatment The project also supported Family Planning services in all seven regions, covering 1,639 HFs and surrounding communities, and as a result, recorded a cumulative uptake of services of 1,218,070, compared to the annual benchmark of 1,079,351 with Couple’s Year Protection (CYP) of 2,231,298. The project also registered improvement in PPFP counseling, reaching 90% in three regions; And the proportion of adolescents and youth (10-24 years) who received FP services increased by 4% by the end of year 2, about 48 hospitals (RRHPs & DHs) from project regions were already providing FP services in CTC sites. A total of 4,887 CTC clients were counselled on FP and 3074 (63%) were served on FP methods of their choice. Malaria achievements included strong gains in the proportion of providers demonstrating adherence to negative test results, increasing from 76% to 92%. A total of 1314 facilities were mentored adherence to national guidelines on malaria case management, to prevent the occurrence of severe morbidity. MSDQI visits, action plans, a targeted mentorship based on performance data, as well as data quality support, formed the engine of the project’s intervention. As a result, there has been a decrease in the number of patients treated based on clinical diagnosis by 68% from 35,161 in 2017 to 11,249 in 2018; the project has seen an increase by 10% points in overall malaria testing for pregnant women, there is a notable increase in IPTp2 from 58% to 70% and IPTp3 from 20% to 43%. A Summary Description of Boresha Afya Southern Zone Approach This activity is focused on the provision of comprehensive health service delivery for preventative and curative services for technical intervention areas (FP, HIV/AIDS, TB, malaria, and MCH). This activity is being implemented in the following geographic areas with the highest disease burden across the following six regions: Iringa, Lindi, Morogoro, Mtwara, Njombe, and Ruvuma for malaria only. Implementation USAID Boresha Afya Southern Zone project implemented its thematic areas through the three results areas namely; Improved enabling environment for health services provision; Improved availability of quality, integrated health services at facility level; and Increased access to health services at the community level. The project closely collaborates with host government through 78 the Ministry of Health, Community Development, Gender, Elderly and Children (MOHCDGEC), President's Office-Regional Administration and Local Government (PO-RALG), other USAID￾Funded Projects as well as with other implementers and stakeholders. USAID Boresha Afya-Southern Zone Project supports provision of HIV/AIDS services at 599 facilities with services ranging from HIV testing, ART and PMTCT services in the five regions of Iringa, Njombe, Morogoro, Lindi and Mtwara. Among the supported health facilities, 381 (64%) are in 18 PEPFAR scale-up councils while 218 are in xxx sustained councils. As of FY18 Q3, a total of 484,528 clients were tested and received results for HIV in both scale up and sustained councils. A total of 10,920 clients were diagnosed to have HIV infection with the positivity of 2.3%. New clients initiated on ART during the reporting period were 9,802, which is 49% of the annual target (20,004). These newly initiated on treatment clients have contributed to the program reaching a total of 175,298 clients currently on ART in the five supported regions. In TB and TB/HIV, 90% (157,380) of the currently on treatment clients were screened for TB where 2851 (99%) of TB presumptive cases were referred for TB diagnosis and 389 (84%) of the diagnosed TB were initiated on anti-TB drugs. In PMTCT services, the program supported the same 599 health facilities to counsel, test and provide HIV results to a total of 40,245 (99%) among pregnant women attended at RCH clinics. Family planning services continued to be supported at 1,124 health facilities across five regions with 45% of them providing integrated FP/HIV services as one stop shop. The program supported these facilities to provide malaria diagnostic services (mRDT, Microscopy or clinically) to a total of 992,017 suspected malaria cases. Gender & Youth focused care had been an integral component during the implementation, monitoring & evaluation of USAID Boresha Afya – Southern Zone Program. Gender perspectives and youth care were implemented as a cross-cutting strategy for all the three program results areas. Instructions: Summarize the primary interventions or tasks implemented by the activity/project/program. Also include a summary of any substantive changes (modifications) in the evaluated activity/project/program and when they were effective. Describe the specific geographic areas in which the activity/project/program operates and/or targeted groups, as applicable. Attach maps if available. C. Summary of the Activity/Project M&E Plan The CHSD development hypothesis is that if universal access to a minimum range of specific service delivery inputs and processes are established along the continuum of care from service delivery sites to the community, then the primary causes of preventable maternal and child deaths can be averted. Instructions: Specify what relevant documents will be available to the evaluators. In particular, identify the existence and availability of relevant performance information sources, such as performance monitoring indicators and/or previous evaluation reports. In addition, identify any other documents or sources of information from outside of USAID that would be useful to the evaluation team (e.g., government or international data). If this section is long it may also be included in an annex] 79 EVALUATION QUESTIONS Instructions: Include 3-5 specific questions focused on key program areas and/or performance and directly linked to the purpose of the evaluation and its expected use. Sub-questions may be included to elaborate on the main question, but not to add new areas of inquiry. NOTE: Not every aspect of an activity, project, or program needs to be, or should be, the focus of the evaluation. Rather, the evaluation should examine specific aspects of the activity, project, or program where there are questions unanswered by performance monitoring or other data. Guidelines: 1. Questions should be precise. Vague terms that can be defined or applied in a variety of ways (such as “relevance,” “effectiveness,” etc.) should be defined clearly for the evaluand. If any specific terminology or standards are included in the evaluation questions indicate the source or definitions. 2. Questions should be researchable. Questions should have an answer that can be obtained through the use of social science methods and tools (qualitative and quantitative) rather than relying on the evaluators’ judgments. 3. Questions should integrate gender. Questions should identify when sex-disaggregated data are expected. Where appropriate, the evaluation questions can include a separate question aimed at evaluating the gender-specific effects of the activity or project. [See the How-To Note on Engendering Evaluation] 4. Questions should be presented in order of priority, or the priority of questions should otherwise be identified. 5. A request for recommendations is not an evaluation question. If you want the evaluators to provide recommendations, describe what aspects of the activity, project, or program you want recommendations to address in a separate paragraph or following the questions. The evaluation questions will be structured based on three levels of program management and implementation. The first level will be at USAID and the component will be evaluated by USAID HQ integration specialist either as a solo evaluator or integrated in D4D evaluation team. The second and third level (Boresha Afya and Beneficiaries) evaluation will be performed by D4D. There will be one evaluation with three sections related to the three Boresha Afya Zones. USAID will like to answer the following evaluation questions: The purpose of conducting this midterm performance-based evaluation is to learn from and document: i. Whether the three USAID Boresha Afya awards are on track to achieve their program goals. ii. What is the program/intervention coverage and service uptake and specifically results versus targets of the different interventions disaggregated by region and districts. The aim is to understand and document the gaps that still remain. 80 iii. How effective is overall oversight and management across the UBA, within and from USAID to the implementing partner then management within the mechanism over field activities in targeted regions. Integration aspects of Boresha Afya will be addressed by Development Evaluation (DE) implemented by CIRCLE and this will be in a separate report. This midterm evaluation will draw qualitative information from DE. In the design and implementation of UBA, the USAID Health Office has integrated at three different levels: (1) at the service delivery point, where various services (e.g., FP, HIV, malaria, and MCH are being offered at a single delivery point; (2) at the mechanism level, where different funding streams (e.g., HIV, TB, MCH, malaria, FP, Nutrition, gender, and gender-based violence) and objectives are integrated into one award; and (3) at USAID management level, where different USAID technical staff (e.g., MCH, malaria, TB, HIV, FP, nutrition, gender, and gender-based violence) are integrated into a “facilities” team that all support the management of the integrated awards. For levels 2 and 3, USAID wants to understand the results on coverage and service uptake and understand the pros and cons of integration as implemented by USAID and its partners, and what are recommendations to improve performance? What is the optimal configuration of integration? (iii) Beyond just integration of services, awards, and USAID staff, what are the lessons learned that USAID Boresha Afya project lends itself that can be replicated in the design of future programming. I.e., what should be continued? What should be discarded? There are several more specific questions that USAID would also like to answer and these are: 1. At USAID level, effective project oversight a. Does the Health Office’s organizational structure allows for effective management of an integrated program such as Boresha Afya? b. How effective is the Boresha Afya Activity Management Team in managing the three awards? What works/does not work? c. How does having an integrated program like Boresha Afya affect the Health Office’s program management burden as a whole and individually for staff (increase or reduce)? d. Is the organization/management structure sufficient in addressing all program areas of integration, e.g., HIV, TB, MCH, FP, Malaria, Nutrition, gender, and gender-based violence? 2. Mechanism/Implementing partners level a. How effective are the different implementing partners in making needed changes effectively and timely manner? b. Does having an integrated program like Boresha Afya allow for all of the different health elements (HIV, TB, MCH, FP, malaria, nutrition, gender, and gender-based violence) to be sufficiently/effectively addressed? Are there any gaps left addressed? 81 c. How effective are the integrated programs compared to standalone mechanisms/ programs in meeting program performance goals (in HIV, TB, MCH, FP, Malaria, Nutrition, gender, and gender-based violence)? d. Are integrated programs more cost effective compared to single program? e. Do integrated mechanisms make it easier or harder to collaborate/coordinate/partner with USAID and the GOT? f. What are the lessons learned positive and negative in managing integrated programs? (What are the benefits and challenges of managing an integrated program at the mechanism/implementing partner level)? g. Is DE providing useful real time feedback to improve the quality of integrated services? 3. Are service providers receiving integrated technical support at PO-RALG level (program oversight) satisfied with the service provided? a. Are Council Health Management Teams receiving integrated technical support (program implementation, training, mentoring, and supportive supervision) satisfied with the support provided by the integrated mechanism, as compared to the standalone mechanism approach? b. Are Health care managers/providers receiving integrated technical support at the level (service provision and quality) satisfied with the support provided by the integrated mechanism, as compared to the standalone mechanism approach? 4. How effective is having a stand-alone action plan at addressing key gender-related barriers? EVALUATION DESIGN AND METHODOLOGY This is a midterm performance evaluation being undertaken after about 60% of the activities have been implemented so it also is intended to provide significant direction to follow-on activity planning and procurement. Evaluators will design evaluation tools that will collect information/data to address all the evaluation questions. Given the nature of the activities and the types of evaluation questions included in this scope, USAID/Tanzania proposes to utilize “before and after” and/or “time series” evaluation design which will be based on extensive use of pre-and post-quantitative and qualitative data and additional data that can be collected through key informant interviews (KIIs) and focus group discussions (FGDs). Data for Development is expected to propose any other designs which will generate high quality data and evidence to answer the evaluation questions. Data for Development will also use qualitative data generated by DE. The team will design and conduct an evaluation that maximizes participation by Local Government Councils (Council Health management teams), civil society organizations, Boresha Afya North/Central, Lake Western and Southern Zones program staff and health facility staff while ensuring, to the greatest extent possible, the objectivity and validity of inputs and outputs. USAID Mission staff will be included as participants while assuring maximum objectivity. Therefore, only Mission staff with no direct roles or responsibilities for activity management will 82 participate in data collection. Beneficiaries such as clients, Local government leaders, and technical teams including RHMT and DHMT will also be interviewed. Data Collection Methods The evaluator team should consider a range of data collection methods and approaches for collecting and analyzing the information that is required to achieve the evaluation objectives. Proposed data collection methodologies will be discussed with and approved by USAID/Tanzania prior to the start of the evaluation, including KIIs, FGDs, and other specific methods. Data analysis methods The report should include both qualitative and quantitative analysis of the achievements and shortcomings in relation to the objectives and targets for the output indicators for the cooperative agreements. Evaluators should describe any statistical tests that will be used, and how key informant interviews and focus group responses will be documented and analyzed. All information and data shall be disaggregated, to the maximum extent possible, by region, district, age and sex/gender to show differential outcomes between adult and children, regions and district/councils, men and women to meet USAID requirements. A data analysis plan for variables collected by the designed tools should be present for each methodology proposed. Methodological strengths and limitations. Limitations: There will be language constraints as some beneficiaries may have minimal English language capacity and evaluators will likely have limited Swahili language capacities. This limitation will be mitigated by participation of Swahili-English translator on the evaluation team. Another limitation is that there are 49 districts/councils in the six regions for Boresha Southern Zone, 40 districts in the six regions for Boresha North Central and 51 councils in the seven regions and six districts of Zanzibar for Boresha Lake western zone of program implementation. Due to time and funding constraints, only a few districts from each zone will be selected as representatives for the purpose of the evaluation. Instructions: This section may include suggestions or illustrative descriptions about the methodological approaches. If the evaluation design team has depth of experience in methodologies and methods than this section may be quite detailed and include methodological suggestions. Otherwise, it may request the evaluators’ expertise and input in the proposal and during the evaluation design phase. At a minimum this section should confirm that it is a performance evaluation. Guidelines: When drafting this section consider and then include narrative that describes clearly: 1. The suggested or expected data collection methodology and the corresponding data sources that will generate the highest-quality and most credible evidence that corresponds to the evaluation purpose and questions. 2. How suggested methods are linked to at least one evaluation question. 3. Any expectations regarding sites to be visited or groups to be interviewed. 4. Any expectations regarding how the evaluation data collected should be analyzed (e.g., comparison of particular groups or precision of response criteria, such as “margin of error must be less than +/- 10 percent”). 5. If performance monitoring data are to be used. If so, include information about how they have been verified. Or if the data have not been verified, that it is the expectation that the proposed design should include this requirement. 6. When analysis of disaggregated data are needed (e.g., sex, age, or other relevant aspects of beneficiaries). 7. Any known limitations to the data to be collected. The following simple design matrix can be included as a summary of evaluation design and methods, and to supplement the narrative section above, but should not replace the narrative. Questions Suggested Data Sources (*) Suggested Data Collection Methods Data Analysis Methods Suggested Respondents 1. Effective USAID project oversight: i) Is the organizational structure well placed to manage integrated programs. ii) How effective is the activity management teams in managing the program. iii) Does integrated programs influence program management burden (increase or reduce). iv) Is the organization/ management structure sufficient in addressing all program areas of integration, e.g., HIV, TB, MCH, FP, Malaria, and Nutrition 1) End of project Evaluation of Tunajali II and LIFE 2) Quarterly reports. 3) Annual reports. 4) Focus group discussions 5) Development Evaluations (DE) 1) Desk Document review 2) Focus group discussions 3) Key informant interviews 4) Questionnaires Qualitative analysis [To be determined by evaluation team] [Requested level of disaggregation— gender, age, location (district, region), public/FBO/Private sector, etc.c…] USAID health office staff Key management and technical staff from EGPAF, Deloitte, and Jhpiego 83 84 Questions Suggested Data Sources (*) Suggested Data Collection Methods Data Analysis Methods Suggested Respondents 1. How effective are the integrated program compared to single programs in reaching and quality services to beneficiaries? i. How effective are the integrated program compared to single programs in meeting program performance goals? ii. Is integrated programs more cost effective compared to single programs? iii. What are the lessons learned positive and negative in managing integrated programs? 1) End of project Evaluation of Tunajali II and LIFE 2) Quarterly reports 3) Annual reports 4) DATIM 5) IPRS 6) DQA reports 7) Focus group discussions 8) Development Evaluations (DE) 9) National statistics, DHS, THIS 10) MOH and PO￾RALG technical folks, CHMT and RHMT 11) Clients on ART, on Anti-TB med, On FP users, on ITN users, etc. 6) Desk Document review 7) Focus group discussions 8) Key informant interviews 9) Questionnaires 10) Observation and Field visits ditto USAID health office staff Key management and technical staff from EGPAF, Deloitte, and Jhpiego 2. Are beneficiaries receiving integrated technical support at PO-RALG level (program oversight) satisfied with the service provided? i. Are beneficiaries receiving integrated technical support at the Council health management teams (program implementation, training mentoring and supportive supervision) satisfied with the service? ii. Are beneficiaries receiving integrated technical support at the Health care managers/providers level (service provision and quality) satisfied with the service provided? iii. Are beneficiaries receiving integrated services at the client level (Patients and relatives of patients) satisfied with the service provided? ditto ditto ditto USAID health office staff Key management and technical staff from EGPAF, Deloitte, and Jhpiego Staff at CHMTs, PO-RALG, RHMTs, and facilities 85 DELIVERABLES AND REPORTING REQUIREMENTS In-brief and initial meetings, in the first day or two after arrival to review all plans, introduce the team to USAID, and finalize any questions or issues for the inception report. The discussion will review and approve the evaluation questions. The Mission in-brief will be immediately followed by initial meetings with implementing partners. • Inception report structured as follows: i) purpose; ii) team roles and responsibilities; iii) final questions and sub-questions; iv) logistical plan for data collection (KIIs, FGDs, and other methods) and analysis; and v) final report draft outline. The inception report should be approved by USAID/Tanzania by the end of the first week of the evaluation. The report should be no more than 10 pages and can largely be provided by a complete draft evaluation design matrix. • The second Mission briefing will be held immediately following site visits and data collection to make sure that all questions have been addressed and at least 2 weeks prior to departure. Plans for data analysis and report drafting will be reviewed. • A final Mission debrief meeting will be held approximately 3 days before evaluation team departure to review the status of data collection efforts and any preliminary impressions from the data. The outline for the final report will be presented and approved. The evaluation team will incorporate the Mission’s comments from the meeting as it produces the final report. • The draft final report must be completed after analysis of the findings. The written report must clearly describe findings, conclusions and recommendations (using the report format provided in “Reporting Requirements” below). The Mission will provide comments on the draft final report within 10 working days of evaluation team submission. • The final report that incorporates the evaluation team’s responses to Mission comments and suggestions will be submitted 10 days after USAID/Tanzania provides its feedback on the draft report. The report format should be restricted to Microsoft products and 12-point type must be used throughout the body of the report, with page margins 1” top/bottom and left/right. The report must not exceed 30 pages, excluding both references and annexes. b) Report requirements As indicated above, the evaluation findings, conclusions, and recommendations will be presented in a draft report at a briefing with USAID/Tanzania. The Mission may also arrange a financial briefing from the evaluation team with key stakeholders and implementing partners. The format for the final evaluation report is as follows: • Evaluation Abstract providing a brief summary that identifies the purpose, methods, findings and conclusion of the evaluation (not more than 250 words) • Executive Summary: concisely states findings, conclusions and recommendations organized by evaluation question (no more than 4 pages) • Has list of acronyms • Table of Contents (1 page); • Introduction: purpose, audience, and synopsis of task (1 page); 86 • Background: brief overview of Health program in Tanzania, USAID strategies and priorities, brief description of the Boresha Afya program, the purpose of the evaluation (2-3 pages); • Design and Methodology: describe evaluation design methods, including constraints and gaps (2 pages); • Findings/Conclusions/Recommendations in separate sections: for each evaluation question (15-20 pages); • Future Directions of USAID programming based on gaps in results achievement, sustainability of those results, or innovations that should be introduced or scaled up (2-3 pages); • References (sources of information, properly identified and listed including bibliographical documentation, and all notes from stakeholders’ meetings, key informant interviews, and focus group discussions); • Annexes, which should include: o The Evaluation Statement of Work. o Any “statements of differences” regarding significant unresolved difference of opinion from Mission staff, implementing partners, stakeholders, and/or members of the evaluation team o Evaluation design methods and all the tools used in conducting the evaluation, such as questionnaires, checklists, survey instruments, and discussion guides o Disclosure of conflicts of interest forms for all evaluation team members, either attesting to a lack of conflict of interest or describing existing conflict of interest. Instructions: List specific deliverables, reporting requirements, audiences, and timeframes that the evaluation team should know. The only required deliverables are the evaluation design, draft report, and final report, but additional deliverables may be beneficial. Sample text is provided below to be adapted as relevant and useful to your Operating Unit. Please consider the time and location of when the evaluation team can reasonably complete the deliverable. For example, preparation of the draft report requires analysis of the data collected; therefore, the exit-briefing for an international team will likely not be able to include requirements for presentation of recommendations.] 1. Evaluation Work plan: [SUGGESTED] Within [# weeks] of the award of the contract, a draft work plan for the evaluation shall be completed by the lead evaluator and presented to the Agreement Officer’s Representative/Contracting Officer’s Representative (AOR/COR). The work plan will include: (1) the anticipated schedule and logistical arrangements; and (2) a list of the members of the evaluation team, delineated by roles and responsibilities. [The work plan may include the Evaluation Design (a requirement of all evaluations). However, it is not always feasible to complete an evaluation design immediately upon award. Therefore, it is advised to separate the deliverable that kicks-off the evaluation from the design. It can take weeks to develop a good design and prepare data collection instruments that are participatory, utilization-focused, and incorporate all of the existing data.] 2. Evaluation Design: [REQUIRED] Within [# weeks] of approval of the work plan, the evaluation team must submit to the Agreement Officer’s Representative (AOR) an evaluation design (which will become an annex to the Evaluation report). The evaluation design will include: 87 (1) a detailed evaluation design matrix that links the Evaluation Questions in the SOW to data sources, methods, and the data analysis plan; (2) draft questionnaires and other data collection instruments or their main features; (3) the list of potential interviewees and sites to be visited and proposed selection criteria and/or sampling plan (must include calculations and a justification of sample size, plans as to how the sampling frame will be developed, and the sampling methodology); (4) known limitations to the evaluation design; and (5) a dissemination plan. [If applicable add a requirement to include a conflict of interest mitigation plan based on the Disclosure of Conflict of Interests submitted with the awardee’s proposal]. [RECOMMENDED language to include to #2] USAID offices and relevant stakeholders are asked to take up to [# business days] to review and consolidate comments through the AOR/COR. Once the evaluation team receives the consolidated comments on the initial evaluation design and work plan, they are expected to return with a revised evaluation design and work plan within [# days]. [It is best practice to have the design reviewed and accepted by USAID before the evaluation team begins data collection or at a minimum within a period of time when it is still possible to change data collection strategies] 3. kIn-briefing: [OPTIONAL] Within [ # days] of arrival in [specify location], the evaluation team will have an in-briefing with the [insert offices/audience] for introductions and to discuss the team’s understanding of the assignment, initial assumptions, evaluation questions, methodology, and work plan, and/or to adjust the Statement of Work (SOW, if necessary. 4. Mid-Term Briefing and Interim Meetings: [OPTIONAL] The evaluation team is expected to hold a mid-term briefing with [specify USAID offices and/or staff] on the status of the evaluation, including potential challenges and emerging opportunities. The team will also provide the evaluation COR/manager with periodic briefings and feedback on the team’s findings, as agreed upon during the in-briefing. If desired or necessary, weekly briefings by phone can be arranged. 5. Final Exit Briefing: [OPTIONAL] The evaluation team is expected to hold a final exit briefing prior to leaving the country to discuss the status of data collection and preliminary findings. This presentation will be scheduled as agreed upon during the in-briefing. [Specify guidelines of the presentation, e.g., who should be included, such as implementing partner staff or other stakeholders; preferred medium (joint or separate briefings); and expected maximum length] 6. Final Presentation: [OPTIONAL] The evaluation team is expected to hold a final presentation in person/by virtual conferencing software to discuss the summary of findings and recommendations to USAID. This presentation will be scheduled as agreed upon during the in￾briefing. [Specify guidelines of the presentation, e.g., who should be included, such as implementing partner staff or other stakeholders; preferred medium (joint or separate briefings);expected maximum length; and timing (before or after the final report)]. 7. Draft Evaluation Report: [REQUIRED] The draft evaluation report should be consistent with the guidance provided in Section IX: Final Report Format. The report will address each of the questions identified in the SOW and any other issues the team considers to have a bearing on the objectives of the evaluation. Any such issues can be included in the report only after consultation with USAID. The submission date for the draft evaluation report will be determined in the evaluation work plan. Once the initial draft evaluation report is submitted, [insert office/s]will have [number] business days in which to review and comment on the initial draft, after which point the AOR/COR will submit the consolidated comments to the evaluation team. The evaluation team will then be asked to submit a revised final draft report [number] business days hence, and again the [insert office/s]will review and send comments on this final draft report within [number] business days of its submission. [A good practice is for the evaluation team to share an early draft or detailed outline that includes main findings and bullets before finalizing the draft evaluation report] 8. Final Evaluation Report: [REQUIRED] The evaluation team will be asked to take no more than [number] business days to respond/incorporate the final comments from the [insert office/s]. The evaluation team leader will then submit the final report to the AOR/COR. All project data and records will be submitted in full and should be in electronic form in easily readable format, organized and documented for use by those not fully familiar with the project or evaluation, and owned by USAID. EVALUATION TEAM COMPOSITION Instructions: Before the standard language offered below, describe the intended size of an evaluation team, the appropriate expertise related to evaluation approaches (or methodologies), methods, and subject matter expertise required of the team or specific team members. Other skills that may be included as requirements include language, geographic experience, among others. Notes: A typical team should include one team leader who will serve as the primary coordinator with USAID. At least one team member who should be an evaluation specialist. The recruitment of local evaluators is highly encouraged. Requested qualifications and/or skills may relate to: (1) evaluation design, methods, management, and implementation; (2) specific relevant technical subject matter expertise, (c) experience in USAID’s cross-cutting program priorities, such as, gender equality and women’s empowerment, (d) regional or country experience; and (e) local language skills. All team members will be required to provide a signed statement attesting to a lack of conflict of interest or describing any existing conflict of interest. The evaluation team shall demonstrate familiarity with USAID’s Evaluation Policy and guidance included in the USAID Automated Directive System (ADS) in Chapter 200. Optional: The [insert name] will participate in the evaluation team in [describe role]. See Guidance for USAID Staff Participation on External Evaluations for language. Optional: The COR of the Evaluation may observe [insert all or some] of the data collection efforts. EVALUATION SCHEDULE Draft Example of LOE to be modified by the Contractor as necessary Task Evaluation Team Lead M&E LOE Local Consultant (4) Proposed Timeframe Review of background documents – Virtual 5 days 5 days 5 x 4 days July 6- 10 Team planning meeting/In-brief with Mission/Initial IP meetings 2 days 2 days 2 x 4 days July 13 -17 88 89 Task Evaluation Team Lead M&E LOE Local Consultant (4) Proposed Timeframe Finalize evaluation design and methodology and draft/complete Inception Report 3 days 3 days 3 x 4 days July 13-17 Develop and test interview guides, questionnaires and facilitation guides, and other tools for KIIs, FGDs, and other data collection 5 days 5 days 5 x 4 days July 20-24 Conduct field work for KIIs, FGDs, and other data gathering with IPs and stakeholders 15 days (Dar and 4 regions) 15 days (Dar and 4 regions) 15 x 4 days (Dar and 4 regions) July 27 – August17 Analysis of data gathered/ Drafting of out-brief and final report annotated outline 7 days 7 days 7 x 4 days July 24 - 28 Out-brief and other pre￾departure meetings 3 days 3 days 3 x 4 days July 29- August 1 Complete evaluation report after Mission comments 5 days 5 days 5 x 4 days August 2-8 Instructions: Provide an estimated timeframe (in days) for the evaluation (period of performance) to be conducted as well as an anticipated start date. Period of performance should include the time it takes for USAID to review the draft and final evaluation reports and for all work to be completed for the evaluation. Likewise it is very important that the schedule includes time for review throughout the process with key stakeholders and USAID staff. Consider including a timeline table (GANTT chart) or indicative schedule in narrative form. Guidance: The sample table outlines these main phases of a performance evaluation. The guiding questions are: 1. What is the period of time (duration) you expect the evaluation team to take to review activity/project documents and become familiar with the program (prior to travel)? 2. How long will it take to get the necessary clearances for travel and to complete any protocols to visit communities and prepare for data collection? 3. How many sites/regions will the team be expected to visit, and what is a realistic timeframe for such requirements? Will the team be split up into smaller units during data collection to speed up the time required to collect the data? 4. What is the period of time (duration) it take to collect data? 90 5. What is the period of time (duration) allocated to analyze the data following data collection? 6. What is the period of time (duration) to prepare briefings and reports? If data visualization and graphical requirements are included state these. Sample Format: Illustrative Schedule Timing Proposed Activities (SOW) June 19, 2019 Discussion of SOW with USAID Preparation and submission of initial work plan and evaluation design July 5-25, 2019 draft and final USAID review of the work plan and evaluation design; COR and CO approval of key personnel July 30, 2019 Preparation and submission of final work plan and evaluation design July 10-31, 2019 Team planning, Preparation of instruments and field work plan ET starts document review August 6, 2019 August 2, 2019? Submission of protocols to NORC review board (IRB) Travel request for short term technical assistance (STTA) sent to USAID August 9, 2019 NORC IRB meeting on August 21 and approval expected the following week if no major revisions are requested August 5-July 20, 2019 Send instruments to USAID for review and to translator for Swahili translation August 6, 2019 Submit approval for NORC clearance August 9-August 23, 2019 Submit NIMR approval and receive clearance (with expedited processing) August 17, 2019 STTA travel August 19-20 2019 Team building, training and planning meeting/mobilization; data collection pre-test and training; in-brief preparation and delivery; August 26-September 16, 2019 Data collection and fieldwork (2-3 weeks of collection with 3 teams and one day between each site for preliminary analysis of findings) September 16, 2019 Preliminary analysis, Data collection briefing: preliminary findings and conclusions to USAID in out-brief September 16 & 25, 2019 STTA travel September 21-October 11, 2019 Coding of data into Dedoose; Data analysis October 14-30, 2019 Report writing October 31, 2019 Submit Draft Report and Internal Memo November 1-10, 2019 USAID review of Draft Report and submit comments November 11-23, 2019 Incorporate USAID comments; Submit Final Report Week of November 25-29th, 2019 Utilization and action planning, after-action review session/meeting and dissemination TBD Upload to DEC (Mission to give approval to upload) 91 Instructions: The section should also include illustrative information about the level of effort (work days) to complete the evaluation. However, it is not required that specific and detailed level of effort be provided by team members. Requirements associated with the level of specificity for the level of effort are determined by the contracting mechanisms. Level of effort calculations by team members are generally required to prepare an accurate Independent Government Cost Estimate (IGCE). See the Guidance Note on IGCE for Evaluations for a detailed explanation for estimating level of effort. Some key factors for determining the level of effort (number of work days to complete a task) include: 1. Planning/Evaluation Design: How many documents are there to review and how methods of data collection are anticipated? Time is required to review the documentation, prepare a work plan, and design instruments. Each method of data collection will require its own instrument. 2. Preparations for Data Collection: Is there an expatriate team? How long does travel take? How much travel is required outside of the capital city? 3. Data Collection: How many different geographic locations will be required? How many people will travel to each location? How many days per person are rOVC_HIVSTATequired by method for data collection? 4. Analysis: How many different types of data sets are going to be generated? Are there quantitative data? If so, allocate time for data entry and cleaning. 5. Reporting and Briefing: How many different deliverables are required? Allocate time by deliverable and by person (not all team members will spend the same amount of time). The sample table shells are illustrative for a simple evaluation with four team members. Sample Table: Estimated LOE in days by activity for a team of four Task LOE for Expat Team Lead LOE for Expat [subject matter] Specialist LOE for Local [subject matter] Specialist LOE for Local [subject matter] Specialist Total LOE in Days Document review/desk review/work planning (evaluation design remote or in-country) Preparations for travel and organizing data collection (contracting translators, vehicles, etc.). In-brief, Evaluation Design (including meetings with USAID) Preparations for data collection (scheduling) Data collection days by method by site Data analysis 92 Task LOE for Expat Team Lead LOE for Expat [subject matter] Specialist LOE for Local [subject matter] Specialist LOE for Local [subject matter] Specialist Total LOE in Days Briefing Draft final report and debrief to USAID [include time for translation if necessary] Final report Totals Sample Table: Estimated LOE in days by position for a team of four Position Preparation Travel To/From Country In-Country Data Collection Finalization of Report Total LOE in Days Expat Team Leader Expat Specialist Local Specialist Local Specialist Totals FINAL REPORT FORMAT The evaluation final report should include an executive summary; introduction; background of the local context and the projects being evaluated; the main evaluation questions; the methodology or methodologies; the limitations to the evaluation; findings, conclusions, and recommendations; and lessons learned (if applicable) as described here. The report should be formatted according to the evaluation report template. The executive summary should be 3-5 pages in length and summarize the purpose, background of the project being evaluated, main evaluation questions, methods, findings, conclusions, and recommendations and lessons learned (if applicable). The evaluation methodology shall be explained in the report in detail. Limitations to the evaluation shall be disclosed in the report, with particular attention to the limitations associated with the evaluation methodology (e.g., selection bias, recall bias, unobservable differences between comparator groups, etc.) The annexes to the report shall include: • The Evaluation SOW; • Any statements of difference regarding significant unresolved differences of opinion by funders, implementers, and/or members of the evaluation team; • All tools used in conducting the evaluation, such as questionnaires, checklists, and discussion guides; • Sources of information, properly identified and listed; and • Disclosure of conflict of interest forms for all evaluation team members, either attesting to a lack of conflicts of interest or describing existing conflicts. 93 In accordance with AIDAR 752.7005, the contractor will make the final evaluation reports publicly available through the Development Experience Clearinghouse (DEC) within 30 calendar days of final approval of the formatted report. CRITERIA TO ENSURE THE QUALITY OF THE EVALUATION REPORT Per the USAID Evaluation Policy and USAID ADS 203, draft and final evaluation reports will be evaluated against the following criteria to ensure the quality of the evaluation report.85 • The evaluation report should represent a thoughtful, well-researched, and well-organized effort to objectively evaluate what worked in the project, what did not, and why. • Evaluation reports shall address all evaluation questions included in the SOW. • The evaluation report should include the SOW as an annex. All modifications to the SOW—whether in technical requirements, evaluation questions, evaluation team composition, methodology, or timeline—need to be agreed upon in writing by the AOR/COR. • The evaluation methodology shall be explained in detail. All tools used in conducting the evaluation—such as questionnaires, checklists, and discussion guides—will be included in an annex in the final report. • Evaluation findings will assess outcomes and impact on males and females. • Limitations to the evaluation shall be disclosed in the report, with particular attention to the limitations associated with the evaluation methodology (selection bias, recall bias, unobservable differences between comparator groups, etc.). • Evaluation findings should be presented as analyzed facts, evidence, and data and not based on anecdotes, hearsay, or the compilation of people’s opinions. Findings should be specific, concise, and supported by strong quantitative or qualitative evidence. • Sources of information need to be properly identified and listed in an annex. • Recommendations need to be supported by a specific set of findings. • Recommendations should be action-oriented, practical, and specific, with defined responsibility for the action. 85 Integrated approach in this programmatic context has a twofold definition mentioned above on page 6. ANNEX 2: EVALUATION METHODS AND LIMITATIONS EVALUATION DESIGN AND METHODOLOGY 2.1 Purpose of the Evaluation The three-fold purpose of the Boresha Afya midterm performance evaluation is to document: i. Whether the three USAID Boresha Afya awards are on track to achieve their program goals; ii. What is the program service coverage and uptake by region and districts; and iii. How effective is overall oversight and management across the program, within and from USAID to the IPs and their consortia implementing field activities in targeted regions. Points (i) and (ii) will be included in the EQ below. Point (iii) will be part of an internal evaluation by USAID/W. This evaluation will gather only the perceptions of the IP on the management and oversight by USAID/Tanzania. The primary audience for the evaluation is the USAID/Tanzania health team and Boresha Afya program staff across all three awards. The evaluation’s findings, conclusions, and recommendations will be used to improve program direction and implementation over its remaining activity period and to assist USAID/Tanzania in shaping the direction its future health programming. 2.2 Evaluation Questions The EQs articulated in the Boresha Afya performance evaluation SOW cover the themes of program results and effectiveness, implementation approaches and constraints, lessons learned for future programming, and USAID/Tanzania management and oversight. These EQs, along with their associated sub-questions, are listed below.86 1. To what extent has the Boresha Afya program been effectively managed and overseen? a. Is the organization/management structure of the three prime contractors sufficient in addressing all program areas of integration?87 2. How effectively has Boresha Afya’s integrated approach88 been implemented and how adaptively have IPs managed performance across program intervention areas (in HIV, TB, MNCH, FP, Malaria, etc.)? 86 This language reverts to the key word “cost effectiveness” in the SOW. However, it is understood that the ET will not use a quantitative cost effectiveness analysis to determine this (which would require a separate audit/assessment). Analysis of “cost effectiveness” will emerge from coded and triangulated responses from IPs, USAID, and GOT stakeholders as well as budgetary and performance documentation from secondary sources. Financial reporting such as expenditure reporting mandated by PEPFAR will also be used to assess costs over time. 87 The DE is focused on measuring real time results on the integration of services in the activity across the three implementing mechanisms. A small contract team works with M&E staff embedded across all three IP consortia and across regional teams to undertake the DE. Meanwhile M&E staff from each mechanism are responsible for each zone’s ongoing performance measurement of standard and custom indicators. 88 This will be identified in the gender and youth action plan and PEPFAR guidance. 949 b. How effective and timely is each IP in making necessary and adaptive changes to improve performance? c. How effectively has UBA integrated the targeted health areas into a functioning IHC system at district level and nationwide? d. How has the implementation of Boresha Afya’s IHC efforts been influenced by other integrated health care initiatives and funding sources? e. How effective are the integrated programs under Boresha Afya expanding access and coverage in HIV, TB, MCH, FP, Malaria, Nutrition, gender, and GBV? f. How cost effective is the integrated approach to implementation to Boresha Afya IPs in coordination with each other, USAID, and the GOT? 89 g. What are the benefits, challenges, and lessons learned from the Boresha Afya integrated programs? h. How has the DE program contributed to the effectiveness, learning and adaptability of UBA implementers?90 3. To what extent are service providers at the PO-RALG, LGA, and facility levels who are receiving integrated systems strengthening and technical support satisfied with the services and technical assistance provided and are they applying skills and practices to their work? a. Are Council Health Management Teams receiving integrated technical support (e.g., program implementation, training, mentoring, and supportive supervision) satisfied with the support provided by the integrated mechanism? What if any changes in capacity do they see? b. Are health care managers/providers receiving integrated technical support at the facility level satisfied with the service provision and quality provided by the integrated mechanism? How are they applying acquired knowledge and skills in the delivery of services at facilities? What if any changes in capacity do they see in facilities? 4. How effective is Boresha Afya at identifying and addressing key gender and youth related barriers91 to service delivery? 2.3 Evaluation Approach Data for Development proposes to conduct the Boresha Afya evaluation using a participatory approach in which the Data for Development ET engages USAID/Tanzania, Boresha Afya IPs, GOT counterparts, program beneficiaries, and other stakeholders during the various evaluation phases. This includes working collaboratively to: 1. Identify EQs, keeping in mind users and uses of the evaluation for Mission, IPs, and GOT decision-making; 2. Identify pertinent documentation and secondary performance data for the desk review; 89 Dedoose is a cross-platform app for analyzing qualitative and mixed-methods research with text, photos, audio, videos, spreadsheet data, and more (https://www.dedoose.com/). 90 This approach will be verified depending on discussions of data and reporting from the DE. If they already have done client interviews of this nature, we will request and draw upon that data. 91 Snowball sampling is when evaluation participants recommend other persons to participate in the evaluation. 95 3. Plan the fieldwork and identify participants for data collection activities, including KIIs, GIs, and FGDs; 4. Develop and review interview/discussion guides for KIIs, GIs, and guides for FGDs; 5. Participate in briefings to review findings, conclusions, and recommendations to ensure feasibility and utilization; and 6. Review reports and intermediate deliverables. At the same time, the ET will remain independent and take steps to maximize the quality of the information and minimize the impact of various potential sources of bias on the evaluation. Accordingly, Boresha Afya IP staff will not be involved directly in data collection activities for the purpose of maintaining objectivity and ensuring respondent/beneficiary confidentiality. 2.4 Evaluation Design To address the EQs, the ET will employ a mixed-methods approach, which will analyze program related documentation, program monitoring data, and primary source data collected through a desk review of program documents; up to 52 KIIs, 24 GIs, and 30-36 FGDs with diverse stakeholders; facility checklists completed through observation and interviews with facility staff; and exit interviews with clients at each facility. These multiple data sources will allow the ET to triangulate information, ensuring that findings and conclusions are robust and that recommendations are sound. The ET’s goal is to generate not only an overall understanding about the program, its results, and effects but also a detailed assessment of its various components and the effectiveness of their approaches and implementing mechanisms. All primary and secondary data will be disaggregated by appropriate demographics, including sex and age, as well as by client/beneficiary groups, as appropriate. 2.4.1 Desk Review Documents to be reviewed include the Boresha Afya cooperative agreements/contracts and their modifications; relevant sub-agreements; program start-up documentation; work plans; annual and quarterly reports; monitoring and internal reports; and other relevant documents from USAID, the GOT, and other stakeholders. The initial set of documents to be reviewed will include all those received from USAID and the Boresha Afya program (see document checklist of those documents received and pending). Upon completing an initial document review, the ET will request additional documents that are deemed important. Throughout the evaluation, the ET will keep an updated checklist of all documents received noting any missing or partial documentation. The ET will continuously update this checklist based on documents received and request additional documentation as needed. 2.4.2 Performance Monitoring Data The ET will analyze performance monitoring data produced by the Boresha Afya Program and data from any secondary sources, such as from the Tanzanian DHIS2, using descriptive statistics and trend analysis of indicators over time. Upon completing an initial review of the data, the ET will determine if it is possible/required to carry out more advanced statistical analyses. 2.4.3 Key Informant Interviews The ET will conduct up to 52 semi-structured interviews with key informants. Key informants to be interviewed in Dar es Salaam include senior-level staff from each of the Boresha Afya IPs, 96 officials from national-level GOT entities, including the MOHCDGEC, the PO-RALG, and USAID AORs/CORs. In Zanzibar, the Zanzibar National Health Authorities such as Zanzibar National Malaria Control Program (ZNMCP) will be interviewed. At the regional and district levels RMOs and DMOs will be interviewed while undergoing regular reporting to received permission to access facilities (in some cases they may join GIs with RHMTs and DHMTs in lieu of these KIIs). The ET will also interview hospital, dispensary and/or clinic supervisors in charge at facilities. KIIs may also be used for other cadres such as clients and CSO leadership when not possible to conduct these as group interviews. The ET will develop modular KII protocols that members can easily tailor to each informant type. Each questionnaire will include 10-15 open-ended questions. The approach will elicit the interviewee’s experience with the IPs as their individual story. The will help show the convergence of usually complex facts that led to the way the interviewee adapted and applied the UBA approach. The story-based approach will help identify themes and evidence related to each EQ. All KIIs will be recorded with the informed consent obtained prior to the start of the interview. 2.4.4 Group Interviews Approximately 24 GIs will be conducted with groups of 2-6 informants. In each of the data collection sites, groups of informants will include health officers at the regional and district levels. At the regional level, the ET will hold GIs with members of the RHMTs and, at the council level, with the CHMTs. GIs will also be used with CSOs at the council level (in all zones except for North and Central) and with CHWs associated with the CSOs. To the extent possible, GI participants will be at the same level of rank possible to ensure that all participants feel comfortable sharing their opinions. When appropriate and logistically possible, higher ranking members/individuals (DMOs or RMOs) may be asked to do a key informant interview using a similar protocol. The number of GIs (as well as KIIs) may vary depending on the appropriateness of conducting the interviews in group versus individually. As in the case of KIIs, the evaluation team will develop a guide of 10-15 open ended questions for each group of informants and will also elicit personal and shared stories. All GIs will be recorded with informed consent obtained prior to the start of the interview and transcribed. 2.4.5 Focus Group Discussions A total of 19-25 FGDs will be conducted across the six sites. In each site, two FGDs will be conducted with clients from the primary facility visited by the ET. These FGDs will be segregated by gender and will each draw clients which received a particular type of care at the facility (e.g., malaria, HIV, TB, MNCH, FP, etc.) The ET will determine the most appropriate health element to focus on in each location to ensure adequate coverage. The ET will also conduct one mixed￾gender FGD with youth (18+ only) in each site visited. In all FGDs, participants will be able to share their stories with the health system and local providers and asked to reflect on the most significant changes in their community’s access to quality, integrated services, as well as remaining barriers to care. The ET will invite up to 15 participants for each FGD, aiming to have 10-12 participants per FGD (after refusals and no-shows). If a given group has fewer than five participants, the team will change to having a GI instead, following the same FGD discussion guide. FGDs will be conducted 97 by Swahili-speaking facilitators and will be recorded with informed consent obtained prior to the start of the discussion. The discussion guide will include 9-12 questions for a 90-minute FGD, starting with broader questions and moving on to narrower key questions. 2.4.6 Qualitative Analysis The ET will apply thematic content analysis to code the KII, GI, and FGD transcripts using Dedoose to produce a summary of recurrent themes that emerge in response to each topic.92 Key themes across groups and instruments will be tabulated and triangulated across qualitative primary sources and with quantitative data from program performance monitoring results and secondary sources. When appropriate, important findings will be illustrated with quotes. Table 3 summarizes the proposed approach for answering each of the evaluation questions. 2.4.7 Facility Checklist The ET will administer a facility checklist via tablet in each facility visited. This checklist will include a mix of closed- and open-ended questions designed to be answered through a mix of observation by the ET and discussions with facility staff. The purpose of this checklist is to gather evidence of the effect of the interventions of each IP in terms of access and coverage of quality integrated services and improvements in the organization and functioning of the health system that are reported to be likely to be sustained. Staff that may provide specific information for this checklist include supervising practitioners, nurses, clinicians, data clerks, administrative personnel, and diagnostics personnel. Quantitative data collected from the facility checklist will be tabulated and descriptive statistics produced for use in the final evaluation report. The ET will also take photos of the facility, not the clients, to document access issues such as crowded waiting areas, which are a sign of lack of crowd control procedures and efficient appointment system. 2.4.8 Client Exit Interviews-TBD93 The ET will administer exit interviews (EI)via tablet with people leaving each facility during a set observation period. The purpose of the EI is to assess clients’ satisfaction with the services received, perceived quality and how well the client’s experience was in relation to integration of various services at the point of service. Here also, the brief story-based approach will be used. Exit interviews will be approximately five minutes in length and cover topics such as satisfaction with the way the health provider treated them and whether or not they were invited to ask questions and offered additional tests or services beyond the original purpose of their visit. Specific questions will be designed to complement and not duplicate existing data collected by the DE. The ET will conduct a total of ten EIs at each facility to make a total of 120 estimated exit interviews. Dispensaries have fewer clients and for that reason only those present at the time of the evaluation will be interviewed. Data collected from the exit interviews will be tabulated and analyzed using descriptive statistics for use in the evaluation report. 92 TL and Evaluation Advisor will start national interviews in DSM before joining teams in the second round of data collection. The second expat short-term technical assistance (STTA) SME will do the national interviews in Dodoma with Team 1 while the rest of the team conducts field work at the Council level. 93 The following regions were strategically selected based on mission input with two regions per zone; final selection of the councils and facilities will be determined following review of further information on regions councils and facilities. 98 2.4.9 Quantitative Analysis of Secondary Data Sources To analyze the effectiveness of the program and the performance of its intervention areas, the ET will request performance data (compiled raw data in a single data source) from the three IPs. The ET will apply descriptive statistics using longitudinal data from the DATIM and IPRS. 2.4.10 Analysis of Cost Effectiveness To assess cost effectiveness, document review including IP budgets, burn rates and expenditure reports will be used, as available, to qualitatively assess the costs against the effectiveness of the program components. Financial reporting such as expenditure reporting mandated by PEPFAR may be used to assess costs over time. The ET will engage stakeholders in discussions on the expected and unexpected costs during KIIs and GIs with USAID, IPs, and GOT HMTs. When possible, simple cost per client service received ratios will be calculated. The ET does not propose using a quantitative cost effectiveness analysis which would require a separate econometric assessment. Analysis of “cost effectiveness” will additionally emerge from coded and triangulated responses from stakeholders (IPs, USAID, and GOT) as well as budgetary and performance documentation from secondary sources. Table 2: Evaluation Design Matrix Evaluation Questions Sources and Key Informants Data Collection Methods Analysis Methods EQ 1: To what extent has the program been effectively managed and overseen? *EQ 1 with the exception of sub question 1d will be done by USAID/W in a separate management assessment. Program documents USAID, IPs • Document review • KIIs with USAID, IPs • Content analysis of documents; coding by themes • Qualitative analysis and coding of transcripts 1.a) Is the organization/management structure of the three prime contractors sufficient in addressing all program areas of integration? Program documents USAID, IPs • Document review • KIIs with USAID and IPs • Content analysis of documents; coding by themes • Qualitative analysis and coding of transcripts EQ 2: How effectively has UBA’s integrated approach been implemented and how adaptively have IPs managed performance across program intervention areas (in HIV, TB, MNCH, FP, Malaria, etc.)? Project documents and other secondary sources: 1) End of project Evaluation of Tunajali II and LIFE 2) Quarterly reports 3) Annual reports 4) DATIM 5) IPRS 6) DE reporting 7) National statistics, DHIS2 IPs, GOT (national and LGA), Facility supervisors, Clients • Document review • Review of secondary datasets • KIIs • GIs • FGDs • Facility checklist • Client exit interviews • Content analysis of documents; coding by themes • Qualitative analysis and coding of KII, GI, and FGD transcripts • Analysis of secondary program performance data using trend analysis • Descriptive statistics from facility checklist and client exit interviews 99 Evaluation Questions Sources and Key Informants Data Collection Methods Analysis Methods 2a) How effective and timely is each IP in making necessary and adaptive changes to improve performance? IPs, GOT (national and LGA), Facility supervisors and staff, Clients • KIIs/GIs with IPs, GOT, facility supervisors • Facility checklist • Client exit interviews • Qualitative analysis and coding of KII, GI, and FGD transcripts • Descriptive statistics from client exit interviews and facility checklist 2b) How effectively has UBA integrated the targeted health areas into a functioning IHC system? 2c) How has the implementation of UBA’s IHC efforts been influenced by other IHC initiatives and funding sources? Performance data DHS and DHIS2 data IPs, GOT (national and LGA), Facility supervisors and staff, Clients, CSOs • Document review • Review of secondary datasets • KIIs/GIs with IPs, GOT, facility supervisors, CSOs • FGDs with clients • Facility checklist • Client exit interviews • Analysis of secondary program performance data, and descriptive statistics. • Qualitative analysis and coding of transcripts • Descriptive statistics from client exit interviews and facility checklist (2d) How effective are the integrated programs under Boresha Afya in meeting program performance goals (in HIV, TB, MCH, FP, Malaria, Nutrition, gender, and GBV)? Performance data DHS and DHIS2 data IPs, GOT (national and LGA), Facility supervisors and staff, Clients, CSOs • Document review • Review of secondary datasets • KIIs/GIs with IPs, GOT, facility supervisors, CSOs • FGDs with clients • Facility checklist • Client exit interviews • Content analysis of documents • Analysis of secondary program performance data, and descriptive statistics. • Qualitative analysis and coding of transcripts • Descriptive statistics from client exit interviews and facility checklist 2e) How cost effective is the integrated approach to f implementation to Boresha Afya IPs in coordination with each other, USAID, and the GOT? Budget documents, inancial and expense reports USAID, IPs, GOT (national and LGA), Facility supervisors and staff • Document review of budgets, burn rates and expenditure reporting • Expenditures by district and by program area • KIIs/GIs with USAID, IPs, RHMTs and CHMTs, Facility supervisors • Facility checklist • Content analysis of documents • Qualitative analysis and coding of transcripts • Analysis of secondary program performance data, descriptive statistics. • Descriptive statistics from facility checklist • Comparison of approaches of each IP to identify areas of collaboration, synergy, and increased efficiency. • Comparison of cost per unit of output and outcome. E.g. cost per 1% FP increase, cost per one client diagnosed, one child fully immunized, etc. 2f) What are the benefits, challenges, and lessons learned from the Boresha Afya integrated programs? IPs’ written lessons learned compiled from QRs and/or central document from USAID • Document review • KIIs/GIs with IPs, USAID, facility supervisors • Facility checklist • Content analysis of documents to analyze the evidence related to the UBA results framework. • Qualitative analysis and coding of transcripts • Descriptive statistics from facility checklist 100 Evaluation Questions Sources and Key Informants Data Collection Methods Analysis Methods 2g) How has the DE program contributed to the effectiveness, learning, and adaptability of UBA implementers? Documents provided by DE IPs and USAID • Document review • KIIs with IPs and USAID • Content analysis of documents • Analysis of DE objective and achievements in relation to UBA IP contractual performance • Qualitative analysis and coding of transcripts 3. To what extent are service providers at the PO-RALG, LGA, and facility levels who are receiving integrated systems strengthening and technical support satisfied with the services and TA provided and are they applying skills and practices to their work? Performance data USAID, IPs, GOT (national and LGA), facility supervisors and staff, clients, CSOs • Document review • KIIs/GIs with IPs, USAID, PO-RALG, LGAs and supervisors at facilities, CSOs • Facility checklist • Client exit interviews • FGDs with clients • Qualitative analysis and coding of transcripts • Descriptive statistics on satisfaction surveys and indicators on participants trained • Descriptive statistics from facility checklist and client exit interviews • TBD 3a. Are CHMTs receiving integrated technical support (e.g., program implementation, training, mentoring and supportive supervision) satisfied with the support provided by the integrated mechanism? What, if any, changes in capacity do they see? CHMTs and RHMTs, Facility supervisors and staff • KIIs/GIs with CHMTs and RHMTs, facility supervisors • Facility checklist • Qualitative analysis and coding of transcripts • Descriptive statistics from facility checklist 3b. Are health care managers/providers receiving integrated technical support at the facility level satisfied with the service provision and quality provided by the integrated mechanism? How are they applying acquired knowledge and skills in the delivery of services at facilities? What, if any, changes in capacity do they see in facilities? Facility supervisors and staff, CSOs, CHMTs, clients • KIIs/GIs with facility supervisors, CSOs, CHMTs • Facility checklist • FGDs with clients • Client exit interviews • Qualitative analysis and coding of transcripts • Descriptive statistics from facility checklist and client exit interviews EQ 4: How effective is Boresha Afya at addressing key gender and youth related barriers to service delivery? IPs, CSOs, CHWs, facility supervisors and staff, clients, youth • KIIs/GIs with IPs, CHWs, CSOs, facility supervisors • Facility checklist • FGDs with clients, youth • Client exit interviews • Qualitative analysis and coding of transcripts • Descriptive statistics from facility checklist and client exit interviews 2.5 Target Areas and Sampling Field research will be carried out in a total of six council level/district sites in six selected regions selected to ensure geographic balance and diversity across the three Boresha Afya zones/IPs. Each zone will have the balanced representation of two selected regions. At each site, similar groups of Boresha Afya stakeholders and clients/beneficiaries will be recruited purposively to participate in KIIs, GIs, or FGDs. Focusing on regional capitals and districts (DCs) will allow the 101 groups of Boresha Afya stakeholders and clients/beneficiaries will be recruited purposively to participate in KIIs, GIs, or FGDs. Focusing on regional capitals and districts (DCs) will allow the ET to conduct KIIs/GIs with regional, municipal, and facility-level informants as well as clients/beneficiaries at a single site. National-level interviews will be conducted in Dar es Salaam, Dodoma, and Zanzibar. Site Selection Six regions of the Boresha Afya project (two in each of the three zones) were purposively selected by USAID as sites for the evaluation. All three Boresha Afya zones were equally represented in terms of the number of regions, districts, and sites (four sites per zone). Randomization using a random number generator (https://stattrek.com/statistics/random￾number-generator.aspx) was used to obtain one council from each selected region. A similar method was used to select facilities after grouping them into two levels; hospitals and primary health facilities (dispensaries and HCs) to strike a balance across the three facility types. Additional criteria for the selection included selection of facilities with the highest number of interventions. Just two out of three facility types are represented per council. Logistic feasibility given the time constraints of the evaluation timeline was considered. • The six regions purposively selected include: Dodoma and Kilimanjaro (in North and Central Zone), Iringa and Morogoro (in Southern Zone) and Mara and Kagera (in Lake and Western Zone). • The districts selected include: Dodoma MC (Dodoma) and Moshi DC (Kilimanjaro), Iringa DC (Iringa) and Mvomero DC (Morogoro), and Ngara DC (Kagera) and Tarime DC (Mara). • The 12 facilities selected across the six councils are shown in Table 5 below. Table 3: Evaluation Design and Methodology North and Central Lake and West Southern Dodoma Kilimanjaro Kagera Mara Morogoro Iringa Dodoma MC Moshi MC Ngara DC Tarime DC Mvomero DC Iringa DC St. Gemma Gilgan Hospital Mwika Health Center Nyamiaga hospital Nyamwaga HC Chazi Hospital Tosamaganga DDH Matumbulu Dispensary Kahe Dispensary Murusagamba HC Nyang’oto Dispensary Kibati HC Itagutwa Disp KII and GI interviewees will be purposefully selected. FGD participants will be randomly selected from lists of clients and youth provided to the ET by program IPs (for identifying non-facility beneficiaries and youth over 18 years of age) and facilities (identify clients) when possible. Should this strategy not be feasible, we will recruit participants through snowball sampling.94 Table 6 presents a summary of the data collection activities programmed in each location. Three teams of 2-3 data collectors will be assigned to data collection in each zone.95 The ET will conduct 11- 94 Due to the scope of the evaluation and the scale of the Boresha Afya program and its three awards under the evaluation, a 40- 50-page report is expected. 95 This is the proposed estimated cost and is based on costs associated with prior evaluations and STTAs current confirmed daily rates. This price is reduced because of the reduced number of STTAs enabled by the number of internal Data for Development staff working on the evaluation. 102 14 KIIs/GIs, six FGDs, 40 exit interviews, and four facility staff checklists per each zone/IP, with an additional 8-16 national-level KIIs. Figure 3: Map of Selected Regions 103 Table 4: Data Collection Activities by Location96 National Level = 14-22 KIIs Location DODOMA DAR ES SALAAM (DSM)/ ZANZIBAR - TOTAL USAID AOR/CORs (KIIs) 1-3 (DSM) 1-3 PO-RALG (KII) 1-2 - 1-2 MOH (KII) 1-2 - 1-2 Zanzibar National Authorities (KII) - 2-3 (ZAN) 2-3 EGPAF and Consortium (KII) - 1-2 (DSM) 1-2 Jhpiego and Consortium (KII) - 1-2 (Mwanza or DSM) 1-2 Deloitte and Consortium (KII) - 1-2 (DSM) 1-2 DE HQ - 1 1 CDC - 1 1 GHSC and or MSD - 1 1 NACP - 1 1 RCHS - 1 1 WHO - 1 1 - North/Central Zone (EGPAF) Lake/Western Zone (Jhpiego) Southern Zone (Deloitte) - - Dodoma Kilimanjaro Kagera w/ Mwanza Mara Morogoro Iringa - Regional Level = 17-20 KIIs/GIs RMO (KII) 1 1 1 1 1 1 6 RHMT (GI) 1 1 1 1 1 1 6 DE Reg Staff (GI) 1 1 2 Prime and Consortium – Regional Staff (GIs) 1-2 1-2 1-2 3-6 District/Council Level = 16 KII/GIs DMO 1 1 1 1 1 1 6 CHMT (GI) 1 1 1 1 1 1 6 CSO (KII/GI) - - 1 1 1 1 4 CHW–mixed gender and mixed inside and outside facilities (GI) 1 1 1 1 1 1 6 Facility and Community Level = 12 KIIs, 36 FGDs, 12 Staff Checklists, 120 Client Exit Interviews-TBD Primary Facility (6 KIIs, 18 FGDs, 6 Staff Checklists, 60 Exit Interviews) Facility Supervisor (KII) 1 1 1 1 1 1 6 104 Clients – each FGD will be recruited from a specific health intervention area 1M-HIV/TB 1F-HIV/TB, 1F￾Other clients (FP/RH, etc.) 1M-HIV/TB 1F-HIV/TB, 1F- (Other clients FP/RH, etc.) 0-1M 1-2F- (Other clients FP/RH MNCH, etc.) 0-1M 1-2F (Other clients FP/RH MNCH, etc.) 1M-HIV/TB 1-2F-HIV/TB, Other clients (FP/RH, etc.) 1M-HIV/TB 1-2F-HIV/TB, Other F Clients (FP/RH,MN CH, etc.) 18 Youth – mixed gender; may be clients (FGD 18 Yrs + of age) 2 2 2 2 2 2 12 Facility Staff Checklist 1 1 1 1 1 1 6 Client Exit Interviews-TBD 10 10 10 10 10 10 60 Secondary Facility (6 KIIs, 60 Client Exit Interviews, 6 Facility Staff Checklists) Facility Supervisor (KII) 1 1 1 1 1 1 6 Client Exit Interviews-TBD 10 10 10 10 10 10 60 Facility Staff Checklist 1 1 1 1 1 1 6 Total KIIs, GIs and FGDs = 89-93 2.6 Required Mission and Implementer Inputs The ET will need an accurate list with contact information (name, title, institutional affiliation, telephone number, email, physical address, and sex) for Boresha Afya prime contractor and subcontractor staff and CSO partners for the sampling and final site selection. The ET will need assistance from both the mission and IPs to contact GOT counterparts and stakeholders at agencies that are actively working with Boresha Afya in the HIV/TB, FP, and malaria space. The following is requested from USAID: • Letter of introduction from USAID to associated health officers at the regional and district levels in sites selected for fieldwork. • Assistance from the mission (and IPs) to contact GOT counterparts and stakeholders at agencies that are actively working with Boresha Afya in the HIV/TB, FP, and malaria space including participation in Results Workshop. • Request access to DHIS2 data and PO-RALG’s information. All project documents including quarterly and annual reports and other relevant background information. • Reports and other relevant materials related to the Development Evaluation across all three IPs. • Any available raw data from the DATIM databases (ET has access to IPRS) to determine what additional quantitative analysis can be conducted beyond the longitudinal analysis using quarter by quarter performance data. From there we can determine if more rigorous statistical methods such as bivariate or multivariate regression analysis can be included in the evaluation design. 105 • Prior to the submission of clearances (at the end of this month), an introduction letter from USAID will be needed to ensure participation in KIIs and to encourage participation in FGDs. • From the IPs, the ET will require: o List of councils and facilities, with their respective program activities, sub-activities, and status (needed from IPs to finalize the sample selection of councils and facilities). o The ET will need an accurate list with contact information (name, title, institutional affiliation, telephone number, email, physical address, and sex) for Boresha Afya prime contractor and subcontractor staff and CSO partners for the sampling and final site selection. o List of Boresha Afya program field personnel and CSO service providers with contact information. o List of Boresha Afya facility workers and/or points of contact who can assist in organizing FGDs with staff and clients/beneficiaries at sites selected for fieldwork; o Introduction and outreach to reach consortium members for joint GIs with prime contractors. o Assistance reaching CSOs or others in contact with CHWs in and outside of facilities for field work for arranging FGDs. The ET will also need USAID and Boresha Afya IP assistance to encourage participation in data collection efforts to help increase response rates so that the ET has as complete information as possible to evaluate program effectiveness. The ET will also need USAID and Boresha Afya assistance to encourage participation in data collection efforts to help increase response rates so that the ET has as complete information as possible to evaluate program effectiveness. An introduction letter from USAID will be needed to ensure participation in KIIs and to encourage participation in FGDs. The ET will work closely with the Mission and IP staff during the design and scoping phase and will update them regularly on key deliverables. Quick review and turnaround from the Mission and IPs in providing feedback on intermediate deliverables will help the ET meet its tight completion timeline. CAPACITY BUILDING PLAN Data for Development, through the evaluation process, will engage a total of three-four local researchers, two to three local institution sub-contractors, and three local Data for Development staff in two-step capacity building sessions. Before data collection, ET members will participate in a two-day training and team building meeting in Dar es Salaam covering the following topics: roles and responsibility of each person and the three regional teams, data collection using the data collection instruments including pre-testing, data quality and QA, field logistics management, transcriptions and translation of recorded audios, qualitative data collection, and qualitative data coding and analysis using Dedoose. The training will involve participatory, hands-on and adult learning approaches, which will enhance participants’ skills in conducting qualitative research. Secondly, during data collection, Data for Development will provide on-the-job training and instruction in quality assurance for ET members. 106 ROLES OF THE TEAM MEMBERS IN RELATION TO THE STUDY Gary Svenson, Team Lead, has a PhD in Public Health, MSW in Social Work, and MSc in Psychology. He is a Licensed Psychologist and has 24 years of professional experience in developing, implementing, researching, evaluating, leading, managing, and overseeing multi￾sectoral health programs addressing HIV/AIDS, reproductive health, adolescent and youth health, health systems, and strategic analysis in developing countries at local, national and regional levels. He has worked in 30+ countries and has 15 years of experience in Africa working with multiple donors including USAID, CDC, PEPFAR, EU, Global Fund, DFID, Netherlands, SIDA/NORAD, WHO, UNAIDS, UNICEF, UNDP, UNFPA, UNODC, IOM, UN IATT/YP, and private sector donors. As team leader, Dr. Svenson will be principally responsible for completing all deliverables, will manage and lead the development of instruments, will lead the fieldwork and data collection and the analysis and triangulation of qualitative and quantitative data into finding conclusions and recommendations. Dr. Svenson will be the principal writer of the report draft and final deliverables. Elvira Beracochea, Health SME, and Health Systems Expert will serve as a health Subject Matter Expert and lead the quantitative instrument development and implementation. She will also lead the fieldwork in the North and Central Zone. Dr. Beracochea will lead the quantitative analysis (including cost effectiveness analysis) with the support of Data for Development and NORC analysts. She will be a principal author to the reports and draft and final deliverables. Dr. M. Mwita will serve as a GOT Co-PI (affiliated with NIMR); he is a Health and HIV Subject Matter Expert who has lead research with the National Institute for Medical Research (NIMR) and has conducted research as a consultant for several international donors across East Africa. Dr. Dominic Mosha will serve as Public Health Specialists conducting FGDs and exit interviews with clients/beneficiaries. They will also conduct the qualitative analysis and contribute to the draft report (particularly findings on EQ 2 and 3). Dr. Dominic Mosha is an epidemiologist with over ten years’ experience in developing and leading health research projects in health facilities and communities. Dr. Mosha has interest in different clinical interventions that aim at improving maternal and child health. Julie Tumbo will serve as a Public Health Specialist in this evaluation. She will conduct a number of interviews at the district and regional level but also, she will participate in the coding, analysis and report writing. She is a HIV/AIDS, Reproductive Health and M&E expert with over 20 years of experience conducting monitoring and evaluation. Ms. Trumbo worked with the Zanzibar AIDS Commission where she helped to develop the Zanzibar National HIV and Sexual Reproductive Health Strategy. Data for Development staff includes Mr. Nasson Konga, Daud Siwalaze, and Bahati Tenga who will serve as evaluation field managers, will lead logistics and field planning, and will provide technical support to the evaluation team; they will also coordinate the research clearance process with NIMR, and will lead the data collection. They will manage the document review process, the storage of data and tracking of progress on data collection and analysis. To coordinate field work, each of the evaluation managers will be assigned to separate sites in each of the three zones to support their STTA team members. 107 Data for Development staff will support all logistics and data collection efforts in the six program sites and will support the team in overall design, facilitate client and IP communication, design the quantitative survey, conduct sampling, and conduct data analysis to be used in the evaluation report. They will also support the analysis of performance data across secondary sources providing data visualization and GIS for the report deliverables. Data for Development will also facilitate the review of interim findings and draft and final reports. Jacob Laden (NORC Country Representative and Data for Development Evaluation Advisor) oversees Data for Development’s evaluation and assessment component. He will advise and co￾develop the evaluation planning, design, and analysis, providing coordination and management support for fieldwork, and will review all intermediate and final deliverables from the team. Overseeing the timely completion of tasks and deliverables, he will support the team lead in managing team roles and responsibilities. EVALUATION LIMITATIONS The proposed evaluation design includes the following limitations: • Data availability and data quality: While the ET will collect and generate primary data, it will rely on the Boresha Afya IPs to provide comprehensive, good quality performance monitoring data. It will also depend on the availability and access to other secondary sources, such as the DHIS2 with the support of PO-RALG. • Selection bias: As some key informants may decline to be interviewed, there is a possibility of selection bias, i.e., those respondents who choose to be interviewed might differ from those who do not in terms of their attitudes and perceptions, affiliation with government/non-government structures, and socio-demographic characteristics and experience. In addition, the purposive nature of the site selection process introduces additional selection bias. Although the sampling method is purposive, CHW and beneficiaries participating in FGDs will be selected by the ET using randomization when possible from client/participant lists; this will help to mitigate bias that could result from self-selection or selection by implementers. • Recall bias: Since a number of questions raised during the interviews will address issues that took place from 2016-2018, informants might not be able to provide accurate and complete responses. This is particularly the case for perspectives on the differences between Boresha Afya’s single mechanism predecessors, which were active prior to 2016. To mitigate this, respondents will be given the option to opt out or select “don’t know/done remember” responses when responding to these questions. • Halo bias: There is a tendency among respondents to under-report socially undesirable answers or alter their responses to approximate what they perceive as the social norm. The extent to which respondents will be prepared to reveal their true opinions may also vary for some questions that call on them to assess the attitudes and perceptions of their colleagues or people on whom they depend for the provision of services. To mitigate this limitation, the ET will outline confidentiality and anonymity statements to all who participate in KIIs, FGDs, and GIs. The evaluation team will also conduct the interviews in a neutral setting where respondents feel comfortable, will conduct separate FGDs for men and women when appropriate, and will conduct GIs with relatively participants of approximately equal social or professional status. 108 EVALUATION TIMELINE AND DELIVERABLES The ET’s anticipated work schedule is provided below. Data for Development Senior Management and Advisors will work with the ET and will have weekly coordination meetings with it for the duration of the evaluation, supervising and managing the process and ensuring the smooth progress of the evaluation. The schedule is designed to provide USAID with preliminary findings at the end of fieldwork in late September and a first draft of the evaluation by the last week of October. The evaluation is anticipated to be complete by the end of November on submission and dissemination of the final evaluation report. 5.1 Evaluation Timeline The ETs anticipated work schedule is provided in Table 7. The schedule is designed to provide USAID/Tanzania with preliminary findings at the end of September 2019 following fieldwork, which depends on Institutional Review Board (IRB) clearance from the Tanzania National Institute of Medical Research (NIMR). Table 5: Timeline and Deliverables Timing Proposed Activities (SOW) June 19, 2019 Discussion of SOW with USAID Preparation and submission of initial work plan and evaluation design July 5-25, 2019 draft and final USAID review of the work plan and evaluation design; COR and CO approval of key personnel July 30, 2019 Preparation and submission of final work plan and evaluation design July 10-31, 2019 Team planning, Preparation of instruments and field work plan ET starts document review August 6, 2019 August 2, 2019 Submission of protocols to NORC review board (IRB) Travel request for STTA sent to USAID August 9, 2019 NORC IRB meeting on August 21 and approval expected the following week if no major revisions are requested August 5-July 20, 2019 Send instruments to USAID for review and to translator for Swahili translation August 6, 2019 Submit approval for NORC clearance August 9-23, 2019 Submit NIMR approval and receive clearance (with expedited processing) August 17, 2019 STTA travel August 19-20 2019 Team building, training and planning meeting/mobilization; data collection pre-test and training; in-brief preparation and delivery; August 26-September 16, 2019 Data collection and fieldwork (2-3 weeks of collection with 3 teams and one day between each site for preliminary analysis of findings) September 16, 2019 Preliminary analysis; Data collection briefing: preliminary findings and conclusions to USAID in out-brief September 16 & 25, 2019 STTA travel September 21-October 11, 2019 Coding of data into Dedoose; Data analysis October 14-30, 2019 Report writing October 31, 2019 Submit Draft Report and Internal Memo November 1-10, 2019 USAID review of Draft Report and submit comments November 11-23, 2019 Incorporate USAID comments; Submit Final Report Week of November 25-29, 2019 Utilization and action planning, after-action review session/meeting and dissemination TBD Upload to DEC (Mission to give approval to upload) 109 Note: Given the short overall timeline, this schedule is predicated on timely approval of USAID/Tanzania for the Evaluation Team proposed in this document, as well as the timely feedback by USAID/Tanzania of deliverables including the instruments and the draft evaluation report. 5.2 Evaluation Deliverables 1. Inputs and discussion on the draft evaluation scope of work – this has been completed. 2. Evaluation design and work plan (this final inception document) with tentative evaluation work plan/schedule indicating which step/activity should occur and when to ensure that the evaluation team fully understands the assignment approach. The design and work plan will be discussed with the USAID technical team in a pre-evaluation meeting and should be approved by USAID prior to implementation. This includes the evaluation design and methodology clearly articulating how the evaluation questions will be answered. 3. Team debriefing will be held immediately following site visits and data analysis to make sure that all questions have been addressed. A debrief presentation of preliminary findings, conclusions, and recommendations will be presented/shared prior to the return of the ET to their home locations. 4. The draft report. The written report will clearly describe findings, conclusions, and recommendations (using the evaluation report template and USAID reporting checklist). The USAID Tanzania team will provide comments on the draft report within 5 working days of submission. 5. A final report that incorporates the team’s responses to Mission comments and suggestions. The final report format will be as follows: Executive Summary, Methodology, Findings (for each EQ). Estimated report length: 40-50 pages.97 BUDGET The total budget for the evaluation is estimated at roughly $375,00098 under the funding of USAID/Tanzania Mission, through ME&A. (See detailed budget in Table 8.) Table 6: Detailed Budget for the Study Cost Item (Justification) Amount in $ A. LABOR-PROFESSIONAL FEES STTAs labor (Evaluation Experts, Assistance Researchers and Translators) Detailed breakdown to be confirmed after signing contracts with proposed staff 96,972 Subcontractor’s costs (Including fee, international travel, allowances, and accommodation). 144,191 B. OPERATIONAL COST Per diems of staff and team members 15,480 Travel and Transportation Cost (Ground & Flight) 18,071 Accommodation the team 34,200 Transport reimbursement and refreshments to FGD participants 14,250 Stationery and Materials-Lump Sum 3000 Translation and transcription costs 36,000 110 Cost Item (Justification) Amount in $ Other direct costs (Local Venue for FGDs) 4,200 IRB Clearance Applications 1,100 Overhead costs 7,536 TOTAL BUDGET-A+B (EXCL. of VAT) 375,000 PROPOSED STAFFING Data for Development has selected an exceptionally qualified team to conduct the mid-term performance evaluation of the Boresha Afya Program. To complete the field work within a three￾week field plan will require three teams of two to three working concurrently in each of the Boresha Afya zones. Overall, the ET will consist of nine team members, five designated short￾term technical assistance (STTA) members, and four Data for Development staff. The evaluation will include two expatriate STTA members who will travel to Tanzania, Gary Svenson, who will serve as Team Leader, and a second international STTA, Elvira Berachochea who will serve as a health SME and lead in the North and Central Zone. The team will also include three to four local STTAs that have substantial experience working on evaluations of complex health programs. Dr. M. Mwita will serve as a GOT Co-PI and is a Health and HIV SME who has lead research with the National Institute for Medical Research (NIMR). Doctor Dominic Mosha and Julie Tumbo (who also served on the evaluation of the Sauti program) will serve as Public Health Specialists and PLHIV Specialists conducting FGDs with clients/beneficiaries. Bernard Kindoli (TBD) if needed will serve as a Health Systems Specialist, data collector, and analyst in the Lake and Western Zone. Data for Development staff includes Mr. Nasson Konga, Daud Siwalaze, and Bahati Tenga (Evaluation Specialists) who will serve as evaluation field managers, will lead logistics and field planning and will provide technical support to the evaluation team; they will also coordinate the research clearance process with NIMR, and data collection. To coordinate field work, each of the evaluation managers will be assigned to separate sites in each of the three zones to support their STTA team members. Jacob Laden (Evaluation Advisor) oversees Data for Development’s evaluation component and will advise and co-develop the evaluation planning, design, and analysis, providing coordination and management support for fieldwork, and will review all intermediate and final deliverables from the team. Data for Development staff will support all logistics and data collection efforts in the six program sites and will support the team in overall design, facilitate client and IP communication, design the quantitative survey, conduct sampling, and conduct the data analysis to be used in the evaluation report. Data for Development will also facilitate the review of interim findings and draft and final reports. During the formal analysis, qualitative analysis specialist, Zoe Grotophorst and a TBD Analyst, from NORC will assist with the qualitative and quantitative analysis. Ms. Grotophorst will assist in the setup of the Dedoose qualitative coding platform and coding frame and will provide training and quality assurance during the analysis. In addition, a quantitative analyst will serve to conduct descriptive and inferential statistics such as correlation and regression analysis which may be performed for EQ 2 and 3 using available secondary data sources. NORC and ME&A home offices will provide operational and technical support and editing and branding on final deliverables. Table 8 provides the estimated level of effort (LOE) for the ET members including all STTAs and Data for Development staff. 111 - - - - - - Table 7: Estimated LOE Per Team Member (Total STTA LOE: up to 356 Days) ✔ Data for Development Staff STTA Expat STTA Expat STTA Local STTA Local STTA Local ✔ ✔ ✔ ✔ NORC HQ Sub Task Team Lead (Gary Svenson) SME (Elvira Beracochea) HIV Specialist and GOT Co PI (M. Mwita) HIV and KP Specialist (DMosha) Health Specialist (Julie Tumbo) M&E Specialist (Nasson Konga) M&E Specialist (Daudi Siwalaze) M&E Specialist (Bahati Tenga) Evaluatio n Advisor & Sr Research Scientist (Jacob Laden) Qualitative and Quant Analysis Specialist (Grotophorst + Quant Analyst) Translators/ 2 FGD facilitators Document review/desk review/work planning (evaluation design remote or in-country) 5 5 3 3 3 2 2 2 6 4 Preparations for travel and organizing data collection (contracting logistics etc.) 2 2 2 2 2 2 2 2 2 Instrument development, evaluation design, protocol preparation and submission 3 3 3 3 3 2 2 2 2 4 Preparations for data collection (scheduling) 1 1 1 1 1 1 1 1 1 2 In-briefing 1 1 1 1 1 1 1 1 1 Field visits 18 18 18 18 18 18 18 18 5 16-18 Preliminary analysis 4 4 3 3 3 3 3 3 2 Out-briefing 1 1 1 1 1 1 1 1 1 Formal Analysis and coding 10 10 10 10 10 10 10 10 7 30 Report drafting 10 10 4 4 4 4 4 4 4 10 Review and submit Final Report 5 2-5 1 1 1 3 Totals (STTA LOE: up to 346 days) 60 days 57-60 days 47 days 47 days 47 days 44 days 44 days 44 days 34 days 48 days 21/facilitator = 42 days 112 ANNEX 3: PERSONS INTERVIEWED KEY INFORMANTS INTERVIEWED No. Name(s) of Interviewee(s) Title (separated with semicolon) Region District Gender 1 UBA Southern Zone Regional Technical Team-Morogoro(Simon Ng'hoboko; Gloria Kokwijjuka; Aikade Nkini; Patrick Kitali; Dominica Lyamuya; Irene Mutayoba; Amos Scott; Modestus Kamonga) M&E Lead; Regional M&EO; Gender and Youth Adv; HIV Adv; Community Adv; Family Planning Adv; Regional Program Manager; Regional Technical Manager Morogoro N/A 4M, 4F 2 CSO Management and Technical Team(Lugano Mwansasu; Sophia Lyimo; Vicent Philipo; Johnson John; Christina Malawa; Lilian Aloyce; Monica Massawe) Community Level Integrated Service Delivery Focal Person(CLISD-FP); CLISD-FP; CLISD-FP; Program Manager; Project Coordinator; Data Clerk; CLISD-FP Morogoro Mvomero DC 3M, 4F 3 CHMT Members DMO; DMIFP; Ag. DACC; DLT; DTLC; DRCH Co.; DHS; DHMIS; DFP Co.; DHTS FP Morogoro Mvomero DC 3M, 3F 4 RHMT Members RHS; RHMIS; RQI; RTLC; Ag. RMIFP; RNuO; Ag. RRCH Co; Ag. RHO; RMO Morogoro N/A 4M, 5F 5 Venance Odilo; Samuel Nassar Medical Officer; PAMO Morogoro Mvomero DC 2M, 0F 6 Matilda Awedaa; Angelina Dissoile RN; EN Morogoro Mvomero DC 0M, 1F 7 Lilian Mlanga; Happiness Sempindu EN; Health System Morogoro Mvomero DC 0M, 1F 8 Angelina Dissoile; Telesia Kato EN; RN Morogoro Mvomero DC 0M, 1F 9 Livin Shayo Medical Officer Morogoro Mvomero DC 1M, 0F 10 Prisca Mtonga; Mtachila Fani; Nakumbukwa Mjema ART Nurse; Data Clerk; Clinician Morogoro Mvomero DC 1M, 2F 11 Aloycia Duma ANO Morogoro Mvomero DC 0M, 1F 12 Agnether Nyamaganda ANO Morogoro Mvomero DC 0M, 1F 13 Celestine Luziga Facility In Charge Morogoro Mvomero DC 0M, 1F 14 Leonia Patrice Mushi et al. ART Nurse; Clinician Morogoro Mvomero DC 0M, 2F 15 Maganga Sinto et al. OPD Person Morogoro Mvomero DC 1M, 0F 16 Ashura Njenga EN Nurse Morogoro Mvomero DC 0M, 1F 113 No. Name(s) of Interviewee(s) Title (separated with semicolon) Region District Gender 17 UBA Southern Zone Regional Technical Team-Iringa (Nelson Haule; Hosea Mang'ombe; Editha Kashija; Julia Mtandu; George Witulo; Atukuzwe Sanga; John Hila; Ndaiga Peter; Charles Mkude; Owen Mhalila; Zegeli; Bilishanga; John Kimario; Faith Dewas; Kakwaya Jumanne; Emmanuel Makundi; Emmy Ibrahim; Aneth Banongo; George Sikalengo; Herbertus Mutayoba; Sophia Samson; Beatus) FP/MNCH Adv; FP/MNCH Adv; PMTCT Adv; TB & TB/HIV Adv; Lab Adv; Pediatric Adv; CBISD Adv; PSCA; MEL; H/A; RPM; RTM; G&Y Adv; H/A; MEO; M&EO; Pediatric Adv.; TB & HIV Adv; M&EO; Community Adv.; District Coordinator Iringa N/A 15M, 6F 18 CSO Management and Technical Team (Castory David; Frida Kipate; Sesilia Masalu; Sikudhani Kiyeyeu; Abisai Kiyasi; Meshack Danda; John Nkoma) Project Coordinator; Data Clerk; Focal Person; Focal Person; Focal Person; Focal Person; Executive Director Iringa Iringa MC 4M, 3F 19 CHMT Members MMOH; MNO; Ag. MQIFP; Ass. MRCH Co; MLT Iringa Iringa MC 3M, 2F 20 RHMT Members RNO; RHS; RLT; RTLC; Ag. RACC; RRCH Co. Iringa N/A 1M, 5F 21 Pilila Zambi MOI Iringa Iringa MC 1M, 0F 22 Maryam Juma CTC In Charge Iringa Iringa MC 0M, 1F 23 Useli Hosea Kisese RCH In Charge Iringa Iringa MC 0M, 1F 24 Mary Makundi Facility In Charge Iringa Iringa MC 0M, 1F 25 Onesmo Ng'ande CTC In Charge Iringa Iringa MC 1M, 0F 26 Pelagia Mwalongo RCH In Charge Iringa Iringa MC 0M, 1F 27 Beatrice Lugina Facility In Charge Iringa Iringa MC 0M, 1F 28 Rahel Richard CTC In Charge Iringa Iringa MC 0M, 1F 29 Yusta Martin RCH In Charge Iringa Iringa MC 0M, 1F 30 Beati Mboya; Gerald Usika; Deo Mwingizi; Gloria COP; DCOP; ZDE; ZDE Dar es Salaam N/A 3M, 1F 31 RMNCH Unit staff RMNCH Unit staff Zanzibar N/A 0M, 3F 114 No. Name(s) of Interviewee(s) Title (separated with semicolon) Region District Gender 32 UBA Lake/Western Zone HQ Technical Team (Deodatus Tibaijuka; Dr Tesha; Goodluck Mwakitosha; Lucy Ikamba; Goodluck Tesha; Juliana; Mary Rwegasira; Juliana Bandambya; Beatus Chikoti; Agnes Kosiwa; Dr. Lusekelo Njonge; Ponsiano Riziki; Dr. Rita Norohna) Technical Director; Snr Clinical Advisor; Applied Health Advisor; Program Advisor; Malaria Technical Lead; HSS Advisor; FP Technical Lead; Technical Advisor Maternal and Newborn; Director of M&E; Associate Director Program Implementation; Technical Advisor Gender; Youth and Respective Care Mwanza N/A 5M, 7F 33 RHMT Members RNO; RHS; RSWO; Pharmacist; RHMIS; RPPPCO; RRCHCO Mara N/A 5M, 3F 34 UBA Lake/Western Zone Regional Technical Team￾Mara(Godlisten Martin; Neema Mashaka; Merengo Joseph; Mary Mwakyusa; Marwa Kitang'ita; Samwel Kalongoje; Neema Mleli; Tuntufye Mwakajonge) Regional Clinical Officer (CO); Midwifery Tech Officer; Technical Advisor; Maternal and Newborn Health; Technical Advisor; Midwifery; Lab Tech Officer MCM; Regional Program Officer; RPM Mara N/A 5M, 3F 35 CSO Management and Technical Team (Simon Chupa; Christina Peter) Regional Coordinator; Program Officer Mara Tarime DC 1M, 1F 36 CHMT Members DMO; HMIS; CHFCO; DNO; CHWCO; PHARMACY; DSWO;DQTFP; DHS; DNO Mara Tarime DC 7M, 3F 37 Monica Ouku; Kilihona Mndaki; Jumanne Mathias; Joel Faustine; Josephine Ninga; Kimori Moreni; Eunice Fredrick; Kennedy Makonyu; Samwel Nashon; Veronica Kwalevele; Mkamis Mirumbe TNuo; RCH Co; Maternity I/c; Nurse; RCH; Ward 4 I/c.; OPD I/c.; HMIS Focal Person; MOI; OT I/c; Asst. Matron Mara Tarime DC 6M, 5F 38 Dr. Samwel Nashon MOI Mara Tarime DC 1M, 0F 39 Levinatha Rwekenya; Neema Opanga; Bhoke Sumara; John Magesa; Lilian Achieng; Josephat Kelambo; Chausiku Simion; Kerarya Bhoke; Daniel Geteema; Denis Mzaula EN; D/c; D/c; RAD; EN; AMO; EN; Nurse; LT; CO Mara Tarime DC 5M, 5F 115 No. Name(s) of Interviewee(s) Title (separated with semicolon) Region District Gender 40 Josephat Kalambo MOI Mara Tarime DC 1M, 0F 41 Elirehema Mbanga; Isack John; Hassan Mgesi; Samson Samwel; Patrick Mali; Eliud Kihongo; Hawa Kindale; Kunibert Mangosongo; Dr. Amos Manya RN; AMO; CO; EN; EN; EN;RN Mara Tarime DC 8M, 1F 42 Dr. Amos Manya MOI Mara Tarime DC 1M, 0F 43 UBA Lake/Western Zone Regional Technical Team￾Kagera(Fredrick Orembo; Julius Majula; Deozawadi Marandu; Cherry Ikanga; Joachim Mazima; George William; Alphoncina Balongo; Dr. Mkamba; Nathan Bagaya) RPM; MNCH Advisor; M&E Officer; PO; TO-FP+RH; TO￾MCM; Advisor Midwifery; STO￾Comm Engagement; Tech Lead; TO-RMNCH Kagera N/A 8M, 2F 44 RHMT Members RSWO; RNO; RNvO; RHS; RHO; RHMI Kagera N/A 2M, 4F 45 CSO Management and Technical Team (Tonny Mugenya; Beatrice Mtani; Devotha Henericko) PO; PO; PO Kagera Ngara DC 1M, 2F 46 CHMT Members CHW Co.; DNO; DPHARM; DMFP; Asst DRCH Co.; DRCH Co.; DHMISCO; DNDCD Co.; RMA; DNuO; PPP Co.; DHS; DASS Pharm; Ag. DMO; DHO; DCBHS Co. Kagera Ngara DC 10M, 6F 47 Dr William Mnyonyela; Regina Soloma; Aliadina Nestory; Hirary Nkonyagi; Dorothy Mbonamasabo; Geraldina Gerald; Regina John; Happy Mnulisa; Jackson Izengo; Dorica Alphonce; Esther Kabigumila; Pauline Wayda; Suzana Itandula MOI; Health Secretary; Matron; Asst Patron; L&D In Charge; Female ward In Charge; CTC I/c; Pediatric; Pharmacy; RCH; RCH; LAB; Nurse Kagera Ngara DC 3M, 11F 48 Dr. William Mnyonyela MOI Kagera Ngara DC 1M, 0F 116 No. Name(s) of Interviewee(s) Title (separated with semicolon) Region District Gender 49 Dr. Bithia Sizimwe; Helbut G; Juliana Damiano; Marcia Merchades; Kandida Sendegaya; Hadija Hirary; Dorah Joseph; Dorcas Sebuyoya; Sabrina Elia; Jacqueline John; Athanas Samwel AMO; Asst Pharmacy; Matron; Data Officer; RCH I/c; Lab Tech; EN; Acc; RCH; CTC I/c; lAB Kagera Ngara DC 2M, 9F 50 Dr Bithia Sizimwe MOI Kagera Ngara DC 1M, 0F 51 Paul Mafulu; Rahma Mkojela; Sekizia Milanga; Allen George; John Gwassa; Amos Zephrine; Essauh Michael; Themistoclass Rubangwa; Deodatha Senzige; Joyce Dawson; Ladslaus Kabila; Happy Nyoni; Dr. Gaston Beyango; Raston John; Anonciatha Gozbath CO; RN; RN; Pharm; M/Att.; CO; EN; LAB; Data Officer; Lab; Lab; AMO; Matt; RN Kagera Ngara DC 9M, 5F 52 Dr. Gaston Beyango MOI Kagera Ngara DC 1M, 0F 53 UBA Southern Zone HQ Technical and Management Team (Marina Njelekela; Zahra Nensi; Jolia Gamaliel; Ramadhan Ally; Simon Mbele; Caroline Mushi; Albert Ikonje; Beatrice Chriastian) COP; CONSULTING; DCOP; Manager-FACILITY-Based Services; Manager-Quality Improvement; Manager-Gender and Youth; Manager-Community￾Based Services; Director of Strategic Information; Senior Technical and Operation Advisor Dar es Salaam 5M, 4F 54 UBA North/Central Zone HQ Technical and Management Team (Roland Van de Ven; Doroth Matoyo; Bonita Kilama; Chrispine Kimaro) Technical Director; Associate Director Project Implementation; Associate Director Strategic Information and Evaluation; Associate Director; Technical Services Dar es Salaam N/A 2M, 2F 55 Dr. Albert Komba Chief of Party-SAUTI Project Dar es Salaam N/A 1M, 0F 56 Dr. Kakohumbya Kazaura Prevention Branch Chief Dar es Salaam N/A 0M, 1F 57 Dr. Azzah Nofly Program Analyst; SRH/HIV Zanzibar N/A 0M, 1F 58 Joseph Obedi; Masanja Kambenga Senior Project Manager; Health Information System Officer Dodoma N/A 2M, 0F 59 Dr. Felix Bundala Head Newborn Child Health Dodoma N/A 1M, 0F 60 Tufingeni Malambugi Officer Nutrition Section Dodoma N/A 0M, 1F 61 Erick Jackson Kitali Director ICT Dodoma N/A 1M, 0F 117 No. Name(s) of Interviewee(s) Title (separated with semicolon) Region District Gender 62 Dr. Ntuli A. Kapolongwe Director of Health Services (DHS) Dodoma N/A 1M, 0F 63 Dr. Lenard Subi Director of Preventive Dodoma N/A 1M, 0F 64 Zuhura Mbuguni Ag. National Family Planning Coordinator Dodoma N/A 0M, 1F 65 Dr. Isaya Jelly TB/HIV Coordinator Dodoma N/A 1M, 0F 66 Zuiwena Kondo Focal Person for Implementation of TB work plan Dodoma N/A 0M, 1F 67 RHMT Members Regional Reproductive Child Adolescent Health Coordinator; Regional Mental Health Coordinator; Head of Department Pediatric; Asst. Regional Community Based Health Services; Regional TB & Leprosy Coordinator; Regional Community Based Health Services; Regional AIDs Control Coordinator; Reginal Social Welfare Dodoma N/A 2M, 9F 68 CHMT Members City Medical Officer; City Laboratory Technologist; City TB HIV Officer; Ag. Council AIDs Control Coordinator; Asst. Reproductive Child Health Coordinator; Council Pharmacist; Council Social Welfare Officer Dodoma Dodoma MC 3M, 4F 69 Dr. Winfredy Mwafongo Ag. Program Manager Dodoma N/A 1M, 0F 70 Bwigane Afwene Nurse Dodoma Dodoma MC 1M, 0F 71 Zainab Maslombo CTC In Charge Dodoma Dodoma MC 1M, 0F 72 No name Nurse In Charge Dodoma Dodoma MC 0M, 1F 73 Ruth A. Masimba Nurse Dodoma Dodoma MC 0M, 1F 74 Anastazia Maiga Nurse Dodoma Dodoma MC 0M, 1F 75 George Matiko MD Dodoma Dodoma MC 1M, 0F 76 Godray Rujabuke Nurse Dodoma Dodoma MC 1M, 0F 77 Rahma MOHamedi CTC In Charge Dodoma Dodoma MC 0M, 1F 78 Magdalena Hoya OPD In Charge Dodoma Dodoma MC 0M, 1F 79 No name Nurse Dodoma Dodoma MC 1M, 0F 80 Loy Mazengo Clinical Officer Dodoma Dodoma MC 0M, 1F 81 UBA North/Central Zone Technical Team￾Kilimanjaro(Jonathan Yona; Safiel Neneka; Maria Maro; Abdul Mpanga; Gerald Tesha) Associate Project Manager; Family Plan and Reproductive Health Coordinator; Council Project Coordinator; Project Officer Laboratory Services; Associates Health Information Systems Officer Kilimanjaro N/A 4M, 1F 118 No. Name(s) of Interviewee(s) Title (separated with semicolon) Region District Gender 82 RHMT Members Regional Health Secretary; Regional Community Based Health Services Coordinator; Regional Health Laboratory Services; Regional Radiology Department; Regional HIV/Testing Services Focal Person; Regional Reproductive Child Health Coordinator; Ag. Regional AIDs Control Coordinator; Regional Social Welfare Officer; Regional Health Management Information System (RHMIS) Kilimanjaro N/A 5M, 6F 83 CHMT Members District Reproductive Child Health Coordinator; District Community Based Health Services Coordinator; Council and Testing Coordinator; District Quality Improvement Focal Person; District Pharmacist; District Social Welfare Officer; District Laboratory Technician; District AIDs Control Coordinator; District Medical Officers; District Hospital Secretary; District TB/HIV coordinator; District DHIS2 person Kilimanjaro Hai DC 5M, 7F 84 Verediana Michael Registered Nurse Kilimanjaro Hai DC 0M, 1F 85 Jane I. Macha MD Kilimanjaro Hai DC 0M, 1F 86 Lukumbwe R. Masawe Nurse Kilimanjaro Hai DC 0M, 1F 87 Aingaya G. Mlay Enrolled Nurse Kilimanjaro Hai DC 0M, 1F 88 Redfan E. Shao Asst. MD Kilimanjaro Hai DC 1M, 0F 89 Josephina S. Kabululu MD Kilimanjaro Hai DC 0M, 1F 90 Anna T Macha Nurse Kilimanjaro Hai DC 0M, 1F 91 Rose Nemes Mushi Nurse Kilimanjaro Hai DC 0M, 1F 92 Monica Shirima Nurse Kilimanjaro Hai DC 0M, 1F 93 Dr. Escor N. Tweve Dental Officer Dodoma Dodoma MC 1M, 0F 94 Dr. Motto Assistant Medical Officer Dodoma Dodoma MC 1M, 0F 95 Caroline Amos Kingu Nurse Dodoma Dodoma MC 0M, 1F 96 Domina R. Kimaro Registered Nurse Dodoma Dodoma MC 0M, 1F 97 Dr. Richard Amaro Asst. MD Kilimanjaro Hai DC 1M, 0F 98 Fortunata Mchomba Nurse Kilimanjaro Hai DC 0M, 1F 99 Linda Bernard Asst. MO Kilimanjaro Hai DC 0M, 1F 100 Luyce Wilson Ngowi Enrolled Nurse Kilimanjaro Hai DC 0M, 1F 101 Fortunata Mchomba Nurse Kilimanjaro Hai DC 0M, 1F 102 Ms. Jema Bisimba AOR UBASZ – USAID Dar es Salaam N/A 0M, 1F 103 Dr. Patrick Swai Facility Team Lead-USAID Dar es Salaam N/A 1M, 0F 104 Jacqueline Kalimunda AOR UBANCZ – USAID Dar es Salaam N/A 0M, 1F 119 No. Name(s) of Interviewee(s) Title (separated with semicolon) Region District Gender 105 Dr Ashery Barankana Sr. Technical Advisor Dar es Salaam N/A 1M, 0F 106 Jacqueline Larsen COP Dar es Salaam N/A 0M, 1F 107 Lucy Ikamba Program Advisor and Lead in the office; Tech Advisor for Reproductive Health Zanzibar N/A 1M, 1F 108 Head of HMIS Head of HMIS Zanzibar N/A 1M, 0F 109 Deogratius Rutatwa; Agnes Nyoni Chief Executive Officer; Program Manager Dar es Salaam N/A 1M, 1F 110 Gemin Mtei; Desderi Wengaa; Sheila Odougherty Finance Team Lead; Information Systems Team Lead; DCOP Dar es Salaam N/A 2M, 0F 111 Abdullah Ali; Faiza B. Abbas; Safia Mohammed Aei; Majda Hassan Nassor; Said Haji; Kali Abdullah Omar; Bakar Juma Mohammed; Program Manager ZAMEP; Deputy Program Manager; Head; Malaria Diagnosis; Officer; Diagnostic Unit; Diagnostic Officer; Diagnostic Officer; Diagnostic Officer Zanzibar N/A 4M, 3F 112 Dr. Ahmad Makuwani Ag. Director RCH Dodoma N/A 1M, 0F 113 Dr. Phineas Sospeter; Ms. Leyla Bungire Coordinator Safe Mother Initiative; Officer Safe Mother Initiative Dodoma N/A 1M, 1F 114 Dr. Yahaya Hussein RMNCH Coordinator Dodoma N/A 1M, 0F 115 Dr. Andrew Komba Director Sector Coordination Dodoma N/A 1M, 0F 116 Mr. Aminieli Macha Director M&E Dodoma N/A 1M, 0F 117 Ezra Mwijarubi; Selina Mathias; Emmanuel Tluway; Todd Koppenhaver; Chonge Kitojo; Erik Jason Reaves; Naomi Serbantez USAID Health Team(TB; FP/HIV; FP; SI; Malaria) Dar es Salaam N/A 4M, 3F 118 Chonge Kitojo Lake/Western Zone AOR Dar es Salaam N/A 0M, 1F Total 386 120 FOCUS GROUP PARTICIPANTS FGD Participants by Gender Type of Stakeholder Gender Total Participants Stakeholder Type District Male Female Youth Mvomero DC 5 8 13 Male Involvement Mvomero DC 9 0 9 PLHIV Women Mvomero DC 0 10 10 CHW Mvomero DC 4 4 8 Youth Mvomero DC 3 8 11 Male Involvement Mvomero DC 11 0 11 PLHIV Women Mvomero DC 0 13 13 Male Involvement Iringa MC 10 0 10 PLHIV Women Iringa MC 0 15 15 CHW Iringa MC 5 13 18 Youth Iringa MC 5 5 10 Male Involvement Iringa MC 9 0 9 PLHIV Women Iringa MC 0 11 11 RMNCH Women Tarime DC 0 12 12 Male Involvement Tarime DC 8 0 8 CHW Tarime DC 7 5 12 CHW Tarime DC 3 3 6 RMNCH Women Tarime DC 0 16 16 Male Involvement Tarime DC 8 0 8 RMNCH Women Ngara DC 0 13 13 Male Involvement Ngara DC 17 0 17 CHW Ngara DC 7 1 8 RMNCH Women Ngara DC 0 12 12 Male Involvement Ngara DC 15 0 15 CHW Dodoma MC 5 10 15 Youth Dodoma MC 5 9 14 PLHIV Women Dodoma MC 0 7 7 Youth Dodoma MC 7 5 12 CHW Hai DC 3 8 11 Youth Hai DC 8 7 15 PLHIV Women Hai DC 0 15 15 PLHIV Women Hai DC 0 15 15 Male Involvement Hai DC 15 0 15 CHW Ngara DC 6 2 8 - Total 175 227 402 121 ANNEX 4: SOURCES OF INFORMATION/BIBLIOGRAPHY OF DOCUMENTS REVIEWED A Comprehensive Health Service Delivery (CHSD)-Feedback-Rapid Gender Analysis Assessing-the-benefits-of-integrated-HIV-and-SRH-services-Kenya-Swaziland-Intervention￾Report-2012 Baseline Assessment Report, USAID Boresha Afya Lake and Western Zone July 2017 Birdthistle-et-al-2014-Integration-of-HIV-and-maternal BORESHA AFYA LW Y2 WORK PLAN APPROVAL Letter Building integrated health systems lessons CDCS Tanzania Final 2014-2019 Client Satisfaction Assessments, April 2019_REPORT 20June2019 Condom Distribution Guide January 2019 Consolidated PY3 Q1 2018 Implementation Plan v30 09 2018 Cost and technical efficiency HIV SRH integration Cost effectiveness of HIV prevention in Africa Cost-effectiveness-of-Integrating-PMTCT-and-MNCH.2013 Data Management SOP USAID Boresha Afya Lake and western zone Data Management Supportive Supervision Checklist – USAID Boresha Afya DE Knowledge Brief-SSI Delivering community-led integrated HIV and SRH Does HIV SRH service integration improve technical quality. Low Resource countries Effects of Integrated Delivery System on Cost and Quality Final Boresha Afya MEL Plan Approved 24th June 2017 updated 19 October 2017 Final Report for MTR of One Plan I Final Formative assessment report on Gender & Responsiveness services July 2017 FP-HIV-Evidence-Based-Practices-2013 Frameworks for Assessing Integrated Care GENDER INTEGRATION CHECKLIST - Boresha AFYA North and Central Zone - Revised Handbook for Coordinating Gender-based Violence Interventions in Emergencies Health Sector HIV and AIDS Strategic Plan (HSHSP III) (2013-2017) Health Sector HIV and AIDS Strategic Plan (HSHSP IV) (2017-2022) Health Sector Strategic Plan IV (HSSP IV) (2015-2020) Health-Research-Program-Overview How linked are national HIV and SRHR. 60 country review Indicators and measurement tools for health system integration Integrated Assessment and Management of Healthcare Infrastructure and technology 122 Integrated Delivery Systems Ids as a Means of reducing costs and improving healthcare delivery (1) Integrating patients’ perspectives into integrated TB and HIV linkages evidence review 2009 en List of Apps and other digital tools developed by JHPIEGO Boresha Afya project Making Adolescent Friendly _ Developing Quality National Standards for Adolescent Friendly Health Services Malaria Program Performance Review Tanzania Mainland 2012 MATERNAL, NEWBORN, and Child Health In Tanzania-Costs and Impacts of the One Plan II Medical Store Department-FINAL MTSP II DISTRIBUTION 2014-2020 MONITORING & EVALUATION PLAN FOR COLLABORATIVE TB HIV ACTIVITIES April 2018 National Guidelines for Recognition of Implementation Status of Quality Improvement Initiatives in Health Facilities NATIONAL GUIDELINES FOR THE MANAGEMENT OF HIV AND AIDS Sixth Edition October 2017 National Nutrition Strategy (2011- 2016) NATIONAL POLICY GUIDELINES FOR COLLABORATIVE TBHIV ACTIVITIES 2016 National Road Map Strategic Plan to Improve Reproductive, Maternal, Newborn, Child and Adolescent Health in Tanzania (2016-2020) (One Plan II National Strategy Plan V For Tuberculosis and Leprosy Programme (2015-2020) National Training Package on Adolescents Living with HIV and AIDS-Participants_Manual 2018 Numbers, systems, people. How interactions influence integration Overview-of-Research-Components-20111 Pages from USAID Boresha Afya Southern Zone - Results Framework with MEL Plan PEPFAR 2019 Country Operational Plan Guidance for all PEPFAR Countries PEPFAR Updated Gender Strategy 2014 Performing Economic Evaluation of Integrated Care PS3 Annual Report Qual study of integrated HIV and SRH in South Africa Rapid Assessment HIV SRH Tanzania RFA-621-16-000012 - CHSD Notice of Funding Opportunity(NOFO) Social Analysis and Action (SAA) in Food and Nutrition Security Programming SOP for Data Management SRH HIV Linkages Compendium SRH, HIV and other Services Guideline Study- Comprehensive Women’s Healthcare HIV, RH and maternal health in South Africa Study on integrated family planning and HIV care services in South Africa 123 Supplementary Gender Assessment Report April 2018 Tanzania - Malaria Operational Plan FY 2019 Tanzania COP 2018 Strategic Direction Summary April 17, 2018 Tanzania Fourth National Multisectoral Strategic Framework for HIV and AIDS (2018/19 to 2022/23)(NMSF IV) Tanzania FP2020 Commitment 2017 Tanzania Malaria Indicator Survey Report 2017 Tanzania National Family Planning Costed Implementation Plan (2010-2015) Tanzania Third National Multisectoral Strategic Framework for HIV and AIDS (2013/14- 2017/18)(NMSF III) Tanzania-Developmental-Evaluation-Enculturation-and-Inception-Workshop Final 07-31-2018 UBA North_Central Zone - Gender Action Plan Updated USAID Boresha Afya LW Zone Performance Indicator Tracking Table Targets USAID Boresha Afya EGPAF Q1 Report Oct-Dec 2016 USAID Boresha Afya EGPAF Q1 Report Oct-Dec 2018 USAID Boresha Afya EGPAF Q1 Report Oct-Dec 2018 USAID Boresha Afya EGPAF Q2 Report Jan-Mar 2017 USAID Boresha Afya EGPAF Q2 Report Jan-Mar 2018 USAID Boresha Afya EGPAF Q2 Report Jan-Mar 2019 USAID Boresha Afya EGPAF Q3 Report Apr-Jun 2017 USAID Boresha Afya EGPAF Q3 Report Apr-Jun 2018 USAID Boresha Afya EGPAF Q4+Annual Oct 2016-Sep 2017 USAID Boresha Afya EGPAF Q4+Annual Oct 2016-Sep 2018 USAID Boresha Afya LW Q1 Performance Report Oct-Dec 2016 USAID Boresha Afya LW Q1 Performance Report Oct-Dec 2017 USAID Boresha Afya LW Q1 Performance Report Oct-Dec 2018 USAID Boresha Afya LW Q2 Performance Report Jan-Mar 2017 USAID Boresha Afya LW Q2 Performance Report Jan-Mar 2018 USAID Boresha Afya LW Q3 Performance Report Apr-Jun 2017 USAID Boresha Afya LW Q3 Performance Report Apr-Jun 2018 USAID Boresha Afya LW Q4+Annual Oct 2016-Sep 2017 USAID Boresha Afya LW Q4+Annual Oct 2017-Sep 2018 USAID Boresha Afya Southern Zone - FY19 Workplan - Re-Submitted Version After Comments USAID Boresha Afya Southern Zone - Updated MEL Plan USAID Boresha Afya Southern Zone - Year 2 Revised Narrative Work Plan - FY 2018 - 17 Oct 2017 USAID Boresha Afya Southern Zone REVISED WORK PLAN Narrative for FY 2017-25 03 17 124 USAID Boresha Afya Southern Zone Submitted Q1 Performance Report Oct-Dec 2016 USAID Boresha Afya Southern Zone Submitted Q1 Performance Report Oct-Dec 2017 USAID Boresha Afya Southern Zone Submitted Q1 Performance Report Oct-Dec 2018 USAID Boresha Afya Southern Zone Submitted Q2 Performance Report Jan-Mar 2017 USAID Boresha Afya Southern Zone Submitted Q2 Performance Report Jan-Mar 2018 USAID Boresha Afya Southern Zone Submitted Q2 Performance Report Jan-Mar 2019 USAID Boresha Afya Southern Zone Submitted Q3 Performance Report Apr-Jun 2017 USAID Boresha Afya Southern Zone Submitted Q3 Performance Report Apr-Jun 2018.docx USAID Boresha Afya Southern Zone Submitted Q4+Annual Oct 2016-Sep 2017 USAID Boresha Afya Southern Zone Submitted Q4+Annual Oct 2017-Sep 2018 USAID BORESHA AFYA WORK PLAN GANTT CHART - PY1 USAID BORESHA AFYA WORK PLAN GANTT CHART - PY2 Oct 17 USAID BORESHA AFYA WORK PLAN GANTT CHART - PY3-May 2019 USAID Boresha Afya Year 1 Workplan Jhpiego Approved Version 9 March 2017 USAID Boresha Afya-LW PY03 Workplan Final Nov 2018-For Implementation USAID TZ HSS Strategy Sept 30 FINAL Utilization of integrated HIV and sexual and reproductive health services in Uganda WHO Country Cooperation Strategy (2016-2020) WHO technical brief. Integrated health services 125 ANNEX 5: DATA COLLECTION INSTRUMENTS Introduction and Consent Prior to all KIIs, the following consent statement should be read out loud to each respondent: Hello and thank you for agreeing to speak with us. My name is (moderator name) and this is my colleague (note taker name). We work with the Data for Development project, a United States Agency for International Development funded platform that is requested to evaluate the performance of the Boresha Afya program to strengthen and integrate health services in Tanzania. As part of the evaluation, we are conducting interviews with project partners, participating health services and community members in contact with the project. One of the goals of the evaluation is to improve the services provided to clients and communities. We request you to take some minutes to answer some questions, so we can capture your viewpoints. We want to understand what has worked well and what has not worked well in the Boresha Afya project to improve health services for people in this area. The project is implemented by [insert implementer name]. We would like to interview you about the implementation of this project. Your perspective is very important. This discussion will last approximately one hour. Your participation in this interview is entirely voluntary and you can choose not to answer a question and skip it or stop the interview at any moment without providing a reason. The information we will be collecting through this interview will be kept safe by our team. Your responses will be kept anonymous and not linked to your name: each person interviewed will be given a unique identification number so your identity will be kept confidential and will not be shared outside of the evaluation team including USAID or the Tanzanian government. Other information that could identify you (e.g., position, organization, community, and district) will be excluded from reports and other documents produced by our team. With your consent, we would like to record this interview so that we can analyze it along with those of other interviewees. The content will be transcribed into a text format that will not include your name and title. It cannot be traced to you personally and will be destroyed after 6 months. If you have any further questions about the evaluation feel free to contact [Tanzania-based Evaluation Team POC Nasson Konga] at [nkonga@engl.com] or by telephone at [0715 201618]. We will provide this information to you on a piece of paper. Do you agree to participate in this interview today? Yes No May I start the recorder? Yes No [IF THE RESPONDENT SAYS “YES”, BEGIN INTERVIEW The following information should be collected for each KII respondent: 126 1 Type of respondent • Facility Supervisor • Facility Staff • RMO/RHMT • DMO/DHMT • USAID • IP • GOT – National • CSO • National Stakeholder (Other) 2 Agency or institution (if applicable) . 3 Respondent title or position (if applicable) . 4 Zone (if applicable) • North/Central • West/Lake • Southern 5 Region (if applicable) • Dodoma • Mara • Iringa • Morogoro • Kagera • Kilimanjaro 6 Interview location . 7 Interview date . 8 Interviewer . 9 Note taker . 10 Start time . 11 End time . 12 Interview duration (minutes) . 13 Language of interview . 127 Introduction and Consent Prior to all FGDs, the following consent statement should be read out loud to each respondent: Hello and thank you for agreeing to speak with us. My name is_______________ (moderator name) and this is my colleague__________________ (note taker name). We work with the Data for Development project, a USAID funded platform that seeks to improve the quality and use of data in decision-making in Tanzania. We are undertaking an evaluation for USAID to understand what has worked well and what has not worked well in the USAID-funded Boresha Afya activity that aims to strengthen and integrate health services for people in this area. The project is implemented by [insert implementer name]. We would like to interview you about the implementation of this project. Your perspective is very important to help USAID improve its programs. This discussion will last approximately one hour. Your participation in this interview is entirely voluntary and you can choose not to answer a question and leave the discussion at any moment without providing a reason. Your privacy will be protected; we will not include any information in any report that would make it possible to identify you. Please note that we cannot guarantee full confidentiality because of the group setting, as we cannot ensure that participants will not disclose any information shared during the focus group. Once again, we ask that what we discuss during our group talk remains here with us. If you have any further questions about the evaluation feel free to contact [Tanzania-based Evaluation Team POC Nasson Konga] at [nkonga@engl.com] or by telephone at [0715 201618]. We will provide this information to you on a piece of paper. Do you agree to participate in this interview today? Yes No May I start the recorder? Yes No [IF THE RESPONDENT SAYS “YES”, BEGIN INTERVIEW FGD Participant Registration Form: Respondent Category: • Community Health Worker • Youth Group • Women Living With HIV • Men’s Involvement Group Moderator: __________________________ Note-taker: ___________________________ Start time: ___ :___ AM/PM (circle one) End time: ___ :___ AM/PM (circle one) Region: District: Locality: Date: Month: ___ Day: ___ 2019 128 Participant Main Occupation (farmer, shopkeeper, teacher, etc.) Gender (M/F) Age Village 1. 2. 3. 4. 5. 6. 7. 8. 9. 10. Comments on any aspect of the FGD: 129 Community Health Workers (CHW) FGD GUIDE Introduction 1. Kindly describe how you were recruited to be involved in Boresha Afya or its supported facilities to integrate and strengthen health services. a. What type of organization recruited you—government (facility or CHMT), Boresha Afya￾supported CSO/NGO, another CSO/NGO, or some other type of organization? [Note to moderator: Get counts for each] b. How long ago did you start working with Boresha Afya? Boresha Afya Involvement 2. Please describe the activities you do as part of Boresha Afya to integrate and strengthen health services for women and youth in your community. Describe the activities and the health service areas in which you work (e.g., MNCH, HIV, TB, FP, Malaria, Gender, Youth, and GBV, etc.) a. Which activities have been most successful? b. Which activities have been most challenging? 3. Do you make patient referrals on a regular basis? How do you track or ensure that the patient appeared at the referred services? 4. How do you characterize the health facilities and the health system's response to your activities with Boresha Afya? a. Have they been cooperative and respectful of your involvement? 5. What has been the local community’s response to your activities with Boresha Afya? a. Have they given you acceptance and recognition for your contribution? b. Are they aware of your services? 6. What do different members of the community say about the quality of services at the facility? Have they noted any change as a result of Boresha Afya’s support to integrated care? Now I have a few questions about the training and support you have received on integrated care through Boresha Afya. 7. Kindly describe any initial and refresher trainings you have received on integrated care as a CHW/ volunteer. What did you learn, and was useful for your work? a. Who provided that training? [Note to moderator: You must determine whether training was conducted as part of Boresha Afya activities.] b. When was the training and how long was it? 8. How do you apply this involvement and training in your daily work with the clients and community members you serve? 130 9. Please describe the support and mentoring provided to you by Boresha Afya. [Moderator probe if needed: technical support, clear communication on your responsibilities, sufficient autonomy, financial and commodity support, etc.] a. How satisfied are you with this support and mentoring? b. What further support do you need? 10. Please describe any ongoing support you’ve received from Boresha Afya specifically to improve cooperation and collaboration with the health care system, your community, other organizations and your peers (other CHWs). a. How satisfied are you with this support? b. What further support do you need? Closing 11. Boresha Afya is at its midterm point, what types of changes or improvements would you like to see moving forward, including addressing gaps? What do you perceive as the priorities and opportunities for Boresha Afya? LOCAL CSO KII GUIDE Introduction 1. Kindly describe your organization and the activities it carries out to integrate and strengthen health services particularly for women and youth. Boresha Afya Involvement 2. Please describe your organization’s involvement or activities with Boresha Afya. Describe the activities and the health service areas in which your organization works (e.g., MNCH, HIV, TB, FP, Malaria, Gender, Youth, and GBV, etc.) a. How long has your organization been involved with Boresha Afya? b. Who is involved in Boresha Afya from your organization—both in terms of personnel and volunteers? 3. Overall, how would you describe the quality of your interaction with Boresha Afya and its implementers? For instance, have they been responsive to your organization’s needs, input and requests? a. Have there been any challenges? If so, were they addressed? 4. Are your organization’s Boresha Afya activities carried out in cooperation and collaboration with the local or other health facilities? Please describe the activities, associated health service areas, who is involved, and how long they have been active. 5. Are your organization’s Boresha Afya activities carried out in cooperation and collaboration with other CSO’s or community representatives, including youth? Please describe the activities, associated health service areas, who is involved, and how long they have been active. 131 6. How do you characterize the health facilities and the health system’s response to your activities with Boresha Afya? a. Have they been cooperative and respectful of your involvement? b. Have they created any challenges for you? If so, have they been addressed? 7. What has been the local community’s response to your activities with Boresha Afya? a. Are they aware of your services? b. Have they given you acceptance and recognition for your contribution? c. Have they created any challenges for you? If so, have they been addressed? 8. What types of support did your organization receive from Boresha Afya, including any capacity building, technical support, management and program development, working with communities, MEL, and financial and commodity support? a. How satisfied are you with the support received? b. What could be improved? What further support do you need? 9. Kindly describe any support you’ve received from UBA to improve cooperation and collaboration between your organization and other CSOs, stakeholders and the health system at the facility, district, regional, and national levels. Please provide details. 10. How has Boresha Afya helped your organization strengthen and integrate health services, particularly for women and youth? Please give specific examples. a. Please provide examples of when this process has been successful. Which were the most important ones? b. Please provide examples when this process has faced challenges or failed. Which were the most important ones? 11. In your own opinion, what are the key components or processes necessary for the successful integration of health care in Tanzania? Provide examples. [Interviewer probe if needed: policies and regulations; technical support; capacity building and knowledge management; management (cooperation and collaboration); strategic information] a. How well has Boresha Afya addressed these components or processes? Sustainability 12. In your opinion, what steps have been achieved towards the sustainability of Boresha Afya’s contribution, including the current U.S. policy focus on recipient self-reliance? [Interviewer probe if needed: policy and laws facilitating integrated health; improvements in healthcare systems and services; capacity of health service providers and support systems, e.g., MEL, supply chain; involvement of communities and marginalized groups, cooperation and collaboration; structural and social barriers, e.g., gender, stigma, youth involvement; cost-effectiveness] 132 Closing 13. Boresha Afya is at its midterm point, what types of changes or improvements would you like to see moving forward, including addressing gaps? What do you perceive as priorities and opportunities for Boresha Afya? FACILITY STAFF—HOSPITALS, HEALTH CENTERS, AND DISPENSARIES—KII GUIDE (HIV/CTC, TB, MNCH, FP/SRH, YOUTH, MALARIA, IPD, OPD, DATA CLERKS OFFICE, LAB/DIAGNOSTICS, PHARMACY) 1. Kindly provide an overview of the types of health conditions your unit addresses and the specific services it provides. 2. What are the most common health conditions or patient requirements you work with? a. Which of these require the greatest amount of your time and the unit’s resources? 3. How do issues concerning a patient’s gender or young age get taken into account in the delivery of quality services? How do you address these? 4. Does your unit work in direct collaboration with CHWs/volunteers, or youth groups, or carry out community outreach activities? Kindly describe. As you may know, this facility is part of a program called Boresha Afya to integrate health care to improve services for women and youth and increase client and health worker satisfaction. It is implemented in this area by [implementer] and is supported by USAID and the Tanzanian health services. 5. Have you had interactions or trainings with Boresha Afya or [implementer]? [IF YES, CONTINUE] 6. What practices and/or skills did you learn from Boresha Afya or [implementer]? a. What skills or practices have you applied in your work? Kindly explain. 7. How has Boresha Afya or [implementer] changed your unit’s procedures and protocols, e.g., referral systems? Kindly describe. 8. Has the transition to integrated care under Boresha Afya lead to any improvements in collaboration or efficiency in the facility? Kindly describe challenges and successes. 9. Has Boresha Afya affected your work satisfaction in any way? Please describe any positive or negative changes. 10. In your opinion, how has Boresha Afya affected patient care for women and youth? Please describe. [Interviewer probe if needed: positive, negative, no change] 11. Overall, how would you describe the quality of your interaction with Boresha Afya and [implementer]? For instance, have they been responsive to your needs, input and requests? a. Have there been any challenges? If so, have they been addressed? 133 Closing 12. The Boresha Afya integrated health care program is at its midterm point, what types of changes or improvements would you like to see moving forward? What gaps do you perceive as having yet to be met? What in your view are the most important priorities and opportunities? FACILITY SUPERVISOR KII GUIDE Introduction 1. Kindly describe the health services your facility offers, including CHWs’ contributions. [Interviewer note: if items on the list below are not mentioned, ask about them] • HIV prevention, treatment, care • TB, DOTS, MDR • RH/FPMNCH/ANC • Malaria • Gender services (e.g., GBV) • Youth Friendly Health Services (YFHS) • Laboratory and diagnostic unit • Data clerk’s office • Medicine and health commodity dispensing • Operation theater • In-patient department (IPD) • Out-patient department (OPD) • Community outreach services Boresha Afya Involvement 2. Please describe your facility’s involvement or activities with Boresha Afya to integrate and strengthen health services particularly for women and youth. a. How long has your facility been involved with Boresha Afya? b. Which sections or personnel are involved with Boresha Afya? What is their role? c. Are community health workers in this facility involved with Boresha Afya? What is their role? 3. Overall, how would you describe the quality of your interaction with Boresha Afya and its implementers? For instance, have they been responsive to your organization’s needs, input, and requests? a. Have there been any challenges? If so, were they addressed? b. Are Boresha Afya implementers sharing knowledge and lessons learned/best practices with your facility? If so, what? 4. Has your facility carried out information, advocacy, and/or community mobilization activities? If so, kindly describe these activities and when they took place. 134 a. Were these activities carried out in cooperation with CSOs or community representatives, including youth groups? b. Did Boresha Afya provide any support for these activities? If so, please explain the support provided. 5. Please describe any technical assistance or capacity development provided to your facility by Boresha Afya, including training, mentoring and material support for health practitioners, administrators, MEL/Data clerks, laboratory and diagnostics, IT, new procedures, commodity supplies, and health provider and customer satisfaction. a. Please describe any support you or your facility have received in managing integrated health care. 6. Please describe any support you’ve received from Boresha Afya to improve cooperation and collaboration between your facility and the community, CHWs, stakeholders, other facilities, health authorities in the district, regional, and national government. 7. How has Boresha Afya helped your facility strengthen and integrate health services, particularly for women and youth? Please give specific examples. a. Please provide examples of when this process has been successful. Which were the most important ones? b. Please provide examples when this process has faced challenges or failed. Which were the most important ones? 8. What has been the client and local community’s response to your facility’s activities to integrate health services? 9. In your opinion, what are the key components or processes necessary for the successful integration of health care in your facility and other facilities at the district, regional, and national government levels? a. To what extent do you think these components or processes are in place? b. To what extent has Boresha Afya contributed? Sustainability 10. In your opinion, what steps have been achieved towards the sustainability of Boresha Afya’s contribution to your facility and the healthcare system? [Interviewer probe if needed: policy and laws facilitating integrated health; improvements in healthcare systems and services; capacity of health service providers and support systems, e.g., MEL, supply chain; involvement of communities and marginalized groups, cooperation and collaboration; structural and social barriers, e.g., gender, stigma, youth involvement; cost-effectiveness] Closing 11. Boresha Afya is at its midterm point, what types of changes or improvements would you like to see moving forward including addressing gaps? What do you perceive as the priorities and opportunities for Boresha Afya? 135 GOT KII (NATIONAL) Introduction 1. Please describe your involvement with the Boresha Afya program. When did you first get involved? a. Who do you primarily interact with from Boresha Afya, the implementing partners or USAID? 2. Please describe the added value Boresha Afya has contributed within your area of responsibility regarding integrating and strengthening the Tanzanian health system. [Interviewer probe if needed: did it improve effectiveness? Efficiency? Cost-effectiveness?] a. What activities were most responsible for this added value? b. To what extent could these contributions been achieved without Boresha Afya? Collaboration & Coordination 3. How has Boresha Afya contributed to collaboration and cooperation on integrated health between and within geographical zones, regions and districts? a. Which organizations are cooperating? What does that cooperation look like? Please be specific. b. What are the main challenges to cooperation between and within geographical zones, regions, and districts? 4. How has Boresha Afya contributed to collaboration and cooperation on integrated health between the health service areas (e.g., HIV, TB, malaria, reproductive health, MNCH)? a. Which organizations are cooperating? What does that cooperation look like? Please be specific. b. What are the main challenges to cooperation between health service areas? 5. Overall, how would you describe the quality of your interaction with Boresha Afya and its implementers? For instance, have they been responsive to GOT’s needs, input, and requests? a. Have there been any challenges? If so, were they addressed? 6. Overall, how would you describe the quality of your interaction with USAID regarding Boresha Afya? For instance, have they been responsive, engaged and collaborative? a. Have there been any challenges? If so, were they addressed? Performance 7. In your opinion, how effective has UBA been in achieving its targets and goals with the Tanzanian government at the national, regional, and district levels? a. Which targets and goals have been met most successfully? b. Which targets or goals have been the most challenging for Boresha Afya to meet? 136 8. In your own opinion, what are the key components or processes necessary for the successful integration of health care in Tanzania? Provide examples. [Interviewer probe if needed: policies and regulations; technical support; capacity building and knowledge management; management (incl. cooperation and collaboration); strategic information] a. How well has Boresha Afya addressed these components or processes? 9. A primary goal of Boresha Afya is to provide capacity building, knowledge sharing and learning between different health service areas, particularly those addressing women and youth. Kindly provide your opinion on how well these activities have been implemented and contributed to integrating and strengthening health services in Tanzania. Give examples. a. What types of activities were supported (e.g., workshops, manuals, online activities) and who was involved? b. How effective do you think these activities were? Are there any lessons learned? c. Are Boresha Afya implementers sharing knowledge and lessons learned/best practices with the Tanzania government at the national, regional and district levels? Sustainability 10. In your opinion, what steps have been achieved towards the sustainability of Boresha Afya’s contribution in Tanzania? [Interviewer probe if needed: policy and laws facilitating integrated health; improvements in healthcare systems and services; capacity of health service providers and support systems, e.g. MEL, supply chain; involvement of communities and marginalized groups, cooperation and collaboration; structural and social barriers, e.g. gender, stigma, youth involvement; cost-effectiveness] Conclusion 11. What has your department learned from working with the Boresha Afya program? 12. Boresha Afya is at its midterm point, what types of changes or improvements would you like to see moving forward, including addressing gaps? What do you perceive as priorities and opportunities for Boresha Afya? WOMEN LIVING WITH HIV FGD GUIDE 1. Kindly describe your first contacts with the facility and the services you were provided. a. Did the facility meet your expectations? Explain. 2. At the time, did you experience barriers that affected your readiness to return and continue using services? [Moderator probe if needed: fees for health services, long waiting times, unfriendly health workers, unknowledgeable health workers, distance/transportation cost to facility, stigma/discrimination, medications not available] a. Have you returned to the facility since your initial visit? 3. Being an HIV positive woman can affect her entire family. Did you, your partner, or your family members receive any services from the facility after you were diagnosed? This includes support in dealing with community and men’s attitudes. 137 a. If so, what type of services? [Moderator probe if needed: counseling, for individuals, family, or couple; testing and treatment for children; family planning or contraception; preparing for a new baby; dealing with stigma/discrimination; support in receiving and taking medication; HIV and treatment literacy] b. Did these services meet your expectations? c. How could those services be improved? Are there services you were not offered that you’d like to receive? 4. Community health workers often provide similar outreach services and medications as health facilities. Have you interacted with community health workers in this area? If so, how would you describe these outreach services in terms of their quality and friendliness? Kindly describe. As you may know, the local facility and Tanzania’s health services have been working to improve services through the Boresha Afya program since [insert date for specific facility]. I now have a few questions for you about how things may or may not have changed since then. 5. Have you noted or heard of any differences in the quality of health services in this facility since that time? Please describe. 6. Have you noted any changes in the way you are referred to additional services, screening or treatment by health workers? For example, fewer steps, less waiting, faster feedback, and friendlier services. 7. Earlier we spoke about barriers women in this area face in accessing health services. Have you noted any changes in those barriers? a. If so, what do you think is responsible for the change? b. What barriers continue to exist for women in accessing health services? How could they be addressed? As you may know, the Boresha Afya program is also carrying out community activities to empower women and reduce stigma. 8. Have you noticed or had any contact with these activities? If so, do you think they are helping community members to change their attitudes and behaviors? Kindly explain. 9. Lastly, what changes do you think the facility could make in order to provide women like yourself with improved, more integrated health services? IMPLEMENTING PARTNER KII GUIDE Introduction 1. Please describe your involvement with Boresha Afya. What is your specific role or area of responsibility? How long have you been in this role? Program Implementation 2. In what ways has Boresha Afya helped to strengthen the Tanzanian health system to integrate health services at the national, regional and district administrative levels? 138 3. In what specific ways have Boresha Afya activities strengthened the integrated health capacity of individual health facilities, and communities (including CSOs, CHWs, women, and youth)? 4. How have Boresha Afya activities impacted or affected collaboration and cooperation between health service providers, health administration offices, community actors, and customers to integrate health services? 5. How, if at all, have Boresha Afya interventions been adapted to changes in response to local needs/contexts, strategic changes (from USAID or otherwise), or lessons learned on program performance? Share any specific examples. 6. What are the key gender and youth related barriers to service delivery in Tanzanian facilities? How is Boresha Afya addressing these barriers? 7. What have been the successes and challenges working with other non-Boresha Afya integrated health stakeholders and implementers at the local and national levels? 8. How has the Developmental Evaluation contributed to the effectiveness, learning and adaptability of Boresha Afya support? Boresha Afya Management and Structure 9. What lessons have been learned on Boresha Afya’s organizational capacity (structure, management, and strategic capacity) that influence its effectiveness to deliver program results? Kindly describe the challenges and lessons learned, and how Boresha Afya has taken action. 10. How effectively has Boresha Afya coordinated and collaborated with its sub￾partners? a. How has Boresha Afya’s relationship with the sub-partners evolved over the program period to improve integrated health? b. To what extent has the existing communications structure affected management issues? 11. How can Boresha Afya’s design, management, and implementation become more efficient, and effective in achieving program goals within the remaining time period of implementation? Information Sharing and Communication 12. What new information is being generated by Boresha Afya M&E units or the DE and how is it used and shared with sub-partners, other IPs, and national and local stakeholders? 13. What mechanisms or tools are used to promote collaboration, knowledge sharing, and harmonization among Boresha Afya IPs and consortia? Sustainability 14. In your opinion, what steps have been achieved towards the sustainability of Boresha Afya’s contribution, including the current U.S. policy focus on recipient self-reliance? [Interviewer probe if needed: policy and laws facilitating integrated health; improvements in healthcare systems and services; capacity of health service providers and support systems, e.g., MEL, supply chain; involvement of communities and marginalized groups, cooperation and collaboration; structural and social barriers, e.g., gender, stigma, youth involvement; cost-effectiveness] 139 15. Kindly explain how Boresha Afya is successfully transitioning traditional Tanzanian health care to integrated health. What has been learned? Investment 16. How are multiple funding sources (PEPFAR, PMI, GHSP, GFATM, etc.) contributing to the expanded delivery of integrated health services in Boresha Afya’s target regions, councils and facilities across the country? a. What is the scale of coverage? b. What coverage gaps still remain? 17. Are there any preliminary findings on the cost-effectiveness of integrated health in comparison to non-integrated care? Kindly explain. FGD GUIDE for Male Involvement Introduction 1. Kindly describe attitudes in this community towards men being involved in the reproductive, HIV, or other health issues of their wives (and women in general). 2. Other than health facilities, have you received any education in school or from community members about women’s reproductive health, pregnancy or other health conditions of women when you were growing up? Please describe, including when that education took place. Enabling and Behavior Change 3. Have you more recently gained more information or awareness about women’s reproductive health and HIV that led you to feel more positive about your own involvement in their care? What have you learned? a. Where have you received this information and awareness raising? [Moderator probe if needed: health facility, community awareness campaigns, media, traditional or government leaders, peers, etc.] 4. How has this new information and knowledge affected your attitudes and behaviors? What do you do differently now, e.g., with your wife? a. Have you shared any of this new information and knowledge with your friends and community? How do they react? Health Services & Boresha Afya For the past two years, the local health facility and the Tanzanian health authorities have been encouraging male involvement in reproductive health, HIV and women’s health in general. This is being done through the Boresha Afya program, implemented here by [implementer]. 5. Are you familiar with the Boresha Afya program? Please describe any involvement you have had with it. I now have a few questions about how things may or may not have changed in the facility and community over the past two years. 140 6. What changes have you noticed, if any, in facility health worker attitudes and toward male involvement over the past two years? Kindly describe. a. What do you think is most responsible for those changes? b. What challenges still exist in changing attitudes toward male involvement? 7. What changes have you noticed, if any, in community health worker attitudes toward male involvement? Kindly describe. a. What do you think is most responsible for those changes? b. What challenges still exist in changing attitudes toward male involvement? 8. What changes have you noticed, if any, in the variety of male involvement services offered by the facility? Kindly describe. [Moderator probe if needed: individual or family counseling, couples counseling, family planning advice and support (contraception), reproductive health and HIV prevention and treatment literacy] a. Which services are most needed that are not currently available? 9. Have you noted or heard of any differences in the quality of health services in the facility? Please describe. As you may know, the Boresha Afya program is also carrying out community activities to empower women and reduce stigma. 10. Have you noticed or had any contact with these activities? If so, do you think they are helping community members to change their attitudes and behaviors? Kindly explain. Closing 11. Lastly, what changes do you think the facility could make in order to provide women with improved, more integrated health services? KII GUIDE FOR NATIONAL STAKEHOLDERS Introduction 1. Kindly describe your organization’s mandate and involvement with Boresha Afya. Please provide approximate dates and mention any other organizations involved (e.g., USAID, MOHCDGEC, etc.). Impact 2. How have your organization’s activities supported Boresha Afya’s efforts to strengthen integrated health care at the national level in Tanzania? a. In your opinion, how well have these activities been coordinated at the national level? b. What are the remaining challenges to integration at the national level? 3. In your opinion, how have your organization’s activities supported Boresha Afya to strengthen integrated health care at the regional and district levels? 141 a. Kindly describe specific examples of Boresha Afya activities your organization has supported to strengthen local communities for integrated health care. b. What are the remaining challenges to integration at the regional and district levels? 4. How have Boresha Afya activities supported by your organization affected or improved collaboration and cooperation between the three health service tiers of government? With communities? 5. What, if any, new and key information has been generated by Boresha Afya based on the program’s data? a. To what extent is this information shared with national and local stakeholders? b. What is your assessment of this data? 6. Kindly describe how Boresha Afya activities supported by your organization have improved access to integrated health care for women and youth, specifically. a. What barriers to health services for women and youth has Boresha Afya been working to address? How successful were those efforts? Structure 7. Is Boresha Afya’s organizational capacity sufficient in delivering integrated health care results? a. What could be improved? 8. Are Boresha Afya’s human resources, strategic interests, MEL systems, communications and overall structure contributing to the organizations capacity to deliver desired integrated health care results? a. What could be improved? 9. How could Boresha Afya’s design, management and implementation become more efficient, effective and relevant to achieving integrated health care goals in Tanzania? RMO/RHMT & DMO/CHMT KII GUIDE Introduction 1. Please describe your involvement with the Boresha Afya program. When did you first get involved? 2. Has Boresha Afya contributed to integrating and strengthening your Region’s/Council’s health care system? Has it added value in terms of performance, efficiency, or cost effectiveness? a. Which activities have been most effective? Please be specific. b. Which activities have been least effective? Please be specific. Collaboration and Cooperation 3. How has Boresha Afya contributed to collaboration and cooperation on integrated health between geographical areas, including regions and districts? 142 a. Which organizations are cooperating? What does that cooperation look like? Please be specific. b. What are the main challenges to cooperation between and within regions and districts? 4. How has Boresha Afya contributed to collaboration and cooperation on integrated health between the health service areas (e.g., HIV, TB, malaria, reproductive health, MNCH)? a. Which organizations are cooperating? What does that cooperation look like? Please be specific. b. What are the main challenges to cooperation between the health service areas? 5. Overall, how would you describe the quality of your interaction with Boresha Afya and its implementers? For instance, have they been responsive, engaged, and collaborative? a. Have there been any challenges? If so, were they addressed? 6. Overall, how would you describe the quality of your interaction with USAID regarding Boresha Afya (if applicable)? For instance, have they been responsive, engaged, and collaborative? a. Have there been any challenges? If so, were they addressed? Performance 7. In your own opinion, how effective has Boresha Afya been in achieving objectives for the integrated health care system at the national, regional, and district levels? a. Where has it been most successful? Please be specific. b. What has been most challenging for Boresha Afya to achieve? Please be specific. 8. In your own opinion, what are the key components or processes necessary for the successful integration of health care in your region/district? Provide examples. [Interviewer probe if needed: policies and regulations; technical support; capacity building and knowledge management; management (incl. cooperation and collaboration); strategic information] a. How well has Boresha Afya addressed these components or processes? 9. A primary goal of UBA is to provide capacity building, knowledge sharing and learning between different health service areas, particularly those addressing women and youth. Kindly provide your opinion on how well these activities have been implemented and contributed to integrating and strengthening health services in your region/district. Give examples. a. What types of activities were supported (e.g., workshops, manuals, online activities) and who was involved? b. How effective do you think these activities were? Are there any lessons learned? c. Are Boresha Afya implementers sharing knowledge and lessons learned/best practices with your region/council? 143 Sustainability 10. In your opinion, what steps have been achieved towards the sustainability of Boresha Afya’s contribution to your region/district and the health care system? [Interviewer probe if needed: practices in integrated health care institutionalized in facilities; national health guidance disseminated and implemented by local government authorities; policy and laws facilitating integrated health; improvements in healthcare systems and services; capacity of health service providers and support systems (e.g., MEL, supply chain, equipment, etc.); involvement of communities and marginalized groups; cooperation and collaboration; ongoing financial support and funds; structural and social barriers to sustainability (e.g., gender, stigma, youth involvement, etc.)] Conclusion 11. What has your regional/council government staff learned from working with the Boresha Afya program? 12. Boresha Afya is at its midterm in its program cycle; what types of changes or improvements would you like to see moving forward, including addressing gaps? What do you perceive as priorities and opportunities for Boresha Afya? USAID KII GUIDE Introduction 1. Please describe your involvement or activities with Boresha Afya. What is your specific role or area of responsibility? How long have you been in this role? 2. Please describe the added value Boresha Afya has contributed within your area of responsibility regarding integrating and strengthening the Tanzanian health system. [Interviewer probe if needed: did it improve effectiveness? Efficiency? Cost-effectiveness?] a. What activities were most responsible for this added value? Collaboration & Cooperation 3. How has Boresha Afya contributed to collaboration and cooperation on integrated health between and within geographical zones, regions, and districts? a. Which organizations are cooperating? What does that cooperation look like? Please be specific. b. What has been USAID’s specific involvement? c. How has the government been involved in this collaboration and cooperation? How receptive are they? d. What are the main challenges to cooperation between and within geographical zones, regions, and districts? 4. How has Boresha Afya contributed to collaboration and cooperation on integrated health between the health service areas (e.g., HIV, TB, malaria, reproductive health, MNCH)? a. Which organizations are cooperating? What does that cooperation look like? Please be specific. 144 b. What has been USAID’s specific involvement? c. How has the government been involved in this collaboration and cooperation? How receptive are they? d. What are the main challenges to cooperation between health service areas? 5. Overall, how would you describe the quality of your interaction with Boresha Afya and its implementers? For instance, have they been responsive to USAID’s needs, input, and requests? a. Have there been any challenges? Were they addressed? 6. Overall, how would you describe the quality of your interaction with GOT at its various levels regarding Boresha Afya? For instance, have they been responsive, engaged and collaborative with USAID? a. Have there been any challenges? Were they addressed? Performance 7. How effective has Boresha Afya been in achieving its targets and goals? a. Which targets and goals have been met most successfully? b. Which targets or goals have been the most challenging for Boresha Afya to meet? 8. How have annual changes in targets affected effective project implementation, if at all? a. How well have implementers adapted to these changes? Please give examples. 9. In your own opinion, what are the key components or processes necessary for the successful integration of health care in Tanzania? Provide examples. [Interviewer probe if needed: policies and regulations; technical support; capacity building and knowledge management; management (cooperation and collaboration); strategic information] a. How well has Boresha Afya addressed these components or processes? 10. A primary goal of Boresha Afya is to provide capacity building, knowledge sharing, and learning between different health service areas, particularly those addressing women and youth. Kindly provide your opinion on how well these activities have been implemented and contributed to integrating and strengthening health services in Tanzania. Give examples. a. What types of activities were supported (e.g., workshops, manuals, online activities) and who was involved? b. How effective do you think these activities were? Are there any lessons learned? Sustainability 11. In your opinion, what steps have been achieved towards the sustainability of Boresha Afya’s contribution, including the current U.S. policy focus on recipient self-reliance? [Interviewer probe if needed: policy and laws facilitating integrated health; improvements in healthcare systems and services; capacity of health service providers and support systems, e.g., MEL, supply chain; 145 involvement of communities and marginalized groups, cooperation and collaboration; structural and social barriers, e.g., gender, stigma, youth involvement; cost-effectiveness] 12. What is your opinion on the integrated funding mechanism for the three Boresha Afya activity zones? Has this model contributed to cost effectiveness? Please explain. Conclusion 13. What have you learned from working on the Boresha Afya program? 14. Boresha Afya is at its midterm point, what types of changes or improvements would you live to see moving forward? FGD GUIDE for Youth Groups Introduction 1. Kindly provide a description of your group, who sponsors you and the activities you’re involved with (e.g., YFHS, advocacy, mobilization, information campaigns, etc.). 2. Who do you meet at the local facility or in the community as you conduct these activities? 3. What are the most common health issues you find yourself working with and how do you deal with people’s sensitive questions? 4. Kindly say a few works about how well your program is functioning among its youth members. Are people satisfied, motivated, and working together as a team? Why or why not? a. Have there been dropouts? What do you think is the reason for dropouts? 5. What are the main barriers to health services for youth in this area? a. Are you facing any challenges or issues from your parents or peers in the community because of your work? Involvement with Boresha Afya As you may know, the local facility and Tanzania’s health services have been working to improve services through the Boresha Afya program over the past two years. I now have a few questions for you about your involvement with the Boresha Afya program, implemented in this area by [implementer]. 6. Overall, how would you describe the quality of your group’s interaction with [implementer] and its management? For instance, have they been responsive to your needs, ideas, and requests? 7. What type of training have each of you and other members received from [implementer] or elsewhere? What did you learn, and was useful for your work? a. Did you receive any training and mentoring on gender and gender issues? If so, please describe. What were some of the key lessons learned? 8. What type of ongoing direction or mentoring do you receive from [implementer], if any? Please describe. 146 a. What is your assessment of the quality of that support? [Moderator probe if needed: Do they treat you respectfully, share information and listen to your input? Is there someone you can turn to if you need information to help or refer others?] b. What could be improved? What other support do you need? 9. What type of support and guidance you receive from facility health worker staff, if any? a. What is your assessment of the quality of that support? [Moderator probe if needed: Do they treat you respectfully, share information and listen to your input? Is there someone you can turn to if you need information to help or refer others?] b. What could be improved? What other support do you need? 10. Earlier we spoke about barriers youth in this area face in accessing health services. Have you noted any changes in those barriers since Boresha Afya began two years ago? a. What do you think is responsible for the change? b. What barriers continue to exist for youth in accessing health services? How could they be addressed? Conclusion 11. As you may know, the Boresha Afya integrated health care program is at its midterm point. What types of changes or improvements would you like to see from [implementer] moving forward? 147 ANNEX 6: DETAILED DATA ANALYSIS TABLES Table 1: PEPFAR Indicator Summary- Targets Versus Actuals Across Years FY 2017- FY 2019 IP Indicator FY 2017 Results FY 2017 Targets FY 2018 Results FY 2018 Targets FY 2019 Results FY 2019 Targets 18060 – Boresha HTS_TST 2,187,667 2,116,474 3,152,130 2,407,591 760,800 2,596,542 Afya Northern PMTCT_ART 6,559 7,126 7,897 6,440 6,070 7,277 Zone TX_CURR 921,807 258,443 302,122 307,073 146,310 333,788 18237 – Boresha HTS_TST 1,371,325 1,262,311 2,826,725 1,629,256 507,141 1,667,208 Afya Southern PMTCT_ART 8,800 8,527 9,504 7,814 6,723 8,700 Zone TX_CURR 1,185,646 312,169 364,312 365,236 180,980 420,826 Table 2: TX TB-Screening – All Zones FY 2017-FY 2019 (Targets not available for the percent, only numerator – see full data report for TB and TB_Stat) IP FY 2017 % FY 2018 % FY 2019 % 18060 – Boresha Afya Northern Zone 0.8% 0.8% 0.9% 18237 – Boresha Afya Southern Zone 0.3% 0.5% 0.4% Table 3: TX TB for Councils Visited (in Sample) FY 2017-FY 2018 IP Council FY 2017 FY 2018 FY 2019 TX_TB TX_TB TX_TB Results Results Results % % % 18060 – Boresha Afya Northern Zone Dodoma MC 1.0% 0.9% 1.1% Hai DC 0.5% 0.9% 1.2% 18237 – Boresha Afya Southern Zone Iringa MC 0.1% 0.4% 0.2% Mvomero DC 0.6% 0.7% NA 148 Table 4: Performance Comparison of Selected Indicators in Selected Regions and Councils in the Same Quarter of April-June in 2016 and 2019 Performance Indicators Dodoma Region Dodoma Council Kilimanjaro Region Kilimanjaro Council (Hai) Base MTE % Increase or Decrease Base MTE % Increase or Decrease Base MTE % Increase or Decrease Base MTE % Increase or Decrease CYP 76,764.70 124,729.50 62.48 13,662.00 28,925.70 111.72 88,478.60 74,525.60 -15.77 8,981.60 6,286.60 -30.01 PMTCT+ 454.00 571.00 25.77 191.00 300.00 57.07 282.00 285.00 1.06 35.00 42.00 20.00 PMTCT_ART 353.00 516.00 46.18 37.00 41.00 10.81 293.00 289.00 -1.37 37.00 41.00 10.81 HIV counseled and tested 37,864.00 75,176.00 98.54 11,842.00 13,754.00 16.15 39,855.00 82,696.00 107.49 2,034.00 7,677.00 277.43 HIV Care 18,630.00 28,758.00 54.36 9,273.00 14,152.00 52.62 22,278.00 28,868.00 29.58 2,195.00 3,001.00 36.72 ART 17,919.00 28,189.00 57.31 8,914.00 13,872.00 55.62 31,093.00 28,374.00 -8.74 2,127.00 2,857.00 34.32 Persons started on TB Rx 404.00 292.00 -27.72 116.00 150.00 29.31 931.00 201.00 -78.41 89.00 51.00 -42.70 149 Table 5: Non-PEPFAR Indicators – IPRS Across Years FY 2017-FY 2019 – Actuals only Indicator IP FY 2017 Actual FY 2018 Actual FY 2019 Actual NUMBER OF PEOPLE TRAINED IN FAMILY PLANNING/REPRODUCTIVE HEALTH WITH USG FUNDS USAID BORESHA AFYA-DELOITTE 102 152,080 65 USAID BORESHA AFYA-LAKE/WESTERN ZONE-JHPIEGO 71 633 105 USAID BORESHA AFYA NORTH/ CENTRAL REGION-EGPAF 107 495 309 Total . 280 153,208 479 NUMBER OF CLIENTS ACCEPTING A MODERN FAMILY PLANNING METHOD USAID BORESHA AFYA-DELOITTE - 622,373 570,429 USAID BORESHA AFYA-LAKE/WESTERN ZONE-JHPIEGO - 1,220,378 946,444 USAID BORESHA AFYA NORTH/ CENTRAL REGION-EGPAF - 570,270 647,179 Total . . 2,413,021 2,164,052 COUPLE YEARS PROTECTION IN USG SUPPORTED PROGRAMS BY METHOD USAID BORESHA AFYA-DELOITTE 32,700 762,499 754084 USAID BORESHA AFYA-LAKE/WESTERN ZONE-JHPIEGO 150,824 4,016,269 1,548,370 USAID BORESHA AFYA NORTH/ CENTRAL REGION-EGPAF - 1,969,915 1,444,213 Total . 183,524 6,748,683 3,746,667 NUMBER OF YOUTH WHO HAVE RECEIVED FAMILY PLANNING/REPRODUCTIVE HEALTH SERVICES USAID BORESHA AFYA-DELOITTE - 352,523 250,687 USAID BORESHA AFYA-LAKE/ WESTERN ZONE-JHPIEGO 14,951 323,805 403,604 USAID BORESHA AFYA NORTH/ CENTRAL REGION-EGPAF 67,999 - - Total . 82,950 676,328 654,291 NUMBER OF USG-ASSISTED FACILITIES THAT OFFER FP SERVICES IMMEDIATELY AID-621-A-16-00002- USAID BORESHA AFYA-DELOITTE - 100 - AID-621-A-16-00003-USAID BORESHA AFYA-LAKE/WESTERN ZONE-JHPIEGO 95 84 200 AID-621-A-16-00004-USAID BORESHA AFYA NORTH/CENTRAL REGION￾EGPAF 100 100 - Total . 195 284 200 150 Indicator IP FY 2017 Actual FY 2018 Actual FY 2019 Actual PERCENT OF WOMEN RECEIVING MODERN METHOD OF FP IMMEDIATELY USAID BORESHA AFYA-DELOITTE - 90 - USAID BORESHA AFYA-LAKE/ WESTERN ZONE-JHPIEGO 10 30 35 Total . 10 120 35 HL.7.1-1 (3.1.7.1-1)-COUPLE YEARS PROTECTION IN USG SUPPORTED PROGRAMS USAID BORESHA AFYA-DELOITTE - 783,385 1,284,522 USAID BORESHA AFYA-LAKE/ WESTERN ZONE-JHPIEGO 264,357 1,657,924 1,788,134 USAID BORESHA AFYA NORTH/ CENTRAL REGION-EGPAF 336,659 - - Total . 601,016 2,441,309 3,072,656 HL.7.2-2-NUMBER OF USG-ASSISTED COMMUNITY HEALTH WORKERS (CHWS) PROVIDING FAMILY PLANNING (FP) INFORMATION, REFERRALS, AND/OR SERVICES DURING THE YEAR AID-621-A-16-00002- USAID BORESHA AFYA-DELOITTE - 1,348 2,524 AID-621-A-16-00003-USAID BORESHA AFYA-LAKE/WESTERN ZONE-JHPIEGO 748 1,308 1,308 AID-621-A-16-00004-USAID BORESHA AFYA NORTH/ CENTRAL REGION￾EGPAF - 168 208 Total . 748 2,824 4,040 151 Table 6: Targets and Actuals for IPRS Indicator IP FY 2017 Actual FY 2018 Target FY 2018 Actual FY 2019 Target FY 2019 Actual NUMBER OF PEOPLE TRAINED IN FAMILY PLANNING/REPRODUCTIVE HEALTH WITH USG FUNDS USAID BORESHA AFYA￾DELOITTE 102 328 152,080 - 65 USAID BORESHA AFYA￾LAKE/WESTERN ZONE￾JHPIEGO 71 740 633 325 105 USAID BORESHA AFYA (NORTH/CENTRAL REGION￾EGPAF) 107 - 495 - 309 Total . 280 1,068 153,208 325 479 NUMBER OF CLIENTS ACCEPTING A MODERN FAMILY PLANNING METHOD USAID BORESHA AFYA￾DELOITTE - 783,385 622,373 1,002,747 570,429 USAID BORESHA AFYA￾LAKE/WESTERN ZONE￾JHPIEGO - 1,079,351 1,220,378 1,386,003 946,444 USAID BORESHA AFYA (NORTH/CENTRAL REGION￾EGPAF) - - 570,270 - 647,179 Total . . 1,862,736 2,413,021 2,388,750 2,164,052 COUPLE YEARS PROTECTION IN USG SUPPORTED PROGRAMS BY METHOD USAID BORESHA AFYA￾DELOITTE 32,700 - 762,499 1,284,522 754,084 USAID BORESHA AFYA￾LAKE/WESTERN ZONE￾JHPIEGO 150824 - 4,016,269 - 1,548,370 USAID BORESHA AFYA (NORTH/CENTRAL REGION￾EGPAF) - - 1,969,915 6,875,492 1,444,213 152 Indicator IP FY 2017 Actual FY 2018 Target FY 2018 Actual FY 2019 Target FY 2019 Actual Total . 183,524 - 6,748,683 8,160,014 3,746,667 NUMBER OF YOUTH WHO HAVE RECEIVED FAMILY PLANNING/REPRODUCTIVE HEALTH SERVICES USAID BORESHA AFYA￾DELOITTE - 23,905 352,523 181,322 250,687 USAID BORESHA AFYA￾LAKE/WESTERN ZONE￾JHPIEGO 14,951 26,637 323,805 411,782 403,604 USAID BORESHA AFYA￾NORTH/CENTRAL REGION￾EGPAF 67,999 22,336 - 228,822 - Total . 82,950 72,878 676,328 821,926 654,291 PERCENT OF CLIENTS ACCEPTING A LONG￾ACTING AND PERMANENT METHOD USAID BORESHA AFYA￾DELOITTE - 28 - 20 - USAID BORESHA AFYA￾LAKE/WESTERN ZONE￾JHPIEGO - 45 - 22 40 USAID BORESHA AFYA - NORTH /CENTRAL REGION￾EGPAF - 46 - 31 - Total . . 119 . 73 40 NUMBER OF USG-ASSISTED FACILITIES THAT OFFER FP SERVICES IMMEDIATELY AID-621-A-16-00002- USAID BORESHA AFYA-DELOITTE - - 100 298 - AID-621-A-16-00003-USAID BORESHA AFYA￾LAKE/WESTERN ZONE￾JHPIEGO 95 85 84 186 200 AID-621-A-16-00004-USAID BORESHA AFYA -NORTH/ CENTRAL REGION-EGPAF 100 61 100 175 - 153 Indicator IP FY 2017 Actual FY 2018 Target FY 2018 Actual FY 2019 Target FY 2019 Actual Total 195 146 284 659 200 PERCENT OF WOMEN RECEIVING MODERN METHOD OF FP IMMEDIATELY USAID BORESHA AFYA￾DELOITTE 90 USAID BORESHA AFYA￾LAKE/WESTERN ZONE￾JHPIEGO 10 9 30 8 35 USAID BORESHA AFYA - NORTH/CENTRAL REGION￾EGPAF 4 Total 10 9 120 12 35 HL.7.1-1 (3.1.7.1-1)-COUPLE YEARS PROTECTION IN USG SUPPORTED PROGRAMS USAID BORESHA AFYA￾DELOITTE 68,680 783,385 216,776 1,284,522 USAID BORESHA AFYA￾LAKE/WESTERN ZONE￾JHPIEGO 264,357 260,273 1,657,924 2,231,298 1,788,134 USAID BORESHA AFYA￾NORTH/CENTRAL REGION￾EGPAF 336,659 165,114 1,969,496 Total 601,016 494,067 2,441,309 4,417,570 3,072,656 PERCENT OF USG-ASSISTED SERVICE DELIVERY SITES PROVIDING FAMILY PLANNING COUNSELING AND/OR SERVICES AID-621-A-16-00002- USAID BORESHA AFYA-DELOITTE 54 100 AID-621-A-16-00003-USAID BORESHA AFYA-LAKE/ WESTERN ZONE-JHPIEGO 89 42 88 AID-621-A-16-00004-USAID BORESHA AFYA-NORTH/ CENTRAL REGION-EGPAF 26 154 Indicator IP FY 2017 Actual FY 2018 Target FY 2018 Actual FY 2019 Target FY 2019 Actual Total 169 142 88 AID-621-A-16-00002- USAID BORESHA AFYA-DELOITTE 40 1,348 2,524 HL.7.2-2-NUMBER OF USG-ASSISTED COMMUNITY HEALTH WORKERS (CHWS) PROVIDING FAMILY PLANNING (FP) INFORMATION, REFERRALS, AID-621-A-16-00003-USAID BORESHA AFYA-LAKE/ WESTERN ZONE-JHPIEGO 748 696 1,308 698 1,308 AND/OR SERVICES DURING THE YEAR AID-621-A-16-00004-USAID BORESHA AFYA-NORTH/ CENTRAL REGION-EGPAF 168 336 208 Total 748 736 2,824 1,034 4,040 155 - - - - Table 7: UBA Performance by selected indicators and councils in comparison to national indicators Performance Indicators Comparison period Mvomero DC Iringa DC Hai DC Dodoma MC Ngara Tarime National Reference Malaria prevalence rate 2016-2019 Not in the scope Not in the scope Not in the scope Not in the scope 6 pp 7 pp 4 pp Modern contraceptive prevalence rate 2016-2019 Unavailable Unavailable Unavailable Unavailable 13 pp 5 pp - 7 pp % of women who receive IPT for malaria during their last pregnancy 2016-2019 Not in the scope Not in the scope Not in the scope Not in the scope 19 pp 66 pp 26 pp % of children who slept under an ITN the night before the survey 2016-2019 Not in the scope Not in the scope Not in the scope Not in the scope 90 pp 71 pp 94 pp Couple years of protection in USG supported programs 2017-2019 1220% Unavailable Unavailable Unavailable 47% 23% 20% % of live births attended by skilled health personnel 2016-2019 Not in the scope Not in the scope Not in the scope Not in the scope 29 pp 23 pp 18 pp % of HIV+ pregnant women who received ART to reduce the risk of mother to child transmission 2017-2019 1 pp 1 pp 3 pp 7 pp Not in the scope Not in the scope - 4 pp Number of HIV positive adults and children receiving ART. 2017-2019 29% 35% 22% 38% Not in the scope Not in the scope 27% Number of facilities providing BEmONC services per population 2016-2019 Not in the scope 0% Not in the scope Not in the scope 2% 7% 9% 156 - - Performance Indicators Comparison period Mvomero DC Iringa DC Hai DC Dodoma MC Ngara Tarime National Reference Number of individuals counseled, tested and received results 2017-2019 56% Unavailable 145% 24% Not in the scope Not in the scope 25% Number of HIV positive adults and children receiving a minimum of one clinical care. 2017-2019 Unavailable Unavailable 22% 38% Not in the scope Not in the scope 28% Number of HIV infected clients attending HIV care and treatment that are receiving treatment for TB disease. 2017-2019 468% 52% Unavailable 61% Not in the scope Not in the scope -47% % of HIV+ pregnant women who received ART for their own health 2017-2019 Unavailable Unavailable 3 pp -7 pp Not in the scope Not in the scope Unavailable Number of facilities providing IMCI services per population (no. of HFs providing IMCI - project data) 2016-2019 Not in the scope Not in the scope Not in the scope Not in the scope 2% 19% Unavailable 157 - - - Table 8: IP Cumulative Achievements to Date in Malaria, TB, ANC and FP/RH from IPRS Achievements to Date DELOITTE SOUTHERN ZONE JHPIEGO LAKE & WESTERN ZONE EGPAF NORTH & CENTRAL ZONE Indicator title TARGET ACTUAL TARGET ACTUAL TARGET ACTUAL 1. Number of health workers trained in intermittent preventive treatment in pregnancy (IPTP) with USG funds 300 168 1,822 8 NA NA 2. Number and proportion of suspected malaria cases tested for malaria by either MRDT or microscopy 69% 100% Unavailable Unavailable NA NA 3. Number and proportion of confirmed malaria cases (i.e., test positive by either MRDT or microscopy) among the total number of suspected malaria cases 69% 38% 30% 36% NA NA 4. Number and proportion of confirmed malaria cases NA 30% 40% 44% NA NA 5. Number and proportion of confirmed malaria cases >=5 years among total number of confirmed malaria cases NA 38% 60% 56% NA NA 6. Number and proportion of ANC clients tested for malaria by MRDT among total number of clients at first ANC visit 98% 97% 80% 99% NA NA 7. Number and proportion of ANC clients receiving iptp2 among total number of ANC clients 88% 91% 80% 78% NA NA 158 - - - Achievements to Date DELOITTE SOUTHERN ZONE JHPIEGO LAKE & WESTERN ZONE EGPAF NORTH & CENTRAL ZONE Indicator title TARGET ACTUAL TARGET ACTUAL TARGET ACTUAL 8. Number and proportion of ANC clients receiving IPTP3 among total number of ANC clients 67% 80% 60% 60% NA NA 9. Number and proportion of patients who received antimalarial treatment among those who tested positive by either MRDT or microscopy 29,523 1,028,862 1,454,115 2,222,758 NA NA 10. Percentage of confirmed malaria cases 35% 38% 30% 36% NA NA 11. Number and proportion of ANC clients who receive an insecticide treated net on their first ANC visit Unavailable 89% 90% 90% NA NA 12. Number and proportion of health facilities that did not have any stock out of MRDTS, SP and acts in the prior three months 100% 78% 90% 92% NA NA 13. Number and proportion of ANC clients with a positive malaria test by MRDT among those tested at the first ANC visit 12% 43% 8% 10% NA NA 14. Number and proportion of children who receive and ITN at measles vaccination clinic among that total number attending their first visit NA 52% 90% 83% NA NA 15. Number of pregnant women who tested positive for malaria at first ANC visit 0 10,768 64,230 62,080 NA NA 16. Number of health care workers trained in malaria diagnosis (MRDTS or microscopy) 0 10 0 6 NA NA 159 - - - Achievements to Date DELOITTE SOUTHERN ZONE JHPIEGO LAKE & WESTERN ZONE EGPAF NORTH & CENTRAL ZONE Indicator title TARGET ACTUAL TARGET ACTUAL TARGET ACTUAL 17. Number and proportion of suspected malaria cases (confirmed and clinical malaria cases) among total number of OPD visits Unavailable Unavailable 51% 67% NA NA 18. Number and percentage of ANC clients who received 30+ tabs Fe/Fol NA NA 90% 84% NA NA 19. Number of women giving birth who received uterotonic in the third stage of labor (or immediately after birth) through USG-supported programs 35,504 20,642 275,102 200,228 NA NA 20. Number of newborns not breathing at birth who were resuscitated in USG￾supported programs 3682 1700 19,650 10601 NA NA 21. Percent of current PLHIV clients who received family planning services from FPINT_SITE service delivery points Unavailable 9% Unavailable 42% Unavailable 42% 22. Percentage of PEPFAR-supported HIV service delivery points (SDPS) that offer at least three types of modern family planning (FP) methods Unavailable 65% Unavailable 1% Unavailable 1% 23. Percent of facilities providing immediate postpartum family planning services in maternal and child health settings Unavailable 11% Unavailable 0% Unavailable 0% 24. Percent of women provided with immediate postpartum family planning services Unavailable 17% Unavailable 6% Unavailable 6% 160 Name Gary R. Svenson Title Independent Consultant Organization NORC at the University of Chicago (Data for Development) Evaluation Position Team Leader Evaluation Award Number (contract or another instrument) AID-OAA-I-15-00024/AID-621-TO-17-00005 USAID Project(s) Evaluated (Include project name(s), implementer name(s) and award number(s), if applicable) USAID Boresha Afya I have real or potential conflicts of interest to disclose. No If yes answered above, I disclose the following facts: Real or potential conflicts of interest may include, but are not limited to: 1. Close family member who is an employee of USAID operating unit managing the project(s) being evaluated or the implementing organization(s) whose project(s) are being evaluated. 2. Financial interest that is direct, or is significant though indirect, in the implementing organization(s) whose projects are being evaluated or in the outcome of the evaluation. 3. Current or previous direct or significant though indirect experience with the project(s) being evaluated, including involvement in the project design or previous iterations of the project. 4. Current or previous work experience or seeking employment with the USAID operating unit managing the evaluation or the implementing organization(s) whose project(s) are being evaluated. 5. Current or previous work experience with an organization that may be seen as an industry competitor with the implementing organization(s) whose project(s) are being evaluated. 6. Preconceived ideas toward individuals, groups, organizations, or objectives of the particular projects and organizations being evaluated that could bias the evaluation. I certify (1) that I have completed this disclosure form fully and to the best of my ability and (2) that I will update this disclosure form promptly if relevant circumstances change. If I gain access to proprietary information of other companies, then I agree to protect their information from unauthorized use or disclosure for as long as it remains proprietary and refrain from using the information for any purpose other than that for which it was furnished. Signature Date August 09, 2019 ANNEX 7: DISCLOSURE OF ANY CONFLICTS OF INTEREST 161 Name Dr. Elvira Beracochea Title Health Specialist Organization ME&A (Data for Development) Evaluation Position Team member Evaluation Award Number (contract or another instrument) AID-OAA-I-15-00024/AID-621-TO-17-00005 USAID Project(s) Evaluated (Include project name(s), implementer name(s) and award number(s), if applicable) USAID Boresha Afya AID-621-A-16-00002-Southern Zone AID-621-A-16-00003-Lake/Western Zone AID-621-A-16-00004- North/Central Zone I have real or potential conflicts of interest to disclose. No If yes answered above, I disclose the I wish to disclose the following: following facts: 1. I do not have a close family member who is an Real or potential conflicts of interest may include, but employee of USAID are not limited to: 2. I do not have a financial interest in the implementing 1. Close family member who is an employee of organizations USAID operating unit managing the project(s) being 3. Regarding current or previous experience with the evaluated or the implementing organization(s) whose project, I wish to disclose that I worked as a consultant with project(s) are being evaluated. Social Solutions International and worked at USAID/Tanzania 2. Financial interest that is direct, or is significant as a consultant to review the Health Information System though indirect, in the implementing organization(s) (HIS) portfolio of the mission in 2018. This review included whose projects are being evaluated or in the outcome reviewing HIS activities of the Boresha Afya projects. of the evaluation. 4. I do not seek employment with USAID or the 3. Current or previous direct or significant though implementing organizations. indirect experience with the project(s) being evaluated, 5. I have been proposed as a consultant in a proposal to including involvement in the project design or previous USAID/Tanzania by PSI for a future project. The proposal iterations of the project. was submitted a few months ago and it is not related to the 4. Current or previous work experience or Boresha Afya program. seeking employment with the USAID operating unit 6. I do not have any preconceived ideas towards the managing the evaluation or the implementing projects or the implementing organizations. organization(s) whose project(s) are being evaluated. 5. Current or previous work experience with an organization that may be seen as an industry competitor with the implementing organization(s) whose project(s) are being evaluated. 6. Preconceived ideas toward individuals, groups, organizations, or objectives of the particular projects and organizations being evaluated that could bias the evaluation. I certify (1) that I have completed this disclosure form fully and to the best of my ability and (2) that I will update this disclosure form promptly if relevant circumstances change. If I gain access to proprietary information of other companies, then I agree to protect their information from unauthorized use or disclosure for as long as it remains proprietary and refrain from using the information for any purpose other than that for which it was furnished. Signature Date August 9, 2019 162 Name Mwita Wambura Title Researcher Organization ME&A Evaluation Position Team member Evaluation Award Number (contract or other instrument) AID-OAA-I-15-00024/AID-621-TO-17-00005 USAID Project(s) Evaluated (Include project name(s), implementer name(s) and award number(s), if applicable) USAID Boresha Afya AID-621-A-16-00002-Southern Zone AID-621-A-16-00003-Lake/Western Zone AID-621-A-16-00004- North/Central Zone I have real or potential conflicts of interest to disclose. No If yes answered above, I disclose the following facts: Real or potential conflicts of interest may include, but are not limited to: 1. Close family member who is an employee of USAID operating unit managing the project(s) being evaluated or the implementing organization(s) whose project(s) are being evaluated. 2. Financial interest that is direct, or is significant though indirect, in the implementing organization(s) whose projects are being evaluated or in the outcome of the evaluation. 3. Current or previous direct or significant though indirect experience with the project(s) being evaluated, including involvement in the project design or previous iterations of the project. 4. Current or previous work experience or seeking employment with the USAID operating unit managing the evaluation or the implementing organization(s) whose project(s) are being evaluated. 5. Current or previous work experience with an organization that may be seen as an industry competitor with the implementing organization(s) whose project(s) are being evaluated. 6. Preconceived ideas toward individuals, groups, organizations, or objectives of the particular projects and organizations being evaluated that could bias the evaluation. I certify (1) that I have completed this disclosure form fully and to the best of my ability and (2) that I will update this disclosure form promptly if relevant circumstances change. If I gain access to proprietary information of other companies, then I agree to protect their information from unauthorized use or disclosure for as long as it remains proprietary and refrain from using the information for any purpose other than that for which it was furnished. Signature Digitally signed by MWITA WAMBURA, Date August 5, 2019 163 Name Dominic Mosha Title Public Health Specialist Organization NORC at the University of Chicago (Data for Development) Evaluation Position Team member Evaluation Award Number (contract or other instrument) AID-OAA-I-15-00024/AID-621-TO-17-00005 USAID Project(s) Evaluated (Include project USAID Boresha Afya name(s), implementer name(s) and award AID-621-A-16-00002-Southern Zone number(s), if applicable) AID-621-A-16-00003-Lake/Western Zone AID-621-A-16-00004-North/Central Zone I have real or potential conflicts of interest to disclose. No If yes answered above, I disclose the following facts: Real or potential conflicts of interest may include, but are not limited to: 1. Close family member who is an employee of USAID operating unit managing the project(s) being evaluated or the implementing organization(s) whose project(s) are being evaluated. 2. Financial interest that is direct, or is significant though indirect, in the implementing organization(s) whose projects are being evaluated or in the outcome of the evaluation. 3. Current or previous direct or significant though indirect experience with the project(s) being evaluated, including involvement in the project design or previous iterations of the project. 4. Current or previous work experience or seeking employment with the USAID operating unit managing the evaluation or the implementing organization(s) whose project(s) are being evaluated. 5. Current or previous work experience with an organization that may be seen as an industry competitor with the implementing organization(s) whose project(s) are being evaluated. 6. Preconceived ideas toward individuals, groups, organizations, or objectives of the particular projects and organizations being evaluated that could bias the evaluation. I certify (1) that I have completed this disclosure form fully and to the best of my ability and (2) that I will update this disclosure form promptly if relevant circumstances change. If I gain access to proprietary information of other companies, then I agree to protect their information from unauthorized use or disclosure for as long as it remains proprietary and refrain from using the information for any purpose other than that for which it was furnished. Signature Date July 29, 2019 164 Name Nasson Exaudy Konga Title Monitoring & Evaluation Specialist Organization NORC at the University of Chicago Evaluation Position Team member Evaluation Award Number (contract or other instrument) AID-621-A-16-00002-Southern Zone AID-621-A-16-00003-Lake/Western Zone AID-621-A-16-00004-North/Central Zone USAID Project(s) Evaluated (Include project name(s), implementer name(s) and award number(s), if applicable) USAID Boresha Afya I have real or potential conflicts of interest to disclose. No potential conflict of Interest If yes answered above, I disclose the following facts: Real or potential conflicts of interest may include, but are not limited to: 1. Close family member who is an employee of USAID operating unit managing the project(s) being evaluated or the implementing organization(s) whose project(s) are being evaluated. 2. Financial interest that is direct, or is significant though indirect, in the implementing organization(s) whose projects are being evaluated or in the outcome of the evaluation. 3. Current or previous direct or significant though indirect experience with the project(s) being evaluated, including involvement in the project design or previous iterations of the project. 4. Current or previous work experience or seeking employment with the USAID operating unit managing the evaluation or the implementing organization(s) whose project(s) are being evaluated. 5. Current or previous work experience with an organization that may be seen as an industry competitor with the implementing organization(s) whose project(s) are being evaluated. 6. Preconceived ideas toward individuals, groups, organizations, or objectives of the particular projects and organizations being evaluated that could bias the evaluation. I certify (1) that I have completed this disclosure form fully and to the best of my ability and (2) that I will update this disclosure form promptly if relevant circumstances change. If I gain access to proprietary information of other companies, then I agree to protect their information from unauthorized use or disclosure for as long as it remains proprietary and refrain from using the information for any purpose other than that for which it was furnished. Signature Date June 7, 2019 165 Name Daud Siwalaze Title M&E Specialist Organization NORC at the University of Chicago (Data for Development) Evaluation Position Team member Evaluation Award Number (contract or other instrument) AID-OAA-I-15-00024/AID-621-TO-17-00005 USAID Project(s) Evaluated (Include project USAID Boresha Afya name(s), implementer name(s) and award number(s), if AID-621-A-16-00002-Southern Zone applicable) AID-621-A-16-00003-Lake/Western Zone AID-621-A-16-00004-North/Central Zone I have real or potential conflicts of interest to disclose. No If yes answered above, I disclose the following facts: Real or potential conflicts of interest may include, but are not limited to: 1. Close family member who is an employee of USAID operating unit managing the project(s) being evaluated or the implementing organization(s) whose project(s) are being evaluated. 2. Financial interest that is direct, or is significant though indirect, in the implementing organization(s) whose projects are being evaluated or in the outcome of the evaluation. 3. Current or previous direct or significant though indirect experience with the project(s) being evaluated, including involvement in the project design or previous iterations of the project. 4. Current or previous work experience or seeking employment with the USAID operating unit managing the evaluation or the implementing organization(s) whose project(s) are being evaluated. 5. Current or previous work experience with an organization that may be seen as an industry competitor with the implementing organization(s) whose project(s) are being evaluated. 6. Preconceived ideas toward individuals, groups, organizations, or objectives of the particular projects and organizations being evaluated that could bias the evaluation. I certify (1) that I have completed this disclosure form fully and to the best of my ability and (2) that I will update this disclosure form promptly if relevant circumstances change. If I gain access to proprietary information of other companies, then I agree to protect their information from unauthorized use or disclosure for as long as it remains proprietary and refrain from using the information for any purpose other than that for which it was furnished. Signature Date June 7, 2019 166 Name Bahati Tenga Title M&E Specialist Organization NORC at the University of Chicago (Data for Development) Evaluation Position Team member Evaluation Award Number(contract or other instrument) AID-OAA-I-15-00024/AID-621-TO-17-00005 USAID Project(s) Evaluated(Include USAID Boresha Afya project name(s), implementer name(s) and AID-621-A-16-00002-Southern Zone award number(s), if applicable) AID-621-A-16-00003-Lake/Western Zone AID-621-A-16-00004-North/Central Zone I have real or potential conflicts of interest to disclose. No If yes answered above, I disclose the following facts: Real or potential conflicts of interest may include, but are not limited to: 1. Close family member who is an employee of USAID operating unit managing the project(s) being evaluated or the implementing organization(s) whose project(s) are being evaluated. 2. Financial interest that is direct, or is significant though indirect, in the implementing organization(s) whose projects are being evaluated or in the outcome of the evaluation. 3. Current or previous direct or significant though indirect experience with the project(s) being evaluated, including involvement in the project design or previous iterations of the project. 4. Current or previous work experience or seeking employment with the USAID operating unit managing the evaluation or the implementing organization(s) whose project(s) are being evaluated. 5. Current or previous work experience with an organization that may be seen as an industry competitor with the implementing organization(s) whose project(s) are being evaluated. 6. Preconceived ideas toward individuals, groups, organizations, or objectives of the particular projects and organizations being evaluated that could bias the evaluation. I certify (1) that I have completed this disclosure form fully and to the best of my ability and (2) that I will update this disclosure form promptly if relevant circumstances change. If I gain access to proprietary information of other companies, then I agree to protect their information from unauthorized use or disclosure for as long as it remains proprietary and refrain from using the information for any purpose other than that for which it was furnished. Signature Date July 29, 2019 167 Name Ingrid RojasArellano Title Principal Research analyst Organization NORC at the University of Chicago Evaluation Position Team member Evaluation Award Number (contract or other instrument) AID-OAA-I-15-00024/AID-621-TO-17-00005 USAID Project(s) Evaluated (Include project name(s), USAID Boresha Afya implementer name(s) and award number(s), if applicable) AID-621-A-16-00002-Southern Zone AID-621-A-16-00003-Lake/Western Zone AID-621-A-16-00004-North/Central Zone I have real or potential conflicts of interest to disclose. No If yes answered above, I disclose the following facts: Real or potential conflicts of interest may include, but are not limited to: 1. Close family member who is an employee of USAID operating unit managing the project(s) being evaluated or the implementing organization(s) whose project(s) are being evaluated. 2. Financial interest that is direct, or is significant though indirect, in the implementing organization(s) whose projects are being evaluated or in the outcome of the evaluation. 3. Current or previous direct or significant though indirect experience with the project(s) being evaluated, including involvement in the project design or previous iterations of the project. 4. Current or previous work experience or seeking employment with the USAID operating unit managing the evaluation or the implementing organization(s) whose project(s) are being evaluated. 5. Current or previous work experience with an organization that may be seen as an industry competitor with the implementing organization(s) whose project(s) are being evaluated. Preconceived ideas toward individuals, groups, organizations, or objectives of the particular projects and organizations being evaluated that could bias the evaluation. I certify (1) that I have completed this disclosure form fully and to the best of my ability and (2) that I will update this disclosure form promptly if relevant circumstances change. If I gain access to proprietary information of other companies, then I agree to protect their information from unauthorized use or disclosure for as long as it remains proprietary and refrain from using the information for any purpose other than that for which it was furnished. Signature Date June 7, 2019 168 169 Name Zoe Grotophorst Title Principal Research Analyst Organization NORC at the University of Chicago Evaluation Position Team Member Evaluation Award Number (contract or other instrument) AID-OAA-I-15-00024/AID-621-TO-17-00005 USAID Project(s) Evaluated (Include project USAID Boresha Afya name(s), implementer name(s) and award number(s), AID-621-A-16-00002-Southern Zone if applicable) AID-621-A-16-00003-Lake/Western Zone AID-621-A-16-00004-North/Central Zone I have real or potential conflicts of interest to disclose. No If yes answered above, I disclose the following facts: Real or potential conflicts of interest may include, but are not limited to: 1. Close family member who is an employee of USAID operating unit managing the project(s) being evaluated or the implementing organization(s) whose project(s) are being evaluated. 2. Financial interest that is direct, or is significant though indirect, in the implementing organization(s) whose projects are being evaluated or in the outcome of the evaluation. 3. Current or previous direct or significant though indirect experience with the project(s) being evaluated, including involvement in the project design or previous iterations of the project. 4. Current or previous work experience or seeking employment with the USAID operating unit managing the evaluation or the implementing organization(s) whose project(s) are being evaluated. 5. Current or previous work experience with an organization that may be seen as an industry competitor with the implementing organization(s) whose project(s) are being evaluated. 6. Preconceived ideas toward individuals, groups, organizations, or objectives of the particular projects and organizations being evaluated that could bias the evaluation. I certify (1) that I have completed this disclosure form fully and to the best of my ability and (2) that I will update this disclosure form promptly if relevant circumstances change. If I gain access to proprietary information of other companies, then I agree to protect their information from unauthorized use or disclosure for as long as it remains proprietary and refrain from using the information for any purpose other than that for which it was furnished. Signature Date June 7, 2019 170