October 15, 2020 This publication was produced at the request of the United States Agency for International Development (USAID). It was prepared independently by Panagora Group for the USAID/South Africa Monitoring, Evaluation, Research, and Learning Activity. It was authored by Roy Natherson, Cheryl Goldstone, and Lahla Ngubeni. USAID/South Africa Tuberculosis South Africa Project (TBSAP) Midterm Evaluation Report USAID/SOUTH AFRICA TUBERCULOSIS SOUTH AFRICA PROJECT (TBSAP) MIDTERM EVALUATION REPORT October 15, 2020 Cover photo Credit: TBSAP Caption: A health worker provides a tuberculosis patient with treatment This document was produced for review by the United States Agency for International Development. It was prepared by Panagora Group for the USAID/South Africa Monitoring, Evaluation, Research, and Learning Activity, IDIQ No. AID-OAA-I-15-00025, Task Order No. 72067418F00002 Monitoring, Evaluation, Research, and Learning Activity. Recommended citation: USAID Monitoring, Evaluation, Research, and Learning Activity. 2020. Tuberculosis South Africa Project (TBSAP) Midterm Evaluation Report. Silver Spring, Maryland: Panagora Group. Monitoring, Evaluation, Research, and Learning Activity. Panagora Group Contact: Jasmine Buttolph Global Health Director Panagora Group 8601 Georgia Avenue, Suite 905 Silver Spring, MD 20910 USA jasminebuttolph@panagoragroup.net Katie Reichert Panagora South Africa Country Director MERL Chief of Party USAID/South Africa Contractor Monitoring, Evaluation, Research and Learning Activity kreichert@panagorasouthafrica.net Mobile: +27 (0) 64 074 8712 ACKNOWLEDGEMENTS The evaluation team is grateful to the more than 80 individuals who generously gave their time and insights for this report. Throughout this evaluation, the team benefited from the contributions of National Department of Health (NDoH) directors, managers, and health care providers at national, provincial, district, and facility levels, and local organizations that were beneficiaries of the Tuberculosis South Africa Project (TBSAP) Small Grants Program funded by the United States Agency for International Development (USAID). This report represents a collective effort, and incorporates the opinions and observations of a large community of individuals. The team would especially like to thank Cindy Dlamini, USAID/South Africa’s Project Development Specialist – TB and TBSAP’s Contracting Officer’s Representative, for her thoughtful guidance throughout the evaluation process. The team is also grateful to the prime implementer, University Research Co., LLC (URC), and TBSAP’s implementing partners for making their staff and project information readily available. The Panagora Group provided technical, administrative, and logistical support and coordination with USAID, NDoH, and TBSAP throughout the evaluation. Panagora/South Africa Chief of Party, Katie Reichert, provided technical support and direction to the evaluation team, as did Panagora’s headquarters in the United States. To all who are committed to improved TB service delivery capacity and quality of care in South Africa, we offer these observations, analysis, and recommendations in the confidence that further progress will be realized through collective efforts. CONTENTS ACKNOWLEDGEMENTS ......................................................................................................................2 CONTENTS .........................................................................................................................................3 ACRONYMS........................................................................................................................................1 EXECUTIVE SUMMARY .......................................................................................................................3 EVALUATION PURPOSE & EVALUATION QUESTIONS ...........................................................................7 PROJECT BACKGROUND .....................................................................................................................8 EVALUATION METHODS & LIMITATIONS...........................................................................................10 FINDINGS, CONCLUSIONS AND RECOMMENDATIONS .......................................................................13 QUESTION I: TO WHAT EXTENT HAS TBSAP BEEN EFFECTIVE AND SUCCESSFUL IN ACHIEVING ITS STATED OBJECTIVES AND TARGETS? ................................................................................................................................................. 13 QUESTION 2. WHAT KEY INTERVENTIONS AND ACTIVITIES HAVE BEEN MOST EFFECTIVE/HIGH IMPACT IN ACHIEVING RESULTS? (WHICH ARE LEAST OR LESS EFFECTIVE?) .......................................................................................... 18 QUESTION 3. WHAT CHALLENGES AND CONSTRAINTS DOES THE PROJECT FACE AND WHAT COURSE CORRECTIONS MIGHT IMPROVE THIS? ......................................................................................................................................... 27 QUESTION 4. WHAT HAS CHANGED IN THE EPIDEMIOLOGY AND ENVIRONMENT OF TB IN SOUTH AFRICA, AND HOW HAS THIS AFFECTED THE PROGRAM? .................................................................................................................... 32 CONCLUSIONS........................................................................................................................................... 36 RECOMMENDATIONS.................................................................................................................................. 39 ANNEXES .........................................................................................................................................43 ANNEX I: EVALUATION STATEMENT OF WORK ................................................................................................ 43 ANNEX II: EVALUATION METHODS AND LIMITATIONS ....................................................................................... 63 ANNEX III: DATA COLLECTION INSTRUMENTS.................................................................................................. 68 ANNEX IV: SOURCES OF INFORMATION.......................................................................................................... 84 ANNEX V: DISCLOSURE OF ANY CONFLICTS OF INTEREST ................................................................................... 91 ANNEX VI: ADDITIONAL TBSAP MATERIALS ................................................................................................... 94 Figures and Tables FIGURE 1: TBSAP LOGIC MODEL AND INDICATORS ................................................................................................................ 9 FIGURE 2: TBSAP IMPLEMENTATION MAP, 2019................................................................................................................. 9 FIGURE 3: TBSAP EVALUATION TIMELINE, JANUARY-AUGUST 2020....................................................................................... 10 FIGURE 4: PERFORMANCE OF 8 TBSAP AND 8 COMPARISON DISTRICTS, 2017/18 ................................................................... 16 FIGURE 5: MATERIALS PRODUCED IN COLLABORATION WITH THE NDOH ................................................................................. 17 FIGURE 6: IMPACT OF COVID-19 ON THE PEPFAR TB/HIV PROGRAM IN SOUTH AFRICA .......................................................... 34 FIGURE 7: QUALITATIVE DATA CODES.............................................................................................................................. 64 FIGURE 8: EXAMPLE OF QUALITATIVE DATA TRANSCRIPTION................................................................................................. 65 FIGURE 9: TBSAP LOCAL-LEVEL MULTISECTORAL ACCOUNTABILITY FRAMEWORK........................................................................ 98 FIGURE 10: TB CASCADE ANALYSIS ................................................................................................................................. 98 TABLE 1: KEY INFORMANT INTERVIEWS CONDUCTED........................................................................................................... 11 TABLE 2: TBSAP PERFORMANCE AGAINST 4 OUTCOME INDICATOR TARGETS: FY17‒FY19......................................................... 13 TABLE 3: TBSAP PERFORMANCE AGAINST TARGETS: FY17‒FY19......................................................................................... 14 TABLE 4: CASCADE INDICATOR PERFORMANCE IN DIRECTLY SUPPORTED FAST HOSPITALS 2017‒2019......................................... 21 TABLE 5: TBSAP INTERVENTION AND COMPARISONS DISTRICTS ............................................................................................ 66 TABLE 6: EVALUATION CHALLENGES AND MITIGATION STRATEGIES......................................................................................... 67 TABLE 7: LIST OF KEY INFORMANTS INTERVIEWED............................................................................................................... 84 TABLE 8: TBSAP ACTIVITY ALIGNMENT WITH THE NATIONAL TB PROGRAM STRATEGY ............................................................... 94 ACRONYMS ACSM Advocacy, Communications, and Social Mobilization ART Antiretroviral Therapy CBO Community-based Organization CDCS Country Development Cooperation Strategy CHW Community Health Worker COVID-19 Coronavirus Disease CQI Continuous Quality Improvement DHIS District Health Information System DoH Department of Health DR-TB Drug-resistant Tuberculosis DS-TB Drug-susceptible Tuberculosis DSP District Support Partners ETR Electronic Tuberculosis Register FAST Finding TB Actively, Separating safely, Treating effectively FY Fiscal Year GOSA Government of South Africa GP General Practitioner HAST HIV/AIDS, STI and TB HIV Human Immunodeficiency Virus IEC Information, Education, and Communications IPC Infection Prevention and Control KI Key Informant KII Key Informant Interview LoE Level of Effort LTFU Lost-to-Follow-up M&E Monitoring and Evaluation MDR-TB Multidrug-Resistant Tuberculosis MERL Monitoring, Evaluation, Research and Learning NDoH National Department of Health NGO Non-governmental Organization NHI National Health Insurance NHLS National Health Laboratory Service NTP National Tuberculosis Program PDoH Provincial Department of Health PEPFAR United States President’s Emergency Plan for AIDS Relief PHC Primary Health Care PMP Performance Monitoring Plan PMTCT Prevention of Mother-to-child Transmission PPP Public-Private Partnerships QA Quality Assurance QI Quality Improvement RR-TB Rifampicin-resistant Tuberculosis RTC Regional Training Center SOP Standard Operating Procedure STI Sexually Transmitted Infection TB Tuberculosis TBQI TB Quality Improvement TBSAP Tuberculosis South Africa Project THP Traditional Health Practitioner u-LAM Urine Test for the Detection of the Lipoarabinomannan Antigen URC University Research Co., LLC USAID United States Agency for International Development USG United States Government WBPHCOT Ward-based Primary Health Care Outreach Team WHO World Health Organization XDR-TB Extensively Drug-resistant Tuberculosis 3 | Page EXECUTIVE SUMMARY Evaluation Purpose and Evaluation Questions The purpose of the Tuberculosis South Africa Project (TBSAP) midterm evaluation is to examine how the project’s interventions helped achieve the National Department of Health (NDoH) and the United States Agency for International Development (USAID) Mission’s goal to reduce the tuberculosis (TB) burden in South Africa and to identify areas for improvement. There are three specific evaluation objectives: 1. To determine the degree or extent to which the TBSAP responds to the Government of South Africa’s (GOSA) TB priorities (National Strategic Plan 2017‒2022 and National TB Program Strategic Plan 2017‒2021). 2. To identify key lessons learned, best practices, and challenges that will inform the strategic design of the follow-on TB project. 3. To learn from the midterm successes and challenges of the TBSAP and recommend course corrections for the final year of the project. The following key questions guided this midterm evaluation: 1. To what extent has TBSAP been effective/successful in achieving its stated objectives and targets? 2. What key interventions and activities have been most effective/high impact in achieving results? (Which are least or less effective?) 3. What challenges and constraints did the project face and what course corrections might improve this? 4. What has changed in the epidemiology and environment of TB in South Africa, and how has this affected the program? Project Background South Africa is one of 14 countries considered to be a high-burden country by the World Health Organization (WHO) for TB, TB/HIV, and multidrug-resistant TB (MDR-TB).1 In 2018, WHO estimated that South Africa had an incidence of 301,000 cases of active TB, or 520 per 100,000 population. 2 Approximately 11,000 people became ill with MDR-TB/ rifampicin-resistant TB (RR-TB) that year.3 The 2018 statistics documented that 59 percent of new and relapse TB cases were coinfected with human immunodeficiency virus (HIV),4 further exacerbating the challenges of providing treatment and care to patients. South Africa’s TB program has made progress sustaining trends to reduce mortality due to TB; lowering 1 World Health Organization (WHO). (2019). Global tuberculosis report 2019. Geneva, Switzerland: WHO. https://apps.who.int/iris/handle/10665/329368. 2 WHO. (2018). Global tuberculosis report 2018. Geneva, Switzerland: WHO. https://apps.who.int/iris/handle/10665/274453. 3 Ibid. 4 World Health Organization. (2019). Global tuberculosis report 2019. World Health Organization. https://apps.who.int/iris/handle/10665/274453. 4 | Page TB incidence rates, including reducing the incidence of co-diagnosed TB/HIV; and improving treatment success rates for most categories of cases. However, challenges remain before South Africa can reach its TB elimination goals, with the country falling short of the 2025 End TB goals. The overarching goal of the TBSAP is to reduce the national TB, MDR-TB/extensively drug-resistant TB (XDR-TB), and TB/HIV burden in South Africa. The primary objectives of the project are to: (1) reduce TB infections; (2) increase the sustainability of effective TB response systems; and (3) improve care and treatment of vulnerable populations. TBSAP contributes to the achievement of the USAID Mission’s Development Objective 1: “Health outcomes for South Africans improved,” as expressed in the Country Development Cooperation Strategy (CDCS). It also responds to the Government of South Africa National Strategic Plan for HIV, TB and STIs (2017‒2022) and National TB Program Strategic Plan (2017‒2021); United States Government Global TB Strategy (2015‒2019); and the White House National Action Plan to Combat Multidrug-Resistant TB (Goal 2): improving international capacity and collaboration to combat multidrug-resistant tuberculosis (MDR-TB). Evaluation Design, Methods and Limitations The midterm evaluation was conducted from December 4, 2019 to August 31, 2020, covering the period from the TBSAP’s inception in March 2016 through September 2019. It used a mixed methods methodology of primary and secondary qualitative and quantitative data collection and analysis. Eight of the 14 TBSAP intervention districts across the six provinces were selected for key informant interviews (KIIs): four high burden TB districts, two medium burden, and two low burden. A quasi-experimental comparison of key indicators was also conducted using eight comparison districts matched to the eight TBSAP districts. In addition, the team did a comprehensive literature review of key TBSAP documents and GOSA and US Government (USG) policy documents. The COVID-19 pandemic and response prohibited the team’s travel to the field. This constraint required the team to conduct interviews remotely. Moreover, data could not be collected directly from facilities, limiting the overall analysis to the district level only, and the resulting quantitative analysis. Findings and Conclusions The TBSAP’s overall goal is to contribute to the reduction of the TB burden in South Africa, which will be achieved through progress in four outcome indicators, and other key indicators across the TB cascade. The TBSAP achieved its fiscal year (FY)19 target for one of its four outcome indicators: drug￾resistant TB (DR-TB) treatment initiation. The project was close to achieving its DR-TB treatment success target, and approximately 10 percentage points lower that its targets for both drug-susceptible (DS) TB (DS-TB) treatment success and antiretroviral therapy (ART) coverage for TB/HIV coinfected clients. The TBSAP performed well in TB screening and testing, with variable performance in rates of clients lost to follow-up (LTFU). The DS-TB treatment success rate and ART initiation among TB clients were affected by data entry and data quality issues. In 2017, the selected data elements of the TB module transitioned between two of South Africa’s health information systems, from ETR.net to TIER.Net, with associated data and system errors. This change resulted in a similar downward trend in the national TB data during this period. Moreover, the TBSAP noted a tendency to prioritize HIV program data for entry in TIER.Net, with data entry backlogs at multiple sites. The project is implementing data clean-up and other measures across its districts to mitigate these challenges. A contributing factor in the underperformance on ART initiation among TB clients cited by the project was differences in data reporting protocols in the Western Cape province (West Coast, City of Cape Town, and Cape Winelands districts), which reported ART initiations with TB outcomes and not with case finding. The TBSAP is aligned with and responsive to global and USG guidelines and strategies, and to GOSA 5 | Page strategies and policies. Several TBSAP interventions are currently being scaled up, integrated, and institutionalized in national policy. The TBSAP has implemented several key interventions and activities that have had a positive effect on TB outcomes in the selected districts. The evaluation team assessed the effectiveness of TBSAP interventions based on two factors: performance indicators that could be linked to a specific activity, and the frequency with which specific interventions were mentioned as project successes or strengths during the qualitative interviews. Based on these criteria, the three most effective interventions were: (1) building partnerships for multisectoral accountability; (2) infection prevention and control (IPC) activities; and (3) continuous quality improvement (CQI). Less effective interventions were: (1) information system management; (2) advocacy, communications, and social mobilization (ACSM); and (3) the ConnecTB app. The TBSAP’s technical assistance model has been well received by all districts supported. However, challenges related to project design and management have inhibited the project and may have unintentionally prevented it from reaching scale and delivering fully on its targets. The challenges became constraints over the course of the project. They are: (1) TBSAP contract and management; (2) sub￾grantee management; and (3) human resource allocation during project planning. The project also experienced several constraints related to South Africa’s health systems, outside its control, which posed considerable challenges to its success. They are: (1) information system management; (2) funding and resource constraints; (3) ongoing capacity development needs; and (4) geographical and socioeconomic factors. Although key data sources on TB epidemiology in South Africa (such as the latest National TB Prevalence Study and 2018 and 2019 TB data sets) are pending public release, TB program policy changes indicate the integration and uptake of advances in DR-TB treatment and decentralized service delivery. However, the COVID-19 pandemic has adversely affected the National Tuberculosis Program (NTP) in terms of management, resource allocation (including human resources), and service delivery, with significant losses in TB outcomes across the cascade. The TBSAP has contributed to the national COVID-19 response, such as screening and education efforts, and leveraged its existing IPC and capacity building measures to strengthen the health system at the facility level. Recommendations The TBSAP ends in March 2021 and USAID has started the planning process for the design of its follow￾on TB project. This evaluation offers recommendations for key stakeholders in the short, medium, and long term, as summarized below. Short-term recommendations for USAID and TBSAP through the end of TBSAP’s contract in March 2021 include: (1) intensified, direct support to the NTP in data management and systems; (2) fast-track approval and continuity of non-governmental organization (NGO) contracts; (3) leveraging TBSAP strengths to help the NTP adapt to the COVID-19 pandemic; and (4) TBSAP documentation of end of project results and transition plans. Medium-term recommendations for USAID to ensure consistency of support in TBSAP’s 14 high TB burden districts include: (1) an extension to TBSAP’s contract, from April 2021 through the anticipated start date of USAID’s follow-on project(s); (2) coordination with the DoH and TBSAP to ensure USAID’s contract extension includes coverage of high-impact priority interventions to sustain gains in TB outcomes achieved in the last five years; (3) alignment of key interventions with the NTP’s “TB Recovery Plan” in response to COVID-19. 6 | Page Long-term recommendations for USAID in the design of its follow-on TB project(s) include: (1) a cooperative agreement as a more appropriate and flexible funding mechanism, especially in the context of COVID-19; (2) a comprehensive indicator set with appropriate measures of project accountability; (3) building on successful approaches from the TBSAP, including the Finding TB Actively, Separating safely, Treating effectively (FAST) approach and CQI; (4) build on and expand public-private engagement and partnerships; (5) ensure continuity of NGO contributions; and (6) strengthen TB/HIV/COVID-19 integration and coordination for improved program outcomes. Recommendations for the NDoH include: (1) address the country’s information management system challenges; (2) oversee an assessment of and manage health system software applications; and (3) strengthen NTP coordination and ACSM. 7 | Page EVALUATION PURPOSE & EVALUATION QUESTIONS The purpose of the Tuberculosis South Africa Project (TBSAP) midterm evaluation, as stated in the Scope of Work (Annex I), is to examine how the project’s interventions contributed to the National Department of Health (NDoH) and the United States Agency for International Development (USAID) Mission’s goal to reduce the tuberculosis (TB) burden in South Africa and to identify areas for improvement. There are three specific evaluation objectives: 1. To determine the degree or extent to which the TBSAP responds to the Government of South Africa’s (GOSA) TB priorities (National Strategic Plan 2017‒2022 and National TB Program Strategic Plan 2017‒2021). 2. To identify key lessons learned, best practices, and challenges that will inform the strategic design of the follow-on TB project. 3. To learn from the midterm successes and challenges of the TBSAP and recommend course corrections for the final year of the project. The target audiences for this evaluation are: • USAID/South Africa and the United States President’s Emergency Plan for AIDS Relief (PEPFAR) • GOSA, NDoH, and National Tuberculosis Program (NTP) stakeholders at all levels • Other donors, implementing partners, and key stakeholders engaged in TB prevention and care The TBSAP midterm evaluation consolidates lessons learned, collaboration, and adaptation across the project, identifying key factors, interventions, and investments that contribute to the successes and challenges in reducing the burden of TB in South Africa. The evaluation also documents opportunities for future improvements, including possible approaches beyond the scope of current activities. Findings and recommendations from this evaluation will inform course corrections in TBSAP’s final year, and USAID/South Africa’s design and investments in future TB projects and interventions. The results of this evaluation will contribute to the learning agenda for TB for USAID/South Africa’s Country Development Cooperation Strategy (CDCS), and will assist the NDoH and other GOSA stakeholders to better identify the most effective approaches and interventions, with the continued goal of reducing the burden of TB across South Africa. The following key questions guided this evaluation, as outlined in the Scope of Work (Annex I): 1. To what extent has TBSAP been effective/successful in achieving its stated objectives and targets? 2. What key interventions and activities have been most effective/high impact in achieving results? (Which are least or less effective?) 3. What challenges and constraints did the project face and what course corrections might improve this? 4. What has changed in the epidemiology and environment of TB in South Africa, and how has this affected the program? 8 | Page PROJECT BACKGROUND TB Burden in South Africa South Africa is one of 14 countries considered to be a high-burden country by the World Health Organization (WHO) for TB, TB/HIV, and multidrug-resistant TB (MDR-TB).5 In 2018, WHO estimated that South Africa had an incidence of 301,000 cases of active TB, or 520 per 100,000 population. 6 Approximately 11,000 people became ill with MDR-TB/ rifampicin-resistant TB (RR-TB) that year.7 The 2018 statistics documented that 59 percent of new and relapse TB cases were coinfected with human immunodeficiency virus (HIV), 8 further exacerbating the challenges of providing treatment and care to patients. The TB program has made progress in sustaining trends to reduce TB mortality. This has been done by lowering TB incidence rates, including reducing the incidence of co-diagnosed TB/HIV; and improving treatment success rates for most categories of cases. WHO confirms a clear and sustained downward trend in TB case notifications in South Africa, which may be driven by high antiretroviral treatment (ART) coverage rates. Of the 120,862 TB patients who were known to be HIV-positive in 2018, 104,625 (87%) were reported to be on ART.9 Moreover, 65 percent of HIV-positive people newly enrolled in HIV care were on TB preventive treatment that same year. 10 However, challenges remain before South Africa can achieve its TB elimination goals. The country is falling short of the 2025 End TB goals. Treatment coverage was 68 percent in 2017 and the treatment success rate was 82 percent. In 2017, South Africa reported that of the 15,986 laboratory confirmed MDR-TB/RR-TB cases, only 10,259 (64%) began treatment. Moreover, the treatment success rate in 2016 of 54 percent for patients with MDR-TB remains low. 11 South Africa’s TB efforts are primarily funded by domestic sources, at 87 percent, and the remaining 13 percent from international sources. 12 The Tuberculosis South Africa Project The TBSAP is a five-year (March 2016‒March 2021) project funded by USAID/South Africa. It has a funding ceiling of $64,800,000. TBSAP’s overarching goal is to reduce the burden of TB, MDR-TB/ extensively drug-resistant TB (XDR-TB), and TB/HIV in South Africa. The primary objectives of the project are to: (1) reduce TB infections; (2) increase the sustainability of effective TB response systems; 5 World Health Organization (WHO). (2019). Global tuberculosis report 2019. Geneva, Switzerland: WHO. https://apps.who.int/iris/handle/10665/329368. 6 WHO. (2018). Global tuberculosis report 2018. Geneva, Switzerland: WHO. https://apps.who.int/iris/handle/10665/274453. 7 Ibid. 8 World Health Organization. (2019). Global tuberculosis report 2019. World Health Organization. https://apps.who.int/iris/handle/10665/274453. 9 World Health Organization. (2019). Global tuberculosis report 2019. World Health Organization. https://apps.who.int/iris/handle/10665/274453. 10 Ibid. 11 Ibid. Due to the longer duration of DR-TB treatment, data from 2016 is the latest available. 12 Ibid. 9 | Page and (3) improve care and treatment of vulnerable populations. Figure 1 presents the project’s logic model and its indicators. The TBSAP contributes to the achievement of the USAID Mission’s Development Objective 1: “Health outcomes for South Africans improved,” as expressed in the Country Development Cooperation Strategy. The TBSAP also responds to the GOSA National Strategic Plan for HIV, TB and STIs (2017‒2022) and National TB Program Strategic Plan (2017‒2021); USG Global TB Strategy (2015‒2019); and the White House National Action Plan to Combat Multidrug-Resistant TB (Goal 2): improving international capacity and collaboration to combat multidrug-resistant tuberculosis (MDR-TB). TBSAP is implemented by the University Research Co., LLC (URC) and its sub-partners in 14 districts across six provinces. Figure 2 shows the districts, the population coverage, the number of TB patients, and the number of drug-resistant tuberculosis (DR-TB) patients by district in 2019. Figure 1: TBSAP logic model and indicators Figure 2: TBSAP implementation map, 2019 10 | Page EVALUATION METHODS & LIMITATIONS This midterm evaluation was conducted from December 4, 2019 to August 31, 2020. It covers the period from the TBSAP’s inception in March 2016 through September 2019. The evaluation team included three technical experts: Roy Natherson (Team Leader), Cheryl Goldstone (TB Expert), and Lahla Ngubeni (Evaluation Specialist). Figure 3 outlines key milestones in the evaluation timeline. Figure 3: TBSAP evaluation timeline, January-August 2020 Study Design The midterm evaluation used a mixed methods methodology of primary and secondary qualitative and quantitative data collection and analysis. The evaluation team selected eight of the 14 TBSAP intervention districts across the six provinces for key informant interviews (KIIs): four high burden TB provinces, two medium burden, and two low burden. A quasi-experimental comparison of key outcome indicators was also conducted using eight comparison districts matched to the eight TBSAP districts. Key Data Sources The following sources were used to collect data to address the evaluation objectives and questions: • Key informant interviews. In-depth interviews were conducted with key informants (KIs) at national, provincial, and district levels. At the national level, USAID, NDoH, other donors and implementing agencies, and partners were interviewed. At provincial, district, facility, and 11 | Page community levels, the team interviewed a range of Department of Health (DoH) managers and frontline health workers, TBSAP staff, community partner organizations, and other implementing partners working in TB. A total of 82 KIIs were conducted (Table 1). Annex IV provides a complete list of the KIIs. • TB data sources. TBSAP project data and District Health Information System (DHIS) district data were used to respond to the evaluation objectives and questions. • Key documents. A comprehensive literature review was conducted. Annex IV lists the documents reviewed, including: o TBSAP documents: Original USAID request for proposal, the TBSAP contract, and its modifications; TBSAP quarterly reports, annual reports, and annual workplans; Performance Monitoring Plan (PMP) and monitoring and evaluation (M&E) plans; and other TBSAP presentations, reports, and materials. o Global and South African guidance documents: The desk review included South Africa’s National Strategic Plan for HIV, TB, STIs (2012‒2016 and 2017‒2022), National TB Strategic Plan (2012‒2016 and 2017‒2022), and the NDoH Joint TB, HIV, PMTCT, STI and Hepatitis Program Review, 2019. Global TB guidance documents from WHO, STOP TB, and End TB, and other relevant journal articles and reports from partners working in TB in South Africa were included in the desk review. Table 1: Key Informant Interviews Conducted National-level # of KIs Respondents by Type and Location NDoH 4 NDoH Directors USAID 1 USAID TB Program Office Other National-Level Key Stakeholders 6 Global Fund, Bill and Melinda Gates Foundation, TB Think Tank, WHO, THINK, and Interactive Research & Development TBSAP Executive Team 15 TBSAP Executive Team - Headquarters Provincial & District-level # of KIs Respondents by Type and Location Provincial TB Management Teams 6 Free State, Gauteng, Western Cape, and Eastern Cape District TB Management Teams 11 City of Johannesburg, eThekwini, OR Tambo, Sekhukhune, Waterberg, uMkhanyakude, Mangaung, Cape Winelands, and Sarah Baartman Facility & Community-level # of KIs Respondents by Type and Location Facility-level TB Staff 2 Heidendal CHC and Hlabisa Gateway Clinic Community Organizations 5 Asiphile eUganda, Kgatelelopele Community Home Based Care, Care Ministries, Mosamaria, and Mfesane TBSAP Operational Management 16 TBSAP Team Managers across 6 provinces: Western Cape, Limpopo, Eastern Cape, Free State, Gauteng, and KwaZulu-Natal (KZN) Implementing Partners 16 Anova Health Institute, Aquity Health Innovations, MatCH, Isibani Development Partners, and TB/HIV Care Total KIIs Conducted 82 12 | Page Data Analysis The evaluation team applied a mixed methods approach to collect qualitative and quantitative data on the four evaluation questions. A comparative analysis of the performance of TBSAP districts and epidemiologically similar non-intervention districts was carried out to triangulate the findings and build convergence. Additional information on the evaluation methods, data collection instruments, and sources of information is provided in Annexes II, III, and IV. Constraints and Limitations Implementation of the evaluation experienced two key constraints. First, in March 2020, the GOSA enacted a national lockdown in response to the coronavirus disease (COVID-19) pandemic, which prevented the evaluation team from planned travel to districts and facilities for qualitative interviews and data collection. This constraint required the team to conduct interviews remotely. Moreover, data could not be collected directly from facilities. This constraint limited the overall analysis to the district level only. Second, TB datasets were available at the district level only, and through 2017 only, limiting the resulting quantitative analysis of comparative data. The evaluation team therefore conducted a comparative analysis of results from eight TBSAP districts with eight comparison districts through the first year of TBSAP implementation only, not through the three-year period of implementation as initially planned. Secondary data from the TBSAP and qualitative data from KIIs provided further insights on successes and limitations at the facility and community levels. Constraints and limitations are further detailed in Annex II. Ethical Considerations Verbal consent was obtained from KIs, per USAID guidelines. All interview data were kept confidential and were used for evaluation purposes only. 13 | Page FINDINGS, CONCLUSIONS AND RECOMMENDATIONS Question I: To what extent has TBSAP been effective and successful in achieving its stated objectives and targets? The TBSAP’s overall goal is to contribute to the reduction of the burden of TB through the achievement of its four outcome indicators and other key indicators across the TB cascade (TB screening, testing, and treatment completion, among others). The evaluation sought to assess the TBSAP’s effectiveness by: (1) comparing the project’s results with its annual targets across the TB cascade of care, including its four outcome indicators; (2) comparing the TBSAP’s results in eight selected intervention districts with results in eight comparison districts across the TB cascade of care, including the project’s four outcome indicators; and (3) assessing the project’s alignment with key national and donor strategies as benchmarks of best practice guidance. TBSAP Performance (TBSAP results compared with annual targets) The TBSAP’s performance was assessed against its four outcome indicators and other indicators across the TB cascade. Results against the four outcome indicators are presented in Table 2. Table 2: TBSAP Performance against 4 Outcome Indicator Targets: FY17‒FY19 INDICATOR Baseline 2016 FY17 FY18 FY19** FY19 Target Final Target (FY21) Achieved Achieved Achieved Treatment success rate for DS-TB cases 82.4% 82.6% 75.0% 77.0% 87.7% 90% Initiation of confirmed DR-TB cases on appropriate treatment 58.7% 64.2% 72.4% 82.0% 77.5% 100% Treatment success rate for DR-TB cases 53.7% 53% 75.0%** 64.0% 66.5% 75% % ART coverage for TB/HIV co￾infected patients 91.0% 85.6% 81.9% 85.0% 96.0% 100% Source: USAID TBSAP PMP October 31, 2019 ** Data through March 2019 only The TBSAP achieved its fiscal year (FY)19 targets for one of the four outcome indicators: DR-TB treatment initiation. The project was within three percentage points of achieving its FY19 target for DR￾TB treatment success, and approximately 10 percentage points lower than its targets for both DS-TB treatment success and ART coverage for TB/HIV coinfected clients. The TBSAP performed well in TB screening and testing, with variable performance in rates of clients lost to follow-up (LTFU) (and rates steadily increasing among DS-TB clients). These latter results are summarized and explained in Table 3. 14 | Page Table 3: TBSAP Performance against Targets: FY17‒FY19 INDICATOR Baseline 2016 FY17 FY18 FY19 (see specific period) Target Actual Target Actual Target Actual Period TB incidence (WHO 2018 Global Report) 6.6% Drug-susceptible TB Screen for TB symptoms 68.7% 73.0% 75.0% 77.2% 82.0% 81.5% 86.0% Oct 2019‒Jun 2020 TB symptomatic client with sputum sent 84.2% 85.4% 76.0% 86.5% 88.0% 87.7% 91.0% TB client start on treatment 94.0% 95.0% 96% 96.0% 96.0% 98.0% 97.0% TB client successfully completed Rx* 82.4% 85.4% 82.6% 86.5% 75.0% 87.7% 77.0% Oct 2018‒Mar TB client lost to follow up 6.7% 6.4% 6% 6.0% 9.4% 5.7% 11.2% 2019 Drug-resistant TB Number of DR-TB cases diagnosed 19,073 15,986 13,199 2,367 Oct 2019‒Mar DR-TB client confirmed 2020 initiated on Rx* 58.7% 66.0% 64.2% 71.2% 72.4% 77.5% 82.0% DR-TB treatment success* 53.7% 58.0% 53% 62.2% 75.0%** 66.5% 64.0% Jan‒Mar 2019 short regimen DR-TB client lost to follow up 18.8% 18.0% 21.5% 17.3% 11.0% 16.5% 13.0% TB/HIV Coinfection Proportion of TB clients tested for HIV 95.8% 90.0% 94.4% 90.0% 82.9% 90.0% 71.0% Oct 2019‒Mar % of TB/HIV coinfected clients 2020 on ART* 91.0% 93.0% 85.6% 95.0% 81.9% 96.0% 85. 0% Source: USAID TBSAP PMP October 31, 2019 *four outcome indicators **for bedaquiline short course regimen Targets were initially set at the baseline for the project in 2016 but were subsequently updated to take into account the 90/90/90 methodology employed in the Stop TB Strategy. Midway through the project, the TBSAP was on track to achieve 5 of its 10 key indicators. Drug-susceptible TB The TBSAP districts performed well in DS-TB screening, testing, and initiation between FY17 and FY19, but underperformed in treatment success and LTFU. The TBSAP’s districts TB screening rate increased from a baseline of 68.7 percent in 2016 to 86 percent in FY19, surpassing the target of 81.48 percent for FY19. The testing rate also increased year by year, surpassing its FY19 target of 87.68 percent and benefitting from enhanced diagnostic services, including the expansion of GeneXpert, which facilitates faster detection of both DS- and DR-TB. The treatment initiation rate was maintained at above the 90 percent target from baseline to FY19, with a DS-TB treatment initiation rate of 97 percent in FY19. The TBSAP districts’ treatment success rate for DS-TB varied, declining from 82.6 percent in FY17 to 75 percent in FY18 and rising slightly to 77 percent in FY19. Increasing DS-TB LTFU contributed to this performance, increasing from 6 percent in FY17 to 9.4 percent in FY18 to 11.2 percent in FY19. TBSAP cited adherence challenges and deaths as contributing factors. The DS-TB treatment success rate was also affected by data entry and data quality issues. In 2017, the TB module was transitioned between two of South Africa’s health information systems, from ETR.net to TIER.Net, with associated data and system errors. This change resulted in a similar downward trend in the national TB data during this period. Moreover, the TBSAP noted a tendency to prioritize HIV program data for entry in TIER.Net, 15 | Page with data entry backlogs at multiple facilities. The project is implementing data clean-up and other measures across its districts to mitigate these challenges. Drug-resistant TB The TBSAP districts performed well in DR-TB initiation and LTFU between FY17 and FY19, but underperformed in treatment success. The DR-TB treatment initiation rate surpassed the FY19 target, at 82 percent, with progressive improvement year-by-year from FY17 to FY19. Of note, TBSAP reported that 74 percent of South Africa’s DR-TB patients enrolled on new drugs and regimens were from supported districts. The TBSAP met its target for DR-TB treatment success only in FY18, although FY19 performance of 64 percent against the 66.48 percent target was close. TBSAP’s FY18 and FY19 results, which were notably higher than baseline and FY17 performance, were facilitated by the introduction of short-course bedaquiline therapy. The TBSAP met its FY18 and FY19 targets for DR-TB LTFU, with 13 percent LTFU in FY19 compared with a target of 16.52 percent. DR-TB reporting was not integrated in the TB module in TIER.Net; therefore, performance reporting was not affected by the same data quality issues as were DS-TB data, as discussed above. The project supported the decentralization of DR-TB management in all 14 districts and 63 sub-districts, in line with GOSA policy of “one functional decentralized DR-TB site per sub-district.” As a result, patients receiving the short course regimen for DR-TB showed a marked reduction in the initial LTFU rates (including a 50% reduction in the death rate). Moreover, the proportion of XDR-TB patients initiated on treatment increased from 65 percent in FY17 to 98 percent in FY19. TB/HIV Coinfection TBSAP districts underperformed in terms of the proportion of TB clients who were tested for HIV and the percentage of TB/HIV coinfected clients on ART. It did not meet the annual targets for either indicator from FY17 to FY19. TBSAP performance in the proportion of TB clients who were tested for HIV declined year by year, from 94.4 percent in FY17 to 82.9 percent in FY18 to 71 percent in FY19. The TBSAP achieved 85 percent ART coverage compared with a 96 percent target in FY19. The project cited differences in data reporting protocols in the Western Cape province (West Coast, City of Cape Town, and Cape Winelands districts), which reported ART initiations with TB outcomes and not with case finding. This was a contributing factor in the underperformance of ART initiation among TB clients in these districts. Comparative Analysis of TBSAP Intervention Districts and non-TBSAP Districts The second approach used to determine the effectiveness of the TBSAP was comparing project results with results achieved in non-TBSAP districts based on 2014‒2017/2018 DHIS data. To do this, eight districts (four high TB burden, two medium burden, and two low burden) were selected to match the eight intervention TBSAP districts. The districts were paired on the basis of TB incidence (number of new district TB notifications). The evaluation team analyzed 15 indicators, comparing the performance of the intervention and comparison districts. This district-level analysis is available in Annex II. An overall analysis of the eight TBSAP districts compared with the eight comparison districts across the four outcome indicators reveals that the TBSAP performed slightly better in DR-TB treatment success and slightly worse in ART initiation for TB/HIV coinfected clients, with equivalent performance in DR￾TB initiation and DS-TB treatment success (Figure 4). 16 | Page “But the challenge in the district, it was death, our death rate for both TB and MDR are really giving us a challenge. TBSAP is helping us with our audits every time there is a death and we are very happy now as the numbers are coming down.” – District TB coordinator The district-level analysis yielded more nuanced performance differences in indicators across the cascade. For example, TBSAP districts had lower treatment failure rates than their comparison districts by 10 to 20 percent. TBSAP districts outperformed their comparison districts for the TB death rate (2016 cohort). For this indicator, the best performers were the TBSAP districts of eThekwini, City of Johannesburg, and West Coast, with death rates lower than their comparison districts by as much as 40 to 50 percent, evidence that the TBSAP interventions were yielding positive results. This finding was confirmed by the qualitative interviews. Respondents from the TBSAP and the DoH working in Sekhukhune and uMkhanyakude districts referred to the decline in TB deaths in their respective districts. Further evidence of progress made is the DR-TB treatment failure rate: five of the eight TBSAP districts out-performed their comparison districts. Performance for TB/HIV is not as clear cut; the comparison districts were on par (equivalent in performance) with the TBSAP districts. TBSAP districts were only better at screening HIV patients for TB and for initiating them on ART by a small margin (~ 5%). This speaks to the efficacy of their screening programs and the attention paid to finding and treating vulnerable populations, such as people living with HIV. Several unforeseen challenges and limitations in this exercise severely constrained the evaluation team’s ability to draw any substantive conclusions from the comparative analysis. DHIS data were available through 2017 only (and for a handful of data points in 2018); as such, the comparison assesses the initial year of TBSAP implementation only. It is therefore premature to expect outcome-level performance differences. In addition, the TBSAP was assigned some of the highest burden districts, with the highest caseloads and the most challenging contextual factors. This includes the major metropolitan areas of the City of Johannesburg, City of Tshwane, eThekwini, and Cape Town. As metros, these districts may have better-resourced facilities and systems, but they also have higher population density and client mobility. The comparison districts are all high burden districts, and they all had some level of partner support. Figure 4: Performance of 8 TBSAP and 8 Comparison Districts, 2017/18 17 | Page “The NDoH is happy with the contributions made by the partners to the TB program. TBSAP has been managing the whole case finding program and has contributed immensely. In IPC, TBSAP has made huge strides…the adherence support and management of adverse events has been critical.” – NDoH KI Figure 5: Materials Produced in Collaboration with the NDoH Given the additional constraints and challenges faced by conducting this evaluation during lockdown (in terms of travel restrictions, limited access to data, etc.), the comparison was based solely on the DHIS data for the period indicated (year 1 of TBSAP implementation) and did not control for other factors or predictors of performance, such as the relative strengths of the DoH, data management systems, etc. TBSAP Alignment with Key Global, Donor, and National Strategies The evaluation team assessed the TBSAP’s alignment with key global, USG (USAID and PEPFAR), and GOSA TB strategies and priorities, an indicator of project implementation compliance with TB evidence-based best practice. The TBSAP is aligned with and responsive to global and USG guidelines and strategies, including the WHO End TB Strategy, USG Global TB Strategy, and the White House National Action Plan to Combat Multidrug-Resistant TB. This evaluation reviewed the TBSAP’s strategies and technical action plans, and confirmed that they are aligned with and responsive to GOSA and NDoH strategies and other official documents, such as the National Strategic Plan for HIV, TB and STIs (2012-2016 and 2017-2022), and TB data at national and sub-national levels. The NTP has five interventions under the Find-Treat-Prevent Framework and two cross-cutting interventions, which were used to evaluate alignment. The following TBSAP interventions are the most pertinent contributions to the NTP and are aligned with its strategies, as described in the Find-Treat￾Prevent Framework: 1. Finding missing patients 2. Key populations 3. Urine test for the detection of the lipoarabinomannan antigen (u-LAM) diagnostics implementation 4. Enhanced case detection 5. DR-TB decentralization: one functional decentralized DR-TB site per sub-district The TBSAP piloted interventions demonstrated to be successful in other countries, testing them in the South African context to confirm their feasibility and effectiveness. The interventions that show great promise if taken to scale in the country have been extensively documented by the TBSAP. They are being integrated in national policy to institutionalize their practice and accredited for nationally recognized health worker training (Figure 5). The most effective TBSAP interventions are discussed in Question 2. Summary The TBSAP achieved its FY19 target for one of its four outcome indicators (DR-TB treatment initiation). The project was close to achieving its DR-TB treatment success target, but was significantly behind in its targets for DS-TB treatment success and ART coverage for TB/HIV coinfected clients. The TBSAP performed well in TB screening and testing, with variable performance in LTFU rates. The TBSAP is aligned with and responsive to global and USG guidelines and strategies, and to GOSA strategies and policies. Several TBSAP interventions are currently being scaled up, integrated, and institutionalized in national policy. 18 | Page “….The working relationships, I would say they are very good. The approach from the staff, you know, we don't feel like they are just intruders. We are a team; we work as a team. There is great consultation with regards to whatever work that needs to be done….They were here to assist us. We sat down with them…they have seen what we have identified [as gaps], we plan how are we going to improve our health care delivery, especially with regards to TB management.” – DoH KI Question 2. What key interventions and activities have been most effective/high impact in achieving results? (Which are least or less effective?) The TBSAP has implemented several interventions and activities that have had a positive effect on TB outcomes in the selected districts. Whereas the project description lists three objectives or intermediate result areas, TBSAP was designed and is implemented using a “whole systems” approach, whereby interconnected interventions are carried out in collaboration with partners to achieve system￾level changes through incremental steps. The evaluation team assessed the effectiveness of TBSAP interventions based on two factors: (1) performance indicators that could be linked to a specific activity; and (2) the frequency with which specific interventions were mentioned as project successes or strengths during the qualitative interviews. More Effective Interventions Based on these criteria, the three most effective interventions were identified as: (1) building partnerships for multisectoral accountability; (2) infection prevention and control (IPC) activities; and (3) continuous quality improvement (CQI). Building Partnerships for Multisectoral Accountability Activities under this intervention include: (1) working with national, provincial, and district DoHs to harmonize activities and assure the judicious and efficient use of resources; (2) provision of support to the Departments of Agriculture, Forestry and Fisheries, and to Social Development and Basic Education to identify opportunities to strengthen TB responses; (3) collaboration with PEPFAR District Support Partners (DSPs) to improve TB case finding, linkage to care, and treatment outcomes; (4) collaboration with private general practitioners (GPs) and traditional health practitioners (THPs); and (5) implementation of the non￾governmental organization (NGO) network model to deliver a comprehensive TB care package to the community. 13 The project framework for multisectoral accountability is available in Annex VI. 13 In this report, the term “NGO” serves as an umbrella for community-based organizations supported by the TBSAP. 19 | Page “You know, they follow the structures very well, and then the province will introduce them down to the district level, with meeting our directors. And then from there, they will be introduced into the facilities. You know why this approach was done? It’s not easy for the supporters if they just go straight to the facilities. We find people, they don't accept a person who just comes to their facility without – I mean, being introduced by the principal. But if they are introduced well, you find that their relationship with the facilities becomes very good. So, if you go through the right level, you find that your reception will be better.” – DoH KI Over 80 percent of KIs (70/82) mentioned that the TBSAP places emphasis on establishing and maintaining working relationships with the DoH at all levels, other government departments, development partners, and civil society organizations. When asked about successful interventions to build multisectoral accountability, 100 percent of the TBSAP respondents (31/31) and 91 percent of NDoH respondents (21/23) mentioned the TBSAP’s participation in established DoH TB program review structures at all levels, such as the TB Think Tank at the national level, and provincial- and district￾level donor coordination and TB program review meetings. This collaborative approach, along with a healthy respect for government protocols, have ensured buy-in and have been critical success factors for the project. NGO network model: KIs at district and facility levels were especially appreciative of the assistance provided by TBSAP￾supported NGOs: 78 percent of all KIs (64/82) and 79 percent of KIs working at the sub-national level in the DoH (15/19) agreed that the organizations provided significant support to communities and households.14 They contribute to efforts to find missing TB patients through improved contact management, case finding, and treatment adherence for patients across all districts supported. The model uses a targeted approach based on the identification of TB hotspots and high TB burden facilities to identify the communities that are most affected by TB. It then links them with a network of community health workers (CHWs) who are familiar with the communities and their needs. The identification of gaps and priorities in the community is done in consultation with and under the guidance of the DoH at the district level, with facilities referring patients likely to default from TB treatment and who may require support for treatment adherence to NGOs. The CHW develops a close relationship with the patient and offers a comprehensive patient-centered package of services that considers the patient’s level of risk, personal circumstances, and preferences. Home visits provide an opportunity to identify the need for social and financial protection and for education on TB infection control measures. The NGO network model has contributed meaningfully to TBSAP achievements. During FY18/19, through the efforts of 49 NGOs funded by TBSAP under the NGO network model, 810,454 people were reached and screened for TB, of which 5,962 were diagnosed with TB (yield=0.74%) and 5,543 (93%) were initiated on treatment. A small cohort analysis of DR-TB patients initiated on treatment from 2016‒2018 demonstrated significantly better treatment success rates and lower LTFU and death rates among patients supported by the NGOs compared with the standard of care, an indication of the potential of the community model when implemented well. 14 The notable exception was the City of Johannesburg, where, at the request of Gauteng province, TBSAP FAST activities are confined to hospitals because 18 other organizations support HIV/TB activities at primary health care (PHC) and community levels. 20 | Page “…our small grants project….is actually one of the big successes of the project. To the point where we actually requested through the [TB quality improvement, TBQI] steering committee to actually work with the ward-based outreach teams, which is the community￾based services structure at the DoH, to also try and get them at the level that our NGOs are working. And they're actually looking at us documenting some of the lessons learned in terms of how we're working with the patients, what M&E frameworks are being used to track these patients and can actually see them through to treatment completion.” – TBSAP KI Although it has significantly contributed to TB outcomes in TBSAP districts, the success of the NGO network model is dependent on consistent and continuous funding and ongoing support to the organizations. Inefficient management of the small grants program has hindered the NGOs’ sustained contributions, which is discussed further in Question 3. Nevertheless, the GOSA will be adapting the TBSAP package of support developed for the NGOs for scale-up through the Ward￾based Primary Health Care Outreach Team (WBPHCOT) program, with assistance from the TBSAP. Scale up of this program will continue through the end of the TBSAP and beyond. Reaching key and vulnerable populations: The private sector has a wide reach and potential for participation in TB care and prevention, especially in accessing hard-to-reach vulnerable populations. The TBSAP piloted an effective model with private medical practitioners, and with workplaces (such as farms and small mining companies) to offer their workers primary health care (PHC) services. This model demonstrated a valuable contribution to case finding. Infection Prevention and Control Activities The TBSAP IPC package is a comprehensive set of activities implemented at national, provincial, district, facility, and community levels. The activities include contributions to national IPC policies, tools, and guidelines; capacity-building and mentorship support for TB program managers and implementers at provincial and district levels; facility and household risk assessments; and district- and facility-level support for the implementation of the Finding TB Actively, Separating safely, Treating effectively (FAST) approach at selected hospitals. Project staff carry out all activities using a technical assistance approach to capacitate district-level DoH staff to deliver appropriate, quality TB services in an environment that complies with national infection control standards. All training is provided using NDoH policies, treatment algorithms, and guidelines. CHWs from project-funded NGOs (and/or WBPHCOTs) conduct home-based IPC risk assessments using tools developed by the project, and provide education to improve infection control measures in households and to reduce the transmission of TB. Nearly 80 percent of KIs (63/82) recognized FAST as a TBSAP flagship strategy. They included 87 percent (20/23) of DoH respondents, 87 percent (27/31) of TBSAP respondents, and 56 percent (15/27) of respondents from other support partners. Although the approach is centered on the fact that prompt diagnosis and effective treatment are the most important activities in the health care facility setting to prevent TB transmission to health workers and other susceptible patients, effective implementation spans activities across the entire TB care cascade. 15 The NDoH endorsed FAST as part of the TBSAP’s support in 2016, and the project was instrumental in packaging existing NDoH IPC policy and guidelines16 in an implementable program using the FAST approach. The project’s activities include orientation and training of health workers at various levels; the development of an implementation guide outlining basic information on the resources, processes, and procedures needed for the successful implementation of FAST; and a checklist to monitor compliance with IPC measures. The FAST approach 15 This includes early detection through active case finding, diagnosis, and follow-up of confirmatory laboratory results to prompt initiation on treatment. 16 These include the National Policy and Strategy for Infection Prevention and Control (2007) and National Infection Prevention and Control Guidelines for TB, MDR-TB and XDR-TB (2015). 21 | Page “…So I would say our greatest success has been the introduction and the implementation of FAST within the district. Which on its own, it's created awareness, TB awareness within the staff, because even when it comes to infection prevention and control practices, they got to improve quite a lot…” – TBSAP KI was first implemented in April 2017, with facilities added in a phased rollout, and the strategy has shown such good results that it has been adopted by hospitals outside the TBSAP-supported districts. As of September 2019, implementation of FAST had been scaled up to 142 health facilities in five provinces: Eastern Cape, Gauteng, Free State, KwaZulu-Natal, and Limpopo. Results from TBSAP-supported hospitals implementing FAST from 2017 to 2019 show a 33 percent increase in the in-hospital rate of screening for TB, from 131,112 patients screened in 2017 to 1,994,604 patients screened in 2019 (Table 4). The number of patients diagnosed with TB increased from 880 (839 DS-TB and 41 DR-TB) in 2017 to 5,840 (5,600 DS-TB and 240 DR-TB) in 2019, although the percentage of presumptive TB patients remained at 3 to 4 percent in 2018 and 2019, and the TB positivity rate remained stable at 13 to 14 percent. With this growth in case load, the DS-TB LTFU rate increased from one percent in 2017 to five percent in 2019, and from two percent in 2017 to five percent in 2019 for DR-TB. A review of district-level TBSAP hospital data for FAST shows interesting variations. For example, in the City of Johannesburg, FAST was scaled up from two district hospitals (South Rand and Bheki Mlangeni Hospitals) in 2017/18 to one regional facility, one specialist mother and child hospital, and three tertiary/academic hospitals (Chris Hani Baragwanath, Helen Joseph, and Charlotte Maxeke academic hospitals) in 2019/20. Although screening rates were similar to the national FAST hospital average in the City of Johannesburg, and testing rates were slightly higher (66%‒83%), the TB positivity rate increased incrementally from 15 percent in 2017/18 to 20 percent in 2018/19 to 32 percent in 2019/20. Facilities were added in a phased rollout of FAST (7 hospitals in year 1 [2017], 40 in year 2, and 72 in year 3). Table 4 presents the overall results across the care cascade for all hospitals at which the TBSAP provided direct support during 2017 to 2019. Table 4: Cascade Indicator Performance in Directly Supported FAST Hospitals 2017‒2019 Source: URC. (2019). TBSAP Annual Report 2019 22 | Page “FAST reduces the issues of cross￾infection in health care workers in facilities. And also, FAST reduces time gaps between diagnosis and initiation; at the very same time, it reduces infection from one person to another or from one person to a nurse in that way. So, FAST is not only for patients but also for health care workers.” – TBSAP KI These findings indicate that despite improved access to PHC services, many TB cases in South Africa are still diagnosed at hospitals. 17 The TBSAP established screening stations in every hospital area and ward to institutionalize the practice of universal TB screening, which facilitates more effective testing and increased yield— an overall better use of limited resources. The TBSAP also sought to increase initiation and to mitigate LTFU by starting patients on treatment before being discharged from the hospital and being referred to PHC level for management. Nearly 30 percent of KIs (23/82) were of the opinion that the introduction of the u-LAM test as part of the TBSAP’s FAST strategy to aid the prompt diagnosis of TB in selected patients has contributed to improved TB diagnosis in patients with disseminated TB, those coinfected with HIV who have a low CD4 count, and those unable to produce sputum. Using u-LAM, TB was detected in 38 percent of patients (925 of 2,460 patients tested). The NDoH is in the process of developing guidelines to expand the use of u-LAM from a doctor-driven test used at hospitals to include its use at selected high TB burden PHC facilities. TB infection among health workers has become a standard item reported in monthly facility IPC reports, which highlight them as a TB risk group. 18 Fifty percent of KIs (41/82), including 53 percent (10/19) of implementation-level DoH staff, perceived FAST and its emphasis on administrative control measures for IPC as providing protection for health workers. Continuous Quality Improvement The TBSAP supports the implementation of the national TB quality improvement (TBQI) approach by strengthening DoH management and service delivery capacity at several levels through the following activities: (1) support the national rollout of the TBQI approach through the secondment of a national TBQI advisor, representation on the national TBQI Steering Committee, and membership in the TB Think Tank; (2) provision of leadership training and technical assistance to provincial and district managers on CQI; (3) build the capacity of program implementers in DS-TB and DR-TB management, FAST, IPC, and quality assurance/quality improvement (QA/QI) using a combination of didactic training, coaching, and mentoring; (4) support the development of clinical and management training modules, tools, and standard operating procedures (SOPs) to guide baseline assessments, and ongoing monitoring and supervision of QI initiatives; (5) support the introduction of new TB treatment regimens, including pharmacovigilance to monitor serious adverse events; (6) support the introduction of new TB diagnostic techniques (such as u-LAM); and (7) support the decentralization of DR-TB services. Development of a structured process for establishing CQI teams. Under the umbrella of a formal CQI program, TBSAP programming at all levels focuses on planning and coordination. A clearly documented step-by-step process guide begins with the establishment of buy-in from HIV/AIDS, STI and TB (HAST) leadership/management structures at provincial and district levels, followed by structured learning sessions addressing specific topics, and then onsite mentoring support. Several examples of the success of this approach in extending the implementation of QI beyond the 14 TBSAP implementation districts were cited by 31 percent of TBSAP implementation-level respondents (5/16 TBSAP provincial- and 17 Clients continue to present at health facilities when they are very sick and need advanced care, partly due to stigma. 18 For example, a sub-district in Limpopo reported infections in two health professionals and a general worker in 2019. 23 | Page district-level respondents). For example, strong buy-in from the Member of the Executive Council for Health in Limpopo province was mentioned, which has ensured that the CQI approach has also been applied in all districts (including Capricorn, Vhembe, and Mopani, and at all 37 hospitals in the province). Another example is the Provincial Department of Health (PDoH) in Gauteng requesting the TBSAP to conduct workshops for managers from all five districts (including Ekurhuleni, Sedibeng, and West Rand). Cluster learning approach. Facilities are clustered by sub-district and are supported to conduct baseline assessments. Structured workshop learning sessions then assist clinical and program staff to deal with bottlenecks and issues along the TB management cascade (See Annex VI) that they identify at their own facilities. This makes learning relevant and promotes the development of local solutions. For those who do not participate in the structured sessions, a peer learning approach provides indirect benefits to districts and facilities that do not receive direct project support. For example, after managers from Yusuf Dadoo, Leratong, and Sedibeng hospitals in Gauteng attended the training workshops, these facilities also implemented and reported on FAST with “virtual” TBSAP support. Capacity building for health professionals: The TBSAP collaborated with provincial regional training centers (RTCs) to build the capacity of health care workers to implement and use the self-directed learning modules to strengthen management of the TB program and cascade the knowledge and skills to facility staff. The intervention targets high TB burden facilities with sub-optimal performance, identifies bottlenecks in service delivery, and facilitates the development of QI plans for improved TB management. In FY18/19, a total of 3,527 health workers at different levels of the health system, including program managers and coordinators, attended various training courses and other capacity￾building sessions. TBSAP also supervised 30 QI clusters in its supported districts. Among the nurse respondents in district-level management roles, 54 percent (6/11) confirmed that without training from the TBSAP, the TB program would not have taken off. A nurse managing the TB program at a Free State hospital, who was deployed to the facility without any TB knowledge, stated how crucial this capacity building was: “…I was thrown in the deep end and was trained and mentored from the beginning by the TBSAP staff…….on TB case management, computers, data management, quality control.” This nurse started the hospital’s FAST program, working closely with the facility’s medical doctors to acquire their buy-in and support, and improved TB case management. Based on the success of the train-the-trainer approach, TBSAP’s capacity building targets have increased by over 50 percent for FY20 to ensure increased QI coverage and impact. The TBSAP learning modules, which are accredited by the South African Medical Association, have been recorded in the databases of USAID’s TraiNet, the RTCs’ SkillMART, and the Western Cape’s Professional Development Course training database. Institutionalizing the QI methodology: TBSAP is supporting the drive to institutionalize the QI methodology in the NTP, seconding a QI Advisor at the national level, and two QI advisors at the district level (Nelson Mandela Bay and West Coast) in FY19. TBSAP has enabled provinces to commit to tracking key indicators along the TB care cascade to monitor progress. The project developed a QI SOP and is contributing to the national QI change package to share lessons learned, which can be implemented and scaled up nationally. Although the imperative to measure, assure, and improve the quality of TB services is now well recognized, systematic attempts to integrate QI in TB service delivery and in TB programs require additional resources to scale up and to sustain the current momentum in the districts. The TBSAP does not have additional resources in its districts for this purpose. Less Effective Interventions The less effective interventions were: (1) information system management; (2) advocacy, communications, and social mobilization (ACSM); and (3) the ConnecTB app. 24 | Page “We do support the districts in terms of data management, we conduct data verifications, we conduct data clean ups, the QAs, trying to improve the quality of data that is being reported by our district. But when you go back again to those facilities, things go back to square one.” – TBSAP KI “We have shortage of data capturers, yes. And also, mostly they are more focused with the HIV program than [the] TB program.” – PDoH professional nurse Information System Management The TBSAP sought to strengthen TB program data management (including recording and reporting) by supporting the transition of TB data from South Africa’s health information system for TB (ETR.net) to the country’s TB/HIV system (TIER.Net) as part of national integration efforts, and training staff to facilitate its implementation. However, progress was hindered by system errors and inadequate processes and human resources at the facility level. In an interview, an above-national donor described the TB data management system at the NDoH as a “crisis.” The donor cited the following example: recently the NTP had to ask the WHO for an extension on reporting relevant 2019/2020 TB statistics because the NTP was unable to produce verifiable and accurate data on time. The TBSAP worked to support the TB/HIV information systems integration process, which was initiated in 2015 and began in earnest in 2017. This involved transitioning the TB module (DS-TB patient data) from ETR.net to TIER.Net.19 The project facilitated the expansion of the TB module in TIER.Net to 1,031 of the 1,063 facilities in the five provinces that are implementing TIER.Net (97% coverage). 20 The TBSAP also trained 1,398 health workers (surpassing its target of 1,367) to ensure the smooth implementation of the new system. However, the transition from ETR.net to TIER.Net caused data and system errors, negatively affecting the quality and accuracy of TB data. Despite the TBSAP’s capacity-building efforts, respondents reported problems translating knowledge gained to implementation. TBSAP and DoH KIs reported that facilities are often short staffed and that data capturers are deployed to perform other administrative tasks or to capture data for other programs. In some cases, the TBSAP contracted additional staff resources on a temporary basis as a stop-gap measure; for example, the temporary appointment of data clerks to assist with administrative tasks, TB patient record filing, and capturing data backlogs. Such efforts provided temporary relief and assisted with locating TB patients LTFU, for example. However, the effects have been short-lived, because once the additional manpower is withdrawn at the end of the data capturers’ contracts, the filing situation and the management of data backlogs have recurred. Advocacy, Communications, and Social Mobilization ACSM was a contractual part of the TBSAP to promote public awareness of the TB epidemic and its impact on national productivity and individual lives. The activity was allocated nearly $2.5 million.21 It 19 Based on a WHO recommendation on systems integration and an independent assessment of systems in use, the NDoH decided that in￾facility TB data collection, using the TB module in TIER.Net, should be implemented in addition to the existing ART functionality. TIER.Net maintains a unique record for the patient at the facility level, and records patient data longitudinally. As such, all HIV, ART, and DS-TB data for the patient will be stored together, and reports can be drawn and actions can be taken to facilitate improved patient management. Source: NDoH TB/HIV Information Systems Support Portal: https://www.tbhivinfosys.org.za/#:~:text=The%20TB%2FHIV%20information%20systems,and%20is%20led%20by%20. 20 Western Cape is the only province that did not transition the TB module to TIER.Net, instead introducing an alternative facility-based health information system for reporting. 21 USAID. (2016). University Research CO. LLC Contract: AID-OOA-I-14--00035 (page 13, paragraph 2). 25 | Page “Because we have a lot of NGOs and it turned out that the system then could not handle that level of patient or that level of data set.... We had to come to a decision to say, you know what, as great as this is but it's currently not serving our needs as a project. So, for that one, we need to go back to the drawing board and fall back onto what we know and use, which are the paper based…. and it's currently still under revision where it's trying to be adapted to what we need to do, so that's able to serve its purpose….” – TBSAP KI involved the production of IPC and information, education, and communication (IEC) materials ($1.8 million), and the provision of targeted TB messages via media platforms ($661,000).22 For the first two years of implementation, TBSAP invested heavily in mass media campaigns on national radio, television, and billboards, and included one indicator for “Number of people reached” for these activities. However, the project did not include any outcome measures for these efforts and the considerable funds invested in them, notably in the costly mass media (TV, radio billboards) and social media platforms. In FY19, TBSAP developed and implemented a more targeted, district- and community-level strategy as a more cost-effective approach. Without the standard measures of reach, coverage, penetration and recall for ACSM activities, it is impossible to assess or analyze whether this significant investment in ACSM from the project’s resources had any effect, and more importantly, reached key populations with the right messages. In FY19, TBSAP integrated ACSM in its NGO network model, with the identification of TB hotspots and the use of the CHWs from its sub-grantees to conduct advocacy and communication activities in the communities. ACSM funding was then mostly devoted to the production of IPC and IEC materials (73% of the budget spent). TBSAP also scaled up its investment in ACSM training for districts, their coordinators, and local organizations, which is in line with the project’s mandate to build capacity at provincial, district, and local levels. A TBSAP senior executive explained the decision to transition the project’s approach, stating, “ACSM was deemed too expensive and unnecessary in the context of the total program and its funding [was] diverted to pay for training materials.” The project’s course correction to a more targeted approach paid better dividends in terms of TB program results: in FY19, 324 community campaigns screened a total of 121,500 people for TB, with 1,850 referred for testing and 376 new TB cases diagnosed and initiated on treatment. TBSAP training of provincial and district DoH staff resulted in the development of 12 district-level ACSM implementation plans, which focus on addressing specific TB program challenges, such as low screening rates among men and high rates of LTFU. Nonetheless, this correction was done over halfway into implementation, and the decision made primarily based on budget constraints rather than the strategic value of its investments. Given the importance of ACSM activities for successful TB programs, this represents a missed opportunity to better target ACSM efforts and invest funds more effectively. ConnecTB App The ConnecTB App is an online platform and mobile application developed by URC under TB CARE II South Africa, the predecessor project to the TBSAP. ConnecTB is intended to support the provision of treatment adherence support by CHWs. App functions include geo-location of patient adherence visits, the ability to record patient information to ensure treatment validation, prompting for side effects and contact management, and real-time monitoring and reporting. The ConnecTB application showed promise during its pilot phase, especially as a tool to improve adherence among DR-TB patients, with reported adherence rates of more than 95 percent and a LTFU 22 The budget also included $17,000 to develop an IPC strategy. 26 | Page rate of one percent. 23 In 2018, the application was scaled up to all project-supported NGOs. The ConnecTB platform was also piloted among the WBPHCOTs, and the household visit tools aligned to include TB indicators. However, in late 2019, the TBSAP reported that during scale-up, the project encountered technical problems with the implementation of the ConnecTB application that compromised the integrity of the data and implementation was halted. An internal review recommended a change of service provider, and the new platform is undergoing testing and review. This was confirmed by 58 percent (18/31) of TBSAP respondents. Summary The TBSAP’s efforts to build partnerships for multisectoral accountability Building Partnerships for Multisectoral Accountability with various sectors and across several levels generated critical goodwill needed for implementation of the project. This foundation has enabled the project to support its districts to work towards the objectives and targets set by USAID and the NTP. This has been largely achieved through a whole systems approach aimed at promoting CQI, efficient TB screening using the FAST approach, and improving IPC practices. Contracting NGOs to provide services in the communities has considerably improved the TBSAP’s effectiveness in finding missing cases and linking them to care, despite the funding interruption challenges (discussed in Question 3). Less effective interventions are related to information system management, inadequate strategic planning and impact measures of the project’s ACSM component and use of funds, and the ConnecTB application. While the health information system is an issue largely outside TBSAP control (and discussed further in Question 3), the project identified issues with the remaining two interventions and either adjusted its strategy (ACSM) or halted implementation for an assessment to determine future direction (ConnecTB application). 23 URC. (2017). USAID TBSAP Annual Report 2017. 27 | Page “The contract must allow for flexibility. Some issues may not be listed in the contract. Rigidity is a shortcoming in contract management by USAID. There should be an allowance for addressing emerging issues.” – NDoH KI Question 3. What challenges and constraints does the project face and what course corrections might improve this? The TBSAP encountered challenges and constraints that have hindered implementation and slowed the progress that the project could have made. Some challenges are systemic issues experienced at various levels of the health system in implementing the NTP. As such, they are impediments to achieving success in the nation’s TB response and lie outside the control of the TBSAP. Other constraints relate exclusively to the TBSAP, its implementation, and the progress it will likely show by the end of its contract. TBSAP Design and Implementation Challenges and Constraints The technical assistance model employed by the TBSAP has been well received by all districts supported. However, challenges related to project design and management have inhibited the project and may have unintentionally prevented it from reaching scale and delivering fully on its targets. The challenges became constraints over the course of the project. They include: (1) TBSAP contract and management; (2) sub-grantee management; and (3) human resource allocation during project planning. TBSAP Contract and Management Funding mechanism: The TBSAP functions under a contract between URC and USAID. Contracts imply higher expectations on the part of the donor for the implementing partner and are less flexible than other mechanisms, such as cooperative agreements or grants. This may contribute to unintended consequences; for example, because the TBSAP’s contract set a limit on the level of effort (LoE) or number of person-days that the project can use, URC project planning specified the number of personnel to be placed in the supported districts, which was later determined to be inadequate to provide the necessary level of mentorship. Performance framework: Although the TBSAP has an extensive PMP, the focus on results achieved toward the four very top-level outcome indicators versus performance against other project indicators across the TB cascade is not well balanced. A larger indicator set would enable implementing organizations to account for the full spectrum of their activities, such as multisectoral collaboration with other partners working in the same districts and performing similar functions. The latter consideration is important to avoid duplication of effort and for reasons of economy. Leadership and management changes: The TBSAP experienced high turnover of key staff, including three Chiefs of Party, with other high-level positions unfilled for long periods of time. There were also project management issues related to high-level decision-making, such as hiring by the “home office” and not by the local executive team. The project also experienced frequent management transitions at USAID, with three Contract Officer’s Representatives since inception. Stronger, more consistent management of the contract and project by both USAID and TBSAP might have led to the earlier identification of issues and course corrections; for example, shifting the ACSM budget to more cost-effective activities earlier, or working with the project to adjust the district LoE allocation to better suit the project’s needs. 28 | Page “TBSAP is also bring new ideas. They provide hands-on support. The use of “smaller NGOs” which are largely community-based may be a noble concept but will require that the NGOs are provided with the critical technical, financial and other support to ensure they remain viable and are then able to deliver the services.” – NDoH KI “And I think TBSAP is more technical, they are not mostly on the ground. They would give us trainings, which is good, but the mentoring side of it, you can be mentored for the first week and what happens for the remaining period of the year? So, I think if they could be more hands on. Secondly the person who is mentoring us, there is only one person for the whole district. We can only see her once in two months because we have got a lot of facilities around here in [district name]. So, the first thing that can be done, appointing of other people to assist.” – Facility professional nurse “I would very much want a TBSAP coordinator in every sub-district. We could then triple the successes we are having, particularly with DR-TB in vulnerable populations.” – PDoH KI Sub-grantee Management The primary goal of the TBSAP’s NGO network model is to provide TB services to people at the community level. The project also acts as a capacity-building mechanism for local organizations, helping them build their technical, financial, and management systems for long-term sustainability. However, there were challenges bringing this intervention to scale and securing continuous funding for the NGOs. One-year grants, lengthy granting processes, and inadequate grant administration resulted in delays in grant periods and a halt in critical relationships, community-level work, and gains made at the community level in screening, testing, and patient tracing (addressing and mitigating LTFU). This situation may have been exacerbated by the leadership and management changes at both USAID and TBSAP. All DoH provincial managers (6/6) and TBSAP coordinators (16/16) raised this as a major disruptive issue in the TBSAP districts in which the NGO network model was deployed. For example, the TBSAP reported that DR-TB patient treatment outcomes were noticeably better among those supported by NGOs in the Eastern Cape. Treatment outcomes for facilities supported by one of these organizations in FY17 included a 90 percent treatment success rate among supported patients (compared with 53% among unsupported patients), five percent LTFU among supported patients (compared with 15% among unsupported patients), and three percent death rate (compared with 9% in supported patients). Project Planning: Human Resource Allocation LoE limitations: TBSAP realized during implementation that its project planning did not fully take certain factors into account. In particular, the KIs agreed that the project’s staff allocation limited their ability to reach a wider population in the districts. The TBSAP established a large national office, involving 90 percent of the overall project LoE, providing each of the 14 districts with a single coordinator. The districts were allocated only 10 percent of the overall TBSAP contract man-days (LoE), based on the assumption that “because the project was providing technical assistance only, district coordinators would only have to train facility staff” (URC executive). In practice, in addition to their work with district and facility staff, district coordinators are involved in data cleaning and management, and training local organizations and CHWs on community outreach. Geographical context: In its initial planning, the TBSAP focused only on the number of training sessions it would have to conduct per district to implement its programs. It did not consider the distances to be traveled between facilities and the volume of preparatory work to be done. Several of the 14 TBSAP districts are rural and are characterized by poor road infrastructure, with long travel times between facilities. For example, Sarah Baartman district in the Eastern Cape is one of the TBSAP’s most sparsely populated districts, at just under nine people per square kilometer. The district spans over 58,000 square kilometers, and the district’s 82 health facilities 29 | Page “…The challenge is that we are having system issues; number 1 is the system issues… the electronic system, either the DHIS or TIER, but…also the lost to follow-up both [for] ART and TB. Again, we are accusing the systems because we have got the linkage officers, they are doing their job but there is something that is [does] not tally when you compare what is being done versus the data that is coming from the system.” – District DoH KI “And in one of the engagements, it came out clear that even the data capturers at facility level, they would prioritise their time only on the HIV program…. They would …deactivate the TB patients… So when they run the reports [from Tier.Net], it would only give them people who missed appointments for HIV, and those that missed appointments for TB would not appear on the list….that would be shared with their tracer teams,…., for them to go and trace this patient…and try to reduce loss to follow up rate.” – District implementing partner “You need to listen to me, what are my challenges? And then I will tell you my challenges is 1, 2, 3, 4 and then I need your intervention here. You cannot come and tell me that you want to intervene…they were supposed to consult me on what are my needs. I would have told them that I need the community caregivers….and I know which areas that are heavily burdened in this particular area, so that they can go and track, because I would be looking at the data at the district level, where do we have a high loss to follow up in both TB and HIV.” – District TB manager “…. what are your treatment gaps and how can we work together to resolve them.” – TBSAP district coordinator are situated up to 175 kilometers apart. This limits the number of supervisory visits that district coordinators can make and the number of facilities they can serve. In the Western Cape, one of the TBSAP’s district coordinators spends 26 percent of her work month driving to and from facilities. The KIs in Limpopo, KwaZulu-Natal, Eastern Cape, and Western Cape raised this constraint as deserving attention and resolution. Health System Challenges and Constraints The TBSAP also experienced several constraints related to South Africa’s health systems, outside the project’s control, which posed considerable challenges to its success. They are: (1) information system management; (2) funding and resource constraints; (3) ongoing capacity development needs; and (4) geographical and socioeconomic factors. Information System Management The challenges associated with TB information systems was one of the greatest system-level barriers. TBSAP’s ability to collect high￾quality data to monitor the TB cascade and use these data as evidence for implementation was curtailed by system errors related to the transition of the TB module to TIER.Net, and inadequate data management processes and human resources at the facility level. Although the GOSA recognizes the high TB burden and existing gaps in TB programming, and expresses the critical need for TBSAP assistance in this area, conflicting priorities with HIV and the relative number of implementing partners, funding levels, technical assistance, and key indicators result in less time and ability to focus on the full TB cascade and associated data management. Despite efforts to integrate services, 68 percent (56/82) of respondents, including 63 percent (35/56) of respondents in all six provinces, reported that HIV and TB data management has not been fully integrated in practice, with a higher priority given to HIV data management (such as capturing). Funding and Resource Constraints All provincial and district managers and district staff (19/19) agreed that there are common barriers to delivering satisfactory health services, including a lack of material and human resources, problems with communications and information sharing on TB programming, and limited guidance on current TB policy. A lack of health system funding was a familiar refrain among all provincial staff interviewed (19/19). For example, the inability of health facilities to repair and supply ultraviolet lights, sputum booths, and CO2 monitors. All provinces (6/6) pointed out how 30 | Page “…we realized challenges around the implementation of algorithm guidelines, TB trainings for staff…the biggest problems were with staff members were not in permanent posts. Most of the time, staff members who were working on the TB program were just temporary…they would move the person and bring in somebody else. So, there was instability for the TB program.” – District DoH KI the scarcity of staff (due to limited funding) was creating enormous setbacks for the TB program. In an attempt to remove these barriers, all respondents representing implementing partners (16/16) and all NGOs (5/5) interviewed agreed that the TBSAP and its peers need to agree on priority areas to be addressed in TB programming with the district health authorities where they provide technical assistance. In this way, it becomes easier to address the barriers and to fit in and adapt to district realities. More recently, the pressing priority of the COVID-19 pandemic necessitated the reallocation of funding, materials, and human resources from the NTP. This is addressed in Question 4. Ongoing Capacity Development Needs The limited number of TBSAP staff allocated at subnational levels, coupled with high DoH staff turnover and poorly functioning systems (such as those for data management), impede efforts to embed recommended (and policy-based) TB management practices. This finding is especially the case at the facility level, where an effective dose, frequency, duration, and coverage of support visits are required for new practices and interventions to be institutionalized. Of note, districts continue to prefer technical assistance because it offers a route to up-skill their staff in a sustainable fashion; direct service delivery is viewed as “taking away jobs.” It helped the TBSAP that several of their coordinators are ex-district HAST coordinators who know how to fit in. At the same time, a sufficient absorptive capacity of the recipient organization is key to successful capacity-building. TBSAP respondents and the district coordinators alike lamented the need to constantly train “new” staff members—instead of the planned focus on mentorship and supportive supervision— because of the instability of the staffing situation at PHC facilities. Some provinces (e.g., Limpopo) have a staff rotation policy whereby clinical staff rotate through different clinical services every six months, which is compounded by staff shortages and high staff turnover. In some districts, such human resource-related challenges are aggravated by poor systems, weak infrastructure, and the lack of leadership. TBSAP staff reported on a surprising finding in some districts, in which the old injectable regimens were still used in TB care. This was ascribed to substandard communication pathways and weak internal knowledge sharing (training) strategies in the districts. New guidelines are not optimally disseminated to health care workers, who might be unaware that use of evidence to manage TB implementation was inadequate and that changes in TB care were required. Geographical and Socioeconomic Factors Every province and its districts presented a unique environment for TBSAP implementation. Although all project districts have a high TB burden, they comprised both urban and rural communities. Some operate near mining areas, others in farming communities, or in densely populated inner cities. As previously noted, many present geographical challenges, covering vast distances and with poor road infrastructure. The major metros may have better-resourced facilities and systems, but they also have higher population density and client mobility. These factors complicate service delivery and TB outcomes. For example, the provincial staff in the Free State and Limpopo districts raised concerns about the high LTFU rates due to migrant populations from neighboring countries, such as Lesotho, which present difficulties with treatment adherence and follow-up (client tracking and tracing). Similarly, 31 | Page informants in Gauteng noted that the City of Johannesburg and City of Tshwane have large migrant populations that are “hidden” and are often unable to access TB care. Course Corrections: Data Management TBSAP has less than one year before the project ends, in March 2021. Because data management is one of the most significant constraints, the TBSAP has the opportunity to contribute meaningfully to the current situation by providing direct assistance to clear the data backlogs and help place the recordkeeping systems on a sound footing. This approach, advocated by an above-national donor despite acknowledgement of the TBSAP’s technical assistance model, would need to be explored in close collaboration with USAID and with relevant levels of the DoH.24 See the Recommendations section for additional information. Summary Many of the constraints experienced by the TBSAP are related to project planning and design. For example, despite previous experience in the provinces and districts, the allocation of LoE at the national level versus the district level meant that the project was not fully able to scale up and embed recommended interventions in full-fledged sustainable programs in the districts and then, through the NTP, in the rest of the country. All six PDoHs confirm this lack of scale up in their districts. Health system constraints, such as the information management system, funding and resource limitations, ongoing capacity development needs, and geographical and socioeconomic factors, also hindered project performance. Although the TBSAP worked to address and adapt to these issues, for example, supporting facilities to transition to TIER.Net for TB data and training users, these are national-level factors that will likely continue to affect the success of the NTP. Such project challenges and constraints—both those related to TBSAP itself and those associated with South Africa’s health systems—indicate that much still remains to be done to make the standard practice of TBSAP interventions a reality in the care and treatment cascade throughout the country. 24 This effort is underway; the TBSAP advertised for 67 data clerks across supported districts in May 2020. http://www.ngopulse.org/opportunity/2020/05/21/urc-data-clerks 32 | Page Question 4. What has changed in the epidemiology and environment of TB in South Africa, and how has this affected the program? What has changed in TB epidemiology in South Africa that has affected the program? TB Prevalence Survey and National TB Datasets The results of South Africa’s latest National TB Prevalence Study, conducted by the NDoH with support from its partners, were scheduled for release on World TB Day in March 2020, but this was delayed by the COVID-19 crisis. Once available, key findings from the National TB Prevalence Survey, along with other relevant studies (such as the TB Think Tank’s “Targeting Universal Testing for TB” report) will provide new data and a strong evidence base for insights to inform future TB programming. Pending the availability of more recent (2018‒19) national TB datasets, coupled with the delays in the National TB Prevalence Survey and other key studies, the evidence base available during the timeframe of this evaluation was too limited to draw any substantive conclusions about shifts in TB epidemiology. TB Program Policy Environment The primary changes in the TB program policy environment are focused on DR-TB: (1) the strong emphasis on the decentralization of DR-TB services; (2) the introduction of nurse-initiated DR-TB treatment; and (3) the introduction of new short-course all-oral DR-TB treatment regimens. South Africa began scaling up its 2011decentralization strategy to treat DR-TB in district hospitals closer to patients’ homes in 2017. The strategic implementation of QI measures in these district hospitals resulted in better teamwork, patient outcomes, and staff morale, and reduced barriers to DR-TB care provision, especially in rural settings.25 In June 2018, South Africa became the first country to replace injectable drug therapy for DR-TB with a short-course regimen (nine months) of bedaquiline tablets. Initial evidence (mainly from South Africa) on treating DR-TB with bedaquiline indicates both improved clinical outcomes in people living with DR-TB and increased safety/reduced toxicity. 26 South Africa is introducing 3HP (once-weekly isoniazid￾rifapentine for 12 weeks) as a new TB preventative therapy in line with the WHO’s recommendation for scale-up as part of the End TB Strategy. 3HP is a shorter regimen, less toxic, and with similar efficacy to isoniazid preventative therapy. It is also safe in combination with efavirenz-based ART regimens. 3HP will be available at select health facilities from July 2020, with scale-up to 23 districts supported by the Global Fund, PEPFAR, and Operation Phuthuma in 2021. 27 The NDoH Joint TB, HIV, PMTCT, STI and Hepatitis Program Review in 2019 identified achievements and good practices for both DS-TB and DR-TB, and challenges hindering progress. The NDoH issued updated policy and clinical reference guidelines in 2019, aligned with WHO’s consolidated guidelines on DR-TB treatment, maintaining the focus on DR-TB services as a priority. 25 NDoH. (2017). Summary Report on Decentralized and Deinstitutionalized Management of Drug Resistant Tuberculosis Services in South Africa. 26 Sikhumbuzo Hlabangane. (2018, June 21). SA First to Roll Out Revolutionary Treatment for Drug-resistant TB. eHealth News. https://ehealthnews.co.za/sa-first-roll-revolutionary-treatment-drug-resistant-tb/. 27 The Aurum Institute. (2020, March 12). New, shorter treatment to prevent TB to be rolled out in five high-burden TB countries. https://www.impaact4tb.org/press-release-wtd-2020/ 33 | Page “Truly speaking, what is called program, we are not looking at it since March. All of us, be it Mother and Child…we are not talking of the project, we are talking of COVID now. As I am busy now, all the managers, all the districts, we are allocated duties for COVID-19. I am supporting…a COVID site, on COVID, daily reporting, daily coordination, nothing that talks of the program.” “We were given tablets and cellphones for the TB program, but these were taken away for COVID, and we do not know what is happening in our TB program.” – Provincial TB managers What has changed in the TB environment and situation in South Africa that has affected the program? Changes in the TB Environment: the Impact of COVID-19 The first case of COVID-19 in South Africa was confirmed on March 5, 2020. In the last six months, South Africa’s total number of confirmed COVID-19 cases has increased significantly. By late August 2020, over 600,000 cases had been confirmed. The GOSA implemented one of the world’s strictest national lockdowns on March 27, 2020, encompassing a vast range of restrictions to limit transmission while simultaneously urgently and strategically preparing health systems to cope with the potential COVID-19 case load and adapt existing service delivery. The NDoH’s emergency preparedness and planning began in February 2020 and relied heavily on the country’s strong TB/HIV epidemic control systems and partners to rapidly shift, incorporating and integrating COVID-19 in existing case finding, contact tracing, and case and data management systems. The NTP was tasked with the overall management and coordination of these efforts, given the similarities in the two airborne, respiratory infections and the network of existing isolation wards for DR-TB. Although this decision was strategically sound in the short-term and South Africa deserves much credit for its ability to rapidly adapt its systems and respond to this pandemic, it has had a significant and detrimental impact on the NTP since March 2020, from service delivery to patient management and data management, with potentially severe longer-term implications for TB outcomes. Of note: • NTP oversight and management and its TB programming and activities have been effectively halted due to competing priorities related to the COVID-19 emergency response. • The diversion of TB resources in response to COVID-19 has been considerable; for example, TB wards in hospitals have been converted to COVID-19 wards, and CHWs and data capturers have been deployed to conduct COVID-19 screening services, leaving TB services relatively understaffed. • Drop in TB case finding: South Africa’s National Health Laboratory Services (NHLS) confirmed that the level 5 lockdown restrictions (during the period March 27, 2020 to April 30, 2020) resulted in an approximately 48 percent average weekly decline in the number of TB tests using GeneXpert, and that the number of TB positive cases detected declined by 33 percent. 28 • Resulting declines across the TB cascade: National and international civil society organizations have cautioned about the potential negative impact that the redeployment of resources, coupled with social distancing measures, may have on the TB program in terms of TB case findings, treatment adherence, and stigma. These findings were confirmed by respondents representing five of the six provinces. 28 National Institute for Communicable Diseases. (2020). Impact of COVID-19 intervention on TB testing in South Africa. https://www.nicd.ac.za/wp-content/uploads/2020/05/Impact-of-Covid-19-interventions-on-TB-testing-in-South-Africa-10-May-2020.pdf. 34 | Page In South Africa, the COVID-19 pandemic is expected to result in a drop of 11,000 new HIV patients from TB entry points and a 10 percent decline in the completion of TB preventive therapy in PEPFAR’s TB/HIV program in quarter 3 of FY20 (July‒September 2020) alone, a significant decrease in the achievement of targets (Figure 6).29 These unintended consequences are likely to have an increasingly negative impact on previous efforts to control and reduce the TB burden, which remains the leading cause of disease death in South Africa, killing an average of 5,000 people per month.30 Figure 6: Impact of COVID-19 on the PEPFAR TB/HIV Program in South Africa Interventions to Mitigate the Impact of COVID-19 on TB in South Africa To mitigate the negative impact of the COVID-19 pandemic on TB services, WHO recommends that in addition to the restoration of routine services, there is a need for “catch-up” measures to focus on reducing the pool of undetected and unreported TB cases that have been increasing during the periods of lockdown. Such measures include intensive ACSM and TB awareness raising, intensified active case finding, and contact tracing activities. The NTP, with support from the national TB Think Tank, is in the process of developing a TB Recovery Plan, which will soon be launched in conjunction with the TB Prevalence Survey mentioned above. The TB Recovery Plan encompasses: 1. Intensified TB case finding, including universal GeneXpert testing for people living with HIV at ART initiation sites, and TB self-screening. 2. TB case holding, including scale-up of digital health technologies (NDoH mHealth tool). 3. TB/HIV and data management, including exploring out-of-facility TB/HIV integration models (e.g., routine TB screening among clients in differentiated care) and institutionalizing weekly verification of TB and TB/HIV data. 29 USAID/South Africa. (2020). Impact of COVID-19 on TB Control. 30 WHO. (2019). WHO Global TB Report 2019. 35 | Page “Now with the COVID-19 pandemic…in the hospitals, there are tents, where each and every patient or each and every person who comes to the hospital, they are screened for COVID. So, this also includes screening for TB, because those patients that are symptomatic, when they are screened at the tents, they will fall in the orange zone. So those are also your TB patients, your TB presumptives [sic]. So, which then, we really need to make sure that we follow them up, because now everybody will be now talking COVID or thinking COVID, now we forget about these other ones.” “We trained healthcare workers, those that would be conducting household screenings and it did work well because from the training, they went out to conduct trainings to the district. So working with the district for me as a project, we were dealing with the district on what needs to be done in terms of implementing the COVID-19 training and screening.” – TBSAP provincial managers TBSAP Interventions during the COVID-19 Crisis Because COVID-19 can also cause severe illness in immunocompromised people and primarily attacks the respiratory system, it is imperative that health services for all co-morbidities, and especially TB, are not compromised. Locating and linking diagnosed TB patients to quality care is now more critical than ever to lower the risk of these patients succumbing to COVID-19. To ensure the continuous quality of TB management services, TBSAP is working closely with the NDoH toward an integrated approach to service delivery. The project has also been actively involved in supporting the DoH rapid response teams to conduct IPC assessments at the facility level. These assessments were informed by the project’s best practices and lessons learned from implementing IPC measures at health facilities across the country. Despite restrictions in movement and the high risk of infection, TBSAP staff have been active in providing technical assistance to support both TB and COVID-19 efforts. TBSAP initiatives include: • Integrated COVID-19 and TB screenings in communities: TBSAP and its local organizations conducted community screening campaigns in collaboration with the DoH, which helped to actively identify COVID-19 patients at the household level. • Capacity building for COVID-19: Partnering with PEPFAR-funded DSPs, TBSAP supported capacity building activities nationwide, including training nearly 800 health care workers on integrated case finding initiatives. • Strengthening IPC measures at health facilities: TBSAP supported 425 facilities across six provinces on IPC activities, including FAST, IPC implementation and monitoring, and IPC risk assessments. • Informing the public about COVID-19: TBSAP took an active role in knowledge and information sharing about COVID-19, leveraging the opportunity to promote TB messaging to audiences across a range of platforms, including local radio stations and on social media. 31 Summary Although key data sources on TB epidemiology in South Africa (such as the latest National TB Prevalence Survey and 2018 and 2019 TB data sets) are pending public release, TB program policy changes indicate the integration and uptake of advances in DR-TB treatment and decentralized service delivery. However, the COVID-19 pandemic has adversely affected the NTP in terms of management, resource allocation (including human resources), and service delivery, with projections of poorer TB outcomes across the cascade. The TBSAP has contributed to the national COVID-19 response, such as screening and education efforts, and leveraged its existing IPC and capacity building measures to strengthen the health system at the facility level. 31 TBSAP. (2020). USAID TB South Africa Project Interventions during the COVID-19 crisis. www.urc￾chs.com/sites/default/files/Integrating%20TB%20COVID-19%20interventions%20in%20South%20Africa%20May%202020.pdf 36 | Page Conclusions At a global level and in South Africa, fewer than 50 percent of people with TB are successfully treated.32 The biggest gaps are in diagnosis and getting people on treatment. Although global advances have been made, progress has been slow and sub-optimal. The number of people developing TB is not declining at a pace that would place countries on the path to end TB by 2030. Within this context, the TBSAP evaluation team has reached the following conclusions: 1. The TBSAP has laid a sound platform of interventions and contributed meaningfully to the NTP The TBSAP has laid a sound platform of interventions and protocols in the districts in which it works. The project has produced credible results toward its four outcome indicators and across the TB cascade, with demonstrated value of interventions to improve screening, care, and treatment. The “whole systems” approach implemented by the TBSAP has made it possible to identify leverage points at all levels of government to locate appropriate interventions in the TB cascade and package them in a comprehensive basket of technical assistance services. Despite the challenges discussed throughout this report, the TBSAP’s implementation has provided valuable lessons and tangible results that form a solid foundation for sustainable future efforts against the TB epidemic in the country. If the interventions are taken to scale in the project’s districts, and if they are implemented in the rest of the country by the NTP, they have the potential to reduce TB incidence and mortality in South Africa to levels set out in the WHO End TB Strategy: reduce the number of TB deaths by 95 percent by 2030; and cut new cases by 90 percent between 2015 and 2035 with a focus on reaching key and vulnerable populations. 2. The most effective components of TBSAP’s approach identified are: • Relationship building: The evaluation team documented that the TBSAP is skilled and well versed in forming and maintaining relationships by working within the structures of district management instead of imposing its own agenda, and by communicating regularly, clearly, and openly with provinces, districts, facilities, project donors, and the NTP. • Mentoring and capacity development: District KIs unanimously agreed that the TBSAP sought to foster district-level ownership of the interventions, largely because of the training provided by the project. This is especially true in terms of community engagement via community institutions, whereby the project also invested in training and mentorship of NGOs supporting TB control. • Measures to reach key and vulnerable populations: The TBSAP has demonstrated the inherent value of its NGO network model in reaching, testing, and inducting vulnerable populations into care and treatment. The TBSAP’s public-private partnerships, including an effective model with private medical practitioners and workplace PHC services for such populations as farm workers and miners, demonstrated particular value for case finding. The evaluation also identified TBSAP’s less effective interventions, which include: 32 World Health Organization (WHO). (2019). Global tuberculosis report 2019. Geneva, Switzerland: WHO. https://apps.who.int/iris/handle/10665/329368. 37 | Page • Information system management: Although the TBSAP made significant efforts to strengthen TB program data management and support the transition of TB data from ETR.net to TIER.net, the challenges to the health information system are an issue that is larger than the project itself and outside the TBSAP’s control. • Advocacy, communications, and social mobilization): Inadequate strategic planning and the lack of appropriate impact measures for the project’s ACSM component was a missed opportunity to identify and implement a more cost-effective approach. • The ConnecTB app: The ConnecTB application showed great progress and promise but was halted during roll-out due to technological problems. Both the ACSM and ConnecTB components were missed opportunities to improve project results. The project adjusted its ACSM strategy and initiated an assessment to determine the future direction for the ConnecTB app. However, both course corrections occurred over halfway through project implementation. 3. The TBSAP was constrained by design and planning deficiencies and health system challenges Although the value of TBSAP approaches has been demonstrated, the project has been unable to scale them up to cover all facilities in project-supported districts. The TBSAP team has had to deliver technical assistance on complex TB interventions in South Africa’s high-burden environment, shaped by global policy, guided by the NTP guidelines, and delivered in varied district settings. Notable challenges affecting the TBSAP include: • Project planning and design: Structural deficiencies dating from the initial project planning stages, such as project LoE allocated to districts, led to constrained performance, especially in terms of mentoring and institutionalizing its interventions at district and facility levels. There was unanimous agreement among all district-level KIs from both DoH and the TBSAP (35/35) that the TBSAP had fallen short in terms of mentoring DoH staff to implement what they had been taught due to a lack of adequate TBSAP staff in the districts. URC is experienced in working in South Africa’s provinces, districts, and facilities, and is familiar with the district health system challenges. Although baseline assessments were carried out for each district, they could have better contextualized their response to the operating environment in each district. • NGO network model funding: The manner of contracting NGOs (through a one-year agreement), requires resolution. At the end of the year, the loss of the NGOs’ contributions and the consequent cost to the project is reflected in higher LTFU rates. • Funding mechanism and performance expectations: The TBSAP functions under a contract between URC and USAID. Contracts are typically less flexible than other mechanisms, which constrained the TBSAP’s capacity to adapt; for example, to adjust the LoE allocated for district-level support. The four project outcome indicators set as the metrics for success do not well reflect or measure the breadth and scope of the work implemented by the TBSAP. However, these indicators are what the TBSAP is held accountable for, that is, the obligation to justify its actions and expenditures in pursuit of performance. The contract does not hold the project accountable, for example, for the number and types of vulnerable populations reached, the success of its efforts to incorporate the private sector in case finding, and its success assisting the NDoH to resolve data quality issues in districts and at facilities. NTP data management and systems are the most significant challenge adversely affecting effective TB programming in South Africa’s health system. One KI described data management as the “lifeblood 38 | Page of effective TB treatment.” Another described the current difficulties with data management in the NTP as “a crisis.” Data collection, data analysis, and data use are essential to implementing an effective and efficient TB control system. It is essential for districts, provinces, and the NTP to be able to report timely, accurate, and complete data to all levels of the system, and to use the data locally to strengthen management of both TB patients and TB control activities. The data management issues, which span all levels of the health system—from inadequate information systems to insufficient processes and staff for facility-level data capturing and reporting—urgently need to be addressed and resolved. 4. The COVID-19 pandemic has the potential to derail NTP gains and exacerbate the country’s TB epidemic The COVID-19 pandemic poses a serious and severe threat to the NTP in South Africa. TBSAP’s support to the national response leveraged the similarities between COVID-19 and TB (in terms of transmission and, thus, prevention), strengthening IPC measures and conducting TB screening in conjunction with COVID-19 screening and awareness. However, as a largely facility-based program, the NTP will need to implement significant adaptations and deepen the integration between the two diseases to maximize finite material and human resources. 39 | Page Recommendations Short-term course corrections and recommendations for USAID and TBSAP through the end of the contract (March 2021): • Intensified, direct support to the NTP in data management and systems: Data management and systems present the most significant constraint to the NTP. They are further exacerbated by the impact of COVID-19. The provision of direct, hands-on assistance at facility and district-level DoHs in data clean-up and record-keeping are of critical importance, and are in line with WHO “catch-up” measures. o The TBSAP’s appointment of data capturers should be context specific to the needs of each district and facility. o To be effective, data capturers will need:  Adequate training and supervision to ensure the quality of data capture at all levels.  Access to hardware (computers, printers), software, and internet connections.  Access to physical space for desks, data teams, files, and filing cabinets with support and buy-in from the DoH and facility staff to work efficiently.  USAID and TBSAP should apply lessons learned from experiences with previous and on-going HIV data mop-up and systems shifts exercises. Experience with HIV partners shows that in-person, hands-on, skilled support yields significantly better results. • Fast-track approval and continuity of NGO contracts: The TBSAP should ensure the continuity of NGO contracts to support efforts toward integrated COVID-19 and TB screening, intensified case finding, contact tracing, and reduced LTFU in communities, also in line with WHO “catch￾up” measures. • Leverage the project’s strengths to help the NTP adapt to the COVID-19 pandemic: The TBSAP should further strengthen facility-level IPC measures and health worker capacity. The project should also identify and pilot program adaptations pivoted toward differentiated models of care and digital/remote services across the TB cascade. This includes self-screening, alternatives for medication access (such as the Central Chronic Medicine Dispensing and Distribution program and using NGOs for home or community medicine delivery), and virtual treatment adherence support (such as SMS reminders and “talking” pillboxes). • End of project documentation and transition plans for district and provincial DoH: The TBSAP has documented its work extensively. It has a complete record of its outputs, which should be widely shared with all stakeholders as valuable lessons learned and “how-to” guides, and made easily accessible via USAID, the NDoH and PDoH websites, URC’s website, and globally as journal publications. Medium-term recommendations to USAID to ensure consistency of support in the TBSAP’s 14 high TB burden districts: • Ensure continuity of support: Provide a cost or no-cost extension to the current TBSAP contract, from April 2021 through the anticipated start date and initiation of USAID’s follow-on TB project(s). In coordination with TBSAP and the DoH, USAID’s contract extension should ensure coverage of high-impact, priority interventions most critical to sustain gains in TB outcomes made over the last five years. Key interventions and activities in the medium (and long) term should align with the NTP and TB Think Tank “TB Recovery Plan,” and, to the extent possible, allow for flexibility and programmatic shifts in response to the rapidly changing TB and COVID￾19 landscape. 40 | Page Long-term recommendations for USAID in the design of its follow-on TB project: • Funding mechanism: A cooperative agreement mechanism would allow the follow-on TB project more flexibility in the use of funds, course corrections, and timely responses to new challenges, notably in the context of COVID-19. The challenges of resource-poor health systems often require additional support from development partners to refashion programs and to build more functional ways to effect change. A cooperative agreement is a more appropriate mechanism for technical assistance than a contract in this context. • Appropriate measures of accountability: A comprehensive indicator set should be identified that spans the TB follow-on project(s)’ intermediate results and interventions. M&E should be integrated in every intervention, such as ACSM, with clear and measurable outcomes from these activities. Data management processes and expectations should account for South Africa’s ongoing context and limitations in this regard for project management (using data to inform programming) and reporting. USAID should also identify opportunities to address and support this health system constraint; for example, by providing short-term, hands-on support for facility-level data management or roving district-level data quality assurance teams to facilitate quarterly data reporting. • Incorporate and build on successful approaches from the TBSAP. The systematic screening of high￾risk groups through outreach and the use of NGOs has led to the prompt initiation of treatment for DS-TB and DR-TB clients. The FAST approach and CQI enhancements have substantively contributed to improvements in outcomes in the districts. The follow-on project(s) should build on these achievements. Moreover, the capacity development and mentoring approach should be applied at multiple levels of the DoH to circumvent challenges related to staff turnover, rotation, and reallocation toward institutionalizing CQI. • Build on and expand public-private engagement and partnerships. The private sector has a wide reach and potential for valuable participation in the management of TB. Future follow-on project(s)’ sub-grantees (such as NGOs and other partners) should be specifically tasked with collaboration with the private sector in an effort to find missing cases. This will: (1) improve screening and TB case management; (2) harmonize TB programming and build National Health Insurance (NHI) networks; (3) reduce TB in key populations; and (4) improve occupational health services for TB (especially in smaller mines) by building on lessons from the HIV program. Specific TBSAP public-private partnerships identified for scale-up include: o The private medical practitioner project in the Eastern Cape, linking it to the NHI. o The work with THPs in Limpopo and KwaZulu-Natal. o The “TB in Farms” intervention, especially in such provinces as the Eastern Cape and KwaZulu-Natal. o Reaching miners as a population vulnerable to TB through their workplace, expanding especially to Free State and Limpopo. • Ensure continuity of NGO contributions: To optimize and sustain TB control contributions from the NGO network model, the follow-on TB project(s) should issue contracts of the same duration as that of the project, or alternatively, have strong contracting and administrative processes to prevent service gaps. • Strengthen TB/HIV/COVID-19 integration and coordination for improved program outcomes. The follow-on TB project(s) should adopt successful program adaptations from the TBSAP as it pilots alternative approaches relevant to the COVID-19 response. USAID should assess the hardware (such as computers) and internet connectivity needs to facilitate data management and remote/online learning and support, and incorporate a related investment component in the follow-on project(s). Pivoting to remote learning using existing platforms, such as the NDoH’s Knowledge Hub for professional development, which incorporate offline functionality, may reach more health workers while allowing them to learn at their own pace and saving on in-person 41 | Page training costs. At a donor (USAID) and USG interagency level, USAID and the Centers for Disease Control and Prevention should increase coordination for program planning and resource allocation across activities (projects), including built-in mandates and platforms for collaboration. 42 | Page Recommendations for the NDoH and NTP: • Address the country’s information management system challenges. Resolving the information management system issues is crucial to the national effort to reduce TB incidence and mortality. It will also contribute to improved case management, treatment initiation, and adherence support; enhance digital patient tracing, tracking, and monitoring systems developed for the COVID-19 response; and reduce data backlogs for TB and HIV programs. Specific actions include: o Strengthen data recording and reporting through continued training and mentorship of M&E personnel at district and facility levels. o Oversee completion of the transition of TB data from ETR.net to TIER.Net, ensuring the provision of adequate training and clear guidance to all stakeholders on access. o Support the development of the Unique Patient Identifiers to avoid data duplication and to connect lab data to individual patient care. o Strengthen laboratory information management systems by improving the use of laboratory data for case finding and linkage to care (providing the results directly to the patient), and expanding the Rif Alert to all districts. • Oversee an assessment of and manage health system software applications. The NDoH should conduct a “best return on investment” analysis of mobile applications, including ConnecTB, toward streamlining and aligning existing technology. Applications should be device, software, and platform-agnostic (able to interact with all file types or able to exchange data with any other software). Software should run on any type of device, such as desktops, tablets, and smartphones. This will minimize the frequent over-investment in hardware and software to run the apps. • Strengthen NTP coordination and ACSM. Given the relatively small number of TB implementing partners, the NDoH should facilitate a technical working group to align interventions and messaging and enhance coordination and collaboration to better leverage stakeholders’ respective strengths and use of limited resources. For example, this technical working group should develop a centralized ACSM strategy for the NTP to which all implementing partners contribute (financially and technically) and implement. This strategy should incorporate marketing principles, such as specific audience targeting, and the most relevant platforms for each audience and in the context of COVID-19 (such as social media). 43 | Page ANNEXES Annex I: Evaluation Statement of Work SCOPE OF WORK Monitoring, Evaluation, Research, and Learning Activity Prime Contract or Subcontract Number: IDIQ No. AID-OAA-I-15-00025, Task Order No. 72067418F00002 Title of SOW: USAID/SOUTH AFRICA BILATERAL HEALTH OFFICE TB SOUTH AFRICA PROJECT MIDTERM EVALUATION 44 | Page I. PURPOSE OF THE EVALUATION Purpose USAID/South Africa seeks to carry out a midterm evaluation of its Tuberculosis (TB) South Africa Project (TBSAP), to examine how the project’s interventions have helped achieve the National Department of Health (NDoH) and Mission’s goal to reduce TB burden in South Africa and to identify areas for improvement. The objectives of this evaluation are three-fold: 1) To determine the extent to which the TBSAP responds to the Government of South Africa’s TB priorities (National Strategic Plan 2017-22 and National TB Program Strategic Plan 2017-2021) 2) To identify key lessons learned, best practices and challenges which will inform the strategic design of the follow-on TB project 3) To learn from midterm program successes and challenges in the TBSAP program and recommend course corrections for the second half The TBSAP midterm evaluation will consolidate learnings, collaboration and adaptation across the project, identifying key factors, interventions and investments that contribute to the successes and challenges of the project in reducing the TB burden (mortality and morbidity) across South Africa. This evaluation will also identify and document opportunities for future improvement, including possible approaches that were not part of the current activities. Audiences There are three distinct audiences who should receive and make use of these evaluation findings. The first includes USAID and PEPFAR, to inform their current and future programming and implementing partners currently working in TB. The second is the Government of South Africa stakeholders at central and provincial/district levels, notably those working at the NDoH and in the TB/HIV program. Third are other implementing partners and nongovernmental stakeholders (including the TB steering committee, other donors, and private sector partners engaged in TB prevention and care). Intended Uses This evaluation seeks to inform current and future TB interventions in South Africa, especially given the high burden of disease, and its disproportionate contribution to morbidity and mortality in South Africa. USAID will use the TBSAP midterm evaluation findings to understand better the most effective approaches to inform the second half of this project and future programming with the continued goal of reducing the burden of TB across South Africa. The evaluation will also inform NDoH and other GOSA stakeholders about activity achievements, successes, failures, and opportunities going forward. The midterm evaluation will also document results achieved against national and global targets, promising practices and lessons learned from the project in a way that can be easily consumed by key stakeholders across South Africa. II. SUMMARY INFORMATION In keeping with USAID’s global TB strategy, USAID/South Africa is committed to supporting the Government of South Africa’s efforts to prevent and control TB, addressing local needs and priorities as well as fostering sustainability of the National Department of Health’s TB approaches. USAID is also keen to ensure investments are coordinated well with the NDoH and other donors (such as the Global Fund, World Bank, BMGF) to increase their impact and sustainability. USAID identifies and invests in programs and policies that have the greatest potential to end the epidemic. USAID/South Africa works in partnership with the NDoH and its partners to identify and provide high- 45 | Page quality, patient-centered care and treatment to individuals with active TB and those most at risk of developing active TB and MDR and XDR-TB. USAID works closely with the NDoH to develop, implement, and track the success of comprehensive, sustainable national TB strategies and programs. This support includes helping GoSA to identify and address gaps in their TB programs and adopt new, evidence-based, context-specific and locally led practices and policies. Consistent with WHO’s End TB Strategy and the national TB strategy, USAID works to: ● Improve access to high-quality, patient-centered TB, DR-TB, and TB/HIV services ● Prevent TB transmission and disease progression ● Strengthen TB service delivery platforms ● Accelerate research and innovation ● Improve international capacity and collaboration to combat multi-drug resistant tuberculosis Toward this end, USAID/South Africa’s TBSAP project is summarized below: Activity Objective, Scope, Reach TBSAP 03/17/2016 – 03/16/2021 $64.8 million IP: URC The TB South Africa Project is a five-year USAID-funded activity implemented by URC and partners. TBSAP “provides technical assistance to the Government of South Africa in order to reduce the burden of tuberculosis and multi-drug-resistant tuberculosis (MDR￾TB) in the country. The primary objectives of the activity are to: 1) reduce TB infections; 2) increase the sustainability of effective TB response systems; and 3) improve care and treatment of vulnerable populations.” The project is implemented in 14 districts across 6 provinces. III. BACKGROUND a. Description of the Problem and Context South Africa is one of the highest-ranking countries in the world in terms of TB burden according to WHO estimates, with a high incidence rate of 567 new infections per 100,000 persons in 2017. South Africa’s TB challenge is compounded by high numbers of multidrug-resistant TB (MDR-TB) and extensively drug-resistant TB (XDR-TB) cases. WHO further estimated that 14,000 of the roughly 322,000 incident cases are MDR or rifampicin resistant. Likewise, approximately 193,000 (60%) of the incident cases are estimated to be coinfected with HIV, further exacerbating challenges in providing treatment and care to patients. Guided by and consistent with global strategies and policies (WHO End TB, USG Global TB Strategy, White House National Action Plan for TB), South Africa’s TB program has made progress in sustaining trends in reducing mortality due to TB; lowering TB incidence rates, including reducing the incidence of co-diagnosed TB/HIV; and improving treatment success rates for most categories of cases. WHO confirms a clear and sustained downward trend in TB case notifications in South Africa, which WHO postulates might be driven by high ART coverage rates. However, challenges remain before South Africa can reach its TB elimination goals. A key gap is the estimated 101,837 incident cases that were not notified or not diagnosed in 2017. The country is falling short of 2025 End TB goals. Treatment coverage was 68% in 2017 and treatment success was 82% (among cases who initiated treatment in 2004) versus a 2025 target of 90%. In 2017 South Africa reported that of the 15,986 laboratory-confirmed MDR/rifampicin resistant cases, only 10,259 (64%) began treatment. Furthermore, the treatment success rate in 2014 of 54 per cent for patients with multidrug-resistant TB (MDR-TB) remains low. 46 | Page Of the 123,148 TB patients who are known to be HIV-positive, 109,799 (89%) of these patients were reported to be on antiretroviral therapy for HIV.33 Additionally, 53% of HIV positive people newly enrolled in HIV care are on TB preventive treatment. South Africa’s TB efforts are largely funded by domestic sources (91%), international (8%), and unfunded (less than 1%).34 b. Description of the Interventions to be Evaluated and Theories of Change The overarching project goal of TBSAP is to reduce the national TB, M/XDR-TB, and TB/HIV burden in South Africa. This activity contributes to achieving USAID Mission’s Development Objective 1: ‘Health outcomes for South Africans improved’, as expressed in the Country Development Cooperation Strategy (CDCS). This activity also responds to the National Strategic Plan for HIV, TB and STIs (2017- 2022); National TB Program Strategic Plan (2017 – 2021); USG Global TB strategy (2015-2019); and the White House National Action Plan to combat Multidrug Resistant TB (Goal 2) - improving international capacity and collaboration to combat multidrug-resistant tuberculosis (MDR-TB). The primary objectives of the activity are to: 1) reduce TB infections; 2) increase the sustainability of effective TB response systems; and 3) improve care and treatment of vulnerable populations.” The specific IRs and sub-IRs are described below. IR1: TB Infections Reduced IR 1.1 Increased public awareness of the TB epidemic IR 1.2 Effective implementation of infection prevention and control IR 1.3 Improved TB screening, including among key populations IR2: Sustainability of Effective TB Response Systems Increased IR 2.1 Strengthened management capacity at all levels IR 2.2 Strengthened service delivery capacity at all levels IR 2.3 Improved data reporting and recording at all levels IR 3: Care and Treatment of Vulnerable Populations improved IR 3.1 Increased contact tracing of key populations IR 3.2 Improved TB case management among key populations IR 3.3 Strengthened comprehensive systems and partnerships for care 33 https://www.tbfacts.org/tb-statistics-south-africa/ 34 “Global Tuberculosis Report, 2018”, World Health Organization, 2018. 47 | Page IV. EVALUATION QUESTIONS This evaluation seeks to answer the following key evaluation questions. Sub-lines of inquiry for each question are included in Annex II. 1. To what extent has TBSAP been effective/successful in achieving its stated objectives and targets? 2. What key interventions and activities have been most effective/high impact in achieving results? (Which are least or less effective?) 3. What challenges and constraints does the project face and what course corrections might improve this? 4. What has changed in the epidemiology and environment of TB in South Africa, and how has this affected the program? Recommendations will stem from key findings and conclusions within the report and will include at a minimum: recommended continuation and/or scale up of priority activities as well as course corrections for the second half of the TBSAP project, and recommendations for the design of the follow-on flagship TB program. V. EVALUATION DESIGN AND METHODOLOGY The evaluation team will propose a comprehensive design for addressing the evaluation questions and will address how choices will be made related to selecting country/geographic focus and intervention activities/results, to maximize the ability of the evaluation to develop findings based on evidence and that will be relevant to the broadest possible range of activities. Within this design, the evaluation team will propose the best and most rigorous methods for data collection appropriate to address the evaluation questions. This will include a mix of secondary data and desk review, and site visits to collect qualitative (key informant interviews, small group and focus group discussions) and quantitative (community, partner and/or facility-based surveys, electronic surveys) primary data collection. The evaluation team will conduct Team Preparation Meetings to discuss further detailed of data collection and analysis methods. Annex III provides an illustrative “Illustrative Analytical Framework for the Evaluation” matrix, including a range of potential data collection methods that may 48 | Page be suitable for each evaluation question. This matrix will be further developed by the evaluation team in the Evaluation Design Proposal. Existing Information Sources The following is an illustrative, non-exhaustive list of materials that will be available for review as part of this evaluation’s secondary documentation and data review: • Annual, quarterly and monthly reports from the TBSAP project • USAID’s RFP/RFA, Cooperative Agreement/contract, and modifications for the TBSAP project • USAID/South Africa CDCS and current mission TB indicators • M&E plans, PMPs, work plans, and data and narratives reported against these, including indicator data tables covering each year of implementation • Research, innovations, best practices, and other documents, papers, highlights and reports published/disseminated by the TBSAP • Baseline, mid-project and other research initiatives (from TBSAP and other TB programs in SA relevant to this project) • The final evaluation of the former “TB programme for South Africa” • Global guidelines and strategies, namely the WHO End TB Strategy, USG Global TB Strategy, White House National Action Plan for TB • Government of South Africa and NDoH strategies and other official documents, such as the National TB Strategic Plan (both 2012-2016 and 2017-2022) and TB data at the national and sub￾national level Data Analysis Methods Data analysis methods to be proposed by the evaluation team will follow closely from the methods used to collect each type of data needed to answer the evaluation questions. Whatever data analysis methods are chosen for this evaluation, they should be justified in terms of their fit with the data collected for each question and the types of answers that USAID seeks. Time and cost considerations are also important in this area. The evaluation team’s proposed design for this evaluation should consider the most appropriate sequence for answering each question, based on the data that will be collected and analyzed for each question. Gender Considerations In accordance with USAID policy (Automated Directive System (ADS) 201 point 7), the research design for this evaluation will consider gender-specific and differential effects of the TBSAP project. The evaluation team will explore gender aspects of the activities per the questions and data sources available but will not seek to aggregate or integrate data across those activities. USAID Participation Regular communication between the evaluation team and the designated USAID Activity Manager will be essential to the successful execution of the TBSAP midterm evaluation. The evaluation team will keep the MERL office and USAID apprised of changes and developments that necessitate any significant decision-making or modification of the approved evaluation design. Possible USAID participation in the data collection tool development phase of the evaluation will be discussed in the meetings, prior to the start of fieldwork. Weekly check-ins from the Team Lead or MERL Evaluation Support Consultant will ensure activities remain of quality and on course. As per USAID policy, the inception report/design proposal, draft report, final report and all PowerPoint presentations will receive formal approval prior to the next phase of the evaluation. VI. DELIVERABLES AND REPORTING REQUIREMENTS The evaluation team will be responsible for the following deliverables. Specific due dates will be 49 | Page proposed in the final version of this scope of work and included in the evaluation team’s Evaluation Design: a. In-briefing b. Inception Report i. Evaluation Work Plan ii. Evaluation Design c. Mid-term Briefing d. Final Exit Briefing e. Draft Evaluation Report f. Final Evaluation Report g. Submission of Datasets to the Development Data Library h. Submission of Final Evaluation Report to the Development Experience Clearinghouse For a detailed description of USAID requirements for each deliverable listed here, please refer to the USAID Evaluation SOW Template (https://usaidlearninglab.org/library/evaluation-statement-work￾template). Deliverable Estimated Due Date 1. In-briefing with USAID, IPs 1. Within first week of in-country time 2. Evaluation Design Draft Report, including the evaluation framework, detailed research methodology, drafts of data collection tools and instruments, sampling plan, matrix for key indicators, report template, work plan and schedule. 2. Two weeks after completing the in￾brief in Pretoria and associated meetings with USAID and PEPFAR. 3. Final Evaluation Design Report, with workplan. 3. One week after receiving USAID and PEPFAR comments on Design Proposal. 4. Mid-term briefing with USAID 4. Informal mid-term briefing (by phone, possibly with a short summary of data collection activities and key findings to date in writing) conducted midway through fieldwork (estimated end of week 2 out of 3 weeks of primary data collection) 5. Draft Evaluation Report w/ draft set of infographics highlighting TBSAP results and key findings. 5. Initial Draft Evaluation Report within two weeks after completion of fieldwork. 6. Exit-briefing: Oral presentation(s) of key findings, etc. on draft findings and recommendation to USAID – 2 versions – 1 for internal USAID consumption and 1 for external use. 7. PowerPoint presentation on how 6. Within two weeks after completion of fieldwork. 7. Together with oral presentation to USAID and PEPFAR. 50 | Page Deliverable Estimated Due Date USAID programmatic and administrative processes could be improved for future TB projects and activities (e.g., financial investment, management, leadership support, and technical intervention design. 8. Oral presentation(s) of key findings, etc. on evaluation findings and recommendation to IPs and NDoH. 8. Following presentation to USAID and PEPFAR, and their approval of external presentation materials. Timing TBD based on IP and NDoH availability. 9. Final Evaluation Report w/ final set of infographics. 9. Two weeks after receiving last USAID comments on the Draft Evaluation Report and associated materials. Note: there may be more than one revision before the Final Report is completed. 10. Submission of Datasets to the Development Data Library; Submission of Final Evaluation Report to the Development Experience Clearinghouse 10. Two-four weeks after receipt of USAID approval of the final evaluation report Throughout the Evaluation have bi-weekly check-in meetings with USAID either in-person or virtually. A full calendar will be shared with USAID within one-week of team starting. All documents and reports will be provided electronically to USAID no later than the dates indicated in the approved Evaluation Design Proposal. All qualitative and quantitative data will be provided in electronic format to USAID in a format consistent with Automated Directives System (ADS) 579 requirements. All debriefs will include a formal presentation with slides delivered both electronically and in hard copy for all attendees. The format of the evaluation report should follow USAID guidelines set forth in the USAID Evaluation Report Template (http://usaidlearninglab.org/library/evaluation-report-template) and the How-To Note on Preparing Evaluation Reports (http://usaidlearninglab.org/library/how-note-preparing-evaluation￾reports). Evaluation team members will be provided with USAID’s mandatory statement of the evaluation standards they are expected to meet (see Annex IV). Dissemination actions are expected to include some or all of the following items, per agreement with USAID: • Executive Summary, summarizing the TBSAP project, evaluation methodology, and main findings and recommendations. • Developing a final version of a power point presentation for USAID internal use, summarizing the TBSAP, the evaluation methodology, and evaluation findings, and recommendations. • Developing a final version of a power point presentation for external use, summarizing the TBSAP project, the evaluation methodology, and evaluation findings, and recommendations. • Working in close collaboration with USAID, organize a dissemination event where the final external (non-procurement sensitive) power point and other methods, e.g. a panel discussion, 51 | Page are presented and used to disseminate findings and recommendations to a broad range of stakeholders, including the Government of South Africa, USAID and PEPFAR, implementing partners, TBSAP sub-steering committee, NGO's and civil society organizations working in TB/HIV, academia, private sector organizations, and others are represented. VII. EVALUATION TEAM COMPOSITION The evaluation team will comprise of three people, all of whom are South African nationals with considerable technical expertise and experience in TB: • Team Lead • TB Expert • Evaluation Specialist The Team Lead is ultimately responsible for the overall management of the evaluation team and the completion of all deliverables in accordance with the deliverable schedule. In addition, the Team Lead is responsible for coordinating evaluation activities and ensuring the production and completion of an evaluation report in conformance with this scope of work, USAID evaluation policy and timelines. The Team Lead must also work with the team to guarantee technical analysis quality, writing quality, and seamless report integration. (see annex V for detailed Position Descriptions for each position above.) In addition, the MERL office will provide one Evaluation Support Consultant, Jennifer Peters, to provide support in the recruitment and lead up to this evaluation, as well as in the initial team planning, communication and coordination, and final analysis and presentation stages of this work. Selection criteria for team members include the following: • Strong knowledge of South Africa, both the health sector and TB/HIV programming • Expert knowledge of TB, with strong understanding of global strategy, priorities and challenges in implementation • Knowledge of USAID programming practices, experience working with USAID preferred • For the team lead, minimum seven years’ experience monitoring and evaluating TB programs and prior successful experience in leading teams • For the TB Expert, demonstrated expertise and experience in TB programming as well as the M&E components of TB, MDRTB and TB HIV • Knowledge of qualitative and quantitative evaluation practices. • One specialist with a background in clinical trials preferred. • Across the team, a mix of expertise in the following areas preferred: Programmatic Management of Drug-Resistant TB (PMDT), private sector engagement, TB/HIV, ASCM, laboratory and diagnostics, and policy planning. • Excellent writing and inter-personal communication skills. VIII. EVALUATION SCHEDULE With the aim of having key findings and recommendations available by the end of March/early April 2020, and in time to inform the development of the next TB project (and as part of the larger TB portfolio design), this evaluation will take place over an estimated 5 months period, from December 2019 – April 2020. In-country team planning, data collection, analysis and preliminary findings debrief will comprise the bulk of the work, and is anticipated to occur over an 8 week period from January – March 2020. 52 | Page Date Task Nov - Dec 2019 Team recruited and employment agreements finalized. 4 – 20 Dec 2019 Literature review documents collected and distributed; Desk study begins. Key stakeholders and informants identified. In-briefs with USAID & IP, and other initial meetings and TPMs held. 6 – 14 Jan 2020 DRAFT Evaluation Design & Inception report, including timeline, deliverable deadlines and data collection tools and instruments submitted to Panagora. 20 Jan 2020 Deliverable: DRAFT Inception Report submitted to USAID. 20 – 31 Jan 2020 Site visits and schedules set and confirmed in coordination with TBSAP and NDoH while USAID reviews draft Inception Report. 27 Jan 2020 Inception Report presentation & review with USAID for final submission. 30 Jan 2020 FINAL Deliverable: Inception Report. 3 – 21 Feb 2020 Upon Inception Report approval, Team mobilizes to field, for site visits, key informant interviews, and other primary data collection. 24 – 29 Feb 2020 Preliminary data analysis conducted to identify key findings and conclusions 2 – 6 Mar 2020 Draft report and preliminary PPP briefing prepared for USAID Program Office, submitted and reviewed by Panagora. 10 Mar 2020 Deliverable: Draft report submitted and PPP on preliminary findings presented to USAID Program Office. 10 – 16 Mar 2020 USAID internal technical review completed; comments sent to Panagora and the evaluation team. 16 – 20 Mar 2020 USAID comments incorporated and edits to draft done for final report review by Panagora, prior to submission to USAID. 25 Mar 2020 Deliverable: Final Evaluation Report submitted to USAID Estimated LoE in days by activity for each of the 3 TBSAP team members Task LoE for Team Lead LoE for TB Expert LoE for Evaluation Specialist Total LoE in days Desk review/work planning and evaluation design drafting 5 5 3 13 In-brief, meetings, inception report and evaluation design draft 6 6 6 18 Data collection – field days 18 18 18 54 Data analysis 6 6 6 18 53 | Page Task LoE for Team Lead LoE for TB Expert LoE for Evaluation Specialist Total LoE in days De-Briefing, report draft 5 5 5 15 Edits, Final report 5 4 2 11 Totals 45 44 40 129 IX. FINAL REPORT FORMAT a. Abstract b. Executive summary c. Evaluation purpose d. Background on the context and the strategies/projects/activities being evaluated e. Evaluation questions f. Methodology g. Limitations to the evaluation h. Findings, conclusions, and (if applicable) recommendations i. Annexes The evaluation abstract of no more than 250 words should describe what was evaluated, evaluation questions, methods, and key findings or conclusions. The executive summary should be 2–5 pages and summarize the purpose, background of the project being evaluated, main evaluation questions, methods, findings, and conclusions (plus recommendations and lessons learned, if applicable). The evaluation methodology shall be explained in the report in detail. Limitations to the evaluation shall be disclosed in the report, with particular attention to the limitations associated with the evaluation methods (e.g., in sampling; data availability; measurement; analysis; any potential bias such as sampling/selection, measurement, interviewer, response, etc.) and their implications for conclusions drawn from the evaluation findings. Annexes to the report must include: • Evaluation SOW (updated, not the original, if there were any modifications) • Evaluation methods • All data collection and analysis tools used in conducting the evaluation, such as questionnaires, checklists, and discussion guides • All sources of information or data, identified and listed • Statements of difference regarding significant unresolved differences of opinion by funders, implementers, and/or members of the evaluation team, if applicable • Signed disclosure of conflict of interest forms for all evaluation team members, either attesting to a lack of or describing existing conflicts of interest • Summary information about evaluation team members, including qualifications, experience, and role on the team X. CRITERIA TO ENSURE THE QUALITY OF THE EVALUATION REPORT Per ADS 201maa, Criteria to Ensure the Quality of the Evaluation Report, draft and final evaluation reports will be evaluated against the following criteria to ensure quality. 54 | Page • Evaluation reports should represent a thoughtful, well-researched, and well-organized effort to objectively evaluate the strategy, project, or activity; • Evaluation reports should be readily understood and should identify key points clearly, distinctly, and succinctly; • The Executive Summary should present a concise and accurate statement of the most critical elements of the report; • Evaluation reports should adequately address all evaluation questions included in the SOW, or the evaluation questions subsequently revised and documented in consultation and agreement with USAID; • Evaluation methodology should be explained in detail and sources of information or data properly identified; • Limitations to the evaluation should be disclosed in the report, with particular attention to the limitations associated with the evaluation methodology (selection bias, recall bias, unobservable differences between comparator groups, etc.); • Evaluation findings should be presented as analyzed facts, evidence, and data and not based on anecdotes, hearsay, or simply the compilation of people’s opinions; • Conclusions should be specific, concise, and include an assessment of quality and strength of evidence to support them supported by strong quantitative and/or qualitative evidence; • If evaluation findings assess person-level outcomes or impact, they should also be separately assessed for both males and females; and • If recommendations are included, they should be supported by a specific set of findings and should be action-oriented, practical, and specific. See ADS 201mah, USAID Evaluation Report Requirements and the Evaluation Report Checklist and Review Template from the Evaluation Toolkit for additional guidance. LIST OF ANNEXES I. TBSAP PMP, Indicators II. Position Descriptions for the Evaluation Team 55 | Page Annex I. TBSAP PMP, Indicators 56 | Page 57 | Page 58 | Page 59 | Page Annex II. Position Descriptions Company Profile: Panagora is a women-owned, small business based in the Washington, DC metro area providing novel and integrated solutions in health and development. Panagora is dedicated to providing high-quality, high￾impact international health, development and learning consulting focused on market-based and integrated solutions that build country capacity and independence; and on learning, communications, and technology solutions to accelerate and heighten impact. Project Summary: The Monitoring, Evaluation, Research, and Learning (MERL) Activity supports USAID/South Africa’s bilateral Health Office by working collaboratively with health office staff to achieve their overall goals by creatively finding innovative solutions, supplementing and complementing their existing staff capacity, and as appropriate enhancing capacity in performance management; evaluation and research; and systematic collaboration, continuous learning, and adaptive management. MERL will support USAID/South Africa’s Bilateral Health Office in carrying out a midterm evaluation of its TB South Africa Project (TBSAP) for the past 3 years (March 2016 – March 2019). This midterm evaluation will examine how these activities collectively have helped achieve the mission’s goal of reduced TB morbidity and mortality and identify areas for improvement. Team Lead Position Summary: MERL is seeking a team lead to oversee design, implementation, and reporting activities for the TBSAP midterm evaluation in close collaboration with the MERL team. The team lead will have extensive experience leading evaluation teams and designing and implementing mixed-methods evaluations for USAID or similar international donor agencies. The team lead will also have experience in TB/HIV, ideally in South Africa. Responsibilities: The team lead will be responsible for the following: ● Oversee desk research and provide inputs to inform required evaluation design documents. ● Lead the implementation of all evaluation activities based on the approved evaluation design proposal. ● Supervise evaluation team members in carrying out required design, data collection, and analysis tasks. ● Ensure that evaluation processes and products meet the requirements and quality standards of Panagora/MERL and USAID. ● Carry out and oversee research and analysis based on the approved evaluation design and methodology. This may include reviewing and analyzing secondary documents, conducting key informant interviews in person in South Africa and remotely, facilitating group discussions in South Africa, supervising data collection activities of other team members in multiple areas of South Africa where the TBSAP activities are being implemented, analyzing implementation data, and other approaches to collect and analyze data, including ensuring necessary mechanisms for data quality assurance. ● Lead an internal findings/conclusions/recommendations workshop with the MERL team prior to drafting the evaluation report. ● Lead the quantitative data analysis for this evaluation. 60 | Page ● Serve as lead author for drafting and finalizing the draft and final evaluation report, including responding to comments from USAID and the IPs. ● Lead presentations to USAID regarding findings, conclusions, and recommendations from the evaluation. Qualifications: ● Must have a master’s degree in a relevant field of study with a minimum 8 years of relevant experience, including at least 5 years’ experience conducting evaluations of donor-funded projects in Africa. ● Good understanding of TB control in high burden countries with experience working with National TB Control Programs in South or Southern Africa. ● Demonstrated expertise with qualitative research methods for data collection and analysis and extensive experience conducting key informant interviews and facilitating focus group discussions. ● Proven evaluation team management and leadership experience, including overseeing data collection teams. ● Sound knowledge of USAID program cycle and extensive familiarity with USAID evaluation policy and relevant Agency guidance. ● Experience utilizing and combining quantitative and qualitative data. ● Prior experience working in South Africa preferred. ● Excellent English skills including speaking, writing, and reading. ● Outstanding analytical and report writing skills. TB Expert Position Summary: MERL is seeking a TB expert to assist the team lead with design, implementation, and reporting activities for the TBSAP midterm evaluation in close collaboration with the MERL team. The TB expert will have extensive experience in in global TB control with an excellent understanding of the global strategy and its implementation. S/he should have at least seven years of experience monitoring and evaluating various TB programs in high burden countries in Southern Africa. Responsibilities: The TB expert will be responsible for the following: ● Undertake desk research and provide inputs to inform required evaluation design documents. ● Lead the implementation of selected evaluation activities based on the approved evaluation design proposal. ● Assist the team lead in carrying out required design, data collection, and analysis tasks. ● Ensure that evaluation processes and products meet the requirements and quality standards of Panagora/MERL and USAID. ● Carry out research and analysis based on the approved evaluation design and methodology. This may include reviewing and analyzing secondary documents, conducting key informant interviews in person in South Africa and remotely, facilitating group discussions in South Africa, supervising data collection activities of other team members in multiple areas of South Africa where the TBSAP activities are being implemented, analyzing implementation data, and other approaches to collect and analyze data, including ensuring necessary mechanisms for data quality assurance. ● Assist with an internal findings/conclusions/recommendations workshop with the MERL team prior to drafting the evaluation report. ● Provide inputs for drafting and finalizing the draft and final evaluation report, including responding to comments from USAID and the IPs. 61 | Page ● Assist with presentations to USAID regarding findings, conclusions, and recommendations from the evaluation. Qualifications: ● Expert in global TB control with excellent understanding of the global strategy and its implementation. ● Good understanding of TB control in high burden countries with experience working with National TB Control Programs in South or Southern Africa. ● S/he should have at least seven years of experience monitoring and evaluating various TB programs. ● Previous experience working with USAID is highly preferred. ● In addition to expertise in TB strategy, s/he should have expertise in one of the following areas: Programmatic Management of Drug-resistant TB (PMDT), private sector engagement, TB/HIV, ASCM, laboratory and diagnosis, or policy planning. ● Previous experience in conducting evaluations for public health programs. ● Background in clinical trials preferred. ● Excellent English including speaking, writing, and reading. ● Outstanding analytical and report writing skills. Evaluation Specialist Position Summary: MERL is seeking a research specialist to assist the team lead with design, implementation, and reporting activities for the TBSAP midterm evaluation in close collaboration with the MERL team. The research specialist will have experience in in TB control with an excellent understanding of research methodologies, both qualitative and quantitative. S/he should have at least 5 years of experience in research, monitoring and evaluation, with some experience working with TB/HIV programs in South or Southern Africa. Responsibilities: The research specialist will be responsible for the following: ● Undertake desk research and provide inputs to inform required evaluation design documents. ● Implementation of selected evaluation activities based on the approved evaluation design proposal. ● Assist the team lead and TB expert in carrying out required design, data collection, and analysis tasks. ● Carry out research and analysis based on the approved evaluation design and methodology. This may include reviewing and analyzing secondary documents, conducting key informant interviews in person in South Africa and remotely, facilitating group discussions in South Africa, supervising data collection activities of other team members in multiple areas of South Africa where the TBSAP activities are being implemented, analyzing implementation data, and other approaches to collect and analyze data, including ensuring necessary mechanisms for data quality assurance. ● Assist with an internal findings/conclusions/recommendations workshop with the MERL team prior to drafting the evaluation report. ● Provide inputs for drafting and finalizing the draft and final evaluation report, including responding to comments from USAID and the IPs. ● Assist with presentations to USAID regarding findings, conclusions, and recommendations from the evaluation. 62 | Page Qualifications: ● Knowledge of qualitative and quantitative evaluation practices. ● Experience in designing, implementing, analyzing and presenting qualitative and quantitative research methodologies. ● Strong knowledge of South Africa, both the health sector and TB/HIV programming ● Knowledge of USAID programming practices, experience working with USAID preferred ● Excellent writing and inter-personal communication skills. 63 | Page Annex II: Evaluation Methods and Limitations Key Informant Interviews In-depth interviews and small group discussions were conducted with key informants at national, provincial, and district levels. At the central/national level, USAID, NDoH, other donors, implementing agencies, and partners were interviewed. At provincial, district, facility, and community levels, the team interviewed a range of NDoH managers and frontline health workers, TBSAP staff, community partner organizations, and other implementing partners working in TB. Data Sources The evaluation drew on relevant national and international documents; these are further outlined in Annex IV. Data Analysis Qualitative Data Analysis: Qualitative data were collected from consenting participants, and adherence to confidentiality and ethical practice were maintained. An interview tool/questionnaire was sent to the KIs to enable them to prepare for the interview, to fill in the questionnaire, and to send back it to the evaluation team in those cases when interviews could not be conducted. Data were collected using voice recorders and mobile phones for interviews with all KIs. Some informants, such as TBSAP personnel and NGO staff, responded by both filling in the questionnaire and via interviews. A codebook was developed using the QDA Miner software program (a Computer Assisted Qualitative Data Analysis tool – CAQDAS). The codes were designed to respond to the project themes on reduced TB infections, care, and treatment of vulnerable populations, and sustainability of effective TB response systems. The codes are indicated in Figure 7. 64 | Page Figure 7: Qualitative Data Codes Recorded interviews were transcribed and codes were assigned to each response using QDA Miner (Figure 8). The results of the qualitative analysis were used to establish corroborating evidence from the quantitative analysis. Connections between the project themes enabled the evaluators to see how relationships across the multiple role players were managed to implement key TB interventions and to identify gaps and opportunities for improvement. 65 | Page Figure 8: Example of Qualitative Data Transcription Quantitative Data Analysis: The performance of the TBSAP was assessed on the basis of baseline data from the DHIS, the District Health Barometer, TBSAP reports, and multiple sources from health departments and other researchers. Four treatment outcome indicators were the major focus of the evaluation: i. 90% Treatment Success Rate for drug-susceptible TB (DS-TB) cases ii. 75% Treatment Success Rate for DR-TB cases iii. 100% initiation of confirmed DR-TB cases on appropriate treatment iv. 90% ART coverage for TB/HIV coinfected patients In addition to reviewing these indicators, the evaluation sought to examine the trends in the implementation of the project over the contract period. The performance of TBSAP was further assessed by comparing its performance against eight comparison districts, including four high TB burden, two medium, and two low burden districts. The eight comparison districts were selected from the remaining nine of the original 23 TBSAP districts allocated to the project in 2016, but where TBSAP does not currently operate. Several unforeseen challenges and limitations severely constrained the evaluation team’s ability to draw any substantive conclusions from this comparative analysis. DHIS data were available through 2017 only (and for a handful of data points in 2018); as such, the comparison assesses the initial year of TBSAP implementation only. It is therefore premature to expect outcome-level performance differences. In addition, the TBSAP was assigned some of the highest burden districts, with the highest caseloads and the most challenging contextual factors. This includes the major metropolitan areas of the City of Johannesburg, City of Tshwane, eThekwini, and Cape Town. As metros, these districts may have better￾resourced facilities and systems, but they also have higher population density and client mobility. The comparison districts are all high burden districts; they all had some level of partner support. Given the added constraints and challenges faced in conducting this evaluation during lockdown (in terms of travel 66 | Page restrictions, limited access to data, etc.), the comparison was based solely on DHIS data for the period indicated above, and did not control for other factors or predictors of performance, such as the relative strengths of the DoH, data management systems, etc. The list of TBSAP and comparison districts is given in Table 5. Table 5: TBSAP Intervention and Comparisons Districts Sampling Frame TBSAP Districts Comparison Districts 4 high TB burden districts • eThekwini • City of Cape Town • City of Johannesburg • Nelson Mandela Bay • Ekurhuleni • Ehlanzeni • Bojanala • Buffalo City 2 medium TB burden districts • Mangaung • West Coast • Kenneth Kaunda • Chris Hani 2 low TB burden districts • Waterberg • uMkhanyakude • Lejweleputswa • West Rand Total Districts Selected 8 TBSAP Districts 8 Comparison Districts 67 | Page Triangulation: The mixed methods approach proved useful in providing multiple lines of sight and in allowing the convergence of the varied contexts in which the project provides support. The use of multiple measures revealed unique similarities and variations in the selected areas that may have been missed had this methodology not been used. Collecting both qualitative and quantitative data allowed the evaluation team to compare responses across sources and data types. An iterative analysis enabled the integration of the various data sources and findings. It was possible to identify contradictions between data sources and to consolidate conclusions based on the KIIs. Triangulation created an awareness of issues that were not anticipated, and the validation of findings by looking for similarities and discrepancies across data sources. Evaluation Challenges and Mitigation Strategies The evaluation team identified the following evaluation challenges and mitigation strategies. Table 6: Evaluation Challenges and Mitigation Strategies Challenge Mitigation Strategy COVID-19 The prioritization of the COVID-19 emergency response and timing of the lockdown led to delays in fieldwork and KIIs conducted. In May, Panagora and USAID agreed that fieldwork should proceed with all available KIIs, the bulk of which would be from national-level stakeholders and TBSAP staff. In June-July, the evaluation team, with assistance from the TBSAP project, was ultimately successful in interviewing KIs from provincial and district DoHs, whose contributions are included in this report. Panagora and USAID agreed to include key stakeholders (e.g., the NDoH and NTP) not available during the evaluation timeframe as an annex to follow, pending availability. Interviews were conducted virtually and not in person, with technological challenges in reaching KIs, and in the clarity of calls and audio recordings. Panagora contracted CADRE, which provided technical guidance on the best technological options for the clarity of calls and audio recordings. The evaluation team and TBSAP worked closely to confirm what platform best suited the various interviewees (phone, WhatsApp, Zoom, etc.) in advance of each call. Data Availability TB datasets were available at the district level only; therefore, the analysis of quantitative results at the facility level could not be conducted as initially planned. DHIS data for TB were available through 2017 only— for the first year of project implementation only. The TB team conducted district-level analysis of results with comparison districts through 2017 only. Project data from the TBSAP and qualitative data from the KIIs provided insights on successes and limitations, serving to triangulate the evaluation team’s findings. 68 | Page Annex III: Data Collection Instruments35 Data Collection Instrument: Community Level Interview Tool Introduction: My name is ________ and I am an independent, external evaluator hired by the Panagora Group to assist the NDoH and USAID to conduct a midterm evaluation to assist in improving TB programming/interventions in South Africa. Your participation in this evaluation is voluntary and all responses from you and the hundreds of people to be interviewed will remain confidential. Can we proceed with this interview? (Respondent response- check one: Yes / No). Section 0: Information about the Interview & the Key Informant 0.1 Date of interview: 0.2 Time began: Time ended: 0.3 Name of Interviewer: 0.4 Name of Key Informant: 0.5 Organization: 0.6 Facility link: 0.7 Title/Designation/Position: 0.8 Length of time in that position: OBSERVATION and SUPPORTING INFORMATION Note the general atmosphere in the organization, staff efficiency and interactions with DOH staff and patients (where possible). Request supporting documentation (stats, training materials, registers etc.) for statements and claims made during the interview wherever appropriate. SECTION A: OPENING - GENERAL 1. Tell me about your organization, the population you serve with a focus on your TB program (facility and community based)? 2. Please describe your relationship with TBSAP and your role in the project 35 Due to COVID-19, the evaluation team conducted all KIIs virtually; no actual forms were used 69 | Page over the past three years. SECTION B: RELEVANCE 3. What interventions/changes have been implemented in this district/facility/community in the last three years to support the TB program and improve (DS and DR TB) treatment outcomes? Probe regarding program community components: Case finding, Infection prevention, Contact management, Testing and referral, Treatment initiation, Linkage to care, Adherence support, Adverse events monitoring, Family education. 4. What was the role of the project in supporting/bringing about this change? SECTION C: EFFECTIVENESS - SYSTEMS STRENGTHENING & CAPACITY BUILDING 5. What do you regard as the major successes of the project? Probe regarding activities undertaken in these areas - Case finding, Infection prevention, Contact management, Testing and referral, Treatment initiation, Linkage to care, Adherence support, Adverse events monitoring, Family education. 6. What do you regard as the major weaknesses/failures of the project? As above. 7. What evidence have you seen or heard about the effectiveness of the project? Probe regarding acceptability of public community-based activities e.g. campaigns/dialogues versus household/family visits. Ask for success stories or challenges and how these were overcome. 8. Are there the opportunities that the project did not act on? What would you have done/do differently to improve the program? Probe for the reasons behind this. SECTION D: FOCUS ON VULNERABLE POPULATIONS – PLHIV, household contacts, HCWs, inmates, pregnant women, children >5 years, diabetics, informal settlements, mining & peri-mining communities 9. Describe how your facility has improved access to services for vulnerable populations. Probe for specific activities or events to facilitate access and/or remove barriers, referral and linkages. 10. How has TBSAP contributed to these? Probe regarding financial or in-kind contributions, admin support, venues, trainings, materials etc. 70 | Page 11. Do you have any comments you would like to make about the resources that were made available to implement the project? 12. Do you think there was a way that the project could have reached more beneficiaries? How could this have happened? What could have been done to make improvements? SECTION E: PUBLIC AWARENESS & COMMUNITY INVOLVEMENT 13. What ACSM activities does your organization conduct in the community? Probe regarding interpersonal communication, campaigns, information dissemination through various communication media including social media. 14. How has the project contributed to these? Probe regarding financial or in-kind contributions, admin support, venues, trainings, materials etc. 15. How are the lives of TB patients, their families and communities changed because of the project? Probe: Is there a positive story you can share? For example, re. referral and linkages? SECTION F: PARTNERSHIPS & SUSTAINABILITY – Probe for stories that illustrate a good or poor response 16. What government, private and NGO stakeholders do you collaborate with in implementing the TB program to support patients, their families and communities? Probe regarding Government (such as education, correctional services), mining, partners working with TB vulnerable groups, private health practitioners, and the corporate sector including occupational health interventions. 17. How has the work of your organization been changed because of this project? In what ways have you mainstreamed TB prevention and care in your work? For organizations no longer supported by TBSAP. 18. How has the project contributed to building human resource and organizational and system capacity? Are these changes sustainable? Have you been able to access other grants or funding opportunities? Probe: Are you able to share any stories (positive or negative of sustainable change? 19. Do you have any other comments or questions? THANK YOU FOR YOUR TIME! 71 | Page SECTION G: MOST EFFECTIVE/IMPACTFUL INTERVENTION 20. Please share your view on the effectiveness of each TBSAP area of intervention by providing a score on a scale of 1-10 (where 1 represents not effective at all and 10 is extremely effective) and a comment against the score. Activity Score Comment PILLAR 1: PREVENTION AND CARE Case finding · TB awareness campaigns · Community TB screening · Community dialogues · Patient and family education Infection prevention · Home IPC assessments · Education on cough hygiene, ventilation, hand washing etc. Contact management · Screening · Referral for testing · Family TB education 72 | Page PILLAR 2: DIAGNOSIS AND TREATMENT Testing and Referral · Sputum collection · Referral for testing · Follow up on results Treatment initiation · Linkage to care · Support treatment initiation · Treatment education Linkage to care · Referral of diagnosed patients to NGOs · Tracking of ILTFU patients and treatment LTFU PILLAR 3: FOLLOW UP CARE AND SUPPORT Adherence support · Adherence counselling · Referrals for special care 73 | Page · Community DOT · Treatment adherence Adverse events monitoring · Side effect reporting · Referral for management Family education · Educating contacts of diagnosed patients 21. Do you have any other comments or questions? THANK YOU FOR YOUR TIME! 74 | Page Data Collection Instrument: Facility Level Interview Tool Introduction: My name is ________ and I am an independent, external evaluator hired by the Panagora Group to assist the NDoH and USAID to conduct a midterm evaluation to assist in improving TB programming/interventions in South Africa. Your participation in this evaluation is voluntary and all responses from you and the hundreds of people to be interviewed will remain confidential. Can we proceed with this interview? (Respondent response- check one: Yes / No). Section 0: Information about the Interview & the Key Informant 0.1 Date of interview: 0.2 Time began: Time ended: 0.3 Name of Interviewer: 0.4 Name of Key Informant: 0.5 Facility: 0.6 Title/Designation/Position: 0.7 Length of time in that position: OBSERVATION and SUPPORTING INFORMATION Note the general atmosphere of the clinic and the TB section – Queues; patient flows, health staff efficiency and interactions with patients, TB infection control measures and evidence of other stakeholders e.g. CBOs working with the facility. Also observe and note the availability of IEC materials, paying particular attention to posters and adverts for patient hotlines, availability of complaints boxes-where are these placed, language, visibility. Request supporting documentation (stats, training materials, registers etc.) for statements and claims made during the interview wherever appropriate. SECTION A: OPENING - GENERAL 1. Tell me about your facility, the population you serve with a focus on your TB program (facility and community based)? 2. Please describe your relationship with TBSAP and the role of the project in the last three years. SECTION B: RELEVANCE 3. What interventions/changes have been implemented in this district/facility/catchment population in the last three years to strengthen the 75 | Page various aspects of the TB program and improve DS and DR TB treatment outcomes? Probe regarding program components: management, clinical, support services (laboratory, pharmaceutical - medicines & diagnostic), data management as well as treatment cascade: prevention (ACSM, IPC, FAST), screening, testing (GeneXpert, U-LAM, X-rays, clinical), treatment, and retention in care/LTFU. 4. What was the role of the project in supporting/bringing about this change? 5. Describe the response of your facility staff (management, clinical, support, outreach) and patients/community to the project. SECTION C: EFFECTIVENESS - SYSTEMS STRENGTHENING & CAPACITY BUILDING 6. What do you regard as the major successes of the project? Probe regarding building management and service delivery capacity, ACSM, key populations, service integration, contact tracing, finding missing patients, quality improvement, support services, data management & reporting, research and innovation, and referral and linkages. 7. What do you regard as the major weaknesses/failures of the project? As above. 8. What evidence have you seen or heard about the effectiveness of the project? Probe regarding quality improvement initiatives. 9. Are there the opportunities that the project did not act on? Probe for the reasons behind this. SECTION D: FOCUS ON VULNERABLE POPULATIONS – PLHIV, household contacts, HCWs, inmates, pregnant women, children >5 years, diabetics, informal settlements, mining & peri-mining communities 10. Describe how your facility has improved access to services for vulnerable populations. Probe for specific activities or events to facilitate access and/or remove barriers, referral and linkages. 11. How has TBSAP contributed to these? Probe: financial or in-kind contributions, admin support, venues, trainings, materials etc. 12. Do you have any comments you would like to make about the resources that were made available to implement the project? 76 | Page 13. Do you think there was a way that the project could have reached more beneficiaries? How could this have happened? What improvements could have been made? SECTION E: PUBLIC AWARENESS & COMMUNITY INVOLVEMENT 14. What ACSM activities are conducted in the facility and community? Probe regarding interpersonal communication, campaigns and information dissemination through various communication media including social media. 15. How has the project contributed to these? Probe regarding financial or in-kind contributions, admin support, venues, trainings, materials etc. 16. How are the lives of TB patients, their families and communities changed because of the project? Probe: Is there a positive story you can share? Referral and linkages? SECTION F: PARTNERSHIPS & SUSTAINABILITY 17. What other government, private and NGO stakeholders do you collaborate with in implementing the TB program to support patients, their families and communities? Probe regarding Government (such as education, correctional services), mining, partners working with TB vulnerable groups, private health practitioners, and the corporate sector including occupational health interventions. 18. How has the project contributed to building human resource and organizational and system capacity? Are these changes sustainable? Probe: Are you able to share any stories of (positive or negative) of sustainable change? 19. Do you have any other comments or questions? THANK YOU FOR YOUR TIME ! SECTION G: MOST EFFECTIVE/IMPACTFUL INTERVENTION 20. Please share your view on the effectiveness of each TBSAP area of intervention by providing a score on a scale of 1-10 (where 1 represents not effective at all and 10 is extremely effective) and a comment against the score. 77 | Page Activity Score Comment IR1: TB infections reduced · Increased public awareness of the TB epidemic · Effective implementation of infection control · Improved TB screening, including KPs IR2: Sustainability of effective TB response systems increased · Strengthened management capacity · Strengthened service delivery capacity · Improved data reporting and reporting systems IR3: Improved care and treatment of vulnerable populations · Increased contact tracing of KPs · Improved case management in KPs · Strengthened comprehensive systems and partnerships for care 21. To what extent has the project met the expectations of your staff and patients? Please rate on a scale of 1 to 5 where 1 is far below expectation, 2 is below expectation, 3 is as expected, 4 is above expectation and 5 is far above expectation. 78 | Page CADRE of staff SCORE COMMENT Management Clinical Support Outreach Patients Community Which category of staff has found the project most useful/benefited most? Do you have any success/positive stories to share? 22. Do you have any other comments or questions? THANK YOU FOR YOUR TIME! 79 | Page Data Collection Instrument: TBSAP Executive Interview Tool Introduction: My name is ________ and I am an independent, external evaluator hired by the Panagora Group to assist the NDoH and USAID to conduct a midterm evaluation to assist in improving TB programming/interventions in South Africa. Your participation in this evaluation is voluntary and all responses from you and the hundreds of people to be interviewed will remain confidential. Can we proceed with this interview? (Respondent response- check one: Yes / No). Section 0: Information about the Interview & the Key Informants 0.1 Date of interview: 0.2 Time started: Time ended: 0.3 Name of Interviewer: 0.4 Names of Key Informants: 0.5 Site: 0.6 Title/Designation/Position: OBSERVATION and SUPPORTING INFORMATION Introduce yourself and thank the team for making the time to meet with you. Explain the objectives of the evaluation and the process of the evaluation. Remind them of additional supporting documentation that may be required (stats, training materials, registers etc.) for statements and claims made during the interview wherever appropriate. SECTION A: OPENING - GENERAL 1. In your view how has the TBSAP project been received by the principals (funders/SAG/recipients)? 2. What have been the major achievements in general? 3. What have been the major constraints? 4. Please describe your relationship with your principals and their role in the project in the last three years. SECTION B: RELEVANCE 5. What interventions/changes have been implemented in the selected districts in the last three years to strengthen the various aspects of the TB program and improve DS and DR TB treatment outcomes? Probe regarding program components: management, clinical, support services (laboratory, pharmaceutical - 80 | Page medicines & diagnostic), data management as well as treatment cascade: prevention (ACSM, IPC, FAST), screening, testing (GeneXpert, U-LAM, X-rays, clinical), treatment and retention in care/LTFU. (NB. This will be available in their reports - you may ask specific questions picked up in their reports) 6. What was the role of the project executives in supporting/bringing about this change? 7. Describe, in general, the response of your principals (management, clinical, support, outreach) and patients/community to the project? SECTION C: EFFECTIVENESS - SYSTEMS STRENGTHENING & CAPACITY BUILDING 8. What do you regard as the major successes of the project? Probe regarding building management and service delivery capacity, ACSM, key population, service integration, contact tracing, finding missing patients, quality improvement, support services, data management & reporting, research and innovation, referral and linkages. 9. What do you regard as the major weaknesses/failures of the project? As above. 10. What evidence have you seen or heard about the effectiveness of the project from beneficiaries? Probe regarding quality improvement initiatives 11. Are there other opportunities that the project did not act on? Probe for the reasons behind this. SECTION E: PUBLIC AWARENESS OF TB & COMMUNITY INVOLVEMENT 12. What has been the best approach to working with the NDoH and other national level role-players on ACSM activities? Probe regarding interpersonal communication, campaigns and information dissemination through various communication media including social media. 13. How has TBSAP created linkages with the private sector to improve awareness of TB in the community? 14. How have you monitored the results and effectiveness of ACSM activities in the districts? Probe: financial expenditure, in-kind contributions by partners, admin 81 | Page support, venues, trainings, materials etc. 15. What can be done to strengthen the impact of ACSM in the districts in the coming period? SECTION D: INFECTION PREVENTION AND CONTROL AND TB SCREENING OF KEY POPULATIONS (PLHIV, household contacts, HCWs, inmates, pregnant women, children >5 years, diabetics, informal settlements, mining & peri-mining communities) 16. Describe how TBSAP has improved IPC guidelines and implementation at all levels of its operations. What lessons were learnt? Probe regarding specific activities/training/mentorship. 17. How has the TBSAP strengthened contributed to the collaboration and harmonization of TB screening and testing in the selected districts? 18. What specific activities are being carried by the project to improve GXP implementation? 19. How has TBSAP supported the NDoH in improving linkages at PHC level/ MCH/HIV programming[DP7] ? 20. Do you feel there was a way that the project could have reached more beneficiaries? How could this have happened? What could have been done to make improvements? 21. Which curriculum reviews on TB screening have been conducted by the TBSAP and how were they received by SAG? SECTION F: INCREASED SUSTAINABILITY OF EFFECTIVE TB RESPONSE 22. How has the project contributed to building human resource and organizational and system capacity? Are these changes sustainable? Probe: Are you able to share any stories (positive or negative) of sustainable change[DP8] ? 23. TBSAP conducted a baseline assessment of the performance low performing facilities. How has TBSAP addressed the shortcomings in performance in these facilities? 24. What support has been provided by TBSAP for the procurement and supply of drugs? Is there evidence for improvement? 25. How has TBSAP supported the introduction the Unique Patient Identifier at NDoH? 82 | Page 26. Has TBSAP assisted in review of data management in collaboration with the NDoH? 27. Has TBSAP assisted in design of a system for TB data review and analysis? Evidence[DP9] ? 28. Do you have any other comments or questions? SECTION G: MOST EFFECTIVE/IMPACTFUL INTERVENTION 29. Please share your view on the effectiveness of each TBSAP area of intervention by providing a score on a scale of 1-10 (where 1 represents not effective at all and 10 is extremely effective) and a comment against score. Activity Score Comment IR1: TB infections reduced · Increased public awareness of the TB epidemic · Effective implementation of infection control · Improved TB screening, including KPs IR2: Sustainability of effective TB response systems increased · Strengthened management capacity · Strengthened service delivery capacity · Improved data reporting and reporting systems IR3: Improved care and treatment of vulnerable populations 83 | Page · Increased contact tracing of KPs · Improved case management in KPs · Strengthened comprehensive systems and partnerships for care 21. To what extent has the project met the expectations of your staff and patients? Please rate on a scale of 1 to 5 where 1 is far below expectation, 2 is below expectation, 3 is as expected, 4 is above expectation and 5 is far above expectation. CADRE of staff SCORE COMMENT Management Clinical Support Outreach Patients Community Which category of staff has found the project most useful/benefited most? Do you have any success/positive stories to share? 22. Do you have any other comments or questions? THANK YOU FOR YOUR TIME! 84 | Page Annex IV: Sources of Information The following sources were used to collect data to address the evaluation objectives and questions: TB Data Sources TBSAP project data and DHIS district data were used to respond to the evaluation objectives and questions. Key Documents A comprehensive literature review included: TBSAP documents: • Original USAID request for proposals • TBSAP contract and modifications • TBSAP quarterly and annual workplans and reports • PMP plans • M&E plans • TBSAP presentations, reports, and materials Global and South African guidance documents: • South Africa’s National Strategic Plan for HIV, TB, STIs (2012‒2016) • South Africa’s National Strategic Plan for HIV, TB, STIs (2017‒2022) • National TB Strategic Plan (2012‒2016) • National TB Strategic Plan (2017‒2022) • NDoH Joint TB, HIV, PMTCT, STI and Hepatitis Program Review, 2019 • Global TB guidance documents from WHO, STOP TB, and END TB • Additional relevant journal articles and reports from partners working in TB in South Africa were considered. Key Informant Interviews In-depth interviews and small group discussions were conducted with a total of 82 key informants at national, provincial and district levels. Table 7: List of Key Informants Interviewed Organization Name Title National/central-level (7 KIIs) USAID Cindy Dlamini TB Program Manager Global Fund Donna Lee Public Health, M&E Specialist, SA Country Team THINK Kristina Wallengren Executive Director & Founder TB Think Tank Neil Martinson Executive Director Pediatric HIV Research Unit; National TB Think Tank Implementation Workstream Lead WHO Owen Laws Kaluwa WHO Country Representative Interactive Research & Development (IRD) Thulani Mbatha Chief of Party Bill & Melinda Gates Zameer Brey SA Country TB Lead 85 | Page Organization Name Title Foundation DoH National-level (4 KIIs) NDoH Norbert Ndjeka DR-TB Director GDoH Ntombizodwa Mntambo TB Coordinator, GDoH NDoH/USAID Phyllis Pholoholo USAID TA to NDoH NDoH Yulene Kock Deputy Director DR-TB Provincial & District-level DoH (19 KIIs) City of Johannesburg Edwin Madibogo TB Coordinator Cape Winelands District Fransisca Wisani District Manager Heidendal CHC, Free State Katlego Seemahale TB Nurse OR Tambo District, Eastern Cape Kholeka Mhlakaza HAST Manager Eastern Cape DoH Linda Ngqunge Operations Manager (TB) Free State DoH Lumka Ntloko TB Nurse Free State DoH Matshidiso Mongihlane HAST Director Gauteng DoH Mogorosi Mathlodi HAST Manager Sarah Baartman, Eastern Cape Nobantu Jack District TB Manager Eastern Cape DoH Nokuthula Sopiseka Senior Manager, TB services, Johannesburg District, Gauteng Nomsa Sebitlo HAST Coordinator Hlabisa Gateway Clinic, uMkhanyakude District, KwaZulu-Natal Nomusa Ntombela Operational Manager Sekhukhune District, Limpopo Patricia Makoane HAST Manager Western Cape DoH Prudence Kani M&E Officer Mangaung District, Free State Rebecca Limo TB Nurse Waterberg District, Limpopo Rhulani Madali HAST Manager Ethekwini District, KwaZulu Natal Sikisiwe Mdabe TB Supervisor uMkhanyakude District, KwaZulu-Natal Thabile Maphumulo TB Coordinator Ethekwini District, KwaZulu￾Natal Thandiwe Mbhele TB Supervisor TBSAP Central & Provincial-level (31 KIIs) TBSAP Alice van Zyl Provincial Manager – Western Cape TBSAP Carol Padi District Coordinator – Free State TBSAP Difuro Masemola Team Lead: Clinical Services TBSAP Dolly Moleba Provincial Manager - Limpopo TBSAP Elisia Dyson District Coordinator – Cape Winelands TBSAP Evelyn Mhlope Health Systems Strengthening/QA & QI Advisor TBSAP Gregory Jagwer Chief of Party TBSAP Ina Johanna Cillie West Coast District Coordinator TBSAP James Mmekwa District Coordinator – Tshwane, Gauteng TBSAP Keitumetse Mbengo QA Advisor, TBSAP HQ TBSAP Kgomotso Moema Project Administrator TBSAP Khaebana Mamorao ACSM Manager TBSAP Lerato Lebona PPP Advisor TBSAP Liziwe Geqe Provincial Manager, Eastern Cape TBSAP Margaret Molefe Provincial Manager - Gauteng TBSAP Neeltjie le Roux West Coast District Coordinator TBSAP Neliswa Soxa M&E Officer 86 | Page Organization Name Title TBSAP Nobesuthu Ramawela QA/QI Advisor TBSAP Nothemba Kula Deputy Chief of Party TBSAP Ntokozo Zuma Clinical Advisor TBSAP Panganai Dliwayo M&E Advisor TBSAP Pitso Makhetha Provincial Manager – Free State TBSAP Princess Mabota￾Rapholo Interpersonal Communication & Counseling Specialist TBSAP Samuel Thamaga District Coordinator – Sekhukhune, Limpopo TBSAP Simphiwe Mayaphi District Manager, Eastern Cape TBSAP Sinenhlanhla Memela uMkhanyakude District Coordinator, KZN TBSAP Siyabonga Kave M&E Officer, Nelson Mandela Bay District, Eastern Cape TBSAP Timothy Makazha M&E Officer TBSAP Tshepo Marelo M&E Officer, Gauteng TBSAP Tinyiko Baloyi District Coordinator – Waterberg, LP TBSAP Zamani Dlamini Provincial Manager – KwaZulu-Natal TBSAP-funded NGOs (5 NGOs) Asiphile eUganda Fidelia Mhlongwa Professional Nurse Mosamaria, Free State Florina Tladi Manager Kgatelelopele Community Home Based Care Robert Leshelele Director Care Ministries, Eastern Cape Siobhan Dooley Manager Mfesane, Eastern Cape Wendy Ndima Manager Provincial & District Level Other Key Stakeholders/Implementing Partners (16 KIIs) TB/HIV Care Amanda Fononda TB Project Manager – KZN MatCH Arthi Ramkissoon CEO Isibani Development Partners Daniella Simpson Finance Manager TB/HIV Care Deliwe Rene Phethlu TB Program Manager Anova Health Institute Diana Mokwena Program Manager APACE, City of Johannesburg Isibani Development Partners Frederica Mkwanazi Program Manager MatCH Lorna Paulus District Coordinator – eThekwini, KwaZulu-Natal Anova Health Institute Pauline Maluleke Hospital Services Manager Isibani Development Partners Phila Gaga M&E Officer MatCH Prineetha Naidoo Technical Lead: HIV Community Services at Maternal Aquity Health Innovations Refiloe Matji Director Anova Health Institute Sebenzile Thuso Technical Advisor - TB Anova Health Institute Simon Mtwampe Technical Advisor - TB Anova Health Institute Susan Kekane Executive Director Government Relations MatCH Victoria Mubaiwa Director Clinical Services Isibani Development Partners Violet Baloyi Executive Director 87 | Page List of Documents Reviewed Ekpe, B. (2017). Request for Approval of Key Personnel - Chief of Party. USAID Southern Africa. Ekpe, B. (2018). URC Approval of Grants Under Contract: AID 674-TO-16-00002. USAID Southern Africa. Ekpe, B. (2018). URC Request for Approval of Grant: AID-674-TO-06-00002. USAID Southern Africa. Ekpe, B. (2018). URC Request for Approval of Grants dated December 12, 2017: AID-674-TO-06-00002. USAID Southern Africa. Ekpe, B. (2018). URC Request for Grant Approval: AID-674-TO-06-00002. USAID Southern Africa. Enroth, B. (2017). Approval letter: Salary Approval for Personnel. USAID Southern Africa. Enroth, B. (2017). Request for Key personnel approval - Financial Management; Ops Dir. USAID Southern Africa. Faber, J. (2016). Request for Sub Grants Consent. USAID Southern Africa. USAID Southern Africa. Gqwaru, N. (2017). AID-674-TO-06-00002: Tuberculosis South Africa (TBSAP) Grant Approval. USAID Southern Africa. Gqwaru, N. (2017). AID-674-TO-06-00002: Tuberculosis South Africa (TBSAP) Grant Approval 2. USAID Southern Africa. Gqwaru, N. (2017). AID-674-TO-06-00002: Tuberculosis South Africa Project (TBSAP) Grants Approval 3. USAID Southern Africa. Gqwaru, N. (2017). AID-674-TO-06-00002: Tuberculosis South Africa Project (TBSAP) Grants Approval 4. USAID Southern Africa. Gqwaru, N. (2017). AID-674-TO-06-00002: Tuberculosis South Africa Project (TBSAP) Grants Approval Small Grants Approval. USAID Southern Africa. Hasha, C. (2016). Request for Grant Sub Approval: AID-674-TO-06-00002. USAID Southern Africa. Hasha, C. (2016). Request for Approval of Key Personnel. USAID Southern Africa. McCord. M. (2015). Tuberculosis South Africa Project Authorization Memo. USAID Southern Africa. National Department of Health. (2017). Let our Actions Count: South Africa’s National Strategic Plan For HIV, TB and STIs 2017-2022. The White House. (2015). National Action Plan For Combating Multidrug-Resistant Tuberculosis. Washington. United States Government. (2015). United States Government Global Tuberculosis Strategy. University Research Co. LLC (URC). (2016). USAID Tuberculosis South Africa Project (TBSAP): Quarterly Report - October to December 2016. University Research Co. LLC (URC). (2018). USAID Tuberculosis South Africa: Project Annual Report. University Research Co. LLC (URC). (2016). USAID TB SOUTH AFRICA PROJECT (TBSAP): Quarterly Report - July to September 2016. University Research Co. LLC (URC). (2016). USAID TB SOUTH AFRICA PROJECT (TBSAP): Quarterly Report – October to December 2016. University Research Co. LLC (URC). (2016). USAID/Tuberculosis South Africa Project (TBSAP): Quarterly Report - 17 March to 30 June 2016. University Research Co. LLC (URC). (2017). USAID TB South Africa Project (TBSAP): Quarterly Report - January to March 2017. University Research Co. LLC (URC). (2017). USAID TB South Africa Project: Quarterly Report - April to June 88 | Page 2017. University Research Co. LLC (URC). (2017). USAID Tuberculosis South Africa: Project Annual Report. University Research Co. LLC (URC). (2018). USAID Tuberculosis South Africa Project: Quarter Three Report - April to June 2018. University Research Co. LLC (URC). (2018). USAID Tuberculosis South Africa Project: Quarterly Report - January to March 2018. University Research Co. LLC (URC). (2019). USAID Tuberculosis South Africa Project Draft 0 Project Quarterly Report - April to June 2019. University Research Co. LLC (URC). (2019). USAID Tuberculosis South Africa Project: Project Quarterly Report - January to March 2019. University Research Co. LLC (URC). (2019). USAID Tuberculosis South Africa: Project Annual Report. University Research Co. LLC (URC). (n.d.). USAID Tuberculosis South Africa Project: Quarterly Report - July to September 2017. University Research Co. LLC (URC). (2016). USAID TB South Africa Project (TBSAP): Project Annual Report. University Research Co. LLC (URC). (2017). TBSAP Workplan: Financial Years 2016-2017. University Research Co. LLC (URC). (2017). USAID Tuberculosis South Africa Project: Quarterly Report - October to December 2017. University Research Co. LLC (URC). (2018). TBSAP Workplan: Financial Year 2017-2018. University Research Co. LLC (URC). (n.d.). Tuberculosis South Africa Project (TBSAP): 2015 - 2020. University Research Co. LLC (URC). (2018). USAID Tuberculosis South Africa Project: A guide for facilitators using the interpersonal communications manual for workshops with health care workers. USAID. (2016). TBSAP Amendment of Solicitation of Contract. USAID. (2016). University Research CO. LLC Contract: AID-OOA-I-14--00035. USAID. (2017). URC Amendment of Solicitation of Contract. USAID. (2018). Basic TB Management Participant Manual. USAID. (2018). Basic TB Management Participant Manual - References. USAID. (2019). USAID TB SAP PMP. USAID & National Department of Health. (2018). Basic TB Management Course Slides. USAID TB South Africa Project. (2017). Training Templates: 2017. USAID TB South Africa Project. (2018). An Interpersonal Communications Manual for Health Care Workers. USAID TB South Africa Project. (2019). Training Tracer Teams in KZN. USAID. (n.d.). TBSAP Grantee Database: 2016 - 2019. USAID. (n.d.). TBSAP_URC Technical Approach. USAID Tuberculosis South Africa Project. (2017). Addressing drug-resistant tuberculosis by implementing the United States Government National Action Plan. USAID Tuberculosis South Africa Project. (2017). Improving patient retention through ConnecTB. USAID Tuberculosis South Africa Project. (2017). Successes of the USAID TB South Africa Project. USAID Tuberculosis South Africa Project. (2017). TB and Women in South Africa: From TB Survivor to TB Ambassador: The Story of Lolo Kekana. USAID Tuberculosis South Africa Project. (2017). The creation of Buddy Beat TB: Supporter of pediatric patients undergoing treatment. 89 | Page USAID Tuberculosis South Africa Project. (2018). Basic Tuberculosis Management Manual for Community Healthcare Workers: Participant’s Manual. USAID Tuberculosis South Africa Project. (2018). South African Districts supported by TBSAP - 2018. USAID Tuberculosis South Africa Project. (2018). Training Delivery Standard Operating Procedure. USAID Tuberculosis South Africa Project. (2019). 2018 High DS-TB burden facilities per District - Maps. USAID Tuberculosis South Africa Project. (2019). 2018 High DS-TB burden facilities per District Graphs. USAID Tuberculosis South Africa Project. (2019). Collaboration between Regional Training Centers and the USAID TB South Africa Project. USAID Tuberculosis South Africa Project. (2019). Community-Based Tuberculosis Management through Local Non-Governmental Organizations. USAID Tuberculosis South Africa Project. (2019). ConnecTB, a mobile health application for locating TB patient contacts. USAID Tuberculosis South Africa Project. (2019). District Profile - TB Epidemiology: Western Cape. USAID Tuberculosis South Africa Project. (2019). District Profiles - TB Epidemiology: Eastern Cape. USAID Tuberculosis South Africa Project. (2019). District Profiles - TB Epidemiology: Free State. USAID Tuberculosis South Africa Project. (2019). District Profiles - TB Epidemiology: Gauteng. USAID Tuberculosis South Africa Project. (2019). District Profiles - TB Epidemiology: Kwa-Zulu Natal. USAID Tuberculosis South Africa Project. (2019). District Profiles - TB Epidemiology: Limpopo. USAID Tuberculosis South Africa Project. (2019). Drug-resistant Tuberculosis Service Package: A Patient￾Centered Care Case Management Approach. USAID Tuberculosis South Africa Project. (2019). Facilities per Sub-District and Type with 2018 high burdens. USAID Tuberculosis South Africa Project. (2019). Finding Missing TB Patients Report. USAID Tuberculosis South Africa Project. (2019). Focus On Community Based TB Management: Contact management as an effective strategy for finding missing TB cases in community settings in South Africa. USAID Tuberculosis South Africa Project. (2019). Focus on key populations: TB in Farms intervention. USAID Tuberculosis South Africa Project. (2019). Implementation of FAST Approach in South Africa. USAID Tuberculosis South Africa Project. (2019). Innovations in Community-Based Care and Strengthening Public-Private Partnerships. USAID Tuberculosis South Africa Project. (2019). One year of cross-promotion of TB information using traditional and new media to #EndTB and #BeatTB. USAID Tuberculosis South Africa Project. (2019). Project Overview. USAID Tuberculosis South Africa Project. (2019). Reducing TB infections in South Africa. USAID Tuberculosis South Africa Project. (2019). Strategic Partnerships to improve TB management. USAID Tuberculosis South Africa Project. (2019). Sustainability by Design: Successes in increasing capacity, strengthening service delivery and improving data systems in South Africa. USAID Tuberculosis South Africa Project. (2019). TB and women in South Africa. USAID Tuberculosis South Africa Project. (2019). TBSAP LOE by District. USAID Tuberculosis South Africa Project. (2019). URC's Commitment to Support USAID's New TB Business Model, the "Global Accelerator to End Tuberculosis" in South Africa. USAID Tuberculosis South Africa Project. (2019). USAID Tuberculosis South Africa Project Brief. USAID Tuberculosis South Africa Project. (2019). USAID Tuberculosis South Africa Project: Mid Term 90 | Page Review Meeting. USAID Tuberculosis South Africa Project. (n.d.). Missing Cases Map. USAID Tuberculosis South Africa Project. (n.d.). TBSAP Budget Breakdown by District. 2019. USAID/ South Africa. (2015). Request for Task Order Proposal - TBSAP. World Health Organization. (2014). The End TB Strategy. World Health Organization. (2018). Global tuberculosis report 2018. World Health Organization. (2019). South Africa Country Profile: Estimates of TB and MDR-TB burden. World Health Organization. (2019). Global tuberculosis report 2019. 91 | Page Annex V: Disclosure of any Conflicts of Interest 92 | Page Name Cheryl Goldstone Title Dr. Organization Panagora Evaluation Position? Team Leader Team member Evaluation Award Number (contract or other instrument) IDIQ No. AID-OAA-1-15-00025 Task Order No. 72067418F00002 USAID Project(s) Evaluated (Include project name(s), implementer name(s) and award number(s), if applicable) TB South Africa Project Midterm Evaluation I have real or potential conflicts of interest to disclose. Yes No If yes answered above, I disclose the following facts: Real or potential conflicts of interest may include, but are not limited to: 1. Close family member who is an employee of the USAID operating unit managing the project(s) being evaluated or the implementing organization(s) whose project(s) are being evaluated. 2. Financial interest that is direct, or is significant though indirect, in the implementing organization(s) whose projects are being evaluated or in the outcome of the evaluation. 3. Current or previous direct or significant though indirect experience with the project(s) being evaluated, including involvement in the project design or previous iterations of the project. 4. Current or previous work experience or seeking employment with the USAID operating unit managing the evaluation or the implementing organization(s) whose project(s) are being evaluated. 5. Current or previous work experience with an organization that may be seen as an industry competitor with the implementing organization(s) whose project(s) are being evaluated. 6. Preconceived ideas toward individuals, groups, organizations, or objectives of the particular projects and organizations being evaluated that could bias the evaluation. I certify (1) that I have completed this disclosure form fully and to the best of my ability and (2) that I will update this disclosure form promptly if relevant circumstances change. If I gain access to proprietary information of other companies, then I agree to protect their information from unauthorized use or disclosure for as long as it remains proprietary and refrain from using the information for any purpose other than that for which it was furnished. Signature Date 17th August 2020 93 | Page Name Lahla Ngubeni Title Mr. Organization Panagora Evaluation Position? Team Leader X Team member Evaluation Award Number (contract or other instrument) IDIQ No. AID-OAA-1-15-00025 Task Order No. 72067418F00002 USAID Project(s) Evaluated (Include project name(s), implementer name(s) and award number(s), if applicable) TB South Africa Project Midterm Evaluation I have real or potential conflicts of interest to disclose. Yes No If yes answered above, I disclose the following facts: Real or potential conflicts of interest may include, but are not limited to: 7. Close family member who is an employee of the USAID operating unit managing the project(s) being evaluated or the implementing organization(s) whose project(s) are being evaluated. 8. Financial interest that is direct, or is significant though indirect, in the implementing organization(s) whose projects are being evaluated or in the outcome of the evaluation. 9. Current or previous direct or significant though indirect experience with the project(s) being evaluated, including involvement in the project design or previous iterations of the project. 10. Current or previous work experience or seeking employment with the USAID operating unit managing the evaluation or the implementing organization(s) whose project(s) are being evaluated. 11. Current or previous work experience with an organization that may be seen as an industry competitor with the implementing organization(s) whose project(s) are being evaluated. 12. Preconceived ideas toward individuals, groups, organizations, or objectives of the particular projects and organizations being evaluated that could bias the evaluation. I certify (1) that I have completed this disclosure form fully and to the best of my ability and (2) that I will update this disclosure form promptly if relevant circumstances change. If I gain access to proprietary information of other companies, then I agree to protect their information from unauthorized use or disclosure for as long as it remains proprietary and refrain from using the information for any purpose other than that for which it was furnished. Signature Date 17 August 2020 94 | Page Annex VI: Additional TBSAP Materials TBSAP Alignment with the National TB Program The goal of the NTP Strategy 2017‒2021 is “To reduce the estimated incidence and mortality of TB in South Africa” and the TBSAP project goal is to “reduce the TB burden in South Arica,” a close alignment. The NTP’s Find-Treat-Prevent Framework includes case identification through increased facility￾level TB screening and active case finding in special populations; treating TB clients by scaling up appropriate short course treatments for DR-TB and reducing initial LTFU for DS-TB and DR-TB cases; and decreasing TB infections by scaling up preventive therapy for people living with HIV and household contacts. The NTP strategy’s cross-cutting interventions include optimizing systems for data use and quality improvement to close gaps in the TB care cascade. The TBSAP project activities were designed to complement this framework through the three Intermediate Results: 1) reduce TB infections 2) increase the sustainability of effective TB response systems 3) improve care and treatment of vulnerable populations Activities through these three Intermediate Results complement the NTP’s strategy interventions, as summarized in the table below. Table 8: TBSAP Activity Alignment with the National TB Program Strategy NTP Strategy Aligned TBSAP Project Activities FIND: Increased facility level TB screening • Expansion and consolidation of the FAST strategy in 102 FAST eligible facilities in the 11 supported districts. • Expansion of community involvement interventions and ensure bi-directional and functional links with the Primary Health Care (PHC) system. Formal and informal linkages between the PHC and organizations and institutions that work with TB key populations has been enhanced for each catchment area of the PHC facility/Health facility. • Micro-targeted IPC campaigns in hotspots and ensure functional linkage with the TB care cascade to increase facility level screening. The IPC campaigns educate communities about the risks of TB transmission and the importance of seeking care. Demand generation, advocacy and social mobilization encourage communities to utilize and to access TB testing and treatment services. • A focus on health managers, primary health area supervisors and local area managers: strengthening leadership, management and governance capacities through technical assistance and continuous quality improvement. FIND: Improve Active Case Finding in special populations • Optimized TB Screening • Efficient contact tracing of index cases • Enhanced case detection among • Optimized TB Screening: FAST approach targeting 102 hospitals and 8 CHCs in 11 supported districts. • Efficient contact tracing of index cases: community mobilization, including door to door campaigns. • Enhanced case detection among key populations: A focus on marginalized, vulnerable populations, as well as those most at-risk, through contact tracing, management and TB case-monitoring; expanding community involvement in and links with the PHC system for DOTS delivery; and increasing formal and informal linkages with organizations and institutions that work with these populations. These also includes: o TB in farms initiative: A total of 88 farms were reached across Sarah 95 | Page NTP Strategy Aligned TBSAP Project Activities key populations • Improving diagnostic yield through new diagnostic tools and revised algorithms • Improved quality standards in recording, reporting and tracking patient transfers between facilities Baartman (5), uMkhanyakude in Waterberg (5) and West Coast (7) districts. o Private Public Partnerships: Facilitating finding the missing TB patients at community level and increasing TB case detection and management by private GPs in OR Tambo District. • Improving diagnostic yield through new diagnostic tools and revised algorithms: New diagnostics such as the LF-LAM tool for PLWHIV and other immunocompromised individuals seen at hospital level. This is an effective intervention considering that 60% of PLWHIV are also coinfected with TB. Improved quality standards in recording, reporting and tracking patient transfers between facilities: A focus on linkages to care and reducing loss to follow-up; updating of records and data clean-up exercises at all levels of GoSA TB data and reporting systems (Tier.Net and EDR.Web) at facility, province and national levels. TREAT: Scale up appropriate short course treatments for DRTB Strengthen programmatic and clinical management of DRTB: TBSAP support to the NTP to ensure improved access to high-quality, patient-centered diagnostic and treatment services, enhancing adherence to TB and MDR-TB treatment and preventing the transmission of TB and MDR-TB within health care facilities and community settings. At district level TBSAP has concentrated on improving the capabilities of clinicians at facility level to improve clinical management and quality of care via onsite mentorship and capacity building. Improve DRTB treatment Outcomes: TBSAP does this by: (1) Providing TA to strengthen patient education, support and counselling for DRTB patients with an emphasis on treatment adherence and minimization of transmission; (2) Addressing loss to follow up through patient tracking and tracing and updating treatment outcomes on the system. Strengthen Capacity to manage DRTB through decentralization of services: • More technical staff have been contracted for the high burden and hotspots in the 14 supported districts to provide technical assistance and support to the 202 decentralized facilities • All 202 DRTB decentralized sites are supported with an emphasis on the key elements of health and community systems and services to ensure improved DRTB care and management: o Baseline and quality assessment o Management support o DR-TB implementation and operations team o Model (MOC) and Standard of Care (SOC) o Referral pathways design  Up and down referral o Community linkage  Project funded NGOs  WBOTs  Other Community Structures o Monitoring, evaluation and reporting  Primary data sources – tools  EDR.Web registration o Quality assurance and improvement o Pharmacovigilance o Capacity building interventions  Mentoring and support supervision  Training – including in-services  Clinical audits TREAT: Reduce initial LTFU for DSTB and DRTB Implement at every site, facility-specific DR-TB packages, including in-service training and clinical systems mentorship support focusing on teams and individual providers. Reporting systems have been established where absent and revitalized where already 96 | Page NTP Strategy Aligned TBSAP Project Activities cases available, including activation of electronic TB registers; the EDR.web. Regular review forums are also part of district and provincial level management dashboards. • Improving the patient experience/journey by ensuring the injectable regimen is phased out in all the supported districts • Improving clinical governance • Ensuring access to appropriate care of DRTB along the cascade from community to health facility and back to community • Interpersonal counselling and communication • Buddy beat package support for pediatric patients • ConnecTB application and roll-out • Geo-mapping of TB hotspots and management • WBOTs and other community institutions engagement in community management of TB • Rif-alerts to link TB patients • Improving quality standards in recording, reporting and tracking patient transfers between facilities PREVENT: Scale up preventive therapy for PLWHIV and household contacts • TBSAP is part of the technical working group for the revision of the South African guidelines to include 3HP in the TPT guidelines • Contact investigations as part of contact management is ongoing through the funded NGOs • WBOTs under the PHC re-engineering strategy included with an emphasis on TB screening among key populations including PLWHIV and household contacts • Collaboration with HIV DSPs and other partners to strengthen linkage into care for key populations and PLWHIV that are newly diagnosed or not in care and household contacts, to increase the number of people accessing TPT. Key Partnerships established include: o Right to Care and University of Pretoria: a campaign to ‘close the gap’ by finding missing TB patients and contacts in Mamelodi, City of Tshwane. o Anova Health: information on men’s health, HIV counselling and testing and distribution of condoms. o Match, Aurum, Health Systems Trust (HST) and South Africa Catholic Bishops Conference (SACBC): Collaborative development of a TB/HIV integrated work plan, which services eThekwini district and its municipality. o Aurum, Humana People to People, Childline Limpopo and Zakheni Training and Development Centre: received capacity building on the CQI and FAST approach. o TB in Farms Initiative: scale-up of support for farms and farming communities. o Engagement with Government Departments: identify opportunities to strengthen TB responses with Department of Agriculture, Forestry and Fisheries; Department of Transport; Department of Social development; and Department of Basic Education. o Engagement with public agencies: identify opportunities for partnership to strengthen TB and TB/HIV collaborated response with SAMA, Public Health Association South Africa and South African National AIDS Council. o National Institute for Communicable Diseases: an MOU was signed between the USAID TBSAP and National Institute for Communicable Diseases in 2018 to work on the “Interferon Gamma Release Assay (IGRA) study” and Linkage to care, initial lost to follow-up (ILTFU). o South African National AIDS Council: utilize local councilors to mobilize communities to participate and create awareness on TB initiatives. • A collaboration with NICD, QIAGEN and the NDOH for a feasibility study conducted in three hospitals (Pelonomi Regional in Free State, Zithulele District in 97 | Page NTP Strategy Aligned TBSAP Project Activities Eastern Cape and Pretoria West District in Gauteng) to examine the usability of the IGRA test for the prevalence of LTBI and active TB and progression from latent to active TB among HCWs • USE QIP to improve delivery of TPT for children under 5 years of age • Strengthen existing adherence support systems with supported NGOs • Facilitate TB screening of Key Populations including: Pregnant women; People living with diabetes, Children under 5 years-old; People living with HIV and initiate preventive treatment for eligible cases found during screening CROSS￾CUTTING: Optimize systems for data utilization • District and facility meetings to develop an evidence-based planning system to evaluate performance and develop remediation plans at least quarterly • Every district is provided with technical and logistical support to conduct review meetings and to plan for remediation. Example: the main gap identified is poor quality data to generate information to be used for program decision making • A short-term support plan is presently in place to ensure availability of valid and reliable data; as a more permanent solution, TBSAP has embarked on extensive data cleaning to ensure valid, reliable and clean data for the GOSA TB databases. This includes an intensified systems mentorship for operational managers and data capturers to increase the knowledge, skills and proficiencies in capturing of TB data in the Tier system • Work with DSPs data capturers to ensure procedurally correct capturing of TB data into the integrated information systems (TIER.Net) and appoint sessional data capturers where not possible • Scale-up the implementation of mHealth applications ConnecTB and IPConnect to improve patient management and reporting capabilities among all the 35 NGOs • Together with the National Department of Health, assess the quality of the data being generated CROSS￾CUTTING: A quality improvement initiative to close gaps in the TB care cascade • Technical support to the national roll-out of the TBQI interventions including secondment of a Technical Advisor and two Quality Improvement Officers (Nelson Mandela Bay Municipality and West Coast Districts) to the NDOH to drive and institutionalize the QI methodology within the TB program • New tools/approaches related to the QI and program management are now in place at all districts • TBSAP targets high TB burden facilities with sub-optimal performance: bottlenecks in service delivery are identified and the development of quality improvement plans for improved TB management is supported. Mentorship is also used to improve the improve these facilities’ performance • A QI SOP is now available and is contributing towards the national QI change package 98 | Page TBSAP Frameworks and Diagrams Figure 9: TBSAP local-level multisectoral accountability framework Figure 10: TB Cascade Analysis 99 | Page U.S. Agency for International Development 1300 Pennsylvania Avenue, NW Washington, DC 20523