EVALUATION REPORT – FINAL External Evaluation of the USAID/South Africa “Systems Strengthening for Better HIV/TB Outcomes” Project (2012-2017) July 15, 2017 This publication was produced at the request of the United States Agency for International Development. It was prepared independently by Mary Pat Selvaggio, Edna Berhane, Puveshni Crozier, Stephen van Houten, and Tabitha Kibuka of Khulisa Management Services (Pty) Ltd. EXTERNAL EVALUATION OF THE USAID/SOUTH AFRICA “SYSTEMS STRENGTHENING FOR BETTER HIV/TB OUTCOMES” PROJECT COMPREHENSIVE DISTRICT SUPPORT MODEL FINAL: July 15, 2017 Funded under AID-674-A-16-00003 with the Baylor College of Medicine Children’s Foundation; sub-recipient agreement award CFDA no. 98.001 to Khulisa Management Services Prepared by Khulisa Management Services, Pty (Ltd) Parktown North, Johannesburg South Africa Tel: +27.11.447.6464 info@khulisa.com www.khulisa.com Authors: Mary Pat Selvaggio, Evaluation Team Leader Edna Berhane, Evaluation Expert, Public Health Puveshni Crozier, HIV Specialist Evaluator Stephen van Houten, HIV/AIDS and Capacity Building Expert Tabitha Kibuka, Public Health Expert DISCLAIMER The authors’ views expressed in this publication do not necessarily reflect the views of the United States Agency for International Development or the United States Government. External Evaluation: USAID/South Africa “Systems Strengthening for Better HIV/TB Outcomes” Project (2012-2017) P a g e | iii ACKNOWLEGEMENTS Khulisa Management Services (Pty) Ltd would like to thank USAID/South Af rica and Baylor College of Medicine Children’s Foundation–Malawi for the opportunity to conduct this Evaluation. Special thanks to the seven (7) District Support Partners – ANOVA, Broadreach, Foundation for Professional Development, Kheth'Impilo, Maternal, Adolescent & Child Health (MatCH), Right to Care, and Wits Reproductive Health Institute – for their proactive, very responsive, and valuable support, particularly in sharing data and documentation, facilitating access to facilities, participating in interview, and providing critical input. In particular, we would like to thank the following individuals for their support in arranging site visits for the piloting of the evaluation tool and training of evaluators:  Oyebola Oyebanji (Kheth’Impilo)  Pappie Mjuba (ANOVA), and  Louise Gilbert (Right to Care) We are also grateful to the DOH staff at all levels – Directors, Deputy Directors, Area Managers, Operation Managers, Facility Managers, Doctors, nurses, pharmacists, pharmacist assistants, HIV/TB counsellors, WBOT teams and data capturers – as well as the PEPFAR Liaison Officers who participated in the evaluation, assisted in scheduling key interviews, and openly discussed and shared their knowledge and experience with the evaluation team. Last but not least we thank our wonderful evaluation and logistics team for their hard work and commitment to ensuring that due processes were followed, interviews conducted and tools completed. Indeed, the short timelines required patience and endurance. Team Khulisa consisted of the following individuals:  Mary Pat Selvaggio – Team Leader  Edna Berhane – Evaluation Expert  Zandile Mthembu- HIV Expert  Stephen van Houten HIV/AID and Capacity Building Expert  Salome Omolo – Evaluator  Puveshni Crozier – Evaluator  Kris Eale – Evaluator  Sabatha Madondo- Fieldworker  Welcome Bongani Manzini- Fieldworker  Boitumelo Desiree Morakile- Fieldworker  Lynette Byrne- Finance Manager  David Ndou- Project Support and fieldworker  Margaret Zwane – Logistics Coordinator External Evaluation: USAID/South Africa “Systems Strengthening for Better HIV/TB Outcomes” Project (2012-2017) P a g e | iv TABLE OF CONTENTS Executive Summary .............................................................................................................. 1 Evaluation Purpose & Evaluation Questions ..................................................................... 4 Evaluation Purpose ................................................................................................................................................ 4 Evaluation Questions ............................................................................................................................................ 4 Project Background ............................................................................................................. 6 Evaluation Methodology & Limitations ............................................................................. 7 Data Mining ............................................................................................................................................................ 8 Key Informant Interviews (KIIs) ........................................................................................................................ 8 Limitations .............................................................................................................................................................. 8 Findings, Conclusions & Recommendations ....................................................................10 Q1. To What Extent, and How, did Partners Strengthen Health Systems at DOH Management Levels? .................................................................................................................................................................... 10 Q2. To What Extent, and How, did the Partners Improve Patient Outcomes at Health Facilities? .... 33 Q3. How did the Program Design Influence the Achievement of Results? ............................................. 38 Q4. Partners’ Linkages with Other PEPFAR Programs to Provide Beneficiaries with Complementary Care ........................................................................................................................................................................ 41 Conclusions ........................................................................................................................45 Q5. Recommendations for future USAID/South Africa HIV Projects and Strategic Directions ........ 46 Evaluation Statement of Work .........................................................................48 Evaluation Methods .........................................................................................56 KII Tool............................................................................................................60 List of sites and Interviews...............................................................................70 HSS / CB Investments by DSP (FY2014-FY2016)...........................................74 Indicator Performance by DSP ........................................................................81 District Case Study – eThekwini......................................................................85 District Case Study – City of Johannesburg.....................................................94 Disclosure of any Conflicts of Interest...........................................................107 External Evaluation: USAID/South Africa “Systems Strengthening for Better HIV/TB Outcomes” Project (2012-2017) P a g e | v ACRONYMS AIDS Acquired Immune Deficiency Syndrome ART Anti-Retroviral Treatment ARV Anti-Retroviral BHC Broadreach Health Care CB Capacity Building CBO Community Based Organisation CCMDD Centralised Chronic Medicines Dispensing and Distribution CEO Chief Executive Officer CHC Community Health Centre COJ City of Johannesburg DATIM Data for Accountability Transparency and Impact DBE Department of Basic Education DHIS District Health Information System DHMT District Health Management Team DHP District Health Plan DIP District Implementation Plan DOH Department of Health DOTS Directly Observed Treatment, Short-course DQA Data Quality Assessment DQI Data Quality Improvement DSD Direct Service Delivery DSP District Support Partner EBF Exclusive Breastfeeding EPWP Expanded Public Works Program ETR Electronic TB Register FBO Faith Based Organisation FPD Foundation for Professional Development FTE Full-time Equivalent FY Fiscal Year GOSA Government of South Africa GP Gauteng Province HAST HIV, AIDS, STIs, and TB HCT HIV Counselling and Testing HIS Health Information Systems HIV Human Immunodeficiency Virus HMIS Health Management Information System HRH Human Resources for Health HSS Health System Strengthening HTS HIV Testing Services HWS Health Workforce Strengthening IPT Isoniazid Preventive Therapy KII Key Informant Interview External Evaluation: USAID/South Africa “Systems Strengthening for Better HIV/TB Outcomes” Project (2012-2017) P a g e | vi KZN KwaZulu-Natal LMG Leadership, Management and Governance LP Limpopo Province MCH Maternal and Child Health MER Monitoring, Evaluation, and Reporting MMC Medical Male Circumcision MSM Men having Sex with Men MTCT Maternal to Child Transmission NDOH National Department of Health NGO Non-Governmental Association NHLS National Health Laboratory System NIMART Nurse Initiated Management of ART NW Northwest Province OVC Orphans and Vulnerable Children PEPFAR President’s Emergency Plan for AIDS Relief PHC Primary Health Care PICT Provider Initiated Counselling and Testing PIMS Partnership Information Management System PLHIV People Living with HIV PMTCT Prevention of Mother to Child Transmission RFP Request for Proposal RTC Right to Care SIMS Site Improvement Monitoring Systems SMS Short Message Service SOP Standard Operating Procedure SOW Statement of Work STI Sexually Transmitted Infection SVS Stock Visibility Solution TA Technical Assistance TB Tuberculosis USAID U.S. Agency for International Development USD US Dollar USG US Government UTT Universal Test and Treat VMMC Voluntary Medical Male Circumcision WBOT Ward Based Outreach Teams WC Western Cape Province WHO World Health Organisation WISN Workload Indicators of Staffing Needs WRHI Wits Reproductive Health and HIV Institute External Evaluation: USAID/South Africa “Systems Strengthening for Better HIV/TB Outcomes” Project (2012-2017) P a g e | 1 EXECUTIVE SUMMARY Evaluation Purpose and Evaluation Questions USAID/South Africa commissioned this external evaluation of the “Systems Strengtheni ng for Better HIV/TB Outcomes” project to (i) assess the progress that the project has made towards achieving set goals and objectives, (ii) assess the quality of the District Support Partners’ (DSPs) project implementation and (iii) determine which approa ches and activities are working (and why). The evaluation is intended to inform USAID’s future strategic directions of its HIV Care and Treatment investments in South Africa, particularly for achieving maximum impact under the 90-90-90 goals. The evaluation was focused on answering 5 evaluation questions (with numerous sub questions), as summarised below: 1. To what extent and how did the DSPs strengthen health systems at the District, Provincial and National levels? 2. To what extent and how did the District Support Partners improve patient outcomes at public health facilities and district hospitals. 3. How did the program design influence the achievement of results at community, facility, district, provincial, and national levels? 4. How well did DSP partners link with other PEPFAR programs to provide beneficiaries with HIV prevention, care and OVC services? 5. What recommendations need to be factored into USAID-SA HIV future project design and strategic directions that will enable the HIV program to provide a broad range of high quality support for diagnosis, linkages to care, treatment initiation, maintenance and viral suppression, treatment adherence and retention in care, and supportive systems in line with the 90-90-90 PEPFAR strategic thinking? Project Background The “Systems Strengthening for Better HIV/TB Outcomes” project is implemented through 7 District Support Partners (DSPs) working in 8 provinces and 21 districts of South Africa. The project consists of a wide range of technical assistance and capac ity building activities to the South African Department of Health (DOH) at national, provincial, district, sub￾district, facility, and community levels. The project was designed to support the DOH in improving patient outcomes; planning; management of facilities, commodities/ supplies, and data; in defining core standards and state of the art practices and in ensuring their application. Each DSP implements a comprehensive model of support at the district level to strengthen DOH systems for improving HIV/TB patient outcomes and to prevent HIV/TB. DSPs supported their allocated DOH districts in core HIV services and management functions, including: care and support, viral monitoring, clinical lab interface for appropriate patients’ monitoring, TB screening, early diagnosis, and treatment, promoting adherence and retention, data quality improvement, supply chain management and commodities. Design and Methods Khulisa used a non-experimental evaluation design that excluded the use of a comparison group, but which allowed for measurement of project trends and achievements. In addition to answering the 5 evaluation questions and their sub-questions, USAID/South Africa External Evaluation: USAID/South Africa “Systems Strengthening for Better HIV/TB Outcomes” Project (2012-2017) P a g e | 2 requested the evaluation team to also try to quantify the HSS/CB activities and programmatic focus undertaken by the District Support Partners since FY2014. Two data collection approaches were employed to answer the evaluation questions: 1. Data Mining –partners were requested to provide two types of data for the period FY2014-FY2016: a. Performance indicator data from the Scope of Work’s (SOW’s) 29 key Indicators of Success (both PEPFAR and DOH indicators). b. Quantification of the volume of HSS/Capacity Building Activities delivered by the partner from FY2014-FY2016. For both data sets, we calculated trends and any association between the two data sets (i.e. whether more HSS and capacity building is associated with improved performance measures). 2. Key Informant Interviews (KIIs) – to obtain key stakeholder perceptions and feedback (from DOH, DSP, and donors) around the design and implementation of the project. More than 183 interviews were carried out in 106 locations/sites, reaching 389 respondents, most of whom were DOH staff and managers. Findings, Conclusions, and Recommendations The recent accelerated pace of the HIV/TB program as a result of South Africa’s adoption of the 90-90-90 goals and UTT requires a commensurate increase in capacity within the health system. USAID’s focus on Health Systems Strengthening/Capacity Building (HSS/CB) has been a relatively sound strategy for improving and expanding HIV/AIDS services in South Africa. DSPs have positively contributed to strengthening the six health system buildi ng blocks in their respective districts, but most especially the 3 building blocks of service delivery, health workforce, and information management. While there is an overall consensus that the DSPs’ efforts have indeed contributed to observed improvements, there are differences of opinion among DOH respondents regarding the magnitude of the changes. Views range from those who consider DSP efforts to have significantly contributed to most of the changes observed (usually DOH respondents at facility, sub-district, and district levels), to those who on the opposite side of the spectrum, consider DSPs as having affected little to no change (usually at DOH respondents at provincial and national levels). The differences in views are even more diverse when unpacked by individual DSPs and by provinces. It remains to be seen if the strategy has truly achieved the desired outcomes of improved quality of service delivery, and ultimately, improved patient outcomes. The performance of the 29 indicators that were the focus of the evaluation showed mixed results over the FY2014-FY2016 period. Indicators that show the most progress relate mostly to initiation on ART, PMTCT, reductions of TB defaulter rates, and use of Tier.net at facility level. This is consistent with the partners’ focus on workforce, services delivery, and information systems strengthening with a strong focus on initiating patients on ART. Across the 3 years, ART enrolment rates increased by 26%, but retention rates increased by only 5%, and viral load suppression rates remained basically unchanged. This points to the need for greater emphasis on differentiated models of care that better meet clients’ needs for retaining them on treatment. The most effective HSS/capacity building activities, as correlated with indicator performance, are those that involve adding staff to DOH services (i.e. secondment of staff for direct services delivery) and mentoring of DOH staff (mentoring, roving clinical teams). External Evaluation: USAID/South Africa “Systems Strengthening for Better HIV/TB Outcomes” Project (2012-2017) P a g e | 3 These were strongly associated with improvements in clinical performance indicators (e.g. HCT, ART initiation, TB patients on ART, circumcisions) and patient level information systems (e.g. use of Tier.net). Training by itself was weakly associated with better indicator performance. PEPFAR, through the DSPs, has supported improved skills of professional nurses to initiate and manage HIV treatment, and some support for planning, management, and monitoring capacity amongst the various DOH management levels. However, the impact of this support is limited by the health system’s ability to absorb it, mainly due to the chronic shortage of staff which acts as a bottleneck to expanding services, and which often constricts existing services by leading to high turnover due to high workloads and demotivatio n. This is further aggravated by an HR performance management system that does not recognise/reward good performance or penalise poor ones. The principal recommendations for PEPFAR/USAID for future HSS/CB projects are the following: 1. Engage in a transparent and open dialogue with Provincial DOHs about USAID’s mandate to, and expectations of, the partners it finances in the province. 2. Support DOH in improving overall HR Performance Management 3. Align PEPFAR and DOH planning/implementation processes and MER indicators/ processes. 4. Continue to support M&E to address critical capacity gaps and data backlogs. 5. Identify key HSS/CB indicators to be reported on by DSPs when technical assistance/capacity building is the main focus of the project. 6. Review the performance of districts that have “graduated” from DSP support to identify lessons learned and success/sustainability factors. External Evaluation: USAID/South Africa “Systems Strengthening for Better HIV/TB Outcomes” Project (2012-2017) P a g e | 4 EVALUATION PURPOSE & EVALUATION QUESTIONS Evaluation Purpose USAID/South Africa commissioned this external process and outcomes evaluation of the “Systems Strengthening for Better HIV/TB Outcomes” project to: i. assess the progress that the project has made towards achieving set goals, objectives, expected outputs and/or outcomes, ii. assess the quality of the District Support Partners’ (DSPs) project implementation and iii. determine which approaches and activities are working (and why). The evaluation is intended to inform USAID’s future strategic directions of its HIV Care and Treatment investments in South Africa, particularly for achieving maximum impact for HIV epidemic control in line with the Joint United Nations Program on HIV/AIDS (UNAIDS), the President’s Emergency Plan for AIDS Relief (PEPFAR), and the Department of Health (DOH) 2020 strategic directions and aspirational targets of achieving 90-90-90 targets by 2020. The evaluation examined both the patient-centred and health system strengthening aspects of the project and the DSPs’ district support model from FY2014 through FY2016, with priority given to the FY2014-FY2016 periods. Case Studies for two district – eThekwini and City of Johannesburg – were also prepared. Evaluation Questions The RFP lists five evaluation questions to be answered, with several sub -questions: 1. To what extent and how did the DSPs strengthen health systems at the District, Provincial and National levels? a. What have been the partners’ contribution to the following health systems building blocks: services delivery, district leadership and governance, district health plan, district implementation plans, laboratory and pharmaceutical systems strengthening, health workforce, and health information systems? b. How well have the DSPs strengthened the capacity of DOH at each level (facility, district, provincial and national level) to plan design, implement, manage, monitor, and sustain HIV/TB programs? c. Assess the partners’ approach to training/ mentoring of the DOH as a part of capacity building for HIV programming at district and facility level. d. Have the capacity building, training, and mentoring activities contributed to improved HIV related patient outcomes at facility and district levels? e. What is the gold standard for technical assistance and support at District, Provincial, and National levels? 2. To what extent and how did the District Support Partners improve patient outcomes at public health facilities and district hospitals. a. Has the program achieved the targeted results? b. Has the program helped to achieve a reduction of the estimated treatment gap, and an increase in the overall retention rate and viral load suppression rate for patients on ART? External Evaluation: USAID/South Africa “Systems Strengthening for Better HIV/TB Outcomes” Project (2012-2017) P a g e | 5 c. How have Health Systems Strengthening activities implemented by the DSPs contributed to improving HIV-related patient outcomes indicators? d. What is the gold standard for technical assistance and service delivery at PHC clinics, CHC, district hospitals? 3. How did the program design influence the achievement of results at community, facility, district, provincial, and national levels? a. What were the strengths in the program design for facilitating achievement of results? b. What were the gaps in the program design which hindered performance? c. What areas require additional investment to reach 90-90-90? 4. How well did DSP partners link with other PEPFAR programs to provide beneficiaries with HIV prevention, care, and OVC services? a. What partnerships and linkages were established? b. What innovative practices were used to establish linkages? c. What could be improved in terms of linkages? 5. What recommendations need to be factored into USAID-SA HIV future project design and strategic directions that will enable the HIV program to provide broad range of high quality support for diagnosis, linkages to care, treatment initiation, maintenance and viral suppression, treatment adherence and retention in care, and supportive systems in line with the 90-90-90 PEPFAR strategic thinking? External Evaluation: USAID/South Africa “Systems Strengthening for Better HIV/TB Outcomes” Project (2012-2017) P a g e | 6 PROJECT BACKGROUND USAID/South Africa finances the “Systems Strengthening for Better HIV/TB Outcomes” project through 7 District Support Partners (DSPs) working in 8 provinces and 21 districts of South Africa (Table 1). The project is complex and layered, consisting of a wide range of technical assistance and capacity building activities to DOH staff at national, provincial, district, sub￾district, facility, and community levels. The project is designed to support the Government of South Africa (GOSA) in improving patient outcomes; planning; management of facilities, commodities/ supplies, and data; in defining core standards and state of the art practices and in ensuring their application. Each DSP implements a comprehensive model of support at the district level to strengthen Government systems for improving HIV/TB patient outcomes and to prevent HIV/TB. DSPs provide training and technical assistance to their allocated districts on core HIV services and management functions, including care and support, viral monitoring, clinical lab interface for appropriate patients’ monitoring, TB screening, early diagnosis, and treatment, promoting adherence and retention, data quality improvement, supply cha in management and commodities. Table 1. USAID-supported DSPs, Provinces, and Districts under the Project District Support Partner (N=7) Provinces (N=8) Districts (N=21) 1. ANOVA Gauteng 1. City of Johannesburg Region C 2. City of Johannesburg Region D 3. City of Johannesburg Region E 4. City of Johannesburg Region G Limpopo 5. Mopani 2. Broadreach Eastern Cape 6. Alfred Nzo Mpumalanga 7. Gert Sibande KwaZulu-Natal 8. UGu 9. uThungulu 3. Foundation for Professional Development) Gauteng 10. Tshwane Mpumalanga 11. Nkangala Limpopo 12. Capricorn 4. Kheth'Impilo KwaZulu-Natal 13. Umgundgundlovu Western Cape 14. Cape Town 5. Maternal, Adolescent & Child Health KwaZulu-Natal 15. eThekwini 6. Right to Care Mpumalanga 16. Ehlanzeni Gauteng 17. City of Johannesburg Region A 18. City of Johannesburg Region B Free State 19. Thabo Mofutsanyane 7. Wits Reproductive Health Institute North West 20. Dr Kenneth Kaunda Gauteng 21. City of Johannesburg Region F External Evaluation: USAID/South Africa “Systems Strengthening for Better HIV/TB Outcomes” Project (2012-2017) P a g e | 7 EVALUATION METHODOLOGY & LIMITATIONS This evaluation was designed to measure the Health Systems Strengthening (HSS) support provided by DSPs to DOH1 against the WHO HSS building blocks shown in Figure 1. Figure 1. WHO Building Blocks for Health Systems Strengthening To answer the evaluation questions and sub questions, Khulisa used a non-experimental evaluation design that excluded the use of a comparison group, but which allowed for measurement of project trends and achievements. Our roadmap to answering the evaluation questions was elaborated in an Evaluation Matrix, which defined key indicators for each evaluation sub-question as well as the data collection and analytical method to be used. In addition to answering the 5 evaluation questions and their sub-questions, USAID/South Africa requested the evaluation team to try to quantify the HSS/CB activities and programmatic focus undertaken by the DSPs since FY2014. Given the difficulty in establishing consistent units of measure across the different DSP programs and HSS/CB activities, the evaluation team decided to use two proxy measures for quantifying HSS/CB -- the amount of money and human resources (in full time equivalent or FTE) dedicated to HSS/CB activities. Thus, two data collection approaches were employed to answer the evaluation questions: 1. Data Mining – we requested partners to provide us with data for the period FY2014- FY2016 for two types of data: a. Performance indicator data from the SOW’s 29 key Indicators of Success which consist of both PEPFAR and DOH indicators. b. Quantification of the volume of HSS/Capacity Building Activities delivered by the partner from FY2014-FY2016. For both data sets, we calculated trends and any association bet ween the two data sets (i.e. whether more HSS and capacity building is associated with improved performance measures). 2. Key Informant Interviews (KIIs) – to obtain key stakeholder perceptions and feedback (from DOH, the DSPs and donors) around the design and implementation of the project. 1 The USAID/South Africa “Systems Strengthening for Better HIV/TB Outcomes” project focused on all the WHO HSS building blocks except Health Care Financing. External Evaluation: USAID/South Africa “Systems Strengthening for Better HIV/TB Outcomes” Project (2012-2017) P a g e | 8 We incorporated quantitative measures in the KIIs in the form of Likert scales (e.g. Strongly Agree to Strongly Disagree) to allow for comparisons between respondent groups. Data Mining DSP partners completed spreadsheets with the values for the 29 Performance Indicators listed in the SOW (see page 51 of Annex 1) for the three-year period FY2014-FY2016, and the volume of HSS/Capacity Building activities – measured by financial expenditure and human resources (i.e. Full-Time-Equivalent or FTE) allocated to HSS/CB – delivered over the same period. We analysed the trends for both data sets across the three years, and the association of the HSS/CB activities to trends in the performance Indicators. Key Informant Interviews (KIIs) A representative sample of locations at national, provincial, district, sub-district, and facility levels was selected. Sampling of individuals targeted for KIIs was purposive where individuals were chosen because of their roles and involvement in the project and partnerships. The sampling approach is further detailed in Annex 2. More than 183 interviews were carried out in 106 locations/sites, reaching 389 respondents. The planned vs actual KII fieldwork is presented in Table 2 and Table 3. The full list of sites visited for the KIIs is presented in Annex 4. Table 2. Planned vs Actual Locations for KII Site Visits Table 3. Planned vs Actual No. Persons to be Interviewed Limitations Evaluation Question 2.b asks “Has the program achieved the targeted results?” Because, we were unable to get targets for the 29 indicators of success stated in the SOW, there is no analysis around this sub-question. However, trends in indicator performance (from FY2014- FY2016) are presented in the analysis on page 33. In designing our KII tool, we faithfully followed the structure of the SOW in terms of the Evaluation Questions and their focus. Question 1 asks the evaluation team to review several building blocks including “Laboratory and Pharmaceutical Systems Strengthening”. In our haste to meet tight deadlines, we asked the combined question (i.e. Laboratory and National Office Provincial Office District Office Sub￾district office Health Facilities Total National Office District Office Sub￾district Team TOTAL Planned 1 8 16 16 32 73 7 16 0 23 2 98 Actual 1 8 11 19 33 72 7 13 7 27 7 106 % of Planned 100% 100% 69% 119% 103% 99% 100% 81% #DIV/0! 117% 350% 108% Grand Total No. Locations for Site visits Funder/ Donor South African Government PEPFAR DSP (N=7) National Office Provincial Office District Office Sub￾district office Health Facilities Total National Office District Office Sub￾district Team TOTAL Planned 4 32 64 64 128 292 28 64 0 92 8 392 Actual 3 25 42 50 136 256 28 78 19 125 8 389 % of Planned 75% 78% 66% 78% 106% 88% 100% 122% #DIV/0! 136% 100% 99% South African Government PEPFAR DSP (N=7) Funder/ Donor Grand Total No. Persons to be Interviewed (estimated 4 respondents per location) External Evaluation: USAID/South Africa “Systems Strengthening for Better HIV/TB Outcomes” Project (2012-2017) P a g e | 9 Pharmacy) in our KIIs. However, this did not provide us with sufficient data around Laboratory Strengthening, and as such there are few findings for this. Tight timelines for carrying out the evaluation limited the calendar for data collection, and this further limited our access to National Department of Health respondents who often need several weeks’ advance notice for meetings. Because we were unable to obtain the PEPFAR performance indicator data directly from USAID, nor the DOH performance indicator data from DOH, we were asked to request this information directly from the DSPs. This took a considerable amount of effort on the part of the evaluation team as well as the partners, many of whom submitted the requested data only very late and only after repeated requests from the evaluation team. Moreover, m any DSPs were unable to provide values for certain PEPFAR indicators listed in our SOW because they said they were not required to report on these to PEPFAR and thus had no data. The accuracy of the performance indicator data received from the DSPs could not be verified. USAID also requested the evaluation team to quantify the amount of HSS/CB delivered under the program, although this was not part of the SOW. As explained above, to do this we requested values for HSS/CB expenditures and FTEs across 3 years (2014 -2016) from the DSPs. Again, this took a considerable effort on the part of the partners, many of whom submitted the requested data only very late and after repeated requests from the evaluation team. Again, the accuracy of this HSS/CB data could also not be verified. External Evaluation: USAID/South Africa “Systems Strengthening for Better HIV/TB Outcomes” Project (2012-2017) P a g e | 10 FINDINGS, CONCLUSIONS & RECOMMENDATIONS Q1. To What Extent, and How, did Partners Strengthen Health Systems at DOH Management Levels? Evaluation Question 1 has numerous sub-questions around the partners’ contributions to strengthening the HSS building blocks and to improving DOH capacity for designing, managing and implementing HIV/TB programmes, as well as the DPSs’ approach to DOH training and mentoring and their contribution to improved patient outcomes. The following discussion first describes the types and trends of DSPs’ Health Systems Strengthening/Capacity Building (HSS/CB) activities over the 3-year period FY2014- FY2016. Thereafter, we present findings around the Government’s satisfaction with the partners’ HSS/CB inputs, a summary of the DSPs’ approach to reaching the 90-90-90 goal, and the gold standard for Technical Assistance and Support at DOH management levels. TRENDS IN PARTNERS’ HSS/CB INVESTMENTS (FY2014 - FY2016) In addition to answering the 5 evaluation questions, USAID/South Africa requested the evaluation team to try to quantify the HSS/CB activities and programmatic focus undertaken by the DSPs since FY2014. Given the difficulty in establishing consistent units of measure across the different DSP programs and HSS/CB activities, the evaluation team (with USAID/South Africa) decided to use two proxy measures for resources dedicated to HSS/CB: 1. Expenditure (in US Dollars or USD), and 2. Human resources (in full-time equivalent or FTE). Because all FTE and expenditure data were self-reported by the DSPs, the evaluation team was unable to validate the numbers provided. Nevertheless, a general understanding of the DSPs’ resource allocation toward HSS/CB can be deduced from the analyses below. Program Areas - Trends in HSS/CB Expenditure and FTE Table 4 and Table 5 present the programmatic areas where partners directed their HSS/CB resources. The two proxy measures depict an overall trend of increasing resources commitment from FY2014 to FY2016 – with HR commitments (i.e. FTE) tripling and expenditure increasing by 19% over the 3-year period. The 3-year trends are not smooth or consistent, as indicated by the distinct change in 2015 by the tables’ spark lines. This reflects the PEPFAR/South Africa pivot toward achieving the 90-90-90 goals, and the DSPs’ emphasis on increasing coverage of HIV and TB services. Most resources were directed at HIV/TB services delivery (Table 4 and Table 5) 2 – such as HCT, PMTCT, and Facility Based Care, Treatment, and Support – and this is consistent with PEPFAR/South Africa’s 2015 pivot toward achieving the 90-90-90 strategic goals, emphasising expanded services delivery. Although less intensively, DSPs also committed HR resources to programmatic areas that enhance services delivery, such as support for health information systems and supply chain management. Annex 5 presents a disaggregated analysis of HSS/CB programme investments by DSP. 2 The trends in FTE and expenditure by programme area (Table 4 andTable 5) and capacity building activities (Table 6 and Table 7) do not necessary align. In part this is due to the exclusion of RTC in the FTE calculations which compromises comparisons of trends between the two data sets. External Evaluation: USAID/South Africa “Systems Strengthening for Better HIV/TB Outcomes” Project (2012-2017) P a g e | 11 Capacity Building Activities - Trends in HSS/CB Expenditure and FTE DSPs used different models for allocating their HSS/CB resources to achieve the greatest impact – focusing on a wide range of HSS/CB activities, especially training, mentoring, direct services delivery/staff secondment and roving clinical teams, which across all partners see the greatest increases from FY2014 to FY2016 (Table 6 and Table 7). Annex 5 presents a disaggregated analysis by DSP. Figure 2 summarises the top three capacity building activities in which the pa rtner invested3. For most partners, adding additional staff through direct services delivery or seconded staff was their primary HSS/CB strategy, followed by mentoring or roving clinical teams. As discussed later in this report – “DSP Training / Mentoring Approaches and Effects on Patient Outcomes” (page 30) – these are the HSS/CB activities that are most correlated with positive changes in indicator values. Figure 2. Top 3 Capacity Building Activities where DSPs invested most FTE (2014 -2016) HSS Capacity Building Activity District Support Partner (DSP) –top CB activities receiving FTE investment Anova BRHC FPD Kheth’Impilo MatCH WRHI Direct Service Delivery (DSD) 1 2 (tied) 2 1 Temporary Seconded Staff 2 (tied) 2 2 Mentoring 3 1 Supportive Supervision 1 Roving Clinical Teams 2 2 1 3 3 Training 3 1 Other 3 1=most FTE investment; 2=second most FTE investment; 3=third most FTE investment 3 While all DSPs implemented most of the capacity activities, Figure 2 highlights the top 3 capacity building activities that were the focus of the DSPs programme External Evaluation: USAID/South Africa “Systems Strengthening for Better HIV/TB Outcomes” Project (2012-2017) P a g e | 1 Table 4. Programmatic Domain - Human Resources (in FTE) dedicated to HSS/CB (2014-16)4 5 6 7 Table 5. Programmatic Domain - Expenditures (in USD) dedicated to HSS/CB (2014-16)5 7 4 Right to Care was excluded from the FTE analysis because they did not present their values in FTE. 5 Green shaded cells = the top 5 programmatic domains in terms of overall increase over the 3-year period. 6 Other = community-based testing and counselling, VMMC, Gender programmes, Prevention 7 Spark lines indicate direction of the change s, but not magnitude . Programmatic Domain 2 014 2 015 2 016 Total Trend % Change since 2014 HTC-PITC 275 467 390 1 131 42% HTC-VCT 127 272 521 920 310% PMTCT 77 271 538 885 602% Facility-based care, treatment and support 388 850 1 787 3 025 361% HSS - District Planning / supervision 60 57 71 189 19% HSS - Pharmacy/ Supply Chain Management 19 16 869 904 4517% HSS-Strategic Information 145 408 497 1 050 243% Infection Control 24 26 72 121 202% Laboratory 7 9 15 32 109% Other 40 64 193 298 377% Total 1 162 2 439 4 952 8 553 326% All Partners excluding RTC (FTE) Programmatic Domain 2 014 2 015 2 016 Total Trend % Change since 2014 HTC-PITC 3 199 577 3 382 114 4 998 685 11 580 375 56% HTC-VCT 3 146 916 3 188 740 5 638 799 11 974 455 79% PMTCT 4 049 374 4 430 922 5 300 044 13 780 340 31% Facility-based care, treatment and support 16 919 262 12 436 833 18 054 625 47 410 720 7% HSS - District Planning / supervision 6 669 013 5 737 864 4 889 655 17 296 532 -27% HSS - Pharmacy/ Supply Chain Management 1 254 144 692 872 679 793 2 626 809 -46% HSS-Strategic Information 3 733 813 4 140 851 4 894 751 12 769 416 31% Infection Control 2 730 500 3 011 462 4 370 302 10 112 263 60% Laboratory 115 520 116 149 3 352 047 3 583 716 2802% Other 2 884 118 3 116 005 2 116 081 8 116 205 -27% Total 41 502 660 36 871 698 49 296 097 139 250 830 19% All Partners (expenditure in USD) External Evaluation: USAID/South Africa “Systems Strengthening for Better HIV/TB Outcomes” Project (2012-2017) P a g e | 2 Table 6. Capacity Building Activities - FTE for HSS/CB (2014-16)4 5 7 8 Table 7. Capacity Building Activities - Expenditures (in USD) for HSS/CB (2014-16) 5 7 8 Other = coaching, mentoring HSS Capacity Building Activity 2014 2015 2016 Total Trend % Change since 2014 Direct Service Delivery (DSD) 142 160 725 1 027 412% Temporary Seconded Staff 93 159 307 558 232% Mentoring 107 64 145 316 36% Supportive Supervision 95 118 126 339 33% Roving Clinical Teams 218 250 404 872 85% Training 503 1 682 3 230 5 415 541% Other 4 6 16 26 284% Total 3 176 4 454 6 968 8 553 119% All Partners excluding RTC (FTE) HSS Capacity Building Activity 2014 2015 2016 Total Trend % Change since 2014 Direct Service Delivery (DSD) 4 223 743 3 528 264 10 418 115 18 170 123 147% Temporary Seconded Staff 4 107 753 5 055 742 8 605 363 17 768 858 109% Mentoring 9 072 350 5 162 482 3 893 064 18 127 896 -57% Supportive Supervision 9 868 069 8 289 115 6 549 197 24 706 382 -34% Roving Clinical Teams 9 796 890 10 790 165 16 279 878 36 866 933 66% Training 5 002 151 3 872 594 4 559 030 13 433 775 -9% Other 2 631 280 3 555 449 3 990 134 10 176 864 52% Total 44 702 237 40 253 812 54 294 781 139 250 830 21% All Partners External Evaluation: USAID/South Africa “Systems Strengthening for Better HIV/TB Outcomes” Project (2012-2017) P a g e | 3 PARTNERS’ CONTRIBUTION TO STRENGTHENING HEALTH SYSTEMS BUILDING BLOCKS DOH respondents credit (to varying degrees) the DSPs with improving all the HSS building block areas, with the most credit being given to them for strengthening the health workforce, HIS, services delivery, and district implementation planning (Figure 3). This is consistent with the DSPs’ investments in HSS/CB described above, which emphasised strengthening the health workforce, health information systems, and services delivery. Figure 3. Changes in HSS Building Blocks Areas Attributed to DSP Efforts (DOH respondents only) Each of the six HSS building blocks that were the focus of the project evaluation are discussed on the pages below: 1. Services Delivery on page 4, 2. Leadership, Management, and Governance (LMG) on page 7, 3. District Health Planning (including Implementation Planning) on page 10, 4. Laboratory and Pharmaceutical Systems Strengthening on page 14, 5. Health Workforce Strengthening on page 19, and 6. Health Management Information Systems on page 22 In each of these discussion, we present the most significant changes in the building block since FY2014, the extent to which those changes can be attributed to DSP Efforts, and the challenges faced in strengthening the building block. One clear pattern in the findings is that DOH respondents at district, sub district, and facilities levels acknowledge DSP contributions in far greater numbers than DOH respondents at provincial or national level, the vast majority of whom “didn’t know” whether the DSP contributed to improvements in the HSS building block area . While this points to the project’s focus on strengthening district and facility level, it also indicates a lack of project engagement at provincial and national levels. External Evaluation: USAID/South Africa “Systems Strengthening for Better HIV/TB Outcomes” Project (2012-2017) P a g e | 4 Services Delivery Most Significant Changes in Services Delivery Ensuring good health services at Primary Health Care (PHC) level is an important objective for the South African Government, as is evidenced by the Government’s numerous policies, guidelines and standard operating procedures (SOPs) governing service delivery standards. The most significant changes in service delivery observed over the past few years can be grouped as per Table 8 below. Table 8. Most Significant Changes in Services Delivery Areas of Significant Change Reasons for Change, as Noted by DOH Respondents 1. Improved patient care/management  Expanded treatment coverage including linka ges with community-based organisations targeting key and priority populations.  Reduced patient waiting times  Improved patient results monitoring  Increased ability to interpret and act on laboratory results  Improved patient demand and uptake of HIV and TB related services.  Increased accessibility of HIV and TB related services. 2. Improved DOH staff capacity to manage HIV and TB  DSP support for NIMART training.  Onsite mentorship and patient file audits to identify skill gaps  DSP staff secondment including roving clinical teams 3. Improved patient outcomes.  .DSP support with preparing, implementing and monitoring quality improvement plans. 4. Improved quality and safety of HIV and TB testing  Increased access to and initiation onto ART.  Proficiency testing of HIV test kits 5. Increased viral suppression rates.  Increased rate of linkage to ART.  DSP support for patient flow and treatment process maps as well as easy-to-reference ‘cheat sheets’. 6. Reduced rate of lost-to￾follow-up.  Increased retention to treatment rates.  Improved defaulter tracking and tracing systems supported by DSPs 7. Decanting of stable patients  Expansion of community -based adherence clubs. 8. Decongestion of health facilities.  Roll-out of CCMDD models of drug distribution.  Reduced waiting times  Increased coverage of treatment. 9. Integration of HIV with other services  DSP support of DOH’s Ideal Clinic initiative that promotes integration of HIV with other chronic diseases. Role of District Support Partners in Services Delivery Changes DOH respondents at district, sub-district, and facility level consider the above-cited changes to be largely attributed to DSP technical support efforts (Figure 4). In contrast, DOH respondents at national and provincial level don’t regard the changes as resulting from DSPs technical support. Nearly all DOH respondents (96%) credit DSPs for improving quality of HIV and TB services. DSPs are very involved in, and provide support for, interrogating and analysing performance data for creating quality improvement processes/plans at the WHO definition of good health services: … those that “deliver effective, safe, quality, personnel, and non-personnel health interventions to those who need them, when and where needed, with minimum waste of resources.” External Evaluation: USAID/South Africa “Systems Strengthening for Better HIV/TB Outcomes” Project (2012-2017) P a g e | 5 various DOH levels. In addition, monthly review meetings held at facility and sub-district levels and quarterly review meetings held with district-level DOH have contributed to improving quality of services. The variation in attribution is exaggerated when the analysing by partner (Figure 5) with certain DSPs such as FPD and WRHI given more credit for service delivery improvements compared to other partners. Likewise, DOH respondents in more resource constrained provinces (such as NW, EC, and LP) were more likely to attribute the improvements in service delivery to DSP Efforts. Figure 4. Attribution of Changes in Service Delivery to DSP Efforts (DOH respondents only) Figure 5. Attribution of Changes in Service Delivery to DSP Efforts, by partners External Evaluation: USAID/South Africa “Systems Strengthening for Better HIV/TB Outcomes” Project (2012-2017) P a g e | 6 Figure 6. Attribution of Changes in Service Delivery to DSP Efforts, by Province Most DOH respondents (61%) believe DSPs have helped to increase coverage to key populations, with Anova, MatCH, WRHI, and RTC cited as the DSPs most involved in this area. Some of the ways DSPs have contributed to this include: skills development programs designed to reduce discrimination and ensure youth- and key population- friendly services; and technical support for treatment strategies such as, adherence clubs for youth, condom distribution initiatives, campaigns and outreach via Community-Based Organisations (CBOs). Challenges around Strengthening Services Delivery The major service delivery challenge to fully implementing the HIV and TB program centres around Human Resources for Health (HRH). Staff shortages are a major constraint due to: a DOH moratorium on hiring staff, high nurse to patient ratios resulting in ex cessive workloads, limited allocation of resources to support implementation of newly -introduced DOH mandates and initiatives, and vacancies in leadership positions. The vacancies in leadership positions were largely filled by acting personnel, sometimes over extended periods of time, who are saddled with the responsibility of the position, without the level of authority or benefits. Despite tracking and tracing initiatives, retaining ART-patients also remains problematic because of patient-level (e.g. resistance to provide accurate personal information) as well as health system factors (e.g. lack of DOH resources earmarked for tracking and tracing of defaulters). Other identified bottlenecks included the lack of planning for adequate infrastructure and equipment as well as shortages of TB drugs and HIV test kits. Within the DSPs’ sphere of influence, they addressed these challenges through various ways:  Developing management and leadership skills for facility managers, and sub - district/district management teams to improve planning and resource allocation,  Implementing DSD models where DSP technical and support staff are seconded to DOH to support service delivery,  Technical assistance to improve patient flow and reduce waiting times, e.g. a scheduling system for patients and integration of HIV and other chronic conditions, and.  Introducing and/or supporting innovative PHC facility decongestion strategies, External Evaluation: USAID/South Africa “Systems Strengthening for Better HIV/TB Outcomes” Project (2012-2017) P a g e | 7 including decanting of stable ART patients, community adherence support groups and CCMDD models. Leadership, Management, and Governance (LMG) Leadership and governance is the most complex yet critical building block of the health system. A key component of DSP support to DOH has been around strengthening leadership, management, and governance at provincial, district and sub-district levels. Most Significant Changes in LMG Respondents cited the most significant changes in leadership, management, and governance since 2014 as per Table 9 below. Table 9. Most Significant Changes in LMG Areas of Significant Change Reasons for Change, as Noted by DOH Respondents 1. Increased commitment and engagement by DOH leaders and managers to HIV and TB and their integration into the management of other chronic conditions  Launching the Ideal Clinic initiative 2. Increased recognition of HIV/TB program implementation as an integral part of the health system.  DSP support for implementing the NDOH’s Integrated Chronic Disease Model and Primary Care 101.  Launching the 2016 Adherence Guidelines for HIV, TB and other NCDs  NDOH adopting the UNAIDS 90- 90-90 targets  Adopting the World Health ‘Universal Test and Treat’ Guidelines in 2016 3. Improved management skills, ownership and accountability, especially at facility level.  DSPs provided official management training and supplemented training with onsite mentoring. 4. More focused and strategic planning including the district planning processes.  Increased use of program data for planning, resource allocation and program management.  Better collaboration between DOH and DSPs. Role of District Support Partners in Changes in LMG Overall, 35% of DOH respondents attribute the above-cited changes to the efforts of DSPs; however, this masks variations in perceptions among the various categories of respondents (Figure 7), with more DOH respondents at district level crediting the DSP than DOH respondents at other levels. Interestingly, DSP respondents, particularly at National/Head and District offices, generally believe they have played a greater role in the above LMG changes than do DOH respondents. More credit for the LMG changes is given to certain DSPs (FPD, WRHI, and ANOVA) for the LMG changes (Figure 8). Likewise, DOH respondents in lower resourced provinces (LP and NW) were more likely to credit the LMG improvements to the DSPs compared to other provinces (Figure 9). It should be noted that large numbers of respondents, particularly in the Western Cape and Eastern Cape, indicated that they did not know enough about the DSPs’ efforts in this area to comment. Challenges around Strengthening LMG The main LMG challenges faced by the DSPs can be grouped into the following four broad categories: 1. Management/Leadership Structure, Capability, and Functions: The GOSA leadership and management structure, according to respondents, is weak due to the high External Evaluation: USAID/South Africa “Systems Strengthening for Better HIV/TB Outcomes” Project (2012-2017) P a g e | 8 Figure 7. Attribution of Changes in LMG to DSP Efforts (DOH respondents only) Figure 8. Attribution of Changes in LMG to DSP Efforts, by Partner Figure 9. Attribution of Changes in LMG to DSP Efforts, by Province External Evaluation: USAID/South Africa “Systems Strengthening for Better HIV/TB Outcomes” Project (2012-2017) P a g e | 9 number of positions that remain vacant or are occupied by acting personnel who do not have the level of authority to act in their positions. In addition, the high turnover rate within the DOH District Health Management Team (DHMT) causes instability at lo wer levels, thus further weakening the organisational structure. This situation is exacerbated by the lack of communication and coordination and priorities of DOH structures, i.e. the provincial, district, sub-district and local municipality DOH structures. Additionally, despite DSP management training, respondents reported weak DOH management skills leading to ineffective and inefficient resource allocation. These factors limit the capacity for the DOH to fulfil its stewardship function. 2. An HR Culture that Resists Ownership and Lacks Accountability: Personnel dynamics within DOH poses a major challenge to the efficient and effective implementation of strategies, policies and guidelines. DOH staff motivation and commitment is generally poor and there is a general lack of ownership and accountability. An ineffective Human Resource (HR) performance management system which rarely rewards/recognises good performance and allows poor performers to remain employed without undergoing appropriate performance m anagement creates additional discordance. Furthermore, an organisational “blame culture” constrains open communication and leads to resistant to change. Lastly, several respondents noted that when DSP staff are seconded to facility level to support servi ces delivery (e.g. Direct Services Delivery or seconded staff), an over-reliance quickly develops on the DSP staff to do the HIV and TB work while designated DOH staff leave to attend to other tasks at the facility. 3. DOH Human Capital Management: The issue of human capital management has been an ongoing challenge for the DOH for several years. With the moratorium on hiring staff, DOH has been faced with chronic staff shortages and high turnover rates, causing facility managers, many of whom are professional nurses, to spend increasing amounts of time providing clinical support and paying less attention to their management responsibilities. Where staff are in place, they receive insufficient support/supervision from higher DOH levels. Notably, some DSP respondents reported that a DSP￾conducted analysis of Workload Indicators of Staffing Needs (WISN) revealed that existing DOH staff are allocated inefficiently, and that staff shortages are less of a problem than believed. 4. DSP-DOH collaboration: The relationship between DSPs and DOH has evolved over the past few years from initial DOH resistance/caution (during PEPFAR’s first pivot away from direct service delivery towards the HSS/CB model) to the present case of active involvement. In fact, the pendulum might have swung too far. As noted by many respondents, there appears to be an over-reliance on PEPFAR partners by DOH for many aspects of program implementation, including monitoring and evaluation (M&E). Part of this is fostered by the DOH’s “unrealistic expectations” of the DSPs’ mandate: DSPs are often seen as “miracle workers,” able to solve major and minor emergency issues. However, some DOH counterparts have not fully embraced DSP support, citing lack of transparency around the DSPs’ level of funding as a challenge to an open and truly collaborative relationship. Also cited as a challenge was the issue of competing DIP and PEPFAR targets (which are, in some cases, up to three times that of the DIP). While many of these challenges are beyond the control of DSPs, DSPs continue to engage their DOH counterparts to address and/or influence resolution of these challenges. Some ways they have done this are as follows: 1. Management/Leadership Structure, Capability, and Functions. DSPs rely on continuous engagement and collaboration across DOH levels and structures through: a. Participating in coordinating forums at provincial, district, sub-district, and ward External Evaluation: USAID/South Africa “Systems Strengthening for Better HIV/TB Outcomes” Project (2012-2017) P a g e | 10 levels which allows for coordination and communication between levels. These forums provide a platform to raise and resolve key issues, e.g. the various AIDS Fora operating at different DOH levels; b. Supporting relevant levels in data quality and use to manage and plan for performance; c. Assisting and guiding DOH in the efficient and effective use of resources; and d. Providing intensive technical support at facility-level in planning; data management and use; pharmaceutical supply management and developing; and implementing Quality Assurance (QA) / Quality Improvement Plans (QIPs). 2. HR Culture that Resists Ownership and Lacks Accountability: Many of the challenges noted above with regards to DOH staff attitude and motivation are beyond the control of DSPs. However, where these attitudes are due to lack of knowledge (e.g. new policies and initiatives), skills or confidence, DSPs provide targeted training, mentoring and guidance. DSPs have supported DOH WISN analyses, using the results to advocate for changes through continuous engagement with the relevant DOH levels. Furthermore, DSPs have provided management training and mentorship to the various levels of DOH management. Management is further supported during the interrogation of performance data for planning, either short term QIPs or medium-term DIPs and District Health Plans (DHPs). DOH performance is actively tracked and supported. This has resulted in greater ownership of and accountability for performance at facility, sub-district and district-levels of the DOH. 3. DOH Human Capital Management: DOH’s HR challenges are systemic in nature and beyond the DSPs’ control. These challenges have become even more visible since the adoption of the 90-90-90 goals and the related need for an accelerated pace to reach greater volumes in order to meet these targets. In response, DSPs, with PEPFAR approval, have provided additional support to DOH through deploying roving teams and seconding staff for direct service delivery to mentor, monitor, and assist facility staff to deliver and administer and manage key services. Management training and mento rship has also been provided to Facility Managers and other DOH managers in an effort to address critical capacity gaps. Several DSPs regularly organise seminars and symposia on leadership and relevant topics. 4. DSP-DOH engagement: Over the years, there has been increasing collaboration between DSPs and DOH. All DSPs reported having MOUs with their respective DOH counterparts around the HSS/CB programs. DSPs are invited to be part of Technical Working Groups and provide input in improving DOH policies and guidelines. They provide support in the roll-out of DOH initiatives such as the 90-90-90 goals and Universal Test and Treat (UTT). DSPs try to manage expectations through continuous engagement with the various DOH levels. Joint planning meetings, especially with the advent of District Implementation Plans, as well as monthly and/or quarterly review meetings between DSPs and DOH serve to further clarify roles and responsibilities and enhance accountability amongst those involved. However, there is still a feeling among DOH respondents that there needs to be more transparency about the scope of work that DSPs have been entrusted with by PEPFAR (terms of reference, budget, work -plans and reports) as this would go a long way towards managing expectations an d fostering effective collaboration. District Health Planning (including Implementation Planning) The District Health System is the fundamental unit for South Africa’s health system since 1995 and is the main mechanism for delivering comprehensive package of primary health care services. In 2003, guidelines were developed for District Health Management Teams External Evaluation: USAID/South Africa “Systems Strengthening for Better HIV/TB Outcomes” Project (2012-2017) P a g e | 11 (DHMTs) to develop sound District Health Plans (DHPs) which in turn feed into the provinces’ 3-year strategic plans and rolling annual plans. In the past few years, the DHPs have been complemented with District Implementation Plans (DIPs) which, although primarily focused on HIV and TB, seek to integrate HIV/TB with other key health programs, e.g. Maternal and Child Health. As envisioned, the DIPs include input from all relevant actors including DOH, non-DOH, and non-governmental partners, and outline the targets, interventions and activities to be undertaken as well as responsible entities. Monthly and quarterly DIP review meetings are recommended to monitor progress towards the DIP targets and identify any corrective action required. Most Significant Changes in District Health and Implementation Planning The most significant changes in District Health and Implementation Planning cited by KII respondents reflect changes in both the planning process as well as the content of the plans, as depicted in Table 10 below. Table 10. Most Significant Changes in District Level Planning (DHPs and DIPs) Areas of Significant Change Reasons for Change, as Noted by DOH Respondents 1. More participatory district health planning with increased involvement of partners and sub￾district DOH.  DSPs are invited by DOH to actively participate and may, on occasion, facilitate district planning sessions.  DSPs support and coordinate planning inputs from PHC facility level to sub-district and district levels of the DOH. 2. Planning is now data-driven.  DSPs participate in the monthly and quarterly district, sub-district and facility performance reviews to identify bottlenecks and threats to implementation. 3. Plans increasingly aim for efficiency  DSPs have supported a more outcomes-based approach to planning. 4. Integration of TB and HIV programming/planning.  TB and HIV data elements are better aligned to allow for a more ‘horizontal’ program.  DOH adopted the 5 pillars, i.e. prevention, case finding, adherence, treatment and care as well as HSS) to integrate HIV and TB programs. 5. Planning focused on achievement of the UNAIDS 90-90-90 targets  Increased budget allocation for HIV/TB programs. 6. Continuous improvement between the various stages or phases of the DIP from Phases 1, 2 to currently, Phase 3.  DSPs have supported a more structured approach to planning.  DSPs support the setting of clear performance indicators.  More integration and cohesion between different ‘vertical’ programs. 7. Improved alignment between the DOH’s Annual Performance Plan, the District Health Plan, and the District Implementation Plan.  DSPs facilitate better coordination between the various planning processes. 8. DOH staff have improved awareness/understanding of the DHP and DIP processes.  DOH staff have been trained on the DHP and are encouraged to engage with the plan.  DSPs have been trained on the National Indicator Datasets 9. Increased ownership of district health plans.  DOH appointed additional staff to support the DHP/DIP processes, e.g. a public health specialist and individuals from the Local Health Authority. Role of District Support Partners in Changes in District Planning Processes DSPs have assisted the DOH with both DHPs and DIPs, but are perceived to have been more influential in improving and strengthening DIPs. This may be because the DOH considers DHPs to be more strategic and not within the purview of implementation partners. More DOH respondents (>50%) credited DSPs with improvements in DIPs than with improvements in DHPs (only 30%). External Evaluation: USAID/South Africa “Systems Strengthening for Better HIV/TB Outcomes” Project (2012-2017) P a g e | 12 District and sub district DOH respondents were more likely to credit DSPs with positive improvements in district planning than DOH respondents from other levels (Figure 10 and Figure 11), mainly because most (roughly half) DOH provincial and facility respondents, and all the national respondents, didn’t know about DSPs’ role in strengthening district planning. Figure 10. Attribution of Changes in DHPs to DSP Efforts (DOH respondents only) Figure 11. Attribution of Changes in DIPs to DSP Efforts (DOH respondents only) Many DOH and DSP respondents referred to the increasingly participatory nature of district health planning – a “more participatory” and “bottoms up” planning process which seeks input from all levels – but in order to for planning to satisfy a bottoms-up approach, input from PHC facility level should be actively sought. However, yet at the health facility level, most respondents noted that they are not involved in the DHP planning process; the only plan they produce or have input into is the Facility Operational P lan. It is worth noting that almost two thirds of facility respondents (63%) felt they did not know enough to comment, further contradicting the notion that district plans have indeed adopted a “bottoms up” approach. DSPs are particularly active in DIP planning processes, where they model and encourage the External Evaluation: USAID/South Africa “Systems Strengthening for Better HIV/TB Outcomes” Project (2012-2017) P a g e | 13 use of data for decision-making and for monitoring the achievement of DIP targets. Some DSP partners serve a secretariat role for the monthly program/progress review meetings. DSPs also work closely with facilities in the preparation of Facility operational plans based on targets set out in the DIPs. Although joint planning between DOH and DSPs has significantly improved due to DIPs, respondents cited that there is still a need for more strategic plan ning, collaboration and alignment between DOH and DSPs, particularly around the differences between PEPFAR and DOH targets. In unpacking DOH responses, it becomes clear that not all DSPs are credited with strengthening district planning – WRHI and FPD stand out for strengthening district planning (Figure 12 and Figure 13), especially DIPs. In disaggregating the findings by province (Figure 14 and Figure 15), NW province stands out from all other provinces for crediting WRHI with improvements in district planning. Figure 12. Attribution of Changes in DHPs to DSP Efforts by DSP Figure 13. Attribution of Changes in DIPs to DSP Efforts by DSP External Evaluation: USAID/South Africa “Systems Strengthening for Better HIV/TB Outcomes” Project (2012-2017) P a g e | 14 Figure 14. Attribution of Changes in DHPs to DSP Efforts by Province Figure 15. Attribution of Changes in DIPs to DSP Efforts by Province Challenges around Strengthening District Planning The main challenge with the DHP process is the limited time set aside to formulate the plans. This, in turn, limits the level of discourse and subsequent alignment across the various program areas, as the process ends up being pushed through for the purpo se of complying with deadlines. The main challenge with DIPs is one of sustainability. So far, DSPs have been driving the process although some partners are starting to shift responsibility and ownership of this process to DOH managers. In summary, while DOH undertook much of the district strategic planning, DSPs contributed significantly to the design and monitoring of district implementation plans. Laboratory and Pharmaceutical Systems Strengthening Access to essential medical products and diagnostics of assured quality, safety and efficacy and cost-effectiveness is another health system building block that was a focus of the project. External Evaluation: USAID/South Africa “Systems Strengthening for Better HIV/TB Outcomes” Project (2012-2017) P a g e | 15 Most Significant Changes in Pharmaceutical Systems According to most DOH respondents, the most significantly changes attributed to the DSPs relate to supporting the implementation of several key National Department of Health (NDOH) policies/guidelines. These include:  The NDOH’s Universal Test and Treat strategy (2016) 9. DOH respondents credited the DSPs for their support of the differentiated care facility decongestion strategies that ultimately improved service delivery and decreased patient waiting times.  The NDOH’s Adherence Guidelines for HIV, TB and Non-Communicable Diseases (NCDs)10.  The NDOH’s Advisory11 (January 2016) regarding the use of Fixed Dose Combination drugs in place of single agents as contributing towards improved patient outcomes in terms of adherence and viral suppression rates. Table 11 below outlines the areas of most significant change in pharmaceutical systems as perceived by DOH respondents as well as reasons for these changes. Table 11. Changes in Pharmaceutical Systems Areas of Significant Change Reasons for Change, as Noted by DOH Respondents 1. Strengthened pharmaceutical supply management systems  Improved pharmaceutical information management including the routine use of the Stock Visibility Solution for reporting and monitoring of consumption data as well as re-distribution of excess stock between facilities.  The introduction of Rx Solutions and electronic scripting in the Free State Province 2. Decreased incidences of stock￾outs of essential TB and ART medicines  Improved accuracy of procurement of medicines as informed by improved consumption data  Direct procurement of medicines from manufacturers  DSD staff secondment and/or roving multi-disciplinary teams that support pharmaceutical supply management 3. Strengthened pharmaceutical human resources for health  Onsite mentorship by roving multi-disciplinary teams  The Pharmacy Learnership Program in WC Province.  Secondment of qualified Basic and Post Basic, as well as student, Pharmacy Assistants to PHC and other DOH facilities  Secondment of DSP Pharmacist to supervise seconded Pharmacy Assistants  DSP Rotational staff, e.g. pharmacists placed at facilities once a week 4. Achievement of ideal clinic status  Alignment to Ideal Clinic Pharmaceutical and Laboratory 5. Decongestion of health facilities  DSPs provided technical assistance to analyse, revise and adapt the supply chain to support Central Chronic Medicine Dispensing and Distribution (CCMDD) program 6. Increased art retention rates  Central Dispensing Unit where medicines are pre-packed  Fast track lanes at PHC facilities  Pharmacy Dispensing Units as alternate pick-up-points/community￾based distribution models  Community-based distribution models using a courier service, Pharmacy Direct. 9 Department of Health, South Africa (2016) N DOH circular: Implementation of the universal Test and Treat strategy for HIV positive patients and differentiated care for stable patients’ http://www.sahivsoc.org/Files/22%208%2016%20Circular%20UTT%20%20%20De congestion%20CCMT%20Director ate.pdf 10 Department of Health, South Africa (2016) ‘Adherence Guidelines for HIV, TB and NCDs’ https://www.nacosa.org.za/wp-content/uploads/2016/11/Integrated -Adherence-Guidelines-NDOH.pdf 11 Department of Health, South Africa (2016) N DOH Advisory: use of FDCs to reduce use of single -agent lamivudine tablets‘ [online]: http://www.sahivsoc.org/Files/FDC%20in%20place%20of%203TC%20updated.pdf External Evaluation: USAID/South Africa “Systems Strengthening for Better HIV/TB Outcomes” Project (2012-2017) P a g e | 16 Areas of Significant Change Reasons for Change, as Noted by DOH Respondents 7. Expansion of adherence support initiatives  Expanding community adherence clubs/groups, including youth adherence clubs  Two months multi-month scripting and dispensing at facilities and community-based distribution models Most Significant Changes in Laboratory Systems DOH respondents acknowledged that the DSPS’ most significant contribution was their support for the NDOH’s roll-out of the 2015 National Consolidated ART Guidelines12 and the 2016 Universal Test and Treat Strategy which called for targeted HIV testing and viral load monitoring for:  Monitoring HIV treatment success, or  Supporting early identification of treatment failure, and  Informing the switch to second and/or third line ART treatment regimens. While partners were unable to influence the actual analytics, they were instrumental in strengthening the Clinic-Laboratory Interface (CLI)13, i.e. the pre- and post- analytic testing processes. The CLI includes the completion of laboratory request forms, specimen identification, phlebotomy, sample handling and transportation to the laboratory. Most DOH respondents credited the DSPs with reducing the lab turnaround time to 24-48 hours and, at most, 72 hours. In addition, DSPs were acknowledged for their support towards the NHLS’ roll-out of GeneXpert machines, recommended by the World Health Organisation14 to diagnose Multi-Drug (MDR-TB), eXtreme Drug Resistant TB and HIV and TB co￾infections (Table 12). Table 12. Most Significant Changes in Laboratory Systems Areas of Significant Change Reasons for Change, as Noted by DOH Respondents 1. Improved understanding/knowledge of pathology and laboratory processes.  Onsite mentoring by DSP roving clinical teams. 2. Strengthened pre-analytic phase of the CLI  Test Ordering. Through onsite mentoring and technical assistance to reduce the number of inappropriate/excessive/miss-timed orders  Patient/Specimen Identification. By ensuring adequate availability and accurate completion of clinical stationery to reduce the number and frequency of wrong patient/wrong specimens/erroneous patient or specimen errors  Specimen Collection. By supporting the implementation of the 2015 NHLS Handbook15 (Standard Operating Procedures) to ensure appropriate/consistent specimen type, volume or application to testing surface or chamber  Specimen Evaluation. To ensure attributes compromising patient ID/collection quality are recognised 3. Strengthened post-analytic phase of the CLI  Reporting Formatting. By supporting and/or following up to ensure NHLS reports contain accurate units and reference intervals as well as to resolve human transcription errors.  Critical Value Reporting. Through clinical support, either through 12 Department of Health, South Africa (2015) ‘New Department of Health National Consolidated ART Gu idelines’ http://www.sahivsoc.org/Files/ART%20Guidelines%2015052015.pdf 13 Strategic Evaluation, Advisory and Development Consulting (2010) ‘Integrated Systems Analysis of Clinic - Laboratory Interface’ Available at: http://www.sead.co.za/downloads/clinic -part-a.pdf 14 World Health Organization (2011) ‘Rapid Implementation of the Xpert MTB/RIF diagnostic test’ Availab le at: http://apps.who.int/iris/bitstream/10665/44593/1/9789241501569_eng.pdf 15 National Health Laboratory Service (2015) ‘Standard Operating Procedure: NHLS Handbook’ Available at: http://www.health.uct.ac.za/sites/default/files/image_tool/images/116/documents/NHLS_Handbook_2015.pdf External Evaluation: USAID/South Africa “Systems Strengthening for Better HIV/TB Outcomes” Project (2012-2017) P a g e | 17 Areas of Significant Change Reasons for Change, as Noted by DOH Respondents DSP staff secondment or roving clinical teams, to ensure that critical values are recognised and brought to the attention of appropriate health practitioners.  Other Result Reporting. By supporting NHLS’ roll-out of the SMS printers and Lab Track software to improve overall turnaround time and to avoid delayed or lost to retrieval results  Recording Results. Through back-capturing of test results into patient files as well as on Tier.net. Role of District Support Partners in Changes in Laboratory and Pharmaceutical System Strengthening Overall, 43% of DOH respondents credit DSP efforts for improvements in laboratory and pharmaceutical systems (Figure 16), most of whom (47%) were at the Health Facility level and Sub-District level (48%). Respondents from higher DOH management levels (national and provincial) attributed less credit to DSPs for the improvements cited above, mainly because of a lack of knowledge around the DSPs’ contributions. Figure 16. Attribution of Changes to Laboratory and Pharmaceutical Supply Systems to DSP Efforts (DOH respondents only) Although DOH respondents have varying levels of awareness around partner contributions to lab and pharmaceutical changes, FPD working in Gauteng, Mpumalanga and Limpopo provinces as well as Anova, working in Gauteng and Limpopo provinces, are credited most for the improvements in laboratory and pharmaceutical in their districts in those provinces (Figure 17). In other words, most DOH respondents in Gauteng and Limpopo provinces credited DSPs for improvements in laboratory and pharmaceutical systems in their provinces (Figure 18). External Evaluation: USAID/South Africa “Systems Strengthening for Better HIV/TB Outcomes” Project (2012-2017) P a g e | 18 Figure 17. Attribution of Changes in Lab and Pharmacy to DSP Efforts by Partner Figure 18. Attribution of Changes in Lab and Pharmacy Supply Systems to DSP Efforts by Province Challenges around Strengthening Lab and Pharmaceutical Systems Table 13 below provides a comprehensive list of challenges mentioned by DOH respondents along with DSP solutions introduced to remove implementation bottlenecks. Table 13. Challenges and DSP Solutions related to Lab and Pharmaceutical Systems Identified Challenges Solutions introduced by DSPs Pharmaceutical Systems The inability of the DOH to absorb newly trained and appointed Pharmacy Assistants due to the DOH’s moratorium on hiring. Ongoing engagement with PDOH on budget requirements with the provisional commitment the line item will be included in provincial budget. Incidences of essential drug stock-outs.  As an interim strategy, facilities are encouraged by DSPs to communicate with each other to share/re-distribute stock-on-hand while emergency orders are placed. 0% 25% 50% 75% 100% EC FS GP KZN LP MP NW WC National Pharmacy/Lab - Attribution of District Support Partner's efforts to Change by Province (DOH respondents only) Most of the change (60% and above) Equal effort in bringing the change (50-50) Don’t Know External Evaluation: USAID/South Africa “Systems Strengthening for Better HIV/TB Outcomes” Project (2012-2017) P a g e | 19 Identified Challenges Solutions introduced by DSPs Pharmaceutical Systems  Direct procurement by facilities from manufacturers  Poor pharmaceutical information and supply management by Enrolled Nurses at facility level including inadequate:  Appropriate use of stock cards  Management of the dispensary, and,  Balancing of pharmacy stock  Use of Stock Visibility Solution  DSPs seconded Pharmacy Assistants and similar cadres to PHC facilities.  DSPs supported the Provincial DOH to develop and implement SOPs for pharmaceutical information management as well as rational drug utilisation.  Quality Improvement strategies based on SIMS assessments.  DSP participation in Pharmacovigilance Committees.  Training across all levels on the Stock Visibility Solution.  Implementation of Rx Solutions for electronic scripting. High patient volumes with resultant long waiting times and congested facilities.  DSPs have supported:  Central Chronic Dispensing and Distribution models aimed at decongesting facilities to reduce patient waiting times and improve accessibility of drugs,  Initiatives involving decanting of stable patient to community-based adherence clubs. Laboratory Systems Poor specimen collection, identification and results reporting to patients. DSPs have appointed Lab Advisors to provide training as well as onsite technical assistance and mentorship. High attrition rate of trained laboratory staff, e.g. medical technicians DSPs are now providing ongoing training.  Delayed turnaround times:  Hospitals do not have onsite laboratories and samples are, therefore, sent to a centralized lab some distance away;  Delayed results reporting to facilities  DSPs are attempting to strengthen the CLI using a tool that checks whether facilities are meeting the NHLS Handbook SOP.  DSPs have supported the implementation of Lab Track software that allows service delivery point-level practitioners to access the NHLS database to obtain results.  DSPs have supported the roll-out of NHLS’ SMS printers by ensuring adequate supply of printer paper. Lab results are not captured into patient files and/or Tier.net timeously. DSPs have seconded staff to support back- and ongoing capturing of lab results into patient files and on Tier.net The management of Multi-Drug Resistant TB (MDR-TB), eXtremely Drug Resistant TB (XDR￾TB) and HIV/TB co-infection remains a challenge. DSPs have supported NHLSs roll-out of their GeneXpert machines by strengthening pre- and post- analytic processes. Health Workforce Strengthening Most Significant Changes in Health Workforce Strengthening 16 Health workface strengthening is the health system building block where DSP contributions are most acknowledged and recognised. DOH respondents cited the most significant changes around health workforce strengthening as developing confidence and skills at DOH district and facility levels in data management, service delivery and management/planning (Table 14). Table 14. Most Significant Changes in Health Workforce Areas Of Significant Change Reasons for Change, as Noted by DOH Respondents 16 Additional analyses of the DSPs’ approach to Training and Mentoring is presented in the discussion around “DSP Approaches to DOH Training/ Mentoring ” beginning on page 29. External Evaluation: USAID/South Africa “Systems Strengthening for Better HIV/TB Outcomes” Project (2012-2017) P a g e | 20 1. Greater competence and more confidence among health workers and health managers 2. Enhanced quality of service delivery (fewer mistakes) 3. Expanded services delivery due to more trained health workers 4. Improved management and planning  Training and mentoring (both clinical and management) have strengthened skills, knowledge, confidence, and attitudes  NIMART training in particular has allowed for task shifting and an increase in the number of nurses able to initiate patients on ART  Tier.net training/rollout and other data management support has contributed to improved patient management  Staff secondments / Direct Services Delivery (DSD) assisted in reaching more patients and clearing the backlog of data entry Role of District Support Partners in Changes in Heath Workforce Strengthening Across all levels of the health system, but most especially at facility level, DOH respondents credit DSPs for strengthening the Health Workforce (Figure 19). This is consistent with the DSPs’ focus on building capacity around services delivery at facility level, particularly for achieving the 90-90-90 goals. Most of this support, according to DOH respondents, centred on Nurse Initiated Management of Antiretroviral Therapy (NIMART) and data management training. Several respondents mentioned management and leadership training; however, strengthen DOH management capacity was inconsistent within and across DSPs. There is little difference between DSPs in terms of DOH attribution for health workforce strengthening (Figure 20). But DOH respondents in low resource provinces (e.g. NW, LP) were more likely to attribute changes in health workforce strengthening to DSPs’ efforts (Figure 21). Figure 19. Attribution of Changes in Health Workforce Strengthening to DSP Efforts (DOH respondents only) External Evaluation: USAID/South Africa “Systems Strengthening for Better HIV/TB Outcomes” Project (2012-2017) P a g e | 21 Figure 20. Attribution of Workforce Strengthening Changes by DSP Figure 21. Attribution of Changes to Health Workforce Strengthening by Province Challenges around Health Workforce Strengthening Issues around the DOH’s management structure and capacity limit its ability to fulfil its stewardship function. In assessing health worker strengthening, several challenges were cited:  High workload and staff shortages. The workload is anticipated to increase as the country aims to achieve its 90-90-90 targets. Staff shortages are made worse by the DOH moratorium on appointing new staff.  High staff turnover rate. There is an exceedingly high turnover of DOH staff, with most leaving for higher salaries and benefits in the private and other sectors. This is a particular challenge with data capturers since DOH uses Expanded Public Works Program (EPWP) data capturers who are employed only on annual contracts. Therefore, despite training provided to EPWP data capturers, there is a loss of institutional understanding and memory when their contracts end.  Unsustainability of improvements. Due to the DOH’s moratorium as well as a general 0% 25% 50% 75% 100% EC FS GP KZN LP MP NW WC National Staff/Workforce - Attribution of District Support Partner's efforts to Change by Province (DOH respondents only) Most of the change (60% and above) Equal effort in bringing the change (50-50) Don’t Know External Evaluation: USAID/South Africa “Systems Strengthening for Better HIV/TB Outcomes” Project (2012-2017) P a g e | 22 lack of resource capacity, DSP-seconded staff who presently provide direct services delivery will not be absorbed or replaced by DOH. This will inevitably lead to an HR crisis in the DOH and will severely impact DOH’s ability to continue delivering quality and safe health care.  Inadequate performance management of staff. As discussed in the LMG challenges section (starting on page 7), despite an HR performance management system in place in the DOH, DOH staff are rarely acknowledged for high levels of performance, nor are they are placed under performance management or disciplinary proceedings for lack of performance. Health Management Information Systems A well-functioning Health Management Information System (HMIS) is a key building block for Health Systems Strengthening and for good reason. According to the WHO, “The best measure of a health system’s performance is its impact on health outcomes” (Margaret Chan – Everybody’s Business, WHO 2007), and the way to monitor this performan ce is through a well-functioning HMIS. PEPFAR’s HIV/AIDS partnership has included sustained support to improve health information systems at all levels of the health system. Most Significant Changes in Health Information Systems Strengthening South Africa’s HIV/AIDS program has seen significant changes in its Health Information Management Systems (HMIS) over the past few years, particularly in the introduction of new/improved platforms (Tier.net) which make it easier to manage and track patient s. DSPs have been very active in supporting the rollout of this system, through training, mentoring, and hands-on support to get the system operational in facilities. Recently, there has been a major push by DOH for all PHC facilities to be on Phase 6 of Tier.net and the DSPs have been working to support this shift. DSPs have also been actively supporting the successful implementation of District Health Information Software (DHIS) version 2. Other significant changes around HIS include technical suppor t in data management at various levels within the health system, which has resulted in better quality data, and an increased appreciation and understanding of the importance of data for program planning and management, which also improved the use of data for performance measurement and informed decision making. At the facility level, DSPs roving M&E teams provided assistance for back capturing data through secondment of data capturers or by training and mentoring EPWPs in data capturing. Table 15. Most Significant Changes in Health Information Systems Areas of Significant Change Reasons for Change, as Noted by DOH Respondents 1. Better patient management and tracking 2. Better district level information for planning  Introduction of new and improved platforms (e.g. Tier.net)  Implementation of DHIS2 3. Reduced backlog of data capturing  M&E staff secondment (data capturers)  Training of EPWPs in data capturing 4. Better Data Quality  Roving M&E teams that provide M&E technical support 5. More Data Use  M&E training / mentoring around interpreting data and use of data for target setting Role of District Support Partners in Changes in HIS Strengthening DOH respondents at district, sub-district and health facility levels attribute most of the External Evaluation: USAID/South Africa “Systems Strengthening for Better HIV/TB Outcomes” Project (2012-2017) P a g e | 23 changes in HMIS to DSP Efforts (Figure 22). It must be noted, however, that this attribution varies across DSPs (Figure 23), with WRHI, FPD and Broad Reach (BHC) credited the most for HIS improvements in their districts. WRHI is recognised for supporting data interrogation and data use for strategic planning and decision making while BHC is recognised for the use of their VANTAGE system that draws on and analyses DHIS data for informed planning. In looking at provincial differences (Figure 24), more DOH respondents from the low resource provinces of EC and NW credited HIS changes to DSPs than respondents in other provinces. Figure 22. Attribution of HIS changes to DSP Efforts (DOH respondents only) Figure 23. Attribution of HIS Changes to DSP Efforts, by Partner External Evaluation: USAID/South Africa “Systems Strengthening for Better HIV/TB Outcomes” Project (2012-2017) P a g e | 24 Figure 24. Attribution of HIS Changes to DSP Efforts, by Province In addition to the support for rolling-out the Tier.net system, DSPs trained, mentored and technically supported/assisted various DOH levels in data management and data quality – particularly around interpreting and analysing programme data to inform decision making. At facility level, DSPs assisted with clearing the backlog of uncaptured data through deploying roving M&E teams, seconding data capturers, or by training and mentoring EPWPs in data capturing. Challenges around HIS Strengthening The key challenges around Health Information Systems mentioned by respondents were:  Data quality issues. Despite major improvements in data quality, challenges remain. These centre on the correct use of clinical stationery by clinicians as well as their ability to capture service delivery using the correct data points. The consistent use of clinical stationery also remains a challenge, although DSPs have assisted with printing to ensure supply. It is increasingly important to ensure data verification checks throughout the data management system to continue improving data quality.  Unsustainable staff shortages. The DOH’s moratorium on appointment of new staff has seriously constrained HIS functions at service delivery and sub-district and district levels. Although the DOH has employed EPWP data capturers, they are only on a year contract, it is unable to appoint additional long-term data management staff, or to absorb DSP seconded staff. Thus, the improvements made by the HSS/CB program are likely to be lost once the program ends.  Equipment and connectivity issues. The lack of Information Technology (IT) equipment and connectivity remain a challenge at PHC facility level. These challenges have led to backlogs in data capturing resulting in delayed or incomplete data which, in turn, affects the availability of key information for decision making purposes and effective patient management. Among the indicators affected by these data management issues are the ‘lost to follow up’ and ‘viral load suppression rates’. As the DOH endeavours to reach its 90-90-90 targets, it will be increasingly important that these two indicators are accurately monitored and measured. Special Issue around HIS DOH respondents at national level noted that there are discrepancies between the HIV-TB External Evaluation: USAID/South Africa “Systems Strengthening for Better HIV/TB Outcomes” Project (2012-2017) P a g e | 25 indicator values captured in Tier.net compared to the values reported by DSPs directly to PEPFAR (which were often greater). Several other district and provincial DOH respondents also raised the issue about the lack of transparency in DSP HIS reporting . While we were unable to explain the discrepancies due to lack of access to the master datasets (PEPFAR, Tier.net, and DHIS), this does raise questions about the accuracy of the data reported by both parties, the alignment of datasets for data -driven planning and decisions, and calls into question the level of DSP-DOH collaboration and trust. DSP Contribution to the 90-90-90 Goals Table 16 summarises the DSPs’ contributions to the six HSS building blocks, by the 90-90- 90 goals. Table 16. DSP Contribution HSS Building Block HSS/CB Interventions HIV Testing (1st 90) Health Workforce CHCW, WBOTs HIS Back-Capture LMG Revised HCT Guidelines (2015) Facility Manager’s Commitment DHP/DIP Adoption of UNAIDS 90-90-90 Strategic Goals Lab and Pharmaceutical Lab TAT Service Delivery Linkages to CBOs QA for Test Kits Key Population(KP) and Priority Population (PP) Friendly Services Linkage to ART (2nd 90) Health Workforce NIMART HIS Tier.net (Pre-ART Tracking) LMG FM’s Support for NIMART UTT Strategy (2016) DHP/DIP Focus on 90-90-90 Lab and Pharmaceutical PSM - SVS, VAN Service Delivery Linkage to CBOs Linkage Officers, CHCWs KP and PP Friendly Services Retention/Viral Suppression (3rd 90) Health Workforce Compliance with Treatment Protocols (Interpreting and Acting Upon VL Results) HIS Defaulter Tracking/Tracing LMG Adherence Guidelines for HIV, TB & NCDs (2016) DHP/DIP Focus on 90-90-90 goals Lab and Pharmaceutical Back-Capture of VL Results Supply Chain for alternate service delivery models Service Delivery Adherence Strategies: • Youth and Key Population Services • Community Adherence Groups • Alternate modes of ART delivery, e.g. fast track lanes • Community-based ART distribution models, CCMDD Multi-Month Scripting/Dispensing (2-months) External Evaluation: USAID/South Africa “Systems Strengthening for Better HIV/TB Outcomes” Project (2012-2017) P a g e | 26 In addition to the above, DSPs played active roles in the integration of TB and HIV services as highlighted in Figure 25. Figure 25. Existing DSP TB/HIV Integration Activities DSP APPROACHES TO DOH TRAINING/ MENTORING As presented in Figure 2 at the beginning of this report, most partners invested heavily in adding additional staff through direct services delivery or seconded staff , followed by mentoring or roving clinical teams, and training. The initial design of the partners’ HSS/CB programs was usually based on some sort of formative research (e.g. baseline assessment, skills audit, or WISN analysis). In KZN, MatCH was already providing technical assistance to the DOH and had established an in-depth understanding of skills gaps. DOH views on DSP Training / Mentoring Approaches and Effects on Health System Functioning The vast majority of DOH respondents (75%) expressed high levels of satisfaction with the DSPs’ HSS/CB program. In close collaboration with the DOH, and particularly Facility Managers and the Regional Training Centres (in provinces where they exist), DOH staff were identified for official training. DSP Training and Skills Development Plans were closely aligned to the DOH’s structure and processes and most DOH respondents agreed that the partners’ HSS/CB approach considered the local context in the design of the HSS/CB interventions. Most agreed that the partners’ HSS/CB program maximised learning (Figure 26), citing examples such as increased staff professionalism, confidence, and new knowledge in areas like NIMART and M&E. And nearly all DOH respondents (90%) felt that they could apply what they learned to their jobs (Figure 27). The length and frequency of training was considered reasonable by most DOH respondents (80%), but some felt that onsite training or mentoring was more useful than off -site training, which took too much time away from work. Health Workforce •TB/HIV Proportion on ART Treatment (correlated with training) •Mentorship ICDM and PC101 protocols and processes Service Delivery •Patient Flow and Process Maps •Implementation Guidance on HIV/TB Rx Protocols ‘Cheat Sheets’ LMG •Launch of Ideal Clinic Initiative •ICDM Strategy •Primary Care 101 HIS •For TB, Shift from ETR to TB Module on Tier.net Lab/Pharmacy •Lab TAT •Supply Chain for Test Kits/Screening Equipment and TB and HIV drugs •Roll-out of GenXpert Laboratory Equipment Key critical skills identified in the partners’ HSS/CB programmes  NIMART training  health information management  pharmaceutical and laboratory supply management  Leadership and management External Evaluation: USAID/South Africa “Systems Strengthening for Better HIV/TB Outcomes” Project (2012-2017) P a g e | 27 Figure 26. Percent of DOH Respondents Who Agreed that the DSP Program Maximised Learning Figure 27. Percentage of DOH Respondents Who Felt They Could Apply Learning To institutionalise newly-gained knowledge and to further support trained clinical staff, DSPs employed various capacity building strategies, such as:  Ongoing onsite mentoring through the deployment of multi-disciplinary roving teams;  Patient Case Management strategies. Particular patients were identified by clinic staff and brought to the attention of DSPs. These patients were used as cases for additional mentoring and training.  Patient File Audits. Roving DSP teams would conduct random patient file audits to ensure treatment protocols were being adhered to. If discrepancies were uncovered, and hoc onsite training was provided to ensure quality of care.  Treatment Protocol ‘Cheat’ Sheets. DSPs developed and implemented simplified cheat sheets for various treatment protocols. These were clearly visible in the consulting rooms of certain clinics and DOH respondents felt that this provided additional guidance.  Secondment of temporary DSP staff. Temporary staff were frequently seconded to facilities to support either service delivery and/or administrative tasks including data 0% 25% 50% 75% 100% Anova BHC FPD Kethimpilo Match Multiple RTC WHRI DOH participants able to apply learning? Percent of DOH Respondents who "agreed" Strongly agree Agree External Evaluation: USAID/South Africa “Systems Strengthening for Better HIV/TB Outcomes” Project (2012-2017) P a g e | 28 management and capturing. DOH respondents were generally positive about the quality and quantity of DSP technical support staff (Figure 28) for the HSS/CB activities, especially at sub-district and district levels. But again, provincial and national DOH respondents were unable to comment on this, given their relative lack of engagement with the DSP training program. Figure 28. Percent of DOH Respondents Who Were Satisfied with the Number and Competence of Deployed DSP Staff DOH respondents were very positive of the DSPs’ training and mentoring , citing DSPs as excellent trouble shooters who are able to identify and remove implementation obstacles. The perceived effects of training and mentoring were noted in the following areas :  DOH internal monitoring systems and processes. Most DOH respondents (83%) ranked DSP training highly for strengthening internal management and monitoring, and this was especially true of DOH respondents in NW province (92%) who had been assisted by WRHI.  Quality and safety of service delivery. DSP training had the effect of improving the confidence of clinical staff, and as such linkage to treatment rates improved. Clinical staff also reported being more confident when initiating children onto ART and were able, with support from the DSP, to more effectively switch clients between first and second line ARV treatment protocols. Likewise, there is a perception that this significantly improved the interpretation and use of viral load results as part of client clinical management. Through training and mentoring of pharmaceutical staff, pharmaceutical supply management improved and there were fewer incidents of stock￾outs and/or expired stock.  Program planning, monitoring, evaluation and reporting. 75% of DOH respondents stated that the DSPs strengthened information systems and overall monitoring of HIV/TB programs, with TIER.NET being singled out as the best example of strengthened information systems. DSPs also supported program performance reviews and helped Facility Managers develop short term Quality Improvement Plans to ramp up performance against particular indicators. Support was also provided by DSPs during DHPs and DIPs in the future design of HIV/TB programs.  Increased linkage to treatment rates. With the introduction of UTT in 2016, positive patients who earlier did not meet eligibility criteria for ART are now being tracked with External Evaluation: USAID/South Africa “Systems Strengthening for Better HIV/TB Outcomes” Project (2012-2017) P a g e | 29 the either TA or DSD support from DSPs. Under the DSD approach, DSPs have seconded Linkage Counsellors to clinics with high numbers of ‘pre -ART’ patients.  Improved adherence counselling and support. DSPs have implemented context￾specific adherence support strategies through training and mentoring of DOH staff implementing adherence support.  Decanting of stable ART patients. Once a patient is successfully retained in treatment to be considered stable, they are decanted to community-based adherence support groups that are also supported by DSPs. This is a known strategy for decongesting PHC facilities, freeing up clinic staff to attend to new clients.  Alternative modes of ART distribution. DOH respondents described DSPs as being exceedingly innovative in this space from rolling out automated Pharmacy Dispensing Units (PDUs) to CCMDD models. This is another known strategy to decongest clinics allowing clinicians to attend to new cases.  Fast track dispensing. Certain DSPs have launched appointment systems at PHC facilities allowing medicines to be pre-packaged for stable ART patients. These patients queue in the ‘fast track’ lane and are able to move through the facility much faster.  Improved viral suppression rates. Ultimately, the above-mentioned strategies have resulted in improved viral suppression rates. Here, DSPs have assisted pre - and post￾laboratory to ensure that the correct clinical stationery is used and that the viral load result is documented in patient files.  Increased coverage of Key Populations (KPs). DSPs provided training to clinical and non-clinical staff to ensure KP-friendly service delivery. They are also engaging with and supporting Community-Based Organisations (CBOs) to improve coverage. DOH respondents considered Anova the highest contributor to key populations.  Defaulter tracking/tracing. DSPs share best practices and are supporting DOH to develop innovative strategies to track and trace treatment defaulters so as to link them to the health system.  Improved youth-friendly strategies. Several DOH respondents felt that the DSPs have supported and mentored DOH staff to develop context-specific and appropriate youth-friendly strategies.  Achievement of Ideal Clinic status. Other DOH respondents felt that when DOH launched the Ideal Clinic initiative, DSPs played a critical role in supporting implementation by providing the necessary skills and knowledge to fill existing DOH gaps. As such, an increasing number of facilities are achieving Ideal Clinic status. Lastly, it is important to note that the USAID policy to not prescribe specific technical assistance interventions might have its logic and politics, but it comes with serious drawbacks. A majority of respondent responses illustrates that:  Training was important but only in tandem with the onsite mentoring and supervision.  Training was most effective in specific technical areas, ensuring, for example NIMART, TIER.NET.  Training was less appreciated than mentoring and direct services delivery. Staff appreciated roving clinical teams and staff secondment and lamented the future without these.  Sustainability of HSS/CB interventions is unlikely mostly due to the lack of DOH External Evaluation: USAID/South Africa “Systems Strengthening for Better HIV/TB Outcomes” Project (2012-2017) P a g e | 30 HR capacity.  Training worked best when negotiated with facility staff and when given in the afternoons after busy clinic mornings.  Training worked best when linked to specific needs e.g. initiation, data capturing and pharmacy. DSP Training / Mentoring Approaches and Effects on Patient Outcomes Quantified HSS/CB activities (as measured by FTE) were analysed against the trends of the 29 performance indicators that were the focus of the evaluation (Figure 29). We calculated the Pearson’s correlation coefficients between each of the 29 performance indicators and the HSS/CB activities to determine which HSS/CB activities were more correlated with improved indicator performance. As shown in Figure 29, better indicator performance is most associated with HSS/CB interventions that involve adding staff or mentoring/supervising DOH staff: • Adding Staff = Direct Services Delivery, Temporary Seconded staff • Mentoring / Supervising Staff = Mentoring, Supervision, Roving Clinical Teams This is particularly true for the performance indicators focused on clinical outputs and outcomes (e.g. HCT, ART initiation, TB patients on ART, circumcisions) and patient level information systems (e.g. use of Tier.net). Other performance indicators show less relationship with HSS/CB activities. Overall, training is only weakly associated with better indicator performance. These quantitative findings are supported by the views of most DOH respondents (75%) who agree that the DSPs’ HSS/CB interventions contributed to improved HIV-related patient outcomes (Figure 30). External Evaluation: USAID/South Africa “Systems Strengthening for Better HIV/TB Outcomes” Project (2012-2017) P a g e | 31 Figure 29. Pearson’s Correlation Coefficient for the 29 Performance Indicators and HSS/CB Activities 4 17 17 Green Coloured cells represent correlation coefficients of 0.70 or great er External Evaluation: USAID/South Africa “Systems Strengthening for Better HIV/TB Outcomes” Project (2012-2017) P a g e | 32 Figure 30. Percent of DOH Respondents Who Agree that DSPs Contributed to Improving Patient Outcomes GOLD STANDARD FOR TA AND SUPPORT TO DOH MANAGEMENT LEVELS In the earlier discussion on Leadership, Management, and Governance (LMG) on page 4, DOH respondents credited DSPs with improving LMG, particularly around increased DOH commitment and engagement to HIV and TB; improved management skills, ownership and accountability; and increased use of program data for planning, resource al location and program management. In contrast, many DSP respondents, particularly at national/head office levels, expressed disappointment in the lack of sustainable skills transfer at DOH management level, especially when DOH managers selected for management training did not complete the training without explanation. Partners acknowledged the busy work schedules of DOH managers as a limiting factor, as well as the high turnover rate among management staff which threatens the sustainability of skills transfer. DOH often appoints ‘acting’ personnel who are uncertain and therefore, unable, to fully execute their role and responsibilities. Ultimately, the DSPs role in HSS/CB was to strengthen the health system so as to ensure high quality and safe HIV/TB-related service delivery. With the most recent PEPFAR pivot to the 90-90-90 strategic goals, DSPs emphasised a hybrid TA/DSD model to support service delivery as PEPFAR measures them by key service delivery indicators. As such, leadership and management strengthening was not a DSP principal focus in this project. Using the correlation between key performance indicators and HSS/CB (in Figure 29) does not provide sufficient guidance around the best approaches for management level strengthening, as only 3 of the 29 performance indicators are related to management level (“Costed district MMC plan”, “In each sub-district support at least one clinic to achieve ideal clinic status; core standards everywhere”, and “appropriately documented minutes of quarterly data review meetings”), but data was missing from all partners on these indicators. Nevertheless, given the correlation results for the other indicators, we surmise that training alone would not be very effective, and that interventions that emphasise mentoring / supervising or adding staff would be better for strengthening at management level. But this would take considerable diplomacy on the part of the DSP, as national, provincial and district managers might not be receptive to technical assistance or mentoring. In summary, the findings suggest the need for a more robust and consistent approach across the project cycle of planning, implementation, monitoring and evaluation at all levels of the health care system. 0% 25% 50% 75% 100% Anova BHC FPD Kethimpilo Match Multiple RTC WHRI DSP support has contributed to improving patient outcomes? Percent of DOH Respondents who "agreed" Strongly agree Agree External Evaluation: USAID/South Africa “Systems Strengthening for Better HIV/TB Outcomes” Project (2012-2017) P a g e | 33 Q2. To What Extent, and How, did the Partners Improve Patient Outcomes at Health Facilities? Evaluation Question 2 has numerous sub-questions around how the partners’ HSS/CB activities contributed to improving patient outcomes in health facilities; and whether key indicators (namely estimated treatment gap, retention rates, and viral load suppression rates) improved. The following discussion first describes the trends of the 29 indicators listed in the SOW. Thereafter, we present findings around the Government’s satisfaction with the partners’ HSS/CB inputs, and the gold standard for Technical Assistance and Support at DOH health facilities. TRENDS IN PERFORMANCE INDICATORS (FY2014-FY2016) Khulisa obtained reported values directly from the DSPs for the 29 PEPFAR and DOH performance indicators (also referred to as indicators of success) included in the SOW for this evaluation (see page 51, Annex 1). Figure 31 presenting the overall results and trends across all partners shows that indicator performance was mixed over the three years. However, it must be noted that there was missing information for certain indicators because DSPs indicated they were not required to track the indicators for either PEPFAR or DOH. Indicators that show steady progress relate mostly to initiation on ART, PMTCT, reductions of TB defaulter rates, and use of Tier.net at facility level. This is consistent with the discussion above around the partners’ contribution to HSS building blocks which showed the partners’ emphasis on workforce strengthening, services delivery, and information systems (per Figure 3). Interestingly, most performance indicators show a shift in FY2015 (either po sitive or negative), reflecting the PEPFAR pivot away from only technical support to the achievement of the 90-90-90 strategic goals through direct services delivery. A breakdown of indicator performance by DSP is presented in Annex 6. Over the 3 years, and across all partners, Figure 31 shows that:  ART enrolment rates (indicator 5) increased by 26%,  Retention rates (indicator 6) increased by 5% (from 73% to 77% ), and  Viral load suppression rates (indicator 7) remained basically unchanged. While these individual results are generally positive, the emphasis on enrolment has not been matched by an equal emphasis on retention, or viral load suppression. One issue faced by DSPs around retention concern the difficulty in tracking and tracing highly mobile populations when there are no unique patient identification numbers . HOW HAVE THE DSPS’ HSS/CB ACTIVITIES CONTRIBUTED TO IMPROVING HIV-RELATED PATIENT OUTCOMES INDICATORS? As previously discussed, DSP training and mentoring efforts are correlated with improvements in certain performance indicators (Figure 29 above), especially clinical output and outcome indicators (e.g. HCT, ART initiation, TB patients on ART, circumcisions) and indicators around the use of patient level information systems (e.g. use of Tier.net). Most DOH respondents credit DSP skills development (clinical and non-clinical) with improving the quality and safety of service delivery, patient management, and overall patient outcomes. External Evaluation: USAID/South Africa “Systems Strengthening for Better HIV/TB Outcomes” Project (2012-2017) P a g e | 34 Notable HSS/CB approaches that are associated with improved patient outcomes include:  DSP onsite quality assurance activities, e.g. regular patient file audits, are successful in identifying additional training needs. DSPs provided onsite tutoring to address gaps in understanding based on patient file reviews. External Evaluation: USAID/South Africa “Systems Strengthening for Better HIV/TB Outcomes” Project (2012-2017) P a g e | 35 Figure 31. Trends in Performance Indicators 2014-20167 18 18 Green highlighted cells show positive performance Ind Domain Intervention Strategy no. Indicator 2014 2015 2016 Trend % Change since 2014 HIV HIV Treatment 1 Cohort analysis for 12, 24, 36 months 0.591 0.72 0.65 10% HIV HIV Treatment 2 Estimated district need for treatment met (males and females) 3 2 2 -18% HIV HIV Treatment 3 Estimated district need for treatment met (children) 3 3 2 -15% HIV HIV Treatment 4 Number of adults and children currently receiving antiretroviral therapy 607 693 471 168 504 159 -17% HIV HIV Treatment 5 Number of adults and children newly enrolled on ART 260 051 302 150 327 218 26% HIV HIV Treatment 6 Percentage of adults and children known to be alive and on treatment 12 months after initiation of antiretroviral 0.73 0.74 0.77 5% HIV HIV Treatment 7 Proportion of viral load tests with undetectable viral load (1000copies/ml) 0.83 0.84 0.81 -1% TB TB Treatment 8 Proportion of TB screening and IPT for PLHIV and HTS for all presumptive and diagnosed/confirmed TB patients 0.92 0.92 0.87 -6% TB TB Treatment 9 Sputum conversion rates 0.67 0.71 0.64 -5% TB TB Treatment 10 TB success rates 0.74 0.78 0.63 -15% TB TB Treatment 11 TB/HIV proportion on ART treatment 0.81 0.86 0.89 10% HIV Prevention PMTCT 12 Percentage of HIV-positive women who received antiretroviral to reduce risk of mothers -to-child transmission during pregnancy and delivery 0.91 0.90 0.95 5% HIV Prevention Distribution of male and female condoms 13 Costed district condom distribution plan. 0 0 0 HIV Prevention Distribution of male and female condoms 14 Male condom distribution rate 0.29 0.42 0.42 44% HIV Prevention Male medical circumcision 15 Costed district MMC plan 0 0 0 HIV Prevention Male medical circumcision 16 Number of circumcisions performed 192 256 180 540 186 515 -3% HIV Prevention PICT 17 Number of Individuals who received HIV testing and Counseling services for HIV and received their test result 4 133 766 3 934 549 4 375 875 6% MCH MCH EBF 18 Proportion infants on EBF at 14 weeks 0.49 0.49 0.38 -23% MCH MCH FP 19 Couple year protection rate; Proportion of clients on implanon 0.40 0.49 0.45 14% Health Systems Strengthening HSS - DHIS use 20 Appropriately documented minutes of quarterly data review meetings 0 0 0 Health Systems Strengthening HSS - Improving 3-Tier M&E 21 Proportion of all facilities that export monthly signed off ART data to DHIS 0.54 0.82 0.96 79% Health Systems Strengthening HSS - Improving 3-Tier M&E 22 Proportion of Tier 2 facilities reporting appropriately signed off cohort data quarterly 0.45 0.74 0.95 109% Health Systems Strengthening HSS - Support DHP 23 DHP incorporating PEPFAR DSP and other NGO plans; has targets and relevant methods to achieve all the priorities on this list 0 0 0 Health Systems Strengthening HSS - Support ETR-net 24 Proportion of facilities with up to date ETR-net data appropriately signed off and exported to DHIS 0.51 0.55 0.61 19% Health Systems Strengthening HSS - Support Ideal Clinic 25 In each sub-district support at least one clinic to achieve ideal clinic status; core standards everywhere 0 0 0 Health Systems Strengthening HSS - Supporting nurses 26 Written monthly reports of supervision visits to clinics. 61.2 61.56 58.46 -4% Other Capacity Building 27 No. persons trained 6 318 4 242 9 100 44% TB TB Treatment 28 TB defaulter rates 0.05 0.05 0.04 -25% HIV Prevention PMTCT 29 Early infant transmission rate 0.68 0.65 0.61 -10% Indicators expected to INCREASE over time Indicators expected to DECREASE over time All Partners External Evaluation: USAID/South Africa “Systems Strengthening for Better HIV/TB Outcomes” Project (2012-2017) P a g e | 36  WRHI’s unique ‘Train the Trainer’ approach to skills development, wherein a WRHI staff member was seconded to the Regional Training Centre, was particularly successful. This approach was further supplemented by ongoing onsite mentoring.  Adding staff to facilities (DSD), seconding temporary staff and roving clinical teams are strongest in the training, placement and support of data capturers and pharmacy assistants. The former contributed to improved results monitoring and the use of data in planning. The latter contributed to improved access to me dication and adherence.  Developing efficient and effective patient flow charts and process maps resulting in decreased waiting times and facilitated several facilities towards their Ideal Clinic status.  Supporting the use of strategic information. On a monthly and quarterly basis, Facility and District Managers are supported for analysing and using program data to inform planning and resource allocation. DSP support for the use of SVS also resulted in fewer incidents of stock-outs and stock wastage. Laboratory results and, in particular, viral load results are now being used to clinically manage ART patients. Therefore, DSPs have supported interpretation and management of viral load failure.  Reducing patient waiting times. This allows clinical staff to attend to more critical and/or complicated patients. In addition to the above general approaches, DSPs were acknowledged for specific accomplishments related to patient level results as listed in the table below. ANOVA  Expanded access to Key Population-friendly services by training DOH staff at PHC level  Supported increased uptake of HIV and TB-related services by key and priority populations by implementing an interactive online magazine, Health4Men BHC  Supported the use of graphic representations of program data to help improve patient management and program performance  Increased viral suppression rates through tracking and tracing of lost-to-follow-up cases FPD  Supported a successful PMTCT program in Mpumalanga where the early infant positivity rate is now below 2%, as well as its pharmacovigilance strategies to improve patient outcomes Kheth’Impilo  Increased viral load suppression rates by training to improve blood sampling practices and supporting better laboratory turnaround times  Capturing of results on Tier.net MatCH  Improving TB cure rates since TB is detected and managed earlier  Better managing co-infected patients according to treatment protocols RTC  Improved adherence support and counselling through an Adherence Facilitator which has increased the uptake of ARVs.  Improved tracking and tracing of defaulters through drawing a list of defaulters or patients due for tests from Tier.net and sharing this list with the Ward Based Outreach Teams (WBOTs) for active tracing and follow-up.  Improved program monitoring, evaluation and reporting such that the sub district DOH met their DIP targets for HCT, ARV initiation and viral load suppression (including children). WRHI  Decreased mortality rate with more patients being initiated onto, and retained on, ART  Successful Prevention of Mother to Child Transmission (PMTCT) as evidenced by the very low early infant positivity rate External Evaluation: USAID/South Africa “Systems Strengthening for Better HIV/TB Outcomes” Project (2012-2017) P a g e | 37 GOLD STANDARD FOR TECHNICAL ASSISTANCE AND SUPPORT AT HEALTH FACILITIES Numerous DOH respondents suggested that the most effective skills development approach for health facilities involves DSPs providing official classroom-based training supplemented with ongoing, onsite mentoring and guidance. The evaluation findings support this suggestion – consistent and relevant mentoring and supervision directed toward specific, measurable areas of work – such as NIMART, medical male circumcision, medical supply management, HMIS platforms (e.g. TIER.net), HIV/TB integration and VL suppression – leads to improved coverage, prevention, testing, treatment initiation and adherence, reduced loss to follow up rates, service integration and improved VL suppression. Given this as well as the other findings presented earlier, the proposed gold standard for technical assistance and support at health facilities consists of 5 steps as depicted in Figure 32: Figure 32. Suggested Gold Standard Approach for TA & Support at Health Facilities Baseline Assessments / Formative Research to ID Capacity Gaps •WISN Analysis •Skills Audits Joint DOH-DSP planning around HSS/CB goals •Measurable HSS/CB targets •Transparent budgets / resource allocation •Signed MOU to document joint goals and objectives Selection of Staff for Training and CB •in collaboration with DOH •Considers Context, Absorbtive Capacity, Ability to Apply Knowledge Coninous Mentoring / Coaching of Trained Staff •on-going onsite mentoring Patient File and Case Management Audits •ID additional capacity needs •Provide ad hoc onsite training and mentoring External Evaluation: USAID/South Africa “Systems Strengthening for Better HIV/TB Outcomes” Project (2012-2017) P a g e | 38 Q3. How did the Program Design Influence the Achievement of Results? The design of the program shifted (or “pivoted”) twice over the 2012-2016 period. The first pivot was in 2013 when USAID decided to assign partners to priority districts/sub districts to provide comprehensive technical support and capacity building for the HIV/TB program. Each partner thus became a “District Support Partner” providing technical assistance. Staff that were employed by the partners before the DSP assignments were absorbed into the DOH workforce during this pivot. The second pivot came in 2015 when the South African Government and PEPFAR adopted the 90-90-90 strategic goals as the guiding framework for the HIV/TB program. Recognising that staff shortages were the greatest bottleneck to implementing 90 -90-90, USAID agreed that District Support Partners would return to staff secondments/direct services delivery as a capacity building strategy for the program. STRENGTHS IN PROGRAM DESIGN The main strengths in the program’s design related to the achievement of results were the following:  A programmatic focus on HSS/CB at district and facility levels, especially around strengthening the health workforce, services delivery, and health information systems for patient tracking helped to expand coverage of clinical services delivery.  Direct services delivery and staff secondments compensating for DOH staff shortages that otherwise would have constrained the achievement of increased clinical performance indicators  The comprehensive nature of the services to be delivered – in terms of the 6 HSS building blocks and the full cascade of HIV-TB services from prevention to treatment – ensures a more systematic approach to achieving results.  The assignment of one key PEPFAR partner (except for COJ district in GP) helped to streamline the support given to district DOHs and assisted the DOH in simpler coordination of PEPFAR support. GAPS IN PROGRAM DESIGN A major gap in program design was the lack of a strategic focus around engaging with, and capacity building of, provincial DOH (PDOH) management who are gatekeepers to successful program implementation. Of all the DSPs, only WRHI explicitly included engagement with PDOH in its implementation plan. Provincial DOH makes strategic decisions around resource allocation for the DIPs and DHPs, but the funds, HR, and material resources are not always adequately ear marked for fully implementing those plans. If the program and DSPs were to establish closer relationships with PDOH management, with the goal of better managing and allocating resources, the achievement in patient outcomes might have been greater because PDOH would have filled more resource gaps rather than depending on the DSPs to do so. This is especially relevant given DSP WISN analyses that revealed inefficient allocation of DOH staff, rather than just staff shortages. In districts where multiple DSPs simultaneously implement the HSS program (such as in the City of Johannesburg), several DOH respondents suggested that USAID identify and appoint one DSP as an overall managing and coordinating partner to ensure coverage without duplication of effort. Others felt that a DOH Partner Manager (or equivalent) role should External Evaluation: USAID/South Africa “Systems Strengthening for Better HIV/TB Outcomes” Project (2012-2017) P a g e | 39 be created and filled to support a managed and coordinated approach to HSS. Finally, some felt that rather than mandating strengthening of all HSS building blocks to all DSPs, USAID should recognise and work with DSP strengths or expertise. Thereafter, the approach developed by the ‘expert’ DSP should be adopted and rolled out across all DSPs. For example, RTC has an innovative automated Pharmacy Dispensing Units (PDUs) which could support pharmacy and laboratory supply management strengthening initiatives in all districts, not just the district it supports. Rolling out effective HSS/CB interventions (including innovations) would help standardise the support provided to DOH across all supported districts and make it easier to develop/monitor HSS/CB indicators and compare progress across regions.19 ADDITIONAL INVESTMENT PRIORITIES FOR REACHING 90-90-90 One investment required to reach 90-90-90 is to emphasise more strengthening of DOH management at provincial, district, and sub-district levels to properly manage, coordinate and supervise service delivery 20. Across the health system, there is a need for a strategic focus on building management and leadership capacity, including HR performance management for reaching and managing significantly higher volumes of clients upon which the 90-90-90 targets depend. Despite the use of alternative strategies such as the ideal clinic, decanting, and alternative ART distribution strategies, current patient to DOH staff ratios remain excessive and PHC facilities remain congested. A comprehensive HRH plan, informed by the WISN analyses, should be developed and actively implemented by DOH supported by the necessary funds. Importantly, many DOH staff interviewed in this evaluation requested support for additional infrastructure, including the building of new clinics in high burden areas. Although this is outside PEPFAR’s current mandate, expanded infrastructure, including spaces designated for community adherence clubs, is critical to reaching 90-90-90. The number of HIV assigned consulting rooms at PHC facilities also needs to increase significantly so as to maintain confidentiality of HIV-related service delivery. Numerous DSP and DOH respondents acknowledged that the facility-based model of treatment is not adequate to support the 90-90-90 goals. Using routine health screening as an entry point, treatment models need to shift towards a community outreach approach so as to reach the hard to find population groups. 21 Furthermore, treatment approaches need to be more dynamic so as to target known and emerging hotspots. However, such a community-oriented approach, will require more and different resources including vehicles, point-of-care laboratory equipment, as well as routine health screening equipment. Currently, NDOH policies and guidelines limit HIV confirmation and initiation onto ART to PHC facility-level, in particular to Enrolled Nurses. Once patients are considered stable, however, they are allowed to receive their ART through alternative modes of ART distribution. Also, current guidelines allow for multi-month scripting but not multi-month dispensing resulting in patients having to pick up their ART on a monthly basis. In order to reach 90-90-90, DSPs and other stakeholders will need to use their strategic positions to advocate for changes in the national policy/guidelines so as to allow for: 19 The need for standardisation of general support as a way to ensure sustainability was also identified during a consultative process between a USAID Consultant and key HIV/AIDS leaders and profession als in South Africa. Rodgers, Roxana. Trip Report. November 7-18, 2016. USAID/South Africa. 20 This finding/opinion is consistent with the findings from Roxana Rogers’ consultative process . Rodgers, Roxana. Trip Report. November 7 -18, 2016. USAID/South Africa 21 This finding is consistent with the findings from Roxana Rogers’ consultative process. One modality proposed during this consultation was a “demonstration” CHW activity which could show the effectiveness of strategically -placed and focused CHW.” Rodgers, Roxana. Trip Report. November 7-18, 2016. USAID/South Africa External Evaluation: USAID/South Africa “Systems Strengthening for Better HIV/TB Outcomes” Project (2012-2017) P a g e | 40  ART initiation by other health care cadres;  Maximising the role of Community Healthcare Workers as a key untapped resource in the HIV continuum of care;  Multi-month scripting and dispensing models for both individuals and community adherence groups (on a rotation basis). External Evaluation: USAID/South Africa “Systems Strengthening for Better HIV/TB Outcomes” Project (2012-2017) P a g e | 41 Q4. Partners’ Linkages with Other PEPFAR Prog rams to Provide Beneficiaries with Complementary Care The UNAIDS 90-90-90 strategy (2014) calls for investments in community system strengthening as essential to realising the promise of decentralised, community-based treatment delivery. South Africa’s 2016 Adherence Guidelines for HIV, TB and Non￾Communicable Diseases (NCD) 22, strategies are covered by the Integrated Chronic Disease Model (ICDM) which necessitates strong program linkages between community structures and health facilities across the HIV care cascade so as to maximise the impact of interventions. Additionally, the South Africa PEPFAR Country Operational Plan (2016) requires DSPs to actively support the 2016 adherence guidelines by fostering strong linkages between other PEPFAR programs including those operating from Community-Based (CBOs) and Faith-Based Organisations (FBOs). Across the HIV clinical cascade, these linkages may involve the following:  HIV Prevention Interventions. Linking DOH PHC-facilities with existing PEPFAR￾funded Orphan and Vulnerable Children and Youth (OVCY) programs, the Determined, Resilient, Empowered, AIDS-free, Mentored and Safe (DREAMS) as well as Voluntary Medical Male Circumcision (VMMC) initiatives to reach priority populations for HIV prevention interventions. Concurrently, PEPFAR OVCY and other programs can be used as distribution points for distributing DOH’s condoms and other preventive measures.  HIV Testing. Linking DOH WBOT and Community Health Care Worker (CHCWs) to existing PEPFAR-funded community-based testing initiatives.  Linkage to Treatment. Establishing and promoting linkages between DOH’s Buddy System and/or Peer Mentorship programs with existing OVCY and DREAMS initiatives, for example, to ensure newly diagnosed are linked to treatment.  Retention in Treatment and Viral Suppression. This may involve linking PEPFAR￾funded CBOs and FBOs as sites for community adherence clubs. Additionally, community-based staff currently employed under other PEPFAR programs, e.g. Child and Youth Care Workers currently implementing OVCY initiatives, can also be used for tracking and tracing of defaulters. PARTNERSHIPS AND LINKAGES ESTABLISHED To enhance their work in reaching targets for the various performance indicators, DSPs utilised partnerships or linkages for specific services to advance their programs and activities (Figure 33). More specifically, DSPs have established and/or strengthened partnerships most frequently related to the areas shown in Figure 34. Some DSP-specific partnerships and linkages under this project include: Community Testing Programs. District Support Partners were credited by DOH respondents for establishing community-based testing programs such as:  FPD partnered with Society for Family Health through the “New Start Initiative” to pilot a home-based testing program.  Anova piloted a home-based testing program, “Friends for Life”, with HIVSA as its community-based partner. 22 Department of Health, SA (2016) ‘Adherence Guidelines for HIV, TB and NCDs’ [online] Available at: https://www.nacosa.org.za/wp-content/uploads/2016/11/Integrated-Adherence-Guidelines-NDOH.pdf External Evaluation: USAID/South Africa “Systems Strengthening for Better HIV/TB Outcomes” Project (2012-2017) P a g e | 42 Figure 33. Strategic Areas for DSP Linkages and Partnerships  Several DSPs partnered with Orphan and Vulnerable Children (OVC) organisations, e.g. NACCW and NACOSA for community-based testing programs.  Other DSPs partnered with other PEPFAR-funded organisations for community￾based testing, e.g. Humana and Hospice Palliative Care Association of South Africa for the “Care and Support to Improve Patient Outcomes” (CASIPO) Project.  Expanding coverage of the Ward Based Outreach Teams (WBOTs), Community Caregivers (CCGs) and Community Healthcare Workers (CHCWs) cadres for outreach initiatives targeting farmworkers, industry, prisoners and per i-mining areas. In these situations, partners provided equipment, including appropriate vehicles.  DSPs partnered with private and NHI General Practitioners (GPs) as well as private hospital chains to reach those individuals, who reside in the DSP’s allott ed catchment area, but who choose to access healthcare through the private healthcare system.  Other private-public partnerships targeting taxi ranks and other known/emerging hotspots. HIV Prevention Programs. Supporting the DOH’s Integrated Access to Care and Treatment (I ACT) strategy, as per the 2016 adherence guidelines, DSPs partnered with community-based organisations to deliver HIV prevention campaigns. Examples of such partnerships include:  VMMC services were provided using mobile units.  Clinics in determined hotspots offered extended clinic hours so as to offer VMMC services.  DSPs supported soccer clubs to sensitise and mobilise young men for testing and VMMC.  Partnering with the Centre for HIV and AIDS Prevention Studies (CHAPS) Project for VMMC.  Public-private partnerships included the ‘Catch Them Young’ Project with UPower Africa as well as partnerships with the South African Clothing and Textile Workers Union (SACTWU) to conduct camps for VMMC  DSPs worked with traditional male medical circumcisers to mobilise boys in the community for VMMC. Community Testing Programs Linkages with CBOs and other PEPFAR IPs Targeted Community-Based Prevention Programs Utilizing CHCWs , WBOTs and Health Promoters Targeting Hotspots Outreach Support for DOH’s I ACT Strategy Voluntary Male Medical Circumcision Linkages with Appropriate Stakeholders Utilizing other DSP Funding Streams Community Health Care Workers and WBOTs Training using National Curriculum WBOT Database External Evaluation: USAID/South Africa “Systems Strengthening for Better HIV/TB Outcomes” Project (2012-2017) P a g e | 43  Young women-friendly services catering for ages 15 – 25 were offered on Saturday mornings or as ‘Friday Clinics’ at pre-determined PHC facilities. Community Health Care Workers and WBOTs. DSPs have provided training to CHCWs using the national curriculum. WBOTs have also undergone training, mentorship and support. At sub-district level, DSPs have shifted a paper-based WBOT system to the WBOT database. CHCWs and WBOTs are an important structure to mobilise and reach the community for testing, linkage and retention. Community-based Treatment Programs. A further component of the DOH’s I ACT strategy is to enrol ART patients into I ACT support groups at CBO level for follow up and retention in care. For example:  Linkage to Treatment Initiatives: o WRHI worked with I ACT Facilitators to ensure newly diagnosed patients were linked to support groups. o DSPs trained CCGs and other youth care worker cadres for linkage of OVCs and youth to Care and Treatment programs.  Retention to Treatment/Adherence Strategies: o Expansion of I ACT adherence clubs o DSPs were integral in implementing CCMDD models including private-public partnerships with Clicks, Shoprite, other private pharmacies and workplaces as alternate pick-up points. Figure 34. Number of Linkages/Partnerships by Type of Service Partner Number of Linkages or Partnerships by type of service VMMC CHWs, etc. Community HCT Community Treatment OVC Prevention Other GOSA Priv. Sector Grand Total Anova 1 1 1 1 1 1 1 1 8 BHC 5 5 5 5 5 5 5 5 40 FPD 4 4 4 4 4 4 4 4 32 Kheth’Impilo* - - - - - - - - - MatCH* - - - - - - - - - Multiple 4 4 4 4 4 4 4 4 32 RTC 3 3 3 3 3 3 3 3 24 WHRI 2 2 2 2 2 2 2 2 16 Grand Total 19 19 19 19 19 19 19 19 152 INNOVATIONS The DSPs’ innovative practices complement direct service delivery, promoting or strengthening the community- facility linkages and promoting better communication and collaboration between partners. These practices are linked to effectiveness (increasing coverage of services as well as extend continuum of services) and efficiency (improved coordination of partners and their resources to minimise duplication). Notably, DSPs are only able to work within the regulatory framework of the DOH, but were able to use newly launched DOH policies and strategies as leverage points for innovation. These include, amongst others:  The RSA HCT Updated Guidelines (2015) 23 recommended community-based testing 23 National Department of Health (2015) ‘RSA HCT Updated Guidelines) [online] Available at: https://www.health-e.org.za/wp-content/uploads/2015/07/HCT-Guidelines-2015.pdf External Evaluation: USAID/South Africa “Systems Strengthening for Better HIV/TB Outcomes” Project (2012-2017) P a g e | 44 strategies as well as mobile and outreach testing strategies. As su ch, DSPs were able to establish partnerships for door-to-door testing using trained CHCWs as well WBOTs. DSPs also loaned facilities gazebos and other necessary equipment to conduct testing outside facilities. For example, RTC sub-contracted with 5 CBOs for community-based testing using CHCWs.  Universal Test and Treat strategy (2016) requires all HIV positive individuals to be linked to treatment regardless of the CD4 count. In response to the launch of this national strategy, DSPs have trained, supported and mentored CHCWs and WBOTs to develop and implement tracking and tracing of all those individuals who historically did not qualify for ART. DSPs used Tier.net and/or DHIS to draw lists of individuals who have defaulted or missed appointments and hav e supported the active tracing of these individuals. Additionally, DSPs seconded Linkage Officers (or equivalent personnel) to support this initiative.  Adherence Guidelines for HIV, TB and NCDs (2016) introduced new strategies for retention including expansion of community-based adherence support clubs and alternate ART distribution models. DSPs have engaged both the private sector and CBOs to support CCMDD models. Discussions are currently underway with private retailors for the distribution of ARTs. In addition, DSPs have launched innovative Pharmacy Dispensing Units, which are entirely automated, are this is currently being piloted in two informal settlements in the City of Johannesburg. HOW TO IMPROVE LINKAGES Community-level structures, according to the DOH’s Integrated Chronic Disease Program, are a key component of health systems. Interestingly, several DOH respondents reported too many partners, some known and others unknown, working in different areas and at various levels within their regions. Additionally, several senior DOH participants felt that one of the DSPs, particularly when multiple where operating in the same region, should be identified by USAID to serve a coordinating, management and supervision role so as to ensure effective and efficient program implementation. By extension, therefore, building effective and efficient linkages and partnerships also requires coordination, supervision and management. In addition to continuous engagement between DSP and local PEPFAR - funded community-structures, this might entail:  Mapping of each CBO, FBO or other implementing partners at community level as well as their coverage and the services they provide. This will allow various stakeholders to understand, identify and address gaps in coverage a nd service provision.  Community-level structures becoming active participants in DSP and DOH DIP/DHP planning and monitoring, evaluation and reporting processes.  DSPs and other local PEPFAR partners should formalise their relationships with clear definitions of roles and responsibilities in the region.  DSPs should appoint a Community Partner Manager (or equivalent person) to serve as a point of contact and ensure a managed and coordinated approach to program implementation.  Community Partners should also attend monthly and quarterly review meetings.  All partners operating within a region should also contribute to AIDs Fora across all levels. External Evaluation: USAID/South Africa “Systems Strengthening for Better HIV/TB Outcomes” Project (2012-2017) P a g e | 45 CONCLUSIONS The recent accelerated pace of the HIV/TB program as a result of South Africa’s adoption of the 90-90-90 goals and UTT requires a commensurate increase in capacity within the health system. USAID’s focus on Health Systems Strengthening/Capacity Building (HSS/CB) has been a relatively sound strategy for improving and expanding HIV/AIDS services in South Africa. It is evident from our findings that DSPs have positively contributed to strengthening the six health system building blocks in their respective districts, but most especially the 3 building blocks of service delivery, health workforce, and information management. The degree to which the DSPs have been successful has been a function of their efforts, as well as the DOH structural and HR challenges they’ve faced, and the degree to which they’ve been able to collaborate with the various DOH levels. While there is an overall consensus that the DSPs’ efforts have indeed contributed to observed improvements, there are differences of opinion among DOH respondents regarding the magnitude of the changes. DOH respondents’ views range from those who consider DSP efforts to have significantly contributed to most of the changes observed (usually at facility, sub-district, and district levels), to those who on the opposite side of the spectrum, consider DSPs as having affected little to no change (usually at provincial and national levels). The differences in views are even more diverse when unpacked by individual DSPs and by provinces. What is clear, however, is that DOH faces numerous staffing and management challenges across the various districts and provinces which limit optimal implementation of otherwise sound policies, and maximising the support provided by the DSPs. It remains to be seen if the strategy has truly achieved the desired outcomes of improved quality of service delivery, and ultimately, improved patient outcomes. The performance of the 29 indicators that were the focus of the evaluation showed mixed results over the FY2014-FY2016 period. Indicators that show the most progress relate mostly to initiation on ART, PMTCT, reductions of TB defaulter rates, and use of Tier.net at facility level. This is consistent with the partners’ focus on workforce, services delivery, and information systems strengthening with a strong focus on initiating patients on ART. Across the 3 years, ART enrolment rates increased by 26%, but retention rates increased by only 5%, and viral load suppression rates remained basically unchanged. This points to the need for greater emphasis on differentiated models of care that better meet clients’ needs for retaining them on treatment. The most effective HSS/capacity building activities, as correlated with indicator performance, are those that involve adding staff to DOH services (i.e. secondment of staff for direct services delivery) and mentoring of DOH staff (mentoring, roving clinical teams). These were strongly associated with improvements in clinical performance indicators (e.g. HCT, ART initiation, TB patients on ART, circumcisions) and patient level information systems (e.g. use of Tier.net). Other performance indicators showed less association with HSS/CB activities. Training by itself was weakly associated with better indicator performance. PEPFAR, through the DSPs, has supported improved skills of professional nurses to initiate and manage HIV treatment, and some support for planning, management, and monitoring capacity amongst the various DOH management levels. However, the impact of this support is limited by the health system’s ability to absorb it, mainly due to the chronic shortage of staff which acts as a bottleneck to expanding services, and which often constricts existing services by leading to high turnover due to high workloads and demotivation. T his is further aggravated by an HR performance management system that does not recognise/reward good External Evaluation: USAID/South Africa “Systems Strengthening for Better HIV/TB Outcomes” Project (2012-2017) P a g e | 46 performance or penalise poor ones. Q5. Recommendations for future USAID/South Africa HIV Projects and Strategic Directions The recommendations below are made with the full knowledge that some will require the removal of obstacles before they can be implemented. Recommendations are not limited to USAID but are also directed at DSPs and the DOH. USAID 1. Engage in a transparent and open dialogue with Provincial DOHs about USAID’s mandate to, and expectations of, the partners it finances in the province. 2. Support DOH in improving overall HR Performance Management 3. Align PEPFAR and DOH planning/implementation processes and MER indicators/ processes. 4. Until the DOH has adequate numbers of M&E personnel to support the HIV/TB program at all levels, continue to support M&E to address critical capacity gaps and data backlogs. 5. Appoint a Regional Managing/ Coordinating DSP wherever multiple DSPs work in the same district or province. 6. Consider allocating HSS/CB work relative to DSP expertise rather than by geographic areas. 7. Identify key HSS/CB indicators to be reported on by DSPs when technical assistance/capacity building is the main focus of the project. 8. Encourage more substantive consultations between DOH and DSP’s during the formative (e.g. baseline) planning phase (beyond MOUs) prior to the initiation of work. This is particularly urgent at the provincial level. 9. Review the performance of districts that have “graduated” from DSP support to identify lessons learned and success/sustainability factors. DSP 1. Share USAID-funded SOWs, performance targets, and implementation plans with DOH counterparts to increase transparency and trust. 2. Establish stronger feedback loops with DOH as part of Quality Improvement including Data Quality Reports and Progress Reports 3. Emphasise leadership and management training/post-training mentorship for facility, sub-district, and district managers to address critical gaps in management skills. 4. Engage with district management teams (DHMTs) to review/plan programs in a manner similar to that undertaken in developing PEPFAR Country Operational Plans. 5. Continue actively supporting the development, implementation, and monitoring of DIPs. 6. Tailor training to address specific, evidence-based capacity gaps within DOH’s health workforce. Design and deliver training in close consultation with facilities such that it minimises the burden on service delivery (e.g. afternoon training sessions instead of full multi-day trainings as well as on-site versus off-site training). External Evaluation: USAID/South Africa “Systems Strengthening for Better HIV/TB Outcomes” Project (2012-2017) P a g e | 47 DOH 1. Improve its HR management system, upskilling and retaining DOH staff, by creating a culture where good performance is appreciated and rewarded and poor performance is not tolerated. 2. Relieve the burden on clinicians at facility level by shifting as many time-consuming, but not clinically-challenging tasks to auxiliary health workers, e.g. lay counsellors for HCT. 3. Given patient mobility, roll out a nation-wide unique patient identification system (possibly with biometric dimensions) to better track patients on ART and ensure their continued care (the Western Cape system could serve as a model). 4. Ensure that DSP direct service delivery at the facility level is not viewed as a replacement of DOH staff (break/“chilling” time), but rather as an addition which will help increase the numbers reached with HIV/TB services. 5. Make the DHP and DIP planning process more participatory and “bottoms up” with relevant support from DSPs. 6. Take greater ownership of the DIP process, given its critical role in managing progress towards key HIV and TB targets. 7. Provide more time for preparing both DHPs and DIPs thereby al lowing for more strategic thinking and optimal consultation across different health departments/areas – especially those programs where integration is desired. 8. Ensure sufficient numbers of capable data capturers at the facility level as this is the entry point into the health information system. Ensure that data captured at the facility level meets all the criteria for data quality. 9. Ensure adequate M&E capacity at DOH management levels (sub-district and above) to enhance data for decision making. External Evaluation: USAID/South Africa “Systems Strengthening for Better HIV/TB Outcomes” Project (2012-2017) P a g e | 48 EVALUATION STATEMENT OF WORK Scope of Work: External Participatory Evaluation of South Africa PEPFAR funded USAID HIV Treatment Program (District Comprehensive HIV Program Support Activity) ACTIVITY TO BE EVALUATED: Comprehensive District Based Support Model Project name: Systems Strengthening for Better HIV/TB Patient Outcomes Project Dates: October 2012 - September 2017 Agreement Value: $510,123,387 Obligated to Date: $377,985,594 Implementing Organisations: USAID Comprehensive District Support Partners Contracting Officer (CO): Camille Hasha Type of Analytic Activity (process and outcome program evaluation of USAID South Africa Care and Treatment program from 2012-2017) Process Evaluation: This aspect of the USAID program evaluation will focus on program or intervention implementation by DSPs in USAID supported districts in South Africa, including, but not limited to access to services, whether services reach the intended population, how services are delivered, client satisfaction and perceptions about needs and services, management practices. Outcome Evaluation: This aspect of the USAID program evaluation will determine if and by how much, intervention activities or services achieved their intended outcomes. It will focus on outputs and outcomes to judge program effectiveness, and will also assess program process to understand how outcomes are produced. Brief Program Description (Comprehensive District Support Model) The purpose of this program is to strengthen the Government of South Africa’s GOSA (GoSA) systems in order to improve patient outcomes and prevent HIV by supporting comprehensive clinic-based (hospitals, community health centers, and primary health care clinics) HIV-related services -district level in 27 Districts in South Africa. The comprehensive HIV district support program was launched in 2012 to strengthen GOSA systems in order to improve patient outcomes and prevent HIV by providing capacity building and supporting comprehensive clinic-based HIV-related services, as well as support goals and objectives of the South African National Strategic Plan on HIV/STI/TB 2012-16. Currently, USAID District Based Partners are providing focused technical assistance and capacity building to districts, facilities and communities on all aspects of care and treatment activities. These partners are supporting a standard service delivery package and providing technical assistance to districts, facilities and communities on core activities, including implementation of PEPFAR care and support activities, viral monitoring, clinical lab interface for appropriate patients monitoring, TB screening, early diagnosis and treatment, promoting adherence & retention, DQA/DQI, supply chain management and commodities. The comprehensive District Based Partners are currently implementing activities in the 14 districts listed below: District Support Partner Districts Provinces ANOVA City of Johannesburg Region C,D E and G Gauteng Mopani Limpopo Broadreach Alfred Nzo Eastern Cape Gert Sibande Mpumalanga Ugu KwaZulu Natal uThungulu KwaZulu Natal Foundation for Professional Tshwane Gauteng Development Nkangala Mpumalanga Capricorn Limpopo Kheth'Impilo Umgundgundlovu KwaZulu Natal Cape Town Western Cape Maternal, Adolescent and Child Health eThekwini KwaZulu Natal External Evaluation: USAID/South Africa “Systems Strengthening for Better HIV/TB Outcomes” Project (2012-2017) P a g e | 49 District Support Partner Districts Provinces Right to Care Ehlanzeni Mpumalanga City of Johannesburg Region A and B Gauteng Thabo Mofutsanyane Free State Wits Reproductive Health Dr Kenneth Kaunda North West Institute City of Johannesburg Region F Gauteng Strategic or Results Framework for the project/program/intervention The strategic objectives and results framework for the project are briefly described below: Strategic Objective 1: Improve HIV-related patient outcomes by strengthening health and patient management systems at facility, sub-district, and district levels. Illustrative activities: HEALTH SYSTEMS STRENGTHENING  Implement tracking and tracing systems to reduce loss to initiation and loss to follow-up for HIV and TB.  Collaborate and coordinate with GOSA and other PEPFAR partners on monitoring resistance  Intensify training , mentorship, and supervision to increase awareness, diagnosis, and treatment of high incidence co-infection (e.g., tuberculosis, Cryptococcal meningitis, hepatitis B and C)  Implement strategies to address specific needs of adolescent HIV positive populations  Reduce barriers for pediatric treatment in the Primary Health Care clinic setting  Identify clear strategies and target setting for elimination of MTCT transmission  Support uniform and routine prevention messages linked to service delivery throughout healthcare access points, including treatment as prevention Strategic objective 2: Build capacity of facility, sub-district, and district management systems in coordination with provinces to strengthen health systems in support of HIV-related services Illustrative activities: TRAINING AND MENTORSHIP  Conduct, as appropriate, pre-service and in-service training and mentorship for managers, doctors, nurses, data capturers, pharmacists, pharmacy assistants, counselors, CHWs, and other health care workers to support rollout of services at PHC clinics, CHCs, and district and other hospitals on: o Facility management o Provision of technical updates for facility-based staff as identified by gap analyses o Support of Nurse Initiated Management of Anti-Retroviral Therapy (NIMART) o Support of nurse mentorship for PHC clinics o Implementation of effective mentorship models o Data collection, analysis, use, and reporting Illustrative activities: HEALTH SYSTEMS STRENGTHENING  Conduct and support comprehensive population-based assessment to identify gaps and needs at sub￾district and district levels  Assist with the formulation of costed work planning at a district level with district target setting and measurable benchmarks  Conduct routine onsite reviews of all DHIS reported data  Conduct quarterly reviews of results related to QI tools with district management and fill identified gaps  Provide support, as necessary, to forecast pharmaceutical and commodity needs, order them in a timely manner, maintain appropriate stock levels, maintain appropriate storage conditions, and strengthen data management in support of pharmacy systems Strategic objective 3: Provide support for development and successful implementation of GOSA policies, External Evaluation: USAID/South Africa “Systems Strengthening for Better HIV/TB Outcomes” Project (2012-2017) P a g e | 50 guidelines, and standards for HIV-related interventions Illustrative activities:  Participate in relevant technical committees to review and update policies, guidelines, and protocols to ensure they reflect state of the art practice  Consolidate guidelines, tools, and curricula to ensure core standards of practice and intensified application of measured better practices SCOPE OF WORK A. Purpose: This is a process and outcomes based evaluation. The primary objective/purpose of this evaluation is to assess progress that has been made by the projects toward achieving set goals, objectives, expected outputs and/or outcomes. The evaluation will also assess the quality of project implementation by DSPs in the USAID supported districts, determine which approaches and activities are working (and why), and to make recommendations and develop lessons learned to inform future awards and refocusing of USAID HIV country program. Findings and recommendations from this evaluation will be used to inform the future strategic directions of USAID/SA HIV Care and Treatment related investments in the country, including ensuring maximum epidemic control and impact in supporting GOSA to achieve 90-90-90 - 90% of people tested for HIV, 90% of those eligible for treatment on treatment, with at least 90% of those on treatment virally suppressed. B. Audience The primary audience of the evaluation report will be the US Government, specifically the United States Agency for International Development (USAID/SA). The secondary audience are DSPs and appropriate government ministries. USAID, USG and DSPs will use the report and lessons learned to inform and improve the performance of future activities as USAID strategically shifts its program to high impact districts/facilities for program saturation and HIV/AIDS epidemic control. C. Applications and use The evaluation will include both the patient-centered and health system strengthened aspects of the project. The final evaluation report produced by the consultant for this project shall: (1) Use evidence from existing data and primary data collected to assess the performance of the USAID District support partners in USAID-supported districts; (2) Identify best practices and lessons learned; and (3) Make recommendations for future interventions that will enable the USAID HIV/AIDS program achieve maximum impact for HIV epidemic control in line with the new UNAIDS, PEPFAR and Department of Health (DOH) 2020 strategic directions and aspirational targets of achieving 90-90-90 targets by 2020. D. Evaluation questions The evaluation questions to be addressed are listed below. Each evaluation question must be answered in the final evaluation report, using evidence provided by systematic methods. Each question and its associated answer (with findings) will form a separate section in the evaluations report. Evaluation question #1: To what extent and how did the DSPs strengthen health systems at the District, Provincial and National systems? What is the gold standard for technical assistance and support at District, Provincial and National level? Scope of work for evaluation question 1: A. Describe partners contribution to the following health systems building blocks listed below:  Service Delivery;  District leadership and governance;  District health plans; External Evaluation: USAID/South Africa “Systems Strengthening for Better HIV/TB Outcomes” Project (2012-2017) P a g e | 51  District Implementation Plans  Laboratory and Pharmaceutical Systems Strengthening;  Health workforce; and  Health information systems. B. How well has the DSPs strengthened the capacity of DOH at each level (facility, district, provincial and national level) to plan design, implement, manage, monitor, and sustain HIV/TB programs? C. Assess the types of training/mentoring provided to DOH as part of capacity building for HIV programming at Districts and facility level. D. Assess (through plausibility analysis) whether any of these capacity building, training and mentoring efforts contributed to improving HIV-related patient outcomes at facility, and district level? Recommended data sources: Approved workplans, quarterly reports, summary of PIMS and DATIM reports, District Implementation Plans. Contractor can develop a survey instrument for partners to complete to collect some of the information for the process evaluation. Evaluation Question #2: To what extend and how did the District Support Partners improve patient outcomes at public health facilities and district hospitals. What is the gold standard for technical assistance and service delivery at PHC, CHC, district hospitals? Scope of work for evaluation question #2: A. Assess whether the activity achieved targeted results focusing on quality/quantity of outputs for this activity. B. Assess if the program helps to achieve reduction of the estimated treatment gap, increase overall retention rates, and viral load suppression rates for patients on ART. C. Finally, assess (through plausibility analysis) whether Health Systems Strengthening activities implemented by the DSPs contributed to improving HIV-related patient outcomes indicators. The Table of indicators combines indictors provided to partners from PEPFAR and NDOH. Partners were instructed in June 2013 to prioritize NDOH activities; which have related indicators in italics. Recommended data sources include DATIM, PEPFAR Panorama, DHIS for district/facility lists, partners’ Annual progress report etc. Intervention Strategy Indicators of success Suggested Sources HIV Prevention Distribution of male and female condoms Costed district condom distribution plan. DHP Male condom distribution rate DHP, DIP Male medical circumcision Costed district MMC plan DHP,DIP Number of circumcisions performed DATIM PMTCT Early infant transmission rate DATIM, Tier.net Percentage of HIV-positive women who received antiretroviral to reduce risk of mothers -to-child transmission during pregnancy and delivery DHIS, DATIM, PIMS PICT Number of Individuals who received HIV testing and Counseling services for HIV and received their test result DHIS, DATIM, DIP HIV Treatment Estimated district need for treatment met (males and females) DIP Estimated district need for treatment met (children) DIP Cohort analysis for 12, 24, 36 months DHIS, DATIM, PIMS Number of adults and children newly enrolled on ART DHIS, DATIM, PIMS Number of adults and children currently receiving antiretroviral therapy DHIS, DATIM, PIMS Percentage of adults and children known to be alive and on treatment 12 months after initiation of antiretroviral DHIS, DATIM, PIMS Proportion of viral load tests with undetectable viral load (1000copies/ml) DHIS, DATIM, PIMS TB Treatment TB success rates; TB defaulter rates; Sputum conversion rates DHIS, DATIM, PIMS External Evaluation: USAID/South Africa “Systems Strengthening for Better HIV/TB Outcomes” Project (2012-2017) P a g e | 5 2 Proportion of TB screening and IPT for PLHIV and HTS for all presumptive and diagnosed/confirmed TB patients DHIS, DATIM, PIMS TB/HIV proportion on ART treatment DATIM, Tier.net MCH Support Early Breast Feeding Proportion infants on EBF at 14 weeks District Health Office Family Planning Couple year protection rate; Proportion of clients on implanon DHIS Health Systems Strengthening Improving 3-Tier M&E Proportion of Tier 2 facilities reporting appropriately signed off cohort data quarterly DHP, DIP Proportion of all facilities that export monthly signed off ART data to DHIS Tier.Net report Supporting nurses Written monthly reports of supervision visits to clinics. District Health Office, Partners Reports Support ETR-net Proportion of facilities with up to date ETR-net data appropriately signed off and exported to DHIS District Health Office, Partners Reports Support DHP DHP incorporating PEPFAR DSP and other NGO plans; has targets and relevant methods to achieve all the priorities on this list DIP Support Ideal Clinic In each sub-district support at least one clinic to achieve ideal clinic status; core standards everywhere HSS Reports DHIS use Appropriately documented minutes of quarterly data review meetings HAST Managers Contractor can develop a survey instrument for partners to complete for outcome evaluation. Recommended data sources: DATIM, PEPFAR Panorama, DHIS for district/facility lists, partners’ Annual progress report etc. Contractor can develop a survey instrument for partners to complete for collecting some of the information for the outcome evaluation. Evaluation question #3: What were the strengths in the program design for facilitating achievement of results? What were the gaps in the program design which hindered performance at the community, facility, district, provincial and national levels? And where are the gaps in the current program or what areas require additional investment to reach 90-90-90? Recommended data sources- Same as stated above for evaluation questions #1-2 Evaluation question #4: How well did the DSP link with other PEPFAR in-country programs to provide beneficiaries with HIV prevention, care and OVC services? How can this be improved? And what are the innovative practices that should be integrated across the entire in-country PEPFAR portfolio? Recommended data sources- Same as stated above for evaluation questions #1-2 Evaluation question #5: What recommendations need to be factored into USAID-SA HIV future project design and strategic directions that will enable the HIV program to provide broad range of high quality support for diagnosis, linkages to care, treatment initiation, maintenance and viral suppression, treatment adherence and retention in care, and supportive systems in line with the 90-90-90 PEPFAR strategic thinking? E. Methods To answer the evaluation questions, USAID expects contractors/evaluators to apply a non- experimental design approach that employs both quantitative and qualitative methods. Methods may include reviews of project documents, key informant interviews, and extensive use of routinely collected program data. The evaluators will have access to routinely collected program data, facilitated by USAID. If appropriate and feasible, evaluators will also collect additional primary interview data to get the most objective evaluation possible. The Evaluation should consider both process and outcome indicators. Process Evaluation: This aspect of the USAID program evaluation will focus on program or intervention implementation by DSPs in USAID supported districts in South Africa, including but not limited to access to External Evaluation: USAID/South Africa “Systems Strengthening for Better HIV/TB Outcomes” Project (2012-2017) P a g e | 53 services, whether services reach the intended population, how services are delivered, client satisfaction and perceptions about needs and services, management practices. Outcome Evaluation: This aspect of the USAID program evaluation will determine whether intervention activities or services achieved their intended outcomes. It will focus on outputs and outcomes to judge program effectiveness, and will also assess program process to understand how outcomes are produced. Performance measurements and trend analysis should be done for some of the performance indicators listed in the evaluation questions and matrix. Document Review (list of documents recommended for review) Evaluation source document: The following source documents/systems should be considered as evidenced based tools for evaluation and performance measurements of the DSPs programs / interventions: 1. Desk Review: The evaluation team is expected to conduct an in-depth review of background documents and relevant materials prior to arrival in the country. The following documents will be provided by USAID/SA: i. Program Descriptions ii. PEPFAR semiannual and annual reports iii. Partners work plans iv. Partners PEPFAR DATIM/PIMS reports v. Selected project research and technical reports, publications, and tools (where applicable) 2. Activity and progress reports: DSPs submit quarterly, semi-annual and annual activity progress reports with updates on supported programs, activities and interventions and outlining completed deliverables. 3. Site Improving Monitoring Strategy (SIMS) reports: The standard SIMS tools were introduced in late 2014 and has been used to routinely monitor DSPs program performance in core program areas/elements at facility, community and above facility/site level. The SIMS reports for facilities and districts already visited by USAID￾SA country staff can be found on the USAID SIMS database. Secondary analysis of existing data Data Source (existing dataset) Description of data Recommended analysis I. PEPFAR DATIM/MER indicators dataset DSPs report quarterly MER indicators on PIMS. This is used by the USAID country program/office to track measurable indicators submitted by all DSP to monitor measurable performance indicators and outcomes submitted that are part of the PEPFAR￾SA Monitoring, Evaluation and Reporting (MER) indicators which the DSPs reported against targets, on a quarterly basis, through the South Africa Partners Information Management System (DATIM). Cascade analysis 3. DHIS (district M&E and program performance dataset) The DHIS in South Africa is the acronym used to describe both the District Health Information System in the broad sense, and the District Health Information Software (used to manage the data collected by this system). The emphasis on District in both terms was chosen to encourage the decentralized design and control of information management and use. Nevertheless, the data collected are also available and used at Provincial and National levels. The system includes the procedures and formats used in all health facilities to collect and report the data, as well as the roles and authority enabling health workers to use their data to improve health service performance. Recommended analysis: Quantitative analysis Cascade analysis 4. Site Improving Monitoring Strategy (SIMS) reporting database. The standard SIMS tools (facility and above sites tools) were introduced in late 2014 and has been used to routinely monitor DSPs program performance in core program areas/elements at facility level. Recommended analysis: Review use for improvement External Evaluation: USAID/South Africa “Systems Strengthening for Better HIV/TB Outcomes” Project (2012-2017) P a g e | 54 Key Informant Interviews Key Informant Interviews: The evaluation team will conduct qualitative, in-depth and structured interviews with key stakeholders, partners and beneficiaries such as DSPs leadership and staff; USAID/SA management; national, provincial and district department of health representatives; HCWs who received training/mentoring, U.S.G and other international partners; local implementing partners and service providers, as well as project beneficiaries. It would be preferable for the interviews to be conducted face-to-face. However, if that is not possible, some of the interviews can be conducted over the phone or through other means such as email. Site or Service Assessment Field Visits: The evaluation team will conduct visits to a sample of clinic and district health office to assess process and outcome of Technical Assistance. The contractor will propose a sampling methodology and specific facility and implementation sites will be finalized during the debriefing process and prior to the country visit. Data abstraction Below are the key tasks the evaluation team will conduct during the data collection phase: a) Pre-evaluation Briefing: Preliminary discussions (prior to arrival in country) with the USAID/SA management team to review SOW, agree on key evaluation questions, evaluation design and data collection methods, finalize schedule and logistics. As an output, it is expected that a detailed work plan will be developed, including milestones and deliverables with due dates, responsible parties clearly established. b) Desk Review: The evaluation team is expected to conduct an in-depth review of background documents and relevant materials prior to arrival in the country. The following documents will be provided by USAID/SA for the desk review purposes: 1. Program Descriptions 2. Project quarterly, semi-annual and annual reports 3. Work plans 4. M&E plan 5. DATIM and PIMS reports 6. Selected project research and technical reports, publications, and tools c) Key Informant Interviews as described above d) Site Visits: The evaluation team will conduct a sample of clinic and district management site visits. The contractor/evaluator must propose an efficient and unbiased sampling method for the data collection methods mentioned above. The contractor should use the following (and the lists in appendix 2) as a sample frame:  List of Public Health Facilities where DSPs provide  DSP workplans Additionally, the contractor should discuss data disaggregation and analysis (by gender and other relevant categories), particularly how qualitative and quantitative data will be pulled together to generate high quality evidence and findings for this evaluation. Before data collection, the contractor will work with USAID to finalize the data analysis methods as part of the methodology plan. The methodology plan should encompass both data collection and analysis. DELIVERABLES AND PRODUCTS Select all deliverables and products required on this analytic activity. For those not listed, add rows as needed or enter them under “Other” in the table below. Provide timelines and deliverable deadlines for each. Deliverable / Product Launch briefing Work plan with timelines External Evaluation: USAID/South Africa “Systems Strengthening for Better HIV/TB Outcomes” Project (2012-2017) P a g e | 55 Analytic protocol with data collection tools In-brief with Mission or organising business unit In-brief with target project / program Routine briefings Findings review workshop with stakeholders with Power Point presentation Out-brief with Mission or organising business unit with Power Point presentation Draft report Final report Raw data Dissemination activity Other (specify): External Evaluation: USAID/South Africa “Systems Strengthening for Better HIV/TB Outcomes” Project (2012-2017) P a g e | 56 EVALUATION METHODS To answer the evaluation questions and sub questions, Khulisa used a non-experimental evaluation design that excluded the use of a comparison group, but which allowed for measurement of project trends and achievements against pre-defined project targets and objectives. Our “roadmap” to answering the evaluation questions was elaborated in an Evaluation Matrix, which defined key indicators for each evaluation sub-question as well as the data collection and analytical method to be used. Two main data collection approaches were employed to answer the evaluation questions: 1. Data Mining – we requested partners to provide us with data for the period FY2014- FY2016 for two types of data: a. Performance indicator data from the project’s 29 key Indicators of Success which consist of both PEPFAR and DOH indicators. b. Quantification of the volume of HSS/Capacity Building Activities delivered by the partner from FY2014-FY2016. For both data sets, we calculated trends as well any association between the two data sets (i.e. if more HSS and capacity building is associated with improved performance measures). 2. Key Informant Interviews (KIIs) – to obtain key stakeholder perceptions and feedback (from DOH, the DSP partners and donors) around the design and implementation of the project. We incorporated quantitative measures in the KIIs in the form of Likert scales (e.g. Strongly Agree to Strongly Disagree) to allow for comparisons betw een respondent groups. Data Mining DSP partners completed spreadsheets with the indicator data for FY2014-FY2016, and the volume of HSS/Capacity Building activities – measured by financial expenditure and human resources (as measured by Full-time-Equivalent or FTE) allocated to HSS/CB – delivered over the same period. We analysed the trends for both data sets from FY2014-FY2016, and the association of the HSS/CB measures to trends in patient outcomes. Key Informant Interviews (KIIs) Sampling: A representative sample of locations at national, provincial, district, sub -district, and facilities levels was selected. Sampling of individuals targeted for KIIs was purposive where individuals were chosen because of their roles and involvement in the project and partnerships. The sampling approach is detailed as follows. National Level: The national offices of the DOH, USAID, and each of the 7 DSPs, were targeted for the KIIs. Individuals targeted for KIIs at national level include the following:  At national DOH: members of the HIV cluster  At DSP head office: National program manager and team  At donors (i.e. USAID/PEPFAR): Relevant program staff External Evaluation: USAID/South Africa “Systems Strengthening for Better HIV/TB Outcomes” Project (2012-2017) P a g e | 57 Provincial Level: The sample includes all 8 provinces where DSPs work. Individuals targeted for KIIs at provincial level were:  At provincial DOH: relevant provincial DOH managers for interviews, e.g. HAST, pharmacy, lab, and M&E managers  At DSP provincial office (where it exists): Provincial program manager and team  At donors: Provincial PEPFAR Liaison Officers District Level: The 7 USAID-funded DSPs work in 21 districts across 8 provinces of South Africa, with some DSPs working in multiple districts in a single province. As such, the sample was based on ensuring that every DSP working in a province has at least 1 district - level operation included in the sample = i.e. 1 DSP operation per province. This sampling approach was based on the assumption that the DSPs’ design and implementation approach is more dependent on provincial priorities; and if the DSP supports multiple districts in a province, we assumed that there would be little significant difference in the DSP’s approach employed in multiple districts in the same province. As such, the sample ensures that each partner is represented in each province, as geography is assumed to be the most important driver of the implementation approaches and ultimately outcomes. Where a DSP worked in multiple districts in a single province, one DSP operation was randomly selected, resulting in the selection of 16 district-level DSP operations which approximates the distribution of the 21 district-level DSP operations funded by USAID throughout South Africa. Individuals targeted for KIIs at district level include the following:  At district DOH: relevant district DOH managers for interviews, e.g. district manager, HAST coordinators, lab, pharmacy, M&E district coordinators, training managers  At DSP district offices: District DSP program manager and team and sub-district manager and team (for the sampled sub-district only) Breakdown of DSP operations and sample by Province Provinces USAID-funded DSP Operations / districts District Sample No. DSP operations in province DSP operations (% of total) No. DSP operations Sampled DSP operations(% of total) Eastern Cape 1 4.8% 1 6.3% Free State 1 4.8% 1 6.3% Gauteng 8 38.1% 4 25.0% KwaZulu-Natal 4 19.0% 3 18.8% Limpopo 2 9.5% 2 12.5% Mpumalanga 3 14.3% 3 18.8% North West 1 4.8% 1 6.3% Western Cape 1 4.8% 1 6.3% Grand Total 21 100.0% 16 100.0% External Evaluation: USAID/South Africa “Systems Strengthening for Better HIV/TB Outcomes” Project (2012-2017) P a g e | 58 Breakdown of DSP operations and sample by Partner District Support Partner (DSP) USAID-funded DSP Operations / districts District Sample No. DSP operations DSP operations (% of total) No. DSP operations Sampled DSP operations(% of total) ANOVA Health Institute 5 23.8% 2 12.5% Broadreach Healthcare (BHC) 4 19.0% 3 18.8% Foundation for Professional Development (FPD) 3 14.3% 3 18.8% Kheth'Impilo 2 9.5% 2 12.5% Maternal, Adolescent and Child Health(MatCH) 1 4.8% 1 6.3% Right to Care (RTC) 4 19.0% 3 18.8% Wits Reproductive Health Institute (WRHI) 2 9.5% 2 12.5% Grand Total 21 100.0% 16 100.0% Selection of sub-districts: in each of the 16 sampled districts, we randomly selected 1 DOH sub-district for site visits and KIIs. Individuals targeted for KIIs at district level included:  At sub-district DOH: relevant sub-district PHC supervisor, PHC teams, sub-district family physician, sub-district trainers, sub-district M&E manager Information Officers, and other as appropriate  At DSP sub-district offices, where relevant: Sub-district program manager and team. Selection of Health Facilities: In each of the 16 sub-districts, we randomly selected 2 health facilities for site visits and interviews. Individuals targeted for KIIs at health facility included the Facility/Operational manager or CEO, professional nurses, pharmacists or pharmacist assistants, counsellors, CHWs, and data capturers. Summary: The above sampling approach resulted in 120 target groups for primary data collection (see table below). Three-quarters of these target groups are DOH management offices or health facilities. One quarter are related to DSPs. An estimated 400 individuals were targeted for interviews (3-4 per site) – mostly in group interviews. Target groups for KIIs Province Donors / PPLs DSP Partner South African Government (DOH) Head Total Office District Office Sub District National Office Provincial Office District Office Sub District Office Facility EC 1 1 1 1 1 2 7 FS 1 1 1 1 1 1 2 8 GP 2 4 5 4 1 1 4 4 8 33 KZN 2 1 3 3 1 3 3 6 22 LP 1 1 2 2 1 2 2 4 15 MP 1 2 3 1 3 3 6 19 NW 1 1 1 1 1 2 7 WC 1 1 1 1 1 1 1 2 9 Sub Total 9 7 15 16 1 8 16 16 32 120 Grand Total 9 38 73 120 External Evaluation: USAID/South Africa “Systems Strengthening for Better HIV/TB Outcomes” Project (2012-2017) P a g e | 59 External Evaluation: USAID/South Africa “Systems Strengthening for Better HIV/TB Outcomes” Project (2012-2017) P a g e | 60 KII TOOL Key Informant Interview Tool Interviewer Name____________ Date of Interview ____________ Time Interview started: (HH:mm): ______________________________ Location/Site Details: Location / Site / Organisation Name ______________ Name of DSP Assisting sites _________________ Province name____________________________ District name _____________________________ Sub-district name ________________________ Type of Site: (tick one) o Funder/ Donor o SA Government: National DOH o SA Government: Provincial DOH o SA Government: District DOH o SA Government: Sub-district DOH o SA Government: Health Facility o DSP Partner: National/Head Office o DSP Partner: District Office o Other (please specify) __________________ Names and Positions of Persons Interviewed: Person 1____________________________________________________ Position ___________________________________ How Long Person 1 has been in the position __________________________________________ Person 2___________________________________________________ Position ______________________________________ How Long Person 2 has been in the position __________________________________________ Person 3____________________________________________________ Position _____________________________________ How Long Person 3 has been in the position __________________________________________ Person 4____________________________________________________ Position _____________________________________ How Long Person 4 has been in the position __________________________________________ Person 5____________________________________________________ Position _____________________________________ How Long Person 5 has been in the position __________________________________________ External Evaluation: USAID/South Africa “Systems Strengthening for Better HIV/TB Outcomes” Project (2012-2017) P a g e | 61 Consent form Thank you for taking the time to participate in this evaluation. This evaluation is being conducted by Khulisa Management Services for USAID/South Africa and the Department of Health in South Africa. Your input is important for understanding the successes and challenges of the USAID-South African partnership for strengthening HIV and TB services delivery. The purpose of the interview is to understand how the USAID-funded District Support Partners have supported the achievement of the 90-90-90 goals. What you tell us about the District Support Partners will help to identify how this support can be strengthened. The interview should take approximately 90 minutes to complete. Your participation in this interview is voluntary. You are free to decline to answer any particular question you do not wish to answer for any reason, however, we want to assure you that your responses are completely anonymous. You may refuse to take part in the research or exit the interview at any time without penalty. Your responses will be combined with those of others and analysed as a group, to further protect your anonymity. If you have questions at any time about the study or the procedures, you may contact Mary Pat Selvaggio at Khulisa Management Services (011.447.6464 or via email at mpselvaggio@khulisa.com) CONSENT: I understand the above information and I voluntarily agree to participate. PERSON 1 PERSON 2 PERSON 3  Agree  Disagree  Agree  Disagree  Agree  Disagree Signature Signature Signature PERSON 4 PERSON 5  Agree  Disagree  Agree  Disagree Signature Signature External Evaluation: USAID/South Africa “Systems Strengthening for Better HIV/TB Outcomes” Project (2012-2017) P a g e | 62 Eval Matrix Ind No. Question Response Options Answers HSS/Capacity Building program overall effects 1.a.iii 1. PEPFAR has supported DOH at national, provincial, district and facility levels in Health System Strengthening and capacity building for HIV and TB. In your opinion, what was the Most Significant Change you have seen since 2014? Open-ended 2. What led to the change? Open ended Overall effects and challenges per HSS Building Block 1.a.iii 3. Since 2014, what has been the Most Significant Change in HEALTH INFORMATION SYSTEMS (including M&E)? Open-ended 4. How much of this change in HEALTH INFORMATION SYSTEMS and M&E do you attribute to the efforts and support of since 2014?  Most of the change (60% and above)  Equal effort in bringing the change (50-50)  Some of the change (From 30-40%)  A little bit of the change (less than 30%)  None of the change (0%)  Don’t Know 5. Please explain. [If the answer is “none” probe around who helped with the change.] Open-ended 6. Since 2014, what have been your main challenges around HIV and TB HEALTH INFORMATION SYSTEMS and M&E if any? Open ended 7. How were these challenges addressed? Open ended 1.a.iii 8. Since 2014, what has been the Most Significant Change in HIV and TB LABORATORY and PHARMACEUTICAL SYSTEMS? Open-ended 9. How much of the change in LABORATORY AND PHARMACEUTICAL SYSTEMS do you attribute to the efforts and support of since 2014?  Most of the change (60% and above)  Equal effort in bringing the change (50-50)  Some of the change (From 30-40%)  A little bit of the change (less than 30%)  None of the change (0%)  Don’t Know 10.Please explain. [If the answer is “none” probe around who helped with the change.] Open-ended 11.Since 2014, what have been your main challenges around HIV and TB LABORATORY AND PHARMACEUTICAL SYSTEMS if any? Open ended 12.How were these challenges addressed? Open ended 1.a.iii 13.Since 2014, what has been the Most Significant Change in HEALTH WORKFORCE STRENGTHENING for HIV and TB? Open-ended 14.How much of this change in HEALTH WORKFORCE STRENGTHENING do you attribute to the efforts and support of since 2014?  Most of the change (60% and above)  Equal effort in bringing the change (50-50)  Some of the change (From 30-40%) External Evaluation: USAID/South Africa “Systems Strengthening for Better HIV/TB Outcomes” Project (2012-2017) P a g e | 63 Eval Matrix Ind No. Question Response Options Answers  A little bit of the change (less than 30%)  None of the change (0%)  Don’t Know 15.Please explain. [If the answer is “none” probe around who helped with the change.] Open-ended 16.Since 2014, what have been your main challenges around HIV and TB HEALTH WORKFORCE STRENGTHENING if any? Open ended 17.How were these challenges addressed? Open ended 1.a.iii 18.Since 2014, what has been the Most Significant Change in HIV and TB SERVICES DELIVERY (services delivered to clients)? Open-ended 19.How much of this change in SERVICE DELIVERY do you attribute to the efforts and support of since 2014?  Most of the change (60% and above)  Equal effort in bringing the change (50-50)  Some of the change (From 30-40%)  A little bit of the change (less than 30%)  None of the change (0%)  Don’t Know 20.Please explain. [If the answer is “none” probe around who helped with the change.] Open-ended 21.Since 2014, what have been your main challenges around services delivery if any? Open ended 22.How have these challenges been addressed? Open ended 1.a.iii 23.Since 2014, what has been the Most Significant Change in HIV and TB LEADERSHIP AND GOVERNANCE/MANAGEMENT at this level (provincial/ district/sub-district/facility)? (Select the appropriate level) Open-ended 24.How much of the change in LEADERSHIP AND GOVERNANCE/MANAGEMENT do you attribute to the efforts and support of since 2014?  Most of the change (60% and above)  Equal effort in bringing the change (50-50)  Some of the change (From 30-40%)  A little bit of the change (less than 30%)  None of the change (0%)  Don’t Know 25.Please explain. [If the answer is “none” probe around who helped with the change.] Open-ended 26.Since 2014, what have been your main challenges around HIV and TB LEADERSHIP AND GOVERNANCE/ MANAGEMENT if any? Open ended 27. How were these challenges addressed? Open ended 1.a.iii 28.Since 2014, what has been the Most Significant Change in DISTRICT HEALTH PLANNING (DHP) for HIV and TB at this level? Open-ended 29.How much of this change in DISTRICT HEALTH PLANNING do you attribute to the efforts and support of since 2014?  Most of the change (60% and above)  Equal effort in bringing the change (50-50) External Evaluation: USAID/South Africa “Systems Strengthening for Better HIV/TB Outcomes” Project (2012-2017) P a g e | 64 Eval Matrix Ind No. Question Response Options Answers  Some of the change (From 30-40%)  A little bit of the change (less than 30%)  None of the change (0%)  Don’t Know 30.Please explain. [If the answer is “none” probe around who helped with the change.] Open-ended 31.Since 2014, what have been your main challenges around HIV and TB DISTRICT HEALTH PLANNING if any? Open ended 32.How were these challenges addressed? Open ended 1.a.iii 33.Since 2014, what has been the Most Significant Change in DISTRICT IMPLEMENTATION PLANNING (DIP) for HIV and TB at this level? Open-ended 34.How much of this change in DISTRICT IMPLEMENTATION PLANNING (DIP) do you attribute to the efforts and support of since 2014?  Most of the change (60% and above)  Equal effort in bringing the change (50-50)  Some of the change (From 30-40%)  A little bit of the change (less than 30%)  None of the change (0%)  Don’t Know 35.Please explain. [If the answer is “none” probe around who helped with the change.] Open-ended 36.Since 2014, what have been your main challenges around HIV and TB DISTRICT IMPLEMENTATION PLANNING (DIP), if any? Open ended 37.How were these challenges addressed? Open ended HSS/Capacity Building program -- Design, Planning and Implementation at DOH management level 3.a.i & 3.a.ii 38.Did ’s conduct Formative Research in designing its HSS/capacity building program/activities (e.g. a baseline needs assessment, a skills audit etc.)? Yes/ No /Don’t Know/Unsure 39.Please describe. Open ended 3.a.i & 3.a.ii 40.In the design of its HSS / capacity building program, did consider the ability of the DOH to use the HSS/CB inputs, DOH workload/responsibilities, enabling environment (e.g. Absorptive capacity)? Yes/ No /Don’t Know/Unsure 41.Please elaborate. Open ended 3.a.i & 3.a.ii 42.How well aligned are the HSS/capacity building activities to the DOH’s own processes at this level – including workplace skills plan and/or internal staff development plan?  Not at all aligned  Somewhat aligned  Well aligned  Don’t know 43.Please elaborate. 3.a.i & 3.a.ii 44.Were the number and competence of the ’s paid HSS/capacity building staff sufficient to ensure effective implementation of their HSS/capacity building activities? Yes – Number only Yes – competence only External Evaluation: USAID/South Africa “Systems Strengthening for Better HIV/TB Outcomes” Project (2012-2017) P a g e | 65 Eval Matrix Ind No. Question Response Options Answers Yes – both No. and competence No Don’t Know 45.Please elaborate for each. Open ended 3.a.i & 3.a.ii To what extent did collaborate with the DOH at this level in the following overarching design elements: 3.a.i & 3.a.ii 46.GOAL SETTING and PROGRAM PLANNING for ’s HSS/capacity building program/activities?  No collaboration at all  Some or limited collaboration  Active collaboration  No goal setting  Don’t know/Unsure 47.Please explain. Open-ended 3.a.i & 3.a.ii 48.Establishment of MONITORING PROCESSES for ’s HSS/capacity building?  No collaboration at all  Some or limited collaboration  Active collaboration  No monitoring process  Don’t know/Unsure 49.Please explain. Open ended 50. What challenges or disruptions have you experienced in achieving the agreed upon DOH-DSP HSS/Capacity building program? Open ended 3.a.i & 3.a.ii 51.Did the ’s HSS/Capacity building program/activities consider the local context in its design? Yes /Partly / No /Don’t Know /Unsure 52.Please elaborate. Open ended 53.Does have a Memorandum of Understanding (or equivalent) with the DOH for the HSS and Capacity Building program in the districts where it operates? Yes/ No /Don’t Know/Unsure 54.If no, please explain. Training feedback around HSS/Capacity Building 1.c.i In terms of training, mentoring and technical support for HIV and TB programs, please indicate your agreement with the following statements: 1.c.i 55.The ’s approach to training, mentoring and technical support was effective in maximising learning.  Strongly disagree to Strongly Agree  Don’t know 56.Please elaborate. Open-ended 1.c.i 57.The training was reasonable in terms of length and frequency.  Strongly disagree to Strongly Agree  Don’t know External Evaluation: USAID/South Africa “Systems Strengthening for Better HIV/TB Outcomes” Project (2012-2017) P a g e | 66 Eval Matrix Ind No. Question Response Options Answers 58.Please elaborate.  Open-ended 1.c.i 59.The DOH participants were able to apply what they learned around HIV and TB programs.  Strongly disagree to Strongly Agree  Don’t know 60.Please elaborate.  Open-ended 1.b.iii 61.As a result of ’s HSS/capacity building, the DOH‘s internal management and monitoring processes were strengthened at this level.  Strongly disagree to Strongly Agree  Don’t know 1.b.iii 62.Overall, has strengthened the planning and design of DOH’s HIV and TB programs – including setting of goals/targets at this level.  Strongly disagree to Strongly Agree  Don’t know 63.Please explain. [Probe for the DSP’s capacity building process, impact.] Open ended 1.b.iii 64. has strengthened the DOH’s implementation and management of HIV and TB programs.  Strongly disagree to Strongly Agree  Don’t know 65.Please explain. [Probe for DSP capacity building process, impact.] Open ended 1.b.iii 66.The strengthened the DOH’s monitoring of HIV and TB programs at this level in the following: a. Tier.Net: Health Patient Registration System  Strongly disagree to Strongly Agree  Don’t know b. Patient Tracking System  Strongly disagree to Strongly Agree  Don’t know c. HCT Module on Tier.net  Strongly disagree to Strongly Agree  Don’t know d. Pre-ART module on Tier.net  Strongly disagree to Strongly Agree  Don’t know e. Workload Indicators of Staffing Need (WISN)  Strongly disagree to Strongly Agree  Don’t know f. DHIS 2  Strongly disagree to Strongly Agree  Don’t know g. Stock Visibility Solution (SVS)  Strongly disagree to Strongly Agree  Don’t know h. Other (please describe)  Strongly disagree to Strongly Agree  Don’t know i. Please explain. [Probe for DSP capacity building process, impact.] Open ended 1.c.i 67.Have you personally received training, mentoring or technical support from ? Yes /No / Don’t know/unsure 1.c.i 68.If yes, how satisfied were you with the training, mentoring and technical support provided by .  Very dissatisfied to Very satisfied External Evaluation: USAID/South Africa “Systems Strengthening for Better HIV/TB Outcomes” Project (2012-2017) P a g e | 67 Eval Matrix Ind No. Question Response Options Answers  Don’t know  N/A Effects of ’s work on patient outcomes 2.c.ii 69.Do you agree that ’s support (e.g. technical assistance, capacity building, HSS, etc.) has contributed towards improving patient outcomes in this province/district/sub-district/facility since 2014?  Strongly disagree to Strongly Agree  Don’t know 3.a.i & 3.a.ii 70.Please elaborate [Probe for theory of change.] Open ended 3.a.i & 3.a.ii 71.Has the ’s HSS/capacity building program helped to increase your coverage for key populations? [e.g. MSM, adolescents and young women, prisoners, sex workers, OVCs etc.] Yes/ No /Don’t Know/Unsure 72.Please elaborate. Open ended 3.a.i & 3.a.ii 73.Has ’s HSS/capacity building program helped to increase your quality of services for HIV and TB (Quality Assurance/Quality Improvement)? Yes/ No /Don’t Know/Unsure 74.Please elaborate. Open ended 2.c.ii 75. For this province/district/sub-district/facility, what are the three top contributions that has made towards improving patient outcomes since 2014? Open ended ’s linkages and partnerships with other partners and work at community level 4.a.ii 76.Has the established linkages or partnerships with any of the following providers: a. Voluntary Male Medical Circumcision (VMMC) partners (PEPFAR funded) Yes/No/Don’t Know-Unsure b. Any Community-based cadres such as Youth health care workers, Community Health Workers, Community-Care Givers, DOTS supporters? Yes/No/Don’t Know-Unsure c. Community testing programs Yes/No/Don’t Know-Unsure d. Community-based Treatment programs Yes/No/Don’t Know-Unsure e. OVC Programs (PEPFAR funded) Yes/No/Don’t Know-Unsure f. HIV Prevention programs (PEPFAR funded) Yes/No/Don’t Know-Unsure g. Other government departments e.g. DBE, Higher education sector, Social Development Yes/No/Don’t Know-Unsure h. Private sector Yes/No/Don’t Know-Unsure 4.c 77.If linkages or partnerships were established, what innovative practices were used? Open ended 4.c. 78.What systematic or structural barriers/ challenges have limited from collaborating efficiently with other PEPFAR partners to provide complementary and comprehensive HIV, prevention and OVC services? Open-ended 4.c. 79.What recommendations can you provide to address these barriers, challenges or gaps? Open ended External Evaluation: USAID/South Africa “Systems Strengthening for Better HIV/TB Outcomes” Project (2012-2017) P a g e | 68 Eval Matrix Ind No. Question Response Options Answers 4.a.ii 80.Does the participate in any of the following AIDS fora? Select all that apply  Yes, Provincial Committee on AIDS  Yes, District AIDS Council  Yes, Ward AIDS Councils  Other (please describe, e.g. War rooms, etc.)  No  Don’t’ know 4.a.ii 81.At this level, did the provide support to strengthen: … [Fieldworker: Pls provide notes on any items selected as “yes” below.] a. …Facility/Community referral system for HIV and TB, e.g. bi-directional referral systems to support patients at all levels? Yes/No/Don’t Know-Unsure b. …Mom-connect Yes/No/Don’t Know-Unsure c. ….Capacity of Ward-Based Outreach Teams (WBOT) and community cadres Yes/No/Don’t Know-Unsure d. ….WBOT’s database Yes/No/Don’t Know-Unsure e. ….Linkages between DOH services and community- based/faith-based organisations (CBOs/FBOs) Yes/No/Don’t Know-Unsure f. …Improved reporting by CBOs/FBOS and sharing to the DOH Yes/No/Don’t Know-Unsure g. …CHWs and Home-Based Caregivers Yes/No/Don’t Know-Unsure h. …Expansion of adherence support clubs Yes/No/Don’t Know-Unsure i. …Decanting of stable patients Yes/No/Don’t Know-Unsure j. …Community-based drug pick up points Yes/No/Don’t Know-Unsure k. …Electronic Stock Management System Yes/No/Don’t Know-Unsure l. …youth- and adolescent-friendly HIV C&T programs Yes/No/Don’t Know-Unsure m. …HIV Rapid test quality assurance: Proficiency testing for HIV testing sites Yes/No/Don’t Know-Unsure Fieldworker: Please provide notes on any items selected as “yes” above. Open ended. Closing questions 3.a.iii 82.In your opinion, what additional support in HSS/capacity building is required to achieve 90-90-90 goals? Open-ended 3.a.ii 83.In your opinion, if stops its support, will the DOH be ready to continue the HSS and Capacity building support? Open-ended 84.Is there anything else you would like to add about ’s HSS/Capacity Building program? Open-ended External Evaluation: USAID/South Africa “Systems Strengthening for Better HIV/TB Outcomes” Project (2012-2017) P a g e | 69 Eval Matrix Ind No. Question Response Options Answers Thank you for your time and cooperation. Would it be possible to get a copy of the District Implementation Plan (DIPs) and District Health Plans (DHPs) for 2014, 2015, and 2016? Statement about next steps: The evaluation report will be prepared for USAID by late May 2017 who will share the main findings with District Support partners and the national Department of Health. Time Interview Ended: (HH:mm)_____________________________________________ Fieldworker’s additional notes / comments/ observations. [Please also indicate if you were able to obtain DHPs/DIPs for 2014-2016.] External Evaluation: USAID/South Africa “Systems Strengthening for Better HIV/TB Outcomes” Project (2012-2017) P a g e | 70 LIST OF SITES AND INTERVIEWS Interviews conducted as part of Fieldwork 183 interviews were carried out at 106 locations or sites. 389 people participated in the interviews, which were often conducted as group interviews. Location or Site Name No. Locations Number of Interviews No. Persons Interviewed SA Government: National DOH 1 3 3 National DOH 1 3 3 SA Government: Provincial DOH 8 14 25 EC-PDOH 1 2 2 FS-PDOH 1 1 1 GP-PDOH 1 2 5 KZN-PDOH 1 3 5 LP-PDOH Polokwane 1 1 4 MP-PDOH 1 1 4 NW-PDOH 1 2 2 WC-PDOH 1 2 2 SA Government: District DOH 11 21 42 ec Alfred Nzo DDOH 1 2 3 gp COJ District DOH 1 4 4 kz Ethekwini DDOH 1 1 1 kz Ugu DDOH 1 2 5 kz Umgungundlovu DDOH 1 1 5 lp Capricorn DDOH 1 2 6 lp Mopani DDOH (Giyani) 1 1 1 mp Elanzeni DDOH 1 2 6 mp Gert Sibande DDOH (Ermelo) 1 2 3 mp Nkangala District Office 1 2 6 nw Kenneth Kaunda DDOH 1 2 2 SA Government: Sub-district DOH 19 26 50 ec ANzo-Maluti Sub DDOH 1 1 1 fs Dihlaben sub DDOH (Bethlehem) 1 1 1 fs Setsoto Sub DDOH (Ficksburg) 1 2 2 gp COJ Reg A 1 1 1 gp COJ Reg E 1 1 4 gp COJ Region F Municipality 1 1 2 gp Tshwane- Hammanskral Sub-DDOH 1 1 1 kz eThekwini sub DDOH 1 1 5 kz Ugu-Hib Coast sub DDOH 1 1 4 kz Umungundlovu - Msunduzi Sub DDOH 1 1 1 lp Ben Farm sub DDOH 1 1 2 lp Lepell Nkumpi sub DDOH 1 1 2 mp Dipaleseng sub DDOH 1 2 3 mp J.S. Moroka sub DDOH 1 2 4 mp Umjindi Sub DDOH 1 1 1 nw Matlosana sub-DDOH 1 1 2 nw Tlokwe Sub DDOH 1 1 1 nw Ventersdorp sub DDOH 1 4 6 wc CCT-Eastern Sub DDOH 1 2 7 SA Government: Health Facility 33 62 136 8th Avenue Clinic - Alexandra 1 2 4 Barberton TB Hospital 1 1 2 Balfour clinic 1 2 3 External Evaluation: USAID/South Africa “Systems Strengthening for Better HIV/TB Outcomes” Project (2012-2017) P a g e | 71 Location or Site Name No. Locations Number of Interviews No. Persons Interviewed Barberton Gateway Clinic 1 2 4 Clocolan 1 2 3 DOH Maluti CHC 1 1 3 Edenvale Hospital – Tsakani ART Support Center 1 2 3 Grootvlei clinic 1 2 3 Ikhwezi Clinic Strand 1 3 9 Jubilee Hospital 1 2 10 Kekanastad clinic 1 2 4 Kibler Park Clinic 1 2 2 Kleinvlei Clinic 1 2 4 Lulekani CHC 1 2 3 Magadla Clinic 1 1 2 Mamello Cinic 1 2 3 Maputha Malatjie Hospital Namakgale 1 2 4 Mason Clinic / Lazarus Dve Copesville/PMB 1 2 4 Mayfair Clinic 1 2 4 Mtentweni 1 1 3 Mvutshini Clinic 1 2 4 Nokaneng CHC 1 2 6 Nyaniso Clinic 1 2 4 OR Tambo Clinic 1 2 4 Rabie Ridge Clinic 1 2 3 Rakgoatha clinic 1 2 4 Shallcross Clinic/ Municipality Clinic 1 2 5 Seabe CHC 1 2 5 Sobantu Clinic 1 2 5 UMLAZI H CLINIC / ETHEKWINI/ DOH Facility/ KZN 1 2 5 Unit R clinic Lebowakgomo 1 2 6 Welgevonden Clinic 1 1 3 Wolmarastad Town Clinic 1 2 5 DSP Partner: National/Head Office 7 16 28 Anova HO 1 1 1 Broadreach HO 1 2 5 FPD HO 1 3 4 Kheth’Impilo HO 1 3 7 MatCH HO 1 1 1 Right to Care - HO 1 2 3 WRHI HO 1 4 7 DSP Partner: District Office 13 26 78 Anova COJ - Reg E 1 3 7 Anova LP Mopani 1 1 3 BHC A Nzo 1 1 5 BHC Gert Sibande 1 1 1 BHC Ugu 1 2 7 FPD Capricorn 1 2 5 FPD Nkangala 1 2 6 Kheth’Implo Umgungundlovu 1 3 11 MatCH Ethekwini 1 4 17 RTC Bethlehem 1 2 3 RTC City of Joburg 1 1 3 RTC Ehlanzeni 1 2 5 WRHI K Kauanda 1 2 5 DSP Partner: Sub-district 7 7 19 Anova - LP- Mopani sub district 1 1 2 External Evaluation: USAID/South Africa “Systems Strengthening for Better HIV/TB Outcomes” Project (2012-2017) P a g e | 72 Location or Site Name No. Locations Number of Interviews No. Persons Interviewed BHC Gert Sibande Sub district 1 1 1 FPD Lepelle Sub District 1 1 3 FPD Moroka sub district 1 1 3 FPD Nkangala sub district 1 1 1 RTC COJ Sub-District 1 1 5 RTC Ehlanzeni Sub-District 1 1 4 Funder/ Donor 7 8 8 PEPFAR Liaison - EC 1 1 1 PEPFAR Liaison - FS 1 1 1 PEPFAR Liaison - GP 1 1 1 PEPFAR Liaison - KZN 1 1 1 PEPFAR Liaison - LP 1 1 1 PEPFAR Liaison - MP 1 1 1 USAID/South Africa 1 2 2 Grand Total 106 183 389 33 Health Facilities visited during Fieldwork by Province, District, and Partner Province District Sub District DSP Name Site Name EC Alfred Nzo Maluti BHC 1. DOH Maluti CHC 2. Magadla Clinic 3. Nyaniso Clinic FS Thabo Mofutsanyane Setsoto RTC 4. Clocolan 5. Mamello Cinic GP City of Joburg Region A RTC 6. OR Tambo Clinic 7. Rabie Ridge Clinic Region E Anova 8. 8th Avenue Clinic - Alexandra 9. Edenvale Hospital – Tsakani ART Support Center Region F WRHI 10. Kibler Park Clinic 11. Mayfair Clinic Tshwane Region 2 FPD 12. Jubilee Hospital 13. Kekanastad clinic KZN eThekwini South 8 MatCH 14. Shallcross Clinic/ Municipality Clinic UMLAZI H MatCH 15. UMLAZI H CLINIC / ETHEKWINI/ DOH Facility/ KZN Ugu Hibiscus Coast BHC 16. Mtentweni 17. Mvutshini Clinic Umgungundlovu DC 22 Kheth’Impilo 18. Mason Clinic / Lazarus Dve Copesville/PMB MSUNDUZI Kheth’Impilo 19. Sobantu Clinic LP Capricorn Lepelle￾Nkumpi FPD 20. Rakgoatha clinic 21. Unit R clinic Lebowakgomo Mopani Ba-Phalaborwa Anova 22. Lulekani CHC 23. Maputha Malatjie Hospital Namakgale MP Ehlanzeni Umjindi RTC 24. Barberton TB Hospital 25. Barberton Gateway Clinic Gert Sibande Dipaleseng BHC 26. Balfour clinic 27. Grootvlei clinic Nkangala Dr. J.S. Moroka FPD 28. Nokaneng CHC External Evaluation: USAID/South Africa “Systems Strengthening for Better HIV/TB Outcomes” Project (2012-2017) P a g e | 73 Province District Sub District DSP Name Site Name 29. Seabe CHC NW Dr Kenneth Kaunda Maquassi Hills WRHI 30. Wolmarastad Town Clinic Ventersdorp WRHI 31. Welgevonden Clinic WC City of Cape Town Eastern Kheth’Impilo 32. Ikhwezi Clinic Strand 33. Kleinvlei Clinic External Evaluation: USAID/South Africa “Systems Strengthening for Better HIV/TB Outcomes” Project (2012-2017) P a g e | 74 HSS / CB INVESTMENTS BY DSP (FY2014-FY2016) PROGRAMME AREA - HUMAN RESOUCES (IN FULL-TIME EQUIVALENT = FTE) Programmatic Domain 2 014 2 015 2 016 Total Trend % Change since 2014 2 014 2 015 2 016 Total Trend % Change since 2014 2 014 2 015 2 016 Total Trend % Change since 2014 2 014 2 015 2 016 Total Trend % Change since 2014 HTC-PITC 275 467 390 1 131 42% - - - - #DIV/0! 5 6 12 24 123% 8 3 25 36 223% HTC-VCT 127 272 521 920 310% 10 10 72 92 620% 3 4 8 15 123% 8 3 25 36 223% PMTCT 77 271 538 885 602% 7 7 18 32 157% 2 2 5 9 123% 2 7 20 29 869% Facility-based care, treatment and support 388 850 1 787 3 025 361% 63 66 200 329 217% 34 39 76 149 123% 60 37 125 222 110% HSS - District Planning / supervision 60 57 71 189 19% 11 11 22 44 100% 9 10 20 39 123% 24 15 3 42 -88% HSS - Pharmacy/ Supply Chain Management 19 16 869 904 4517% 9 9 12 30 33% 2 2 5 9 123% 7 4 2 12 -75% HSS-Strategic Information 145 408 497 1 050 243% 7 7 10 24 43% 10 11 21 42 123% 41 33 165 239 302% Infection Control 24 26 72 121 202% 10 10 46 66 360% 1 2 3 6 123% 3 3 10 16 199% Laboratory 7 9 15 32 109% 6 6 - 12 -100% 1 2 3 6 123% - 0 0 1 42% Other 40 64 193 298 377% - - - - #DIV/0! - - - - #DIV/0! 5 7 - 12 -100% Total 1 162 2 439 4 952 8 553 326% 123 126 380 629 209% 68 78 152 298 123% 157 112 375 644 139% All Partners excluding RTC (FTE) Anova BRHC FPD External Evaluation: USAID/South Africa “Systems Strengthening for Better HIV/TB Outcomes” Project (2012-2017) P a g e | 75 Programmatic Domain 2 014 2 015 2 016 Total Trend % Change since 2014 2 014 2 015 2 016 Total Trend % Change since 2014 2 014 2 015 2 016 Total Trend % Change since 2014 HTC-PITC 4 5 6 15 70% 225 420 292 937 -30% 33 33 54 120 66% HTC-VCT 4 5 6 14 70% 63 220 312 595 395% 39 30 98 168 150% PMTCT 4 5 6 14 67% 40 230 447 717 1018% 22 20 42 84 92% Facility-based care, treatment and support 4 5 6 15 72% 159 633 1 269 2 061 698% 69 69 111 249 62% HSS - District Planning / supervision 1 1 1 3 17% - 2 8 10 300% 15 18 18 52 16% HSS - Pharmacy/ Supply Chain Management 0 0 0 0 160% 1 1 850 852 84900% - - - - #DIV/0! HSS-Strategic Information 3 4 7 14 105% 5 207 164 376 -21% 79 145 130 355 -10% Infection Control 3 4 6 13 109% - - - - #DIV/0! 6 7 7 20 8% Laboratory - - 6 6 #DIV/0! - 1 6 7 #DIV/0! - - - - #DIV/0! Other - - - - #DIV/0! 4 36 105 145 2525% 32 21 88 140 178% Total 21 29 44 95 107% 497 1 750 3 453 5 700 595% 295 344 548 1 187 86% Kethimpilo Match WRHI External Evaluation: USAID/South Africa “Systems Strengthening for Better HIV/TB Outcomes” Project (2012-2017) P a g e | 76 PROGRAMME AREA – EXPENDITURE (IN USD) Programmatic Domain 2 014 2 015 2 016 Total Trend % Change since 2014 2 014 2 015 2 016 Total Trend % Change since 2014 2 014 2 015 2 016 Total Trend % Change since 2014 2 014 2 015 2 016 Total Trend % Change since 2014 HTC-PITC 3 199 577 3 382 114 4 998 685 11 580 375 56% - - - - #DIV/0! 363 562 239 185 241 687 844 434 -34% 778 424 243 952 1 087 567 2 109 943 40% HTC-VCT 3 146 916 3 188 740 5 638 799 11 974 455 79% 444 613 304 342 1 132 607 1 881 562 155% 227 226 149 491 151 054 527 771 -34% 778 424 243 952 1 087 567 2 109 943 40% PMTCT 4 049 374 4 430 922 5 300 044 13 780 340 31% 1 529 241 908 837 755 071 3 193 149 -51% 136 336 89 695 90 632 316 663 -34% 256 061 564 121 852 332 1 672 513 233% Facility-based care, treatment and support 16 919 262 12 436 833 18 054 625 47 410 720 7% 3 344 017 2 177 737 4 530 429 10 052 183 35% 2 272 262 1 494 909 1 510 541 5 277 712 -34% 6 497 227 3 075 851 5 317 252 14 890 329 -18% HSS - District Planning / supervision 6 669 013 5 737 864 4 889 655 17 296 532 -27% 441 262 584 311 24 000 1 049 573 -95% 590 788 388 676 392 741 1 372 205 -34% 2 718 177 1 343 865 125 150 4 187 192 -95% HSS - Pharmacy/ Supply Chain Management 1 254 144 692 872 679 793 2 626 809 -46% 60 787 77 957 15 600 154 344 -74% 136 336 89 695 90 632 316 663 -34% 750 256 346 961 72 894 1 170 111 -90% HSS-Strategic Information 3 733 813 4 140 851 4 894 751 12 769 416 31% 101 816 135 479 268 838 506 133 164% 636 233 418 575 422 952 1 477 759 -34% 902 444 892 617 360 386 2 155 447 -60% Infection Control 2 730 500 3 011 462 4 370 302 10 112 263 60% 772 018 456 619 755 071 1 983 708 -2% 90 890 59 796 60 422 211 108 -34% 419 739 269 209 416 301 1 105 249 -1% Laboratory 115 520 116 149 3 352 047 3 583 716 2802% 23 359 30 273 377 536 431 168 1516% 90 890 59 796 60 422 211 108 -34% - 24 886 12 700 37 586 -49% Other 2 884 118 3 116 005 2 116 081 8 116 205 -27% - - - - #DIV/0! - - - - #DIV/0! 427 155 622 409 - 1 049 564 -100% Total 41 502 660 36 871 698 49 296 097 139 250 830 19% 6 717 113 4 675 554 7 859 152 19 251 820 17% 4 180 961 2 750 633 2 779 396 9 710 990 -34% 12 749 482 7 383 870 8 244 582 28 377 935 -35% All Partners (expenditure in USD) Anova BRHC FPD External Evaluation: USAID/South Africa “Systems Strengthening for Better HIV/TB Outcomes” Project (2012-2017) P a g e | 77 Programmatic Domain 2 014 2 015 2 016 Total Trend % Change since 2014 2 014 2 015 2 016 Total Trend % Change since 2014 2 014 2 015 2 016 Total Trend % Change since 2014 2 014 2 015 2 016 Total Trend % Change since 2014 HTC-PITC 1 436 135 2 233 867 3 030 761 6 700 763 111% 19 032 15 573 24 230 58 835 56% 1 655 25 018 10 159 36 832 514% 600 769 624 519 604 280 1 829 569 1% HTC-VCT 1 376 689 2 170 140 2 933 016 6 479 845 113% 19 580 8 557 32 414 60 550 66% - - - - #DIV/0! 300 384 312 260 302 140 914 784 1% PMTCT 1 477 145 2 190 861 2 935 240 6 603 246 99% 46 902 51 455 61 464 159 822 31% 2 920 1 434 1 025 5 379 -65% 600 769 624 519 604 280 1 829 569 1% Facility-based care, treatment and support 1 553 050 2 366 930 3 138 379 7 058 358 102% 180 230 191 878 302 795 674 903 68% 789 555 756 355 958 964 2 504 873 21% 2 282 922 2 373 174 2 296 266 6 952 361 1% HSS - District Planning / supervision 655 426 747 655 856 619 2 259 700 31% - 17 430 4 310 21 740 -75% 2 143 207 2 531 022 3 365 979 8 040 208 57% 120 154 124 904 120 856 365 914 1% HSS - Pharmacy/ Supply Chain Management 294 875 129 750 481 090 905 715 63% 6 038 8 885 4 698 19 621 -47% 5 852 39 625 14 878 60 355 154% - - - - #DIV/0! HSS-Strategic Information 1 255 617 1 941 144 3 059 942 6 256 703 144% 12 139 14 910 5 164 32 213 -65% 224 795 113 607 173 189 511 592 -23% 600 769 624 519 604 280 1 829 569 -3% Infection Control 1 147 328 1 913 578 2 836 368 5 897 274 147% 140 - - 140 -100% - - - - #DIV/0! 300 384 312 260 302 140 914 784 1% Laboratory - - 2 891 734 2 891 734 #DIV/0! 1 270 1 193 9 655 12 118 660% - - - - #DIV/0! - - - - #DIV/0! Other - - - - #DIV/0! 51 600 23 229 13 620 88 449 -74% 1 203 826 1 221 329 893 900 3 319 055 -26% 1 201 538 1 249 039 1 208 561 3 659 137 1% Total 7 760 129 11 460 058 19 132 388 38 352 575 147% 317 899 317 536 434 120 1 069 555 37% 4 370 155 4 663 372 5 407 935 14 441 462 24% 5 406 920 5 620 675 5 438 524 16 466 118 1% Kethimpilo Match RTC WRHI External Evaluation: USAID/South Africa “Systems Strengthening for Better HIV/TB Outcomes” Project (2012-2017) P a g e | 78 CAPACITY BUILDING ACTIVITIES - HUMAN RESOUCES (IN FULL-TIME EQUIVALENT = FTE) HSS Capacity Building Activity 2014 2015 2016 Total Trend % Change since 2014 2014 2015 2016 Total Trend % Change since 2014 2014 2015 2016 Total Trend % Change since 2014 2014 2015 2016 Total Trend % Change since 2014 Direct Service Delivery (DSD) 142 160 725 1 027 412% 14 14 216 244 1443% - - 45 45 #DIV/0! 6 5 75 86 1176% Temporary Seconded Staff 93 159 307 558 232% - - - - #DIV/0! - - 45 45 #DIV/0! 12 10 30 52 155% Mentoring 107 64 145 316 36% 25 25 58 108 132% 1 1 3 5 188% 59 37 60 155 2% Supportive Supervision 95 118 126 339 33% - - - - #DIV/0! 58 66 22 146 -63% 24 21 60 105 152% Roving Clinical Teams 218 250 404 872 85% 69 72 58 199 -16% 5 6 19 31 264% 18 31 75 124 324% Training 503 1 682 3 230 5 415 541% 15 15 48 78 220% 3 4 9 16 188% 39 7 76 122 93% Other 4 6 16 26 284% - - - - #DIV/0! 0 1 9 10 1992% - - - - #DIV/0! Total 3 176 4 454 6 968 8 553 119% 2 137 2 141 2 396 629 12% 2 082 2 093 2 168 298 4% 2 171 2 127 2 391 644 10% All Partners excluding RTC (FTE) Anova BRHC FPD External Evaluation: USAID/South Africa “Systems Strengthening for Better HIV/TB Outcomes” Project (2012-2017) P a g e | 79 HSS Capacity Building Activity 2014 2015 2016 Total Trend % Change since 2014 2014 2015 2016 Total Trend % Change since 2014 2014 2015 2016 Total Trend % Change since 2014 Direct Service Delivery (DSD) 2 2 3 7 81% 25 81 100 206 23% 95 58 286 439 201% Temporary Seconded Staff 5 6 11 23 138% - - 93 93 #DIV/0! 76 142 128 346 68% Mentoring 1 1 0 2 -47% 21 - 24 45 14% - - - - #DIV/0! Supportive Supervision 3 4 4 10 15% 10 27 41 78 310% - - - - #DIV/0! Roving Clinical Teams 6 8 18 32 185% 28 20 122 170 510% 92 112 112 316 22% Training 1 2 2 5 79% 413 1 622 3 073 5 108 89% 32 32 22 86 -31% Other 4 5 6 15 72% - - - - #DIV/0! - - - - #DIV/0! Total 2 035 2 044 2 060 95 1% 2 511 3 765 5 469 5 700 118% 2 309 2 359 2 564 1 187 11% Kethimpilo Match WRHI External Evaluation: USAID/South Africa “Systems Strengthening for Better HIV/TB Outcomes” Project (2012-2017) P a g e | 80 CAPACITY BUILDING ACTIVITIES – EXPENDITURE (IN USD) HSS Capacity Building Activity 2014 2015 2016 Total Trend % Change since 2014 2014 2015 2016 Total Trend % Change since 2014 2014 2015 2016 Total Trend % Change since 2014 2014 2015 2016 Total Trend % Change since 2014 Direct Service Delivery (DSD) 4 223 743 3 528 264 10 418 115 18 170 123 147% 1 193 662 738 326 4 530 429 6 462 417 280% - - 889 413 889 413 #DIV/0! 536 445 342 264 1 949 974 2 828 683 263% Temporary Seconded Staff 4 107 753 5 055 742 8 605 363 17 768 858 109% - - - - #DIV/0! - - 889 413 889 413 #DIV/0! 1 076 900 689 204 779 989 2 546 093 -28% Mentoring 9 072 350 5 162 482 3 893 064 18 127 896 -57% 2 387 325 1 476 647 1 132 607 4 996 578 -53% 67 880 44 658 59 274 171 812 -13% 5 376 481 2 423 903 1 559 979 9 360 363 -71% Supportive Supervision 9 868 069 8 289 115 6 549 197 24 706 382 -34% - - - - #DIV/0! 3 889 786 2 559 070 430 488 6 879 344 -89% 2 185 868 1 401 786 1 559 979 5 147 633 -29% Roving Clinical Teams 9 796 890 10 790 165 16 279 878 36 866 933 66% 2 387 325 1 476 647 1 132 607 4 996 578 -53% 353 510 232 572 387 620 973 703 10% 1 617 354 2 062 937 1 949 974 5 630 264 21% Training 5 002 151 3 872 594 4 559 030 13 433 775 -9% 748 801 983 935 1 063 510 2 796 246 42% 203 640 133 974 177 821 515 435 -13% 2 734 859 707 727 1 532 255 4 974 841 -44% Other 2 631 280 3 555 449 3 990 134 10 176 864 52% - - - - #DIV/0! 29 707 19 544 187 053 236 305 530% - - - - #DIV/0! Total 44 702 237 40 253 812 54 294 781 139 250 830 21% 6 717 113 4 675 554 7 859 152 19 251 820 17% 4 544 523 2 989 818 3 021 083 10 555 424 -34% 13 527 907 7 627 822 9 332 149 30 487 877 -31% All Partners Anova BRHC FPD HSS Capacity Building Activity 2014 2015 2016 Total Trend % Change since 2014 2014 2015 2016 Total Trend % Change since 2014 2014 2015 2016 Total Trend % Change since 2014 2014 2015 2016 Total Trend % Change since 2014 Direct Service Delivery (DSD) 660 562 883 509 1 481 494 3 025 565 124% 77 991 131 143 68 719 277 853 -48% 235 223 179 684 199 252 614 159 -15% 1 519 860 1 253 337 1 298 835 4 072 032 -15% Temporary Seconded Staff 1 857 248 2 673 794 5 435 970 9 967 012 193% - - 77 536 77 536 #DIV/0! - - - - #DIV/0! 1 173 606 1 692 743 1 422 454 4 288 803 21% Mentoring 295 212 345 376 248 941 889 529 -16% 47 368 20 065 35 896 103 328 -24% - - - - #DIV/0! 898 085 851 834 856 368 2 606 287 -5% Supportive Supervision 2 063 470 2 629 860 2 709 962 7 403 292 31% 77 187 87 696 92 003 256 886 19% - - - - #DIV/0! 1 651 758 1 610 704 1 756 766 5 019 227 6% Roving Clinical Teams 2 375 935 3 644 946 8 317 181 14 338 062 250% 130 006 85 686 167 554 383 246 96% 2 932 762 3 287 377 4 324 943 10 545 081 47% - - - - #DIV/0! Training 546 091 1 201 863 1 060 420 2 808 375 94% 4 380 8 519 16 642 29 541 95% - - - - #DIV/0! 764 379 836 576 708 382 2 309 337 -15% Other 1 397 748 2 314 576 2 909 181 6 621 504 108% - - - - #DIV/0! 1 203 826 1 221 329 893 900 3 319 055 -26% - - - - #DIV/0! Total 9 196 265 13 693 925 22 163 149 45 053 338 141% 336 931 333 109 458 350 1 128 390 36% 4 371 810 4 688 390 5 418 094 14 478 294 24% 6 007 689 6 245 194 6 042 804 18 295 686 1% Kethimpilo Match RTC WRHI External Evaluation: USAID/South Africa “Systems Strengthening for Better HIV/TB Outcomes” Project (2012-2017) P a g e | 81 INDICATOR PERFORMANCE BY DSP Ind Domain Intervention Strategy no. Indicator 2014 2015 2016 Trend % Change since 2014 2014 2015 2016 Trend % Change since 2014 HIV HIV Treatment 1 Cohort analysis for 12, 24, 36 months 0.591 0.72 0.65 10% #DIV/0! #DIV/0! #DIV/0! HIV HIV Treatment 2 Estimated district need for treatment met (males and females) 3 2 2 -18% - 1 1 HIV HIV Treatment 3 Estimated district need for treatment met (children) 3 3 2 -15% - - - HIV HIV Treatment 4 Number of adults and children currently receiving antiretroviral therapy 607 693 471 168 504 159 -17% 65 857 95 372 102 459 56% HIV HIV Treatment 5 Number of adults and children newly enrolled on ART 260 051 302 150 327 218 26% 21 877 21 107 12 554 -43% HIV HIV Treatment 6 Percentage of adults and children known to be alive and on treatment 12 months after initiation of antiretroviral 0.73 0.74 0.77 5% 0.73 0.74 0.82 13% HIV HIV Treatment 7 Proportion of viral load tests with undetectable viral load (1000copies/ml) 0.83 0.84 0.81 -1% 0.83 0.81 0.85 3% TB TB Treatment 8 Proportion of TB screening and IPT for PLHIV and HTS for all presumptive and diagnosed/confirmed TB patients 0.92 0.92 0.87 -6% 0.95 0.97 0.96 1% TB TB Treatment 9 Sputum conversion rates 0.67 0.71 0.64 -5% 0.69 0.72 0.61 -11% TB TB Treatment 10 TB success rates 0.74 0.78 0.63 -15% 0.81 0.84 0.35 -57% TB TB Treatment 11 TB/HIV proportion on ART treatment 0.81 0.86 0.89 10% 0.91 0.92 0.91 0% HIV Prevention PMTCT 12 Percentage of HIV-positive women who received antiretroviral to reduce risk of mothers -to-child transmission during pregnancy and delivery 0.91 0.90 0.95 5% 0.86 0.97 0.91 6% HIV Prevention Distribution of male and female condoms 13 Costed district condom distribution plan. 0 0 0 0 0 0 HIV Prevention Distribution of male and female condoms 14 Male condom distribution rate 0.29 0.42 0.42 44% 0.27 0.45 0.51 86% HIV Prevention Male medical circumcision 15 Costed district MMC plan 0 0 0 0 0 0 HIV Prevention Male medical circumcision 16 Number of circumcisions performed 192 256 180 540 186 515 -3% 77 430 63 822 56 534 -27% HIV Prevention PICT 17 Number of Individuals who received HIV testing and Counseling services for HIV and received their test result 4 133 766 3 934 549 4 375 875 6% 348 917 440 782 226 871 -35% MCH MCH EBF 18 Proportion infants on EBF at 14 weeks 0.49 0.49 0.38 -23% 0.45 0.32 0.19 -59% MCH MCH FP 19 Couple year protection rate; Proportion of clients on implanon 0.40 0.49 0.45 14% 0.41 0.48 0.50 22% Health Systems Strengthening HSS - DHIS use 20 Appropriately documented minutes of quarterly data review meetings 0 0 0 0 0 0 Health Systems Strengthening HSS - Improving 3-Tier M&E 21 Proportion of all facilities that export monthly signed off ART data to DHIS 0.54 0.82 0.96 79% 0.79 0.94 0.95 21% Health Systems Strengthening HSS - Improving 3-Tier M&E 22 Proportion of Tier 2 facilities reporting appropriately signed off cohort data quarterly 0.45 0.74 0.95 109% 0.79 0.94 0.99 25% Health Systems Strengthening HSS - Support DHP 23 DHP incorporating PEPFAR DSP and other NGO plans; has targets and relevant methods to achieve all the priorities on this list 0 0 0 0 0 0 Health Systems Strengthening HSS - Support ETR-net 24 Proportion of facilities with up to date ETR-net data appropriately signed off and exported to DHIS 0.51 0.55 0.61 19% 0.50 0.50 0.50 0% Health Systems Strengthening HSS - Support Ideal Clinic 25 In each sub-district support at least one clinic to achieve ideal clinic status; core standards everywhere 0 0 0 0 0 0 Health Systems Strengthening HSS - Supporting nurses 26 Written monthly reports of supervision visits to clinics. 61.2 61.56 58.46 -4% 19.2 19.56 20.46 7% Other Capacity Building 27 No. persons trained 6 318 4 242 9 100 44% - - - TB TB Treatment 28 TB defaulter rates 0.05 0.05 0.04 -25% 0.04 0.05 0.01 -71% HIV Prevention PMTCT 29 Early infant transmission rate 0.68 0.65 0.61 -10% 1.01 0.96 0.41 -60% Indicators expected to INCREASE over time Indicators expected to DECREASE over time All Partners Anova External Evaluation: USAID/South Africa “Systems Strengthening for Better HIV/TB Outcomes” Project (2012-2017) P a g e | 82 Ind Domain Intervention Strategy no. Indicator 2014 2015 2016 Trend % Change since 2014 2014 2015 2016 Trend % Change since 2014 HIV HIV Treatment 1 Cohort analysis for 12, 24, 36 months #DIV/0! #DIV/0! #DIV/0! 0.591 0.67 #DIV/0! HIV HIV Treatment 2 Estimated district need for treatment met (males and females) - - - - - - HIV HIV Treatment 3 Estimated district need for treatment met (children) - - - - - - HIV HIV Treatment 4 Number of adults and children currently receiving antiretroviral therapy 168 106 187 770 212 845 27% - - - HIV HIV Treatment 5 Number of adults and children newly enrolled on ART 42 347 45 439 44 668 5% 50 826 61 321 60 335 19% HIV HIV Treatment 6 Percentage of adults and children known to be alive and on treatment 12 months after initiation of antiretroviral #DIV/0! #DIV/0! 0.72 0.70 0.70 0.77 9% HIV HIV Treatment 7 Proportion of viral load tests with undetectable viral load (1000copies/ml) #DIV/0! #DIV/0! 0.87 0.83 0.87 0.43 -48% TB TB Treatment 8 Proportion of TB screening and IPT for PLHIV and HTS for all presumptive and diagnosed/confirmed TB patients #DIV/0! #DIV/0! 0.94 #DIV/0! 0.80 0.76 TB TB Treatment 9 Sputum conversion rates 0.63 0.68 0.68 7% 0.66 0.71 0.60 -8% TB TB Treatment 10 TB success rates 0.72 0.74 0.74 3% 0.64 0.78 0.23 -64% TB TB Treatment 11 TB/HIV proportion on ART treatment 0.77 0.93 0.94 22% 0.87 0.90 0.92 6% HIV Prevention PMTCT 12 Percentage of HIV-positive women who received antiretroviral to reduce risk of mothers -to-child transmission during pregnancy and delivery 0.84 0.91 0.92 9% 0.85 0.97 0.98 16% HIV Prevention Distribution of male and female condoms 13 Costed district condom distribution plan. 0 0 0 0 0 0 HIV Prevention Distribution of male and female condoms 14 Male condom distribution rate 0.22 0.35 0.35 57% 0.26 0.39 0.41 61% HIV Prevention Male medical circumcision 15 Costed district MMC plan 0 0 0 0 0 0 HIV Prevention Male medical circumcision 16 Number of circumcisions performed 23 785 30 448 43 084 81% - - - HIV Prevention PICT 17 Number of Individuals who received HIV testing and Counseling services for HIV and received their test result 516 448 520 066 562 876 9% 1 019 902 1 338 735 1 485 967 46% MCH MCH EBF 18 Proportion infants on EBF at 14 weeks 0.43 0.51 0.34 -19% 0.51 0.48 0.47 -8% MCH MCH FP 19 Couple year protection rate; Proportion of clients on implanon 0.32 0.40 0.36 15% 0.31 0.45 0.43 35% Health Systems Strengthening HSS - DHIS use 20 Appropriately documented minutes of quarterly data review meetings 0 0 0 0 0 0 Health Systems Strengthening HSS - Improving 3-Tier M&E 21 Proportion of all facilities that export monthly signed off ART data to DHIS #DIV/0! #DIV/0! #DIV/0! - 0.63 0.96 Health Systems Strengthening HSS - Improving 3-Tier M&E 22 Proportion of Tier 2 facilities reporting appropriately signed off cohort data quarterly #DIV/0! #DIV/0! #DIV/0! - 0.54 0.96 Health Systems Strengthening HSS - Support DHP 23 DHP incorporating PEPFAR DSP and other NGO plans; has targets and relevant methods to achieve all the priorities on this list 0 0 0 0 0 0 Health Systems Strengthening HSS - Support ETR-net 24 Proportion of facilities with up to date ETR-net data appropriately signed off and exported to DHIS #DIV/0! #DIV/0! #DIV/0! - - - Health Systems Strengthening HSS - Support Ideal Clinic 25 In each sub-district support at least one clinic to achieve ideal clinic status; core standards everywhere 0 0 0 0 0 0 Health Systems Strengthening HSS - Supporting nurses 26 Written monthly reports of supervision visits to clinics. 0 0 0 0 0 0 Other Capacity Building 27 No. persons trained 3 075 1 436 2 264 -26% 2 577 2 800 3 200 24% TB TB Treatment 28 TB defaulter rates 0.04 0.04 0.04 -10% 0.06 0.05 0.02 -68% HIV Prevention PMTCT 29 Early infant transmission rate 0.02 0.02 0.02 6% 2.07 1.60 2.13 3% Indicators expected to INCREASE over time Indicators expected to DECREASE over time BRHC FPD External Evaluation: USAID/South Africa “Systems Strengthening for Better HIV/TB Outcomes” Project (2012-2017) P a g e | 83 Ind Domain Intervention Strategy no. Indicator 2014 2015 2016 Trend % Change since 2014 2014 2015 2016 Trend % Change since 2014 HIV HIV Treatment 1 Cohort analysis for 12, 24, 36 months #DIV/0! #DIV/0! #DIV/0! #DIV/0! 0.77 0.65 HIV HIV Treatment 2 Estimated district need for treatment met (males and females) - - - 1 1 1 16% HIV HIV Treatment 3 Estimated district need for treatment met (children) - - - 0 1 0 114% HIV HIV Treatment 4 Number of adults and children currently receiving antiretroviral therapy 373 730 188 026 188 855 -49% - - - HIV HIV Treatment 5 Number of adults and children newly enrolled on ART 60 546 31 608 34 652 -43% 34 530 57 084 55 208 60% HIV HIV Treatment 6 Percentage of adults and children known to be alive and on treatment 12 months after initiation of antiretroviral 0.82 0.70 0.73 -11% #DIV/0! 0.92 0.91 HIV HIV Treatment 7 Proportion of viral load tests with undetectable viral load (1000copies/ml) 0.82 0.87 0.94 14% #DIV/0! 0.86 0.86 TB TB Treatment 8 Proportion of TB screening and IPT for PLHIV and HTS for all presumptive and diagnosed/confirmed TB patients #DIV/0! #DIV/0! #DIV/0! 0.87 0.95 0.97 11% TB TB Treatment 9 Sputum conversion rates #DIV/0! #DIV/0! #DIV/0! #DIV/0! #DIV/0! #DIV/0! TB TB Treatment 10 TB success rates #DIV/0! #DIV/0! #DIV/0! #DIV/0! #DIV/0! #DIV/0! TB TB Treatment 11 TB/HIV proportion on ART treatment 0.62 0.82 0.85 37% 0.76 0.71 0.69 -9% HIV Prevention PMTCT 12 Percentage of HIV-positive women who received antiretroviral to reduce risk of mothers -to-child transmission during pregnancy and delivery 0.86 0.70 0.96 11% 0.82 0.95 0.97 18% HIV Prevention Distribution of male and female condoms 13 Costed district condom distribution plan. 0 0 0 0 0 0 HIV Prevention Distribution of male and female condoms 14 Male condom distribution rate 0.39 0.34 0.31 -22% 0.20 0.38 0.40 100% HIV Prevention Male medical circumcision 15 Costed district MMC plan 0 0 0 0 0 0 HIV Prevention Male medical circumcision 16 Number of circumcisions performed 11 329 4 895 9 594 -15% 22 161 25 672 22 056 0% HIV Prevention PICT 17 Number of Individuals who received HIV testing and Counseling services for HIV and received their test result 1 355 168 366 864 436 901 -68% 337 032 462 081 754 723 124% MCH MCH EBF 18 Proportion infants on EBF at 14 weeks #DIV/0! #DIV/0! #DIV/0! 0.72 0.26 0.48 -33% MCH MCH FP 19 Couple year protection rate; Proportion of clients on implanon 0.64 0.55 0.48 -26% 0.23 0.63 0.52 126% Health Systems Strengthening HSS - DHIS use 20 Appropriately documented minutes of quarterly data review meetings 0 0 0 0 0 0 Health Systems Strengthening HSS - Improving 3-Tier M&E 21 Proportion of all facilities that export monthly signed off ART data to DHIS 0.50 0.61 0.96 93% 0.51 0.96 0.97 90% Health Systems Strengthening HSS - Improving 3-Tier M&E 22 Proportion of Tier 2 facilities reporting appropriately signed off cohort data quarterly 0.50 0.61 0.96 93% 0.51 0.96 0.97 90% Health Systems Strengthening HSS - Support DHP 23 DHP incorporating PEPFAR DSP and other NGO plans; has targets and relevant methods to achieve all the priorities on this list 0 0 0 0 0 0 Health Systems Strengthening HSS - Support ETR-net 24 Proportion of facilities with up to date ETR-net data appropriately signed off and exported to DHIS 1.00 1.00 1.00 0% - - 0.04 Health Systems Strengthening HSS - Support Ideal Clinic 25 In each sub-district support at least one clinic to achieve ideal clinic status; core standards everywhere 0 0 0 0 0 0 Health Systems Strengthening HSS - Supporting nurses 26 Written monthly reports of supervision visits to clinics. 24 24 24 0% 0 0 0 Other Capacity Building 27 No. persons trained 666 - - -100% - - 3 376 TB TB Treatment 28 TB defaulter rates #DIV/0! #DIV/0! #DIV/0! #DIV/0! #DIV/0! #DIV/0! HIV Prevention PMTCT 29 Early infant transmission rate 0.01 0.01 0.00 -61% 0.01 0.01 - -100% Indicators expected to INCREASE over time Indicators expected to DECREASE over time Kethimpilo Match External Evaluation: USAID/South Africa “Systems Strengthening for Better HIV/TB Outcomes” Project (2012-2017) P a g e | 84 Ind Domain Intervention Strategy no. Indicator 2014 2015 2016 Trend % Change since 2014 2014 2015 2016 Trend % Change since 2014 HIV HIV Treatment 1 Cohort analysis for 12, 24, 36 months #DIV/0! #DIV/0! #DIV/0! #DIV/0! #DIV/0! #DIV/0! HIV HIV Treatment 2 Estimated district need for treatment met (males and females) - - - 2 0 1 -64% HIV HIV Treatment 3 Estimated district need for treatment met (children) - - - 3 2 2 -25% HIV HIV Treatment 4 Number of adults and children currently receiving antiretroviral therapy - - - - - - HIV HIV Treatment 5 Number of adults and children newly enrolled on ART 39 392 58 047 95 896 143% 10 533 27 544 23 905 127% HIV HIV Treatment 6 Percentage of adults and children known to be alive and on treatment 12 months after initiation of antiretroviral 0.68 0.64 0.67 -2% #DIV/0! 0.88 0.91 HIV HIV Treatment 7 Proportion of viral load tests with undetectable viral load (1000copies/ml) 0.83 0.86 0.90 9% #DIV/0! 0.79 0.88 TB TB Treatment 8 Proportion of TB screening and IPT for PLHIV and HTS for all presumptive and diagnosed/confirmed TB patients #DIV/0! #DIV/0! 0.46 #DIV/0! #DIV/0! #DIV/0! TB TB Treatment 9 Sputum conversion rates #DIV/0! #DIV/0! #DIV/0! 0.72 0.73 0.63 -13% TB TB Treatment 10 TB success rates #DIV/0! #DIV/0! #DIV/0! 0.76 0.78 0.81 6% TB TB Treatment 11 TB/HIV proportion on ART treatment 0.85 0.90 0.95 11% #DIV/0! 0.75 0.86 HIV Prevention PMTCT 12 Percentage of HIV-positive women who received antiretroviral to reduce risk of mothers -to-child transmission during pregnancy and delivery 0.91 0.90 0.96 5% 1.25 0.93 0.97 -22% HIV Prevention Distribution of male and female condoms 13 Costed district condom distribution plan. 0 0 0 0 0 0 HIV Prevention Distribution of male and female condoms 14 Male condom distribution rate 0.25 0.37 0.35 43% 0.47 0.72 0.66 39% HIV Prevention Male medical circumcision 15 Costed district MMC plan 0 0 0 0 0 0 HIV Prevention Male medical circumcision 16 Number of circumcisions performed 38 749 40 030 38 871 0% 18 802 15 673 16 376 -13% HIV Prevention PICT 17 Number of Individuals who received HIV testing and Counseling services for HIV and received their test result 388 343 598 324 618 063 59% 167 956 207 697 290 474 73% MCH MCH EBF 18 Proportion infants on EBF at 14 weeks 0.54 0.62 0.36 -34% 0.39 0.44 0.37 -5% MCH MCH FP 19 Couple year protection rate; Proportion of clients on implanon 0.34 0.42 0.38 12% 0.54 0.70 0.61 14% Health Systems Strengthening HSS - DHIS use 20 Appropriately documented minutes of quarterly data review meetings 0 0 0 0 0 0 Health Systems Strengthening HSS - Improving 3-Tier M&E 21 Proportion of all facilities that export monthly signed off ART data to DHIS 1.00 1.00 1.00 0% 0.47 0.85 0.92 97% Health Systems Strengthening HSS - Improving 3-Tier M&E 22 Proportion of Tier 2 facilities reporting appropriately signed off cohort data quarterly 0.63 0.74 0.91 46% 0.47 0.85 0.92 97% Health Systems Strengthening HSS - Support DHP 23 DHP incorporating PEPFAR DSP and other NGO plans; has targets and relevant methods to achieve all the priorities on this list 0 0 0 0 0 0 Health Systems Strengthening HSS - Support ETR-net 24 Proportion of facilities with up to date ETR-net data appropriately signed off and exported to DHIS 1.00 1.00 1.00 0% 0.59 0.59 0.97 66% Health Systems Strengthening HSS - Support Ideal Clinic 25 In each sub-district support at least one clinic to achieve ideal clinic status; core standards everywhere 0 0 0 0 0 0 Health Systems Strengthening HSS - Supporting nurses 26 Written monthly reports of supervision visits to clinics. 0 0 0 18 18 14 -22% Other Capacity Building 27 No. persons trained - 6 4 - - 256 TB TB Treatment 28 TB defaulter rates #DIV/0! #DIV/0! #DIV/0! 0.08 0.07 0.07 -14% HIV Prevention PMTCT 29 Early infant transmission rate 0.06 0.02 0.02 -72% 1.20 1.75 1.25 4% Indicators expected to INCREASE over time Indicators expected to DECREASE over time RTC WRHI External Evaluation: USAID/South Africa “Systems Strengthening for Better HIV/TB Outcomes” Project (2012-2017) P a g e | 85 DISTRICT CASE STUDY – ETHEKWINI Among South Africa’s eight (8) Metropolitan municipalities (Metros), eThekwini Metro, located in eastern KwaZulu-Natal (KZN), has the highest HIV prevalence. eThekwini has a population of 3.5 million and an HIV prevalence rate of 14.5% - above the national average of 12.2%24, but lower than the 16.9% for KZN as a whole, the highest rate among all provinces. MatCH (Maternal Adolescent and Child Health) is the District Support Partner (DSP) providing Health System Strengthening / Capacity Building (HSS/CB) support to eThekwini Metro’s HIV and TB program. MatCH has actively supported the metro since 2013. The purpose of this case study is to summarise the evaluation findings specific to eThekwini Metro and not to repeat the findings presented in the main body of the evaluation report. As with the full report, this case study focuses on the three (3) year period 2014-2016, and briefly describes:  MatCH’s Health Systems Strengthening model,  the trends of its HIV/TB capacity building support to the district (in terms of human and financial resources),  perceptions of DOH respondents around MatCH’s contribution to strengthening the key building blocks of Health Systems and the most significant changes that in each of these blocks since 2014,  partnerships / linkages formed by MatCH in an effort to further strengthen the HIV/TB program. MatCH Health System Strengthening /Capacity Building (HSS/CB) Model MatCH’s HSS/CB implementation model is similar to that of other partners and includes a combination of the elements below:  Developing management and leadership skills for facility managers, and sub - district/district management teams to improve planning and resource allocation,  Implementing Direct Service Delivery (DSD) models where DSP technical and support staff are seconded to DOH to support service delivery,  Technical assistance to improve patient flow and reduce waiting times, e.g. a scheduling system for patients and integration of HIV and other chronic conditions, and.  Introducing and/or supporting innovative PHC facility decongestion strategies, including decanting of stable ART patients, community adherence support groups and CCMDD models. 24 Shisana, O., Rehle, T., Simbayi, L.C. et al. (2014) ‘South African National HIV Prevalence, Incidence and Behaviour Survey, 2012’ http://www.hsrc.ac.za/uploads/pageContent/4565/SABSSM%20IV%20LEO%20final.pdf External Evaluation: USAID/South Africa “Systems Strengthening for Better HIV/TB Outcomes” Project (2012-2017) P a g e | 86 Design of the MatCH HSS/CB Program Few DOH respondents (Figure 35) were aware of any formative research carried out to inform the development of the MatCH HSS/CB program. One DOH respondent who answered “no” felt strongly that there was a general lack of appreciation among partners for engagement with the DOH and its needs prior to the development of their programs , stating “They should identify with the DOH what we are looking for before they come to us after getting the funding… They then start putting pressure on DOH to provide them with work while we have many other partners. Some just decide what area they are going to work.” Interestingly, however, most DOH respondents (67%) said that MatCH considered the absorptive capacity of the DOH in designing its HSS/CB program and activities (Figure 36). The same DOH respondent as above, now speaking in glowing terms about MatCH’s activities, states: “MatCH does not do what they want to do, they build on us to do what we need to do and help us improve our work and reach our target… They meet the DOH’s needs in terms of the right skills, human capacity, and technology.” Half of DOH respondents (50%) believe MatCH’s HSS/CB activities is aligned with DOH policies, guidelines and priorities and a third (33%) consider it somewhat aligned ( Figure 37 ) Figure 35. eThekwini - DSP Formative Research Conducted Prior to HSS Strategy Design (DOH Respondents only) Figure 36. eThekwini - DSP Considered Absorptive Capacity of DOH Prior to HSS Strategy Design (DOH Respondents only) External Evaluation: USAID/South Africa “Systems Strengthening for Better HIV/TB Outcomes” Project (2012-2017) P a g e | 87 Figure 37. eThekwini – MatCH Aligned HSS Strategy Design to DOH’s Structures and Processes (DOH Respondents only) The following section will highlight the HSS/CB investment, human and financial, made by MatCH, over time, towards strengthening the HIV/TB program. It will also highlight the most significant changes in each of the health system building blocks since 2014 as well as the DOH respondent’s perceptions about the extent to which MatCH contributed to the changes. HSS/CB Investment – Human and Financial Resources and Associated Trends Given the difficulty in establishing consistent units of measure across the different DSP programs and HSS/CB activities, the evaluation team decided to use two proxy mea sures for quantifying HSS/CB -- the amount of money and human resources (in full time equivalent or FTE) dedicated to HSS/CB activities. Because all FTE and expenditure data were self - reported by the District Support Partners, the evaluation team was unab le to validate the numbers provided. Nevertheless, a general understanding of the DSPs’ resource allocation toward HSS/CB can be deduced from the analyses below. The trends in human and financial investment made by MatCH to strengthen eThekwini’s HIV/TB program is presented in Table 17 and Table 18 below. The human investment is measured in Full-time Equivalent (FTE) while the financial investment is shown in US dollars. Table 17 illustrating MatCH’s FTE investment by programmatic domains of HIV and TB, and the % change over time shows that human resource investment grew dramatically from 2014 to 2016 in all programmatic domains except for Provider Initiated Testing and Counselling and HSS-Strategic Information. However, the patterns observed in FTE changes are not mirrored in the changes in associated expenditure over time. The spark lines in the tables show the trend from 2014 to 2016 for both forms of investment and illustrate a sharp change corresponding to the DOH & PEPFAR pivot in 2015 towards 90-90-90 and UTT. External Evaluation: USAID/South Africa “Systems Strengthening for Better HIV/TB Outcomes” Project (2012-2017) P a g e | 88 Table 17. eThekwini - MatCH: Human Resources (in FTE) and Associated Expenditure (US$) Dedicated to HSS/CB by Programmatic Domain (2014 – 2016) Table 18 below, presents MatCH’s human and financial investment by Capacity Building Activities. Since 2014, there has been a large increase in personnel providing hands -on support to the DOH in the form of seconded staff, Direct Service Deliver (DSD), and roving clinical teams. In terms of expenditure, all types of HSS/CB activities have enjoyed increased investment except for DSD whose increased budget dropped by 48% in 2016 to below 2014 levels. Similar to Table 17 above, the spark lines in the table below illustrate the change over time as well the 2015 pivot. Table 18. eThekwini - MatCH: Human Resources (in FTE) and Associated Expenditure (US$) Dedicated to HSS/CB by Capacity Building Activity (2014 – 2016) To What Extent, and How, did MatCH Strengthen Health Systems at DOH Management Levels? In general, DOH respondents have favourable views of MatCH’s HSS/CB support and activities. All (100%) of DOH respondents consider MatCH’s approach to training, mentoring, and technical support to be effective in maximising learning. Furthermore, they all indicated that they were able to apply what they had learned. Most DOH supporters believe that MatCH has strengthened the DOH’S internal management and monitoring processes – particularly at the facility level (Figure 38) – and credit MatCH for strengthening the implementation and management of HIV/TB program s. However, only half (50%) believe that the DOH’s ability to plan and design HIV/TB programs has been strengthened by MatCH, suggesting that the focus of MatCH support was largely on strengthening services delivery and not DOH capacity to plan or design services. . Programmatic Domain 2014 2015 2016 Total Trend % Change since 2014 2014 2015 2016 Total Trend % Change since 2014 HTC-PITC 225 420 292 9 37 -30% 1 9,580 8,557 3 2,414 6 0,550 279% HTC-VCT 6 3 220 312 5 95 395% 4 6,902 5 1,455 6 1,464 159,822 31% PMTCT 4 0 230 447 7 17 1018% 180,230 191,878 302,795 674,903 68% Facility-based care, treatment and support 159 633 1,269 2,061 698% - 1 7,430 4,310 2 1,740 #DIV/0! HSS - District Planning / supervision - 2 8 10 300% 6,038 8,885 4,698 1 9,621 -47% HSS - Pharmacy/ Supply Chain Management 1 1 850 8 52 84900% 1 2,139 1 4,910 5,164 3 2,213 -65% HSS-Strategic Information 5 207 164 3 76 -21% 140 - - 140 #DIV/0! Infection Control - - - - #DIV/0! 1,270 1,193 9,655 1 2,118 660% Laboratory - 1 6 7 #DIV/0! 5 1,600 2 3,229 1 3,620 8 8,449 -74% Other 4 36 105 1 45 2525% - - - - #DIV/0! Ethekwini Ethekwini FTE for HSS by Capacity Building Activity Associated Expenditure HSS Capacity Building Activity 2014 2015 2016 Total Trend % Change since 2014 2015 2016 Total Trend % Change since 2014 Direct Service Delivery (DSD) 25.00 81.00 100.00 206.00 23% 7 7,991 131,143 6 8,719 277,853 -48% Temporary Seconded Staff - - 93.00 9 3.00 #DIV/0! - - 7 7,536 7 7,536 #DIV/0! Mentoring 21.00 - 24.00 4 5.00 14% 4 7,368 2 0,065 3 5,896 103,328 -24% Supportive Supervision 10.00 27.00 41.00 7 8.00 310% 7 7,187 8 7,696 9 2,003 256,886 19% Roving Clinical Teams 28.00 20.00 122.00 170.00 510% 130,006 8 5,686 167,554 383,246 96% Training 413.00 1 ,622.00 3 ,073.00 ###### 89% 4,380 8,519 1 6,642 2 9,541 95% Other - - - - #DIV/0! - - - - #DIV/0! Match Match FTE for HSS by Capacity Building Activity Associated Expenditure External Evaluation: USAID/South Africa “Systems Strengthening for Better HIV/TB Outcomes” Project (2012-2017) P a g e | 89 Figure 38. DSP Strengthened DOH's Internal Management and Monitoring System (DOH Respondents Only) In terms of monitoring processes, many DOH respondents (67%) credit MatCH with significantly improving the various M&E information systems in eThekwini district (Figure 39). Figure 39. M&E systems strengthened by the MatCH (DOH respondents only) DOH Perspectives on MatCH’s Contribution to HSS Building Blocks and Patient Outcomes According to DOH respondents, MatCH has contributed to the strengthening of all health system building blocks – albeit to differing degrees (Figure 40). Health Workforce Strengthening is the building block for which MatCH is given the most credit, followed closely by Health Information Systems. Some of the most significant changes which DOH respondents have identified as having taken place since 2014 (and the implications for service delivery) are presented in Table 19. DOH respondents credit MatCH with strengthening health systems overall, and also positively influencing key indicators such as improved patient outcomes (83% of DOH respondents) (Figure 41), increasing health coverage for key population (67%), and improved quality of HIV and TB services as a whole (100%). MatCH was particularly acknowledged for strengthening the health workforce and health information services, leading to: • Improving TB cure rates since TB is detected and managed earlier • Better management co-infected patients according to treatment protocols pre_ART_modul HCT_Module Tier_net Pt_Tracking SVS DHIS2 WISN Other eThekwini: Most Commonly Mentioned M&E system that was Strengthened (DOH respondents only) External Evaluation: USAID/South Africa “Systems Strengthening for Better HIV/TB Outcomes” Project (2012-2017) P a g e | 90 Figure 40. eThekwini - DSP Contribution to HSS Building Blocks (DOH Respondents Only) Figure 41. eThekwini - DOH Respondents who agreed that DSP Support contributed to Improved Patient Outcomes Table 19. eThekwini - Most Significant Changes (MSC) in HSS Building Blocks and Implications for Service Delivery Most Significant Change (MSC) Implications for Improved Service Delivery Health Workforce Strengthening (83% of DOH respondents recognised MATCH’s contribution to MSC) Increased skills transfer (and related increases in competence and confidence) Improved quality of care Increased variety of patients treated e.g. mothers, children, adolescents and adults Increased number of nurses that can initiate and manage HIV/TB case (NIMART): Increased ART initiation Workload shared amongst trained staff Reduced work overload; happier staff Increase in number and capacity of people working in HIV/TB Better integration of HIV and TB services Provision of extra staff from MATCH via secondment or roving supporting teams (Data capture, nurses, pharmacist assistant, etc.) Increased provision of HIV/TB services Reduced workload for DOH staff Improved quality of care (via on-site mentoring and support) Health Information System Building Block (67% of DOH respondents credit MatCH for MSC) Implementation of Tier.net in facilities Improved access to patient data Back capturing of data onto Tier.net via seconded Improved data for program planning and monitoring Workforce Health Information Systems Service Delivery District Implementation Planning Pharmaceutical and Lab District Health Planning Leadership, Management, Governance eThekwini: DOH Attribution of HSS Building Block Change to DSP efforts None of the change M ost of the Change External Evaluation: USAID/South Africa “Systems Strengthening for Better HIV/TB Outcomes” Project (2012-2017) P a g e | 91 Most Significant Change (MSC) Implications for Improved Service Delivery staff/DSD (e.g. VL data) Improved patient monitoring by clinicians leading to better case management e.g. VLs Improved defaulter tracing Service Delivery (50% of DOH respondents recognised MATCH’s contribution to MSC) Increase number of NIMART trained nurses Improved access to care Improved quality of care Better access to ARVs, better control on facilities stock out Decanting of clinics allowing patients to access ARVs in places they are more comfortable Decongestion of clinics Reduced Waiting time for patients Improved access of treatment for patients Increased adherence Support from MATCH for adherence clubs, identification of convenient pick up points, and improved monitoring of patients’ adherence Improvement of patient care Improvement of clinic standards Improvement in patient clinical management (VL, PCR, defaulters) Increased adherence Increased retention Reduced congestion of clinics Improved staff capacity to manage HIV and TB Improved access to and quality of patient care Leadership, Management and Governance (33% of DOH respondents credit MatCH for MSC) Increased ownership and commitment of HIV/TB program by DOH (reflected in changes in policies such as UTT and 90-90-90) Increased accountability for results e.g. UTT, 90-90-90 strategy Increased access to treatment Targeted approach centred on patient outcomes Improved patient care Improved implementation of policies and guidelines Improved quality of care and patient outcomes Increased understanding, management and use of data Improved data quality Improved evidence-based decision making at all levels of the health system Improved facility management skills - Patient Flow - Team Management - Structural Changes Improved efficiency and effectiveness in service delivery District Health Planning (33% of DOH respondents credit MatCH for MSC) DHP guided by 90-90-90 Increased accountability for HIV/TB results An integrated and synergistic district plan with 5 pillars to structure/ guide HIV/TB activities (prevention, case finding, adherence, treatment and care and HSS) Integration of HIV/TB interventions Improved facility (clinic) operational plans as a result of MATCH support Increased evidence based planning Increased ownership and accountability for results Better alignment across different district plans – DHPS, DIPs and DOPs Improved performance management of the HIV/TB program - increased efficiency and effectiveness District Implementation Plan (17% of DOH respondents credit MatCH for MSC) Increased cooperation between DOH and partners (the DIP development process is funded and facilitated by MatCH) Improved performance management of the HIV/TB program External Evaluation: USAID/South Africa “Systems Strengthening for Better HIV/TB Outcomes” Project (2012-2017) P a g e | 92 Most Significant Change (MSC) Implications for Improved Service Delivery Joint planning between, and uniformity across, metro and provincial DIP development (MATCH supported the alignment of the Metro and provincial DIPs) - increased efficiency and effectiveness - Increased accountability for HIV/TB results Harmonisation between the DIP and facility operational plans (with MATCH support) DIP is disease specific (e.g. HIV/TB): Increased accountability for HIV/TB results Pharmacy and Laboratory Building Block (17% of DOH respondents recognised MATCH for MSC) MSC in Laboratory Systems Decrease in number of blood specimen sample rejected by the Lab Improved quality of care to patients Turnaround time to get result reduced SMS printer 50- 50% Availability of SMS printers and support from MATCH staff in accessing and sharing lab results with facilities Improvement in PEDHIV patient blood sampling Increased care to Paediatric cases MSC in Pharmaceutical Systems CCMDD and decanting mechanism Improvement in stock management especially in Primary Health Care Clinics (PHCC) Improved patient access to treatment Decongestion of facilities reduced waiting time decreased work load for nurses Implementation of Rx solution Improved tracking and management of stock Pharmacist Assistant seconded by MATCH Linkages and Partnerships Match established a wide range of partnerships and linkages with other PEPFAR -supported programs (Figure 42) most notably with VMMC programs. Community treatment, however, was the least cited program, reflecting MatCH’s emphasis on facility -based care and treatment services. Figure 42. Linkages and Partnerships established by MatCH Recommendations KZN has made significant progress in its HIV and TB program and particularly in achieving VMMC OVC CHWs, etc Other GOSA Prevention Community HCT Community Treatment eThekwini: MatCH established Linkages and Partnerships External Evaluation: USAID/South Africa “Systems Strengthening for Better HIV/TB Outcomes” Project (2012-2017) P a g e | 93 the first 90. While there is still room for improvement in identifying new patients, the focus now needs to be on the 2nd and 3rd 90s as individuals are not linking to care following testing. The following recommendations are made in consideration of this: • Strengthen the referral process • Strengthen adherence to treatment • Strengthen monitoring of VL and VL suppression results, speed LAB process, consider innovative ways of testing VL • Implement a unique patient identifier system to track patients and identify defaulters • Build on the progress made in pharmacovigilance e.g. promote good practices External Evaluation: USAID/South Africa “Systems Strengthening for Better HIV/TB Outcomes” Project (2012-2017) P a g e | 94 DISTRICT CASE STUDY – CITY OF JOHANNESBURG Gauteng comprises the largest share of South Africa’s population with approximately 12.9 million people (approximately 24%) living in the province. The City of Johannesburg (COJ) is the district characterized by rapid urbanization with a highly transient population resulting in high HIV prevalence and incidence rates, particularly in urban informal areas. The sexually active group (males and females 15 – 49 years) represents almost 60% of Gauteng’s population and HIV prevalence amongst this group is 17.8%25. The HIV prevalence rate in COJ was found to be 11.1%, which is lower than the national average of 12.2%25 . Within the COJ district, three District Support Partners (DSPs) were assigned to the regions as follows: Anova Health Institute (Anova): • City of Johannesburg Region C • City of Johannesburg Region D • City of Johannesburg Region E • City of Johannesburg Region G Right to Care (RTC): • City of Johannesburg Region A • City of Johannesburg Region B Wits Reproductive Health and HIV Institute (WRHI): • City of Johannesburg Region F DSPs Health System Strengthening / Capacity Building (HSS/CB) Model Against the backdrop of the COJ’s diverse, highly mobil e, and informal urban communities, where social conditions are marked by poverty, gender asymmetries and instability, it was critical that DSPs designed their HSS models to be responsive to existing conditions. This view is supported by the recognition that HIV programs need to be differentiated and locally adapted to the relevant epidemiological, socioeconomic, and cultural contexts in which they are to be implemented26. Additionally, interventions are more likely to succeed if they are built on genuinely participatory approaches and an understanding of the local dynamics of HIV risk and vulnerability. As per Figure 43 below, 64% of DOH respondents agreed that DSPs implementing HSS programs in COJ had conducted formative research with most respondents (83%) especially recognising the formative research conducted by RTC and WRHI. The types of formative 25 Shisana, O., Rehle, T., Simbayi, L.C. et al. (2014) ‘South African National HIV Prevalence, Incidence and Behaviour Survey, 2012’ http://www.hsrc.ac.za/uploads/pageContent/4565/SABSSM%20IV%20LEO%20final.pdf 26 Scorgie, F., Veary, J.,Oliff, M. et al. (2017) ‘Leaving no one behind: reflections on the design of a community-based HIV prevention for migrants in Johannesburg’s inner-city hostels and informal settlements’ https://bmcpublichealth.biomedcentral.com/articles/10.1186/s12889 -017-4351-3 External Evaluation: USAID/South Africa “Systems Strengthening for Better HIV/TB Outcomes” Project (2012-2017) P a g e | 95 research conducted by the three DSPs included: • Skills audits at PHC facility level • Baseline assessments • PHC Facility Staff Needs Assessments DOH respondents appeared to be most satisfied with a participatory approach, ongoing facility assessments, and skill audits wherein assessment results are shared with DOH staff. RTC is credited for using a continuous version of the “Plan, Do, Study, Act” (PDSA) model of program implementation. Figure 43. COJ - DSP Formative Research Conducted Prior to HSS Strategy Design (DOH Respondents only) Absorptive capacity of the health system, including pre-existing DOH tools, skills, staff and support systems, represents the ability to identify, assimilate, transform and apply newly gained external knowledge. During the design of their HSS mod els, it was important that partners considered the DOH’s absorptive capacity so as to ensure the HSS/CB strategy achieved the desired outcomes of improved quality of service delivery, and ultimately, improved patient outcomes. Figure 44 below shows that almost 75% of DOH respondents believed that DSPs did consider and adapt the HSS program design to the DOH’s absorptive capac ity; thus, improving efficiency and effectiveness of program implementation. Importantly, DSPs’ work within the policy/guideline landscape of both the National and provincial Departments of Health. As such, it is imperative that the HSS program was align ed to the DOH structures and processes. Indeed, more than 70% of DOH respondents felt that the HSS program design was well-aligned with DOH’s structures and processes (Figure 45). The following sections focus on the contribution made by DSPs towards strengthening the individual HSS building block within the HIV/TB Program. The discussion combines Health Systems Strengthening / Capacity Building (HSS/CB) activities at both management level and facility level, with a focus on Government satisfaction with the partners’ various inputs. 0% 25% 50% 75% 100% Anova RTC WHRI Multiple COJ: DSP Conducted Research to Design HSS Strategy? Percent of DOH respondents who said "yes" Yes Don't Know / Unsure External Evaluation: USAID/South Africa “Systems Strengthening for Better HIV/TB Outcomes” Project (2012-2017) P a g e | 96 Figure 44 COJ: DSP Considered Absorptive Capacity of DOH Prior to HSS Strategy Design (DOH Respondents only) Figure 45 COJ: HSS Strategies Designed to DOH’s Structures and Processes (DOH Respondents only) To What Extent, and How, did Partners Strengthen Health Systems Given the difficulty in establishing consistent units of measure across the different DSP programs and HSS/CB activities, the evaluation team decided to use two proxy measures for quantifying HSS/CB – the amount of money and human resources (in full time equivalent or FTE) dedicated to HSS/CB activities. Because all FTE and expenditure data were self - reported by the District Support Partners, the evaluation team was unable to validate the numbers provided. Nevertheless, a general understanding of the DSPs’ resource allocation toward HSS/CB can be deduced from the analyses below. ANOVA Looking at the programmatic areas where ANOVA targeted its HSS/CB investments (Figure 46), there is an overall trend of increasing resource allocation, with HR commitments more than doubling over the 2014- 2016 period. ANOVA allocated most HR and expenditure to facility-based care, treatment and support. This is consistent with the overall DSP aim of strengthening primary health care so as to improve service delivery and patient outcomes. 0% 25% 50% 75% 100% Anova RTC WHRI Multiple COJ: DSP considered DOH Absorbtive Capacity in HSS Design? (Percent of DOH Respondents who said "yes" ) Yes Don't Know / Unsure 0% 25% 50% 75% 100% Anova RTC WHRI Multiple COJ: The DSP's HSS approach is aligned to DOH structures and processes? Percent of DOH Respondents who said "yes" Well aligned Somewhat aligned External Evaluation: USAID/South Africa “Systems Strengthening for Better HIV/TB Outcomes” Project (2012-2017) P a g e | 97 Examining the specific capacity building areas where ANOVA invested its resources, direct services delivery/staff secondment, mentoring, and training received the greatest increases from 2014-2016 (Figure 47). This is consistent with the 2015 PEPFAR/South Africa pivot toward achieving the 90-90-90 strategic goals, which emphasise increasing services delivery. Figure 46. COJ Anova - FTE and Expenditure by Programmatic Domain (2014 – 2016) FTE by Programmatic Domain Expenditure by Programmatic Domain Figure 47. COJ Anova – FTE and Expenditure by Capacity Building Activity (2014-2016) FTE by Capacity Building Activity Associated Expenditure by Capacity Building Activity External Evaluation: USAID/South Africa “Systems Strengthening for Better HIV/TB Outcomes” Project (2012-2017) P a g e | 98 Right to Care Right to Care, in contrast, invested most of its HR resources to Practitioner Initiated Testing and Counselling (PITC) which aims to improve the yield rate of testing programs. It can be seen that most expenditure was consumed towards facility-based care, treatment, and support, which is consistent with attempts to strengthen targeted testing programs (Figure 48). Right to Care’s investments demonstrate a general trend of increased resource allocation such that HR commitments almost tripled between 2014 and 2016. In terms of capacity building activities, Right to Care targeted human resources (FTE) towards direct service delivery and roving clinical teams (Figure 49). Unfortunately, the expenditure data received from RTC was not in the correct format. Despite this, it is clear from the trend lines in the figure below, that expenditure towards direct service delivery/staff secondment and roving clinical teams increased between 2014 and 2016. Figure 48. COJ Right to Care: Human Resources (in FTE) and Associated Expenditure by Programmatic Domain (2014 – 2016) FTE by Programmatic Domain Expenditure by Programmatic Domain External Evaluation: USAID/South Africa “Systems Strengthening for Better HIV/TB Outcomes” Project (2012-2017) P a g e | 99 Figure 49. COJ: Right to Care FTE and Associated Expenditure by Capacity Building Activity (2014-2016) FTE by Capacity Building Activity Associated Expenditure by Capacity Building Activity Wits Reproductive Health and HIV Institute WRHI’s programmatic areas focus for HSS/CB activities (Figure 50) show a broad trend of increased resource allocation with more than a 50% increase in HR commitments between 2014 and 2016. While WRHI increased expenditure across all programmatic domains fairly equally (2% increase) – except for HR commitments to strengthen the health information system which saw a 6% decrease in expenditure – the biggest focus appears to be on HTC; voluntary testing and counselling (a 120% increase) over the period 2014 t o 2016. This is consistent with the UNAIDS strategy since individuals have to be tested to enter the 90 -90- 90 clinical cascade. Since 2014, Wits Reproductive Health Institute has mostly directed its HR and associated expenditure, towards the provision of temporary seconded staff (Figure 51). External Evaluation: USAID/South Africa “Systems Strengthening for Better HIV/TB Outcomes” Project (2012-2017) P a g e | 1 0 0 Figure 50. COJ Wits Reproductive Health Institute: Human Resources (in FTE) and Associated Expenditure by Programmatic Domain (2014 – 2016) FTE by Programmatic Domain Expenditure by Programmatic Domain External Evaluation: USAID/South Africa “Systems Strengthening for Better HIV/TB Outcomes” Project (2012-2017) P a g e | 1 0 1 Figure 51. COJ: Wits Reproductive Health Institute - FTE and Associated Expenditure by Capacity Building Activity (2014-2016) FTE by Capacity Building Activity Associated Expenditure by Capacity Building Activity The DOH Perspective on the DSP’s Contribution to Main HSS Building Blocks and Patient Outcomes The following discussion outlines COJ DOH perceptions around the contributions made by the three DSPs to improve patient outcomes and service delivery by strengthening the various health system building blocks. The most significant changes that DOH credits the DSPs for are: • All facilities are able to initiate adults onto ART; • Improved TB cure rate – an increase from 69-70% to 80-82%; and, • Improved data and performance management. This finding is corroborated by the figure below (Figure 52) which shows that majority (75%) of DOH respondents felt that, in general, the 3 DSPs contributed to improved patient outcomes with 100% of DOH respondents crediting RTC and WRHI for this improvement and 60% of respondents recognising Anova’s contribution. Aligned to the most significant changes above, DOH recognised DSPs for: • Formal NIMART training and ongoing facility-based mentoring, o Nurses are now able to initiate children • Improved viral load monitoring and suppression o Back capturing of viral load results o Support to interpret and act on viral load results External Evaluation: USAID/South Africa “Systems Strengthening for Better HIV/TB Outcomes” Project (2012-2017) P a g e | 1 0 2 • Establishment of adherence clubs • Support for CCMDD models that support decongestion of PHC facilities and improve adherence including: o Multi-month scripting/dispensing, o Fast track lanes, with pre-packed medicines, o Pharmaceutical Dispensing Units, and, • Improved performance management o Data interrogation to inform QA/QI strategies o Support for DIP/DHP processes o Direct service delivery/staff secondment initiatives to assist with excessive workloads. Figure 52. COJ: DOH Respondents who agree that DSP Support contributed to Improved Patient Outcomes Figure 53. COJ: DOH Respondents who agreed that DSP Support contributed to Improved Service Delivery 0% 25% 50% 75% 100% Anova RTC WHRI Multiple COJ: DSP support has contributed to improving patient outcomes? Percent of DOH Respondents who "agreed" Strongly agree Agree Don't Know / Unsure 0% 25% 50% 75% 100% Anova RTC WHRI Multiple COJ: Services Delivery- Attribution of District Support Partner's efforts to Change by PEPFAR Partner (DOH respondents only) Most of the change (60% and above) Equal effort in bringing the change (50-50) Don't Know / Unsure External Evaluation: USAID/South Africa “Systems Strengthening for Better HIV/TB Outcomes” Project (2012-2017) P a g e | 1 0 3 Nearly half of all DOH respondents credited the DSPs for improving service delivery (Figure 53), with 60% recognising the contribution made by Anova and half (50%) of respondents acknowledging RTC’s and WRHI’s contribution. Specific contributions made to strengthen each of the building blocks so as to improve service delivery and patient outcomes are outlined below: Table 20. COJ: Most Significant Change to Each HSS Building Block and its Implication for Service Delivery Most Significant Change (MSC) Implication for Improved Service Delivery Health Workforce Strengthening (72% of DOH respondents recognised DSPs contribution to MSC) NIMART support, mentoring and formal training  Increased number of HIV positive patients newly initiated onto ART  Improved ART retention rates  Improved access to HIV-related services by priority population (men)  Improved TB/HIV co-infection clinical management  Improved capacity to manage greater volumes of clients for HIV-related services. Adherence counselling support and training of PHC staff Routine TB screening of all HIV positive patients Support for improved men-friendly care and treatment DSP staff secondment (temporary and/or fixed) Training of WBOT Team Leaders to support HCT-VCT at community level  Increased number of individuals newly tested and who know their results (first 90)  Increased understanding for improved productivity of facility staff Health Information System Building Block (74% of DOH respondents credit the DSPs for MSC) Rationalization of Registers  Decreased administrative workload – more time to focus on service delivery  Improved availability of program data for clinical and administrative decision-making Support for implementation of Tier.net - In COJ, 100% of PHC facilities using Tier.net Staff secondment/DSD models to support data capturing, including back-capturing of VL results Improved data quality  Improved ability to make accurate strategic decisions regarding program implementation/course correction. Support for increased use of program data for planning Pharmacy and Laboratory Building Block (68% of DOH respondents recognised DSPs for MSC) MSC in Laboratory Systems Specimen collection done twice daily  Improved laboratory turnaround time (mostly 48 hours with exception of 72 hours)  Supports Universal Test and Treat strategy for, as soon as possible, ART initiation. Introduction/implementation of Lab Track software Improved quality assurance procedures Implementation of SMS printers Back-capturing of lab results into patient files Increased access to VL results by clinicians and support to interpret and act on VL results MSC in Pharmacy Systems Introduction/implementation of Stock Visibility Solution and Rx Solution software to monitor stock levels/consumption rates  Improved pharmaceutical supply management  Decreased stock-outs and shrinkage of stocks Temporary secondment of Pharmacy staff to PHC facilities to support pharmaceutical supply management Support for roll-out of Fixed Dose Combination drug  Improved ART adherence rates and associated increased viral suppression/patient survival rates Support for CCMDD models, e.g. fast-lane tracks, multi￾month scripting/dispensing, pharmaceutical dispensing External Evaluation: USAID/South Africa “Systems Strengthening for Better HIV/TB Outcomes” Project (2012-2017) P a g e | 1 0 4 Most Significant Change (MSC) Implication for Improved Service Delivery units  Decreased adverse reactions due to launch of FDC drugs improves patient adherence rates.  Decongestion of PHC facilities allows nurses to attend to ‘new’ clients. Expansion of community- and facility-based adherence clubs Support for WISN analysis Leadership, Management and Governance (38% of DOH respondents credit DSPs for MSC) DOH introduced/Implemented the DIP process  Improved accountability between DSPs and DOH  Improved coordination between DSPs and DOH initiatives  Improved capacity of Facility Managers and sub￾district DOH staff to identify and manage emerging bottlenecks DOH established Technical Working Groups DSP technical assistance Formal management training DSP support for development of strategic plans including APP and DIP Improved buy-in from management staff across DOH Support for M&E and data utilisation for decision-making District Implementation Planning (29% of DOH respondents attribute MSC to DSPs) Root cause analysis of performance data, including facility￾level data, to support DIP  Improved performance management of program implementation  Increased accountability of both DOH and DSPs  Improved efficiency and effectiveness Targeted approach (for improved impact/efficiency) to planning Increased buy-in from sub-district and PHC facility staff Improved target setting Integration of HIV/TB response with other key programs Inability of District Management Team to support and coordinate implementation of DIP  Implementation failure  HRH and client frustration District Health Planning (21% of DOH respondents attribute MSC to DSPs The adoption of the 90-90-90 strategy  Improved, but inadequate, resource allocation for 90-90-90 targets, and,  Implementation failure Increased facility-level involvement in DHP process Integration and alignment of targets and indicators between DHP and DIP Planning and targets more reasonable and achievable Improved availability of information for planning Partnerships, Linkages and Innovative Practices DSPs used partnerships, community linkages and other innovative practices to enhance their work so as to reach targets for various performance indicators. As per Figure 54 below, almost 90% of DOH respondents agreed that DSPs working in COJ had established linkages for the various programmatic areas. Most DOH participants felt that linkages were established for community-based testing, HIV prevention and VMMC programs. External Evaluation: USAID/South Africa “Systems Strengthening for Better HIV/TB Outcomes” Project (2012-2017) P a g e | 1 0 5 Figure 54. COJ: Percent of DOH Respondents who agreed that DSP Established Linkages and Partnerships by type of Linkage In terms of community-testing programs, DSPs are credited for the following:  Anova established a partnership with HIVSA to pilot a home-based testing program, ‘Friends for Life’,  DSPs partnered with PEPFAR-funded Orphan and Vulnerable Children (OVC) organisations such as NACCW for community-based testing initiatives,  Expanding the coverage of existing community-based cadres, e.g. Ward Based Outreach Teams (WBOTs) and Community Healthcare Workers for outreach activities targeting hard-to-reach areas such as farms.  Establishing Public-Private Partnerships, e.g. o With Discovery Health for HCT services at taxi ranks o With Vodacom to supply necessary infrastructure and equipment for a maternity clinic in Hillbrow. HIV prevention partnerships included:  Linkages with Department of Basic Education and Social Development for VMMC interventions  Social mobilisation to increase uptake of VMMC services using community-based OVC partners, e.g. NACOSA. In addition to the above linkages/partnerships, DSPs were recognised for employing innovative practices to scale up and improve program implementation as follows:  By leveraging off the existing DREAMs program, DSPs supported the supply of youth-friendly HIV-related services to girls and young women during ‘Friday Clinics’ and on Saturday mornings, Community HCT Prevention VMMC Other GOSA CHWs, etc OVC Community Treatment Private Sector COJ: DSP established Linkages and Partnerships? Percent of DOH Respondents who said "yes" External Evaluation: USAID/South Africa “Systems Strengthening for Better HIV/TB Outcomes” Project (2012-2017) P a g e | 1 0 6  Young boys were also targeted for testing and other HIV prev ention activities using soccer clubs, e.g. Anova support for the ‘Score 4 Life’ in Alexandra informal settlement,  M-Health strategies, including: o RTC’s adherence application o Anova’s Centre for HIV and AIDS Prevention Studies (CHAPS) to increase uptake of VMMC services o Anova’s Health4Men which is an eHealth resource for MSM. Recommendations PEPFAR to: 1. Engage provincial department of health as well as the COJ Metropolitan Municipality to: a. Better understand the needs of both the provincial and metropolitan mun icipality departments of health to inform program design, b. Establish buy-in from critical government structures who act as gatekeepers to the service delivery level, i.e. PHC facilities c. Improve alignment of PEPFAR and DOH structures planning, implementation and MER d. Provide technical support to key DOH structures throughout the project cycle of planning, implementation, monitoring and evaluation and course correction. 2. Identify and appoint a managing partner who, in addition to their HSS program implementation, coordinates and collaborates across DSPs for effective and efficient program implementation. 3. Determine DSPs’ expertise and assign HSS building blocks to specific partners with the appropriate knowledge and technical skill. Thereafter, all DSPs to roll-out interventions developed by ‘expert’ partners across all COJ regions. DSPs to 1. Share and engage DOH structures on their scopes of work and implementation plans 2. Establish robust feedback loops on issues related to PEPFAR program reviews and data quality. DOH to: 1. Roll-out unique patient identifiers, including biometrics, for COJ’s highly transient and mobile population 2. Strengthen overall HR performance management system. External Evaluation: USAID/South Africa “Systems Strengthening for Better HIV/TB Outcomes” Project (2012-2017) P a g e | 1 0 7 DISCLOSURE OF ANY CONFLICTS OF INTEREST External Evaluation: USAID/South Africa “Systems Strengthening for Better HIV/TB Outcomes” Project (2012-2017) P a g e | 1 0 8 External Evaluation: USAID/South Africa “Systems Strengthening for Better HIV/TB Outcomes” Project (2012-2017) P a g e | 1 0 9 External Evaluation: USAID/South Africa “Systems Strengthening for Better HIV/TB Outcomes” Project (2012-2017) P a g e | 110 U.S. Agency for International Development 1300 Pennsylvania Avenue, NW Washington, DC 20523