EVALUATION Evaluation of USAID/South Africa Tuberculosis Program (FY2010-FY2014) FINAL: December 2014 This publication was produced at the request of the United States Agency for International Development. It was prepared independently by Mary Pat Selvaggio, Catherine Rogers, Jennifer Peters, Zach Akiy, and Salome Omolo of Khulisa Management Services, Pty (Ltd). Evaluation of the USAID/South Africa Tuberculosis Programme (FY2010-FY2014 Page | ii ACKNOWELDGEMENTS The evaluation team is grateful to the more than 169 individuals who generously gave their time and insights for this evaluation report. Throughout this evaluation, the team benefited from the contributions of dozens of Department of Health (DOH) directors, managers and health care providers, at central level and at all levels of the health system across South Africa’s nine provinces. The team is extremely grateful to University Research Corporation (URC) for making their staff so readily available to the team and for assisting us with the scheduling and introduction to key informants during the fieldwork and data collection. We would also like to thank the CDC, PEPFAR and NGO grantees who took part in the evaluation. The team would especially like to thank Nellie Gqwaru, COR and Paula van Dyk, Program Officer at USAID/South Africa for their thoughtful guidance throughout the evaluation process. This report represents a collective effort and incorporates opinions and observations of a large community of individuals. To all who are committed to improving the situation of TB (and HIV/AIDS) in South Africa, we offer these observations, analyses, and recommendations in the confidence that further progress will be realized through collective efforts. Evaluation of the USAID/South Africa Tuberculosis Programme (FY2010-FY2014 Page | iii EVALUATION OF THE USAID TB PROGRAM IMPLEMENTED BY UNIVERSITY RESEARCH CORPORATION (URC) (FY2010-FY2014) December 2014 Project Number: Cooperative Agreement No. 674-C-00-09-00121-00 Evaluation Contract Number: AID-674-I-12-00002, Order No. AID-674-TO-14-00008 DISCLAIMER The authors’ views expressed in this publication do not necessarily reflect the views of the United States Agency for International Development or the United States Government. Evaluation of the USAID/South Africa Tuberculosis Programme (FY2010-FY2014 Page | i TABLE OF CONTENTS Acronyms ..................................................................................................................................................... iv 1 Executive Summary ..............................................................................................................................1 2 Evaluation Purpose & Evaluation Questions.........................................................................................8 3 Project Background ...............................................................................................................................9 3.1 Context ...............................................................................................................................................9 3.2 Project Description ...........................................................................................................................10 4 Evaluation Methods & Limitations .......................................................................................................16 5 Findings ...............................................................................................................................................20 5.1 Evaluation Question 1: To what extent did the project achieve its intended results as stated in the contract’s objectives? What were the reasons for any shortfalls?....................................................20 5.2 Evaluation Question 2: To what extent is the design of this project valid? How successful have been the programmatic and management approaches, structures and systems in carrying out the project’s activities? ...........................................................................................................................29 5.3 Evaluation Question 3A: Did the project strengthen the capacity of NTP sufficiently to ensure its sustainability?...................................................................................................................................37 5.4 Evaluation Question 3B: What role has the Small Grants program played in improving TB, TB/HIV and MDR-TB programs? ..................................................................................................................46 5.5 Evaluation Question 4: What strategies were used to reach vulnerable populations? E.g. Mines and work place programs .................................................................................................................58 5.6 Evaluation Question 5: How has the project integrated technology? e.g. mobile health work in KZN, Active TB case finding......................................................................................................................64 6 Conclusions .........................................................................................................................................65 7 Recommendations ..............................................................................................................................67 Annex 1: Evaluation Statement of Work ....................................................................................................69 Annex 2: Detailed Methodology .................................................................................................................74 Annex 3: Sources of Information ................................................................................................................81 Annex 4: Data Collection Instruments ........................................................................................................89 Annex 5: Team Members’ Disclosure of Conflicts of Interest ....................................................................98 List of Tables Table 1. Key Project Strategies and Activities ...........................................................................................11 Table 2. Consortium Partnerships .............................................................................................................12 Table 3. Summary of Respondents and Topics for Data Collection Methods ...........................................16 Table 4. KIIs – Targeted and Actual ..........................................................................................................17 Table 5. Online Surveys – Targeted and Actual ........................................................................................18 Table 6. Achievement of Project Outcome Indicators ................................................................................21 Table 7. Summary of Respondents and Topics for Data Collection Methods ...........................................74 Table 8. KIIs – Targeted and Actual ..........................................................................................................75 Table 9. Online Surveys – Targeted and Actual ........................................................................................76 Evaluation of the USAID/South Africa Tuberculosis Programme (FY2010-FY2014 Page | ii Table 10. Sample of Districts and Sub-districts visited for KIIs ................................................................. 77 Table 11. List of CSOs to be targeted for KIIs. ..........................................................................................78 Table 12. Documents and Data Reviewed ................................................................................................. 81 List of Figures Figure 1. Implementation Structure of the TB Program .............................................................................11 Figure 2. The Evaluation Team’s Understanding of the Project’s Theory of Change................................12 Figure 3. DOH districts supported by URC over the Life of the Project (LOP) ..........................................13 Figure 4. Grantees funded per province ....................................................................................................14 Figure 5. No. small grants issued by wave ................................................................................................14 Figure 6. Amount of Funding for small grants by wave .............................................................................15 Figure 7. Summary of Project Progress towards Indicators ......................................................................20 Figure 8. Smear Conversion Rate .............................................................................................................22 Figure 9. Treatment Success Rate ............................................................................................................22 Figure 10. TB Cure Rate ............................................................................................................................23 Figure 11. Defaulter Rate ...........................................................................................................................23 Figure 12. HIV Testing Amongst TB Cases ...............................................................................................23 Figure 13. TB Screening for HIV Positive Clients ......................................................................................23 Figure 14. Ave Treatment Success Rates at District Level: 2010-2012 ...................................................25 Figure 15. Average Defaulter Rate at District Level: 2010-2012 ..............................................................25 Figure 16. Ave Treatment Success Rates at Sub-district Level: 2010-2012 ............................................25 Figure 17. Average Defaulter Rate at Sub-district Level: 2010-2012 .......................................................25 Figure 18. Percent of TB patients tested for HIV by District: 2010-2012 ...................................................26 Figure 19. Percent of HIV patients screened for TB by District: 2010-2012 ..............................................26 Figure 20. Percent of TB patients tested for HIV by Sub-district: 2010-2012 ............................................26 Figure 21. Percent of HIV patients screened for TB by Sub-district: 2010- 2012 ......................................26 Figure 22. Quality of URC Coordination with DOH ....................................................................................31 Figure 23. Fit of URC’s TB project with the NTP .......................................................................................32 Figure 24. Views on Project Capacity Building Strengths ..........................................................................33 Figure 25. No. Training Activities by FY and Funding Source ...................................................................38 Figure 26. No. Trainees Reached by FY and Funding Source ..................................................................38 Figure 27. The type of online trainings that grantees report receiving .......................................................39 Figure 28. The perceived effectiveness of the online training received ......................................................39 Figure 29. DOH respondent’s views on areas where the project significantly improved capacity in the NTP ..................................................................................................................................................41 Figure 30. Grantee respondent’s views on areas where the project significantly improved their organization’s capacity ..............................................................................................................43 Figure 31. Project’s implementation approach vis-à-vis capacity building ................................................44 Figure 32. Types of activities conducted by small grantees ......................................................................47 Figure 33. Grantee perceptions on the most significant change the Small Grants Program has made ...48 Figure 34. Grantee opinions on alternative TB support .............................................................................50 Evaluation of the USAID/South Africa Tuberculosis Programme (FY2010-FY2014 Page | iii Figure 35. Grantee perceptions of URC communication with their organization. ......................................51 Figure 36. NGOs’ rating of the quality of formal training received .............................................................52 Figure 37. Effectiveness of the Grants Management Team in supporting the CSOs ................................. 52 Figure 38. Grantee perceptions of the URC's responsiveness to financial and contractual concerns ......53 Figure 39. Grantee perceptions on the responsiveness of URC’s project team. .......................................54 Figure 40. Effectiveness of the Technical Team support to CSOs .............................................................55 Figure 41. Description of URC technical support .......................................................................................55 Figure 42. Shortcomings of URC support ..................................................................................................56 Figure 43. Vulnerable populations accessed by CSOs .............................................................................62 Figure 44. CSO constraints to accessing vulnerable populations .............................................................62 Figure 45. Map of districts visited during the evaluation. ...........................................................................88 Evaluation of the USAID/South Africa Tuberculosis Programme (FY2010-FY2014 Page | iv ACRONYMS ACSM Advocacy, Communication & Social Mobilization AIDS Acquired Immunodeficiency Syndrome ART Antiretroviral Therapy ARV Antiretroviral CBO Community Based Organization CDC Centers for Disease Control and Prevention CPT Co-trimoxazole Preventative Therapy CSO Civil Society Organizations DCS Department of Correctional Services DDOH District Department of Health DENOSA Democratic Nursing Organization of South Africa DFID Department of International Development DHET Department of Higher Education and Training DHIS District Health Information System DOH Department of Health DOTS Directly Observed Treatment Shortfalls DRAT District Rapid Appraisal Tool DR-TB Drug Resistant Tuberculosis EC Eastern Cape Province eHealth Electronic Health EOP End of the Project ETR Electronic TB Register FS Free State Province FY Fiscal Year GF Global Fund GFATM Global Fund to Fight AIDS, Tuberculosis and Malaria GP Gauteng Province HAST HIV/AIDS, STIs and TB HCT HIV/AIDS Counselling and Testing HCW Health Care Worker HIV Human Immunodeficiency Virus HST Health Systems Trust ICF Intensified Case Finding IEC Information Education and Communication ILO International Labour Organization IPT Isoniazid Preventive Therapy IR Intermediate Result IUATLD International Union Against TB & Lung Disease Evaluation of the USAID/South Africa Tuberculosis Programme (FY2010-FY2014 Page | v JHHESA John Hopkins Health Education South Africa KII Key Informant Interview KZN KwaZulu-Natal LOP Life of the Project LP Limpopo Province M&E Monitoring and Evaluation MDR-TB Multi-Drug Resistant Tuberculosis mHealth Mobile Health MP Mpumalanga Province NC Northern Cape Province NDOH National Department of Health NGO Non-Governmental Organization NHLS National Health Laboratory Services NIMART Nurse Initiated Management of ART NJH National Jewish Health NTP National Tuberculosis Control Program NW North West Province PDOH Provincial Department of Health PEPFAR The President’s Emergency Plan for AIDS Relief PMP Performance Monitoring Plan PPM Public Private Mix QI Quality Improvement RFA Request for Applications RSA Republic of South Africa SABCOHA The South African Business Coalition on Health and AIDS SAG South African Government SAQA South African Qualification Authority SPSS Statistical Package for Social Sciences STI Sexually transmitted disease TB Tuberculosis THP Traditional Health Practitioners UN United Nations UNEP United Nations Environmental Program URC University Research Co., LLC USAID United States Agency for International Development USD United States Dollar USG United States Government WB World Bank WC Western Cape Province WHO World Health Organization Evaluation of the USAID/South Africa Tuberculosis Programme (FY2010-FY2014 Page | vi XDR-TB Extremely Drug Resistant Tuberculosis ZAR South African Rand Evaluation of the USAID/South Africa Tuberculosis Program (FY2010-FY2014) Page | 1 1 EXECUTIVE SUMMARY EVALUATION PURPOSE AND EVALUATION QUESTIONS The purpose of this evaluation was to review the performance of the USAID/South Africa TB project for developing a follow-on project to support South Africa’s National TB Programme (NTP). The evaluation aimed to answer five questions: 1. To what extent did the project achieve its intended results as stated in the contract’s objectives? What were the reasons for any shortfalls? 2. To what extent is the design of this project valid? How successful have been the programmatic and management approaches, structures and systems in carrying out the project’s activities? 3. (a) Did the project strengthen the capacity of NTP sufficiently to ensure its sustainability? (b) What role has the small grants program played in improving Tuberculosis (TB), TB/ Human Immunodeficiency Virus (HIV) and Multi-Drug Resistant Tuberculosis (MDR-TB) programs? 4. What strategies were used to reach vulnerable populations? 5. How has the project integrated technology? In answering these questions, the evaluation sought to: assess the quality of the project’s design, technical inputs, and implementation to improve TB outcomes; determine which approaches and activities were employed, which were successful and why; assess the effectiveness of the project in strengthening TB prevention and control in the Republic of South Africa (RSA); identify actual outcomes achieved; and summarize key results and effectiveness of the project in achieving its intended objectives. Key audiences for this report include: USAID/South Africa, the Department of Health (DOH), University Research Council Co., LLC (URC) and partners and other donors and implementing partners working in HIV and TB in South Africa. PROJECT BACKGROUND Globally, South Africa ranks second in TB incidence and fifth in TB prevalence (per 100,000 population)1 . The country’s TB incidence rate means that 1 of every 100 persons (or 1% of the general population) develops TB annually. South Africa has the highest burden of drug resistant tuberculosis in the world2 . In 2012, there were an estimated 15,000 cases of MDR-TB of which nearly half (42%) of detected cases were not on treatment3 . TB/HIV co-infection is extremely common in South Africa with 65% of all TB patients also infected with HIV. From 2009-11, South Africa received donor funds totaling USD8.3 million for TB versus USD595.11 1 WHO. Global TB data (2014). https://extranet.who.int/tme/generateCSV.asp?ds=estimates 2 WHO. Global Tuberculosis Report 2013. apps.who.int/iris/bitstream/10665/91355/1/9789241564656_eng.pdf 3 Global Fund, Program Scorecard: South Africa HIV and TB (2013). www.theglobalfund.org/ProgramDocuments/ZAF/Common/SAF-H-T-13_GSC_0_en/ Evaluation of the USAID/South Africa Tuberculosis Program (FY2010-FY2014) Page | 2 million for HIV; as such, TB funding represents only 1% of the total funds between the two disease areas, although it accounts for a far greater proportion of the country’s mortality rate4 . The USAID TB Program is a 5-year (2009-2014) USAID-funded project implemented by the University Research Corporation (URC) that provides technical support to South Africa’s NTP for improving: early case detection, access to diagnostics, patient compliance with treatment regimens, and care and treatment, including Antiretroviral (ARVs), for TB/HIV co-infected patients. The project’s goal is to build the capacity of Government and non-government entities to deliver more effective TB prevention and control efforts, with a focus on strengthening health systems at national level (namely policies, and data/information systems) and improving TB service delivery at provincial, district and community level. The project has five Intermediate Results (IRs) including: Increased quality, availability and demand for TB services (IR1-3); Improved management of TB support systems (IR4); and testing of new approaches for expanding Directly Observed Treatment Shortfalls (DOTS) coverage (IR5). EVALUATION DESIGN, METHODS AND LIMITATIONS This evaluation was conducted by a team of six consultants over a three month period (8 September - 10 December 2014) and covered a four-year period from project inception through September 2014. The evaluation utilized a non-experimental design using a mix of qualitative and quantitative data collection and analysis: Document and data review: including project documents and reports, donor and government reports, and data from the Electronic TB register (ETR.net) to assess progress towards selected TB indicators and other achievements as well as constraints; Key Informant Interviews: with 111 respondents across four categories: URC staff, the DOH, USAID and other donors and Non-Governmental Organization (NGO) grantees. Online Surveys: with 60 respondents for 3 cadres of respondents: President Emergency Plan for AIDS Relief (PEPFAR) partners, NGO grantees, and facility managers. Limitations to the evaluation included: missing data (including project data for IR indicators); data presented in inconsistent structures or formats; and difficulty in conducting fieldwork or accessing personnel in “graduated” districts. FINDINGS, CONCLUSIONS AND RECOMMENDATIONS Evaluation Question 1: To what extent did the project achieve its intended results as stated in the contract’s objectives? What were the reasons for any shortfalls? The TB Project reports on 21 indicators: 6 at outcome level and 15 across the project’s five IRs. Of these 21 indictors, 6 (or 29%) were achieved or likely to be achieved by the end of project (EOP), while 5 (or 24%) were not or not likely to be achieved at EOP. There was insufficient data in Years 3-4 to evaluate progress for the remaining 10 indicators. However the majority of Key Informant Interviews (KIIs) felt the project was likely to achieve its IR indicators. 4 Mapping the Donor Landscape in Global Health: HIV and TB Reports, Kaiser Foundation, 2013. Evaluation of the USAID/South Africa Tuberculosis Program (FY2010-FY2014) Page | 3 Three of the project’s 6 outcome indicators were achieved at the end of Year 4, and the project was close to achieving the other three. While the project may not achieve all its indicators by EOP, URC’s Performance Monitoring Plan (PMP) and annual reports show progressively positive trends for all six outcomes in Years 1–4. Although not captured by PMP indicators, URC was widely applauded for their high level of technical expertise in TB, bottom-up approach, responsiveness, and support to the DOH, both in capacity building and skills transfer as well as in systems building. In examining the performance of project-supported districts vs districts that received no support (using NTP data), there is evidence that the project contributed to better TB outcomes, such as Treatment Success Rate, Default Rate, and TB screening among HIV patients. Overarching constraints to the achievement of key NTP (and URC) TB outcomes include:  Defaulter rate: Limited or no systems or methodology for community workers, notably in defaulter tracing and early case detection;  TB cure rate: The emergence and increasing caseload of drug resistant (DR) TB and inadequate coverage of HIV and TB services for vulnerable populations;  HIV testing amongst TB cases: Little/no training staff nurses who serve as primary TB focal points, and late presentation of TB patients. Evaluation Question 2: To what extent is the design of this project valid? How successful have been the programmatic and management approaches, structures and systems in carrying out the project’s activities? Project design: The TB Project is closely aligned with the strategic objectives of the NTP, and 85% of respondents felt the project’s broad scope was an advantage allowing the necessary flexibility to respond to shifts in the TB epidemic and context-specific DOH requests. Over 80% of DOH respondents felt the project struck the right balance between systems and individual capacity building. Key constraints in project design included the geographical targeting of 24 of the country’s total 52 districts without flexibility to shift geographic focus over the life of the project (LOP). The relative attention, focus and funding for HIV and resulting subordination of the importance of TB in overall HIV/TB programming was a further constraint. Lack of involvement of the National Health Laboratory System (NHLS) in project design and/or as a formal partner hampered the project’s ability to engage or successfully improve laboratory functioning as desired. Programmatic and management approaches: Across all KII respondents, URC received unprecedented acclaim, praise, and recognition for its strong relationship building and close collaboration with the DOH at all levels. Notably 100% of DOH respondents praised URC for its close and collaborative relations with the DOH, as well as with partners and donors. 94% of all KIIs, and notably 100% with the DOH, praised URC’s bottom-up approach. 90% of all KIIs (and 60% of DOH respondents) cited limited and/or high turnover of URC staff, particularly at provincial and sub-district levels as the primary internal constraint to implementation. The project has no clear minimum criteria for “graduating” facilities or sub￾districts once TB outcomes have improved and new skills are deemed to be institutionalized and/or likely to be sustained. All KIIs cited the PEPFAR realignment and resulting shift from district to provincial level partner as the primary external constraint to the project. Challenges Evaluation of the USAID/South Africa Tuberculosis Program (FY2010-FY2014) Page | 4 stemming from this include: limited ability to monitor and supervise implementation of new skills and systems at district and sub-district levels; dependence on the varying levels of interest and engagement from PEPFAR partners in implementing the project’s broader TB mandate; and increasing requests for formal training which further stretched project resources. Constraints in DOH staffing and other resources also presented a considerable challenge. Evaluation Question 3A: Did the project strengthen the capacity of NTP sufficiently to ensure its sustainability? The project assisted the NDOH to standardize and co-facilitate over 300 trainings for more than 20,000 health care workers. Respondents praised the project’s mix of formal training, mentoring and supervision, noting that trainings and mentoring greatly expanded their TB knowledge and capacity. Of the project’s 9 focal intervention areas, respondents perceived HIV/TB integration and Monitoring and Evaluation (M&E) to be the strongest, with the least improvements achieved in drug supply management and laboratory functioning5 . Respondents cited lack of training certificates and South African Qualification Authority (SAQA) accreditation as the primary constraint in the sustainability of individual capacity building efforts. Limited progress in improving laboratory functioning was also noted; more work is needed to improve routine use of the GeneXpert algorithms as well as turnaround times in some facilities. Systems strengthening efforts included: supporting the development of national TB Guidelines and MDR treatment protocols; joint data analysis exercises with DOH staff at district level using the District Rapid Appraisal Tool (DRAT); quarterly TB/HIV meetings with DOH staff to improve TB and TB/HIV services; development and dissemination of numerous tools and registers for TB diagnosis and M&E. Successes in capacity building efforts were identified at facility level: there have been significant improvements in screening for co-infection amongst TB and HIV patients. Uptake of Co-trimoxazole Preventative Therapy (CPT) has improved significantly. ART uptake also improved though more work is needed. Isoniazid Preventive Therapy (IPT) uptake remains a significant challenge though activities were undertaken in 2013 to improve this6 . These results suggest that while training and systems strengthening have had a positive effect on TB services, there is more to be done. Most respondents felt the project’s approach was sufficient to strengthen the NTP and that strides had been made towards sustainability of efforts, but that elements of this support were not institutionalized and thus unlikely to be sustained. Indeed, 100% of DOH respondents felt URC support should be continued – particularly given constraints and turnover in DOH staffing and the need for on-going training and support – before the NTP could reasonably sustain improved TB outcomes without further assistance. Evaluation Question 3B: What role has the Small Grants program played in improving TB, TB/HIV and MDR-TB programs? The objective of the small grants program was to implement community based strategies to improve early case detection, contact and defaulter tracing and treatment adherence, and general TB and TB/HIV knowledge and support in communities. The project issued 100 small 5 URC’s mandate did not include drug supply, though they provided facility level support to avoid stock outs and improve supply chain management. 6 USAID TB Project South Africa, Annual Report (1 October 2012 to 30 September 2013) Evaluation of the USAID/South Africa Tuberculosis Program (FY2010-FY2014) Page | 5 grants to 75 NGOs over the LOP; the vast majority of which focused on intensified case finding, DOTs and Advocacy, Communication and Social Mobilization (ACSM). NGOs reported that the most significant changes resulting from these grants were: improved capacity of the grantees’ organization and staff; increased availability of TB services; increased ACSM and thus community knowledge of TB; increased TB specific expertise; development and strengthening of relationships with facilities and the DOH; and employment of local community members. SAG and URC respondents consistently recognized the value of community-based grantees in extending the reach of the NTP and increasing access to health services at community level. Most (83%) small grantee respondents indicated that the support received from URC was effective, though the long distance management of grantees (from the Pretoria based grants team) was cited by 25% of respondents – and recognized by URC – as a constraint in supporting and supervising grantees as frequently as desired. 95% of grantees felt URC’s financial and contractual support was effective, while considerably less (71%) indicated that URC’s programmatic support was effective, citing the long distance, infrequent and/or “unpredictable” support as key constraints. In contrast, DOH respondents felt the small grants program was insufficiently scaled to adequately address needs at community level and unanimously requested greater focus and investment in community level work in the future. The project’s open solicitation process which did not prioritize NGOs according to location, coupled with the stringent eligibility criteria required of applicants, resulted in many grantees being funded for work outside the project’s focal geographic areas. Ideally, grantees would have been purposively selected according to their location so as to pair them with priority districts/sub-districts that were the focus of the project’s DOH support. Furthermore, the project had no minimum package of services to be implemented in communities, resulting in variations in scope and activities implemented, regardless of community needs or DOH requirements. Numerous SAG respondents reported few or no grantees operating in their area and some expressed frustration at the lack of coordination and communication with local grantees. In addition, the NTP’s data collection and reporting system is not able to “tag” data from NGOs nor to correlate or quantify the contributions made by NGOs at community level. Others were limited in their ability to manage and implement more technical activities (e.g. MDR support) if they did have a trained health care worker (e.g. professional nurse) on their staff. Grantees and SAG respondents alike reported that the one-year grant funding cycle was too short to demonstrate a marked or sustainable impact. Gaps and delays between contracts and funding further constrained their work. 90% of NGO respondents felt that if URC support ended it would negatively TB outcomes and 80% would not be able to continue their work, as they have no alternative sources of funding. Evaluation Question 4: What strategies were used to reach vulnerable populations? The project’s vulnerable populations initially include prisoners, miners and farm workers, though children were added later in response to the growing caseload amongst this age group. While the project did not explicitly define strategies for achieving this objective, the evaluation team identified three cross-cutting strategies that responded to this. ACSM activities to improve awareness in communities, including mobile populations such as miners and farm workers who resided there. This was an indirect strategy, as the project’s ASCM activities were not specifically designed or implemented to target vulnerable populations. Evaluation of the USAID/South Africa Tuberculosis Program (FY2010-FY2014) Page | 6 Public Private Mix (PPM) activities, which involved the private sector (mines, industries, medical schemes and traditional health practitioners (THP) in TB service delivery. The Public/Private mix strategy directly engaged mines, prisons, transport and other private companies to address challenges in access to TB care, treatment and support in the private sector. The small grants program to increase access to rural, hard to reach communities, vulnerable populations and informal settlements. Some grantees were funded to target mines, farms, general industries and correctional institutes while others targeted rural and remote communities and informal settlements. Several grantees targeted MDR-TB patients in high burden areas. Evaluation Question 5: How has the project integrated technology? Technology was not a specific objective of the project; it was an activity under IR5 for testing new approaches to expand DOTS coverage. In 2011, the project presented a mobile health (mHealth) application in response to the DOH’s desire to improve community-based management of DR-TB through development of appropriate tools and real-time information systems. Jointly with GeoMed, the project developed an innovative web-based mHealth solution to more effectively bring healthcare directly to the community and enable mobile health workers to link mapped households with home based care services. Geo-mapping of DR-TB cases was piloted in three high burden provinces: Eastern Cape (EC), Gauteng province (GP), and Free State (FS). The main limitation cited by KIIs in FS was the difficulty in identifying respondents’ addresses; patients often provided a work address rather than one for the communities in which they lived. As a result, vulnerable populations were spread more widely than expected, and the Geo-mapping information was therefore not put to use by the DOH. The project piloted another mHealth solution for contact tracing of DR-TB patients in KwaZulu Natal (KZN) which unfortunately ended and was never scaled up. The University of Pretoria reportedly requested use of the KZN technology under their grant, but never received permission and thus was unable to implement this. As technology was not a focal area of the project, URC, understandably, has no technologically trained staff to advocate for new technological solutions. Recommendations The current project should: 1. Document the successes and lessons learned in the current project for scale-up and increase institutionalization and sustainability of approaches and activities in the future. The follow on project should: 2. Re-establish the balance between formal training and mentoring, onsite capacity building and coaching; consider alternative mentoring approaches; as well as an increase in provincial staffing, possibly roving teams to provide timely and adequate support to poor performing districts/sub-districts. 3. Provide accreditation and certification for TB training and improve database management of trained Health Care Workers (HCWs) to facilitate tracking of skilled HCWs and NTP managers. 4. Develop and define clear criteria for “graduating” facilities, sub-districts and districts from project support. Evaluation of the USAID/South Africa Tuberculosis Program (FY2010-FY2014) Page | 7 5. Develop and maintain effective and efficient internal M&E and knowledge management systems. Ensure management, accumulation, and dissemination of project best practices. 6. Increase PPM efforts to assist with early case finding, contact tracing and treatment adherence. Adopt a holistic approach targeting all vulnerable groups in the community. 7. Assist NTP to expand community level work in TB prevention and treatment, including: a. building on best practices and lessons learnt from 20 years of HIV community￾based programmes b. improved continuity of NGO and community grants so as not to compromise services delivery required to achieve NTP goals by: i. facilitating longer term grants; ii. increasing sustainability of NGOs efforts through capacity building in fund-raising from non-URC sources; iii. purposively selecting grantees in priority districts and building their capacity to improve linkages to facilities and expand community work; iv. utilizing intermediate or umbrella NGOs to coordinate smaller community based organizations where possible. 8. Include the NHLS as a formal partner to improve early and accurate diagnosis and appropriate treatment regimes. 9. Pursue technological innovations and new approaches to improve defaulter and contact tracing in collaboration with a team of mHealth experts to ensure successful implementation is taken to scale. Recommendations for USAID/PEPFAR: 10. Increase alignment/inclusion of key TB outcome level indicators across all HIV/TB programs and partners. Specifically, include the following two TB indicators: (1) TB related mortality amongst HIV co-infected and (2) TB defaulters amongst co-infected. Evaluation of the USAID/South Africa Tuberculosis Program (FY2010-FY2014) Page | 8 2 EVALUATION PURPOSE & EVALUATION QUESTIONS USAID/South Africa requires this evaluation to assist it in developing a follow-on project for support to South Africa’s national TB programme. This evaluation aims to answer five questions: 1. To what extent did the project achieve its intended results as stated in the contract’s objectives? What were the reasons for any shortfalls? 2. To what extent is the design of this project valid? How successful have been the programmatic and management approaches, structures and systems in carrying out the project’s activities? 3. (a) Did the project strengthen the capacity of NTP sufficiently to ensure its sustainability? (b) What role has the Small Grants program played in improving TB, TB/HIV and MDR￾TB programs? 4. What strategies were used to reach vulnerable populations? E.g. Mines and work place programs. 5. How has the project integrated technology e.g. mobile health work in KZN, Active TB case finding? In answering the evaluation questions, the evaluation seeks to: i. assess the quality of the project’s design and implementation; ii. determine which approaches and activities are working and why; iii. assess the effectiveness of the URC contract in strengthening of TB prevention and control efforts in South Africa; iv. ascertain the technical quality of the inputs and activities used to improve TB outcomes; v. identify the actual outcomes achieved; vi. ascertain the approaches employed to carry out activities and management systems utilized to implement contract obligations, including managing funding sources and communication strategies; and vii. provide results on the likely effectiveness of the project and whether it has met its intended objectives. Evaluation of the USAID/South Africa Tuberculosis Program (FY2010-FY2014) Page | 9 3 PROJECT BACKGROUND 3.1 Context Of all countries in the world, South Africa ranks second in TB incidence (per 100,000 population) and fifth in TB prevalence (per 100,000 population)7,8 . The 2012 TB incidence rate for the general population indicates that 1 of every 100 people in South Africa develop TB every year. South Africa had an estimated 15,000 cases of MDR-TB in 20129 , the highest burden of drug resistant tuberculosis in the world8 . However, a significant number of detected cases (42%)8 were not on treatment in 2012. TB/HIV co-infection is extremely common in South Africa with 65% of all TB patients also infected with HIV8 . South Africa’s response to TB is embedded in the goals of the National Strategic Plan on HIV, STIs and TB 2012-201610:  Halving the number of new HIV infections;  Ensuring that at least 80% of people who are eligible for treatment for HIV are receiving it (at least 70% should be alive and still on treatment after five years);  Halving the number of new TB infections and deaths from TB;  Ensuring that the rights of people living with HIV are protected, and;  Halving the stigma related to HIV and TB. When comparing donor assistance available for TB and HIV in South Africa, the contrast is stark. South Africa ranks in the top five countries in sub-Saharan Africa who received the largest share of funding (14% of total, from 23 donors). From 2009-11, South Africa had a total of USD 595.11 million donor funding (86% from USAID and 5% from Global Fund to Fight AIDS, Tuberculosis and Malaria (GFATM) allocated to it for HIV. In comparison, SA received 8.31 million or 4% of all donor funding for TB programming from 2009-2011 (84% from the USA). Yet TB accounts for a far greater proportion of South Africa’s mortality rate11 . Indeed, worldwide, fewer donors provide TB assistance compared to HIV/AIDS and malaria assistance, and donor funding for TB is highly concentrated among a small number of donors - the GFATM alone providing more than half of all 7 WHO. Global TB data (2014). Available from: https://extranet.who.int/tme/generateCSV.asp?ds=estimates 8 WHO. Global Tuberculosis Report 2013. Available from: http://apps.who.int/iris/bitstream/10665/91355/1/9789241564656_eng.pdf 9 Global Fund, Program Scorecard: South Africa HIV and TB. July 2013. Available from: http://www.theglobalfund.org/ProgramDocuments/ZAF/Common/SAF-H-T-13_GSC_0_en/ 10 National Strategic Plan on HIV, STIs and TB 2012-2016. www.sahivsoc.org 11 Mapping the Donor Landscape in Global Health: HIV and TB Reports, Kaiser Foundation, 2013 8.31 595.11 Funding for TB vs HIV/AIDS in South Africa, 2009-2011 (USD millions) TB HIV/AIDS Evaluation of the USAID/South Africa Tuberculosis Program (FY2010-FY2014) Page | 10 TB assistance worldwide and the top five donors together providing over 95%. More than 15 years ago, dire predictions of the impact of HIV on TB and MDR-TB in South Africa were made. What at the time were mere assumptions are now coming true. Substandard care, fertile conditions for transmission and the rapidly progressing HIV epidemic all impede the ability of South Africa to reach the required targets for TB control; they also contribute to establishing the endemicity and spread of drug-resistant TB. As donors and recipient countries look forward to the future, and seek to achieve the ambitious goals laid out in the Global Stop TB Plan12, it will be more important than ever to ensure there is adequate and effective coordination between donors and recipients in order to achieve the greatest return possible on the global investments being made in the TB response. A dynamic and exceptionally strong collaboration between HIV and TB control programs will be required to avert large-scale HIV-associated epidemics of drug-resistant TB. 3.2 Project Description The USAID TB Program – implemented by University Research Co., LLC (URC) – is a 5-year (September 2009- September 2014), USAID-funded technical assistance project that supports South Africa’s National Tuberculosis Control Program in expanding and improving: early case detection, access to laboratory diagnostics, patient compliance with TB treatment regimens, and appropriate and timely HIV care, including ART treatment for eligible patients co-infected with HIV13. The project’s strategies and activities are designed around 5 objectives or intermediate results (IRs):  IR1: Increasing the quality of TB services  IR2: Increasing the availability of TB services  IR3: Increasing the demand for TB treatment  IR4: Improving management of TB support systems; and  IR5: Testing and scaling up new approaches for expanding Directly Observed Treatment Shortfalls (DOTS) coverage. The USAID TB program builds on the successes of an earlier 5-year TB project and incorporates best practices identified under that project. The current project builds the capacity of Government and non-government entities to deliver more effective TB prevention and control efforts in both health facilities and at community levels, with a focus on strengthening health systems at national level (namely policies and data/information systems) and improving TB services delivery at provincial, district and community level (Figure 1). The project employed a wide range of strategies and activities to achieve its overall goals and objectives, as depicted in Table 1. 12 WHO and Stop TB Partnership. The Global Plan to STOP TB, 2011 -2015. Available from: http://www.stoptb.org 13 Available from: http://tbsouthafrica.org/content/mission Evaluation of the USAID/South Africa Tuberculosis Program (FY2010-FY2014) Page | 11 Table 1. Key Project Strategies and Activities  knowledge and skills enhancement of health care workers;  improved systems for case management at facility level;  strengthened linkages between laboratories within the National Health Laboratory Service (NHLS);  improved DOTS strategies to reduce MDR-TB as well as improved management of MDR-TB patients;  strengthened infection control to reduce nosocomial MDR/ Extremely Drug Resistant (XDR) TB infections at healthcare facilities;  improved program management at district and provincial level;  improved capacity of HIV testing sites to provide TB screening to all HIV+ clients  improved capacity of primary health care clinics to provide on-site HIV testing to TB patients;  national and local ACSM strategies;  strategies to engage local communities in the management of TB patients; and  private sector involvement (mines, industries, and traditional healer practitioners) in TB service delivery. The expected long-term results (outcomes) of the project’s technical inputs are to:  achieve a case detection rate of 70 percent  treatment success rate of 85 percent;  improve capacity to plan and implement TB DOTS at community, facility, district, municipality, provincial and national levels;  improve surveillance system resulting in early detection of MDR-TB and other problems in the TB service delivery system;  improve understanding and support among the general population regarding TB signs, symptoms, referral, and treatment. Figure 1. Implementation Structure of the TB Program Communities National Provincial District Small Grants Program South African Departments of Health and Partners NTP Evaluation of the USAID/South Africa Tuberculosis Program (FY2010-FY2014) Page | 12 Figure 2. The Evaluation Team’s Understanding of the Project’s Theory of Change URC partnered with six organizations (Table 2) to achieve its objectives under the project. Table 2. Consortium Partnerships PRIME CONTRACTOR PARTNER ORGANIZATIONS Name of the Organization Headquarters Location Implementation Role UNIVERSITY RESEARCH CO., LLC (in the USA) BEA Enterprises, Inc. USA  Web portal development Health Systems Trust (HST) Durban, South Africa  Data management strengthening  Evaluative Operations Research International Union Against TB & Lung Disease (IUATLD) USA  Clinical training courses on drug￾resistant TB, and TB/HIV Johns Hopkins Health & Education South Africa (JHHESA) Pretoria, South Africa  Health communications for community based activities to prevent TB and TB/HIV Karensoft Consulting Group USA  System design  Training in the use of information systems, and  Monitoring and evaluation National Jewish Health (NJH) USA  Drug-resistant TB, infection control, and laboratory capacity building. Evaluation of the USAID/South Africa Tuberculosis Program (FY2010-FY2014) Page | 13 Over the life of the project, assistance was given to DOH structures in 9 provinces, 27 districts (Figure 3), and 91 sub-districts – mostly in KZN, EC, and FS. Project support began in most districts (N=19) in the first year of implementation, with the remaining 8 districts added in years 2 and 3. Of the 27 directly supported districts over the LOP, nearly all were still being supported by September 2014 – i.e. only 4 districts were no longer receiving project support, although the reasons for the cessation of support are not clear. Figure 3. DOH districts supported by URC over the Life of the Project (LOP) In terms of community-level work through small grantees, the project issued 100 small grants to 75 different organizations over the life of the project. The grantees were located in all 9 provinces, with most grantees being located in the high burden provinces of KZN and the EC. Figure 4 and Figure 5 illustrate the number of grants issued per province and each wave of funding over the LOP. In the first three waves of small grant funding, approximately 20-25 million South African Rands (ZAR) were disbursed per wave, but this doubled in Wave 4 (per Figure 6). A total of ZAR119,905,862.63 was disbursed through small grants over the LOP. EC - Amathole EC - Buffalo City Metro EC - Nelson Mandela Bay EC - Chris Hani FS - Fezile Dabi * FS - Lejweleputswa * FS - Mangaung FS - Xhariep FS - Thabo Mofutsanyane GP - Sedibeng GP - Tshwane Metro * KZN - eThekwini KZN - ILembe KZN - uMgungundlovu KZN - uMkhanyakude KZN - Zululand LP - Sekhukhune LP - Waterberg MP - Gert Sibande MP - Nkangala NC - Siyanda NC - Pixley ka Seme NW - Bojanala NW - Dr Kenneth Kaunda NW - Dr Ruth Segomotsi Mompati NW - Ngaka Modiri Molema WC - Cape Winelands Q Q1 Q2 Q3 Q4 Q5 Q6 Q7 Q8 Q9 Q10 Q11 Q12 Q13 Q14 Q15 Q16 Q17 Q18 Q19 Q20 Q21 Fiscal Year (FY) and Quarter Timeline of URC Support in DOH Districts FY 2010 FY 2011 FY 2012 FY 2013 FY 2014 Evaluation of the USAID/South Africa Tuberculosis Program (FY2010-FY2014) Page | 14 Figure 4. Grantees funded per province Figure 5. No. small grants issued by wave 51 44 31 21 17 16 15 14 5 0 10 20 30 40 50 60 Grantees per Province (Includes grants implemented across multiple provinces) 17 17 20 37 9 0 10 20 30 40 Wave 1 Wave 2 Wave 3 Wave 4 Wave 5 No. of Small Grants Issued by Wave (Total = 100 small grants) Evaluation of the USAID/South Africa Tuberculosis Program (FY2010-FY2014) Page | 15 Figure 6. Amount of Funding for small grants by wave The grantees were one mechanism used by the project to conduct ACSM activities. These activities are essential for IR3: increased demand for TB services. Examples of ACSM activities conducted by the project include:  Community dialogues,  School outreach programs through KICK TB campaigns,  Mass Media campaigns on national television, radio announcements including Public Service Announcements such as ”We Beat TB!” and “Cough Etiquette”,  Campaigns in Public taxi rank, taxis and buses,  Supporting National Events such as the World AIDS Day and World TB Day,  Distributing IEC (Information Education and Communication) materials such as the TB soccer ball (a soccer ball with educational messages regarding TB). R 23.08 R 25.57 R 20.28 R 48.88 R 0.94 0 10 20 30 40 50 Wave 1 Wave 2 Wave 3 Wave 4 Wave 5 millions of South AFrican Rands (ZAR) Total Funding for Small Grants by Wave Evaluation of the USAID/South Africa Tuberculosis Program (FY2010-FY2014) Page | 16 4 EVALUATION METHODS & LIMITATIONS To answer the 5 evaluation questions, the evaluation team utilized a non-experimental design that excluded a rigorously-defined counterfactual. However, using data contained in annual reports and the SAG’s information system, comparisons were undertaken to define the project’s contributions to TB services in South Africa, such as examining actual vs planned project outcomes over time and using the SAG’s ETR.net data to compare performance in URC￾supported districts vs non-URC supported districts. Four TB outcome indicators were the focus of these comparisons: 1. Treatment Success Rate 2. Lost to Follow Up 3. HIV testing rate amongst TB cases, and 4. TB screening rate amongst HIV positive clients In addition to document and data review (see list in Annex 3), the evaluation team conducted key informant interviews and online surveys. Four key informant interview guides and three online surveys were created (Annex 4 contains the tools). The target respondents and topics explored in the KIIs and online surveys are summarized in Table 3 below. Table 3. Summary of Respondents and Topics for Data Collection Methods Tools Target Respondents Topics to be explored Key Informant Interview Guides (N=4) USAID and other donors  USAID  Centers for Disease control and Prevention (CDC)  PEPFAR Provincial Liaison Officers (PPLs)  World Health Organization (WHO)  URC project management and leadership (general, staffing, consortium partners),  Project design  Implementation model and implementation process  Use of technology  Capacity building  Community work and small grants activity  Project results  Needs for future TB support. URC project staff  URC management and technical staff  Consortium partners DOH  National TB managers  Provincial TB Managers  Provincial TB M&E officers  District TB Managers  District TB M&E officers  Sub-district TB managers Grantees /Civil society Organizations (CSOs)  Grantee managers Same instrument as online survey (see below) but administered face-to-face Online Questionnaires (N=3) PEPFAR partners  Those trained by URC on TB/HIV integration  URC capacity building  Project results  Needs for future TB support. Evaluation of the USAID/South Africa Tuberculosis Program (FY2010-FY2014) Page | 17 Tools Target Respondents Topics to be explored Grantees / CSOs  All CSOs  Implementation process  URC capacity building  Quality of relationship with URC,  Use of technology  Needs for future TB support. Facility Managers  A sample of DOH Facility Managers  URC capacity building  Training results  Needs for future TB training Because the project provided two kinds of support to DOH at district/sub-district levels – direct or indirect – the team sampled only areas that were directly supported by URC. No indirectly￾supported districts or sub-districts were targeted for the evaluation. A total of 18 districts and 18 sub-districts were visited as part of fieldwork as shown in Figure 45 in Annex 2. In-depth KIIs were planned with more than 100 key informants, mostly with Government at district and sub-district levels (Table 4). Sampling for KIIs was purposive, where individuals were chosen because of their roles and involvement in the project. Government interviews at provincial, district and sub-district levels were carried out in all 9 provinces, in 1-3 districts per province, and 1 sub-district per selected district. At provincial, district and sub-district levels, many KIIs were conducted as group interviews with more than 1 individual – the TB coordinator, the TB M&E officer (where applicable), and others. In each province, two CSO grantees supported by the project were also targeted for KIIs (i.e. 2 grantees per province). The remaining CSOs were reached through the online survey. Table 4. KIIs – Targeted and Actual Targeted Respondents Actual Respondents Response Rate (%) No. KIIs South African Government 51 57 112% 45 - National Level 6 6 100% 5 - Provincial Level 9 14 156% 10 - District Level 18 21 117% 17 - Sub-district Level 18 16 89% 13 Donor 13 8 62% 8 - USAID 2 1 50% 1 - PPLs 9 4 44% 4 - Other Donors 2 3 150% 2 URC Staff (sample) 16 37 231% 23 URC Consortium Partners 5 1 20% 1 CSOs Receiving URC support (Grantees) 18 8 44% 8 Total No. KIIs 103 111 108% 85 Evaluation of the USAID/South Africa Tuberculosis Program (FY2010-FY2014) Page | 18 Three online surveys were developed to collect the following feedback: 1. Small grants recipients to obtain their views on achievements since the inception of the project as well as the quality of engagement with partners and the effects at facility and community levels. The survey was sent to all directors/programme managers at CSOs who have email addresses. 2. PEPFAR partners who received project training on HIV/TB integration. 3. DOH Facility Managers who received training from URC on TB management and services delivery during 2014. Table 5. Online Surveys – Targeted and Actual Targeted Respondents Actual Respondents Response Rate (%) No. of surveys Small Grants Recipients 115 38 33% 38 PEPFAR Partners 16014 4 3% 4 DOH Facility Managers 108 7 7% 7 Fieldwork took place from 29 September 2014 until 2 November 2014. Data Capturing and Analysis: KIIs were manually captured into electronic forms (Adobe Forms) which allowed the data to be exported into Excel for coding and analysis. Quantitative data were analyzed using Excel/Statistical Package for the Social Sciences (SPSS). The online survey data were exported from the online survey provider as an excel worksheet for analysis. The mix of qualitative and quantitative data generated through fieldwork were analyzed using methods appropriate to each. Data Limitations Missing Data: The evaluation team was unable to access certain documents per the list provided in Annex 3. However, the missing URC information that most critically affected data analysis are as follows:  A complete list of PEPFAR partners trained (which resulted in our difficulty in analyzing the training provided);  A breakdown of the project budget by key programme areas (e.g. grants, staffing, etc.); and  IR indicator data for Years 3 and 4 (as this was not presented in Annual Reports) Data in difficult structures/formats: In addition to the above missing data, the team had difficulties with some programme data provided by the project, as it was either consolidated (which prevented us from conducting meaningful analysis), or it wasn’t consistently reported from year to year. For example, each of the project’s five Annual Reports were different in 14 The evaluation team never received the full list of PEPFAR partners trained by the project, and therefore was unable to fully explore the views of this target group Evaluation of the USAID/South Africa Tuberculosis Program (FY2010-FY2014) Page | 19 content and format, such that the same indicator data could not be followed over time from one report to another. Difficulty in accessing “graduated” districts: Fieldwork was initially planned such that the 2 “graduated” districts would be visited to see how the project’s inputs had been sustained. However, the team found it nearly impossible to conduct fieldwork in these locations as relevant DOH personnel had departed and staff at the district or sub-district had little knowledge of the project’s support. One district in North West (NW) province was completely substituted. Evaluation of the USAID/South Africa Tuberculosis Program (FY2010-FY2014) Page | 20 5 FINDINGS 5.1 Evaluation Question 1: To what extent did the project achieve its intended results as stated in the contract’s objectives? What were the reasons for any shortfalls? To answer this evaluation question, the evaluation team conducted two procedures: (1) a comparison of the project’s actual vs planned outcomes over time as a measure of project performance, and (2) a comparison of trends in URC-supported districts vs non-URC supported districts using the National TB Programme’s ETR.net data against 4 outcome indicators15:  Treatment Success Rate  Lost to Follow Up  HIV testing rate amongst TB cases, and  TB screening rate amongst HIV positive clients 5.1.1 URC PROGRESS IN ACHIEVING ITS PROJECT TARGETS The TB Project has a robust PMP and reports on 21 routinely reported indicators: 6 overarching, outcome level indicators and 15 indicators across the project’s five IR areas. Of these 21 indicators, six (or 29%) were achieved or likely to be achieved by the End of Project (EOP), while 5 (or 24%) were not achieved or not likely to be achieved at EOP (Figure 7). Insufficient project data exists in Years 3 and 4 to evaluate progress for the remaining 10 indicators for the IRs (except for IR4). However, from the interviews, the perception is that the project is successfully meeting the IRs. Figure 7. Summary of Project Progress towards Indicators 15 These outcome indicators were specified in the evaluation Terms of Reference. Evaluation of the USAID/South Africa Tuberculosis Program (FY2010-FY2014) Page | 21 Half of the project’s outcome indicators were achieved at the end of Year 4 (Table 6), and the project came close to achieving the other three. Table 6. Achievement of Project Outcome Indicators Outcome Indicator Baseline 2009 Year 4 Target (FY2013) Year 4 Actual Achieved (FY 2013) Smear Conversion Rate 63 % 75 % 75.8 % Treatment Success Rate 74 % 80 % 80 %16 TB Cure Rate 62 % 75 % 71.8 % Defaulter Rate 8.3 % <5 % 6.8 % HIV Testing Amongst TB Cases 90 % 84.7 % TB Screening Amongst HIV Positive Clients 90 % 97.5 % It is worthwhile noting that while all 6 of the project’s outcome indicators may not be achieved by the EOP, data extracted from URC’s PMP and annual reports show that trends for the outcome indicators are progressively positive from Fiscal Year (FY) 2010 through FY2013 as depicted in Figure 8 through Figure 13 below. These data include the 4 districts where the project stopped providing direct support, as well as (new) districts added midway17 . Similarly, though not captured by PMP indicators, URC is widely applauded for their high level of technical expertise in TB, bottom-up approach, responsiveness, and support to the DOH in both skills transfer and capacity building as well as systems building. 16 This was achieved 3 months after the Year 4 end date – i.e. in December 2013 rather than September 2013. 17 Figure 3 in the Project Description of this document provides explanation on the districts added and subtracted from direct project support over the LOP. BEST PRACTICES:  Joint assessment & planning with Provincial DOHs and District DOHs (DRAT, risk assessments, Quality Improvement (QI) plans)  Infection control policies/stickers developed at facility levels  Joint supervisory and mentoring visits to facilities  Data verification at facility level with DOH  Research, data, study helps inform programme INNOVATIONS:  Screening Tools; TB diary; Registers  ACSM: Soccer balls, Lap desks for kids Evaluation of the USAID/South Africa Tuberculosis Program (FY2010-FY2014) Page | 22 Trends in Project Performance at the Outcome Level: Targets vs. Actual FY 2010-FY 2013 Figure 8. Smear Conversion Rate Figure 9. Treatment Success Rate 60% 70% 80% FY 2010 FY 2011 FY 2012 FY 2013 Target Actual Linear (Actual) 60% 70% 80% FY 2010 FY 2011 FY 2012 FY 2013 Target Actual Linear (Actual) Evaluation of the USAID/South Africa Tuberculosis Program (FY2010-FY2014) Page | 23 Figure 10. TB Cure Rate Figure 11. Defaulter Rate Figure 12. HIV Testing Amongst TB Cases Figure 13. TB Screening for HIV Positive Clients 60% 70% 80% FY 2010 FY 2011 FY 2012 FY 2013 Target Actual Linear (Actual) 0% 5% 10% FY 2010 FY 2011 FY 2012 FY 2013 Target Actual Linear (Actual) 50% 75% 100% FY 2010 FY 2011 FY 2012 FY 2013 Target Actual Linear (Actual) 40% 60% 80% 100% FY 2010 FY 2011 FY 2012 FY 2013 Target Actual Linear (Actual) Evaluation of the USAID/South Africa Tuberculosis Program (FY2010-FY2014) Page | 24 5.1.2 COMPARATIVE PERFORMANCE IN PROJECT-SUPPORTED DISTRICTS VS NON-SUPPORTED DISTRICTS The evaluation team examined data from the NTP (contained within ETR.net) covering the period 2010-2012 for the four key outcome indicators that are the focus of this evaluation. 2013 data was not ready and could not be provided. Treatment Success Rate (TSR) ETR.net data shows that from 2010-2012, South Africa has seen an overall improvement of 7% in TB Treatment Success Rates – from 72% in 2010 to 79% in 2012. Districts that were directly supported by the project generally started off with lower treatment success rates, as the DOH and URC purposefully targeted poor-performing districts most in need of capacity building support. However, the improvement in project supported areas was 1+ percentage point greater than seen nationally – 8.1% for districts and 8.7% for sub-districts (Figure 14 and Figure 16 respectively). One notable finding in this data is the “smoothing” of the trends in project￾supported areas; non-project areas or indirectly supported areas show a dip in 2011, but this is less evident in districts and sub-districts where the project was working. A test of means shows the difference in performance on treatment success by districts (directly supported, indirectly supported and non-supported) is statistically significant (p<0.05). Defaulter Rates South Africa has seen little improvement in defaulter rates in the 2010-2012 period, and indeed, the average defaulter rate for this period increased by 0.26% across all provinces. Although project-supported districts and sub-districts generally had higher defaulter rates than seen in other areas (as expected given that these areas were purposefully selected for their greater need), the change in defaulter rate was slightly better in project-supported areas than in non￾supported areas (Figure 15 and Figure 17). TB Patients Who Test For HIV Overall HIV testing in TB patients increased by 13% from 2010 to 2012 in South Africa – from 76% to 89%. Although directly supported districts and sub-districts started off with slightly lower rates of testing amongst TB patients, by 2013 these districts matched the other districts/sub￾districts in performance/ achievement for the indicator. However, there was no significant difference in the change over time between project-supported districts and non-supported districts (Figure 18 and Figure 20) – all areas improved in this overall. HIV Patients Who Test For TB Data for this indicator obtained from the NDOH DHIS seems to be problematic and may not be correct. Nonetheless, trend analysis over 2010-2012 indicates that overall there was an increase (5.7% change) in the rate of HIV patients screened for TB in URC-supported districts, and an even larger increase (change of 28.2%) in the rate of TB screenings amongst HIV patients in indirectly-supported districts. In contrast, non-supported districts recorded a drop in the rate of screening amongst HIV patients. Figure 19 and Figure 21show linear graphs of Trends in TB screening rates amongst HIV patients. It is important to note that even though the districts and sub districts show similar trends, the value of the change varies between the two when looking at the different category of support. Evaluation of the USAID/South Africa Tuberculosis Program (FY2010-FY2014) Page | 25 Figure 14. Ave Treatment Success Rates at District Level: 2010-2012 Figure 15. Average Defaulter Rate at District Level: 2010-2012 Figure 16. Ave Treatment Success Rates at Sub-district Level: 2010-2012 Figure 17. Average Defaulter Rate at Sub-district Level: 2010-2012 65% 80% 2010 2011 2012 Ave. Treatment Success Rate: District Trends by Type of Project Support URC direct support URC indirect support No URC support 8.1% change 6.9% change 7.1% change 4.00 6.00 8.00 2010 2011 2012 Ave. Defaulter Rate: District Trends by Type of Project Support URC direct support URC indirect support No URC support +0.04% change -0.03% change +0.31% change 65% 80% 2010 2011 2012 Ave. Treatment Success Rate: Sub District Trends by Type of Project Support URC direct support URC indirect support No URC support 8.7% change 10.4% change 6.4% change 4.00 6.00 8.00 2010 2011 2012 Ave. Default Rate: Sub District Trends by Type of Project Support URC direct support URC indirect support No URC support -0.13% change 0.16% change 0.40% change Evaluation of the USAID/South Africa Tuberculosis Program (FY2010-FY2014) Page | 26 Figure 18. Percent of TB patients tested for HIV by District: 2010-2012 Figure 19. Percent of HIV patients screened for TB by District: 2010-2012 Figure 20. Percent of TB patients tested for HIV by Sub-district: 2010-2012 Figure 21. Percent of HIV patients screened for TB by Sub-district: 2010- 2012 70 80 90 100 2010 2011 2012 Ave HIV Testing Rate in TB patients: District Trends by Type of Project Support URC direct support URC indirect support No URC support +12.3% change 14.0% change +13.8% change 0 500 1000 1500 2000 2010 2011 2012 Trends in TB Screening in HIV patients: District Trends by Type of Project Support Linear (URC direct support) Linear (URC indirect support) Linear (No URC support) 70 80 90 100 2010 2011 2012 Ave HIV Testing Rate in TB patients: Sub District Trends by Type of Project Support URC direct support URC indirect support No URC support +14.1% change +11.0% change +12.8% change 0 500 1000 1500 2000 2010 2011 2012 Trends in TB Screening in HIV patients: Sub District Trends by Type of Project Support Linear (URC direct support) Linear (URC indirect support) Linear (No URC support) Evaluation of the USAID/South Africa Tuberculosis Program (FY2010-FY2014) Page | 27 5.1.3 REASONS FOR PROJECT SHORTFALLS The NDOH, NTP, URC and other HIV/TB partners have jointly made impressive strides in the rapid scale up of key interventions and improved implementation of HIV and TB programs. The impact of these efforts is beginning to show: the country has declining numbers of new HIV and TB infections and is beginning to see lower rates of new infections in children. HIV and TB mortality is declining, with a corresponding decline in all natural cause mortality. However, a number of overarching constraints continue to hamper the achievement of key NTP (and URC) TB outcomes, notably the defaulter rate, TB cure rate and HIV testing amongst TB cases: Defaulter Rate:  There is no guidance for community caregivers on handling defaulters and no systems for reporting or managing side-effects. Seasonal workers in particular have higher defaulter rates, but there are few examples of service adaptations to support migrants.  The project did not develop/introduce adequate methodology or technology to facilitate contact tracing especially amongst migrants (both domestic and international). TB Cure Rate:  There was insufficient community level work in the project to increase early case detection, improve defaulter and contact tracing, and mobilize communities to assist in prevention and treatment.  This was exacerbated by lack of a systematic and institutionalized methodology for tracking defaulters (as above),  The emergence, and increasing caseload, of DR patients challenged the project’s ability to achieve its TB Cure Rate target. Poor treatment outcomes are further exacerbated by the diagnostic and therapeutic challenges of the increasing treatment failure for XDR-TB.  There was inadequate programming and coverage of HIV and TB services for vulnerable populations (including within correctional services, mining, farms and informal settlements). HIV testing amongst TB Cases:  Despite the significant scale up of integrated TB/HIV activities facilitated by the expansion of HIV testing and the decentralization of ART through Nurse Initiated Management of ART (NIMART), this training reportedly excludes staff nurses who serve as primary TB focal points. This is a barrier to prompt initiation of ART among TB patients including children.  Late presentation of TB patients (due to stigma or the need for more intensified early case detection) contributes to co-infected patients being unaware of their HIV status. TB and HIV data:  Substantial progress has been made towards reliable reporting of both TB and HIV data. The District Health Information System (DHIS) and ETR.net are well established and have national coverage. However, there is a lack of inter-operability between the two Evaluation of the USAID/South Africa Tuberculosis Program (FY2010-FY2014) Page | 28 systems, and a multiplicity of registers at health facilities confound inadequate data capturing by facility staff. More detailed analysis and summary of both internal and external constraints and challenges to the project’s design, approach, implementation and management is presented in section 5.2 below. Evaluation of the USAID/South Africa Tuberculosis Program (FY2010-FY2014) Page | 29 5.2 Evaluation Question 2: To what extent is the design of this project valid? How successful have been the programmatic and management approaches, structures and systems in carrying out the project’s activities? 5.2.1 PROJECT DESIGN Successes Overall, KII respondents from the DOH, donors, partners and URC felt the initial project design was strong. 85% of KII respondents felt the project’s broad scope was an advantage that allowed the necessary flexibility to respond to variations in local context and DOH requests, as well as to shifts in the epidemic. Respondents felt that in order to adequately and appropriately respond to the complexities and highly technical aspects of TB, a broad spectrum of interventions was required. The TB Project is closely aligned with the goals, objectives and strategic objectives of the NTP. The project aligned its activities with the strategic objectives of the SAG. Key strategic policies included:  Point 7 of the Health Sector 10 Point Plan, which emphasizes accelerated implementation of the National Strategic Plan for HIV and AIDS, sexually transmitted infections (STI) and TB (HAST) 2012-2016 reduction of mortality due to TB;  Decentralization of services to Primary Health Care (PHC) and nurse initiated diagnosis, treatment, and monitoring of HIV infected patients including management of TB/HIV co-infected patients. The change in criteria allowing ART initiation for all co￾infected patients;  Revised National DR-TB Control Policy: Amendment addressing decentralization of DR-TB care and community based management of DR-TB. The TB project tracks and utilizes the same outcome indicators as the NTP, which KII respondents from both URC and the DOH indicated helped the project to maintain the appropriate technical focus, prioritization of assistance, and relevance of its support activities to the NDOH. URC is widely regarded as the technical expert in tuberculosis, both in South Africa and globally: 100% of DOH respondents indicated that URC was the only partner whose core competency and primary focus was TB, and with the highly specialized technical skills needed to assist the NTP across all TB functional areas and technical interventions. The type and mix of capacity and systems building initiatives included in URC’s project design was also deemed to be good: the project included a mix of formal training, online training and mentoring and supervision. 83% of DOH respondents felt the project was designed to System Individual Don't Know Both Emphasis on Systems vs Individual Capacity Building (DOH, N = 41) Evaluation of the USAID/South Africa Tuberculosis Program (FY2010-FY2014) Page | 30 focus on both systems and individual capacity building equally. Of these respondents, 89% felt the emphasis and balance between these two capacity building approaches was appropriate. Of note, many KIIs – with URC staff, NGO grantees and the DOH, particularly at sub￾district levels – indicated that this was the first project that made a real effort to engage and involve communities (through small grants to NGOs) in efforts to prevent the spread of TB, identify new cases, and increase treatment compliance amongst TB patients. Constraints & Challenges KIIs revealed a number of key constraints and challenges in the project’s design. The project was initially designed to target 18 of the country’s total 52 districts (midway into implementation, the project’s target districts increased to 24). However, URC also received requests from both national and provincial levels to assist in rolling out new guidelines and systems across provinces and beyond their direct support districts. When URC was made a provincial level partner as a result of the PEPFAR realignment, this resulted in an increase in the number of requests for training and support province-wide, but limited their scope/ability to implement and provide the technical support intended at district and sub-district levels in their initial target districts. As a result, progress towards indicators and attribution of successes was dependent on the interest and engagement of the PEPFAR partners implementing in each of these. The project was challenged by the relative weight, attention and funding focused on HIV relative to TB. For example, URC is the only TB focused United States Government (USG) partner in South Africa: PEPFAR alone funds approximately 50 international and local partners (with many, many more organizations receiving HIV funding from USAID, GFATM, Department for International Development (DFID), the United Nations (UN) and others). Of URC’s 6 outcome indicators, one third (i.e. 2) are HIV related. In comparison, of PEPFAR’s 22 impact indicators, only 2 (or 9%) are TB related. While these two TB/HIV indicators are aligned across URC and PEPFAR partners, the relative emphasis amongst donors, implementing partners, and within the DOH, is heavily weighted towards HIV. This has the result of subordinating TB’s importance in overall HIV and TB programming as represented by the following quote. “TB is still seen as the ‘little sister’ or orphan child, largely viewed as playing a supportive role in the achievement of HIV indicators in South Africa.” – KII with DOH The project was initially designed to improve laboratory functioning through a joint gap assessment with the NHLS and the development of technical support plans to follow. However, the NHLS was not formally involved in the design of the TB Project, nor was it included as a formal partner. When approached by the NDOH and URC at the beginning of project implementation, the NHLS viewed the “gap assessment” as a threat and potential exposure of weaknesses, and was unwilling to participate. The NHLS is also a separate and autonomous Evaluation of the USAID/South Africa Tuberculosis Program (FY2010-FY2014) Page | 31 entity to the NDOH, and thus, neither party was able to successfully convince them to engage with the project. URC’s work in communities and through NGO grantees was constrained by the relatively high standards of eligibility criteria for small grants. The project was unable to work with many CBOs that had the potential to assist in improving TB outcomes, but did not meet the minimum organizational requirements for funding. In addition, the open competition process for small grants meant that funding did not always go to CBOs based in URC-supported districts, and this limited the possibilities for building greater linkages and partnerships between DOH and CBO efforts. 5.2.2 . PROGRAMMATIC APPROACH Successes Across all KII respondents (including donors, partners, URC staff, and national and district health staff), URC received unprecedented acclaim, praise, and recognition for its strong relationship building and close collaboration with the DOH at all levels. The evaluation team has never, to date, in the course of hundreds of evaluations spanning decades, come across such unparalleled and consistently high praise and appreciation for a project’s working relations, both with the NTP and other stakeholders. Nearly all KIIs, and notably 100% of DOH respondents, praised URC for their close and collaborative relations with the DOH (Figure 22), as well as with partners and donors. “URC is a part of the DOH” – SAG respondent “They’re one of us” – SAG respondent Some DOH respondents stated that while the mandate of other partners was not always clearly defined or shared with the DOH, and some partners implemented activities independently and without consulting or coordinating with them, URC was always transparent, clear, supportive and collaborative. 94% of all KIIs, and 100% with the DOH, praised URC’s bottom-up approach. At the start of the project, this approach reduced resistance and increased buy-in from the DOH through baseline assessments conducted jointly. From these assessments, the DOH could identify its own gaps and from these, adapt and plan the technical support most needed from URC. Quarterly and annual reviews using the district rapid assessment tool (DRAT) allowed the DOH and URC to continually assess changes and adapt the assistance and support from URC as needed. This led to the development of Quality Improvement (QI) plans, which were ultimately rolled out nationwide. Figure 22. Quality of URC Coordination with DOH 26% 74% How well does URC coordinate with the DOH? (N = 43) Well Very Well Evaluation of the USAID/South Africa Tuberculosis Program (FY2010-FY2014) Page | 32 The vast majority of KIIs also reported a good “fit” between the project’s objectives and the National TB Programme (Figure 23), with technical expertise and skills transfer cited as URC’s core competency and reasons for success. DOH respondents also commended URC for its focus on systems and capacity building and skills transfer, and not “doing the work of the DOH themselves”. Examples of key capacity and systems building initiatives included: a. Facility-level QI processes (mentoring and training/tools) for case management; b. Capacity building of managers at sub-district/district/provincial level for improved data collection, reporting, planning and management; c. Policies, guidelines, protocols and registers created or updated with the NDOH; d. Updates to pre-service nursing training and curricula for TB; e. Strengthened capacity and efficiency of laboratories for case identification; f. Increased engagement of private sector and communities in TB control. As shown in Figure 24, KII respondents perceived HIV/TB integration and TB M&E to be the strongest of nine focal intervention areas of the project, while the least improvements were achieved in drug supply management and laboratory functioning18 . 18 URC’s mandate did not include drug supply, though they provide support at facility level to avoid stock outs and improve supply chain management. Similarly, lack of engagement by the NDLS resulted in limited ability to improve laboratory functions. Figure 23. Fit of URC’s TB project with the NTP 8% 93% How well does the TB Project align with the National TB Programme? (N = 40) Well Very Well Evaluation of the USAID/South Africa Tuberculosis Program (FY2010-FY2014) Page | 33 Figure 24. Views on Project Capacity Building Strengths Constraints and Challenges Although largely praised for their bottom-up approach and close collaboration and supportive implementation, a number of constraints in project approach and implementation were also noted. First, the majority of DOH KIIs (60%) cited limited URC staff at sub-district levels as the primary challenge to successful implementation; particularly following the PEPFAR realignment and URC’s “shift” from district to provincial partner, which resulted in an increase in geographic coverage and reduced manpower and reach at sub-districts levels. A great majority of KIIs with DOH and URC staff felt that this shift constrained the project from implementing the requisite/planned level of mentoring and support - particularly following formal training or introduction of new tools, protocols and systems. Second, the project has no clear definition of the minimum criteria for “graduating” facilities or sub-districts once TB outcomes have improved and new skills are deemed to be institutionalized and/or likely to be sustained. Indeed, the project ended support to only 4 of the 27 districts where it has worked over the LOP, but the reasons or circumstances for this cessation of support cannot be firmly defined as “graduation”. Likewise, the project’s “champion facilities” were reported from many KIIs as “a good model”, yet the minimum criteria for designating a champion facility is not clearly defined. Third, there was some confusion in the “branding” of the project. Its title, “The USAID TB Project”, can be misleading and at times led partners and DOH staff to confuse or equate URC with USAID. Similarly, URC works with a consortium of partners, yet it was extremely difficult 0% 25% 50% 75% 100% Respondents (n = 48) Who Agree/Strongly Agree that the TB Project Significantly Improved Capacity by Area % strongly agree % agree N = 48 (USAID: 1; Other donors: 3; DOH: 44) ** variations in denominator by area, from 44 – 48 Evaluation of the USAID/South Africa Tuberculosis Program (FY2010-FY2014) Page | 34 for the evaluation team to get any clarity on the mandates, roles, and responsibilities of these partners, either from URC reports or the partners themselves. Alongside the internal constraints to the project’s approach listed above, several external changes and challenges also negatively impacted the project. The most significant of these – as mentioned by nearly 100% of KIIs conducted – was the PEPFAR realignment in 2012. Prior to this, URC worked as a district level partner with staff supporting sub-district, and at times facility, levels. With the realignment of PEPFAR partners – intended to streamline efforts and reduce the volume of partners working in each district to one – it was jointly agreed that URC would become a provincial-level partner, and utilize PEPFAR partners in each district to assist with implementation, monitoring, reporting and ultimately, achieving its objectives. For a number of reasons, this approach posed a major challenge to the project and greatly constrained its ability to achieve all that was initially included in their mandate. Key examples include:  The re-alignment severely challenged and complicated URC’s ability to implement and institutionalize new systems/skills at sub-district and facility levels since they no longer had a presence or staffing at this level;  With the same staffing structure, but increased geographic coverage, URC’s provincial coordinators were suddenly responding to increasing requests for training, technical input, assistance and other support from all districts within a province, and not the 2-3 initially designated in the project design;  Lack of project personnel at sub-district and facility levels hampered/restricted the project’s ability to provide ongoing mentorship and support supervision as originally planned; where limited or no follow-up support was provided, DOH and URC respondents indicated that improvements in service quality, availability, reporting and ultimately, TB outcomes, suffered;  The realignment of PEPFAR partners did not fully consider a realignment of TB efforts required as a result;  The assumption that PEPFAR partners would help with implementation and improvements in TB outcomes was erroneous; despite trainings and increased coordination, partners are still primarily HIV focused;  Varying levels of engagement from PEPFAR partners for training and follow￾up/implementation at district and facility levels resulted in variations in TB outcomes beyond the control of URC;  With a generalized TB epidemic, and the “removal” of priority districts, provincial district health departments wanted increased input and support, notably trainings, across all districts.  URC was also tasked with training PEPFAR partners, further increasing the emphasis of its efforts on formal training (often, for reduced numbers of days in an effort to achieve all trainings as requested) in lieu of mentoring and supervision. After the realignment, it was a huge challenge for URC to maintain its highly praised, bottom-up and skills transfer approach, as “they were in the wrong place to implement this successfully Evaluation of the USAID/South Africa Tuberculosis Program (FY2010-FY2014) Page | 35 and dependent on the willingness of partners to do so”19. Ultimately, the decision to make URC a provincial level partner led to an increase in formal trainings, but a more significant decrease in the likelihood of institutionalization of new skill sets and systems, sustainability of new approaches in the longer-term and ultimately, sustained improvements in TB outcomes. “As part of the PEPFAR realignment, URC was displaced as a district partner and thus lost its direct relationships with districts unless the province requested it. This was further complicated by tricky relationships with other PEPFAR partners working at district level.” KII, donor “The project doesn't fit well into the re-aligned PEPFAR partner comprehensive approach as it is a specialized TB project.” - KII, donor The Project was also constrained by human, financial and other resource limitations within the DOH. Staff turnover and vacancies of key DOH TB managerial positions hampered the project’s ability to achieve its objectives effectively and efficiently. Limited or no resources within the DOH for the logistical and administrative support required to rollout new tools and systems or skill sets was a constraint. In particular, while URC developed national as well as facility protocols for infection prevention and control, without the resources for minor refurbishments (such as improved ventilation, or cordoned off sections for MDR-TB patients), the project was only able to advise the DOH, but successful implementation of these protocols was restricted. 5.2.3 MANAGEMENT STRUCTURE AND SYSTEMS URC’s highly skilled and extremely dedicated, passionate staff were acknowledged by many respondents within the DOH, as well as partners and donors. Many DOH respondents noted that URC’s staff was primarily comprised of former DOH staff – retired nurses or former DOH managers. As a result, they were not only technical experts in TB, they also had in-depth knowledge and understanding of DOH systems and functioning. In sum, URC’s technical expertise combined with their local knowledge and experience within the DOH country-wide was a key factor in their success. URC staff at all levels was deemed to be flexible/responsive to the local context, needs and requests from DOH as well as the changing landscape of the TB epidemic in general. Despite the challenges faced by the limited staffing at provincial and district levels, the technical expertise and decentralized nature of support provided from URC’s central office in Pretoria was a significant factor in overcoming both shortages in local staffing structure and the high turnover of URC staff (similar to that of the DOH). Still, while this was not identified through KIIs, the evaluation team felt the staffing at central level compared to that at provincial, district and sub-district levels to be out of balance and seemingly “top-heavy”. Of note, some grantees received little local or provincial URC support given that the 19 Quote from KII with donor Evaluation of the USAID/South Africa Tuberculosis Program (FY2010-FY2014) Page | 36 support was being managed remotely and infrequently by the staff in Pretoria. Section 4.4.3 expands on this finding. The project was also challenged by a high turnover of staff; over 90% of KIIs listed staff turnover as a key constraint to the project. Over the LOP, the project has had 5 Chiefs of Party (2 of which were acting while the project was recruiting a permanent replacement) and at least 4 M&E officers. Currently, 2 of 9 Provincial Coordinator positions are vacant (WC, NC), there is a lack of mentors at the sub-district level, and the project has many staff with less than one year’s experience with URC. Factors explaining staff turnover include USAID restrictions in annual salary increases (limited to 5%) which adversely affects the project’s ability to retain staff in a tight labor market, and a demanding/challenging work load. High staff turnover negatively affects the project’s ability to provide sufficient continuity/coverage of support to provinces/districts. Project financial management is the responsibility of URC’s Washington DC headquarters. Reportedly, the DC office only provides the project with funding on a weekly basis. For a project of this size, scope, and scale, with multiple partners as well as NGO grantees to manage, this reportedly resulted in funding delays and constraints, particularly to the NGO grantees, presenting another considerable challenge to the project. Evaluation of the USAID/South Africa Tuberculosis Program (FY2010-FY2014) Page | 37 5.3 Evaluation Question 3A: Did the project strengthen the capacity of NTP sufficiently to ensure its sustainability? The project carried out capacity building at both individual and systems levels, and nearly all respondents (90%) felt that the balance between these was appropriate. 5.3.1 INDIVIDUAL CAPACITY BUILDING ACTIVITIES The project envisioned three types of individual capacity building:  formal training workshops,  mentoring and support, and  online training. Both the mentoring and online training were designed to support the formal training to ensure that knowledge gained in formal training was applied correctly on the job. Training and mentoring were envisioned as a means to accomplishing IR1: Improved Quality of Services and IR2: Increased Availability of TB Services. Formal Training and Mentoring From FY 2011, the project jointly hosted the National Training Task Team with the NTP to ensure standardization of TB training as well as to plan and monitor training activities across the country20. Many project trainings were co-facilitated by the DOH. Working closely with the DOH in this manner was an important step in institutionalizing TB knowledge and skills in the DOH and improving the sustainability of efforts. The project conducted 300 trainings over the LOP (Figure 25) – mainly in EC, Gauteng, and KZN, and mainly during year 1 – reaching more than 20,000 people (Figure 26), including health practitioners (doctors, nurses), DOH managers at provincial/district/sub-district/facility levels, PEPFAR partners, nurse educators, master trainers for Clinical MDR-TB, and NGO/CBO staff (e.g. Home Based Carers, and Grantee Managers). The USAID TB Program facilitated formal training workshops with IUATLD, NJH, CDC, and the Pretoria University, and were focused on basic TB-HIV management, clinical management of DR-TB, GeneXpert, DOT support, ETR.net, HIV/AIDs counselling and testing (HCT), MDR-TB management, monitoring and evaluation, data capturing, data validation, and management and finance of small grants. Respondents praised the formal training and mentoring and expressed great appreciation for the training and support. Many DOH respondents noted that they knew little about TB until the project started working with them and that the training/mentoring has greatly expanded their TB knowledge and capacity. “They helped me a lot, I thought I knew TB before the training but I didn't” – SAG Respondent 20 USAID TB Program – Annual Report (1 October 2011 to 30 September 2012) Evaluation of the USAID/South Africa Tuberculosis Program (FY2010-FY2014) Page | 38 Figure 25. No. Training Activities by FY and Funding Source Figure 26. No. Trainees Reached by FY and Funding Source However, respondents noted the inconsistent provision of training certificates and lack of SAQA accreditation as project shortcomings. Evaluation of the USAID/South Africa Tuberculosis Program (FY2010-FY2014) Page | 39 Online Training Online training was not a large component of the individual capacity building model. It was designed to supplement rather than replace formal face-to-face training, and was targeted at grantees to provide supplemental guidance on Basic TB, Project, and Financial management. It was developed and piloted by USAID TB Program before being sent to the grantees. The roll￾out of the online training and the uniformity of the training sent to grantees appears to be irregular as a number of grantees, with internet access, do not report receiving the training and others report receiving different types of training as seen in Figure 24. Figure 27. The type of online trainings that grantees report receiving Figure 28. The perceived effectiveness of the online training received 0 5 10 15 20 No online training received Basic TB training Financial Management training Reporting requirements training TB-HIV co-infection training Monitoring and Evaluation ACSM (advocacy, communication, social… Don’t know Were you provided with any online training from URC around the implementation of your grant? 0 1 2 3 4 5 6 7 8 9 Basic TB training TB-HIV co-infection training Reporting requirements training Monitoring and Evaluation ACSM Financial Management training How effective was the online training that you received? Very effective or Effective Ineffective Evaluation of the USAID/South Africa Tuberculosis Program (FY2010-FY2014) Page | 40 The majority of the grantees who received the online training report it to be effective or very effective (seen in Figure 28). URC has not recorded the roll-out or use of the online survey in their training data and it appears that the online training was not monitored to assess its feasibility and track its use. 5.3.2 SYSTEMS CAPACITY BUILDING ACTIVITIES The project undertook a wide range of other activities to build capacity of systems for TB services delivery and management, such as:  conducting joint data analysis exercises with DOH staff at district level using the District Rapid Appraisal Tool (DRAT)  quarterly TB/HIV collaborative meetings with health providers and managers from different facilities who were brought together to share ideas and experiences of how to better provide TB and TB/HIV services;  creating and disseminating numerous tools for TB diagnosis and M&E: the TB diary, TB/HIV Screening Tool, and IPT register;  assisting in the development of national TB Guidelines and MDR treatment protocols;  conducting joint data verification exercises with DOH, supporting use of ETR.net at facilities, sub-districts and districts; and  supplying needed laboratory equipment (e.g. GeneXpert machines). The system’s capacity building efforts contribute to IR4: Improved management of TB support systems. 5.3.3 QUALITY AND EFFECTS OF CAPACITY BUILDING ACTIVITIES Measuring the quality of trainings is limited without pre- and post-test results. However, the evaluation measured perceptions of the training quality which are reported on below. Furthermore, measures of facility level activities such as TB screening of HIV positive patients can indicate if the training is being implemented. As mentioned previously, there has been significant improvements in TB screening of HIV positive patients and HIV screening of TB patients in supported districts suggesting that the TB/HIV training has yielded results21 . From 2011 onward, Co-trimoxazole Preventative Therapy (CPT) uptake showed significant improvement. Improving ART uptake was initially challenging but gradually improved (though more still needs to be done). Improving Izoniazid Preventive Therapy (IPT) uptake was a significant challenge during the LOP, and a number of activities were undertaken in FY2013 to improve this22 . These results suggest that though the training is having an effect on screening and some drug uptake, there is more to be done. Most DOH respondents felt that the project significantly increased capacity in the NTP particularly in (Figure 29): 21 USAID TB Project South Africa, Annual Report (1 October to 30 September 2011) 22 USAID TB Project South Africa, Annual Report (1 October 2012 to 30 September 2013) Evaluation of the USAID/South Africa Tuberculosis Program (FY2010-FY2014) Page | 41  TB related M&E, HIV/TB integration,  TB Treatment service quality,  Infection Control,  TB Case Finding,  ACSM,  Planning and Management, and  TB Treatment Adherence. Figure 29. DOH respondent’s views on areas where the project significantly improved capacity in the NTP However, the project’s impact on improving drug supply management and laboratory functioning was perceived to be lower than in the other areas. In 2010, URC appointed a Laboratory Advisor and began meeting with the NHLS and permission was granted for the project to engage with laboratories. After completing a desktop review, plans were put in place to 0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100% Laboratory functioning Drug Supply Management Reaching Vulnerable Populations TB Treatment service availability TB Treatment adherence Planning and management ACSM TB Case finding Infection Control TB Treatment service quality HIV/TB integration TB-related Monitoring and Evaluation The USAID TB program significantly improved capacity in the following areas (N=43): Agree Disagree Evaluation of the USAID/South Africa Tuberculosis Program (FY2010-FY2014) Page | 42 strengthen laboratory services. However, an MOU with NHLS was only signed a year later. Support provided by the project included:  TB diagnostic tools, GeneXpert leaflet and training material, training materials on TB diagnostics,  GeneXpert training,  DRAT in two NW districts,  Monitoring of diagnostic criteria,  Purchase of 11 GeneXpert Machines to improve sputum turnaround time23 . Though work was done to improve laboratory functioning, this may have been limited by the MOU only being signed in 2011. Furthermore, a SAG respondent noted that; “Part of their (the project’s) scope of work was to assist in improving laboratory services but the NHLS was resistant to this. They felt threatened or exposed if they accepted the "assessments" to identify gaps. As they are autonomous to the DOH, they had little ability to convince them to cooperate so URC could not take this on. We still need this assistance.” According to the Joint Review of HIV, TB and Prevention of Mother to Child Transmission of HIV (PMTCT) Programmes in South Africa24, though Laboratory services are generally good, there is still work to be done on routine use of GeneXpert algorithms, and turnaround time is still problematic in some facilities. This suggests that there is a need for the project to continue working on improving laboratory functioning in the future. URC respondents described the work done in Drug Supply Management to be “behind-the￾scenes” monitoring of drug supplies in facilities and actions were only taken if and when drug￾supply issues arose. This was not a major focus of the program, however, the Joint Review of HIV, TB and PMTCT Programmes in South Africa report suggests that constant drug supply management is a significant problem24 requiring attention. The majority of the grantees ‘agree’ that the USAID TB program significantly improved their organizations’ capacity in relevant areas as seen in Figure 30, which mirrors the DOH views on capacity building provided by URC. 23 USAID TB Project South Africa, Annual Reports FY2010 to FY 2013 24 Joint Review of HIV, TB and PMTCT Programmes in South Africa, April 2014, Main Report Evaluation of the USAID/South Africa Tuberculosis Program (FY2010-FY2014) Page | 43 Figure 30. Grantee respondent’s views on areas where the project significantly improved their organization’s capacity 5.3.4 SUSTAINABILITY Most respondents (from URC, SAG, consortium partners, and USAID) believe the project’s implementation approach is sufficient to strengthen the NTP (Figure 31). Indeed, the project has taken a number of steps towards institutionalization and sustainability of TB efforts as evidenced by:  Incorporating URC training in the National Training Plan  Upgrading pre-service training on TB at nursing colleges  Training of DOH Trainers  Ensuring the project’s screening and M&E tools are adopted by the DOH However, there is still much work to be done before the NTP can be sustained without further contribution from URC, particularly in on-going training and support to individuals (DOH staff and communities) for delivery and management of TB services. As described by an SAG respondent: “URC are the experts on TB and we are not yet the experts”. 0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100% Laboratory functioning Drug Supply Management TB Treatment service availability TB Treatment service quality Infection Control Reaching Vulnerable Populations Planning and management HIV/TB integration ACSM TB Treatment adherence TB Case finding TB-related Monitoring and Evaluation The URC TB grant significantly improved my organization’s capacity in the following areas: Agree disagree Evaluation of the USAID/South Africa Tuberculosis Program (FY2010-FY2014) Page | 44 Indeed, all (100%) government respondents stated that URC support should continue and increase. However, the degree of concern as to the impact of a discontinued USAID TB program differed: “It would be a disaster, TB program will die, people will die. Everything will collapse, we need their support” – SAG Respondent “Because we worked together, we took ownership of initiatives, so we can sustain it. But please don’t take them away, there is much to do especially with MDR-TB” – SAG Respondent. Figure 31. Project’s implementation approach vis-à-vis capacity building The degree of dependency on URC for support with TB implementation was linked to the human resources of the DOH. For example, in a district without a TB coordinator, the responsibility for managing the TB program in that district is largely left to a data capturer who relied heavily on the URC provincial coordinator (PC) for support. In Limpopo (LP), the DOH has been ‘under administration’25, and as a result the URC PC was heavily relied upon to provide resources to speed-up implementation. [Mpumalanga has recently gone under administration and is likely to require more assistance in future.]. Yet, in other areas, DOH respondents were more comfortable with continuing to operate without URC but wished for URC support as they ventured into new areas such as MDR-TB decentralization of care and nurse initiated MDR treatment. Constraints to sustaining the project’s capacity building efforts largely centers around the high turnover/rotation of SAG staff at district and facility levels which results in the need for constant training and re-training. Effective capacity building of individuals also requires follow-on 25 Executive power is taken from the department and given to the cabinet. The cabinet is responsible for approving and monitoring expenditure which slows the approval processes and thus implementation down. 69% 7% 24% Was the project's implementation approach at provincial and district sufficient to strengthen TB Program capacity? (N=68) yes no partially Evaluation of the USAID/South Africa Tuberculosis Program (FY2010-FY2014) Page | 45 mentoring, but this has been restricted by the shortage of URC staff on the ground. For example, in the Northern Cape (NC) there is no URC PC and, as a result, there is very limited post-training mentoring provided at facilities. Though it was not part of the project mandate, a lack of DOH staff at some facilities, districts and sub-districts increased the workload of the project’s PCs who end up filling these gaps and doing the job of the DOH staff. This has led to an increased dependency on the project. Another constraint to sustainability involves the movement of patients across geographic areas, and especially from areas where capacity has been improved to areas where capacity is poor. This often creates clinical challenges whereby patients default or are lost to follow-up, thereby potentially compromising national TB control efforts. Some project activities were reliant on DOH adopting the efforts of URC but this did not take place. For example, the mobile tracking system piloted by URC in KZN was expected to be adopted by DOH but it was not (see further discussion on this in section 5.6 of this report). In Mpumalanga, TB data capturers were trained and employed on URC contracts with the expectation that they would be hired by the DOH, but they were not. 5.3.5 CONCLUSIONS AND RECOMMENDATIONS The program has implemented capacity building efforts at all levels of the health care system and the efforts have mostly been well received with consistent requests for more capacity building in the future. The program has made efforts towards sustainability but more still needs to be done before their efforts are institutionalized. Recommendations:  Utilize alternative mentoring approaches such as peer mentoring and communities of practice, to reduce URC workload and wean trainees/SAG staff off of URC  Hold more Train-the-trainer sessions  Increase pre-service TB training  Include sustainability indicators in QI plans and assessments  Get accreditation for the training Evaluation of the USAID/South Africa Tuberculosis Program (FY2010-FY2014) Page | 46 5.4 Evaluation Question 3B: What role has the Small Grants program played in improving TB, TB/HIV and MDR-TB programs? 5.4.1 SMALL GRANTEE PROGRAM DESIGN According to the project’s FY 2010 Annual Report, the small grants program plays a critical role in meeting IR 2: Increased Availability of TB Services and IR 3: Increased Demand of TB Services. The role of the small grants program is to: “…implement community-based strategies that will identify TB suspects and ensure early referrals for testing and treatment. Building community-based support also ensures treatment adherence by patients and reduction of stigma and discrimination and also creates demand for improved services for TB-HIV co-infected people.”26 The grants are initially granted for a 12 month period, however, a number of organizations have received funding for additional months or have been funded for consecutive waves. The waves correspond with the fiscal year (FY) and there have been 5 waves from 2009 to 2014. Though the waves run by FY, the actual start and end dates of the grants vary substantially from one grant to another. For example, in Wave 1 some grants started in April 2010 while others began in July 2010. In the second wave, some grants started in September 2010 while others started in January 2011. The solicitation method of seeking grantees involved the circulation of Requests for Applications (RFAs) in four waves. The RFAs originally required proposals for strengthening DOTS through various means (from TB case detection to ACSM). In FY2011, a series of Proposal Writing Workshops were conducted as a means of identifying potential organizations in all 9 provinces. More than 120 organizations participated in these workshops which had the particular aim of “…orienting participants in the URC proposal writing template, providing an overview of the USAID TB Program objectives, and identifying key areas that the USAID TB Program can benefit through support at community level and health care centres”27 . 100 small grants were issued to 75 organizations over the LOP. Though the primary aim of the small grants programme was to strengthen community level implementation, grants were issued to a wide range of organizations varying from localized organizations operating at grass roots level to national NGOs who primarily conducted research or other activities not directly related to community services delivery. Indeed, a number of grantees appear to be chosen to fill in other necessary gaps in the NTP such as research, capacity building of nurses, etc. For example, Democratic Nursing Organization of South Africa (DENOSA) was financed to conduct TB training of nurse educators at nursing colleges. Figure 32 presents the types of activities conducted by grantees over the LOP. Most organizations used their grants for carrying out Intensified Case Finding, DOTs, and ACSM. Many included other complementary activities; for example Phaphamani Home Based Care conducted DOTS support, Intensified Case Finding (ICF), HIV/TB integration and ACSM 26 USAID TB Project South Africa, Annual Report FY 2010 (1 October 2009 to 30 September 2010). 27 USAID TB Project South Africa, Annual Report FY 2011 (1 October 2010 to 30 September 2011). Evaluation of the USAID/South Africa Tuberculosis Program (FY2010-FY2014) Page | 47 campaigns. Other grantees implemented only one specialized activity with their grant funding; for example Impangele only conducted ACSM activities involving dramas acted out at schools and the creation and dissemination of educational comic books. Figure 32. Types of activities conducted by small grantees According to grantees (Figure 33), the most significant change produced by the grants were:  improved capacity of the grantees’ organization and staff,  increased availability of TB services,  increased ACSM and thus community knowledge of TB,  increased TB specific expertise,  the development and strengthening of relationships with facilities and the DOH, and  employment of local community members. These responses suggest that the grants program contributed to IRs 2 and 3 as planned. Though the small grants program was not aimed at improving IR 1 (increased quality of TB services), a number of the small grantees report significant change in TB Service quality as well. 0 5 10 15 20 25 30 35 40 45 50 TB referal system IPT screening Data management Research Nutritional support HCT Unknown MDR/XDR Defaulter tracing Working with vulnerable populations Training / Capacity Building HIV-TB DOT ACSM Intensified Case Finding Types of activities conducted by Small Grantees (N= 80) Evaluation of the USAID/South Africa Tuberculosis Program (FY2010-FY2014) Page | 48 Figure 33. Grantee perceptions on the most significant change the Small Grants Program has made 5.4.2 STRENGTHS AND CHALLENGES OF SMALL GRANTEES Extension of NTP to the Community Level SAG and URC respondents consistently recognized the value of community-based grantees in extending the NTP to community level and in closing the gap between patients and health facilities. Strengths of small grantees in supporting the NTP include the fact that they:  Can readily access rural areas and vulnerable populations such as children, farmers, miners, migrant workers etc.,  Are more trusted by communities than government is,  Trace defaulters and reduce defaulting through improved adherence (DOTS),  Conduct door-to-door campaigns and ICF for increased awareness,  Provide home based care for community members who would otherwise travel long distances to clinics, are bedridden and do not have the money for transport,  Integrate HIV/TB screening, treatment and education, 0 2 4 6 8 10 12 14 improved indicators promoted CSO brand relationship with community children focus No significant change research MDR expertise increased resources financial reporting system employment quality of TB services ACSM Relationship with facilities TB knowledge availability of TB servceis improved capacity Grantees' views on most significant change due to Grants (N= 38) Evaluation of the USAID/South Africa Tuberculosis Program (FY2010-FY2014) Page | 49  Prevent DR-TB development by improving adherence,  Provide employment to the community members in the communities they serve. Most SAG respondents believe the small grants program was insufficiently scaled to meet needs at community level, reporting the programme’s limited geographic coverage as a weakness. They unanimously recommended a greater focus on community level work in future investments. One respondent describes TB as a community problem requiring a community response. As TB and MDR-TB decentralized care is rolled out, the small grantees become even more fundamental in this process: "If we can get more support at community level, and conduct outreach with a full-fledged, well-resourced team, this is where we can make a real difference" – SAG respondent Ideally, grantees would have been purposively selected according to their location so as to pair them with priority districts/sub-districts that were the focus of the project’s DOH support. NGOs/CBOs operating in these areas could have been purposefully selected and linked to DOH services to provide the community-level support needed. A successful example of this is seen in Mafikeng, a high burden area, where a grantee worked closely with the URC provincial coordinator and the sub-district DOH, forming an essential partnership to deliver comprehensive TB prevention and control activities in the area. Unfortunately, many grantees were funded to implement in areas other than the high priority geographic areas. For example; the grantees assisting with MDR-TB in one province were not located in the MDR-TB hotspots of the province. This is partly explained by the project’s open solicitation process, which did not restrict proposals for implementation in priority areas, as well as the eligibility criteria for small grants which favored NGOs with strong(er) organizational capacity for managing the funding. As a result, the award process resulted in the rejection of many potential (albeit organizationally weaker) grantees based in project-supported districts who could have been funded with the provision of additional capacity building at start-up. Numerous SAG respondents reported that no grantees operated in their area or that they were not introduced to the grantees working in their area, with some expressing frustration at the lack of coordination and communication between the grantees and themselves, and requesting more communication and joint planning in the future. Even where grantees are working well with DOH, and where the grantee submits data on its services to the facility’s/sub-district’s monthly report, the DOH’s information system has no ability to “tag” the data coming from NGOs or community level, thereby making it impossible to quantify the contributions being made by NGOs at community level. This however, is being examined by the NDOH, as it requires more data on community level services delivery for its annual TB report to WHO. Technical Scope of Grantee Programmes Though many grantees carried out DOT, ICF and ACSM activities as prioritized by the small grants programme, some did not, which limited their usefulness to the DOH. In this regard, because the project did not define a minimum package of services to be implemented at community level, funding was provided for what the organizations proposed as long as it had Evaluation of the USAID/South Africa Tuberculosis Program (FY2010-FY2014) Page | 50 relevance to the overall project. This meant that the type and scope of grantee activities was unevenly distributed throughout the country, regardless of community needs or DOH requirements. Some grantees did not specifically focus on accessing vulnerable populations, although they were ideally suited to do so. Other grantees found that they were limited in their ability to manage and implement more technical activities (e.g. MDR support) without the addition of higher level health care workers (e.g. professional nurses) on their project teams. Funding Cycle and Cash Flow Grantees and SAG respondents alike reported that the 1-year grant funding cycle was problematic in that it was too short to demonstrate a marked or sustainable impact. Many reported that it took the year for program start-up and for capacitating their staff. Although several grantees received consecutive grants, these often did not dovetail the previous grant, leading to gaps in funding that were highly problematic for service delivery. Grantees that did not receive follow-on grants reported that their programs were unable to continue at the same capacity without the follow-on funding, resulting in a marked negative reduction in service delivery and unemployment for their staff which they had spent time and funds training. Less than 10% of grantee respondents (4 of 47) indicated that there would be (or was) no impact when URC support ends (ended). However, the remaining 90% noted a negative impact without the grants, particularly as alternatives for TB funding are limited to non-existent – 80% of respondents indicated no “replacement” for the URC support (Figure 34). Given that the South African Department of Social Development provides grants to NGOs for community-level HIV and health work, this might be an alternative source of funding for NGOs looking to carry on with their TB work. In addition to the constraints experienced by the one year implementation period, numerous grantees reported delays in receiving reimbursement payments within the grant period, and this negatively affected their ability to offer continuous services. For example, delays in reimbursement for one NGO caused such cash flow problems that the Director was forced to reach into her own pocket to pay for staff salaries and volunteer stipends on more than one occasion. Figure 34. Grantee opinions on alternative TB support 80% 11% 3% 6% Are there other options for accessing TB-related funding and technical support that could be better than the URC TB support? no yes partially Don't know Evaluation of the USAID/South Africa Tuberculosis Program (FY2010-FY2014) Page | 51 5.4.3 FACTORS AFFECTING THE PERFORMANCE OF THE SMALL GRANTS PROGRAM: Relationship with URC staff The vast majority (83%) of the small grantee respondents report that the support received from URC staff was effective. Figure 35 shows that while most respondents rate URC’s communication as good, some grantees received more hands-on support than others, who mainly experienced long-distance and infrequent support. Just as the grantee activities varied, the grantee management appears to have varied. URC staff acknowledged that they did not have a sufficient staffing complement to supervise the grantees as frequently as required. Figure 35. Grantee perceptions of URC communication with their organization. NGO Capacity Building: Through its proposal writing workshops, URC made strides towards its objective of capacitating NGOs in the proposal writing process. In addition, once the grants were awarded, the project conducted initial verification visits to the grantees and identified capacity building requirements. Training was then provided to the organizations in basic TB, project and financial management. One grantee respondent states that: “The URC TB grant added value to the fight against TB and has enabled my organization to grow in an area we were not involved before. The support from the technical team and the provincial team was very helpful in building our capacity to achieve set targets and conduct activities effectively.” Despite annual reports noting the training provided to grantees, the capacity building of grantees does not seem to have been uniform. Grantee respondents gave mixed reports around receiving capacity building, with some noting that they did not receive, or received only limited, formal training, mentoring or visits. Some were not aware of the online training course aimed at supplementing the formal training. When asked to rate the quality of the formal training received, the small grantee respondents rated the training highly as seen in Figure 36 below: 14% 10% 76% There was good communication between my organisation and URC Strongly disagree and disagree Neutral Strongly Agree or Agree Evaluation of the USAID/South Africa Tuberculosis Program (FY2010-FY2014) Page | 52 Figure 36. NGOs’ rating of the quality of formal training received Support in Financial Management The effectiveness of the Grants Management Team in supporting the small grantees was considered to be effective in 95% of the responses as seen in Figure 37. Figure 37. Effectiveness of the Grants Management Team in supporting the CSOs 0 5 10 15 20 25 30 35 40 Basic TB training TB-HIV co-infection training Financial Management training Reporting requirements training ACSM (advocacy, communication, social mobilisation) Monitoring and Evaluation Quality of the formal CSO training received from URC Very effective Effective Neither effective or ineffective Ineffective Very Ineffective 95% 5% How effective was the Grants Management Team in supporting your financial management and contractual compliance? (N=39) Very effective and effective Neutral Evaluation of the USAID/South Africa Tuberculosis Program (FY2010-FY2014) Page | 53 When asked if the Grants Management Team could have made any improvements in their support a small grantee responded “None because her support was excellent.” Indeed most grantee respondents viewed the financial team to be responsive (Figure 38). Figure 38. Grantee perceptions of the URC's responsiveness to financial and contractual concerns However, as discussed previously, numerous grantees mentioned funding difficulties, including:  Delays in funding disbursements (delays up to 6 months have been reported)  Gaps between monthly disbursements: “It was hard to manage an ambitious grant when dependent on monthly disbursements. This made it difficult to plan and manage moving forward. It would have been better to have quarterly advances.” – Small Grantee  Poor communication regarding follow-on grants and proposals  Poor communication about disbursement or funding delays These funding difficulties had a marked impact on the grantees’ organizations and implementation including:  inconsistent delivery of key services (especially DOTS and defaulter tracing),  demotivation and unrest of staff who were awaiting wages  high staff turnover as staff members sought predictable employment  damaged relationships between grantees and health facilities/communities they had been serving  debt for the organizations and their staff members who took out loans to maintain activities while awaiting funds 7% 12% 81% The URC TB project team responded to all our financial management/contractual needs in a timely manner. Strongly disagree and disagree Neutral Strongly Agree or Agree Evaluation of the USAID/South Africa Tuberculosis Program (FY2010-FY2014) Page | 54 “Delays in funding this year has affected our work. They owe us for June. This affects our work plans and quarterly reports. We have to submit expense claims but you need money to spend money.” – Small Grantee URC staff noted that the one factor in this regard is the financing policies of the URC headquarters in Washington (i.e. weekly fund disbursements to the project) which at times created bottlenecks that affected the grantees: “There are limitations stemming all the way from the URC office in the US that result in delays and gaps between contracts, which is a constraint/challenge” – URC respondent. Technical Support The majority of grantee respondents viewed the Technical Team to be responsive to their programmatic needs (Figure 39) while also considering the Technical Team to be effective in their support of the grantees programs (75%) (Figure 40). When asked to elaborate on the technical support received from URC (Figure 41), 68% of the CSO respondents described the URC technical and grant-management staff as responsive and supportive, 18% described the support as being telephonic without hands-on interaction while 14% described them as unsupportive and unresponsive citing a complete lack of support being received or URC staff/turnover resulting in unpredictable support. Figure 39. Grantee perceptions on the responsiveness of URC’s project team. 10% 19% 71% The URC TB project team responded to all our programmatic needs in a timely manner. Strongly disagree and disagree Neutral Strongly Agree or Agree Evaluation of the USAID/South Africa Tuberculosis Program (FY2010-FY2014) Page | 55 Figure 40. Effectiveness of the Technical Team support to CSOs Figure 41. Description of URC technical support Overall areas of Improvement Figure 42 presents a list of shortcomings for the Small Grant program, which serves as a guide to areas where URC can improve: 75% 15% 10% How effective was this Technical Team in supporting your technical work? (N=42) Very effective and effective Neutral Ineffective 68% 14% 18% CSO description of URC technical support (N=22) Responsive and supportive Unresponsive/ unsuppotive Telephonic support only Evaluation of the USAID/South Africa Tuberculosis Program (FY2010-FY2014) Page | 56 Figure 42. Shortcomings of URC support Implementation difficulties Though some grantees report no difficulty in accessing vulnerable populations, others mention a number of challenges when trying to access these populations. Difficulties include:  obtaining parental consent to screen children,  tracing patients particularly in rural areas with migrant laborers ,  covering large geographical areas with limited resources such as staff and transport,  concerns regarding the safety of their HBC staff,  varying “red tape” at prisons, facilities and workplaces,  a lack of patient willingness, compliance and availability; “Patients need to learn to care for their own health. They are focusing on earning money and not their health.” – Small Grantee Manager  the stigma associated with TB/HIV preventing screening and treatment. Other reported factors that affect grantees abilities to implement the TB activities include:  Quality of relationships/linkages between CBOs and health facilities varied,  Insufficient transport to cover large rural areas,  Staff retrenchment when funding ended, 0 1 2 3 4 5 change in expected work monthly disbursements unclear requirements activities not aligned with CSO limited training poor URC planning Technical staff expertise/assistance gaps in funding limited mentoring/support short funding period delay in funding limited communication Frequency of Mentions Shortcomings of URC support to Grants (N=34) Evaluation of the USAID/South Africa Tuberculosis Program (FY2010-FY2014) Page | 57  Some CSOs lack clinical staff (e.g. nurses) and are thus restricted in their programme efforts. URC has worked with grantees to address some of these issues. In a district where patients providing incorrect addresses to facilities, thereby making it difficult for DOT supporters to find the patients, URC arranged that DOT supporters should accompany the patients to their homes when discharged thus ensuring that the DOT supporters had the correct addresses. Furthermore, with regards to patient compliance and community stigma, ACSM campaigns assist with reducing this problem. Some grantees were provided with funding to hire professional nurses so that they could expand their services, allowing them to treat MDR-TB infected patients. 5.4.4 CONCLUSIONS AND RECOMMENDATIONS The project’s small grantee program is an essential aspect of the USAID TB Program as it provides essential community based TB initiatives. Recommendations for the future grantee program are:  Build the capacity of all Home Based Care community level organizations receiving the grant so that they all perform the same functions (so long as those activities are relevant to the area they are in and the community they serve).  Purposively choose grantees that link to facilities, provide services required in the specific areas, and ensure even coverage of services throughout the country.  Increase the number of grantees to cover more of the country.  Ensure that there is hands-on support for grantees either by increasing the number of provincial coordinators, hiring nurse mentors or by choosing and capacitating one grantee to oversee the others in each province.  Collaborate with government grantee system  Plan ahead for smooth transition from one grant to the next  Implement sustainable practices at grantees  Quarterly fund disbursement Evaluation of the USAID/South Africa Tuberculosis Program (FY2010-FY2014) Page | 58 5.5 Evaluation Question 4: What strategies were used to reach vulnerable populations? E.g. Mines and work place programs 5.5.1 DEFINITION AND STRATEGY One objective of the NTP in line with WHOs objective 2 is to protect poor and vulnerable populations from TB, TB/HIV and MDR-TB. Objective 2 of the NTP STOP TB strategy aims to specifically address prisoners, refugees, and other high risk groups and situations. Similarly, URC has an objective to protect the poor and vulnerable populations from TB, TB/HIV and MDR-TB although it does not explicitly define particular strategies for the objective. Despite the absence of an explicit definition of these populations and strategies to reach them, the evaluation team found that URC could be said to have had three strategies that responded to this objective. 1. ACSM activities in communities where URC would indirectly create awareness amongst workers and family members of mobile vulnerable populations such as miners and farm workers. 2. Public Private Mix (PPM) activities which falls under the project’s IR5 i.e. “Tested new approaches for expanding DOTS coverage”. PPM activities involved the private sector (mines, industries, medical schemes and traditional health practitioners (THP) in TB service delivery. 3. Small grants programme, under IR3 “Increased demand of TB services). Under this project vulnerable populations that were initially focused on included prisoners, mine laborers and farm laborers; however, in the middle of the project, children were added as an additional vulnerable population given growing TB caseloads amongst children and increasing numbers, particularly in children under five years of age, with MDR-TB. Because defaulter rates were also high in children, the project identified the need for more education of parents and caretakers. 5.5.2 SUCCESSES AND CHALLENGES ACSM Strategy: The project’s indirect strategy for reaching vulnerable populations was through non-targeted ASCM activities for indirectly creating awareness among the target groups. For example, the Free State has a large mining industry and therefore ACSM activities conducted at community level would invariably reach the mining population. Likewise, Mpumalanga Province has a large farming industry, and ACSM activities in this Province would indirectly reach vulnerable farming populations. Little was recorded in URC’s annual reports around the ACSM strategy with regards to indirectly reaching vulnerable populations. According to the FY 2010 annual report, North-West Province was recorded as having many platinum mines with a highly mobile population. The evaluation team expected to read about stepped-up ACSM activities in NW in subsequent years but there is no mention of ACSM activities indirectly targeting these populations in the Province. KIIs revealed that general challenges with ACSM activities consisted of people refusing to participate in TB screening at big gatherings and only agreeing to screen when it was done Evaluation of the USAID/South Africa Tuberculosis Program (FY2010-FY2014) Page | 59 door-to-door. The evaluation team suspects that these populations are largely vulnerable populations, but there is no supporting evidence for this assertion. Public Private Mix (PPM) The Public/Private mix strategy directly engaged Mines, Prisons and Transport organizations for the addressing challenges of access to TB care, treatment and support in the private sector. The PPM strategy falls under IR2 of the project, i.e. “Increased availability of TB services” and IR5 through the indicator “Number and Types of PPM Models developed using NGOs and other private sector entities.” URC measures this indicator through case finding, treatment, follow-up; lab service support, supervision of patients and health education undertaken in all private sector activities. In this section we will only mention PPM activities as they pertain to addressing vulnerable populations. Engaging Traditional Health Practitioners (THPs): URC’s Annual Report FY2012 notes the initiation of Engaging THPs as part of its PPM strategy. In collaboration with the KwaZulu Natal Premier’s Office and Provincial AIDS Council, the project hosted a Traditional Health Practitioner Summit, attended by 100 THPs. The project’s presentation highlighted the role of THP in integrating TB in the HIV/AIDS services currently offered by THPs. Following the Summit, the project collaborated with THP to:  Raise awareness among THPs to protect both THPs and clients from TB transmission  Build THP’s capacity in providing community TB education and care  Develop relevant IEC materials to improve referrals and messages on TB collaborative bridges between traditional and modern health systems To date, THPs have been provided with the project’s TB screening tool for use in case finding. In addition THPs refer TB suspects to health facilities, follow up patients and encourage them to take their drugs thus protecting patients from graduating to MDR-TB. During the KII with the PPM Advisor, the evaluation team was informed that there were efforts to link THPs with mine workers. This is because being sick is considered to be a weakness amongst mine workers who tend to present late at health facilities. The evaluation team assessed PPM’s effort of encouraging THPs to screen their mine-worker clients and referring them to facilities for treatment as an attempt to obtain treatment quickly thus protecting their families and co-workers (vulnerable populations) from contracting TB. Engaging the Private Sector: When the PPM coordinator was employed in 2011, he started by developing models for private sector participation in TB service delivery. These models included working with small and medium sized mines, working with correctional services through the use of the small grants programme, and working with SABCOHA, an umbrella organization targeting private sector in terms of workplace programmes. URC’s Annual Reports record that strategic plenary and advocacy meetings were conducted with the International Labour Organization (ILO), Mine Medical Practitioners Association, and SABCOHA in an effort to increase access to TB services within private sector and mining industry. In KwaZulu Natal collaboration expanded to include the Premier’s office, where the Provincial Council on AIDS is coordinated. Evaluation of the USAID/South Africa Tuberculosis Program (FY2010-FY2014) Page | 60 Implementation of this model reached vulnerable populations through training and mentoring staff in the sugar industry in KZN, which led to TB screening, sputum collection, adherence support to TB patients, and contact tracing in employees’ households. The project’s FY2013 Annual Report states that TB in the mining sector is a massive challenge and a top priority for DOH given that there are half a million mine workers with TB in the country. SABCOHA provided a platform through which the project encouraged mines in 4 provinces – namely LP, NW, KZN, and EC– to design implementation plans to support the objectives and interventions contained in the National and Provincial Strategic Plans28. Activities included providing technical advice in the development of the Mining Led Stop Project Business Case, which was presented to chief executive officers from some mining companies. The project’s PPM unit also developed a mine rapid assessment tool to determine the TB/HIV services in mines. The tool provides baseline information on TB/HIV services provided and was piloted and used at Ezulwini Mine in Gauteng. It is reported that the project’s approach has emphasized developing programs, and TB service linkages within small and medium size mines in LP and GP. Jointly with the DOH, the project also facilitated local community based TB awareness and treatment services with mining communities. Technical support provided to small and medium size mines included:  Technical assistance on integrating TB into HIV workplace policies  Conducting assessment of TB management in selected mines in LP province  Strengthen TB programme management in the mining sector  Capacity building of clinical staff  Provision of IEC materials on TB There are numerous examples of successful PPM activities in the mining industry. The latest URC quarterly report indicates that the project with support from district health officials, Eskom Electricity Company, and local organizations working in TB and HIV, led a one-week active TB case finding campaign in the community of Lephalale near Eskom’s Medupi power station, LP Province. During the campaign, 3324 employees were screened for TB, 477 were presumptive of TB, 422 sputum were collected from those who were presumptive of TB, 4 employees were found to be positive for TB, 51 samples were not tested because the sputum was insufficient, 19 had unsuccessful results, and three were put on TB treatment. The TB Program PPM Advisor was approached by Harmony Gold Mine to propose a strategy for supporting the TB Program within that mine. The project developed a strategy document, which was presented to the health team at Harmony Gold. One of the challenges in working with the Private Sector was reluctance especially from Managers of Mines to open doors for employee empowerment in terms of TB since their main priority is empowerment against HIV/AIDs. Engaging the Public Transport Sector: During implementation with the Private Sector, URC realized that the Public Transport sector was also vulnerable and thus approached the 28 In the FS, the project was awaiting its introduction by the province to the mines. However, due to the lack of small grantees working in the province, PPM activities with mines were never implemented in FS. Evaluation of the USAID/South Africa Tuberculosis Program (FY2010-FY2014) Page | 61 Department of Transport to discuss ways of reaching actors in the Transport industry. The PPM Advisor has indicated that is currently devising a “Public-Public” model whereby public sector organizations such as Department of Health and Department of Transport worked jointly on developing solutions for common problems, such as TB in the Transport sector. As such, URC has begun work with the Taxi Association using NDOH ACSM activities to reach commuters at taxi ranks. The PPM Advisor also indicated that another successful example can be seen in the project’s work in correctional service facilities, where it encouraged TB patients be separated from other prisoners, and to improve TB case finding and treatment. This is now being implemented in a number of prisons through small grantees. Further support to the Department of Correctional Services (DCS) has been provision of TB screening tool, IPT registers, and IEC material to facilities. Small Grants Programme Strategy: The third strategy for reaching vulnerable populations was the use of small grants for funding community-level services. According to the FY2011 Annual Report, small grants to local NGOs is one of primary ways in which the project reaches vulnerable populations. Small grantees also target workplace programs e.g. mining, general industries and farming. One or two grantees worked in prisons, a grantee working in a peri-urban area received a grant from the project to work with miners who were disengaged by their employers and had returned home. This grantee supported DOTs using the miners’ relatives. A few grantees worked with juvenile prisons to conduct active case finding, HIV TB screening treatment, while others worked at old age homes and conducted screening on a quarterly basis. Activities undertaken by grantees that worked with TB in children was to mainly DOT children on IPT. Several vulnerable populations including children, farmers, prisoners, miners and DR-TB patients were targeted by grantees as seen in Figure 43. Out of the 47 CSO respondents, 10 organizations did not specifically target vulnerable populations. Unfortunately, the project information on grantees scope of work was in numerous formats and the information provided does not specifically state the number of grantees tasked with accessing vulnerable populations. However, from the scope of works provided 10 grants descriptions describe working with these populations. The grantee online survey revealed numerous constraints to accessing vulnerable populations, chief of which was tracing, limited resources, patient compliance and safety of staff (Figure 44). Evaluation of the USAID/South Africa Tuberculosis Program (FY2010-FY2014) Page | 62 Figure 43. Vulnerable populations accessed by CSOs Figure 44. CSO constraints to accessing vulnerable populations 0 5 10 15 20 25 30 35 Not Applicable Farmers Miners Prisoners Do not work directly with vulnerable populations Work with communities that contain vulnerable populations Other DR-TB patients Number of mentions Vulnerable Populations Accessed by CSOs as reported by grantee respondents 0 1 2 3 4 5 6 7 8 9 facility resources stigma patient knowledge lack of URC support parental consent size of area served prison/facility relationship/protocol safety of staff patient compliance and availability limited resources tracing no challenges accessing vulnerbal populations Number of Mentions CSO contraints to accessing vulnerable populations Evaluation of the USAID/South Africa Tuberculosis Program (FY2010-FY2014) Page | 63 KII interviews also pointed out pointed out the following perceived successes and challenges of using the small grants programme to reach vulnerable populations. Successes:  Increased community awareness of TB in prisons, mines, informal settlements  Site visits and training in correctional facilities, use of DRAT  Increased TB screening and services in correctional services.  In mines where permanent employees are on Medical Aid it’s relatively easier to coordinate TB activities  Community radio slots had TB discussions where people phoned in with questions. Challenges:  Stigma around TB  Follow up of mobile populations  Some people did not want to go for TB screening at big gatherings and would only agree to screening if it was done door-to-door. 5.5.3 ADDITIONAL FINDINGS There is no mechanism for disaggregating services to vulnerable populations in TB data. Given other donor-funded projects (GF and WB) that focus on mines and prisons, there may be duplication of efforts with the work of the project, particularly as the project’s work with vulnerable populations appears to be of limited scale. There were mixed results with PPPs and mines/farms – where management was supportive or where there were onsite health facilities available, the project was able to gain traction. But where management was not supportive or there were no health facilities, less success was seen. Mines need further assistance to strengthen data sharing (TB forms and data) between mine health service and the DOH to track defaulters. More investment is needed to effectively reach vulnerable populations with – Active case finding, – Contact tracing, – Following-up defaulters Evaluation of the USAID/South Africa Tuberculosis Program (FY2010-FY2014) Page | 64 5.6 Evaluation Question 5: How has the project integrated technology? e.g. mobile health work in KZN, Active TB case finding 5.6.1 TYPES OF TECHNOLOGY USED BY THE PROJECT The evaluation team defines “technology” as mHealth or electronic Health (eHealth) solutions as URC did not have an explicit definition of technology for the project. Technology was not a specific project objective but was included as an activity under IR 5: Tested New Approaches for Expanding DOTS Coverage. The project’s FY2011 Annual report states that URC presented a mHealth application in May to the National Quarterly meeting after the DOH’s determination that community-based management of DR-TB needed support with appropriate tools, particularly with information systems, which had poor quality and outdated data that did not provide real time information for meaningful management decisions. The project in collaboration with GeoMed, a private sector service provider, developed an innovative web-based mHealth solution combining Google™ Earth and smart phone technology for remote data gathering, patient information management, and workforce management. It effectively brought healthcare directly to the community by enabling mobile health workers to link mapped households with home based care services. The annual report indicated that Geo-mapping of DR-TB cases were undertaken in Eastern Cape, Gauteng, and Free State, where the highest numbers of DR-TB were found in Lejweleputswa in the Free State with mines and migrant workers. Key Informant interviews in Free State reported that the main limitation in the project was identifying respondents’ home addresses, as the patients mainly provided workplace addresses and not necessarily the communities where they lived. As a result it was found that vulnerable populations were more widely spread than in the immediate area around the mines. The Geo-mapping information was therefore not put to use by the FS DOH Key Informants also noted that the project piloted another mHealth solution contact tracing of DR-TB patients in KZN. Unfortunately, this pilot was never scaled and eventually ended, although the University of Pretoria reportedly requested to take the KZN technology and use it in their grant, but because they never received permission from the KZN DOH, it was never implemented. 5.6.2 TECHNOLOGICAL CONSTRAINTS The project faced a number of constraints in the area of introducing new technology or adapting new technology introduced by other role players in TB:  Integration of technology solutions was not a focus of the project. There was no technologically-trained staff on the Pretoria team to advocate for the DOH to take these solutions forward. Evaluation of the USAID/South Africa Tuberculosis Program (FY2010-FY2014) Page | 65 6 CONCLUSIONS The USAID TB project, and URC in particular, is highly regarded, both in South Africa and globally, for its expertise in TB. The DOH and other partners recognize its primary position as the main partner focused on TB as its core competency with specialized technical skills needed to assist the NTP. It is extremely well respected, appreciated, and liked by the DOH as well as other stakeholders. The project has seen positive progress and trends toward achieving its six outcome indicators, and will likely achieve four by end of project. In the districts and sub-districts it supported (52% of all districts in the country), the project appears to have stabilized the implementation of the NTP, which reportedly tended to have fluctuating performance without project support. The project also increased outcomes to the same degree or slightly more compared to districts/sub￾districts where it did not provide direct support. The project’s achievement of most of its objectives, as well as its positive performance at district level (relative to non-supported districts) is in part due to its programmatic approach that focused on:  using TB data for analyzing and subsequent planning (e.g. DRAT and quality improvement plans),  Extensive training of HCWs and community care workers, especially in Basic & MDR￾TB; HIV/TB co-infection; infection control; intensified case finding; defaulter and contact tracing;  Strengthening key systems and tools (e.g. TB/HIV and IPT registers, TB diaries, Quality Improvement Plans, Data Verification Exercise, etc.), and  Support for national policy and guidelines. The project’s breadth was seen as an advantage in that it gave flexibility in responding to specific needs at local level. However, limited project staffing at district and sub-district levels did not always allow for deep enough support to ensure “graduation”, as only a fraction of districts stopped receiving URC support over the LOP. Project success was compromised by the fact that many DOH staff at facility and management levels who were trained and mentored by the project, often left during the LOP (i.e. were rotated). This then required heavy investment in basic capacity building throughout the LOP. Another constraint was movement of patients in and out of the districts, making it difficult to ensure continuous case detection, increased treatment success, and reduced defaulter rates. The project was adversely affected by the PEPFAR realignment mid-way through the LOP, which moved it from a district-based partner to a provincial partner, posing challenges in carrying out its district-based model, and in achieving indicators though other PEPFAR organizations over which it had no authority. Moreover, the greatest need for TB support is at sub-district, facility and community levels, yet as a provincial partner, its ability to provide follow￾up support (in the form of mentoring and support supervision) after formal training suffered as a result. Linkages between URC and other PEPFAR partners were largely based on the other PEPFAR partners’ willingness and interest in better HIV/TB integration, rather than strengthening TB specifically. Related to this is the lack of alignment between the NTP’s key Evaluation of the USAID/South Africa Tuberculosis Program (FY2010-FY2014) Page | 66 indicators for TB and the PEPFAR indicators under TB-HIV integration; PEPFAR partners are not required to report against TB outcome indicators (e.g. default and TB death rates) which, if they were to track, might spur them to invest more in TB. The project’s work with NGOs has proved helpful, and initial activities have shown some successes. However, a weakness was inadequate linkages between the small grants programme and the support being provided to DOH at district and sub-district level. The project’s open solicitation process for grants did not ensure that CBOs located in the 27 directly￾supported districts were prioritized for funding to extend TB services from facilities to community-level. In addition, organizational capacity requirements at the solicitation stage appear to have eliminated many (weaker) CSOs who were based in the priority districts. In this regard, the project would have been required to invest more energy in organizational capacity development, but presumably this would have paid off in terms of program performance, particularly in improving patient compliance, treatment success, and reducing default rates. Although 75 organizations were funded with 100 small grants, the grants’ 1-year timeframe precluded the community activities from reaching any significant scale. In this regard, nearly all respondents in the evaluation emphasized the need to expand and strengthen community￾based TB services (both prevention and treatment), highlighting the importance of CBO efforts in the NTP. The main strategies for increasing access to TB services for vulnerable populations were through general public ACSM activities (by grantees), through engaging traditional health practitioners, and through technical support to mining companies and prisons for workplace programmes. These strategies, while appropriate, were also limited in scale, and as such it is unclear what the impact of these have been on stemming TB incidence. As a means to improving DOTS, the project attempted to introduce mHealth solutions through a pilot in KZN. Unfortunately, this did not succeed in being implemented beyond the pilot stage, in part because the relevant technical expertise in the project team was based in the US, and the South African project staff lacked capacity to create momentum for a more favorable outcome. Numerous respondents noted the need for the project to better document its best practices and successes, suggesting the need for an improved knowledge management and communications function in the project. Indeed, the evaluation team experienced some difficulties in accessing data and information from the project (such as M&E data around IRs), further reinforcing the need for better data and knowledge management. Better documentation of successes and lessons learned is needed if future projects and the DOH are to take successful elements to scale. While some of project’s work has been institutionalized and is likely to be sustained (systems, tools, etc), the DOH needs more technical assistance and support in the next five years if SA is to adequately address the growing TB epidemic and reach the global targets set for post-2015. Evaluation of the USAID/South Africa Tuberculosis Program (FY2010-FY2014) Page | 67 7 RECOMMENDATIONS Current Project Recommendations 11. Build on the successes and lessons learned in the current project, to scale-up and increase institutionalization and sustainability of approaches and activities. Follow on Project Recommendations The follow on project should: 12. Re-establish the balance between formal training and mentoring, onsite capacity building and coaching. Project design should consider alternative mentoring approaches such as peer mentoring, communities of practice, coaching etc., or increasing provincial level staffing to include roving teams who can focus on and quickly support poor performing districts/sub-districts. 13. Provide accreditation and certification for formal TB training and improve management of trained health personnel database to facilitate and tracked skilled TB HCW and managers within NTP. 14. Develop and define clear criteria for “graduating” facilities, sub-districts and districts from project support. 15. Develop and maintain effective and efficient internal M&E and knowledge management systems. Ensure management, accumulation, and dissemination of project best practices. 16. Increase PPM efforts to assist with early case finding, contact tracing and treatment adherence. Ensure a holistic approach that targets all vulnerable groups in the community. 17. Assist NTP to expand community level work in TB prevention and treatment by: a. building upon the best practices and lessons learned from 20 years of HIV community-based programmes; b. ensuring continuity of NGO and community grants so as not to compromise services delivery required to achieve NTP goals by; i. facilitating longer term grants (more than 1 year); ii. increasing the capacity building of NGOs for sustainability such as training on accessing funding from non-URC sources; iii. purposively selecting grantees who are located in priority districts and building their capacity (where necessary) to link to facilities and expand community work. iv. Using an intermediate (larger) NGO to coordinate smaller community based organizations where possible. 18. Include the NHLS as a formal partner to improve early and accurate diagnosis and appropriate treatment regimes. Evaluation of the USAID/South Africa Tuberculosis Program (FY2010-FY2014) Page | 68 19. Pursue technological innovations and new approaches to improve defaulter and contact tracing in collaboration with a team of mHealth experts to ensure successful implementation is taken to scale. “The strengths of NGOs and other CSOs active in health care and other development interventions at the community level include their reach and spread and their ability to engage marginalized or remote groups. These organizations have a comparative advantage because of their understanding of the local context. Greater collaboration between NGOs and other CSOs and local and national governments could greatly enhance development outcomes. A more decentralized approach that formally recognizes the critical role of NGOs and other CSOs as partners addressing gaps through support to community-based actions will expand TB prevention, diagnosis, treatment, and care activities.”– www.who.int/tb/people_and_communities Recommendations for USAID/PEPFAR 20. Increase alignment/inclusion of key TB outcome level indicators across all HIV/TB programs and partners. Specifically, include the following two TB indicators: (1) TB related mortality amongst HIV co-infected and (2) TB defaulters amongst co-infected. Evaluation of the USAID/South Africa Tuberculosis Program (FY2010-FY2014) Page | 69 ANNEX 1: EVALUATION STATEMENT OF WORK SECTION C – DESCRIPTION/SPECIFICATIONS/STATEMENT OF WORK PROJECT TO BE EVALUATED: Project name: USAID TB Program Cooperative Agreement No.: 674-C-00-09-00121-00 Project Dates: September 28, 2009 – September 30, 2014 Agreement Value Range: $64,616,586 Implementing Organizations: University Research Co., LLC (URC) I. Purpose and Use of the evaluation The overall goal is to assess the quality of project design and implementation, determine which approaches and activities are working and why. Findings and recommendations from this evaluation will be used to inform the future direction of USAID/SA TB related investments in the country. The purpose of this external participatory evaluation is to assess the effectiveness of the URC contract in strengthening of TB prevention and control efforts in South Africa. The evaluation should focus on both the technical aspects of the project, including inputs and activities to improve TB outcomes, the actual outcomes achieved, as well as approaches employed to carry out activities and management systems utilized to implement contract obligations, including managing funding sources and communication strategies. This information will help determine the impact of USAID/SA supported interventions and guide the development of a new TB procurement cycle. II. PROGRAM BACKGROUND The USAID TB Program South Africa supports the National TB Control Program (NTP) strategies for improving early case detection, increasing access to diagnostics, ensuring treatment support for patient on TB treatment and ensuring that there is provision of appropriate and timely HIV care for TB patients and ART treatment for all TB/HIV co-infected patients. Overall, the USAID TB Program has continued to develop its multi-level support working closely with: • The NTP to build national support by mobilizing resources and creating a conducive environment for expansion of TB services. This includes: TB/HIV and developing strategic interventions that could rapidly address MDR/XDR-TB threats; • Provincial and district health departments to support the collaborative development of need based strategies to combat TB, TB/HIV and DR-TB, as well as communities to create appropriate social mobilization and service delivery models for rapid directly observed treatment short-course (DOTS) expansion in the country. The Project’s technical and financial inputs are expected to contribute to the following results:  Treatment success rate of 80%;  Improved capacity to plan and implement TB DOTS at community, facility, district, municipality, provincial and national levels;  Improved surveillance system resulting in early detection of TB cases, MDR-TB cases, coinfected patients for ART as well as to prevent treatment defaulters and reduce mortality; Evaluation of the USAID/South Africa Tuberculosis Program (FY2010-FY2014) Page | 70  Improved understanding and support among the general population regarding TB and TB/HIV signs, symptoms, referral, and treatment. Since October 2012, the Project has been implementing work plan activities that coincide with strategic policies for the South African Government (SAG). These key strategic policies include the following: • Health Sector 10 Point Plan, in which Point 7 specifically emphasizes accelerated implementation of the National Strategic Plan for TB, HIV and AIDS and sexually transmitted infections (STI) 2012-2016 reduction of mortality due to TB; • Decentralization of services to Primary Health Care (PHC) and nurse initiated diagnosis, treatment, and monitoring of HIV infected patients including management of TB/HIV co-infected patients. The criteria for initiation of ART for co-infected patients which changed to allow ART initiation for all co-infected patients; • Revised National DR-TB Control policy: amendment addressing decentralization of DR- TB care and community based management of DR-TB. The project’s activities fall under the following Intermediate Results (IRs): IR 1: Increased Quality of TB Services IR 2: Increased Availability of TB Services IR 3: Increased Demand for TB Services IR 4: Improved Management of TB Support Systems IR 5: Tested New Approaches for Expanding DOTS Coverage Partner Organizations Technical Area/s Regional coverage 1. Johns Hopkins Health and Education South Africa Advocacy, Communication and Social Mobilization National 2. International Union Against TB and Lung Disease Training (Principles of TB Management, MDR-TB, TB/HIV, Finance & Management) National III. Project Supported Areas Project supported areas information is available as part of background information. IV. Key Evaluation Questions The key evaluation questions to be addressed are: 1. To what extent did the project achieve its intended results as stated in the contract’s objectives? What were the reasons for any shortfalls? 2. To what extent is the design of this project valid? How successful have been the programmatic and management approaches, structures and systems in carrying out the project’s activities? 3. Did the project strengthen the capacity of NTP sufficiently to ensure its sustainability? What role has the Small Grants program played in improving TB, TB/HIV and MDR-TB programs? 4. What strategies were used to reach vulnerable populations? E.g. Mines and work place programs. 5. How has the project integrated technology e.g. mobile health work in KZN, Active TB case finding? V. Evaluation Design and Data Collection Methodology Evaluation of the USAID/South Africa Tuberculosis Program (FY2010-FY2014) Page | 71 To answer questions on the extent to which expected results were achieved and , the quality of implementation, USAID expects the offerors to apply a non-experimental design approach with extensive use of quantitative and qualitative methods. These will include reviews of project documents and extensive use of routinely collected program data. The evaluators will also be expected to collect primary qualitative data. The evaluation team should use a variety of methods, both quantitative and qualitative, for collecting information and data. The following essential elements should be included in the methodology at a minimum. The evaluation team is expected to develop a more detailed methodology and discuss with USAID/SA prior to evaluation initiation. It is recommended that the methodology assess outcomes achievement, as well as a comparative basis (e.g., Districts with URC and Districts) to assess the results. Below are the key tasks the evaluation team will conduct during the data collection Phase: a) Pre-evaluation Briefing: Preliminary discussions (prior to arrival in country) with the USAID/SA management team to review SOW, agree on key evaluation questions, evaluation design and data collection methods, finalize schedule and logistics. As an output, it is expected that a detailed work plan will be developed, including milestones and deliverables with due dates, responsible parties clearly established. b) Desk Review: The evaluation team is expected to conduct an in-depth review of background documents and relevant materials prior to arrival in the country. The following documents will be provided by USAID/SA for the desk review purposes: i. Contract agreement, amendments ii. Project quarterly, semi-annual and annual reports iii. Work plans and any management reviews iv. M&E plan v. USAID/SA South Africa CDCS vi. Selected project research and technical reports, publications, and tools c) Key Informant Interviews: The evaluation team will conduct qualitative, in-depth and structured interviews with key stakeholders, partners and beneficiaries such as URC leadership and staff; USAID/SA management; national, provincial and district department of health representatives; U.S.G and other international partners; local implementing partners and service providers, as well as project beneficiaries. It would be preferable for the interviews to be conducted face-to-face. However, if that is not possible, some of the interviews can be conducted over the phone or through other means such as email. d) Field Visits: The evaluation team will conduct field visits at each of the provinces and districts where the project has been operational. The specific facility and implementation sites will be selected and finalized during the Debriefing process and prior to the country visit. Prior to arriving in country and conducting field work, the team will review project documents and reports, to be made available by USAID/SA Upon award, but before fieldwork is conducted, the contractor will submit a detailed evaluation design, methodological framework, and implementation plan for review and approval by USAID. The following are illustrative data sources to be used by the evaluation team. Offerors are requested to propose other data sources and collection methods based on their understanding of the work to be done and proposed evaluation approach Evaluation of the USAID/South Africa Tuberculosis Program (FY2010-FY2014) Page | 72 - Document review (project reports and project generated data; capacity building tools and institutional strengthening plans; relevant national documents). A detailed bibliography will be provided by USAID - Key Informant Interviews with URC management and technical staff - Key Informant Interviews with sub-grantee management and staff - Key Informant Interviews with volunteer caregivers affiliated to sub-grantee programs - Key Informant Interviews with relevant government staff in line ministries (national/provincial level); National Department of Health (NDoH) and other line departments, - Focus Group Interviews/discussions with beneficiaries of TB programs The evaluators will conduct structured interviews with project staff, stakeholders and beneficiaries. To ensure that comparable information is collected during interviews, the team will develop interview guidelines for different groups of interviewees reflecting the evaluation questions. Field site visits to sub-grantee head/field offices and implementation sites will be identified by USAID in consultation with URC. Beneficiary interviews should concentrate on TB programs with a focus on selected provinces where the majority of URC partners operate. Offerors are requested to complete the evaluation matrix below based on their proposed evaluation approach and data collection methods. Before data collection, the contractor in coordination with USAID will finalize the matrix and include it in the overall evaluation design and methodology plan. Matrix: Evaluation Questions What evidence would you look for and what indicators or other assessment criteria will you use to qualify that evidence Data Source(s) and Collection Methods Data Analysis Methods 1. To what extent did the project achieve its intended results as stated in the contract’s objectives? What were the reasons for any shortfalls? 2. To what extent is the design of this project valid? How successful have been the programmatic and management approaches, structures and systems in carrying out the project’s activities? 3. Did the project strengthen the capacity of NTP sufficiently to ensure its sustainability? What role has the Small Grants program played in improving TB, TB/HIV and MDR-TB programs? 4. What strategies were used to reach vulnerable populations? E.g. Mines and work place programs. 5. How has the project integrated technology e.g. mobile health work in KZN, Active TB case finding? Existing Program Data Data available Brief description of data Successful Treatment Rate 1. New cases 2. Retreatment cases Lost to follow up 1. New cases 2. Retreatment cases Evaluation of the USAID/South Africa Tuberculosis Program (FY2010-FY2014) Page | 73 HIV testing rate amongst TB cases % TB patients tested for HIV in supported districts. TB screening rate amongst HIV positive clients % HIV positive clients screened for TB. END OF SECTION C Evaluation of the USAID/South Africa Tuberculosis Program (FY2010-FY2014) Page | 74 ANNEX 2: DETAILED METHODOLOGY Data Collection Methods To answer the 5 evaluation questions, the evaluation team undertook data collection in three ways: (1) document and data review, (2) key informant interviews, and (3) online surveys. Four key informant interview guides and three online surveys were created (see Annex 4 for copies of the tools). The target respondents and topics explored in the KIIs and online surveys are summarized in Table 7 below. Table 7. Summary of Respondents and Topics for Data Collection Methods Tools Target Respondents Topics to be explored Key Informant Interview Guides (N=4)  USAID and other donors  USAID  CDC  PEPFAR Liaison Officers  WHO  URC project management and leadership (general, staffing, consortium partners),  Project design  implementation model and implementation process  use of technology  capacity building  community work and small grants activity  project results  needs for future TB support.  URC Project staff  URC management and technical staff  Consortium Partners  DOH  National TB managers  Provincial TB Managers  Provincial TB M&E officers  District TB Managers  District TB M&E officers  Sub-district TB managers  Grantees /CSOs  Grantee managers Same instrument as online survey (see below) but administered face-to-face Online Questionnaires (N=3)  PEPFAR partners  Those trained by URC on TB/HIV integration  URC capacity building  project results  needs for future TB support.  Grantees /CSOs  All CSOs  Implementation process  URC capacity building  Quality of relationship with URC,  use of technology  needs for future TB support.  Facility Managers  A sample of DOH Facility Managers  URC capacity building  Training results  needs for future TB training DATA/DOCUMENT REVIEW This evaluation utilized a non-experimental design that excluded a rigorously-defined counterfactual. However, using data contained in annual reports and the South African Government’s information system, comparisons were undertaken to define the project’s contributions to TB services in South Africa. One comparison involved examining actual vs Evaluation of the USAID/South Africa Tuberculosis Program (FY2010-FY2014) Page | 75 planned project outcomes over time as a measure of project performance. Another comparison was the use of ETR.net data to compare performance in URC-supported districts vs non-URC supported districts in 4 TB outcome indicators:  Treatment Success Rate;  Lost to Follow Up;  HIV testing rate amongst TB cases; and  TB screening rate amongst HIV positive clients. Annex 3 lists the key data and documents reviewed for the evaluation. KEY INFORMANT INTERVIEWS (KIIS) In-depth KIIs were planned to be conducted with more than 100 key informants, mostly with Government at district and sub-district levels (Table 8). Sampling for KIIs was purposive, where individuals are chosen because of their roles and involvement in the project. Government interviews at provincial, district and sub-district levels were carried out in all 9 provinces, in 1-3 districts per province, and 1 sub-district per selected district (see discussion below on sampling for further detail). At provincial, district and sub-district levels, many KIIs were conducted as group interviews with more than 1 individual – the TB coordinator, the TB M&E officer (where applicable), and others. In each province, two CSO grantees supported by the project were also targeted for KIIs (i.e. 2 grantees per province). The remaining CSOs were reached through the online survey described in the table below. Table 8. KIIs – Targeted and Actual Targeted Respondents Actual Respondents Response Rate (%) No. KIIs South African Government 51 57 112% 45 - National Level 6 6 100% 5 - Provincial Level 9 14 156% 10 - District Level 18 21 117% 17 - Sub-district Level 18 16 89% 13 Donor 13 8 62% 8 - USAID 2 1 50% 1 - PPLs 9 4 44% 4 - Other Donors 2 3 150% 2 URC staff (sample) 16 37 231% 23 URC Consortium Partners 5 1 20% 1 CSOs receiving URC support (Grantees) 18 8 44% 8 Total No. KIIs 103 111 108% 85 KII tools for URC and CSOs were pretested in and around Gauteng and this informed the subsequent revision of all the tools. KII data was captured manually and then transcribed into Adobe Forms. Data in Adobe forms was automatically deposited in a database from where coding was done for analysis. Evaluation of the USAID/South Africa Tuberculosis Program (FY2010-FY2014) Page | 76 A detailed list of the specific individuals interviewed through KIIs is presented in Annex 3. Tools used for the KIIs are presented in Annex 4. ONLINE SURVEY Three online surveys were developed to collect the following feedback: 1. URC small grants recipients to obtain their views on achievements since the inception of the project as well as the quality of URC engagement with partners and the effects at facility and community levels. The survey was sent to all directors/programme managers at CSOs who have email addresses. 2. PEPFAR partners who received project training on HIV/TB integration 3. DOH Facility Managers who received training from URC on TB management and services delivery during 2014. These were voluntary surveys and as a result the response rates were less than expected. Table 9. Online Surveys – Targeted and Actual Targeted Respondents Actual Respondents Response Rate (%) No. of surveys Small Grants Recipients 115 38 33% 38 PEPFAR Partners 16029 4 3% 4 DOH Facility Managers 108 7 7% 7 The tools used for the three online surveys are presented in Annex 4. Sampling Framework KEY INFORMANT INTERVIEWS – GOVERNMENT Provincial Level: no sampling; all provinces are included in the evaluation. District Level: The project provided two kinds of support to districts and sub-districts – direct or indirect. Directly supported districts received the full range of technical support and training provided by the project. Indirectly-supported districts received only minimal support – usually participation in a training held for a directly supported district, or one-off support from URC staff. For this evaluation, sampling at district level focused only on the number of districts that were directly supported by URC. No indirectly-supported districts were targeted for the evaluation. Across the country, provinces had 1, 2, 3, 4 or 5 districts that were directly supported by the project.  Where the province had only 1 directly supported district (NC and WC), that district was selected for the sample.  Where the province had 2 directly supported districts (LP, MP, NW, GP) or 3 districts (FS), those districts were all selected for the sample. 29 The evaluation team never received the full list of PEPFAR partners trained by the project, and therefore was unable to fully explore the views of this target group Evaluation of the USAID/South Africa Tuberculosis Program (FY2010-FY2014) Page | 77  Where provinces had 4 or 5 directly supported districts (EC, KZN), we selected 3 districts for the sample. Sampling was purposeful, selecting one predominantly urban and one predominantly rural district, and randomly selecting a third. The final sample of districts to be included in the sample is presented in Table 10 listed below. Sub-district level: Among the selected districts, 1 sub-district per district was randomly selected. The sample of sub-districts is presented in Table 10 listed below. Of the 18 sub-districts in the sample, URC began supporting 9 in 2009 (under the previous project), 7 since 2010, and the remainder in 2012 and 2013. One sub-district ended their support in 2010 and the remainder were still supported in September 2014. Table 10. Sample of Districts and Sub-districts visited for KIIs Province District Sub-district Project Start Date Project End Date 1. Eastern cape 1. Amathole 1. Mbhashe 2009-10-01 2014-09-01 2. Chris Hani 2. Sakhisizwe 2009-10-01 2014-09-01 3. Nelson Mandela Bay 3. B 2012-07-01 2014-09-01 2. Free State 4. Mangaung 4. Bloemfontein 2010-01-01 2014-09-01 5. Thabo Mofutsanyane 5. Maluti a Phofung 2013-01-01 2014-09-01 6. Xhariep 6. Letsemeng 2013-03-01 2014-09-01 3. Gauteng 7. Sedibeng 7. Lesedi 2010-01-01 2014-09-01 4. KZN 8. eThekwini 8. North 2009-10-01 2010-09-01 9. uMkhanyakude 9. Mtubatuba 2010-02-01 2014-09-01 10. ILembe 10. Kwadukuza 2010-07-01 5. Limpopo 11. Sekhukhune 11. Elias Motswaledi 2009-10-01 2014-09-01 12. Waterberg 12. Mogalakwena 2009-10-01 2014-09-01 6. Mpumalanga 13. Nkangala 13. Emakhazeni 2009-10-01 2014-09-01 14. Gert Sibande 14. Lekwa 2009-10-01 2014-09-01 7. Northern Cape 15. Siyanda 15. !Kheis 2009-10-01 2010-10-01 8. North West 16. Ngaka Modiri Molema 16. Mahikeng 2010-10-01 2014-09-01 17. Kenneth Kaunda 30 17. Matlosana 2010-10-01 2014-09-01 9. Western Cape 18. Cape Winelands 18. Stellenbosch 2010-10-01 2014-09-01 Figure 45 on page 88 presents a map of the sub-districts visited as part of the fieldwork. KEY INFORMANT INTERVIEWS – OTHER Donor (USAID, CDC, WHO, Provincial PEPFAR Liaisons): Purposeful sampling was undertaken for these stakeholders per the list presented in Annex 3. URC Staff: Purposeful sampling was be undertaken for these respondents per the list presented in Annex 3. Given workloads, travel schedules, and vacancies, the Evaluation Team 30 substitute for Bonjala which was initially selected by stopped being supported by the project in FY2011 Evaluation of the USAID/South Africa Tuberculosis Program (FY2010-FY2014) Page | 78 endeavored to interview 16 staff from all levels of the team – management, programme, finance, M&E, small grants, etc. URC Consortium Partners: Purposeful sampling was undertaken for interviewing respondents from the five subcontractors per the list presented in Annex 3. CSO KIIs: According to information provided by URC, there were 100 grants issued to 75 CSOs in 4 “waves” since the beginning of the project. The Evaluation Team planned to interview 2 CSOs per province (18 total CSOs) when collecting data at provincial, district, and sub-district level. The 2 CSOs per province were randomly selected from the master list of 96, and are presented in Table 11. However, there were no current CSOs in the FS, and logistical restrictions resulted in 1 CSO interview in LP and 2 CSO interviews in the NW province being cancelled. However, these CSOs answered the online survey instead. Table 11. List of CSOs to be targeted for KIIs. Grantee Name Wave Province 1. Phaphamani Home Based Care 2 & 4 MP 2. TB HIV Care Association 1 NC 3. Octavovect Association 2 EC 4. Durban University of Technology 3 KZN 5. Centre for Positive Care 2 LP 6. Killer Disease Targeter 3 NW 7. Impangele Projects 2 WC 8. DENOSA 3 FS 9. Eastern Cape Gender Development Programme 4 EC 10. SABCOHA 4 GP 11. University of Pretoria (Department of Family medicine) 4 GP 12. Mpilonhle (A Good Life) 4 KZN 13. Humana People to people 4 LP 14. Isiphephelo HBC 4 MP 15. Ethembeni (place of Hope) Community and Trauma Centre 4 NC 16. Maboloka HIV&AIDS Organization 4 NW 17. SACTWU 4 WC ONLINE SURVEY CSOs: As indicated in above, 18 of 75 CSO grantees were contacted for face-to-face interviews using the online survey tool. The remaining 78 CSO grantees were requested to provide feedback on their grants through the online survey. Health Facility Managers: Numerous health facility managers were trained by the project in TB management and services delivery. All managers trained in 2014 were invited to complete the online survey to provide their feedback on the training. PEPFAR Partners Trained in TB: The project trained a wide range of PEPFAR partners on integration of TB into their HIV-focused work. The evaluation team never received the full list of Evaluation of the USAID/South Africa Tuberculosis Program (FY2010-FY2014) Page | 79 PEPFAR partners trained by the project, and therefore was unable to fully explore the views of this target group. Those PEPFAR partners, whose details were received by the team were invited to complete the online survey to provide their feedback on the URC support. . Fieldwork Fieldwork took place from 29 September 2014 until 2 November 2014. Data Analysis KIIs were manually captured into electronic forms (Adobe Forms) which allowed the data to be exported into Excel for coding and analysis. Quantitative data were analyzed using Excel/SPSS. The online survey data were exported from the online survey provider as an excel worksheet for analysis. The mix of qualitative and quantitative data generated through fieldwork were analyzed using methods appropriate to each. Some of the techniques which the evaluation team used to analyze the data are briefly described below. Qualitative analysis: Thematic Analysis: viewing the data several times as a whole, identifying patterns and themes, reorganizing the data (e.g. coding the data according to the themes identified); Triangulation: Cross-checking and synthesizing the data in order to increase the confidence in the findings; and use of multiple data sources; Quantitative analysis: Descriptive statistics: Frequency response analysis and percentages; and Graphs, for example: plotting of results against baseline or over project lifespan and trend analysis. Data Limitations Missing Data: The evaluation team was unable to access certain documents per the list provided in Annex 3. However, the missing information that most critically affected data analysis area as follows:  From URC: o A complete list of PEPFAR partners trained (which resulted in our difficulty in analyzing the training provided); o A breakdown of the project budget by key programme areas (e.g. grants, staffing, etc.) o IR indicator data for Years 3 and 4 (as this was not presented in Annual Reports) Data in difficult structures/formats: In addition to the above missing data, the team had difficulties with some programme data provided by the project, as it was either consolidated (which prevented us from conducting meaningful analysis), or it wasn’t consistently reported from year to year. For example, each of the project’s five Annual Reports were different in content and format, such that the same indicator data could not be followed over time from one report to another. Evaluation of the USAID/South Africa Tuberculosis Program (FY2010-FY2014) Page | 80 Difficulty in accessing “graduated” districts: fieldwork was initially planned such that the 2 “graduated” districts would be visited to see how the project’s inputs had been sustained. However, the team found it nearly impossible to conduct fieldwork in these locations as relevant personnel had departed and staff at the district or sub-district had little knowledge of the project’s support. One district in Northwest province was completely substituted. Evaluation of the USAID/South Africa Tuberculosis Program (FY2010-FY2014) Page | 81 ANNEX 3: SOURCES OF INFORMATION Documents and Data Reviewed Table 12. Documents and Data Reviewed Category Document Title / Date Received? USAID Country Development Cooperation Strategies South Africa CDCS 2013 -2017 Yes Regional RDCS 2011-2016 Yes Contract agreement and amendments Contract Yes Amendments NDOH TB Data Treatment Success Rate 2009-2012 Yes Default Rate 2009-2012 Yes HIV testing in TB patients 2009-2012 Yes TB testing in HIV patients 2009-2012 NTB Program Guidelines SAG National TB Management Guideline 2014 Yes Project quarterly, semi-annual and annual reports APRs FY 2010 Yes FY 2011 Yes FY 2012 Yes FY 2013 Yes Quarterly Project reports Q1 2011 Yes Q2 2011 Yes Q3 2011 Q1 2012 Yes Q2 2012 Q3 2012 Q1 2013 Yes Q2 2013 Yes Q3 2013 Yes Q1 2014 Yes Q2 2014 Yes Q3 2014 Yes Best practices 2009 Yes Training report 2014 Q1 Yes 2014 Q2 Yes 2014 Q3 Yes other training reports? Project M&E Data Indicator Data (excel) By district for the LOP Yes M&E Plan USAID TB PROGRAM ME PLAN OCT 2009 Yes PMP USAID TB Program – PMP_july_2014 Yes Evaluation of the USAID/South Africa Tuberculosis Program (FY2010-FY2014) Page | 82 Category Document Title / Date Received? Monitoring and Evaluation Annual report 2009 2010 2011 2012 Yes 2013 2014 Monitoring and Evaluation Quarterly report 2009 Q1 2009 Q2 2009 Q3 2010 Q1 2010 Q2 2010 Q3 2011 Q1 2011 Q2 2011 Q3 2012 Q1 Yes 2012 Q2 Yes 2012 Q3 2013 Q1 Yes 2013 Q2 Yes 2013 Q3 Yes 2014 Q1 Yes 2014 Q2 Yes 2014 Q3 Yes DQAs Amahlathi & Nkonkobe DQA report Yes Amathole DQA report Yes Capricorn DQA report Yes Dr K Kaunda DQA report Yes 2013 Q2 DQA Yes FIELD REPORT Eastern Cape Support Yes Mpumalanga-Emalahleni DQA report Yes Mpumalanga-Nkangala DQA report Yes Ngaka Modiri Molema DQA Yes Nkangala DQA report Yes NMBM DQA report Yes Sekhukhune DQA report Yes uMkhanyakhude DQA Yes Ethekwini North DQA report Yes Baseline Reports Dr Kenneth Kaunda District Baseline Assessment Yes Gauteng Sedibeng BA Report 25-30 June 2010) Yes Mafikeng Baseline Assessment Yes Metsweding District Baseline Assessment TB data feedback 05-11-2010 Yes Mkhanyakhude Baseline presentation Yes NMBM Baseline report Yes Sedibeng TBHIV Assessment Report Yes Evaluation of the USAID/South Africa Tuberculosis Program (FY2010-FY2014) Page | 83 Category Document Title / Date Received? TBHIV assessment Prelim Report 18 02 2011 Yes Tshwane District Baseline Assessment TB data feedback 11- 11-2010 Yes Umgungundlovu BA Report April 2010 Yes Zululand BA Report April 2010 Yes DVEs 2014 04 03 Mahikeng DVE Reporting Template Yes 2014.06.05 DVE for sharing with NTP_v3 Yes BCMHD DVE Yes DVE Report (Mangaung 17-21 February2014) Yes DVE Reporting KZN in Ethekwini District April 2014 Yes MATLOSANA DVE Yes Project Work plans and management reviews over the LOP Workplans 2010 Yes 2011 Yes 2012 Yes 2013 2014 Yes PEPFAR training reports Oct 10 - Sep 11 Yes Oct 11 - Sep 12 Yes Oct 12 - Sep 13 Yes Oct 13 - Sep 14 Yes USAID training reports Oct 09 - Sep 10 Yes Oct 10 - Sep 11 Yes Oct 11 - Sep 12 Yes Oct 12 - Sep 13 Yes Oct 13 - Sep 14 Yes Conference Presentations Lille- France 2011 Utilizing Mass Media to mainstream TB/HIV messaging to Change Behaviour Yes Community Dialogues as a Tool for Stakeholder Commitments towards TB/HIV Interventions Yes MDR XDR-TB Household Contact Investigation Yes Kuala Lumpur - MALAYSIA 2012 CBOs as key players in tuberculosis (TB) control: a success story from South Africa Yes Improving TB program performance through championing selected health facilities in South Africa Yes Partnering with district and facility staff to decrease smears not done pre-treatment in 4 facilities in Motheo district, Free State, South Africa Yes Multidisciplinary interventions for a successful Tuberculosis/Human Immunodeficiency Virus (TBHIV) integration in 178 South African health facilities Yes Towards improved tuberculosis (TB) treatment data outcomes in umKhanyakhude district, KwaZulu Natal, South Africa: a monitoring and evaluation approach Yes Paris - FRANCE 2013 Advocacy and Social Mobilization Strategies to improve early presentation of TB /HIV co-infected patients for care in Sedibeng District, Gauteng Province. Yes Evaluation of the USAID/South Africa Tuberculosis Program (FY2010-FY2014) Page | 84 Category Document Title / Date Received? Improving HIV care for TB/HIV co-infected patients in Ventersdorp Sub-District,Dr Kenneth Kaunda District, and North West Province. Yes Association between new smear positive pulmonary TB patients remaining positive at the end of intensive phase with other determinants in Matlosana Sub-District- Dr. Kenneth Kaunda Yes Improving TB treatment success through commitment of the district management team: UMgungundlovu Health District in KwaZulu Natal Yes TB case detection, prevention, and treatment in small mines in South Africa: A response to the SADC declaration. Yes Household contact tracing of Extensively and Multidrug resistant tuberculosis (MDR XDR-TB) patients in Tugela Ferry, KwaZulu Natal, South Africa, 2006-2012. Yes RSA - 2014 An Integrated ACSM framework to respond to TB in children Yes DUT USAID TB Yes Strengthening Linkages Between Communities And Primary Health Care Facilities For Early Mdr-Tb Treatment Yes Tracing TB contacts and treatment defaulters in mining communities in the North West Yes Leaving No Child Behind: Zero TB Deaths Improving TB Case Detection among children at community level The contribution of NGOs in strengthening health system for TB control at community level Yes Decentralised Vs. Centralised Care For MDR-‐TB Pa8ents: A Prospec8ve Cohort Study Comparing Final Treatment Outcomes In KwaZulu-‐Natal, South Africa (1 July 2008 – 30 June 2012) Yes Selected project research and technical reports, publications and tools Evaluations 2011 evaluation Yes End of Project Report TASC-II Tuberculosis, South Africa EOP report 2004-2009, URC Yes Grantee Special Studies Tuberculosis Initial Default study in South Africa - DTTC Yes A schools-based programme to increase awareness of TB/HIV and early case detection of TB (MRC-OVSA) - MRC (Atom) Yes Enhancing DOT support, treatment adherence and contact tracing in DR-TB outpatients - MRC (Atom) Yes Strengthening the paediatric cycle for TB care: Kid-Care - DTTC Yes Community Oriented Primary Care (COPC): A municipal ward based primary care intervention to increase TB case detection and improve TB cure rates (FY2013-G89-4740) - University of Pretoria Yes Evaluation of the USAID/South Africa Tuberculosis Program (FY2010-FY2014) Page | 85 Category Document Title / Date Received? TB and HIV integration: obstacles and possible solutions to implementation in South Africa - MRC Loveday Article Yes TB in Children Baseline Assessment Report Yes WHO reports WHO Global TB Reports 2013-2014 Yes WHO online country profiles Yes WHO Stop TB Strategy Yes WHO Engage-TB Summary Yes “TB Financing and Funding Gaps”, WHO 2013. Yes Other “Mapping the Donor Landscape in Global Health: HIV/AIDS (June 2013) and TB (August 2013), Kaiser Foundation Yes Financing the SA National Strategic Plan for HIV, STIs and TB 2012-2016: An Analysis of Funding Gaps and Financing Considerations, SANAC March 2014 Yes Persons Interviewed Organization Location Position Consortium Partner 1. BEA Enterprises USA President Other Donor 2. WHO/South Africa National National Professional Officer TB USG 3. CDC/South Africa National Chief, Care and Treatment Branch 4. CDC/South Africa National TB Officer 5. State Dept KwaZulu Natal PEPFAR Liaison 6. State Dept Gauteng PEPFAR Liaison 7. State Dept Eastern Cape PEPFAR Liaison 8. State Dept Western Cape PEPFAR Liaison 9. USAID National Project COTR SAG 10. National National Chief Director: TB Control and Management 11. National National Director Research, Information, Monitoring, Evaluation, and Surveillance (RIMES) 12. National National DR, TB & HIV Director 13. National National National Director of Susceptible TB (Director of TB Control and Management Cluster) 14. National National TB ACSM Director 15. National National Technical Advisor for MDR-TB 16. Provincial Eastern Cape Deputy TB Manager/Coordinator 17. Provincial Eastern Cape M&E Coordinator 18. Provincial Eastern Cape TB Coordinator 19. Provincial Free State Director TB, MDR and Communicable Disease Control 20. Provincial Gauteng Chief Director: TB Programme 21. Provincial KwaZulu Natal TB Coordinator 22. Provincial Limpopo HAST Director 23. Provincial Mpumalanga TB Department 24. Provincial Mpumalanga TB Department Evaluation of the USAID/South Africa Tuberculosis Program (FY2010-FY2014) Page | 86 Organization Location Position 25. Provincial Mpumalanga TB Department 26. Provincial North West PDOH Director for programme implementation TB and ART 27. Provincial Northern Cape Deputy Director TB Programme 28. Provincial Western Cape Provincial Coordinator for MDR-TB 29. District Amathole Deputy Directorn TB Programme 30. District Botshabelo Local Area Manager 31. District Cape Winelands District HAST Coordinator 32. District Chris Hani District TB Coordinator 33. District Dr. Kenneth Kaunda TB/HIV Coordinator 34. District Durban Municipality Senior Manager & TB Communicator 35. District Durban Municipality Vector Control Advisor 36. District Elias Motswaledi Sub-district TB coordinator 37. District Ethekwini District TB Supervisor & Acting TB Coordinator 38. District Ilembe TB Control Programme Manager (TB Coordinator) 39. District Nelson Mandela District TB Coordinator 40. District Ngaka Modiri Molema HAST District Manager 41. District Nkangala District TB Coordinator 42. District Sedibeng District TB coordinator 43. District Sekhukhune District TB Manager 44. District Thabo Mofutsanyana District Coordinator 45. District Umkhanyakude Clinical & Programme manager 46. District Umkhanyakude TB Coordinator 47. District Waterberg District TB Manager 48. District Xhariep District Coordinator 49. District Xhariep PHC Manager 50. District ZFM District TB Coordinator 51. Sub-District Emakhazeni Sub-district TB coordinator 52. Sub-District Emfuleni Sub-district TB coordinator 53. Sub-District Govan Mbeki TB Coordinator 54. Sub-District Hlabisa Previous TB Coordinator 55. Sub-District Hlabisa TB coordinator 56. Sub-District Lekwa TB Data Clerk 57. Sub-District Mahikeng TB Coordinator (and communicable diseases) 58. Sub-District Matlosana TB Coordinator 59. Sub-District Mbhashe TB Program Manager 60. Sub-District Mogalakwena TB/HIV Coordinator 61. Sub-District Ndwedwe Clinical Nurse Practitioner TB 62. Sub-District Sakhisizwe General Program Manager 63. Sub-District Sakhisizwe TB Program Manager 64. Sub-District Stellenbosh HAST Coordinator 65. Sub-District Sub-district B TB Coordinator 66. Sub-District ZFM Operational Manager Nursing URC Staff 67. Pretoria National Technical Advisor for Public Private Mix 68. Pretoria National Chief Technical Advisor 69. Pretoria National Data Capturer 70. Pretoria National Director of Grants Programme 71. Pretoria National Grants Administrator Evaluation of the USAID/South Africa Tuberculosis Program (FY2010-FY2014) Page | 87 Organization Location Position 72. Pretoria National Grants Administrator 73. Pretoria National Grants Team Leader 74. Pretoria National Lab Tech Advisor 75. Pretoria National M&E Officer 76. Pretoria National TB/HIV Advisor 77. Pretoria National Technical Advisor for MDR-TB and Infection Prevention 78. Pretoria National ACSM Officer 79. Pretoria National ACSM Advisor 80. Pretoria Eastern Cape NEC 81. Pretoria Eastern Cape and Mpumalanga NEC 82. Pretoria Gauteng Provincial Coordinator 83. Pretoria KwaZulu Natal NEC 84. Pretoria Limpopo and North West NEC 85. Pretoria Mpumalanga and Eastern Cape NEC 86. Pretoria Northern Cape and Gauteng NEC 87. Pretoria Western Cape NEC 88. Provincial Free State NEC 89. Provincial KwaZulu Natal District Nurse Coordinator 90. Provincial Amathole, Buffalo City and Chris Hani Provincial Coordinator 91. Provincial Amathole, Buffalo City and Chris Hani Provincial Coordinator 92. Provincial KwaZulu Natal Provincial Coordinator 1 93. Provincial KwaZulu Natal Provincial Coordinator 2 94. Provincial KwaZulu Natal Provincial Coordinator 3 95. District Eastern Cape Nurse Mentor 96. District Xhariep-Letsemeng Nurse Mentor 97. District Free State Provincial Coordinator 98. District Gert Sibande Provincial Coordinator 99. District Nelson Mandela Provincial Coordinator 100.District Nkangala Provincial Coordinator 101.District North West Provincial Coordinator 102.District Skhukhune Provincial Coordinator 103.District Waterberg Provincial Coordinator Sites visited Per Table 10, the evaluation team visited 18 districts and 18 sub-districts that were directly supported by the URC project. In the map presented in Figure 45, the “Dark Gold” coloured sub-districts are those that were visited as part of the fieldwork. Evaluation of the USAID/South Africa Tuberculosis Program (FY2010-FY2014) Page | 88 Figure 45. Map of districts visited during the evaluation. Evaluation of the USAID/South Africa Tuberculosis Program (FY2010-FY2014) Page | 89 ANNEX 4: DATA COLLECTION INSTRUMENTS Key Informant Interview Guide for URC respondents 1. What were your roles and responsibilities in the project? 2. What qualifications/experiences do you have that support your responsibilities? 3. Did URC provide you with sufficient training to carry out your roles and responsibilities? Please elaborate. 4. What are your thoughts on the organisational structure of the project team? Was it streamlined enough? Were roles clearly delineated? How has it evolved over time? 5. Did the project have the right mix of staff to meet its goals (probe for both management and technical staff)? 6. If not, how did URC ensure sufficient skills were on the team? 7. What, if any, were the obstacles in hiring local staff with sufficient technical expertise? 8. Was there a high staff turn-over? At which levels? What were the reasons? 9. What were your greatest challenges in carrying out your responsibilities? 10. What were your greatest successes in carrying out your responsibilities? 11. Which of the following intermediate results did your efforts directly contribute to and how? Please describe.  IR1: Increased Quality of TB Services  IR2: Increased availability of TB services  Increased demand of TB services  Improved management of TB support systems  Tested new approaches for expanding DOTS coverage 12. Was the project fully aligned with national strategies? Please describe. 13. What were the strengths and weaknesses of the project’s model/approach to working with government? Grantees? PEPFAR Partners? Consortium Partners? 14. Was the broad scope of the project an advantage or a disadvantage? Please elaborate. 15. Was the project’s implementation approach at provincial and district level —e.g. data review and analysis, action planning, small grants, etc – sufficient to achieve program objectives? 16. How did URC determine which activities to pursue in each local setting and organization? Were there standardized approaches that were promoted or did URC tailor the approach to the local situation/organization? 17. Was the funding for the program sufficient to achieve its goals? 18. Were there any issues in reporting against the 2 different funding streams (PEPFAR and USAID)? 19. Question for NEC/PC staff: Describe the provincial/district political climate and how it may have influenced project implementation. 20. How well were you able to engage the provinces/districts? 21. How well resourced were the provinces/districts to manage and implement the strengthened TB services? 22. How well did project implementation progress since 2010? What inputs were important for implementation and what was the quality of these inputs? 23. On average, how long did it take to initiate work with each province/district/sub-district/grantees? What were some of the difficulties in start-up? Were there solutions for these difficulties that were instituted, and by whom? 24. Did the project strike the right balance of flexibility and structure to be effective? (internal, external) 25. What internal and external factors required changes in the program? Was the program sufficiently responsive to the change requirements 26. How were URC consortium partners involved in the program? Did they have clear roles and responsibilities in the program? [BEA enterprises; Health Systems Trust; IUATLD; JHHESA; Karensoft Consulting; National Jewish Health]. Did they have the right mix of skills for implementing their sub-contracts? What were their strengths and weaknesses of the partners' contributions to the program? 27. What were the strengths and weaknesses of the approaches for reaching vulnerable populations? E.g. miners, prisoners etc. 28. Given the broad number of TB/HIV partners and players in South Africa, what is URC’s comparative advantage? Evaluation of the USAID/South Africa Tuberculosis Program (FY2010-FY2014) Page | 90 29. How did URC coordinate with other actors (not government or grantees) working in TB at the national, provincial, or district level? Are there examples of successful coordination? E.g. other donor funded projects. 30. What new technologies were introduced by the program? How were they employed and with whom? How successful was the technology effort? Were these technologies helpful in achieving the program/NTP goals? 31. Was there an emphasis on individual or systems capacity building? 32. Was this emphasis appropriate? If no, why not? 33. What worked well with training and what could have been improved? Comments: 34. How could URC’s capacity building be improved (probe for formal training vs on-the-job mentoring, systems development, etc)? 35. When the program began, training of other PEPFAR implementation partners was not part of the program. What was the evolution of adding this activity into the program? What have been the strength, weaknesses, and challenges of this activity? 36. The program provides small grants to NGOs; how does this mechanism help the NTP? What are the limitations of this mechanism? 37. What contributions have the small grantees made to the fight against TB in South Africa? To DR-TB? To HIV/TB integration? 38. What common challenges did the grantees experience in implementing their TB efforts? 39. What common challenges did the grantees experience in financial reporting compliance? 40. The URC TB program significantly improved capacity in ….?  Planning and management  TB related Monitoring and evaluation  Laboratory functioning  TB treatment service quality  HIVTB integration  TB case finding  Tb treatment adherence  Drug supply management  Infection control  Reaching vulnerable populations  ACSM (advocacy, communication and social mobilisation)  Other (please specify) 41. What factors contributed to these changes? 42. What were most the significant strengths and challenges of the project overall? 43. Did the program effectively address the weaknesses? If so, how? 44. 38. What are the most significant changes the Program has made to the Intermediate Results and what factors contributed to these changes?  Increased quality of TB Services  Increased availability of TB Services  Increased demand of TB Services  Improved management of TB Support Systems E.g. laboratory, drug and logistics management)  Tested New Approaches for Expanding DOTS Coverage  Reaching vulnerable populations 45. What aspects of the program are sustainable and will continue without further USAID funding? 46. What could URC have done differently/better? 47. Do you have any additional comments regarding the program? Key Informant Interview Guide for SAG respondents 1. What is your role at the DOH? 2. How much interaction have you had with the USAID TB Program/URC? 3. Describe any support the USAID TB Program has given you? 4. In your opinion, how well did the program coordinate with DOH? Evaluation of the USAID/South Africa Tuberculosis Program (FY2010-FY2014) Page | 91 5. In your opinion, how well did the USAID TB Program fit in with the overall National TB Programme? 6. How did the USAID TB Program determine which activities to support? Were there standardized approaches that were promoted or did the USAID TB Program tailor the approach to the sub-district, district, provincial level and local context? 7. Was the project's implementation approach at provincial and district levels—e.g. data review and analysis, action planning, small grants, etc -- sufficient to strengthen TB service delivery and management? 8. On average, how long did it take to start work with the USAID TB Program? Were there difficulties in start-up? If so, what were these challenges? Were there solutions for these difficulties that were instituted, and by whom? 9. How well did project implementation progress since 2010? What project inputs were important for implementation and what was the quality of these inputs? 10. Where any new technologies introduced by the Program? How were they employed and with whom? How successful was the technology effort? Were these technologies helpful in achieving the project/NTP goals? 11. How well did the program build capacity in TB? (probe for planning and management, laboratory functioning, treatment services availability and quality, HIV/TB integration, case finding, treatment adherence, advocacy/communication/social mobilisation) 12. What capacity building approaches were most effective? Was this emphasis appropriate? (e.g., formal training, mentoring, supportive supervision, online training, others)? 13. Was there an emphasis on individual or systems capacity building? Please give examples. 14. Was this emphasis appropriate? If no, why not? 15. What could have been improved in the programme’s capacity building efforts? (Probe for formal training versus on-the-job-training; systems development) 16. The URC TB program significantly improved capacity in ….?  Planning and management  TB related Monitoring and evaluation  Laboratory functioning  TB treatment service quality  HIVTB integration  TB case finding  Tb treatment adherence  Drug supply management  Infection control  Reaching vulnerable populations  ACSM (advocacy, communication and social mobilisation)  Other (please specify) 17. The USAID TB Program provided small grants to NGOs; What have been the strengths and weaknesses of the small grants program? 18. What contributions have the small grantees made to:  the fight against TB in South Africa?  DR-TB?  HIV/TB integration? 19. Given the broad number of TB/HIV partners and players in South Africa, what is the USAID TB Program’s comparative advantage? 20. Are there other options for accessing TB support that could be better than the USAID TB Program support? 21. In general, what have been the successes and challenges in working with the USAID TB Program? 22. Were these challenges addressed? 23. What are the most significant changes the Program has made to the Intermediate Results and what factors contributed to these changes? 24. Increased quality of TB Services 25. Increased availability of TB Services 26. Increased demand of TB Services 27. Improved management of TB Support Systems E.g. laboratory, drug and logistics management) Evaluation of the USAID/South Africa Tuberculosis Program (FY2010-FY2014) Page | 92 28. Tested New Approaches for Expanding DOTS Coverage 29. Reaching vulnerable populations 30. What aspects of the program are sustainable and will continue without further USAID funding? 31. If the project was to be discontinued today, how would this affect DOH and the fight against TB? 32. What could the Program have done better? 33. Do you have any last comments on the USAID TB Project support you received? Key Informant Interview Guide for USAID Respondents 1. In general, what have been the successes and challenges in working with the URC management team in Pretoria? 2. How did URC contribute to technical leadership, knowledge building, and collaboration in TB? 3. What are the project’s technical leadership strengths? Shortcomings? How might they be improved? 4. Was the project’s broad scope an advantage or a disadvantage? Please elaborate. 5. responsive to changes in national policies/ strategies? Please describe 6. What internal and external factors required changes in the project? 7. Given the broad number of TB/HIV partners and players in South Africa, what is URC’s comparative advantages? 8. How did URC coordinate with other actors working in TB at the national, provincial, or district level? 9. Are there examples of successful coordination? Please explain. 10. Was the project’s overall model/framework conducive to achieving the objectives and desired results? Why or why not? 11. Was the funding for the project sufficient to achieve its goals? 12. Was the project’s organisational structure sufficiently streamlined? 13. Were roles and responsibilities of project team members sufficiently delineated? Please elaborate. 14. Was the program managed in a manner that facilitated successful work at all levels? 15. Please describe the project’s strengths and weaknesses in reporting and management. 16. Were there any issues in reporting against the two funding streams? Please elaborate. 17. Did URC complete work plans, and reports in a timely manner? 18. Did URC have the right mix of staff to meet its goals (probe for both management and technical staff)? 19. What, if any, were the obstacles in hiring local staff with sufficient technical expertise? 20. What were the reasons for staff turnover in the project team? 21. How were URC subcontractors involved in the project? 22. Did they have clear roles and responsibilities in the project? 23. What were the strengths and weaknesses of the sub-contractors’ contributions to the project? 24. Was the project's implementation approach at provincial and district levels—e.g. data review and analysis, action planning, small grants, etc – sufficient to achieve project objectives? 25. How well did project implementation progress since 2010? 26. What project inputs were important for implementation and what was the quality of these inputs? 27. What have been the strengths and weaknesses of the small grants program? 28. In what ways did URC's training and technical support strengthen TB management and services delivery (probe for planning and management, laboratory functioning, treatment services availability and quality, HIV/TB integration, case finding, treatment adherence, ACSM, M&E, procurement)? 29. What were the most effective capacity building approaches used by URC (e.g., formal training, online training, mentoring, supportive supervision, regular visits, others)? 30. When the URC project began, training of other PEPFAR implementation partners was not part of the project. What was the evolution of this activity in the project? 31. What have been the strengths, weaknesses, and challenges of this activity? 32. What new technologies were introduced by the project? How were they employed? With Whom? 33. How successful was the technology effort? 34. Were these technologies helpful in achieving the project/NTP goals? 35. Were these technologies helpful in achieving the project/NTP goals? Evaluation of the USAID/South Africa Tuberculosis Program (FY2010-FY2014) Page | 93 36. List any significant shortcomings in the project? 37. Did URC effectively address these? If so, how? 38. Did the URC TB grant significantly improve capacity in the following areas?  Planning and management  TB related Monitoring and evaluation  Laboratory functioning  TB treatment service quality  HIVTB integration  TB case finding  Tb treatment adherence  Drug supply management  Infection control  Reaching vulnerable populations  ACSM (advocacy, communication and social mobilisation)  Other (please specify) 39. What factors contributed to these changes? 40. What could the USAID TB project have done better? 41. Any additional comments? Key Informant Interview Guide for Other Donor Respondents 1. Describe any relationship you have with the USAID TB Project? 2. How well did the USAID TB programme fit in with the overall national TB programme? 3. How does the USAID TB Project coordinate with your organization or other actors working in TB in the country – at national, provincial, or district level? 4. Did the URC TB grant significantly improve capacity in the following areas?  Planning and management  TB related Monitoring and evaluation  Laboratory functioning  TB treatment service quality  HIVTB integration  TB case finding  Tb treatment adherence  Drug supply management  Infection control  Reaching vulnerable populations  ACSM (advocacy, communication and social mobilisation)  Other (please specify) 5. What factors contributed to these changes? 6. Given the broad number of TB/HIV partners and players in South Africa, what is the USAID TB Program’s comparative advantage? 7. Are there other options for accessing TB support that could be better than the USAID TB Project support? 8. What could the USAID TB project do better? 9. Any additional comments? Key Informant Interview Guide for PEPFAR Provincial Liaison Respondents 1. How did the USAID TB Project contribute to technical leadership, knowledge building, and collaboration in TB in this province? 2. What are the project’s technical leadership strengths? Shortcomings? How might they be improved? 3. How well did the USAID TB Project fit in with the Government’s TB Programme in this province? Evaluation of the USAID/South Africa Tuberculosis Program (FY2010-FY2014) Page | 94 4. How did the USAID TB Project coordinate with other actors working in TB at the provincial, or district level? 5. Are there examples of successful coordination? 6. What were the strengths and weaknesses of the project’s model/approach to working in this province, at provincial and district levels as well as with CSOs? 7. Was the project's implementation approach at provincial and district levels—e.g. data review and analysis, action planning, small grants, etc – sufficient to facilitate improved Government TB services delivery and programme management? 8. How well did project implementation progress since 2010? 9. How well did project implementation progress since 2010? What project inputs were important for implementation and what was the quality of these inputs? 10. Were there any issues or difficulties with project start-up in this province or at district level? How were these resolved? 11. The URC project provides small grants to NGOs. Can you describe any strengths and weaknesses of the small grants program? 12. What new technologies, if any, were introduced by the URC project? How were they employed? With Whom? 13. How successful was the technology effort? 14. Were these technologies helpful in achieving the project/NTP goals? 15. When the USAID TB Project began, training of other PEPFAR implementation partners was not part of the project. What was the evolution of adding this activity into the project? 16. What have been the strengths, weaknesses, and challenges of this activity? 17. List any significant shortcomings in the project? Did USAID TB Project effectively address these? If so, how? 18. Did the URC TB grant significantly improve capacity in the following areas? What factors contributed to these changes?  Planning and management  TB related Monitoring and evaluation  Laboratory functioning  TB treatment service quality  HIVTB integration  TB case finding  Tb treatment adherence  Drug supply management  Infection control  Reaching vulnerable populations 19. Are there other options for accessing TB support that could be better than the USAID TB Project? 20. What could the USAID TB project do better? 21. Any additional comments? Online Survey of Facility Managers 1. The TB training I received from the USAID TB Program was of a high quality. (Strongly agree, agree, neutral, disagree, strongly disagree, Not applicable) 2. After the formal training, I received enough on the job TB training from the USAID TB Program. For example; mentoring and supervision. (Strongly agree, agree, neutral, disagree, strongly disagree, Not applicable) 3. I gained new and relevant information from the training that I did not already know. (Strongly agree, agree, neutral, disagree, strongly disagree, Not applicable) 4. The USAID TB Program training has improved my ability to:  Record TB data accurately  Integrate HIV/TB  Control infections  Detect cases  Manage laboratory support for TB Evaluation of the USAID/South Africa Tuberculosis Program (FY2010-FY2014) Page | 95  Increase TB treatment adherence 5. I faced challenges in using the training information in my job. (Strongly agree, agree, neutral, disagree, strongly disagree, Not applicable) 6. I use the skills I learned in TB training every day. (Strongly agree, agree, neutral, disagree, strongly disagree, Not applicable) 7. Any final comments? Online Survey of PEPFAR Partners Trained 1. Did your organization implement HIV and/or TB activities prior to receiving training/technical support from the USAID TB project?  No  Yes, HIV activities only  Yes, TB activities only  Yes, both HIV and TB activities  Don’t know 2. Please indicate the type of TB-related capacity building support given by the USAID TB programme to your organization/programme (tick all that apply):  Formal training  Online training  Mentoring  Supportive supervision  Introduction of new technology  Systems development  None  Other (please specify) 3. How effective was the TB capacity building support provided to your organization/programme?  Formal training  Online training  Mentoring supportive supervision  Introduction of new technology  Systems development  None  Other (please specify) 4. Did the USAID TB project emphasise individual or systems capacity building?  Individual capacity  Systems capacity  Both  Neither  Don’t know 5. Was this emphasis appropriate for your organization/programme? 6. Why was this emphasis not appropriate for your organization/programme? 7. The USAID TB Project significantly…improved my organisation’s/programme’s capacity in the following areas:  improved my organisation’s/programme’s capacity in Planning and management.  strengthened my organisation’s/programme’s capacity to deliver TB services.  Improved the skills of relevant staff in TB activities  Enabled my orgnanisation/programme to expand out TB offerings (i.e. TB related training technical support, services delivery, etc)  Enabled my organization/programme to expand the reach/coverage of TB services to target populations Evaluation of the USAID/South Africa Tuberculosis Program (FY2010-FY2014) Page | 96  Contributed to my organisation’s ability to deliver or support the delivery of, high quality TB services.  Was sufficient to integrate TB services with HIV services  Improved Monitoring and evaluation of my organisation’s/programme’s TB data  Helped us better align our services to the TB needs of the community 8. In your opinion, could the project’s capacity building be improved? 9. How could the project’s capacity building be improved? 10. The USAID TB Project and my organization/programme work productively together. (Strongly agree, agree, neutral, disagree, strongly disagree, Not applicable) 11. There was good communication between my organization/programme and the USAID TB project. (Strongly agree, agree, neutral, disagree, strongly disagree, Not applicable) 12. The USAID TB Project responded to all our technical/programmatic needs for TB support in a timely manner. (Strongly agree, agree, neutral, disagree, strongly disagree, Not applicable) 13. My organization is more sustainable due to the support of the USAID TB Project. (Strongly agree, agree, neutral, disagree, strongly disagree, Not applicable) 14. Did the USAID TB project introduce you to any new technologies for TB? How were they employed and with whom? How successful was the technology effort? Were these technologies helpful in achieving the project/NTP goals? 15. What were the benefits of the USAID TB Project support to your organization/programme? 16. What were the significant shortcomings of the USAID TB Project support to your organization/programme? 17. Did the USAID TB Project effectively address these? If so, how? 18. Are there other options for accessing TB support that could be better than the USAID TB Project support? Please elaborate. 19. If the USAID TB Project was to be discontinued today, how would this affect your organization/programme and the fight against TB? 20. If USAID support for TB continues, what should be the priorities? 21. What could the USAID TB project do better? 22. Any final comments? 23. Thank you for your time and feedback. We greatly appreciate your participation in this survey. Online Survey of Small Grantees 1. How many grants have you received from URC? Please Specify. 2. Did your organization implement HIV and/or TB activities prior to receiving the URC grant? 3. In what month and year did your most recent grant with URC begin? 4. In what month and year did your most recent grant with URC end (including any extensions)? 5. What activities were funded under the most recent URC grant? (click all that apply) 6. Please indicate If your organization works directly with vulnerable populations under the URC grant 7. Please describe any challenges experienced in accessing these populations. 8. Did you receive any formal (face-to-face) training from URC around the implementation of your grant? 9. How effective was the formal training that you received? 10. Were you provided with any online training from URC around the implementation of your grant? 11. How effective was the online training that you received? 12. Please indicate if you received any mentoring or supportive supervision from URC staff around your grant 13. Was there an emphasis on individual or systems capacity building? Please elaborate. 14. Was this emphasis appropriate? 15. If not, Why was this emphasis not appropriate? 16. How could URC’s capacity building be improved 17. Was there a main point of contact within URC designated for technical support to your grant? 18. How effective was this individual in supporting your technical work? 19. What was the most positive support you received from this individual? 20. What could this individual have done better? Evaluation of the USAID/South Africa Tuberculosis Program (FY2010-FY2014) Page | 97 21. Was there a main point of contact within URC designated for the contractual and financial support to your grant? 22. How effective was this individual in supporting your financial management and contractual compliance? 23. Was this person sufficiently knowledgeable to support you in financial management and contract compliance? 24. What was the most positive support you received from this individual? 25. What could this individual have done better? 26. The URC Project team and my organization worked effectively together. 27. There was good communication between my organization and URC. 28. Please elaborate on communication shortcomings. 29. The URC TB project team responded to all our programmatic needs in a timely manner. 30. The URC TB project team responded to all our financial management/contractual needs in a timely manner. 31. The URC grant enabled my organization to expand the technical scope/offering of programme activities. 32. The URC grant enabled my organization to expand the reach/coverage of programme activities to target populations. 33. URC’s technical training/support strengthened my organisation’s programme management and implementation. 34. URC’s financial management/contract training/support strengthened my organisation’s financial management and reporting systems. 35. The URC TB grant significantly improved my organisation’s capacity in the following areas:  Planning and management  TB related Monitoring and evaluation  Laboratory functioning  TB treatment service quality  HIVTB integration  TB case finding  Tb treatment adherence  Drug supply management  Infection control  Reaching vulnerable populations  ACSM (advocacy, communication and social mobilisation)  Other (please specify) 36. URC project provided my organization with TB-related technological solutions for our programme. If yes, please elaborate 37. My organization is more sustainable as a result of the URC TB grant 38. What are the most significant changes the URC TB grant has made to your organization? 39. List any significant shortcomings and/or inconveniences of the URC TB grant and/or URC support to your organization? 40. Are there other options for accessing TB-related funding and technical support that could be better than the URC TB support? 41. If the URC TB support was to be discontinued today, how would this affect your organization and the fight against TB? 42. If URC TB support to civil society organisations continues, what should be the priorities? 43. What could URC do better? 44. Any final comments? Evaluation of the USAID/South Africa Tuberculosis Program (FY2010-FY2014) Page | 98 ANNEX 5: TEAM MEMBERS’ DISCLOSURE OF CONFLICTS OF INTEREST Evaluation of the USAID/South Africa Tuberculosis Program (FY2010-FY2014) Page | 99 Evaluation of the USAID/South Africa Tuberculosis Program (FY2010-FY2014) Page | 100 Evaluation of the USAID/South Africa Tuberculosis Program (FY2010-FY2014) Page | 101 Evaluation of the USAID/South Africa Tuberculosis Program (FY2010-FY2014) Page | 102 U.S. Agency for International Development 100 Totius Street P.O. Box 43 Pretoria 0027 Gauteng South Africa