South Africa TASC II TB Mid Term Review Joy Riggs-Perla December 2006 2 Table of Contents Page List of Acronyms 3 Executive Summary 4 I. Background and Rationale for Review 6 II. Summary Project Description 7 III. Review Methodology 8 IV. Key Findings and Issues 9 a. Progress and Pace of Implementation 9 b. Strategic Approach and Achieving Outcomes 12 c. Monitoring and Measuring Progress 13 d. Quality of Activities 15 e. Capacity Building and Sustainability 18 f. Management Issues 19 V. Principal Recommendations 20 a. Strategic/Programmatic 20 b. Managerial 21 VI. Conclusion 22 3 List of Acronyms AIDS Acquired Immunodeficiency Syndrome AFB Acid Fast Bacteria (Sputum) Test BTC Belgian Technical Cooperation DFID UK Department for International Development DOH Department of Health DOTS Directly Observed Treatment Short-course DRAT District Rapid Appraisal Tool ETR Electronic TB Register HHDT Human Health Development Trust HIV Human Immunodeficiency Virus HST Health Systems Trust MDR TB Multi Drug Resistance Tuberculosis M&E Monitoring and Evaluation MSH Management Sciences for Health MTDP Medium Term Development Plan NDOH National Department of Health NGO Non Governmental Organization NHLS National Health Laboratory Service NTCP National Tuberculosis Control Program OR Operations Research PDOH Provincial Department of Health PEPFAR President’s Emergency Program for AIDS Relief PMTCT Prevention of Mother to Child Transmission PMP Program Monitoring Plan PPM Public-Private Mix QA Quality Assurance SO Strategic Objective STI Sexually Transmitted Infection TA Technical Assistance TADSA TB Alliance for DOTS in South Africa TB Tuberculosis URC University Research Co., LLC USAID United States Agency for International Development VCT Voluntary Counseling and Testing WHO World Health Organization XDR TB Extremely Drug Resistant Tuberculosis 4 Executive Summary The mid-term review was commissioned by URC in December 2006 to assess progress of the TASC II TB Project in South Africa. The review objective was to analyze the pace and quality of implementation, and identify any adjustments or changes needed. Because the project is roughly half way through its life, an assessment at this time allows time for any adjustments needed. The tasks included in the scope of work were to: document accomplishments and identify any problems, assess whether objectives have been met and systems established, assess quality of activities on site, identify issues and challenges for capacity development of the health care system, and recommend any improvements needed. The review concluded that TASC II TB, developed in collaboration with the National TB Control Program, was well-conceived and already showing some evidence of strengthening the TB programs at the facility level. Because of delays in launching the project, it is too soon to cite solid results across districts and provinces on outcome indicators such as improvements in cure rates, smear conversion, successful treatment rates or reducing default rates. However, the project is well positioned to do so, if the pace of work is accelerated and the focus tightened. Efforts to focus need to include support for a similar set of high priority interventions across the five provinces to enable achieving impact on a large scale. To date, a significant amount of work has been accomplished. Future effort needs to emphasize developing ‘systems’ for training, supervision, cross referrals between TB and HIV/AIDS and facility performance monitoring to ensure that the improvements will last well beyond the end of the TASC II TB Project. The review also recommended some ways of strengthening the program in the areas of TB counseling and defaulter tracking, expanding tools for supervision and monitoring, strengthening evidence-based public education efforts, and modifying the approach to operations research. It also suggested a few management improvements that should be considered. There is an urgent need to finalize the plan for collecting project monitoring and tracking data and to revise the Monitoring and Evaluation Plan accordingly. This is important to ensure that the same information is collected over time to assess progress and to do so in a way that is manageable. The primary emphasis in recording and reporting should be on strengthening the DOH’s Electronic TB Register (ETR) and on coaching district, sub￾district and facility staff on using the information for program management purposes. Nevertheless, an additional data collection system is necessary until the ETR becomes more reliable and timely. This is important not only for assessing problems and progress but also for reporting to USAID which requires accurate data to measure project impact. Because South Africa has a functioning public health care system with reasonable organizational and financial resources, the potential for the project having impact on the four selected outcome level indicators by September 2008 is good if the pace of implementation accelerates and span of key activities is broaden across provinces. Even 5 improvements in recording and reporting will yield results since management of TB patients appears to be better at the facilities than the data indicate. Patient compliance and defaulting behavior and key human resource constraints, however, present challenges for the program. TB and HIV/AIDS program coordination which is most difficult at the national and provincial levels is improving at the sub-district and facility level, as is the reporting of cross referrals. This program review makes a number of recommendations that can help strengthen the potential for significant impact and help TASC II TB become an even more effective partner to the NTCP. 6 I. Background and Rationale for Review South Africa is currently ranked 7th among the 22 high TB burden countries in the world. Furthermore, roughly half of those identified TB+ from sputum tests are also found to be HIV+ due to the problems with co-infection. The key national TB statistics for all provinces are as follows: National TB Data (2004) All Provinces1 The National Department of Health (NDOH) in 1996 launched the National TB Control Program (NTCP) including the DOTS strategy. A Medium Term Development Plan (MTDP) was also put in place with the targets of 85% cure rates and 70% case detection rates by 2005. The plan was officially endorsed by the MOH in January 2002.2 National TB Control Program (NTCP) Report estimates that the prevalence rate of all TB cases is 670/100 000 population, although a national population based TB prevalence survey has not been conducted in S. Africa for many years. The HIV/AIDS epidemic poses significant challenges to making progress on TB and therefore the national policy is to link the programs and address TB and HIV together. As in other countries, HIV and TB affect the most economically productive age group in the country resulting in a serious impact on productivity and a steady declining life expectancy. In 2006, life expectancy in S. Africa has dropped from the previous level of 64 to just over 50. To date, HIV infection rates continue to grow and have shown no decline in any age group. Multi-Drug Resistant (MDR) TB is also on the rise and more recently Extremely Drugs Resistant (XDR) TB has been identified in several provinces of the country leading to the development of a National Crisis Management Plan to address the XDR TB problem. South Africa has a functioning public health system which provides services free-of￾charge at the primary care level to all citizens. The government estimates that 23.3% of total government expenditures are allocated for health.3 TB and HIV services, including ARV therapy, are provided free-of-charge to encourage utilization by all patients who meet the criteria. Unlike some countries, while private practitioners provide a significant 1 DOH TB Program (http://www.doh.gov.za/tb/) 2 URC TASC II TB Technical Proposal, 2004 3 HST website (www.hst.org.za/health statistics) Indicator National Data (%) TB Cure Rate 50.8 Smear Conversion Rate 48.6 Successful Treatment Completion 65.5 Default Rate 10.3 7 proportion of health care in South Africa, their role in treating TB, especially among the poor, is limited, and most refer poor patients to the government system. TB drugs are available from pharmacies only with prescriptions which helps limit the amount of self treatment, a serious problem challenging other countries. Traditional practitioners are, however, a heavily-utilized source of care for the poor, including using traditional cures for TB and HIV. This may delay access to effective treatment by the public health system. In the government health care system, the provinces function in a largely autonomous fashion with the national level providing policy guidance, coordination and evaluation. Programs, including TB, are funded and supervised by provinces. Each province now has a designated TB Manager, as do some Districts. The district level provides the front line supervisors and managers of the TB program as well as other PHC services. Human resource constraints, however, are a serious problem for the DOH. Staff trained in TB frequently move to new jobs, and Professional Nurses find well-paying jobs abroad. Many nurses at the clinic and health center level struggle to manage huge patient loads in addition to their TB and HIV/AIDS related work. It appears that the availability of free services has, in many places, creating a problem of over-utilization, which creates a greater burden for the clinicians. I. Summary Project Description In August 2004, USAID awarded URC and its partners Health Systems Trust (HST) and Management Sciences for Health (MSH) a Task Order contract which launched the TASC II TB project in S. Africa. This was the outcome of the USAID project design process in which the National TB Control Program was heavily involved. The project is funded at roughly $10 million for the period August 2004 to September 2008. Part of the project funding comes from the President’s Emergency Program for AIDS Relief (PEPFAR) sources which are focused on TB-HIV activities. PEPFAR funds also come with a prescribed set of indicators reported on semiannually by TASC II TB beginning this year. The project is guided by periodic inputs from a Technical Advisory Group. It also attends the DOH quarterly TB meetings, which are reported to be very useful. Five provinces were chosen by the National DOH to participate in TASC II TB because they were felt to be in greatest need of additional support. The provinces are: KwaZulu Natal, Eastern Cape, Limpopo, Mpumalanga, and North West Province. TASC II TB works in selected districts of each province, and sometimes not in all sub-districts of each district. There are currently 252 facilities (both hospitals and clinics) in the five provinces. Roughly 4 million people live in the catchment areas of those 252 clinics, which represents about 8% of the total population of S. Africa (54.6 million in 20044 ) TASC II TB functions with a core staff of technical experts and administrative staff at the Pretoria office with one Provincial Coordinator in each of the five provinces. USAID has recently approved an increase in staffing for the project which includes additional 4 USAID/ S. Africa website 8 provincial staff, given the difficulties of having one person manage a growing set of responsibilities. TASC II TB contributes to achieving USAID’s Strategic Objective “Increased Use of HIV/AIDS and other PHC Services” and the intermediate result of “Treatment of TB and AIDS Improved”. Five key “tasks” contribute to achieving the desired results as follows: Figure 1 TASC II TB – South Africa Results Framework SO Level: Increased use of HIV/AIDS and other PHC Services Improved quality of TB services Increased availability of TB treatment Increased demand For TB treatment IR Level: Treatment of TB and AIDS Improved Improved management of TB support systems Tested innovative approaches for DOTS expansion Use of sputum Registers Institutionalized Access to and quality of microscopy improved Availability of HIV C&T for TB patients increased Training of health workers on TB increased Number of facilities offering comprehensive TB/HIV increased Number of facilities offering daily TB services increased Number of health facilities with outreach increased Number of health facilities offering TB/HIV increased Knowledge about TB control and treatment increased Number of facilities experience TB stock outages decreased Number of districts with HAST Committees functional Number of districts with functional Electronic TB Register Developed/tested new PPM models for TB expansion Model for pre-service and in-service training for doctors & nurses developed Work-based TB programs increased Number of NGOs providing DOTS increased According to the TASC II TB four year work plan, by the end of the project, the following results are expected: ¾ Increase case detection rate to 70% ¾ Increase treatment success rate to 85% ¾ Improved capacity to plan and implement TB/DOTS at facility, district, provincial and national levels ¾ Increase early detection of MDR ¾ Increased patient compliance and lower default rates. The Work Plan for TASC II TB includes a Monitoring and Evaluation (M&E) Plan which specifies the indicators to be monitored and the data collection plans. II. Review Methodology A consultant was hired to conduct this mid term review in December 2006. Because the consultant was hired by URC, this can be considered an internal review of progress and lessons learned to date. It represents an important opportunity to take an objective look at project progress and identify any adjustments or changes needed. Because the project is 9 roughly half way through its life, this is an appropriate time to make any adjustments needed. The analytic framework was based on the overall Project Work Plan, and the Monitoring and Evaluation Plan, included in the Work Plan as Annex A. The following are the tasks in the scope of work (see Annex 2 for details): • Document progress and identify any problems • Assess to what degree objectives have been met and systems established • Assess quality of activities on site • Assess accuracy and usefulness of information generated • Identify issues and challenges for capacity development of health care system • Determine whether focus is on the most promising activities based on experience to date and identify external constraints. • Recommend any improvements needed in technical/strategic oversight • Make recommendations about any changes that will strengthen implementation. The following methodology was employed: • Interview TASC II Pretoria and Provincial project staff • Interview TASC II contractual partners • Interview external partners and central government officials • Interview Provincial, District and facility staff and NGO counterparts using a structured set of questions (see Annex 4) • Review data and written documents available • Presentation of findings to TASC II staff and DOH/USAID at end of visit • Submit draft report for comments before December 20. The work in country was conducted from November 30 to December 15, 2006 with visits to all five provinces. Because of the limited time provided for this review, this assessment focuses on the broader strategic and technical issues rather than delving into the technical details of the program in each province. III. Key Findings and Issues A. Progress and Pace of Implementation It is appropriate to note at the onset that TASC II TB is an important investment for USAID and is already showing early signs of making measurable improvements at the clinic and in some cases at the district level. Because the project began work at the provincial level during 2005 (with activities in Mpumalanga province delayed until early 2006), it is still relatively early to document results at the outcome level. However, on a facility by facility basis and in some sub-districts, record keeping, for instance, is improving and case detection increasing due to project support work with counterparts on supervision and community mobilization. One cannot expect to see much improvement yet in the outcome level indicators given that the full TB treatment regimen is at least six months so generally program performance is not captured until the following year. Figure 10 Facility data: SCR Q1 to Q4 2005 0 20 40 60 80 100 120 Macambini Sundumbili CHC Mandini Ohwebede thukela kearsney maqumbi amandlalathi otimati Q1 Q2 Q3 Q4 2 shows data gathered by the project from Ilembe District in KwaZulu Natal Province in which some facilities are showing improvements in Smear Conversion Rates in the four quarters of 2005. Figure 2 TASC II TB staff have done a substantial amount of work in each of the five provinces to support TB management training, work closely with counterparts on improving the frequency and effectiveness of the supervision visits, and on improving the record keeping as a means to strengthen the Electronic TB Register (ETR). Work in the five provinces has varied somewhat depending on the local needs, the willingness of local counterparts to focus on particular tasks and the skills of the Provincial Coordinators themselves. At the national level, project staff, at the request of the DOH, assisted in the development of the TB Crisis Management Plan. The Crisis Management Plan has generated additional staff positions at the provincial level as well as, in some cases, additional budgetary resources allocated by provinces. Other partners such as BTC and DFID are funding some these positions. At the national level, a laboratory information system was developed but has not yet to been adopted by the National Health Laboratory Service (NHLS), an autonomous institution responsible for lab services in most provinces. TASC II staff have worked on improving cross referrals and record keeping for TB-HIV, and have assisted with modifying supervision material. The introduction of ‘suspect registers’ (to record all patients who are suspected of having TB based on clinical signs and the results of the AFB tests) gives facilities a good sense of whether they need to step up case detection efforts among patients presenting with other conditions or as the result of community TB awareness campaigns. Other important contributions and areas of work to date are: ¾ Provided training in TB/HIV management for doctors and nurses ¾ Helped establish or revive HIV/AIDS, STI and TB (HAST) Committees ¾ Developed advocacy material and social mobilization events for World TB Day 11 ¾ Developed and piloted low literacy TB and TB/HIV educational materials ¾ Developed broad cast media messages for local radio stations ¾ Provided co-funding with the DOH of two NGOs to assist with TB activities in two provinces The project is also in the process of conducting an analysis of trends in performance of TASC II assisted sites versus sites with no assistance from TASC II TB to determine if project interventions are helping to accelerate progress. The result of this analysis may be quite useful. Relationships between TASC II TB and counterparts at all levels, as well as other partners in the TB program, appear to be excellent. The response to questions about the usefulness of the TASC II TB was most enthusiastic at the facility level where they were quick to point out the support they had received. When questioned about whether there were important problems that TASC II was not addressing, one provincial TB manager said that it would be useful to have more information on the scope and mandate of the project to better understand the full range of activities that could be included. There did not appear to be significant problems associated with overlap of donor supported projects or activities. While the project has accomplished much good work to date, it has faced some important challenges, not all of which are easily addressed by the project or the National DOH. These constraints which have affected the pace of implementation are: ¾ Project Staffing Delays: URC has experienced difficulties filling positions. The project director began in April 2005, almost 9 months after the project officially began. Fortunately, the URC Regional Director was able to step in and keep activities moving ahead. Other key staff began at different times, most in 2005, but some later. ¾ Implementation Delays: On the DOH side, while the provinces were selected promptly by the NDOH, each province had to be consulted, a determination made about the specific project sites and approvals granted for beginning project work. In some cases those approvals were granted very late by provinces. Mpumalanga was the last province to come on board in January of 2006. While these approvals and the preparatory work caused delays, it was extremely important to undertake this preparatory work to ensure the provinces were willing and prepared to take on the collaborative work with the project. Any short cutting of this process would have been unwise. The NDOH was also instrumental in trying to expedite the initiation of work at the provincial level. Another DOH issue at the provincial, district and sub-district levels has sometimes been the availability of counterparts who are available to travel and work with the Provincial Coordinators. Because there are few staff at the lower levels in the system whose responsibility is limited only to TB, many of them are busy with other programs and have sometimes not been available to participate in 12 project initiated work. Because the emphasis of TASC II TB is capacity development of the TB program managers at the sub district and district level, when these staff are not available, the Provincial Coordinators are somewhat constrained. In some cases, the vacancies of staff at the national level have caused problems. An example of this is the Training Advisor position at the DOH has been vacant and was only very recently filled, making progress on developing a national TB training plan difficult. Furthermore, some activities outlined in the original work plan may be delayed due to factors beyond their control. An example is the plan to develop a laboratory quality control program, although it is not clear at this juncture that the NHLS is interested in working with the DOH or URC on this issue. ¾ Because of these delays, the TASC II TB project has only recently begun to take off with a full set of activities in most provinces. Unfortunately, September of 2008 is looming near. Only 21 months of time remain to demonstrate that the interventions supported by the project will result in improvement in the four key outcome indicators, i.e., cure rates, smear conversion rate, default rate and successful treatment completion. The project may or may not be extended but the staff must assume that it will not and plan accordingly. This puts tremendous pressure on project timetables. ¾ The other issue which increases the pressure for TASC II TB is the fact that, for a number of reasons cited earlier, the baseline surveys in each province were conducted between February and November of 2005, in some cases more than a year after the formal initiation of the project. This delay reduces the amount of time available to implement activities and demonstrate that those interventions will result in the desired (and promised) program improvements. B. Strategic Approach and Achieving Outcomes The strategic framework (Figure 1) not only provides the strategic logic of the project but is also used as an organizational framework for work plans and reporting. In that way, it is very clear how activities fit together and how they contribute to achieving the four key outcomes. At this juncture, however, there is a need to begin the process of focusing attention and efforts on those activities which will contribute most directly to achieving the project outcomes and put aside those activities which are either peripheral to the outcomes or are important but whose progress may be constrained by factors beyond the project’s control. Clearly the work related to TB and TB-HIV training, supervision and cross referral systems, strengthening counseling, reducing default rates, strengthening record keeping and the use of data to manage the program, social mobilization and DOTS supporters to increase case detection and patient compliance, assisting in national efforts for the Crisis Management Plan.are all highly important and directly related to achieving impact. 13 The effort to focus should also include maximizing the strategic coherence of the program across the five provinces. In other words, the key interventions need to be applied consistently in all project sites to ensure that the interventions are being applied widely enough to make a difference across the five provinces. A proactive effort to expand successful approaches, techniques or tools used in one location to all provinces should be a high priority, with adaptations as needed in each area. This is sometimes challenging as each province may have different priorities and preferences, however, the project staff should not be purely reactive and where appropriate, push for approaches that have demonstrated to be successful elsewhere. Examples include the use of patient￾provider contracts (see section IV D) if they prove effective, the District Rapid Appraisal Tool (DRAT) now used in Eastern Cape (or other better tools if available) and the application of ‘collaboratives’ as a means for meeting periodically to reinforce skills in data analysis and utilization. Potential candidates for activities that might be tabled are: ¾ Medical school curriculum reform to introduce DOTS could have good long term benefits but is notoriously difficult and would not likely to contribute significantly to achieving the outcome within a four year period. ¾ Pursuing PPM models as described in the original work plan to work with private practitioners. The ‘private sector’ efforts for this program should focus more on the mines and other large employers who do already diagnose and treat TB patients among their employees or should be referring and tracking employees infected with TB and/or HIV. ¾ NHLS QA program, only because it is not clear yet whether any progress is possible. The project leadership recognizes this challenge and has started the process of trying to consolidate efforts. It needs to be more explicit with an emphasis on the major themes across all provinces. C. Monitoring and Measuring Progress TASC II TB is organized around the strategic framework illustrated in Figure 1. The framework is logical, and given sufficient time and effort, should result in significant improvements in the four key outcomes, at least in the project areas and beyond if the DOH applies the same techniques and systems to neighboring areas or provinces. The project itself is expanding gradually at the request of the provinces to areas beyond the original 252 health facilities as well. Much to their credit, TASC II TB has organized its work plans and reporting to flow from the strategic framework facilitating an understanding of how activities relate to achieving the project targets. A highly important part of TASC II TB interventions are those related to improving recording and reporting, and the use at facility and district levels of the information generated. In this regard, improving the accuracy and timeliness of the Electronic TB 14 Register (ETR) data is the most important focus for institutionalizing improvements in collecting and using program data. Ideally, the impact of project activities should be captured by analyzing ETR data. However, there are still sufficient difficulties with that system which makes it necessary (and wise) for the project to collect some data independently. This is in part due to USAID’s requirement to demonstrate the achievement of results using reliable data. And it is an especially rigorous requirement for the President’s Emergency Program for AIDS Relief (PEPFAR) funded activities within TASC II TB. The revisions in the VCT registers now in use and the TB registers (now being revised) to add information about TB-HIV cross referrals are another important improvement to the formal recording and reporting system. PEPFAR indicators: The PEPFAR program has a prescribed set of indicators related the TB-HIV activities funded through that source. They are fairly rigid in the sense that modifications cannot be made in the indicators and the project must report to USAID semiannually on all of the relevant indicators. Furthermore, many of those data are required to be submitted as numbers rather than percentages or coverage of specific populations. There are targets that have been established and as long as the project meets those targets, there appears to be less concern about the distribution or coverage they represent. TASC II TB, therefore, conducted a survey of 75 of the original 252 facilities using a “Facility Tool” and will do so every six months to capture the needed information. They also work with the facilities to ensure that those data are carefully recorded in the event that they need to be verified. In the next round of the “Facility Tool” the plan is to begin with ten facilities in each province and gather data until the targets are met and then stop. Given the nature of those information requirements, this seems like an unusual but appropriate way to proceed. Non-PEPFAR indicators: The other indicators listed in the M&E plan are similarly gathered using the Facility Tool. The plan has been (as described in the M&E plan) to collect the information on a quarterly basis from at least 75 facilities. The Provincial Coordinators, however, have reported that this special data collection effort poses significant challenges for them because it is extremely time consuming and takes precious time away from other activities that are more geared to capacity development and working on improving the ETR, which of course is the information system that will last beyond the life of this project. There has been a lack of clarity about the frequency and scope of this data collection effort which needs to be resolved quickly and decisively. The M&E plan then needs to be revised to accurately reflect the decisions made. The recommendation from this assessment is that a quarterly data collection effort is not needed and that it should be conducted only semi annually to coincide with the PEPFAR data collection efforts. The project should hire additional short term staff, guided by the Provincial Coordinators, whose sole task is the gather the needed information. In that way, the Provincial Coordinators can focus on the capacity development and mentoring activities on data use as well as improving the ETR that will serve the best interest of the DOH over the long term. The decision about whether to collect the data from the 252 facilities covered by the baseline survey or a reduced number should be made based on feasibility and cost. Clearly the most useful would be to have this information from all 15 252 as it would provide the most complete data on progress and problems both for the Provincial Coordinators, their DOH counterparts as well as those at the national level. One of the most effective ways of strengthening the ETR is to help facility staff understand the importance and relevance of the information. The “collaboratives” work supported by TASC II TB, where facility level staff are brought together at the district or sub-district level to analyzed data from their own clinics and talk about issues is very useful in this regard. On site mentoring and feedback on performance using the data will also help reinforce for facility staff that even when they are managing effectively and curing the patients, they will not get credit or recognition until the key information is accurately recorded. Currently, it appears that while some facilities do report that supervisors do show them the clinic level data and discuss performance, few facility staff have either the time or ability to do this analysis on their own. This is especially true in the busier urban clinics. Another issue that emerged during this review is that the baseline data reported to USAID was by necessity incomplete because at the time it was due, all five provinces had not been surveyed. Therefore the results from three provinces were reported. Furthermore, in the baseline studies, one of the key outcome indicators, successful treatment completion, was not collected and therefore the figure reported was probably taken from the ETR rather than from survey information. The Facility Tool now includes that indicator so future reporting will include it. During the review visit, the TASC II TB M&E officer recalculated the baseline data, this time including all five provinces and made an estimate from the ETR for the successful treatment completion, using data only from project sites. This has resulted in the need to update and correct the baseline information both with USAID and in the M&E plan. Figure 3 shows the corrections: Figure 3 TASC II TB Baseline Data Aggregated 2005 (%) Note: data aggregated using simple averages, ranges for values in each indicator are sometimes large. *Updated based using baseline survey data from 5 provinces ** Original baseline data submitted to USAID in March 2005 For three of the four indicators, the values are fairly close. The exception is the “successful treatment completion” which is lower that the earlier figure reported which included non-project areas within the 5 provinces. Analysis of the data collected in 2006 using the facility tool will help update and hopefully provide a more accurate estimate for that indicator. Indicator 2005 Baseline (5 provinces)* 2005 Baseline (3 provinces)** Cure Rate 52.7 51.3 Smear conversion 54.7 52.3 Successful treatment completion 57 (ETR) 69 (ETR) Default rate 13.9 12.5 16 It is important to keep in mind is that these aggregates are done primarily to simplify and reduce the volume of data that goes to USAID, however, they are not especially useful for program management or oversight. The facility and sub-district level data are of greatest value to the project itself and to the DOH. In addition, while these four indicators are the outcome level ones, there are 53 other key indicators (PEPFAR and Non￾PEPFAR) which help assess progress and are looked at carefully by staff and their counterparts. The original M&E plan also includes a number of indicators which are not very useful and should not be collected. The project had also realized that and had therefore not included these in the Facility Tool. Again, the M&E does need to be amended to reflect this decision. D. Quality of Activities There are a number of observations from this assessment which fall under the category dealing with the quality of interventions. Laboratory quality assurance: The original work plan places a fair amount of emphasis on improving the quality of laboratory services. Currently, in all but one province (KwaZulu Natal), laboratory services for TB are provided by the National Health Laboratory Service (NHLS), an autonomous body which is not part of the Department of Health. KwaZulu Natal will, however, by next year, also use NHLS to manage their laboratory services. Therefore any significant intervention to analyze or strengthen the current services or establish an external QA system would need the agreement of NHLS. TASC II TB has worked on the development of a laboratory information system but this is currently on hold. The national DOH has had discussions with the NHLS about such activities but there is no specific agreement to date about a role for TASC II TB. Under these circumstances, the best that TASC II TB can do is to work with health facilities and NHLS staff at the local level on more localized issues such as improving the turn around time (TAT) for sputum test (AFB) results or adjusting schedules for sputum pick up. During this review, the visits to the facilities and an examination of the records showed that, for the most part, TATs have improved significantly, and in many facilities, specimens were picked up daily by NHLS couriers. Some facilities did voice concern about the quality of results but at this juncture, working in that area in any systematic way would require a more formal relationship with NHLS. The recommendation here is to await guidance from the NDOH and move forward with NHLS only if an opportunity arises. TB counseling and defaulter tracking: Patient behavior is clearly one of the most important challenges for the National TB Control Program in South Africa. Defaulter rates are high for a variety of reasons, and staff shortages and generally heavy patient loads mean that patient counseling is often not ideal. Improvements have been made in some facilities by having a separate section of the clinic for TB patients and dedicating one of the nurses on duty to dealing with just those clients, but not all nurses have formal 17 training on counseling or any specific tools to use. TASC II TB has developed some patient education brochures but they are not yet available in all of the local languages. Clinic staff also have a difficult time with tracing defaulters due to lack of transportation and often just simply lack of time. In some cases, tracing patients is made more difficult between provincial DOH owned facilities and the clinics owned and managed by municipal government units. While DOTS supporters, where they are functional, help with tracking defaulters, in some areas visited, DOTS supporters had stopped working in TB in favor of becoming HIV/AIDS community workers because they are compensated with monthly stipends from government or NGOs. In other places, DOTS supporters received monthly stipends but the clinics were allotted a limited number of DOTS supporters per facility causing those who do not receive stipends to stop working as volunteer DOTS supporters. During the review, clinic staff sometimes reported that patients living in distant communities were sent home with the initial one month supply of drugs with no DOTS supporter available to monitor patient compliance or liaise with the facility. In other cases, family members were used as DOTS supporters even though many felt that this was not an ideal solution. Whenever possible, family members were included in the counseling sessions. Sigma associated with TB also influenced whether patients would accept either family members or neighbors as DOTS supporters. In some provinces, facility staff used traditional practitioners as DOTS supporters in an effort to incorporate them into the program. Clearly, TASC II TB should intensify its work on improving the quality of counseling and defaulter tracing across all 5 provinces. In one province, there is a plan to introduce a patient – provider “contract”, which would outline the responsibilities of both patient and facilities during period of TB treatment as a way to formalize patient responsibilities for notifying the facility when they move or discontinue treatment for whatever reason. It would also include cell phone numbers or other means of finding defaulters. Defaulting on ARVs is less of a problem in South Africa given the intensive preparatory work and attention to counseling. While it is unlikely that TB will have the same resources available as the national HIV/AIDS program, strengthening the counseling programs should receive priority attention given the problem with defaulters given the increasing danger represented by MDR and XDR in the country. Another issue which is apparently a problem influencing defaulters in some regions is one related to the stipends that government provides to poor or unemployed patients who are confirmed to have TB and/or are HIV+. For TB, those subsidies are discontinued after the six month period if treatment was successful. Some facility staff interviewed were convinced that patients sometimes have little incentive to continue their drugs after they begin to feel better, especially knowing that a full cure will result in an end to stipends. This is a broad public policy issue that needs further investigation and discussion. Tools for supervision and monitoring performance: Improving the supervision of facility level TB staff and monitoring performance of the facility in TB and TB-HIV is a major area of emphasis for TASC II TB. The aim is to work intensively with sub-district 18 and district level counterparts to improve the frequency and quality of the supervision provided. The project has worked on modifying and improving the supervision tool provided by the NDOH which includes a supervision checklist covering all areas that need to be checked and discussed with facility staff during supervision visits. These supervision visits are at the heart of improving program performance and are highly important. Care must be taken to ensure that the tools used are as simple and functional as possible. Highly detailed manuals that are difficult to use, and even more difficult for facility staff to understand, should be avoided. Checklists that can guide discussions and where possible, copies left with facility staff are much more useful. The purpose should be to help facility staff understand the areas in which they need improvement and the elements of the program that will be monitored during supervision visits. In many facilities, where there are problems retaining professional nurses, staff nurses are managing a number of TB and TB-HIV program activities making it even more important to keep the supervision tools as straight forward as possible. This is offered more as a suggestion for the future than because any specific problems were identified. Social Mobilization: Some of the mass media related work on the project does not appear to be completely evidenced-based in that the strategic plan does not include information on whether or how there will be any measurement of behavioral change as a result of the public education efforts. This review recommends that media efforts focus on local media on a pilot basis to determine whether the efforts are having any impact on increasing case detection or patient compliance to therapy rather than general increase in knowledge about TB which is already likely to be high. Promising approaches that demonstrate impact can then be expanded and the DOH will know for its broader program the kinds of public education interventions that are the most effective. Operations Research: While the original work plan did not place a lot of emphasis on operations research, subsequently, there has been an effort to identify key issues that would benefit from some field intervention studies. Not much has been accomplished to date due to the long clearance procedures related to ethical review and approvals at several levels within the DOH. Because building research capacity is not the aim of this project, the recommendation is to keep all such studies as practical and short as possible. Studies and pilot interventions should be used as a tool for solving field level problems that constrain achieving the key outcomes effectively. Since S. Africa already has substantial research capacity and a long history of research activities, any study should be preceded by a literature search to determine whether the research questions have already been addressed by others. Rapid assessments and other shorter term studies should be emphasized given the limited amount of time remaining on the project to use the findings to make programmatic changes or improvements. E. Capacity Building and Sustainability TASC II TB project staff all appear to be highly cognizant of their role as technical advisors whose function is to build capacity of the system at the peripheral levels. Activities have sometimes been delayed when counterparts were not available to participate. However, there needs to be more emphasis on building “systems” for training 19 and supervision and the other areas of project focus. The approach should begin with building the components of a program that will last beyond the life of the TASC II TB project. For example, in training, rather than simply improving training skills of the provincial or district staff or funding training sessions, TASC II TB should systematically help develop (as appropriate) a group of core training staff in the provinces with skills in adult education techniques, improve the training curriculum, establish a regular system for follow-up with trainees to reinforce skills, and provide on-site mentoring, develop training needs and schedules, and so on. The same applies to supervision systems. The Pretoria based staff need to assist the Provincial Coordinators to ensure that the approach is ‘systems’ oriented instead of a more ‘opportunistic’ approach. In some provinces this may be more difficult where they have strong preferences about how to use the project. The project leadership is very much aware of these issues, and it has been a point of discussion during the last annual meeting. Another avenue to help build longer term capacity and sustainability, as well as to build better support for TB programs at the community level, is to work with local NGOs. This especially important in an environment where stigma associated with TB, compounded by fears about TB patients being HIV infected, very much affect social and community level support for TB programs. It also affects the willingness of TB patients to get tested for HIV. Patients referred to VCT are generally willing to be counseled but often refuse the HIV testing. NGOs can help develop DOTS supporters, help reduce stigma and greatly assist the facilities to maintain continuous tracking of TB patients. Furthermore, provincial governments in South Africa are already providing grants to NGOs and are likely to continue to do so. Two NGOs have been provided sub-grants by TASC II TB. One in KwaZulu Natal, HHDT, works on the expansion of DOTS supporters at the community level while the other in Limpopo province, TADSA, is involved in training for nursing staff. HHDT has had some success in other projects providing sub-grants to smaller NGOs to build their capacity to receive donor funds directly. TASC II TB Project must be careful about expanding the number of grants given how management intensive such grant programs can become. However, if there are opportunities with HHDT or other NGOs to take on the management of smaller grantees, this may be one way of increasing the involvement of NGOs efficiently. Following the current pattern, co-funding these grants with the DOH should be continued. F. Management Issues The following are a few management related suggestions for consideration at this juncture in the project: ¾ Place high priority on hiring the new staff as time is a running out and finding capable staff willing to join the project will decrease with time. The NDOH has suggested that in some cases, full time staff may not be needed but rather hiring 20 short term staff and longer term local consultants to take on specific pieces of work. This is appealing because it will allow the project to remain lean and flexible from the managerial perspective. It is clear, however, that the pace of activity will be constrained unless there are more people to assist with the field activities given the number of facilities and provinces involved. From looking at the financial pipeline, it appears that resources will not be a constraint. ¾ Ensure that all senior staff report to the URC Project Director. To maintain a clear chain-of-command and strategic direction, all key sub-contractor staff should be co-located and be supervised by the Project Director. That is not to say that the Project Director needs to supervise all staff. Even now her supervision responsibilities are extremely heavy; and there is a plan to shift supervisory workloads to some of the more experience and senior staff. ¾ All project staff need to be held accountable for accomplishing key activities, not just the Provincial Coordinators. While the Provincial Coordinators should be involved in and direct all activities within their provinces for the project, Pretoria based staff are more than just advisors and need to ensure that the nature, pace and quality of implementation in each province is sufficient to achieve the targets. Work plans and staff performance reviews should reflect this expectation. ¾ TASC II TB should buy project digital cameras; develop a project documentation center (both electronic and hard copy) and a photo library. Documents from other organizations that are relevant and useful for the project should be readily available for use by project staff. This includes the district profiles and other material from HST which appear to be potentially very useful as well as other documents from WHO, and so on. Hard copies of all key project documents need to be readily accessible. IV. Summary of Principal Recommendations The following in bulletized form are the key recommendations and actions emanating from this program review. a. Strategic/Programmatic i. Focus on fewer activities that have potential for direct contribution to achieving the four project outcomes (i.e., improved cure rate, improved smear conversion, increased successful treatment completion and reduced default rate). Emphasize systems likely to continue beyond the life of the project. ii. Drop or suspend activities that have a less direct contribution to the outcomes or which are beyond the control of the project. 21 iii. Continue to use the Facility Tool to collect data from the project sites as the ETR is being strengthened. Collect and report all data only every six months for both PEPFAR and Non-PEPFAR indictors. iv. Make final decisions on the frequency and methodology of collecting the key indicators for the project and formalize and document this by revising the M&E plan. v. Submit the updated baseline data to USAID. vi. Ensure the focus of activity by Provincial Coordinators is on the program monitoring activities is on the capacity development work with the ETR and the facility staff. Hire additional short term people for the project data collection using the Facility Tool. vii. Intensify work on improving counseling and defaulter tracking. viii. Measure behavioral impact of social mobilization/media activities in relation to key outcome indicators. ix. Use Operations Research to solve important field problems using rapid assessment and limited interventions to obtain results within a short period of time. b. Managerial i. Hire additional full time staff as soon as possible. As appropriate and feasible, consider use of local consultants and short term or part time staff for certain tasks (e.g. compiling and drafting quarterly or annual reports, etc.) ii. Broaden staff supervision responsibilities to other senior staff beyond the Project Director and hold all Pretoria staff responsible for key project activities in each province. iii. Continued planned improvements in teamwork among the TASC II TB contractual partners. iv. Improve the organization and availability of project documents and photos. 22 V. Conclusions The TASC II TB Project is a well –conceived and potentially highly useful mechanism for the DOH to strengthen its TB and TB-HIV program in at least five of the provinces in the country whose performance needs special attention. Because South Africa has a functioning public health care system with reasonable organizational and financial resources, the potential for the project having impact on the four selected outcome level indicators by September 2008 is good if the pace of implementation accelerates and span of key activities is broaden across all provinces. Even improvements in recording and reporting will yield results since management of TB patients appears to be better at the facilities than the data indicate. Patient compliance and defaulting behavior and key human resource constraints, however, present challenges for the program. TB and HIV/AIDS program coordination which is most difficult at the national and provincial levels is improving at the sub-district and facility level, as is the reporting of cross referrals. This program review makes a number of recommendations that can help strengthen the potential for significant impact and help TASC II TB become an even more effective partner to the NTCP.