USAID/SOUTH AFRICA PEPFAR TREATMENT PROGRAM FINAL EVALUATION SEPTEMBER 2011 This publication was produced for review by the United States Agency for International Development. It was prepared by Jane Gruber von Kerenshazy, Lois Eldred, Ntombi Bandezi, Elna Hirschfeld, Peter Njaramba, Mary Pat Selvaggio, and Robin Wood, with additional field inputs from Zandile Wanda, Chris Chetty, and Jenna Kamps through the Global Health Technical Assistance Project. USAID/SOUTH AFRICA PEPFAR TREATMENT PROGRAM FINAL EVALUATION DISCLAIMER The authors‘ views expressed in this publication do not necessarily reflect the views of the United States Agency for International Development or the United States Government. This document (Report No. 11-01-530) is available in printed or online versions. Online documents can be located in the GH Tech website library at http://resources.ghtechproject.net. Documents are also made available through the Development Experience Clearing House (http://dec.usaid.giv).Additional information can be obtained from: The Global Health Technical Assistance Project 1250 Eye St., NW, Suite 1100 Washington, DC20005 Tel: (202) 521-1900 Fax: (202) 521-1901 info@ghtechproject.com This document was submitted by The QED Group, LLC, with CAMRIS International and Social & Scientific Systems, Inc., to the United States Agency for International Development under USAID Contract No. GHS-I-00-05-00005-00 USAID/SOUTH AFRICA PEPFAR TREATMENT PROGRAM: FINAL EVALUATION i ACKNOWLEDGMENTS The PEPFAR SA program evaluation team wishes to express its thanks to the many individuals and organizations that have facilitated its work. Dr. Yogan Pillay of the National Department of Health was instrumental in providing the impetus for the evaluation; his colleagues Dr. Peter Barron and Lillian Disek are also to be thanked for their contributions during the early planning phase. Members of staff at USAID and CDC have provided many technical inputs and overall support to make the evaluation happen. The team would like to thank Roxana Rogers, Dr. Win Brown, Dr. Melinda Wilson, and most particularly Charles Mandivenyi of USAID and Dr. Thurma Goldman,Dr. Jeff Klausner, and Heidi O‘Bra of CDC for their assistance. In addition, Christy Mulinder of USAID accompanied the Eastern and Western Cape teams and is to be thanked for her contributions. The team was fortunate to hold interviews with three PEPFAR Provincial Liaison Officers: Jessica Rebert in the Western Cape, Coceka Nogoduka in the Eastern Cape, and Chalone Savant in KwaZulu Natal. The team was able to hold a large number of key informant interviews, make upwards of 30 site (health facility) visits, and lead four ―hub meetings‖ in Gauteng, Eastern and Western Cape, and KwaZulu Natal in which PEPFAR implementing partners participated. We are grateful to the large number of people from the Department of Health at the provincial and district levels, PEPFAR partner organizations, and health facilities who made time in their busy schedules to meet us, often at short notice. Annex H provides details on those individuals with whom the team met. Khulisa Consultants took on the task of assembling a team of national consultants at extremely short notice. Two Khulisa staff members and evaluation team members also took the lead on making the field visits and hub meetings happen, in addition to providing their own technical inputs: Mary Pat Selvaggio and Elna Hirschfeld, who are to be thanked most particularly for their logistical work. Zanele Nkabinde and Marge du Plessis from Khulisa are to be thanked for making flight and accommodation arrangements. The evaluation team was much assisted in its second and last weeks of work by the hospitality and administrative support offered by ESI. This was much appreciated, and the team would like to thank Derek Kunaka, Robin Martin, Hodene Goliath, and others. Tabana Mailula and Brennan Walsh of ESI created the five maps used in the report. Finally, the team would like to express its sincere thanks to the GH Tech team, particularly to Julie Klement, Allison Korman, Carrie Keefe, and Michelle Ferng, for their inputs and support. ii USAID/SOUTH AFRICA PEPFAR TREATMENT PROGRAM: FINAL EVALUATION USAID/SOUTH AFRICA PEPFAR TREATMENT PROGRAM: FINAL EVALUATION iii CONTENTS ACRONYMS ............................................................................................................ v EXECUTIVE SUMMARY ..................................................................................... ix Introduction..................................................................................................................................ix Art Technical Areas: Key Recommendations ........................................................................ x Findings For the Over-Arching Issues ................................................................................... xii I. INTRODUCTION ............................................................................................ 1 II. BACKGROUND ............................................................................................... 3 III. EVALUATION APPROACHES AND METHODS ....................................... 5 The Evaluation Sow and Its Clarification ................................................................................ 5 Sow Limitations ............................................................................................................................ 5 The Evaluation Team ................................................................................................................... 6 The Evaluation Approach ........................................................................................................... 7 The Evaluation Methodology ..................................................................................................... 7 The Evaluation Tools ................................................................................................................... 7 The Evaluation Schedule and Work Plan ................................................................................ 8 IV. EVALUATION FINDINGS .............................................................................. 9 Adult and Pediatric Art ............................................................................................................... 9 Key Recommendations .............................................................................................................12 PMTCT .........................................................................................................................................13 TB/HIV Integration ....................................................................................................................16 Monitoring and Evaluation........................................................................................................18 Four ―Quick Looks‖: Pharmacy, Laboratory, Infrastructure, and Infection Control..21 V. DISCUSSION OF OVER-ARCHING ISSUES ............................................. 25 Introduction.................................................................................................................................25 The 10 Sow Questions .............................................................................................................25 Sow Forward-looking Points ...................................................................................................35 The Two CDC Points ...............................................................................................................37 NDOH Cross-cutting Issues ...................................................................................................38 The PEPFAR Transition to TA ................................................................................................42 The PEPFAR Alignment: Lessons Learned............................................................................43 iv USAID/SOUTH AFRICA PEPFAR TREATMENT PROGRAM: FINAL EVALUATION ANNEXES ANNEX A. SCOPE OF WORK .......................................................................... 47 ANNEX B. CLARIFICATIONS TO THE SCOPE OF WORK ........................ 55 ANNEX C. PEPFAR SA ART PROGRAM EVALUATION JUNE-AUGUST 2011 ...................................................................... 65 ANNEX D. EVALUATION METHODOLOGY AND TOOLS ....................... 67 ANNEX E. EVALUATION SITE VISIT SCHEDULE AND DETAILED PROVINCIAL MAPS (GP, KZN, EASTERN AND WESTERN CAPE)................................................................................................ 73 ANNEX F. ADDITIONAL DISCUSSION POINTS, AS REQUESTED BY USAID AND CDC IN AUGUST 2011 ........................................... 77 ANNEX G. REFERENCES .................................................................................. 85 ANNEX H. THE COMPREHENSIVE PEPFAR SA ART PROGRAM EVALUATION SCHEDULE (JUNE-JULY 2011) .......................... 91 FIGURES Figure 1. Sites Visited During PEPFAR SA ART Program Evaluation ........... 8 Figure 2. ART Program Sites Visited ................................................................. 26 Figure 3. Cohort Losses and Retention Proportions ....................................... 30 Figure 4. Annual Rate of Cohort Losses ........................................................... 31 Figure E-1. Gauteng ............................................................................................. 74 Figure E-2. KwaZulu-Natal .................................................................................. 75 Figure E.3. Eastern Cape ..................................................................................... 75 Figure E-4. Western Cape ................................................................................... 76 USAID/SOUTH AFRICA PEPFAR TREATMENT PROGRAM: FINAL EVALUATION v ACRONYMS ACR Adult Clinical Record ACTS Advise, consent, test, support AHF AIDS Healthcare Foundation AIDS Acquired immune deficiency syndrome ANC Antenatal care AZT Azidothymidine ARK Absolute Return to Kids (now Kheth‘Impilo) ART Antiretroviral therapy ARV Antiretroviral CAPRISA Centre for the AIDS Programme of Research in South Africa CBO Community-based organization CCM Chronic care management CDC Centers for Disease Control and Prevention CHAI Clinton Health Access Initiative CHC Community health center CHF Community healthcare facilitator CHW Community health worker CD4 Cluster of differentiation 4 DDG Deputy Director General DHIS: District Health Information System DOH Department of Health DOT Directly observed therapy (also DOTS: directly observed therapy short-course) DQA Data quality assurance EFV Efavirenz EGPAF Elizabeth Glaser Pediatric Healthcare Foundation EID Early infant diagnosis ENA Enrolled nursing assistant EPWP Expanded Public Works Programme FBO Faith-based organization FGD Focus group discussion FIO Facility information officer FP Family planning GHI Global Health Initiative vi USAID/SOUTH AFRICA PEPFAR TREATMENT PROGRAM: FINAL EVALUATION GP General practitioner HAST HIV/AIDS, STIs, and TB HCT HIV counseling and testing HIV Human immunodeficiency syndrome HR Human resources HRH Human resources for health HSS Health systems strengthening iDART Intelligent dispensing of ART IEC Information, education, and communication IMCI Integrated management of childhood illnesses INH Isoniazid IPT Isoniazid preventive therapy IYDSA Institute for Youth Development in South Africa KII Key informant interview KIP Kheth‘Impilo Information System KZN KwaZulu Natal LTFU Lost to follow up M&E Monitoring and evaluation M2M Mother to mother MARP Most-at-risk population MCC Medicines Control Council MCH Maternal and child health MDR/TB Multidrug-resistant tuberculosis MDT Multidisciplinary teams MMC Male medical circumcision MSM Men who have sex with men MTCT Mother-to-child transmission M/XDR Multidrug and extensively drug-resistant TB NASA National AIDS Spending Assessment NDOH National Department of Health NGO Non-governmental organization NHC National Health Council NHLS National Health Laboratory Service NIMART Nurse-initiated-and-managed antiretroviral therapy NSP HIV, AIDS and STI National Strategic Plan 2007-2011 USAID/SOUTH AFRICA PEPFAR TREATMENT PROGRAM: FINAL EVALUATION vii NTP National TB Programme NVP Nevirapine OGAC Office of the Global AIDS Coordinator OPD Out-patient department OVC Orphans and vulnerable children PA Patient advocate PCR Polymerase chain reaction PDS Patient Data System PEPFAR President‘s Emergency Plan for AIDS Relief PHC Primary health care center PICT Provider-initiated counseling and testing PMTCT Prevention of mother-to-child transmission PPTCT Prevention of parent-to-child transmission PN Professional nurse PPLO PEPFAR Provincial Liaison Officer QA Quality assurance QI Quality improvement RDQA Routine data quality assurance RHRU Reproductive Health Research Unit SA South Africa SAG South African Government SASI South Africa Strategic Information SI Strategic information SMS Short message service SOP Standard operating procedure SOW Scope of work SRH Sexual and reproductive health STI Sexually transmitted infection TA Technical assistance TasP Treatment (ART) as prevention TB Tuberculosis TBDU Tuberculosis discharge unit TE Therapy edge TFR Tenofovir TO Transfer out viii USAID/SOUTH AFRICA PEPFAR TREATMENT PROGRAM: FINAL EVALUATION UGM Umbrella grants management USAID United States Agency for International Development VCT Voluntary counseling and testing VFM Value for money WHO World Health Organization USAID/SOUTH AFRICA PEPFAR TREATMENT PROGRAM: FINAL EVALUATION ix EXECUTIVE SUMMARY INTRODUCTION A performance evaluation of the U.S. Government‘s PEPFAR anti-retroviral treatment (ART) program in South Africa was undertaken between June and September 2011, with in-country work conducted in June and July. A total of six consultants (two international and four local) constituted the core evaluation team, with short-term inputs from an additional four local consultants. The evaluation team had expertise in clinical aspects of HIV, such as ART, prevention of mother-to-child transmission (PMTCT), tuberculosis (TB) and TB/HIV integration, and infection control; in addition, the team was equipped to evaluate the program from the perspective of monitoring and evaluation (M&E), health system strengthening, and community/demand-side aspects of HIV. This report is meant to serve as the basis from which the U.S. Government will build on past efforts while creating new avenues and directions for support through the South African Government (SAG). Decentralization of provision of ART to the primary health care level and management to the district level represent major shifts in SAG responses to the epidemic, as does the increased focus on integrated management of HIV and TB. Changes to provision of ART have coincided with the National Department of Health (NDOH) primary health care (PHC) re-engineering plan, which requires strengthening of PHC and the district health system. It is in this context of change that the PEPFAR ART program is completing an alignment of its implementing partners at district and sub-district levels. A transition is in process in which PEPFAR implementing partners are shifting from a previous focus on direct ART service provision to prioritization of technical assistance (TA) inputs. This evaluation addresses the following technical areas: adult and pediatric ART; PMTCT; TB/ HIV integration; M&E; and the ―four quick looks‖ of pharmacy, laboratory services, infrastructure, and infection control. The evaluation team was also requested to address over-arching issues. These were: Ten questions set out in the scope of work (SOW), which chiefly address the historical perspective to date of the PEPFAR ART program and lessons learned Three SOW forward-looking points Two CDC points Three NDOH points In August USAID and CDC provided comments on the draft report, which was submitted on July 11, 2011. Attention to an additional six areas was requested: gender, test-and-treat, unique identifier, SAG infrastructure, TB services (i.e., separate from TB/HIV integration) and the human resources for health transition. Discussion on these areas is not included in the Executive Summary, as their consideration did not form part of the evaluation. Annex F provides further information and recommendations. USAID and CDC also asked for identification of PEPFAR ―game changers‖ with regard to ART. Section III in the main body of the report and Annexes C, D, E, and H provide detailed discussion of the evaluation methodology, tools (key informant interview and focus group discussion guides and a clinical assessment questionnaire), the evaluation schedule, maps, and x USAID/SOUTH AFRICA PEPFAR TREATMENT PROGRAM: FINAL EVALUATION details of sites visited. The two PEPFAR ART program evaluation field teams made a total of 36 health facility site visits in four provinces (Gauteng, KwaZulu-Natal, and Eastern and Western Cape) and conducted four ―hub meetings‖ with representatives of PEPFAR implementing partners. Nearly130 key informant interviews (KII), clinical questionnaires, and focus group discussions were held. The evaluation process was challenging for a number of reasons. Section III and Annexes A and B include discussions of SOW limitations. ART TECHNICAL AREAS: KEY RECOMMENDATIONS The recommendations noted below are those considered by the evaluation team to be the most significant and pressing for each of the eight technical areas. Section IV includes a discussion upon which all these recommendations are based, along with further recommendations relevant to each technical area. Key Adult and Pediatric Art Recommendations 1. Supportive supervision and close mentoring of nurse-initiated-and-managed antiretroviral therapy (NIMART) nurses, e.g., in pediatric ART service delivery, are potentially critical roles that PEPFAR partners can continue to play during and after the transition from PEPFAR direct support to TA, harmonized with NDOH‘s transition into PHC ART service delivery. 2. Task-shifting should be effectively supported over the long term, with close attention to retaining key workers and volunteers, such as social workers and patient advocates. 3. Innovative ways for follow up of stable patients while maintaining adherence may reduce the burden of high patient volumes at the PHC level. 4. Quality improvement (QI) starts with a single data point, such as numbers screened for TB at an ART visit, to begin the quality improvement process. There is currently a concerted effort to review and use data captured from clinics at the district and sub-district district levels; the DOH needs to institutionalize and effectively support such activities at a facility level. One possible approach is the QI model applied by Reproductive Health Research Unit (RHRU) and the Hillbrow community health center (CHC). Key PMTCT Recommendations 1. Integration of family planning (FP), reproductive health, antenatal care (ANC), PMTCT, and integrated management of childhood illnesses (IMCI) needs to be strengthened at the primary health care level to improve maternal and child health. 2. Additional targeted support should be provided to midwives and PMTCT nurses in terms of training, mentoring, and quality improvement/assurance. 3. Appropriate strengthening of community engagement in PMTCT needs to take place, e.g., through community health worker involvement. 4. The issue of male involvement in reproductive health issues, PMTCT, and pediatric ART should be addressed. Key TB/HIV Integration Recommendations 1. Patients should be down-referred only when TB-stabilized; such links should be strengthened at the district and sub-district levels. 2. At PHC level, clear guidelines and well-trained staff are needed to refer up complex cases. USAID/SOUTH AFRICA PEPFAR TREATMENT PROGRAM: FINAL EVALUATION xi 3. There is a need for a strong health facility-community directly observed therapy short￾course (DOTS) link, such as takes place through the Patient Advocates‘ model, as primarily developed by Kheth‘Impilo. Key Monitoring and Evaluation Recommendations 1. SAG and PEPFAR need to harmonize all indicators, definitions, and standard operating procedures (SOPs) for the ART program; performance indicators need to be clearly defined, monitored, and evaluated. 2. PEPFAR partners‘ transition to the provision of TA requires creation of an M&E plan that includes process and outcome indicators and milestones of progress. TA cannot be measured entirely or even primarily through quantitative methods. 3. Data management and use of data should be improved to support evidence-based planning and reporting; this includes making the data warehouse better fit for their purpose, more user-friendly, and available to all relevant partners. 4. The implementation of the 3-Tier Strategy can be enhanced by finalization and dissemination of the implementation plan and by expanded training. Key Recommendations for the “Four Quick Looks” Key Pharmacy Recommendations 1. Integration of patient tracking with pharmacy systems will lead to better patient management and reporting. 2. The SAG should provide legislative and policy clarity on whether pharmacists can accept nurse-signed prescriptions. 3. The SAG needs to enhance pharmacy space to allow for larger bulk orders; it also needs to take steps to minimize the number of patient visits for stable patients and increase clinic efficiency. Key Laboratory Service Recommendations 1. Further studies should be conducted on the cost-effectiveness of decentralized labs in the short and longer term. 2. The use of the National Health Laboratory Service (NHLS) should be expanded. The service collects and analyzes a wealth of data that could be explored for monitoring, planning, and improving ART services, cluster of differentiation 4 (CD4), and viral load could be tracked over time by province, district, sub-district, and/or facility. 3. The development of a national unique identifier system, as used in the Western Cape, would further enhance data specificity, as patients accessing ARVs for the first time could be differentiated from ―transfers out‖ who may be moving between programs. Further discussion of this issue is found in SOW Question 6 and in Annex F. Key Infrastructure Recommendations 1. PEPFAR‘s transition to TA must clarify where responsibility for infrastructure development and maintenance will reside. 2. PEPFAR partners should work with SAG to ensure continued and proper use of buildings, equipment, and supplies. 3. The refurbishment of health facilities should form part of the SAG‘s Expanded Public Works Programme. xii USAID/SOUTH AFRICA PEPFAR TREATMENT PROGRAM: FINAL EVALUATION Key Infection Control Recommendations 1. Control the spread of TB through information, education, and communication (IEC) and provide training on cough etiquette and respiratory hygiene. 2. Minimize the time TB patients spend at health facilities, e.g., through fast tracking. 3. Make effective and enhanced use of respiratory protective equipment and cough control in high-risk situations. 4. Consider opportunities for PEPFAR to become more process-oriented, address evidence￾based approaches, and optimize coherence with GHI principles. FINDINGS FOR THE OVER-ARCHING ISSUES Section V provides a detailed discussion of all issues overviewed below. The 10 Scope of Work Questions SOW Question 1: Did the program help to achieve reduction of the estimated treatment gap? The answer is an unequivocal yes. SOW Question 2: Did the program support the SAG in developing recognized standard public health practice for ART? Yes, because PEPFAR implementing partners have sat on all DOH HIV Guideline committees and provided input on public health practices, e.g., adherence clubs, tracking in retention and care, patient literacy, cohort follow up, and so forth. PEPFAR has assisted in mobilizing the public health community to become fully functional in its response to HIV. SOW Question 3: Did the program play a lead role in reducing costs of delivering quality ART services? PEPFAR did not start as a cost-efficient model; this continues to have implications. One frequent question asked by partners and DOH counterparts is the following: How does one evaluate economies of scale when the cost data are obscured by PEPFAR? Regardless of such issues, PEPFAR has influenced the reduction of costs in truly significant respects. SOW Question 4: Did the program capacitate cadres of health workers to ensure a sustainable program? Through the activities of its implementing partners, PEPFAR has supported considerable training inputs: e.g., NIMART, assistant pharmacist and post-basic pharmacy assistants, and strategic information/M&E training. CDC has been instrumental in supporting the development the program for clinical associates, a key cadre facing human resources for health (HRH) constraints in South Africa. PEPFAR‘s transition from direct clinical support to TA is having significant impact on national, provincial, and district HRH structures. SOW Question 5: Did the program enhance the SAG’s ability to monitor the progress of its ART program? A key stated PEPFAR objective is to work with the SAG and fund initiatives that support the government in enhancing monitoring of its ART program. However, the emergency phase of PEPFAR was characterized by a number of challenges that adversely affected PEPFAR‘s realization of that objective. Separate reporting systems to provincial DOH and PEPFAR were developed, with parallel systems, lack of data coherence, limited capacity to undertake joint (e.g., district-wide) monitoring and evaluation, and limited attention to the development of evidence￾based planning and M&E. USAID/SOUTH AFRICA PEPFAR TREATMENT PROGRAM: FINAL EVALUATION xiii SOW Question 6: Did the program increase overall retention rates and decrease mortality rates? A quantitative answer with regard to retention is difficult in the context of this performance evaluation; however, it can be stated that overall support to retention interventions has increased. Increased overall coverage has been associated with marked increases in the transfer out from services, but this has been poorly tracked. Secular trends indicate increasing loss to follow up. The system to track lost to follow up (LTFU) and adherence is fallible and is minimally reported; this has had a potentially critical impact on the overall quality of care in terms of ART service delivery. Because TB is a major cause of ART program deaths, efforts to integrate HIV and TB care represent a positive strategy. SOW Question 7: Did the program take a population-based approach that emphasizes coverage and reach of ART services? The program was unable to initiate such approaches in 2004 due to the context of the times. An emergency intervention, as was PEPFAR in Phase I, could not have been expected to use population-based models. A population-based approach requires baseline data and close planning at the national and lower levels, with attention to geographical, epidemiological, societal, gender, and other key factors. Such inputs were not available to PEPFAR partners as they initiated implementation. SOW Question 8: Did the program accurately account for its progress with respect to numbers reached? Delivering carefully measured results is one of the seven guiding principles of the South African PEPFAR program. As such, PEPFAR requires all its ART implementing partners to report, on a quarterly basis, on a limited set of program monitoring indicators that are used to track the progress of ART program activities. The definition of what constitutes indirect and direct PEPFAR support and the attribution of numbers to each is imprecise. Partners have previously been pushed by the imperatives of funding that was linked to head count-based performance. Thus PEPFAR, and by extension the NDOH, have had major problems in obtaining an accurate overview of total numbers on ART due to issues related to double counting and the lack of a clear validation process at various reporting levels. In addition, acquiring or establishing the correct denominators to calculate rates for some indicators has not been easy. Moreover, upwards of 40 patient management systems linked to PEPFAR were developed by partners, leading to incoherence of reported data. SOW Question 9: Did the program design and implement its activities with transparency, appropriately sharing plans and progress with counterparts at all levels? There appears to have been ―selective‖ internal and external transparency from the outset of PEPFAR‘s 2004 ART program in South Africa. No criteria existed in PEPFAR I regarding partners‘ obligations to plan for transparency, mutual accountability, and joint stewardship. Lack of transparency continues to be a difficult issue. A frequent remark from SAG partners is that there has always been, and continues to be, a significant lack of transparency on PEPFAR‘s part with regard to the overall budget and individual partners‘ annual budget allocations. Another much-cited example of deep current concern is the perceived insufficiency in planning and discussion on the almost completed PEPFAR district alignment process and the ongoing transition to TA. xiv USAID/SOUTH AFRICA PEPFAR TREATMENT PROGRAM: FINAL EVALUATION SOW Question 10: Did the program invest wisely in sustainable solutions that support South Africa in its current and future responsibilities for ART provision? Due to the political environment at the beginning of the PEPFAR ART program in April 2004, non-governmental organizations became key implementing partners and interventions were developed with little if any engagement with government structures. That emergency approach failed to facilitate prioritization of sustainable approaches and solutions: Each PEPFAR partner worked largely in isolation at that time, with at times considerable duplication and competition and without the best matching of inputs with needs. PEPFAR now increasingly supports moves toward sustainability, notably via the PEPFAR alignment model and the transition to TA, support to development of SAG guidelines, and clinical and other studies (e.g., the costing study). The Three Scope of Work Forward-looking Points 1. Outstanding issues in the policy arena related to ART The following outstanding policy issues were identified in the course of the evaluation: There is a need for continued advocacy and policy direction for a unique identifying patient number to effectively track and manage LTFU/transfer out at the national level, with provincial buy-in (see also Annex F). More effective linkages are required at the policy and practice levels between health systems and community systems strengthening. There should be thorough attention paid at the policy level to the supply and demand-side implications of test-and-treat and treatment as prevention (see Annex F). Quality of care, from both the supply and demand-side perspectives, requires detailed policy direction, further development of instruments such as supportive supervision, and further attention to the health rights of patients. More effective policy guidance is required on hitherto somewhat neglected aspects of ART provision, e.g., for adolescents and older people (both pre-ART and ART), overall pre-ART support and maintenance, and support to care givers. Many key NDOH policy documents fail to adequately address gender issues in the context of ART provision and more widely in the shift to a PHC focus; this is a significant policy gap. Annex F provides a more detailed consideration of gender aspects. 2. Important Gaps that Are Not Being Addressed by USAID/CDC, NDOH, or Other Donors Greater, more focused attention to pre-initiation and pre-ART patients is required. Dedicated support for adolescents, pre-ART, and treatment is essential. Dedicated support for older/old HIV+ and care givers is also essential. An absence of gender mainstreaming, both internal and external, represents a significant gap. Inter and multisectoral initiatives require further attention and are critical in the move toward effective chronic care management. NHLS data are not optimally utilized at the provincial and district level to evaluate population impact over time and inform evidence-based planning. There is lack of coherence between District Health Information System (DHIS) and other population-based datasets and national birth and death registries. Despite site-specific best practices, there is currently insufficient use of pharmacy data systems for program management and monitoring. USAID/SOUTH AFRICA PEPFAR TREATMENT PROGRAM: FINAL EVALUATION xv 3. Identify and Document Best Practices, Lessons Learned, and Recommendations to Inform Follow-on Activities Focusing on Sustainability There is now a considerable cultural shift to more health systems strengthening (HSS) and chronic care management-focused approaches; this has resulted in the need for change management processes within the U.S. Government and in partners. Aspects to consider with regard to sustainability include: how might this have been instituted into implementing partners‘ ART service delivery models and what was the time frame on this? There should be a focus on the fact that the SAG is reforming the health system to make it a PHC model. How can sustainability-related lessons learned and best practices from PEPFAR inputs help to achieve this? The transition to TA continues to require transparent planning and definition, given the continued confusion as to TA parameters, the implications for partner inputs, and the potential impact on overall ART service delivery. ―Nerve centre meeting‖ model processes should be closely supported, documented, and institutionalized as a key HSS component and a central plank of the alignment to district support. This is especially pertinent in light of the NDOH‘s transition to PHC-level ART service delivery. The Two CDC Points CDC Point 1: What is the partners’, government’s, and civil society’s understanding of technical versus direct assistance, the benefits and risks/harms/costs of both as PEPFAR transitions to the former? Most stakeholders accept that PEPFAR cannot and should not play a role that facilitates continued dependency. Therefore, there is widespread recognition that PEPFAR‘s transition to TA represents a necessary step toward greater government ownership and stewardship of the ART program. However, there remains considerable confusion and concern among many stakeholders with regard to the fit between PEPFAR‘s transition and the SAG‘s transition of ART to the PHC level/NIMART model. For instance, and crucially: who will fill the inevitable clinical gaps in down referral/NIMART service delivery? The on-the-ground reality is that many districts will still need clinical inputs, e.g., in management of pediatric ART and supportive supervision and mentoring of NIMART nurses. A further issue is that because provinces and districts vary so considerably, the balance between continued direct support versus transition to TA may also differ. There are grave concerns that pre-ART and ART patients will suffer if the transition is too hasty. Another area of concern is the lack of clarity as to the future role of non-clinical staff in the provision of TA. There are genuine fears that the complex, interwoven nature of ART service delivery that has developed since 2004 with PEPFAR‘s support may now be in danger of being fragmented and weakened. Transition Recommendations 1. Essential areas should be considered for continued direct support by implementing partners, e.g. pediatric ART, optimal TB/HIV integration, targeted clinical research, and close mentoring of NIMART nurses. 2. Change-management approaches are essential in the transition process and are still timely, despite ongoing rollout. 3. A key area for support during and beyond transition is health manager training at the district and sub-district levels. 4. Advocacy is needed for competence-based recruitment and performance-based management. 5. Advocacy is also needed for required service contracts. xvi USAID/SOUTH AFRICA PEPFAR TREATMENT PROGRAM: FINAL EVALUATION 6. It may be advisable to consider the re-introduction/expansion of the additional community service years currently applied to medical students; this could be extended to pharmacists and others to address shortages, especially in rural health facilities. CDC Point 2: Consider the significance of PEPFAR support into the medium-term future in the overall context of the HIV/TB response in South Africa This point refers to the fact that PEPFAR provides 10% of overall inputs – so how relevant, necessary, and important is that support, and how can it achieve maximum impact? The current challenge is how to implement cost efficiencies and achieve maximum impact, especially in line with the 5% year-on-year reduction in the PEPFAR grant from 2012 on and a further significant reduction starting in 2015. With PEPFAR‘s support remaining pivotal, major opportunities exist for effectively supporting the transition to the HSS approach, chronic care management (CCM) of HIV and more focused PHC integration. In the context of PEPFAR‘s alignment and transition processes, as well as the reduction in financial inputs, it is imperative to manage change effectively, transparently, and as guided by the DOH. The Three NDOH Points 1. The Extent to which the PEPFAR ART Program Has Supported Increased Coverage A truly major achievement of PEPFAR implementing partners is that their staffing capacity has allowed for greater coverage in ART provision than would otherwise have been possible. Another, more recent achievement has been support for the creation and maintenance of roving teams, in the context of the move toward district-level provision of ART. While these teams may represent a stop-gap measure in terms of individual visits and health facilities, they are providing services where none would otherwise be available. In addition, roving X-ray machines facilitate enhanced coverage. PEPFAR support to PMTCT has undoubtedly enhanced coverage. Continuing challenges include human resources for health constraints. There are also issues related to PEPFAR‘s alignment with government-provided services, in that as ART patients are transferred from implementing partners‘ services to SAG, there may be absorptive constraints that result in the loss of some patients (this concern was expressed by the South African Catholic Bishops‘ Conference). This would compromise coverage and usher in fundamental questions of equity in service provision and continued support. Future attention to increased coverage would include the following: Provide more target group- and location-specific interventions (e.g., education and other prevention, HIV care) at work places and other sites. Further develop and implement programs that reach MARPs, including migrant labor, illegal immigrants, orphans and vulnerable children (OVC), and deep rural populations. Consider a greater role for private general practitioners (GPs), possibly linked to the eventual National Health Insurance scheme rollout. Private GPs often cater to professional groups, e.g., health workers, teachers, etc., who tend to avoid local clinics. 2. The Extent to which the PEPFAR ART Program Has Supported Efficiency One major achievement is that PEPFAR implementing partners have significantly contributed to the efficiency component of appropriately matching ART service provision with patients‘ needs. Patients are now being given ARVs for upwards of three months. This approach speaks to the efficiency of service and scale in a resource-constrained environment as well as to the responsiveness of care. USAID/SOUTH AFRICA PEPFAR TREATMENT PROGRAM: FINAL EVALUATION xvii The SAG is seeking efficiencies of scale through its district and PHC models. PEPFAR alignment and transition processes require a well-managed transition and change management focus, as well as an entirely realistic evaluation of HRH and other constraints. The danger is that otherwise district and sub-district health systems will be unable to cope and service delivery efficiencies will be compromised. Future attention to efficiency issues should include: Instituting a scheduled, properly managed transition to a TA focus (doing this too quickly and without proper planning may result in patient loss). Key within the transition will be HSS linkages and strong down and up referral systems, all of which support strengthened efficiencies of service delivery. 3. The Extent to which the PEPFAR ART Program Has Supported Quality of Services Clinical and supply side components of quality of care have been closely addressed under the PEPFAR ART program, as have viral load suppression and emphasis on program retention. There has been attention to the quality of training provision, while quality improvement initiatives were not an initial PEPFAR priority. Demand-side quality criteria (i.e., patient/client and community aspects) have not always and universally been closely addressed within the PEPFAR ART program, despite much focus from a number of partners, including Anova and Kheth‘Impilo. A major quality issue is: how most equitably and effectively to balance quantity and quality of service delivery, from both the supply and demand sides. Attention to such matters is essential as decentralization proceeds and as PEPFAR implementing partners make their transition to TA. The need will be to identify an approach that balances expanded coverage with increased delivery quality. Future attention to quality issues should include: Greater coherence of PEPFAR quality criteria in terms of national requirements. Closer attention to issues of quality in the context of clinical care provision: overall PEPFAR attention should be increased with regard to key indicators of quality, e.g., tracking, adherence, action on LTFU, support to pre-ART, and so forth. Enhanced responsiveness to demand-side perspectives: this is a core criterion for quality of care and one that has received insufficient overall attention. Future USG Support to Art Programming in South Africa The following points represent potential areas of future focus, in addition to those discussed above under recommendations and over-arching issues: ART service delivery components require additional focus on integration, e.g., TB and HIV and also PMTCT and ANC, FP, IMCI, and sexual and reproductive health (SRH) services. There needs to be increased focus on health systems strengthening within the overall context of ART service delivery, e.g., for referral, HRH strategy, and continued attention to optimization of data and patient management systems. The transition to implementing partners‘ prioritizing TA support must allow for continued direct service provision inputs in key ART areas – for example, the mentoring of NIMART nurses. There needs to be clear and agreed definition of TA and space for continued, close mentoring by PEPFAR implementing partners. xviii USAID/SOUTH AFRICA PEPFAR TREATMENT PROGRAM: FINAL EVALUATION One key area for TA support is enhanced assistance to district-level health systems, e.g., data management, meetings, and HIV/AIDS, STIs, and TB (HAST) managers Task shifting requires strategic and realistic planning and support. PEPFAR implementing partners have developed programs in which non-medical staff members and volunteers (e.g., social workers and patient advocates) are integral to effective, quality ART service delivery. It is essential that attention be given to the retention of such cadres in future ART interventions. There are serious issues of data management coherence and harmonization that need to be jointly addressed by PEPFAR and the SAG/NDOH, including further strengthening of an evidence-based approach to planning, M&E, and reporting, ART programming now requires additional attention to demand-side (patient and community) interventions that further strengthen quality of service provision, access to and uptake of those services, and patient retention. The changing nature of the South African HIV epidemic necessitates closer consideration of chronic care management models of ART support. Within this context, PEPFAR should further support interventions that address the continuum of prevention/care/support/treatment and demand-side engagement. Proper, dedicated attention to gender issues in the context of program management, service delivery, and patient/community engagement is now more than timely. Enhanced commitment to transparency in funding, budget lines, and other key areas is important in terms of optimal partnership between the SAG and PEPFAR. USAID/SOUTH AFRICA PEPFAR TREATMENT PROGRAM: FINAL EVALUATION 1 I. INTRODUCTION The President‘s Emergency Plan for AIDS Relief (PEPFAR) was funded by the U.S. Government in 2004 as an emergency measure to provide HIV prevention, care, and treatment for 15 countries with high burdens of HIV disease. The largest disease-specific program funded by a single country in history, PEPFAR has greatly influenced the global landscape in terms of care and treatment of HIV-infected patients. As the PEPFAR-supported country with the highest burden of HIV disease (in both prevalence and absolute numbers), South Africa has benefited from an influx of funds, programs, and expertise since the early years of PEPFAR. From its inception, PEPFAR‘s mandate has been to support national governments in implementing country public health HIV programs. In South Africa this coincided with the antiretroviral rollout plan of the South African National Department of Health (NDOH). The South African Government‘s decision to provide antiretroviral therapy (ART) drugs and laboratory services though the public sector enabled the PEPFAR program to expand its reach in supporting ART programs. The U.S. Congress reauthorized the PEPFAR program in 2008 as the Lantos-Hyde Act. With this reauthorization, the program‘s direction shifted dramatically. Not only was the scope expanded to include tuberculosis (TB) and malaria but the focus shifted to program sustainability within countries, strengthening health care systems, training health care workers, working with other funders and global partners to sustain programs, and decreasing drug costs. The U.S. Department of State also developed the Global Health Initiative (GHI) in 2008 with seven key principles, including a focus on country-led planning and health systems strengthening. In December 2010, the governments of South Africa and the United States signed a non-binding Partnership Framework to improve effectiveness, efficiency, and sustainability of the South African HIV/AIDS and TB response. This cooperative strategic plan aims to improve the coordination between the South African Government and PEPFAR, and enhance planning and implementation of HIV and TB programs at the national, provincial, district, and local levels. It also emphasizes PEPFAR‘s role in providing technical assistance while decreasing direct support for services. 2 USAID/SOUTH AFRICA PEPFAR TREATMENT PROGRAM: FINAL EVALUATION USAID/SOUTH AFRICA PEPFAR TREATMENT PROGRAM: FINAL EVALUATION 3 II. BACKGROUND The estimated adult South African HIV prevalence for adults aged 15-49 is 17.8%, with upwards of 5.6 million adults and children living with HIV at the end of 2009.This represents approximately one-sixth of the estimated 33 million global HIV cases (UNAIDS: Report on the Global AIDS Epidemic 2010). The HIV, AIDS and STI National Strategic Plan (NSP) for 2007-2011 observes that women bear the brunt of the HIV and AIDS epidemic in South Africa; this phenomenon is especially marked in the age groups 20-24 and 25-29, where HIV prevalence rates are 23.9 and 33.3% for women and 6.0 and 12.1 % for men, respectively. The 2010 UNAIDS Global Report estimates that out of the total 5. 3 million adults aged 15 years and over living with HIV, 3.3 million are female. Gender-based and sexual violence in the context of the epidemic represents an area of grave and continuing concern. There is a clear correlation between poverty and high HIV prevalence, with people living in urban and rural informal settlements often among the most vulnerable to infection. Other areas requiring particular attention with regard to the HIV epidemic in South Africa include HIV and TB co-infection, prevention of mother-to-child transmission (PMTCT), and focused support to adolescents and most-at-risk populations (MARPs). UNAIDS 2010 estimates that 330,000 boys and girls under the age of 14 are living with HIV in South Africa and that there are 1.9 million children aged 0-17 who are orphaned due to the epidemic. Thus many children are both infected and deeply affected by HIV, with reduced life opportunities. Since 2007 the South African Government (SAG) has dramatically increased its support to public sector HIV/AIDS and tuberculosis programs. Over 900,000 South Africans were receiving antiretroviral therapy (ART) as of November 2010.The National Department of Health (NDOH) estimates that by the end of April 2011, approximately 1.4 million South Africans were receiving ART and approximately 470,000 had initiated therapy in the previous year. The NSP, a multisectoral response to South Africa‘s AIDS epidemic, states that the objective is to reach 80%of all those eligible for treatment, care, and support by 2011. In 2010 the World Health Organization (WHO) recommended that all countries, including resource-poor countries, start treatment at a cluster of differentiation 4 (CD4) count of <350 cells/mm3.The SAG updated ART treatment guidelines in partial alignment with WHO recommendations: thus SAG guidelines continue to state that ART treatment begins at ≤ 200 cells/mm3.It is only certain groups – for example, pregnant women and the TB/HIV co- infected – for whom ART treatment is started at ≤ 350 cells/mm3. A number of recent studies have reported a problem of late stage HIV diagnosis (with CD4 <100) in South Africa; research also indicates many challenges with regard to delays in initiating ART (e.g., Bassett 2009). The SAG started providing ART at public health facilities in 2003. However, until recently ART was not widely available in all health facilities, in part due to a detailed accreditation process. PEPFAR implementing partners have been instrumental in facilitating the process for most health facilities now accredited. The previous complex accreditation tool has been replaced by an assessment tool to gauge ―readiness to serve,‖ which is intended to facilitate scale up of ART service provision at all facilities. In recent years the SAG has developed a number of key HIV and linked policies, with much work undertaken to expand ART services. On December 1, 2009, President J. G. Zuma 4 USAID/SOUTH AFRICA PEPFAR TREATMENT PROGRAM: FINAL EVALUATION emphasized government commitment to fight the HIV and AIDS epidemic. He stated that ―In order to meet the need for testing and treatment, we will work to ensure that all the health institutions in the country are ready to receive and assist patients and not just a few accredited ARV centres. Any citizen should be able to move into any health centre and ask for counseling, testing, and even treatment if needed.‖ Many challenges exist in the provision of ART to eligible South Africans. The SAG has sought to address the daunting shortages at all levels in human resources for health (HRH) and to initiate health systems strengthening approaches that will sustain the potentially greatly increased numbers of people who will initiate ART. One such approach has been to expand and support health worker task shifting to optimize best use of existing HRH. The following measures have been approved: nurses as well as doctors can initiate ART (NIMART); lay counselors, rather than nurses, can carry out HIV tests and can also provide the support to orphans traditionally undertaken by social workers; and pharmacy assistants, rather than pharmacists themselves, can prescribe antiretroviral drugs. Decentralization of provision of ART to the primary health care level and management to the district level represent further major shifts in SAG responses to the epidemic, as does the increased focus on integrated management of HIV and TB. Changes to provision of ART have coincided with the NDOH primary health care center (PHC) re-engineering plan, which requires strengthening of PHC and the district health system. In 2010 new SAG HIV counseling and testing (HCT) guidelines were launched, with an emphasis on provider-initiated HIV counseling and testing. It is anticipated that the 2010 HCT campaign will facilitate early (or earlier) diagnosis. The SAG has also committed to early diagnosis of all HIV-exposed children, with immediate access to treatment for all HIV-infected children eligible for ART.1 1 Section V discusses the historical background of the PEPFAR South Africa ART program and provides additional background information and contextualization. USAID/SOUTH AFRICA PEPFAR TREATMENT PROGRAM: FINAL EVALUATION 5 III. EVALUATION APPROACHES AND METHODS THE EVALUATION SOW AND ITS CLARIFICATION The PEPFAR South Africa ART program evaluation was preceded by considerable discussion of the scope of work (SOW) to achieve mutual understanding, clarification, and agreement on the way forward. This process continued during the first two weeks of the evaluation, with implications for the evaluation‘s overall scope in terms of time for literature review, analysis of raw data generated during field work, and preparation of the out brief and draft report. In addition, a number of technical areas were added to the SOW once the evaluation had started: the so-called ―four quick looks‖ at PEPFAR South Africa ART program support for pharmacies, laboratories, infrastructure, and infection control. Other expansion occurred, e.g., through the requirement by the NDOH that the evaluation address issues of coverage, efficiency and quality in the context of the ART program. All such developments had impact on the evaluation‘s process and outputs. SOW LIMITATIONS As is common practice when conducting evaluations, a section is included in this report on limitations to the SOW. This refers to constraints and unavoidable changes due to circumstances beyond the control of the evaluation team and/or its counterparts. Annex B provides a full discussion of this issue. Pre-planning Due to a number of factors, pre-planning (i.e., before the full evaluation team was under contract) was extremely short and less than ideal. This resulted in unavailability of documents, considerably shortened preparation time before and after evaluation start, and extremely limited time for team planning and other key meetings. The first week in country (June 13-17) was largely taken up with activities that should either have been completed before the start of the assignment or were substantially in hand. The full list of core documents had still not been made available from any of evaluation counterparts by the assignment‘s end. Time Pressure The evaluation was initially scheduled to last six weeks, but was compressed into four. The evaluation team had less than 1.5 weeks (June 23 to July 1, 2011) to conduct the great majority of key informant interviews, site visits, and hub meetings. Such time constraints inevitably had a bearing on the comprehensiveness of the evaluation. National Consultants Khulisa Consultants was not engaged to begin work on the evaluation team until Monday, June 13, 2011. This resulted in inevitable scheduling conflicts and limited availability of consultants to participate in the evaluation‘s in-country activities. This outcome was less than ideal in terms of continuity, joint learning, and optimal inputs to the out brief and report work. Logistic Support A local consultant was retained to provide logistic support a week before the international consultants arrived in South Africa. His work proved unsatisfactory, which led to the contracting of Khulisa Consultants on June 17, 2011, to provide logistic services in addition to technical 6 USAID/SOUTH AFRICA PEPFAR TREATMENT PROGRAM: FINAL EVALUATION inputs. These delays had an inevitable effect on early in-country planning for the evaluation, e.g., the work plan could not be completed until Week 2 of the assignment. They also put considerable pressures on the entire evaluation team, in particular on two of its members who shouldered responsibility for ensuring Gauteng, KwaZulu Natal (KZN), Western and Eastern Cape meetings, and site visits became possible and effective (see discussion below). Considerable amounts of time up to June 24, 2011, had to be dedicated to planning schedules and hub meetings, rather than to having the full team working on the evaluation‘s technical aspects. Incomplete Technical Inputs Due to the logistical challenges, Mary Pat Selvaggio, Director, Health, and Elna Hirschfeld, Associate Director, both of Khulisa Consultants, were unable to participate fully in the technical aspects of the evaluation before June 24, 2011. Mary Pat Selvaggio was only available until that day to work on the evaluation, while Elna Hirschfeld had the opportunity to be part of the team from that date. Professor Robin Wood was only able to participate intermittently in the evaluation due to other schedule commitments. Limitations of the Fieldwork Due to the short lead-time and curtailed time in the field, the evaluation team was unable to achieve the most complete range of interviews and site visits. A major shortfall was the limited number of meetings held with individuals or groups representing district and sub-district structures. In addition, engagement with community members and groups was severely restricted. It was also not possible to hold a meeting with a McKinsey representative; this was unfortunate, given that McKinsey is providing inputs to the SAG transition into PHC and nurse￾managed ART service delivery. Limitations of the Debrief and Draft Report Due to all the above limitations, the evaluation team was only able to undertake the most top line discussion of data collected during the field visits in preparation for the debrief and draft report. The team reconvened from the field on Monday, July 4, and presented the out brief on Thursday, July 7. Detailed analysis and discussion were simply not possible in the time available; this has inevitably had repercussions in terms of the depth of findings as discussed in the debrief and draft report. A further major issue is that a full and detailed literature review was also impossible, due to key documents not having been provided to the evaluation team. This too has had implications in terms of the breadth and depth of the out brief, the draft report, and this final report. THE EVALUATION TEAM The evaluation team was composed of two international consultants: Dr. Lois Eldred (KZN Team Leader) and Janet Gruber von Kerenshazy (Team Leader and Eastern and Western Cape Team Leader) and national South African consultants. The latter were Professor Robin Wood and the following Khulisa employees and consultants: Mary Pat Selvaggio, Elna Hirschfeld, Peter Njaramba, Dr. Ntombi Bandezi, Zandile Wanda, Chris Chetty, and Jenna Kamps. The latter three consultants participated only in the one-week intensive field visit part of the evaluation, held between June 27 and July 2, 2011. It should be noted that due to the extreme time constraints of this evaluation, the only consultants available throughout the evaluation were Lois Eldred and Janet Gruber von Kerenshazy. Thus Professor Wood participated June 14-17, June 22-23, and July 6-8. USAID/SOUTH AFRICA PEPFAR TREATMENT PROGRAM: FINAL EVALUATION 7 THE EVALUATION APPROACH The evaluation was a performance evaluation as per the 2010 USAID Evaluation Guidelines; its objective was to provide a snapshot of the PEPFAR ART program, using primarily qualitative methodology, with some limited quantitative analysis of PEPFAR data. As a performance evaluation, findings and recommendations cannot address the longitudinal and higher-level impact of the program. The evaluation focuses on descriptive and normative questions: what the PEPFAR South Africa ART program has achieved to date, how it is being implemented, how it may be valued, and whether expected and planned results are being achieved. THE EVALUATION METHODOLOGY Key informant interviews (KII) were conducted with a range of representative stakeholders, including members of staff from the NDOH, Centers for Disease Control (CDC), and USAID, PEPFAR partners and their sub-partners, donor partners, SAG (e.g., Treasury) and district staff members, and also at sites with the following respondents, as per agreement with USAID and CDC: the facility Medical Director; the clinician/health worker responsible for ART service delivery; the health worker responsible for PMTCT; the monitoring and evaluation (M&E) officer responsible for reporting using PEPFAR systems; and, where appropriate, health workers responsible for TB/HIV integration or services. Hub meetings: Four meetings were held, one in each of the provinces visited for the purposes of this evaluation. The rationale for these meetings was to optimize time efficiency in conducting interviews. A total of 16 PEPFAR implementing partners participated in the various hub meetings. Focus group discussions (FGD) were conducted where appropriate with PEPFAR partners and subs, e.g., at the hub meetings. In addition, the following data collection methodologies were applied: 1. Document review: USAID, CDC, PEPFAR, partner, and other relevant national and international literature 2. Further quantitative analysis of key PEPFAR data stored on the Data Warehouse System The methodology was designed to enable triangulation of findings from a range of sources while acknowledging the scope of this performance evaluation and its constraints. THE EVALUATION TOOLS A total of eight standard KII questionnaires were developed (see Annex C for further details; copies of all KIIs are available on request). Clinical check lists were developed for adult and pediatric ART, PMTCT, TB/HIV integration, and M&E, while a list of key questions was developed for discussions on subjects related to pharmacies, laboratories, infrastructural development and refurbishment, and infection control. In addition, a number of KIIs were conducted where specific questions were asked, with reference to the position and expertise of the respondent, e.g. with the Clinton Health Access Initiative (CHAI). 8 USAID/SOUTH AFRICA PEPFAR TREATMENT PROGRAM: FINAL EVALUATION THE EVALUATION SCHEDULE AND WORK PLAN The initial two weeks of the evaluation (June 13-22, 2011) were spent on SOW clarification, logistical planning, and arrangements for Gauteng KII and the first hub meeting. Substantive evaluation work began in the middle of the second week. In brief: two teams (eight team members altogether) visited a total of 36 sites over a period of seven days. In Gauteng: 3 sites; in KwaZulu Natal: 18 sites; in Eastern Cape: 10 sites; and in Western Cape: 5 sites were visited. Annexes C and H contain the work plan and comprehensive schedule for the evaluation; Annex E provides a site visit schedule and four detailed provincial maps showing all sites (health facilities) visited. Table 1. Catalogue of Site Visits Summary KIIs FGD KII Sites Gauteng 10 2 19 KZN 1 1 54 WC 5 1 18 EC 1 1 35 Totals 17 5 126 Figure 1. Sites Visited During PEPFAR SA ART Program Evaluation USAID/SOUTH AFRICA PEPFAR TREATMENT PROGRAM: FINAL EVALUATION 9 IV. EVALUATION FINDINGS This section discusses the following components of the PEPFAR South Africa ART program, as required in the SOW and its clarification: 1. Adult and pediatric ART 2. PMTCT 3. TB/HIV integration 4. Monitoring and evaluation 5. ―Four quick looks:‖ pharmacy, lab, infrastructure, and infection control Consideration is given to achievements, gaps, challenges, and, where appropriate, best practices. Three or four key recommendations are made for each area, with further recommendations noted where relevant. ADULT AND PEDIATRIC ART Achievements The level of clinical knowledge among implementing partners met by the evaluation team is consistently high, as is their understanding of the revised NDOH ART guidelines. A review of medical files at partner PHCand at district clinic and hospital indicates that national guidelines are being followed. Even clinicians who may not have fully agreed with an NDOH guideline (e.g.,isoniazid preventive therapy: IPT) realize the importance of consistency and followed national guidelines. Standard operating procedures (SOPs) are available at most PEPFAR￾supported sites. Challenges Supervision of NIMART-trained Nurses Nurses who have attended three or five day NIMART training courses but who have not received supportive supervision and mentoring are frequently reluctant to see patients. A number of nurses feel uncertain of their skills and do not want to see patients without a doctor present. Several nurses interviewed for the evaluation stated they need a consistent and regular doctor mentor with whom there is phone contact and regularly scheduled visits. The need was also felt for regular patient case reviews, even in routine cases. The evidence of the evaluation indicates that doctors who participate in district meetings often feel that the expectation is that supervision will be their responsibility, even though they personally manage full caseloads. The implications of this lack of clarity and support are significant in terms of the NDOH transition to PHC approaches to ART service delivery, and should be addressed as a matter of urgency by all relevant parties. Quality Improvement One finding of the evaluation is that there is great diversity in implementation of quality improvement (QI) at the facility level. PEPFAR has supported a number of larger partners‘ QI development and implementation for a number of years; there was evidence of QI integration at health facilities supported by those partners. However, site visits to district and sub-district facilities produced little evidence of effective use of data to support evidence-based planning or improve the quality of service delivery. Respondents at one site mentioned there had been QI when there was a nurse reviewing charts; such inputs collapsed once she transferred. 10 USAID/SOUTH AFRICA PEPFAR TREATMENT PROGRAM: FINAL EVALUATION When a number of facility managers, doctors, and nurses were asked specifically about quality, there seemed to be little understanding of its value and best use. There often appears to be a disconnect between data capture as a routine (rote) activity and the actual use of data for quality improvement and assurance purposes. This is likely to be related to the volume of data collection required by sites by both government and PEPFAR. This represents a missed opportunity to support the development of an information culture, with evidence-based planning and effective iteration at its core. Many of the data capturers at health facilities have been hired by PEPFAR implementers and are not seen as an integral part of the clinic. In other sites where nurses are responsible for data capture and register entry, they are frequently too overwhelmed by other responsibilities to fully capture, analyze, and use the data. As the majority of quality improvement programs are data-based, there needs to be a concerted effort to utilize data at the facility level. Task Shifting to Nurses The evolution of HRH development with task shifting to nurses for initiation and management of ART is laudable and necessary in the context of the epidemic in South Africa. The same is true for the focus on increased responsibility of professional nurses. However, there is concern that if task shifting stops at the professional nurse level, the degree and type of work will be overwhelming. At the district hospital level, nurses interviewed for evaluation purposes carry both administrative and clinical responsibilities. Clinical tasks are supposed to account for a minority of such nurses‘ time, yet they are frequently called to fill in because of continual nursing shortages. These are the same nurses being trained in NIMART, yet there is no clear plan to shift some of their administrative responsibilities. It is essential that attention be given within the overall re-engineering of PHC to the realities and implications of task shifting and down referral of ART to PHC clinics, in terms of complex case management, volume of stable patients, HRH constraints, and the continued role of non-medical staff, to name a few issues. Use of Lay Counselors and Community Outreach Workers Many PEPFAR partners have hired lay counselors and community outreach workers (e.g., in the latter cadre patient advocates in the Kheth‘Impilo approach) who function well beyond their assigned responsibilities. They serve as health literacy educators, adherence counselors, data capturers, default trackers, and community advocates, filling in many of the gaps created by the rapid scale-up of ART. There is now grave concern that comparable posts do not exist within DOH structures, and that these well-trained persons will be underutilized in the government system, or sacked. At some of the larger sites visited, facility managers, doctors, and nurses mentioned that their work would be literally unsustainable without the inputs of lay counselors and community outreach workers. Pediatrics Doctors at a number of sites expressed considerable concerns regarding ART service delivery to infants and children. Many feel that they do not have the clinical skills to care for children. One general practitioner interviewed for the purposes of the evaluation, who has cared for children and adults through PEPFAR support, is now taking the HIV Diploma Course; this is sponsored though the Colleges of Medicine of South Africa. He feels strongly that general practitioners and other clinicians should not be providing pediatric ART without at least the Diploma Course. In other sites, doctors caring for children consider their clinical capability has been greatly enhanced by phone support from pediatricians. Considering this perceived need for extra support, the relative lack of ability of NIMART-trained nurses to provide pediatric care has been underscored during the course of the evaluation. USAID/SOUTH AFRICA PEPFAR TREATMENT PROGRAM: FINAL EVALUATION 11 This has created a dilemma for PHC facilities working to provide family-based care, as it fragments care between adults cared for by NIMART-trained nurses and their children, because the latter may receive pediatric ART and linked services at a higher-level facility. The issue is: how can pediatric cases be referred down to the PHC level, when their management cannot currently be effectively managed at such facilities? Adolescents Adolescents are another group of concern. As the epidemic matures, perinatally infected children are reaching adolescence and facing adherence challenges, as is seen in the management of many other chronic diseases during this life stage. Adolescents require identification with a peer group, sex education, and dedicated spaces differentiated from adult treatment, with trained staff. Many clinicians are caring for adolescents who are not comfortable in pediatric or adult clinics, or who may feel they no longer fit comfortably into the family-centered care model. Through PEPFAR support, a few sites have created adolescent centers, in an effort to provide separate, private space for health and sexual education, care, and group support. Working with General Practitioners PEPFAR implementing partners can play a critical role in strengthening public-private partnerships for ART, e.g., through facilitating general practitioners (GPs)in providing mentoring and supportive supervision to local PHCs and NIMART nurses to improve the limited capacity that currently exists in district health teams. Best Practices Numerous best practices were seen at clinical facilities visited during the evaluation, largely developed by PEPFAR partners. While these practices are often disseminated within all sites supported by an individual implementing partner, there is little evidence that best practices are known and shared across partners. This lack of effective lesson sharing and focus on the process aspects of the overall PEPFAR ART program represents lost opportunities for making best use of comparative advantage as well as poses the potential for parallel activities and resulting duplication of effort. Tailoring of Frequency of Visits Many sites visited are transitioning to less than monthly patient visits for ART medication refills. Some sites set criteria of two- or three-month visits, based on length on ART medication. Others use more sophisticated patient clinical and behavioral parameters; patients with stable CD4 and viral loads and evidence of adherence are scheduled for medication-only visits every two or three months. If patients on a two- or three-month visit schedule have a change in status, visit frequency is increased. At down-referral sites where newly trained NIMART nurses are practicing, visits are held monthly to allow the nurse time to become comfortable with newly acquired skills. Defaulter Tracking The evaluation found numerous examples of defaulter tracking that tie into existing household or community-based programs. All sites that conduct defaulter tracking attempt to reach patients by phone before employing other strategies. Some PEPFAR partners have contracted with community-based organizations (CBOs) for prevention and other services; they also work with these community workers to track defaulters at the household level. Other partners have attached the responsibility for household defaulter tracking to co-existing research projects at the household level. Yet another route into community tracking involves utilizing TB program contact tracers; this appears particularly effective where TB and HIV services were co-located. The Department of Social Development also employs community social workers for this service at some sites. Programs that conduct community tracing can to an extent determine why 12 USAID/SOUTH AFRICA PEPFAR TREATMENT PROGRAM: FINAL EVALUATION patients are not returning for ART visits. At one site opportunity costs such as taxi fare and food while waiting were the main reasons given by patients for not returning; these issues have been addressed. Patient-flow Systems Assessing and improving patient-flow systems was identified as a key process for preparing facilities to handle a large influx of HIV-infected patients. One site uses repeated audits as a component of a quality improvement process to assess and improve patient flow. Other sites use the medical record system to assist with patient flow. One example, the Cohort System, also assists in preparing for patient visits, decreasing patient time in the clinic and defaulter tracking (this was observed during the evaluation site visits at Murchison Hospital in KZN, a Broadreach-supported facility). Patients have scheduled appointments in week blocks. Files are numbered and stored in cabinets according to the appointment date. This is intended to reduce waiting times while charts are obtained. If the medical file remains in its cabinet at the end of the week, the patient is considered a defaulter and is contacted. A ―fast track‖ system for initial and subsequent visits for patients with low CD4 cell count is implemented at many sites (e.g., Hillbrow Community Health Centre), thereby decreasing the waiting time for sicker patients by ensuring they are seen first. Adherence Clubs As seen in action at such facilities as Wallacedene PHC in Western Cape, these clubs encourage the self-management of stable patients and are more cost-effective than individual counseling. As self-management is the first step toward patient empowerment in the chronic care model, consideration of expansion of these clubs to other partners and to DOH facilities is encouraged. KEY RECOMMENDATIONS 1. Supportive supervision and close mentoring of NIMART nurses, e.g., in pediatric ART service delivery, represent potentially critical roles that PEPFAR partners can continue to play during and after the transition from PEPFAR direct support into TA, harmonized with the NDOH transition into PHC ART service delivery. 2. Ensure task-shifting is effectively supported over the long term, with close attention to how best to retain key workers and volunteers, such as social workers and patient advocates. 3. Innovative ways for follow-up of stable patients while maintaining adherence may reduce the burden of high patient volumes at the PHC level. Examples include adherence clubs for stable patients (Kheth‘Impilo and Anova), a community-based approach to support adherence tracking and household-based teams (Africare), and working with patient advocates (as developed by Kheth‘Impilo). 4. Quality improvement: start with a single data point, such as numbers screened for TB at an ART visit, to begin the quality improvement process. There is currently a concerted effort to review and use data captured from clinics at district and sub-district district levels; the DOH needs to institutionalize and effectively support such activities at the facility level. One possible approach is the QI model applied by the Reproductive Health Research Unit (RHRU) and the Hillbrow community health center (CHC). Further Recommendations Pediatric guidelines need to be updated to include regimens for when the second line regimen fails. Medical doctors and NIMART trained staff need more training in management of pediatric HIV infection. USAID/SOUTH AFRICA PEPFAR TREATMENT PROGRAM: FINAL EVALUATION 13 Prevention of nurse and other provider burn-out. In integrated clinics, having professional nurses rotate through FP, TB, HIV, and chronic illness clinics supports professional development, strengthens moves toward greater normalization of HIV, and can reduce staff member stress and burnout. Options for psychosocial support (e.g., health worker counseling) should also be explored. Focus on targeted support to adolescents within the context of ART service delivery. Widely promote adolescent-friendly services (e.g., the model of Adolescent Clubs as developed by ANOVA) Strategies need to be developed and implemented to sensitize health workers on the right to health for all, including vulnerable groups/MARPs such as migrants and same-sex couples, need to be implemented so that people are not marginalized and can also easily access HIV services. Support the genuine development of an evidence-based approach to health planning, with effective iteration of analysis (i.e., feedback to all levels of facility that are providing data). PMTCT The South African PMTCT program started in 2001, using a single dose Nevirapine regimen. This was updated in 2008 to a dual therapy protocol (using AZT and Nevirapine). Infant feeding guidelines have also been updated according to the latest evidence. The South African HIV, AIDS and STI National Strategic Plan 2007-2011 (NSP) aims to scale up access and improve quality of PMTCT services to reduce mother-to-child transmission (MTCT) to less than 5% and also to make provision for the special treatment needs of women and children. The 2009 PEPFAR South Africa Country Operational Plan also emphasizes the need for PEPFAR partners to work with the Department of Health to scale up facility and population-based maternal and child health (MCH) to achieve these goals. It further outlines the need for all programs to demonstrate referrals and linkages to care and treatment, progress toward family-centered care and treatment programming, and improved maternal and child healthcare service quality. Despite increased PMTCT services at facilities (> 90% of facilities), in 2008 the MTCT rate was still higher than the expected <5%, with some areas as high as 20%. Fewer than 80% of women were accessing PMTCT services. In addition South Africa continued to have one of the highest under-5 mortality rates, due in part to MTCT and high mortality among HIV-infected pregnant women. These factors further heighten the urgent need for access to quality PMTCT services at the PHC level. An Accelerated PMTCT plan was developed by the NDOH in collaboration with PEPFAR and other development partners. This A-Plan, in line with NDOH and PEPFAR strategic direction for PMTCT integration with MCH and PHC reengineering, provides clear implementation guidelines on how to use data and district systems to improve PMTCT implementation. Its key priority areas are to reduce MTCT and maternal and child mortality.2 In 2010 the NDOH and partners implemented the new PMTCT guidelines in which all HIV￾positive pregnant women with a CD4 count of ≤ 350/mm3 commence lifelong ART earlier, or women who are not eligible for lifelong ART commence ART prophylaxis earlier, at 14 weeks pregnancy. In addition, and for the first time, HIV-positive women can safely breastfeed their children provided the child is on ART during the breastfeeding period (NDOH & South African National AIDS Council. Clinical Guidelines: PMTCT (Prevention of Mother -to- Child 2 See the NDOH Operational Plan for Accelerating Scale-up and Improvement of the Quality of Services for Prevention of Mother to Child Transmission (PMTCT) in the Context of Integrated Maternal and Child Health Care in South Africa. 2009. 14 USAID/SOUTH AFRICA PEPFAR TREATMENT PROGRAM: FINAL EVALUATION Transmission). The first prevention of parent-to-child transmission survey reported a decline in HIV prevalence to 3.5% at the first immunization visit (four to eight weeks post-partum) showing promising results supporting UNAIDS/WHO strategy for the virtual elimination of pediatric HIV by 2015 (HIV & AIDS Treatment in Practice. Issue 178, July 1, 2011). With this background and consideration of the key elements of the South African PMTCT strategy (primary prevention among parents and parents to be, prevention of unwanted pregnancies, prevention of vertical transmission and care and treatment of mothers, children, and families with HIV), this evaluation focused on the following PMTCT areas: knowledge of PMTCT guidelines and alignment of PMTCT services to national guidelines; availability of those guidelines, along with job aids and standard operating procedures; integration and coordination of ANC; and labor, delivery, and post natal care of mother and baby, including a follow-up plan up to 18 months. Attention was also given to comprehensive clinical care of mother and baby, efficiency of laboratory services, community involvement, quality assurance supervision, and mentoring and monitoring and evaluation. The trend in the most facilities visited is that they are not PMTC sites per se. The sites are mostly involved with the care of pregnant women on treatment (i.e., not specifically PMTCT prophylaxis) and postnatal care, mainly of mothers. Achievements Some sites have a dedicated PMTCT nurse to enhance the connection between HIV treatment and PMTCT. PMTCT is the most accepted and integrated component of HIV management and represents a good opportunity to support acceptance and normalization of HIV. A number of PEPFAR I partners use PMTC as an opportunity to enhance coverage and an entry point to a range of services (for example: facilitation of identity documents for mothers; issuing of parent‘s death certificates; grants and foster care for orphans and vulnerable children (OVC); male partner involvement; and comprehensive care for care givers). There is thorough knowledge of the latest PMTCT guidelines. PEPFAR partners have contributed considerably to the training of a number of clinicians and ancillary health workers on HIV management, including ongoing updates on PMTCT. Partners provide on-site mentoring; this practice enhances transfer of skills and increases capacity within the facility to the point that some facilities have their own on-site mentors. PEPFAR partners have facilitated the availability of PMTCT guidelines, job aids, protocols, clinical record forms, and registers, and have additionally been instrumental in the promotion of community involvement through different community-based approaches. These include collaborating with and capacitating a local CBO, as well as use of patient advocates and the mother-to-mother (M2M) model (the latter observed during the evaluation at Imbalenhle CHC (a facility supported by Kheth‘Impilo). Challenges In general, laboratory services are working efficiently at most facilities, especially in the turnaround time for CD4 counts; however, in some facilities there was a reported a delay (up to four weeks) for polymerase chain reaction (PCR) results. Staff retention remains a challenge, e.g., with continuous training or retraining needed in many facilities. The burden of the number of registers at facilities undermines the value of the registers for quality improvement at the facility level; there is under use of data of PMTCT data by facilities. Decentralization of services requires careful planning for some of the underserved PHC facilities. HIV-positive mothers and their exposed infants often require more individualized attention; with limited services they can fall between the cracks. Reluctance to manage pediatric USAID/SOUTH AFRICA PEPFAR TREATMENT PROGRAM: FINAL EVALUATION 15 care even among NIMART and integrated management of childhood illnesses (IMCI)-trained nurses together with limited pediatric expertise poses a real challenge to early clinical diagnosis and management of HIV-infected children. This also hampers the current decentralization strategy of making ART services accessible at the patient‘s doorstep. Long-term mother and infant care and follow up are challenged by care of babies by grandmothers (with limited education and parenting capacity) and out migration of parents. There is a need for unique identifying numbers to facilitate the tracking of mother and babies who remain within the health system. Availability of psychosocial services linked to poor care of children remains a big challenge. Gaps Currently there is no fully applied model for joint follow-up of the mother and baby up to 18 months (although Kheth‘Impilo-supported facilities are moving toward this goal). There seems to be poor emphasis on integration of reproductive health and HIV services in general; proper family counseling is scarcely practiced and the choice of contraceptive methods is limited. There are very few health services for adolescents (some of whom are HIV-infected due to MTCT), including contraceptive needs. Yet there is an increasing problem of teenage pregnancy and the 2008 antenatal survey reported an HIV prevalence of 14% among pregnant teenagers aged 15-19 years (National Department of Health. National HIV and Syphilis Prevalence Survey South Africa 2008. Pretoria: NDOH DOH June 2009). The inconsistent involvement of men (e.g., the absence of male-friendly ANC and labor wards) in HIV, FP, and reproductive health programs represents an opportunity missed to strengthen family-oriented health-seeking behavior and male support to PMTCT and pediatric care of HIV￾positive infants and children. Key Recommendations 1. Integration of FP, reproductive health, ANC, PMTCT, and IMCI needs to strengthened at the primary health care level to improve maternal and child health. The choice of contraceptive methods needs to be expanded. All such expansion needs to take into account the realities of capacity and health systems at this level of service delivery. This may represent an area for continued direct support by PEPFAR partners. 2. Provide additional targeted support to midwives and PMTCT nurses in training, mentoring, and quality improvement/assurance. 3. Address appropriate strengthening of community engagement in PMTCT, e.g., through community health worker involvement. Communities need to be sensitized on the relevance of FP counseling for potential parents, regardless of the couple‘s HIV status (concordant/negative or concordant/discordant). Added to this is the rejuvenation of couple HIV counseling and testing from both demand (community level) and supply side; couples of reproductive age need to know their HIV status before conception. 4. Seek to address male involvement in reproductive health issues, PMTCT, and pediatric ART. Further Recommendation Further address the role of caregivers other than mothers in PMTCT and pediatric ART. 16 USAID/SOUTH AFRICA PEPFAR TREATMENT PROGRAM: FINAL EVALUATION TB/HIV INTEGRATION A thorough assessment of TB/HIV integration is beyond the scope of this evaluation. Further, the evaluation team was not able to assess the degree of implementation, as there was no IPT register or consistent documentation. However, a separate and comprehensive evaluation of South African TB/HIV integration with an emphasis on implementation of IPT was conducted in early 2011 and provides data on 49 sites throughout all provinces (Chebal et al., 2011). The following is a short overview of the current situation. Overview TB and HIV are intimately intertwined at both an individual and epidemic level. The burden of HIV/TB disease on both TB and HIV care services has been particularly high in South Africa. Large and rapidly growing numbers of patients with HIV and TB co-infection require coordinated diagnosis, care, and treatment. HIV/TB cases receive care in two separate public￾health systems, resulting in duplication of patient-health provider interactions and a lack of integration between the services, resulting in missed opportunities for prevention and prophylaxis. Barriers to integration of HIV and TB services include the historical development of two separate disease-management structures with different characteristics and divergent philosophies. The South African National TB Programme (NTP) has developed over decades, using a centrally organized public-health approach. Treatment is strongly tied to population control of TB transmission, concentrating on standardized diagnosis and chemotherapy of self￾referring individuals with predominantly smear-positive pulmonary TB. The NTP has incorporated pragmatic targets for directly observed therapy short-course (DOTS) coverage, cure rates (>80%), and case finding (70% of smear-positive disease). Medication is typically dispensed daily under direct supervision for a period of six to nine months, and usually given at a community level of care. NTP has successfully made complex TB treatment scalable and effective. By contrast, HIV services have developed more recently and are patient-orientated, with a strong emphasis on human rights and social justice. Individual HIV management is weakly linked to epidemic disease control, as many HIV-infected individuals are unaware of their status and are not in care. The HIV epidemic is undermining TB control in many high TB burden countries. Increasing numbers of co-infected patients have resulted in limited collaboration between HIV and TB programs at different levels of the healthcare system. HIV testing of TB cases currently provides a major portal for entry into HIV care; however, HIV testing is still far from universal within TB programs. Advanced HIV disease makes TB a diagnostic challenge, requiring increased access to diagnostic modalities additional to sputum smear. Earlier access to HIV care including screening for TB will also require expansion and reconfiguration of HIV services. A key indicator of successful integration will be the documentation of the delay between presenting to the health system and initiation of ART for patients presenting with TB. Postponement of ART in these patients results in increased mortality. Delayed ART initiation is comprised of TB diagnostic delays, health systems referral delays, and ART initiation delays. It therefore could be a good reflection of effective integration. Treatment as prevention (TasP) might in the near- or medium-term future offer another opportunity for TB and HIV integration. Despite TasP being primarily targeted at decreasing HIV incidence, inclusion of regular population TB screening would additionally impact on community TB transmission. USAID/SOUTH AFRICA PEPFAR TREATMENT PROGRAM: FINAL EVALUATION 17 Achievements There was near universal awareness at sites visited of the high TB/HIV co-infection rate in South Africa and a good understanding of the importance and need for TB/HIV integration. All PHCs visited had integrated TB/HIV services. New diagnostics tools, e.g., Gene Xpert, are being rapidly incorporated in PHC sites visited (at least in Kwazulu-Natal) and are more accessible than chest x-rays. Mobile chest x-rays that visit a number of sites at scheduled intervals have helped with access to this service. One PEPFAR￾supported facility is a TB ―step- down‖ hospital; the only access the facility has to chest radiography is through the mobile van. Challenges Pediatric TB diagnosis was cited as a common problem, with clinicians needing to up refer to district hospitals due to poor diagnostic tools for infants and children with TB. Infection control is a concern in many facilities visited in the course of the evaluation, due to small clinical spaces, use of park homes, and inadequate infrastructure. In one facility five health workers had recently contracted tuberculosis. Although awareness of the need for respiratory infection control was high, the practices observed were less than optimal. Again, much of this is due to poor infrastructure. One district hospital visited for the evaluation has good facility infection control in the TB clinic, while in the adult HIV clinic there is a high ceiling and high windows but patients are crowded together. The pediatric clinic is a park home with clinicians sharing offices and exam rooms and an extremely cramped waiting room. Further discussion of this issue is provided later in this section. Gaps While staff members at most sites know of the 2010 NDOH guidelines for IPT, there appears to be variability in implementation across sites. Best Practices One facility in the Western Cape that specializes in TB and multidrug resistant TB (MDR-TB) has a dedicated entrance for TB patients so that they do not need to queue with other HIV￾infected patients. The PHCs with full TB and HIV integration had generally good clinical processes, registers were up to date, and defaulted tracking was implemented. In facilities without full integration, ―close proximity‖ was noted to enhance full care for each disease. One facility manager noted that patients with TB were ―walked across the street‖ to the TB clinic; this also encouraged regular meetings of the staffs of both programs. In KwaZuluNatal a PEPFAR implementing partner (MATCH) and district health services each purchased a mobile digital x-ray van. Visits to HIV and TB clinics were coordinated so that physicians could access chest x-rays. While not absolutely necessary to diagnose and manage TB, chest x-rays are useful and encourage HIV clinicians to include TB as part of a diagnostic work-up of pulmonary or constitutional symptoms. Another example of best practice in terms of TB/HIV integration is the Imbalenhle Community Health Centre in KZN, which began to pilot TB/HIV integration from 2004, initially under the auspices of ARK and now taken over by Kheth‘Impilo. 18 USAID/SOUTH AFRICA PEPFAR TREATMENT PROGRAM: FINAL EVALUATION Key Recommendations 1. Down refer only when the patient is TB stabilized – strengthen such links at district and sub-district levels. 2. At PHC level, have clear guidelines and well-trained staff to refer up complex case. 3. There is a need for a strong health facility-community DOTS link, e.g., such as happens through the Patient Advocates Model, as primarily developed by Kheth‘Impilo. Further Recommendations It should be standard practice to have a joint tracking team for ART defaulters and TB. PEPFAR partners should provide dedicated TA to support the DOH on multidrug and extensively drug-resistant TB (M/XDR), because epidemiologically such cases will only increase. More information must be provided to health workers and patients regarding infection control. MONITORING AND EVALUATION The ART program evaluation focused primarily on four M&E focus areas: M&E capacity and functions, ART data management systems, data quality, and data use. Achievements Without support of PEPFAR partners as described below, there would be very few data at site level; this was especially the case in the early years of ART programming. M&E Capacity Building Much recognition was given to PEPFAR during the evaluation for creating M&E capacity in the form of manual and electronic systems, filing systems, and computers. Through PEPFAR support, M&E staff, including data capturers, data quality mentors, and M&E officers, have been recruited, trained, mentored, and provided with refresher training as necessary. M&E staff were provided with SOPs and South Africa Strategic Information (SASI) Manual with indicator definitions and reporting guidelines. Enhanced Data Management Systems E-systems for both pharmacies and clinics have been made possible with PEPFAR funding. These systems include: Therapy Edge, a for-profit system that tracks patients‘ outcomes; and the Intelligent Dispensing of ART (iDART), a tool used by pharmacists. Effective use of the iDART system supports enhanced dispensing and reduced client waiting time. Other value-added inputs include the introduction of technology, e.g., use of short message services (SMS) to report results from the labs, a PMTCT tool for tracking HIV-exposed babies (in pilot), and improvement in filing systems. 3-tier Strategy As there is a great need for an advanced, standardized, and integrated ART recording e-system to be used by all facilities, PEPFAR has worked with the SAG to develop the new 3-tier strategy to strengthen routine and clinical monitoring of data used for ART patient management. Tier 1 is a paper-based system that is aligned with WHO recommendations and functions in sites with fewer than 500patients and limited infrastructure. Tier 2 is an electronic implementation of a facility-based register and entails the use of a non-networked electronic register (e-Register) USAID/SOUTH AFRICA PEPFAR TREATMENT PROGRAM: FINAL EVALUATION 19 that is aligned to the paper-based register. Tier 3 is a fully networked patient management system that produces the same data as tiers 1 and 2, but has much more capability. eKapa, currently used and owned by the Western Cape Department of Health, has been selected as the national 3-tier HIV patient management system. Data Quality PEPFAR has assisted facilities since 2004 with patient management systems that could track patients and minimize double counting. Data are required to be verified at various levels of reporting. Data quality audits have been conducted with PEPFAR partners since 2005 and improvement plans developed to improve identified weaknesses. Routine data quality assessments (self-assessments) have been introduced to partners and SAG facilities to enable identification of data quality issues and corrective actions. Research Studies PEPFAR has supported research studies at the national and facility level. At the national level PEPFAR has supported studies such as those on cost of ARVs, ART scale-up, and human resources. The National ART Cost Modeling exercise in 2009/2010 identified two mechanisms for cost savings: a change to drug procurement at internationally competitive prices; and changes to staffing levels and tasks whereby nurses initiate and manage ART at the PHC level under physician supervision and ARVs are dispensed by pharmacy assistants under pharmacist supervision. PEPFAR has also supported research at the facility level using routinely collected data. Challenges Parallel Reporting All sites and partners have been required to report to the DOH and PEPFAR, often by using different indicators. The two reporting systems have led to parallel reporting and appointment of additional M&E staff to alleviate this reporting burden. The two M&E systems are frequently not coherent, which has led to imprecision, e.g., regarding the actual numbers on ART. Lack of Standardized Indicators Frequent PEPFAR indicator changes have been challenging for PEPFAR partners, because tools, systems, and training have had to be modified accordingly. Explosion of a Variety of Patient Management Systems Through PEPFAR there has been a mushrooming of patient management systems. The ―explosion‖ of 30-40 database systems at the health facility level is profoundly unsustainable and also inimical to SAG ownership of data. Lack of Data Use Limited access to and equally constrained ability to use data continue to be primary deterrents to data use. Instilling the culture of data for evidence-based planning and effective M&E in both PEPFAR partners and SAG is still a work in progress and is closely linked to the building of the capacity of (especially) HIV/AIDS, STIs, and TB (HAST) managers on overall information management. Likewise, comfort with data and analysis encourages use of specific parameters for clinical and programmatic quality improvement. Implementation of the New 3-tiered Monitoring Strategy The new ART monitoring system has not been implemented in some provinces. For example, the DOH made the decision to abandon the Adult Clinical Record (ACR) form used in Eastern Cape. Outside Western Cape there continues to be resistance to the use of eKapa. Partners with Tier 3 systems are still waiting for the implementation plan for Tier 3. 20 USAID/SOUTH AFRICA PEPFAR TREATMENT PROGRAM: FINAL EVALUATION Gaps PEPFAR developed the web-based Data Warehouse to which PEPFAR partners submit their plans and reports. However, data are primarily consolidated for PEPFAR management and are not shared with the SAG. The Data Warehouse is not user-friendly. There is a lack of data capturers in some health facilities, lack of reliable and timely DHIS data, and lack of triangulation of DHIS or Data Warehouse data with pharmacy and lab data. There is lack of transparency in sharing of targets, numbers reached, and budgets. There has been limited use of data for evidence-based planning. There is a lack of harmonization of indicator definitions between SAG and PEPFAR indicators. Best Practices PEPFAR partners are slowly beginning to share data with SAG. Roving teams that include an M&E officer build M&E capacity of staff at PHCs. Other best practices include formation of M&E structures at both district and sub-district levels and implementation of data quality assurance (DQA) and routine data quality assurance (RDQA) at all levels of reporting. Key Recommendations 1. SAG and PEPFAR need to harmonize all indicators, definitions, and SOPs for the ART program. 2. PEPFAR partners‘ transition to provision of TA requires the creation of an M&E plan that includes process and outcome indicators and milestones of progress. TA cannot be measured entirely or even primarily through quantitative methods. In this context, the ART program should be informed by GHI principles and approaches (more discussion of this issue is provided later in Section V). 3. Improve data management and use data to support evidence-based planning and reporting; this includes making the Data Warehouse more fit for purpose and user-friendly, available to all relevant partners. 4. The implementation of the 3-Tier Strategy can be enhanced by finalization and dissemination of the implementation plan and by expanded training. Further Recommendations Tier 1 needs to be standardized, especially as many PHCs will be at this level for a long time. All Tiers 1-3 should fit into the District Health Information System (DHIS). Reported data should be cross-checked with pharmacy and lab data. Integrate patient tracking and pharmacy systems—pharmacies can quickly pick up missed appointments, while such tracking in registers can take three to six months in the worst cases. i-DART— this provides the option of serving solely as a pharmacy system, or it can expand; the most effective mechanism should be explored. The SAG should consider absorbing partner-trained data management staff as partners make the transition to the technical assistance model of support. Data management staff should be provided with on-going training and support for enhanced data management and data use, including data capturers. Such training should include analysis, so that M&E staff move beyond mere collection of raw data. USAID/SOUTH AFRICA PEPFAR TREATMENT PROGRAM: FINAL EVALUATION 21 The DHIS should be updated and networked, as the upward and downward flow of data is currently slow. There is a need to educate all health workers and M&E staff members to understand why data are important and how and why their use can make service delivery easier. FOUR “QUICK LOOKS”: PHARMACY, LABORATORY, INFRASTRUCTURE, AND INFECTION CONTROL Pharmacy PEPFAR works closely with NDOH at the national and provincial levels to assist with procurement and distribution of antiretrovirals (ARVs) and other drugs. PEPFAR procurement of ARVs is limited as SAG procures drugs for the public sector, which serves the vast majority of the patients on ART. Achievements PEPFAR has facilitated a major expansion of pharmacy resources and inputs. ART could not have been rolled out as speedily or as comprehensively as it was without such support. PEPFAR funds ARVs for non-governmental organizations (NGOs) and private facilities that directly serve about 10% of total number of clients on ARVs in the country. PEPFAR spent a one-off $120 million on ARVs to avert stock-outs in public health facilities in 2010 and 2011.Through technical assistance and supporting cost studies, PEPFAR has contributed to reduction of ARV drug costs. At the facility level, PEPFAR has supported the implementation of the iDART pharmacy system that supports pharmacists in dispensing accurately and processing clients faster. The system helps identify, and hence follow up on, patients who have missed appointments. Through the provision of TA and equipment and the use of pharmacy assistants, PEPFAR has genuinely strengthened pharmaceutical services, despite the continuing grave shortage of fully trained pharmacists. Challenges As NIMART is rolled out, there is an ever-increasing volume of clients on ART without a concomitant increase in human and infrastructural resources. Lack of adequate space, refrigerators, and air conditioners for adequate and optimal drug storage are other challenges. In most facilities it is difficult to procure and store several months‘ supply of ARVs, affecting the dispensing of more than one month‘s supply of ARVs to stable patients. Some facilities have experienced stock-outs largely due to inefficient practices of procurement from the district depots by some facilities or to sudden high volumes of patients during transition. However, there have been no observed instances of patients going without antiretrovirals. Rather, PEPFAR partners have gone to great lengths to ensure ART coverage. Examples of how this was managed included: 1) using pediatric formulations in adult dosages and crushing adult tablets for pediatric dosages; 2) refilling for two weeks supply only; and 3) transferring medication from pharmacies supported by the same PEPFAR partner. While creative solutions have prevented treatment interruption, the very need to resort to such practices raises concern about some of the facilities‘ ability to plan for increasing patient burdens and down referrals. Defaulting, even when patients are on a second line therapy, is a serious problem especially for patients who have been transferred out from other facilities. Pediatric medication is challenging due to lack of temperature-regulated storage, lack of treatment supporters for orphaned children, and incorrect administration of dosages by guardians. 22 USAID/SOUTH AFRICA PEPFAR TREATMENT PROGRAM: FINAL EVALUATION Gaps With the rollout of NIMART, PHCs will have to contend with the increased workload without additional staff. Another gap is the lack of integrating patient tracking and pharmacy systems, as pharmacies can quickly pick up missed appointments. Registration of ARVs by the Medicines‘ Control Council (MCC) has been slow, limiting the types of ARVs that can be procured. Best Practices ARV procurement is aligned to DOH procurement requirements. Some facilities have dedicated ART pharmacies. Other best practices include the use of pharmacy assistants and roving pharmacists, use of iDART pharmacy system, pre-packaging of drugs, and running of adherence clubs for stable patients. Recommendations 1. Integration of patient tracking with pharmacy systems will lead to better patient management and reporting. 2. The SAG should provide legislative and policy clarity on whether pharmacists can accept nurse- signed prescriptions. 3. The SAG needs to enhance pharmacy space to allow for larger bulk orders, minimizing the number of patient visits for stable patients and increasing clinic efficiency. Further Recommendation Provide on-going patient-friendly education on branded and generic ARVs, their side effects, consequences of non-adherence, and drug resistance. Laboratory Services Achievements PEPFAR supports the NHLS and has also supported point of care diagnostic services. District NHLS laboratories supports a network of local clinics that provide primary care services. PEPFAR also funds Toga Laboratorie‘s establishment of a network of HIV monitoring laboratories and associated service access tools in resource-constrained ART settings in areas where public NHLS coverage is limited. Challenges Facilities without on-site NHLS labs are improving turnaround times, but this remains an issue. A Toga lab in one facility was closed in April 2011, as the operation was not cost-effective. Gaps Lack of integrating lab data into patient management systems for improved patient management and triangulation of reported results. Best Practices The use of park homes to support rapid establishment of decentralized labs and the use of lab data to manage and track missed appointments. USAID/SOUTH AFRICA PEPFAR TREATMENT PROGRAM: FINAL EVALUATION 23 Recommendations 1. Conduct further studies on the cost-effectiveness of decentralized labs in the short and longer term 2. Expand use of the National Health Laboratory Service (NHLS): this collects and analyzes a wealth of data that could be explored for monitoring, planning, and improving ART services. CD4 and viral load could be tracked over time by province, district, sub-district, or facility. Such data could be de-linked from patient identifiers and analyzed in aggregate. As the data would be used only for program evaluation, there should not need for an extensive Institutional Review Board process The development of a national unique identifier system, as is used in the Western Cape, would further enhance the specificity of the data, as patients accessing ARVs for the first time could be differentiated from ―transfers out‖ who may be moving between programs. This would allow CD4 and viral load to be tracked longitudinally per patient, providing information on CD4 when first receiving HIV testing, when ART is first initiated, and at regular intervals.3 Infrastructure4 Achievements PEPFAR infrastructure support and use of mobile services demonstrate innovative models of providing treatment in resource-poor settings. PEPFAR created infrastructure for the ART program by extending buildings, providing much needed equipment (computers, filing cabinets, furniture, x-ray boards, and communication equipment) and vehicles and trailers for mobile services. Challenges The major and perennial constraint of space is a cross-cutting challenge, with implications in terms of overall quality of care, e.g., infection control, privacy, confidentiality, patient flow, drug storage, and child- and youth-friendly services. Infrastructural gaps are likely to adversely influence the new PEPFAR TA approach. Some PEPFAR partners‘ view is that TA will be futile in the absence of the right infrastructure while facilities are likely to request infrastructural support before accepting TA. Gaps The DOH is not yet ready to take on the full responsibility of maintaining PEPFAR-funded structures. There are still considerable infrastructural gaps in the facilities. One point made several times by both provincial DOH staff members and implementing partners is that little consideration appears to have been given yet by PEPFAR or the SAG on how the presumed increase in ART patient numbers will impact on physical infrastructures, with regard to overall waiting space, infection control, waiting times, quality of services, pressures on staff members, and maintenance of buildings and grounds. The issue of maintenance budgets was raised on a number of occasions, invariably linked to a lack of clarity on how such monies would be allocated on a long-term basis and from which sources. PEPFAR infrastructure assistance appears to have been provided, especially to district/gateway facilities, less so to PHC facilities. With the increased focus on PHC-level service delivery there will be a need for greater, sustained infrastructural refurbishment and maintenance. Several implementing partners and DOH staff members expressed concern over the apparent lack of clarity as to the overall lead responsibility for such activities. 3 See also SOW Question 6 in Section V and Annex F. 4 See also Annex F. 24 USAID/SOUTH AFRICA PEPFAR TREATMENT PROGRAM: FINAL EVALUATION Best Practices Buildings were extended through the use of park homes (containers) and equipped with sophisticated ventilation systems and furniture. One facility visited during the course of the evaluation has several park homes to serve adolescents at a separate location that remains open from 10 am to 4 pm for HCT and youth support groups. Recommendations 1. The PEPFAR transition to TA must clarify where responsibility for infrastructure development and maintenance will reside. 2. PEPFAR partners should work with SAG to ensure continued and proper utilization of buildings, equipment, and supply. 3. The refurbishment of health facilities should form part of the SAG‘s Expanded Public Works Programme (EPWP). Infection Control Achievements In one facility there are no staff known to have been infected with TB, all staff wear masks, and there is tight control over ventilation. In addition the facility had a sputum booth and separate queues and management systems for TB/HIV/pediatric patients. Challenges The evaluation team established there are structural challenges related to space and compliance with infection control measures in some of the visited facilities. Major space constraints are a cross-cutting challenge that has implications in terms of overall quality of care, e.g., infection control. The risk of infection due to congestion is high. In one Eastern Cape facility supported by a PEPFAR implementing partner, five staff members were reported to have contracted TB. Best Practices Best practices include the TB/HIV Association and RHRU/HCHC models of addressing infection control, especially in light of MDR/XDR TB. At Hannan Crusaid facility patients reporting coughing episodes are given protective masks and TB patients are fast-tracked. Key Recommendations 1. Control the spread of TB through IEC and training on cough etiquette and respiratory hygiene. 2. Minimize the time TB patients spend at health facilities, e.g., through fast tracking. 3. Make effective and enhanced use of respiratory protective equipment and cough control in high-risk situations. Further Recommendations Where feasible, review the use of available spaces and consider renovation of existing structures to improve ventilation. Ensure ventilation systems are used appropriately, especially in the cold seasons. USAID/SOUTH AFRICA PEPFAR TREATMENT PROGRAM: FINAL EVALUATION 25 V. DISCUSSION OF OVER-ARCHING ISSUES INTRODUCTION The following section addresses strategic and systemic issues, primarily looking to the future in light of the significant changes taking place in South African health service delivery. The SAG and the NDOH have made the decision to support the primary health care level to deliver most ART services. Recognizing the inefficiencies and redundancies inherent in their method of funding, PEPFAR has undertaken an alignment of the districts (a process now close to 90% complete), in which its implementing partners work to enable more appropriate and effective geographic coverage of ART service delivery, and seek to align PEPFAR partners in supporting district and sub-district ART service delivery needs. PEPFAR is also requiring its implementing partners to make a transition from provision of direct ART service delivery providers to a greater focus on technical assistance. The first part of this section considers the 10 scope of work questions and predominantly addresses the historical environment in which PEPFAR Phase 1 activities occurred (the emergency phase), with the intention of providing a degree of understanding of the processes to date. The section then goes on to address the three SOW forward-looking points, and then considers two questions submitted by CDC. Next, the section examines the three NDOH issues of coverage, efficiency, and quality in the context of the overall PEPFAR ART program. The next subsection addresses PEPFAR‘s transition from direct support to technical assistance. To conclude, the section briefly examines lessons learned to date from the almost completed PEPFAR district-level alignment process. THE 10 SOW QUESTIONS SOW Question 1: Did the program help to achieve reduction of the estimated treatment gap? The answer is an unequivocal yes. In September 2004, the coverage rate for ART was 7%. This had increased to 36% by February 2007, at which time 257,108 people were receiving ART through the national ART program (i.e., with all support, not solely from PEPFAR). This number had risen to 458,951 by the end of the year. By the end of 2008, WHO (2009) reported that 700,500 people were receiving ART through the national ART program, which represents a 53% increase (241,549 people) in the proportion of people on ART in South Africa between December 2007 and December 2008. See Figure 2 below.5 5 All data from HEARD 2009. 26 USAID/SOUTH AFRICA PEPFAR TREATMENT PROGRAM: FINAL EVALUATION Figure 2. ART Program Sites Visited In 2005 there was 8% coverage of people requiring ART; by 2010 80% coverage had been achieved. NDOH data state that currently 1.4 million are on treatment, with 470,000 added in the last 12 months, of whom 100,000 are children. PEPFAR provides 10% of all ART inputs, the South African government 85%, and other donors 5%. In 2004 PEPFAR provided impetus to start up and scale up ART in a challenging national environment. Its commitments have included support to accreditation, considerable inputs to infrastructure, training, mentoring, lobbying of the U.S. FDA to approve generics, and a one-off emergency input of $120 million for drugs, etc. Therefore, the PEPFAR program as a whole and in terms of its individual partners and their sub-partners has supported SAG action to achieve reduction of the treatment gap. Because the SAG has in the past year set a higher CD4 threshold for entry on to ART than is given in the WHO treatment guidelines, this means more people can gain access. This action also ushers in service delivery challenges, including human and other resource challenges at health facilities. This is one challenge that PEPFAR will need to consider in its transition from direct service provision to TA. One final point is that a number of PEPFAR partners and other stakeholders strongly stated during the course of the evaluation that the emergency phase is not over, due to the volume of new patients and late presenters. As a result, the treatment gap will continue to be challenging until such time as the epidemic reaches a plateau.6 6 See also Atun & Bataringaya 2011 for one recent contribution to the debate over the continued need for mixed responses that incorporate emergency elements within an overall move toward chronic care management models for HIV and ART. USAID/SOUTH AFRICA PEPFAR TREATMENT PROGRAM: FINAL EVALUATION 27 SOW Question 2: Did the program support the SAG to develop recognized standard public health practice for ART? Yes, because PEPFAR implementing partners have sat on all DOH HIV Guideline committees. public health practices – e.g., adherence clubs, tracking in retention and care, patient literacy, cohort follow up – all these have been informed by the PEPFAR inputs. PEPFAR has assisted in the mobilization of the public health community to become fully functional in response to the HIV emergency phase and beyond. SOW Question 3: Did the program play a lead role in reducing costs of delivering quality ART services? It is essential to bear in mind the emergency phase, along with the fact that PEPFAR did not start as a cost-efficient model; these considerations continue to have implications. One frequent question asked by partners and DOH counterparts is: how does one evaluate economies of scale when the cost data are obscured by PEPFAR? Regardless of such issues, PEPFAR has influenced the reduction of costs in truly significant respects, e.g., ART costs were reduced as a result of the Ambassador Goosby program together with the Clinton Foundation initiatives. There has also been support to NIMART training and to monitoring patients so individuals need fewer viral load tests. Since 2004 cost-efficiency studies have been conducted by Boston University, with funding from PEPFAR, e.g., the SAG requested that PEPFAR estimate the cost of ARV drugs as well as the cost of ART program scale up. PEPFAR and CHAI have influenced the efficiency of drug tendering and pricing. PEPFAR also provided a one-off emergency infusion of $120 million for ARVs, with the overall condition that drug costs would be reduced. Much PEPFAR support has had positive implications in terms of cost efficiency. Considerable work has also been undertaken on certain specific inputs, e.g., reduction in the cost of viral load tests. It is difficult to disentangle the value for money (VFM) and cost per patient data from the systems implemented by PEPFAR. CHAI has been involved in studying such issues through the Global Health Financing team, which included an expenditure analysis of where monies have been spent on HIV. There is a National AIDS Spending Assessment (NASA) under way and also costing of a number of treatment models, looking at 30 sites across South Africa – this is linked to the DOH transition to PHC-level delivery of ART. Boston University supported the National AIDS Costing Model in 2009; a key finding was that the unit cost per patient on ART was rand 5,500 (clinical service delivery only). SOW Question 4: Did the program capacitate cadres of health workers to ensure a sustainable program?7 All such support must be viewed in light of the ongoing and considerable HRH constraints that persist in the South African health sector (see, for example, HEARD 2009 for an overview of the challenges). While such matters are the responsibility of the South African Government, the presence of PEPFAR and the recruitment activities of its implementing partners have undoubtedly had an impact on HRH constraints. Comments were made several times during the course of the evaluation that health workers and ancillary cadres, e.g., social workers and outreach workers, have increasingly come to prefer working for PEPFAR partners, due to factors such as supportive structures for capacity development and a more inclusive management approach. 7 See also Annex F. 28 USAID/SOUTH AFRICA PEPFAR TREATMENT PROGRAM: FINAL EVALUATION Therefore, while PEPFAR has undoubtedly supported its own program, there are repercussions in terms of the overall HRH outlook. The PEPFAR transition from direct clinical support to TA also has had a significant impact on national, provincial, and district HRH structures. One trenchant comment made in the Western Cape is that it is only relatively recently that the DOH has been informed that PEPFAR has provided some 90 million rand to HRH inputs. A key question is: if PEPFAR were to withdraw tomorrow, would the DOH be expected to take over those salaried posts, despite never having been consulted or participating in any HRH planning that PEPFAR and its partners may have undertaken? Another question: What will happen to those PEPFAR partner staff members who are now superfluous in the new TA dispensation? While it may be the case that the DOH need not absorb all such posts, there will be a number whose loss will have a negative impact on ART service delivery. This ―transition gap‖ may have serious implications for the continuity, integrity, and quality of ART service delivery. Through its implementing partners‘ activities, PEPFAR has supported considerable training inputs: e.g., NIMART, assistant pharmacist and post-basic pharmacy assistants, and strategic information/M&E training. CDC has been instrumental in supporting the development of the Clinical Associates Program, a key cadre affected by overall HRH constraints in South Africa. One best practice is that staff rotation has become institutional policy within a number of health facilities supported by PEPFAR partners (e.g., those supported by Kheth‘Impilo). Impacts have included overall capacity development, enhancement of integration of HIV within wider clinical service delivery, and reduction (or at least support for the potential reduction) of staff member burnout and stress. There appears to be confusion among partners and DOH staff members with regard to what the situation is concerning accreditation of PEPFAR training courses. If these are not accredited, then does this mean PEPFAR training may not be recognized by the SAG? SOW Question 5: Did the program enhance the SAG’s ability to monitor the progress of its ART program? 8 Partnering with the SAG, supporting the South African National Strategic Plan, and developing local capacity are three of the seven guiding principles of PEPFAR South Africa. A key stated PEPFAR objective is, therefore, to work with the SAG and fund initiatives that support the SAG in enhancing its monitoring of its ART program, as stipulated in the HIV and AIDS and STI Strategic Plan for South Africa and the Comprehensive Plan for HIV and AIDS Care, Management and Treatment. PEPFAR‘s emergency phase was characterized by a number of challenges that adversely affected PEPFAR‘s realization of its objective to enhance the SAG‘s ability to monitor the progress of its ART program. These challenges – a number of which persist to this day – include the following: Partners were funded in developing separate information systems. Separate reporting systems to provincial, PEPFAR, and USAID authorities were developed, with many implications, e.g., parallel systems, lack of data coherence, limited capacity to undertake joint (e.g., district-wide) evaluation M&E, and all too limited attention to the development of evidence-based planning and M&E. One view, forcibly expressed, is that there has been an ―explosion‖ of M&E systems at the health facility level through PEPFAR, a development that is ―profoundly unsustainable and also inimical to SAG ownership of data and strategic information (SI).‖ Early PEPFAR funding is said to have addressed performance through tick box-type reporting. Upwards of 40 M&E/SI systems 8 See also the discussion on this issue in Section III. USAID/SOUTH AFRICA PEPFAR TREATMENT PROGRAM: FINAL EVALUATION 29 are linked to PEPFAR, which has resulted in parallel systems and reporting, data loss, and lack of coherence. There is poor data quality in the DHIS, exacerbated by the fact that several DHIS systems are in use. In addition, the DHIS is not networked and data flow is slow and incomplete, resulting in poor quality raw aggregate data. While the DHIS is owned by government, it appears that few initial attempts were made by PEPFAR to support its strengthening. There was initially limited access and use of PEPFAR information by SAG to monitor its ART service delivery. The SAG had no input into development of PEPFAR indicators, which resulted in an absence of harmonization of SAG and PEPFAR indicators and definitions. The SAG initially had no clear data verification procedures for ART services. The Data Warehouse system has primarily been used by PEPFAR; however, partners without information technology systems have been able to access aggregated data, while it appears that government partners have not had access. PEPFAR partners did not on the whole support the development of an information and data management culture, in which there is an integrated and iterative understanding of the use and value of quality data in evidence-based planning and M&E. Therefore, it can be stated that in its emergency phase, the PEPFAR ART program did enhance the SAG‘s ability, but mainly by default. However, such inputs were relatively limited in relation to its size and funding streams. More recently, there has been significant collaboration between the SAG and PEPFAR. In 2009 PEPFAR responded to a SAG request to conduct an independent assessment of nine of the many PEPFAR-supported patient management systems to determine which best align with SAG priorities. SAG has worked with PEPFAR and the DHIS to review indicators, revised the ART monitoring systems, and has developed the 3-Tier strategy. The development of the 3 Tier system and its rollout were frequently mentioned during the evaluation as key building blocks on the road toward a more integrated, user-friendly data resource. PEPFAR partners in Western Cape have provided constructive feedback on eKapa, selected as the national HIV patient management system. Other positive inputs by PEPFAR and its partners include capacity-building activities, e.g., the Foundation for Professional Development PEPFAR Fellowship Program, where recent South African master‘s level graduates are placed within the SAG or with PEPFAR partners. In collaboration with NDOH and other key stakeholders, PEPFAR has contributed to the development of the 2007-2011 (HIV) National Strategic Plan M&E framework. One significant development that was supported by PEPFAR is the ACR: the Adult Clinical Record card. This was initially developed by ICAP and Africare and is now more widespread. Its use is considered to be instrumental in providing clear patient information. Further examination of data-specific inputs by PEPFAR partners to the district model of ART provision appears later in this section under the discussion of PEPFAR‘s transition to TA. SOW Question 6: Did the program increase overall retention rates and decrease mortality rates? A quantitative answer with regard to retention is difficult in the context of this performance evaluation, due to the absence of any appropriate comparison group. There is no doubt that the inclusion of retention as a PEPFAR indicator has increased awareness of retention within individual partners‘ programs and support to ART service delivery. Increased overall coverage has been associated with marked increases in the transfer out from services. Transfers out have 30 USAID/SOUTH AFRICA PEPFAR TREATMENT PROGRAM: FINAL EVALUATION been poorly tracked. Secular trends indicate increasing numbers of patients who are lost to follow up (LTFU). There were opportunities during the evaluation to analyze the quantitative data from PEPFAR partners that is stored on the Data Warehouse, itself a resource funded by PEPFAR. The following data indicate trends in terms of retention and death rates. ART program losses have been incorporated into PEPFAR indicators and are available from the Data Warehouse for the 6-month, 12-month, 24-month, and 36-month cohorts. Figure 3 shows the proportion of each cohort alive and on ART as of the second quarter of 2011, together with the proportions of patients who have stopped ART, died, are LTFU, or transferred out of the cohorts. Figure 3. Cohort Losses and Retention Proportions 0 0,1 0,2 0,3 0,4 0,5 0,6 0,7 0,8 0,9 36 mth 24 mth 12 mth 6 mth Cohort Losses & Retention Proportions Stopped ART Died Transfer out Lost to FU Alive on ART Source: USAID The 6-month cohort reported in the second quarter of 2011 represents individuals who started ART between April and June 2010; the 12-month, 24-month, and 36-month cohorts reported patients recruited to ART in the last quarter of 2007, 2008, and 2009, respectively. Therefore, the minimum period in which the 6-month, 12-month, 24-month, and 36-month cohorts could sustain program losses would be 9, 15, 27, and 39 months, respectively. The proportion of losses during each of these time periods gives an indication of the rate of ART program losses (and also indicates the rate of retention) during the last three years during which ART services have been rapidly expanded. One evaluation finding is that the system to track LTFU and adherence is a fallible one that incorporates minimal reporting; this fact has a potentially critical impact on overall quality of care in terms of ART service delivery. Figure 4 shows the annual rate of stopping ART, deaths, loss to follow up, and transfers out for each cohort. While the rate of ART stoppage and death has remained stable, there has been an increase in the rate of transfer out and loss to follow up in each successive cohort. A number of districts have dedicated transfer out forms, developed specifically to track and to achieve an overview of transfers out versus LTFU. There is also a standard SAG form for down referral, created to support the transition to PHC and nurse-provider lead ART provision. One USAID/SOUTH AFRICA PEPFAR TREATMENT PROGRAM: FINAL EVALUATION 31 PEPFAR partner best practice observed during the evaluation is that of Africare, which provides a formal transfer form/letter when its ART patients are down referred to PHC clinics; a copy of each patient‘s file goes to the PHC, while the original is retained at the higher level health facility. Figure 4. Annual Rate of Cohort Losses 0 0,02 0,04 0,06 0,08 0,1 0,12 0,14 0,16 36 mt h 24 mth 12 mth 6 mt h Annual Rate of Cohort Losses Stopped ART Died Transfer out Lost to FU Source: USAID It should be noted that supply and demand-side inputs from PEPFAR have facilitated deeper integration of focus on retention and understanding of its role in quality service delivery. Interventions such as patient literacy, patient advocates, community health workers, and M2M all have the potential to have a positive impact on retention rates. Adherence clubs represent a key aspect of retention in the move toward chronic care management: PEPFAR partners such as Kheth‘Impilo have facilitated their development, as was seen during a site visit to Wallacedene PHC in Western Cape. Training to health workers in, for example, ART and HIV management, provide an opportunity to enhance ART service delivery, with potential for positive impacts on retention rates. A further finding is that as TB is a major cause of ART program deaths, efforts to integrate HIV and TB care represents a positive strategy. SOW Question 7: Did the program take a population-based approach that emphasizes coverage and reach of ART services? The program was unable to initiate such approaches, due to the context of the times in 2004. An emergency intervention, as was PEPFAR in Phase I, could not have been expected to use population-based models. A population-based approach requires baseline data and close planning at national and lower levels, with attention to geographical, epidemiological, societal, gender, and other key factors. Such inputs were not available to PEPFAR partners as they initiated implementation. In the light of the SAG‘s move toward district-level implementation of ART and the initiatives included in the 2009 PEPFAR Country Operational Plan, there are now opportunities to address more population-based approaches. 32 USAID/SOUTH AFRICA PEPFAR TREATMENT PROGRAM: FINAL EVALUATION Such approaches should be informed by epidemiological criteria that balance the needs of the general population with those of the so-called ―drivers of the epidemic,‖ those known as MARPs in U.S. Government parlance – the most-at-risk populations, such as sex workers, migrants, and illegal immigrants, and men who have sex with men (MSM). In addition, there is currently a considerable need to integrate prevention and treatment perspectives in this area, with the inclusion of gender-sensitive approaches that provide services to both males and females. There exists experience in depth among PEPFAR partners in this area that can readily respond to the focused demands of the SAG. SOW Question 8: Did the program accurately account for its progress with respect to numbers reached? 9 Delivering carefully measured results is one of the seven guiding principles of the South African PEPFAR program. As such, PEPFAR requires all its ART program partners to report, on a quarterly basis, on a limited set of program monitoring indicators that are used to track the progress of ART program activities. The U.S. Government in South Africa evaluates ART program progress by monitoring the selected indicators and assessing these in relation to the targets and overall objectives set by ART program staff. To provide partners with standardized definitions and guidelines, PEPFAR developed a South Africa Strategic Information manual for PEPFAR 1 and PEPFAR 2 (next generation) indicators. To enhance the level of detail and hence the precision of reported ART data, PEPFAR set age and sex-disaggregated targets for reporting ART indicators. To establish if partners‘ reported data met the minimum standards for data quality, PEPFAR incorporated external DQAs into partners‘ performance cycles from 2005.The DQAs resulted not only in identifying data quality issues, but served to build the M&E capacity of PEPFAR partners and affiliated NDOH health facilities, with the objective of ensuring that quantitative results reported to the Office of the Global AIDS Coordinator (OGAC) are valid and reliable. Since 2009 PEPFAR has funded RDQAs at the district, PEPFAR partner, and facility levels. RDQAs are internal self-assessment processes where districts, partners, and facilities lead the DQA process and thus theoretically take ownership of their data quality improvement plans – with the proviso that their data management capacity may limit effectiveness. There has been some improvement over time with regard to absolute numbers reached in terms of verification. The incompleteness and imprecision of such data is a national issue with regard to numbers reached; SAG data are also fallible. The recent HCT campaign experienced such challenges: how to differentiate between first time and repeat testers and a lack of clarity on the need to count per test, rather than per person. Another issue in this context is the imprecise definition of what constitutes indirect and direct PEPFAR support and the attribution of numbers to each; PEPFAR deems all support to be direct and counts all numbers as such. Partners have previously been pushed by funding imperatives linked to head count-based performance. Thus PEPFAR, and by extension the NDOH, have a major problem in obtaining an accurate overview of total numbers on ART due to issues of double counting and lack of a clear validation process at the various levels of reporting. While facilities strive to avoid double counting, poor understanding of indicators and an inadequately designed and used data tool compromises accurate reporting. In addition, acquiring or establishing the correct denominators to calculate rates for some indicators has not been easy. 9 See above, in Section III for further discussion of this topic. USAID/SOUTH AFRICA PEPFAR TREATMENT PROGRAM: FINAL EVALUATION 33 Moreover, upwards of 40 patient management systems linked to PEPFAR were developed by partners, leading to incoherence of reported data. Although patient tracking within a facility improved somewhat with the advent of patient management systems and training of data management staff, tracking of patients between facilities has been problematic. This has led to double counting of clients who were transferred out of facilities. PEPFAR has made efforts to address double counting. At the service delivery level partners are advised to track patients‘ progress in the program and report on the number of individuals reached, regardless of the number of visits made. PEPFAR also developed an electronic, web￾based partner reporting system (Data Warehouse) to which PEPFAR partners submit their plans and reports. The issue of double counting may be reduced following the alignment of PEPFAR partners by districts. The 3-tier strategy of monitoring the national ART program and the collaboration of PEPFAR and SAG is intended to strengthen routine and clinical monitoring of data used for ART patient management, with the objective of achieving enhanced, accurate reporting. The evaluation team found various forms of data quality checks being implemented at the service provider level, including data staff checking on one another‘s work; an error record for identification of training gaps; an M&E coordinator verifying information with the ARV nurses monthly; and senior management level validation of data by M&E unit. However, these best practices were not institutionalized as standard. SOW Question 9: Did the program design and implement its activities with transparency, appropriately sharing plans, and progress with counterparts at all levels? There appears to have been ―selective‖ internal and external transparency from the outset of the PEPFAR South Africa ART program in 2004. No criteria existed in PEPFAR 1 regarding partners‘ obligations to plan for transparency, mutual accountability, and joint stewardship. In this context, it should be noted that there was no apparent criterion in Phase 1 from the SAG side for such planning. Lack of transparency continues to be a difficult issue for government partners, according to discussions during the evaluation. Thus a frequent remark from SAG partners is that there has always been, and continues to be, a significant lack of transparency from PEPFAR with regard to the overall budget and individual partners‘ annual budget allocations. Another much cited example of deep current concern is the perceived insufficient planning and discussion on the almost completed PEPFAR district alignment process and the ongoing transition of PEPFAR implementing partners from a predominantly direct support model to one that prioritizes TA. Moreover, ART program design and implementation processes were often initially driven by partners, with provincial and district partners only recently having significant inputs. While it is acknowledged that internal South African debates and positions were instrumental in shaping such imperatives, the result has been a certain degree of continued lack of transparency. However, there is now palpable demonstration of a significant shift toward embedded SAG stewardship and ownership. This has required PEPFAR partners to re-assess their roles and responsibilities, not only vis-à-vis government partners, but also in terms of overall programming. Good examples of this closer and more equal relationship are those of Broadreach, MATCH, TB/HIV Association, and Kheth‘Impilo models, where each partner in different ways has embraced the more district-focused approach and the move toward sustainability. 34 USAID/SOUTH AFRICA PEPFAR TREATMENT PROGRAM: FINAL EVALUATION There is also an onus on government to lead in this context: There is an overall lack of transparency from the donor community in South Africa with regard to financial inputs and overall outcomes. In other words, this situation is not unique to PEPFAR, although its size and influence render its position on this topic more visible. SOW Question 10: Did the program invest wisely in sustainable solutions that support South Africa in its current and future responsibilities for ART provision? Due to the political environment at the beginning of the PEPFAR ART program in April 2004, non-governmental organizations became key implementing partners and interventions were developed without much if any engagement with government structures. PEPFAR funded implementing partners to such an extent that some observers consider the NGO market as significantly distorted. An issue here is that if a PEPFAR partner has more budget than the provincial DOHs, then power relations become skewed – and some PEPFAR implementing partners have budgets of upwards of $30 million per annum. Such large partners are now profoundly uneasy about the implications of the alignment to PHC model (budgetary concerns and also in terms of that model‘s very sustainability). The emergency approach did not facilitate prioritization of sustainable approaches and solutions. Furthermore, there was no engagement with the SAG at the beginning, to provide a baseline, to undertake health facility and catchment area mapping, needs assessments from both supply and demand sides, and other such initial health systems strengthening (HSS) building blocks that would have supported movement toward sustainability. Therefore, each PEPFAR partner worked largely in isolation from 2004, with at times considerable duplication and competition and without the best match of inputs with needs. This situation has now changed, in certain regards considerably, e.g., with the 2006 burden of disease study in Western Cape, which has facilitated a needs-based and public health approach to overall planning, into which PEPFAR partners have the opportunity to make timely and jointly planned inputs. PEPFAR is increasingly supporting moves toward sustainability, notably via the PEPFAR alignment model, where implementing partners have been obliged to re-allocate their support to optimize geographical coverage and to move away from earlier circumstances where several partners could be supporting one health facility and/or be over-represented in one district, while neighboring districts might be entirely lacking in assistance. In addition, sustainability is a moving target and those targets and objectives that applied in 2004 will no longer be entirely relevant, or will have been subsumed within others that now have greater validity, as ART provision responds to the changing epidemiological environment. More sustainable inputs include capacity development, infrastructural developments, support to development of SAG guidelines, clinical and other studies (e.g., the costing study), etc. USAID/SOUTH AFRICA PEPFAR TREATMENT PROGRAM: FINAL EVALUATION 35 SOW FORWARD-LOOKING POINTS SOW Forward-looking Point 1: Outstanding Issues in the Policy Arena Related to ART The following outstanding policy issues were identified in the course of the evaluation: There is need for continued advocacy and policy direction for a unique identifying patient number to track and manage LTFU/transfers out effectively at the national level, with provincial buy-in (see also Annex F). More effective linkages are required at the policy and practice level between HSS and community systems‘ strengthening. There should be thorough attention at the policy level to the supply- and demand-side implications of ―test-and-treat‖ and treatment as prevention (again, see Annex F for further discussion). Quality of care, from both the supply- and demand-side perspectives, requires detailed policy direction, further development of instruments such as supportive supervision, and further attention to patients‘ health rights. More effective policy guidance on hitherto somewhat neglected aspects of ART provision is required, e.g., for adolescents and older people (both pre-ART and ART), on overall pre￾ART support and maintenance, and on support to care givers. Many key NDOH policy documents fail to adequately address gender issues in the context of ART provision and more widely in the shift to PHC focus; this is a significant policy gap (such documents include the 2010 Re-engineering PHC in South Africa and the 2011 National Care Standards and the Draft Strategy for the Health Sector 2012/13-2016/17). See Annex F for a more detailed, though brief consideration of gender aspects. SOW Forward-looking Point 2: Important Gaps that are Not Being Addressed by USAID/CDC, MOH, or Other Donors The following important gaps were identified by evaluation respondents: Greater and more focused attention to pre-initiation and pre-ART patients is required to minimize the danger of patients being lost. Dedicated support to adolescents, both pre-ART and during treatment, is essential. Dedicated support to older/old HIV+ and those who are care givers is also essential. An absence of gender mainstreaming, both internal and external, represents another significant gap. Inter and multisectoral initiatives require further attention, critical in the move toward chronic care management. NHLS data are not optimally utilized at the provincial and district level to evaluate population impact over time and to inform evidence-based planning. There is lack of coherence between DHIS and other population-based datasets and national birth and death registries. Despite site-specific best practices, there is currently insufficient use of pharmacy data systems for program management and monitoring. 36 USAID/SOUTH AFRICA PEPFAR TREATMENT PROGRAM: FINAL EVALUATION Identified gaps specific to the PEPFAR transition from direct support to technical assistance include the following: There has been inadequate discussion between PEPFAR, its implementing partners, and the N/P DOH with regard to the optimal approach for best ART service delivery – is the public health system entirely, fully ready to manage direct service delivery? Insufficient consideration has been given by PEPFAR to planning for its HSS inputs in support of the transition. Far too little attention appears to have been given to how best to maintain the close mentoring relationship developed between PEPFAR implementing partners and health workers and facilities once the transition from direct service provision to TA occurs. The point was forcefully made on a number of occasions during evaluation hub meetings and KII that mentoring is very different from TA—it is longer-term, more incremental and organic, as well as more embedded in partnership approaches. There is no M&E plan in support of the major changes being instituted by PEPFAR in terms of the transition from direct support to TA, yet it is essential to document processes as well as eventual outcomes. Each implementing partner‘s transition will be unique, which also needs to be taken into account. Otherwise lessons learned, best practices, and approaches that support sustainability will not be adequately captured and documented. There is lack of clarity—and a deep concern – over infrastructure development and maintenance in the PEPFAR transition from direct support to TA. SOW Forward-looking Point 3: Identify and Document Best Practices, Lessons Learned, and Recommendations to Inform Follow on Activities Focusing on Sustainability There is currently a considerable cultural shift away from the PEPFAR emergency phase to more HSS and chronic care management-focused approaches; this has resulted in the need for change management processes within the U.S. Government and in partners. Aspects to consider with regard to sustainability include: how might this have been instituted into implementing partners‘ ART service delivery models and what was the time frame on this? Focus on the fact that the SAG is reforming the health system to make it a PHC model – how can any sustainability lessons learned and best practices from PEPFAR inputs help achieve this? The transition to TA continues to require transparent planning and definition, as much confusion still pertains as to parameters of TA, the implications for partner inputs, and the potential impacts on overall ART service delivery. If such planning and discussion does not occur, opportunities for sustainability are likely to be compromised. The ―nerve centre meeting‖ model processes should now be closely supported documented and institutionalized as a key HSS component and a central plank of the alignment to district support. This is especially pertinent in light of the NDOH transition to PHC-level ART service delivery. USAID/SOUTH AFRICA PEPFAR TREATMENT PROGRAM: FINAL EVALUATION 37 THE TWO CDC POINTS CDC Point 1: What Is the Partners’, Government’s, and Civil Society’s Understanding of Technical Versus Direct Assistance, the Benefits, and Risks/Harms/Costs of Both as PEPFAR Transitions to the Former?10 A general evaluation finding is that most stakeholders, be they government, PEPFAR implementing partners, or other civil society actors – for example, faith-based organizations (FBOs)—accept that PEPFAR cannot and should not have a role that facilitates continued dependency. Such dependency may be linked to direct clinical support in terms of staff members, human resources, and other inputs to information systems, infrastructural support, and equipment, among others. Therefore, there is widespread recognition that the PEPFAR transition to TA represents a necessary step toward greater government ownership and stewardship of the ART program. There is linked acceptance that an overall transition to a more HSS approach and genuinely enhanced SAG ownership/stewardship is essential. The PEPFAR transition also potentially supports movement toward chronic care management, e.g., through links to CSS and sub/district management. However, a considerable degree of confusion and concern remains among many stakeholders with regard to the fit between PEPFAR transition and the SAG transition of ART to PHC level/the NIMART model. For instance, and crucially: Who will fill the inevitable clinical gaps in down referral/NIMART service delivery? The on-the-ground reality is that so many districts will still need clinical inputs, e.g., management of pediatric ART and supportive supervision/mentoring of NIMART nurses, to name just two key issues. Moreover, what is the transition timetable? This appears not to have been discussed. Where is a clear and mutually agreed definition of TA that encompasses attention to critical issues, including the following: the needs of patients and facilities in a careful move away from direct support; partners‘ comparative advantage; provincial and district differences; societal issues; coverage challenges; and the realities of health system capacities at the district and sub￾district levels? One further issue is that provinces and districts vary considerably, so the balance between continued direct support versus transition to TA may also differ. An example of this is in Western Cape, where Metro will not require anywhere near as much TA as, for example, Overburg, which needs all the direct support and TA it can get. There are also grave concerns that pre-ART and ART patients will suffer if the transition is too hasty – this challenge should be viewed within the context of such issues as coverage, retention, transfer out, and LTFU. Another area of major disquiet is the lack of clarity on the future role of non-clinical staff in the provision of TA. To give one example, a number of implementing partners have supported the training and mentoring of social workers, who have become integral, valued members of ART service delivery and support teams. What will their future role be? Which line ministry will assume overall responsibility and management of that intersectoral, cross-cutting input? In addition, how will volunteers be assimilated, as well as demand-side, community workers such as patient advocates? There are genuine fears that the complex, interwoven nature of ART service delivery that has developed since 2004 with PEPFAR support may now be in danger of being fragmented and weakened. 10 More information on this topic is included later in this section in the subsection describing PEPFAR‘s transition to TA as well as in Annex F. 38 USAID/SOUTH AFRICA PEPFAR TREATMENT PROGRAM: FINAL EVALUATION CDC Point 2: Consider the Significance of PEPFAR Support into the Medium-term Future in the Overall Context of the HIV/TB Response in South Africa This point refers to the fact that PEPFAR provides 10% of overall inputs. How relevant, necessary, and important is that support, and how can it achieve its maximum impact? The SAG was given $600 million per annum for three years from the outset of PEPFAR starting April 1, 2004, with no cost criteria attached. A somewhat ―gold-plated‖ service was instituted as standard. The challenge now is how to implement cost efficiencies, especially in line with the year-on-year reduction in the PEPFAR grant from 2012 of 5% and a further significant reduction starting in 2015? PEPFAR support remains pivotal, with major opportunities for providing effective support to transition to an HSS approach, CCM of HIV, and more focused PHC integration. In the context of supporting PEPFAR alignment and transition processes, as well as reduction of financial inputs, it is imperative to manage change effectively, transparently, and in accordance with guidance from the DOH. A one-size-fits-all approach would be inappropriate in the context of a general population epidemic, but becomes even less workable in the context of South Africa, where vulnerable groups require targeted attention and provincial and district differences must be acknowledged and addressed. NDOH CROSS-CUTTING ISSUES Introduction Evaluation attention to the three issues of coverage, efficiency, and quality has been shaped by the requirements of the NDOH in the context of the imminent RFA process, with the NDOH requesting that these three issues receive particular focus. An overall introductory comment is relevant here: PEPFAR ART partners have supported and implemented a wide range of services, i.e., not a standard model of treatment or solely that of clinical service provision. Partners have supported infrastructural upgrades at many health facilities nationwide; they have also provided and facilitated training on all aspects of clinical care, M&E, infection control, made inputs where appropriate to SAG policy development, and so on. Therefore, discussion of ―PEPFAR inputs‖ to the complexities of coverage, efficiency, and quality can only be broad brush in the context of this performance evaluation: Impacts cannot be measured. Moreover, as this evaluation covers the clinical and related components addressed in Section IV of this report, and does not address the continuum of HIV mitigation from prevention into care, treatment, and support, it cannot adequately reflect all contributions made by PEPFAR partners. The evaluation sought to assess how issues of coverage, efficiencies of scale, and quality are being addressed in the environment of the SAG district and PHC methodology11 and with acknowledgement of SAG attention to burden of disease, social determinants of health, and MDG imperatives. In addition, key instruments such as the National Service Delivery Agreement have been factored into the discussion. 11See NDOH 2010: ―Re-engineering Primary Health Care in South Africa.‖ USAID/SOUTH AFRICA PEPFAR TREATMENT PROGRAM: FINAL EVALUATION 39 Coverage Definitions The discussion of coverage earlier in this section links into SOW Question 1 (reduction of the treatment gap); issues such as whether any key populations are falling behind within overall coverage should be addressed in any such discussion. These key populations include children, commercial sex workers, truckers, MSM, and women having sex with women (WSW). There are many, varied definitions of coverage, including per population, geographically, and per service rendered. The working definition for this report is grounded in demand-side perspectives, where potential and existing patients can gain timely access to relevant ART and linked services, without undue constraints and opportunity costs – in other words, service delivery coverage seeks to match need and demand created by initiation of services. This definition is based on evaluation stakeholders‘ feedback; it is also sensitive to the SAG imperative to ―take the services to the people‖ and to initiatives relating to, for example, e.g., male medical circumcision (MMC) and the HCT campaign. It also answers to the more traditional epidemiological perception of coverage, where services are theoretically focused where need is greatest – with due acknowledgement that the epidemic is within the general population in South Africa. Further acknowledgement is given to the fact that movement into genuinely widespread coverage, e.g., into the deeper rural areas and further into workplace and other group-targeted interventions, remains a work in progress. Many issues such as matching economies of scale with equity of access to services continue to be discussed and addressed. A recent SAG target for coverage was that 80% of those needing ART would be reached (the definition of ―reached‖ was not provided, but does not seem to refer to actually being tested) by 2010.The target appears to have been achieved due to the HCT campaign, changes in government guidelines, and inputs by partners such as PEPFAR. Achievements A truly major achievement of PEPFAR implementing partners is that their staffing capacity has allowed them to achieve greater coverage in terms of provision of ART than would otherwise have been possible. Such coverage was made possible due to PEPFAR funding from April 2004. A further, more recent achievement has been support for creation and maintenance of roving teams, in the context of the move toward district-level provision of ART. While these teams may represent a stopgap measure in terms of individual visits and health facilities, they nonetheless provide services where none would otherwise be available. In addition, the roving x-ray machines enhance coverage. PEPFAR support to PMTCT has undoubtedly enhanced coverage. PMTCT services represent a key entry point to ART services, not only for infants, but also for mothers and on occasion for fathers as well. Challenges One major, perennial constraint for SAG and PEPFAR partners is difficulty recruiting and retaining health workers for service provision in rural areas. Therefore, in an overall situation of grave HRH issues (perhaps especially for doctors and pharmacists), there is limited opportunity to create and maintain an enabling environment to supports genuinely complete coverage. There are also epidemiological constraints that may be beyond the capacities of any partner to resolve, whether government, PEPFAR, or other. The maturing epidemiological profile linked to the absence of ART in the 1990s and early 2000s and current infection rates present challenges 40 USAID/SOUTH AFRICA PEPFAR TREATMENT PROGRAM: FINAL EVALUATION in terms of numbers that will require treatment. Optimal coverage may continue to be an objective rather than a complete achievement. The forecast is that 3.5 million people will be on ART by 2017, at which point it is estimated that the epidemic may plateau: this is based on increased prevalence and more people presenting for ART with a CD4 count greater than 100. A further challenge related to precise definitions of coverage is insufficient accuracy in terms of numbers of people on treatment. There are also issues linked to PEPFAR alignment, in that as ART patients are transferred from partner to SAG services, there may be absorptive constraints that result in some patients being lost (such fears have been expressed by the South African Catholic Bishops‘ Conference). Such losses would compromise coverage, as well as raise fundamental questions of rights related to health care, equity in service provision, and continued support. Recommendations 1. Provide more target group- and location-specific interventions – e.g., education and other prevention, HIV care): at work places, in the community – as further routes into increasing coverage. 2. In addition, further develop and implement programs that reach MARPs, including migrant labor, illegal immigrants, OVCs, and deep rural populations. 3. Consider a greater role for private GPs, possibly linked into the eventual National Health Insurance scheme rollout. Private GPs often cater for professional groups, e.g., health workers, teachers, etc, who are unlikely to visit local clinics. Efficiency As per the SOW clarification, found in Annex B, efficiency in the context of this evaluation does not refer to financial and health economics-based models of efficiencies, as such a study is beyond the scope of this evaluation. Rather, it refers to issues such as chronic care management and the service delivery and other efficiencies potentially or actually associated with this model. In this context primary attention should be focused on lessons learned. In addition, the opportunity costs for all those eligible for ART should be borne in mind. The evaluation team decided to define efficiency as seen through a health systems‘ strengthening lens, based on discussion with a wide range of stakeholders. Close attention was not given to financial management, cost effectiveness and efficiencies, cost-benefit analyses, etc. Therefore, training and more coherent and functioning systems—including data management, patient flows, appropriate matching of service provision with need, task shifting, and distance specialty support—were considered. Achievements PEPFAR partners have contributed significantly to this efficiency component by matching provision of ART services with patients‘ needs at the most appropriate level. Patients are now being given ARVs for upwards of three months. This approach speaks to both efficiencies of services and scale in a resource-constrained environment as well as to the delivery of responsive care. Challenges The SAG is seeking efficiencies of scale through its district and PHC models. PEPFAR implementing partners have been aligned at district level and are in the process of transferring from primarily direct ART service provision to a greater focus on TA. All such processes require a well-managed transition and change management focus, as well as an entirely realistic USAID/SOUTH AFRICA PEPFAR TREATMENT PROGRAM: FINAL EVALUATION 41 evaluation of HRH and other constraints. The danger is otherwise that district and sub-district health systems will be unable to cope. It is at the point of transition to districts where there are the most challenges in terms of efficiency of transfer to the DOH and to districts: if efficient systems are not transferred or supported during that phase, along with ownership and stewardship, the very building blocks of HSS will be compromised. Gaps There appear not to have been uniform national or provincial cost-benefit analyses undertaken to support the transition to the district and PHC/nurse-initiated ART model; as a result, no nationwide mapping of health resources or evidence-based planning has been done. This absence has implications in terms of effective mapping of need vis-à-vis coverage and also has a bearing on the most efficient allocation and use of scarce human, financial, and other resources. In addition, it has major implications with regard to PEPFAR partners‘ proposals and the future direction of the overall transition. Recommendations 1. ART provision is in an era of major change: careful joint planning is essential to enhance efficiency. 2. Institute a scheduled, properly managed transition to a TA focus (doing this too quickly and without proper planning may result in patient loss). Key within the transition are HSS linkages and strong down and up referral systems, all of which support strengthened service delivery efficiencies. Quality Achievements The clinical and supply side components of quality of care have been closely addressed under the PEPFAR ART program, as have viral load suppression and program retention. There has been attention to the quality of training provision. At the same time, quality improvement programs were not an initial priority of PEPFAR, with ongoing repercussions. Gaps Demand-side (i.e., patient/client and community aspects) quality criteria have not always been universally addressed within the PEPFAR ART program, despite much focus from a number of partners, e.g., Anova and Kheth‘Impilo. Challenges A major quality issue is how most equitably and effectively to balance quantity and quality of service delivery, from both the supply and demand sides. Attention to such matters is essential as decentralization and alignment proceed and as PEPFAR implementing partners make their transition to TA. The question is how to best to balance the needs for expanded coverage and increased delivery quality. Recommendations 1. There should be greater coherence of PEPFAR quality criteria with national requirements. 2. Closer attention should be paid to issues of quality in the context of clinical care provision: overall PEPFAR attention should be increased with regard to key indicators of quality, e.g., tracking, adherence, action on LTFU, and support to pre-ART. 42 USAID/SOUTH AFRICA PEPFAR TREATMENT PROGRAM: FINAL EVALUATION 3. Enhanced responsiveness to demand-side perspectives should be a priority: This is a core criterion for quality of care and one that has received insufficient attention overall (with notable exceptions from civil society and others). THE PEPFAR TRANSITION TO TA12 The transition required for implementing partners to move from a primarily direct ART service provision approach to one of predominantly technical assistance was clearly an issue of considerable concern for many of the informants interviewed during the evaluation. While most partners were able to understand the rationale for the district alignment (see the following subsection); as individually challenging and fraught as that process might be, the same cannot currently be said for the transition. DOH personnel at the provincial level were similarly confused and concerned. The discussion below indicates that a number of concerns were put forward on the optimal (and, where appropriate, cross-cutting) management of the three major processes currently underway – the SAG transition to a primarily PHC service delivery model, the linked alignment process at the district health level for PEPFAR implementing partners, and the implementing partner transition to TA. Key points raised by implementing partners (and on occasion also by DOH personnel and other key informants) were as follows: Does anyone within PEPFAR know the exact PEPFAR transition timetable? If so, this should be tabled, discussed, and agreed upon. There was frequent mention of the current confusion over how best to harmonize the district-level alignment with PEPFAR‘s transition to TA. This refers to issues such as PEPFAR partners having previously provided direct clinical and other support to health facilities, often in rural areas, where DOH staff members continue to be reluctant to serve. Now that the transition to TA is under way, service delivery gaps are said to have arisen, e.g., in one health facility visited in Eastern Cape. Thus patient care may be interrupted or disrupted in certain instances due to PEPFAR‘s transition to TA, especially in light of the parallel transition to PHC ART service delivery. The transition to TA is considered by a number of evaluation respondents not to have prioritized demand-side quality criteria. One example given was the possibility that mentoring, previously a significant component in many implementing partners‘ direct service provision, could be watered down and become less consistent once TA is fully implemented. Mentoring has included a strong focus on quality assurance, incorporating attention to patients‘ own criteria and definitions. The question is how this can continue to be a priority in the TA approach, perceived by many as requiring less consistent inputs. DOH personnel raised the additional key points: PEPFAR is seen as having failed to properly discuss with the SAG its transition from direct support to TA or the DOH transition to PHC service delivery of ART. There is a lack of understanding and certainly no strategic mapping on the government side on the best way for PEPFAR to support the transition to PHC and what the implications may be for its own transition processes. This area of debate links into the wider question of PEPFAR‘s lack of transparency over HRH. Districts and provinces are expected to absorb health workers (but apparently not ancillary human resources such as social and community outreach workers). However, they have not 12 See also the discussions appearing earlier in this section, particularly regarding the SOW in Forward￾looking Point 2) and CDC Point 1. USAID/SOUTH AFRICA PEPFAR TREATMENT PROGRAM: FINAL EVALUATION 43 been involved in earlier discussion or decisions on how many permanent staff members and visiting technical support staff there are or should be at any one health facility or in individual sub-districts or districts. Another HRH issue is that many staff members currently employed by implementing partners absolutely do not wish to transfer to the public sector. Professional development opportunities, mentoring and supportive supervision, and working environments are said to be far more attractive for employees of implementing partners than for government workers. Key Recommendations 1. Essential areas for continued direct support by implementing partners should be considered, e.g., pediatric ART, optimal TB/HIV integration, targeted clinical research, close mentoring of NIMART nurses. 2. Change management approaches are essential in the transition process and are still timely, despite the ongoing rollout. 3. A key area for support during and beyond transition: health manager training at district and sub-district levels. Further Recommendations Advocate for competence-based recruitment and performance-based management Advocate also for required service contract Perhaps consider re-introduction/expansion of the additional community service years currently applied to medical students– extend to pharmacists, etc., to address shortages, especially in rural health facilities THE PEPFAR ALIGNMENT: LESSONS LEARNED The PEPFAR alignment process has been closely linked to the SAG re-engineering of the primary health care system, which will consolidate PHC as the primary mode of health care delivery, focusing on prevention of disease and promotion of good health. The alignment process is estimated to be 90% complete. The impetus behind the PEPFAR alignment (known to many PEPFAR implementing partners, DOH managers, and staff members as ―the re-alignment‖) was seen by most evaluation respondents as seeking to ensure that partners achieve more equitable, widespread coverage in terms of ART provision, through attention to more balanced geographical spread. In the past, it has been the case that two or more PEPFAR partners have operated in the same district or sub￾district, and occasionally even in the same health facility. Such lack of effective planning to ensure optimal ART service provision in a resource-poor setting has resulted in patchy coverage and support, frequently exacerbated by the sometimes extreme HRH constraints prevailing in the public health sector. Most PEPFAR partners, DOH representatives, and others met by the team in the course of the evaluation have accepted the broad thrust of the alignment. The alignment‘s logic – to institute and sustain more consistent provision of PHC services in the closest possible partnership with the SAG and N/PDOH structures and approaches – is one that resonates with organizations with considerable experience in the provision of health services in settings that are often challenging. An overall alignment perspective is provided by a number of PEPFAR implementing partners in KZN, whose view is that the alignment, although painful and in many instances severing deep relationships built up over time with health facilities, will demonstrably make ART service 44 USAID/SOUTH AFRICA PEPFAR TREATMENT PROGRAM: FINAL EVALUATION delivery more effective in that both partners‘ and districts‘ capacities can be matched more appropriately. A somewhat more negative view is that the PEPFAR alignment process is ―very cosmetic‖; while the intention that implementing partners and sub-partners no longer work in the same district is appropriate, as this does not enhance optimal coverage, there continues to be much concern from all sides as to eventual outcomes. In addition, the overall sub-district ―lead partner‖ approach is disliked, especially as partners are said to have had minimal say in decision making. A further concern noted by implementing partners was the degree to which the alignment may have adopted a ―one-size-fits-all‖ approach, yet districts and sub-districts within the same province, let alone within South Africa as a whole, vary considerably with regard to ART service delivery capacity and needs, again from both supply and demand sides. Faith-based organizations that are PEPFAR implementing partners noted deep concern over the following matters in connection with the alignment, matters that continue to need attention. ART patients have been transferred from FBO health centers to public sector clinics as part of the alignment. While public sector clinics can serve large numbers of patients and achieve economies of scale, the closure of faith-based and other NGO health centers means that all patients must be seen at public clinics. There are problems with this model, including capacity shortage at some government clinics, limited availability of clinics for persons who work during the day, and discomfort with the use of government clinics on the part of vulnerable patient groups such as migrants who note a stigmatizing attitude on the part of the nursing staff. Some patients reported being required to provide identifying documents to be seen and being specifically excluded from care. The faith-based clinics also noted that HIV-infected health care workers will not go to government clinics for fear of being identified. The PEPFAR Alignment: Potential Lessons Learned There was lack of clarity on planning and overall lead of the alignment. There is perceived to be an overall lack of transparency (these views were expressed by provincial DOH staff and employees and volunteers of PEPFAR implementing partners). If there is to be genuine SAG ownership and stewardship at all levels of the health system, as well as appropriate governance and accountability, then engagement with district and sub￾district structures is essential (e.g., joint development and implementation of SOW for district alignment). Joint PEPFAR/DOH assessments should have been undertaken in advance of the alignment - No needs assessment was said to have been carried out for both the supply and demand side or any HRH capacity assessment. It was mentioned that this apparent lack of an evidence base may have implications for achieving the most appropriate and effective coverage of services, geographically, epidemiologically and in societal terms, e.g., attention to MARPs. What have partners‘ experiences been when exiting a district? Have there been processes to evaluate the exit and measured outcomes? Where has the alignment been documented? The decisions taken for alignment of PEPFAR partners have not always met ART service needs: - The PEPFAR inventory was said to have formed the basis (perhaps also justification) for the alignment, yet that process was described as ―abominable‖ and ―deeply flawed.‖ The overall perception expressed was that there was either no or at best inadequate USAID/SOUTH AFRICA PEPFAR TREATMENT PROGRAM: FINAL EVALUATION 45 participatory alignment planning; ideally, all relevant stakeholders should have worked together to plan for alignment. The view was expressed that both PEPFAR and the NDOH have shown a lack of interest in the slow, incremental, learning process that should be integral to such a major change. Inadequate attention was given to demand-side criteria: - Attention to demand-side quality criteria is essential in chronic care management approaches to HIV and ART – and these approaches are widely considered to be appropriate in the current ART context in South Africa. - The needs of the most vulnerable and marginalized may not have been adequately considered. Cost effectiveness criteria may have been too dominant in the alignment process: – Cost effectiveness in the context of the alignment was described by several implementing partners as equal to the DOH retrenchment of staff. This was defined as ―delegation by neglect.‖ If there is to be retrenchment, then a detailed, gender-aware HRH strategy is essential (see also Annex F). There cannot merely be an expectation that lower cadres (nurses) will shoulder most of the PHC burden at the district and sub-district levels. 46 USAID/SOUTH AFRICA PEPFAR TREATMENT PROGRAM: FINAL EVALUATION USAID/SOUTH AFRICA PEPFAR TREATMENT PROGRAM: FINAL EVALUATION 47 ANNEX A. SCOPE OF WORK PEPFAR/SA Treatment Program Evaluation Statement of Work (Revised: 05-31-11) I. SUMMARY This Statement of Work sets forth guidelines for a performance evaluation of the USG/PEPFAR anti-retroviral treatment (ART) program in South Africa and serves as the basis from which the USG will build upon past efforts while creating new avenues and directions for support through the South African Government. It is proposed that this evaluation commence in June 2011. Since 2004, USG has supported a range of approaches aimed to increase access to and to improve the quality of ART programs in South Africa. Although much progress has been made, the USG is preparing for a potentially significant revision of its implementation strategy for the next five years in accordance with the Partnership Framework mandate that emphasizes country ownership and long-term sustainability. As such, there will be an incremental phasing over of program management including funding, human resources and technical assistance. An important first step in this new direction is the recent request by the South African Government (SAG) for an evaluation of the USG ART program. In response, the USG is coordinating an evaluation team to review the achievements and lessons learned of its ART investments to date and to identify future opportunities focused on sustainability. II. BACKGROUND The PEPFAR/South Africa ART program was developed in 2004 by the principal USG Agencies (USAID and CDC) receiving PEPFAR funding in South Africa. The two agencies manage the implementation of the ART program through a variety of international and South African organizations (henceforth termed ―implementing partners. The purpose of the program has been to support comprehensive clinical HIV-related services in-line with South African national, provincial, and district policies, standards, and implementation plans. The program has continued to build and capitalize on the accomplishments and lessons learned since the South Africa national ARV rollout began in 2004. Implementing partners have provided support to facilities in the following areas: Adult Care and Treatment services; Pediatric Care and Treatment services; PICT/HCT; Prevention from Mother to Child Transmission; TB/HIV; Prevention with Positives; Male Circumcision (where designated by the Province); Post-exposure prophylaxis access and provision; Cervical cancer screening; Linkage to community social mobilization and health promotion efforts and facility-based services to ensure effective implementation of referral, tracking, and adherence strategies; Linkages to Sexually Transmitted Infections, Maternal and Child Health and reproductive health services; and 48 USAID/SOUTH AFRICA PEPFAR TREATMENT PROGRAM: FINAL EVALUATION District and Provincial management capacity building. III. RECENT ACTIVITIES By the end of March 2010, the number of PEPFAR-supported individuals receiving ART had grown to 832,800 with approximately 84,000 individuals having newly enrolled on treatment by September 2010.Based upon recent trends and quarterly results, PEPFAR SA anticipates the scale-up of treatment will continue during FY 2011 with an estimated 1,500,000 individuals on ART by the end of the FY 2011 implementation period. These results are based on PEPFAR monitoring guidance that informs how implementing partners are able to ―count‖ the numbers of individuals supported. The overall PEPFAR/South Africa ART program objectives for FY 2011 are to support the SAG in providing antiretroviral treatment at Primary Health Care clinics, transition the Track 1 ART program, strengthen the NDOH capacity to hire and retain staff including implementing Nurse Initiated Management of ART, conduct the extensive HCT campaign to test 15 million South Africans for HIV by June 2011, and effectively link individuals to care and treatment services when they test positive and initiating ART earlier. In addition, the PEPFAR/South Africa program in South Africa will support SAG with the implementation of the revised treatment guidelines introduced in the Presidential announcement on World AIDS Day 2009.The revised eligibility criteria allow patients to be eligible earlier for treatment at all government clinics and hospitals: 1. HIV positive pregnant women with a CD4 count ≤ 350 cells/mm³ 2. HIV positive pregnant women receive PMTCT regimens at 14 weeks 3. HIV positive infants (from birth to one year irrespective of their CD4 count) 4. Individuals co-infected with HIV/AIDS and TB with a CD4 count ≤ 350 5. HIV positive individuals with a CD4 count ≤ 200. IV. PROPOSED SCOPE OF WORK The main purpose of this evaluation will focus on assessing the effectiveness, efficiency, quality and sustainability of the USG ART program interventions; identifying what has been successfully incorporated into the NDOH‘s ongoing programs and what challenges remain; establish evidence of project results and impact and provide lessons and recommendations for the planning and management of future interventions that focus on HIV/AIDS treatment. Drawing on international literature and experience, the evaluation team will review existing documentation on USG support for HIV treatment services in South Africa and meet with a variety of implementers and governmental officials, both at the national and local levels. Building on previous evaluations and strategies, and based on consultations with USAID, CDC, the NDOH, SANAC, and other relevant organizations, the team will review USG efforts to date with a view to documenting successes and lessons learned. In addition, the team will provide recommendations on remaining gaps and needs for consideration in future programming. This would include conceptualizing the overall USG framework for addressing ART issues in South Africa and the principal programmatic and management structure(s) to achieve impact. The team will consider the specific environment in which USAID and CDC support is given, with a special focus on contributions to national program achievements. Key interview questions, methods, and tools: The Evaluation Team will primarily rely on an efficient structured questionnaire as a guide for discussions with key respondents. At the discretion of the Team Leader, USG and NDOH staff may be asked to excuse themselves from certain interviews to allow the team to collect the necessary data. Following each day of interviews the Evaluation Teams will meet separately to summarize key points and issues USAID/SOUTH AFRICA PEPFAR TREATMENT PROGRAM: FINAL EVALUATION 49 introduced during the interviews. For this purpose, the Team will use an interview summary form for each interview and site visited. At the end of each week of site visits, the Team will meet together to summarize findings associated with their site visits and to work incrementally on the draft of the evaluation report. This is a brief summary of the process; it is expected that the Team will elucidate and fully outline its methodology before the work commences. The objectives of this evaluation include: Whether programmatic objectives were achieved Determine lessons learned from the program Evaluate sustainability of the program Identify any future programming opportunities In addressing these objectives the following key questions will be addressed: Did the program: 1. Help to substantially reduce the estimated Treatment Gap? 2. Support the SAG to develop recognized standard public health practice for ART? 3. Play a lead role in reducing costs of delivering quality ART services? 4. Capacitate cadres of health workers to ensure a sustainable program? 5. Enhance the SAG‘s ability to monitor the progress of its ART program? 6. Increase overall retention rates and decrease mortality rates? 7. Take a population-based approach that emphasizes coverage and reach of ART services? 8. Accurately account for its progress with respect to numbers reached? 9. Design and implement its activities with transparency, appropriately sharing plans and progress with counterparts at all levels? 10. Invest wisely in sustainable solutions that support South Africa in its current and future responsibilities for ART provision? For a forward look, the evaluation should consider: Outstanding issues in the policy arena related to ART Important gaps that are not being addressed by either USAID/CDC, MOH, or other donors Identify and document best practices, lessons learned and recommendations to inform follow on activities focusing on sustainability. To achieve these objectives, the USG requires a team of four people, split into two teams (Team A and Team B) to conduct a performance evaluation of the treatment program. The two teams would be based in two key provinces, Gauteng and KwaZulu-Natal, and would work concurrently. The team covering Gauteng could potentially also cover a few Mpumalanga sites. It is expected that the two teams will each be able to visit at least two clinics per day with the expectation that a total of 25 clinics (15 PEPFAR sites and 10 DOH clinics) will be visited as observation sites. V. TEAM COMPOSITION The Team shall consist of a mix of international and local health experts with expertise in HIV/AIDS care and treatment, health systems management, and community systems of care. Combined, the team should have expertise in monitoring and evaluating large-scale programs, and should fully understand the role of health systems strengthening in PEPFAR ART programs. A representative from USAID, CDC, and the National Department of Health will join the team 50 USAID/SOUTH AFRICA PEPFAR TREATMENT PROGRAM: FINAL EVALUATION during field visits to provide an in-country perspective. These individuals will not participate in the writing of the draft or final reports and will be responsible for their own logistics and expenses. The Team Leader will be a consultant with extensive experience in HIV/AIDS, prevention, care, and treatment expertise and a person with proven experience as a team leader for evaluations. In addition, the team leader should be fluent in English and have strong communication, writing, and presentation skills. The Team Leader will hold meetings with the other three core team members, key representatives from USAID/Southern Africa, Health Office, CDC, and the National Department of Health staff prior to the commencement of the evaluation. The Team Leader will: Finalize the work plan for the assignment Establish assignment roles, responsibilities, and tasks for the members of the team Ensure that the logistics arrangements in the field are complete with assistance from USAID Health Office and CDC Office Facilitate the Team Planning meeting Take the lead on preparing, coordinating team members‘ input, submitting, revising, and finalizing the assignment report Manage the process of writing the final report Manage team coordination meetings in the field Coordinate the workflow, team tasks and ensure that the team schedule works; and Ensure that the team field logistics are arranged. The Evaluation team members should each have an advanced degree in health and 15 years of experience in their specialties. In addition, each team member should have, at minimum, the following skills and experience 1. Demonstrated skill in written and oral communication. 2. Demonstrated knowledge of international HIV/AIDS mitigation approaches, Treatment and Care including strategies for health systems strengthening and promoting host-country ownership of programs. 3. Working in Southern Africa 4. Ability to work effectively in, and communicate with a diverse set of professionals. 5. Excellent English language skills (both written and verbal) This scope envisions two international team members and a minimum of two South African consultants as members of the team. In addition to the team members described above, a logistics coordinator will be contracted to assist the team with all logistical matters including setting up key informant interviews, arranging travel, arranging site visits, etc to ensure that the team leader‘s attention will not diverted from focusing on technical substantive matters. USAID/SOUTH AFRICA PEPFAR TREATMENT PROGRAM: FINAL EVALUATION 51 VI. LEVEL OF EFFORT (IN NUMBER OF DAYS) Task Team Leader Team Member 2 Team Member 3 Team Member 4 Logistics Coordinator Potential Dates (illustrative) Background Document Review/ background work 3 3 3 3 5 June 1 -3 Travel to country 2 2 0 0 June 8-9 Team Planning Meeting; USAID/CDC briefings (following TPM, travel to the fieldwork location) 2 2 2 2 2 June 10 - 11 Meetings and Interviews with Key Stakeholders and Field Visits 15 15 15 15 15 June 13 – June 29 Analysis and writing draft report 4 4 4 4 June 30 – July 4 Debrief and submission of draft report 1 1 1 1 July 5 Depart country 2 2 0 0 July 6 -7 USAID/CDC have 14 working days to submit written comments 0 0 0 0 July 26 Revision/finalization of report based on USAID/CDC comments 5 1 1 1 July 27 – Aug. 2 USAID/CDC approval of final report content 0 0 0 0 Aug. 5 GH Tech has final report edited and formatted for submission (approx 30 working days) 0 0 0 0 Sept. 9 TOTAL LOE 34 days 29 days 26 days 26 days 22 days Note: A 6-day work week is authorized while team members are working in country. 52 USAID/SOUTH AFRICA PEPFAR TREATMENT PROGRAM: FINAL EVALUATION VII. TIMELINE The entire review should be completed in approximately 5 weeks. This would include preparation days, in-country work in Pretoria and the two main provinces selected for field observations, and report writing and finalization. The tentative start date is June 10, 2011 in Pretoria. VIII. LOGISTICS The evaluation team, in collaboration with the staff of National Department of Health and USAID/South Africa and CDC will arrange all meetings, interviews, site visits, in briefing and out briefing in advance. USAID/Southern Africa and CDC will provide contact information for suggested interlocutors but in all other respects the evaluation team should be self-sufficient. IX. DELIVERABLES 1. Pre-trip Background Document Reading/Briefing: Prior to arrival, Team Leader and Evaluation team will review all relevant documentation and schedule a conference call with USAID/Southern Africa, CDC, and National Department of Health. 2. Team Planning Meeting and Work Plan: After the Team Leader‘s arrival in country a work plan will be developed during the team planning meeting and briefings with USAID/Southern Africa, CDC, and the National Department of Health. The work plan should include should include, but not be limited to, the following items: 1. Milestones and deliverables with due dates clearly established 2. Key interview questions that ensure quantifiable data, methods, and tools 3. Parameters for secondary analyses of existing data 4. Schedule of in-briefing and formal debriefing presentations 5. Tentative schedule for informant interviews 6. Tentative schedule of travel to field sites 7. Timeline for drafting the assessment report, requesting feedback, and finalizing the final report 3. Debriefing: Prior to departure the Evaluation Team will make a formal oral presentation to USAID/Southern Africa Program Office, Health Office, CDC and National Department of Health members. 4. Draft Report: Prior to departure, the Team Leader will submit a draft evaluation report to USAID/Southern Africa Health Office, CDC, and the National Department of Health that incorporates comments and information from the debriefing. The team will provide one hard copy and one electronic copy on a CD Rom or flash drive. The report (not including attachments) will be no longer than 30 pages with an Executive Summary, Introduction, Methodology, Findings, Lessons Learned, Conclusions, and Recommendations. 5. Final Report: After the Evaluation Team departs, USAID/South Africa and CDC has 14 working days to review the draft report and provide one single set of written comments on the draft report. The Team Leader will submit the final report to the Program Office after receiving comments from the USG team and the National Department of Health. GH Tech will provide the edited and formatted final document approximately 30 days after USAID and CDC provides final approval of the content. USAID/Southern Africa and CDC requests both an electronic version of the final report (Microsoft Word 2003 format) and 5 hard copies of the report. The report will be released as a public document on the USAID USAID/SOUTH AFRICA PEPFAR TREATMENT PROGRAM: FINAL EVALUATION 53 Development Experience Clearinghouse (DEC) (http://dec.usaid.gov) and the GH Tech project web site ww.ghtechproject.com). X. RELATIONSHIPS AND RESPONSIBILITIES (USAID AND CONSULTANTS) GH Tech will coordinate and manage the evaluation team and will undertake the following specific responsibilities throughout the assignment: Recruit and hire the four-person evaluation team (Note: Any USG officials will be financially supported by their respective agencies.) Make logistical arrangements for the consultants, including travel and transportation, country travel clearance, lodging, and communications; The USG team will provide overall technical leadership and direction for the Evaluation Team throughout the assignment and will undertake the following specific roles and responsibilities: Before In-Country Work Respond to all points included in the SOW, including the submission of the final report. Consultant Conflict of Interest. To avoid conflicts of interest or the appearance of a COI, review previous employers listed on the CV‘s for proposed consultants and provide additional information regarding potential COI with the project contractors or NGOs evaluated/assessed and information regarding their affiliates. Documents. Identify and prioritize background materials for the consultants and provide them, preferably in electronic form. Local Consultants. Assist with identification of potential local consultants and provide contact information. Site Visit Preparations. Provide a list of site visit locations, key contacts, and suggested length of visit for use in planning in-country travel and accurate estimation of country travel line items costs. Lodgings and Travel. Provide guidance on recommended secure hotels and methods of in￾country travel (i.e., car rental companies and other means of transportation) and identify a person to assist with logistics (i.e., visa letters of invitation etc.) if appropriate. During In-Country Work Mission Point of Contact. Throughout the in-country work, ensure constant availability of the Point of Contact person and provide technical leadership and direction for the team‘s work. Meeting Space. Provide guidance on the team‘s selection of a meeting space for interviews and/or focus group discussions (i.e. USAID space if available, or other known office/hotel meeting space). Meeting Arrangements. The Logistics coordinator embedded with the evaluation team will arrange meetings for contacts outside the Health Office. Formal and Official Meetings. Arrange key appointments with national and local government officials and accompany the team on these introductory interviews (especially important in high-level meetings). Other Meetings. If appropriate, assist in identifying and helping to set up meetings with local professionals relevant to the assignment. 54 USAID/SOUTH AFRICA PEPFAR TREATMENT PROGRAM: FINAL EVALUATION Facilitate Contact with Partners. Introduce the Evaluation Team to implementing partners, local government officials, and other stakeholders, and where applicable and appropriate prepare and send out an introduction letter for team‘s arrival and/or anticipated meetings. After In-Country Work Timely Reviews. Provide timely review of draft/final reports and approval of the deliverables XI. MISSION CONTACT PEOPLE/PERSON Mission Contact for this assignment: Charles Mandivenyi, Program Office, USAID/Southern Africa cmandivenyi@usaid.gov+27 12 452 2273 +27823217808 Final report to be sent to: Charles Mandivenyi, Program Office, USAID/Southern Africa cmandivenyi@usaid.gov and Win Brown, Health Office, USAID wbrown@usaid.gov XII. COST ESTIMATE USAID: GH Tech will provide a cost estimate. XIII. REFERENCES (PROJECT DOCUMENTS) Reviewers will be provided with the following background documents in preparation for the assignment: Key Resource Documents: 1. The USG Mission‘s current strategies 2. The 2006 (2007?) APS that funded many of our treatment partners 3. The USAID and CDC aggregated COP treatment summaries for 2007, 2008, and 2009 4. The draft RFP for the big treatment partners 5. The IPHC Evaluation, which will serve as a useful reference in this activity 6. Quarterly, Semi-Annual, and Annual Progress Reports to OGAC that summarize the achievements of the USAID and CDC treatment partners 7. Partnership Framework Mandate (per request of the team) 8. PEPFAR monitoring guidance that informs how implementing partners are able to ―count‖ the numbers of individuals supported (per request of the team) 9. SAG HIV/AIDS M&E framework with the national indicators (per request of the team) 10. All relevant NDOH strategic frameworks, guidelines and the most recent NDOH reports (per request of the team) 11. Resource documents about the other key development partners supporting the NDOH HIV/AIDS program, including the recently approved GF HIVAIDS grant (per request of the team) USAID/SOUTH AFRICA PEPFAR TREATMENT PROGRAM: FINAL EVALUATION 55 ANNEX B. CLARIFICATIONS TO THE SCOPE OF WORK This appendix is included because clarifications were sought of USAID, CDC, and the NDOH in South Africa before the start of the in-country phase of the evaluation. These clarifications were sought both by the consultants and by GH-Tech. Discussion continued in Pretoria during the first week of the assignment. The Appendix contains the following documents: 1. The international consultants‘ questions, prepared in advance of the evaluation assignment 2. GH-Tech discussion (led by Julie Klement, Director) 3. Clarifications to the SOW, provided by USAID and CDC South Africa and by the NDOH. This document was signed off in the week of 20 June 2011. CONSULTANTS’ QUESTIONS REGARDING THE SOW PEPFAR/SA Treatment Program Evaluation, June-August 2011 [GH-Tech 1.530] Technical Questions A. Overall scope of the evaluation A1. Please define the parameters of the evaluation and its scope. What exactly is meant by evaluation of the PEPFAR ART program in terms of mission expectations? What are the ART Program Goal and Strategic Objectives (intermediate results)? [Comments: the SOW in itself and also vis-à-vis other key documents covers a wide range of priorities. Thus the SOW (section I) describes an ‗evaluation of the USG/PEPFAR ART program‘. The SOW (section II) lists 12 areas of support provided by Implementing Partners to facilities; it also describes in section III the 5 ‗overall PEPFAR/SA ART program objectives for FY 2011‘ and the implementation of revised treatment guidelines and eligibility criteria. Section IV lists objectives and key questions. The PEPFAR SA COP Report FY 2010 discusses a total of 14 technical areas (not entirely coherent with the 12 areas of support listed in the SOW). The SAG/USG Partnership Framework sets out priority goals and objectives, again not entirely coherent with the SOW. The FY 2011 APR and SAPR Indicator Templates discuss objectives and priorities, and changes to these relative to earlier FY, changes based on SAG priorities.] A2. Which PEPFAR technical areas and IP support components (as set out in section II of the SOW) are to be evaluated during this assignment? A3. Is the evaluation to address additional components, objectives, initiatives, and programs? A4. Is the evaluation to be health facility and supply-side focused, i.e. primarily or solely addressing clinical components? A5. Are demand-side perspectives to be addressed (e.g. in light of previous priorities and the current SAG position that PEPFAR funding should now focus more on prevention)? A6. Degree of focus in the evaluation on Health Systems Strengthening, women and girl￾centered approach and the other five GHI principles? E.g. just two points: are we to address 56 USAID/SOUTH AFRICA PEPFAR TREATMENT PROGRAM: FINAL EVALUATION gender aspects of access and/or implications of HIV/TB and ART focus for wider health service delivery? B. Impetus for the evaluation B1. What is the impetus for the evaluation? Was this a request centrally from OGAC, or from in–country? Is an evaluation also occurring in other PEPFAR funded countries/ southern Africa? C. Time line C1. Focus of the evaluation in terms of time: is the evaluation to review activities since 2004 or primarily since the PEPFAR re-alignment in 2008? C2. Is the evaluation to prioritize review of more recent activities – what is the time line in that case? D. Definitions D1. Is there an agreed working definition of sustainability among all partners, including Implementing Partners? Or might this key concept be interpreted differently; if so, whose definition has priority? Is there an agreed exit strategy and/or milestones for this program? D2. Is the discussion on sustainability in the Partnership Framework to guide the evaluation? E. Implementation Partners E1. How many IPs are there and which require visits from the evaluation team (beyond site visits)? E2. What is the role of SANAC in this program? Line ministries? Province and district-level public sector IPs? Private sector and CSO IPs? F. Review vis-à-vis recommendations F1. What is the balance to be in the evaluation between review of past and current activities and recommendations for future programming? (See also questions A1-6 above.) F2. The team will need to have opportunity to review any new interventions (FY 2011 and post￾Partnership Framework), so as to have proper understanding of processes, lessons learned and potential for change/recommendations. F3. National policy: is the evaluation linked to the development of the new SA National Strategic Plan and the new TB NSP, and/or other national policy and strategy instruments? Will its recommendations inform the development of such documents? Please note that team members have additionally developed more focused technical questions for each key question, to be addressed during in-country meetings with USAID, CDC, and NDOH. Administrative and Logistical Questions 1. The current SOW (dated 31st May 2011) states that participants from the NDOH, USAID and CDC will not contribute to the writing of the draft and final reports. At the time of writing this document (3rd June), the full composition of the evaluation team is not known. Therefore, the full range of team skills, individuals‘ comparative advantage, etc, remains to be clarified. These matters will have a bearing on the evaluation approach. Once the full team is identified we shall need to discuss the SOW and our roles and responsibilities, both internally and also with USAID, CDC and SAG partners. Limitations may be identified. This is necessary before interviews and fieldwork begin. USAID/SOUTH AFRICA PEPFAR TREATMENT PROGRAM: FINAL EVALUATION 57 2. Please define what is meant by a ‗draft‘ report. 3. Please refer to the preliminary criteria for site selection. Will the teams visit any of the 18 priority districts? GH-TECH DISCUSSION Julie Klement, Director, GH-Tech, sent the following table of action points as an e-mail on June 9, 2011. Question/Item POC/Lead Deadline Resolution Clarify meeting date with CDC, NDOH, and USAID; Monday the 13th or Tuesday the 14th? Charles ASAP Notify the team Clarify definition of ―stakeholder,‖ per discussions of the stakeholder list Alli 6/9/11 By this, GH Tech is referring to implementing partners, local government officials, and other stakeholders. These are individuals/groups who will be interviewed and/or receive site visits so the evaluation team can gather information. Prepare the stakeholder list so that Themba can begin setting up appointments Charles ASAP Clarification on the SOW. (Using Janet‘s document as a guide, discussions between Charles and the Evaluation Team must help clarify the SOW, expectations, parameters, etc). Charles/Eva luation Team 6/13 – 6/14 Discussions to take place during the TPM & meetings w/ USAID – prior to any interviews or site visits. Written summary of key SOW clarifications and expectations, to be signed off by Charles/Eval Team before beginning fieldwork. Charles/ Evaluation Team 6/14 Must be done prior to any site visits or interviews Confirm local consultants on Eval Team Charles/ Win ASAP Must be on board by Monday, June 13th (share CVs with GH Tech and Team Leader) 58 USAID/SOUTH AFRICA PEPFAR TREATMENT PROGRAM: FINAL EVALUATION Question/Item POC/Lead Deadline Resolution Provide specifics on Robin Wood‘s involvement Alli 6/9/11 Robin has confirmed his interest in participating on the team as his schedule allows. Alli is awaiting confirmation from Robin that he is available on the following dates: June 14-18 in Pretoria June 22-23 in the field (exact location TBD by the team planning meeting but likely Jo‘ burg, Pretoria or Durban) July 5 in Pretoria or remotely (TBD) and potentially a few days of remote follow-up work later in July USAID/SOUTH AFRICA PEPFAR TREATMENT PROGRAM: FINAL EVALUATION 59 CLARIFICATIONS TO THE SCOPE OF WORK PEPFAR SA ART program evaluation (June-August 2011) Clarification of the SOW, definition of stakeholders and discussion of SOW limitations 06-17-11 Introduction The intention of this document is to present finalized clarification of questions arising from the Scope of Work (SOW) and definition of stakeholders; in addition, given the tight time frame and other evaluation constraints, it addresses limitations in terms of SOW requirements. This document is based on technical and planning discussions held in Pretoria during the week of 13 June, 2011, with Dr. Yogan Pillay, Ms Lilian Disek and Dr. Peter Barron from the NDOH, Roxana Rogers, Win Brown, Charles Mandivenyi, Wendy Benzerga and Melinda Wilson (all USAID), and Dr. Thurma Goldman and Dr. Jeff Klausner of CDC. The document does not provide a complete record of discussions to date; it addresses the three points listed above. It was signed off in the week of 20 June 2011. Clarification of the Sow Overall Scope of the Evaluation This evaluation is a performance evaluation, not an impact evaluation. Please see the current USAID Evaluation Policy for definitions. As a performance evaluation, it cannot be an in-depth research study; it is a partial, primarily qualitative snapshot of current programmatic and linked activities. Site selection (health facilities and a few communities) will be based on as random a sample as possible, managed by the evaluation team of local and international consultants, with inputs from its three counterparts, these being the NDOH, USAID and CDC. Methodology, site and key informant selection and tools form part of the final report annexes that provide summaries of key aspects of conducting an evaluation (annexes C, D, E, and H). The evaluation approach is always to be guided by the 4 objectives and 10 key questions listed in the SOW. The South Africa CDC and USAID PEPFAR antiretroviral treatment program is in a period of transition. The primary impetus for the evaluation is increased SAG ownership and stewardship. The intention is for ―a sensible transition by PEPFAR to TA focus,‖ informed in part by this evaluation. The SAG is making a transition into enhanced focus on a Primary Health Care (PHC) model of service delivery and greater focus on USG technical inputs; these key imperatives will inform the evaluation. At the core of this evaluation are the questions: how PEPFAR inputs have added value to overall ART service delivery; how has this translated into mainstreaming the program, into institutionalizing its inputs. 60 USAID/SOUTH AFRICA PEPFAR TREATMENT PROGRAM: FINAL EVALUATION Key Issues These three issues were developed during the meeting held at the NDOH on 15th June 2011, articulated by Dr. Yogan Pillay. Key Issue 1: Coverage This will link into key question 1 (reduction of the treatment gap); moreover, issues such as whether any key populations are falling behind within overall coverage should be addressed. These key populations might include children, commercial sex workers, truckers, MSM and WSW. Any lessons learned and/or best practices derived from PEPFAR inputs are to be solicited where possible. Key Issue II: Efficiency This does not refer to financial and health economics‘ models of efficiencies, as such attention is beyond the scope of this evaluation. Rather, it refers to issues such as Chronic Care Management and the service delivery and other efficiencies potentially or actually associated with this model. In this context primary attention should be given to lessons already learned. In addition, opportunity costs for all those eligible for ART should be borne in mind. Key Informant Interviews (KII) will be held with Treasury and CHAI – not to consider financial aspects, but looking into cost reduction/efficiency. Key issue III: Quality How most equitably and effectively to balance quantity and quality of service delivery, from both the supply and demand sides. Attention to such matters is essential as decentralization proceeds: how to balanced expanded coverage with increased quality of delivery. Additional Issues put forward by CDC, as Raised during the Week of 13th June 2011 1) The partners‘, government and civil society‘s understanding of technical versus direct assistance, the benefits and risks/harms/costs of both as PEPFAR transitions to the former. 2) Consider the significance of PEPFAR support into the medium-term future in the overall context of the HIV/TB response in South Africa. 3) Finally, given the HIV/TB care and treatment programs are increasingly led, managed, and funded by the provinces, it is important to receive input from the SAG provincial leadership.KZN, Gauteng, and WC have strong ideas about the direction and future needs of the relationship. USAID/SOUTH AFRICA PEPFAR TREATMENT PROGRAM: FINAL EVALUATION 61 Specific Responses to the SOW Clarification Questions These responses have been provided through discussion with the three evaluation counterparts (NDOH, USAID, and CDC). Question A1: Please define the parameters of the evaluation and its scope. What exactly is meant by evaluation of the PEPFAR ART program in terms of mission expectations? Please also see 2.1 above. When the ART program began 1 April 2004 the focus was on the emergency phase of treatment. Thus 2004 objectives were to provide treatment and to get as many people on such treatment. There were solely numerical targets, broadly applying the ‗head count model‘. PEPFAR phase II began in 2009 and has expanded beyond the emergency treatment phase. Question A2: Which PEPFAR technical areas and IP support components (as set out in section II of the SOW) are to be evaluated during this assignment? Primary focus is to be on clinical interventions; the clinical interventions to be addressed are: adult and pediatric ART, TB/HIV integration, PMTCT and M&E. If possible, there is also to be (secondary) attention to prevention interventions (e.g., PwP) and to crosscutting interventions such as gender. Clinical interventions will be evaluated through visits to health facilities, while prevention and crosscutting issues will be evaluated through visits to communities and engagement with facility-based HIV Support Groups, etc. Question A3: Is the evaluation to address additional components, objectives, initiatives, and programs? No. Question A4: Is the evaluation to be health facility and supply-side focused, i.e. primarily or solely addressing clinical components? See A2 above. Question A5: Are demand-side perspectives to be addressed (e.g. in light of previous priorities and the current SAG position that PEPFAR funding should now focus more on prevention)? See also A2 above. Thus demand-side/community perspectives are to be considered, with appropriate follow up during health facility visits (e.g. if access issues are raised) and particularly during community visits. Entry point may be through adherence. See also the key issues discussed in 2.2. Question A6: Degree of focus in the evaluation on Health Systems Strengthening, women and girl￾centered approach and the other five GHI principles? E.g. just two points: are we to address gender aspects of access and/or implications of HIV/TB and ART focus for wider health service delivery? The evaluation is to be cognizant of such issues; however, these do not constitute core priorities. For instance, it is not to prioritize HSS, as PEPFAR does not fund such interventions. Question B: Impetus for the Evaluation This has been directed by the NDOH, led by Dr. Yogan Pillay, Deputy Director General of the Strategic Health Programmes‘ Directorate in the NDOH. Question C: Time Line Question C1: Focus of the evaluation in terms of time: is the evaluation to review activities since 2004 or primarily since the PEPFAR re-alignment in 2008? 62 USAID/SOUTH AFRICA PEPFAR TREATMENT PROGRAM: FINAL EVALUATION The major focus is to be recommendations for the transition; these will inform the RFAs. Lessons are to be learned from both past and current Best Practice and challenges/gaps. Question C2: Focus of the evaluation in terms of time: is the evaluation to review activities since 2004 or primarily since the PEPFAR re-alignment in 2008? See C1. Definitions Question D1: Is there an agreed working definition of sustainability among all partners, including IPs? Or might this key concept be interpreted differently; if so, whose definition has priority? Is there an agreed exit strategy and/or milestones for this program? The PEPFAR ART program does not have an agreed exit strategy or currently defined milestones; however its targets inform process and outcomes.Questions to ask during the evaluation include: what have partners‘ experiences been when exiting; are there measured outcomes? Aspects to consider with regard to sustainability include: time frame on this; how quickly did this happen and how – which components, institutionalization, HSS. Focus on the fact that the SAG is reforming the health system to make it a PHC model – how can sustainability lessons learned from PEPFAR inputs help to achieve this? It is important to link consideration of exit and sustainability with the current transition into PHC and lessons learned and best practices in those contexts. Question D2: Is the discussion on sustainability in the Partnership Framework to guide the evaluation? TBD. The NDOH has requested information on what has been achieved, what has been recorded and what can all relevant partners can do to assist enhanced sustainability. Implementation Partners (For Key Informant Interviews) Question E1: How many IPs are there and which require visits from the evaluation team (beyond site visits)? This is being determined through field trip planning (as of 17 June 2011). Question E2: What is the role of SANAC in this program? Line ministries? Province and district-level public sector IPs? Private sector and CSO IPs? These points remain to be addressed. Review Vis-à-Vis Recommendations Question F1: What is the balance to be in the evaluation between review of past and current activities and recommendations for future programming? (See also questions A1-6 above.) See C1 above. Question F2: The team will need to have opportunity to review any new interventions (FY 2011 and post-Partnership Framework), so as to have proper understanding of processes, lessons learned and potential for change/recommendations. To be discussed. USAID/SOUTH AFRICA PEPFAR TREATMENT PROGRAM: FINAL EVALUATION 63 G. Changes to wording of the SOW These changes were requested by CDC on 22 June 2011 and referred to questions 2 and 10 of the 10 key SOW questions. Definition of Stakeholders This includes, inter alia: Treatment partners Treatment implementing/sub partners SAG and NDOH counterparts at national, provincial and district levels PEPFAR and non-PEPFAR health facility staff members at secondary and primary facility levels – Medical Director – ART physician – M&E Officer – Nurse/physician in charge, PMTCT – Health worker with responsibility for PMTCT – Nurse-in-charge (at PHC level) Members of facility-based HIV Support Groups Patients Community members Civil society organization representatives SOW Limitations As is common practice when conducting an evaluation, a section is included in the report on limitations to the SOW. This refers to constraints and unavoidable changes due to circumstances beyond the control of the evaluation team and/or its counterparts. Pre-planning Due to a number of factors, pre-planning (i.e. before the full evaluation team was under contract) was short and less than ideal. This resulted in unavailability of documents, considerably shortened preparation time before and after commencement of the evaluation and extremely limited time to have team planning and other key meetings. The first week in country (13-17 June) was largely taken up with activities that should have been either completed before the start of the assignment, or substantially in hand. Time Pressure The evaluation was initially scheduled to last six weeks; in the event, it was compressed into four. Moreover, the evaluation team had a maximum of two weeks (20 June – 2 July 2011) to conduct all key informant interviews, site visits, and hub meetings. Such time constraints inevitably had a bearing on the comprehensiveness of the evaluation. Local Consultants Khulisa Consultants was engaged to begin work as part of the evaluation team only on Monday 13 June 2011. This resulted in inevitable scheduling conflicts and limited availability of all consultants to participate in the evaluation throughout its in-country activities. This was less 64 USAID/SOUTH AFRICA PEPFAR TREATMENT PROGRAM: FINAL EVALUATION than ideal in terms of continuity, joint learning and optimal inputs to the out brief and report work. Logistic Support A local consultant was retained as logistic support in the week prior to the international consultants‘ arrival in South Africa. His work proved unsatisfactory, which led to the contracting of Khulisa Consultants on June 17 2011 to provide logistic services in addition to technical inputs. These delays also had an inevitable effect on the early in-country planning for the evaluation, e.g. the work plan could not be completed until week two of the assignment. They also put very considerable pressures on the entire evaluation team and in particular two of its members who shouldered responsibility for ensuring Gauteng, KZN, Western and Eastern Cape meetings and site visits became possible and effective (see below). Very considerable amounts of time up to 24 June 2011 had to be dedicated to planning schedules and hub meetings, rather than to having the full team working on the technical aspects of the evaluation. Incomplete Technical Inputs Due to the logistical challenges, Mary Pat Selvaggio, Director, Health and Elna Hirschfeld, Associate Director, both of Khulisa Consultants, were unable to participate fully in the technical aspects of the evaluation before 24 June 2011. Mary Pat Selvaggio was only available until that day to work on the evaluation, while Elna Hirschfeld had the opportunity to be part of the team until 8 July. Limitations of the Debrief and Draft Report Due to all the above limitations, the evaluation team was only able to undertake the most top line discussion of data collected during the field visits in preparation for the debrief and the draft report. The team reconvened from the field on Monday 4 July and presented an out brief on Thursday 7 July. Detailed analysis and discussion were simply not possible in the time available; this has inevitably had repercussions in terms of the depth of findings as discussed in the debrief and the draft report. An additional major issue is that a literature review was also impossible, due to key documents not having been provided to the evaluation team and insufficient time. This too had implications in terms of the breadth and depth of the out brief and the draft report. USAID/SOUTH AFRICA PEPFAR TREATMENT PROGRAM: FINAL EVALUATION 65 ANNEX C. PEPFAR SA ART PROGRAM EVALUATION JUNE-AUGUST 2011 WORK PLAN Introduction This work plan constituted a key early deliverable as set out in the SOW for the PEPFAR SA ART program evaluation. Due to limitations as set out in the SOW Clarification document, signed off by USAID in the week of 20 June 2011, the work plan continued to be a work in progress during week 2 of the assignment (20-25 June). Thus the work plan here below partly sets out a description of process and progress as of 20 June 2011, when the terms of the evaluation contract obliged its submission. This annex has been updated to reflect the finalization of the report as of 2 September 2011. Please see Annex H (separately appended) for a full record of work undertaken during the evaluation. MILESTONES FOR EVALUATION DELIVERABLES Milestone Date due Progress as of 6/20/11 Progress as of 9/2/11 1. Work Plan Week of 6/20/11 Completed Completed 2. Schedule of in briefing meetings Week of 6/13/11 In-briefings: completed week of 6/13/2011 Completed 3. Schedule of out￾briefing meetings Week of 6/20/11 Out-briefing: date TBD; either 7/8/11 or 7/11/11 Completed; out briefing to USAID & CDC conducted 7/8/11. Out briefing to NDOH completed in July/August 2011 by CDC & USAID (without evaluation team participation) 4. KII schedule Week of 6/20/11 In progress; to be provided to assignment partners (NDOH, USAID & CDC) when completed. Please also see section 3 below. Completed and KII undertaken 5. Field visit schedule Week of 6/20/11 In progress; to be provided to assignment partners (NDOH, USAID & CDC) when completed. Please also see section 3 below. Completed and field visits undertaken 6. Draft evaluation report Before departure of international consultants from SA Dependent on date of out￾briefing: date TBD Draft evaluation report submitted 8/11/11 7. Final 14 + 5 days Partially dependent on date Out briefing done on 7/8/11. 66 USAID/SOUTH AFRICA PEPFAR TREATMENT PROGRAM: FINAL EVALUATION Milestone Date due Progress as of 6/20/11 Progress as of 9/2/11 evaluation report after draft (assignment partners have 14 working days to comment; final report due 5 days later) of out-briefing: date TBD. Final report due from evaluation team in early August 2011. Comments received from USAID & CDC 8/10/11; responses sent by evaluation team 8/18/11. Final report submitted to GH-Tech 9/2/11. USAID/SOUTH AFRICA PEPFAR TREATMENT PROGRAM: FINAL EVALUATION 67 ANNEX D. EVALUATION METHODOLOGY AND TOOLS SCOPE OF THE EVALUATION The evaluation is a performance evaluation as per the USAID Evaluation Guidelines; its objective is to provide a snapshot of the PEPFAR ART program, using primarily qualitative methodology, with quantitative analysis of PEPFAR data. The baseline for the evaluation has been agreed at 2007, when the ―HIV Care and ART Consultation‖ report was finalized. As a performance evaluation, findings and recommendations cannot address the longitudinal and higher-level impact of the program. EVALUATION TEAM COMPOSITION AND SCHEDULE The evaluation team was composed of two international consultants (Dr. Lois Eldred and Janet Gruber, Team Leader) and local South African consultants. The latter were Professor Robin Wood and the following Khulisa consultants: Mary Pat Selvaggio, Elna Hirschfeld, Dr. Ntombi Bandezi, Jenna Kamps, and Peter Njaramba. It should be noted that due to the extreme time constraints of this evaluation, the only consultants available throughout the evaluation were Lois Eldred and Janet Gruber. The evaluation team worked together in Gauteng during the week of June 20, 2011; it then split into two teams, one traveling to KZN, the other to Eastern and Western Cape. SITE SELECTION This was guided by the ESI Data Warehouse system and its GSI co-ordinates and also assisted by facilitation of introductions by CDC and USAID evaluation counterparts. The intention was to achieve as representative a site (health facility) sample as possible that speaks to the requirement to evaluate the PEPFAR ART program, given the severe time and forward planning constraints that apply for this evaluation (in this context, please see the SOW Clarification document, signed off by USAID in the week of June 20, 2011). Not only PEPFAR-supported sites were visited, but also NDOH facilities. DATA COLLECTION METHODOLOGY Key informant interviews (KII) were conducted with a range of representative stakeholders, including members of staff from the NDOH, CDC and USAID, PEPFAR partners and their sub-partners, donor partners, SAG (e.g., Treasury), and also at sites (health facilities) with the facility Medical Directors. Clinical Services’ Tool: this was applied at sites with the clinician/health worker responsible for ART service delivery; the health worker responsible for PMTCT; the M&E officer responsible for reporting using PEPFAR systems; and, where appropriate, the health worker responsible for TB/HIV integration and/or services. Hub meetings: four were held, one in each of the provinces visited for the purposes of this evaluation. The rationale for these meetings was to optimize time efficiency in terms of conducting interviews. In Gauteng, PEPFAR partners attended; in the three other provinces both PEPFAR partners and subs were present. Focus group discussions were held at each of the four hub meetings; during these KII adapted for PEPFAR implementing partners were used. 68 USAID/SOUTH AFRICA PEPFAR TREATMENT PROGRAM: FINAL EVALUATION Focus group discussions (FGD) were conducted where appropriate with PEPFAR partners and subs, e.g., at the hub meetings. In addition and where applicable, FGDs were held with facility-based HIC support groups and community representatives. The following data collection methodologies were also applied: 1. Document review: USAID, CDC, PEPFAR, partner, and other relevant national and international literature 2. Further quantitative analysis of key PEPFAR data stored on the Data Warehouse system. TRIANGULATION AND ANALYSIS The methodology was designed to enable triangulation of findings from a range of sources (while acknowledging the scope of this performance evaluation and its constraints). Analysis of all findings was a joint team effort, as was the writing of the evaluation report. To this end daily team meetings were conducted in Gauteng and in the field. This enabled initial analysis as per the report structure (completed on June 19, 2011). TOOLS: KEY INFORMANT INTERVIEW (KII) GUIDES AND FOCUS GROUP DISCUSSION GUIDES Two examples of KII guides are provided here; a total of seven were used, with the following categories of respondent: 1. (N) DOH representatives 2. USAID and CDC 3. PEPFAR Provincial Liaison Officer 4. Health Facility Manager and other staff 5. M&E representative at health facility 6. GP 7. PEPFAR Implementing Partners In addition a number of unique KII were conducted, e.g., with representatives of CHAI and the National Treasury. All KII addressed the 10 key SOW questions and the forward looking SOW issues, as well as being informed by the priority issues provided by Dr. Pillay of the NDOH (coverage, efficiency, and quality). FIRST EXAMPLE OF A KII Key Informant Interview for PEPFAR Partner Organizations Please ensure that you always obtain the name of the respondent, his/her position within the organization, and the organization‘s name and address. Please also obtain the respondent‘s contact details, in case follow up is required. Introduction Thank you for finding the time to see us at such short notice. We are members of the team that has been asked to conduct an independent evaluation of the PEPFAR ART program in South USAID/SOUTH AFRICA PEPFAR TREATMENT PROGRAM: FINAL EVALUATION 69 Africa. The National Department of Health and USAID/CDC PEPFAR in South Africa have requested the evaluation. We are meeting a wide range of treatment partners, visiting health facilities and communities. The objectives of our evaluation are to examine whether PEPFAR programmatic objectives were achieved, to look at lessons learned and best practices and to make recommendations that will help to inform the way forward for the NDOH in its planning for transition to Primary Health Care rollout of ART. Question 1 Can you give us a brief history of your organization‘s work in ART in this site/district/ province/ South Africa? [Probe on how long it has been in existence in SA, its size, and its key activities in the field of ART, geographical spread] Question 2 When did you first receive PEPFAR support specific to ART? Question 3 How has PEPFAR ART support changed your organization and how you support or deliver ART services? (Positively and/or negatively) [If relevant to the organization: probe about the organization‘s relationship with its implementing/sub partners specific to ART provision. Get an understanding of number of subs, spread, type of relationship, management, reporting, iteration, lead partner support to capacity development, etc.] Question 4 What has been the most positive aspect of PEPFAR support for ART?(This could be added value to the organization‘s delivery of ART [and its integration into other services] or more general PEPFAR influence in the policy arena e.g. on harmonization and alignment.) Question 5 What has been the most challenging aspect of PEPFAR support for ART? [Probe, e.g., has PEPFAR support helped or hindered ART provision/integration of services and in what manner?] Question 6 What do you believe has been your organization‘s greatest success in ART in this site/district/province/South Africa? [Probe as to whether this can be attributed to PEPFAR support or not] Question 7 Please tell us more about how your organization addresses coverage in terms of ART provision. [Don‘t lead – see how the respondent defines coverage. Then probe on the degree to which the organization considers and addresses such matters, e.g., changes to its way of working, any outreach to difficult-to-reach groups (and which those might be, e.g., children, MSM), any focus on pre-ART, transition out, gender aspects of access, etc. Does the organization have an active plan to target improved coverage?] Question 8 Please tell us more about how your organization has addressed efficiency in terms of ART provision. [Don‘t lead—see how the respondent defines efficiency. Then probe on action to date and any consideration by the organization with regard to issues such as movement out of the emergency 70 USAID/SOUTH AFRICA PEPFAR TREATMENT PROGRAM: FINAL EVALUATION phase of ART provision into a potentially more stable treatment environment, e.g. Chronic Care and the efficiency implications.] Question 9 Please tell us more about how your organization addresses quality of service in terms of ART provision. [Again, don‘t lead; allow the respondent to define quality. Be alert for issues such as staff members‘ attitudes and behavior, whether there is any supportive supervision and/or quality assurance of this (including from the patient/client side), best use of data for planning, tracking of pre-ART patients and those on ART, links to other services (e.g. TB, PMTCT, SRH), integration (or not) with DOH structures, other service providers and organizations to address maximum coverage and efficiency and minimal duplication.] Question 10 How does your organization manage data collection, analysis, reporting, and use in planning for ART services provision? [Probe on issues such as multiple reporting, iteration, disaggregation, and use of analysis in planning for improved ART service delivery.] Question 11 Looking back on PEPFAR support to date, what have been the top five lessons learned specific to ART? Question 12 Looking to the future and the planned transition into a PHC model of ART provision, which recommendations would you like to make and which role/s would you like your organization to play? Which best practices would you like to see maintained and mainstreamed? Second Example of a KII KII with USAID and CDC Counterparts PEPFAR SA ART program evaluation June-August 2011 Key Informant Interview for USAID and CDC counterparts (> 60 minutes) Please ensure that you always obtain the name of the respondent, his/her position within the organization, and the organization‘s name and address. Please also obtain the respondent‘s contact details, in case follow up is required. Introduction Thank you for finding the time to see us at such short notice. The objectives of our evaluation are to examine whether PEPFAR programmatic objectives were achieved, to look at lessons learned and best practices and to make recommendations that will help to inform the way forward for the NDOH in its planning for transition to Primary Health Care rollout of ART. Question 1 Can you give us a brief overview of your role in the PEPFAR ART program? Question 2 What do you think is the value added by the PEPFAR ART program to SAG ART provision? [Probe] USAID/SOUTH AFRICA PEPFAR TREATMENT PROGRAM: FINAL EVALUATION 71 Question 3 How far do you feel the 10 key SOW questions have been achieved, been unachievable or fallen short of target? [See attached list.] Question 4 How and to what extent do you think PEPFAR ART program support has contributed to effective coverage? Can you describe any gaps, lessons learned, and best practice? [Probe on definition. Points from Dr. Pillay (NDOH) on coverage: this will link into SOW key question 1 (reduction of the treatment gap); moreover, issues such as whether any key populations are falling behind within overall coverage should be addressed. These key populations might include children, commercial sex workers, truckers, MSM, and WSW. Any lessons learned and/or best practices derived from PEPFAR inputs are to be solicited where possible.] Question 5 How and to what extent do you think PEPFAR ART program support has contributed to efficiency? Can you describe any gaps, lessons learned, and best practice? [Again, probe on definition. Points from Dr. Pillay: this does not refer to financial and health economics‘ models of efficiencies, as such attention is beyond the scope of this evaluation. Rather, it refers to issues such as chronic care management and the service delivery and other efficiencies potentially or actually associated with this model. In this context primary attention should be given to lessons already learned. In addition, opportunity costs for all those eligible for ART should be borne in mind, as should cost reduction.] Question 6 How and to what extent do you think PEPFAR ART program support has contributed to quality? Can you describe any gaps, lessons learned, and best practice? [Probe on definition – e.g., is quality defined solely from a clinical/supply-side perspective, or does it include patient/demand-side issues of quality assurance too? Dr. Pillay‘s points: how most equitably and effectively to balance quantity and quality of service delivery, from both the supply and demand sides? Attention to such matters is essential as decentralization proceeds: how to balanced expanded coverage with increased quality of delivery.] Question 7 Do you think that the PEPFAR transition from direct support to TA will help increase the reach of government ART programs? What do you think the challenges are for transition? How do you think they should be addressed? Question 8 (from CDC) Consider the significance of PEPFAR support into the medium-term future in the overall context of the HIV/TB response in South Africa. Question 9 How sustainable do you think PEPFAR ART program inputs, outputs and outcomes will be in the context of transition to the PHC ART service delivery model? 72 USAID/SOUTH AFRICA PEPFAR TREATMENT PROGRAM: FINAL EVALUATION PEPFAR ART EVALUATION: THE 10 KEY QUESTIONS IN THE SOW: Did the program: 1. Help to substantially reduce the estimated treatment gap? 2. Develop a recognized standard of public health practice for ART? 3. Play a lead role in reducing costs of delivering quality ART services? 4. Capacitate cadres of health workers to ensure a sustainable program? 5. Enhance the SAG‘s ability to monitor the progress of its ART program? 6. Increase overall retention rates and decrease mortality rates? 7. Take a population-based approach that emphasizes coverage and reach of ART services? 8. Accurately account for its progress with respect to numbers reached? 9. Design and implement its activities with transparency, appropriately sharing plans and progress with counterparts at all levels? 10. Invest wisely in sustainable solutions that will enable South Africa to assume responsibility in managers? USAID/SOUTH AFRICA PEPFAR TREATMENT PROGRAM: FINAL EVALUATION 73 ANNEX E. EVALUATION SITE VISIT SCHEDULE AND DETAILED PROVINCIAL MAPS (GP, KZN, EASTERN AND WESTERN CAPE) SCHEDULE OF SITES VISITED GAUTENG Team A Team B Tuesday 21 June (am) Thembisa Site Visit Friday 24 June (am) RTC Alexandra clinic WRHI Hillbrow EAST LONDON Team A1 Team A2 Tuesday 28 June Dimbaza Africare: Victoria Hospital, Alice Kethimpilo: Dimbaza Clinic Butterworth IYDSA: Great Kei Health Centre, Komga Hospital (en route to B-worth hospital) Kethimpilo: Butterworth Gateway Clinic Kethimpilo: Nqamakwe Wednesday 29 June East London /Mdantsane ICAP: Duncan Village Day Hospital BroadReach: Dr. Nadaraju Kwelitsha /Bisho / King William’s Town Kethimpilo: Zwewlitsha clinic Kethimpilo: Grey Gateway clinic CAPE TOWN Team A1 Team A2 Friday 1 July Durbanville / Winelands: ANOVA: Eesterivier Clinic Kethimpilo: Wallacedene Cape Town / Cape Flats TB/HIV Assoc: Hout Bay ANOVA: G F Jooste Hospital ANOVA: Desmond Tutu: Hanan Crusades Clinic KZN Team B1 Team B2 Tuesday 28 June Pietermaritzburg EGPAF: Grey‘s hospital (with M2M program?) Kethimpilo: Imbalenhle CHC Umlazi EGPAF/AIDS Health Care Foundation: ―H‖ Clinic Caprisa: Vulindela Clinic Aurum: Dr. Ramlutchman Wednesday 29 June Durban St Mary's Hospital, Marianhill McCord/Zoe Life: Westville Municipal Clinic McCord/Zoe Life: Pinetown Municipal Clinic Durban MATCH: Kwamakhuta Clinic MATCH: Lwasi Clinic and 6B (in Addington Hospital) MATCH: Charles James TB Hospital 74 USAID/SOUTH AFRICA PEPFAR TREATMENT PROGRAM: FINAL EVALUATION KZN Team B1 Team B2 Thursday 30 June Port Shepstone BroadReach: Murchison Gateway clinic BroadReach: Murchison Hospital Mthubathuba Africa Centre: Awankhuta Clinic Africa Centre: Sipho Zungu Clinic Friday 1 July Ugu TB/HIV Assoc: Christ the King ARV Gateway Clinic, Ixopo Richards Bay URC: Engwelazani Hospital DETAILED PROVINCIAL MAPS SHOWING ALL SITES VISITED Figure E-1. Gauteng USAID/SOUTH AFRICA PEPFAR TREATMENT PROGRAM: FINAL EVALUATION 75 Figure E-2. KwaZulu-Natal Figure E.3. Eastern Cape 76 USAID/SOUTH AFRICA PEPFAR TREATMENT PROGRAM: FINAL EVALUATION Figure E-4. Western Cape USAID/SOUTH AFRICA PEPFAR TREATMENT PROGRAM: FINAL EVALUATION 77 ANNEX F. ADDITIONAL DISCUSSION POINTS, AS REQUESTED BY USAID AND CDC IN AUGUST 2011 BACKGROUND This annex has been written in response to comments received by the evaluation team in August 2011. After reviewing the draft report submitted on July 11, 2011, USAID and CDC requested attention to six areas not included in the assignment scope of work or discussed during assignment work in South Africa June-July 2011. Thus the six areas are post hoc, i.e., the team did not address these issues during the assignment. The six areas are: 1. Gender 2. Test-and-treat 3. Unique identifiers 4. SAG infrastructure 5. TB services 6. HR transition The evaluation team, in discussion with GHTech, decided on the following response: to be responsive to U.S. Government and the NDOH, this annex will provide absolutely top line comments on the six areas, based on team expertise and local knowledge. Any and all such comments are not based on evaluation fieldwork findings. GENDER It should be noted that clarification was sought by the evaluation team in advance of work in-country on opportunities for undertaking a limited gender analysis; this was not forthcoming. In addition, gender and HIV overall and gender and ART in particular in the South African context are extremely wide-ranging, complex, and challenging subjects that have already been well researched and documented and which could only be discussed in the most superficial way in this annex. Therefore, the decision was taken to attempt to provide specific gender-related comments on key SAG and U.S. Government/PEPFAR SA policy documents and reports. As is the case with all six areas under discussion in this annex, all comments are based on evaluation field site visits in four provinces and can in no way be viewed as representative or indeed generalizable. Please also see sections II and V in the report for brief discussions of a number of gender-related issues in the context of the development of the PEPFAR SA ART program over time. It is important to underline that all discussion here on gender refers to both women and men, boys and girls. SAG Documents and Gender Focus The South Africa CDC and USAID PEPFAR antiretroviral treatment program is in a period of transition. The primary impetus for this evaluation is increased SAG ownership and stewardship. The intention is for ―a sensible transition by PEPFAR to TA focus.‖ The SAG is making a 78 USAID/SOUTH AFRICA PEPFAR TREATMENT PROGRAM: FINAL EVALUATION transition into enhanced focus on a primary health care (PHC) model of service delivery and greater prioritization of U.S. Government technical assistance. All such changes are likely to have profound gender implications for both supply (health workers and managers) and demand-side (clients/patients) aspects of ART service delivery. Transition to a PHC model of service delivery is discussed in two key documents: ―Re-engineering Primary Health Care in South Africa: Discussion Document‖ (NDOH 2010), ―Re-vitalizing Primary Health Care in South Africa: Review of the PHC Package, Norms and Standards‖ (Rispel et al. 2010). The re-engineering document states: ―Many of the health problems are linked to the social determinants of health (―upstream factors‖) such as education and water which require a primary health care approach.‖ The social determinants of health approach usually also includes issues of gender, e.g., in relation to access to health education and services as an integral element. While neither the re-engineering nor the re-vitalizing document addresses gender, there is reference to the importance of equity, social justice, and rights. However, it might be pertinent to consider the potential supply- and demand-side specific gender ramifications of the major shifts planned in delivery of health services using a PHC model. The 2011 NDOH HRH SA 2030 Strategy (currently in draft) makes no mention of gender in its 114 pages, and gives minimal attention to issues of equity. As one example of why it is relevant to consider gender aspects of HRH: There is need to address health systems support provided to nurses who will deliver NIMART, who will overwhelmingly be women – issues of enrollment, retention, and supportive supervision all too often lack effective gender analysis. A general point that could be noted is that a major and growing criticism of the health systems‘ strengthening approach is its gender blindness. This is an example of the WHO building block framework can too easily enable a ―one-size-fits-all‖ position, where locally specific issues can receive too little attention. PEPFAR SA and Other U.S. Government Documents And GENDER FOCUS A priority gender focus of the Office of the Global AIDS Coordinator is ―Increasing Gender Equity in HIV/AIDS Activities and Services.‖ The chief document reviewed for this section of the annex is the U.S Government Office of the Inspector General. ―Audit of USAID/Southern Africa‘s Gender-related HIV/AIDS Activities.‖ Audit Report 4-674-11-004-P; January 4, 2011 (the Gender Audit). Another document partially reviewed was the 2010 ―South Africa PEPFAR Partner HIV/AIDS Facility-based Alignment Plan‖ – its Executive Summary and Attachment 2 (alignment scopes of work) were made available to the evaluation team. To discuss the Gender Audit first. It notes that in order to address women‘s greater vulnerability to HIV and AIDS, USAID/Southern Africa has implemented a number of activities in South Africa focusing on five high-priority gender strategies set forth by the Office of the Global AIDS Coordinator. These strategies include increasing women‘s legal protection, addressing male norms, and reducing violence against women. The Gender Audit reviewed five programs, none of which addresses increasing access to ART. The Gender Audit does not refer to any effective read-across between PEPFAR implementing partners specific to gender-focused interventions. Thus it is difficult to gauge whether there has been any sharing of gender-specific lessons learned, best practices, and gender analysis. Yet such read-across should be standard and monitored and evaluated effectively by teams that include dedicated gender expertise. USAID/SOUTH AFRICA PEPFAR TREATMENT PROGRAM: FINAL EVALUATION 79 There is reference in the Gender Audit to a Gender TWG Gender Self-assessment Tool, yet its use appears not to be a criterion that is consistently applied in terms of U.S. Government oversight of its PEPFAR implementing partners – or indeed internally administered by PEPFAR and CDC in their management of PEPFAR. There also appears to be scope for closer attention to analysis of sex-disaggregated data, which is the first essential step toward monitoring and evaluating the processes that might lead toward achievement of more gender-equitable approaches to ART service delivery, access, and uptake. The Gender Audit states: ―...targets for individual-level indicators should be disaggregated to establish expectations about a program‘s intended impact on men and women. This guidance was reinforced in the FY 2009 country operational plan...increasing gender equity is one of PEPFAR‘s five high-priority gender strategies. Despite this intent...none of the indicator targets set by partners for the activities reviewed in this audit were disaggregated by gender. This was generally true for activities beyond the scope of the audit as well.‖ This recommendation to disaggregate is entirely apposite, yet it is often at the level of sex￾disaggregated data that attention to gender begins and ends, which is insufficient. There is in addition need for a more social development equity and rights approach to provision of ART, which incorporates gender analysis. The Alignment Plan is the cornerstone of the district-level support by PEPFAR implementing partners to facility-based ART services. The PEPFAR alignment process promotes accountability to SAG district health work plans; the intention is that PEPFAR implementing partners will thereby support the SAG more effectively by responding to priorities identified within districts. The Alignment Plan is responsive to key SAG/NDOH policies and approaches. As previously discussed above, key SAG/NDOH documents such as HRH SA 2030 and the 2010 Re-engineering and Re-vitalising PHC discussion paper and report do not explicitly address gender aspects of the major shift occurring in provision of ART, whether from the supply or the demand-side and give minimal consideration to equity and health rights issues. The sections of the Alignment Plan reviewed by the evaluation team do not consider gender issues. Gender Recommendations Key Recommendations 1. Any organization seeking to participate in future implementation of the PEPFAR SA ART program should be required to demonstrate its prior attention to gender issues and to put forward action-oriented plans to mainstream such attention in future programming, with gender-sensitive and time-specific process and outcome indicators. 2. The U.S. Government in SA should itself review the extent to which gender has been internally mainstreamed within USAID and CDC and address any shortcomings and policy/ practice gaps. 3. Implementation should include as standard attention to how best to share gender lessons learned and best practices among PEPFAR implementing partners. 80 USAID/SOUTH AFRICA PEPFAR TREATMENT PROGRAM: FINAL EVALUATION Further Recommendations Because the HRH SA 2030 Strategy is currently in draft, there are opportunities for there to be proper gender analytical approaches to be applied to its finalization. Should opportunity still exist to review the Re-engineering and Re-vitalising PHC documents through a gender lens, this should be taken. There is a wealth of gender expertise in South Africa, with much of that expertise focused on health issues, gender-based and sexual violence in the context of HIV, and other issues entirely relevant to the PEPFAR SA ART program. Therefore, now is an opportune time to harness and make better use of such local expertise. The PEPFAR SA ART program should undertake gender analysis of how overall program indicators are monitored and evaluated. PEPFAR Alignment Recommendations Despite the need for the PEPFAR Alignment to be coherent with SAG approaches, there is nonetheless scope for discussion of national, provincial, and district-level gender implications of the move to district-based service delivery. There are many potential HRH and PHC re-engineering gender issues that PEPFAR implementing partners can play a key part in addressing. These include: The role of PEPFAR implementing partners in the alignment specific to TA that might strengthen gender awareness in district planning and implementation The possibility of providing gender training to District Management Teams Gender-specific support to implementation of the School Health Programme Consideration of the development of gender-specific supply and demand-side indicators that reflect district-level issues Focus on developing and implementing district-level initiatives that increase access to ART for the underserved and the most vulnerable TEST-AND-TRAT The current situation is that ―test-and-treat‘ is speculative, based on highly theoretical modeling and has not been part of any U.S. Government or SAG activities to date. Moreover, the SAG is currently striving toward providing ART access to all individuals who require treatment. Until that objective has been met and stability achieved in patient support and follow-up (especially in the context of the transition to PHC focus), test-and-treat is not only speculative but a distant goal for South Africa. Test-and-treat also leans more toward HIV prevention than treatment; therefore, it should be assessed in the context of HIV prevention, which is beyond the scope of this evaluation. The comments that follow set out some of the current thinking on test-and-treat and treatment as prevention. It should be noted that the points are presented from a biomedical, supply-side perspective. USAID/SOUTH AFRICA PEPFAR TREATMENT PROGRAM: FINAL EVALUATION 81 TREATMENT AS PREVENTION Antiretroviral therapy(ART) has been remarkably effective in ameliorating HIV-associated morbidity and mortality. However, each year there are more than twice as many new infections as there are people starting ART.13Ever increasing numbers of HIV-positive people maintained on expensive drugs that must be taken regularly and for life, is not sustainable. However, the rapid decline in viral load during ART presents an opportunity to develop a ―treatment as prevention‖ [TasP] strategy in order to reduce HIV transmission at a population level. Modeling exercises have demonstrated that for TasP to be effective, a ―test-and-treat‖ policy of universal and regular HIV-testing, combined with immediate initiation and continued adherence to ART, will be required.1415Effectiveness of TasP is dependent on achieving a very high coverage of ART in the HIV-infected population. The arguments against TasP have generally been concerned with practical issues of implementation including stigma, acceptability, compliance, side effects, viral load suppression, viral rebound, treatment failure and drug resistance.16171819 Additionally it will be important to ensure a reliable and sustainable drug supply-chain is in place, that the tools for monitoring the impact of TasP on both individual patients and the population are available, and that the short￾and long-term costs of using TasP can be met.20 For all the difficulties, acknowledged and to come, treatment may offer the best hope for rapidly reducing HIV transmission. UNIQUE IDENTIFIERS The development of a national unique identifier system, currently used only in the Western Cape, would strengthen monitoring of patients‘ health and tracking, as patients accessing ARVs for the first time could be differentiated from ―transfers out‖ who may be moving between programs. This would allow CD4 and viral load to be tracked longitudinally per patient, providing information on CD4 when first receiving HIV testing, when ART is first initiated, and at regular intervals. Health providers require patient data for individual case management, including a history of major clinical events and laboratory parameters, which identify the level of immune 13WHO/UNAIDS/UNICEF. Towards Universal Access: Scaling up priority HIV/AIDS interventions in the health sector. Geneva: United Nations Joint Programme on AIDS. 14Granich, R.M., Gilks, C.F., Dye, C., De Cock, K.M., Williams, B.G. ―Universal voluntary HIV testing with immediate antiretroviral therapy as a strategy for elimination of HIV transmission: a mathematical model.‖ Lancet 373:48-57 2009. 15Montaner, J.S., Hogg, R., Wood, E., Kerr, T., Tyndall, M., Levy, A.R., Harrigan, P.R. ―The case for expanding access to highly active antiretroviral therapy to curb the growth of the HIV epidemic.‖ Lancet 368:531-536 2006. 16Walensky, R.P., Paltiel, A.D., Losina, E., Morris, B.L., Scott, C.A., Rhode, E.R., Seage, G.R., Freedberg, K.A. ‗Test and treat DC: forecasting the impact of a comprehensive HIV strategy in Washington DC.‘ Clin Infect Dis 51:392-400 2010. 17Wagner, B.G., Kahn, J.S., Blower, S. ―Should we try to eliminate HIV epidemics by using a 'Test and Treat' strategy?‖ AIDS 24:775-776 2010. 18Smith, R.J., Okano, J.T., Kahn, J.S., Bodine, E.N., Blower, S. ―Evolutionary dynamics of complex networks of HIV drug-resistant strains: the case of San Francisco.‖ Science 327:697-701 2010. 19Garnett, G.P., Baggaley, R.F. ―Treating our way out of the HIV pandemic: could we, would we, should we?‖ Lancet 373:9-11 2009. 20Montaner, J.S., Hogg, R., Wood, E., Kerr, T., Tyndall, M., Levy, A.R., Harrigan, P.R. ―The case for expanding access to highly active antiretroviral therapy to curb the growth of the HIV epidemic.‖ Lancet 368:531-536 2006. 82 USAID/SOUTH AFRICA PEPFAR TREATMENT PROGRAM: FINAL EVALUATION compromise, adherence, and response to therapy. Consolidation of data within clinics has enabled some program evaluation, including temporal changes in clinic caseload and on-treatment analysis of population responses to therapy. At the same time, the absence of data has been recognized as a potential measure of program losses; data may be missing for a variety of reasons. PEPFAR has supported clinics in investing resources to maintain these clinic-based databases. Amalgamation of PEPFAR indicators from multiple providers has been collated in the data warehouse and used as a measure of overall program performance. With the expansion of the national ART program to all health districts, treatment is now widely available to a mobile population. Analysis of the database maintained by one of the largest PEPFAR implementing partners shows that up to 38% of patients starting ART within the program had already suppressed viral loads (<400 cpm), an indication of a high proportion of patients transferring into the program. This same program reported 12.4% of patients had transferred out of the program. It is probable that a proportion of patients identified as lost to follow up will access care in new clinics. The current national ART program is now an interconnected network rather than a series of isolated clinics; there is increasing need to track patients throughout this network. Effective patient tracking requires use of a unique personal identifier, which would be valid at all clinics visited by patients. Benefits of such a national system for program evaluation would include decreased multiple counting of patients and increased ability to match district and regional resources to need. A unique identifier would also allow linkage of ARV program to other national databases such as the electronic tuberculosis and death registers. Potential benefits would accrue to patients if the unique identifier could also facilitate relevant health information transfer to new care providers. However, potential harms include breaches of confidentiality and denied access to treatment of vulnerable groups, such as illegal immigrants without national identification documents. SAG INFRASTRUCTURE This question is interpreted by the evaluation team to refer to actual physical infrastructure of public health facilities. This subject has already been considered very briefly in Section IV and there is no further information that can be provided here. Any attention to SAG infrastructure in terms of institutional structures and management frameworks is beyond the scope of the evaluation. TB SERVICES It was made clear to the evaluation team that TB services were not to be addressed; focus was to be on the integration of TB and HIV services. In addition, the reader is referred to the comprehensive, recent review on TB and HIV services undertaken by the NDOH and CDC – it is currently in draft (Chebab et al. (June) 2011). Please see Section III of the main report. HUMAN RESOURCE TRANSITION The understanding of the evaluation team is that this request primarily refers to the transition of staff members currently working with PEPFAR-supported organizations to employment by the NDOH as a result of the transition from service provision to technical assistance. Proper attention to this topic is far beyond the scope of the evaluation and would require a separate study. Despite these constraints, here too the team has sought to make comments. A major issue was frequently mentioned by PEPFAR implementing partners in the context of human resource transition. It is one that is pertinent to the PEPFAR transition from direct USAID/SOUTH AFRICA PEPFAR TREATMENT PROGRAM: FINAL EVALUATION 83 service delivery to technical assistance: not all members of staff currently employed by PEPFAR implementing partners to provide services cannot easily or appropriately make a shift to provision of technical assistance. To make an obvious point: not all biomedical staff can provide TA. One subject brought up by several implementing partner representatives and other key informants, and one that was said to have led to considerable disquiet, is that there appears to be a PEPFAR perception that the transition to TA will allow the same close attention to mentoring that has been a feature of direct service delivery. This was felt not to be the case at all; one reason given was that service delivery at dedicated sites allows the development over time of close, supportive mentoring. A further issue that may be of importance is: what steps have been taken by PEPFAR to ensure that the NDOH, provincial DOHs, and district-level management teams have had sufficient lead time to assimilate all relevant staff members whose salaries have previously been paid by implementing partners? Comments were made during evaluation hub meetings, at which implementing partner representatives were present, regarding the overall absence of sufficient information and opportunities for joint planning. Several points are made here in the context of wider human resources for health (HRH) challenges. Especially concerned comments about overall HRH challenges were made in Western Cape. The situation is apparently that PEPFAR has provided significant funding for payment of salaries, yet there has been no discussion with the Western Cape DOH as to the degree to which it might in the future be expected to take over any or all of those salaried posts. Thus there is perceived to have been minimal joint HRH planning and a lack of PEPFAR transparency on this matter. This issue of staff transfer/absorption is clearly one of real concern and some confusion for a number of implementing partners: What happens to staff members if the provincial DOH does not employ them once the implementing partner moves out of a district due to the alignment process? One topic that was mentioned several times by implementing partners and also by provincial DOH representatives was: lower cadres [nurses] simply cannot be expected to shoulder most of the PHC burden. Yet this appears to be what is envisaged in the thrust toward delivery of ART at the PHC level, where nurses will still be required to fulfill all other PHC duties. Social workers and community outreach workers: indications were found during evaluation fieldwork of good intentions on the part of both PEPFAR implementing partners and the relevant government department/ministry to absorb such cadres; the need to continue to employ such expertise was widely acknowledged. However, issues of budget allocations for salaries remain unclear, as does management within a necessarily intersectoral framework (DOH and other ministries). The danger was perceived of restricting social and community outreach workers to only one department/ ministry in the future, compared to previously creative and intersectoral approaches initiated by PEPFAR implementing partners: silos might emerge, with consequent restricted practice. Creative mechanisms are required to ensure optimal utilization of this cadre of support staff to the benefit of the various community members they serve. One option might be for them to be absorbed in local government structures than provincial departments. 84 USAID/SOUTH AFRICA PEPFAR TREATMENT PROGRAM: FINAL EVALUATION Best Practice Within Broadreach KZN staff identification and appointment was done within the existing structure of the DOH, so no additional staff members were appointed. This seems to be a more sustainable approach, as the KZN DOH will not need to absorb staff and take them on to the payroll, as those health workers are already employed by the department. This was the only example of such close partnership observed during the course of the evaluation fieldwork (this is not to claim that other such approaches do not exist or have not been successfully applied). USAID/SOUTH AFRICA PEPFAR TREATMENT PROGRAM: FINAL EVALUATION 85 ANNEX G. REFERENCES It should be noted that, while not all references in this annex are mentioned in the body of the report, all documents listed here have been reviewed for the purposes of the PEPFAR SA ART program evaluation. Adam, M.A., and L.F. Johnson. ―Estimation of adult antiretroviral treatment coverage in South Africa.‖ South African Medical Journal 99: 661–667 2011. Aids2031 Costs and Financing Working Group. The Long-run Costs and Financing of HIV/AIDS in South Africa. Washington, DC: Results of Development Institute 2010. Anova Health Institute. 2010 Annual Report. Johannesburg: AHI 2010. Aurum Institute. Interim Progress Report 21 April 2011. Johannesburg: The Aurum Institute 2011. Bärnighausen, T., F. Tanser, A. Malalza,and M.L. Newell. ―Antiretroviral treatment and participation in HIV surveillance in rural KwaZulu-Natal.‖ Paper prepared for the 5th South African AIDS Conference. Durban 7-9 June 2011. Bärnighausen, T., V. Xolo, and G.S. Cooke. ―Evaluating the performance of antiretroviral treatment programs: mortality and loss to follow-up.‖ PLoS Med, e298 2007. Bassett, I. ―Who starts ART in Durban, South Africa?...Not everyone who should.‖ Paper prepared for 5th IAS Conference on HIV Pathogenesis, Treatment and Prevention. Cape Town: 19-22 July, 2009. Bedelu, M., N. Ford, K. Hilderbrand, H. Reuter (2007). ―Implementing Antiretroviral Therapy in Rural Communities: The Lusikisiki Model of Decentralized HIV/AIDS Care.‖ J Infect Dis, 196: S464-S468 2007. Bor, J., T. Bärnighausen, C. Newell, F. Tanser & M.L. Newell. ―Social exposure to an antiretroviral programme in rural KwaZulu-Natal. Online publication (?) Tropical Medicine and International Health 2011. Boulle, A., P. Bock, M. Osler, K. Cohen, L. Channing, et al. ―Antiretroviral therapy and early mortality in South Africa.‖ Bulletin of the World Health Organization 86: 678–687 2008. Boulle, A., P. Bock, M. Osler, K. Cohen, L. Channing, K. Hilderbrand, E. Mothibi, V. Zweigenthal, N. Slingers, K. Cloete, F. Abdulla. ―Antiretroviral therapy and early mortality in South Africa.‖ Bull. World Health Organization 2008, 86:678-687 2008. Boulle, A., G. Van Cutsem, K. Hilderbrand, C. Cragg, M. Abrahams, et al. ―Seven-year experience of a primary care antiretroviral treatment programme in Khayelitsha, South Africa.‖ AIDS 24: 563–572 2010. Chebab, J.C., K. Vilakazi-Nhlapo, P. Vranken, A. Peters, J.D. Klausner. Current Implementation of the Republic of South Africa National TB HIV Integrated Services’ Program. Draft report. Pretoria: NDOH & CDC, June 2011. Colvin, C.J., L. Fairall, S. Lewin, D. Georgeu, M. Zwarenstein, et al. ―Expanding access to ART in South Africa: The role of nurse-initiated treatment.‖ South African Medical Journal 100: 210–212 2010. Cooke, G.S., F.C Tanser, T. Bärnighausen, M.L. Newell. ―Population uptake of antiretroviral treatment through primary care in rural South Africa.‖ BMC Public Health 2010 10: 585 2011. 86 USAID/SOUTH AFRICA PEPFAR TREATMENT PROGRAM: FINAL EVALUATION Cristofides, N., D. Muirhead, R. Jewkes, L. Penn-Kekana & N. Conco. Including post-exposure prophylaxis to prevent HIV/AIDS into post-sexual assault health services in South Africa: costs and cost￾effectiveness of user preferred approaches to provision. Pretoria: Medical Research Council, 2006. Department of Health, Western Cape. The phased implementation of the District Health System in the Western Cape Province. Cape Town: DOH (undated). Fatti, G. et al. ―Increased vulnerability of rural children on antiretroviral therapy attending public health facilities in South Africa: a retrospective cohort study.‖ Journal of the International AIDS Society 13:46 2010. Fatti, G., P. Bock, A. Grimwood, B. Eley. ―Increased vulnerability of rural children on antiretroviral therapy attending public health facilities in South Africa: a retrospective cohort study.‖ Journal of the International AIDS Society, 13:46 2010. Fatti, G., A. Grimwood, P. Bock. ―Better Antiretroviral Therapy Outcomes at Primary Healthcare Facilities: An Evaluation of Three Tiers of ART Services in Four South African Provinces.‖ PLoS ONE 5(9): e12888.doi:10.1371/journal.pone.0012888 2010. Fatti, G., P. Bock, B. Eley, E. Mothibi & A. Grimwood. ―Temporal trends in baseline characteristics and treatment Outcomes of children starting antiretroviral treatment: An analysis in four provinces in South Africa, 2004 – 2009.‖ Paper & poster prepared for 5th South African AIDS Conference (Track Two). Durban 7-9 June 2011. Forster, M., C. Bailey, M. Brinkhof, C. Graber, A. Boulle, et al. ―Electronic medical record systems, data quality and loss to follow-up: survey of antiretroviral therapy programmes in resource-limited settings.‖ Bulletin of the World Health Organization 86: 939–947 2008. Foundation for Professional Development. Annual Report2010/2011. Pretoria 2011. George, G., T. Quinlan &C. Reardon. Human Resources for Health: A Needs and Gap Analysis of HRH in South Africa. Durban: University of KwaZulu-Natal, Health Economics and HIV & AIDS Research Division, November 2009. Government of South Africa & National Department of Health. HIV & AIDS and STI Strategic Plan for South Africa 2007-2011. Pretoria: 2007. Government of South Africa and Government of the USA. Partnership Framework in Support of South Africa’s National HIV & AIDS and TB Response 2012/2013 – 2016/2017 between The Government of the Republic of South Africa and the Government of the United States of America. Pretoria & Washington, DC: SAG & USG, December 2010. Grimwood, A., E. Mothibi, G. Fatti, R. Ally. ―Encouraging PMTCT outcomes over an 18 month period, Q1 October 2009 to end Q2 March 2011, at Kheth‘Impilo(KI) supported PHC and district hospital sites.‖ Poster 706 prepared for the 5th South African AIDS Conference (Track Two). Durban, 7-9 June 2011. Herbst, A.J., G.S. Cooke, T. Bärnighausen, A. Kanykany, F. Tanser & M.L. Newell. ―Adult mortality and antiretroviral treatment roll-out in rural KwaZulu- Natal, South Africa.‖ Bull. World Health Organization 87(10): 754-762 2009. Igumbor, J. O. , E. Scheepers, R. Ebrahim, A. Jason, & A. Grimwood. ―An evaluation of the impact of a community-based adherence support programme on ART outcomes in selected government HIV treatment sites in South Africa.‖ AIDS Care 23: 2, 231-236 2011. Johnston, V.J., K. Fielding, S. Charalambous, M. Mampho, G. Churchyard, A. Phillips & A. D. Grant. ―Second-line Antiretroviral Therapy in Community and Workplace-based Treatment Programmes in South Africa: determinants of virological outcome.‖ (Undated - draft paper?). USAID/SOUTH AFRICA PEPFAR TREATMENT PROGRAM: FINAL EVALUATION 87 Jooste, J.P., A.J.M van Zyl, A. Baker, W. Crawford & A. Jassen. ―Antiretroviral therapy in the Northern Cape.‖ S Afr Med J 95:12 2008. Kheth‘Impilo. PEPFAR/SA Treatment Program Evaluation 2011. Cape Town: Kheth‘Impilo AIDS-free Living 2011. Khulisa Management Services (Pty) Ltd. Summarized Findings of 20 PEPFAR Partner On-site Performance Assessments. Summary Report – TO4- 4 April 2011. Johannesburg: 2011. Klausner, J.D., C. Serenata, H. O‘Bra, C.L. Mattson, J.W. Brown, et al. ―Scale-Up and Continuation of Antiretroviral Therapy in South African Treatment Programs, 2005–2009.‖ J Acquir Immune Defic Syndr 56:292–294 2011. Lesser, R.J., P.C. Mutevedzi, G.S Cooke, M. L. Newell. ―Retention in HIV Care for Individuals Not Yet Eligible for Antiretroviral Therapy: Rural KwaZulu Natal, South Africa.‖ J Acquir Immune Defic Syndr S6; e79-e86, 2011. Mutevedzi, P.C., R.J. Lessells, T. Heller, T. Bärnighausen, G.S. Cooke, M. L. Newell. ―Scale-up of a decentralized HIV treatment programme in rural KwaZulu-Natal, South Africa: does rapid expansion affect patient outcomes?‖Bull. World Health Organization 2010. National AIDS Council, South Africa. HIV & AIDS and STI Strategic Plan for South Africa: 2007– 2011. Pretoria: NAC 2007. National AIDS Council, South Africa. Increasing Access to Integrated Tuberculosis and HIV Services at the Primary Health Care and Community Levels. South Africa Round 10 Global Fund Proposal (2010). Pretoria: NAC 2010. National Department of Health, South Africa. Operational Plan for Comprehensive HIV and AIDS Care, Management and Treatment for South Africa. Pretoria: NDOH 2003. National Department of Health, South Africa. South African National Antiretroviral Treatment Guidelines. Pretoria: NDOH 2004. National Department of Health, South Africa: Guidelines for the management of HIV infected children.[http://www.doh.gov.za/docs/factsheets/ guidelines/hiv/index.html]. Pretoria: NDOH 2005. National Department of Health, South Africa. Operational Plan for Accelerating Scale up and Improvement of the Quality of Services for Prevention of Mother to Child Transmission (PMTCT) in the context of Integrated Maternal and Child health care in South Africa 2009. Pretoria: NDOH 2009a. National Department of Health, South Africa. National HIV and Syphilis Prevalence Survey South Africa 2008. Pretoria: NDOH, 2009b. National Department of Health, South Africa. The National Integrated Prevention of Mother-To-Child Transmission (PMTCT) of HIV Accelerated Plan at a Glance. Pretoria: NDOH, 2010a. National Department of Health, South Africa. Re-engineering Primary Health Care in South Africa: Discussion Document. Pretoria: NDOH November, 2010 (2010b). National Department of Health, South Africa. The South African Antiretroviral Treatment Guidelines 2010. Pretoria: NDOH November, 2010 (2010c). National Department of Health, South Africa. HRH SA 2030. Draft Strategy for the Health Sector 2012/13-2016/17. Pretoria: NDOH, 2011a. National Department of Health, South Africa. “Towards Quality Care for Patients’. National Care Standards for Health Establishments in South Africa. Pretoria: NDOH, 2011b. 88 USAID/SOUTH AFRICA PEPFAR TREATMENT PROGRAM: FINAL EVALUATION National Department of Health, South Africa & South African National AIDS Council. Clinical Guidelines: PMTCT (Prevention of Mother -to- Child Transmission) Pretoria: NDOH & NAC, 2010. National Treasury, South Africa. Vote 16 Health. Estimates of National Health Expenditures 2011. Pretoria: National Treasury, 23 February 2011. Odendal L. ―Mothers2mothers: mothers helping mothers to save babies.‖ HIV & AIDS Treatment in Practice Issue 17.8. 1 July 2011. Patel, S.D., E. Larson, T. Mbengashe, H. O‘Bra & J.W. Brown.―Pediatric Outcomes of the President‘s Emergency Plan for AIDS Relief in South Africa, 2005-2010.‖ AIDS 2011. PEPFAR South Africa Team. PEPFAR South Africa Care and Support Review. Pretoria: July 26 – August 6, 2010. PEPFAR South Africa Team. PEPFAR South Africa Country Operational Plan Report FY 2010. Pretoria: 2011a. PEPFAR South Africa Team. ―From District Alignment to Comprehensive District Support.‖ PEPFAR PPL Meeting March 30, 2011. Pretoria: 2011b. PEPFAR South Africa Team. South Africa PEPFAR Partner HIV/AIDS Facility-Based Alignment Plan. Attachment 2: Alignment Scopes of Work. (undated). Right To Care. PEPFAR Support for ART Programme: Evaluation Hub Meeting, Cape Town. Cape Town: RTC 30 June 2011. Presentation prepared by Dr. Theo Van Handel. Rispel, L., J. Mourman, M. Chersich, J. Goudge, N. Nxumalo & T. Ndou. Re-vitalising Primary Health Care in South Africa: Review of the PHC package, norms and standards. Johannesburg: University of the Witwatersrand, School of Public Health, Centre for Health Policy, November 11, 2010. Rosen, S. Innovations in Costing Health Care Delivery Models. Boston University: Experience Center for Global Health & Development. Boston, USA: BU ECGH&D, February 10, 2011. Smart, T. ―South Africa is moving towards an AIDS-free generation, and improving survival in children living with HIV.‖ Editorial in HIV & AIDS Treatment in Practice (HATIP) 178, 1 July 2011. Smart, T. ―South African AIDS Conference: Special Issue.‖ HATIP Issue 178, 1 July 2011. Tanser, F: ―Methodology for optimising location of new primary health care facilities in rural communities: a case study in KwaZulu-Natal, South Africa.‖ J Epidemiol Community Health 60(10):846-850 2006. UNAIDS. Global report: UNAIDS report on the global AIDS epidemic 2010. Geneva: UNAIDS, 2010. USAID Office of the Inspector General. Audit of USAID/southern Africa’s Gender-related HIV/ AIDS Activities. Pretoria: Audit Report # 4-674-11-004-P. January 4, 2011. U.S. Government Mission to South Africa. Inter-Agency Annual Program Statement UNAIDS/WHO AIDS epidemic update: November 2009. Pretoria: USG, 2009. U.S. Government, Office of the Global AIDS Coordinator. PEPFAR 5-year strategy. Washington, DC: OGAC, 2009a. U.S. Government, Office of the Global AIDS Coordinator. PEPFAR Next Generation Indicators: Reference Guide. Version 1.0. Washington, DC: OGAC, 2009b. U.S. Government, State Department. Implementation of the Global Health Initiative; Consultation Document. Washington, DC: State Department, 2009. USAID/SOUTH AFRICA PEPFAR TREATMENT PROGRAM: FINAL EVALUATION 89 US Government Strategic Information Team in South Africa. The United States President‘s Emergency Plan for AIDS Relief (PEPFAR) South Africa Strategic Information (SASI) Manual. Version 5.01 March 2010. Pretoria: USG, 2010. Welz, T., V. Hosegood, S. Jaffar, J. Batzing-Feigenbaum, K. Herbst & M. L. Newell. ―Continued very high prevalence of HIV infection in rural KwaZulu-Natal, South Africa: a population-based longitudinal study.‖ AIDS 21:1467-1472 2007. World Health Organization. Antiretroviral therapy of HIV infection in infants and children in resource￾limited settings: towards universal access. Recommendations for a public health approach. Geneva: WHO, 2006. World Health Organization. Antiretroviral therapy for HIV infection in adults and adolescents: recommendations for a public health approach. Geneva: WHO, 2010. 90 USAID/SOUTH AFRICA PEPFAR TREATMENT PROGRAM: FINAL EVALUATION USAID/SOUTH AFRICA PEPFAR TREATMENT PROGRAM: FINAL EVALUATION 91 ANNEX H. THE COMPREHENSIVE PEPFAR SA ART PROGRAM EVALUATION SCHEDULE (JUNE-JULY 2011) Table of Contents Sheet N# Description 1 Team and Member Details 2 Gauteng Key Stakeholders - Clients; KIIs etc 3 Gauteng Meeting Schedule for KIIs 4 Gauteng Hub Meeting: 110623 5 Gauteng Site Visit Schedule and KIIs Summary 6 KwaZulu Natal Key Stakeholders 7 KwaZulu Natal Hub Meeting: 110627 8 KwaZulu Natal Site Visit Schedule and KIIs Summary 9 Western Cape Key Stakeholders 10 Western Cape Hub Meeting: 110630 11 Western Cape Site Visit Schedule and KIIs Summary Sheet N# Description 12 Eastern Cape Key Stakeholders 13 Eastern Cape Hub Meeting: 110627 14 Eastern Cape Site Visit Schedule and KIIs Summary Total Summary 15 Summary of Agencies; Prime Partners & Projects 16 Full List: PEPFAR Partners; Subpartners; Projects; Project Site Info 92 USAID/SOUTH AFRICA PEPFAR TREATMENT PROGRAM: FINAL EVALUATION Full Name Organisation Designation Email Address / Website Janet Gruber GHTech Team Leader Lois Eldred GHTech Consultant Mary Pat Selvaggio Khulisa Director mpselvaggio@khulisa.com Robin Wood GHTech Consultant Ntombi Bandezi Khulisa Consultant Elna Hirschfeld Khulisa Associate Director ehirschfeld@khulisa.com Peter Njaramba Khulisa Sen Associate pnjaramba@khulisa.com Jenna Kamps Khulisa Research Consultant jkamps@khulisa.com Ntombi Bandezi Khulisa Khulisa consultant Zandile Wanda Khulisa Khulisa consultant Chris Chetty Khulisa Khulisa consultant Dr Wim Brown Strategic Information Strategic Information jwbrown@usaid.gov Dr Melinda Wilson mwilson@usaid.gov Roxana Rogers rrogers@usaid.gov Christy Mulinder Charles Mandivenyi CMandivenyi@usaid.gov Wendy Benzerga Likza Iglesias Regional Inspector General‘s Office lIglesias@usaid.gov Department of Health: National Dr. Yogan Pillay Selokela (PA) leshas@health.gov.za Assit Dir Magda Fouri fourim@health.gov.za Dr TobieMbengashe Director mbengt@health.gov.za Deneo PA kekand@health.gov.za Dr Lilian Diseko Dr Peter Barron On leave Dr Thurma Goldman GoldmanT@sa.cdc.gov PA for Dr Goldman: Saroj Pillay USAID/SOUTH AFRICA PEPFAR TREATMENT PROGRAM: FINAL EVALUATION 93 Full Name Organisation Designation Email Address / Website Dr Jeff Klausner klausnerj@sa.cdc.gov Heidi O‘Bra CDC/PEPFAR Coordinator ObraH@sa.cdc.gov Derek Kunaka ESI COP for the JSI ESI Project derek@enhancesi.co.za James Maloney US Embassy PEPFAR Coordinator Various Provincial PEPFAR Liaison Officers (PPLOs) Masilo Marumo PEPFAR Gauteng PPL marumom@state.gov Coceka Nogoduka PEPFAR Eastern Cape PPL CNogoduka@pactsa.org.za Jessica Rebert PEPFAR Western Cape PPL rebertj@sa.cdc.gov Chalone Savant PEPFAR KZN Cape PPL savantcr@state.gov Lauren Marks PEPFAR NDOH PEPFAR Liaison lmarks@usaid.gov Celicia M. Serenata Clinton Health Access Initiative (CHAI) Deputy Country Director cserenata@clintonHealthAccess.or g Bob Fryatt Karen Tobias (PA of Bob Fryatt) Pending Pending Dr. Kisbey-Green Tsohane Puso The Aurum Institute Operations Manager: Ekurhuleni North tpuso@auruminstitute.org HE2RO Dr. Gesine Meyer-Rath Health Economics & Epidemiology Research Office (HE2RO) Themba Lethu Clinic, Helen Joseph Hospital Perth Road, Westdene, 2092, South Africa. Assistant Professor: Center for Global Health and Development Boston University Boston, US gesine@bu.edu 94 USAID/SOUTH AFRICA PEPFAR TREATMENT PROGRAM: FINAL EVALUATION DATE TIME ACTIVITY WHERE Janet Lois Robin Ntombi Elna Peter Jenna Mary Pat Province: Gauteng Mon 20 June KIIs 8:30 Team engagements and preparations X X X X X 10:00 Celicia M. Serenata (Confirmed) Deputy Country Director Clinton Health Access Initiative (CHAI) 2nd Floor Hilda Chambers 339 Hilda Street Hatfield Tel: +27 (82) 413-2381 X X X 11:30 Team engagements and preparations X X X X X 1:00 LUNCH 2:30 Win Brown (Confirmed) USAID Mobile: 083 443 6603 X X X 4:00 Charles Mandivenyi (Postponed) USAID Mobile: 082 321 7808 X X X 6:00 Team Meeting ESI Training Venue X X X X X Tues 21 June Site Visit & KIIs 8:45 Masakhane Clinic: Tembisa hospital Tsohane Puso: Operations Manager: Ekurhuleni North 3 Thami Mnyele Dr, Tembisa 1632 X X X X X 1:00 LUNCH 2:30 Team meeting X X X X X X 4:00 DOH (Lilian; Peter; Tobi not confirmed) X X X X X X 6:00 Team Meeting ESI Training Venue X X X X X X Wed 22 June KIIs 8:30 Melinda Wilson, USAID (Confirmed) USAID Mobile: 083-443- 6613 X X X And USAID/SOUTH AFRICA PEPFAR TREATMENT PROGRAM: FINAL EVALUATION 95 DATE TIME ACTIVITY WHERE Janet Lois Robin Ntombi Elna Peter Jenna Mary Pat DOH (Lilian; Peter; Tobi not confirmed) X X 10:00 Derek Kunaka (Confirmed) ESI X X X And X DOH (Lilian; Peter; Tobi not confirmed) X X 11:30 Roxana Rogers (postponed) USAID X X X X 1:00 LUNCH & Team Leading 2:30 Charles Mandivenyi￾M&E Officer of the Mission (Confirmed) USAID Mobile: 082 321 7808 X X X X ? 4:00 Dr Jeff Klausner (Confirmed) CDC SA Mobile: 082 655 0728 X X X X ? And DOH (Lilian; Peter; Tobi not confirmed) X X 6:00 Team Meeting ESI Training Venue X X X X ? Thurs 23 June Hub Meeting & KIIs 8:30 Hub Meeting – Gauteng Province(Morning and working lunch 8:30-2:30) with GP based PEPFAR ART Partners: (head office reps) Venue: RTC - Helen Joseph Hospital X X X X X X 2:00 Team Meeting Khulisa Management Services X X X X X X X X X X X X Fri 24 June Site Visits and KIIs 8:30 Alexandra Clinic (RTC) Team A X X X Hillbrow Clinic (WRHI) Team B X X X 96 USAID/SOUTH AFRICA PEPFAR TREATMENT PROGRAM: FINAL EVALUATION DATE TIME ACTIVITY WHERE Janet Lois Robin Ntombi Elna Peter Jenna Mary Pat 1:00 Team Meeting Khulisa Management Services X X X X X X 2:00 Bob Fryatt (Confirmed) DFID: Department of Health, Room 2128, Civitas Building, Cnr Struben and Andries Street. Call Ext. 8851 X X X X X X And DOH (Lilian; Peter; Tobi not confirmed) X X X X X X Dr Gesine Meyer-Rath (Confirmed) Mobile: 076-528 -1767 Health Economics & Epidemiology Research Office (HE2RO) Themba Lethu Clinic, Helen Joseph Hospital Perth Road, Westdene, 2092, South Africa. X X X X X X Sat 25 June REST DAY Sun 26 June Travel Day to Provinces Mon 27 June - 2 July Fieldwork in Provinces: Western Cape; Eastern Cape and KwaZulu-Natal Sub 3 July REST DAY Mon 4 July Drafting of Report 8:30 Team Meeting: Pre￾drafting & Joint drafting: Full day X X X X X X Tues 5 July Drafting of Report 8:30 Drafting up to midday together X X X X X 2:00 Dr Thurma Goldman (Confirmed) CDC 012-424-9025 X X USAID/SOUTH AFRICA PEPFAR TREATMENT PROGRAM: FINAL EVALUATION 97 DATE TIME ACTIVITY WHERE Janet Lois Robin Ntombi Elna Peter Jenna Mary Pat & KIIs 2:00 PPLO Joint Meeting (Confirmed) Premium Hotel X X Wed 6 July Drafting of Report & KIIs 8:30 Drafting of report X X X X X X 2:00 Drafting of draft report & prep for debriefing meeting X X X X X Thurs 7 July Drafting of Report & Briefing 8:30 Finalization of draft report & prep for debriefing meeting X X X X X 3:00 Debriefing meeting: USAID X X X X X Fri 8 July Finalizat ion of Draft Report 8:30 Finalization of draft report & prep for debriefing meeting X X X X X Outstanding KIIs 1 Likza Iglesias (USAID/Regional Inspector General‘s Office) 2 If not in office contact: 3 Robin Mason: rmason@usaid.gov or 4 Emily Mhlanga: 012-452-2135 5 McKinsey (Ownership study) 6 Tulani Masimela: NDOH - Strategic Planning - Information Systems: 012 395 8420; masim@health.gov.za 98 USAID/SOUTH AFRICA PEPFAR TREATMENT PROGRAM: FINAL EVALUATION For more information, please visit http://www.ghtechproject.com/resources.aspx Global Health Technical Assistance Project 1250 Eye St., NW, Suite 1100 Washington, DC20005 Tel: (202) 521-1900 Fax: (202) 521-1901 www.ghtechproject.com