December 2014 This publication was produced at the request of the United States Agency for International Development. It was prepared independently by Edna Berhane, Peter Njaramba, Shalote Chipamaunga and Cheryl Goldstone with support from Mary Pat Selvaggio of Khulisa Management Services (Pty) Ltd. EVALUATION Mid-term Performance Evaluation of the USAID Sexual HIV Prevention Program (SHIPP) Mid-term Performance Evaluation of the SHIPP Project Evaluation Report ACKNOWLEDGEMENTS The evaluation team wishes to thank all the staff members at various South African Government departments who met with us, including the Office of the Deputy President, the South African National AIDS Council (SANAC), the National Department of Health (NDOH), the Department of Basic Education (DBE), the Department of Social Development (DSD), Department of Public Service Administration (DPSA), and Department of Higher Education and Training (DHET). Thanks also to all SAG department respondents in Gauteng, KwaZulu-Natal and Mpumalanga. The team extends its appreciation to Zululand, Gert Sibande and Ehlanzeni District Municipalities as well as City of Johannesburg for their information. The KwaZulu-Natal Provincial AIDS Council, Mpumalanga AIDS Council, and Gauteng Province AIDS Council as well as AIDS Councils in Johannesburg, Zululand, Gert Sibande, and Ehlanzeni are also thanked for their time and support. The team also wishes to thank the USAID/South Africa staff who supported this evaluation, especially the Contracting Officer’s Representative for this evaluation, Paul Pleva, and the Contracts Officer, Adam Walsh. The team highly appreciates the valuable information provided during the evaluation from Mathata Madibane, SHIPP’s Activity Manage; Joan Littlefield, SHIPP Prevention Program Office; Shalone Savant, and Marumo Masilo PEPFAR Liaison Officers in KwaZulu-Natal and Gauteng respectively. Special thanks go to the SHIPP staff who supported this evaluation and provided us with the documentation and information we needed. Particular thanks go to the Chief of Party, Doris Macharia, who coordinated all of SHIPP’s support to the evaluation team. Thanks also go to SHIPP’s small grant recipients who responded to the online survey as well as the sampled small grant recipients that participated in the evaluation, including in Gauteng, Azali Health Care, The PhilaMzantsi Diepsloot Consortium (ESST and Diamond Life) and Camp La Thuso; in KwaZulu-Natal Sinethemba, Khulanathi, and Thuthukisa Abantulayo; and in Mpumalanga GRIP, Sifiso Sethu, AIDS Consortium, Seboka, Christelike, and Thembalethu. Mid-term Performance Evaluation of the SHIPP Project Evaluation Report Page | iii MID-TERM PERFORMANCE EVALUATION OF THE USAID SEXUAL HIV PREVENTION PROGRAM (SHIPP) DISCLAIMER The views of the authors expressed in this publication do not necessarily reflect the views of the United States Agency for International Development or the United States Government. Mid-term Performance Evaluation of the SHIPP Project Evaluation Report Page | iv CONTENTS Acronyms ...................................................................................................................................... vi 1 Executive Summary ............................................................................................................... 1 2 Introduction............................................................................................................................ 6 2.1 Evaluation Background and Scope ...............................................................................................9 2.1.1 Evaluation Approach: ...................................................................................................10 2.2 SHIPP Program Description ........................................................................................................12 3 Methodology........................................................................................................................ 17 4 Key Findings for Each of the 7 Evaluation Questions –.................................................... 19 4.1 To what extent has SHIPP achieved its stated objectives/results so far? What are the reasons for any shortfalls? ........................................................................................................................24 4.1.1 SO1: Strengthening Leadership Capacity of SAG and Other Structures ....................25 4.1.2 SO2: Strengthening Priority HIV Prevention Service Delivery Systems in NDOH and DBE ..............................................................................................................................33 4.1.3 SO3: Improving the Quality, Effectiveness and Coverage of HIV Prevention Programs at the Community Level................................................................................................39 4.1.4 Recommendations .......................................................................................................45 4.2 To what extent have combination HIV prevention approaches been incorporated into policy, planning, and strategy in partner SAG departments? .................................................................46 4.2.1 Findings ........................................................................................................................46 4.2.2 National Strategic Plan on HIV, STIs and TB 2012-2016 ............................................50 4.2.3 Examples of SHIPP support for Combination HIV Prevention.....................................51 4.2.4 Recommendations .......................................................................................................54 4.3 What are the strengths and weaknesses of the program so far?................................................54 4.3.1 Strengths ......................................................................................................................55 4.3.2 Weaknesses.................................................................................................................57 4.3.3 Recommendations .......................................................................................................60 4.4 What key aspects of the SHIPP model should be continued in future HIV/AIDS programming? What should be discontinued? What should be scaled up?........................................................60 4.4.1 Activities to Continue and Scale Up .............................................................................60 4.4.2 Discontinue...................................................................................................................61 4.5 What systems have been established due to the technical assistance provided by SHIPP?.....62 4.5.1 Leadership and governance within government structures .........................................63 4.5.2 Strengthening Human Resources - Secondment of staff.............................................64 4.5.3 Strengthening Supply Chain Management - Condom distribution plans and tools......64 4.5.4 Strategic information ....................................................................................................65 4.5.5 Prevention service delivery ..........................................................................................65 4.5.6 Budget planning ...........................................................................................................66 Mid-term Performance Evaluation of the SHIPP Project Evaluation Report Page | v 4.5.7 Community systems .....................................................................................................66 4.5.8 Recommendations .......................................................................................................66 4.6 What are the examples of working partnerships established with SAG structures? ...................67 4.6.1 DSD partnership...........................................................................................................67 4.6.2 Partnerships with PEPFAR partners ............................................................................68 4.6.3 Partnership with Operation SUKUMA SAKHE.............................................................68 4.6.4 Partnership with AgriAIDS to target farm workers in Gert Sibande district..................68 4.7 What aspects of small grant activities are successful to inform future prevention models? .......68 4.7.1 Findings ........................................................................................................................68 4.7.2 Recommendations .......................................................................................................72 5 Conclusions and Recommendations .................................................................................. 73 5.1 Conclusions .................................................................................................................................73 5.2 Issues, challenges, and accomplishments ..................................................................................75 5.2.1 Issues ...........................................................................................................................75 5.2.2 Challenges ...................................................................................................................76 5.2.3 Accomplishments .........................................................................................................76 5.3 Lessons learned ..........................................................................................................................77 5.4 Future direction for the program ..................................................................................................79 Annex 1: Evaluation Scope of Work ......................................................................................... 82 Annex 2: Detailed Methodology............................................................................................... 83 Annex 3: Sources of Information .............................................................................................. 84 Annex 4: Signed Statements Attesting To Lack of Conflict of Interest ................................ 85 Mid-term Performance Evaluation of the SHIPP Project Evaluation Report Page | vi Acronyms AIDS Acquired Immune Deficiency Syndrome ART Antiretroviral Therapy CADRE Centre for AIDS Development, Research and Evaluation CATs Community Action Teams CBO Community-based Organization CC Community Capacity CCEP Community Capacity Enhancement program CCE Community Capacity Enhancement CHW Community health workers COGTA Department of Co-operative Governance and Traditional Affairs COJ City of Johannesburg COP Chief of Party CSO Civil Society Organization CSTL Care and Support for Teaching and Learning DAC District AIDS Council DBE Department of Basic Education DG Director General DHET Department of Higher Education and Training DOH Department of Health DPME Department of Performance Monitoring and Evaluation DPSA Department of Public Service and Administration DSD Department of Social Development FBO Faith Based Organization GBV Gender-based Violence GIBS Gordon Institute of Business GP Gauteng Province HAST HIV/AIDS, STIs and TB HCT HIV Counseling and Testing HIV Human Immunodeficiency Virus HPV Human Papilloma Virus HSRC Human Sciences Research Council HTA High Transmission Areas Mid-term Performance Evaluation of the SHIPP Project Evaluation Report Page | vii IEC Information, Education and Communication ILO International Labour Organization IOM International Organization for Migration IPT Intermittent Preventative Therapy IR Intermediate Result KYE Know Your Epidemic KYR Know Your Response KZN KwaZulu-Natal Province LAC Local AIDS Council LCT Leadership Capacity Team LEAP Local Epidemic Assessment Process tool M&E Monitoring and Evaluation MMC Medical Male Circumcision MP Mpumalanga Province MPAC Mpumalanga AIDS Council MRC Medical Research Council MTCT Mother to Child Transmission NDOH National Department of Health NDOT National Department of Transport NGO Non-governmental Organization NHC National Health Council NSP National Strategic Plan for HIV, STIs and TB 2012–2016 OSS Operation SUKUMA SAKHE OTDP Office of the Deputy President PEPFAR U.S. President’s Emergency Program for AIDS Relief PFIP Partnership Framework Implementation Plan SA South Africa SADC Southern African Development Community SAG South African Government SANAC South African National AIDS Council SGR Small Grant Recipient SLA Service Level Agreements SHIPP Sexual HIV Prevention Program Mid-term Performance Evaluation of the SHIPP Project Evaluation Report Page | viii SO Strategic Objective SSD Social and Structural Determinants STI Sexually Transmitted Infection STTA Short term technical assistance TA Technical Assistance TasP Treatment as Prevention TB Tuberculosis TO Technical Assistance UNAIDS United Nations Programme on HIV/ Acquired Immune Deficiency Syndrome USAID U.S. Agency for International Development USG United States Government VMMC Voluntary Medical Male Circumcision WAC Ward AIDS Council WHO World Health Organization Wits RHI Wits Reproductive Health and HIV Institute ZDM Zululand District Municipality Mid-term Performance Evaluation of the SHIPP Project Evaluation Report Page | 1 1 EXECUTIVE SUMMARY Evaluation Purpose: USAID/South Africa commissioned this mid-term performance evaluation of the Sexual HIV Prevention Program (SHIPP) to establish the effectiveness of the project, its overall model/approach, the extent to which it met its intended objectives, and to highlight elements that worked well (or not) and the reasons why. The evaluation covers SHIPP’s performance from September 2010 to March 2014. Project Background: SHIPP provides technical assistance (TA) support for the prevention of HIV/AIDS in South Africa. TA services were aimed at strengthening South Africa’s capacity to deliver HIV prevention services for reducing HIV incidence in selected high-prevalence districts in the country. The project provided technical support to the South African Government (SAG) at national level and in selected provinces and districts. SHIPP’s primary objectives were to: 1. Strengthen the leadership capacity of SAG Departments and other structures that effectively coordinate, implement and evaluate HIV prevention programming at national, provincial and district levels; 2. Strengthen priority HIV prevention service delivery systems in the National Department of Health (NDOH) and Department of Basic Education (DBE) at all levels, including the community level; and 3. Improve the quality, effectiveness and coverage of HIV prevention programs at community level by: a. Focusing on key drivers of the epidemic and using evidence based-interventions b. Engaging leadership at all levels and mobilizing communities to action. c. Seizing all opportunities for increasing quality HIV prevention interventions. d. Effectively linking mass media and national campaign efforts with community norm change initiatives. e. Enhancing the coordination and synergies between and among sexual HIV prevention, treatment, and care partners SHIPP was implemented by the Futures Group in conjunction with its consortium partners: Engender Health, Wits Reproductive Health and HIV Institute (Wits RHI), the Center for AIDS Research (CADRE), and Futures Institute. Evaluation Questions, Design, Methods and Limitations: The evaluation team was tasked with answering 7 overarching evaluation questions: 1. To what extent has SHIPP achieved its stated objectives/results so far? What are the reasons for any shortfalls? 2. To what extent have combination HIV prevention approaches been incorporated into Mid-term Performance Evaluation of the SHIPP Project Evaluation Report Page | 2 policy, planning, and strategy in partner SAG departments? 3. What are the strengths and weaknesses of the program so far? 4. What key aspects of the SHIPP model should be continued in future HIV/AIDS programming? What should be discontinued? What should be scaled up? 5. What systems have been established due to the technical assistance provided by SHIPP? 6. What are the examples of working partnerships established with SAG structures? 7. What aspects of small grant activities are successful to inform future prevention models? Evaluation Approach: Two evaluation approaches were utilised for determining the changes that have taken place as a result of SHIPP’s efforts:  Collective Impact. Successfully addressing complex social problems (such as HIV prevention) across multiple partners is achieved through attention to key domains that facilitate the process for collective change (www.fsg.org).  Most Significant Change. This approach provides an indication of a program’s perceived impact and is useful for explaining how change takes place and under what circumstances (www.betterevaluation.org). The evaluation team undertook the following data collection activities targeting the SHIPP Consortium members and a wide range of SAG managers and service delivery personnel: a) Data/document review, b) Key informant interviews (KIIs) with checklist., and c) Online quantitative survey with all 33 of SHIPP’s small grant recipients (SGRs). Limitations: The evaluation examined the project’s work over a 3½ year period, but due to high turnover of project staff, several respondents lacked the institutional memory to respond to some evaluation questions - limiting the extent of triangulation. The timing of the evaluation occurred at the end of the project’s 4thyear, with much uncertainty around the staff’s continuity with the project for the 5th year option, possibly affecting the objectivity of some respondents. Overall Findings and Conclusions: SHIPP operated in a complex and layered environment, providing technical support to seven government stakeholders across three or more layers of government – national, provincial, district and/or sub-district levels. It provided grants to 33 community-based organizations in three provinces, across four districts and 13 sub-districts. Furthermore, it was designed to provide TA in HIV prevention with combination prevention as the core – an approach that requires multi-level implementation of bio-medical, structural and behavioral/social interventions focused on reducing HIV incidence by breaking the chain of transmission. The evaluation team established the following:  SHIPP was designed in consultation with NDOH, DBE and SANAC. Mid-term Performance Evaluation of the SHIPP Project Evaluation Report Page | 3  SHIPP’s award occurred at a time when there were significant new initiatives in HIV prevention both globally and within South Africa, including an increasing appreciation of combination prevention as an important approach for reducing HIV incidence.  The National Strategic Plan (NSP) for HIV and AIDS (2012-2016) identified additional key SAG departments critical to the fight against HIV. Over time, four of these were added to the project – more than doubling the number of stakeholders eligible for SHIPP’s TA.  Despite the establishment of a project stakeholders group that tried to meet quarterly, the project did not have an effective overarching Steering Committee to guide, prioritize and coordinate the requests for support from the 7 SAG departments.  Annual project workplans were drawn up in consultation with SAG departments and implemented following approval from USAID. Workplans were aligned to the NSP as well as departmental HIV/AIDS strategies and PEPFAR’s Partnership Framework.  The project experienced high staff turnover at all levels of the organization. This turnover, especially at the level of the COP, introduced a high degree of instability which in turn negatively affected smooth implementation.  SHIPP’s geographical focus – 4 districts within 3 provinces (Gauteng, KwaZulu-Natal and Mpumalanga) - was the result of consultations between USAID, SHIPP and SAG, initially at national level, and later at provincial and district levels. Although high HIV prevalence was the main criteria, lengthy negotiations to achieve agreement between different SAG departments and different levels of government contributed to a delay in implementation.  Measuring the effect of the project’s TA proved to be a challenge as there were no PEPFAR indicators for the provision of technical assistance. This in turn delayed the development of a Performance Monitoring Plan (PMP). It was not until more than half￾way through the project period that a suitable mix of indicators was agreed upon to track the results arising from SHPP’s TA and capacity building efforts. EVALUATION QUESTION 1 - PROJECT ACHIEVEMENTS: The project had 3 strategic objectives (SOs) in its Scope of Work: SO1: Strengthening Leadership Capacity of SAG and Other Structures SO2: Strengthening Priority HIV Prevention Service Delivery Systems in NDOH and DBE SO3: Improving the Quality, Effectiveness and Coverage of HIV Prevention Programs at Community Level Across the three SOs, there are 17 output indicators, of which SHIPP had only met 7 (41%) by March 2014, and most of these were in SO1, focused on the project’s work in developing policies, strategies, and plans. Three indicators were not met, and the remaining 7 indicators (41%) had no LOP targets specified in the PMP, and therefore it was not possible to determine Mid-term Performance Evaluation of the SHIPP Project Evaluation Report Page | 4 the percent achievement for these. EVALUATION QUESTION 2 - COMBINATION PREVENTION POLICY: To date, SHIPP has provided technical support in incorporating/ strengthening combination prevention in the HIV/AIDS response for numerous SAG departments: National Department of Health (NDOH), Department of Basic Education (DBE), South African National AIDS Council(SANAC), Department of Social Development (DSD), Department of Public Service and Administration (DPSA), Department of Higher Education and Training (DHET) and the Office of the Deputy President (OTDP). SHIPP’s contribution towards the inclusion of combination prevention in the national NSP 2012-2016 as well as in the departmental and sub-national strategic plans aligned to it, has helped to keep combination prevention in the spotlight. Incorporating combination HIV prevention approaches into various policies and strategies strengthened a common agenda and commitment to combination HIV prevention across SAG departments. Indeed, combination HIV prevention is in the foreground of SAG HIV prevention efforts at all levels. EVALUATION QUESTION 3 - STRENGTHS AND WEAKNESSES: SHIPP’s consortium and program structure employed multiple technical offerings which made it possible for the project to be flexible enough to address emerging needs. That, coupled with the multiple channels of delivery available to it, made it a source of ready technical assistance and operational support to South African Government (SAG) departments. Other key strengths included strengthening of political leadership and governance – particularly at district level; facilitating dialogue on combination HIV prevention at district and community levels; and developing capacity of Community Based Organizations (CBO) in condom mapping and distribution. In its work with a large number of SAG stakeholders, the effort of gaining consensus and getting everyone onboard resulted in a prolonged post-award engagement process (approximately 24 months). Furthermore, SHIPP did not demonstrate the operationalization of combination prevention based on the local epidemic. On M&E, the opportunity to strengthen data management and usage across SAG departments and assist AIDS councils was not fully exploited as LEAP was not implemented as originally planned. Furthermore, within the SHIPP consortium itself, there was a high staff turnover of key personnel. EVALUATION QUESTION 4 - ACTIVITIES TO BE SCALED UP: Activities that the evaluation team deemed noteworthy and candidates for scale up were those implemented at the district or community level. These include: Condom mapping and distribution; strengthening of AIDS Councils in other districts; and continued capacity building in costing of operational plans. EVALUATION QUESTION 5 - SYSTEMS THAT HAVE BEEN STRENGTHENED: Project TA contributed to strengthened existing SAG systems at various levels, but no new SAG systems were established per se. SHIPP strengthened leadership and governance within SAG structures through: developing prevention policies, strategies and technical working groups; seconding staff to bolster human resources, improving supply chain management through condom distribution plans and tools; building capacity for strategic information and M&E and support to local Mid-term Performance Evaluation of the SHIPP Project Evaluation Report Page | 5 epidemic modeling; and strengthening budget planning through development of costing models. SHIPP strengthened community systems through its support to District AIDS Councils (DACs), Local AIDs Councils (LACs) and Ward AIDS Councils (WACs) for building their capacity to assess local drivers of HIV transmission and to monitor the SAG response toward HIV. EVALUATION QUESTION 6 - WORKING PARTNERSHIPS: Factors unrelated to technical skills that contributed to successful working relationships with SAG structures and counterparts include six commonly cited ‘relationship builders’: (1) open communication; (2) mutual respect and collaboration; (3) sense that SHIPP was not prescriptive and added value ;(4) commitment from both partners; (5) effective coordination from SAG counterpart; and (6) continuity of key contacts or champions within SAG departments. EVALUATION QUESTION 7 – SMALL GRANTS: - SHIPP provided small grants to 33 CBOs in the target districts to improve the quality, effectiveness, and coverage of sexual HIV prevention programs at community level. SAG and SGR respondents alike report the most successful aspect of the SGR model to be (i) strengthened linkages between CBOs and SAG departments through WAC and DAC structures and (ii) strengthened community services through community dialogues for social mobilization, peer education in schools, condom distribution, and referrals and linkages between communities and other HIV prevention service providers. Future Direction for the Program SHIPP:  Learn the lessons thus far and continue/scale up the activities that worked well and where SHIPP adds value, e.g. condom mapping, geo-spatial mapping, and strengthening AIDS councils.  Focus future efforts on working operationalizing combination prevention.  Provide TA to develop and refine M&E indicators and tools for combination HIV prevention coverage and behavior change. SAG:  Build on the investments made thus far by the SHIPP project e.g. its work with strengthening the DACs and WACs. Furthermore, scale up those activities that enable other provinces and districts to better understand their local epidemic and tailor appropriate responses to reduce HIV incidence.  Ensure robust M&E indicators and tools to track combination HIV prevention interventions – across sectors in line with existing SAG and PEPFAR M&E systems. USAID:  Continue support for AIDS Councils, especially at district, local and ward levels.  For future projects operating in a complex and multi-sectoral area, ensure that there is an effective and representative project-specific steering committee with clear terms of reference to provide guidance and coordination.  Clearly define and communicate to stakeholders the project’s definition of TA and develop a “menu” of TA services. Mid-term Performance Evaluation of the SHIPP Project Evaluation Report Page | 6 2 INTRODUCTION In 2012, it was estimated that 12.2% of South Africa’s population of 52 million people were HIV positive (6.4 million persons)1 , more than any country in the world. HIV prevalence differs substantially by province with prevalence rates over 14% in four provinces -- KwaZulu-Natal (16.9%), Mpumalanga (14.1%), Free State (14.0%), and the North West (13.3%) – and lower rates in the Western Cape (5.0%), Northern Cape (7.4%), Limpopo (9.2%), Eastern Cape (11.6%), and Gauteng (12.4%). HIV prevalence rate among pregnant women attending antenatal clinics2 is 29.5%. KwaZulu￾Natal Mpumalanga, and Free State have prevalence rates greater than 30.0%, while Limpopo, Northwest, Gauteng and the Eastern Cape recorded prevalence rates between 20.0% and 30.0%. Only Northern Cape and Western Cape have HIV prevalence rates below 20.0%. Figure 1: HIV Prevalence among antenatal women Source: The 2012 National Antenatal Sentinel HIV and Herpes Simplex type-2 prevalence Survey South African HIV epidemic is a generalized hyper-endemic there are still higher levels of infection and transmission within certain geographic areas, as well as among some key populations. In 2012, 25 of 52 districts recorded antenatal HIV prevalence rates below the national average of 29.5. . Two districts had antenatal HIV prevalence rates of more than 40% - 1 Shisana, O, Rehle, T, Simbayi LC, Zuma, K, Jooste, S, Zungu N, Labadarios, D, Onoya, D et al. (2014) South African National HIV Prevalence, Incidence and Behaviour Survey, 2012. Cape Town, HSRC Press. 2 The 2012 National Antenatal Sentinel HIV and Herpes Simplex type-2 prevalence Survey, South Africa, National Department of Health. Mid-term Performance Evaluation of the SHIPP Project Evaluation Report Page | 7 uMgungundlovu (40.7%) in KwaZulu-Natal and Gert Sibande (40.5%) in Mpumalanga. The South African National Strategic Plan (NSP) 2012-2016 defines the following populations as being at highest risk of acquiring or transmitting HIV infection: :  young women between the ages of 15 and 24 years;  people living close to national roads and in informal settlements;  young people not attending school and girls who drop out of school before matriculating;  people from low socio-economic groups;  uncircumcised men;  people with disabilities and mental disorders;  sex workers and their clients;  people who abuse alcohol and illegal substances;  men who have sex with men; and  transgender individuals. When the SHIPP Project was awarded in September 2010, many changes were beginning to occur in the policy space around HIV/AIDS prevention in South Africa. The year had already seen the launch of the HCT campaign in April 2010 as well as the launch of the Male Medical Circumcision campaign in the same month. There was a good understanding of the drivers of the HIV epidemic and an increasing appreciation of the various evidence-based prevention interventions which, if implemented well, could significantly reduce new HIV infections in the country. The first NSP for HIV and AIDS (2007 to 2011), which called for the reduction of new HIV infections by half by 2011, was more than halfway through the period covered but the infection rates had not declined as hoped. The NDOH was busy preparing an implementation plan to accelerate the scale up of HIV prevention in order to meet the target set forth in the first NSP. Around the same time, consultations and initial drafts of the new NSP (2012 to 2016) were underway. Meanwhile, a key shift was taking place in the global HIV Prevention policy arena. There was a sense that HIV prevention programs were not keeping up with the epidemic. For every new person starting on ARVs, two new people were being infected by HIV. There was a realization that the approach to HIV prevention had to be improved quickly taking into consideration the interacting causes of HIV risk and vulnerability as shown in Figure 2below. Mid-term Performance Evaluation of the SHIPP Project Evaluation Report Page | 8 Figure 2 - Interacting Causes of HIV Risk and Vulnerability Source: Towards Combination HIV Prevention Tailoring and Coordinating Biomedical, Behavioural and Structural Strategies to Reduce New HIV Infections. UNAIDS.2010 In response, a UNAIDS Discussion Paper published in 2010 proposed the use of Combination Prevention as a promising approach to significantly reduce new infections. As defined by the UNAIDS Prevention Reference Group, combination prevention programs are: ... rights-based, evidence-informed, and community-owned programmes that use a mix of biomedical, behavioural, and structural interventions, prioritized to meet the current HIV prevention needs of particular individuals and communities, so as to have the greatest sustained impact on reducing new infections. Well-designed combination prevention programmes are carefully tailored to national and local needs and conditions; focus resources on the mix of programmatic and policy actions required to address both immediate risks and underlying vulnerability; and they are thoughtfully planned and managed to operate synergistically and consistently on multiple levels (e.g. individual, relationship, community, society) and over an adequate period of time. They mobilize community, private sector, government and global resources in a collective undertaking; require and benefit from enhanced partnership and coordination; and they incorporate mechanisms for learning, capacity building and flexibility to permit continual improvement and adaptation to the changing environment. It was onto this national and global landscape that the SHIPP project was introduced. SHIPP’s Mid-term Performance Evaluation of the SHIPP Project Evaluation Report Page | 9 overarching mandate was to support SAG to achieve adequate coverage of combination HIV prevention interventions that address the sources of new infections at a scale and scope likely to reduce sexual HIV infections. SHIPP is a complex and layered program with a wide range of activities under a common intervention structure including: technical assistance and capacity building in multiple domains across multiple levels of government, multiple prevention interventions, with partnerships and networks. By its nature, SHIPP’s interventions must be “adaptive” to change in circumstances and cannot always be planned from the beginning, making it difficult to determine the extent to which the program has achieved its objectives. Moreover, SHIPP’s work is embedded in an environment where multiple governmental and non-governmental organizations at various levels work to prevent new HIV/AIDS infections, making it challenging to attribute change specifically to the SHIPP program in the absence of a control or comparison group. 2.1 Evaluation Background and Scope USAID/South Africa commissioned this external, mid-term performance evaluation of the Sexual HIV Prevention Program (SHIPP) project to: 1. Assess the quality of the program implementation, particularly in relation to support to SAG departments, 2. Explain SHIPP’s effectiveness in achieving outputs and outcomes and detail the program elements that work and those that do not, while explaining why this is the case 3. Document lessons learned, 4. Explore challenges and accomplishments, and 5. Provide strategic guidance for the program going forward. The mid-term evaluation covers the SHIPP’s period of performance from September 2010 to March 2014. USAID indicated that these results will aid in determining whether the program should be ‘scaled-up’ or which elements of the program should be removed or reinforced. The evaluation team was tasked with answering 7 overarching evaluation questions: 1. To what extent has SHIPP achieved its stated objectives/results so far? What are the reasons for any shortfalls? 2. To what extent have combination HIV prevention approaches been incorporated into policy, planning, and strategy in partner SAG departments? 3. What are the strengths and weaknesses of the program so far? 4. What key aspects of the SHIPP model should be continued in future HIV/AIDS programming? What should be discontinued? What should be scaled up? 5. What systems have been established due to the technical assistance provided by SHIPP? Mid-term Performance Evaluation of the SHIPP Project Evaluation Report Page | 10 6. What are the examples of working partnerships established with SAG structures? 7. What aspects of small grant activities are successful to inform future prevention models? Given the complexity of the SHIPP project, evaluation was required across various levels of government and the supported community projects and their service delivery efforts. In addition, the evaluation required an examination of:  The quality of engagement between SHIPP and government as well as between SHIPP and civil society, e.g. the effectiveness of technical assistance and capacity building efforts, the extent of a shared agenda, etc.;  Improvements in performance in Government – i.e. enhanced leadership, management planning; and  Improvements in HIV prevention service delivery through civil society. 2.1.1 EVALUATION APPROACH: Khulisa’s approach incorporates elements of different evaluation theories selected for their utility in answering the evaluation questions and in estimating SHIPP’s contribution to changes that have occurred since the beginning of the project. Because our technical approach utilizes a non-experimental design, a guiding principle throughout the evaluation was the consideration of the SHIPP’s direct and indirect contributions to observed changes. Where SHIPP has had a direct influence, more certain conclusions can be made about SHIPP’s contribution, than when evaluating areas of indirect influence. Two evaluation approaches were utilized for determining the changes that have taken place as a result of SHIPP’s efforts:  Collective Impact. Successfully addressing complex social problems (such as HIV prevention) across multiple partners is achieved through attention to 5 core domains and 7 additional domains (Table 1) that facilitate the process for collective change (www.fsg.org).  Most Significant Change. This approach provides an indication of a program’s perceived impact and is useful for explaining how change takes place and under what circumstances (www.betterevaluation.org). Table 1: Collective Impact Domains 1 Common Agenda Across all partnerships, there is a common understanding of the HIV prevention problem and a shared approach and agreed upon actions for solving it. 2 Continuous Communication Mid-term Performance Evaluation of the SHIPP Project Evaluation Report Page | 11 Consistent and open communication exists between SHIPP and its numerous partners and stakeholders to develop trust, build mutual objectives, and create mutual motivation. 3 Shared Measurement System Consistent data collection across all participants ensures that efforts are aligned and partners hold each other accountable. 4 Mutually Reinforcing Activities Partner activities are differentiated while also coordinated through a shared action plan. 5 Backbone Function There is a backbone structure to the entire initiative which ensures that the effort is managed by dedicated staff and strong leaders who have a specific set of skills 6 Learning Culture Learning is embedded in the SHIPP program. 7 Capacity Interstitial elements keep the SHIPP process dynamic and progressive (e.g., funding, human resources). 8 Behavior Change: Professional Practice Formal actors and organizations/institutions make changes in their work as it relates to the goals of SHIPP. 9 Behavior Change: Individual Behavior Individuals change their behavior in relation to the SHIPP goals. 10 Systems Change: Funding Flows The flow of philanthropic and public funding shifts to support the goals of SHIPP. 11 Systems Change: Cultural Norms The social and cultural norms change in ways that support the goals of SHIPP. 12 Systems Change: Advocacy and Public Policy Progress is made on SHIPPs advocacy and public policy goals. Mid-term Performance Evaluation of the SHIPP Project Evaluation Report Page | 12 2.2 SHIPP Program Description The Sexual HIV Prevention Program (SHIPP) is a 4-year (2010-2014) USAID-funded technical assistance program with an option of a 5th year. The program supports multiple levels of the South African Government (SAG) as well as communities in selected high-prevalence districts (Figure 5) to strengthen “combination HIV prevention”3 for reducing HIV incidence overall. Implemented by Futures Group International with its consortium partners - the Centre for AIDS Development, Research and Evaluation (CADRE), the Wits Reproductive Health and HIV Institute (Wits RHI), Engender Health, and the Futures Institute, SHIPP provided a wide range of technical assistance services, including building leadership and management capacity and strengthening prevention service delivery systems with the aim of improving the quality, effectiveness and coverage of HIV prevention programs at specific levels. The SHIPP model had 4 main components which are:  Multi-sectoral and multi-level TA ,  Multiple technical offerings, and  Multiple channels of delivery  Provision of small grants to CBOs to increase coverage at community level4 . SHIPP established a South African government multi-sectoral collaboration strategy with SAG departments and structures identified key counterparts in each of these departments. SHIPP’s technical support took the form of short, medium and long-term support (via seconded staff). In several instances, SHIPP also provided support in the facilitation of consultation workshops and in the printing and dissemination of approved guidance documents. The SHIPP Project had three strategic objectives (SOs) and three Intermediate Results (IRs) corresponding to the SOs (Figure 3) 3 Combination prevention approach addresses behavioral risk, social norms, biomedical interventions, and school-based programming. http: //futuresgroup.com/projects/sexual_hiv_prevention_program_shipp_south_africa#sthash.cMIwL0mr.dpuf 4The CBOs that received small grants are referred to as small grant recipients (SGR) in this document. “(SHIPP) support aims to help the SAG achieve adequate coverage of key interventions that address the sources of new infections at a scale and scope likely to reduce sexual HIV transmission” Pg. 3 RFTOP Mid-term Performance Evaluation of the SHIPP Project Evaluation Report Page | 13 Figure 3: SHIPP Strategic Objectives and Intermediate Results SO 1: Leadership capacity building approach: SHIPP adopted a multi-sectoral and multi-level TA approach targeted at SAG structures (e.g. health, education, social development, AIDS councils, and others) to strengthen coordination, implementation, and evaluation of combination HIV prevention programming at national, provincial, and district levels. SHIPP’s primary mechanism for building leadership capacity in Government was through seconded experts and STTA. TA support from SHIPP head office along with technical staff secondment of staff aimed to immediately address current staffing shortages within the SAG and build long￾term capacity. At the assignment location, seconded technical officers, with expertise in health, education, engagement and strategic planning, were supposed to form cross-functional Leadership Capacity Teams (LCTs) to maximize learning, coaching, and mentoring, and to build SAG ownership of leadership tools and approaches for HIV prevention. As shown in (Figure 4), SHIPP’s approach uses tools, technologies, and technical support to build the leadership capacity of SAG. This skills transfer is aimed at national and local leaders to achieve results by effectively planning, coordinating, guiding, implementing, and mainstreaming interventions that address the drivers of the epidemic and ultimately reducing HIV transmission. SO 2: Strengthen priority HIV prevention service delivery systems in NDOH and DOE at all levels: SHIPP’s approach for achieving SO 2 was to strengthen policies and programs (i.e. delivery systems) for HIV prevention services from national to local levels. SHIPP’s intention was to support new skills in strategic planning, evidence-based decision-making, and effective SO1: Strengthen the leadership capacity of SAG Departments and other structures that effectively coordinate, implement and evaluate HIV prevention programming at national, provincial and district levels; IR1: Improved policy environment that supports implementation of comprehensive, quality, evidence-based HIV sexual prevention interventions. SO2 Strengthen priority HIV prevention service delivery systems in the National Department of Health (NDOH) and Department of Basic Education (DBE) at all levels, including the community level IR2: Capacities built in SAG and coordinating bodies to coordinate, monitor and evaluate HIV sexual prevention policy and programming at national, provincial, district and sub-district levels SO3: Improve the quality, effectiveness and coverage of HIV prevention programs at the community level IR3: Improved quality, effectiveness and coverage of HIV sexual prevention interventions at the community level Mid-term Performance Evaluation of the SHIPP Project Evaluation Report Page | 14 engagement of stakeholders leading to development of policies and programs that are endorsed by SAG leadership. SO 3: Improve quality, effectiveness, & coverage of HIV prevention programs at the community level: To achieve SO 3, SHIPP focuses on SAG implementation leadership at district levels, supporting DACs to build community ownership and to coordinate action on education guidelines, curricula, and behavioral and medical HIV prevention services. By building the leadership capacity of DACs to support and coordinate interventions, communities engage in addressing the drivers of the epidemic, leading to long-term impact. SHIPP also supports community level implementation through awarding small grants to local CBOs. The project’s Theory of Change is presented in Figure 4. Figure 4: Theory of Change Mid-term Performance Evaluation of the SHIPP Project Evaluation Report Page | 15 Figure 5: Implementation Structure of SHIPP In implementing SHIPP, the focus was on SAG Departments and structures such as the National Department of Health (NDOH), Department of Basic Education (DBE), Department of Higher Education and Training (DHET), Department of Social Development (DSD), South African National AIDS Council (SANAC), Department of Public Service and Administration (DPSA), and the Office of the Deputy President (OTDP), and their relevant counterparts at provincial, district and local levels. SHIPP worked in three provinces, four districts, and 13 sub-districts, selected primarily because they are areas with the highest HIV prevalence rates in South Africa (Table 2). Table 2: SHIPP's Levels of Engagement National Province District Sub-district/Region NDOH DBE DHET DSD SANAC DPSA OTDP Gauteng City of Johannesburg Regions A Region G KwaZulu-Natal Zululand district Nongoma, Abaqulusi, eDumbe, uPongolo, Ulundi Mpumalanga Gert Sibande Albert Luthuli, Msukaligwa, Mkhondo Ehlanzeni Mbombela, Umjindi, Nkomazi Mid-term Performance Evaluation of the SHIPP Project Evaluation Report Page | 16 Figure 6: Timeline of the SHIPP Program Mid-term Performance Evaluation of the SHIPP Project Evaluation Report Page | 17 3 METHODOLOGY The SHIPP Project is implemented by Futures Group International through its four 4 sub￾contractors (including WHRI which is no longer involved in the project) and 33 small grant recipients. These entities, along with a wide range of relevant South African government managers and service delivery personnel were targeted for data collection. The Evaluation Team undertook the following data collection activities: d) Data/document review; e) Key informant interviews (KIIs) with checklist. f) Online quantitative survey. All 33 of SHIPP’s small grant holders as well as U.S. President’s Emergency Program for AIDS Relief (PEPFAR) partners working in the SHIPP supported districts and sub-districts were invited to answer the online survey to obtain their views on the grant and technical support provided by SHIPP. Table 3: Data Collection Methods – Planned vs. Actual Outputs 10. Target Planned Output Actual Output Response Rate Desk Review SAG Documents N/A N/A -- PEPFAR and USAID Documents SHIPP Documents Key Informant Interviews SAG Structures 75 Key Informant interviews 54 Key Informant Interviews 72% USAID Staff SHIPP Consortium Staff Small Grant Recipients (SGR) Group Interviews As for KII 6 group interviews with 14 individuals 12 group interviews with 40 individuals completed 200% Online Surveys With SGR 33 SGR 21 SGR 63% With PEPFAR Partners 11 PEPFAR Partners 0 PEPFAR Partners 0% Mid-term Performance Evaluation of the SHIPP Project Evaluation Report Page | 18 The Inception Report presented a list of 90+ KII respondents5 purposefully chosen because of their roles and involvement in the SHIPP program and partnerships. Of the 90+ respondents, 54 were interviewed individually while a further 40 respondents were interviewed in groups. Confidential online surveys were used to collect feedback from the 33 SHIPP small grant recipient organizations, and other PEPFAR partners working in each of the 4 SHIPP-supported districts to evaluate their perceptions of SHIPP’s contribution to HIV prevention leadership, management, and services delivery. The mix of qualitative and quantitative data generated through fieldwork was analyzed using methods appropriate to each. Some techniques which the evaluation team used to analyze the data included thematic analysis, triangulation, and descriptive statistics. The detailed methodology can be found in Annex 2. 5 Because some KIIs involved more than one respondent (i.e. group interview), there were more respondents than KIIs. Mid-term Performance Evaluation of the SHIPP Project Evaluation Report Page | 19 4 KEY FINDINGS FOR EACH OF THE 7 EVALUATION QUESTIONS – The findings for this mid-term evaluation cover SHIPP’s performance from September 2010 to March 2014. As per the SOW, the evaluation team was required to present the findings for each evaluation question. Prior to discussing the evaluation questions’ findings an overview of SHIPP’s design and implementation is presented to properly contextualize the findings. Overview of SHIPP Design and Implementation Design of SHIPP: USAID conceptualized SHIPP as vehicle for providing technical assistance (TA) for targeted HIV prevention and to move forward HIV prevention efforts in South Africa. SHIPP was designed to adopt combination prevention approaches for scaling up by SAG and other partners. USAID ensured the design of SHIPP was collaborative by involving SAG during the development of the project’s SOW. SHIPP initially focused on assisting SANAC and the SAG Departments of Education and Health in their prevention efforts. The project’s design took into consideration combination HIV prevention starting from the national level and cascading to lower levels including the communities. SHIPP’s contract was awarded when PEPFAR was transitioning from direct service delivery to TA, but some SAG structures initially did not understand the TA model and what technical support to request; as a result some requests were unclear and not systematic. The evaluation team found that SHIPP not only provided TA, but also organized workshops and carried out relevant administrative activities to support the smooth running of the workshops. Theory of Change: The evaluation team found SHIPP’s Theory of Change complex. But this complexity was defended by SHIPP project respondents who noted that “the theory of change was (necessarily) complex as it described the South African epidemic and prevention response in total. SHIPP responded to opportunities and activities that the SAG requested technical assistance for and which fit within the overall SAG defined response as enshrined in the NSP”. SHIPP used this logic framework to develop its project-specific results framework which in turn guided its selection of activities. SHIPP’s pathways of change are not linear and there are overlaps due to the interrelated nature of many individual activities and intermediate results. Some IRs contribute to more than one SO. For example, while IR2 (Capacities built in SAG and coordinating bodies to coordinate, monitor and evaluate HIV sexual prevention policy and programming at national, provincial, district and sub-district levels) is intended to correspond to SO2 (Strengthen priority HIV prevention service delivery systems in NDOH and DOE at all levels) it also contributes to SO1 (Strengthen the leadership capacity of SAG departments and other structures that effectively coordinate, implement and evaluate HIV prevention programming at national, provincial and district levels). Mid-term Performance Evaluation of the SHIPP Project Evaluation Report Page | 20 Consortium Partners: The lead consortium partner is Futures Group; a global leader in policy diagnostics and formulation, capacity building, and implementation. The other consortium partners were selected for their respective strengths. Engender Health was selected for its experience in gender integration, health and education service delivery systems strengthening, and expertise in sex and HIV education curricula in schools and communities. Wits RHI is a South African organization and was selected for its strong capabilities in evaluation and health implementation; and vast experience working with SANAC and NDOH. CADRE, a South African NGO, was selected for its experience in HIV research. CADRE’s role was to provide STTA in Operational Research in order to understand which prevention models were having impact so that they could be expanded. Futures Institute was included as a consortium partner to provide STTA on costing and financial modelling. CADRE started its activities by developing a synthesis paper on what is known about the drivers of HIV infection and an understanding of South Africa’s HIV prevention needs, opportunities, challenges and priorities in general and for the three specific provinces where SHIPP was to operate - Gauteng, KwaZulu-Natal (KZN) and Mpumalanga. After developing the synthesis paper, CADRE had little to do in terms of Operational Research and Futures Group asked them to build the capacity of AIDS Councils and to train political leaders in supporting AIDS Councils. Another notable change in the SHIPP’s Consortium was withdrawal of Wits RHI at the end of 2012. According to some key informants, Wits RHI pulled out of the partnership citing poor management by the then Chief of Party, but this could not be confirmed with Wits RHI, as they were unavailable to respond to this evaluation. However, all respondents emphasized that the withdrawal was amicable. SHIPP’s Technical Units and Staffing Arrangements: At the beginning SHIPP was organized around 3 service delivery units (Engagement and Strategic Planning, Gender Integration, and Health Implementation and Evaluation Units) and a fourth unit responsible for Quality Assurance. Futures Group, as the lead partner, provided the Deputy and the Chief of Party and staff for the Engagement and Strategic Partnership and Quality Assurance Units. Engender Health provided staff for the Gender Integration Unit, while Wits RHI was responsible for staffing the Health Implementation and Evaluation Unit. Each service delivery unit was staffed with a Technical Director, Senior Technical Officer/s (STOs) and Technical Officers (TOs). Some STOs and TOs were seconded to SAG national and district structures. Mid-term Performance Evaluation of the SHIPP Project Evaluation Report Page | 21 Figure 7: SHIPP's Organogram in March 2014 The SHIPP management structure took long to stabilize initially due to problems of getting an effective Chief of Party. Initially constituted with an interim Chief of Party, the project’s first Chief of Party was only hired 7 months into the project. This individual was seen as unsatisfactory and was replaced about 1.5 years later. Respondents indicated that the turnover in the leadership at SHIPP was a major challenge which in turn affected project management and some delays in project implementation. After Wits RHI’s exit and the second Chief of Party coming on board, SHIPP restructured by splitting the Health Implementation and Evaluation Unit into two - Health Implementation and Strategic Information Units. The Social and Structural Determinant Unit was also added to focus on social and structural aspects of combined HIV prevention. Stakeholders: HIV Prevention is implemented through many sectors that have different leadership, strategies and policies. With NSP 2012-2016, many SAG departments identified their role in HIV prevention. When the SHIPP contract was issued (before NSP 2012-2016), the NDOH, DOE, and SANAC were identified as key beneficiaries of SHIPP support. The NDOH was part of the original three stakeholders since it is responsible for health systems and sexual HIV prevention service delivery. DBE is pivotal to HIV prevention services to learners. SANAC was included as a stakeholder as it coordinates government and civil society responses to the epidemic and leads multi-sectoral HIV prevention efforts. During the discussions to determine where SHIPP would work, NDOH directed SHIPP to include •Senior Technical Director (of Programs Costing •Senior Pgm Advisor (Vacant) •STO •TO SI •Technical Director •STO M&E •Knowledge R Manager •QA Officer •TO ESP •Tech Director (Vacant) •Senior Pgm Advisor (Vacant) •STO x3 •TO SSD •Technical Director HI •Technical Director •STO NDOH •Health Economist •Senior Technical Officer •STO- VMMC Coordination •Project Manager - HPV SANAC •STO - Migrant and Mobile Men pgm •STO – Girls and Young Women pgm •STO M & E (Vacant) DSD •Senior Technical Officer (Vacant) OTDP •Senior Technical Officer – Research Analyst (Vacant) DBE •STO-Education Policy & Pgm Implementation •TO-Education Implementation (Vacant) DPSA •STO-Prevention and Research •STO- HIV Prevention Project Mngt Capacitation (Vacant) Gert Sibande District •Technical Officer ESP •Technical Officer Gender x2 •Technical Officer HI x3 •Technical Officer M&E Ehlanzeni District •Technical Officer ESP Vacant •Technical Officer Gender x2 (1 Vacant) •Technical Officer HI x2 (1 Vacant) •Technical Officer M&E Vacant Zululand District •Technical Officer ESP •Technical Officer Gender x2 •Technical Officer HI •Technical Officer M&E City Of Johannesburg •Technical Officer ESP Vacant •Technical Officer Gender x2 •Technical Officer HI x2 •Technical Officer M&E Secondments STTA Futures Institute CADRE Gender •Technical Director •STO - (Vacant) Chief of Party Futures HQ Deputy Chief of Party HR Finance Grants Management IT Mid-term Performance Evaluation of the SHIPP Project Evaluation Report Page | 22 OTDP and DPSA due to their leading roles in coordinating the mainstreaming of HIV prevention in SAG departments and public service respectively. In 2010, DOE split into DBE and DHET and while SHIPP engaged with both DHET only came on board for SHIPP support in 2014. SHIPP further engaged with DSD for its leading role in protecting the rights of orphans, vulnerable children and youth, as well as its potential role in the implementation of social and structural components of combination HIV prevention. SHIPP’s other stakeholders at sub-national level included leaders in the selected provinces, districts and sub-district officials and communities. Thus, over the LOP, SHIPP’s stakeholders increased dramatically from the initial 3 SAG departments to 7. SHIPP confirms that if the project had stuck to the original number of stakeholders, they would not have been stretched in terms of priorities. On the other hand, the evaluation team’s analysis of the situation indicates SHIPP created a larger footprint for its HIV prevention efforts because of its engagement with these many stakeholders. First, bringing all the departments together to discuss each department’s HIV prevention priorities was an achievement. For the first time a platform for multi-sectoral HIV prevention debate was created. Secondly, SHIPP consulted widely with the key contacts in these departments on program initiation and roll-out as well as seeking their assistance to mitigate implementation issues. Thirdly, through DPSA, SHIPP supported the National Department of Transport (NDOT) HIV and TB Operational Plan and the development of guidelines to implement Annual Sport and Wellness Event for the Department of Sports and Recreation. SHIPP also engaged with the Department of Co-operative Governance and Traditional Affairs (COGTA) in Zululand resulting in a needs and resource analysis of HIV/AIDS prevention strategies. In partnership with SANAC SHIPP supported the work on girls and young women that was initiated by the Department of Women, Children and People with Disabilities (DWCPD). Therefore through this form of hierarchy arrangements, where SHIPP directly interacted with fewer SAG departments that represented the needs of other departments, SHIPP should not have found dealing with many stakeholders a problem. Stakeholders met 2-3 times a year although ideally these stakeholder meetings should have been held every quarter. The meetings included the SAG departments, USAID and SHIPP consortium partners. The agenda included review of SHIPP’s progress, departments’ priorities and requests, and information sharing among participants. The evaluation team established there were no documented clear terms of reference for these stakeholder meetings to guide or coordinate departmental requests and SHIPP’s responsiveness. Geographic Focus: Following the SHIPP contract award in late September 2010, USAID and SHIPP worked with SAG to clarify provincial and district level support and to jointly develop SHIPP’s first year workplan. In the contract, USAID had identified 18 needy districts from which the project was to select (collaboratively with the SAG) specific districts in which to work. In January 2011, three provinces (KZN, Mpumalanga, and Gauteng) were identified as target provinces based on their high HIV prevalence rates. The process of defining the range and scale of interventions at the provincial level was negotiated individually with the provinces. The Mid-term Performance Evaluation of the SHIPP Project Evaluation Report Page | 23 negotiations were lengthy and initial recommendations for intervention districts were reviewed and amended or added to by the provinces’ Office of the Premier and provincial departments of health and basic education. SHIPP involvement in three provinces was seen by both SHIPP and SAG stakeholders as a challenge as it spread itself too thin. Some respondents were of the view that SHIPP might have been better working in only one province. By working in one province, SHIPP would have avoided protracted negotiations with more than one province, would have started earlier and reached more sub-districts in the selected districts. Respondents pointed out that at sub-district levels, the politics of service delivery is intense – “if it’s implemented in one area it must be implemented in other needy areas – that’s how prioritization of need in SA works politically”. It is noteworthy that SHIPP focused on some sub-districts following decisions set by SAG and USAID, based on high transmission areas, availability of funds and HIV prevention activities by other funders and stakeholders in the districts. That said, the evaluation team established that despite SHIPP’s focus on selected sub-districts in Gert Sibande district, the other (“non-focus”) sub-districts received some support from SHIPP. For example, SHIPP’s technical assistance led to all seven sub-districts setting condom distribution targets and Social Mobilization Officers were appointed to scale up social mobilization activities and increase condom use. In addition, in the non-focus sub-districts, a LAC was established. , a capacity building workshop was held and a community dialogue on HIV/AIDS was held. . Work Planning, Performance Monitoring and Reporting: SHIPP began developing its Year 1 workplan immediately after the contract award in September 2010. In the first stakeholder meeting, held October 2010, NDOH, DBE, SANAC and DPSA provided inputs and requests for TA which were incorporated into the workplan. A second stakeholder meeting held in January 2011 allowed finalization of the workplan with further inputs from the SAG departments. USAID eventually approved the Year 1 workplan in April 2011. Following the launch of the NSP 2012-2016, SHIPP’s workplans were further aligned to the NSP, the DBE and DSD Draft HIV Strategies, and by the USAID Partnership Framework Implementation Plan. SHIPP consulted with SAG during stakeholders meetings and individual departmental meetings to ensure that SHIPP’s workplan activities supported SAG’s HIV prevention activities. After SHIPP’s workplans were approved, SAG departments made further requests, requiring SHIPP to integrate the new requests into the existing workplan where applicable or carry over requests into the following year’s workplan. After transitioning from direct service delivery to provision of TA, PEPFAR had not created any technical assistance indicators. As such SHIPP had difficulties with developing its Performance Monitoring Plan (PMP) to track the provision of TA and related outputs. The first version was approved by USAID only in October 2012 after several revisions in which the project sought to find the right mix of indicators to measure tangible results from technical assistance and capacity building. Mid-term Performance Evaluation of the SHIPP Project Evaluation Report Page | 24 The first approved PMP was later revised in September 2013. Although the first PMP had targets for the output and outcome indicators, the revised PMP had 17 output indicators and 3 outcome indicators with no targets. SHIPP explained that target setting in the updated 2013 version was not the aim, since SAG did not have targets for the number of policies, strategies or guidelines planned to be developed, costed, and implemented. In essence, there was paucity of information that SHIPP had to operate within, and this was acknowledged by USAID and hence the approval of the revised PMP. Other changes made in the revised PMP include:  Indicators for IR1 were changed to better monitor and report implementation progress  The numbers of output indicators for IR2 were reduced from 11 to 3.  New Generation PEPFAR indicators to monitor progress of the small grants program in Partnership Information Management System (PIMS) were included  Indicator reference sheets were included to provide information on the indicators’ definitions, purpose, measurement, analysis and reporting and data quality risks. The evaluation team analysis of SHIPP’s quarterly semi-annual and annual reports revealed that generally SHIPP prepared in-depth narrative reports in a standard reporting format. These reports included a summary of current activities; achievements broken down by IRs; report on operations; challenges and solutions, and anticipated activities for the following quarter. However, some inconsistencies were found such as:  Not all progress reports were organized by workplan activities.  Some progress reports did not have tables for reporting PMP indicator data  Some tables for reporting PMP indicators did not have current data for output indicators  Some outputs counted towards some indicators were not valid- for example in its Semi￾annual report for Year 4, the indicator Number of national, provincial and district HIV prevention policies and/or strategic plans developed or revised with support from SHIPP and aligned with the NSP, participation of SHIPP in the Sex Worker & Long Distance Truck Drivers Surveillance study on transportation routes (N2 & N3 in KZN) was counted as a strategy developed, while SHIPP’s reported role in the study was to ensure the study was aligned with the NSP. For the same indicator, SHIPP counted Draft Terms of reference for Governance Model for Mpumalanga AIDS Council as a developed strategy 4.1 To what extent has SHIPP achieved its stated objectives/results so far? What are the reasons for any shortfalls? Findings for this evaluation question are organized according to the project’s three strategic objectives in its Scope of Work instead of SHIPP’s stated Intermediate Results (IRs): Mid-term Performance Evaluation of the SHIPP Project Evaluation Report Page | 25 SO1: Strengthening Leadership Capacity of SAG and Other Structures SO2: Strengthening Priority HIV Prevention Service Delivery Systems in NDOH and DBE SO3: Improving the Quality, Effectiveness and Coverage of HIV Prevention Programs at the Community Level Across the three SOs, there are 17 output indicators, of which SHIPP had only met 7 (41%) by March 20146 , and most of these were in SO1. Three output indicators were not met, and the remaining 7 indicators had no LOP targets specified, and therefore it was not possible to determine the percent achievement for these. As such, much of the discussion below on each SO’s achievements is based on document reviews and key informant interviews. .. 4.1.1 SO1: STRENGTHENING LEADERSHIP CAPACITY OF SAG AND OTHER STRUCTURES SHIPP’s first objective was to intensify the HIV prevention response by strengthening leadership capacity across a wide range of relevant stakeholders –SAG departments, SANAC, AIDS councils, NGOs, FBOs, private sector, political, traditional, community and faith-based leaders. The activities undertaken for this, included:  Supporting development of HIV prevention policies/strategic plans  Supporting coordination of HIV combination prevention efforts across government departments Based on desk review and interviews, the evaluation team established that SHIPP also contributed to building leadership capacity through additional activities that were not tracked by SHIPP’s PMP, such as:  Seconding experts and providing STTA at national, provincial and district levels.  Supporting development of national level HIV programs Each of these bullets is further discussed below. Supporting Development of HIV Prevention Policies/Strategic Plans To achieve SHIPP’s IR 1, Policy environment support implementation of comprehensive, quality, evidence based interventions for HIV sexual prevention, SHIPP supported the development, or the revision, of a number of national, provincial and district HIV prevention policies and/or strategic plans aligned with the NSP. SHIPP’s support included technical inputs to prevention sections of policy documents, financial and operational support for consultative workshops, as well as copy editing, printing and dissemination of the approved policies/strategies. SHIPP supported dissemination of approved policies/strategies through facilitating workshops for the relevant 6 “Met” = a least 70% achievement at the project’s 3.5 year mark. Mid-term Performance Evaluation of the SHIPP Project Evaluation Report Page | 26 stakeholders and through training stakeholders on how to implement such policies/strategies. SHIPP’s stepwise approach to supporting the development or revision of policies and strategic plans is shown in Figure 8 below. SHIPP’s aim was to ensure the policies/and strategies were aligned to the NSP and that they were adequately translated into an operational plan for implementation at provincial, district and community levels. However, SHIPP responded to SAG’s requests to support development or revision of policies and strategies as well as implementing the associated steps in the cascade. Figure 8: SHIPP's Approach to responding to SAG requests to support a Policy/Strategy Development and Implementation Achievements The evaluation team assessed SHIPP’s performance in supporting development/revision of HIV prevention policies and strategic plans by comparing actual mid-term achievement (derived from SHIPP’s progress reports) against the project’s life of project (LOP) targets by end of March 2014 (from the October 2012 PMP) Table 4 below shows that SHIPP exceeded LOP targets for five of its seven indicators with targets. Two indicators did not have targets and the team could not assess their progress. Two indicators (number of costed operational plans and number of multi-sectoral meetings held) scored 50% and 88% respectively and they were still on track to 100% achievement by the end of project. Table 4: IR1 Output Indicators, LOP Targets and Mid-term Performance Results Output Indicators LOP Targets (Oct 2012 PMP) Actual by March 2014 Percent Achievement Output Indicators in the latest PMP (September 2013) 1. Number of national, provincial and district HIV prevention policies and/or strategic plans are developed or revised with support from SHIPP and are aligned with the NSP 6 14 233% 2. Number of national, provincial and district HIV prevention policies and/or strategic plans are adopted by respective authority 6 11 183% 3. Number of provincial, district, and sub-district HIV prevention operational pans developed with support from SHIPP 4 8 200% 4. Number of provincial, district, and sub-district HIV prevention operational plans approved by respective authority 4 7 175% 5. Number of provincial, district, and sub-district HIV prevention operational plans costed with support from SHIPP 4 2 50% Respond to SAG request to support Policy development Wait for Policy/strate gy to be adopted by authority Support developm ent of Operation al Plan Wait for Operation al Plan approval Support Costing of Operation al Plan Support developm ent of M&E Plan Support developm ent of Technical Guidelines Support populariza tion of Technical Guidelines Mid-term Performance Evaluation of the SHIPP Project Evaluation Report Page | 27 Output Indicators LOP Targets (Oct 2012 PMP) Actual by March 2014 Percent Achievement 6. Number of provinces, districts and sub-districts that developed and adopted an monitoring plan for tracking implementation of the HIV prevention operational plan N/A 3 N/A 7. Number of technical products (guidelines, tools, training modules) and analytical reports produced with SHIPP support 3 26 867% 8. Number of technical products developed with SHIPP support that have been popularized N/A 9 N/A Additional Output Indicators Number of multi-sectoral meetings held 8 7 88% Number of seconded staff 22 27 123% SHIPP’s high performance in the area of supporting policy formulation is in part due to the fact that the project’s LOP targets did not include targets for stakeholders (DPSA, DHET, and DSD) who joined SHIPP later. In addition, lack of information about the number of policies/strategies for which SAG required support, could have led SHIPP setting low targets for some indicators. Table 5: Policies developed with support from SHIPP Policy Level Policies Developed with SHIPP support Formally Adopted by 31 March 2014 National-level policies, strategies and plans National Strategic Plan for HIV, STIs and TB 2012-2016 Yes NDOH HIV Prevention Strategy for the Health Sector 2012-16 No DBE National TB/HIV policy No The DBE HIV and AIDS, STI and TB Strategy Yes NDOT HIV Prevention Strategic Plan for HIV, STI and TB (2012-16) Yes DSD Strategy on HIV, AIDS and STIs Yes National Condom Distribution Strategy Yes National Sex Worker Strategy Yes Sub-national￾level policies COJ Strategic Plan for HIV, STI and TB (2012-16) Yes COJ HIV Prevention Strategic Plan Yes Gert Sibande strategic plan for HIV, STI and TB 2012-16 Yes Gert Sibande DOH HIV and AIDS, STI and TB Control Business Plan No Ehlanzeni District Municipality HIV and AIDS, STI and TB Strategy Yes Zululand District’s strategic plan for HIV, STI, and TB 2012-16 Yes As noted in Table 5, some policies developed with SHIPP support had not yet been adopted by respective authorities by end of March 2014, including:  NDOH HIV prevention strategy for the Health Sector 2012-16 Mid-term Performance Evaluation of the SHIPP Project Evaluation Report Page | 28  DBE national TB/HIV policy  Gert Sibande DOH HIV and AIDS, STI and TB Control Business Plan The evaluation team notes that while progress had been made in developing 14 HIV prevention policies/strategies and 7 operational plans, SAG had made little progress in translating these policies into action. For example, only two of the developed operational plans (Ehlanzeni and Gert Sibande DSP Implementation plans) had been costed and only three had M&E plans for tracking implementation (M&E Plans for Mpumalanga PSP, Gert Sibande DSP, and National Sex Worker Strategy), Supporting coordination of HIV combination prevention efforts across government departments Seven stakeholder meetings were held from inception of SHIPP to March 2014, and through these meetings, SHIPP was able to bring different SAG departments together and create a forum for information sharing and a better understanding of combination prevention as a way to accelerate HIV prevention. SHIPP also supported OTDP and DPSA to provide strategic leadership across SAG departments including supporting consultations and development of partnerships with key stakeholders involved in advancing HIV mainstreaming in the public sector. SHIPP also supported efforts to get DOH, DBE and DSD to integrate their HIV prevention efforts national, provincial and district levels. For example, SHIPP supported inter-sectoral collaboration with the local DOH, DBE and DSD, and other stakeholders in Region A and G. While much work remains to be done to achieve significant integration of leadership, SHIPP‘s efforts towards this goal are noteworthy. Seconding Experts and providing STTA Initially SHIPP planned to second 22 experts at various SAG departments and structures at national, provincial and district levels, but by March 2014, USAID had approved 37 secondment positions and SHIPP had filled 27 positions. . Some of the 10 vacant positions had previously been filled but the seconded staff left SHIPP as the project was being implemented. Mid-term Performance Evaluation of the SHIPP Project Evaluation Report Page | 29 Figure 9: Filled and Vacant Secondment Posts by 31 March 2014 SHIPP’s seconded staff alleviated staff shortages at SAG structures as well as enabled SAG departments to access technical assistance for HIV prevention. The seconded staff with technical expertise in health, education, engagement and strategic planning, created cross￾functional Leadership Capacity Teams (LCTs) at their level of operations. Three LCTs were to be formed at national level one each for NDOH, DBE, and SANAC. Each LCT was to be comprised of two experts each at NDOH and DBE and three experts at SANAC. At district AIDS council level, five LCTs were envisaged with each LCT comprising of three experts. The LCTs were to receive head office SHIPP support through respective Senior Technical Officers, Technical Directors, and Deputy and Chief of Party. The purpose of the LCTs was to improve learning, coaching, and mentoring and thus build SAG leadership capacity. Overall, the evaluation team established that the LCTs did not function as planned. LCTs failed to work due to:  Inadequate numbers of seconded staff at SAG structures during the first two years of SHIPP’s operation. It was only in Year 3 that the number of seconded staff increased to about 30.  Ensuring that the SAG beneficiary department was part of the recruitment process sometimes led to lengthy recruitment processes.  Vacancies within SAG limiting counterparts to mentor  Not all seconded staff were deemed experts.  SAG sometimes assigned non-TA work to seconded staff. Filled and Vacant Secondment Posts by 31 March 2014 18 Mid-term Performance Evaluation of the SHIPP Project Evaluation Report Page | 30 Supporting national level HIV programs and activities SHIPP supported activities also contributed to strengthening national level leadership at various SAG departments and structures. By 31 March 2014, SHIPP had supported several programs and activities at national level for the following SAG departments and structures: OTDP: SHIPP key assistance to the OTDP included support to the OTDP in its role as co-chair of the SAG–United States Government (USG) Partnership Framework Implementation Plan (PFIP) Management Committee. Consultations were held with the various PFIP work-streams and PEPFAR Technical Working Groups (TWG). SHIPP’s support to the OTDP is to ensure better alignment of and cross-sectoral planning, implementation and monitoring of all government programs across all departments and across PEPFAR, in accordance with South Africa’s priorities (including program sustainability) SHIPP seconded a Senior Technical Officer (STO) who was an expert in research analysis. SHIPP provided research support for publications and presentations by the Deputy President (DP) and the Special Advisory to the DP in various international meetings. In addition, SHIPP assisted OTDP to provide strategic leadership across SAG departments DPSA: Support to DPSA included: development of partnerships with key stakeholder groups involved in advancing HIV mainstreaming in the public sector. These included discussions with key SAG social and economic clusters, international partner organizations (e.g., SADC) and other international organizations (e.g., ILO and IOM). DPSA was also supported in its plans to establish an advocacy and support group for people living with HIV in public service. SHIPP assisted the DPSA to mainstream HIV prevention activities in Sports and Wellness event in the public service. DPSA was also supported to pilot local epidemic assessment tools and models to assist SAG structures to appropriately develop Know Your Epidemic/Know Your Response/Know Your Local Capacity and Institutions (KYE/KYR/KYC&I). In further support to DPSA, SHIPP participated in analysis of Coordination Framework and Mainstreaming Indicators for all SAG national departments. SANAC: SANAC was the first SAG department to receive SHIPP support with the seconding of a Research Analyst in April 2011. SANAC support included finalizing the NSP 2012-2016 through technical inputs on combination HIV prevention interventions. SHIPP’s other contributions were around TB-HIV integration leading to adoption in the NSP 2012-2016 of the strategies proposed by SHIPP for integrating HIV and TB in South Africa. Following the launch of the NSP, SHIPP continued to support its implementation. SHIPP participated in the technical reference group for the governance of SANAC, and provided further support to SANAC in:  Young Women and Girls Response  National Mobile Men Program  National Sex Worker Program Mid-term Performance Evaluation of the SHIPP Project Evaluation Report Page | 31  National Transport Coordination Sector Support NDOH: SHIPP responded to NDOH’s request to develop a synthesis paper on identification of HIV prevention priorities for South Africa in general and for the 3 SHIPP focus provinces. SHIPP provided short-term support through facilitating workshops on health needs of adolescents and informal settlements. As part of leadership strengthening at NDOH, SHIPP contributed to (i) finalizing the NDOH HIV Prevention Strategy, (ii) developing the NDOH’s Condom Supply and Distribution Strategy and Implementation Plan and (iii) developing the Home to Home HIV Counselling and Testing guidelines and Couple HIV Counselling & Testing implementation guidelines. SHIPP supported the NDOH in a Human Papilloma Virus (HPV) Vaccination Program aimed at reaching 500,000 girls in 17,000 schools. SHIPP seconded a technical expert at the NDOH to help support the HPV vaccination campaign. The campaign’s major components included planning and modeling, monitoring and evaluation, communication and social mobilization, training and capacity development as well as partner engagement. DBE: SHIPP supported a number of DBE’s activities including: Costing of DBE-Integrated HIV, TB and STI Strategy and developing the DBE HIV and TB Policy as well as TB field guides. SHIPP provided TA to develop the Peer Education Guidelines for facilitators and participants. In further support to DBE’s leadership, SHIPP assisted in developing scripted lessons on Sexuality Education Plans for the Life Orientation curriculum. SHIPP also supported the design of the Positive Prevention and Psychosocial Support intervention for HIV-positive students, but this had not yet been incorporated in the schooling system. DSD: SHIPP support to DSD started in Year 2 and included the finalization and dissemination of a costed DSD HIV/AIDS Strategy. SHIPP further supported DSD to define a gender integration framework and approach to HIV prevention as well as facilitating costing capacity building workshops for DSD. DHET: By March 2014, March 2014SHIPP was in the process of supporting DHET and HEAIDS to develop a national HIV, STI and TB strategy for the Further Education and Training (FET) sector. Most Significant Changes Reported by SAG “Capacity building of structures in Zululand district - First of its kind in KZN set a landmark that people can learn from. All districts are using the manual developed by SHIPP. Development of DSP on HAST” ~ Zululand District Respondent “Better coordinating departments at level of SANAC Better capacity within coordinating departments at level of SAG” ~ SAG National Department Respondent “Definitely had impact on relationships with government. There was a history of tension between Mid-term Performance Evaluation of the SHIPP Project Evaluation Report Page | 32 SAG & PEPFAR partners” ~ USG Respondent “City of Johannesburg AIDS Council is stable, directed, strategic” ~ USG Respondent “Building leadership capacity - government, district municipalities and structures “ ~ SHIPP Consortium Respondent Shortfalls and Reasons SHIPP did not achieve the SO for strengthening leadership capacity as intended. While it contributed to specific improvements in policy that support SAG HIV prevention priorities, some policies (e.g. the NDOH Prevention Strategy) are yet to be translated into action. Only two implementation plans have been costed and three M&E plans developed. The evaluation team notes that failure to translate the policies into action is not SHIPP’s fault as it is the responsibility of SAG to operationalize the policies. The other shortfall was in monitoring, evaluation, research of supporting leadership capacity. The evaluation team found that SHIPP provided limited TA at national, provincial, and district levels for data analysis and use to effectively translate data to inform implementation. For example, during the June 2012 stakeholder meeting, SAG asked SHIPP to assist SANAC to develop a multi-sectoral M&E framework to facilitate the planning and evaluation of combination HIV prevention response; however, this had not happened (except for MPAC’s, Gert Sibande and Ehlanzeni M&E plans) by March 2013. March 2014. SHIPP was supposed to document what works and feed-back to decision-makers to enable leaders decide on what to scale up. Some stakeholders expected they would work with SHIPP to design innovative biomedical, structural, social and behavioral communication for piloting through the project’s Small Grants Recipients (SGRs) in SHIPP’s intervention districts. “SAG expected that SHIPP would talk to the Steering Committee (referring to SHIPP’s Stakeholder Meetings) and then would decide on combination prevention projects design them and work with local municipalities and NGOs to test, track, document, and determine what works. And each of the NGOs would have been selected for a group of interventions which would have been tested” ~ SAG Respondent There are many reasons why SHIPP did not fully achieve the objective of strengthening leadership capacity as intended. The reasons are both internal and external to SHIPP and they are discussed further under Section 4.3, but the immediate reasons can be attributed to:  delays in seconding staff  LCTs did not function as planned.  high turnover of both seconded staff and head office staff that were supposed to Mid-term Performance Evaluation of the SHIPP Project Evaluation Report Page | 33 support the seconded staff  delayed adoption of policies due to SAG structures having other competing demands; however the evaluation team notes that some documents were not drafted to the satisfaction of stakeholders and were subsequently revised and updated several times. For example, in May 2012, SHIPP was asked to assist with developing the NDOH Prevention Strategy which had been started a few years earlier. However, the strategy’s final draft was only completed in the first quarter of Year 4 (October to December 2013).  dependence on SAG making requests for TA that was within SHIPP’s mandate. SHIPP responded to SAG requests for secondments and technical assistance and SHIPP used these TA requests for work planning. In some cases, such as with DHET, negotiations for appropriate SHIPP support were protracted leading to delayed provision of SHIPP support. 4.1.2 SO2: STRENGTHENING PRIORITY HIV PREVENTION SERVICE DELIVERY SYSTEMS IN NDOH AND DBE Determining the project’s achievement in this SO was problematic due to the fact that indicators for this SO were inconsistent between the first and revised versions of the PMP. SO2 aimed to strengthen priority HIV prevention delivery systems within NDOH and DBE, however, support to DSD was included from February 2012 – over a year after SO2 had been formulated in SHIPP’s Project Documents. SHIPP also engaged DHET in February 2012 but it was not until the first quarter of Year 4 that SHIPP officially started work with DHET. SHIPP’s stated approach for achieving SO 2 was to strengthen policies and programs as a means to improve implementation, but project documents indicate a focus on institutional assessments (of LACS, WACS, etc.) for determining capacity for implementing prevention programmes, and epidemiological analysis (LEAP) at the local level. The evaluation team also established that SHIPP strengthened priority delivery systems through:  Supporting development of HIV prevention guidelines and tools  Capacity building of HIV prevention services providers  Supporting monitoring and evaluation activities Achievements Table 6: IR2 Output Indicators, LOP Targets and Mid-term Performance Results Output Indicators LOP Targets (Oct 2012 PMP) Actual by March 2014 Percent Achievement Output Indicators in the latest PMP (September 2013) 9. Number of local SAG structures and/or coordinating bodies assessed by SHIPP 25 9 36% Mid-term Performance Evaluation of the SHIPP Project Evaluation Report Page | 34 Output Indicators LOP Targets (Oct 2012 PMP) Actual by March 2014 Percent Achievement 10. Number of local SAG structures and/or coordinating bodies that took steps to address gaps and/or barriers identified by the assessment N/A 15 N/A 11. Number of localities that were engaged by SHIPP to undertake an analytical process for identifying program priorities 13 4 31% 7. Number of technical products (guidelines, tools, training modules) and analytical reports produced with SHIPP support 3 26 867% Additional Output Indicators Number of PACs whose capacity was built 3 1 33% Number of LACs created 3 4 133% Number of LACs reconstituted 12 2 17% Number of WACs formed N/A 45 N/A Number of people trained 800 3,472 434% Assessment of SAG structures and/or coordinating bodies: SHIPP’s objective for assessing SAG structures and coordinating bodies was to build the structures’ capacity to:  Use assessment tools (e.g. SWOT analyses, baseline assessment tools, use of geospatial mapping).  Analyze, interpret and communicate findings of assessments conducted to improve service delivery.  Monitor and supervise HIV prevention service delivery  Advocate for resources based on assessment outcomes including ensuring that activities for HIV prevention are included in the Integrated Development Plan (IDP) for the activities to be budgeted. Based on SHIPP’s reported results, the project underperformed on the indicator for tracking assessment of SAG structures and other coordinating bodies by achieving only 36% after three and half years (70%) of the LOP. However, the team’s closer scrutiny of the indicator results showed that SHIPP did not track the indicator adequately leading to invalid reporting. For example in its Year 3 report, SHIPP reported training sessions provided to AIDS Councils as assessments. In addition, SHIPP underreported for this indicator. During FY 2012, SHIPP reported 9 assessments (per Table 6), but actually assessed the functionality of four DACs and six LACs as well as the following 6 assessments (i.e. 16 assessments):  Rapid assessment of the HIV prevention needs in SHIPP’s 5 local municipalities in Zululand Mid-term Performance Evaluation of the SHIPP Project Evaluation Report Page | 35  Health SWOT analysis in Regions A and G to identify the regions health priorities  Assessment of current HIV Prevention services in five facilities in Region A using the Baseline Assessment Tool  Assessment of the functionality of clinic health committees in City of Johannesburg  Assessment of HIV High Transmission Areas (HTAs) using national baseline tool in Muskaligwa  Situational analysis and Monitoring and Evaluation (M&E) capacity assessments of HIV prevention programs in Mpumalanga Number of local SAG structures and/or coordinating bodies that took steps to address gaps and/or barriers identified by the assessment: SHIPP also used the number of training sessions provided to AIDS Councils to report on this indicator. The initial intention for this indicator was to count concrete steps undertaken by the assessed SAG structures/coordinating bodies based on the assessment recommendations, such as hiring M&E Officers/HIV Coordinators, and increasing funding for supervision and M&E. Supporting districts and sub-districts to undertake analytical processes for identifying program priorities: SHIPP underperformed on the indicator for tracking number of districts and sub-districts undertaking analytical processes for identifying program priorities. SHIPP reported 31% achievement on this indicator after 70% of implementation period had elapsed. Only 4 of 13 localities were assisted in this analytical process, which is comprised the following steps: 1. Creating a team that will lead the process 2. Collecting secondary and primary data, including demographic information, and epidemiological and behavioral data 3. Analyzing the data using a model-based analysis tool 4. Determining which scenario would have the highest impact on the HIV epidemic in the locality and develop a plan of action or strategy accordingly Due to lack of information the evaluation team was not able to establish how far along each locality was in their analytical processes. SHIPP intended to support districts in using Local Epidemic Assessment for Prevention (LEAP) to guide the development of a combination prevention approach appropriate for each district. Districts were to use LEAP to define their local epidemic, identify appropriate and targeted combination prevention activities, and monitor and report on the impact of their interventions. This was to be in line with the Know Your Epidemic, Know Your Response (KYE/KYR) framework. However, SHIPP failed in this approach as the LEAP process did not materialize as intended. SAG asked SHIPP to stop using LEAP until the tool was interrogated and aligned to existing Mid-term Performance Evaluation of the SHIPP Project Evaluation Report Page | 36 tools with similar utilities. SHIPP also introduced geospatial mapping as a tool to facilitate the understanding of the geographic distribution of HIV prevalence and services in the SHIPP localities and hence inform the design and delivery of combined HIV prevention response. In collaboration with the University of KwaZulu-Natal, SHIPP used DHIS, PEPFAR partner and other administrative data to create geospatial maps for Gert Sibande and Nkomazi sub-districts to identify hot spots and available services in these SHIPP supported areas. This work was highly regarded by respondents. Supporting development of HIV prevention guidelines and tools: SHIPP supported priority HIV prevention services and key delivery systems for numerous government departments as shown in Table 7 below: Table 7: SAG Structures and Supported Systems SAG Structure Supported HIV Delivery System Location / Level DOH Condom Supply and Distribution Strategy NDOH Home to Home HIV Counselling and Testing guidelines NDOH Voluntary Medical Male Circumcision (VMMC) activities at national level NDOH A model for VMMC was set up in Region G Region G, Gauteng Condom Distribution Plan Gert Sibande district STI services and condom distribution data collection tools for the GP health districts Gauteng Province Integration of STI prevention activities in care and treatment services Johannesburg, Gauteng Training CHWs to scale up delivery of HIV prevention services Johannesburg, Gauteng DBE Scripted Lessons on Sexuality Education Plans DBE Peer Education Manuals DBE TB Field Guides DBE Combination Prevention Training of peer educators Region G, Gauteng Training of educators and learners on life skills Msukaligwa sub-district in Gert Sibande Workshop on Peer Education Guidelines Vryheid District Peer Education training life orientation educators ZDM five sub-districts M&E workshop for Provincial & District HIV/AIDS and Life Skills Coordinators DBE Pre- and in-service training for Care and Support Forums, Care and Support for Teaching and Learning (CSTL) educators and support staff Ehlanzeni and Gert Sibande Districts DSD CCE-CC workshops for SGRs to improve implementation/ outcomes of community dialogues Regions A and G, Gauteng Mid-term Performance Evaluation of the SHIPP Project Evaluation Report Page | 37 Capacity building of HIV prevention service providers: The team noted that SHIPP did very well in its capacity building of HIV prevention service providers At least 3,472 people were trained on better coordination, service delivery, data demand and data use, monitoring and evaluation functionalities; community mobilization for HIV prevention; using LEAP; use the spatial mapping system for decision making. The largest numbers of people trained were AIDS Council members followed by Peer Educators. SHIPP facilitated workshops for Ward AIDS Committee (WAC) members in 89 wards of Zululand District Municipality (ZDM) to revive and strengthen the WAC in which a total of 1313 participants from the 89 wards were trained. SHIPP, in partnership with CADRE, provided capacity building training sessions to Gert Sibande and Ehlanzeni LACs in SHIPP’s focus sub-districts. In some wards, SHIPP supported the establishment of a total of 45 WACs and resuscitation of some structures (e.g., Msukaligwa LAC) that had been formed but were dysfunctional. SHIPP also engaged and trained community healthcare workers (CHWs) to scale up delivery of HIV prevention services in the City of Johannesburg’s Region G. The CHWs were trained on how to create and increase demand for voluntary HIV testing and counselling, male circumcision, and consistent condom use within the informal settlement areas in Region G. Other support provided was training of educators and learners on life skills and Peer Education Guidelines as well as conducting an M&E workshop for Provincial & District HIV/AIDS and Life Skills Coordinators. Monitoring and evaluation support: One anticipated function of SHIPP was to build the capacity of SAG to effectively use qualitative and quantitative data at all levels of decision-making, especially at district levels. SHIPP’s activities in this area included:  Providing technical assistance in coordinating the quarterly data review in Abaqulusi (Zululand), to ensure that the data reported are of high quality, and that feedback is given to the facilities.  Conducting an M&E workshop for Provincial & District HIV/AIDS and Life Skills Coordinators to facilitate monitoring of the implementation of the Peer Education Programme as guided by the National Peer Education Guidelines  SHIPP supported the Zululand district health team in analyzing and interpreting the district performance on HIV prevention indicators  Geospatial mapping  Analyzing 2012 Antenatal clinic survey data focusing on why Gert Sibande’s HIV prevalence continued to be high. This informed the work done in Gert Sibande However, the evaluation team’s findings indicate SHIPP provided limited technical assistance for analysis and use of epidemiological, behavioral and program data especially at national and Mid-term Performance Evaluation of the SHIPP Project Evaluation Report Page | 38 provincial levels. In addition, there was little TA to establish mechanisms for provinces and districts to effectively translate data to inform implementation. Key informants views on the most significant change attributable to SHIPP are illustrated in the following Box: “Development of TB/HIV policy in the education sector is ground breaking, first ever in the region. Prevention for positives in schools - looking at the education sector's response to HIV+ students and a minimum package of psychosocial support needed. Peer Education guidelines” ~ SAG Respondent “Mapping of HTAs and secondary distribution sites (non-traditional sites). It would be great “if they could train the condom logistics people in GP and KZN”. They would in turn train the partners SHIPP is unable to train. “Everyone recognizes the need. Wherever we go, people are asking when we are coming”. MP is using it. In MP, the condom distribution in MP has “increased exponentially” and “we believe that it’s because they have streamlined their reporting, their distribution” and they are able to collect the data. “The demand has gone high and the distribution is able to meet that demand”. The reporting is more accurate” ~ SAG Respondent “Contribution to the National HIV prevention strategy -Contribution to the HPV vaccination plan and execution -Development of sex education guidelines” ~ SHIPP Consortium Respondent “DBE and HIV/TB strategy in 2012. Scripted lesson plans” ~ SHIPP Consortium Respondent “SHIPP has given advise to stakeholders and reached out to youth through training and peer education” ~ SHIPP Consortium Respondent According to respondents, SHIPP’s greatest contributions to services delivery systems were the DBE TB/HIV policy and scripted lesson plans, the mapping of High Transmission Areas (HTAs), and secondary condom distribution sites (non-traditional sites) for the NDOH. Shortfalls and Reasons While SHIPP did very well capacitating AIDS Councils, coaching and mentoring is still needed to consolidate the capacity building and ensure AIDS Councils are using the HIV strategic documents that have been developed. SHIPP had also intended to facilitate AIDS Council to use dashboards to monitor program indicators, and uses data for decision making, but this was not done. SHIPP should follow up on this vital support to the AIDS Councils. SHIPP’s entry into the provinces and, hence sites of operation, were delayed due to lengthy provincial engagements for political buy-in. This is largely because the project’s early engagements with stakeholders were focused at national level, and when SHIPP attempted to roll out its work at district level, it met with resistance from 2 of the 3 provinces who had not been adequately consulted. In addition, SHIPP’s seconded staff only increased substantially in SHIPP’s third year of operation. Mid-term Performance Evaluation of the SHIPP Project Evaluation Report Page | 39 As of March 2014, SHIPP had only supported 31% of its targeted localities (districts and sub￾districts) to undertake analytical processes for identifying prevention needs in the local community and program priorities, mainly because LEAP was not implemented as planned. It did not demonstrate a minimum combination HIV prevention package for locally-identified key populations. SHIPP explains this by stating that because the NDOH prevention policy is still not approved, it is not at liberty to pilot or demonstrate minimum packages at service level. The evaluation team also noted that SHIPP did not strengthen peer education programming in some schools where SHIPP-funded SGRs operated. In such schools, the SGRs continued with their old ways of providing peer education which was not always aligned to DBE priorities and guidelines. SHIPP, as supporter of NSP implementation, should have ensured the SGR peer education activities were aligned to DBE guidelines. The SGRs state that they did not use recommended DBE guidelines due to limited/lack of direction from SHIPP’s Technical Officers. 4.1.3 SO3: IMPROVING THE QUALITY, EFFECTIVENESS AND COVERAGE OF HIV PREVENTION PROGRAMS AT THE COMMUNITY LEVEL The objective of community level efforts was to obtain adequate scale and to create critical mass needed to have a positive impact on individual behaviors, social norms and reduce HIV infection rates. SHIPP aimed to strengthen quality, effectiveness and coverage of HIV programs at the community level through the following: SHIPP aimed to strengthen quality, effectiveness and coverage of HIV programs at the community level by:  Supporting districts and sub-districts to strengthen the quality, coverage and effectiveness of their combination HIV prevention services for key populations  Competitively awarding small grants to selected CBOs and providing technical support to other CBOs to create demand for and provide HIV prevention services  Supporting the integration of gender and youth activities into HIV prevention services  Supporting capacity to create community networks and services referral systems  Engaging local community-based leaders to promote effective and sustainable HIV prevention efforts  Enhancing the coordination and synergies between and among sexual HIV prevention, treatment, and care partners Mid-term Performance Evaluation of the SHIPP Project Evaluation Report Page | 40 Achievements Table 8: IR3 Output Indicators, LOP Targets and Mid-term Performance Results Indicators LOP Targets (Oct 2012 PMP) Actual by March 2014 Percent Achievement Output Indicators in the latest PMP (September 2013) 12. Number of districts and sub-districts that have taken steps to strengthen the quality, coverage and effectiveness of their combination HIV prevention services for key populations N/A 4 N/A 13. Number of CBOs receiving financial and TA from SHIPP to provide combination HIV prevention services 40 33 83% 14. Number of individuals from target audience who participated in community‐wide event (PEPFAR P8.5.D) N/A 182,770 N/A 15. Male norms and behaviors: Number of people reached by an individual, small‐group, or community‐level intervention or service that explicitly addresses norms about masculinity related to HIV&AIDS(PEPFAR P12.1.D) N/A 0 N/A 16. Gender-based Violence and Coercion: Number of people reached by an individual, small group or community‐level intervention or service that explicitly addresses gender‐based violence and coercion related to HIV&AIDS(PEPFAR P12.2.D) N/A 6,211 N/A 17. Number of condoms distributed by CBOs supported by SHIPP N/A 3,253,512 N/A Additional Output Indicator Number of social mobilization campaigns, community meetings held to promote combination HIV prevention 80 132 165% Supporting districts and sub-districts to strengthen the quality, coverage and effectiveness of their combination HIV prevention services for key populations: All four districts where SHIPP operates had taken steps to provide HIV prevention services. The major achievements for this activity was the mapping of condom distribution in local municipalities and support the scale-up of combination HIV prevention programs, including mobilizing targeted communities for action and behavior change. To improve quality of HIV programs, SHIPP carried out a number of assessments. For example, SHIPP collaborated with district DOH coordinators to develop an HIV High Transmission Areas (HTA) assessment tool and HTA assessments were completed in Gert Sibande district. In addition, sexually transmitted infections (STI) and condom distribution assessments were completed in five districts of Gauteng province. Competitively awarding small grants to selected CBOs and providing technical support to other to create demand for and provide HIV prevention services: SHIPP performed well on this indicator with an achievement of 83%. The first small grants were issued only toward the end of 2012 to 5 CBOs in KZN. In 2013, a further 28 CBOs were contracted including 5 in KZN, 14 in MP and 9 in GP. The process of awarding the small grants to CBOs entailed SHIPP holding consultations with the provinces and districts. For example, SHIPP participated in men’s dialogue on gender based violence (GBV) held in eDumbe, Zululand. Three hundred men representing several SAG Mid-term Performance Evaluation of the SHIPP Project Evaluation Report Page | 41 structures and civil society organizations from all five sub-districts in the Zululand district participated in the dialogue. The delegates agreed to the areas which SHIPP would support to address gender based violence. SHIPP completed needs assessment and reviews of the current HIV prevention activities in the specific districts and sub-districts. SHIPP also held consultative and briefing workshops with CBOs that had expressed interest in the small grants program. Based on the findings of the HIV prevention needs assessment and stakeholder consultations, the scope of work for the small grants program was developed. The SGRs supported social and behavior change communication at community and interpersonal-levels with a focus on promoting HIV prevention activities and community-based public health services. The SGRs specific objectives were to:  Incorporate gender and youth transformative strategies to increase awareness of and uptake of HIV combination prevention interventions  Address stigma, discrimination and disclosure among people living with HIV and AIDS  Provide HIV/AIDS prevention education including activities specifically targeted to the men’s sector on cultural norms and behavior change  Increase uptake of HIV counselling and testing in the supported regions, with focus on informal settlements  Promote and distribute condoms in hard to reach areas  Reduce vulnerability to HIV and TB infection through social, structural and behavioral change; With SHIPP support, the CBOs expanded their staff numbers, increased field worker stipends, and increased geographical areas of operation. To increase awareness and create demand for HIV prevention services, SHIPP supported HIV education in schools and social mobilization via a variety of mechanisms, including:  Door-to-door campaigns  Workshops  Peer education in schools and facilitating camps for youth  Conducting dialogues with specified populations e.g. men and youth, and  Linking with national campaign efforts e.g. during World AIDS Day, STI and Condom Week, and 16 Days of Gender Activism for no violence against women and children. In KZN, SHIPP also supported HIV awareness during the traditional reed dance. The reed dance is a Zulu traditional ceremony where young girls are encouraged to maintain their virginity until marriage by delaying engaging in sexual activity. The reed dance is therefore an important strategy to prevent HIV/AIDS and other sexually transmitted diseases among the youth. Mid-term Performance Evaluation of the SHIPP Project Evaluation Report Page | 42 The SGRs reached a total of 182,770 individuals (men, women, and children) participated in community‐wide events which focused on creating awareness on how to prevent HIV. Nearly 7,300 youth were reached youth outreach and in-school education programs. Over 6,200 people were reached by individual, small group or community‐level intervention or service that explicitly addresses gender‐based violence and coercion related to HIV/AIDS. The SGRs also opened secondary condom distribution channels and distributed over 3.2 million condoms. To enable the community to take ownership of HIV prevention, SHIPP facilitated capacity building workshop for SGRs in the Community Action Teams (CATs) Model to facilitate broader community involvement and participation in taking action. The CATs model builds on community dialogues in formulating and undertaking actions to prevent GBV and HIV. The specific aims of CATs were to:  Educate men and women to understand how they have been socialized into gender roles that limit their full potential as human beings  Encourage communities to ignite social mobilization movement to change the social norms that perpetuate GBV and the spread of HIV infection  Increase the number of men, women, and organized social structures advocating or taking collective action to prevent GBV and HIV Figure 10: Three Phases of Forming CATs SHIPP put in place Gender CATs in over 80 marginalized wards in the thirteen sub-districts it works in. SHIPP district technical officers identified and trained local community facilitators from the SGRs. Following the training, the community facilitators identified and organized willing community residents into CATs to combat GBV and reduce HIV transmission. The teams usually comprised of men only, women only, or mixed groups of between ten and twenty volunteers and included people living with HIV/AIDS and religious leaders. Apart from working with the SGRs SHIPP facilitated capacity building workshops for 57 CBOs focusing on condom mapping and distribution in Gert Sibande district. The purpose of Mid-term Performance Evaluation of the SHIPP Project Evaluation Report Page | 43 mapping was to increase secondary distribution sites and increase demand and supply of condoms in the community. SHIPP also supported six organizations in Chief Albert Luthuli sub￾district. Capacity building focused on integration of key HIV prevention activities into their operational plans. In COJ Region A, SHIPP worked with and supported two local organizations (Philisa Isizwe, and Jozi Ihlomile) to plan and implement ward level HIV prevention activities. Supporting the integration of gender and youth activities into HIV prevention services: SHIPP supported the targeting of youth to address HIV prevention among young people in the context of teenage pregnancy, STI’s, HIV testing, condom usage, and alcohol abuse. For example, SGRs were required to incorporating gender and youth transformative strategies in their activities to increase awareness of and uptake of HIV combination prevention interventions. Through the SHIPP Youth and Gender Integration Unit, capacity building workshops were held for SGRs. In collaboration with DOH, SHIPP identified youth dialogues as a priority to achieve the outcomes for the DBE’s life skills and sexual and reproductive health curricula. By March 2014, SHIPP was in the process of developing a Gender Operational Framework for Gender and Youth Integration across SHIPP activities in all districts. Supporting capacity to create community networks and services referral systems: The Zululand DAC requested SHIPP to develop a community network and referral system to strengthen the Premier’s model of utilizing “war rooms” to collect information at community level. SHIPP led a number of visits to the Operation Sukuma Sakhe (OSS) war rooms to conduct the situational analysis of the current ZDM HIV prevention referral system. Some challenges identified from the situational analysis included lack of awareness of available HIV prevention services. As a result Zululand District Municipality (ZDM) requested SHIPP to support the development of a ZDM Referral Directory. The COJ requested SHIPP to review referral systems in the community, and as a result SHIPP enabled the mapping of 18 NGOs providing services in Region A. SHIPP also supported the setting up of an integrated and comprehensive referral system in Gert Sibande district to ensure adequate co-ordination within the HTA mobile clinics and the CBOs. The referral system was to assist in distribution of condoms and IEC materials by CBO for all the truck stops and sex worker hot spots in Gert Sibande. Engaging local community-based leaders to promote effective and sustainable HIV prevention efforts: SHIPP engaged with religious leaders in Zululand and obtained buy-in from the religious fraternity to promote HIV prevention. The project also targeted the traditional health sector (i.e. traditional health practitioners and leaders) with advocacy to increase awareness of men’s health and the safety of VMMC. In collaboration with the HIV & AIDS, STIs and TB (HAST) Unit in Gauteng Province and HIV South Africa, SHIPP organized a capacity building training workshop for 80 traditional healers in Region G. The traditional healers committed to work with the DOH in HIV/AIDS prevention strategies. Mid-term Performance Evaluation of the SHIPP Project Evaluation Report Page | 44 Enhancing coordination and synergies between and among sexual HIV prevention, treatment, and care partners: SHIPP participated in stakeholder workshop co-facilitated by SHIPP, ANOVA Health Institute, and UNAIDS to clarify the roles of all partners in Mpumalanga province. SHIPP established working partnerships with AgriAIDS to scale up HIV prevention among farm worker populations in Gert Sibande district. SHIPP also established strategic partnerships with BroadReach Health Care to coordinate prevention activities in the district. SHIPP initiated activities to support the roll out of HIV prevention activities for key populations. Shortfalls and Reasons The main shortfalls include:  At the time of the evaluation, SHIPP had not yet supported the definition of minimal or optimal combination HIV prevention packages through an analytical process for key populations identified at locality level.  The implementation of the small grants program was delayed  Inadequate support for linkages of HIV prevention with mass and social media.  SHIPP did not empower communities to sustain community-led efforts.  Failure to support AIDS Councils mechanisms to measure coverage through measurement and reporting on denominators (potential number of beneficiaries) at its area of operation. The purpose of the small grants program is to provide support to CBOs to enhance and better align their services within the newly-formulated prevention needs of the community. However, by 31 March 2014, SHIPP had not yet supported the AIDS Councils to define optimal combination HIV packages for key populations identified at district and sub-district level through an analytical process. SHIPP had intended to support the definition of combination HIV prevention packages using Local Epidemic Assessment for Prevention (LEAP) process but SAG asked SHIPP to stop using the tool until SAG interrogated and aligned it to other existing tools. LEAP was supposed to use available HIV and AIDS research and local surveillance and program data in the intervention districts to identify and address multiple levels of HIV and AIDS transmission risks and informs the best use of available resources to maximize prevention impact in a given locality. The implementation of SGR model was delayed due to the slow-paced nature of provincial political consultations for political buy-in and access to intervention sites. Availability of appropriate CBOs experienced in HIV prevention services was another challenge. Some SGRs that SHIPP funded were turned from HBC organizations to organizations that provide HIV prevention services. Most SGR reported that they had not yet covered all the areas that they were supposed to cover either due to the geographical vastness of some localities and/or the short duration of Mid-term Performance Evaluation of the SHIPP Project Evaluation Report Page | 45 implementation. Dysfunctional or non-existent DACs, LACs and WACs meant that instead of training these structures on combination prevention SHIPP instead spent time (1) establishing structures where there were none (2) and capacitating new and old structures so that they could be take up the role of identifying and coordinating implementation of combination interventions SHIPP in its workplans intended to support AIDS Councils with targets for quality, coverage and effectiveness. SHIPP did not support AIDS Councils to measure coverage through measurement and reporting on denominators (potential number of beneficiaries) at its area of operation. In addition, SHIPP did not support AIDS Councils to measure behavior and social change and therefore effectiveness could not be demonstrated. SHIPP did not empower communities to sustain community-led efforts. While efforts were made to have municipalities fund the LACs and WACs these efforts are yet to bear fruits for most of these structures. Issues reported by some SGRs are summed in the following Box. “The process of the contracting of our organization was delayed. It was good to receive the pre-award visit but we struggled to finalize the contract. It would be good to have had a little more help with some of the contract nuances. We got support throughout the grant period although we did not need much organizational and technical support. So although the answers provided during the question section may be more neutral or disagree, this does not mean that there was insufficient support from SHIPP, just that we had a particular model and our grant supported that model and our expertise is well established. And we were confident in what we did. SHIPP provided great support to access to GDE at the beginning of the grant and opened all those communication channels which set the positive tone for the grant period” ~ SGR Respondent 4.1.4 RECOMMENDATIONS  Improve quality assurance of SHIPP’s technical inputs into policy documents to decrease the cycles of revision and time needed to finalize a document.  Develop a ‘menu’ of TA areas that are within SHIPP’s mandate even while waiting for SAG requests.  Develop guidelines for implementing HIV combination prevention activities before funding SGRs.  Support alternative LEAP process such as the DPSA’s Local Assessment Model based on KYE/KYR.  Build on the capacity developed for DACs, LACs and WACs to enable them identify, Mid-term Performance Evaluation of the SHIPP Project Evaluation Report Page | 46 implement and coordinate optimal HIV prevention packages.  Establish mechanisms to measure coverage through measurement and reporting on denominators (potential number of beneficiaries) at its area of operation.  Support AIDS Councils to establish mechanisms to measure behavior and social change.  Document and share with stakeholders what works and what doesn’t during SHIPP’s fifth year of operation 4.2 To what extent have combination HIV prevention approaches been incorporated into policy, planning, and strategy in partner SAG departments? The policy circle in Figure 11 below depicts an overview of the steps involved in the process of policy formulation and implementation in Government. With regards to the inclusion of combination prevention in SAG policies, strategies, and plans, the evaluation team has established, that SHIPP has contributed to all the steps outlined. SHIPP provided technical input (via seconded staff, STTA, or participation in consultative workshops) into the incorporation of combination prevention into policies and strategic plans at the different spheres of government it works with. It also provided support for developing costed implementation plans and operational plans. 4.2.1 FINDINGS While SHIPP’s core mandate was to provide TA to support the implementation of combination prevention, a large portion of SHIPP’s activities during the first four years were focused on supporting stakeholders within different spheres of government to develop strategic plans aligned to the NSP 2012-2016. SHIPP’s contribution towards including combination prevention in the national NSP 2012-2016, as well as into departmental and sub-national strategic plans, has helped keep combination prevention in the spotlight. SHIPP’s main challenge was demonstrating how combination prevention approach works in reality. Many key respondents noted that they fully understood the rationale behind combination prevention, but had expected, and would still like, SHIPP to demonstrate the tailoring and implementation of combination prevention packages for key populations. SHIPP complemented its policy-level work with activities aimed at strengthening policy implementation as evidenced by support provided to SAG departments at district level, and to a Mid-term Performance Evaluation of the SHIPP Project Evaluation Report Page | 47 Figure 11: Policy Circle Source: Adapted from THE POLICY CIRCLE: A Framework for Analyzing the Components of Family Planning, Reproductive Health, Maternal Health, and HIV/AIDS Policies. 2004. lesser extent, at provincial level. SHIPP’s work with strengthening PACs, DACs and WACs is particularly notable. SHIPP provided ongoing input around implementation through its participation in strategy review processes at national and provincial levels, stakeholder consultation workshops, etc. An early contribution made by SHIPP in the policy arena was its support to SANAC and the Prevention Task Team for developing the prevention component of the NSP. Through SHIPP’s technical input, via a seconded staff member and other consultations, it contributed to the inclusion of combination prevention as the main prevention strategy. When the current NSP 2012-2016 was launched on World AIDS Day, 1 December 2011, combination prevention was featured as the approach which should be used to achieve the goal of reducing HIV incidence by at least 50%. The weight given to HIV prevention in the NSP and the specific inclusion of combination prevention, created an enabling environment for SHIPP to assist SAG departments in fulfilling their respective roles in the reduction of HIV incidence. To date, SHIPP has provided technical support in incorporating/ strengthening combination prevention in the HIV/AIDS response for the following SAG departments: DOH, DBE, SANAC, Political, Social, Cultural and Economic Context Stakeholders Consultation (People) Process of Policy Development Costing of the Policy (Price Tag) Policy Dissemination Implementation and M&E Problem Mid-term Performance Evaluation of the SHIPP Project Evaluation Report Page | 48 DSD, DPSA, DHET and the Office of the Deputy President (ODP). Between September 2010 and March 2014, SHIPP’s support led to combination HIV prevention approaches being incorporated into numerous other SAG policies, strategies, plans and guidelines shown in Table 9 below. Table 9: Key SHIPP Supported Policies, Strategies, and Plans Incorporating Combination Prevention Policy, Strategy, Plans, Guidelines Developed Approved SHIPP’s Role National Level National Strategic Plan for HIV, AIDS and TB (2012- 2016) X X Provided technical input via a seconded person to the SANAC Secretariat. SHIPP was part of a task team which contributed to the draft on combination prevention which was eventually incorporated into the NSP 2012-2016. National DSD Strategy on HIV, AIDS and STIs X X SHIPP provided support in finalizing and disseminating the DSD HIV/AIDS strategy - including the development of a gender-equity framework. National DOT Strategic Plan for HIV, STIs and TB (2012- 2016) X X Provided technical input National Condom Distribution Plan 2013/2014 X X SHIPP provided technical support in developing and finalizing the plan. National DBE integrated HIV, TB and STI strategy. X X Financial support (Design, layout and printing) NDOH HIV prevention strategy for the Health Sector 2012-16 X Pending Provided technical support to finalize the prevention strategy Operational plan for NSP 2012-16 X X Information not available National Strategic Plan for HIV Prevention, Care and Treatment for Sex Workers X Provided technical input via seconded staff at SANAC. Sub-national level Gauteng Gauteng Strategic Plan on HIV, TB and STIs 2012-2016 X X Provided technical input during the consultative conference, facilitated a session on reducing HIV infections. The session resulted in strong recommendation to adopt combination prevention as the provincial HIV prevention strategy. This was later adopted. COJ Strategic Plan for HIV, STI and TB 2012-16 X X Information on specific SHIPP support not available COJ HIV Prevention Strategic Plan X X Provided support to developing the strategic plan. Mid-term Performance Evaluation of the SHIPP Project Evaluation Report Page | 49 Policy, Strategy, Plans, Guidelines Developed Approved SHIPP’s Role Operational plan for COJ’s district strategic plan for HIV, STI and TB 2012-16 X X Information on specific SHIPP support not available Operational plan for COJ’s HIV prevention strategic plan X X Information on specific SHIPP support not available KwaZulu-Natal Zululand district’s strategic plan for HIV, STI, and TB 2012-16 X X Facilitated several consultations which led to the development of the district strategic plan. Operational plan for Zululand’s district strategic plan for HIV, STI, and TB 2012-16 X X Information on specific SHIPP support not available Zululand district’s implementation plan on HAST 2012-16 X X Information on specific SHIPP support not available Mpumalanga Mpumalanga Provincial Strategic Plan (2012-2016) X X Provided support in finalizing the Provincial Strategic Plan (PSP) review including writing the prevention component of the PSP and support with copy editing. Mpumalanga Provincial Implementation Plan for HIV, STIs and TB X X Provided support in developing and finalizing the implementation plan. Mpumalanga AIDS Council (MPAC) Operational Plan X X SHIPP seconded an MPAC Secretariat Director to assist in implementing the PSP Gert Sibande District’s Strategic Plan for HIV, STI and TB 2012-16 X X Provided technical support for developing and finalizing the district strategic plan. Gert Sibande DOH HIV and AIDS, STI and TB Control Business Plan X Pending Information on specific SHIPP support not available Ehlanzeni District Strategic Plan2013 to 2016 X X Provided support to developing and finalizing the district strategic plan. Other Eastern Cape Provincial Department of Social Development Operational Plan for HIV, AIDS and TB (2012-2016) X X Information on specific SHIPP support not available A select number of policies and strategic plans which SHIPP provided technical inputs into and the extent to which they incorporate combination prevention is discussed below. Mid-term Performance Evaluation of the SHIPP Project Evaluation Report Page | 50 4.2.2 NATIONAL STRATEGIC PLAN ON HIV, STIS AND TB 2012-2016 The South African National Strategic Plan on HIV, STIs and TB 2012-2016 was launched on 1st December 2011 replacing the National Strategic Plan 2007 – 2011. The plan has four strategic objectives designed to guide South Africa’s response to HIV, STI and TB as follows: Strategic Objective 1: Address social and structural barriers to HIV, STI and TB prevention, care and impact; Strategic Objective 2: Prevent new HIV, STI and TB infections; Strategic Objective 3: Sustain health and wellness; and Strategic Objective 4: Increase protection of human rights and improve access to justice. With regards to SO2, Prevention of new HIV, STI and TB infections, the NSP 2012-2016 identifies combination preventions as the approach that should be used to meet the objective. The NSP defines combination prevention as an approach which “…seeks to achieve maximum impact on HIV prevention by combining behavioral, biomedical and structural strategies that are human rights-based and evidence-informed, in the context of a well-researched and understood local epidemic”. SHIPP provided input into the development of Strategic Objective 2 and its work was guided by SO2’s six sub-objectives outlined under this strategic objective:  Sub-objective 2.1: Ensure everyone in South Africa tests voluntarily for HIV and is screened for TB annually, and subsequently enrolls in relevant wellness and treatment, care and support programs;  Sub-objective 2.2: Make accessible a package of sexual and reproductive health services to prevent HIV and STIs, with emphasis on key populations, including strengthening of syndromic management of STIs in both the public and private health sectors;  Sub-objective 2.3: Prevent transmission of HIV from mother to child to reduce MTCT to less than 2% at six weeks post-birth and to less than 5% at 18 months of age by 2016;  Sub-objective 2.4: Implement a national social and behavioral change communication program with a focus on key populations to shift social norms (especially those related to gender), attitudes, promote healthy behaviors, and increase demand and uptake of services;  Sub-objective 2.5: Prepare for the potential implementation of innovative biomedical prevention strategies, such as microbicides, PrEP and treatment as prevention; and  Sub-objective 2.6: Prevent new TB infection and disease through intermittent preventative therapy (IPT), infection control, early identification and treatment of TB and an improved TB cure rate. Mid-term Performance Evaluation of the SHIPP Project Evaluation Report Page | 51 The NSP discusses each sub-objective in detail explaining which types of interventions would help meet the respective sub-objectives as well as core indicators for measuring progress against the strategic objective. SHIPP support to the NSP included provision of technical input via a research officer seconded to the SANAC Secretariat. The technical input included was focused on the inclusion of combination prevention as a specific strategy which was subsequently approved by SANAC as a key approach for NSP 2012-2016. SHIPP further provided input for the development of associated provincial operational plans. 4.2.3 EXAMPLES OF SHIPP SUPPORT FOR COMBINATION HIV PREVENTION SHIPP support for combination prevention is evident from its work with three SAG departments: Health Sector – HIV Prevention Strategy and Guideline 2013-2016 (Pending Adoption) SHIPP provided technical support for refining and finalizing the Health Sector’s HIV Prevention Strategy and Guideline 2013-2016. Aligned to the NSP 2012-2016, this strategy outlines the DOH’s combination prevention strategy for reducing the HIV incidence within the general population as well as among key populations. The strategy is intended to guide health care managers in tailoring prevention interventions to the needs of the people they serve within their catchment areas and the local context. The strategy defines a package of HIV prevention interventions to be provided by the health sector and comprised of a combination of biomedical, socio-behavioral and structural interventions. The strategy and guideline is intended to be implemented by health sector mangers (district managers) in close collaboration with other sectors. District Health Managers are expected to7 :  Understand the epidemic in their respective districts and sub-districts and tailor HIV prevention interventions accordingly  Know the guidelines and policies that are conducive for the implementation of selected interventions  Invest their efforts in line with the priority or key populations present in their district  Work with all relevant stakeholders including government departments, the private sector and non-governmental, traditional, and community-based organizations. The strategy recognizes that while South Africa has a generalized HIV epidemic, the prevalence and rate of HIV incidence and transmission is higher in some sub-populations. The strategy 7 Department of Health. Draft Health Sector HIV Prevention Strategy and Guideline 2013-2016. Mid-term Performance Evaluation of the SHIPP Project Evaluation Report Page | 52 defines a package of interventions for these populations as well as the government department which needs to take the lead in providing the interventions. Ehlanzeni District Municipality Strategic Plan for HIV & AIDS, STIs and TB (June 2013 – June 2016) SHIPP provided both technical and financial support for developing and finalizing the Ehlanzeni District’s Strategic Plan (DSP) for HIV & AIDS, STIs and TB (2013 to 2016) The Municipality’s strategic plan is aligned with the NSP 2012-2016 and the Mpumalanga Provincial Strategic Plan (2012-2016) and has the same four strategic goals. Under Strategic Objective 2 – Prevention of new HIV related, TB and Sexually Transmitted infections, the Ehlanzeni DSP outlines the following sub-objectives:  Sub-objective 2.1: Reduce new HIV, STI and TB infections  Sub-objective 2.2: Prevent vertical transmission of HIV to reduce Mother to Child Transmission to Less than 2% at 6 weeks and less than 5% at 18 months by 2016.  Sub-objective 3: Universal Screening and Testing for HIV, STIs and TB for all consultations The Municipality’s strategic plan recognizes the need to use combination prevention as the approach for preventing new HIV, TB and sexually transmitted infections. It states, “A combination of biomedical, behavioral and structural interventions will be implemented to strengthen inter-sectoral collaboration across the five municipalities” of Ehlanzeni District. The Municipal plan further identifies and describes indicators for measuring each sub-objective and proposes targets. Some indicators under the sub-objective covering sexual transmission of HIV include those that measure condom use by age and gender and sexual contact (e.g. first sex, non-marital, non-cohabiting partner, etc.), distribution of male and female condoms, people on ART, people counselled and tested for HIV by age and gender as well as prevalence rates among antenatal attendees, key populations as well as the population at large. Department of Basic Education Integrated Strategy on HIV, STIs and TB 2012-2016 The national DBE Integrated Strategy on HIV, STIs and TB 2012-2016 is the DBE’s response to the government’s call for action outlined in the NSP 2012-2016. The DBE strategy is based on seven “key imperatives” including “Education as a protective factor – the role of prevention,” and “Alignment with the NSP 2012-2016”. The strategy also identifies the following three strategic outcomes as follows:  Increased HIV, STIs and TB knowledge and skills among learners, educators and officials  Decrease in risky sexual behavior among learners, educators and officials; and  Decreased barriers to retention in school, in particular for vulnerable learners. Mid-term Performance Evaluation of the SHIPP Project Evaluation Report Page | 53 Figure 12: Impact and Outcomes for DBE's Integrated Strategy on HIV, STIs and TB 2012- 2016 The DBE strategy notes the lead role of the South African education sector in HIV prevention and in addressing the structural factors in the NSP 2012-2016. Furthermore, DBE acknowledges the direct role it plays in achieving the SOs and sub-objectives outlined in the NSP 2012-2016. In keeping with its imperative to learn “Lessons from available evidence on effective responses,” the DBE’s strategy discusses lessons learned on the importance of comprehensive and combination prevention. It stresses the importance of addressing social and structural drivers both HIV and TB in order to have an effective impact. SHIPP provided technical input and supported the design, layout and printing of the DBE strategic plan. Furthermore, SHIPP provided the DBE with technical support in developing peer education guidelines for school learners and educators which are meant to address some elements contained in the DBE strategic plan. More specifically, SHIPP provided technical support to Mpumalanga, Gauteng and KwaZulu-Natal in designing and facilitating peer education training for learners and educators. Improved quality of basic education Reduction in new HIV, STI and TB infections among learners, educators and officials Increased HIV, STI and TB knowledge and skills among learners, educators and officials Decreased risky, sexual behavior among learners, educators and officials Increased learner, educator and officals' retention within the education system supported Decreased barriers to retention in schools particularly for vulnerable learners Mid-term Performance Evaluation of the SHIPP Project Evaluation Report Page | 54 Gauteng Provincial Strategic Plan (GSP) 2012-2016 SHIPP provided technical and financial support for a consultative conference to finalize the Gauteng Provincial Strategic Plan (2012-2016) which outlines goals and strategic objectives that mirror those in the NSP 2012-2016. SHIPP facilitated a session on reducing HIV infections which resulted in a strong recommendation to adopt combination prevention as the provincial HIV prevention strategy. This was later adopted. The GSP’s strategic objective addressing HIV prevention states: “Reduce new HIV infections by at least 50% using combination prevention”. The plan also identifies key populations which are at higher risk of HIV infections. The GSP includes the following sub-objectives: 1. Everyone tests for HIV and screens for TB each year. 2. Provide sexual and reproductive health services as part of all primary health care services. “Safe sex” campaigns combine HIV prevention with contraception/ family planning. 3. Reduce transmission of HIV from mother to baby to eliminate HIV infections in babies by 2016. 4. Prevent TB infections and disease. 5. Provide health care and counselling for survivors of sexual assault. 6. Safe supply of blood for transfusion. 7. Prevention of sexual abuse with comprehensive care after sexual assault including medical care, counselling and access to justice. The support SHIPP provided in developing/revising policies and strategic plans was guided by the requests it received from SAG. Many SAG departments saw a need to align their response to HIV to the NSP 2012-2016 and SHIPP’s assistance was sought in addressing this, particularly in regards to HIV prevention. While providing support in the HIV prevention policy arena was relevant in the early days of the SHIPP project, there is a general sense that what is more useful now is for SHIPP to show how these policies are best implemented at local and district levels. 4.2.4 RECOMMENDATIONS  Shift focus towards demonstrating the implementation of optimal combination prevention packages for key populations at the local level  Contribute to the mid-term review of the NSP and related provincial and district strategic plans 4.3 What are the strengths and weaknesses of the program so far? SHIPP had several strengths and weaknesses as reported by key informants from the SHIPP Consortium, SAG, and SGRs as well as reported in project documents. These are discussed Mid-term Performance Evaluation of the SHIPP Project Evaluation Report Page | 55 below and summarized in Table 10. 4.3.1 STRENGTHS Promoting Combination Prevention in SAG policies and Strategies: SHIPP’s approach to promoting combination HIV prevention was a novel idea as it had never been implemented before in South Africa. Combination HIV prevention enabled South Africa to better respond to the drivers of the epidemic. SHIPP facilitated intergovernmental relationships which resulted in consensus around the ultimate goal – that is using combination prevention as the core approach to preventing HIV infections. Since combination HIV prevention is now incorporated into SAG policies and strategies, all stakeholders (i.e. USG, SAG, SANAC and AIDS Councils) are committed to a shared vision for change. Building M&E for HIV Prevention: SHIPP also assisted SAG departments like DBE, DSD and DPSA with incorporating HIV Prevention indicators into the M&E System. For SGRs, SHIPP trained all relevant staff in M&E and data management. SHIPP technical officers and M&E specialist visited SGR and updated them on data collection and M&E templates. The outcome of this support is that SGR respondents now report improved data management skills. Establishment of Coordination Mechanisms: SHIPP facilitated the formation of quarterly stakeholder meetings whereby different provincial and district SAG departments met for information sharing and a better understanding of combination prevention as a way to accelerate HIV prevention. In addition, SHIPP used other communication channels to coordinate activities and feedback to and from SAG stakeholders at provincial and district levels. At the community level, planning meetings at the level of working teams, emails, telephone and general open communication in between regular meetings served as coordination mechanisms. Combined with the presence of seconded staff, some SGR found these communication channels convenient and beneficial. In some districts, representatives from all relevant sectors attended meetings where they discussed their different activities. SHIPP made use of existing structures and systems that were working well. For example in Zululand, SHIPP worked through Operation Sukuma Sakhe (OSS) which was seen as a mutually reinforcing activity whereby multi-sectoral collaboration between government departments and internal stakeholders work together to discuss progress and share lessons on HIV prevention activities. Strengthening Political Leadership and Governance at national and district levels: Since SHIPP has bilateral engagements with SAG departments at national, provincial and district levels, the activities have all been distinct while contributing towards the goal of HIV Prevention. At community level, there are no significant programmatic or geographic overlaps amongst SGRs. Project Responsiveness, Flexibility and Leadership: SHIPP had good vision and leadership to advocate for such a complex program. Some senior staff were resilient in managing complex relationships. The project demonstrated flexibility and willingness to adapt strategies to Mid-term Performance Evaluation of the SHIPP Project Evaluation Report Page | 56 respond to emergent issues and the needs of SAG at various levels. For example, a rapid assessment of DACs and LACs conducted in late 2011/early 2012 found that 80% of DAC/LAC and civil society respondents did not understand the mandates, roles and responsibilities of DACs and LACs. In response, CADRE conducted capacity building workshops for these structures with a focus on M&E, data management, program coordination, program design, implementation and management. Whilst implementing this program there was a request that political leaders should also be capacitated on the HIV and AIDS program governance so as to allow them to exercise stewardship in their respective areas of leadership, which was then added to the training program. Some SHIPP head office staff, technical officers and consultants delivered TA of high quality, and some SAG respondents appreciated the secondment model for delivery of technical assistance, M&E, and combination prevention. The presence of seconded SHIPP staff ensured ready technical assistance and handholding where necessary on short, medium and long term basis. This was also seen as a quality assurance system and an effective way of ensuring objectives were met. One respondent referred to SHIPP as a “very valuable partner” who was always there to meet their needs. Some reported that seconded staff were deployed strategically and some DGs showed interest in absorbing them, reflecting favorably on the quality of SHIPP staff as well as project sustainability. District and Community level impact: The project’s impact was felt at community level through its work with SGRs. Many national and district level respondents reported strengthening of LACS in political leadership and governance as strength. Facilitating dialogue on combination prevention at the district and community levels was also a reported strength and Engender Health was seen as having fielded “very skilled people, very conversant with gender”. Through the Community Action Teams model, community members were empowered to discuss HIV prevention and their role and actions they need to take in order to mitigate the spread of HIV infection. SHIPP developed the capacity of CBOs in condom mapping and distribution. As a result, ensuring access to condoms is perceived by SGRs as a key activity in HIV prevention activities. SGRs also reported increased knowledge and skills in data management and project management and governance as a result of SHIPP’s support. Capacity building in the Education sector: The capacity building activities that SHIPP organized for the education sector were well executed. The development of scripted lesson plans, school dialogues and training of peer educator supervisors fostered a learning culture. The Community Capacity Enhancement Program (CCEP) methodology and Community Action Teams encourages shared learning of real experiences and challenges and lessons are learned all the time. There is an opportunity to respond better to real problems due to these methodologies. Mid-term Performance Evaluation of the SHIPP Project Evaluation Report Page | 57 Costing workshops: By the time of this evaluation, SHIPP had initiated support for the development of costing models for various prevention activities, e.g. the HPV vaccine, MMC, Treatment as Prevention (TasP), and the Prevention Costing Model. This helped some SAG departments like DSD to appreciate the importance of costing strategies and plans and orientate others in the department. 4.3.2 WEAKNESSES Developing a Shared Agenda: While SHIPP tried to develop a common agenda with its stakeholders, there were several challenges which delayed the start of implementation. At the beginning of the project, SHIPP engaged key SAG stakeholders in conceptualizing project activities and in selecting geographical areas. However, with high staff changeover, SHIPP did not put in mechanisms for continued engagement and updating of new SAG staff. This resulted in a prolonged post-award engagement process (approximately 24 months). The provinces did not communicate adequately to some stakeholders at the district level about the provincial consultations and these led to disgruntlements and undercurrents that negatively affected SHIPP’s relationships with some districts thereafter. Combination HIV prevention was a new idea which was not well understood by all key stakeholders, and the combination HIV prevention package was never clearly operationalized during the life of the project. On the geographical spread, some respondents felt that SHIPP “put in very high resources into small areas”, suggesting that it would have been more efficient for SHIPP to cover wider areas with the same amount of resources SHIPP’s workplans involved working with many SAG stakeholders at various levels. The Stakeholder Meetings were seen by some as the convening authority or steering committee but there were no clear terms of reference for the stakeholder meetings to coordinate and regulate SAG requests. Workplans with some departments were revised several times leading to some agreed upon plans, which had already been shared with districts and schools, being significantly delay or not being implemented at all. Some stakeholders did not fully understand what SHIPP “technical assistance” entailed and this may have resulted in too many requests from the SAG. Respondents had the following to say about the understanding of TA: Mid-term Performance Evaluation of the SHIPP Project Evaluation Report Page | 58 TA was not well articulated. For example when SHIPP facilitates 2-3 hour meeting people would request for catering ~ Consortium Respondent TA was not clearly understood ~ SAG Respondent If you are going to run a workshop one does not know whether SHIPP should cater or not for the workshops. I still need a workshop to tell us seconded staff the dos and don’ts ~ Consortium Respondent The design was not well understood ~ SAG Respondent Lack of clarification of boundaries - Lack of understanding of collaboration at onset ~ Consortium Respondent The district forgot we are there for TA- they would ask for money ~ Consortium Respondent Poor Communication: Communication between SHIPP and some of its SAG stakeholders, as well as within the SHIPP Consortium, was cited as an issue by respondents. Working groups did not hold regular meetings as planned, and the project encountered challenges in mobilizing multiple sectors for stakeholder meetings. Information on the program did not flow easily through all levels of collaboration. The program did not share progress reports with all relevant SAG and Consortium partners as per agreement. This left some SAG partners like COJ uncertain of SHIPP’s implementation and progress. Missed M&E opportunities: The opportunity to strengthen data management and usage across SAG departments and assist SANAC and AIDS Councils was not fully exploited. LEAP was not implemented due to a request from the DOH - at the conceptualization stage - to not use the LEAP process until it could be aligned to existing tools. SHIPP did not support the development of combination prevention indicators to enable the monitoring of behavior and socio-cultural changes. Some SHIPP-supported M&E activities and tools were not complete by March 2014. .For example, SHIPP did not support the integration of HIV prevention data across SHIPP-supported sectors. In addition SHIPP did not support Operational Research to measure behavior change in people reached in the areas SHIPP operated in. This would have helped to identify working models for scale up. . HR and consortium management issues: There was a high staff turnover of key personnel within the consortium. There were weak synergies within consortium partners and weak coordination and openness as they operated in silos. Some SAG partners were not aware of the roles and responsibilities of some consortium partners like CADRE and Engender Health. Furthermore, although qualified staff had been identified at the proposal stage, some consortium staff lacked the technical competence for the required TA once implementation started necessitating hiring of consultants. In the initial years, SHIPP leadership did not possess adequate expertise and skills in change management for navigating such a complex and multi-sectoral program. Mid-term Performance Evaluation of the SHIPP Project Evaluation Report Page | 59 Table 10: Summary of Strengths and Weaknesses Strengths Weaknesses Common Agenda  Combination HIV prevention - novel idea never before implemented in South Africa  Combination HIV prevention incorporated into SAG policies and strategies.  Flexibility and willingness to adapt strategies to respond to emergent issues and needs of SAG at all levels  Some SAG stakeholders were not fully conversant with the conceptualization of project activities and the process of selecting geographical areas  Optimal combination HIV prevention packages not demonstrated for key populations  Too many stakeholders and overly ambitious project targets Communication  Use of a variety of communication channels: planning meetings at the level of the working teams, emails, telephone and general open communication in between regular meetings  Regular coordination of activities and feedback to and from SAG partners through technical officers  Irregular stakeholder meetings at the national level  Insufficient communication within the SHIPP Consortium M&E System  Assistance with incorporating HIV Prevention indicators into SAG departments M&E System  Training of SGR relevant staff in M&E and data management  Improved data management skills of SGR respondents  Regular on-site M&E technical assistance  M&E missed opportunities: - strengthening data management and use of LEAP  HIV prevention indicators for monitoring behavior and socio-cultural changes were not developed. SHIPP Project Management  Good vision and leadership to advocate for such a complex program  Resilience in managing complex relationships  High quality head office staff, technical officers and consultants  Delivery of TA through secondment model￾handholding where necessary on short, medium and long term basis  High staff turnover within the consortium  Inadequate expertise in change management for navigating a complex and multi-sectoral program that required tactful engagement at multiple levels, and small grants management.  Insufficient in-house technical competence to provide on-going technical inputs and supervise/quality assure work done by STTA. Community level  Capacitating local AIDS councils in political leadership and governance  Facilitating dialogue on combination prevention at the district and community levels through CCEP and CATs model  Capacitating CBOs and SGRs in condom mapping  The opportunity to build the sustainability of CBOs was not explored Mid-term Performance Evaluation of the SHIPP Project Evaluation Report Page | 60 Strengths Weaknesses and distribution  Well executed capacity building activities for the education sector  Development of scripted lesson plans, school dialogues and training of peer educator supervisors fostering a learning culture.  Increased knowledge & skills in data management, project management & governance of SGR. 4.3.3 RECOMMENDATIONS Based on the strengths and weaknesses identified above, the following are the recommendations for the program: 1. Work with SAG to demonstrate combination HIV prevention packages for the different key social groups and develop the respective strategies for implementation of targeted interventions. 2. Work through a steering committee for coordination of many stakeholders and focus combination HIV prevention efforts at the community level where there is high potential for behavior change. 3. Design longer term funding cycles for CBOs, minimum 3 years for achievement and measurement of outcomes – particularly with regards to behavior change. 4. Develop M&E indicators and tools that enable tracking of combination HIV prevention behavior change. The tools need to correlate with the existing SAG M&E systems. 4.4 What key aspects of the SHIPP model should be continued in future HIV/AIDS programming? What should be discontinued? What should be scaled up? This section discusses what elements or activities of the SHIPP project should be continued or scaled up and which should be discontinued (Figure 13). 4.4.1 ACTIVITIES TO CONTINUE AND SCALE UP The activities that should be continued and scaled up include:  Condom mapping and distribution. SHIPP was instrumental in providing technical and financial support for developing the National Condom Distribution Plan and assisting with provincial training dissemination workshops and distributing the document. SHIPP facilitated condom mapping exercises in the districts of Ehlanzeni in MP and the sub-district of Pongola – a high transmission area (HTA) in Zululand. These led to the Mid-term Performance Evaluation of the SHIPP Project Evaluation Report Page | 61 establishment of secondary condom outlets, which match the increased demand for condoms created by community mobilization activities. In Zululand a condom distribution warehouse was rented in Ulundi to help prevent stock outs. Some specific achievements as a result of community mobilization include: o Condoms distributed - 3,109,256 o VMMC referrals - 1,270 o Persons reached by community awareness programs - 119,772 This capacity built through geospatial mapping for identifying HTA and condom mapping and distribution is strong and sustainable. It should be scaled up to enable expansion to other geographical areas not currently within SHIPP funding  Staff secondments providing that staff possess the technical expertise required and that their SOWs and reporting structures are clearly defined.  Strengthening AIDS Councils in other districts. SHIPP strengthened the capacity of DACs, LACs and WACs to identify, implement and coordinate optimal HIV prevention packages.  Costing of operational plans to include other departments. SHIPP supported the development of costing models for various prevention activities and this helped some SAG departments like DSD to appreciate the importance of costing strategies and plans and orientate others in the department. 4.4.2 DISCONTINUE SHIPP should focus its attention on working with communities rather than developing/refining policies at the national level. When dealing with many stakeholders around policy work, engagements were drawn out and often went beyond expected timelines with little outputs. There is higher potential for making impact through community dialogues as these impacted on people’s behaviors. Mid-term Performance Evaluation of the SHIPP Project Evaluation Report Page | 62 Figure 13: Recommendations on Key Aspects of SHIPP model 4.5 What systems have been established due to the technical assistance provided by SHIPP? Although SHIPP’s TA contributed to strengthening SAG systems at various levels, no new SAG systems were established per se. However, SHIPP supported the establishment of organizational and reporting systems for its SGRs. The emphasis in this section, therefore, is on systems strengthening rather than the establishment of new systems. The findings in this section are arranged according to the six World Health Organization (WHO) building blocks for health systems strengthening. A critical, but often forgotten seventh building block PEOPLE, has been added, recognizing the fact that communities must be aware of their right to health care and how they can contribute to their own wellbeing if we are to achieve optimal health outcomes. The SHIPP community systems strengthening activity, through the capacitation of small community based organizations (CBOs), was an attempt to address this need. Continue and/or scale up: Geospatial mapping for identifying HTAs Condom mapping & distribution to HTAs Costing of operational plans to include other SAG departments SGR model Strengthening AIDS councils in other districts Staff secondments Discontinue: Policy work Work with large number of stakeholders Strong focus at national – rather focus on HTAs Mid-term Performance Evaluation of the SHIPP Project Evaluation Report Page | 63 Figure 14: Seven building blocks for effective health care8 4.5.1 LEADERSHIP AND GOVERNANCE WITHIN GOVERNMENT STRUCTURES SHIPP provided support to leadership, oversight and governance structures at all levels within several government structures, including the OTDP which is charged with leadership and coordination of the SAG response to HIV; SANAC which has the responsibility for coordination and oversight of the response to the epidemic amongst government, private and civil society sectors; and the DPSA which represents all public sector government departments at all levels. The DPSA has begun engaging with sectors outside the health and social clusters in performing its lead role in mainstreaming HIV within SAG departments. SHIPP supported DPSA in its work with the Department of Transport collaborating with other partners including the International Labour Organization (ILO), International Organization of Migration (IOM) and the Southern African Development Community (SADC) Transport Forum to support development of an Integrated HIV Prevention and Mainstreaming Plan in the Maputo Transport Corridor. 8USAID. Health Systems 2020. Presentation by Ann Lion, DLI HSS Workshop, August 2011. Overview of Health Systems Strengthening. http: //www.healthsystems2020.org/content/resource/detail/79741/ Mid-term Performance Evaluation of the SHIPP Project Evaluation Report Page | 64 Policies and strategies As shown in Table 5 above, SHIPP contributed to several key national level policies and strategies mainly within the health and social clusters:  the National HIV Prevention Strategy which was completed in early 2014, but is awaiting ratification by the National Health Council (NHC) and sign-off by the Minister of Health;  National Condom Distribution Plan  updated HIV Counselling and Testing (HCT) Guidelines for the National Department of Health (NDOH),  the HIV and TB Prevention Policy for the Department of Basic Education (DBE) and  the Department of Social Development (DSD) HIV/AIDS Strategy that includes a gender equity framework. SHIPP also supported the development of district strategic plans, which include combination prevention strategies, in all three implementation districts. SANAC Technical Working Group on strategic information for HIV prevention Through its LTTA to SANAC, SHIPP led the establishment of a SANAC Technical Working Group (TWG) on strategic information for HIV prevention. The TWG identified priority interventions to strengthen HIV prevention in the targeted key populations. SHIPP led the technical inputs on the development of a national sex worker sector plan. The project also provided TA to draft a concept paper on girls and women. 4.5.2 STRENGTHENING HUMAN RESOURCES - SECONDMENT OF STAFF The project has recruited and seconded about 33 technical staff to support government departments at national and district levels. This worked well when the staff were assigned to undertake particular activities, but the long-term capacity building benefits were limited because of staff shortages within government departments, and the lack of SAG staff to whom skills could be transferred, particularly when the and seconded individual cannot (or is unwilling to) be absorbed into the SAG staff establishment. 4.5.3 STRENGTHENING SUPPLY CHAIN MANAGEMENT - CONDOM DISTRIBUTION PLANS AND TOOLS As discussed above, SHIPP was instrumental in providing technical and financial support for developing the National Condom Distribution Plan and in assisting with provincial training dissemination workshops and distribution of the document. These activities led to establishment of secondary condom outlets in Mpumalanga and KwaZulu-Natal. A condom distribution warehouse is being established in Ulundi to help prevent stock-outs. Mid-term Performance Evaluation of the SHIPP Project Evaluation Report Page | 65 4.5.4 STRATEGIC INFORMATION M&E capacity building Through the OTDP, SHIPP supported strategic leadership training across all government departments to ensure alignment and cross sectoral planning. Training covered a variety of topics related to M&E in HIV policy, planning and programming9 and was facilitated by staff from SADC, the Department of Performance Monitoring and Evaluation (DPME) in the Office of the Presidency, and the Gordon Institute of Business (GIBS) at the University of Pretoria. Synthesis paper In 2012, SHIPP released the National Synthesis Paper on Prevention of Sexual HIV transmission in South Africa, which is focused on the three SHIPP implementation provinces. The paper discussed key interventions being implemented and assists in identifying priority areas supported through SHIPP. The document was used, for example, to inform the HIV Turnaround Strategy for Gert Sibande District which SHIPP supported. KYE/KYR and LEAP SHIPP initially put forward a tool called the Local Epidemic Assessment for Prevention (LEAP) as an approach for collating all the available information from a locality to generate the appropriate combination of HIV prevention interventions. LEAP was not taken up by SAG as it required information sets that were not readily accessible at a local level for the modelling. SHIPP subsequently supported DPSA to hold two Local Epidemic Modelling seminars to help stakeholders at different levels (national, provincial and district) to respond to local needs and realities. The project is currently developing tools I (based on the Medical Research Council (MRC) work conducted in Buffalo City Municipality) to enable development of appropriate Know Your Epidemic (KYE), Know Your Response (KYR) and Know Your Local Capacity and Institutions responses. 4.5.5 PREVENTION SERVICE DELIVERY Training SHIPP provided training to equip community health workers (CHWs) to improve combination HIV prevention service delivery. For example, 80 peer educators in Johannesburg were trained on: HIV transmission, treatment and adherence counselling, key drivers of HIV/TB collaboration, PMTCT, HCT, PEP, MMC, sexual debut, multiple sexual partners, woman and children, multiple 9Training areas included definitions, benefits, linkages, contents, revision and development of M&E plans, M&E systems, National HIV M&E planning, M&E work plan, Results chain, Logic Model, Logical Framework, Indicators, Data quality dimensions, and Policy cycle. Mid-term Performance Evaluation of the SHIPP Project Evaluation Report Page | 66 partners, alcohol and drug abuse, transactional sex and gender inequality. This enabled frontline lay health workers in the city to engage community members in conversations on combination prevention and provide accurate information. Implementation materials development SHIPP supported the development of materials to help ensure effective implementation of HIV prevention strategies. Prime examples include the new Peer Education Guidelines and Scripted Lesson Plans for the DBE. Demand creation through community mobilization CHWs were trained to conduct household visits and provide health education and promote health seeking behaviors and HIV prevention activities such as condom use, as well as STI and cervical cancer screening 4.5.6 BUDGET PLANNING SHIPP supported the development of costing models for various prevention activities e.g. the HPV vaccine, MMC, Treatment as Prevention (TasP) and the Prevention Costing Model. SHIPP conducted costing capacity building workshops for SAG program managers and supported the development and costing of HIV prevention implementation plans in a number of departments including national DBE, as well at the district level (e.g. Ehlanzeni and Zululand). In addition the project provides continuous support for the review of the HIV/AIDS Conditional Grant, Business Plans and Operational Plans for all 9 provinces through the services of the Health Economist seconded to the NDOH. 4.5.7 COMMUNITY SYSTEMS Strengthening systems at community level SHIPP was in the process of strengthening local systems focused on HIV prevention, such as DACs, LACs and WACs, to strengthen their ability to assess the current effect of local HIV transmission drivers and monitor the responses of SAG agencies in meeting these needs. SHIPP supported with establishing and training these bodies in MP and KZN. The project also provides TA (e.g. condom mapping exercise) to identify areas for intervention and will then assist the DAC to monitor the impact of these interventions through integrated strengthened monitoring and evaluation systems. The M&E systems have yet to be fully developed. 4.5.8 RECOMMENDATIONS  Include coaching and mentoring as an integral part of capacity building (SAG would need to have staff in place to be coached and mentored)  Review mechanism for seconding staff – scope of work, reporting, sustainability of skills Mid-term Performance Evaluation of the SHIPP Project Evaluation Report Page | 67 they bring  Support ongoing functionality of local AIDS structures, including their ability to participate in planning and monitoring district strategic plans for HIV  Strengthen capacity to conduct local epidemic modelling and develop appropriate combination prevention approaches at sub-district level  Continue costing support and strengthen linkages between financial & program reporting 4.6 What are the examples of working partnerships established with SAG structures? A number of key themes emerged from our KIIs indicating critical success factors, unrelated to technical skills, which contributed to successful working relationships between SHIPP and SAG structures and counterparts. All too often, not enough attention is paid to these ‘soft’ skills in projects of a ‘hard’ or technical nature. The six most commonly cited ‘relationship builders’ are indicated in Box 1 below. We have defined working partnerships as relationships between SAG and SHIPP which yielded (or have the potential to yield) results AND also demonstrated some or all of the factors contributing to a good working relationship as listed below. In this section we describe illustrative examples of such relationships across the different levels and localities in which the project is being implemented. Box 1: Factors contributing to a good working relationship between SHIPP and SAG departments  Open communication  Mutual respect and collaboration  Sense that SHIPP was not prescriptive and added value  Commitment from both partners  Effective coordination from SAG counterpart  Continuity of key contacts or champions 4.6.1 DSD PARTNERSHIP The mutually appreciative relationship between SHIPP and DSD demonstrated all the soft issues presented in Box 1 above. The new NSP (2012-2016) defined a pivotal role for DSD in addressing the social determinants in the prevention of HIV. The combination prevention approach advocated by SHIPP includes behavioral, social as well as structural interventions and the project was perfectly positioned to partner with DSD as that department explored its expanded NSP mandate. Both SHIPP and DSD respondents commended the open nature of their bilateral communication. Mid-term Performance Evaluation of the SHIPP Project Evaluation Report Page | 68 4.6.2 PARTNERSHIPS WITH PEPFAR PARTNERS SHIPP created strategic partnerships with other PEPFAR partners working in the three provinces (e.g. BroadReach in Gert Sibande, Anova, and Right to Care) to coordinate prevention support in the districts. In the spirit of respect and mutual collaboration, SHIPP and these PEPFAR partners coordinated HIV prevention activities, so as to avoid duplication of effort (i.e. ‘being TA’d’ by more than PEPFAR partner) and reducing the ‘burden’ on government staff who are busy with routine activities. This enables the support to be provided at a pace that SAG counterparts have the capacity to absorb. This ‘development partner competition’ is particularly evident in the NHI pilot districts like Gert Sibande and clarification of partner roles was beneficial to both partner and beneficiary organizations. 4.6.3 PARTNERSHIP WITH OPERATION SUKUMA SAKHE Operation Sukuma Sakhe is a KZN initiative that links projects and activities being implemented by different government departments. SHIPP supported a benchmarking visit to KZN and assisted the Mpumalanga AIDS Council (MPAC) to adopt and adapt the OSS model. The MPAC team attended meetings with the KZN Premier’s Office to understand the successes of OSS including how data is gathered, collated and reported as well as visits to war Rooms to understand the importance of community based structures in an effective HIV response. The SHIPP team stationed in Zululand District Municipality (ZDM) works through the OSS structures and demonstrated commitment to identifying, disseminating and supporting adaptation of local good practice models while OSS showed openness and commitment to sharing lessons learned and MPAC exhibited commitment to learning and doing. 4.6.4 PARTNERSHIP WITH AGRIAIDS TO TARGET FARM WORKERS IN GERT SIBANDE DISTRICT Farm workers are prioritized as a key population in the NSP 2012-2016. An effective working relationship has been established with AgriAIDS to target and scale up prevention activities to farm workers in Gert Sibande district in Mpumalanga. SHIPP has added value in extending coverage to a key population group which is hard to reach through routine health services. 4.7 What aspects of small grant activities are successful to inform future prevention models? 4.7.1 FINDINGS The goal of the small grants program is to improve the quality, effectiveness, and coverage of sexual HIV prevention programs at community level. The call for expression of interest was issued in November 2011 and in Q4 2012 the first 5 grants to CBOs working in Zululand District Municipality were issued for a period of 12 months. Another 28 grants were issued in September 2013 for implementing activities to August 2014. A Mid-term Performance Evaluation of the SHIPP Project Evaluation Report Page | 69 total of 33 SGRs were supported. The maximum grant amount awarded was USD 250,000 as shown in Table 11. Table 11: Funding levels of SGRs per province Province Number of SGR awards <$100,000 Number of SGR awards $100,000 - $200,000 Number of SGR awards >$200,000 Total number of grants issued Gauteng 1 7 1 9 KwaZulu-Natal 1 9 10 Mpumalanga 2 8 4 14 TOTAL 33 The CBO projects varied in size, but focused on the following key objectives:  Increase uptake of HIV counselling and testing in the supported regions, with focus on informal settlements;  Promote and distribute condoms in hard to reach areas;  Reduce vulnerability to HIV and TB infection through social, structural and behavioral change;  Incorporate gender and youth transformative strategies to increase awareness and uptake of HIV prevention services; and  HIV/AIDS prevention education, targeting men’s sector on cultural norms and behavior. The main activities undertaken by CBOs with SHIPP funding involved:  Social mobilization through community dialogues  Formation of community action teams (CATs) to discuss various issues including gender based violence (GBV)  Peer education in schools  Referrals and linkages with Government structures  Condom distribution and demonstration of their use through secondary condom distribution sites and during household visits. The process for selecting and supporting SGRs is depicted in Figure 15 below. Mid-term Performance Evaluation of the SHIPP Project Evaluation Report Page | 70 Figure 15: Process for selecting and supporting SGRs Besides financial support, SGRs also received organizational development support from Avante Advisory Services, a service provider appointed by SHIPP, as well as ongoing in-house training and mentoring support from the SHIPP capacity building and strategic information units. Overall, the CBOs indicated that SHIPP’s financial and technical support helped them to expand their reach and improved community mobilization, M&E, and quality of prevention services (Figure 16). SAG respondents and SGRs themselves report that the most successful aspects of the SHIPP SGR model were:  Supporting local CBOs in target districts, particularly those operating in remote rural areas to improve access to and coverage of hard to reach populations in a culturally sensitive manner.  Strengthening linkages between CBOs and SAG departments through WAC and DAC structures. Through linking SGRs with primary health care (PHC) facilities, the clinics supplied commodities (such as condoms and IEC materials) and the CBOs became the department’s ‘’hands, eyes and ears in the community’’. Engage with districts to identify priorities SGR selection process: call for EOI, screen proposals, convene evalauation panels (with district DOH) Select CBO CBO capacity building: audit to identify gaps; tailored organisational development and technical support Ongoing monitoring and mentoring by district Technical Officers top up training by SHIPP capacity building unit Routine reporting to SHIPP and district municipality Mid-term Performance Evaluation of the SHIPP Project Evaluation Report Page | 71 Figure 16: SGRs’ views on the Effects of SHIPP Support  Strengthening community based services through activities such as social mobilization through community dialogues, peer education in schools, referral and linkages between communities and other HIV prevention service providers, and condom distribution.  Strengthening linkages and referral systems between organizations carrying out complementary activities and creating synergies between CBOs through the establishment of Civil Society Forums  SHIPP’s organizational and technical capacity building strengthened the CBOs position in terms of future funding applications. Khulanathi (a SGR) success stories (1) A mobile clinic for Babanango – taking health services to the people: Babanango is a deep rural village in a hard to reach area situated in a valley surrounded by steep hills in ZDM. Mobile health services were provided from the nearest clinic in Ulundi sub-district. The village is inaccessible by road from that side of the mountain because of the steep terrain so the clinic would park on the crest of the hill and the patients would walk up to it; even the elderly and disabled had to make the climb if they wanted medical attention. Khulanathi took this issue to nearest clinic and advocated that the matter be taken up to the health authority. It was decided that Babanango world be serviced by a mobile from the neighboring sub-district which can access the valley. “Because of our 0 2 4 6 8 10 12 14 16 Support increased SGR's understanding of combination HIV prevention approach SHIPP has increased HIV/AIDS prevention capacity in my organisation Increased HIV/AIDS prevention capacity Ability to deliver higher quality HIV prevention services Use of organisation's M&E data to improve services Improved community mobilisation Helped to expand the number of beneficiaries supported Number of respondents Strongly agree Agree Neutral Disagree Mid-term Performance Evaluation of the SHIPP Project Evaluation Report Page | 72 intervention now the mobile clinic can reach to the community rather than people coming to the clinic”. Success stories (2) Changing social norms/breaking the silence on GBV in Entsubeni: “When we went to that community we started with situation analysis and observed that rate of HIV may increase because there are social norms that perpetuate the spread. Women getting beaten by her partner becomes a social norm when they think that is life and they have nothing to do about it. We talked to the ward councilor & local traditional leadership. Gathered community into a meeting and after talking, talking, talking - one woman stood up & disclosed that she is the victim of GBV and eventually 5 stood up and disclosed. At 4th meeting spoke to women only on rights and what is acceptable, violence in a relationship is never acceptable….some think that if my boyfriend does not beat me he does not love me. In last meeting it was men & women….let us do this together. We need to change for the young ones to change we need to change first. So what we did in Entsubeni we changed the social norms of gender based violence because we understood that GBV can increase the spread of HIV”. 4.7.2 RECOMMENDATIONS  Increase duration of SGR funding to three years for the same amount as that provided in the current one year grant  Strengthen linkages between SGRs and SAG departments to ensure improve community based services Mid-term Performance Evaluation of the SHIPP Project Evaluation Report Page | 73 5 CONCLUSIONS AND RECOMMENDATIONS 5.1 Conclusions SHIPP, USAID’s flagship HIV prevention project in South Africa, is aimed at establishing combination HIV prevention at different levels of the South African Government (through TA in response to SAG requests) and at community level. The project’s start-up was considerably delayed for a number of reasons. SHIPP’s original design anticipated working with 3 government departments at different levels (DOH, DBE and SANAC), but then within the first 2 years, it expanded to include 4 additional departments (OTDP, DPSA, DSD, and DHET) – resulting in more complex management and implementation than initially envisioned. In addition, not all SAG departments fully understood the project’s concepts of ‘Technical Assistance’ or ‘Combination HIV Prevention,’ in part because SHIPP did not fully define them, and this contributed to lengthy negotiations around the scope of SHIPP support to SAG. SAG partners also did not capitalize on SHIPP’s TA in year 1 because they were unclear about their role in HIV prevention until the NSP was issued a year later, at which point there was an upsurge in requests to the project for assistance. If SHIPP could have coordinated these 7 SAG departments through a South African coordination body, it would have helped to prioritize and streamline its support. However, intra￾departmental government coordination is generally weak in South Africa at all levels, and the project ended up responding directly to 7 different departments, in 3 provinces, 4 districts and 13 sub districts – each of whom wanted direct engagement with the project. As some SAG departments are stronger than others, there was a perception that SHIPP did not support all departments equally and that some departments had more project privileges than others. Many respondents believe that the Project needed a steering committee to moderate and coordinate the project’s work, to ensure that certain departments didn’t get preferential treatment, and to ensure that all departments were working toward the same goal of reducing HIV incidence. It is widely noted that the project provided strong technical support and guidance around HIV prevention. But it is also evident that the project was not equipped enough to deal with the political dynamics and change management requirements for introducing new interventions into SAG structures. This is especially true in the project’s interactions with provinces – SHIPP’s focus on national and district levels meant that provinces were insufficiently engaged at the beginning of the project in decisions around project support. While the process was not without its difficulties, only one province had significant reservations about the districts selected. The ensuing negotiations resulted in significant delays, such that implementation at sub-national level didn’t truly pick-up until SHIPP’s third year. Part of this may be due to the absence of strong South African leadership within the project team to effectively navigate the political landscape. The initial SHIPP consortium included two established South African organizations who could have assisted in this regard, however, their Mid-term Performance Evaluation of the SHIPP Project Evaluation Report Page | 74 roles did not include liaising directly with the South African Government Departments. SHIPP’s ‘responsiveness’ to SAG requests was viewed as a strength by most respondents. However, it was also a weakness, as it meant the project spread itself very thin, and didn’t always achieve the depth of change desired. In addition, SHIPP sometimes delayed or failed to honor requests, and this was seen by some as the project making false promises The project’s work at district and sub-district levels in years 3 and 4 are regarded as significant contributions to HIV prevention, specifically the geospatial and condom mapping interventions, but the delay in getting started meant that these interventions were only rolled out to some project areas. In terms of its achievements and results, SHIPP contributed to changes in leadership, planning, and management at national, provincial, and district levels through facilitating the inclusion of combination HIV prevention in various policies and strategic plans. Staff secondments and short-term TA were instrumental in building capacity in these areas, as well as in alleviating staff shortages at SAG structures. Leadership capacity building at the district level (capacity building of DACs and LACs in political leadership/governance) and the support for costing prevention plans are seen as the project’s greatest achievements in this area. SHIPP’s work led to somewhat better coordination between SAG departments, although much remains to be done to remove barriers to achieving robust cross-sectoral leadership for HIV prevention. M&E was a shortcoming for the project. While SHIPP strengthened indicators for prevention activities and outputs (e.g. number of community dialogues, etc.), there were no indicators developed to track the outcomes or effects of these services (e.g. behavior change, coverage of prevention services delivered). Moreover, the revised PMP lacked targets for many output level indicators, making it very difficult for the evaluation team to determine if the project had achieved its overall objectives. The project never tested the concept of a “minimum or optimal package” of HIV prevention services directed at specific populations. Although SHIPP helped the NDOH to define this in the national Prevention Strategy, it never took the next step to pilot it at local level. In part this is due to the fact that the Prevention Policy is not yet formally approved by the NDOH, but respondents indicated that this should not have precluded the project from proceeding with piloting/demonstrating this concept at local levels. SHIPP’s work at community level was significantly delayed until year 3 because of the need to involve DACs and LACs in CBO selection. Many SGRs had limited time to implement their programs supported by the project, although they did receive organizational capacity building (OD, M&E, and financial management) from SHIPP. SAG and SGR respondents alike report the most successful aspect of the SGR model to be (i) strengthened linkages between CBOs and SAG departments through WAC and DAC structures Mid-term Performance Evaluation of the SHIPP Project Evaluation Report Page | 75 and (ii) strengthened community services through community dialogues for social mobilization, peer education in schools, condom distribution, and referrals and linkages between communities and other HIV prevention service providers. While SHIPP did help with improving the quality of their work (e.g. community dialogues, or peer education) as well as referrals between CBOs and health facilities, there was little effort directed toward ensuring that CBOs offered a comprehensive or minimum package of prevention services to their target beneficiaries. 5.2 Issues, challenges, and accomplishments 5.2.1 ISSUES  Lack of clear definitions and a common understanding of SHIPP’s mandate- Combination HIV prevention was a new idea which was not well conceived by all key stakeholders and a combination HIV prevention package was not defined. There was lack of clarity of what technical assistance entailed and the specific roles of seconded staff. This resulted in unrealistic expectations of the project and SAG ‘asks’ that were outside the project’s scope.  Missed opportunities to strengthen M&E systems - The opportunity to strengthen data management across all levels and to assist SANAC and AIDS Councils were not fully realized. LEAP was not implemented as planned as the model was not well accepted by SAG as it was perceived as complex and required data that were not readily available at local level. Definition of combination prevention indicators that would have enabled monitoring of behavior and socio-cultural changes did not occur. SHIPP supported M&E tools were not complete by the cut-off date of this evaluation.  HR and consortium management issues– High staff turnover within the consortium resulted in lack of continuous engagement and some senior staff were perceived not to possess adequate skills in change management to steer such a complex and multi￾sectoral program and navigate engagement at multiple levels tactfully. The consortium partners operated in silos and opportunities to coordinate and create synergies were not exploited fully. Some SAG partners were not aware of the roles and responsibilities of CADRE and Engender Health. Gender integration was delayed.  ‘Shortened’ implementation phase– Protracted engagement and slow start up of activities has artificially shortened the implementation phase and SHIPP’s impact on behavior change cannot be ascertained at this stage. SGRs acknowledge that SHIPP’s organizational and technical capacity building support has strengthened these organizations and places them in a better position in terms of future funding applications. Mid-term Performance Evaluation of the SHIPP Project Evaluation Report Page | 76 5.2.2 CHALLENGES  Slow/delayed policy implementation - While progress was made in developing policies and operational plans, there was little progress in translating these policies into action. This, however, lies outside of the remit of SHIPP, which was to provide technical assistance; the responsibility for policy and program implementation lies with the SAG. Some policies developed with SHIPP support had not yet been adopted by respective authorities by 31 March 2014.  Prolonged period of engagement with some departments before initiation of project support - In some instances, this was due to competing priorities within SAG but a contributing factor was that SHIPP relied on SAG to make requests for assistance that were within SHIPP’s mandate. In some cases, this lead to protracted negotiations regarding the nature of support or the decline of some requests creating a sense of ‘unfulfilled expectations’.  Uneven coverage of intervention districts - SHIPP’s activities to strengthen HIV prevention delivery systems at sub-national level did not cover all sub districts in the intervention districts. For example, the condom mapping exercise was conducted district-wide in Gert Sibande, but only in one sub-district in Ehlanzeni (Nkomazi) and a sub-district (Pongola) in Zululand. Lack of coverage in some areas of operation maybe attributed to the short period of implementation and inadequate SHIPP staff numbers. SHIPP’s reliance on SAG requests may have also contributed to slow uptake in some sectors and geographic areas. 5.2.3 ACCOMPLISHMENTS  Building leadership capacity - SHIPP’s approach to building leadership capacity is through seconded staff and STTA. This alleviated staff shortages within SAG structures and enabled departments to access technical assistance for HIV prevention. Seconded staff from diverse technical backgrounds created cross-functional Leadership Capacity Teams at their level of operations at national, provincial and district levels. SHIPP supported activities also contributed to strengthening national level leadership for purposes of planning, coordinating, implementing and evaluating HIV prevention programs.  Supporting the prevention policy environment - SHIPP support included the development or revision of a number of national, provincial and district HIV strategic plans and/or prevention policies to ensure that these were aligned with the NSP.  Strengthening provision of HIV prevention services in DOH and DBE - Priority HIV prevention services for NDOH are mostly biomedical interventions that include provision of condoms, HCT, VMMC, Post-Exposure Prophylaxis, ART and STI screening, TB screening, and care. These services are usually integrated at service delivery level to Mid-term Performance Evaluation of the SHIPP Project Evaluation Report Page | 77 optimize their uptake. Priority HIV prevention delivery systems for DBE are targeted at learners and educators. Services provided include capacity-building for educators and school-management teams; and the development of teaching and learning materials.  Enhancing community capacity - CBOs utilized a variety of mechanisms (e.g. door to door campaigns, workshops, and dialogues) to mobilize communities, increase awareness and create demand for HIV prevention services. SHIPP built the capacity of CBOs on the Community Action Team (CAT) approach to facilitate broader community involvement and participation in taking action. The CATs model builds on community dialogues towards formulating and undertaking actions to prevent and stop GBV and HIV. SHIPP also engaged with religious leaders and strengthened the capacity of traditional practitioners and leaders on the health of men and safety of VMMC.  Mapping ‘hotspots’ – To improve quality of HIV programs SHIPP carried out a number of assessments. These included geo-spatial mapping to identify hot spots and services as well as sexually transmitted infections (STI) and condom distribution assessments in some SHIPP supported areas. SHIPP collaborated with district DOH coordinators to develop a high transmission area (HTA) assessment tool.  Collaborating with partners– SHIPP co-facilitated a workshop with ANOVA Health Institute, and UNAIDS to clarify the roles of all partners in Mpumalanga province and established working partnerships with AgriAIDS to scale up HIV prevention among farm worker populations in Gert Sibande district and with BroadReach Health Care to coordinate prevention activities in the district. 5.3 Lessons learned The very nature of SHIPP – the mandate to promote a common agenda, among multiple stakeholders and multiple levels – meant that SHIPP frequently encountered issues and challenges to be dealt with or taken into consideration in decision-making. Some key lessons learned include the following:  Need for wide consultation and buy-in when designing and implementing a project whose success depends on the effective collaboration and sense of ownership of multiple stakeholders.  Importance of open, regular, and mutually respectful communication between stakeholders  Alignment of expectations on the part of all stakeholders  Clear roles and responsibilities among the various stakeholders  Fewer and more targeted stakeholders  An effective coordination mechanism that all stakeholders can be held accountable to Mid-term Performance Evaluation of the SHIPP Project Evaluation Report Page | 78  Commitment to the common goal from the highest levels within each department  Clear articulation of the support needed from a project like SHIPP  Importance of allowing for a long engagement period – especially if the groundwork has not been finalized before project start-up Mid-term Performance Evaluation of the SHIPP Project Evaluation Report Page | 79 5.4 Future direction for the program The evaluation team’s recommendations are presented in the Table 12 below and are organized by the respective stakeholder they apply to. Table 12 - Recommendations SAG SHIPP (Year 5) USAID Combination HIV prevention package Scale up alternative LEAP process such as the DPSA’s Local Assessment Model based on KYE/KYR to inform optimal HIV combination packages at the local level In collaboration with SAG operationalize combination prevention packages for the SHIPP-supported districts based on an understanding of the local epidemic To support the demonstration of optimal combination HIV prevention packages in geographical areas where SHIPP/SAG has generated local epidemic information based on alternative LEAP process Build on the capacity developed for DACs, LACs and WACs to enable them to identify, implement, monitor, and coordinate optimal HIV prevention packages Appoint sufficient staff to manage HIV prevention structures; provide appropriate levels of resources Strengthening AIDS Councils Ensure the presence and functionality of AIDS Councils especially at ward levels Replicate SHIPP’s work with AIDS Councils to other districts e.g. development of strategic plans, establishment of local and ward AIDS Councils, and capacity building Coach and mentor AIDS Councils which have been trained by SHIPP to consolidate the capacity building Continue providing technical support to strengthen AIDS Councils especially at district, local and ward levels Ensure political support and funding to Mid-term Performance Evaluation of the SHIPP Project Evaluation Report Page | 80 SAG SHIPP (Year 5) USAID AIDS Councils at the district, local and ward levels Mapping Scale up geospatial and condom mapping and implement combination approaches based on the problems identified from the mapping. Ensure skills transfer through documenting the geospatial and condom mapping and distribution process and sharing the lessons learned with SAG Support further scale up of condom mapping and distribution process Support further scale up of geospatial mapping Costing of Plans Strengthen the capacity of relevant SAG departments to cost HIV prevention plans Continue costing support and strengthen linkages between financial and program reporting Continue to support technical assistance for developing SAG’s capacity for developing costed implementation plans Ensure skills transfer through documenting the costing process and sharing lessons learned M&E Ensure M&E indicators and tools track coverage of combination HIV prevention interventions by measuring the number of people reached (numerator) and the number of potential beneficiaries (denominator). The tools need to correlate with the existing SAG and PEPFAR M&E systems. Continue providing TA to develop and refine M&E indicators and tools that track combined HIV prevention coverage and behavior change Ensure M&E indicators and tools that track coverage of combination HIV prevention interventions and behavior change are incorporated in all combination HIV prevention programs Conduct Operational Research to measure behavior change in people reached in the areas where SHIPP is operating Integrate HIV prevention data from key sectors to enable better coordination Provide TA to support the integration of HIV prevention data across SHIPP- Mid-term Performance Evaluation of the SHIPP Project Evaluation Report Page | 81 SAG SHIPP (Year 5) USAID supported sectors Improve access to key sectors’ data systems (e.g. DHIS, EMIS, CBIMS) to enable data use for decision making across all levels i.e. national, provincial, district, sub-district Continue providing TA to improve data management and data quality in all sectors Conduct external data quality audit for SHIPP, as best practice, to ensure the data generated by the project meets required standards and to identify areas for improvement Project Management Document lessons learned from managing SHIPP which is complex and works at different sectors, levels, and localities with complicated dynamics Include change management components into complex and novel projects to facilitate implementation and increase chances of success Small Grant Recipients (SGRs) Involve the SHIPP-supported SGRs in the implementation of the optimal HIV prevention package Document lessons learned in working with SGRs in improving the quality, effectiveness and coverage of sexual HIV interventions at community level Stakeholders Encourage virtual attendance at stakeholder meetings Ensure that future-funded projects with multiple stakeholders have an effective and representative project-specific steering committee with clear terms of reference to provide guidance and coordination Develop criteria and clear process for evaluating requests for TA Share documented lessons learned Technical Assistance Clearly define and communicate what is meant by TA to all stakeholders. Develop a “menu” of TA services Annex 1: Evaluation Scope of Work Annex 1. Scope of Work.pdf Annex 2: Detailed Methodology Annex 2. Detailed Methodology.pdf Annex 3: Sources of Information Annex 3. Sources of Information.pdf Annex 4: Signed Statements Attesting To Lack of Conflict of Interest Annex 4a. Edna Berhane_COI.pdf Annex 4b. Peter Njaramba_COI.pdf Annex 4c. Cheryl Goldstone_COI.pdf Annex 4d. Shalote Chipamaunga_COI.pdf Annex 4e. Mary Pat Selvaggio_COI.pdf U.S. Agency for International Development 100 Totius Street P.O. Box 43 Groenkloof, 0027 Pretoria, Gauteng South Africa Mid-term Performance Evaluation of the SHIPP Project Evaluation Report Page | 87