May 2013 This publication was produced at the request of the United States Agency for International Development. It was prepared independently by University Research South Africa (URSA). John Hopkins University (JHU) HIV Communication Programme Project Performance Evaluation Acknowledgement The University Research South Africa Evaluation team would like to acknowledge the support and guidance of the following: The United States Agency for International Development The Johns Hopkins Health and Education South Africa JHU HIV Communication Programme Partners and beneficiaries URSA evaluation team members JHU HIV COMMUNICATION PROGRAMME PROJECT PERFORMANCE EVALUATION BEHAVIOR CHANGECOMMUNICATIONINITIATIVE IN SOUTH AFRICA May 2, 2013 RFTOP-674-13-000003 DISCLAIMER The author’s views expressed in this publication do not necessarily reflect the views of the United States Agency for International Development or the United States Government. CONTENTS Executive Summary ........................................................................................................................................................... 7 Evaluation Purpose & Evaluation Questions.............................................................................................................. 12 Introduction and Background ....................................................................................................................................... 14 Evaluation Methods & Limitations ............................................................................................................................... 17 Findings, Conclusions & Recommendations .............................................................................................................. 22 Annexes - Annex I: Evaluation Statement of Work - Annex II: Evaluation Methods and Limitations - Annex III: Data Collection Instruments o INTERVIEW GUIDE FOR USAID/SA ACTIVITY MANAGER o INTERVIEW GUIDE FOR SUB-GRANTEE: CONTENT DEVELOPMENT o INTERVIEW GUIDE: NDOH/DBE/DSD/SANAC/PRIVATE SECTOR PARTNERS o INTERVIEW GUIDE: JHHESA NATIONAL OFFICE – CEO o FOCUS GROUP DISCUSSION GUIDE: MASS MEDIA CAMPAIGN o FOCUS GROUP DISCUSSION GUIDE: COMMUNITY INTERVENTIONS - Annex IV: Sources of Information - Annex V: List of Sites Visited - Annex VI: List of Sites Excluded - Annex VII: Disclosure of Any Conflicts of Interest - Annex VIII: JHU HIV Communication Programme Strategic Partners 5 ACRONYMS AB Abstinence, Be Faithful ACSM AIDS ANC Advocacy, Communication, Social Mobilization Acquired Immunodeficiency Syndrome Antenatal Care ARV ART Anti-Retroviral Anti-Retroviral Therapy BCC CADRE C&S C&T Behavior Change Communication Centre for AIDS Development, Research and Evaluation Care and Support Counseling and Testing CBO Community Based Organization CD4 CEO Cluster of Differentiation 4 Chief Executive Officer CHAPS CHMT Centre for HIV and AIDS Prevention Studies Community Health Media Trust CMMB COP Catholic Medical Mission Board Country Operational Plan DBE Department of Basic Education DCS DOE Department of Correctional Services Department of Education DOH Department of Health DSD Department of Social Development FBO Faith Based Organization FGD GBV GDRT HDA Focus Group Discussions Gender Based Violence Gauteng Department of Roads and Transport Health and Development Africa HCT HIV Counseling and Testing HIV Human Immunodeficiency Virus IEC Information, Education and Communication JHHESA Johns Hopkins Health and Education South Africa JHU-CCP KZN Johns Hopkins University Center for Communication Programs Kwa-Zulu Natal M&E MCP Monitoring and Evaluation Multiple Concurrent Partners MMC MOU MSM Medical Male Circumcision Memorandum of Understanding Men who have sex with men NDOH NCS National Department of Health National Communication Survey NGO NRASD NSP Non-Governmental Organization National Religious Association for Social Development National Strategic Plan OVC Orphans and Vulnerable Children PEPFAR United States President’s Emergency Plan For AIDS Relief 6 PHC Primary Health Care PLWH People Living with HIV PMTCT PPP Prevention of Mother to Child Transmission Public Private Partnership PwP Prevention with Positives SABC South African Broadcasting Corporation SAG South African Government SANAC SANCA SAPS South African National AIDS Council South African National Council on Alcoholism and Drug Dependence South African Police Services SEM SMS Social Ecology Model Short Message Service STI TA TAMS Sexually Transmitted Infections Technical Assistance Television Audience Measurement System TB Tuberculosis TLPP TOCOR Treatment Literacy and Prevention Practitioners Task Order Contracting Officer’s Representative URSA University Research South Africa USAID/SA United States Agency for International Development VCT VMMC XDR-TB Voluntary Counseling and Testing Voluntary Male Medical Circumcision Extremely Drug Resistant Tuberculosis 7 EXECUTIVE SUMMARY With the life of the current United States Agency for International Development Southern Africa (USAID/SA) implementing mechanism for the JHU HIV Communication Program scheduled to come to an end in December 2013, University Research South Africa (URSA) was contracted by USAID/SA to conduct an independent end of project evaluation of the JHU HIV Communication Programme project. Johns Hopkins Health and Education South Africa (JHHESA) is the South Africa NGO affiliate of Johns Hopkins University Center for Communication Programs (JHU-CCP) and was awarded a five-year cooperative agreement by USAID/SA (2008-2013) with a total estimated cost of $80,000,000 to support the USAID/SA’s Health Strategic Objective: “Increased use of HIV/AIDS and other Primary Health Care services.” JHU HIV Communication Programme works with the South African government and civil society partners to undertake strategic communication that combines mass media and interpersonal approaches to improve the health and wellbeing of all South Africans. In the five year period JHU HIV Communication Programme coordinated the work of more than 30 South African partners that use communication interventions to reduce the number of new HIV infections, promote HIV counseling and testing (HCT) and emphasize the importance of treatment adherence. Its reach is nationwide in all provinces, but efforts are concentrated in high transmission areas in Kwa-Zulu Natal (KZN), Gauteng, Mpumalanga, and Free State provinces. PURPOSE OF EVALUATION: The purpose of the evaluation was to: • Learn to what extent JHU HIV Communication Programme’s goals and objectives have been achieved; and • Inform the design of a new community-driven Human Immunodeficiency Virus (HIV) prevention project in the country. The findings are expected to: 1) describe the performance of the JHU HIV Communication Programme project and its contribution to the USAID/SA Health Strategic Objective “Strengthened Capacity to Deliver Sustainable and Integrated Primary Health Care and HIV and AIDS Services;” and 2) provide concrete recommendations for the Mission’s new community-driven HIV prevention project by outlining critical components to be included in the design, serve as the basis for a concept paper for the new design, and form the basis of the project description to be developed for any follow-on project. The evaluation focused on JHU HIV Communication Programme’s most strategic communication interventions and uses a combination of strategies, including: 1) Mass media – TV and radio spots; TV dramas; posters, pamphlets; music videos, etc. 2) Community mobilisation; and 3) Interpersonal communication and counseling. 8 USAID/Southern Africa: JHU HIV Communication Programme Performance Evaluation METHODOLOGY: The evaluation utilized a combination of qualitative and quantitative evaluation methods, including: document review of program activities; review and analysis of program monitoring and evaluation data; assessment of sub grantees’ performance on key outcomes; and assessments conducted with target populations, beneficiaries, policy makers, administrators, and stakeholders. SELECTED KEY FINDINGS: Following is a summary of the most significant findings of the evaluation. Additional findings are presented in the body of the report from page 22. JHU HIV Communication Programme implemented a highly successful and effective health communication program. Findings from the desk review and interviews with stakeholders and beneficiaries revealed that the JHU HIV Communication Programme developed technically feasible interventions which were largely politically and culturally acceptable and allow for future replication. With its socio-ecological approach and utilization of the Pathways to a Health Competent Society conceptual framework, JHU HIV Communication Programme utilized numerous interventions that address various factors affecting an individual’s behavior change. The major components of JHU HIV Communication Programme’s program addressed most of the key drivers of the epidemic such as multiple concurrent partnerships, low condom use and low medical male circumcision and alcohol and substance abuse. A key gap that was identified as requiring more innovative interventions is alcohol and drug abuse and many target audience informant did not report any change in their behaviour particularly in relation to their alcohol use. JHU HIV Communication Programme and its partners have implemented health communications campaigns that have reached its intended audience. The use of mass media resulted in innovative, high quality productions that have stimulated dialogue and created awareness among its audiences. In relation to the appropriateness of strategies to change social and gender norms and individual behaviours there is a consistent message that served a real need. JHU HIV Communication Programme reached substantial number of its target audience through mass media and community mobilization activities with the latter being more interactive with its target audiences. However, the interventions fell short in addressing some of the prevalent harmful cultural practices eg forced marriage that have bearing on the HIV epidemic. Minority race groups viz Whites, Indians and Coloureds are not being specifically targeted or reached by the communication activities. HIV prevention messages are complementary and synergy is achieved within the campaign as well as with other South African Government (SAG) and key stakeholder campaigns. As a result of JHU HIV Communication Programme leadership and involvement in the conceptualization, development and dissemination of the activities, synergy of the JHU HIV Communication Programme’s campaigns is maintained by developing tools that are utilized for mass media, community outreach and interpersonal communication. JHU HIV Communication Programme developed successful health communications to target gender norms among men and women. Brothers for Life is an innovative campaign that uses multimedia platforms that demonstrated successes among men with extended benefits for their women partners. However, the communication targeting men did not address the needs of men who have sex with men (MSM) or sexual violence targeting lesbian women i.e. corrective rape, an issue that is being increasingly reported in South Africa. 9 USAID/Southern Africa: JHU HIV Communication Programme Performance Evaluation As a key partner JHU HIV Communication Programme contributed to key policies at national and provincial levels and significantly built up the capacity of other United States Government (USG) implementing partners, local organizations and private sector partners but no clear pathway to sustainability. Regarding engagement with diverse stakeholders in South Africa, JHU HIV Communication Programme played a key role in advocacy and supporting the Department of Health (DOH) and South African National AIDS Council (SANAC) in health communications campaigns although relationships with Department of Education (DOE) and Department of Social Development (DSD) are not well established due to lack of availability by the said Departments. The absence of a clear pathway to a measurable level of sustainability at current levels of activity by DOH was articulated by key stakeholders. Through capacity building activities, JHU HIV Communication Programme built mechanisms within organisations to support sustainability although sub-grantees indicated that activities would not continue at the same scale without JHU HIV Communication Programme’s support. Strategic partnerships with local organizations and private sector partners have allowed JHU HIV Communication Programme to leverage non-USAID/SA/PEPFAR resources to meet its objectives. JHU HIV Communication Programme promotes linkages in its mass media campaigns and community mobilization activities. Linkages are primarily promoted by JHU HIV Communication Programme’s community mobilization partners through outreach activities as well as service provision for HCT, Tuberculosis (TB) screening and Sexually Transmitted Infections (STI) treatment as well the mass media campaigns. While the demand for services was created, efforts were hampered by the absence of clear referral mechanisms to link individuals to health services was apparent resulting in missed opportunities of appropriate management. The effect of overall structure and management of JHU HIV Communication Programme on performance the approach had a largely favourable effect on performance. The project established regular coordination meeting with sub-grantees and its public and private partners at all levels for information sharing. JHU HIV Communication Programme built institutional capacity at tertiary institutions for by supporting post-graduate education which yielded important publications. While the linkage with Johns Hopkins University –Center for Communications Programs has been beneficial delays in disbursements of funds to JHU HIV Communication Programme has occasionally affected programs adversely. CONCLUSIONS The findings from this evaluation demonstrate that the JHU HIV Communication Programme met the needs of stakeholders in relation to the reach of target audiences, appropriateness of messages and strategies used to address the key drivers of the HIV epidemic. The project’s overall approach and activities implemented were effective to meet the project’s objectives as stipulated in the Cooperative Agreement. The project contributed significantly to the priorities of the SAG as espoused in the National Strategic Plan (NSP) for STIs, HIV, and TB for 2012-2016. Unmet needs included; referral systems were not formalized and feedback mechanisms between services by JHU HIV Communication Programme’s partners and public and private health facilities to which patients were being referred were not clear. There were challenges in services provided by sub-grantees including low pretest counseling, low TB screening rates among newly diagnosed HIV positive and low HIV testing rates among TB patients. Low uptake of Medical Male Circumcision (MMC) among men also needs to be addressed as well as identifying better approaches to address cultural practices such as traditional medicine, ukuthwala (forced marriage) and polygamy in the context of HIV prevention. 10 USAID/Southern Africa: JHU HIV Communication Programme Performance Evaluation RECOMMENDATIONS • Improve linkages to health services by including a documented referral system. Future interventions should ensure clients are linked to support, care and treatment services beyond HCT. Scaling up the Helping Hands model as a best practice would support this. Other activities to improve linkages include; identifying barriers to MMC at individual, community and societal levels, developing health communication strategies targeting negative/judgmental attitudes of health care providers and strengthening in-facility health communication. Linkages between health communication activities in the communities and those in the health facilities should strengthened to promote synchrony and continuity in communication • Scale up campaigns that address gender an identify barriers to MMC, address needs of Men Who Have Sex with Men and address sexual violence. JHU HIV Communication Programme should continue to expand Brothers for Life particularly among older men. The campaign’s model can be used to also address the needs of Men Who Have Sex with Men as well as the challenges pertaining to rape and sexual violence. The root causes of rape and sexual violence as well as barriers to MMC at service delivery levels should be identified, explored and addressed in future health communication interventions. • A clear sustainability plan that focus on building capacity and not creating dependence among SAG departments for future programs. Investments in capacity building, particularly of the SAG structures and departments tasked with health communications, should be initiated and implemented by JHU HIV Communication Programme, in future, to promote sustainability. Activities should aim to build a critical mass of people within the SAG with the capacity to manage and implement health communications programs at scale. In subsequent projects the scope of partners should be increased with more participation of the private sector e.g. in the workplace. This will assist the JHU HIV Communication Programme to accomplish HIV/AIDS prevention, care, and treatment goals and help ensure sustainability of programs, facilitate scale-up of interventions, and leverage private-sector cash and in-kind resources. • Cultural and racial context: Future programs should address prevalent cultural practices and target all race groups. The JHU HIV Communication Programme should integrate more messages into their programming that takes into account the role of prevalent cultural practices, typically of rural communities e.g. traditional medicine, polygamy and ukuthwala.. Programming should also address the needs of other minority race groups and identify mechanisms to increase their participation in health services. • Increase access to appropriate mass media channels. Judicious use of resources would dictate that placement of mass media campaigns in future follow on programs should prioritise channels that have the widest audience. More focus should be placed on TV and radio spots rather than TV dramas. While a multipronged, multi-channeled approach is important, future program designs should be cognisant of the limitations of social media for rural audiences and should be tailored channels that are more accessible for rural audiences. • Future programs should increase interventions to address alcohol and substance abuse as a key driver of the HIV epidemic. More interventions are required to target alcohol and the increase substance abuse in the context of HIV prevention. Health communication programs particularly social mobilization activities should scale up interventions 11 USAID/Southern Africa: JHU HIV Communication Programme Performance Evaluation and identify linkages with alcohol and drug dependence treatment organizations to facilitate referrals. • Program Logic: The future project should articulate specific timelines in its objectives and quantify the expected measurable changes of each objective, determine realistic, achievable targets for each of its partners. Unintended negative effects of the interventions should be routinely monitored. • Address the structural drivers of HIV. Future programs need to more comprehensively address community and societal factors that interface with the individual to increase their risks of HIV acquisition and transmission. Integrating HIV prevention activities into the microenterprise approach as a public health strategy to target women in particular that is responsive to the broader socioeconomic and structural context should be considered. 12 USAID/Southern Africa: JHU HIV Communication Programme Performance Evaluation EVALUATION PURPOSE & EVALUATION QUESTIONS EVALUATION PURPOSE With the life of the current USAID/SA implementing mechanism for the Johns Hopkins Health and Education South Africa JHU HIV Communication Programme scheduled to come to an end in December 2013, University Research South Africa (URSA) was contracted by USAID/SA to conduct an independent end of project evaluation of the JHU HIV Communication Programme project. The purpose of the evaluation was to: • Learn to what extent JHU HIV Communication Programme’s HIV communication interventions in South Africa goals and objectives have been achieved; and • Inform the design of a new community-driven HIV prevention project in the country. The findings are expected to: i) describe the performance of the JHU HIV Communication Programme project and its contribution to the USAID/SA/South Africa Health Strategic Objective “Strengthened Capacity to Deliver Sustainable and Integrated Primary Health Care and HIV and AIDS Services”; and ii) provide concrete recommendations for the Mission’s new community-driven HIV prevention project by outlining critical components to be included in the design, serve as the basis for a concept paper for the new design and finally form the basis of the project description to be developed for any follow-on project. While the JHU HIV Communication Programme project encompasses a diverse set of activities, it was expected that the evaluation concentrates its focus on the major level of effort components of the project – e.g., mass media and community mobilization. The evaluation sought to identify follow-on activity needs to be integrated as well as describe the most critical components of the JHU HIV Communication Programme project in order to maximize performance. EVALUATION QUESTIONS In response to the USAID/SA scope of work, the activities of this evaluation aimed to answer the following four key questions below related to the development hypothesis; appropriateness of strategies to change gender norms; engagement with diverse partners; and, how the overall structure and management of JHU HIV Communication Programme affected performance. 1. Did the development hypothesis of the JHU HIV Communication Programme program relate to the achievement of expected results as articulated in the original scope of work? If not, why not? 2. Has JHU HIV Communication Programme implemented the most appropriate strategies to change social and gender norms and individual behaviors? If so, how? a. Have the major components (e.g., mass media, community mobilization) been complementary and been able to reinforce key messages to maximize performance? If so, how? b. Have the major components strengthened linkages across the continuum of response (Prevention, Care, and Treatment)? If so, how? c. To what extent has JHU HIV Communication Programme been able to integrate gender throughout its approach and how did this affected performance? d. Has JHU HIV Communication Programme implemented the most appropriate strategies to reach its target populations? e. To what extent has JHU HIV Communication Programme addressed the key drivers of the epidemic (multiple concurrent partners, low condom use, drug and alcohol abuse, 13 USAID/Southern Africa: JHU HIV Communication Programme Performance Evaluation low prevalence of male circumcision)? 3. How has JHU HIV Communication Programme’s engagement with the diverse stakeholders in South Africa affected the performance of the project? a. Coordination with and Technical Assistance to the South African Government (Department of Health, Department of Basic Education, Department of Social Development, South Africa National AIDS Council-Communications Technical Task Team) b. Coordination with and Technical Assistance to USG implementing partners and local organizations c. Partnerships with the Private Sector (e.g., Levi’s, South African Broadcasting Corporation (SABC)) 4. To what extent has the overall structure and management of JHU HIV Communication Programme affected performance? a. How has the sub-grantee model (providing some sub-grants to organizations responsible for content development and other responsible for community mobilization) affected the performance and sustainability of the HIV response? b. How has this model strengthened the capacity of the local organizations supported directly through JHU HIV Communication Programme? c. How has the linkage with Johns Hopkins University-Center for Communications Programs been a value added? 14 USAID/Southern Africa: JHU HIV Communication Programme Performance Evaluation INTRODUCTION AND BACKGROUND OVERVIEW OF THE PROJECT South Africa has more people living with HIV (approximately 5.6 million) than any other country in the world. It ranks 3rd highest globally in terms of the TB burden, with an incidence that has increased over 400% in the last 15 years.1 The dual HIV/TB epidemics pose the largest health challenges to the country. In response, the Johns Hopkins University Center for Communication Programs (JHU-CCP) was awarded a five-year cooperative agreement by USAID/SA (2008-2013) with an obligated amount of $80 million to support the USAID/SA/South Africa Health Strategic Objective: “Increased use of HIV/AIDS and other Primary Health Care services.” JHU-CCP, through its South Africa Non-Governmental Organisation (NGO) affiliate, JHU HIV Communication Programme works with the South African government and civil society partners to undertake strategic communication that combines mass media and interpersonal approaches to improve the health and wellbeing of all South Africans. JHU HIV Communication Programme’s project beneficiaries are identified as government departments, parastatal organisation and local NGOs, Community Based Organisations (CBOs) and Faith Based Organisations (FBOs) who received resources to support and strengthen their community response to HIV/AIDS and /or TB. In the five year period JHU HIV Communication Programme coordinated the work of more than 30 South African partners that use communication interventions to reduce the number of new HIV infections, promote HIV counseling and testing (HCT) and emphasize the importance of treatment adherence (see Annex VIII for list of JHU HIV Communication Programme’s partners). While JHU HIV Communication Programme’s reach is nationwide, its main activities are concentrated in high transmission areas in Kwa-Zulu Natal, Gauteng, Mpumalanga, and Free State provinces. Collaboration and coordination with the SAG has been at the core of the project’s approach with alignment to the government’s program being integrated through strategies such as provision of technical assistance (TA) with SAG priorities and needs, focus on use of local expertise, and focus on sustainability. The goals of JHU HIV Communication Programme and its partners were guided by the National Strategic Plan (NSP) for South Africa 2007-2011, and now by the new National Strategic Plan (NSP) for STIs, HIV, and TB for 2012-2016. JHU HIV Communication Programme’s communication approach recognized that effective behavior change communication (BCC) is grounded in a particular socio-ecological context, including enabling environments, service delivery systems, communities, husbands and wives, family members and individuals. All of JHU HIV Communication Programme’s work is based on the Pathways to a Health Competent Society framework (see Figure 2). A health competent society is one which values the individual so that s/he can make appropriate health decisions within a healthy participatory community that supports and enables them to do so, enabled by effective health care delivery systems and supporting health policy. The project utilized this framework to reach its target population of out-of￾school youth, sex workers, people with HIV, high risk women and adult men. Aside from the project, the country has had a long history of providing behaviour change communication activities to address HIV and TB among its population. Large scale communication campaigns have been running for many years to raise the awareness of HIV and Acquired Immunodeficiency Syndrome (AIDS). These include Khomanani (“caring together”) which is the SAG’s primary AIDS-awareness campaign since 2001; Soul City and Soul Buddyz, multi-media campaigns targeted at adults and children respectively, to promote good sexual health and well-being; and the campaign loveLife which has run since 1999 and mainly targets teens. Over the years, there has also 15 USAID/Southern Africa: JHU HIV Communication Programme Performance Evaluation been more emphasis on social norms, policies, culture and supportive environments with prevention activities being part of a broader continuum of HIV response in relation to treatment, care, support, rights and social mobilization. The 2009 HIV National Communication Survey, itself an initiative of JHU HIV Communication Programme, showed that communication programs are beginning to impact on knowledge levels and behavior. PROJECT AT A GLANCE • Primary Audiences: o Youth ages 15-24 o Women of sexual and reproductive age o Men aged 25+ • Secondary Audiences o Policy and decision makers o Traditional leaders and structures o Government Departments • Objectives: Behavioral Prevention o Increase the age of sexual debut amongst young people o Increase people’s perception of risk to HIV infection in relation to their behaviours (alcohol, transactional sex, etc.) and sexual partnerships and encouraging the development of risk reduction strategies 16 USAID/Southern Africa: JHU HIV Communication Programme Performance Evaluation o Reduce the number of men who report having multiple and concurrent partners o Increase correct and consistent condom usage with all partners • Objectives: Biomedical Prevention o Promote early antenatal booking amongst pregnant women o Increase the knowledge benefits of exclusive breastfeeding for Prevention of Mother to Child Transmission (PMTCT) o Reduce the number of children born with HIV o Increase the levels of knowledge of the HIV benefits of Medical Male circumcision o Increase the number of men who are circumcised • Objectives: C&T and C&S o Counseling and Testing (C&T)  Increase the number of people who undergo Voluntary Counseling and Testing (VCT ) for HIV and who receive their test results  Increase the number of people who test for HIV on a regular basis o Care and Support (C&S)  Increase awareness of the linkages between HIV/TB  Increase knowledge and awareness of the signs and symptoms of TB  Increase awareness of opportunistic infections and the need for early treatment • Objectives: Treatment and Strategic Information o Treatment  Increase the number of people who are treatment literate 17 EVALUATION METHODS & LIMITATIONS The performance evaluation was conducted in South Africa from 14 January to 1 March, 2013 by a technical team from URSA which was comprised of highly qualified health care professionals who brought a range of expertise in support of the evaluation activities. A number of limitations to the evaluation have been identified and reported results should be viewed in this light. Only activities in KwaZulu-Natal, Free State and Gauteng provinces were evaluated as no partner was based in Mpumalanga and consequently, sampling bias may have been introduced. The short time period assigned to conduct the evaluation limited the scope and the team could only conduct limited observation of actual service delivery which could be considered a gap. Additionally, the data collected was based on interviewee responses, resulting in possible recall biases particularly for mass media communications related information. Focus group discussions are not generalizable and represent the views of the participants only. During the evaluation some of the sites we originally identified to include in our sample were not evaluated as they no longer existed or fell outside the targeted provinces, introducing possible selection bias. These have been listed in Annexes V and VI. Taking into account the limitations that have been identified, the selection of sites represents more than 60% of JHU HIV Communication Programme’s sub-partners at the time of the evaluation. The lists of sites visited and excluded are listed in Annex V and Annex VII. METHODOLOGY To conduct the evaluation, the team combined outcome evaluation with process evaluation. The purpose of the outcome evaluation was to determine the extent to which JHU HIV Communication Programme’s program's specific objectives were achieved, what worked, what did not and why, as well as inform the design of a new community-driven HIV prevention project. Through process evaluation, we sought to describe the program and how it was implemented, and through this, attempt to gain an understanding of why the objectives were or were not achieved. The evaluation focused on the most strategic communication interventions that used a combination of three types of programmes: 1) Mass media – TV and radio spots; TV dramas; posters, pamphlets; music videos, etc.; 2) Community mobilisation and; 3) Interpersonal communication and counseling. Sites were chosen purposively in collaboration with USAID/SA, the SAG, and JHU HIV Communication Programme to reflect the full spectrum of JHU HIV Communication Programme cooperative agreements. Sample sites are listed in Annex V. Study Design To improve the credibility of findings, the evaluation utilized a combination of qualitative and quantitative evaluation methods, including: document review of program activities; review and analysis of program monitoring and evaluation data; assessment of sub grantees’ performance on key outcomes; and assessments conducted with target populations, beneficiaries, policy makers, administrators, and stakeholders. These are described below: Review of project documents In collaboration with USAID/SA and JHU HIV Communication Programme, key documents were 18 reviewed and analyzed including: • Original proposal to USAID/SA • Cooperative Agreement and modifications • Reports to USAID/SA-quarterly, semi-annual and annual • Reports to Government departments as available • Sub-award scopes of work and modifications • Training reports • Annual program workplans • Monitoring, evaluation and reporting plans The purpose of the review was to inform USAID/SA on: • Which project approaches may have worked well and which ones did not work well • Whether the main components of JHU HIV Communication Programme’s strategy were complementary and if and how they strengthened linkages across the HIV prevention, care, and treatment response • The appropriateness of JHU HIV Communication Programme strategies to change gender norms • Engagement with diverse partners • How the overall structure and management of JHU HIV Communication Programme affected performance • Whether planned messaged were produced and delivered to intended audiences • Whether planned activities were carried out as designed, on time, and on budget • Whether intended audiences were being reached • What was happening in the program environment during the intervention period. Review and analysis of program monitoring and evaluation data URSA developed standardized questionnaires to collect and compile monitoring and evaluation data from each sub-grantee and JHU HIV Communication Programme (see Annex III). The focus of this review was to collect information on whether the JHU HIV Communication Programme program achieved its expected level of results as outlined in the original scope of work. URSA’s analysis centered on assessing the inputs, outputs (e.g., knowledge, attitudes, motivations, changes in behaviours, changes in service delivery, skills, community participation) and outcomes (e.g., changes in health indicators), causal attributions (did the program contribute to cause those changes), main accomplishments and challenges of each of JHESSA’s main USAID/SA-funded communication activities. Findings were compared across sub-grantees and project activities to document best practices and lessons learned. Documents reviewed included: • Monitoring and evaluation data from sub-grantees as well as JHU HIV Communication Programme’s monitoring and evaluation plan and outcomes. These included an assessment of the appropriateness and adequacy of indicators (e.g., how gender was incorporated into the indicators); quantitative measures as well narrative reports. • Data quality audit reports • Documents indicating use of radio, TV and print media • Documents indicating evidence of civil society engagement • Independent program evaluations • Research reports 19 Structured interviews/assessments with sub grantees, JHU HIV Communication Programme staff, USAID/SA, and program beneficiaries The study instruments consisted of 6 questionnaires/assessments: Interview guide for USAID/SA activity manager; interview guide for JHU HIV Communication Programme national office; interview guide for sub-grantee (community mobilization); interview guide for sub-grantee (content development); interview guide for key stakeholders; focus group guides for community intervention; and focus group guides for mass media. Questionnaires and all interviews were conducted in English and therefore no translations or back translations were made. The purpose of the questionnaires was to gather additional data on whether the strategies employed by JHU HIV Communication Programme were appropriate to change social and gender norms, JHU HIV Communication Programme engagement with key stakeholders, and the viability of the sub-grantee model. In addition, URSA observed project activities of select sub￾grantee organizations in the JHU HIV Communication Programme supported provinces. Table 1 describes the evaluation questions, the corresponding data sources that were used to address each question, the study populations, and the limitations. Data Collectors Data collectors consisted of the project evaluation team comprised of a team leader, two senior technical advisors, two field supervisor and two dedicated data collectors. The same evaluation team was used for all the interviews to ensure consistency. Questionnaires were reviewed prior to data collection to ensure uniformity in the understanding of the tools. Ethical Considerations Data was collected in a manner that adhered to ethical principles and promoted confidentiality of organizations and interviewees. Verbal consent was obtained for voluntary participation from respondents prior to data collection commencing. Respondent identifier information was removed to protect confidentiality. Data Analysis Interviews were recorded and transcribed to ensure accuracy of data collection. Data was analyzed using Excel for quantitative data; and manually for qualitative data, including focus group discussions. Table 1: Evaluation question and data source Evaluation Question Data Source 1. Did the development hypothesis of the JHU HIV Communication Programme program relate to the achievement of expected results as articulated in the original scope of work? If not, why not? - Review of JHU HIV Communication Programme project documents - Interview with JHU HIV Communication Programme senior managers - Interview guide for USAID/SA activity manager 2. Has JHU HIV Communication Programme implemented the most appropriate strategies to change social and gender norms and individual behaviors? If so, how? a. Have the major components (e.g., mass media, community mobilization) been complementary and been able to reinforce key messages to maximize performance? If so, how? - Interview guide for USAID/SA activity manager - Interview with JHU HIV Communication Programme senior managers - Interviews with subgrantees - Focus Group Discussions (FGD) 20 b. Have the major components strengthened linkages across the continuum of response (Prevention, Care, and Treatment)? If so, how? - Review of JHU HIV Communication Programme project documents - Interviews with sub-grantees - FGD c. To what extent has JHU HIV Communication Programme been able to integrate gender throughout its approach and how did this affected performance? - Review of JHU HIV Communication Programme project documents - Interviews with sub-grantees - Interview guide for USAID/SA activity manager d. Has JHU HIV Communication Programme implemented the most appropriate strategies to reach its target populations? - Interview with JHU HIV Communication Programme senior managers - FGD e. To what extent has JHU HIV Communication Programme addressed the key drivers of the epidemic (multiple concurrent partners, low condom use, drug and alcohol abuse, low prevalence of male circumcision)? - Interview with JHU HIV Communication Programme senior managers - Interviews with sub-grantees 3. How has JHU HIV Communication Programme’s engagement with the diverse stakeholders in South Africa affected the performance of the project? a. Coordination with and Technical Assistance to the South African Government (Department of Health (DOH), Department of Basic Education (DBE), Department of Social Development (DSD, South Africa National AIDS Council-Communications Technical Task Team) - Interview with JHU HIV Communication Programme senior managers - Interview guide for NDOH/DBE/DSD/SANAC/Private Sector Partners - Interview guide for USAID/SA activity manager b. Coordination with and Technical Assistance to USG implementing partners and local organizations - Interview guide for NDOH/DBE/DSD/SANAC/Private Sector Partners - Interview guide for USAID/SA activity manager - Interviews with sub-grantees c. Partnerships with the Private Sector (e.g., Levi’s, South African Broadcasting Corporation (SABC) - Interview with JHU HIV Communication Programme senior managers - Interview guide for National Department of Health (NDOH)/Department of Basic Education (DBE)/Department of Social Development (DSD)/SANAC/Private Sector Partners - Interview guide for USAID/SA activity manager 4. To what extent has the overall structure and management of JHU HIV Communication Programme affected performance? a. How has the sub-grantee model (providing some sub-grants to organizations responsible for content development and other responsible for community mobilization) affected the performance and sustainability of the HIV response? - Interview with JHU HIV Communication Programme senior managers - Interviews with sub-grantees b. How has this model strengthened the capacity of the local organizations supported directly through JHU HIV Communication Programme? - Interview with JHU HIV Communication Programme senior managers - Interviews with subgrantees - Interview guide for USAID/SA activity manager 21 c. How has the linkage with Johns Hopkins University-Center for Communications Programs been a value added? - Interview with JHU HIV Communication Programme senior managers - Interviews with sub-grantees - Interview guide for USAID/SA activity manager - Interview guide for USAID/SA activity manager 22 FINDINGS, CONCLUSIONS & RECOMMENDATIONS This section presents the key findings of the project evaluation and has been organized and regrouped to respond to the evaluation questions. QUESTION 1: DEVELOPMENT HYPOTHESIS Evaluation question 1 asked: “Did the development hypothesis of the JHU HIV Communication Programme relate to the achievement of expected results as articulated in the original scope of work? If not, why not?” JHU HIV Communication Programme’s development hypothesis is based on the Social Ecology Model (SEM) theoretical framework illustrated in Figure 1, 2 as well as the Pathways to a Health Competent Society conceptual framework to inform the health communication program for social and behavior change. The model defines a health competent society as one “possesses the necessary elements for optimal health performance, namely a supportive environment, an effective system, and health literate individuals/societies”. The hypotheses take into account the interrelated complexities of the individual and how relationships with his/her social networks, communities, and larger societal environment bring about and sustain individual behavior change. JHU HIV Communication Programme’s HIV prevention interventions are designed to target multiple levels and address factors that put individuals at risk of HIV infection. JHU HIV Communication Programme worked in three domains: 1) social political environment; 2) service delivery system; and 3) community and individual. Specific activities have included assistance to Government Departments, parastatal organizations, and local NGOs, CBOs and FBOs. A multipronged approach was developed and implemented by JHU HIV Communication Programme with health communication as the vehicle to effecting change. JHU HIV Communication Programme’s health communication model consists of four broad types of communication: advocacy, mass media communication, community mobilization and strategic information, all of which are interlinked and intended to complement each other. The cooperative agreement between USAID/SA and JHU HIV Communication Programme specified the Pathways to a Health Competent Society conceptual framework while the Cooperative Agreement and all subsequent documents made reference to the SEM models. A JHU HIV Communication Programme key informant indicated both models were not contradictory, with the Pathways to a Health Competent Society conceptual framework being a Figure 1. Social Ecology Model & Communication for Social and Behavioural Change [Source :JHU HIV Communication Programme, 2013] 23 predecessor of the SEM model. According to the Cooperative Agreement 3 the aim of the project is “to reduce HIV transmission and to mitigate the impact of HIV and AIDS on families and communities” by implementing a high level prevention, treatment, care and support strategic communication intervention over the five years. Specific outcomes of the project are articulated as; to impact on the attitudes, norms and behaviors of men and women in sustaining high rates of concurrency and partners and the resultant sexual networks, cross-generational sex between older men and younger women, and the need for male involvement in the prevention of mother-to-child transmission interventions; promote counseling and testing as a key intervention that supports prevention, treatment, care and support and to increase treatment literacy, including nutrition, positive prevention, palliative care for those in need and awareness of the linkages between TB-HIV.” In support of this aim JHU HIV Communication Programme’s development objectives are categorized into four key areas; behavioral, biomedical, HCT and care and support and finally treatment each with by intended results.3 JHU HIV Communication Programme’s development objectives were assessed to be aligned to the development hypothesis. Findings • Development objectives aligned with National Strategic Plan, but could be formulated better: JHU HIV Communication Programme successfully moved its program beyond the theoretical framework towards achievement of the development objectives and goals. JHU HIV Communication Programme’s development objectives were targeted and aligned to the priorities of the country as indicated in the National Strategic Plan (NSP) 4 and took into account the social dynamics of the country to ensure investments promote sustainable outcomes. However, we found the project objectives in the original scope of work were not SMART (i.e., did not indicate the expected amount of change and were not time bound), and thus it was difficult to assess impact according to the objectives of the project. Examples of the objectives included: “Increase the number of people who test for HIV on a regular basis” and increase correct and consistent condom usage with all partners,” neither of which indicate the degree to which changes could be made. • Good use of multi-sectoral partnerships: The JHU HIV Communication Programme appropriately capitalized its existing expertise and purposively selected diverse, multi-sectoral and congruous partners to ensure developmental objectives were mutually reinforcing and integrated towards the anticipated aim. The strategic selection of partners from the SAG, private sector and other NGOs accessing non-USAID/SA/United States President’s Emergency Plan for AIDS Relief (PEPFAR) funds resulted in maximization of results in a synergistic manner in support of the development hypothesis. • Ground-breaking research projects conducted: The JHU HIV Communication Programme conducted several groundbreaking research projects such as the National Communication Survey (NCS) pertaining to the impact of health communication programs on health outcomes. • Targets were achieved, but initially were too ambitious: Review of the progress reports showed that in addition to PEPFAR indicators, JHU HIV Communication Programme developed multiple relevant measurable indicators to monitor progress. Several targets had not been achieved particularly prior to FY2010 and had to subsequently be revised in FY2010, after which they were largely attained. This was also influenced by changes in the USAID/SA indicator definitions such the introduction of small group discussions and termination of contracts with partners contributing to certain targets. It is however, possible that the targets initially set were too ambitious. 24 • Structural drivers of epidemic not sufficiently addressed: The Infections Averted Report conducted by JHU HIV Communication Programme 5 indicated that, when compared to the unemployed, those currently employed are slightly more likely to practice HIV prevention behaviors. This was further supported by the 2006 NCS which reported that poorer people were less likely to know that faithfulness, reduction in sexual partners and abstinence are methods to prevent HIV. 6 However, the program activities did not address the role of and the reduction structural drivers of the epidemic – e.g. socioeconomic factors such unemployment, low levels of education and poverty that can result in sexual risk taking and promote multiple concurrent partners (MCP), intergenerational sex and low condom use particularly among women. Conclusion The JHU HIV Communication Programme’s theoretical framework allowed it to select appropriate interventions that address various factors that affect an individual’s HIV-related behavior change. Together with its multi-sectoral partners, JHU HIV Communication Programme implemented a highly successful health communication program that reached large number of its target audience. Building on the Social Ecology Model, a predecessor of the Pathways to a Health Competent Society conceptual framework, JHU HIV Communication Programme identified relevant objectives which were largely achieved as outlined in its scope of work. However, the current objectives were not well articulated and did not reflect the degree of expected change nor were they time bound. Notably, prior to FY2011 most targets were not achieved until revised downward. In order to fully realize and address to a greater degree the structural factors that influence behavior change, future program design should incorporate strategies to address HIV within the context of unemployment and poverty. Recommendation • Address the structural drivers of HIV. Future project design should more comprehensively address community and societal factors that interface with the individual to increase their risks of HIV acquisition and transmission. Integration of HIV prevention activities into the microenterprise approach is recommended as a public health strategy that is responsive to the broader socio￾economic and structural context that particularly affects women for future programming. • Develop measurable objectives with targets. The future project should articulate specific timelines in its objectives as well and quantify the expected measurable changes of each objective and select realistic, achievable targets for each of its partners. Barriers to reaching targets should be identified and addressed in a timely fashion. QUESTION 2: APPROPRIATENESS OF STRATEGIES TO CHANGE SOCIAL AND GENDER NORMS AND INDIVIDUAL BEHAVIORS This section addresses evaluation question 2 which asked: “Has JHU HIV Communication Programme implemented the most appropriate strategies to change social and gender norms and individual behaviors and if so, how?” Findings per sub-component of this question are provided below. The JHU HIV Communication Programme’s activities comprised of two major components: mass media and community mobilization, both Figure 2. USAID/JHU HIV Communication Programme Expenditure FY08 -12 [Source: JHHESA , 2013] 25 aimed at changing the social and gender norms and individual behaviors. For the period FY08 – FY11, the total USAID/SA/PEPFAR funding to JHU HIV Communication Programme was reported to be $51,984,245 with other funding sources excluding Public Private Partnerships (PPP) contributions amounting to $1,669,224.86. 7 The bulk of the program expenditure was distributed between mass media campaigns and interpersonal communications (community mobilization) both of which accounted for 77% of the project expenditure. JHU HIV Communication Programme’s expenditure was primarily towards program activities in almost equal proportions to mass media and community mobilization. JHU HIV Communication Programme innovatively accessed additional significant resources through PPP to support interventions. The evaluation noted that capacity building activities only constituted 1.1 % of the expenditure towards two academic programs implemented by University of the Witwatersrand and University of Kwa-Zulu-Natal. Have the major components (e.g., Mass media, community mobilization) been complementary and been able to reinforce key messages to maximize performance? Mass Media: JHU HIV Communication Programme and its partners developed and implemented four main mass media campaigns: Scrutinise, Brothers for Life, Intersexions (SABC1), and 4play: Sex Tips for Girls (eTV), each addressing themes related to targeted HIV prevention behaviors among specific target groups. The campaigns included the production of two high quality TV dramas: 4Play Sex Tips for Girls (Season 1 and 2 (2010 and rebroadcast in 2011); Season 3 (2012) and Intersexions, a Peabody award winning show. Intersexions is the second highest viewed show in the history of the national broadcaster SABC. Both TV dramas targeted women in different age categories (i.e., Sex Tips targeted women ages 25-39 years; and Intersexions women ages 18-35). To complement the broadcasting of the shows, specific toolkits to reach their adult audiences were developed. For example, for Sex Tips for Girls had a discussion guide and DVD to further explain the themes of the campaign, as well as to encourage discussion about how to be responsible for one’s health and relationships. For Brothers for Life, DOH social mobilisers used the toolkits which allowed field workers to explain and discuss scientific concepts related to MMC to communities. Multimedia platforms: JHU HIV Communication Programme innovatively made use of numerous complementary channels to communicate key messages by using a variety of digital platforms in the mass media campaigns. This included the JHU HIV Communication Programme website, targeting primarily funders, policy-makers, civil society, researchers and the media. The website had recorded more than 23911 visitors and has more than 4000 research reports downloaded in the period FY2008 to FY2012. To reach youth aged 15-24, in addition to TV, JHU HIV Communication Programme utilized websites, blogs, social media such as Facebook and Twitter, and cellphone technology – e.g., a Short Message Service (SMS) site locator, that costs R0,20c per SMS and a ‘Please Call Me’ service, for Male Medical Circumcision services to augment all its mass media campaigns. This demonstrates a novel approach that reaches the target market by matching their most commonly used communication platforms. JHU HIV Communication Programme innovatively utilized social media as a tool for several purposes including monitoring responses to TV shows, as an incubator for new creative ideas and content for storylines and character development. The use of a professional sexologist to interact with users and moderate discussions created an opportunity to address misinterpretation of key messages and address myths and concerns regarding HIV prevention by experts. Social media monitoring and interaction with the target audience also allowed for more complex, technical topics (e.g., HIV discordance among couples) to be addressed and explained by a content specialist. Local media: JHU HIV Communication Programme also implemented 98 billboards situated in high 26 traffic areas such as taxi ranks, clinics and shopping centers as well as small media (e.g., posters and pamphlets in its communications). In addition, other out of home media were used which included Taxi TV and In-Taxi branding. Community mobilization activities: An effective component to JHU HIV Communication Programme’s health communication program was community mobilization, largely implemented through JHU HIV Communication Programme’s financial support as well as technical assistance through mentoring and training to community based organizations. The success of community mobilization activities was attributable to the close proximity and access to the communities to implement community mobilization activities that focus on; prevention, abstinence and be faithful (AB), male norms, gender based violence, PMTCT, MMC, treatment literacy, stigma reduction, and prevention with positives. Community mobilization organizations also participated in activities such as facilitated discussions in primary health care facilities, outreach events in tertiary institutions and community dialogues. JHU HIV Communication Programme also worked closely with religious organizations e.g. National Religious Association for Social Development (NRASD), a network of religious groups, with the aim of fostering the role of religious organizations in social development projects. The organization primarily conducts outreach among local religious institutions to promote HIV prevention within the confines and context of religion. Findings: • Synergy of messages: In all the mass media campaigns, JHU HIV Communication Programme worked with key stakeholders such as the SAG agencies and local communities to identify key messages pertaining to the project’s objectives. This ensured synergy in the key messages with other health communication activities of the SAG. JHU HIV Communication Programme’s role in providing technical assistance to the SAG is further discussed in response to Question 3. JHU HIV Communication Programme actively exercised leadership and control of the development of the mass media messages by its content development partners. Key informants attested to the extensive engagement and oversight undertaken in the conceptualization and development of key messages to avoid contradictory communications with other JHU HIV Communication Programme campaigns as well as other national campaigns (e.g., Soul City and Khomanani). In addition, the participants in FGDs felt that messages were complementary and consistent with other existing mass media content developed by other organizations (e.g., Soul City, Love Life, Room 9 and Untold Stories). • Synergy of mass media channels: The JHU HIV Communication Programme designed each campaign to incorporate multiple media channels including TV, radio, outdoor media, print, social media and cellphone linked through an overarching theme at all levels (e.g., promotion of condom use, reduction in MCP and HCT). Scrutinise and Brothers for Life further incorporated significant interpersonal communications and community mobilization components to augment messaging. In addition, the use of social media was an innovative approach to health communication that complemented the TV dramas and allowed the JHU HIV Communication Programme to monitor immediate feedback by the viewers after each episode as well as for the project’s professionals to engage with the target audience which is considered a best practice for extending dialogue. 27 • Synergy between interpersonal communication and mass media channels: JHU HIV Communication Programme purposively selected a combination of partners that serve diverse but complementary functions in the program. Interviews with key informants \from these CBOs demonstrated synchrony with and extension of the mass media campaigns through the interpersonal communications and provision of clinical services. For example, partners conducted outreach activities targeting youth and developed communication approaches that integrated complementary content e.g., Scrutinise, which conducted communication within school based programs through the life skills lessons, workshops, individual counseling and peer education programs. Organizations’ community mobilization activities were further augmented through the use of community based media (e.g., local, campus radio and community radio spots and talk shows, drama, community dialogues, workshops and peer education mechanisms to promote and encourage dialogue and locally driven communication). JHU HIV Communication Programme developed materials that accompanied TV dramas (e.g., discussion guides that were used as part of the social mobilization activities allowing interactive discussion with viewers). The discussion guides were facilitated by staff trained and mentored by JHU HIV Communication Programme and were used to educate, inform and guide dialogue with community members about the topics raised in the mass media campaigns. In focus group discussions with beneficiaries of JHU HIV Communication Programme’s mass media campaigns as well community mobilization activities, participants indicated that a combination of platforms, mainly TV and radio were the preferred modes of communication to receive information. Participants largely agreed on the entertainment value of TV dramas and acknowledged the fidelity of the storylines to their own observations of the same issues happening in their communities. In addition to entertainment, TV dramas were an important source of information, created awareness about HIV prevention and stimulated dialogue. Radio was reported as an important method of receiving HIV prevention messages and for some, it was preferred for its ability to be interactive and promoting dialogue. A participant said, “We can call, ask questions and get feedback.” While participants valued radio as a means of communication because it was physically accessible as well provided a choice of languages, they indicated that they often did not listen to the radio in the evenings because they watch TV at that time. One participant stated, “[radio] is for old people.” Conclusion: JHU HIV Communication Programme and its partners have implemented multimedia health communications campaigns that have reached its target intended audiences. As a result of JHU HIV Communication Programme’s leadership and involvement in the conceptualization, development and dissemination of the activities, the key messages are complementary and synergy is achieved within each campaign as well as with other SAG and key stakeholder campaigns. Recommendations: • Increase capacity building activities. Investments in capacity building particularly of the SAG structures and departments tasked with health communications should be implemented to promote sustainability. Activities should aim to build a critical mass of people within the SAG with the right capacity to manage and implement health communications programs at the scale initiated by JHU HIV Communication Programme. Capacity building activities are further addressed in Question 3. 28 • Radio is an important, participatory channel for engaging target audiences particularly for rural communities. Radio complements TV shows by extending dialogue pertaining to key messages. While a multipronged, multichannel approach is important, future program designs should be cognizant of the limitations of social media for rural audiences and should be tailor communication to be more accessible for rural audiences. Have the major components strengthened linkages across the continuum of response? Within the context of HIV prevention, JHU HIV Communication Programme’s health communication strategy is intended to support community based services by: creating demand for care and treatment services; maintain the continuum of clinical treatment and referrals; allow for management of opportunistic infections; initiate Anti-Retroviral Therapy (ART/ARV) treatment and provide AIDS care and treatment services including ART; manage complications or side effects; address treatment failure; and make or confirm diagnoses.8 For the purposes of this evaluation referral is defined as the process through which a client is moved or moves through the continuum of treatment, care and support. Linkages are viewed as the formal structures or conduits between institutions or organizations through which the process of referral occurs. Findings: • Linkages created through mass media: JHU HIV Communication Programme promoted linkages to prevention care and treatment services in TV content by providing information at the end of each episode that will direct the viewer to the resources or services that he or she may require e.g. the National AIDS helpline, a SAG information resource. • Linkages created to HCT: JHU HIV Communication Programme promoted HCT as the entry point to care, prevention and support services through all its mass media channels (e.g., a four page HIV Testing supplement in a widely circulated daily newspaper, estimated to reach more than 7, 2 million people, which profiled a man living positively with HIV and highlighted how testing changed his life for the better was published). Many participants in the FGDs reported having sought HCT for themselves and also encouraging others, such as family members to do so directly as a result of viewing Intersexions. A large number of individuals also received HCT through JHU HIV Communication Programme’s community based partners. JHU HIV Communication Programme was a key player in the communication strategy for National HCT campaign. The mass media and community mobilization components of the HCT theme were enhanced by the provision of HCT services conducted by trained and registered professional nurses who also render pre- and post-test counseling of individuals. Community mobilization partners indicated that community members who test HIV positive are then referred to the nearest health facility for further management (e.g. CD4 and clinical staging). However, there was no documented or formal referral pathway between JHU HIV Communication Programme’s community mobilization partners and health facilities to which patients who had received services were being referred. The number of individuals who received HIV pretest counseling was substantially lower than the reported number of individuals who tested for Figure 3: Number of people tested for HIV [Source: JHHESA] 29 HIV. This may be due to poor reporting, poor data quality or non-compliance with HCT process mandates. An innovative example to strengthening linkages and referrals was noted with the NRASD programme through its “Helping Hands” model which encouraged young people to test for HIV but also accompanied them to the health facilities for HCT and PMTCT services and supported HIV positive individual by implementing Prevention with Positives (PwP).This strategy ensured that the individual successfully accessed the required services and also included referral to local pastoral support in churches. • Linkages to MMC were formed, but barriers to MMC can be better addressed: Some of the community mobilization partners shared physical premises with service providers of MMC services thereby promoting both HCT and MMC services. MMC is supported primarily through the Brothers for Life campaign. A significant barrier to MCC usage, reported by focus group participants for this evaluation, was the perception of health care workers’ negative and judgmental attitudes at government institutions. Some participants indicated that at times they did not access the desired services because they anticipated poor service towards patients. A participant noted “They speak anyhow to men.” Men also indicated that in addition, the high number of female staff made them reluctant to visit health facilities for MMC services. Review of the NCS 2012 11 report on the impact of health communications programs on the intention to get circumcised indicates that most uncircumcised males in the sample (more than Figure 5: Intention to get circumcised [Source: JHHESA] Figure 4: Number reached through VMMC outreach [Source:JHHESA] 30 50%) indicated that they did not have the intention to get circumcised despite having being exposed to health communications programs, a factor that may have been impacted by the degree of exposure to the campaigns. The survey further found that despite the campaigns, few men and women knew that circumcision reduces the HIV risk with the spontaneous mention of MMC as an HIV prevention method was still low (7%) although it has increased since 2009 (0.4%).9 A new communication project should also explore other root causes of barriers to MMC particularly among older men. A new communication program could be enhanced by creating training programs that target health care professionals in health services on customer care. Greater interactions and building on existing professional relationships with JHU HIV Communication Programme’s community partners could assist. A new communication program should explore the causes of barriers to service delivery at health facilities particularly for men. • TB/HIV service integration can be improved: Review of data reports indicate that JHU HIV Communication Programme addressed TB screening by integrating this theme into some of its drama series storylines, particularly 4Play. However at the community level, review of JHU HIV Communication Programme’s data indicated that of the 10688 newly diagnosed HIV positive individuals, a total of 7791 were screened for TB. Interviews with key informants from community mobilization partners indicated that the mechanisms for referral of patients who required further TB screening through sputum collection were not formalized between the community based organizations and the facilities. Only a total of 883 TB patients being tested for HIV over a three year period by the community mobilization partners (see Figure 6). Given the high case load of HIV and TB in the districts in which JHU HIV Communication Programme works, scale up of TB and HIV services is recommended. • Good Linkages to Treatment Literacy JHU HIV Communication Programme’s partnership with The Community Health Media Trust (CHMT) was a significant interpersonal communications intervention that promotes treatment literacy including positive living. Treatment Literacy and Prevention Practitioners (TLPPs), some of whom are People Living with HIV, were trained to conduct facilitated discussions with patients in public health facilities (including ARV , ANC, and TB clinics) supported by small media. CHMT developed a comprehensive treatment literacy series, "Siyayinqoba: Beat it", in DVD and print formats which it used to facilitate sessions with HIV positive people, their families and other community members. Review of JHU HIV Communication Programme’s documentation indicated more than one million people were recipients of information on topics such as ARV, opportunistic infections, nutrition, hygiene, PwP and accessing ARV services. In addition, JHU HIV Communication 0 50 100 150 200 250 300 350 400 Q4 COP07 COP08COP09COP10COP11 Figure 6. Number of TB patients Tested for HIV [Source:JHHESA] Number of TB patients tested for HIV Male Number of TB patients tested for HIV Female 31 Programme supported 94 support groups reaching 1769 people during the life of the project. Findings from the interviews indicated that scale up for People Living with HIV/AIDS (PLWHA) is an unmet need. None of the supported organizations seemed to include active involvement of PLWHA in programming conceptualization and implementation. Given that South Africa has the highest number of people on ART in the world this would be a critical component of future communication program. • Successful linkages to PMTCT: JHU HIV Communication Programme played a key role in developing the Accelerated PMTCT communication and social mobilization strategy in support of the SANAC Communications Technical Task Team. The communication strategy was an integral component of the National Department of Health’s “Operational Plan for Accelerating Scale up of PMTCT” in 18 identified districts for optimal supply-demand synergies. JHU HIV Communication Programme combined mass media with interpersonal communication approaches and advocacy to reach pregnant women and women of child-bearing age, with a particular emphasis on women aged 20 – 34 years as well as men and fathers, caregivers (grandparents, midwives and family members) and health care providers. JHU HIV Communication Programme reached 117,908 people through PMTCT outreach activities with 1,603,695 reached at health facilities. JHU HIV Communication Programme also incorporated PMTCT in its Intersexions storyline to promote awareness on the topic. Conclusion: JHU HIV Communication Programme successfully promoted linkages across the continuum of HIV response in its mass media campaigns and TV dramas by providing information to viewers on where to access services. Further linkages are promoted by JHU HIV Communication Programme’s community mobilization partners through outreach activities as well as service provision for HCT, TB screening and STI treatment. However, there are unmet needs in relation to linkages where referral systems were not formalized and feedback mechanisms between services by JHU HIV Communication Programme’s partners and public and private health facilities to which patients were being referred was not clear. While JHU HIV Communication Programme successfully conducted advocacy for HCT and TB but challenges in service provision include low pretest counseling, low TB screening rates among newly diagnosed HIV positive, and low HIV testing rates among TB patients. Low uptake of MMC among men also needs to be addressed. Overall linkages to services have to be improved and referral systems within which social mobilization partners’ work should be established where necessary and strengthened. Recommendation: • Scale up of Helping Hands model as a best practice. Partners should scale up this model that provides support for an individual to ensure that they are linked to support, care and treatment services beyond HCT. In addition the successful secondary prevention activities (e.g., Prevention with Positives and PMTCT) should be scaled up. • Identify barriers to MMC at individual, community and societal levels. The next communication project should explore root causes and develop health communication strategies targeting negative/judgmental attitudes of health care providers. This could include creating training programs that target health care professionals in health services on customer care. • Strengthen communication between health facilities and communities. Linkages between health communication activities in the communities and those in the health facilities should strengthened to promote synchrony and continuity in communication. 32 • Promote pretest counseling prior to testing. Emphasis should be placed on ensuring that HIV testing takes place in the context of counseling and underlying reasons for low pretest counseling should be addressed. • Improve TB and HIV integration. JHU HIV Communication Programme should identify strategies to improve TB and HIV integration among services delivered by its partners. • Establish formal referral systems between partners and health facilities in the community. Partners who deliver clinical services should make use of existing resources such as directories and existing relationships with public health facilities to identify and establish formal pathways for referrals which include feedback mechanisms. To what extent has JHU HIV Communication Programme been able to integrate Gender throughout its approach and how has this affected performance? JHU HIV Communication Programme’s mass media campaign Brothers for Life targets men using the tag line “Yenza Kahle – Do the Right Thing” that aims to support the NSP by developing and implementing a comprehensive package that promotes male sexual health, addresses gender and gender-based violence (GBV), and expands PMTCT service to increase the involvement of men. The intended outcomes of the campaign were to reduce the prevalence of multiple and concurrent partners – including the linkages to transactional intergeneration, sex and alcohol, sex and HIV; increase correct and consistent condom use, increase uptake of HIV testing by men; increase male involvement in PMTCT, and reduce the prevalence of gender-based violence. JHU HIV Communication Programme and its partners actively advocated against gender based violence by supporting national campaigns such as “16 Days of activism of no violence against women and children” with strategic partners such as South African Police Services (SAPS), South African National Council on Alcoholism & Drug Dependence (SANCA) and LifeLine Men’s League. This door-to-door campaign engaged with men on topics such as people’s rights in relation to women and children, the role played by alcohol and drugs in fuelling gender￾based violence and child abuse and types of abuse. Through a combination of these strategies JHU HIV Communication Programme reached more than 400,000 people between FY2009 and FY2012 (Figure 7). As part of the campaign to address GBV, JHU HIV Communication Programme’s partners also engaged other government departments in community dialogues to find solutions to GBV and to encourage the community members not to be silent about abuse but to report it to the South African Police Service (SAPS) and to make use of protection orders if they are in an abusive situation. Findings: • Successfully leveraged mechanisms to gather men: The JHU HIV Communication Figure 7: Numbers reached on gender-based violence and post exposure prophylaxis through social mobilization [Source:JHHESA] 33 Programme’s successfully harnessed existing popular and traditional cultural mechanisms to gather men to disseminate HIV prevention messages. These included providing messages through sports activities particularly football (e.g., FIFA World Cup, Confederations Cup and other sports gatherings such as viewing parks). Through its partners the project supported Footballers for Life, an initiative that aimed to assist footballers to deal with the pressure to engage in HIV risk behaviors, in particular multiple and concurrent partners, inconsistent condom use and their perceived low risk of infection . The initiative appropriately took advantage of the popularity of football celebrities among men by building on the Brothers for Life and Scrutinize brands. Community mobilization partners also targeted taverns which are typically frequented by men. • Addressed male norms: JHU HIV Communication Programme appropriately advocated for changes in male norms by addressing traditional men’s gatherings (e.g., imbizos and community dialogues). URSA interviews with FGD participants indicated that these platforms were appropriate for men, as men were more comfortable to discuss issues of sexuality with their male peers in such gatherings. In addition, male FGD participants felt that JHU HIV Communication Programme’s interventions targeting male norms filled information gaps in their communities where young men were often raised by single mothers and had few male role models. These gatherings became a significant source of information on the expected norms for men. • Addressed gender norms: Male beneficiaries who participated in the FGDs reported a change in the way in which they relate to their female partners following receipt of Brothers for Life campaigns. This included more open communication regarding sexual intercourse with their partners, better self-awareness as a man that encouraged better relationships and discussion. Some beneficiaries reported improved attitudes with regard to their female partner requesting condom use or carrying condoms. In addition, male participants in FGDs indicated that Intersexions had assisted them to better understand abusive behavior particularly towards women and children. • Changed behaviors: Review of research evaluations conducted by JHU HIV Communication Programme on the impact of Brothers for Life 10supported the responses of male beneficiaries in the FGDs who expressed significant changes in their behavior following exposure to both mass media and community based interventions. The main changes in reported behaviors included: correct use of condoms, seeking MMC services and improved communication with female partners. JHU HIV Communication Programme’s partners have also distributed more than 17,185,748 male condoms and 386,849 female condoms in total, which reflects the national challenges in the provision of female condoms. Educating oneself about the role of the father in the family unit, improvement in self-awareness as a male and knowledge regarding health issues affecting men (e.g., prostate cancer) were cited as significant changes among men who were interviewed. • Addressed stigma related to MSM: The evaluation noted that responses from FGD participants who had viewed Intersexions indicated a more positive attitude towards men who have sex with men (MSM). As one participant stated, “I have learnt to be accepting and tolerant.” Conclusion: JHU HIV Communication Programme developed successful health communications to target gender 34 norms among men and women. Brothers for Life is an innovative campaign that uses multimedia platforms and has demonstrated successes among men with extended benefits for their women partners. Recommendations: • Scale up of Brothers for Life Model. Future programming should continue to expand Brothers for Life particularly among in- and out-of-school youth and older men. The campaign’s model can be used to also address the needs of MSM as well as the challenges pertaining to rape and sexual violence. • In addition to addressing the needs of men who have sex with men, the issue of “corrective rape”, an act that can lead to unwanted pregnancy and HIV transmission, particularly targeting women who have sex with women, can be further addressed by engaging with communities and particularly men. Has JHU HIV Communication Programme implemented the most appropriate strategies to reach its target population? In assessing the appropriateness of JHU HIV Communication Programme’s strategies to reach its target population, the evaluation first examined the reach of various interventions. Then we examined whether the strategies chosen were based on thorough situation analyses and explicit, recorded analyses of the advantages and disadvantages of possible alternative ways of addressing specific problems and accomplishing particular objectives. The evaluation’s approach was to also assess whether the strategies were adapted to different situational contexts and took into account issues such as reach of the interventions, the causes of the HIV epidemic, the current health status of the population and their health risks, alignment with national health policies and international standards, promotion of any protection and human rights issues, monitoring of any potential unintended, negative effects that particular strategies might have, cultural sensitivity; and accessibility of services accessible equitably to all population groups including marginalized and disadvantaged groups (disabled people, minorities, etc.). Findings: • Large numbers of people reached through mass media: Through key partners such as SABC, JHU HIV Communication Programme strategically placed Intersexions during TV prime times with high viewership (e.g., following Generations, the most popular television show in the country). This resulted in large numbers of target audiences being reached. A review of the Television Audience Measurement System (TAMS), 11 designed to measure national television audiences in private households with television and mains electricity, indicated that Intersexions reached six million of its intended target audience after airing of 26 episodes rebroadcast in 2012. In addition, Scrutinise and Brothers for Life utilised TV adverts/spots (Table 1) as well as radio spots to target mainly in- and out-of-school youth and men. Between 2009 and 2011, Brothers for Life was able to reach 29 million people and Scrutinize 32 million people through TV spots. Table 2. Brother for Life and Scrutinise TV spots We found that despite more episodes being broadcast 4Play’s reach was limited possibly due to lower viewership on eTV in comparison to SABC1.The decision to air on eTV was informed by Campaign Number of spots People reached Brothers for Life 2 434 spots 29 million people Scrutinize 4 505 spots 32 million people 35 delays for approvals by SABC1. • Access to mass media restricted in rural areas: While the national reach of the mass media campaigns resulted in many South Africans benefitting from the HIV prevention messages, community mobilization activities is primarily in urban and peri-urban communities and appropriately prioritizing two provinces viz. KwaZulu-Natal and Gauteng which jointly have a high burden of disease. KZN houses all five of the districts with the highest HIV prevalence viz. Umkhanyakude, Ethekwini, Umgungundlovu, iLembe and Ugu. A finding of the NCS 2012 indicated that location in tribal settlements, KZN, Eastern Cape, NorthWest and Northern Cape provinces was negatively related to exposure to health communication programs among men and women ages 16-55 years who had had sex in the previous twelve months. While some of JHU HIV Communication Programme’s partners conducted activities in these districts, expansion of JHU HIV Communication Programme’s community mobilization coverage into these districts particularly more rural communities where services can be provided could greatly benefit more rural communities. Many FGD participants, particularly in rural areas, indicated that their utilization of social media platforms was low due to limited internet access. • Fewer beneficiaries reached through community mobilization than originally planned for: The community mobilisation partners reached 12,227,418 and 7,990,970 people through individual and group activities in sexual prevention (Figure 8) and Abstinence and Be Faithful activities respectively. Targets for both these program areas had been progressively revised down significantly from FY2010. • Use of mass media alone can be limiting: Mass media reached a significant number of people with HIV prevention messages. However, beneficiaries stated that while mass media campaigns were effective in giving them information and creating awareness, they felt that Illustrative Social media Achievements- Brothers for Life (since 2009) • Cellphone technology has 65,000 subscribers • Email subscription has 14,200 subscribers • Facebook account followers: The Group (3,228) and the Page (1,700) • Twitter Profile: 200 followers • Information resources downloads page (8398) • Website : 146 000 pageviews 0 500,000 1,000,000 1,500,000 2,000,000 2,500,000 3,000,000 3,500,000 4,000,000 4,500,000 Q4 COP07 COP08 COP09 COP10 COP11 Figure 8. Number reached through community outreach ( Sexual Prevention) [Source, JHHESA] Sexual Prevention Male Sexual Prevention Female Sexual Prevention Small group Sexual Prevention Large Group 36 community based activities provided additional benefits such as providing opportunities to ask questions. Communities also felt familiar with the campaign staff who were typically community members. One beneficiary stated: “If you read you cannot ask questions if you want more information, and there are some words we do not understand.” FGD participants felt that individual engagement with the community mobilisation activities allowed for practical demonstration of how to perform certain activities (e.g., using a condom) something that cannot be adequately done through mass media such as TV. Another participant said: “It is hard to remember all the information even the steps or the ways to use condoms; for instance tear up the condoms with the teeth. We have to tear with the fingers.” • Need for interventions to address cultural practices: JHU HIV Communication Programme significantly matched intervention materials to its target audience by producing materials and mass media in numerous indigenous South African languages, in addition to English, thereby increasing access to information for the beneficiaries. However, there is a need to extend culturally sensitive approaches beyond multilingualism. This includes deeper cultural, social and historical forces that influence health behaviour in the target population. Beneficiaries who had viewed Intersexions noted the dichotomous approach to ARVs and traditional African medicine, a finding also noted in an independent evaluation report conducted on behalf of JHU HIV Communication Programme. Given the widespread use of traditional medicine in its target audience and the role of traditional health practitioners, JHU HIV Communication Programme should identify the involvement of the latter as possible advocates for its key messages as well and provide clear information on the use of traditional medicines in the context of HIV treatment. FGDs with communities identified a need for the mass media productions to more comprehensively address traditional practices that particularly affect rural communities, as for example ukuthwala (forced marriage), virginity testing, and polygamy. This type of communication could be particularly addressed to older, rural women and men. Beneficiaries interviewed in focus groups also expressed that women would greatly benefit from interventions similar to Brothers for Life. In response to the successes of Brothers for Life JHU HIV Communication Programme supported the DOH to develop a draft of a national Women and Girls campaign strategy to address women's sexual and reproductive health entitled “Our future, my rights, our choice”. • Materials were produced for those with impairments: The JHU HIV Communication Programme ensured access to its mass media and community mobilization content through its production of materials in Braille for individuals with visual impairment and blindness. Drama series were subtitled and incorporated sign language to accommodate deaf individuals. • Interventions may not have been targeted sufficiently to different races: While JHU HIV Communication Programme extensively used minorities (e.g., White, Indian and Coloured people) in its drama series particularly Intersexions, targeted interventions beyond mass media, such as outreach, may be required to promote greater participation of these races in HIV prevention activities e.g. HIV testing and exposure to health communication programs, a need that was identified in the NCS 2012. • Need for greater in-depth formative research, audience analysis, and pre-testing for language : JHU HIV Communication Programme implemented innovative and rigorous research methods to ascertain the causes of the HIV epidemic and related health risks. The P- 37 process used by JHU HIV Communication Programme helped ensure a comprehensive process of conducting situational analyses, audience analyses, strategic intervention design, testing and implementation monitoring to address HIV risk behaviors resulting in strategies that would be acceptable the communities in which they worked. Despite this, FGD participants indicated that the use of actors who are known to be urban, the indigenous language inaccuracies and mispronunciations should be addressed. While participants indicated that Intersexions portrayed their communities accurately in relation to high risk behaviors, they highlighted a need to portrayal of more positive role models and “low – risk” characters that they could identify with. JHU HIV Communication Programme’s NCS evaluation also recommended an approach that highlights the positive alternative to MCP rather than focus on the negative consequences. • Innovative use of multimedia platforms reinforce key messages to target audiences: Participants in focus group discussions indicated they derive information from a number of different sources including health care providers. Health workers were found to be an important source of information and clarification of information received through mass media. JHU HIV Communication Programme’s partnership with Mindset built the capacity of health care providers in the necessary technical clinical competences. • No clear strategy for monitoring of any potential unintended negative effects of strategies: While the key informants from organizations supported by JHU HIV Communication Programme indicated no known unintended or negative effects of the interventions, our evaluation found that there was no documented strategy or system for identifying, monitoring or reporting of such. These unintended or negative effects e.g. misconceptions, misinterpretation of health communication messages and behavioral disinhibition following MMC. While JHU HIV Communication Programme’s materials provided correct information on the window period for rapid HIV tests following HIV testing, these messages were not necessarily correctly internalized or understood by target audiences. For example, many FGD participants reported that HIV testing should be done every three months. The HCT national guidelines stipulate the frequency of testing is based on the individual’s risk and recommends annual HIV testing, an area that can be further developed in communication programs pertaining to HCT. In addition, we found that there was stereotyping regarding the impact of HIV in rural communities when compared to urban communities. Participants in FGDs indicated that there are misconceptions in the communities with HIV being regarded as a primarily urban disease associated with urban lifestyles and the stereotyping or rural women as naïve and not engaging in high risk behaviors. • Strategies promoted protection and human rights issues: Review of project data indicates that JHU HIV Communication Programme’s project is underpinned by a human rights approach in its communications. Human rights issues were noted in Intersexions that addressed issues such as xenophobia and right to housing. As previously described JHU HIV Communication Programme’s partners play significant roles in highlighting violence against women and children. The use of memorable key messages such as “Violence is not an Act of Love” and “No one deserves to be abused” ensure simplicity of the message for the target Figure 9. P-process [Source: JHU HIV Communication Programme] 38 audience. Participants from FGDs indicated their concerns about stigma that they believe still exists within their communities an area that should continue to be highlighted in future HIV communication programs. Conclusion: JHU HIV Communication Programme reached a substantial number of its target audience through mass media and community mobilization activities with the latter being more interactive. JHU HIV Communication Programme’s interventions reinforce key messages and promote protection and human rights issues. Unmet needs in JHU HIV Communication Programme’s interventions include: better approaches to address cultural practices such as traditional medicine, ukuthwala (forced marriage) and polygamy in the context of HIV prevention. Despite the research and pretesting of messages potential unintended negative effects of campaigns was not routinely monitored. Recommendation: • Place mass media content on TV channels with high viewership. Judicious use of resources would dictate that placement of mass media campaigns in the future should prioritise channels that have the largest numbers of viewers (e.g., SABC in lieu of eTV, the latter having less accessibility and viewership). Future projects should focus on a combination of short TV and radio spots, which allow for greater frequency in broadcasting and a higher reach when compared to TV dramas. • More focus on cultural and racial context. Future HIV communication programs should integrate more messages into their programming that take into account the role of prevalent cultural practices (e.g., traditional medicine , polygamy and ukuthwala) to address cultural norms and practices, primarily of rural communities. Programming should also address the needs of other minority race groups and identify mechanisms to increase their exposure to health communications. • Radio is an important, participatory channel for engaging target audiences. Radio complements TV shows by extending dialogue pertaining to key messages. While a multipronged, multi-channeled approach is important, future program designs should be cognizant of the limitations of social media for rural audiences and should tailor channels that are more accessible for rural audiences. To what extent has JHU HIV Communication Programme addressed the key drivers of the epidemic? JHU HIV Communication Programme’s original Cooperative Agreement included “developing an evidence-based multilevel and multi-platform mass media effort that addresses the key drivers of the epidemic in South Africa, in particular heightened perception of risk in relation to sexual partnerships and behaviors” as a key activity. 3A review of JHU HIV Communication Programme’s workplans demonstrated a synergistic, complementary and sustained approach in JHU HIV Communication Programme’s strategy to address key drivers of the epidemic in multiple platforms. JHU HIV Communication Programme’s health communication process to develop key messages addressing the drivers of HIV epidemic commence with a well-defined research method (the P-process) the results of which informs synergistic messaging in its campaigns (See Table 3). 39 Findings: • Addressed multiple concurrent partners: JHU HIV Communication Programme’s mass media campaign, Intersexions, addressed to a large extent multiple concurrent partners by demonstrating the movement of HIV infection through a sexual network of different characters and their stories. JHU HIV Communication Programme conducted a post-broadcast qualitative evaluation of the series in six provinces between April - June 2011 comprising focus groups and individual in-depth interviews. The research showed that Intersexions increased participants’ knowledge and perception of risk to HIV infection, a finding also confirmed by our evaluation. Social mobilization campaigns, particularly Brothers for Life, resonated with men with some FGD participants reporting that they reduced the number of partners as a result of the interventions. One participant added “Reducing the number of partners does reduce the chances of getting HIV. But it goes both ways.” • Addressed condom use: The JHU HIV Communication Programme successfully integrated condom use in its mass media and interpersonal communications interventions. The project’s own post broadcast evaluation showed that participants articulated the importance of using condoms with their sexual partners. Our FGDs revealed similar results. A participant commented: “Even, when I had a condom, there were certain girls that just by looking at them, I just knew I would not be using it. Intersexions taught me never to do that. ” “They [Intersexions] were preaching condomize. A lot.” Participants also indicated that information regarding acute HIV infection was added value and an issue which many were unaware of prior to viewing Intersexions. • Addressed MMC: Evaluation findings noted that some participants reported an increase in MMC in the communities with some health facilities implementing MMC ‘camps’. Through its relationships with other organizations such as Centre for HIV and AIDS Prevention Studies (CHAPS) and Catholic Medical Mission Board (CMMB), JHU HIV Communication Programme’s Table 3. Key drivers of HIV Epidemic, by campaign [Source, JHU HIV Communication Programme] 40 partners have built the linkages with organizations that provide MMC clinical services. Many beneficiaries reported a better understanding of MMC following information received as part of Brothers of Life campaign (e.g., that unlike traditional circumcision which is conducted primarily during the winter months MMC can be accessed at any time of the year including summer). As previously alluded to, more needs to be done to increase uptake of MMC. • Alcohol and Drug Use Findings of this evaluation were contradictory from the FGD participants in relation to alcohol and drugs. Whilst participants of the FGDs acknowledged that Intersexions increased awareness and perception of the relationship between alcohol and multiple partners and the risk of HIV acquisition, as was also identified in JHESSA’s post broadcast evaluation, some participants changed their behavior after viewing the show, and others did not. One participant commented: “For me, my clubbing habits changed. I used to go clubbing and I was one of those who went home with women I had never met before. Alcohol is the main issue. If I think about it I was not the only one who used to do it. Many of my peers did that. The series changed me a lot. You know when it comes to clubbing there are two things: clubbing equals sex and money – you give me money, I give you sex! Even in rural areas clubbing happens in taverns. Locally there is a popular tavern that people from even cities visit, because there are a lot of younger girls there.” Another participant indicated a contradictory view with respect to the impact of Intersexions on alcohol use: “Like you watch and you’re like, yho [exclamation] and then the next thing you’re like, okay, let’s go and drink.” In addition, JHU HIV Communication Programme’s NCS 2012 results also highlighted the necessity for heavy drinking to be addressed as it was associated with reduced chances of HIV testing. • Health communication programs had impact on knowledge, uptake of HCT, MMC, and condom use: The National Communication Surveys (2006, 2009, and 2012) implemented by JHU HIV Communication Programme was a significant achievement of the project as a tool to understand the key drivers of HIV epidemic nationally. The aim of the NCS was to “strengthen HIV communication programmes so that they are strategically aligned to important risk behaviors.” The results of the survey (Table 4) are utilized to inform JHU HIV Communication Programme’s mass media and community mobilization interventions. The evaluation noted that this survey was the first of its kind to be conducted at a national level anywhere in the world and reached a representative sample of 10,034 people in 2012. Table 4: Key Findings of Comparison of NCS 2006, 2009 & 2012 [Source: JHU HIV COMMUNICATION PROGRAMME] 41 Conclusion: The major components of JHU HIV Communication Programme’s program address the key drivers of the epidemic and activities were strategically aligned to the important risk factors as stipulated in the cooperative agreement. A key gap that was identified as requiring more innovative interventions is alcohol and drug abuse. Recommendations: • Address alcohol and drug abuse. More interventions are required to target alcohol and drug use in the context of HIV prevention. Health communication programs particularly social mobilization activities should scale up interventions and identify linkages with organization such as SANCA to facilitate referrals for excessive alcohol use. The role of drugs (e.g., marijuana and nyaope) and HIV risk behaviors should be explored as well. QUESTION 3: ENGAGEMENT WITH THE DIVERSE STAKEHOLDERS IN SOUTH AFRICA This section responds to evaluation question 3 which stated: “How has JHU HIV Communication Programme’s engagement with diverse stakeholders in South Africa affected the performance of the project?” To analyze this question, URSA investigated: JHU HIV Communication Programme’s coordination with and technical assistance to the South African Government (DOH, DBE, DSD, SANAC-Communications Technical Task Team); coordination with and technical assistance to USG implementing partners and local organizations; and partnerships with the private sector (e.g., Levi’s, SABC). Coordination with and technical assistance to South African Government JHU HIV Communication Programme’s Cooperative Agreement indicates that a key project activity was to provide technical assistance to the SAG by engaging national, provincial, community, political, and religious policy and decision-makers through initiating and attending policy meetings and committee membership at national, provincial and community levels. Findings: • Good coordination with the DOH and SANAC: The JHU HIV Communication Programme built close working relationships with the DOH at national and provincial levels and played a role in ensuring inclusion of health communication in the NSP 2012-2016. Key informants at both levels reported having benefitted a great deal from the technical assistance provided by JHU HIV Communication Programme. Document reviews and interviews with key informants showed that JHU HIV Communication Programme was the key member of the SANAC Communication Task Team that coordinated the communication efforts within South Africa and was a partner in developing SANAC’s Communication Framework and Strategy on Male Circumcision. JHU HIV Communication Programme further assisted the SAG in the development of the communication and social mobilisation components of the NDOH/SANAC Accelerated PMTCT Plan. In addition, JHU HIV Communication Programme participated in the National Medical Male Circumcision Task Team and the SAG/USAID/SA M&E task team, providing technical assistance on the communication strategy, framework and advocacy. In addition, JHU HIV Communication Programme and its partners ensured active involvement of 42 high profile SAG individuals to promote their campaigns. For example, the Brothers for Life campaign was launched on 29 August 2009 by the late Deputy Minister of Health Dr. Molefi Sefularo, in an event attended by 10,000 people. JHU HIV Communication Programme aligned its health communication activities with those of the national DOH campaign (e.g., facilitating a discussion on MMC at the launch of HIV Counseling and Testing Campaign), an event attended by Minister of Health and the Deputy President of the country. While JHU HIV Communication Programme’s role in HIV prevention was substantial, SAG respondents identified unmet needs with regards to TB and TB/HIV integration communication and family planning. JHU HIV Communication Programme’s close relationship with the DOH, as well as having a health communication strategy linked to the national BC strategy, gives implementing NGOs credibility with local leaders, which is critical for interventions involving community mobilisation. Key informants with program managers at national and provincial levels described the relationship as “very positive, passionate, “and project staff as “open to dialogue, to new ideas, and very inclusive in their approach.” • Collaborated with other SAG Departments: Through its partners JHU HIV Communication Programme extended its scope beyond traditional government departments engaged in healthcare and supported the Gauteng Department of Roads and Transport (GDRT) by participating in a Gender Based Violence (GBV) awareness event for their management and employees. As part of it program, JHU HIV Communication Programme’s partners have engaged Department of Correctional Services ( DCS) and developed activities targeting men in correctional services with HIV prevention activities. Challenges to working with SAG included delayed signing of Memorandum of Understanding (MOU)s with some provinces and difficulties in building relationships with DOE and DSD despite reported attempts from JHU HIV Communication Programme to do so. • Fulfilled important role in informing health communication policy: Interviewed SAG key informants described JHU HIV Communication Programme’s contribution to policies and public awareness in HIV prevention in South Africa as significant. “There would be much more silence in the country [around HIV prevention] if they had not been here.” The NCS was particularly cited as a useful contribution and tool to policy makers. • Capacity building and sustainability of health communications needs to be augmented: Key informants noted significant capacity building contribution by JHU HIV Communication Programme to health communications in South Africa. Key informants indicated that the relationship was collaborative, added value, built the capacity of the government departments and was aligned to the objectives of the Department of Health. The low scale of communication programs developed independently by the SAG identifies a need for JHU HIV Communication Programme to build the capacity within the SAG at all levels for a sustainable impact beyond the life of the project to meet the country’s needs. An additional need is for interventions targeting the SAG health care workers to promote service delivery. A SAG key informant indicated: “For men, we need to understand what the access barriers to seeking care for HIV are.” The SAG identified linkages with care, support and treatment services as a gap that has not been adequately addressed in the partnership with JHU HIV Communication Programme with prevention and treatment programs operating in silos. While the SAG acknowledged ownership and adequate resources being available for the health communication programs nationally and provincially, there was concern regarding the sustainability of JHU HIV Communication Programme’s activities at the current scale, particularly for the NCS, which was viewed as a valuable by costly exercise. It was also noted that the SAG does not have a 43 contingency plan to take over JHU HIV Communication Programme activities in the event that JHU HIV Communication Programme’s project not be funded for a new cycle. Conclusion: JHU HIV Communication Programme played a key role in advocacy and supporting the DOH and SANAC in health communications campaigns although relationships with DOE and DSD were not well established. As a key partner JHU HIV Communication Programme contributed to key policies at national and provincial levels. Recommendation: • Increase capacity building activities. Investments in capacity building particularly of the SAG structures and departments tasked with health communications should be implemented to promote sustainability. Activities should aim to build a critical mass of people within the SAG with the capacity to manage, implement, and sustain health communications programs at the scale initiated by JHU HIV Communication Programme. Coordination with and technical assistance to USG implementing partners and local organizations and partnerships with the private sector JHU HIV Communication Programme provided significant technical assistance to its implementing partners and other local organisations. It conducted a comprehensive capacity building program among its sub-partners that included training (Table 5), and onsite mentoring and coaching by JHU HIV Communication Programme’s program manager. Each partner was assigned a program manager who conducted site visits, capacity building and mentored the organizations on the use of the resources and tools. Some key informants from community mobilization partners indicated that while this type of technical assistance was valuable they felt that the program managers were overstretched in their duties due to the demands of their roles. Findings: • Capacity building benefitted partners. The JHU HIV Communication Programme conducted extensive training as part of it capacity building activities and provision of technical assistance in support of partners with more than 5 million individuals trained particularly in treatment literacy (Table 5). Organizational development and technical assistance to develop behavior change communication strategies were cited as the most important contributions of JHU HIV Communication Programme to its partners. Community mobilization partners, compared to content development partners, found that the technical assistance provided on training and capacity building to be more beneficial. To note is that content development partners, however, benefitted less from financial support. Content development partners reported Training Program Q4 COP07 COP08 COP09 COP10 COP11 Total Sexual Prevention 24,472 17,931 12,221 15,979 70,603 Gender 1,402 7,742 9,144 PMTCT 2,573 3,716 8,626 11,365 26,280 VMMC 3,146 7,863 13,725 24,734 HCT 159 5,333 22,378 7,008 10,358 45,236 ART 106 3,952 143 4,201 TB/HIV 159 2,647 6,040 8,846 Treatment literacy 843,009 1,688,272 516,001 1,768,167 233,286 5,048,735 OVC 1,898 5,759 4,215 705 12,577 Table 5. Capacity building by Program [ Source: JHHESA] 44 mostly having benefitted from the research and development activities supported by JHU HIV Communication Programme. Key informants from the SABC and production company partners indicated that the SABC was able to fulfill its mandate as national broadcaster to build the skills and capacity within the industry as a result of the capacity building support from JHU HIV Communication Programme. This included; increased training opportunities for actors, producers, directors, cameramen and lighting crew trained on the project. The SABC was also able to strengthen a long standing relationship with DOH as a result of the partnership. The JHU HIV Communication Programme’s approach also benefitted another USAID/SA supported education show, Takalani Sesame, where skills learned through the partnership were utilized to successfully contextualize HIV for younger viewers. • Leveraged non-USAID/PEPFAR media resources. JHU HIV Communication Programme’s partnership with the SABC Education has been critical primarily in the success of the mass media campaigns. As previously stated, JHU HIV Communication Programme’s expenditure is largely directed towards mass media content. However, the project managed to leverage non￾USAID/SA /PEPFAR funding by sourcing significant financial contributions in air time from the SABC Education, a key media stakeholder and a SABC business unit of the national broadcaster. As a division of the national broadcaster SABC Education was mandated to create awareness among South African regarding HIV prevention and has been engaged in numerous productions to this end. JHU HIV Communication Programme’s biggest contribution towards SABC was cited by key informants as financially contributing towards production costs resulting in high quality content that has earned high viewership, an achievement unprecedented for the education division and built the capacity to develop educational dramas. For example, SABC successfully applied the approach and lessons learnt through Intersexions, in the development in other educational dramas with unrelated topics (e.g. for development of a series on democracy education and parliamentary education). Conclusion: JHU HIV Communication Programme significantly built up the capacity of other USG implementing partners, local organizations and private sector partners. Strategic partnerships allowed JHU HIV Communication Programme to leverage non-USAID/SA/PEPFAR resources to meet its objectives. Recommendation: • Increase scope of PPP funding. In subsequent projects the scope of partners should be increased to augment participation of the private sector (e.g., in the workplace to support health communication programs). This will assist the next HIV communication project to accomplish HIV/AIDS prevention, care, and treatment goals and help ensure sustainability of programs, facilitate scale-up of interventions, and leverage private-sector cash and in-kind resources. QUESTION 4: EFFECT OF OVERALL STRUCTURE AND MANAGEMENT OF JHU HIV COMMUNICATION PROGRAMME ON PERFORMANCE Sub-grantee model and effect on performance and sustainability of the HIV response As previously mentioned, JHU HIV Communication Programme’s approach combined the reach, 45 education and entertainment value of mass media with the interpersonal communications and community mobilization activities to bring about a synergy in the activities of the partners. Engaging various local partners, each with its own unique expertise, has resulted in complementary approaches that are synergistic and have benefitted the performance of the project. Findings: • Valuable TA provided to sub-partners: Through an annual partners’ meeting JHU HIV Communication Programme created a common platform for all its partners to receive the projects’ key messages and identify gaps. The meetings allowed partners to share activity progress, identify best practices and evaluate progress against project goals. JHU HIV Communication Programme’s technical assistance to partners included assistance with the development of strategic plans, budgets, scopes of work, training material and script reviews and on-site consultations. In addition, assistance to partners to develop outcome based goals, objectives and realistic work plans was provided. JHU HIV Communication Programme also conducted monitoring workshops and random data quality audits which aimed to improve collection and collation methods. Moreover, JHU HIV Communication Programme provided significant support to build M&E capacity through the provision of program implementation and monitoring tools, technical support to data officers, funding of Monitoring and Evalulation (M&E) posts and networking and training opportunities. • Funding for health communications scaled up existing activities: The community mobilization organizations that were interviewed indicated that prior to JHU HIV Communication Programme’s support HIV prevention services for Voluntary Counseling and Testing (VCT) and PMTCT programs were already in place and were funded by other sources (mainly DOH and private sectors- e.g., mines). However, the JHU HIV Communication Programme funding allowed expansion and scale up of these pre-existing services in order to reach more people enabled organizations to add health communication components to pre￾existing work. • Need for a grant application system: No formal competitive sub-grantee application system exists, a statement corroborated by the sub-grantees. Partners are identified to meet the need of the program and largely through recommendations and professional networks. If JHU HIV Communication Programme is to expand its scope and reach of the project, or a project like JHESSA were to be implemented, it would benefit by developing a more formal systematic grant application system that will allow it to identify more partners who may better fit in with the objectives and needs of the program in a transparent manner. • Sustainability of local organizations work post-project questionable: With regard to sustainability of the HIV response, community mobilization organizations indicated that the delivery of direct services (mainly VCT and ART treatment adherence) would not continue at the same level. Capacity building mainly targeted sub-partner staff and there was limited inclusion of district staff to ensure that skills to manage, support and monitor activities are institutionalized and can be sustained after funding for implementing partners ends at district level. JHU HIV Communication Programme’s interventions have created a degree of dependency particularly with the social mobilization organizations, all of whom reported that current JHU HIV Communication Programme funded activities would end in the event that JHU HIV Communication Programme is unable to continue funding. 46 Strengthening the capacity of local organizations supported through JHU HIV Communication Programme The JHU HIV Communication Programme placed emphasis on building the capacity of local organizations it supports. All sub-grantees cited that JHU HIV Communication Programme’s contribution to capacity building in terms of organizational development, financial management, training and capacity building, research and evaluation and development of social and BCC strategies benefitted them a great deal (see Figure 10). • Capacity built in monitoring and evaluation: JHU HIV Communication Programme partnered with organizations such as the Centre for AIDS Development, Research and Evaluation (CADRE), Health and Development Africa (HDA) and Cell-Life to develop and design effective monitoring systems and evaluations of its programs including the development and training of JHU HIV Communication Programme partners on the use of on-line monitoring systems. A detailed M&E plan, linked to the JHU HIV Communication Programme’s national M&E system, was developed which had appropriate output, outcome and impact indicators and targets. As previously mentioned, the JHU HIV Communication Programme also assisted partners through workshops on PEPFAR monitoring requirements and changes to indicator descriptions, coupled with on-going field support through site visits and telephone support. Sub-partners indicated that among the benefits that JHU HIV Communication Programme provided, feedback from research conducted by other JHU HIV Communication Programme partners assisted in evaluating their own impact. For example, a research partner provided technical assistance to JHU HIV Communication Programme partners on mid- and end￾point evaluation activities and aided them in the development of Terms of Reference for research consultants as well as provided feedback and reviews of research reports that were developed. • Capacity strengthened of communication training institutions: A significant contribution of JHU HIV Communication Programme’s project was to strengthen the capacity for evidence based strategic communication interventions by supporting two post￾graduate interventions. The University of Kwa-Zulu￾Natal offers an Entertainment-Education course as part Illustrative book publications supported by JHU HIV Communication Programme • Meyer, M & Struthers, H (eds) (2012) (Un) Covering Men: Rewriting Masculinity and health in South Africa. Jacana Media: Auckland Park. • Durden E & Govender E (eds) (2012) Investigating Communication, Health and Development. 10 Years of Research in The Centre for Communication, Media and Society (CCMS). • Tomaselli K & Chasi C (eds) (2011) Development and Health Communication. Pearson Education South Africa : Cape Town Figure 10: Extent of the benefits organizations get from linkages from JHHESA – community mobilization and content development sub-partners 47 of its graduate Degrees Program in Communication, Cultural and Media Studies, a course which explores effective ways to design, implement, monitor and evaluate strategic communication programs that promote health, HIV prevention, care and support. Students participating in the course undertook research projects that analyzed existing communication interventions. The JHU HIV Communication Programme also contributed to strengthening the capacity of tertiary education institutions through the HIV & AIDS Media project, which is jointly managed by the Anova Health Institute and the Wits Journalism Programme. The aim of the project included making information accessible to a wide range of journalists and media practitioners, skills building among the media as a whole, improved networking between civil society, medical professionals and the media, and providing feedback to the media via monitoring and analysis of coverage. This was achieved in activities such as annual research and journalism fellowships by: making information accessible to journalists and building their skills; promoting partnerships and networking between medical professionals and the media; and engaging with the news media through monitoring and media analysis particularly as it related to HIV. Linkages with Johns Hopkins University-Center for Communications Programs JHU HIV Communication Programme has benefitted a great deal from the linkage with JHU-CCP in terms of organizational development. The JHHESA board accommodates three members from JHU, with the Chief Executive Officer (CEO) reporting to the board. JHU-CCP also provides overall oversight and contributes to governance of JHHESA and its project activities and support in the contractual management of the project. Key informants from sub-grantees and partners indicated that delays in the disbursement of funds at times have taken up to four months and resulted in adverse effects on program implementation such as reduction in reach figures and not being able to expand program activities. An example cited included suspension of radio programs meant to be flighted in synchrony with TV shows pending funding availability. This resulted in a disjointed product being aired and sub-grantees having to operate on minimum resources pending the finalization of the contractual arrangements between JHU-CCP, JHHESA and USAID/SA. This finding was explained by the program managers as due to “a delay in the signature of the agreement by South African authorities that have resulted in a delay in the issuing of the modification from USAID/SA or in delays at the USAID/SA side.” Conclusion: JHU HIV Communication Programme’s sub-grantee approach had a largely favorable effect on performance of the HIV response. Through mechanisms such as partner’s meetings and capacity building activities, JHU HIV Communication Programme built mechanisms within organizations to support sustainability, although sub-grantees indicated that activities would not continue at the same scale without JHU HIV Communication Programme’s support. In addition, JHU HIV Communication Programme built institutional capacity at tertiary institutions by supporting post-graduate education which yields important BCC capacity building among learners. While overall the linkage with Johns Hopkins University –Center for Communications Programs has been beneficial, delays in disbursements of funds to JHU HIV Communication Programme occasionally affected programs adversely. Recommendations: • Develop sub-grantee application system: A follow on communication project would benefit from a more formal sub-grantee application system, which allows more and a wider array of potential partners to compete for grants within a framework that promotes transparency. 48 • Ensure involvement and capacity building of district health staff in planning and budgeting for health communication. More effort should made to promote sustainability of communication activities at the end of the project by building capacity at district level • Continue working with training institutions to build country ownership and sustainability of behavior change communication activities. 49 This report was commissioned by U.S. Agency for International Development 1300 Pennsylvania Avenue, NW Washington, DC 20523 1 UNAIDS. Global AIDS Response Progress Report 2012 Republic of South Africa. 2 USAID- JHU Cooperative Agreement 3 JHU Programme in South Africa Strategic Plan 2008--‐2013 4 National Strategic Plan 2011-2016 5 Infections Averted Report 6 National Communication Survey 2006 7 Presentation from Richard Delate 8 World Health Orgamisation definition 9 National Communication Survey Ministry’s briefing _9 July 2012 10 JHU HIV Communication Programme Impact Sept12_2012_Scrut-Brothers 141212. Presentation by D. Lawrence Kincaid and Maria Elena Figueroa 11 M&E Documents ANNEXES ANNEX I: EVALUATION STATEMENT OF WORK SECTION C – DESCRIPTION/SPECIFICATIONS/STATEMENT OF WORK 1. Background Development Problem: At the project’s inception, USAID/South Africa’s Health and HIV/AIDS Strategy was responding to the overwhelming challenges posed by the epidemic on individuals, families, communities and society in South Africa. There had been a dramatic rise in HIV infections during the previous decade threatening to undermine many of the advances made since efforts to transform the sector began in 1994. During the fifteen years prior to the project, HIV infection rates among pregnant women in antenatal clinics went from less than one percent (in 1990) to over 30 percent (in 2005). The South African National Department of Health estimated that about five million, or one in ten South Africans, were infected with HIV. This was more than any other country in the world, and each day, more than 1,700 additional people became infected. In 2006, the South African Government declared Tuberculosis (TB) a crisis, which became exacerbated by the emergence of extremely drug resistant TB (XDR-TB). Development Opportunities: The South African Government (SAG) has committed significant financial and institutional resources to transforming the public services to meet the challenges of the HIV/AIDS epidemic and TB. Beginning even a few years before the project, the health budget has increased dramatically and the scale of transformation at district level has proceeded with high institutional and donor support. In many respects, the South African approach to the epidemic had been recognized as being among the most comprehensive programs in the world. However, as the HIV/AIDS and TB continued, there was an increasing demand for a strategic, coordinated approach to the epidemic and integration of quality HIV/AIDS and TB services into the primary health care (PHC) system which was determined the most effective vehicle through which to deliver these services in South Africa. Meanwhile, USAID/South Africa programs aimed to achieve South Africa’s President’s Emergency Plan for AIDS Relief (PEPFAR) targets – 500,000 HIV positive people under treatment; 1.8 million HIV infections averted; and two million HIV/AIDS affected people receiving care and support. The Mission’s PEPFAR and Health strategy focused on activities which leveraged USAID’s core competencies in technical assistance, public-private partnerships, systems strengthening, and identifying and testing “best practices” that could be taken to scale. As a result, the approach of USAID/SA’s program has focused on HIV/AIDS and TB services. 1 By taking this approach, USAID identified important opportunities to leverage both the strong desire on the part of the SAG for a comprehensive HIV response program as well as significant investments made by the host country and the donors to address HIV/AIDS and TB. Target Areas: USAID/JHU HIV COMMUNICATION PROGRAMME has a nationwide reach, but has concentrated efforts in Kwa-Zulu Natal, Gauteng, Mpumalanga, and Free State provinces. Within these provinces, activities are further geographically targeted in high transmission areas including mining, farming, and informal settlements. USAID/JHU HIV COMMUNICATION PROGRAMME utilizes market segmentation and gender-responsive targeted interventions to reach each of its target populations: in- and out-of-school youth; sex workers, people with HIV, high risk women, and adult men. Project Approach and Implementation Project Approach The Johns Hopkins University Center for Communication Programs (JHU-CCP) is the prime awardee of the cooperative agreement. The primary objective of the project is to mobilize the energy and talents of communities, assist with building their capacities to address social norms related to health practices and help them gain the ability to manage key health issues especially those related to HIV/AIDS and TB. The project has a multi-sectoral approach to assist both USG and the SAG programs in efforts to influence social norms and practices that contribute to improved health behavior. The importance of collaboration and coordination with the SAG cannot be overemphasized. Development of the activities to be implemented through the project focus on the following key principles identified by USAID and the Government of South Africa: 1. Activities are structured and implemented to maximize sustainability. 2. NGO activities supported by USAID should: a. Integrate into the government’s program and/or be complementary to the government’s program. b. Minimize overhead by maximizing resources being applied directly to the grants program. 3. New activities should be assessed through the lens of the short time for completion and sustainability. 4. Focus of activities should be on overcoming the challenges faced in the past. 5. Focus of activities should be on the continued provision of technical assistance consistent with the SAG’s priorities and needs. 6. Maximize the use of locally available expertise. JHU-CCP provides a rapid response mechanism to award grants/contracts to local organizations implementing HIV/AIDS and/or TB activities. The JHU-CCP grant/contract management system 2 provides an array of related activities, starting with a solicitation document, program review, assessment of grant worthiness, negotiation, award, administration, monitoring, reporting, and closeout. In addition, JHU-CCP supports related activities to address institutional capacity building and technical assistance. Utilizing the Pathways to a Health Competent Society conceptual framework, JHU-CCP has worked in three domains: 1) social political environment; 2) service delivery system; and 3) community and individual. Specific activities have included assistance to Government Departments, parastatal organizations, and local NGOs, CBOs and FBOs. JHU-CCP is working with a number of these organizations to expand and strengthen HIV/AIDS and/or TB activities at the community levels. Small grants financing combined with timely technical assistance have demonstrated to be an effective combination for enhancing their capacity and effectiveness. JHU-CCP has also continued to work with NGOs, CBOs and FBOs to build the community response to the epidemic. Major Changes and Project Modifications: Presidential Election In April 2009, Jacob Zuma was elected President of South Africa, which marked an extreme shift in the national HIV response in South Africa. Since his inauguration, the South African Government has increased its investment in HIV/AIDS substantially and scaled up HIV services throughout the country. On December 1, 2011, President Zuma officially launched the National Strategic Plan (NSP) on HIV, STIs, and TB, 2012-2016. The NSP highlights four strategic objectives: addressing social and structural barriers that increase vulnerability to HIV, STI and TB infection; preventing new HIV, TB and STI infections; sustaining health and wellness; and increasing protection of human rights and improving access to justice. Confederation’s and World Cup The Republic of South Africa played host to the Confederation’s Cup and World Cup in 2009 and 2010, respectively. These two events attracted nearly a half a million extra tourists to the country. During and leading up to these events USAID/JHU HIV COMMUNICATION PROGRAMME provided substantial support to the increased HIV prevention efforts. USAID/JHU HIV COMMUNICATION PROGRAMME received an additional one million dollars to implement these activities. PEPFAR Portfolio Re-Alignment Since the beginning of the project, USAID/JHU HIV COMMUNICATION PROGRAMME has received funds to implement communication activities related to several diverse components of the HIV response. As priorities have changed the amount of resources from these different components have also changed. In 2009, the South African PEPFAR program launched a process to revamp and refocus the prevention portfolio. For Country Operational Plan (COP) FY 2009, USAID/JHU HIV COMMUNICATION PROGRAMME and other programs received a 30% budget reduction. At 3 that time they were also guided to focus their programming much more directly on the drivers of the epidemic (e.g., multiple concurrent partners, low condom use, gender-based violence, drug and alcohol abuse, low prevalence of male circumcision). In 2010, the SAG gave the green light for Voluntary Medical Male Circumcision (VMMC); USAID programmed some VMMC funds to USAID/JHU HIV COMMUNICATION PROGRAMME to support communication strategy development and subsequent work around demand creation. On April 1, 2010, the SAG launched a massive HIV Testing and Counseling campaign. USAID/JHU HIV COMMUNICATION PROGRAMME provided substantial support for the campaign with no additional funding. In 2010, as the Orphans and Vulnerable Children (OVC) portfolio was consolidating, the OVC component of USAID/JHU HIV COMMUNICATION PROGRAMME was eliminated. Two USAID/JHU HIV COMMUNICATION PROGRAMME sub-partners (Turntable Trust and The Valley Trust) were moved to an OVC partner. In response to dramatic cuts to the Prevention of Mother to Child Transmission (PMTCT) program, the PMTCT component of USAID/JHU HIV COMMUNICATION PROGRAMME was also eliminated in 2012. Key Personnel Change: On August 16, 2011, Richard Delate replaced Patrick Coleman as the Managing Director of USAID/JHU HIV COMMUNICATION PROGRAMME. Prior to this promotion, Mr. Delate served as the Deputy to Mr. Coleman for four years. The internal promotion ensured a smooth transition and was helpful in promoting continuity. Furthermore, the transition was accompanied by a well thought out plan which also included Mr. Coleman serving in an advisory capacity through the end of the calendar year. The current view of USAID staff is that this transition has not had an impact on project performance. 2. Purpose This final evaluation report shall serve a dual purpose: (1) to learn to what extent the project’s objectives and goals have been achieved; and (2) to inform the design of a new community￾driven HIV prevention project. The life of the current USAID implementing mechanism is scheduled to come to an end in March 2013. While the USAID/JHU HIV COMMUNICATION PROGRAMME project encompasses a diverse set of activities, the evaluation will concentrate its focus on the major level of effort components of the project (e.g., mass media, community mobilization). With PEPFAR/South Africa funding levels decreasing, the follow-on activity needs to integrate the most critical components of the USAID/JHU HIV COMMUNICATION PROGRAMME project in order to maximize performance. 4 The final report will provide concrete recommendations for the Mission’s new community-driven HIV prevention project, outlining critical components to be included in the design. Technical recommendations within the evaluation report will serve as the basis for a concept paper for the new design and form the basis of the project description to be developed for any follow-on project. The final evaluation must answer the following four key questions below related to the development hypothesis; appropriateness of strategies to change gender norms; engagement with diverse partners; and, how the overall structure and management of USAID/JHU HIV COMMUNICATION PROGRAMME affected performance. 1. Does the development hypothesis of the USAID/JHU HIV COMMUNICATION PROGRAMME program relate to the achievement of expected results as articulated in the original scope of work? If not, why not? 2. Has USAID/JHU HIV COMMUNICATION PROGRAMME implemented the most appropriate strategies to change social and gender norms and individual behaviors? If so, how? a. Have the major components (e.g., mass media, community mobilization) been complementary and been able to reinforce key messages to maximize performance? If so, how? b. Have the major components strengthened linkages across the continuum of response (Prevention, Care, and Treatment)? If so, how? c. To what extent has USAID/JHU HIV COMMUNICATION PROGRAMME been able to integrate gender throughout its approach and how has this affected performance? d. Has USAID/JHU HIV COMMUNICATION PROGRAMME implemented the most appropriate strategies to reach its target populations? If so, how? e. To what extent has USAID/JHU HIV COMMUNICATION PROGRAMME addressed the key drivers of the epidemic (multiple concurrent partners, low condom use, drug and alcohol abuse, low prevalence of male circumcision)? 3. How has USAID/JHU HIV COMMUNICATION PROGRAMME’s engagement with the diverse stakeholders in South Africa affected the performance of the project? a. Coordination with and Technical Assistance to the South African Government (Department of Health, Department of Basic Education, Department of Social Development, South Africa National AIDS Council-Communications Technical Task Team) b. Coordination with and Technical Assistance to USG implementing partners and local organizations c. Partnerships with the Private Sector (e.g., Levi’s, South African Broadcasting Corporation (SABC)) 4. To what extent has the overall structure and management of USAID/JHU HIV COMMUNICATION PROGRAMME affected performance? 5 a. How has the sub-grantee model (providing some sub-grants to organizations responsible for content development and other responsible for community mobilization) affected the performance and sustainability of the HIV response? b. How has this model strengthened the capacity of the local organizations supported directly through USAID/JHU HIV COMMUNICATION PROGRAMME? c. How has the linkage with Johns Hopkins University-Center for Communications Programs been a value added? The Contractor shall use the USAID Evaluations Policy (http://transition.usaid.gov/evaluation/USAIDEvaluationPolicy.pdf) and any other relevant information. USAID may require representatives from USAID/Washington, USAID/SA, and DOH to participate as observers in parts of the evaluation and/or travel with the consultant team to site visits 3. Implementation Schedule The table below indicates activities to be performed under the task order. The contractor will substantially follow the implementation/work plan in its proposal (See Attachment 1 for Contractor proposed Implementation Plan). In the event of any conflicts between the task order schedule and the implementation plan, the task order will take precedence. Project Tasks Milestone Week Preparatory activities; in-briefing with USAID/SA, Team planning meeting(s) Briefing meeting held with USAID/SA Week 1 Evaluation schedule; tools development and debriefing; review documents, reports and existing materials; complete protocol, pretest and finalize questionnaires, field work/data collection Evaluation protocol and questionnaires complete Data Collection complete End of week 5 Week 2-5 Analysis of findings and preparation of Draft Report. Draft Report shared with USAID End of week 7 Week 6-7 Additional data collection/clarification of findings as needed Feedback received from USAID Week 8 Revision and oral presentation of the Final Evaluation Report Evaluation Report presented to USAID Week 9 Evaluation Report finalized Final Report submitted Week 10 4. Relationships and responsibilities The Contracting Officer has appointed a TO COR. An alternate TO COR may be named upon award of the task order and the Contractor will be informed if this is the case. The Contracting Officer and the TO COR are the only official representatives of USAID for this contract and are 6 the only ones authorized to provide technical direction to the Contractor throughout the evaluation. The Contractor is expected to work together with the TO COR to implement the scope of work. 5. Logistics A six-day work week is authorized if not in conflict with your organization’s policies regarding work week. Local holidays are not authorized. The evaluation team will be responsible for all off￾shore and in-country logistical support. This includes international and in-country travel (including vehicle rentals), hotel bookings, working/office space, computers, printing and photocopying. The evaluation team, in collaboration with USAID/Southern Africa, will arrange all meetings, interviews, site visits, in-briefing and out-briefing. In all other respects, the evaluation team should be self-sufficient. [End of Section C - Statement of Work] 7 ANNEX II: EVALUATION METHODS AND LIMITATIONS The short time period assigned to conduct the evaluation limited the scope. There was limited observation of actual service delivery which could be considered a gap. Data collected was based on interviewee responses with possible recall biases particularly for mass media communications related information. During the evaluation some of the sites originally identified for evaluation were not evaluated as they no longer existed or fell outside the targeted provinces which might have introduced selection bias. These have been listed in Annexes V and VI. Taking into account the limitations that have been identified, the selection of sites represents more than 60% of USAID/JHU HIV COMMUNICATION PROGRAMME’s sub-partners at the time of the evaluation. 8 ANNEX III: DATA COLLECTION INSTRUMENTS INTERVIEW GUIDE FOR USAID ACTIVITY MANAGER Interviewee: Position: Date: Interviewer(s): The focus of this evaluation is to assess the USAID funded program implemented by JHHESA. We’re interested in learning about your experiences, perspectives, and recommendations for improving this intervention which is why you’ve been asked to participate today. Before we start I would like to remind you that there are no right or wrong answers in this discussion. We are interested in knowing what you think, so please feel free to be frank and honest and to share your point of view. Your comments will NOT be linked to your name or identity in the final report. It is very important that we hear your personal opinion. We will be audio-recording today’s discussion, so we will have a record of what is said. I hope you’ll feel free to speak openly and honestly, as everything that is said in this room will be held completely confidential. I’d like the discussion to be informal, so there’s no need to wait for me to call on you to respond. The interview will take about 60 minutes Your participation in this interview is completely voluntary. Are you willing to be interviewed? YES/NO Do you have any questions before we begin? Page 1 of 12 Interview Guide USAID Activity Manager SECTION I: Organizational structure 1. Please describe the key components the USAID funded JHHESA programme in South Africa? ______________________________________________________________________________ ______________________________________________________________________________ ______________________________________________________________________________ ______________________________________________________________________________ 2. Please describe your role as USAID in supporting JHHESA’s activities in terms of : i. Project management ii. Operations iii. Financing iv. Grants management v. Monitoring , Evaluation and reporting vi. Gender mainstreaming vii. Technical assistance viii. Capacity building 3. In your understanding, how is the JHHESA USAID funded program managed and coordinated in relation to the following areas? a) Project management b) Operations c) Financing d) Grants management e) Monitoring , Evaluation and reporting 4. Briefly describe JHHESA’s contribution to the National Strategic Plan on HIV, STIs and TB 2012 – 2016? ______________________________________________________________________________ ______________________________________________________________________________ ______________________________________________________________________________ ______________________________________________________________________________ Page 2 of 12 Interview Guide USAID Activity Manager 5. Please describe support provided through JHHESA by sub-grantees under HIV prevention services including program targets, populations served, and districts targeted in terms of Program area Program target Population served (PROBE: women, youth ,men, sex workers etc) District i. HCT ii. HIV prevention iii. Treatment care and support iv. OVC 6. Briefly describe the grant management system between JHHESA and sub-grantees under this program in terms of the following: i. The number of sub-grantees supported under this program ii. The selection procedure(s) for organizations for funding support? (Probe: what criteria are used?) iii. Capacity building activities for sub￾grantees iv. Tracking and measuring success or outcomes (Probe : How does the JHHESA grant management system allow you to define, measure, and report on the key outcomes important to your organization) v. Monitoring (Probe :How does the system allow JHHESA to quickly monitor the performance and results for any program, grant, to quickly respond and make adjustments) vi. Budgeting (Probe: How does JHHESA keep track of and create budgets and manage the grant pipeline) vii. Grant application tracking (Probe: How does JHHESA allow faster compilation of application components) Page 3 of 12 Interview Guide USAID Activity Manager 7. In your opinion, to what extent has JHHESA benefitted from the linkage with Johns Hopkins University- Center for Communications Programs in terms of? And why? Not at all Somewhat A great deal Comments i. Organizational development ii. Financial management iii. Training and capacity building (Probe: How does JHU-CCP “develop a cadre of public health professionals who are well-versed in strategic communication.” iv. Knowledge management (Probe: how does JHU-CCP’s K4health assist you to synthesize experiential and scientific knowledge, share information broadly, and encourage local use and adaptation as necessary) v. Research and Evaluation vi. Development of social and behavior change communication strategies SECTION II Strategies to change social and gender norms 1. Describe JHHESA’s role in promoting behavior change communication programs in South Africa? ______________________________________________________________________________ ______________________________________________________________________________ ______________________________________________________________________________ Describe the interpersonal communications, community mobilization and mass media campaigns used by JHHESA? ______________________________________________________________________________ ______________________________________________________________________________ ______________________________________________________________________________ 2. How is consistency in messaging between JHHESA , government and sub-grantees ensured? ______________________________________________________________________________ ______________________________________________________________________________ ______________________________________________________________________________ ______________________________________________________________________________ Page 4 of 12 Interview Guide USAID Activity Manager How is the change measured? (Probe: trend analysis, formative evaluation, summative evaluations, post test assessments) Campaign 1 Campaign 2 Campaign 3 Campaign 4 Knowledge Awareness Attitude Change Behaviour change 3. In your opinion please state if the campaign has improved the knowledge, attitude, behavior or intention of the individuals or community. If yes, did behavior change occur in the way it was expected? If no, why not? Campaign 1 Campaign 2 Campaign 3 Campaign 4 Knowledge Y/N Y/N Y/N Y/N Awareness Y/N Y/N Y/N Y/N Attitude Change Y/N Y/N Y/N Y/N Behaviour change Y/N Y/N Y/N Y/N 4. What were the negative campaign effects identified, if any? Campaign 1 Campaign 2 Campaign 3 Campaign 4 5. In your opinion were the campaigns able to assist the DoH in terms of..? Please explain Campaign 1 Campaign 2 Campaign 3 Campaign 4 i. Increased Linkages to care and treatment services ii. Addressing key drivers of the epidemic eg concurrent partners, low condom utilization, drug and alcohol abuse, low MMC 6. Did anything occur during the course of the program that could increase the target audience’s reception to the campaign or desire to engage in competing behaviours? ______________________________________________________________________________ ______________________________________________________________________________ Page 5 of 12 Interview Guide USAID Activity Manager SECTION III: Monitoring and Evaluation 1. Please state how JHHESA monitors programs ? Checking distribution of materials Observing interpersonal outreach Periodic focus groups Other At JHHESA Y/N Y/N Y/N Y/N Of sub-grantees Y/N Y/N Y/N Y/N 2. How does JHHESA support new and existing sub-grantees to strengthen program monitoring and evaluation systems? ______________________________________________________________________________ ______________________________________________________________________________ What reports does JHHESA submit to USAID, to whom, how often? Report submitted To Whom Frequency 3. Please comment on the quality of the data submitted by JHHESA to USAID in relation to? Dimension Comment Reliability Validity Timeliness Completeness Integrity 4. What challenges have you noted in monitoring the programs, if any ? Challenge Comments Page 6 of 12 Interview Guide USAID Activity Manager 5. What suggestions would you make to improve data reporting by JHHESA , if any? ______________________________________________________________________________ ______________________________________________________________________________ ______________________________________________________________________________ ______________________________________________________________________________ SECTION IV: Government collaboration, Advocacy and Liaison 1. Please describe JHHESA’s technical assistance work with Departments of Health, Basic Education, Social Development and SANAC in terms of? Activity DoH DBE DSD SANAC Communications Technical Task Team i. Policies and guidelines ii. Training iii. Monitoring and Evaluation of activities iv. Advocacy 2. What activities does JHHESA undertake to build the capacity of SAG at the following levels? Activity DoH DBE DSD SANAC Communications Technical Task Team i. National ii. Provincial iii. District 3. To what extent does JHHESA support SAG’s broader prevention strategy? ______________________________________________________________________________ ______________________________________________________________________________ ______________________________________________________________________________ What have been some of the successes in these efforts to build capacity of the Department of Health? ______________________________________________________________________________ ______________________________________________________________________________ ______________________________________________________________________________ Page 7 of 12 Interview Guide USAID Activity Manager 4. What have been some of the challenges in these efforts to build capacity of the Department of Health? ______________________________________________________________________________ ______________________________________________________________________________ 5. What recommendation would you make to improve your capacity building efforts with the Department of Health? ______________________________________________________________________________ ______________________________________________________________________________ ______________________________________________________________________________ 6. What have been some of the successes in these efforts to build capacity of the Department of Basic Education? ______________________________________________________________________________ ______________________________________________________________________________ ______________________________________________________________________________ 7. What have been some of the challenges in these efforts to build capacity of the Department of Basic Education? ______________________________________________________________________________ ______________________________________________________________________________ 8. What recommendation would you make to improve your capacity building efforts with the Department of Basic Education? ______________________________________________________________________________ ______________________________________________________________________________ ______________________________________________________________________________ 9. What have been some of the successes in these efforts to build capacity of the Department of Social Development? ______________________________________________________________________________ ______________________________________________________________________________ ______________________________________________________________________________ 10. What have been some of the challenges in these efforts to build capacity of the Department of Social Development? ______________________________________________________________________________ ______________________________________________________________________________ 11. What recommendation would you make to improve your capacity building efforts with the Department of Social Development? ______________________________________________________________________________ ______________________________________________________________________________ ______________________________________________________________________________ Page 8 of 12 Interview Guide USAID Activity Manager 12. What have been some of the successes in these efforts to build capacity of the Department of Communications Technical Task Team? ______________________________________________________________________________ ______________________________________________________________________________ ______________________________________________________________________________ 13. What have been some of the challenges in these efforts to build capacity of the Department of Communications Technical Task Team? ______________________________________________________________________________ ______________________________________________________________________________ What recommendation would you make to improve your capacity building efforts with the Department of Communications Technical Task Team? ______________________________________________________________________________ ______________________________________________________________________________ ______________________________________________________________________________ 14. Which other SAG departments (eg Women, Children and People with Disabilities) does JHHESA support? Describe activities? ______________________________________________________________________________ ______________________________________________________________________________ Briefly describe any research JHHESA has undertaken as part of its collaboration with SAG ______________________________________________________________________________ ______________________________________________________________________________ ______________________________________________________________________________ ______________________________________________________________________________ SECTION V: Training/Human Capacity Development 1. Describe the range of training programs and courses supported by JHHESA under this program in terms of target participants, and number trained? Training program Description (brief) (Probe: duration of training, accredited) Target participants Number of people trained Page 9 of 12 Interview Guide USAID Activity Manager 2. What have been some of the successes of the training program? ______________________________________________________________________________ ______________________________________________________________________________ ______________________________________________________________________________ 3. Have the training programs been evaluated? If so, how? ______________________________________________________________________________ ______________________________________________________________________________ 4. What have been some of the challenges of the training programs? ______________________________________________________________________________ ______________________________________________________________________________ SECTION VI: Strategic partnerships 1. Please describe JHHESA’s role in the partnerships with the private sector? What were the challenges and successes of the partnership? Organisation Role of JHHESA Role of partner Challenges Successes i. Levis ii. SABC iii. etv 2. What are the benefits that JHHESA provides to subgrantees and partners? ______________________________________________________________________________ ______________________________________________________________________________ 3. How does JHHESA identify new strategic partners for development? ______________________________________________________________________________ ______________________________________________________________________________ ______________________________________________________________________________ 4. Describe activities undertaken by JHHESA to build subgrantees capacity to provide community￾driven HIV prevention? ______________________________________________________________________________ ______________________________________________________________________________ 5. How does JHHESA facilitate the development of partnerships between subgrantees and other organizations? ______________________________________________________________________________ ______________________________________________________________________________ ______________________________________________________________________________ Page 10 of 12 Interview Guide USAID Activity Manager 6. Please describe significant successes in providing technical assistance and financial support to sub-grantees ______________________________________________________________________________ ______________________________________________________________________________ ______________________________________________________________________________ 7. Please explain any significant challenges in achieving JHHESA’s targets through sub-grantees? ______________________________________________________________________________ ______________________________________________________________________________ ______________________________________________________________________________ 8. In what way could this support have been improved? ______________________________________________________________________________ ______________________________________________________________________________ ______________________________________________________________________________ SECTION VII : Gender mainstreaming process 1. Which campaigns specifically addresses gender, what is the approximate percentage of funds for these campaigns, what is the target? Campaign 1 Campaign 2 Campaign 3 Campaign 4 i. Percentage of total funds ii. Target group (Probe: women in general , young women, rural women, sex workers, men) 2. What informs the campaign in terms content, implementation strategy, target group, location? What role does USAID play? ______________________________________________________________________________________ ______________________________________________________________________ 3. Is there special support for gender mainstreaming from USAID in terms of? Resources/Activities Y/N Describe i. Staff training/ workshops on gender ii. Gender mainstreaming Policies iii. Work groups iv. Materials Page 11 of 12 Interview Guide USAID Activity Manager 4. In your opinion has the JHHESA program been successful in gender mainstreaming through its activities If yes, explain. If no, why not? ______________________________________________________________________________________ ______________________________________________________________________ SECTION VIII: Program design/Grants Management 1. What are the components of this program that you believe will enhance its sustainability? Why? ______________________________________________________________________________ ______________________________________________________________________________ ______________________________________________________________________________ ______________________________________________________________________________ 2. What have been some of the key lessons in terms of successes of this program? ______________________________________________________________________________ ______________________________________________________________________________ ______________________________________________________________________________ 3. What have been some of the key challenges? ______________________________________________________________________________ ______________________________________________________________________________ ______________________________________________________________________________ 4. In your opinion, did the program achieve its aims? Describe how [Briefly] ______________________________________________________________________________ ______________________________________________________________________________ 5. What recommendation would you make to JHHESA to improve the implementation of this program going forward ______________________________________________________________________________ ______________________________________________________________________________ 6. In your opinion how should USAID improve its support to the JHHESA funded project or similar in the future? ______________________________________________________________________________ ______________________________________________________________________________ THANK YOU FOR TAKING TIME TO TALK TO US! Page 12 of 12 Interview Guide USAID Activity Manager INTERVIEW GUIDE FOR SUB-GRANTEE Content Development Name of organization being interviewed: Interviewee: Position: Date of interview: Interviewer: The focus of this evaluation assessment is to assess the USAID funded HIV communication program implemented by JHHESA. We’re interested in learning about your experiences, perspectives, and recommendations for improving this intervention which is why you have been asked to participate today. Before I start I would like to remind you that there are no right or wrong answers in this discussion. I am interested in knowing what you think, so please feel free to be frank and honest and to share your point of view. It is very important that I hear your personal opinion. I will be audio-recording today’s discussion, only for our own purposes of evaluating the data.I hope you’ll feel free to speak openly and honestly, as everything that is said in this room will be held completely confidential. Your comments will NOT be linked to your name or identity in the final report. The interview will take about 60 minutes. Your participation in this interview is completely voluntary. Are you willing to be interviewed? YES/NO Do you have any questions before we begin? Page 1 of 15 Interview Guide Content Development SECTION I: Organizational structure 1. Could you briefly describe your organizational structure including numbers of full-time staff, part-time staff, and volunteers and their roles? ______________________________________________________________________________ ______________________________________________________________________________ ______________________________________________________________________________ ______________________________________________________________________________ 2. Please briefly describe the JHHESA funded services provided by your organization and the names of the communities served: Services provided Yes/No Communities served Community radio Communication training Communications research Community dialogue Community education and literacy Film and television production Public health communication campaigns Advocacy Other (specify)_____________ Other (specify)______________ Page 2 of 15 Interview Guide Content Development 3. To what extent has your organization benefitted from the linkage with JHHESA in terms of? Please answer either “Not at all,” “somewhat,” “a great deal.” Not at all Somewhat A great deal Organizational development Financial management Training and capacity building Research and evaluation Development of social and behavior change communication strategies Communication material development Communication material dissemination 4. Please describe program areas which your organization supports (JHHESA and non-JHHESA supported) Program area JHHESA supported Non-JHHESA supported (State name of organization) HCT HIV prevention • Intergenerational sex • VMMC • Condom utilization • Multiple and concurrent partners • Transactional sex • Treatment care and support Page 3 of 15 Interview Guide Content Development Gender mainstreaming OVC Treatment Drug and alcohol abuse TB Sexual and reproductive health SECTION II Strategies to change social and gender norms 1. Describe the interpersonal communications, community mobilization and mass media activities your organization implemented with JHHESA funding. ______________________________________________________________________________ ______________________________________________________________________________ 2. Describe which channels/type of media your organization used for interpersonal communication, community mobilization and mass media activities? (Interviewer: Tick all applicable) Channel Channel type Comments TV TV shows TV spots Print Billboards Brochures Newspapers Newletters Magazines Radio Radio spots Radio adverts Page 4 of 15 Interview Guide Content Development Mobile technology Cellphones Email Internet Websites Internet adverts Promotional material Condom packs T shirts, caps and clothing items Interpersonal Strategies Peer education Workshops Hotlines Social media Facebook Twitter YouTube Other (specify) 3. Please describe the goals and objectives of each campaign? Campaign 1 Campaign 2 Campaign 3 Campaign 4 Goal Objective Page 5 of 15 Interview Guide Content Development 4. Describe the key driver of the epidemic that the campaign aims to address? (INTERVIEWER: Tick all applicable) Driver Campaign 1 Campaign 2 Campaign 3 Campaign 4 Multiple concurrent partners Low condom use Drug and alcohol abuse Low prevalence of male circumcision Intergenerational sex Gender Inequality 5. Describe the process your organization uses to identify target audiences for your communication activities. ______________________________________________________________________________ ______________________________________________________________________________ 6. Briefly describe the process that your organisation uses to develop messages for your target audiences. PROBE: Does the community and target audiences participate in the development of the messages? If so, how? ______________________________________________________________________________ ______________________________________________________________________________ ______________________________________________________________________________ ______________________________________________________________________________ 7. Do you pilot your communication activities prior to implementation and roll out? PROBE: If so, how? ______________________________________________________________________________ ______________________________________________________________________________ Page 6 of 15 Interview Guide Content Development 8. Do you address cultural issues in the design and implementation of your communication activities? PROBE: If so, how? ______________________________________________________________________________ ______________________________________________________________________________ 9. Can you describe the expected outcomes of the campaign in terms of the following: Campaign 1 Campaign 2 Campaign 3 Campaign 4 Knowledge Awareness Attitude Change Behaviour change 10. Do you try to establish consistency in messaging between all the interpersonal communications, community mobilization, and mass media activities? (PROBE: If so, how?) ______________________________________________________________________________ ______________________________________________________________________________ 11. Has JHHESA talked to you about maintaining consistency in messaging between your organization, JHHESA and other organizations? PROBE: If so, how do you ensure that consistency? ______________________________________________________________________________ ______________________________________________________________________________ 12. How is the change measured? (Probe: trend analysis, formative evaluation, summative evaluations, post test assessments) Campaign 1 Campaign 2 Campaign 3 Campaign 4 Knowledge Awareness Attitude Change Behaviour change Page 7 of 15 Interview Guide Content Development 13. In your opinion please state if the campaign has improved the knowledge, attitude, behavior or intention of the individuals or community. If yes, did behavior change occur in the way it was expected? If no, why not? Campaign 1 Campaign 2 Campaign 3 Campaign 4 Knowledge Y/N Y/N Y/N Y/N Awareness Y/N Y/N Y/N Y/N Attitude Change Y/N Y/N Y/N Y/N Behaviour change Y/N Y/N Y/N Y/N 14. Did change in knowledge, attitude, behavior or intention of the individuals or community occur in the way it was expected. If no, why not? ______________________________________________________________________________ ______________________________________________________________________________ ______________________________________________________________________________ 15. What were the negative campaign effects identified, if any? Campaign 1 Campaign 2 Campaign 3 Campaign 4 Page 8 of 15 Interview Guide Content Development In your opinion were your activities able to assist the DoH in terms of the following: Why or why not? Response Increased Linkages to care and treatment services Multiple concurrent partners Low condom use Drug and alcohol abuse Low prevalence of male circumcision Intergenerational sex 16. Did anything occur during the course of the program that could increase the target audience’s reception to the campaign or desire to engage in competing behaviours? ______________________________________________________________________________ ______________________________________________________________________________ SECTION III: Monitoring and Evaluation 1. In general, how did you measure outputs and outcomes of the JHHESA-funded communication activities? (Probe: distribution of materials, trend analysis, formative evaluation, summative evaluations, post test assessments) ______________________________________________________________________________ ______________________________________________________________________________ 2. Did JHHESA support your organization to strengthen program monitoring and evaluation systems? PROBE: If so, how? ______________________________________________________________________________ ______________________________________________________________________________ Page 9 of 15 Interview Guide Content Development 3. What reports do you submit to JHHESA and how frequently? Report submitted Frequency 4. How do you ensure data quality of the submitted reports? Dimension Method to ensure quality Reliability Validity Timeliness Completeness SECTION IV: Government collaboration, Advocacy and Liaison To what extent does your organization work with government organisations? What is JHHESA’s role in facilitating this? Not at all Somewhat A great deal JHHESA’s role (if any) Comments i. DoH ii. DBE iii. DSD iv. SANAC Communications Page 10 of 15 Interview Guide Content Development Technical Task Team 1. What have been some of the successes in these efforts to build capacity of the SAG? ______________________________________________________________________________ ______________________________________________________________________________ ______________________________________________________________________________ 2. What have been some of the challenges in these efforts to build capacity of the SAG ? ______________________________________________________________________________ ______________________________________________________________________________ 3. What recommendation would you make to improve your capacity building efforts with SAG? ______________________________________________________________________________ ______________________________________________________________________________ ______________________________________________________________________________ 4. What have been some of the successes in these efforts to build capacity of the SAG? ______________________________________________________________________________ ______________________________________________________________________________ ______________________________________________________________________________ 5. What recommendation would you make to improve your capacity building efforts with SAG? ______________________________________________________________________________ ______________________________________________________________________________ ______________________________________________________________________________ Page 11 of 15 Interview Guide Content Development SECTION V: Training/Human Capacity Development 1. Describe the range of training programs and courses supported by JHHESA under this program? Name of training program Description of training Target participants Number of people trained through JHHESA funding to date 2. What have been some of the successes of the training program? ______________________________________________________________________________ ______________________________________________________________________________ ______________________________________________________________________________ 3. What have been some of the challenges of the training programs? ______________________________________________________________________________ ______________________________________________________________________________ _____________________________________________________________________________ Page 12 of 15 Interview Guide Content Development SECTION VI: Strategic partnerships 1. In your opinion, what are the benefits that JHHESA provides to your organization? ______________________________________________________________________________ ______________________________________________________________________________ 2. In your opinion, what support that you received from JHHESA has provided the biggest impact? ______________________________________________________________________________ ______________________________________________________________________________ 3. What were some of the challenges your organization faced related to carrying out specific activities? • BCC planning ________________________________________________________________________ • IPC/C Trainings ________________________________________________________________________ ComMob and Local Advocacy ________________________________________________________________________ Mass Media ________________________________________________________________________ Organizing and Supporting Health Events ________________________________________________________________________ Training and Capacity Building ________________________________________________________________________ Monitoring and Evaluation ________________________________________________________________________ What could be done differently next time to overcome those challenges? • BCC planning ____________________________________________________________________ IPC/C Trainings ____________________________________________________________________ ComMob and Local Advocacy ____________________________________________________________________ Mass Media ____________________________________________________________________ Organizing and Supporting Health Events ____________________________________________________________________ Page 13 of 15 Interview Guide Content Development Training and Capacity Building ____________________________________________________________________ Monitoring and Evaluation ____________________________________________________________________ What services, programs, activities that you currently implementing will continue without support from JHHESA? Why? ____________________________________________________________________ Services, Programs, Activities With JHHESA Without JHHESA 4. Do you think JHHESA meets an important need for your organization? Please explain (PROBE for their perspective on what their needs are and which of these JHHESA could or could not appropriately address) __________________________________________________________________________________ __________________________________________________________________________________ SECTION VIII : Gender mainstreaming process 1. Which campaigns specifically addresses gender, what is the approximate percentage of funds for these campaigns? Campaign 1 Campaign 2 Campaign 3 Campaign 4 Address gender Percentage of total funds Target group (Probe: women in general , young women, rural women, sex workers) Page 14 of 15 Interview Guide Content Development 2. What is the involvement of women in the conceptualisation, implementation and monitoring of these campaigns? Campaign 1 Campaign 2 Campaign 3 Campaign 4 3. Is there special support for gender mainstreaming? Resources/Activities Y/N Staff training/ workshops on gender Work groups Materials THANK YOU FOR TAKING THE TIME TO TALK TO US! Page 15 of 15 Interview Guide Content Development INTERVIEW GUIDE NDOH/DBE/DSD/SANAC/Private Sector Partners Name of Department/Organization Interviewer Date University Research South Africa is conducting an evaluation to assess the USAID funded HIV and communication programs implemented by JHHESA. We’re interested in learning about your experiences, perspectives, and recommendations for improving this intervention which is why you’ve all been asked to participate today. Before we start I would like to remind you that there are no right or wrong answers in this discussion. We are interested in knowing what you think, so please feel free to be frank and honest and to share your point of view. Your comments will NOT be linked to your name or identity in the final report. It is very important that we hear your personal opinion. We will be audio-recording today’s discussion, so we can better analyze the information we gather. I hope you’ll feel free to speak openly and honestly, as everything that is said in this room will be held completely confidential. In addition, it is important for all participants to respect the confidentiality and privacy of everyone in this discussion. I’d like the discussion to be informal, so there’s no need to wait for me to call on you to respond. The interview will take about 60 minutes Your participation in this interview is completely voluntary. Are you willing to be interviewed? YES/NO Do you have any questions before we begin? Page 1 of 5 Interview Guide NDOH/DBE/DSD/SANAC/Private Sector Partners SECTION I: Program implementation and design 1. Briefly describe your role and the role of your directorate/organization in HIV prevention and care in South Africa? Interviewee role Role of Directorate/Organization 2. Could you describe the work and contribution of JHHESA to your department/organization ? PROBE What coordination and technical assistance roles did JHHESA provide to your department/organization? ______________________________________________________________________________ ______________________________________________________________________________ 3. Describe your experience working with JHHESA ? PROBE What else? ______________________________________________________________________________ ______________________________________________________________________________ 4. In your opinion, what has been JHHESA’s contribution to policy and public awareness in HIV prevention in South Africa, in your organisation? ______________________________________________________________________________ ______________________________________________________________________________ 5. What would you consider to be the significant successes of JHHESA’s program ? (PROBE: What in your mind have been the key components that you believe created value?) ______________________________________________________________________________ ______________________________________________________________________________ 6. Were there any challenges your department/organization faced working with JHHESA? PROBE: Which ones? ______________________________________________________________________________ ______________________________________________________________________________ Page 2 of 5 Interview Guide NDOH/DBE/DSD/SANAC/Private Sector Partners 7. How would you describe JHHESA’s contribution to your organisation’s HIV and TB communication strategy? ______________________________________________________________________________ ______________________________________________________________________________ 8. Are there any strategies in place to ensure that your organisation and JHHESA’s campaign messages are consistent? If so, PROBE, which ones? ______________________________________________________________________________ ______________________________________________________________________________ 9. What recommendations would you make to JHHESA to improve program implementation and activities? ______________________________________________________________________________ ______________________________________________________________________________ SECTION II: Education and training 1. How would you describe JHHESA’s contribution to health communication education and training for HIV prevention nationally and regionally? ______________________________________________________________________________ ______________________________________________________________________________ 2. Briefly describe some specific JHHESA education and training programs you are aware of. ______________________________________________________________________________ ______________________________________________________________________________ 3. How does JHHESA involve your department/organization in their education and training activities? ______________________________________________________________________________ ______________________________________________________________________________ Page 3 of 5 Interview Guide NDOH/DBE/DSD/SANAC/Private Sector Partners SECTION III: Health System strengthening 1. In your opinion, how has JHHESA contributed to health system strengthening in South Africa ? PROBE: -WHO Key component of a well-functioning health system] i. Leadership and Governance Policy ii. Human Resources iii. Health Information Systems iv. Finance v. Access to medical products SECTION IV: Lessons learned 1. In your opinion, what are the lessons that have been learned from your department/organization’s collaboration with JHHESA?” (PROBE: What else)? ______________________________________________________________________________ ______________________________________________________________________________ 2. How can this lessons be replicated and what do you see as your role as government/ organisation? ______________________________________________________________________________ ______________________________________________________________________________ Page 4 of 5 Interview Guide NDOH/DBE/DSD/SANAC/Private Sector Partners SECTION V: Sustainability 1. Do you think the initiatives implemented under the JHHESA’s HIV prevention program are sustainable? If so, how do you think they will be sustained? (PROBE: How will they be technically sustained? How will they be financially sustained?) ______________________________________________________________________________ ______________________________________________________________________________ ______________________________________________________________________________ 2. What are some of the benefits of the JHESSA program that you believe will be long lasting? ______________________________________________________________________________ ______________________________________________________________________________ 3. Which components of this program do you believe still need to be supported? (PROBE: Why?) ______________________________________________________________________________ ______________________________________________________________________________ THANK YOU FOR YOUR PARTICIPATION! Page 5 of 5 Interview Guide NDOH/DBE/DSD/SANAC/Private Sector Partners INTERVIEW GUIDE JHHESA National Office - CEO Interviewee: Position: Date : Interviewer: The focus of this evaluation assessment is to assess the USAID funded program implemented by JHHESA. We’re interested in learning about your experiences, perspectives, and recommendations for improving this intervention which is why you’ve all been asked to participate today. Before we start I would like to remind you that there are no right or wrong answers in this discussion. We are interested in knowing what you think, so please feel free to be frank and honest and to share your point of view. It is very important that we hear your personal opinion. We will be audio-recording today’s discussion, so we will have a record of what is said I hope you’ll feel free to speak openly and honestly, as everything that is said in this room will be held completely confidential. In addition, it is important for all participants to respect the confidentiality and privacy of everyone in this discussion. We ask that opinions and perspectives expressed during this discussion are not shared with others outside of this group. I’d like the discussion to be informal, so there’s no need to wait for me to call on you to respond. Your comments will NOT be linked to your name or identity in the final report. The interview will take about 60 minutes Your participation in this interview is completely voluntary. Are you willing to be interviewed? YES/NO Do you have any questions before we begin? Page 1 of 18 Interview Guide JHHESA National Office – CEO SECTION I: Organizational Structure 1. Could you briefly describe JHHESA’s organizational structure at the national and provincial level? ________________________________________________________________________ ________________________________________________________________________ ________________________________________________________________________ ________________________________________________________________________ ________________________________________________________________________ 2. To what extent has JHHESA benefitted from the linkage with Johns Hopkins University￾Center for Communications Programs in terms of? Not at all Somewhat A great deal Organizational development Financial management Training and capacity building (Probe: How does JHU-CCP “develop a cadre of public health professionals who are well-versed in strategic communication.” Knowledge management (Probe: how does JHU-CCP’s K4health assist you to synthesize experiential and scientific knowledge, share information broadly, and encourage local use and adaptation as necessary) Research and Evaluation Development of social and behavior change communication strategies 3. Please describe the key components the USAID programme in South Africa? ________________________________________________________________________ ________________________________________________________________________ ________________________________________________________________________ Page 2 of 18 Interview Guide JHHESA National Office – CEO 4. How is the USAID program managed and coordinated in relation to the following areas? a) Project management b) Operations c) Financing d) Grants management e) Monitoring , Evaluation and reporting 5. Briefly describe JHHESA’s contribution to the National Strategic Plan on HIV, STIs and TB 2012 – 2016? ________________________________________________________________________ ________________________________________________________________________ ________________________________________________________________________ ________________________________________________________________________ 6. Please describe support provided through JHHESA by sub-grantees under HIV prevention services including program targets, populations served, and districts targeted in terms of: HCT HIV prevention Treatment care and support OVC Page 3 of 18 Interview Guide JHHESA National Office – CEO 7. Briefly describe the grant management system between JHHESA and sub-grantees under this program in terms of the following: i. The number of sub-grantees supported under this program ii. The selection procedure(s) for organizations for funding support? iii. Capacity building activities for sub￾grantees iv. Tracking and measuring success or outcomes (Probe : How does the JHHESA grant management system allow you to define, measure, and report on the key outcomes important to your organization) v. Monitoring (Probe :How does the system allow JHHESA to quickly monitor the performance and results for any program, grant, to quickly respond and make adjustments) vi. Budgeting (Probe: How does JHHESA keep track of and create budgets and manage the grant pipeline) vii. Grant application tracking (Probe: How does JHHESA allow faster compilation of application components) SECTION II: Strategies to Change Social and Gender Norms 1. Describe JHHESA’s role in promoting behavior change communication programs in South Africa? ________________________________________________________________________ ________________________________________________________________________ ________________________________________________________________________ Page 4 of 18 Interview Guide JHHESA National Office – CEO 2. Describe the interpersonal communications, community mobilization and mass media campaigns used? ________________________________________________________________________ ________________________________________________________________________ Describe the channels used for interpersonal communication, community mobilization and mass media campaigns? (tick all applicable) Channel Channel type Comments TV TV shows TV spots Print Billboards Brochures Newspapers Newletters Magazines Radio Radio spots Radio adverts Mobile technology Cellphones Email Internet Websites Internet adverts Promotional material Condom packs T shirts, caps and clothing items Interpersonal Strategies Peer education Workshops Hotlines Social media Facebook Twitter YouTube Page 5 of 18 Interview Guide JHHESA National Office – CEO 3. How are campaign goals defined? ________________________________________________________________________ ________________________________________________________________________ ________________________________________________________________________ 4. Please describe the goals and objectives of each campaign? Campaign 1 Campaign 2 Campaign 3 Campaign 4 Goal Objective 5. Describe the key driver of the epidemic that the camping aims to address? (Tick all applicable) Driver Campaign 1 Campaign 2 Campaign 3 Campaign 4 Multiple concurrent partners Low condom use Drug and alcohol abuse Low prevalence of male circumcision Intergenerational sex Commercial sex 6. Describe the process of identifying the target audience? ________________________________________________________________________ ________________________________________________________________________ ________________________________________________________________________ 7. Briefly describe the process that JHHESA undertakes to develop precise messages? ________________________________________________________________________ ________________________________________________________________________ ________________________________________________________________________ Page 6 of 18 Interview Guide JHHESA National Office – CEO 8. What is the role of the community and target audience in the development of the precise messaging? ________________________________________________________________________ ________________________________________________________________________ ________________________________________________________________________ 9. How are campaigns tested and piloted by JHHESA prior to implementation and roll out? ________________________________________________________________________ ________________________________________________________________________ ________________________________________________________________________ 10. How is cultural sensitivity addressed in the design and implementation of campaigns? ________________________________________________________________________ ________________________________________________________________________ ________________________________________________________________________ 11. Can you describe the expected outcomes of the campaign in terms of the following: Campaign 1 Campaign 2 Campaign 3 Campaign 4 Knowledge Awareness Attitude Change Behaviour change 12. How is consistency in messaging between all the interpersonal communications, community mobilization campaigns and mass media campaigns maintained? ________________________________________________________________________ ________________________________________________________________________ ________________________________________________________________________ 13. How is consistency in messaging between JHHESA , government and sub-grantees ensured? ________________________________________________________________________ ________________________________________________________________________ ________________________________________________________________________ Page 7 of 18 Interview Guide JHHESA National Office – CEO 14. How is the change measured? (Probe: trend analysis, formative evaluation, summative evaluations, post test assessments) Campaign 1 Campaign 2 Campaign 3 Campaign 4 Knowledge Awareness Attitude Change Behaviour change 15. In your opinion please state if the campaign has improved the knowledge, attitude, behavior or intention of the individuals or community. If yes, did behavior change occur in the way it was expected? If no, why not? Campaign 1 Campaign 2 Campaign 3 Campaign 4 Knowledge Y/N Y/N Y/N Y/N Awareness Y/N Y/N Y/N Y/N Attitude Change Y/N Y/N Y/N Y/N Behaviour change Y/N Y/N Y/N Y/N 16. Did change in knowledge, attitude, behavior or intention of the individuals or community occur in the way it was expected. If no, why not? ________________________________________________________________________ ________________________________________________________________________ ________________________________________________________________________ _______________________________________________________________________ 17. What were the negative campaign effects identified, if any? Campaign 1 Campaign 2 Campaign 3 Campaign 4 Page 8 of 18 Interview Guide JHHESA National Office – CEO 18. In your opinion were the campaigns able to assist the DoH in terms of Campaign 1 Campaign 2 Campaign 3 Campaign 4 Increased Linkages to care and treatment services Addressing key drivers of the epidemic eg concurrent partners, low condom utilization, drug and alcohol abuse, low MMC 19. Did anything occur during the course of the program that could increase the target audience’s reception to the campaign or desire to engage in competing behaviours? ________________________________________________________________________ ________________________________________________________________________ ________________________________________________________________________ SECTION III: Monitoring and Evaluation 1. Briefly describe how you monitor programs? Checking distribution of materials Observing interpersonal outreach Periodic focus groups At JHHESA Of sub-grantees 2. How do you support new and existing sub-grantees to strengthen program monitoring and evaluation systems? ________________________________________________________________________ ________________________________________________________________________ ________________________________________________________________________ Page 9 of 18 Interview Guide JHHESA National Office – CEO 3. What reports do you submit, to whom? Report submitted To Whom 4. How do you ensure data quality? Dimension Method to ensure quality Reliability Validity Timeliness Completeness 5. What challenges have you noted in monitoring the programs? Challenge Y/N Comments 6. What suggestions would you make to improve data collection and reporting if any? ________________________________________________________________________ ________________________________________________________________________ ________________________________________________________________________ Page 10 of 18 Interview Guide JHHESA National Office – CEO SECTION IV: Government Collaboration, Advocacy and Liaison 1. Please describe JHHESA’s technical assistance work with Departments of Health, Basic Education, Social Development and SANAC in terms of? Activity DoH DBE DSD SANAC Communications Technical Task Team i. Policies and guidelines ii. Training iii. Monitoring and Evaluation of activities 2. What activities do you undertake to build the capacity of the Department of Health at following levels? Activity DoH DBE DSD SANAC Communications Technical Task Team i. National ii. Provincial iii. District 3. To what extent does JHHESA support SAG’s broader prevention strategy? ________________________________________________________________________ ________________________________________________________________________ ________________________________________________________________________ 4. What have been some of the successes in these efforts to build capacity of the Department of Health? ________________________________________________________________________ ________________________________________________________________________ ________________________________________________________________________ Page 11 of 18 Interview Guide JHHESA National Office – CEO 5. What have been some of the challenges in these efforts to build capacity of the Department of Health? ________________________________________________________________________ ________________________________________________________________________ ________________________________________________________________________ 6. What recommendation would you make to improve your capacity building efforts with the Department of Health? ________________________________________________________________________ ________________________________________________________________________ ________________________________________________________________________ 7. What have been some of the successes in these efforts to build capacity of the Department of Basic Education? ________________________________________________________________________ ________________________________________________________________________ ________________________________________________________________________ 8. What have been some of the challenges in these efforts to build capacity of the Department of Basic Education? ________________________________________________________________________ ________________________________________________________________________ ________________________________________________________________________ 9. What recommendation would you make to improve your capacity building efforts with the Department of Basic Education? ________________________________________________________________________ ________________________________________________________________________ ________________________________________________________________________ ________________________________________________________________________ Page 12 of 18 Interview Guide JHHESA National Office – CEO 10. What have been some of the successes in these efforts to build capacity of the Department of Social Development? ________________________________________________________________________ ________________________________________________________________________ ________________________________________________________________________ 11. What have been some of the challenges in these efforts to build capacity of the Department of Social Development? ________________________________________________________________________ ________________________________________________________________________ ________________________________________________________________________ 12. What recommendation would you make to improve your capacity building efforts with the Department of Social Development? ________________________________________________________________________ ________________________________________________________________________ ________________________________________________________________________ ________________________________________________________________________ 13. What have been some of the successes in these efforts to build capacity of the Department of Communications Technical Task Team? ________________________________________________________________________ ________________________________________________________________________ ________________________________________________________________________ ________________________________________________________________________ 14. What have been some of the challenges in these efforts to build capacity of the Department of Communications Technical Task Team? ________________________________________________________________________ ________________________________________________________________________ ________________________________________________________________________ Page 13 of 18 Interview Guide JHHESA National Office – CEO 15. What recommendation would you make to improve your capacity building efforts with the Department of Communications Technical Task Team? ________________________________________________________________________ ________________________________________________________________________ ________________________________________________________________________ ________________________________________________________________________ 16. Briefly describe any research has JHHESA undertaken as part of its collaboration with government ________________________________________________________________________ ________________________________________________________________________ ________________________________________________________________________ ________________________________________________________________________ ________________________________________________________________________ SECTION V: Training/Human Capacity Development 1. Describe the range of training programs and courses supported by JHHESA under this program? Training program Description (brief) Target participants Number of people trained Page 14 of 18 Interview Guide JHHESA National Office – CEO 2. What have been some of the successes ________________________________________________________________________ ________________________________________________________________________ ________________________________________________________________________ ________________________________________________________________________ 3. What have been some of the challenges? ________________________________________________________________________ ________________________________________________________________________ ________________________________________________________________________ ________________________________________________________________________ SECTION VI: Strategic partnerships 1. Please describe the JHHESA’s role in the partnerships with the private sector? Organisation Role of JHHESA Role of partner Challenges Successes Levis SABC etv 2. What are the benefits that JHHESA provides to subgrantees and partners? ________________________________________________________________________ ________________________________________________________________________ ________________________________________________________________________ 3. How does JHHESA identify new strategic partners for development? ________________________________________________________________________ ________________________________________________________________________ ________________________________________________________________________ ________________________________________________________________________ Page 15 of 18 Interview Guide JHHESA National Office – CEO 4. Describe activities undertaken by JHHESA to build subgrantees capacity to provide community-driven HIV prevention? ________________________________________________________________________ ________________________________________________________________________ ________________________________________________________________________ 5. How do you facilitate the development of partnerships between subgrantees and other organizations? ________________________________________________________________________ ________________________________________________________________________ ________________________________________________________________________ ________________________________________________________________________ 6. Please describe significant successes in providing technical assistance and financial support to sub-grantees ________________________________________________________________________ ________________________________________________________________________ ________________________________________________________________________ ________________________________________________________________________ 7. Please explain any significant challenges in achieving your targets through sub-grantees? ________________________________________________________________________ ________________________________________________________________________ ________________________________________________________________________ ________________________________________________________________________ 8. In what way could this support have been improved? ________________________________________________________________________ ________________________________________________________________________ ________________________________________________________________________ ________________________________________________________________________ Page 16 of 18 Interview Guide JHHESA National Office – CEO SECTION VII: Program Design/Grants Management 1. What are the components of this program that you believe will enhance its sustainability? ________________________________________________________________________ ________________________________________________________________________ ________________________________________________________________________ ________________________________________________________________________ 2. What have been some of the key lessons in terms of successes of this program? ________________________________________________________________________ ________________________________________________________________________ ________________________________________________________________________ ________________________________________________________________________ 3. What have been some of the key challenges? ________________________________________________________________________ ________________________________________________________________________ ________________________________________________________________________ ________________________________________________________________________ 4. Did the program achieve its aims? Describe how [Briefly] ________________________________________________________________________ ________________________________________________________________________ ________________________________________________________________________ 5. What recommendation would you make to USAID to improve the implementation of this program going forward ________________________________________________________________________ ________________________________________________________________________ ________________________________________________________________________ Page 17 of 18 Interview Guide JHHESA National Office – CEO SECTION VIII : Gender Mainstreaming Process 1. Which campaigns specifically addresses gender, what is the approximate percentage of funds for these campaigns? Campaign 1 Campaign 2 Campaign 3 Campaign 4 Address gender Percentage of total funds Target group (Probe: women in general , young women, rural women, sex workers) 2. What is the involvement of women in the conceptualisation, implementation and monitoring of these campaigns? Campaign 1 Campaign 2 Campaign 3 Campaign 4 3. Is there special support for gender mainstreaming? Resources/Activities Y/N Staff training/ workshops on gender Work groups Materials THANK YOU FOR TAKING THE TIME TO TALK TO US! Page 18 of 18 Interview Guide JHHESA National Office – CEO FOCUS GROUP DISCUSSION GUIDE Mass Media Campaign Good morning/afternoon/evening. My name is ____ and I work for University Research South Africa (URSA). This is my colleague_________. We are studying the mass media campaigns implemented by JHHESA. We’re interested in learning about your experiences, perspectives, and recommendations for improving mass media campaigns which is why you’ve all been asked to participate today. Our discussion will take about 2 hours. Before we start I would like to remind you that there are no right or wrong answers in this discussion. We are interested in knowing what you think, so please feel free to be frank and honest and to share your point of view. Although my colleague will be making notes during the discussion and audio-taping it, it is only for us to remember what points have been raised. She/he will not write down any names. So whatever you say will be confidential. Your comments will NOT be linked to your name or identity in the final report. It is very important that we hear your personal opinion. Please remember, you are the experts and we are here to learn from you. Please don't tell us what you think we might want to hear. Tell us your views, whatever they are. It is important for all participants to respect the confidentiality and privacy of everyone in this discussion. We ask that opinions and perspectives expressed during this discussion are not shared with others outside of this group. I’d like the discussion to be informal, like a conversation, so there’s no need to wait for me to call on you to respond. Before we begin, let’s all introduce ourselves. Please tell us your name. Your participation in this interview is completely voluntary. Are you willing to be interviewed? YES/NO Do you have any questions before we begin? Page 1 of 3 Mass Media Campaign 1. Have you ever heard of Intersexions? PROBE: What is Intersexions? What is it about? ______________________________________________________________________________ ______________________________________________________________________________ 2. Did you learn anything from Intersexions? What did you learn? ______________________________________________________________________________ ______________________________________________________________________________ 3. What do you think were the main HIV prevention messages from Intersexions? ______________________________________________________________________________ ______________________________________________________________________________ 4. Was this information useful to you? PROBE: In what way? ______________________________________________________________________________ ______________________________________________________________________________ 5. Was there any information that was new to you that Intersexions provided? ______________________________________________________________________________ ______________________________________________________________________________ 6. After watching Intersexion, did you do anything different to prevent HIV? (PROBE: reduce drug and alcohol use, reduce number of sexual partners, and increase condom use, visit your nearest health facility for HIV prevention services e.g. MMC, HCT, TB screening, PMTCT, FP, etc.) ______________________________________________________________________________ ______________________________________________________________________________ 7. Other than Intersexions, where else have you seen (or heard) information about HIV prevention? (PROBE: TV, radio, print, health workers) ______________________________________________________________________________ ______________________________________________________________________________ 8. Are the messages that you received from Intersexions the same as messages that you receive from other sources? (PROBE: e.g. TV, radio, health provider? If no, which messages are different? If yes, which messages are the same?) ______________________________________________________________________________ ______________________________________________________________________________ Page 2 of 3 Mass Media Campaign 9. Which channel for receiving HIV prevention messages do you prefer? Why? (PROBE: TV, radio, print, health workers) ______________________________________________________________________________ ______________________________________________________________________________ 10. Could you identify with any of the messages that were given during Intersexions? ______________________________________________________________________________ ______________________________________________________________________________ 11. Is there anything from Intersexions that you did not agree with? (PROBE: What did you especially like about Intersexions? What did you dislike about Intersexions?) ______________________________________________________________________________ ______________________________________________________________________________ 12. Was Intersexions accurate in portraying men and women and how their relationships affect HIV prevention e.g. GBV (PROBE: If yes, what do you think the message is?) ______________________________________________________________________________ ______________________________________________________________________________ 13. Do you think Intersexions empowered the community about HIV prevention? If yes, how? ______________________________________________________________________________ ______________________________________________________________________________ 14. Are there any other topic areas you wish Intersexions addressed that it didn’t? ______________________________________________________________________________ ______________________________________________________________________________ THANK YOU FOR YOUR TIME AND ASSISTANCE! Page 3 of 3 Mass Media Campaign FOCUS GROUP DISCUSSION GUIDE Community Interventions Good morning/afternoon/evening. My name is ____ and I work for University Research South Africa (URSA). This is my colleague_________ . We are studying the community interventions implemented by JHHESA. We’re interested in learning about your experiences, perspectives, and recommendations for improving community interventions, which is why you’ve all been asked to participate today. Our discussion will take about 2 hours. Before we start, I would like to remind you that there are no right or wrong answers in this discussion. We are interested in knowing what you think, so please feel free to be frank and honest and to share your point of view. Although my colleague will be making notes during the discussion and audio-taping it, it is only for us to remember what points have been raised. She/he will not write down any names. So whatever you say will be confidential. Your comments will NOT be linked to your name or identity in the final report. It is very important that we hear your personal opinion. Please remember, you are the experts and we are here to learn from you. Please don't tell us what you think we might want to hear. Tell us your views, whatever they are. It is important for all participants to respect the confidentiality and privacy of everyone in this discussion. We ask that opinions and perspectives expressed during this discussion are not shared with others outside of this group. I’d like the discussion to be informal, like a conversation, so there’s no need to wait for me to call on you to respond. Before we begin, let’s all introduce ourselves. Please tell us your name. Your participation in this interview is completely voluntary. Are you willing to be interviewed? YES/NO Do you have any questions before we begin? Page 1 of 3 Community Interventions Questions 1. Tell me what you know about HIV/AIDS prevention. ______________________________________________________________________________ ______________________________________________________________________________ 2. From where did you get most of this information? (PROBE: TV, radio, print, health workers) ______________________________________________________________________________ ______________________________________________________________________________ 3. Which channel of receiving HIV prevention messages do you prefer? Why? (PROBE : TV, radio, print, health workers) ______________________________________________________________________________ ______________________________________________________________________________ 4. Have you ever heard of Sisonke/ TVT/ DramAidE? PROBE: What is Sisonke/ TVT/ DramAidE? ______________________________________________________________________________ ______________________________________________________________________________ 5. Did you learn anything from the [Sisonke/TVT/DramAidE] event? PROBE: What did you learn? ______________________________________________________________________________ ______________________________________________________________________________ 6. What do you think were the main HIV prevention messages from [Sisonke/TVT/DramAidE]? ______________________________________________________________________________ ______________________________________________________________________________ 7. Was this information useful to you? PROBE: In what way? ______________________________________________________________________________ ______________________________________________________________________________ 8. Was there any information that was new to you that [Sisonke/TVT/DramAidE] provided? ______________________________________________________________________________ ______________________________________________________________________________ Page 2 of 3 Community Interventions 9. Are the messages that you received from the event the same as messages that you receive from other sources e.g. TV, radio, health provider? If no, which messages? If yes, which messages? ______________________________________________________________________________ ______________________________________________________________________________ 10. What did you especially like about the event? ______________________________________________________________________________ ______________________________________________________________________________ 11. What did you dislike about the event? ______________________________________________________________________________ ______________________________________________________________________________ 12. Following the event did you have an opportunity to ask questions afterwards? If no, why not? ______________________________________________________________________________ ______________________________________________________________________________ 13. After the event to what extent did you do anything different to prevent HIV? [PROBE: reduce drug and alcohol use, reduce number of sexual partners, increase condom use, visit your nearest health facility for HIV prevention services e.g. MMC, HCT, TB screening , PMTCT, FP etc.] ______________________________________________________________________________ ______________________________________________________________________________ 14. Was the event accurate in portraying men and women and how their relationships affect HIV prevention [PROBE: If yes, what do you think the message is?] ______________________________________________________________________________ ______________________________________________________________________________ 15. To what extent have you been involved in the design, conceptualisation and planning of [Sisonke/TVT/DramAidE]? ______________________________________________________________________________ ______________________________________________________________________________ 16. Do you think the event empowered the community about HIV prevention? If yes, how? ______________________________________________________________________________ ______________________________________________________________________________ THANK YOU FOR TAKING TIME TO TALK TO US! Page 3 of 3 Community Interventions ANNEX IV: SOURCES OF INFORMATION LIST OF DOCUMENTS REVIEWED 1. ACSM Trainings – Workshops undertaken with districts and provinces to develop district and provincial level ACSM Strategies. 2. Advocacy and Capacity Building Partners a. Anova/Wits HIV and the Media Project b. Health-e c. UKZN – CCMS 3. Communication Programmes – Documents pertaining to the USAID/JHU HIV Communication Campaigns being undertaken. a. 4Play Sex Tips for Girls b. Brothers for Life c. Intersexions d. Scrutinize 4. DOH – SANAC Support – Communication Strategy Documents developed for the Department of Health and SANAC. a. National Infant Feeding Strategy b. Khomanani Confederations Cup c. Medical Male Circumcision d. PMTCT A-Plan 5. NCS_Evaluations – Descriptive and combined impact of communication programmes. a. 2009 b. 2012 c. Overview of the NCS 6. Qualitative Research Reports 7. Social Mobilisation Partner Reports – Reports that summarises partner programmes from 2008 - 2011 a. CMT b. Lesedi Lechabile c. Mothusimpilo d. Turntable Trust e. Valley Trust f. Mindset Health – An evaluation of the Mindset Programme 8. USAID/JHU Strategic Documents a. COPs b. M&E Docs c. MOUs d. Quarterly and Annual Progress Reports 1 e. Strategic Documents f. USAID Contract and Mods g. Workplans 9. USAID/JHU HIV COMMUNICATION PROGRAMME program monitoring database 10. Program Indicator List 2 ANNEX V: LIST OF SITES VISITED Province Dates Sites Visited Gauteng 29-Jan USAID Gauteng 30-Jan USAID/JHU HIV COMMUNICATION PROGRAMME Gauteng 04-Feb Mothusimpilo Gauteng 04-Feb Sonke Gender Gauteng 04-Feb ANOVA Gauteng 05-Feb Curious Pictures Gauteng 05-Feb Health & Development Africa Gauteng 05-Feb CADRE Gauteng 06-Feb Joe Public Free State 06-Feb Lesedi Lechabile KwaZulu Natal 11-Feb FGD KwaZulu Natal 11-Feb FGD KwaZulu Natal 11-Feb The Valley Trust KwaZulu Natal 12-Feb Turn Table Trust KwaZulu Natal 12-Feb FGD KwaZulu Natal 13-Feb DramAidE KwaZulu Natal 13-Jan Provincial liaison Gauteng 13-Jan FGD Gauteng 13-Feb FGD Gauteng 13-Feb DOH Gauteng 14-Jan Provincial liaison Gauteng 14-Feb SABC Gauteng 14-Feb FGD Gauteng 14-Feb SANAC KwaZulu Natal 14-Feb Centre for Communication and Media Studies @ UKZN 3 ANNEX VI: LIST OF SITES EXCLUDED Organisation Location Interviewed Reason : if not interviewed ABC Ulwazi - No The organisation went insolvent a year ago Cell-Life Cape Town No It is a monitoring partner and provides primarily the software for the monitoring system, again based outside the selected provinces The Community Health Media Trust (CHMT) Cape Town No Organisation based outside the selected provinces LifeLine Southern Africa Limpopo & Northern Cape No Organisation based outside the selected provinces Health-e News Service Western Cape (Cape Town) No Organisation based outside the selected provinces Matchboxology Western Cape (Cape Town) No Organisation based outside the selected provinces The Mindset Health Channel (MHC) Gauteng No The contract with USAID/JHU HIV COMMUNICATION PROGRAMME ended more than a year ago 4 ANNEX VII: DISCLOSURE OF ANY CONFLICTS OF INTEREST Name Nondumiso Makhunga-Ramfolo Title Director, Research and Evaluation Advisor Organization Evaluation Position? Team Leader Team member Evaluation Award Number (contract or other instrument) USAID Project(s) Evaluated (Include project name(s), implementer name(s) and award number(s), if applicable) Johns Hopkins Health and Education South Africa Project Performance Evaluation I have real or potential conflicts of interest to disclose. Yes No If yes answered above, I disclose the following facts: Real or potential conflicts of interest may include, but are not limited to: 1. Close family member who is an employee of the USAID operating unit managing the project(s) being evaluated or the implementing organization(s) whose project(s) are being evaluated. 2. Financial interest that is direct, or is significant though indirect, in the implementing organization(s) whose projects are being evaluated or in the outcome of the evaluation. 3. Current or previous direct or significant though indirect experience with the project(s) being evaluated, including involvement in the project design or previous iterations of the project. 4. Current or previous work experience or seeking employment with the USAID operating unit managing the evaluation or the implementing organization(s) whose project(s) are being evaluated. 5. Current or previous work experience with an organization that may be seen as an industry competitor with the implementing organization(s) whose project(s) are being evaluated. 6. Preconceived ideas toward individuals, groups, organizations, or objectives of the particular projects and organizations being evaluated that could bias the evaluation. I certify (1) that I have completed this disclosure form fully and to the best of my ability and (2) that I will update this disclosure form promptly if relevant circumstances change. If I gain access to proprietary information of other companies, then I agree to protect their information from unauthorized use or disclosure for as long as it remains proprietary and refrain from using the information for any purpose other than that for which it was furnished. Signature Date 05 March 2013 5 ANNEX VIII: USAID/JHU HIV COMMUNICATION PROGRAMME STRATEGIC PARTNERS Strategic Partners: 1. SANAC 2. Department of Health 3. Provincial Governments 4. JHU--‐CCP Broadcast Partners 5. ABC Ulwazi (Radio)–Community Radio 6. SABC Education (TV and Radio) 7. E--‐TV (Television) 8. Mediology (Media Planning) Research Partners 10. Health and Development Africa 11. Centre for AIDS, Development and Research (CADRE) Media Advocacy Partners 12. Marcus Brewster Publicity (Media Advocacy) 13. Health--‐E (Media Advocacy) Creative Partners 14. JoePublic (Creative Agency) 15. Matchboxology (Creative Agency) 16. Curious/ Quizzical Pictures (Creative Production Company) 17. Paprika Communications (Print Publications) Capacity building Partners 18. Community MediaTrust (Siyayinqoba–Beat It–Training of Community Health Care Workers) 19. Sonke Gender Justice – Training for Men’s Sector on Brothers for Life 20. Wits HIV and the Media Project 21. UKZN – Centre for Cultural and Media Studies Community Outreach Partners working with youth, women, men, traditional structures 22. Lesedi Lechabile – Free State, Lejwelephutswa 23. Mothusimpilo, Guateng – West Rand 24. Turntable Trust, KZN, Sinonke 25. The Valley Trust (KZN, Ethekwini) 26. NRASD (Eastern Cape, KZN, Western Cape) 27. One Voice (KZN, EC and WC) – Project phased out 28. Lighthouse Foundation – Project phased out 29. Footballers for Life – Project phased out 30. Mindset Health – Project phased out 6 This report was commissioned by U.S. Agency for International Development 1300 Pennsylvania Avenue, NW Washington, DC 20523 7